From njs at pobox.com Tue Sep 1 02:45:26 2015 From: njs at pobox.com (Nathaniel Smith) Date: Mon, 31 Aug 2015 23:45:26 -0700 Subject: [Numpy-discussion] np.sign and object comparisons In-Reply-To: <20150831193106.6504744f@fsol> References: <1441009395.1716.14.camel@sipsolutions.net> <20150831193106.6504744f@fsol> Message-ID: On Mon, Aug 31, 2015 at 10:31 AM, Antoine Pitrou wrote: > On Mon, 31 Aug 2015 10:23:10 -0700 > Stephan Hoyer wrote: >> >> My inclination is that return NaN would be the appropriate choice. It's >> certainly consistent with the behavior for float dtypes -- my expectation >> for object dtype behavior is that it works exactly like applying the >> np.sign ufunc to each element of the array individually. >> >> On the other hand, I suppose there are other ways in which an object can >> fail all those comparisons (e.g., NaT?), so I suppose we could return None. > > Currently: > >>>> np.sign(np.timedelta64('nat')) > numpy.timedelta64(-1) > > ... probably because NaT is -2**63 under the hood. But in this case > returning NaT would sound better. I think this is going through the np.sign timedelta64 loop, and thus is an unrelated issue? It does look like a bug though. -n -- Nathaniel J. Smith -- http://vorpus.org From njs at pobox.com Tue Sep 1 02:49:50 2015 From: njs at pobox.com (Nathaniel Smith) Date: Mon, 31 Aug 2015 23:49:50 -0700 Subject: [Numpy-discussion] np.sign and object comparisons In-Reply-To: References: Message-ID: On Sun, Aug 30, 2015 at 9:09 PM, Jaime Fern?ndez del R?o < jaime.frio at gmail.com> wrote: > > There are three ways of fixing this that I see: > > 1. Arbitrarily choose a value to set the return to. This is equivalent > to choosing a default return for `cmp` for comparisons. This preserves > behavior, but feels wrong. > 2. Similarly to how np.sign of a floating point array with nans > returns nan for those values, return e,g, None for these cases. This > is my preferred option. > 3. Raise an error, along the lines of the TypeError: unorderable types > that 3.x produces for some comparisons. > > Having read the other replies so far -- given that no-one seems to have any clear intuition or use cases, I guess I find option 3 somewhat tempting... it keeps our options open until someone who actually cares comes along with a use case to hone our intuition on, and is very safe in the mean time. (This was noticed in the course of routine code cleanups, right, not an external bug report? For all we know right now, no actual user has ever even tried to apply np.sign to an object array?) -n -- Nathaniel J. Smith -- http://vorpus.org -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Tue Sep 1 03:16:17 2015 From: njs at pobox.com (Nathaniel Smith) Date: Tue, 1 Sep 2015 00:16:17 -0700 Subject: [Numpy-discussion] Cythonizing some of NumPy In-Reply-To: References: Message-ID: On Sun, Aug 30, 2015 at 2:44 PM, David Cournapeau wrote: > Hi there, > > Reading Nathaniel summary from the numpy dev meeting, it looks like there is > a consensus on using cython in numpy for the Python-C interfaces. > > This has been on my radar for a long time: that was one of my rationale for > splitting multiarray into multiple "independent" .c files half a decade ago. > I took the opportunity of EuroScipy sprints to look back into this, but > before looking more into it, I'd like to make sure I am not going astray: > > 1. The transition has to be gradual Yes, definitely. > 2. The obvious way I can think of allowing cython in multiarray is modifying > multiarray such as cython "owns" the PyMODINIT_FUNC and the module > PyModuleDef table. The seems like a plausible place to start. In the longer run, I think we'll need to figure out a strategy to have source code divided over multiple .pyx files (for the same reason we want multiple .c files -- it'll just be impossible to work with otherwise). And this will be difficult for annoying technical reasons, since we definitely do *not* want to increase the API surface exposed by multiarray.so, so we will need to compile these multiple .pyx and .c files into a single module, and have them talk to each other via internal interfaces. But Cython is currently very insistent that every .pyx file should be its own extension module, and the interface between different files should be via public APIs. I spent some time poking at this, and I think it's possible but will take a few kluges at least initially. IIRC the tricky points I noticed are: - For everything except the top-level .pyx file, we'd need to call the generated module initialization functions "by hand", and have a bit of utility code to let us access the symbol tables for the resulting modules - We'd need some preprocessor hack (or something?) to prevent the non-main module initialization functions from being exposed at the .so level (like 'cdef extern from "foo.h"', 'foo.h' re#defines PyMODINIT_FUNC to remove the visibility declaration) - By default 'cdef' functions are name-mangled, which is annoying if you want to be able to do direct C calls between different .pyx and .c files. You can fix this by adding a 'public' declaration to your cdef function. But 'public' also adds dllexport stuff which would need to be hacked out as per above. I think the best strategy for this is to do whatever horrible things are necessary to get an initial version working (on a branch, of course), and then once that's done assess what changes we want to ask the cython folks for to let us eliminate the gross parts. (Insisting on compiling everything into the same .so will probably also help at some point in avoiding Cython-Related Binary Size Blowup Syndrome (CRBSBS), because the masses of boilerplate could in principle be shared between the different files. I think some modern linkers are even clever enough to eliminate this kind of duplicate code automatically, since C++ suffers from a similar problem.) > 3. We start using cython for the parts that are mostly menial refcount work. > Things like functions in calculation.c are obvious candidates. > > Step 2 should not be disruptive, and does not look like a lot of work: there > are < 60 methods in the table, and most of them should be fairly > straightforward to cythonize. At worse, we could just keep them as is > outside cython and just "export" them in cython. > > Does that sound like an acceptable plan ? > > If so, I will start working on a PR to work on 2. Makes sense to me! -n -- Nathaniel J. Smith -- http://vorpus.org From njs at pobox.com Tue Sep 1 04:20:32 2015 From: njs at pobox.com (Nathaniel Smith) Date: Tue, 1 Sep 2015 01:20:32 -0700 Subject: [Numpy-discussion] Notes from the numpy dev meeting at scipy 2015 In-Reply-To: References: Message-ID: On Sun, Aug 30, 2015 at 9:12 PM, Marten van Kerkwijk wrote: > Hi Nathaniel, others, > > I read the discussion of plans with interest. One item that struck me is > that while there are great plans to have a proper extensible and presumably > subclassable dtype, it is discouraged to subclass ndarray itself (rather, it > is encouraged to use a broader array interface). From my experience with > astropy in both Quantity (an ndarray subclass), Time (a separate class > containing high precision times using two ndarray float64), and Table > (initially holding structured arrays, but now sets of Columns, which > themselves are ndarray subclasses), I'm not convinced the broader, new > containers approach is that much preferable. Rather, it leads to a lot of > boiler-plate code to reimplement things ndarray does already (since one is > effectively just calling the methods on the underlying arrays). > > I also think the idea that a dtype becomes something that also contains a > unit is a bit odd. Shouldn't dtype just be about how data is stored? Why > include meta-data such as units? > > Instead, I think a quantity is most logically seen as numbers with a unit, > just like masked arrays are numbers with masks, and variables numbers with > uncertainties. Each of these cases adds extra information in different > forms, and all are quite easily thought of as subclasses of ndarray where > all operations do the normal operation, plus some extra work to keep the > extra information up to date. The intuition behind the array/dtype split is that an array is just a container: it knows how to shuffle bytes around, be reshaped, indexed, etc., but it knows nothing about the meaning of the items it holds -- as far as it's concerned, each entry is just an opaque binary blobs. If it wants to actually do anything with these blobs, it has to ask the dtype for help. The dtype, OTOH, knows how to interpret these blobs, and (in cooperation with ufuncs) to perform operations on them, but it doesn't need to know how they're stored, or about slicing or anything like that -- all that's the container's job. Think about it this way: does it make sense to have a sparse array of numbers-with-units? how about a blosc-style compressed array of numbers-with-units? If yes, then numbers-with-units are a special kind of dtype, not a special kind of array. Another way of getting this intuition: if I have 8 bytes, that could be an int64, or it could be a float64. Which one it is doesn't affect how it's stored at all -- either way it's stored as a chunk of 8 arbitrary bytes. What it affects is how we *interpret* these bytes -- e.g. there is one function called "int64 addition" which takes two 8 byte chunks and returns a new 8 byte chunk as the result, and a second function called "float64 addition" which takes those same two 8 byte chunks and returns a different one. The dtype tells you which of these operations should be used for a particular array. What's special about a float64-with-units? Well, it's 8 bytes, but the addition operation is different from regular float64 addition: it has to do some extra checks and possibly unit conversions. This is exactly what the ufunc dtype dispatch and casting system is there for. This also solves your problem with having to write lots of boilerplate code, b/c if this is a dtype then it means you can just use the actual ndarray class directly without subclassing or anything :-). > Anyway, my suggestion would be to *encourage* rather than discourage ndarray > subclassing, and help this by making ndarray (even) better. So, we very much need robust support for objects-that-quack-like-an-array that are *not* ndarrays, because ndarray subclasses are forced to use ndarray-style strided in-memory storage, and there's huge demand for objects that expose an array-like interface but that use a different storage strategy underneath: sparse arrays, compressed arrays (like blosc), out-of-core arrays, computed-on-demand arrays (like dask), distributed arrays, etc. etc. And once we have solid support for duck-arrays and for user-defined dtypes (as discussed above), then those two things remove a huge amount of the motivation for subclassing ndarray. At the same time, ndarray subclassing is... nearly unmaintainable, AFAICT. The problem with subclassing is that you're basically taking some interface, making a copy of it, and then monkeypatching the copy. As you would expect, this is intrinsically very fragile, because it breaks abstraction barriers. Suddenly things that used to be implementation details -- like which methods are implemented in terms of which other methods -- become part of the public API. And there's never been any coherent, documentable theory of how ndarray subclassing is *supposed* to work, so in practice it's just a bunch of ad hoc hooks designed around the needs of np.matrix and np.ma. We get a regular stream of bug reports asking us to tweak things one way or another, and it feels like trying to cover the floor with a too-small carpet -- we end up with an API that covers the need of whoever complained most recently. And there's the thing where as far as we can tell, 99% of the people who have ever sat down and tried to subclass ndarray ended up regretting it :-). Seriously, you are literally the only person who I've ever heard say positive things about the experience, and I can't really see why given how often I see you in the bug tracker complaining about some weird breakage :-). So there aren't many people motivated to work on it... If someone has a good plan for how to fix all this then by all means, speak up :-). But IMO it's better to write some boilerplate that you can control than to import + monkeypatch, even if the latter seems easier in the short run. And there's a lot we can do to reduce that boilerplate -- e.g. when you want to implement a new sequence type in Python you can write your __getitem__ and __len__ and then use collections.abc.Sequence to fill in the rest of the interface; we've been talking about adding something similar for arrays as part of the __numpy_ufunc__ work. -n -- Nathaniel J. Smith -- http://vorpus.org From jdmc80 at hotmail.com Tue Sep 1 06:38:24 2015 From: jdmc80 at hotmail.com (Joseph Codadeen) Date: Tue, 1 Sep 2015 10:38:24 +0000 Subject: [Numpy-discussion] Numpy FFT.FFT slow with certain samples In-Reply-To: References: , <55E0A861.6020001@stsci.edu>, , <87y4gvi6av.fsf@berkeley.edu> <55E0ECC1.8010705@gmail.com>, <55E0F8E9.4020607@gmail.com>, Message-ID: Hi, I cannot see how the following would work when it is np.fft.fft() that takes a long time based on the length of data. In my case my data is non-periodic.> from numpy.fft import fft> from numpy.random import rand> from math import log, ceil> seq_A = rand(2649674)> seq_B = rand(2646070)> fft_A = fft(seq_A) #Long> fft_B = fft(seq_B)>zeropadded_fft_A = fft(seq_A, n=2**(ceil(log(len(seq_A),2))+1))>zeropadded_fft_B = fft(seq_B, n=2**(ceil(log(len(seq_B),2))+1))Ideally I need to calculate a favourable length to pad, prior to calling np.fft.fft() as St?fan pointed out by calculating the factors. In [6]: from sympy import factorint In [7]: max(factorint(2646070)) Out[7]: 367 In [8]: max(factorint(2649674)) Out[8]: 1324837 I will try adding some code to calculate the next power of two above my array length as you suggest;> And while you zero-pad, you can zero-pad to a sequence that is a power of two, thus preventing awkward factorizations.Does numpy have an easy way to do this, i.e. for a given number, find the next highest number (within a range) that could be factored into small, prime numbers as Phil explained? It would help if it gave a list, prioritised by number of factors.Thanks,Joseph Date: Fri, 28 Aug 2015 17:26:32 -0700 From: stefanv at berkeley.edu To: numpy-discussion at scipy.org Subject: Re: [Numpy-discussion] Numpy FFT.FFT slow with certain samples On Aug 28, 2015 5:17 PM, "Pierre-Andre Noel" wrote: > > I had in mind the use of FFT to do convolutions ( > https://en.wikipedia.org/wiki/Convolution_theorem ). If you do not > zero-pad properly, then the end of the signal may "bleed" on the > beginning, and vice versa. Ah, gotcha! All these things should also be handled nicely in scipy.signal.fftconvolve. St?fan _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion at scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion -------------- next part -------------- An HTML attachment was scrubbed... URL: From cournape at gmail.com Tue Sep 1 07:56:21 2015 From: cournape at gmail.com (David Cournapeau) Date: Tue, 1 Sep 2015 12:56:21 +0100 Subject: [Numpy-discussion] Cythonizing some of NumPy In-Reply-To: References: Message-ID: On Tue, Sep 1, 2015 at 8:16 AM, Nathaniel Smith wrote: > On Sun, Aug 30, 2015 at 2:44 PM, David Cournapeau > wrote: > > Hi there, > > > > Reading Nathaniel summary from the numpy dev meeting, it looks like > there is > > a consensus on using cython in numpy for the Python-C interfaces. > > > > This has been on my radar for a long time: that was one of my rationale > for > > splitting multiarray into multiple "independent" .c files half a decade > ago. > > I took the opportunity of EuroScipy sprints to look back into this, but > > before looking more into it, I'd like to make sure I am not going astray: > > > > 1. The transition has to be gradual > > Yes, definitely. > > > 2. The obvious way I can think of allowing cython in multiarray is > modifying > > multiarray such as cython "owns" the PyMODINIT_FUNC and the module > > PyModuleDef table. > > The seems like a plausible place to start. > > In the longer run, I think we'll need to figure out a strategy to have > source code divided over multiple .pyx files (for the same reason we > want multiple .c files -- it'll just be impossible to work with > otherwise). And this will be difficult for annoying technical reasons, > since we definitely do *not* want to increase the API surface exposed > by multiarray.so, so we will need to compile these multiple .pyx and > .c files into a single module, and have them talk to each other via > internal interfaces. But Cython is currently very insistent that every > .pyx file should be its own extension module, and the interface > between different files should be via public APIs. > > I spent some time poking at this, and I think it's possible but will > take a few kluges at least initially. IIRC the tricky points I noticed > are: > > - For everything except the top-level .pyx file, we'd need to call the > generated module initialization functions "by hand", and have a bit of > utility code to let us access the symbol tables for the resulting > modules > > - We'd need some preprocessor hack (or something?) to prevent the > non-main module initialization functions from being exposed at the .so > level (like 'cdef extern from "foo.h"', 'foo.h' re#defines > PyMODINIT_FUNC to remove the visibility declaration) > > - By default 'cdef' functions are name-mangled, which is annoying if > you want to be able to do direct C calls between different .pyx and .c > files. You can fix this by adding a 'public' declaration to your cdef > function. But 'public' also adds dllexport stuff which would need to > be hacked out as per above. > > I think the best strategy for this is to do whatever horrible things > are necessary to get an initial version working (on a branch, of > course), and then once that's done assess what changes we want to ask > the cython folks for to let us eliminate the gross parts. > Agreed. Regarding multiple cython .pyx and symbol pollution, I think it would be fine to have an internal API with the required prefix (say `_npy_cpy_`) in a core library, and control the exported symbols at the .so level. This is how many large libraries work in practice (e.g. MKL), and is a model well understood by library users. I will start the cythonize process without caring about any of that though: one large .pyx file, and everything build together by putting everything in one .so. That will avoid having to fight both cython and distutils at the same time :) David > > (Insisting on compiling everything into the same .so will probably > also help at some point in avoiding Cython-Related Binary Size Blowup > Syndrome (CRBSBS), because the masses of boilerplate could in > principle be shared between the different files. I think some modern > linkers are even clever enough to eliminate this kind of duplicate > code automatically, since C++ suffers from a similar problem.) > > > 3. We start using cython for the parts that are mostly menial refcount > work. > > Things like functions in calculation.c are obvious candidates. > > > > Step 2 should not be disruptive, and does not look like a lot of work: > there > > are < 60 methods in the table, and most of them should be fairly > > straightforward to cythonize. At worse, we could just keep them as is > > outside cython and just "export" them in cython. > > > > Does that sound like an acceptable plan ? > > > > If so, I will start working on a PR to work on 2. > > Makes sense to me! > > -n > > -- > Nathaniel J. Smith -- http://vorpus.org > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From oscar.j.benjamin at gmail.com Tue Sep 1 11:14:41 2015 From: oscar.j.benjamin at gmail.com (Oscar Benjamin) Date: Tue, 1 Sep 2015 16:14:41 +0100 Subject: [Numpy-discussion] Numpy FFT.FFT slow with certain samples In-Reply-To: References: <55E0A861.6020001@stsci.edu> <87y4gvi6av.fsf@berkeley.edu> <55E0ECC1.8010705@gmail.com> <55E0F8E9.4020607@gmail.com> Message-ID: On 1 September 2015 at 11:38, Joseph Codadeen wrote: > >> And while you zero-pad, you can zero-pad to a sequence that is a power of >> two, thus preventing awkward factorizations. > > Does numpy have an easy way to do this, i.e. for a given number, find the > next highest number (within a range) that could be factored into small, > prime numbers as Phil explained? It would help if it gave a list, > prioritised by number of factors. Just use the next power of 2. Pure powers of 2 are the most efficient for FFT algorithms so it potentially works out better than finding a smaller but similarly composite size to pad to. Finding the next power of 2 is easy to code and never a bad choice. To avoid the problems mentioned about zero-padding distorting the FFT you can use the Bluestein transform as below. This pads up to a power of two (greater than 2N-1) but does it in a special way that ensures that it is still calculating the exact (up to floating point error) same DFT: from numpy import array, exp, pi, arange, concatenate from numpy.fft import fft, ifft def ceilpow2(N): ''' >>> ceilpow2(15) 16 >>> ceilpow2(16) 16 ''' p = 1 while p < N: p *= 2 return p def fftbs(x): ''' >>> data = [1, 2, 5, 2, 5, 2, 3] >>> from numpy.fft import fft >>> from numpy import allclose >>> from numpy.random import randn >>> for n in range(1, 1000): ... data = randn(n) ... assert allclose(fft(data), fftbs(data)) ''' N = len(x) x = array(x) n = arange(N) b = exp((1j*pi*n**2)/N) a = x * b.conjugate() M = ceilpow2(N) * 2 A = concatenate((a, [0] * (M - N))) B = concatenate((b, [0] * (M - 2*N + 1), b[:0:-1])) C = ifft(fft(A) * fft(B)) c = C[:N] return b.conjugate() * c if __name__ == "__main__": import doctest doctest.testmod() -- Oscar From jdmc80 at hotmail.com Tue Sep 1 13:13:30 2015 From: jdmc80 at hotmail.com (Joseph Codadeen) Date: Tue, 1 Sep 2015 17:13:30 +0000 Subject: [Numpy-discussion] Numpy FFT.FFT slow with certain samples In-Reply-To: References: , <55E0A861.6020001@stsci.edu>, , <87y4gvi6av.fsf@berkeley.edu>, <55E0ECC1.8010705@gmail.com>, <55E0F8E9.4020607@gmail.com>, , , Message-ID: Ahhhhh, looking back I see what Pierre-Andr? did, the penny drops. Sorry I miss-read.I'm only interested in this part;>zeropadded_fft_A = fft(seq_A, n=2**(ceil(log(len(seq_A),2))+1)) >zeropadded_fft_B = fft(seq_B, n=2**(ceil(log(len(seq_B),2))+1))>You could remove the "+1" above to get faster results, but then that may >lead to unwanted frequencies (coming from the fact that fft assumes >periodic signals, read online about zero-padding). I did it in the following way and it seems to work. I've seen an increase in performance for sure. My Python sucks by the way :)def pad_to_power_of_two(audio): ''' Calculates next power of two and pads the audio sample array with zeros ''' # RIFF spec specifies: ckSize A 32-bit unsigned value identifying the size of ckData. max_shift = 32 shift = 1 power = len(audio) - 1 print "{0:b}".format(power) # Test if a power of two. Turn off the right-most bit (x & (x - 1)) and zero test while ((power + 1) & power) and (shift <= max_shift): power |= power >> shift shift *= 2 print "new power: " + "{0:b}".format(power) power = power + 1 print "{0:b}".format(power) ext = np.array([0x00] * (power - len(audio))) new_array = np.concatenate((audio, ext), axis=0) print "Next power of 2 greater than actual array length is " + str(power) print "Extending current array length (" + str(len(audio)) + ") by " + str(power - len(audio)) + " null bytes" print "New array length is " + str(len(new_array)) return new_array I will try using the suggested methods and see if I can boost the performance further.Thanks. > From: oscar.j.benjamin at gmail.com > Date: Tue, 1 Sep 2015 16:14:41 +0100 > To: numpy-discussion at scipy.org > Subject: Re: [Numpy-discussion] Numpy FFT.FFT slow with certain samples > > On 1 September 2015 at 11:38, Joseph Codadeen wrote: > > > >> And while you zero-pad, you can zero-pad to a sequence that is a power of > >> two, thus preventing awkward factorizations. > > > > Does numpy have an easy way to do this, i.e. for a given number, find the > > next highest number (within a range) that could be factored into small, > > prime numbers as Phil explained? It would help if it gave a list, > > prioritised by number of factors. > > Just use the next power of 2. Pure powers of 2 are the most efficient > for FFT algorithms so it potentially works out better than finding a > smaller but similarly composite size to pad to. Finding the next power > of 2 is easy to code and never a bad choice. > > To avoid the problems mentioned about zero-padding distorting the FFT > you can use the Bluestein transform as below. This pads up to a power > of two (greater than 2N-1) but does it in a special way that ensures > that it is still calculating the exact (up to floating point error) > same DFT: > > from numpy import array, exp, pi, arange, concatenate > from numpy.fft import fft, ifft > > def ceilpow2(N): > ''' > >>> ceilpow2(15) > 16 > >>> ceilpow2(16) > 16 > ''' > p = 1 > while p < N: > p *= 2 > return p > > def fftbs(x): > ''' > >>> data = [1, 2, 5, 2, 5, 2, 3] > >>> from numpy.fft import fft > >>> from numpy import allclose > >>> from numpy.random import randn > >>> for n in range(1, 1000): > ... data = randn(n) > ... assert allclose(fft(data), fftbs(data)) > ''' > N = len(x) > x = array(x) > > n = arange(N) > b = exp((1j*pi*n**2)/N) > a = x * b.conjugate() > > M = ceilpow2(N) * 2 > A = concatenate((a, [0] * (M - N))) > B = concatenate((b, [0] * (M - 2*N + 1), b[:0:-1])) > C = ifft(fft(A) * fft(B)) > c = C[:N] > return b.conjugate() * c > > if __name__ == "__main__": > import doctest > doctest.testmod() > > > -- > Oscar > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion -------------- next part -------------- An HTML attachment was scrubbed... URL: From hodge at stsci.edu Tue Sep 1 13:42:49 2015 From: hodge at stsci.edu (Phil Hodge) Date: Tue, 1 Sep 2015 13:42:49 -0400 Subject: [Numpy-discussion] Numpy FFT.FFT slow with certain samples In-Reply-To: References: <55E0A861.6020001@stsci.edu> <87y4gvi6av.fsf@berkeley.edu> <55E0ECC1.8010705@gmail.com> <55E0F8E9.4020607@gmail.com> Message-ID: <55E5E399.3020703@stsci.edu> On 09/01/2015 11:14 AM, Oscar Benjamin wrote: > Just use the next power of 2. Pure powers of 2 are the most efficient > for FFT algorithms so it potentially works out better than finding a > smaller but similarly composite size to pad to. Finding the next power > of 2 is easy to code and never a bad choice. It would be better to pad to a power of three or five, or some other product of small primes, if the pad length would be significantly less than padding to a power of two. The obvious problem with padding is that it's different from the real data, so its presence will affect the result. At the least, it would be good to pad with the mean value of the original data, or to pad with zero after subtracting the mean from the original data. Phil From oscar.j.benjamin at gmail.com Tue Sep 1 14:06:55 2015 From: oscar.j.benjamin at gmail.com (Oscar Benjamin) Date: Tue, 01 Sep 2015 18:06:55 +0000 Subject: [Numpy-discussion] Numpy FFT.FFT slow with certain samples In-Reply-To: <55E5E399.3020703@stsci.edu> References: <55E0A861.6020001@stsci.edu> <87y4gvi6av.fsf@berkeley.edu> <55E0ECC1.8010705@gmail.com> <55E0F8E9.4020607@gmail.com> <55E5E399.3020703@stsci.edu> Message-ID: On Tue, 1 Sep 2015 18:43 Phil Hodge wrote: On 09/01/2015 11:14 AM, Oscar Benjamin wrote: > Just use the next power of 2. Pure powers of 2 are the most efficient > for FFT algorithms so it potentially works out better than finding a > smaller but similarly composite size to pad to. Finding the next power > of 2 is easy to code and never a bad choice. It would be better to pad to a power of three or five, or some other product of small primes, if the pad length would be significantly less than padding to a power of two. The obvious problem with padding is that it's different from the real data, so its presence will affect the result. At the least, it would be good to pad with the mean value of the original data, or to pad with zero after subtracting the mean from the original data. I meant performance wise it's not a bad choice. If you're concerned about distortion then use the Bluestein algorithm as I showed. -- Oscar -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Tue Sep 1 19:16:03 2015 From: charlesr.harris at gmail.com (Charles R Harris) Date: Tue, 1 Sep 2015 17:16:03 -0600 Subject: [Numpy-discussion] Numpy FFT.FFT slow with certain samples In-Reply-To: References: <55E0A861.6020001@stsci.edu> <87y4gvi6av.fsf@berkeley.edu> <55E0ECC1.8010705@gmail.com> <55E0F8E9.4020607@gmail.com> <55E5E399.3020703@stsci.edu> Message-ID: On Tue, Sep 1, 2015 at 12:06 PM, Oscar Benjamin wrote: > > On Tue, 1 Sep 2015 18:43 Phil Hodge wrote: > > On 09/01/2015 11:14 AM, Oscar Benjamin wrote: > > Just use the next power of 2. Pure powers of 2 are the most efficient > > for FFT algorithms so it potentially works out better than finding a > > smaller but similarly composite size to pad to. Finding the next power > > of 2 is easy to code and never a bad choice. > > It would be better to pad to a power of three or five, or some other > product of small primes, if the pad length would be significantly less > than padding to a power of two. The obvious problem with padding is > that it's different from the real data, so its presence will affect the > result. At the least, it would be good to pad with the mean value of > the original data, or to pad with zero after subtracting the mean from > the original data. > > > > I meant performance wise it's not a bad choice. If you're concerned about > distortion then use the Bluestein algorithm as I showed. > I don't see the problem with padding. After all, the spectrum is an estimate and the effect of the padding, as well as the use of windowing is well understood. However, I would recommend subtracting the average before padding, as otherwise the DC frequency is likely to be of large amplitude and its side lobes will mask the lower frequencies. > -- > Oscar > > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.barker at noaa.gov Tue Sep 1 20:28:56 2015 From: chris.barker at noaa.gov (Chris Barker) Date: Tue, 1 Sep 2015 17:28:56 -0700 Subject: [Numpy-discussion] UTC-based datetime64 In-Reply-To: References: Message-ID: > Googling for a way to print UTC out of the box, the best thing I could > find is: > > In [40]: [str(i.item()) for i in np.array([t], dtype="datetime64[s]")] > Out[40]: ['2015-08-26 11:52:10'] > > Now, is there a better way to specify that I want the datetimes printed > always in UTC? > maybe, but it's a kludge no matter how you do it :-( It's been in the list for a while to fix this, but hasn't happened yet. Partly due to me not finishing writing out the notes from SciPy 2014 where we discussed a way forward on this. Sorry, -Chris > Thanks, > -- > Francesc Alted > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From jbenson at nofs.navy.mil Tue Sep 1 21:00:20 2015 From: jbenson at nofs.navy.mil (Jim Benson) Date: Tue, 1 Sep 2015 18:00:20 -0700 Subject: [Numpy-discussion] UTC-based datetime64 In-Reply-To: References: Message-ID: <201509020100.t820xVY0066364@charon.nofs.navy.mil> On Tue, Sep 01, 2015 at 05:28:56PM -0700, Chris Barker wrote: > > Googling for a way to print UTC out of the box, the best thing I could > > find is: > > > > In [40]: [str(i.item()) for i in np.array([t], dtype="datetime64[s]")] > > Out[40]: ['2015-08-26 11:52:10'] > > > > Now, is there a better way to specify that I want the datetimes printed > > always in UTC? > > > > maybe, but it's a kludge no matter how you do it :-( > > It's been in the list for a while to fix this, but hasn't happened yet. > Partly due to me not finishing writing out the notes from SciPy 2014 where > we discussed a way forward on this. > > Sorry, > > -Chris > Plans for getting times in UTC1 at the ms accuracy? Sorry, a joke, i couldn't resist. Cheers, Jim > > > > > Thanks, > > -- > > Francesc Alted > > > > _______________________________________________ > > NumPy-Discussion mailing list > > NumPy-Discussion at scipy.org > > > > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > > > > > -- > > Christopher Barker, Ph.D. > Oceanographer > > Emergency Response Division > NOAA/NOS/OR&R (206) 526-6959 voice > 7600 Sand Point Way NE (206) 526-6329 fax > Seattle, WA 98115 (206) 526-6317 main reception > > Chris.Barker at noaa.gov > > > > -- > > Christopher Barker, Ph.D. > Oceanographer > > Emergency Response Division > NOAA/NOS/OR&R (206) 526-6959 voice > 7600 Sand Point Way NE (206) 526-6329 fax > Seattle, WA 98115 (206) 526-6317 main reception > > Chris.Barker at noaa.gov > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion From jaime.frio at gmail.com Wed Sep 2 00:26:53 2015 From: jaime.frio at gmail.com (=?UTF-8?Q?Jaime_Fern=C3=A1ndez_del_R=C3=ADo?=) Date: Tue, 1 Sep 2015 21:26:53 -0700 Subject: [Numpy-discussion] np.sign and object comparisons In-Reply-To: References: Message-ID: On Mon, Aug 31, 2015 at 11:49 PM, Nathaniel Smith wrote: > On Sun, Aug 30, 2015 at 9:09 PM, Jaime Fern?ndez del R?o < > jaime.frio at gmail.com> wrote: > >> >> There are three ways of fixing this that I see: >> >> 1. Arbitrarily choose a value to set the return to. This is >> equivalent to choosing a default return for `cmp` for comparisons. This >> preserves behavior, but feels wrong. >> 2. Similarly to how np.sign of a floating point array with nans >> returns nan for those values, return e,g, None for these cases. This >> is my preferred option. >> 3. Raise an error, along the lines of the TypeError: unorderable types >> that 3.x produces for some comparisons. >> >> Having read the other replies so far -- given that no-one seems to have > any clear intuition or use cases, I guess I find option 3 somewhat > tempting... it keeps our options open until someone who actually cares > comes along with a use case to hone our intuition on, and is very safe in > the mean time. > > (This was noticed in the course of routine code cleanups, right, not an > external bug report? For all we know right now, no actual user has ever > even tried to apply np.sign to an object array?) > We do have a user that tried np.sign on an object array, and discovered that our Py3K object comparison was crap: https://github.com/numpy/numpy/issues/6229 No report of anyone trying np.sign on anything other than numbers that we know of, though. I'm starting to think that, given the lack of agreement, I thinking I am going to agree with you that raising an error may be the better option, because it's the least likely to break people's code if we later find we need to change it. Jaime -- (\__/) ( O.o) ( > <) Este es Conejo. Copia a Conejo en tu firma y ay?dale en sus planes de dominaci?n mundial. -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.barker at noaa.gov Wed Sep 2 12:41:47 2015 From: chris.barker at noaa.gov (Chris Barker) Date: Wed, 2 Sep 2015 09:41:47 -0700 Subject: [Numpy-discussion] Comments on governance proposal (was: Notes from the numpy dev meeting at scipy 2015) In-Reply-To: References: <87y4gxllb4.fsf@berkeley.edu> <1440673873.1694.33.camel@sipsolutions.net> <1440754805.11529.109.camel@sipsolutions.net> Message-ID: 1) I very much agree that governance can make or break a project. However, the actual governance approach often ends up making less difference than the people involved. 2) While the FreeBSD and XFree examples do point to some real problems with the "core" model it seems that there are many other projects that are using it quite successfully. 3) at least in the XFree case, maybe the biggest issue was not the core (or the board), but the fact that discussions and decision making process were kept secret (and restriction to CVS, EVEN read-only) -- this does not seem to be inherent to the "core" model at all, and not how nupy will even operate. 4) the biggest issue numpy has faced in years is a lack of people that can/will/do actually contribute code itself. there simply isn't a big group f folks waiting to step up and be the president. And the "leader", president of BDFL, etc, while primarily being a management role, needs to be fully technically competent -- there are very few people qualified at this point, and I suspect none of them want that job. -- in theory, one can manage without the technical competency, but does anyone have a single example of a successful open-source software project run by a good manager that isn't a top technical expert? All the BDFLs I know of are absolutely top of the heap technically. 5) Sadly, at the end of the day, democracy is often a way to make very poor decisions. So -- it seems there is consensus that we need to formalize the governance of numpy. And honestly, I don't think there are any other options that would work at this point with the current community. And the biggest point: while forking may well demonstrate a complete failure of the governance of a project -- it also represents a success of open-source. In fact, when I read Mathew's history of X11 -- it seems the biggest impediment at each stage were the efforts to make it less that truly open-source. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.barker at noaa.gov Wed Sep 2 12:45:48 2015 From: chris.barker at noaa.gov (Chris Barker) Date: Wed, 2 Sep 2015 09:45:48 -0700 Subject: [Numpy-discussion] Numpy helper function for __getitem__? In-Reply-To: References: <1440353282711.d9fa3274@Nodemailer> <1440404602.2051.14.camel@sipsolutions.net> Message-ID: On Sat, Aug 29, 2015 at 12:55 AM, Phil Elson wrote: > Biggus also has such a function: > https://github.com/SciTools/biggus/blob/master/biggus/__init__.py#L2878 > It handles newaxis outside of that function in: > https://github.com/SciTools/biggus/blob/master/biggus/__init__.py#L537. > > Again, it only aims to deal with orthogonal array indexing, not numpy > fancy indexing. > > I'd be surprised if Dask.array didn't have a similar function too. > This all indicates to me that this would be a great thing to have as a stand alone project, or a utility shipped with numpy. It's been said that you really don't want to subclass ndarray, and should rather, wrap and delicate (or duck-type) -- maybe this is a good time to provide utilities to make it easier to do so. -Chris ----- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From allanhaldane at gmail.com Wed Sep 2 17:24:47 2015 From: allanhaldane at gmail.com (Allan Haldane) Date: Wed, 2 Sep 2015 17:24:47 -0400 Subject: [Numpy-discussion] np.sign and object comparisons In-Reply-To: References: Message-ID: <55E7691F.60107@gmail.com> On 08/31/2015 12:09 AM, Jaime Fern?ndez del R?o wrote: > There are three ways of fixing this that I see: > > 1. Arbitrarily choose a value to set the return to. This is equivalent > to choosing a default return for `cmp` for comparisons. This > preserves behavior, but feels wrong. > 2. Similarly to how np.sign of a floating point array with nans returns > nan for those values, return e,g, None for these cases. This is my > preferred option. > 3. Raise an error, along the lines of the TypeError: unorderable types > that 3.x produces for some comparisons. I think np.sign on nan object arrays should raise the error AttributeError: 'float' object has no attribute 'sign' If I've understood correctly, currently object arrays work like this: If a ufunc has an equivalent pure-python func (eg, PyNumber_Add for np.add, PyNumber_Absolute for np.abs, < for np.greater_than) then numpy calls that for objects. Otherwise, if the object defines a method with the same name as the ufunc, numpy calls that method. For example, arccos is a ufunc that has no pure python equivalent, so you get the following behavior >>> a = np.array([-1], dtype='O') >>> np.abs(a) array([1], dtype=object) >>> np.arccos(a) AttributeError: 'int' object has no attribute 'arccos' >>> class MyClass: ... def arccos(self): ... return 1 >>> b = np.array([MyClass()], dtype='O') >>> np.arccos(b) array([1], dtype=object) Now, most comparison operators (eg, greater_than) are treated a little specially in loops.c. For some reason, sign is treated just like the other comparison operators, even through technically there is no pure-python equivalent to sign. I think that because there is no pure-python 'sign', numpy should attempt to call obj.sign, and in most cases this should fail with the error above. See also http://stackoverflow.com/questions/1986152/why-doesnt-python-have-a-sign-function I think the fix for sign is that the 'sign' ufunc in generate_umath.py should look more like the arccos one, and we should get rid of OBJECT_sign in loops.c. I'm not 100% sure about this since I haven't followed all of how generate_umath.py works yet. ------- By the way, based on some comments I saw somewhere (apologies, I forget who by!) I wrote up a vision for how ufuncs could work for objects, here: https://gist.github.com/ahaldane/c3f9bcf1f62d898be7c7 I'm a little unsure the ideas there are a good idea since they might be made obsolete by the big dtype subclassing improvements being discussed in the numpy roadmap thread. Allan From m.h.vankerkwijk at gmail.com Wed Sep 2 20:03:52 2015 From: m.h.vankerkwijk at gmail.com (Marten van Kerkwijk) Date: Wed, 2 Sep 2015 20:03:52 -0400 Subject: [Numpy-discussion] Notes from the numpy dev meeting at scipy 2015 In-Reply-To: References: Message-ID: Hi Nathaniel, Thanks for the detailed reply; it helped a lot to understand how one could, indeed, have dtypes contain units. And if one had not just on-the-fly conversion from int to float as part of an internal loop, but also on-the-fly multiplication, then it would even be remarkably fast. Will be interesting to think this through in more detail. Still think subclassing ndarray is not all *that* bad (MaskedArray is a different story...), and it may still be needed for my other examples, but perhaps masked/uncertainties do work with the collections idea. Anyway, it now makes sense to focus on dtype first. Thanks again, Marten -------------- next part -------------- An HTML attachment was scrubbed... URL: From shoyer at gmail.com Thu Sep 3 16:28:52 2015 From: shoyer at gmail.com (Stephan Hoyer) Date: Thu, 3 Sep 2015 13:28:52 -0700 Subject: [Numpy-discussion] Notes from the numpy dev meeting at scipy 2015 In-Reply-To: References: Message-ID: >From my perspective, a major advantage to dtypes is composability. For example, it's hard to write a library like dask.array (out of core arrays) that can suppose holding any conceivable ndarray subclass (like MaskedArray or quantity), but handling arbitrary dtypes is quite straightforward -- and that dtype information can be directly passed on, without the container library knowing anything about the library that implements the dtype. Stephan -------------- next part -------------- An HTML attachment was scrubbed... URL: From matthew.brett at gmail.com Thu Sep 3 21:33:45 2015 From: matthew.brett at gmail.com (Matthew Brett) Date: Fri, 4 Sep 2015 02:33:45 +0100 Subject: [Numpy-discussion] Comments on governance proposal (was: Notes from the numpy dev meeting at scipy 2015) In-Reply-To: References: <87y4gxllb4.fsf@berkeley.edu> <1440673873.1694.33.camel@sipsolutions.net> <1440754805.11529.109.camel@sipsolutions.net> Message-ID: Hi, On Wed, Sep 2, 2015 at 5:41 PM, Chris Barker wrote: > 1) I very much agree that governance can make or break a project. However, > the actual governance approach often ends up making less difference than the > people involved. > > 2) While the FreeBSD and XFree examples do point to some real problems with > the "core" model it seems that there are many other projects that are using > it quite successfully. > > 3) at least in the XFree case, maybe the biggest issue was not the core (or > the board), but the fact that discussions and decision making process were > kept secret (and restriction to CVS, EVEN read-only) -- this does not seem > to be inherent to the "core" model at all, and not how nupy will even > operate. This was an issue, but not the central one. I would argue that the central problem was that XFree86 had grown up slowly and organically, to find itself at the heart of the Linux ecosystem. At this point the private developers-club aspect of XFree86 became a massive hindrance to the huge pressure from outside to develop new stuff. > 4) the biggest issue numpy has faced in years is a lack of people that > can/will/do actually contribute code itself. there simply isn't a big group > f folks waiting to step up and be the president. And the "leader", president > of BDFL, etc, while primarily being a management role, needs to be fully > technically competent -- there are very few people qualified at this point, > and I suspect none of them want that job. -- in theory, one can manage > without the technical competency, but does anyone have a single example of a > successful open-source software project run by a good manager that isn't a > top technical expert? All the BDFLs I know of are absolutely top of the heap > technically. I can't easily imagine someone without reasonable technical ability wanting to lead the project - although it appears that technical ability is overrated for technical management [1]. > 5) Sadly, at the end of the day, democracy is often a way to make very poor > decisions. I don't think the question here is whether to have democracy or not, but what kind of democracy. In fact the current proposal is a sort of consensus democracy. My interest is whether this really is a good model for numpy. For example, will this model lead to a lack of accountability and coherent direction? Will it make it harder for other projects to interact with numpy and negotiate about numpy direction? > So -- it seems there is consensus that we need to formalize the governance > of numpy. And honestly, I don't think there are any other options that would > work at this point with the current community. It is true, that this discussion needs some commitment. That either has to come from {Chuck, Nathaniel, Ralf} or very strongly from the rest of us, and we don't have either of those conditions, so I think we are going to fall back to the status quo by default. Cheers, Matthew [1] http://www.nytimes.com/2011/03/13/business/13hire.html?_r=0 From josef.pktd at gmail.com Fri Sep 4 10:11:13 2015 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Fri, 4 Sep 2015 10:11:13 -0400 Subject: [Numpy-discussion] meshgrid dtype casting Message-ID: I'm trying to build a meshgrid with small nonnegative integers default is int32 >>> np.meshgrid([0,1,2], [0,1])[0].dtype dtype('int32') If I use uint, then the arrays are upcast to int64 - Why? >>> np.meshgrid(np.array([0,1,2], np.uint), np.array([0,1], np.uint))[0].dtype dtype('int64') broadcast_arrays preserves dtype >>> np.broadcast_arrays(np.array([0,1,2], np.uint)[:,None], np.array([0,1], np.uint)[None, :]) [array([[0, 0], [1, 1], [2, 2]], dtype=uint32), array([[0, 1], [0, 1], [0, 1]], dtype=uint32)] with uint8 >>> np.broadcast_arrays(np.array([0,1,2], np.uint8)[:,None], np.array([0,1], np.uint8)[None, :]) [array([[0, 0], [1, 1], [2, 2]], dtype=uint8), array([[0, 1], [0, 1], [0, 1]], dtype=uint8)] >>> np.meshgrid(np.array([0,1,2], np.uint8), np.array([0,1], np.uint8))[0].dtype dtype('int32') >>> Winpython 64 bit >>> np.__version__ '1.9.2rc1' Josef -------------- next part -------------- An HTML attachment was scrubbed... URL: From matthew.brett at gmail.com Fri Sep 4 16:53:03 2015 From: matthew.brett at gmail.com (Matthew Brett) Date: Fri, 4 Sep 2015 21:53:03 +0100 Subject: [Numpy-discussion] Comments on governance proposal (was: Notes from the numpy dev meeting at scipy 2015) In-Reply-To: References: <87y4gxllb4.fsf@berkeley.edu> <1440673873.1694.33.camel@sipsolutions.net> <1440754805.11529.109.camel@sipsolutions.net> Message-ID: On Fri, Sep 4, 2015 at 2:33 AM, Matthew Brett wrote: > Hi, > > On Wed, Sep 2, 2015 at 5:41 PM, Chris Barker wrote: >> 1) I very much agree that governance can make or break a project. However, >> the actual governance approach often ends up making less difference than the >> people involved. >> >> 2) While the FreeBSD and XFree examples do point to some real problems with >> the "core" model it seems that there are many other projects that are using >> it quite successfully. I was just rereading the complaints about the 'core' structure from high-level NetBSD project leaders: "[the "core" and "board of directors"] teams are dysfunctional because they do not provide leadership: all they do is act reactively to requests from users and/or to resolve internal disputes. In other words: there is no initiative nor vision emerging from these teams (and, for that matter, from anybody)." [1] "There is no high-level direction; if you ask "what about the problems with threads" or "will there be a flash-friendly file system", the best you'll get is "we'd love to have both" -- but no work is done to recruit people to code these things, or encourage existing developers to work on them." [2] I imagine we will have to reconcile ourselves to similar problems, if we adopt the same structures. Cheers, Matthew [1] http://julipedia.meroh.net/2013/06/self-interview-after-leaving-netbsd.html [2] http://mail-index.netbsd.org/netbsd-users/2006/08/30/0016.html From efiring at hawaii.edu Fri Sep 4 17:22:46 2015 From: efiring at hawaii.edu (Eric Firing) Date: Fri, 4 Sep 2015 11:22:46 -1000 Subject: [Numpy-discussion] Comments on governance proposal (was: Notes from the numpy dev meeting at scipy 2015) In-Reply-To: References: <1440754805.11529.109.camel@sipsolutions.net> Message-ID: <55EA0BA6.3030607@hawaii.edu> On 2015/09/04 10:53 AM, Matthew Brett wrote: > On Fri, Sep 4, 2015 at 2:33 AM, Matthew Brett wrote: >> Hi, >> >> On Wed, Sep 2, 2015 at 5:41 PM, Chris Barker wrote: >>> 1) I very much agree that governance can make or break a project. However, >>> the actual governance approach often ends up making less difference than the >>> people involved. >>> >>> 2) While the FreeBSD and XFree examples do point to some real problems with >>> the "core" model it seems that there are many other projects that are using >>> it quite successfully. > > I was just rereading the complaints about the 'core' structure from > high-level NetBSD project leaders: > > "[the "core" and "board of directors"] teams are dysfunctional because > they do not provide leadership: all they do is act reactively to > requests from users and/or to resolve internal disputes. In other > words: there is no initiative nor vision emerging from these teams > (and, for that matter, from anybody)." [1] > > "There is no high-level direction; if you ask "what about the problems > with threads" or "will there be a flash-friendly file system", the > best you'll get is "we'd love to have both" -- but no work is done to > recruit people to code these things, or encourage existing developers > to work on them." [2] This is consistent with Chris's first point. > > I imagine we will have to reconcile ourselves to similar problems, if > we adopt the same structures. Do you have suggestions as to who would make a good numpy president or BDFL and potentially has the time and inclination to do it, or how to identify and recruit such a person? Eric > > Cheers, > > Matthew > > [1] http://julipedia.meroh.net/2013/06/self-interview-after-leaving-netbsd.html > [2] http://mail-index.netbsd.org/netbsd-users/2006/08/30/0016.html > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From matthew.brett at gmail.com Fri Sep 4 17:55:45 2015 From: matthew.brett at gmail.com (Matthew Brett) Date: Fri, 4 Sep 2015 22:55:45 +0100 Subject: [Numpy-discussion] Comments on governance proposal (was: Notes from the numpy dev meeting at scipy 2015) In-Reply-To: <55EA0BA6.3030607@hawaii.edu> References: <1440754805.11529.109.camel@sipsolutions.net> <55EA0BA6.3030607@hawaii.edu> Message-ID: Hi, On Fri, Sep 4, 2015 at 10:22 PM, Eric Firing wrote: > On 2015/09/04 10:53 AM, Matthew Brett wrote: >> On Fri, Sep 4, 2015 at 2:33 AM, Matthew Brett wrote: >>> Hi, >>> >>> On Wed, Sep 2, 2015 at 5:41 PM, Chris Barker wrote: >>>> 1) I very much agree that governance can make or break a project. However, >>>> the actual governance approach often ends up making less difference than the >>>> people involved. >>>> >>>> 2) While the FreeBSD and XFree examples do point to some real problems with >>>> the "core" model it seems that there are many other projects that are using >>>> it quite successfully. >> >> I was just rereading the complaints about the 'core' structure from >> high-level NetBSD project leaders: >> >> "[the "core" and "board of directors"] teams are dysfunctional because >> they do not provide leadership: all they do is act reactively to >> requests from users and/or to resolve internal disputes. In other >> words: there is no initiative nor vision emerging from these teams >> (and, for that matter, from anybody)." [1] >> >> "There is no high-level direction; if you ask "what about the problems >> with threads" or "will there be a flash-friendly file system", the >> best you'll get is "we'd love to have both" -- but no work is done to >> recruit people to code these things, or encourage existing developers >> to work on them." [2] > > > This is consistent with Chris's first point. Do you mean Chris' point that "I very much agree that governance can make or break a project"? Charles Hannum's complaints about NetBSD are very specific in blaming the model rather than the people. I think the XFree86 story supports the same conclusion - that the governance model caused a sense of diffused responsibility that lead to bad decisions and lack of direction. >> I imagine we will have to reconcile ourselves to similar problems, if >> we adopt the same structures. > > Do you have suggestions as to who would make a good numpy president or > BDFL and potentially has the time and inclination to do it, or how to > identify and recruit such a person? That's a good question, and the answer is that in the current situation (zero interest in this discussion from the three current members of the numpy leadership team) - no reasonable person would be interested in that job. That's the situation we're in, and so we have to accept that nothing is going to change, with the consequences that implies. If the situation were different, and we had the interest or commitment to explore this problem, then I guess we could discuss other options including the one I suggested further up the thread. Cheers, Matthew From josef.pktd at gmail.com Fri Sep 4 19:04:38 2015 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Fri, 4 Sep 2015 19:04:38 -0400 Subject: [Numpy-discussion] Comments on governance proposal (was: Notes from the numpy dev meeting at scipy 2015) In-Reply-To: References: <1440754805.11529.109.camel@sipsolutions.net> <55EA0BA6.3030607@hawaii.edu> Message-ID: On Fri, Sep 4, 2015 at 5:55 PM, Matthew Brett wrote: > Hi, > > On Fri, Sep 4, 2015 at 10:22 PM, Eric Firing wrote: > > On 2015/09/04 10:53 AM, Matthew Brett wrote: > >> On Fri, Sep 4, 2015 at 2:33 AM, Matthew Brett > wrote: > >>> Hi, > >>> > >>> On Wed, Sep 2, 2015 at 5:41 PM, Chris Barker > wrote: > >>>> 1) I very much agree that governance can make or break a project. > However, > >>>> the actual governance approach often ends up making less difference > than the > >>>> people involved. > >>>> > >>>> 2) While the FreeBSD and XFree examples do point to some real > problems with > >>>> the "core" model it seems that there are many other projects that are > using > >>>> it quite successfully. > >> > >> I was just rereading the complaints about the 'core' structure from > >> high-level NetBSD project leaders: > >> > >> "[the "core" and "board of directors"] teams are dysfunctional because > >> they do not provide leadership: all they do is act reactively to > >> requests from users and/or to resolve internal disputes. In other > >> words: there is no initiative nor vision emerging from these teams > >> (and, for that matter, from anybody)." [1] > >> > >> "There is no high-level direction; if you ask "what about the problems > >> with threads" or "will there be a flash-friendly file system", the > >> best you'll get is "we'd love to have both" -- but no work is done to > >> recruit people to code these things, or encourage existing developers > >> to work on them." [2] > > > > > > This is consistent with Chris's first point. > > Do you mean Chris' point that "I very much agree that governance can > make or break a project"? Charles Hannum's complaints about NetBSD > are very specific in blaming the model rather than the people. I > think the XFree86 story supports the same conclusion - that the > governance model caused a sense of diffused responsibility that lead > to bad decisions and lack of direction. > > >> I imagine we will have to reconcile ourselves to similar problems, if > >> we adopt the same structures. > > > > Do you have suggestions as to who would make a good numpy president or > > BDFL and potentially has the time and inclination to do it, or how to > > identify and recruit such a person? > > That's a good question, and the answer is that in the current > situation (zero interest in this discussion from the three current > members of the numpy leadership team) - no reasonable person would be > interested in that job. That's the situation we're in, and so we > have to accept that nothing is going to change, with the consequences > that implies. If the situation were different, and we had the > interest or commitment to explore this problem, then I guess we could > discuss other options including the one I suggested further up the > thread. > " Today, the project is run by a different cabal. This is the result of a coup that took place in 2000-2001, in which The NetBSD Foundation was taken over by a fraudulent change of the board of directors. (Note: It's probably too late for me to pursue any legal remedy for this, unfortunately.) Although "The NetBSD Project" and "The NetBSD Foundation" were intended from the start to be separate entities -- the latter supplying support infrastructure for the former -- this distinction has been actively blurred since, so that the current "board" of TNF has rather tight control over many aspects of TNP. " " The existing NetBSD Foundation must be disbanded, and replaced with an organization that fulfills its original purpose: to merely handle administrative issues, and not to manage day-to-day affairs. " It doesn't sound to me like a developer and community driven governance structure to me. Cheers Josef https://jeb2016.com/?lang=es > > Cheers, > > Matthew > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From matthew.brett at gmail.com Fri Sep 4 19:19:30 2015 From: matthew.brett at gmail.com (Matthew Brett) Date: Sat, 5 Sep 2015 00:19:30 +0100 Subject: [Numpy-discussion] Comments on governance proposal (was: Notes from the numpy dev meeting at scipy 2015) In-Reply-To: References: <1440754805.11529.109.camel@sipsolutions.net> <55EA0BA6.3030607@hawaii.edu> Message-ID: On Sat, Sep 5, 2015 at 12:04 AM, wrote: > > > On Fri, Sep 4, 2015 at 5:55 PM, Matthew Brett > wrote: >> >> Hi, >> >> On Fri, Sep 4, 2015 at 10:22 PM, Eric Firing wrote: >> > On 2015/09/04 10:53 AM, Matthew Brett wrote: >> >> On Fri, Sep 4, 2015 at 2:33 AM, Matthew Brett >> >> wrote: >> >>> Hi, >> >>> >> >>> On Wed, Sep 2, 2015 at 5:41 PM, Chris Barker >> >>> wrote: >> >>>> 1) I very much agree that governance can make or break a project. >> >>>> However, >> >>>> the actual governance approach often ends up making less difference >> >>>> than the >> >>>> people involved. >> >>>> >> >>>> 2) While the FreeBSD and XFree examples do point to some real >> >>>> problems with >> >>>> the "core" model it seems that there are many other projects that are >> >>>> using >> >>>> it quite successfully. >> >> >> >> I was just rereading the complaints about the 'core' structure from >> >> high-level NetBSD project leaders: >> >> >> >> "[the "core" and "board of directors"] teams are dysfunctional because >> >> they do not provide leadership: all they do is act reactively to >> >> requests from users and/or to resolve internal disputes. In other >> >> words: there is no initiative nor vision emerging from these teams >> >> (and, for that matter, from anybody)." [1] >> >> >> >> "There is no high-level direction; if you ask "what about the problems >> >> with threads" or "will there be a flash-friendly file system", the >> >> best you'll get is "we'd love to have both" -- but no work is done to >> >> recruit people to code these things, or encourage existing developers >> >> to work on them." [2] >> > >> > >> > This is consistent with Chris's first point. >> >> Do you mean Chris' point that "I very much agree that governance can >> make or break a project"? Charles Hannum's complaints about NetBSD >> are very specific in blaming the model rather than the people. I >> think the XFree86 story supports the same conclusion - that the >> governance model caused a sense of diffused responsibility that lead >> to bad decisions and lack of direction. >> >> >> I imagine we will have to reconcile ourselves to similar problems, if >> >> we adopt the same structures. >> > >> > Do you have suggestions as to who would make a good numpy president or >> > BDFL and potentially has the time and inclination to do it, or how to >> > identify and recruit such a person? >> >> That's a good question, and the answer is that in the current >> situation (zero interest in this discussion from the three current >> members of the numpy leadership team) - no reasonable person would be >> interested in that job. That's the situation we're in, and so we >> have to accept that nothing is going to change, with the consequences >> that implies. If the situation were different, and we had the >> interest or commitment to explore this problem, then I guess we could >> discuss other options including the one I suggested further up the >> thread. > > > " > > Today, the project is run by a different cabal. This is the result of a > coup that took place in 2000-2001, in which The NetBSD Foundation was > taken over by a fraudulent change of the board of directors. (Note: > It's probably too late for me to pursue any legal remedy for this, > unfortunately.) Although "The NetBSD Project" and "The NetBSD > Foundation" were intended from the start to be separate entities -- the > latter supplying support infrastructure for the former -- this > distinction has been actively blurred since, so that the current "board" > of TNF has rather tight control over many aspects of TNP. > > " > > " > > The existing NetBSD Foundation must be disbanded, and replaced with > an organization that fulfills its original purpose: to merely handle > administrative issues, and not to manage day-to-day affairs. > > " > > > It doesn't sound to me like a developer and community driven governance > structure to me. I think that's a separate issue - the distinction between the 'board' and the 'core'. It would be great if the 'core' concept was fine as long as there is no 'board' but I think that's a hard argument to make. Cheers, Matthew From josef.pktd at gmail.com Fri Sep 4 19:47:56 2015 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Fri, 4 Sep 2015 19:47:56 -0400 Subject: [Numpy-discussion] Comments on governance proposal (was: Notes from the numpy dev meeting at scipy 2015) In-Reply-To: References: <1440754805.11529.109.camel@sipsolutions.net> <55EA0BA6.3030607@hawaii.edu> Message-ID: On Fri, Sep 4, 2015 at 7:19 PM, Matthew Brett wrote: > On Sat, Sep 5, 2015 at 12:04 AM, wrote: > > > > > > On Fri, Sep 4, 2015 at 5:55 PM, Matthew Brett > > wrote: > >> > >> Hi, > >> > >> On Fri, Sep 4, 2015 at 10:22 PM, Eric Firing > wrote: > >> > On 2015/09/04 10:53 AM, Matthew Brett wrote: > >> >> On Fri, Sep 4, 2015 at 2:33 AM, Matthew Brett < > matthew.brett at gmail.com> > >> >> wrote: > >> >>> Hi, > >> >>> > >> >>> On Wed, Sep 2, 2015 at 5:41 PM, Chris Barker > > >> >>> wrote: > >> >>>> 1) I very much agree that governance can make or break a project. > >> >>>> However, > >> >>>> the actual governance approach often ends up making less difference > >> >>>> than the > >> >>>> people involved. > >> >>>> > >> >>>> 2) While the FreeBSD and XFree examples do point to some real > >> >>>> problems with > >> >>>> the "core" model it seems that there are many other projects that > are > >> >>>> using > >> >>>> it quite successfully. > >> >> > >> >> I was just rereading the complaints about the 'core' structure from > >> >> high-level NetBSD project leaders: > >> >> > >> >> "[the "core" and "board of directors"] teams are dysfunctional > because > >> >> they do not provide leadership: all they do is act reactively to > >> >> requests from users and/or to resolve internal disputes. In other > >> >> words: there is no initiative nor vision emerging from these teams > >> >> (and, for that matter, from anybody)." [1] > >> >> > >> >> "There is no high-level direction; if you ask "what about the > problems > >> >> with threads" or "will there be a flash-friendly file system", the > >> >> best you'll get is "we'd love to have both" -- but no work is done to > >> >> recruit people to code these things, or encourage existing developers > >> >> to work on them." [2] > >> > > >> > > >> > This is consistent with Chris's first point. > >> > >> Do you mean Chris' point that "I very much agree that governance can > >> make or break a project"? Charles Hannum's complaints about NetBSD > >> are very specific in blaming the model rather than the people. I > >> think the XFree86 story supports the same conclusion - that the > >> governance model caused a sense of diffused responsibility that lead > >> to bad decisions and lack of direction. > >> > >> >> I imagine we will have to reconcile ourselves to similar problems, if > >> >> we adopt the same structures. > >> > > >> > Do you have suggestions as to who would make a good numpy president or > >> > BDFL and potentially has the time and inclination to do it, or how to > >> > identify and recruit such a person? > >> > >> That's a good question, and the answer is that in the current > >> situation (zero interest in this discussion from the three current > >> members of the numpy leadership team) - no reasonable person would be > >> interested in that job. That's the situation we're in, and so we > >> have to accept that nothing is going to change, with the consequences > >> that implies. If the situation were different, and we had the > >> interest or commitment to explore this problem, then I guess we could > >> discuss other options including the one I suggested further up the > >> thread. > > > > > > " > > > > Today, the project is run by a different cabal. This is the result of a > > coup that took place in 2000-2001, in which The NetBSD Foundation was > > taken over by a fraudulent change of the board of directors. (Note: > > It's probably too late for me to pursue any legal remedy for this, > > unfortunately.) Although "The NetBSD Project" and "The NetBSD > > Foundation" were intended from the start to be separate entities -- the > > latter supplying support infrastructure for the former -- this > > distinction has been actively blurred since, so that the current "board" > > of TNF has rather tight control over many aspects of TNP. > > > > " > > > > " > > > > The existing NetBSD Foundation must be disbanded, and replaced with > > an organization that fulfills its original purpose: to merely handle > > administrative issues, and not to manage day-to-day affairs. > > > > " > > > > > > It doesn't sound to me like a developer and community driven governance > > structure to me. > > I think that's a separate issue - the distinction between the 'board' > and the 'core'. It would be great if the 'core' concept was fine as > long as there is no 'board' but I think that's a hard argument to > make. > there is an "esprit de corps" pronounced "esprit de core" but not an "esprit de board" I trust the core developers, but not ... But maybe I don't understand some definitions " The "core" group must be replaced with people who are actually competent and dedicated enough to review proposals, accept feedback, and make good decisions. " I thought that's what the "core" group is. Josef > > Cheers, > > Matthew > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From matthew.brett at gmail.com Fri Sep 4 22:07:15 2015 From: matthew.brett at gmail.com (Matthew Brett) Date: Sat, 5 Sep 2015 03:07:15 +0100 Subject: [Numpy-discussion] Comments on governance proposal (was: Notes from the numpy dev meeting at scipy 2015) In-Reply-To: References: <1440754805.11529.109.camel@sipsolutions.net> <55EA0BA6.3030607@hawaii.edu> Message-ID: On Sat, Sep 5, 2015 at 12:47 AM, wrote: > > > On Fri, Sep 4, 2015 at 7:19 PM, Matthew Brett > wrote: >> >> On Sat, Sep 5, 2015 at 12:04 AM, wrote: >> > >> > >> > On Fri, Sep 4, 2015 at 5:55 PM, Matthew Brett >> > wrote: >> >> >> >> Hi, >> >> >> >> On Fri, Sep 4, 2015 at 10:22 PM, Eric Firing >> >> wrote: >> >> > On 2015/09/04 10:53 AM, Matthew Brett wrote: >> >> >> On Fri, Sep 4, 2015 at 2:33 AM, Matthew Brett >> >> >> >> >> >> wrote: >> >> >>> Hi, >> >> >>> >> >> >>> On Wed, Sep 2, 2015 at 5:41 PM, Chris Barker >> >> >>> >> >> >>> wrote: >> >> >>>> 1) I very much agree that governance can make or break a project. >> >> >>>> However, >> >> >>>> the actual governance approach often ends up making less >> >> >>>> difference >> >> >>>> than the >> >> >>>> people involved. >> >> >>>> >> >> >>>> 2) While the FreeBSD and XFree examples do point to some real >> >> >>>> problems with >> >> >>>> the "core" model it seems that there are many other projects that >> >> >>>> are >> >> >>>> using >> >> >>>> it quite successfully. >> >> >> >> >> >> I was just rereading the complaints about the 'core' structure from >> >> >> high-level NetBSD project leaders: >> >> >> >> >> >> "[the "core" and "board of directors"] teams are dysfunctional >> >> >> because >> >> >> they do not provide leadership: all they do is act reactively to >> >> >> requests from users and/or to resolve internal disputes. In other >> >> >> words: there is no initiative nor vision emerging from these teams >> >> >> (and, for that matter, from anybody)." [1] >> >> >> >> >> >> "There is no high-level direction; if you ask "what about the >> >> >> problems >> >> >> with threads" or "will there be a flash-friendly file system", the >> >> >> best you'll get is "we'd love to have both" -- but no work is done >> >> >> to >> >> >> recruit people to code these things, or encourage existing >> >> >> developers >> >> >> to work on them." [2] >> >> > >> >> > >> >> > This is consistent with Chris's first point. >> >> >> >> Do you mean Chris' point that "I very much agree that governance can >> >> make or break a project"? Charles Hannum's complaints about NetBSD >> >> are very specific in blaming the model rather than the people. I >> >> think the XFree86 story supports the same conclusion - that the >> >> governance model caused a sense of diffused responsibility that lead >> >> to bad decisions and lack of direction. >> >> >> >> >> I imagine we will have to reconcile ourselves to similar problems, >> >> >> if >> >> >> we adopt the same structures. >> >> > >> >> > Do you have suggestions as to who would make a good numpy president >> >> > or >> >> > BDFL and potentially has the time and inclination to do it, or how to >> >> > identify and recruit such a person? >> >> >> >> That's a good question, and the answer is that in the current >> >> situation (zero interest in this discussion from the three current >> >> members of the numpy leadership team) - no reasonable person would be >> >> interested in that job. That's the situation we're in, and so we >> >> have to accept that nothing is going to change, with the consequences >> >> that implies. If the situation were different, and we had the >> >> interest or commitment to explore this problem, then I guess we could >> >> discuss other options including the one I suggested further up the >> >> thread. >> > >> > >> > " >> > >> > Today, the project is run by a different cabal. This is the result of a >> > coup that took place in 2000-2001, in which The NetBSD Foundation was >> > taken over by a fraudulent change of the board of directors. (Note: >> > It's probably too late for me to pursue any legal remedy for this, >> > unfortunately.) Although "The NetBSD Project" and "The NetBSD >> > Foundation" were intended from the start to be separate entities -- the >> > latter supplying support infrastructure for the former -- this >> > distinction has been actively blurred since, so that the current "board" >> > of TNF has rather tight control over many aspects of TNP. >> > >> > " >> > >> > " >> > >> > The existing NetBSD Foundation must be disbanded, and replaced with >> > an organization that fulfills its original purpose: to merely handle >> > administrative issues, and not to manage day-to-day affairs. >> > >> > " >> > >> > >> > It doesn't sound to me like a developer and community driven governance >> > structure to me. >> >> I think that's a separate issue - the distinction between the 'board' >> and the 'core'. It would be great if the 'core' concept was fine as >> long as there is no 'board' but I think that's a hard argument to >> make. > > > there is an "esprit de corps" pronounced "esprit de core" but not an "esprit > de board" > > I trust the core developers, but not ... > > But maybe I don't understand some definitions > > " > > The "core" group must be replaced with people who are actually > competent and dedicated enough to review proposals, accept feedback, > and make good decisions. > > " > > I thought that's what the "core" group is. To avoid repeating myself any more than I have already - see further up this thread : http://mail.scipy.org/pipermail/numpy-discussion/2015-August/073452.html Matthew From njs at pobox.com Fri Sep 4 23:35:48 2015 From: njs at pobox.com (Nathaniel Smith) Date: Fri, 4 Sep 2015 20:35:48 -0700 Subject: [Numpy-discussion] Comments on governance proposal (was: Notes from the numpy dev meeting at scipy 2015) In-Reply-To: References: <87y4gxllb4.fsf@berkeley.edu> <1440673873.1694.33.camel@sipsolutions.net> Message-ID: On Fri, Aug 28, 2015 at 1:46 AM, Matthew Brett wrote: > Hi, > > On Fri, Aug 28, 2015 at 5:59 AM, Jaime Fern?ndez del R?o > wrote: [...] >> >> Are you trying to prove the point that consensus doesn't work by making it >> impossible to reach a consensus on this? ;-) >> > > Forgive me if I use this joke to see if I can get us any further. > > If this was code, I think this joke would not be funny, because we > wouldn't expect to reach consensus without considering all the > options, and discussing their pros and cons. > > Why would that not be useful in the case of forms of governance? > > One reason might be that the specific form of governance can have no > influence on the long-term health of the project. > > I am convinced that that is wrong - that the form of governance has a > large influence on the long-term health of a project. > > If there is some possibility that this is true, then it seems to me > that we would be foolish not to try and come to some reasoned choice > about the form of governance. That seems fair, and I also think that even if we end up not changing anything then there's still some utility in having the discussion. Certainly one of my secret fears in writing all this up is that it would end up being just some words that only I cared about and everyone else ignored, so seeing other people engaging with it is heartening :-). I'm not feeling inspired this Friday evening to write a full white paper reviewing the universe of possible governance, but I'll summarize a bit what the reasoning was that led us to what we ended up with (at least as I remember it and from my perspective). Some of the sources that were mentioned for reference and positive or negative inspiration include Karl Fogel's chapter on governance, the Debian constitution, the gcc/egcs fork+reverse-fork, glibc/eglibc ditto, subversion, Jupyter/ipython, the Linux kernel, python, matplotlib, the Apache foundation's processes, the gnome foundation, nodejs, ... I'm certainly forgetting some. The basic observation that we started from was that all the successful projects seemed to fit a similar template: (1) basically all actual decision making is done by informal and maximally inclusive consensus-based processes, (2) there are some sort of more formal "backstop" rules that are called into play if informal processes break down (which hopefully never even happens). These projects don't tend to look that similar on paper, because people tend to only write down the "backstop" rules, and those vary a lot. So e.g., for some projects like Python or IPython, the formal rule is "the BDFL decides". But then if you look at what successful BDFLs do with this power, they mostly enforce rule #1 above. (Karl discusses this: http://producingoss.com/en/producingoss.html#benevolent-dictator). E.g., if you watch Guido, I don't think I've ever seen him say "I know people don't like X but we're doing it anyway". He mostly sits back to watch the debate, steps in occasionally to prune off unproductive lines of conversation (e.g. in the PEP 465 debate where he watched for a while, stepped in to tell people to knock it off with the pointless bikeshedding about which character to use, and then went back to watching) or encourage people to keep thinking about something, is always very careful not to actually make any strong statements, etc. Or, e.g., the bit in the draft governance about how the steering committee's secondary job is to make decisions, and their primary job is to prevent things reaching the point where they have to? That's a straight paraphrase of Fernando talking about his BDFL philosophy. Or on the other end of things, you have e.g. Subversion, which had an elaborate defined governance system with different levels of "core-ness", a voting system, etc. -- and they were 6 years into the project before they had their first vote. (The vote was on the crucial technical decision of whether to write function calls like "f ()" or "f()".) Or e.g. the massive and famously fractious Debian has their CTTE, which theoretically is the final arbiter of technical decisions in Debian and has immense powers... but they issue like ~5 rulings a year on average, and have gone years without issuing a ruling at all. So, our thought process was: for our purposes, since we don't have a BDFL, we can't depend on them to be the unwritten mechanism for enforcing rule #1, so first we should write down rule #1. And then we should attach some kind of workable backstop procedure. "Consensus among a core group" seemed like a good bet. It does mean you need to have a formal rule for defining who is in the "core group", because the point of having a formal backstop rule is that it be... formal... but otherwise it's about as simple as you can get: it requires minimal voting logistics (it's trivial to tell whether you have consensus, because if someone disagrees, there's your answer), it's minimally gameable (no temptation to form factions), and because in practice doing anything more complicated than consensus would first require consensus to agree on the more complicated system, so it's a safe procedure to agree on now and try to improve later if we have problems. But we don't really expect to have problems, since our goal is to never even use this system, so spending lots of effort imagining hypothetical scenarios didn't seem like a great use of time. And then we took the Jupyter/IPython governance document, ripped out the BDFL-dependent parts, replaced them with the two ideas above, and called it a day. This is an interesting article I came across today, about how "merely" documenting the status quo provides non-trivial benefits: https://kateheddleston.com/blog/the-null-process (Not that I would necessarily say that the draft I sent is just formalizing the status quo -- IMO the biggest problem with the status quo is that no-one is quite sure what it even is! Which leads to things like that time people panicked because I said we should consider all our options before making a decision [1], or the frustrating and unproductive tendency of contentious technical debates to switch over into becoming ad-hoc one-off debates about decision making processes...) -n [1] http://mail.scipy.org/pipermail/numpy-discussion/2015-April/072629.html -- Nathaniel J. Smith -- http://vorpus.org From njs at pobox.com Sat Sep 5 03:46:21 2015 From: njs at pobox.com (Nathaniel Smith) Date: Sat, 5 Sep 2015 00:46:21 -0700 Subject: [Numpy-discussion] Comments on governance proposal (was: Notes from the numpy dev meeting at scipy 2015) In-Reply-To: References: <87y4gxllb4.fsf@berkeley.edu> <1440673873.1694.33.camel@sipsolutions.net> <1440754805.11529.109.camel@sipsolutions.net> Message-ID: On Fri, Sep 4, 2015 at 1:53 PM, Matthew Brett wrote: > On Fri, Sep 4, 2015 at 2:33 AM, Matthew Brett wrote: >> Hi, >> >> On Wed, Sep 2, 2015 at 5:41 PM, Chris Barker wrote: >>> 1) I very much agree that governance can make or break a project. However, >>> the actual governance approach often ends up making less difference than the >>> people involved. >>> >>> 2) While the FreeBSD and XFree examples do point to some real problems with >>> the "core" model it seems that there are many other projects that are using >>> it quite successfully. > > I was just rereading the complaints about the 'core' structure from > high-level NetBSD project leaders: > > "[the "core" and "board of directors"] teams are dysfunctional because > they do not provide leadership: all they do is act reactively to > requests from users and/or to resolve internal disputes. In other > words: there is no initiative nor vision emerging from these teams > (and, for that matter, from anybody)." [1] > > "There is no high-level direction; if you ask "what about the problems > with threads" or "will there be a flash-friendly file system", the > best you'll get is "we'd love to have both" -- but no work is done to > recruit people to code these things, or encourage existing developers > to work on them." [2] > > I imagine we will have to reconcile ourselves to similar problems, if > we adopt the same structures. I guess I just don't see how you think we can legislate ourselves into having a vision. Like... we'll elect a volunteer to produce a roadmap, and then if they don't we'll fire them (accountability!), leaving us with no volunteer and no roadmap? How will that help? The Jupyter/IPython project probably has one of the the most developed roadmaps in all of open-source, since their grant-funded development model requires them to actually commit to plans ahead of time. And AFAIK the way they accomplish this has nothing to do with Fernando sitting down and having visions; it involves dragging the "core" team in front of a whiteboard 2x a year for a week, and seeing what they can come up with. And I guess we have some tentative evidence from the other thread that this strategy may work for us too... I'm not sure how useful this focus on NetBSD is, given the elephant in the room: the fundamental challenge for NetBSD is that NetBSD has no compelling reason to exist; even its old distinctive feature of being the most portable free OS (which is already an intrinsically niche appeal) has been taken over by Linux. (And Linux, notably, has always explicitly refused to have a roadmap [1][2]... it seems to be doing okay.) Nonetheless, looking at the Hannum [3] post in particular, I'm struck by how much it *doesn't* apply to us. In particular, his main two prescriptions aside from "have leadership" are #6 and #7. #6 is that the legal Foundation part of the project should restrict itself to administrative activities and get out of technical decision making; the draft governance document says "[The NumFOCUS] Subcommittee shall NOT make decisions about the direction, scope or technical direction of the Project.". #7 is: The "core" group must be replaced with people who are actually competent and dedicated enough to review proposals, accept feedback, and make good decisions. More to the point, though, the "core" group must only act when *needed* -- most technical decisions should be left to the community to hash out; it must not preempt the community from developing better solutions. (This is how the "core" group worked during most of the project's growth period.) So he has nothing at all against the idea of a "core group", he just thinks it should be competent and... basically follow the rules that we attempted to codify in the draft governance document....? -n [1] http://www.cnn.com/TECH/computing/9906/21/linus.idg/ [2] https://en.wikipedia.org/wiki/Linux_kernel#Development [3] http://mail-index.netbsd.org/netbsd-users/2006/08/30/0016.html -- Nathaniel J. Smith -- http://vorpus.org From pizza at netspace.net.au Sat Sep 5 10:28:26 2015 From: pizza at netspace.net.au (pizza at netspace.net.au) Date: Sat, 05 Sep 2015 22:28:26 +0800 Subject: [Numpy-discussion] review of #6247 Message-ID: Hi all, first attempt at this: small doco patch to address?https://github.com/numpy/numpy/issues/6247 https://github.com/numpy/numpy/compare/master...pizzathief:issue6247?diff=unified&expand=1&name=issue6247 -------------- next part -------------- An HTML attachment was scrubbed... URL: From matthew.brett at gmail.com Sat Sep 5 19:20:37 2015 From: matthew.brett at gmail.com (Matthew Brett) Date: Sat, 5 Sep 2015 16:20:37 -0700 Subject: [Numpy-discussion] Comments on governance proposal (was: Notes from the numpy dev meeting at scipy 2015) In-Reply-To: References: <87y4gxllb4.fsf@berkeley.edu> <1440673873.1694.33.camel@sipsolutions.net> Message-ID: Hi, Thanks for replying, that is helpful. I'm going to write carefully, because I suspect we've only got this one shot at getting it right. I first wanted to come back to an earlier email of yours: On Thu, Aug 27, 2015 at 3:05 AM, Nathaniel Smith wrote: > At least personally, I have to admit that the idea of a governance > model involving elections fills me with creeping horror. The reason is > that whole point of having a governance model (IMHO) is to (a) > minimize the rise of interpersonal drama, (b) when some amount of > interpersonal drama does inevitably arise anyway, provide some > regulated channel for it, hopefully one that leads to a drama sink. I think this is the wrong way to think of the problem. There is no drama when: * everyone agrees (problem is trivial); * no one cares (problem is not important); * everyone who does care has come to the conclusion that it is not worth arguing. The opposite of drama is not efficiency, but boredom and frustration. As evidence, vigorous, substantial and bad-tempered debates are characteristic of successful companies: """ "They were chaos. We would stay there for hours, ironing out the issues, until we came to something ... At times the meetings would get so violent that people almost went across the table at each other.... People yelled. They waved their arms around and pounded on tables. Faces would get red and veins bulged out". """ [1] I'm sure you have hung out with the IPython team as I have. You will then hear their description of how much the team disagree with each other, and often, how important that disagreement is, to finding the right answer. On the other hand, I don't think it is our culture in numpy to value that kind of debate. That brings us to my next comment: On Fri, Sep 4, 2015 at 8:35 PM, Nathaniel Smith wrote: > On Fri, Aug 28, 2015 at 1:46 AM, Matthew Brett wrote: >> Hi, >> >> On Fri, Aug 28, 2015 at 5:59 AM, Jaime Fern?ndez del R?o >> wrote: > [...] >>> >>> Are you trying to prove the point that consensus doesn't work by making it >>> impossible to reach a consensus on this? ;-) >>> >> >> Forgive me if I use this joke to see if I can get us any further. >> >> If this was code, I think this joke would not be funny, because we >> wouldn't expect to reach consensus without considering all the >> options, and discussing their pros and cons. >> >> Why would that not be useful in the case of forms of governance? >> >> One reason might be that the specific form of governance can have no >> influence on the long-term health of the project. >> >> I am convinced that that is wrong - that the form of governance has a >> large influence on the long-term health of a project. >> >> If there is some possibility that this is true, then it seems to me >> that we would be foolish not to try and come to some reasoned choice >> about the form of governance. > > That seems fair, and I also think that even if we end up not changing > anything then there's still some utility in having the discussion. > Certainly one of my secret fears in writing all this up is that it > would end up being just some words that only I cared about and > everyone else ignored, so seeing other people engaging with it is > heartening :-). > > I'm not feeling inspired this Friday evening to write a full white > paper reviewing the universe of possible governance, but I'll > summarize a bit what the reasoning was that led us to what we ended up > with (at least as I remember it and from my perspective). > > Some of the sources that were mentioned for reference and positive or > negative inspiration include Karl Fogel's chapter on governance, the > Debian constitution, the gcc/egcs fork+reverse-fork, glibc/eglibc > ditto, subversion, Jupyter/ipython, the Linux kernel, python, > matplotlib, the Apache foundation's processes, the gnome foundation, > nodejs, ... I'm certainly forgetting some. > > The basic observation that we started from was that all the successful > projects seemed to fit a similar template: > (1) basically all actual decision making is done by informal and > maximally inclusive consensus-based processes, > (2) there are some sort of more formal "backstop" rules that are > called into play if informal processes break down (which hopefully > never even happens). > These projects don't tend to look that similar on paper, because > people tend to only write down the "backstop" rules, and those vary a > lot. > > So e.g., for some projects like Python or IPython, the formal rule is > "the BDFL decides". But then if you look at what successful BDFLs do > with this power, they mostly enforce rule #1 above. (Karl discusses > this: http://producingoss.com/en/producingoss.html#benevolent-dictator). > E.g., if you watch Guido, I don't think I've ever seen him say "I know > people don't like X but we're doing it anyway". He mostly sits back to > watch the debate, steps in occasionally to prune off unproductive > lines of conversation (e.g. in the PEP 465 debate where he watched for > a while, stepped in to tell people to knock it off with the pointless > bikeshedding about which character to use, and then went back to > watching) or encourage people to keep thinking about something, is > always very careful not to actually make any strong statements, etc. > Or, e.g., the bit in the draft governance about how the steering > committee's secondary job is to make decisions, and their primary job > is to prevent things reaching the point where they have to? That's a > straight paraphrase of Fernando talking about his BDFL philosophy. > > Or on the other end of things, you have e.g. Subversion, which had an > elaborate defined governance system with different levels of > "core-ness", a voting system, etc. -- and they were 6 years into the > project before they had their first vote. (The vote was on the crucial > technical decision of whether to write function calls like "f ()" or > "f()".) Or e.g. the massive and famously fractious Debian has their > CTTE, which theoretically is the final arbiter of technical decisions > in Debian and has immense powers... but they issue like ~5 rulings a > year on average, and have gone years without issuing a ruling at all. > > So, our thought process was: for our purposes, since we don't have a > BDFL, we can't depend on them to be the unwritten mechanism for > enforcing rule #1, so first we should write down rule #1. What you are describing in the previous paragraph seems to be a well-known feature of successful leaders. The CEOs of highly successful companies described in [2] differed from less-successful comparators in encouraging debate and acting as the final arbiter for the decisions of the group. I don't think you are right that you can create that effect without having such a leader. The point about the leader is that they are a trusted referee and mentor to the group. They provide stability and space for the debate. For example, in my little church, there is an elected position called the 'moderator' with that purpose. Most groups recognize that someone has to do that job, and that the group cannot do that themselves. On Sat, Sep 5, 2015 at 12:46 AM, Nathaniel Smith wrote: > I guess I just don't see how you think we can legislate ourselves into > having a vision. Like... we'll elect a volunteer to produce a roadmap, > and then if they don't we'll fire them (accountability!), leaving us > with no volunteer and no roadmap? How will that help? I agree it is not sensible for the president or moderator to produce a roadmap, but a roadmap is different from a vision. We often run into trouble on the list, discussing whether to make backwards-incompatible changes. The reason for the trouble is that different people have different ideas about what numpy is. Some people think numpy is a legacy library who's job it is to tread water while waiting for a better alternative. For them, backwards compatibility is of overriding importance. Others think that numpy should be gradually refactored into a better and more extensible version that will continue to be the backbone of scientific computing into the fairly distant future. These are two different visions that are not compatible one with another. Trying to keep both groups happy is likely to lead to dissatisfaction and frustration from both sides. On Sat, Sep 5, 2015 at 12:46 AM, Nathaniel Smith wrote: > I'm not sure how useful this focus on NetBSD is, given the elephant in > the room: the fundamental challenge for NetBSD is that NetBSD has no > compelling reason to exist; That is now true, but I would argue (I think Hannum would argue) that this was the direct effect of very poor management, with no vision for where the project was going. I imagine you'd agree that OpenBSD does have a purpose, and OpenBSD started as a one-person-in-charge fork of NetBSD, where the fork was the result of very bad management by the 'core'. I hate to quote these again but: "[the "core" and "board of directors"] teams are dysfunctional because they do not provide leadership: all they do is act reactively to requests from users and/or to resolve internal disputes. In other words: there is no initiative nor vision emerging from these teams (and, for that matter, from anybody)." "As things are today, the project leaders will almost always settle for either the conservative solution or a solution that compromises on every detail so as to not annoy anyone. " [3] A organization that is governed like this, just has to fail, in the long term. Cheers, Matthew [1] 'Exhibit E' in http://nipyworld.blogspot.co.uk/2012/02/what-is-meritocracy-in-open-source.html [2] https://en.wikipedia.org/wiki/Good_to_Great [3] http://asterisk.dynevor.org/melting-core.html From njs at pobox.com Sat Sep 5 22:14:17 2015 From: njs at pobox.com (Nathaniel Smith) Date: Sat, 5 Sep 2015 19:14:17 -0700 Subject: [Numpy-discussion] Comments on governance proposal (was: Notes from the numpy dev meeting at scipy 2015) In-Reply-To: References: <87y4gxllb4.fsf@berkeley.edu> <1440673873.1694.33.camel@sipsolutions.net> Message-ID: On Sat, Sep 5, 2015 at 4:20 PM, Matthew Brett wrote: > Hi, > > Thanks for replying, that is helpful. > > I'm going to write carefully, because I suspect we've only got this > one shot at getting it right. Not sure why? "Change the governance document" is explicitly listed as one of the steering council's powers, so I don't see what we'd lose by adopting what we have now. > I first wanted to come back to an earlier email of yours: > > On Thu, Aug 27, 2015 at 3:05 AM, Nathaniel Smith wrote: >> At least personally, I have to admit that the idea of a governance >> model involving elections fills me with creeping horror. The reason is >> that whole point of having a governance model (IMHO) is to (a) >> minimize the rise of interpersonal drama, (b) when some amount of >> interpersonal drama does inevitably arise anyway, provide some >> regulated channel for it, hopefully one that leads to a drama sink. > > I think this is the wrong way to think of the problem. There is no drama when: > > * everyone agrees (problem is trivial); > * no one cares (problem is not important); > * everyone who does care has come to the conclusion that it is not > worth arguing. > > The opposite of drama is not efficiency, but boredom and frustration. > > As evidence, vigorous, substantial and bad-tempered debates are > characteristic of successful companies: > > """ > "They were chaos. We would stay there for hours, ironing out the > issues, until we came to something ... At times the meetings would get > so violent that people almost went across the table at each other.... > People yelled. They waved their arms around and pounded on tables. > Faces would get red and veins bulged out". > """ [1] No, by drama I mean situations where technical issues become personal, and where people end up feeling emotionally hurt and disrespected and so forth. I agree that vigorous debate is important, but strongly disagree that veins bulging and violence are necessary or helpful to success. (This is tangential to your main point, but I wanted to point it out.) Since I happened to be reading Kate Heddleston's blog yesterday, here's another relevant link... https://www.kateheddleston.com/blog/argument-cultures-and-unregulated-aggression > I'm sure you have hung out with the IPython team as I have. You will > then hear their description of how much the team disagree with each > other, and often, how important that disagreement is, to finding the > right answer. Indeed, that's... why we have a document that is designed to provide a framework for exactly that sort of debate? I don't really understand the criticism here. My objection to elections is that they create a whole *extra* source of drama that runs the risk of compromising technical debates. Suddenly people have to campaign and have fraught arguments over who gets to vote and how to run the election and it's just borrowing trouble IMO. (Are you by chance on the PSF members mailing list? If not then ask someone how things have been going there vis-a-vis elections -- I'm not myself, but I've heard stories...) Obviously it's a fallacy to think that we can entirely ignore social issues and focus only on technical problems, but still -- I'd rather we spend our difficult debates on technical issues as much as possible. Even worse, elections mean that someone might think "Hmm, should I give in on this point, even though I know I'm right, so as to avoid annoying this person who might otherwise retaliate by telling their friends not to vote for me next year?" Or "hmm, did they give in on that point because they actually agree or because the person they work for wants the person they're agreeing with to win the next election?" Or etc. Hopefully no-one would actually start playing politics to that degree, but IMHO it's poisonous to even set up a situation where people are tempted to perform such calculations. > On the other hand, I don't think it is our culture in numpy to value > that kind of debate. That brings us to my next comment: > > On Fri, Sep 4, 2015 at 8:35 PM, Nathaniel Smith wrote: >> On Fri, Aug 28, 2015 at 1:46 AM, Matthew Brett wrote: >>> Hi, >>> >>> On Fri, Aug 28, 2015 at 5:59 AM, Jaime Fern?ndez del R?o >>> wrote: >> [...] >>>> >>>> Are you trying to prove the point that consensus doesn't work by making it >>>> impossible to reach a consensus on this? ;-) >>>> >>> >>> Forgive me if I use this joke to see if I can get us any further. >>> >>> If this was code, I think this joke would not be funny, because we >>> wouldn't expect to reach consensus without considering all the >>> options, and discussing their pros and cons. >>> >>> Why would that not be useful in the case of forms of governance? >>> >>> One reason might be that the specific form of governance can have no >>> influence on the long-term health of the project. >>> >>> I am convinced that that is wrong - that the form of governance has a >>> large influence on the long-term health of a project. >>> >>> If there is some possibility that this is true, then it seems to me >>> that we would be foolish not to try and come to some reasoned choice >>> about the form of governance. >> >> That seems fair, and I also think that even if we end up not changing >> anything then there's still some utility in having the discussion. >> Certainly one of my secret fears in writing all this up is that it >> would end up being just some words that only I cared about and >> everyone else ignored, so seeing other people engaging with it is >> heartening :-). >> >> I'm not feeling inspired this Friday evening to write a full white >> paper reviewing the universe of possible governance, but I'll >> summarize a bit what the reasoning was that led us to what we ended up >> with (at least as I remember it and from my perspective). >> >> Some of the sources that were mentioned for reference and positive or >> negative inspiration include Karl Fogel's chapter on governance, the >> Debian constitution, the gcc/egcs fork+reverse-fork, glibc/eglibc >> ditto, subversion, Jupyter/ipython, the Linux kernel, python, >> matplotlib, the Apache foundation's processes, the gnome foundation, >> nodejs, ... I'm certainly forgetting some. >> >> The basic observation that we started from was that all the successful >> projects seemed to fit a similar template: >> (1) basically all actual decision making is done by informal and >> maximally inclusive consensus-based processes, >> (2) there are some sort of more formal "backstop" rules that are >> called into play if informal processes break down (which hopefully >> never even happens). >> These projects don't tend to look that similar on paper, because >> people tend to only write down the "backstop" rules, and those vary a >> lot. >> >> So e.g., for some projects like Python or IPython, the formal rule is >> "the BDFL decides". But then if you look at what successful BDFLs do >> with this power, they mostly enforce rule #1 above. (Karl discusses >> this: http://producingoss.com/en/producingoss.html#benevolent-dictator). >> E.g., if you watch Guido, I don't think I've ever seen him say "I know >> people don't like X but we're doing it anyway". He mostly sits back to >> watch the debate, steps in occasionally to prune off unproductive >> lines of conversation (e.g. in the PEP 465 debate where he watched for >> a while, stepped in to tell people to knock it off with the pointless >> bikeshedding about which character to use, and then went back to >> watching) or encourage people to keep thinking about something, is >> always very careful not to actually make any strong statements, etc. >> Or, e.g., the bit in the draft governance about how the steering >> committee's secondary job is to make decisions, and their primary job >> is to prevent things reaching the point where they have to? That's a >> straight paraphrase of Fernando talking about his BDFL philosophy. >> >> Or on the other end of things, you have e.g. Subversion, which had an >> elaborate defined governance system with different levels of >> "core-ness", a voting system, etc. -- and they were 6 years into the >> project before they had their first vote. (The vote was on the crucial >> technical decision of whether to write function calls like "f ()" or >> "f()".) Or e.g. the massive and famously fractious Debian has their >> CTTE, which theoretically is the final arbiter of technical decisions >> in Debian and has immense powers... but they issue like ~5 rulings a >> year on average, and have gone years without issuing a ruling at all. >> >> So, our thought process was: for our purposes, since we don't have a >> BDFL, we can't depend on them to be the unwritten mechanism for >> enforcing rule #1, so first we should write down rule #1. > > What you are describing in the previous paragraph seems to be a > well-known feature of successful leaders. The CEOs of highly > successful companies described in [2] differed from less-successful > comparators in encouraging debate and acting as the final arbiter for > the decisions of the group. > > I don't think you are right that you can create that effect without > having such a leader. The point about the leader is that they are a > trusted referee and mentor to the group. They provide stability and > space for the debate. For example, in my little church, there is an > elected position called the 'moderator' with that purpose. Most > groups recognize that someone has to do that job, and that the group > cannot do that themselves. Having a good moderator is certainly a wonderful thing if you can manage it, but it's just not true that "Most groups recognize [...] the group cannot do that themselves". FOSS projects with a formal leader are in the minority. And there's no particular need for a specific person to be designated to always do the job: in a consensus-based framework, pretty much anyone has the power to step up to act as an arbitrator or to protect a minority opinion. The main requirement for someone to be a successful arbitrator is that the participants trust and respect their judgement as a neutral party. If you don't have such a person, then an election won't create that that trust and respect. If you do have such a person -- or multiple such people! -- then an election is unnecessary. > On Sat, Sep 5, 2015 at 12:46 AM, Nathaniel Smith wrote: >> I guess I just don't see how you think we can legislate ourselves into >> having a vision. Like... we'll elect a volunteer to produce a roadmap, >> and then if they don't we'll fire them (accountability!), leaving us >> with no volunteer and no roadmap? How will that help? > > I agree it is not sensible for the president or moderator to produce a > roadmap, but a roadmap is different from a vision. > > We often run into trouble on the list, discussing whether to make > backwards-incompatible changes. The reason for the trouble is that > different people have different ideas about what numpy is. Some > people think numpy is a legacy library who's job it is to tread water > while waiting for a better alternative. For them, backwards > compatibility is of overriding importance. Others think that numpy > should be gradually refactored into a better and more extensible > version that will continue to be the backbone of scientific computing > into the fairly distant future. These are two different visions that > are not compatible one with another. Trying to keep both groups happy > is likely to lead to dissatisfaction and frustration from both sides. Yep. And like you said above re: IPython and so forth, the solution is to confront that problem head-on and have the debate. Having a dictator come in and dictate the outcome wouldn't help anything. > On Sat, Sep 5, 2015 at 12:46 AM, Nathaniel Smith wrote: >> I'm not sure how useful this focus on NetBSD is, given the elephant in >> the room: the fundamental challenge for NetBSD is that NetBSD has no >> compelling reason to exist; > > That is now true, but I would argue (I think Hannum would argue) that > this was the direct effect of very poor management, with no vision for > where the project was going. I imagine you'd agree that OpenBSD does > have a purpose, and OpenBSD started as a one-person-in-charge fork of > NetBSD, where the fork was the result of very bad management by the > 'core'. I think you might be trying too hard to make history line up with your conclusions here :-/. Theo de Raadt didn't quit; he got kicked out in 1994 because they felt his "long history of rudeness towards and abuse of users and developers" finally outweighed his "long history of positive contributions": https://mail-index.netbsd.org/netbsd-users/1994/12/23/0000.html Obviously I don't know the whole sordid backstory, but "productive + rude + abusive" still pretty much characterizes his reputation today. And in general making this kind of difficult decision is exactly what I would look for as a sign of *good* management. And isn't 1994 well within the period where you and Hannum are both arguing that the NetBSD 'core' was still functional and healthy? Anyway, who knows why NetBSD failed. It's surely some complex and overdetermined combination of their governance structure, the license (Linux's use of the GPL has often been cited as a crucial factor in its becoming "the standard free kernel", b/c it created a level playing field for otherwise competing companies to collaborate and this set off the feedback cycle where it's the obvious one to use -> so people use it and contribute back -> so it's even more obviously the one to use -> ...), the particular individuals who happened to attach themselves to NetBSD vs. the other BSDs vs. Linux, the different approaches to handling userspace (developed alongside the kernel vs. developed separately), and so on and so forth. I just don't think these cherrypicked messages about one project are terribly compelling as a data point. > I hate to quote these again but: > > "[the "core" and "board of directors"] teams are dysfunctional because > they do not provide leadership: all they do is act reactively to > requests from users and/or to resolve internal disputes. In other > words: there is no initiative nor vision emerging from these teams > (and, for that matter, from anybody)." > > "As things are today, the project leaders will almost always settle > for either the conservative solution or a solution that compromises on > every detail so as to not annoy anyone. " [3] > > A organization that is governed like this, just has to fail, in the long term. Sure, but I don't understand why you conclude that these are inevitable consequences of any organization where more developers are "more central" than others without a single strong leader. You're quoting Hannum, but Hannum doesn't think that - his prescription is to get better a better core team. Linux itself -- the project that ate NetBSD's lunch -- doesn't have a core leader providing vision and initiative, and it seems to be doing okay. (They have Linus obviously, but he mostly acts as a moderator -- he definitely does not tell anyone that they need to go write a better filesystem or go invent cgroups or perf or whatever. That's all bottom-up.) Leadership is something that people do, not something people are. You can't legislate a vision into existence any more than you can legislate trust into existence. You can try to create the conditions for it to happen, like by getting people into the same room and providing a clear path for newcomers to propose ideas and so forth, but to me elections seem like exactly the wrong approach. Or instead of debating how to create a vision in the abstract, we could shift our energy over to the other thread that's debating what the vision should actually be :-). -n -- Nathaniel J. Smith -- http://vorpus.org From matthew.brett at gmail.com Sat Sep 5 23:08:47 2015 From: matthew.brett at gmail.com (Matthew Brett) Date: Sat, 5 Sep 2015 20:08:47 -0700 Subject: [Numpy-discussion] Comments on governance proposal (was: Notes from the numpy dev meeting at scipy 2015) In-Reply-To: References: <87y4gxllb4.fsf@berkeley.edu> <1440673873.1694.33.camel@sipsolutions.net> Message-ID: On Sat, Sep 5, 2015 at 7:14 PM, Nathaniel Smith wrote: > On Sat, Sep 5, 2015 at 4:20 PM, Matthew Brett wrote: >> Hi, >> >> Thanks for replying, that is helpful. >> >> I'm going to write carefully, because I suspect we've only got this >> one shot at getting it right. > > Not sure why? "Change the governance document" is explicitly listed as > one of the steering council's powers, so I don't see what we'd lose by > adopting what we have now. In practice, I doubt that there will be any proposal to change the governance in the medium term, especially if the governance does turn out to be risk averse and conservative. >> I first wanted to come back to an earlier email of yours: >> >> On Thu, Aug 27, 2015 at 3:05 AM, Nathaniel Smith wrote: >>> At least personally, I have to admit that the idea of a governance >>> model involving elections fills me with creeping horror. The reason is >>> that whole point of having a governance model (IMHO) is to (a) >>> minimize the rise of interpersonal drama, (b) when some amount of >>> interpersonal drama does inevitably arise anyway, provide some >>> regulated channel for it, hopefully one that leads to a drama sink. >> >> I think this is the wrong way to think of the problem. There is no drama when: >> >> * everyone agrees (problem is trivial); >> * no one cares (problem is not important); >> * everyone who does care has come to the conclusion that it is not >> worth arguing. >> >> The opposite of drama is not efficiency, but boredom and frustration. >> >> As evidence, vigorous, substantial and bad-tempered debates are >> characteristic of successful companies: >> >> """ >> "They were chaos. We would stay there for hours, ironing out the >> issues, until we came to something ... At times the meetings would get >> so violent that people almost went across the table at each other.... >> People yelled. They waved their arms around and pounded on tables. >> Faces would get red and veins bulged out". >> """ [1] > > No, by drama I mean situations where technical issues become personal, > and where people end up feeling emotionally hurt and disrespected and > so forth. I agree that vigorous debate is important, but strongly > disagree that veins bulging and violence are necessary or helpful to > success. (This is tangential to your main point, but I wanted to point > it out.) Since I happened to be reading Kate Heddleston's blog > yesterday, here's another relevant link... > https://www.kateheddleston.com/blog/argument-cultures-and-unregulated-aggression The distinction is between a culture that welcomes expressions of disagreement with one that does not. In practice, if a group is easily alarmed by expressions of strong feeling, most people will be careful about expressing disagreement. This is of course a rather subtle point. It is easy to say that we do welcome disagreement, when in practice what happens is that the person disagreeing will in fact be labeled as being impolite. >> then hear their description of how much the team disagree with each >> other, and often, how important that disagreement is, to finding the >> right answer. > > Indeed, that's... why we have a document that is designed to provide a > framework for exactly that sort of debate? I don't really understand > the criticism here. Er - I was arguing that, by attempting to formalize a system that doesn't have a leader, that will lead to a culture where debate is more difficult and less effective. > My objection to elections is that they create a whole *extra* source > of drama that runs the risk of compromising technical debates. > Suddenly people have to campaign and have fraught arguments over who > gets to vote and how to run the election and it's just borrowing > trouble IMO. (Are you by chance on the PSF members mailing list? If > not then ask someone how things have been going there vis-a-vis > elections -- I'm not myself, but I've heard stories...) Obviously it's > a fallacy to think that we can entirely ignore social issues and focus > only on technical problems, but still -- I'd rather we spend our > difficult debates on technical issues as much as possible. > > Even worse, elections mean that someone might think "Hmm, should I > give in on this point, even though I know I'm right, so as to avoid > annoying this person who might otherwise retaliate by telling their > friends not to vote for me next year?" Or "hmm, did they give in on > that point because they actually agree or because the person they work > for wants the person they're agreeing with to win the next election?" > Or etc. Hopefully no-one would actually start playing politics to that > degree, but IMHO it's poisonous to even set up a situation where > people are tempted to perform such calculations. I suppose these things might happen. The benefit of elections is that it forces us to be explicit about what the issues are, and what is at stake. It seems to me that is something that numpy has to do, and soon, whatever the mechanism. >> On the other hand, I don't think it is our culture in numpy to value >> that kind of debate. That brings us to my next comment: >> >> On Fri, Sep 4, 2015 at 8:35 PM, Nathaniel Smith wrote: >>> On Fri, Aug 28, 2015 at 1:46 AM, Matthew Brett wrote: >>>> Hi, >>>> >>>> On Fri, Aug 28, 2015 at 5:59 AM, Jaime Fern?ndez del R?o >>>> wrote: >>> [...] >>>>> >>>>> Are you trying to prove the point that consensus doesn't work by making it >>>>> impossible to reach a consensus on this? ;-) >>>>> >>>> >>>> Forgive me if I use this joke to see if I can get us any further. >>>> >>>> If this was code, I think this joke would not be funny, because we >>>> wouldn't expect to reach consensus without considering all the >>>> options, and discussing their pros and cons. >>>> >>>> Why would that not be useful in the case of forms of governance? >>>> >>>> One reason might be that the specific form of governance can have no >>>> influence on the long-term health of the project. >>>> >>>> I am convinced that that is wrong - that the form of governance has a >>>> large influence on the long-term health of a project. >>>> >>>> If there is some possibility that this is true, then it seems to me >>>> that we would be foolish not to try and come to some reasoned choice >>>> about the form of governance. >>> >>> That seems fair, and I also think that even if we end up not changing >>> anything then there's still some utility in having the discussion. >>> Certainly one of my secret fears in writing all this up is that it >>> would end up being just some words that only I cared about and >>> everyone else ignored, so seeing other people engaging with it is >>> heartening :-). >>> >>> I'm not feeling inspired this Friday evening to write a full white >>> paper reviewing the universe of possible governance, but I'll >>> summarize a bit what the reasoning was that led us to what we ended up >>> with (at least as I remember it and from my perspective). >>> >>> Some of the sources that were mentioned for reference and positive or >>> negative inspiration include Karl Fogel's chapter on governance, the >>> Debian constitution, the gcc/egcs fork+reverse-fork, glibc/eglibc >>> ditto, subversion, Jupyter/ipython, the Linux kernel, python, >>> matplotlib, the Apache foundation's processes, the gnome foundation, >>> nodejs, ... I'm certainly forgetting some. >>> >>> The basic observation that we started from was that all the successful >>> projects seemed to fit a similar template: >>> (1) basically all actual decision making is done by informal and >>> maximally inclusive consensus-based processes, >>> (2) there are some sort of more formal "backstop" rules that are >>> called into play if informal processes break down (which hopefully >>> never even happens). >>> These projects don't tend to look that similar on paper, because >>> people tend to only write down the "backstop" rules, and those vary a >>> lot. >>> >>> So e.g., for some projects like Python or IPython, the formal rule is >>> "the BDFL decides". But then if you look at what successful BDFLs do >>> with this power, they mostly enforce rule #1 above. (Karl discusses >>> this: http://producingoss.com/en/producingoss.html#benevolent-dictator). >>> E.g., if you watch Guido, I don't think I've ever seen him say "I know >>> people don't like X but we're doing it anyway". He mostly sits back to >>> watch the debate, steps in occasionally to prune off unproductive >>> lines of conversation (e.g. in the PEP 465 debate where he watched for >>> a while, stepped in to tell people to knock it off with the pointless >>> bikeshedding about which character to use, and then went back to >>> watching) or encourage people to keep thinking about something, is >>> always very careful not to actually make any strong statements, etc. >>> Or, e.g., the bit in the draft governance about how the steering >>> committee's secondary job is to make decisions, and their primary job >>> is to prevent things reaching the point where they have to? That's a >>> straight paraphrase of Fernando talking about his BDFL philosophy. >>> >>> Or on the other end of things, you have e.g. Subversion, which had an >>> elaborate defined governance system with different levels of >>> "core-ness", a voting system, etc. -- and they were 6 years into the >>> project before they had their first vote. (The vote was on the crucial >>> technical decision of whether to write function calls like "f ()" or >>> "f()".) Or e.g. the massive and famously fractious Debian has their >>> CTTE, which theoretically is the final arbiter of technical decisions >>> in Debian and has immense powers... but they issue like ~5 rulings a >>> year on average, and have gone years without issuing a ruling at all. >>> >>> So, our thought process was: for our purposes, since we don't have a >>> BDFL, we can't depend on them to be the unwritten mechanism for >>> enforcing rule #1, so first we should write down rule #1. >> >> What you are describing in the previous paragraph seems to be a >> well-known feature of successful leaders. The CEOs of highly >> successful companies described in [2] differed from less-successful >> comparators in encouraging debate and acting as the final arbiter for >> the decisions of the group. >> >> I don't think you are right that you can create that effect without >> having such a leader. The point about the leader is that they are a >> trusted referee and mentor to the group. They provide stability and >> space for the debate. For example, in my little church, there is an >> elected position called the 'moderator' with that purpose. Most >> groups recognize that someone has to do that job, and that the group >> cannot do that themselves. > > Having a good moderator is certainly a wonderful thing if you can > manage it, but it's just not true that "Most groups recognize [...] > the group cannot do that themselves". FOSS projects with a formal > leader are in the minority. I believe you - I was referring to groups like businesses, universities, governments, churches and so on. > And there's no particular need for a > specific person to be designated to always do the job: in a > consensus-based framework, pretty much anyone has the power to step up > to act as an arbitrator or to protect a minority opinion. This is of course the ideal, but I think that just doesn't work in practice. When tempers fray and people feel strongly, they will tend not to accept arbitration from any of the other active participants, or a self-designated outsider. >> On Sat, Sep 5, 2015 at 12:46 AM, Nathaniel Smith wrote: >>> I guess I just don't see how you think we can legislate ourselves into >>> having a vision. Like... we'll elect a volunteer to produce a roadmap, >>> and then if they don't we'll fire them (accountability!), leaving us >>> with no volunteer and no roadmap? How will that help? >> >> I agree it is not sensible for the president or moderator to produce a >> roadmap, but a roadmap is different from a vision. >> >> We often run into trouble on the list, discussing whether to make >> backwards-incompatible changes. The reason for the trouble is that >> different people have different ideas about what numpy is. Some >> people think numpy is a legacy library who's job it is to tread water >> while waiting for a better alternative. For them, backwards >> compatibility is of overriding importance. Others think that numpy >> should be gradually refactored into a better and more extensible >> version that will continue to be the backbone of scientific computing >> into the fairly distant future. These are two different visions that >> are not compatible one with another. Trying to keep both groups happy >> is likely to lead to dissatisfaction and frustration from both sides. > > Yep. And like you said above re: IPython and so forth, the solution is > to confront that problem head-on and have the debate. Having a > dictator come in and dictate the outcome wouldn't help anything. Yes, I see I am not being clear. I was claiming that it is difficult to have this debate without someone who will make the final call. That person doesn't 'dictate' but they have the final say when all the arguments are in. If I am right, then, with the no-leader consensus model you propose, it will be hard to have that debate because it is harder to get to the final decision. >> On Sat, Sep 5, 2015 at 12:46 AM, Nathaniel Smith wrote: >>> I'm not sure how useful this focus on NetBSD is, given the elephant in >>> the room: the fundamental challenge for NetBSD is that NetBSD has no >>> compelling reason to exist; >> >> That is now true, but I would argue (I think Hannum would argue) that >> this was the direct effect of very poor management, with no vision for >> where the project was going. I imagine you'd agree that OpenBSD does >> have a purpose, and OpenBSD started as a one-person-in-charge fork of >> NetBSD, where the fork was the result of very bad management by the >> 'core'. > > I think you might be trying too hard to make history line up with your > conclusions here :-/. Theo de Raadt didn't quit; he got kicked out in > 1994 because they felt his "long history of rudeness towards and abuse > of users and developers" finally outweighed his "long history of > positive contributions": I know that history very well, sorry that I did not unpack what I meant. I believe that the 'core' group made a terrible social mess of trying to deal with TdR's interaction style, in a characteristic 'core' way, which was that they ended up throwing him out with an officious email where no one member of the core prepared to accept the responsibility for doing that, and with no way back once they had done that. I would also argue that the 'core' way tends to a rather conservative and joyless social model that made it particularly difficult for them to deal with TdR. But anyway - that was not my point. My point was that, in 1994, as NetBSD was losing direction, it was perfectly possible to fork it and take it in a new and fruitful direction, with the right governance and leader. Despite his later prescription, Hannum is arguing that the core structure, specifically, was at fault for preventing that happening in NetBSD and in other projects: """ Much of this early structure (CVS, web site, cabal ["core" group], etc.) was copied verbatim by other open source (this term not being in wide use yet) projects -- even the form of the project name and the term "core". This later became a kind of standard template for starting up an open source project. [...] I'm sorry to say that I helped create this problem, and that most of the projects which modeled themselves after NetBSD (probably due to its high popularity in 1993 and 1994) have suffered similar problems. FreeBSD and XFree86, for example, have both forked successor projects (Dragonfly and X.org) for very similar reasons. """ [1[ Cheers, Matthew [1] http://asterisk.dynevor.org/melting-core.html From njs at pobox.com Sun Sep 6 02:36:45 2015 From: njs at pobox.com (Nathaniel Smith) Date: Sat, 5 Sep 2015 23:36:45 -0700 Subject: [Numpy-discussion] Comments on governance proposal (was: Notes from the numpy dev meeting at scipy 2015) In-Reply-To: References: <87y4gxllb4.fsf@berkeley.edu> <1440673873.1694.33.camel@sipsolutions.net> Message-ID: On Sat, Sep 5, 2015 at 8:08 PM, Matthew Brett wrote: > On Sat, Sep 5, 2015 at 7:14 PM, Nathaniel Smith wrote: >> On Sat, Sep 5, 2015 at 4:20 PM, Matthew Brett wrote: >>> Hi, >>> >>> Thanks for replying, that is helpful. >>> >>> I'm going to write carefully, because I suspect we've only got this >>> one shot at getting it right. >> >> Not sure why? "Change the governance document" is explicitly listed as >> one of the steering council's powers, so I don't see what we'd lose by >> adopting what we have now. > > In practice, I doubt that there will be any proposal to change the > governance in the medium term, especially if the governance does turn > out to be risk averse and conservative. Whether there will be a proposal depends entirely on whether someone (perhaps you) makes one :-). Whether the proposal will be successful depends on many things, but I'm guessing you'd have *more* luck getting traction in this hypothetical future where you can point to how the project is suffering from risk aversion and over-conservative decision making. Right now everyone seems to be getting excited about getting stuff done and trying out this proposed model, which seems like the worst possible time to try and convince them that they're doomed to stagnation. [...] >> No, by drama I mean situations where technical issues become personal, >> and where people end up feeling emotionally hurt and disrespected and >> so forth. I agree that vigorous debate is important, but strongly >> disagree that veins bulging and violence are necessary or helpful to >> success. (This is tangential to your main point, but I wanted to point >> it out.) Since I happened to be reading Kate Heddleston's blog >> yesterday, here's another relevant link... >> https://www.kateheddleston.com/blog/argument-cultures-and-unregulated-aggression > > The distinction is between a culture that welcomes expressions of > disagreement with one that does not. In practice, if a group is > easily alarmed by expressions of strong feeling, most people will be > careful about expressing disagreement. This is of course a rather > subtle point. It is easy to say that we do welcome disagreement, when > in practice what happens is that the person disagreeing will in fact > be labeled as being impolite. I worry about this too. But this is a culture thing, not something that seems amenable to quick fixes; it requires buy-in from everyone individually. In particular I don't see what it has to do with whether we have a named leader or not -- being named "project leader" doesn't make someone's emails particularly more or less likely to change how anyone in particular thinks about things. [...] >> Yep. And like you said above re: IPython and so forth, the solution is >> to confront that problem head-on and have the debate. Having a >> dictator come in and dictate the outcome wouldn't help anything. > > Yes, I see I am not being clear. I was claiming that it is difficult > to have this debate without someone who will make the final call. > That person doesn't 'dictate' but they have the final say when all the > arguments are in. If I am right, then, with the no-leader consensus > model you propose, it will be hard to have that debate because it is > harder to get to the final decision. I suggest we try it and see :-) -n -- Nathaniel J. Smith -- http://vorpus.org From pizza at netspace.net.au Sun Sep 6 08:03:16 2015 From: pizza at netspace.net.au (pizza at netspace.net.au) Date: Sun, 06 Sep 2015 22:03:16 +1000 Subject: [Numpy-discussion] review of #6247 In-Reply-To: References: Message-ID: <1562003.6iNdAildBC@minipizza> On Sat, 5 Sep 2015 10:28:26 PM pizza at netspace.net.au wrote: > Hi all, first attempt at this: > small doco patch to > address https://github.com/numpy/numpy/issues/6247 > https://github.com/numpy/numpy/compare/master...pizzathief:issue6247?diff=un > ified&expand=1&name=issue6247 Fixed up next sentence as well. https://github.com/numpy/numpy/compare/master...pizzathief:issue6247?diff=unified&expand=1&name=issue6247 From faltet at gmail.com Wed Sep 9 12:22:23 2015 From: faltet at gmail.com (Francesc Alted) Date: Wed, 9 Sep 2015 18:22:23 +0200 Subject: [Numpy-discussion] ANN: bcolz 0.11.0 released Message-ID: ======================= Announcing bcolz 0.11.0 ======================= What's new ========== Although this is mostly a maintenance release that fixes some bugs, the setup.py is entirely based now in setuptools and has been greatly modernized to use a new versioning system. Just this deserves a bump in the minor version. Thanks to Gabi Davar (@mindw) for such a nice improvement. Also, many improvements to the Continuous Integration part (and hence not directly visible to users) and others have been made by Francesc Elies (@FrancescElies). Thanks for his quiet but effective work. And last but not least, I would like to announce that Valentin Haenel (@esc) just stepped down as release manager. Thanks Valentin for all the hard work that you put in making bcolz a better piece of software! What it is ========== *bcolz* provides columnar and compressed data containers that can live either on-disk or in-memory. Column storage allows for efficiently querying tables with a large number of columns. It also allows for cheap addition and removal of column. In addition, bcolz objects are compressed by default for reducing memory/disk I/O needs. The compression process is carried out internally by Blosc, an extremely fast meta-compressor that is optimized for binary data. Lastly, high-performance iterators (like ``iter()``, ``where()``) for querying the objects are provided. bcolz can use numexpr internally so as to accelerate many vector and query operations (although it can use pure NumPy for doing so too). numexpr optimizes the memory usage and use several cores for doing the computations, so it is blazing fast. Moreover, since the carray/ctable containers can be disk-based, and it is possible to use them for seamlessly performing out-of-memory computations. bcolz has minimal dependencies (NumPy), comes with an exhaustive test suite and fully supports both 32-bit and 64-bit platforms. Also, it is typically tested on both UNIX and Windows operating systems. Together, bcolz and the Blosc compressor, are finally fulfilling the promise of accelerating memory I/O, at least for some real scenarios: http://nbviewer.ipython.org/github/Blosc/movielens-bench/blob/master/querying-ep14.ipynb#Plots Other users of bcolz are Visualfabriq (http://www.visualfabriq.com/) the Blaze project (http://blaze.pydata.org/), Quantopian (https://www.quantopian.com/) and Scikit-Allel (https://github.com/cggh/scikit-allel) which you can read more about by pointing your browser at the links below. * Visualfabriq: * *bquery*, A query and aggregation framework for Bcolz: * https://github.com/visualfabriq/bquery * Blaze: * Notebooks showing Blaze + Pandas + BColz interaction: * http://nbviewer.ipython.org/url/blaze.pydata.org/notebooks/timings-csv.ipynb * http://nbviewer.ipython.org/url/blaze.pydata.org/notebooks/timings-bcolz.ipynb * Quantopian: * Using compressed data containers for faster backtesting at scale: * https://quantopian.github.io/talks/NeedForSpeed/slides.html * Scikit-Allel * Provides an alternative backend to work with compressed arrays * https://scikit-allel.readthedocs.org/en/latest/bcolz.html Installing ========== bcolz is in the PyPI repository, so installing it is easy:: $ pip install -U bcolz Resources ========= Visit the main bcolz site repository at: http://github.com/Blosc/bcolz Manual: http://bcolz.blosc.org Home of Blosc compressor: http://blosc.org User's mail list: bcolz at googlegroups.com http://groups.google.com/group/bcolz License is the new BSD: https://github.com/Blosc/bcolz/blob/master/LICENSES/BCOLZ.txt Release notes can be found in the Git repository: https://github.com/Blosc/bcolz/blob/master/RELEASE_NOTES.rst ---- **Enjoy data!** -- Francesc Alted -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Wed Sep 9 15:56:45 2015 From: charlesr.harris at gmail.com (Charles R Harris) Date: Wed, 9 Sep 2015 13:56:45 -0600 Subject: [Numpy-discussion] Commit rights for Stephan Hoyer Message-ID: Hi All, Stephan Hoyer now has commit rights to Numpy to compliment all the other projects of which he is a member. Congratutlations Stephan. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From ben.v.root at gmail.com Wed Sep 9 23:04:54 2015 From: ben.v.root at gmail.com (Benjamin Root) Date: Wed, 9 Sep 2015 23:04:54 -0400 Subject: [Numpy-discussion] Commit rights for Stephan Hoyer In-Reply-To: References: Message-ID: You realize that this is part of his plans for world domination, right? +1, Congrats! I wish I had enough skill and time to contribute to more projects. Cheers! Ben Root On Sep 9, 2015 3:56 PM, "Charles R Harris" wrote: > Hi All, > > Stephan Hoyer now has commit rights to Numpy to compliment all the other > projects of which he is a member. > > Congratutlations Stephan. > > Chuck > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From chenchao at inhand.com.cn Thu Sep 10 23:55:34 2015 From: chenchao at inhand.com.cn (chenchao at inhand.com.cn) Date: Fri, 11 Sep 2015 11:55:34 +0800 Subject: [Numpy-discussion] cross compile numpy In-Reply-To: <55F24FB5.6050600@inhand.com.cn> References: <55F24FB5.6050600@inhand.com.cn> Message-ID: <55F250B6.8060702@inhand.com.cn> hi: I have to use numpy package. My python runs on my embedded arm device. So, how do i cross compile numpy? ps: my embedded arm device supports floating point operation. From d.manson.11 at ucl.ac.uk Fri Sep 11 09:10:48 2015 From: d.manson.11 at ucl.ac.uk (Daniel Manson) Date: Fri, 11 Sep 2015 13:10:48 +0000 Subject: [Numpy-discussion] feature request - increment counter on write check Message-ID: Originally posted as issue 6301 on github. Presumably any block of code that modifies an ndarray's buffer is wrapped in a (thread safe?) check of the writable flag. Would it be possible to hold a counter rather than a simple bool flag and then increment the counter whenever you test the flag? Hopefully this would introduce only a tiny additional overhead, but would permit pseudo-hashing to test whether a mutable array has changed since you last encountered it. Ideally this should be exposed a bit like python's __hash__ method, lets say __mutablehash__, meaning a hash is returned but be warned that the object is mutable. For an ndarray, X, containing objects that themselves have a __mutablehash__ method (e.g. other ndarrays, or some user object), the X.__mutablehash__ method will need to do the full check over all constituent objects, or simply return None. Defining and API of this sort would make it possible to - for example - let pandas DataFrames also implement this interface. In terms of usage cases, the one I was motivated by was imagining improvements to the "variable explorer" in Spyder - roughly speaking, this widget's job is to always display an up-to-date summary of variables in current scope, e.g. currently it can show max/min and shape, but you could imagine also showing graphical summaries of the contents of an ndarray. If the widget could cache summaries and check which arrays have really changed it should be much faster/offer more features/be simpler internally. Note that pandas DataFrames are relevant here as an example of complex objects, containing ndarrays, which would benefit from being able to have their summaries cached. A more common/general usage case would be as a check in some kind of memoization process... #simple example... @memoize_please def hasnans(x): return np.any(np.isnan(x)) # more complex example... def convolve_fft(a,b, _cache={}): a_hash = mutablehash(a) b_hash = mutablehash(b) if a_hash not in _cache: _cache[a_hash] = fft(a) if b_hash not in _cache: _cache[b_hash] = fft(b) return ifft(_cache[a_hash] * _cache[b_hash]) A quick though on an implementation detail... I'm not sure exactly how to deal with the counter overflowing: perhaps if you treated counter==0 to mean not-writable (i.e. that would be the new version of the old write flag) then you might get some uint-wraparound checking for free (because when it wraps back around to zero the buffer ends up becoming locked)? Alternatively you could just say that no guarantee is given of wraparound being caught..though that might seriously impact on the range of possible uses. In summary... Hopefully the stuff needed to make __mutablehash__ work could be implemented simply by adding a single extra operation to the write-check (and maybe changing the footprint of the ndarray slightly to accomodate a counter). But I suspect someone will tell me that life is never that simple! -------------- next part -------------- An HTML attachment was scrubbed... URL: From sebastian at sipsolutions.net Fri Sep 11 09:20:28 2015 From: sebastian at sipsolutions.net (Sebastian Berg) Date: Fri, 11 Sep 2015 15:20:28 +0200 Subject: [Numpy-discussion] feature request - increment counter on write check In-Reply-To: References: Message-ID: <1441977628.1847.3.camel@sipsolutions.net> On Fr, 2015-09-11 at 13:10 +0000, Daniel Manson wrote: > Originally posted as issue 6301 on github. > > > Presumably any block of code that modifies an ndarray's buffer is > wrapped in a (thread safe?) check of the writable flag. Would it be > possible to hold a counter rather than a simple bool flag and then > increment the counter whenever you test the flag? Hopefully this would > introduce only a tiny additional overhead, but would permit > pseudo-hashing to test whether a mutable array has changed since you > last encountered it. > Just a quick note. This is a bit more complex then it might appear. The reason being that when a view of the array is changed, you would have to "notify" the array itself that it has changed. So propagation from top to bottom does not seem straight forward to me. (the other way is fine, since on check you could check all parents, but you cannot check all children). - Sebastian > Ideally this should be exposed a bit like python's __hash__ method, > lets say __mutablehash__, meaning a hash is returned but be warned > that the object is mutable. For an ndarray, X, containing objects > that themselves have a __mutablehash__ method (e.g. other ndarrays, or > some user object), the X.__mutablehash__ method will need to do the > full check over all constituent objects, or simply return None. > Defining and API of this sort would make it possible to - for example > - let pandas DataFrames also implement this interface. > > > In terms of usage cases, the one I was motivated by was imagining > improvements to the "variable explorer" in Spyder - roughly speaking, > this widget's job is to always display an up-to-date summary of > variables in current scope, e.g. currently it can show max/min and > shape, but you could imagine also showing graphical summaries of the > contents of an ndarray. If the widget could cache summaries and check > which arrays have really changed it should be much faster/offer more > features/be simpler internally. Note that pandas DataFrames are > relevant here as an example of complex objects, containing ndarrays, > which would benefit from being able to have their summaries cached. > > > A more common/general usage case would be as a check in some kind of > memoization process... > > > #simple example... > @memoize_please > def hasnans(x): > return np.any(np.isnan(x)) > > > # more complex example... > def convolve_fft(a,b, _cache={}): > a_hash = mutablehash(a) > b_hash = mutablehash(b) > if a_hash not in _cache: > _cache[a_hash] = fft(a) > if b_hash not in _cache: > _cache[b_hash] = fft(b) > return ifft(_cache[a_hash] * _cache[b_hash]) > > > > > A quick though on an implementation detail... > > I'm not sure exactly how to deal with the counter overflowing: perhaps > if you treated counter==0 to mean not-writable (i.e. that would be the > new version of the old write flag) then you might get some > uint-wraparound checking for free (because when it wraps back around > to zero the buffer ends up becoming locked)? Alternatively you could > just say that no guarantee is given of wraparound being caught..though > that might seriously impact on the range of possible uses. > > > In summary... > Hopefully the stuff needed to make __mutablehash__ work could be > implemented simply by adding a single extra operation to the > write-check (and maybe changing the footprint of the ndarray slightly > to accomodate a counter). But I suspect someone will tell me that > life is never that simple! > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 819 bytes Desc: This is a digitally signed message part URL: From archibald at astron.nl Fri Sep 11 09:32:16 2015 From: archibald at astron.nl (Anne Archibald) Date: Fri, 11 Sep 2015 13:32:16 +0000 Subject: [Numpy-discussion] feature request - increment counter on write check In-Reply-To: <1441977628.1847.3.camel@sipsolutions.net> References: <1441977628.1847.3.camel@sipsolutions.net> Message-ID: On Fri, Sep 11, 2015 at 3:20 PM Sebastian Berg wrote: > On Fr, 2015-09-11 at 13:10 +0000, Daniel Manson wrote: > > Originally posted as issue 6301 on github. > > > > > > Presumably any block of code that modifies an ndarray's buffer is > > wrapped in a (thread safe?) check of the writable flag. Would it be > > possible to hold a counter rather than a simple bool flag and then > > increment the counter whenever you test the flag? Hopefully this would > > introduce only a tiny additional overhead, but would permit > > pseudo-hashing to test whether a mutable array has changed since you > > last encountered it. > > > > Just a quick note. This is a bit more complex then it might appear. The > reason being that when a view of the array is changed, you would have to > "notify" the array itself that it has changed. So propagation from top > to bottom does not seem straight forward to me. (the other way is fine, > since on check you could check all parents, but you cannot check all > children). > Actually not so much. Like the writable flag, you'd make the counter be a per-buffer piece of information. Each array already has a pointer to the array object that "owns" the buffer, so you'd just go there in one hop. This does mean that modifying one view would affect the modified flag on all views sharing the same buffer, whether there's data overlap or not, but for caching purposes that's not so bad. I think a more serious concern is that it may be customary to simply check the writable flag by hand rather than calling an is_writable function, so to make this idea work you'd have to change all code that checks the writable flag, including user code. You'd also have to make sure that all code that tries to write to an array really checks the writable flag. Rather than making this happen for all arrays, does it make sense to use an array subclass with a "dirty flag", maybe even if this requires manual setting in some cases? Anne -------------- next part -------------- An HTML attachment was scrubbed... URL: From d.manson.11 at ucl.ac.uk Fri Sep 11 09:00:55 2015 From: d.manson.11 at ucl.ac.uk (Daniel) Date: Fri, 11 Sep 2015 06:00:55 -0700 (MST) Subject: [Numpy-discussion] feature request - increment counter on write check In-Reply-To: <1441977628.1847.3.camel@sipsolutions.net> References: <1441977628.1847.3.camel@sipsolutions.net> Message-ID: <1441976455133-41018.post@n7.nabble.com> >...you would have to "notify" the array itself that it has changed.. I don't get what you mean..or maybe I do...? Do you mean that if there are two arrays with non-overlapping views of the same data then modifying one will change the mutablehash of both? That's true I guess, and I hadn't considered it, but I don't think it matters hugely because inequality of mutablehashes is not supposed to imply inequality of data. -- View this message in context: http://numpy-discussion.10968.n7.nabble.com/feature-request-increment-counter-on-write-check-tp41015p41018.html Sent from the Numpy-discussion mailing list archive at Nabble.com. From sebastian at sipsolutions.net Fri Sep 11 09:47:02 2015 From: sebastian at sipsolutions.net (Sebastian Berg) Date: Fri, 11 Sep 2015 15:47:02 +0200 Subject: [Numpy-discussion] feature request - increment counter on write check In-Reply-To: <1441976455133-41018.post@n7.nabble.com> References: <1441977628.1847.3.camel@sipsolutions.net> <1441976455133-41018.post@n7.nabble.com> Message-ID: <1441979222.4962.11.camel@sipsolutions.net> On Fr, 2015-09-11 at 06:00 -0700, Daniel wrote: > >...you would have to "notify" the array itself that it has changed.. > > I don't get what you mean..or maybe I do...? Do you mean that if there are > two arrays with non-overlapping views of the same data then modifying one > will change the mutablehash of both? That's true I guess, and I hadn't > considered it, but I don't think it matters hugely because inequality of > mutablehashes is not supposed to imply inequality of data. > > I just meant you need to propagate the "changed" flag to the parent, which yes, means that you also propagate it down again (when testing if a change occured) to other non-overlapping children (which is not a problem). As Anne said, propagating to the parents usually means only the ultimate base (there are some exceptions for subclasses IIRC), so it might be fine. Though there will be failures cases if the ultimate base is not a numpy array. As Anne suggested, I think the writable flag is actually a flag, meaning there is no room for a counter there. There should be room for a "dirty" flag. Another issue is, which may be a non-issue, that this design is inherently not quite thread safe. Anyway, I do not know whether or not it is doable or makes sense in numpy right now, just some things to think about. - Sebastian > > -- > View this message in context: http://numpy-discussion.10968.n7.nabble.com/feature-request-increment-counter-on-write-check-tp41015p41018.html > Sent from the Numpy-discussion mailing list archive at Nabble.com. > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 819 bytes Desc: This is a digitally signed message part URL: From d.manson.11 at ucl.ac.uk Fri Sep 11 09:34:16 2015 From: d.manson.11 at ucl.ac.uk (Daniel) Date: Fri, 11 Sep 2015 06:34:16 -0700 (MST) Subject: [Numpy-discussion] feature request - increment counter on write check In-Reply-To: References: <1441977628.1847.3.camel@sipsolutions.net> Message-ID: <1441978456237-41020.post@n7.nabble.com> Anne Archibald-3 wrote > ... You'd also have to make sure that all code that tries to write to an > array really checks the writable flag. If there is code out there which modifies arrays without checking the write flag then it's already on buggy-territory. In terms of value vs functions, surely adding an extra pair of empty parens is not going to be a big deal, although yes it would be a breaking change? The advantage of piggybacking on the write-check is that it doesn't really matter how liberal the original author was with the write checks - you can increment the counter as many times as you like, though removing unnecessary increments would be vaguely helpful if this does end up being implemented as I suggest - I suppose you could also offer an optional keep_it_clean flag in the is_writeable call if you really wanted. > Rather than making this happen for all arrays, does it make sense to use > an > array subclass with a "dirty flag", maybe even if this requires manual > setting in some cases? I have no idea what the reality is of trying to subclass ndarray, neither at the C level nor at the python level, but perhaps this wouldn't be too bad. I just thought that you could do such a neat job down at the C level it would be nice to make it happen. Also, regarding "dirty flags", it's harder to imagine that working with the examples I gave since their may be more than one "consumers" of the hash and thus an array could be dirty from the point of view of one consumer, but clean from the point of view of another. Sebastian wrote > Another issue is, which may be a non-issue, that this design is > inherently not quite thread safe. In terms of thread safety, I guess it's a bit complicated...I'd hoped that it would be covered by the same logic as the simple write check - although now I think about it again it isn't that simple. It's not really something I know anything about in the context of numpy, so I guess I shouldn't try and make any further wild assertions! -- View this message in context: http://numpy-discussion.10968.n7.nabble.com/feature-request-increment-counter-on-write-check-tp41015p41020.html Sent from the Numpy-discussion mailing list archive at Nabble.com. From njs at pobox.com Fri Sep 11 13:43:31 2015 From: njs at pobox.com (Nathaniel Smith) Date: Fri, 11 Sep 2015 10:43:31 -0700 Subject: [Numpy-discussion] feature request - increment counter on write check In-Reply-To: References: <1441977628.1847.3.camel@sipsolutions.net> Message-ID: On Sep 11, 2015 6:35 AM, "Anne Archibald" wrote: > > > > On Fri, Sep 11, 2015 at 3:20 PM Sebastian Berg wrote: >> >> On Fr, 2015-09-11 at 13:10 +0000, Daniel Manson wrote: >> > Originally posted as issue 6301 on github. >> > >> > >> > Presumably any block of code that modifies an ndarray's buffer is >> > wrapped in a (thread safe?) check of the writable flag. Would it be >> > possible to hold a counter rather than a simple bool flag and then >> > increment the counter whenever you test the flag? Hopefully this would >> > introduce only a tiny additional overhead, but would permit >> > pseudo-hashing to test whether a mutable array has changed since you >> > last encountered it. >> > >> >> Just a quick note. This is a bit more complex then it might appear. The >> reason being that when a view of the array is changed, you would have to >> "notify" the array itself that it has changed. So propagation from top >> to bottom does not seem straight forward to me. (the other way is fine, >> since on check you could check all parents, but you cannot check all >> children). > > > Actually not so much. Like the writable flag, you'd make the counter be a per-buffer piece of information. Each array already has a pointer to the array object that "owns" the buffer, so you'd just go there in one hop. This does mean that modifying one view would affect the modified flag on all views sharing the same buffer, whether there's data overlap or not, but for caching purposes that's not so bad. Weirdly, the writeable flag is maintained on a per-view basis -- you can have a read only view of a writeable array, or even vice-versa (e.g. if you set the base array to be read-only after creating the view). This doesn't matter for this purpose though -- it just means that if you want to detect array changes, you can't assume that read-only arrays never change. > I think a more serious concern is that it may be customary to simply check the writable flag by hand rather than calling an is_writable function, so to make this idea work you'd have to change all code that checks the writable flag, including user code. There is PyArray_FailUnlessWriteable (which must be called with the GIL held, because it can raise an exception, so thread safety is OK). It's used consistently inside numpy itself these days. But that's new api in 1.6, so yeah, there is surely a ton of legacy code out there that is doing its own checking. I guess we'd have to somehow deprecate PyArray_ISWRITEABLE and other methods of direct flag checking, and wait for people to switch over? There are some arguments for doing this anyway I suppose... though it would certainly be a hassle and take a while. > You'd also have to make sure that all code that tries to write to an array really checks the writable flag. For purposes of something like updating Spyder's gui, I guess it might be OK if buggy code that doesn't check the RO flag is also buggy in that it fails to update the GUI? Maybe it would even cause people to finally notice the bug and get it fixed? Agreed that you certainly could not rely on this to be 100% accurate in practice though :-/ > Rather than making this happen for all arrays, does it make sense to use an array subclass with a "dirty flag", maybe even if this requires manual setting in some cases? I think the problem with this would be that you can have an ndarray view of your special subclass, or vice-versa, and code that goes via the regular ndarray will not update your flag. And unfortunately that includes most numpy functions implicitly (e.g. np.sin(a, out=b) doesn't much care whether b is a subclass, it's just going to blindly do its thing). Actually, now that I think about it there is a much worse general version of this problem, which probably kills the idea dead. Via the buffer interface (among others), it is totally legal and encouraged to create non-numpy views of numpy arrays. They'll check the flags once at creation time, but after that they're free to write directly to the underlying buffer whenever they please: a = np.ones(10) a2 = memoryview(a) a2[0] = 0 # now what? -n -------------- next part -------------- An HTML attachment was scrubbed... URL: From jeffreback at gmail.com Fri Sep 11 13:49:35 2015 From: jeffreback at gmail.com (Jeff Reback) Date: Fri, 11 Sep 2015 13:49:35 -0400 Subject: [Numpy-discussion] ANN: pandas v0.17.0rc1 - RELEASE CANDIDATE Message-ID: Hi, I'm pleased to announce the availability of the first release candidate of Pandas 0.17.0. Please try this RC and report any issues here: Pandas Issues We will be releasing officially in 1-2 weeks or so. **RELEASE CANDIDATE 1** This is a major release from 0.16.2 and includes a small number of API changes, several new features, enhancements, and performance improvements along with a large number of bug fixes. We recommend that all users upgrade to this version. Highlights include: - Release the Global Interpreter Lock (GIL) on some cython operations, see here - Plotting methods are now available as attributes of the .plot accessor, see here - The sorting API has been revamped to remove some long-time inconsistencies, see here - Support for a datetime64[ns] with timezones as a first-class dtype, see here - The default for to_datetime will now be to raise when presented with unparseable formats, previously this would return the original input, see here - The default for dropna in HDFStore has changed to False, to store by default all rows even if they are all NaN, see here - Support for Series.dt.strftime to generate formatted strings for datetime-likes, see here - Development installed versions of pandas will now have PEP440 compliant version strings GH9518 - Development support for benchmarking with the Air Speed Velocity library GH8316 - Support for reading SAS xport files, see here - Removal of the automatic TimeSeries broadcasting, deprecated since 0.8.0, see here See the Whatsnew for much more information. Best way to get this is to install via conda from our development channel. Builds for osx-64,linux-64,win-64 for Python 2.7 and Python 3.4 are all available. conda install pandas -c pandas Thanks to all who made this release happen. It is a very large release! Jeff -------------- next part -------------- An HTML attachment was scrubbed... URL: From d.manson.11 at ucl.ac.uk Fri Sep 11 14:29:54 2015 From: d.manson.11 at ucl.ac.uk (Daniel) Date: Fri, 11 Sep 2015 11:29:54 -0700 (MST) Subject: [Numpy-discussion] feature request - increment counter on write check In-Reply-To: References: <1441977628.1847.3.camel@sipsolutions.net> Message-ID: <1441996194394-41023.post@n7.nabble.com> Nathaniel Smith wrote > Actually, now that I think about it there is a much worse general version > of this problem, which probably kills the idea dead. Via the buffer > interface (among others), it is totally legal and encouraged to create > non-numpy views of numpy arrays. They'll check the flags once at creation > time, but after that they're free to write directly to the underlying > buffer whenever they please: > > a = np.ones(10) > a2 = memoryview(a) > a2[0] = 0 > # now what? You could say that when you create a memoryview you set a to being non-mutablehash-able. Ideally you would want a C++ RAII-like/python with-block style implementation so that when the memoryview is destroyed the mutablehash-able-ness is reinstated (although I guess you would need a ref count of memoryviews to do that safely). I don't know how common memoryview is within numpy itself or whether it's only part of the extension API? If it's relatively rare, then given that this hashing thing could easily be designed to fail gracefully it would seem a shame not to implement it simply because of this issue...in my opinion of course! P.S. I had no idea about the view-local write-lock...that's rather a surprise. -- View this message in context: http://numpy-discussion.10968.n7.nabble.com/feature-request-increment-counter-on-write-check-tp41015p41023.html Sent from the Numpy-discussion mailing list archive at Nabble.com. From pizza at netspace.net.au Mon Sep 7 04:43:57 2015 From: pizza at netspace.net.au (pizza at netspace.net.au) Date: Mon, 07 Sep 2015 18:43:57 +1000 Subject: [Numpy-discussion] review of #6247 In-Reply-To: <1562003.6iNdAildBC@minipizza> References: <1562003.6iNdAildBC@minipizza> Message-ID: <4259495.ymQv3CekSj@minipizza> On Sun, 6 Sep 2015 10:03:16 PM pizza at netspace.net.au wrote: > On Sat, 5 Sep 2015 10:28:26 PM pizza at netspace.net.au wrote: > > Hi all, first attempt at this: > > small doco patch to > > address https://github.com/numpy/numpy/issues/6247 > > https://github.com/numpy/numpy/compare/master...pizzathief:issue6247?diff= > > un ified&expand=1&name=issue6247 > > Fixed up next sentence as well. > > https://github.com/numpy/numpy/compare/master...pizzathief:issue6247?diff=un > ified&expand=1&name=issue6247 it was pointed out to me that numpy/lib/npyio.py needed updating as well. https://github.com/numpy/numpy/compare/master...pizzathief:issue6247?diff=unified&expand=1&name=issue6247 (I'm just repeating the same url, aren't I, sigh) From pizza at netspace.net.au Sat Sep 12 00:56:08 2015 From: pizza at netspace.net.au (jason) Date: Sat, 12 Sep 2015 14:56:08 +1000 Subject: [Numpy-discussion] review of #6191 Message-ID: <5136964.rM9CB8ZY2a@minipizza> doco patch to address https://github.com/numpy/numpy/issues/6191 https://github.com/numpy/numpy/compare/master...pizzathief:issue6191 From njs at pobox.com Sat Sep 12 01:15:49 2015 From: njs at pobox.com (Nathaniel Smith) Date: Fri, 11 Sep 2015 22:15:49 -0700 Subject: [Numpy-discussion] review of #6191 In-Reply-To: <5136964.rM9CB8ZY2a@minipizza> References: <5136964.rM9CB8ZY2a@minipizza> Message-ID: Can you submit this as a pull request? That's how we normally review patches. On Sep 11, 2015 21:59, "jason" wrote: > doco patch to address https://github.com/numpy/numpy/issues/6191 > > https://github.com/numpy/numpy/compare/master...pizzathief:issue6191 > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Sat Sep 12 01:15:55 2015 From: njs at pobox.com (Nathaniel Smith) Date: Fri, 11 Sep 2015 22:15:55 -0700 Subject: [Numpy-discussion] review of #6247 In-Reply-To: <4259495.ymQv3CekSj@minipizza> References: <1562003.6iNdAildBC@minipizza> <4259495.ymQv3CekSj@minipizza> Message-ID: Please submit this to a pull request as well :-) On Sep 11, 2015 21:56, wrote: > On Sun, 6 Sep 2015 10:03:16 PM pizza at netspace.net.au wrote: > > On Sat, 5 Sep 2015 10:28:26 PM pizza at netspace.net.au wrote: > > > Hi all, first attempt at this: > > > small doco patch to > > > address https://github.com/numpy/numpy/issues/6247 > > > > https://github.com/numpy/numpy/compare/master...pizzathief:issue6247?diff= > > > un ified&expand=1&name=issue6247 > > > > Fixed up next sentence as well. > > > > > https://github.com/numpy/numpy/compare/master...pizzathief:issue6247?diff=un > > ified&expand=1&name=issue6247 > > it was pointed out to me that numpy/lib/npyio.py needed updating as well. > > > https://github.com/numpy/numpy/compare/master...pizzathief:issue6247?diff=unified&expand=1&name=issue6247 > (I'm just repeating the same url, aren't I, sigh) > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From travis at continuum.io Sat Sep 12 17:51:51 2015 From: travis at continuum.io (Travis Oliphant) Date: Sat, 12 Sep 2015 16:51:51 -0500 Subject: [Numpy-discussion] Looking for a developer who will work with me for at least 6 months to fix NumPy's dtype system. Message-ID: Hi all, Apologies for cross-posting, but I need to get the word out and twitter doesn't provide enough explanation. I've been working on a second edition of my "Guide to NumPy" book. It's been a time-pressured activity, but it's helped me put more meat around my ideas for how to fix NumPy's dtype system -- which I've been contemplating off an on for 8 years. I'm pretty sure I know exactly how to do it --- in a way that fits more cleanly into Python. It will take 3-6 months and will have residual efforts needed that will last another 6 months --- making more types available with NumPy, improving calculations etc. This work will be done completely in public view and allow for public comment. It will not solve *all* of NumPy's problems, but it will put NumPy's dtype system on the footing it in retrospect should have been put on in the first place (if I had known then what I know now). It won't be a grandiose rewrite. It will be a pretty surgical fix to a few key places in the code. However, it will break the ABI and require recompilation of NumPy extensions (and so would need to be called NumPy 2.0). This is unavoidable, but I don't see any problem with breaking the ABI today given how easy it is to get distributions of Python these days from a variety of sources (including using conda --- but not only using conda). For those that remember what happened in Python dev land, the changes will be similar to when Guido changed Python 1.5.2 to Python 2.0. I can mentor and work closely with someone who will work on this and we will invite full participation and feedback from whomever in the community also wants to participate --- but I can't do it myself full time (and it needs someone full time+). Fortunately, I can pay someone to do it if they are willing to commit at least 6 months (it is not required to work at Continuum for this, but you can have a job at Continuum if you want one). I'm only looking for people who have enough experience with C or preferably the Python C-API. You also have to *want* to work on this. You need to be willing to work with me on the project directly and work to have a mind-meld with my ideas which will undoubtedly give rise to additional perspectives and ideas for later work for you. When I wrote NumPy 1.0, I put in 80+ hour weeks for about 6 months or more and then 60+ weeks for another year. I was pretty obsessed with it. This won't need quite that effort, but it will need something like it. Being able to move to Austin is a plus but not required. I can sponsor a visa for the right candidate as well (though it's not guaranteed you will get one with the immigration policies what they are). This is a labor of love for so many of us and my desire to help the dtype situation in NumPy comes from the same space that my desire to work on NumPy in the first place came. I will be interviewing people to work on this as not everyone who may want to will really be qualified to do it --- especially with so many people writing Cython these days instead of good-ole C-API code :-) Feel free to spread the news to anyone you can. I won't say more until I've found someone to work with me on this --- because I won't have the time to follow-up with any questions or comments. Even if I can't find someone I will publish the ideas --- but that also takes time and effort that is in short supply for me right now. If there is someone willing to fund this work, please let me know as well -- that could free up more of my time. Best, -Travis -- *Travis Oliphant* *Co-founder and CEO* @teoliphant 512-222-5440 http://www.continuum.io -------------- next part -------------- An HTML attachment was scrubbed... URL: From travis at continuum.io Sat Sep 12 20:01:53 2015 From: travis at continuum.io (Travis Oliphant) Date: Sat, 12 Sep 2015 19:01:53 -0500 Subject: [Numpy-discussion] Just to this list --- more details of the approach Message-ID: Hey all, To the NumPy list only, I'll at least give the highlights of the surgical approach I would like to get someone to work on -- I can help mentor and guide. These are just the highlights, but it should give someone familiar with the code the general gist. There are some details to work out, of course, but it could be done. It may be very similar to what Nathaniel is contemplating --- except I think breaking the ABI is the only way to really do this --- could be wrong but I'm not wiling to risk *not* just breaking the ABI. 1) Create a new meta-type in C (call it dtype) 2) Create Python Classes (in C) that are instances of this meta-type for each "kind" of data-type 3) Make PyArray_Descr * be a reference to one of these new objects (which can be built either in C or Python) and should be published outside NumPy as well. 4) Remove most of the "per-type function calls" in PyArray_ArrFuncs --- instead replacing those with the Generalized Ufunc equivalents and expand the capability of Generalized Ufuncs 5) Keep the Array Scalar Types but change them so that they also use the dtype meta-type as their foundation and mixin an array-methods type. Also, have these be in a separate project from NumPy itself. 6) The current void* would be replaced with real Python classes instead of structured arrays being shoved through a single data-type. 7) The documented ways to spell a dtype would be reduced --- but backwards compatibility would be preserved. 8) Make sure Numba can create these Descriptor objects with Ahead of Time Compilation and start to move code of NumPy to Numba 9) Ensure the Generalized Ufunc framework can take the data-type as an argument so that *all* data-types can participate in the general multi-method approach. There is more to it, but that is the basic idea. Please forgive me if I can't respond to any feedback from the list in a timely way. I will as I can. -Travis -- *Travis Oliphant* *Co-founder and CEO* @teoliphant 512-222-5440 http://www.continuum.io -------------- next part -------------- An HTML attachment was scrubbed... URL: From travis at continuum.io Sun Sep 13 18:51:12 2015 From: travis at continuum.io (Travis Oliphant) Date: Sun, 13 Sep 2015 17:51:12 -0500 Subject: [Numpy-discussion] The process I intend to follow for any proposed changes to NumPy Message-ID: Hey all, I just wanted to clarify, that I am very excited about a few ideas I have --- but I don't have time myself to engage in the community process to get these changes into NumPy. However, those are real processes --- I've been coaching a few people in those processes for the past several years already. So, rather than do nothing, what I'm looking to do is to work with a few people who I can share my ideas with, get excited about the ideas, and then who will work with the community to get them implemented. That's what I was announcing and talking about yesterday --- looking for interested people who want to work on NumPy *with* the NumPy community. In my enthusiasm, I realize that some may have mis-understood my intention. There is no 'imminent' fork, nor am I planning on doing some crazy amount of work that I then try to force on other developers of NumPy. What I'm planning to do is find people to train on NumPy code base (people to increase the diversity of the developers would be ideal -- but hard to accomplish). I plan to train them on NumPy based on my experience, and on what I think should be done --- and then have *them* work through the community process and engage with others to get consensus (hopefully not losing too much in translation in the process --- but instead getting even better). During that process I will engage as a member of the community and help write NEPs and other documents and help clarify where it makes sense as I can. I will be filtering for people that actually want to see NumPy get better. Until I identify the people and work with them, it will be hard to tell how this will best work. So, stay tuned. If all goes well, what you should see in a few weeks time are specific proposals, a branch or two, and the beginnings of some pull requests. If you don't see that, then I will not have found the right people to help me, and we will all continue to go back to searching. While I'm expecting the best, in the worst case, we get additional people who know the NumPy code base and can help squash bugs as well as implement changes that are desired. Three things are needed if you want to participate in this: 1) A willingness to work with the open source community, 2) a deep knowledge of C and in-particular CPython's brand of C, and 3) a willingness to engage with me, do a mind-meld and dump around the NumPy code base, and then improve on what is in my head with the rest of the community. Thanks, -Travis -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Sun Sep 13 22:54:51 2015 From: njs at pobox.com (Nathaniel Smith) Date: Sun, 13 Sep 2015 19:54:51 -0700 Subject: [Numpy-discussion] [Python-ideas] Should our default random number generator be secure? In-Reply-To: References: <1441821254.2853664.379081313.43B3886D@webmail.messagingengine.com> <1441824901.2867280.379140585.4F25667D@webmail.messagingengine.com> <20150909190757.GM19373@ando.pearwood.info> <20150910015505.GO19373@ando.pearwood.info> Message-ID: [This is getting fairly off-topic for python-ideas (since AFAICT there is no particular reason right now to add a new deterministic generator to the stdlib), so CC'ing numpy-discussion and I'd suggest followups be directed to there alone.] On Thu, Sep 10, 2015 at 10:41 AM, Robert Kern wrote: > On 2015-09-10 04:56, Nathaniel Smith wrote: >> >> On Wed, Sep 9, 2015 at 8:35 PM, Tim Peters wrote: >>> >>> There are some clean and easy approaches to this based on >>> crypto-inspired schemes, but giving up crypto strength for speed. If >>> you haven't read it, this paper is delightful: >>> >>> http://www.thesalmons.org/john/random123/papers/random123sc11.pdf >> >> >> It really is! As AES acceleration instructions become more common >> (they're now standard IIUC on x86, x86-64, and even recent ARM?), even >> just using AES in CTR mode becomes pretty compelling -- it's fast, >> deterministic, provably equidistributed, *and* cryptographically >> secure enough for many purposes. > > > I'll also recommend the PCG paper (and algorithm) as the author's > cross-PRNGs comparisons have been bandied about in this thread already. The > paper lays out a lot of the relevant issues and balances the various > qualities that are important: statistical quality, speed, and security (of > various flavors). > > http://www.pcg-random.org/paper.html > > I'm actually not that impressed with Random123. The core idea is nice and > clean, but the implementation is hideously complex. I'm curious what you mean by this? In terms of the code, or...? I haven't looked at the code, but the simplest generator they recommend in the paper is literally just def rng_stream(seed): counter = 0 while True: # AES128 takes a 128 bit key and 128 bits of data and returns 128 bits of encrypted data val = AES128(key=seed, data=counter) yield low_64_bits(val) yield high_64_bits(val) counter += 1 which gives a 64-bit generator with a period of 2^129. They benchmark it as faster than the Mersenne Twister on modern CPUs (<2 cycles-per-byte on recent x86, x86-64, ARMv8), it requires less scratch space, is incredibly simple to work with -- you can parallelize it, get independent random streams, etc., in a totally trivial way -- and has a *way* stronger guarantee of random-looking-ness than merely passing TestU01. The downsides are that it does still require 176 bytes of read-only scratch storage (used to cache an expanded version of the "key"), the need for a modern CPU to get that speed, and that it doesn't play well with GPUs. So they also provide a set of three more ad hoc generators designed to solve these problems. I'm not as convinced about these, but hey, they pass TestU01... The PCG paper does a much better job of all the other stuff *around* making a good RNG -- it has the nice website, clear comparisons, nice code, etc. -- which is definitely important. But to me the increase in speed and reduction in memory use doesn't seem worth it given how fast these generators are to start with + the nice properties of counter mode + and cryptographic guarantees of randomness that you get from AES, for code that's generally targeting non-embedded non-GPU systems. -n -- Nathaniel J. Smith -- http://vorpus.org From matthew.brett at gmail.com Mon Sep 14 04:18:35 2015 From: matthew.brett at gmail.com (Matthew Brett) Date: Mon, 14 Sep 2015 01:18:35 -0700 Subject: [Numpy-discussion] OK to upload patched 1.9.2 for Python 3.5? Message-ID: Hi, I'm just building numpy 1.9.2 for Python 3.5 (just released). In order to get the tests to pass on Python 3.5, I need to cherry pick commit 7d6aa8c onto the 1.9.2 tag position. Does anyone object to me uploading a wheel built from this patched version to pypi as 1.9.2 for Python 3.5 on OSX? It would help to get the ball rolling for Python 3.5 binary wheels. Cheers, Matthew From cournape at gmail.com Mon Sep 14 04:22:36 2015 From: cournape at gmail.com (David Cournapeau) Date: Mon, 14 Sep 2015 09:22:36 +0100 Subject: [Numpy-discussion] OK to upload patched 1.9.2 for Python 3.5? In-Reply-To: References: Message-ID: On Mon, Sep 14, 2015 at 9:18 AM, Matthew Brett wrote: > Hi, > > I'm just building numpy 1.9.2 for Python 3.5 (just released). > > In order to get the tests to pass on Python 3.5, I need to cherry pick > commit 7d6aa8c onto the 1.9.2 tag position. > > Does anyone object to me uploading a wheel built from this patched > version to pypi as 1.9.2 for Python 3.5 on OSX? It would help to get > the ball rolling for Python 3.5 binary wheels. > Why not releasing this as 1.9.3 ? It does not need to be a full release (with binaries and all), but having multiple sources for a given tag is confusing. David > > Cheers, > > Matthew > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Mon Sep 14 04:26:58 2015 From: robert.kern at gmail.com (Robert Kern) Date: Mon, 14 Sep 2015 09:26:58 +0100 Subject: [Numpy-discussion] [Python-ideas] Should our default random number generator be secure? In-Reply-To: References: <1441821254.2853664.379081313.43B3886D@webmail.messagingengine.com> <1441824901.2867280.379140585.4F25667D@webmail.messagingengine.com> <20150909190757.GM19373@ando.pearwood.info> <20150910015505.GO19373@ando.pearwood.info> Message-ID: [Tim, ping me if you want to get dropped from the reply chain, as we are liable to get more into numpy decision-making. I've dropped python-ideas.] On Mon, Sep 14, 2015 at 4:34 AM, Tim Peters wrote: > > [Robert Kern ] > >> ... > >> I'll also recommend the PCG paper (and algorithm) as the author's > >> cross-PRNGs comparisons have been bandied about in this thread already. The > >> paper lays out a lot of the relevant issues and balances the various > >> qualities that are important: statistical quality, speed, and security (of > >> various flavors). > >> > >> http://www.pcg-random.org/paper.html > >> > >> I'm actually not that impressed with Random123. The core idea is nice and > >> clean, but the implementation is hideously complex. > > [Nathaniel Smith ] > > I'm curious what you mean by this? In terms of the code, or...? In terms of the code that would be necessary to implement this in a simultaneously performant and cross-platform way in numpy.random. > > I haven't looked at the code, I recommend it! Approach it with the practical goal of "how do I stick this in as a new core PRNG for numpy.random?" > > but the simplest generator they > > recommend in the paper is literally just > > > > def rng_stream(seed): > > counter = 0 > > while True: > > # AES128 takes a 128 bit key and 128 bits of data and returns > > 128 bits of encrypted data > > val = AES128(key=seed, data=counter) > > yield low_64_bits(val) > > yield high_64_bits(val) > > counter += 1 > > I assume it's because if you expand what's required to _implement_ > AES128() in C, it does indeed look pretty hideously complex. On HW > implementing AES primitives, of course the code can be much simpler. > > But to be fair, if integer multiplication and/or addition weren't > implemented in HW, and we had to write to C code emulating them via > bit-level fiddling, the code for any of the PCG algorithms would look > hideously complex too. > > But being fair isn't much fun ;-) Actually, I meant all of the crap *around* it, the platform-compatibility testing to see if you have such a hardware instruction or not, and C++ template shenanigans in the surrounding code. It's possible that the complexity is only due to flexibility, but it was too complex for me to begin understanding *why* it's so complex before I succumbed to ennui and moved on to some other productive use of my free time. At least some of the complexity is due to needing software implementations of reduced-round crypto for decent performance in the absence of the hardware instruction. Performing well in the absence of the hardware instruction is very important to me as I do not seem to have the AES-NI instruction available on my mid-2012 Macbook Pro. Exposing counter-mode AES128 as a core PRNG is a nice idea, but it's just low on my wishlist. I want fast, multiple independent streams on my current hardware first, and PCG gives that to me. On the "if I had to implement all of the bit-fiddling myself" count, I'd still prefer PCG. It has a similar problem with 128-bit integers that may or may not be natively provided (should you want a 128-bit state), and I have to say that the software implementation of that bit-fiddling is much nicer to work with than reduced-round Threefry. It's reference implementation is also overly complex for practical use as it is showing off the whole parameterized PCG family to support the paper, even the explicitly unrecommended members. But it's relatively straightforward to disentangle the desirable members; their implementations, including the supporting 128-bit arithmetic code, are small and clean. -- Robert Kern -------------- next part -------------- An HTML attachment was scrubbed... URL: From matthew.brett at gmail.com Mon Sep 14 04:32:13 2015 From: matthew.brett at gmail.com (Matthew Brett) Date: Mon, 14 Sep 2015 01:32:13 -0700 Subject: [Numpy-discussion] OK to upload patched 1.9.2 for Python 3.5? In-Reply-To: References: Message-ID: Hi, On Mon, Sep 14, 2015 at 1:22 AM, David Cournapeau wrote: > > > On Mon, Sep 14, 2015 at 9:18 AM, Matthew Brett > wrote: >> >> Hi, >> >> I'm just building numpy 1.9.2 for Python 3.5 (just released). >> >> In order to get the tests to pass on Python 3.5, I need to cherry pick >> commit 7d6aa8c onto the 1.9.2 tag position. >> >> Does anyone object to me uploading a wheel built from this patched >> version to pypi as 1.9.2 for Python 3.5 on OSX? It would help to get >> the ball rolling for Python 3.5 binary wheels. > > > Why not releasing this as 1.9.3 ? It does not need to be a full release > (with binaries and all), but having multiple sources for a given tag is > confusing. Generally OK with me, but it's quite a bit of extra work for very little gain. We'd have to tag, release a source tarball and OSX wheels, at least. The patch being cherry-picked is just deleting some legacy monkey-patching of gzip: https://github.com/numpy/numpy/commit/7d6aa8c721d5274ac57d0c87685d472cb1fd7948 and it's difficult for me to imagine this could cause a problem with source differences. Cheers, Matthew From solipsis at pitrou.net Mon Sep 14 04:34:31 2015 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 14 Sep 2015 10:34:31 +0200 Subject: [Numpy-discussion] [Python-ideas] Should our default random number generator be secure? References: <20150909190757.GM19373@ando.pearwood.info> <20150910015505.GO19373@ando.pearwood.info> Message-ID: <20150914103431.7d6b43fd@fsol> On Mon, 14 Sep 2015 09:26:58 +0100 Robert Kern wrote: > > Actually, I meant all of the crap *around* it, the platform-compatibility > testing to see if you have such a hardware instruction or not, and C++ > template shenanigans in the surrounding code. It's possible that the > complexity is only due to flexibility, but it was too complex for me to > begin understanding *why* it's so complex before I succumbed to ennui and > moved on to some other productive use of my free time. At least some of the > complexity is due to needing software implementations of reduced-round > crypto for decent performance in the absence of the hardware instruction. > Performing well in the absence of the hardware instruction is very > important to me as I do not seem to have the AES-NI instruction available > on my mid-2012 Macbook Pro. Exposing counter-mode AES128 as a core PRNG is > a nice idea, but it's just low on my wishlist. I want fast, multiple > independent streams on my current hardware first, and PCG gives that to me. Using AES also means emulating it on a GPU will be quite hard. For the record, Numba is currently using a Mersenne Twister on the CPU, to emulate Numpy's behaviour (although some of our distributions may be different): >>> def f(x): ... np.random.seed(x) ... l = [] ... for i in range(10): l.append(np.random.random()) ... return l ... >>> g = numba.jit(nopython=True)(f) >>> f(10) [0.771320643266746, 0.0207519493594015, 0.6336482349262754, 0.7488038825386119, 0.4985070123025904, 0.22479664553084766, 0.19806286475962398, 0.7605307121989587, 0.16911083656253545, 0.08833981417401027] >>> g(10) [0.771320643266746, 0.0207519493594015, 0.6336482349262754, 0.7488038825386119, 0.4985070123025904, 0.22479664553084766, 0.19806286475962398, 0.7605307121989587, 0.16911083656253545, 0.08833981417401027] Currently we don't provide those APIs on the GPU, since MT is much too costly there. If Numpy wanted to switch to a different generator, and if Numba wanted to remain compatible with Numpy, one of the PCG functions would be an excellent choice (also for CPU performance, incidentally). Regards Antoine. From solipsis at pitrou.net Mon Sep 14 04:36:39 2015 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 14 Sep 2015 10:36:39 +0200 Subject: [Numpy-discussion] OK to upload patched 1.9.2 for Python 3.5? References: Message-ID: <20150914103639.18bd26ea@fsol> On Mon, 14 Sep 2015 01:32:13 -0700 Matthew Brett wrote: > >> > >> Does anyone object to me uploading a wheel built from this patched > >> version to pypi as 1.9.2 for Python 3.5 on OSX? It would help to get > >> the ball rolling for Python 3.5 binary wheels. > > > > Why not releasing this as 1.9.3 ? It does not need to be a full release > > (with binaries and all), but having multiple sources for a given tag is > > confusing. > > Generally OK with me, but it's quite a bit of extra work for very > little gain. We'd have to tag, release a source tarball and OSX > wheels, at least. It's always bad to have two silent versions of a single release. People can never know up front which version they got, since it's not written anywhere. That's why the right policy is to bump the version number in some way (be it by incrementing it or adding a ".patchXXX" at the end). Regards Antoine. From robert.kern at gmail.com Mon Sep 14 04:46:28 2015 From: robert.kern at gmail.com (Robert Kern) Date: Mon, 14 Sep 2015 09:46:28 +0100 Subject: [Numpy-discussion] OK to upload patched 1.9.2 for Python 3.5? In-Reply-To: References: Message-ID: On Mon, Sep 14, 2015 at 9:32 AM, Matthew Brett wrote: > > Hi, > > On Mon, Sep 14, 2015 at 1:22 AM, David Cournapeau wrote: > > > > > > On Mon, Sep 14, 2015 at 9:18 AM, Matthew Brett > > wrote: > >> > >> Hi, > >> > >> I'm just building numpy 1.9.2 for Python 3.5 (just released). > >> > >> In order to get the tests to pass on Python 3.5, I need to cherry pick > >> commit 7d6aa8c onto the 1.9.2 tag position. > >> > >> Does anyone object to me uploading a wheel built from this patched > >> version to pypi as 1.9.2 for Python 3.5 on OSX? It would help to get > >> the ball rolling for Python 3.5 binary wheels. > > > > > > Why not releasing this as 1.9.3 ? It does not need to be a full release > > (with binaries and all), but having multiple sources for a given tag is > > confusing. > > Generally OK with me, but it's quite a bit of extra work for very > little gain. We'd have to tag, release a source tarball and OSX > wheels, at least. I think it's highly desirable that we also have a *source* release that builds on Python 3.5, irrespective of whether or not we have binary wheels for a couple of platforms up for Python 3.5. So I would encourage a quick 1.9.3 release that incorporates this patch. -- Robert Kern -------------- next part -------------- An HTML attachment was scrubbed... URL: From cmkleffner at gmail.com Mon Sep 14 05:21:49 2015 From: cmkleffner at gmail.com (Carl Kleffner) Date: Mon, 14 Sep 2015 11:21:49 +0200 Subject: [Numpy-discussion] OK to upload patched 1.9.2 for Python 3.5? In-Reply-To: References: Message-ID: I would like to add patches for the mingwpy windows build as well. There is no Python-3.5 build so far. Carlkl 2015-09-14 10:46 GMT+02:00 Robert Kern : > On Mon, Sep 14, 2015 at 9:32 AM, Matthew Brett > wrote: > > > > Hi, > > > > On Mon, Sep 14, 2015 at 1:22 AM, David Cournapeau > wrote: > > > > > > > > > On Mon, Sep 14, 2015 at 9:18 AM, Matthew Brett < > matthew.brett at gmail.com> > > > wrote: > > >> > > >> Hi, > > >> > > >> I'm just building numpy 1.9.2 for Python 3.5 (just released). > > >> > > >> In order to get the tests to pass on Python 3.5, I need to cherry pick > > >> commit 7d6aa8c onto the 1.9.2 tag position. > > >> > > >> Does anyone object to me uploading a wheel built from this patched > > >> version to pypi as 1.9.2 for Python 3.5 on OSX? It would help to get > > >> the ball rolling for Python 3.5 binary wheels. > > > > > > > > > Why not releasing this as 1.9.3 ? It does not need to be a full release > > > (with binaries and all), but having multiple sources for a given tag is > > > confusing. > > > > Generally OK with me, but it's quite a bit of extra work for very > > little gain. We'd have to tag, release a source tarball and OSX > > wheels, at least. > > I think it's highly desirable that we also have a *source* release that > builds on Python 3.5, irrespective of whether or not we have binary wheels > for a couple of platforms up for Python 3.5. So I would encourage a quick > 1.9.3 release that incorporates this patch. > > -- > Robert Kern > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jtaylor.debian at googlemail.com Mon Sep 14 06:47:30 2015 From: jtaylor.debian at googlemail.com (Julian Taylor) Date: Mon, 14 Sep 2015 12:47:30 +0200 Subject: [Numpy-discussion] OK to upload patched 1.9.2 for Python 3.5? In-Reply-To: References: Message-ID: <55F6A5C2.1080409@googlemail.com> as due to the many incompatiblities in 1.10 many will likely not be able to update anytime soon, so I think putting out another 1.9.3 bugfix release would be a good idea. I can probably do the release management for it, though I haven't been keeping up with bugfixes recently so, please comment on important issues you would want fixed. The np.where upcast regression and np.nanmedian issues come to my mind as should be fixed. On 09/14/2015 11:21 AM, Carl Kleffner wrote: > I would like to add patches for the mingwpy windows build as well. There > is no Python-3.5 build so far. > > Carlkl > > 2015-09-14 10:46 GMT+02:00 Robert Kern >: > > On Mon, Sep 14, 2015 at 9:32 AM, Matthew Brett > > wrote: > > > > Hi, > > > > On Mon, Sep 14, 2015 at 1:22 AM, David Cournapeau > wrote: > > > > > > > > > On Mon, Sep 14, 2015 at 9:18 AM, Matthew Brett > > > > wrote: > > >> > > >> Hi, > > >> > > >> I'm just building numpy 1.9.2 for Python 3.5 (just released). > > >> > > >> In order to get the tests to pass on Python 3.5, I need to cherry pick > > >> commit 7d6aa8c onto the 1.9.2 tag position. > > >> > > >> Does anyone object to me uploading a wheel built from this patched > > >> version to pypi as 1.9.2 for Python 3.5 on OSX? It would help to get > > >> the ball rolling for Python 3.5 binary wheels. > > > > > > > > > Why not releasing this as 1.9.3 ? It does not need to be a full release > > > (with binaries and all), but having multiple sources for a given tag is > > > confusing. > > > > Generally OK with me, but it's quite a bit of extra work for very > > little gain. We'd have to tag, release a source tarball and OSX > > wheels, at least. > > I think it's highly desirable that we also have a *source* release > that builds on Python 3.5, irrespective of whether or not we have > binary wheels for a couple of platforms up for Python 3.5. So I > would encourage a quick 1.9.3 release that incorporates this patch. > > -- > Robert Kern > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From aparker at enthought.com Fri Sep 18 20:16:19 2015 From: aparker at enthought.com (Chip Parker) Date: Fri, 18 Sep 2015 19:16:19 -0500 Subject: [Numpy-discussion] testing Message-ID: Let's see if the instructions enclosed in http://www.jamesh.id.au/articles/mailman-spamassassin/ which was written in 2003 are still biting us. -- Chip Parker DevOps Engineer From cgohlke at uci.edu Mon Sep 14 14:06:01 2015 From: cgohlke at uci.edu (Christoph Gohlke) Date: Mon, 14 Sep 2015 11:06:01 -0700 Subject: [Numpy-discussion] OK to upload patched 1.9.2 for Python 3.5? In-Reply-To: <55F6A5C2.1080409@googlemail.com> References: <55F6A5C2.1080409@googlemail.com> Message-ID: <55F70C89.5000103@uci.edu> On 9/14/2015 3:47 AM, Julian Taylor wrote: > as due to the many incompatiblities in 1.10 many will likely not be able > to update anytime soon, so I think putting out another 1.9.3 bugfix > release would be a good idea. > I can probably do the release management for it, though I haven't been > keeping up with bugfixes recently so, please comment on important issues > you would want fixed. > The np.where upcast regression and np.nanmedian issues come to my mind > as should be fixed. > > On 09/14/2015 11:21 AM, Carl Kleffner wrote: >> I would like to add patches for the mingwpy windows build as well. There >> is no Python-3.5 build so far. >> >> Carlkl >> >> 2015-09-14 10:46 GMT+02:00 Robert Kern > >: >> >> On Mon, Sep 14, 2015 at 9:32 AM, Matthew Brett >> > wrote: >> > >> > Hi, >> > >> > On Mon, Sep 14, 2015 at 1:22 AM, David Cournapeau > wrote: >> > > >> > > >> > > On Mon, Sep 14, 2015 at 9:18 AM, Matthew Brett > >> > > wrote: >> > >> >> > >> Hi, >> > >> >> > >> I'm just building numpy 1.9.2 for Python 3.5 (just released). >> > >> >> > >> In order to get the tests to pass on Python 3.5, I need to cherry pick >> > >> commit 7d6aa8c onto the 1.9.2 tag position. >> > >> >> > >> Does anyone object to me uploading a wheel built from this patched >> > >> version to pypi as 1.9.2 for Python 3.5 on OSX? It would help to get >> > >> the ball rolling for Python 3.5 binary wheels. >> > > >> > > >> > > Why not releasing this as 1.9.3 ? It does not need to be a full release >> > > (with binaries and all), but having multiple sources for a given tag is >> > > confusing. >> > >> > Generally OK with me, but it's quite a bit of extra work for very >> > little gain. We'd have to tag, release a source tarball and OSX >> > wheels, at least. >> >> I think it's highly desirable that we also have a *source* release >> that builds on Python 3.5, irrespective of whether or not we have >> binary wheels for a couple of platforms up for Python 3.5. So I >> would encourage a quick 1.9.3 release that incorporates this patch. >> >> -- >> Robert Kern >> Support for Visual Studio 2015 and Intel Fortran 16 for Python 3.5 on Windows would also be nice. I am using: DEV: Replace deprecated options for ifort remove /GL for vs2015 in check_long_double_representation Enable Visual Studio 2015 C99 features BLD: revert C99 complex for msvc14 Christoph From allanhaldane at gmail.com Tue Sep 15 15:29:30 2015 From: allanhaldane at gmail.com (Allan Haldane) Date: Tue, 15 Sep 2015 15:29:30 -0400 Subject: [Numpy-discussion] new ufunc implementations for object arrays Message-ID: <55F8719A.3080300@gmail.com> Hello all, I've just submitted a PR which overhauls the implementation of ufuncs for object types. https://github.com/numpy/numpy/pull/6320 The motivation for this is that many ufuncs (eg all transcendental functions) can't handle objects. This PR will also make object arrays more customizable, as all ufuncs can now be overridden by the user by defining methods on the object. I'd like to see if there's enough interest/agreement for me to finish it, and also to have more eyes look over the pure-python ufunc implementations since I am sure I have missed some edge cases. I would love suggestions, and I need help for a few cases I couldn't figure out (eg np.nextafter, np.spacing, np.cbrt. Also, np.expm1 np.signbit and others are a bit suspect). Allan From tcaswell at gmail.com Mon Sep 14 23:59:23 2015 From: tcaswell at gmail.com (Thomas Caswell) Date: Tue, 15 Sep 2015 03:59:23 +0000 Subject: [Numpy-discussion] First release candidate for the 1.5.0 series Message-ID: Please give it a try! (linux64 conda builds are available on the tacaswell anaconda.org channel) https://github.com/matplotlib/matplotlib/releases/tag/v1.5.0rc1 This release contains many new features. The highlights include: - the object oriented API will now automatically re-draw the figure when working in the command line - automatic unpacking of labeled data into most plotting functions - arbitrary style cycling - four new perceptually linear color maps - mouse-over for pixel values with `imshow` - many new rcparams In addition there are significant improvements to `nbagg` and a complete overhaul of the c++ bindings to AGG. Please see the drafts of the [full whats new]( http://matplotlib.org/devdocs/users/whats_new.html#new-in-matplotlib-1-5) and [api changes]( http://matplotlib.org/devdocs/api/api_changes.html#changes-in-1-5-0) -------------- next part -------------- An HTML attachment was scrubbed... URL: From matthew.brett at gmail.com Mon Sep 14 13:04:32 2015 From: matthew.brett at gmail.com (Matthew Brett) Date: Mon, 14 Sep 2015 10:04:32 -0700 Subject: [Numpy-discussion] OK to upload patched 1.9.2 for Python 3.5? In-Reply-To: References: <55F6A5C2.1080409@googlemail.com> Message-ID: On Mon, Sep 14, 2015 at 9:42 AM, Matthew Brett wrote: > On Mon, Sep 14, 2015 at 3:47 AM, Julian Taylor > wrote: >> as due to the many incompatiblities in 1.10 many will likely not be able >> to update anytime soon, so I think putting out another 1.9.3 bugfix >> release would be a good idea. >> I can probably do the release management for it, though I haven't been >> keeping up with bugfixes recently so, please comment on important issues >> you would want fixed. >> The np.where upcast regression and np.nanmedian issues come to my mind >> as should be fixed. > > OK - fair enough - it would be good to do this soon as possible, as > Python 3.5 is currently the default download from the Python.org site > (rather than Python 2.7) and most other binary installers depend on > numpy. Is there anything I can do to help? For example, how about a rush-out 1.9.3 release with just the gzip patch, and a very-soon release with further patches? Matthew From matthew.brett at gmail.com Mon Sep 14 12:42:02 2015 From: matthew.brett at gmail.com (Matthew Brett) Date: Mon, 14 Sep 2015 09:42:02 -0700 Subject: [Numpy-discussion] OK to upload patched 1.9.2 for Python 3.5? In-Reply-To: <55F6A5C2.1080409@googlemail.com> References: <55F6A5C2.1080409@googlemail.com> Message-ID: On Mon, Sep 14, 2015 at 3:47 AM, Julian Taylor wrote: > as due to the many incompatiblities in 1.10 many will likely not be able > to update anytime soon, so I think putting out another 1.9.3 bugfix > release would be a good idea. > I can probably do the release management for it, though I haven't been > keeping up with bugfixes recently so, please comment on important issues > you would want fixed. > The np.where upcast regression and np.nanmedian issues come to my mind > as should be fixed. OK - fair enough - it would be good to do this soon as possible, as Python 3.5 is currently the default download from the Python.org site (rather than Python 2.7) and most other binary installers depend on numpy. Is there anything I can do to help? Matthew From ondrej.certik at gmail.com Mon Sep 14 19:59:39 2015 From: ondrej.certik at gmail.com (=?UTF-8?B?T25kxZllaiDEjGVydMOtaw==?=) Date: Mon, 14 Sep 2015 17:59:39 -0600 Subject: [Numpy-discussion] The process I intend to follow for any proposed changes to NumPy In-Reply-To: References: Message-ID: Hi Travis, On Sun, Sep 13, 2015 at 4:51 PM, Travis Oliphant wrote: > Hey all, > > I just wanted to clarify, that I am very excited about a few ideas I have > --- but I don't have time myself to engage in the community process to get > these changes into NumPy. However, those are real processes --- I've > been coaching a few people in those processes for the past several years > already. > > So, rather than do nothing, what I'm looking to do is to work with a few > people who I can share my ideas with, get excited about the ideas, and then > who will work with the community to get them implemented. That's what I > was announcing and talking about yesterday --- looking for interested people > who want to work on NumPy *with* the NumPy community. > > In my enthusiasm, I realize that some may have mis-understood my intention. > There is no 'imminent' fork, nor am I planning on doing some crazy amount of > work that I then try to force on other developers of NumPy. > > What I'm planning to do is find people to train on NumPy code base (people > to increase the diversity of the developers would be ideal -- but hard to > accomplish). I plan to train them on NumPy based on my experience, and on > what I think should be done --- and then have *them* work through the > community process and engage with others to get consensus (hopefully not > losing too much in translation in the process --- but instead getting even > better). > > During that process I will engage as a member of the community and help > write NEPs and other documents and help clarify where it makes sense as I > can. I will be filtering for people that actually want to see NumPy get > better. Until I identify the people and work with them, it will be hard > to tell how this will best work. So, stay tuned. > > If all goes well, what you should see in a few weeks time are specific > proposals, a branch or two, and the beginnings of some pull requests. If > you don't see that, then I will not have found the right people to help me, > and we will all continue to go back to searching. > > While I'm expecting the best, in the worst case, we get additional people > who know the NumPy code base and can help squash bugs as well as implement > changes that are desired. Three things are needed if you want to > participate in this: 1) A willingness to work with the open source > community, 2) a deep knowledge of C and in-particular CPython's brand of C, > and 3) a willingness to engage with me, do a mind-meld and dump around the > NumPy code base, and then improve on what is in my head with the rest of the > community. I don't have time to do the work myself, but I'll be happy to help in terms of the community. Travis has paid me in the past to do the NumPy 1.7 release, and implement/fix some things that Travis had in mind. I had no prior involvement with the NumPy codebase / community, and within few months I had to get myself familiar with the code, fix release critical bugs, get my PRs accepted by the community, eventually get push access (from the core developers, not from Travis) so that I can merge other people's patches, and most importantly, make the community allow me to actually do the release, even though I came from outside. As far as I know, Travis didn't have to pull any strings for me, and as far as I know, there was no friction in the community either --- there was a job to be done, we got it done, and that was it. So I'll be happy to share my experience and help the person (if coming from outside) how to do this. In this case a push access is not necessary, so it's just about sending high quality PRs and good communication. Ondrej From chris.barker at noaa.gov Mon Sep 14 13:46:51 2015 From: chris.barker at noaa.gov (Chris Barker) Date: Mon, 14 Sep 2015 10:46:51 -0700 Subject: [Numpy-discussion] The process I intend to follow for any proposed changes to NumPy In-Reply-To: References: Message-ID: Travis, I'm sure you appreciate that this might all look a bit scary, given the recent discussion about numpy governance. But it's an open-source project, and I, at least, fully understand that going through a big process is NOT the way to get a new idea tried out and implemented. So I think think this is a great development -- I know I want to see something like this dtype work done. So, as someone who has been around this community for a long time, and dependent on Numeric, numarray, and numpy over the years, this looks like a great development. And, in fact, with the new governance effort -- I think less scary -- people can go off and work on a branch or fork, do good stuff, and we, as a community, can be assured that API (or even ABI) changes won't be thrust upon us unawares :-) As for the technical details -- I get a bit lost, not fully understanding the current dtype system either, but do your ideas take us in the direction of having dtypes independent of the container and ufunc machinery -- and thus easier to create new dtypes (even in Python?) 'cause that would be great. I hope you find the partner you're looking for -- that's a challenge! -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From sturla.molden at gmail.com Mon Sep 14 10:56:07 2015 From: sturla.molden at gmail.com (Sturla Molden) Date: Mon, 14 Sep 2015 16:56:07 +0200 Subject: [Numpy-discussion] [Python-ideas] Should our default random number generator be secure? In-Reply-To: <20150914103431.7d6b43fd@fsol> References: <20150909190757.GM19373@ando.pearwood.info> <20150910015505.GO19373@ando.pearwood.info> <20150914103431.7d6b43fd@fsol> Message-ID: On 14/09/15 10:34, Antoine Pitrou wrote: > Currently we don't provide those APIs on the GPU, since MT is much too > costly there. > > If Numpy wanted to switch to a different generator, and if Numba wanted > to remain compatible with Numpy, one of the PCG functions would be an > excellent choice (also for CPU performance, incidentally). We have discussed allowing plugable PRNGs in NumPy on multiple occations. I don't think NumPy should change its default PRNG. The Mersenne Twister MT19937 is state of the art of numerical work. It has excellent numerical accuracy, a huge period, and is very fast. For most users of NumPy, the MT19937 is the first and last word that needs to be said about pseudorandom number generators. We have the default that most users of NumPy will ever want, unless severe flaws are discovered in the algorithm (not very likely). But there are occations where it does not work well. For example for parallel simulations it is (sometimes) nice to use DCMT instead of a vanilla MT19937. There are a lot of other generators that NumPy should consider as well, including the one you mention and Marsaglia's generators. We need to refactor randomkit to use a plugable entropy source instead of its internal MT19937. This is actually very easy to implement. In that case a user could just provide a Cython class or callback function (C, Fortran or Python) for generating a random ulong32. The discussion on python-ideas refers to cryptography. That is not equally relevant for NumPy. But with a plugable design we can let NumPy users use os.urandom as well. Sturla From sturla.molden at gmail.com Mon Sep 14 11:21:55 2015 From: sturla.molden at gmail.com (Sturla Molden) Date: Mon, 14 Sep 2015 17:21:55 +0200 Subject: [Numpy-discussion] [Python-ideas] Should our default random number generator be secure? In-Reply-To: References: <20150909190757.GM19373@ando.pearwood.info> <20150910015505.GO19373@ando.pearwood.info> Message-ID: On 14/09/15 10:26, Robert Kern wrote: > I want fast, multiple independent streams on my > current hardware first, and PCG gives that to me. DCMT is good for that as well. It should be possible to implement a pluggable design of NumPy's mtrand. Basically call a function pointer instead of rk_double. That way any (P)RNG can be plugged in. Then we could e.g. allow a Python callable, a Cython class or an f2py function as callback. We could ship multiple PRNGs with NumPy, and we could allow users to supply their own. Sturla From sturla.molden at gmail.com Mon Sep 14 13:02:58 2015 From: sturla.molden at gmail.com (Sturla Molden) Date: Mon, 14 Sep 2015 19:02:58 +0200 Subject: [Numpy-discussion] [Python-ideas] Should our default random number generator be secure? In-Reply-To: <20150914103431.7d6b43fd@fsol> References: <20150909190757.GM19373@ando.pearwood.info> <20150910015505.GO19373@ando.pearwood.info> <20150914103431.7d6b43fd@fsol> Message-ID: On 14/09/15 10:34, Antoine Pitrou wrote: > If Numpy wanted to switch to a different generator, and if Numba wanted > to remain compatible with Numpy, one of the PCG functions would be an > excellent choice (also for CPU performance, incidentally). Is Apache license ok in NumPy? (Not sure, thus asking. The PCG suite is Apache licensed.) From faltet at gmail.com Mon Sep 14 12:10:27 2015 From: faltet at gmail.com (Francesc Alted) Date: Mon, 14 Sep 2015 18:10:27 +0200 Subject: [Numpy-discussion] ANN: Numexpr 2.4.4 is out Message-ID: ========================= Announcing Numexpr 2.4.4 ========================= Numexpr is a fast numerical expression evaluator for NumPy. With it, expressions that operate on arrays (like "3*a+4*b") are accelerated and use less memory than doing the same calculation in Python. It wears multi-threaded capabilities, as well as support for Intel's MKL (Math Kernel Library), which allows an extremely fast evaluation of transcendental functions (sin, cos, tan, exp, log...) while squeezing the last drop of performance out of your multi-core processors. Look here for a some benchmarks of numexpr using MKL: https://github.com/pydata/numexpr/wiki/NumexprMKL Its only dependency is NumPy (MKL is optional), so it works well as an easy-to-deploy, easy-to-use, computational engine for projects that don't want to adopt other solutions requiring more heavy dependencies. What's new ========== This is a maintenance release which contains several bug fixes, like better testing on Python3 platform and some harmless data race. Among the enhancements, AppVeyor support is here and OMP_NUM_THREADS is honored as a fallback in case NUMEXPR_NUM_THREADS is not set. In case you want to know more in detail what has changed in this version, see: https://github.com/pydata/numexpr/blob/master/RELEASE_NOTES.rst Where I can find Numexpr? ========================= The project is hosted at GitHub in: https://github.com/pydata/numexpr You can get the packages from PyPI as well (but not for RC releases): http://pypi.python.org/pypi/numexpr Share your experience ===================== Let us know of any bugs, suggestions, gripes, kudos, etc. you may have. Enjoy data! -- Francesc Alted -------------- next part -------------- An HTML attachment was scrubbed... URL: From matthew.brett at gmail.com Wed Sep 16 04:08:53 2015 From: matthew.brett at gmail.com (Matthew Brett) Date: Wed, 16 Sep 2015 01:08:53 -0700 Subject: [Numpy-discussion] First release candidate for the 1.5.0 series In-Reply-To: References: Message-ID: Hi, On Mon, Sep 14, 2015 at 8:59 PM, Thomas Caswell wrote: > Please give it a try! (linux64 conda builds are available on the tacaswell > anaconda.org channel) > > https://github.com/matplotlib/matplotlib/releases/tag/v1.5.0rc1 > > This release contains many new features. The highlights include: > > - the object oriented API will now automatically re-draw the figure when > working in the command line > - automatic unpacking of labeled data into most plotting functions > - arbitrary style cycling > - four new perceptually linear color maps > - mouse-over for pixel values with `imshow` > - many new rcparams > > In addition there are significant improvements to `nbagg` and a complete > overhaul of the c++ bindings to AGG. > > Please see the drafts of the [full whats > new](http://matplotlib.org/devdocs/users/whats_new.html#new-in-matplotlib-1-5) > and [api > changes](http://matplotlib.org/devdocs/api/api_changes.html#changes-in-1-5-0) After some struggle, RC1 wheels up at http://wheels.scipy.org/ Built via : https://travis-ci.org/MacPython/matplotlib-wheels/builds/80587383 If you're testing Python 3.5 you'll need the patched numpy 1.9.2 wheel from nipy.bic.berkeley.edu : pip install -f https://nipy.bic.berkeley.edu/scipy_installers numpy Cheers, Matthew From faltet at gmail.com Wed Sep 16 07:12:58 2015 From: faltet at gmail.com (Francesc Alted) Date: Wed, 16 Sep 2015 13:12:58 +0200 Subject: [Numpy-discussion] ANN: python-blosc 1.2.8 released Message-ID: ============================= Announcing python-blosc 1.2.8 ============================= What is new? ============ This is a maintenance release. Internal C-Blosc has been upgraded to 1.7.0 (although new bitshuffle support has not been made public, as it seems not ready for production yet). Also, there is support for bytes-like objects that support the buffer interface as input to ``compress`` and ``decompress``. On Python 2.x this includes unicode, on Python 3.x it doesn't. Thanks to Valentin Haenel. Finally, a memory leak in ``decompress```has been hunted and fixed. And new tests have been added to catch possible leaks in the future. Thanks to Santi Villalba. For more info, you can have a look at the release notes in: https://github.com/Blosc/python-blosc/blob/master/RELEASE_NOTES.rst More docs and examples are available in the documentation site: http://python-blosc.blosc.org What is it? =========== Blosc (http://www.blosc.org) is a high performance compressor optimized for binary data. It has been designed to transmit data to the processor cache faster than the traditional, non-compressed, direct memory fetch approach via a memcpy() OS call. Blosc is the first compressor that is meant not only to reduce the size of large datasets on-disk or in-memory, but also to accelerate object manipulations that are memory-bound (http://www.blosc.org/docs/StarvingCPUs.pdf). See http://www.blosc.org/synthetic-benchmarks.html for some benchmarks on how much speed it can achieve in some datasets. Blosc works well for compressing numerical arrays that contains data with relatively low entropy, like sparse data, time series, grids with regular-spaced values, etc. python-blosc (http://python-blosc.blosc.org/) is the Python wrapper for the Blosc compression library. There is also a handy tool built on Blosc called Bloscpack (https://github.com/Blosc/bloscpack). It features a commmand line interface that allows you to compress large binary datafiles on-disk. It also comes with a Python API that has built-in support for serializing and deserializing Numpy arrays both on-disk and in-memory at speeds that are competitive with regular Pickle/cPickle machinery. Installing ========== python-blosc is in PyPI repository, so installing it is easy: $ pip install -U blosc # yes, you must omit the 'python-' prefix Download sources ================ The sources are managed through github services at: http://github.com/Blosc/python-blosc Documentation ============= There is Sphinx-based documentation site at: http://python-blosc.blosc.org/ Mailing list ============ There is an official mailing list for Blosc at: blosc at googlegroups.com http://groups.google.es/group/blosc Licenses ======== Both Blosc and its Python wrapper are distributed using the MIT license. See: https://github.com/Blosc/python-blosc/blob/master/LICENSES for more details. ---- **Enjoy data!** -- Francesc Alted -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Wed Sep 16 16:41:06 2015 From: njs at pobox.com (Nathaniel Smith) Date: Wed, 16 Sep 2015 13:41:06 -0700 Subject: [Numpy-discussion] The process I intend to follow for any proposed changes to NumPy In-Reply-To: References: Message-ID: Hi all, Thanks, Travis, for the followup. I know some people were confused or concerned by some points in Travis?s recent emails (as was I, initially), but after checking in with Travis and the NumPy steering council, it sounds like the main points of possible confusion are actually things where we at least are actually on the same page. So to avoid any uncertainty or miscommunication, I just want to reiterate some hopefully-uncontroversial points here: - There is currently no plan whatsoever for NumPy to drop Python 2 support; something like ~80% of our users are still using Python 2, and we anticipate that both Python 2 and Python 3 will continue to receive full support for the foreseeable future. - While it is possible that there will eventually be a compatibility-breaking ?NumPy 2.0? release, it won?t happen without a community consensus that this is a good idea, and that consensus has not yet been reached. - As a clarification on how NumPy governance works: While we all continue to be grateful to Travis for his past contributions, in terms of formal authority he stepped down from his leadership role in the project several years ago, and the project has since switched to a community-driven governance model [1]. Under this model, all contributors participate as equal peers, and (outside of specific exceptional circumstances) all contributors should be assumed to be speaking only on behalf of themselves, not the project as a whole. - Discussion about the best way to improve NumPy?s dtype system is ongoing, and several approaches are under consideration. No proposal will be accepted without review and consensus by the broader NumPy community. We welcome anyone who?s interested in these issues to join us on the mailing list -- the more input we have, the better the result will be :-). Apologies for the wide distribution of this message; I suggest any followups be directed to numpy-discussion at scipy.org only. Thanks, - Nathaniel [1] http://thread.gmane.org/gmane.comp.python.numeric.general/61106 (we?ll move it into the repository soon, we promise!) On Sun, Sep 13, 2015 at 3:51 PM, Travis Oliphant wrote: > Hey all, > > I just wanted to clarify, that I am very excited about a few ideas I have > --- but I don't have time myself to engage in the community process to get > these changes into NumPy. However, those are real processes --- I've > been coaching a few people in those processes for the past several years > already. > > So, rather than do nothing, what I'm looking to do is to work with a few > people who I can share my ideas with, get excited about the ideas, and then > who will work with the community to get them implemented. That's what I > was announcing and talking about yesterday --- looking for interested people > who want to work on NumPy *with* the NumPy community. > > In my enthusiasm, I realize that some may have mis-understood my intention. > There is no 'imminent' fork, nor am I planning on doing some crazy amount of > work that I then try to force on other developers of NumPy. > > What I'm planning to do is find people to train on NumPy code base (people > to increase the diversity of the developers would be ideal -- but hard to > accomplish). I plan to train them on NumPy based on my experience, and on > what I think should be done --- and then have *them* work through the > community process and engage with others to get consensus (hopefully not > losing too much in translation in the process --- but instead getting even > better). > > During that process I will engage as a member of the community and help > write NEPs and other documents and help clarify where it makes sense as I > can. I will be filtering for people that actually want to see NumPy get > better. Until I identify the people and work with them, it will be hard > to tell how this will best work. So, stay tuned. > > If all goes well, what you should see in a few weeks time are specific > proposals, a branch or two, and the beginnings of some pull requests. If > you don't see that, then I will not have found the right people to help me, > and we will all continue to go back to searching. > > While I'm expecting the best, in the worst case, we get additional people > who know the NumPy code base and can help squash bugs as well as implement > changes that are desired. Three things are needed if you want to > participate in this: 1) A willingness to work with the open source > community, 2) a deep knowledge of C and in-particular CPython's brand of C, > and 3) a willingness to engage with me, do a mind-meld and dump around the > NumPy code base, and then improve on what is in my head with the rest of the > community. > > Thanks, > > -Travis > > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -- Nathaniel J. Smith -- http://vorpus.org From aparker at enthought.com Fri Sep 18 20:03:37 2015 From: aparker at enthought.com (Chip Parker) Date: Fri, 18 Sep 2015 19:03:37 -0500 Subject: [Numpy-discussion] testing Message-ID: This would be an email. -- Chip Parker DevOps Engineer From njs at pobox.com Fri Sep 18 21:05:37 2015 From: njs at pobox.com (Nathaniel Smith) Date: Fri, 18 Sep 2015 18:05:37 -0700 Subject: [Numpy-discussion] [Python-ideas] Should our default random number generator be secure? In-Reply-To: References: <20150909190757.GM19373@ando.pearwood.info> <20150910015505.GO19373@ando.pearwood.info> <20150914103431.7d6b43fd@fsol> Message-ID: On Mon, Sep 14, 2015 at 7:56 AM, Sturla Molden wrote: > On 14/09/15 10:34, Antoine Pitrou wrote: > >> Currently we don't provide those APIs on the GPU, since MT is much too >> costly there. >> >> If Numpy wanted to switch to a different generator, and if Numba wanted >> to remain compatible with Numpy, one of the PCG functions would be an >> excellent choice (also for CPU performance, incidentally). > > > We have discussed allowing plugable PRNGs in NumPy on multiple occations. > > I don't think NumPy should change its default PRNG. The Mersenne Twister > MT19937 is state of the art of numerical work. It has excellent numerical > accuracy, a huge period, and is very fast. For most users of NumPy, the > MT19937 is the first and last word that needs to be said about pseudorandom > number generators. There are obviously trade-offs here and I haven't looked at the details enough to have an opinion on whether the benefits are *enough* that changing the default would be worth it, but it's simply not true that MT19937 is state of the art. There are faster generators with better numerical properties and more features (including, but not only, the PCG family). It looks like the random123 Philox generator is somewhat popular on GPUs at this point too (e.g. https://uk.mathworks.com/help/distcomp/examples/generating-random-numbers-on-a-gpu.html). -n -- Nathaniel J. Smith -- http://vorpus.org From matthew.brett at gmail.com Fri Sep 18 21:13:57 2015 From: matthew.brett at gmail.com (Matthew Brett) Date: Fri, 18 Sep 2015 18:13:57 -0700 Subject: [Numpy-discussion] OK to upload patched 1.9.2 for Python 3.5? In-Reply-To: <55F70C89.5000103@uci.edu> References: <55F6A5C2.1080409@googlemail.com> <55F70C89.5000103@uci.edu> Message-ID: Hi Christoph, On Mon, Sep 14, 2015 at 11:06 AM, Christoph Gohlke wrote: > On 9/14/2015 3:47 AM, Julian Taylor wrote: >> >> as due to the many incompatiblities in 1.10 many will likely not be able >> to update anytime soon, so I think putting out another 1.9.3 bugfix >> release would be a good idea. >> I can probably do the release management for it, though I haven't been >> keeping up with bugfixes recently so, please comment on important issues >> you would want fixed. >> The np.where upcast regression and np.nanmedian issues come to my mind >> as should be fixed. >> >> On 09/14/2015 11:21 AM, Carl Kleffner wrote: >>> >>> I would like to add patches for the mingwpy windows build as well. There >>> is no Python-3.5 build so far. >>> >>> Carlkl >>> >>> 2015-09-14 10:46 GMT+02:00 Robert Kern >> >: >>> >>> On Mon, Sep 14, 2015 at 9:32 AM, Matthew Brett >>> > wrote: >>> > >>> > Hi, >>> > >>> > On Mon, Sep 14, 2015 at 1:22 AM, David Cournapeau >>> > wrote: >>> > > >>> > > >>> > > On Mon, Sep 14, 2015 at 9:18 AM, Matthew Brett >>> > >>> > > wrote: >>> > >> >>> > >> Hi, >>> > >> >>> > >> I'm just building numpy 1.9.2 for Python 3.5 (just released). >>> > >> >>> > >> In order to get the tests to pass on Python 3.5, I need to >>> cherry pick >>> > >> commit 7d6aa8c onto the 1.9.2 tag position. >>> > >> >>> > >> Does anyone object to me uploading a wheel built from this >>> patched >>> > >> version to pypi as 1.9.2 for Python 3.5 on OSX? It would help >>> to get >>> > >> the ball rolling for Python 3.5 binary wheels. >>> > > >>> > > >>> > > Why not releasing this as 1.9.3 ? It does not need to be a full >>> release >>> > > (with binaries and all), but having multiple sources for a given >>> tag is >>> > > confusing. >>> > >>> > Generally OK with me, but it's quite a bit of extra work for very >>> > little gain. We'd have to tag, release a source tarball and OSX >>> > wheels, at least. >>> >>> I think it's highly desirable that we also have a *source* release >>> that builds on Python 3.5, irrespective of whether or not we have >>> binary wheels for a couple of platforms up for Python 3.5. So I >>> would encourage a quick 1.9.3 release that incorporates this patch. >>> >>> -- >>> Robert Kern >>> > > Support for Visual Studio 2015 and Intel Fortran 16 for Python 3.5 on > Windows would also be nice. > > I am using: > > DEV: Replace deprecated options for ifort > > > remove /GL for vs2015 in check_long_double_representation > > > Enable Visual Studio 2015 C99 features > > > BLD: revert C99 complex for msvc14 > Would you mind making a branch for these, preferably starting from my current branch: git://github.com/matthew-brett/numpy.git branch prepare-1.9.3 ? I had a quick go, but the merge conflicts needed more understanding than I had of the Windows build. Thanks a lot, Matthew From travis at continuum.io Fri Sep 18 21:17:13 2015 From: travis at continuum.io (Travis Oliphant) Date: Fri, 18 Sep 2015 18:17:13 -0700 Subject: [Numpy-discussion] The process I intend to follow for any proposed changes to NumPy In-Reply-To: References: Message-ID: Hey Chris (limiting to NumPy only), I've had some great conversations with Nathaniel in the past few days and I'm glad he posted his thoughts so that there is no confusion about governance or what I was implying. With respect to governance, I'm very supportive of what everyone is doing in organizing a governance document and approach and appreciate the effort of Nathaniel and others to move this forward. Nothing I said was meant to imply differently. I'm sorry if it made anyone nervous. I'm a very enthusiastic person when I get an idea of what to do. I like to see things implemented. In this case, it also turns out that in terms of overall architecture, my ideas are actually very similar to Nathaniel's ideas. That's a good sign. We have different tactical approaches as to how to move forward, but I think it's a good thing to note that we see a very similar path forward. Nothing will be done in NumPy itself except via pull-request and review. My approach for the ideas I'm pursuing will be to organize people around two new prototype packages I'm calling memtype and gufunc. The purpose of these is to allow playing with the design and ideas quickly before looking at how to put them into NumPy itself --- there will also be some training involved in getting people up to speed. There was a long discussion today at this BIDS data-structures for data-science summit part of which talked about how to improve NumPy's dtype system. I would love to these independent objects evolve into independent packages that could even go into Python standard library. Not everyone agrees that is the best idea, but regardless of whether this happens or not, the intent is to do work that could go into NumPy now. I look forward to the activity. -Travis On Mon, Sep 14, 2015 at 10:46 AM, Chris Barker wrote: > Travis, > > I'm sure you appreciate that this might all look a bit scary, given the > recent discussion about numpy governance. > > But it's an open-source project, and I, at least, fully understand that > going through a big process is NOT the way to get a new idea tried out and > implemented. So I think think this is a great development -- I know I want > to see something like this dtype work done. > > So, as someone who has been around this community for a long time, and > dependent on Numeric, numarray, and numpy over the years, this looks like a > great development. > > And, in fact, with the new governance effort -- I think less scary -- > people can go off and work on a branch or fork, do good stuff, and we, as a > community, can be assured that API (or even ABI) changes won't be thrust > upon us unawares :-) > > As for the technical details -- I get a bit lost, not fully understanding > the current dtype system either, but do your ideas take us in the direction > of having dtypes independent of the container and ufunc machinery -- and > thus easier to create new dtypes (even in Python?) 'cause that would be > great. > > I hope you find the partner you're looking for -- that's a challenge! > > -Chris > > > > > -- > > Christopher Barker, Ph.D. > Oceanographer > > Emergency Response Division > NOAA/NOS/OR&R (206) 526-6959 voice > 7600 Sand Point Way NE (206) 526-6329 fax > Seattle, WA 98115 (206) 526-6317 main reception > > Chris.Barker at noaa.gov > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > https://mail.scipy.org/mailman/listinfo/numpy-discussion > > -- *Travis Oliphant* *Co-founder and CEO* @teoliphant 512-222-5440 http://www.continuum.io -------------- next part -------------- An HTML attachment was scrubbed... URL: From phillip.m.feldman at gmail.com Sat Sep 19 00:16:42 2015 From: phillip.m.feldman at gmail.com (Phillip Feldman) Date: Fri, 18 Sep 2015 21:16:42 -0700 Subject: [Numpy-discussion] method to calculate the magnitude squared Message-ID: In communications and signal processing, it is frequently necessary to calculate the power of a signal. This can be done with a function like the following: def magsq(z): """ Return the magnitude squared of the real- or complex-valued input. """ return z.real**2 + z.imag**2 A high percentage of the scripts that I write contain or import a function like this. It would be great if there were a built-in method in NumPy, preferably with a name like `magsq`, `mag2`, or `msq`. Phillip -------------- next part -------------- An HTML attachment was scrubbed... URL: From rays at blue-cove.com Sat Sep 19 00:41:35 2015 From: rays at blue-cove.com (R Schumacher) Date: Fri, 18 Sep 2015 21:41:35 -0700 Subject: [Numpy-discussion] method to calculate the magnitude squared In-Reply-To: References: Message-ID: <201509190441.t8J4fhee027822@blue-cove.com> At 09:16 PM 9/18/2015, you wrote: >In communications and signal processing, it is >frequently necessary to calculate the power of a >signal.? This can be done with a function like the following: > >def magsq(z): >? ? """ >? ? Return the magnitude squared of the real- or complex-valued input. >? ? """ >? ? return z.real**2 + z.imag**2 Is that not the same as np.abs(z)**2 ? - Ray From solipsis at pitrou.net Sat Sep 19 09:26:19 2015 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sat, 19 Sep 2015 15:26:19 +0200 Subject: [Numpy-discussion] [Python-ideas] Should our default random number generator be secure? References: <20150910015505.GO19373@ando.pearwood.info> <20150914103431.7d6b43fd@fsol> Message-ID: <20150919152619.4c8fe13f@fsol> On Mon, 14 Sep 2015 19:02:58 +0200 Sturla Molden wrote: > On 14/09/15 10:34, Antoine Pitrou wrote: > > > If Numpy wanted to switch to a different generator, and if Numba wanted > > to remain compatible with Numpy, one of the PCG functions would be an > > excellent choice (also for CPU performance, incidentally). > > Is Apache license ok in NumPy? While I don't know Numpy's policy precisely, I have no idea why a modern non-copyleft license such as Apache wouldn't be ok. That said, PCG looks simple enough that you can reimplement it (or at least the variants that are of interest to you) independently without too much effort. Regards Antoine. From robert.kern at gmail.com Sat Sep 19 12:06:39 2015 From: robert.kern at gmail.com (Robert Kern) Date: Sat, 19 Sep 2015 17:06:39 +0100 Subject: [Numpy-discussion] [Python-ideas] Should our default random number generator be secure? In-Reply-To: <20150919152619.4c8fe13f@fsol> References: <20150910015505.GO19373@ando.pearwood.info> <20150914103431.7d6b43fd@fsol> <20150919152619.4c8fe13f@fsol> Message-ID: On Sat, Sep 19, 2015 at 2:26 PM, Antoine Pitrou wrote: > > On Mon, 14 Sep 2015 19:02:58 +0200 > Sturla Molden wrote: > > On 14/09/15 10:34, Antoine Pitrou wrote: > > > > > If Numpy wanted to switch to a different generator, and if Numba wanted > > > to remain compatible with Numpy, one of the PCG functions would be an > > > excellent choice (also for CPU performance, incidentally). > > > > Is Apache license ok in NumPy? > > While I don't know Numpy's policy precisely, I have no idea why a > modern non-copyleft license such as Apache wouldn't be ok. GPLv2 incompatibility: http://www.apache.org/licenses/GPL-compatibility.html > That said, PCG looks simple enough that you can reimplement it (or at > least the variants that are of interest to you) independently without > too much effort. The author has agreed in principle to relicense the code as MIT, though has not yet merged the PR that accomplishes this yet. That said, we'd probably end up doing a significant amount of rewriting so that we will have a C implementation of software-uint128 arithmetic. -- Robert Kern -------------- next part -------------- An HTML attachment was scrubbed... URL: From m.h.vankerkwijk at gmail.com Sun Sep 20 13:01:29 2015 From: m.h.vankerkwijk at gmail.com (Marten van Kerkwijk) Date: Sun, 20 Sep 2015 13:01:29 -0400 Subject: [Numpy-discussion] method to calculate the magnitude squared In-Reply-To: <201509190441.t8J4fhee027822@blue-cove.com> References: <201509190441.t8J4fhee027822@blue-cove.com> Message-ID: > > Is that not the same as > np.abs(z)**2 ? > It is, but since that involves taking sqrt, it is *much* slower. Even now, ``` In [32]: r = np.arange(10000)*(1+1j) In [33]: %timeit np.abs(r)**2 1000 loops, best of 3: 213 ?s per loop In [34]: %timeit r.real**2 + r.imag**2 10000 loops, best of 3: 47.5 ?s per loop ``` -- Marten -------------- next part -------------- An HTML attachment was scrubbed... URL: From travis at continuum.io Sun Sep 20 14:20:28 2015 From: travis at continuum.io (Travis Oliphant) Date: Sun, 20 Sep 2015 11:20:28 -0700 Subject: [Numpy-discussion] Governance model request Message-ID: After long conversations at BIDS this weekend and after reading the entire governance document, I realized that the steering council is very large and I don't agree with the mechanism by which it is chosen. A one year time frame is pretty short on the context of a two decades old project and I believe the current council has too few people who have been around the community long enough to help unstuck difficult situations if that were necessary. I would recommend three possible adjustments to the steering council concept. 1 - define a BDFL for the council. I would nominate chuck Harris 2 - limit the council to 3 people. I would nominate chuck, nathaniel, and pauli. 3 - add me as a permanent member of the steering council. Writing NumPy was a significant amount of work. I have been working indirectly or directly in support of NumPy continously since I wrote it. While I don't actively participate all the time, I still have a lot of knowledge, context, and experience in how NumPy is used, why it is the way it is, and how things could be better. I also work with people directly who have and will contribute regularly. I am formally requesting that the steering council concept be adjusted in one of these three ways. Thanks, Travis -------------- next part -------------- An HTML attachment was scrubbed... URL: From othalan at othalan.net Sun Sep 20 15:35:12 2015 From: othalan at othalan.net (David Morris) Date: Sun, 20 Sep 2015 13:35:12 -0600 Subject: [Numpy-discussion] Compiling NumPy for iOS PyQt app? Message-ID: I have a PyQt app running on iOS and would like to add NumPy to improve calculation speed. I see a few python interpreters in the app store which use NumPy so it must be possible, however I have not been able to find any information on the build process for the iOS cross compile. We are building a python with all libraries static linked. Here is the environment: iOS 8.0+ Python 3.4 PyQt 5.5 Qt 5.5 pyqtdeploy Any help getting NumPy compiled into the iOS app? Thank you, David -------------- next part -------------- An HTML attachment was scrubbed... URL: From rays at blue-cove.com Sun Sep 20 15:39:44 2015 From: rays at blue-cove.com (R Schumacher) Date: Sun, 20 Sep 2015 12:39:44 -0700 Subject: [Numpy-discussion] method to calculate the magnitude squared In-Reply-To: References: <201509190441.t8J4fhee027822@blue-cove.com> Message-ID: <201509201939.t8KJdqb8023322@blue-cove.com> At 10:01 AM 9/20/2015, you wrote: >Is that not the same as >? ? np.abs(z)**2 ? > >It is, but since that involves taking sqrt, it is *much* slower. Even now, >``` >In [32]: r = np.arange(10000)*(1+1j) > >In [33]: %timeit np.abs(r)**2 >1000 loops, best of 3: 213 ??s per loop > >In [34]: %timeit r.real**2 + r.imag**2 >10000 loops, best of 3: 47.5 ??s per loop > >-- Marten Ahh yes, a full extra step, "back" Assuming these are spectra, how does timeit do with scipy.signal.periodogram (using appropriate n and scaling)? - Ray -------------- next part -------------- An HTML attachment was scrubbed... URL: From sturla.molden at gmail.com Sun Sep 20 15:48:54 2015 From: sturla.molden at gmail.com (Sturla Molden) Date: Sun, 20 Sep 2015 21:48:54 +0200 Subject: [Numpy-discussion] [Python-ideas] Should our default random number generator be secure? In-Reply-To: References: <20150910015505.GO19373@ando.pearwood.info> <20150914103431.7d6b43fd@fsol> <20150919152619.4c8fe13f@fsol> Message-ID: On 19/09/15 18:06, Robert Kern wrote: > That said, we'd > probably end up doing a significant amount of rewriting so that we will > have a C implementation of software-uint128 arithmetic. This is where a small subset of C++ would be handy. Making an uint128_t class with overloaded operators is a nobrainer. :-) Not that we will ever use C++ in NumPy, but we do have a fair amount of C++ in SciPy. Sturla From sturla.molden at gmail.com Sun Sep 20 15:56:17 2015 From: sturla.molden at gmail.com (Sturla Molden) Date: Sun, 20 Sep 2015 21:56:17 +0200 Subject: [Numpy-discussion] [Python-ideas] Should our default random number generator be secure? In-Reply-To: References: <20150910015505.GO19373@ando.pearwood.info> <20150914103431.7d6b43fd@fsol> <20150919152619.4c8fe13f@fsol> Message-ID: On 20/09/15 21:48, Sturla Molden wrote: > This is where a small subset of C++ would be handy. Making an uint128_t > class with overloaded operators is a nobrainer. :-) Meh... The C++ version of PCG already has this. From solipsis at pitrou.net Sun Sep 20 16:41:32 2015 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sun, 20 Sep 2015 22:41:32 +0200 Subject: [Numpy-discussion] method to calculate the magnitude squared References: Message-ID: <20150920224132.0d3bc8af@fsol> On Fri, 18 Sep 2015 21:16:42 -0700 Phillip Feldman wrote: > In communications and signal processing, it is frequently necessary to > calculate the power of a signal. This can be done with a function like the > following: > > def magsq(z): > """ > Return the magnitude squared of the real- or complex-valued input. > """ > return z.real**2 + z.imag**2 > > A high percentage of the scripts that I write contain or import a function > like this. It would be great if there were a built-in method in NumPy, > preferably with a name like `magsq`, `mag2`, or `msq`. Are you asking for speed or convenience reasons? Regards Antoine. From njs at pobox.com Mon Sep 21 00:13:30 2015 From: njs at pobox.com (Nathaniel Smith) Date: Sun, 20 Sep 2015 21:13:30 -0700 Subject: [Numpy-discussion] draft NEP for breaking ufunc ABI in a controlled way Message-ID: Hi all, Here's a first draft NEP for comments. -- Synopsis ======== Improving numpy's dtype system requires that ufunc loops start having access to details of the specific dtype instance they are acting on: e.g. an implementation of np.equal for strings needs access to the dtype object in order to know what "n" to pass to strncmp. Similar issues arise with variable length strings, missing values, categorical data, unit support, datetime with timezone support, etc. -- this is a major blocker for improving numpy. Unfortunately, the current ufunc inner loop function signature makes it very difficult to provide this information. We might be able to wedge it in there, but it'd be ugly. The other option would be to change the signature. What would happen if we did this? For most common uses of the C API/ABI, we could do this easily while maintaining backwards compatibility. But there are also some rarely-used parts of the API/ABI that would be prohibitively difficult to preserve. In addition, there are other potential changes to ufuncs on the horizon (e.g. extensions of gufuncs to allow them to be used more generally), and the current API exposure is so massive that any such changes will be difficult to make in a fully compatible way. This NEP thus considers the possibility of closing down the ufunc API to a minimal, maintainable subset of the current API. To better understand the consequences of this potential change, I performed an exhaustive analysis of all the code on Github, Bitbucket, and Fedora, among others. The results make me highly confident that of all the publically available projects in the world, the only ones which touch the problematic parts of the ufunc API are: Numba, dynd-python, and `gulinalg `_ (with the latter's exposure being trivial). Given this, I propose that for 1.11 we: 1) go ahead and hide/disable the problematic parts of the ABI/API, 2) coordinate with the known affected projects to minimize disruption to their users (which is made easier since they are all projects that are almost exclusively distributed via conda, which enforces strict NumPy ABI versioning), 3) publicize these changes widely so as to give any private code that might be affected a chance to speak up or adapt, and 4) leave the "ABI version tag" as it is, so as not to force rebuilds of the vast majority of projects that will be unaffected by these changes. This NEP defers the question of exactly what the improved API should be, since there's no point in trying to nail down the details until we've decided whether it's even possible to change. Details ======= The problem ----------- Currently, a ufunc inner loop implementation is called via the following function prototype:: typedef void (*PyUFuncGenericFunction) (char **args, npy_intp *dimensions, npy_intp *strides, void *innerloopdata); Here ``args`` is an array of pointers to 1-d buffers of input/output data, ``dimensions`` is a pointer to the number of entries in these buffers, ``strides`` is an array of integers giving the strides for each input/output array, and ``innerloopdata`` is an arbitrary void* supplied by whoever registered the ufunc loop. (For gufuncs, extra shape and stride information about the core dimensions also gets packed into the ends of these arrays in a somewhat complicated way.) There are 4 key items that define a NumPy array: data, shape, strides, dtype. Notice that this function only gets access to 3 of them. Our goal is to fix that. For example, a better signature would be:: typedef void (*PyUFuncGenericFunction_NEW) (char **data, npy_intp *shapes, npy_intp *strides, PyArray_Descr *dtypes, /* NEW */ void *innerloopdata); (In practice I suspect we might want to make some more changes as well, like upgrading gufunc core shape/strides to proper arguments instead of tacking it onto the existing arrays, and adding an "escape valve" void* reserved for future extensions. But working out such details is outside the scope of this NEP; the above will do for illustration.) The goal of this NEP is to clear the ground so that we can start supporting ufunc inner loops that take dtype arguments, and make other enhancements to ufunc functionality going forward. Proposal -------- Currently, the public API/ABI for ufuncs consists of the functions:: PyUFunc_GenericFunction PyUFunc_FromFuncAndData PyUFunc_FromFuncAndDataAndSignature PyUFunc_RegisterLoopForDescr PyUFunc_RegisterLoopForType PyUFunc_ReplaceLoopBySignature PyUFunc_SetUsesArraysAsData together with direct access to PyUFuncObject's internal fields:: typedef struct { PyObject_HEAD int nin, nout, nargs; int identity; PyUFuncGenericFunction *functions; void **data; int ntypes; int check_return; const char *name; char *types; const char *doc; void *ptr; PyObject *obj; PyObject *userloops; int core_enabled; int core_num_dim_ix; int *core_num_dims; int *core_dim_ixs; int *core_offsets; char *core_signature; PyUFunc_TypeResolutionFunc *type_resolver; PyUFunc_LegacyInnerLoopSelectionFunc *legacy_inner_loop_selector; PyUFunc_InnerLoopSelectionFunc *inner_loop_selector; PyUFunc_MaskedInnerLoopSelectionFunc *masked_inner_loop_selector; npy_uint32 *op_flags; npy_uint32 iter_flags; } PyUFuncObject; Obviously almost any future changes to how ufuncs work internally will involve touching some part of this public API/ABI. Concretely, the proposal here is that we avoid this by disabling the following functions (i.e., any attempt to call them should simply raise a ``NotImplementedError``):: PyUFunc_ReplaceLoopBySignature PyUFunc_SetUsesArraysAsData and that we reduce the publicly visible portion of PyUFuncObject down to:: typedef struct { PyObject_HEAD int nin, nout, nargs; } PyUFuncObject; Data on current API/ABI usage ----------------------------- In order to assess how much code would be affected by this proposal, I used a combination of Github search and Searchcode.com to trawl through the majority of all publicly available open source code. Neither search tool provides a fine-grained enough query language to directly tell us what we want to know, so I instead followed the strategy of first, casting a wide net: picking a set of search terms that are likely to catch all possibly-broken code (together with many false positives), and second, using automated tools to sift out the false positives and see what remained. Altogether, I reviewed 4464 search results. The tool I wrote to do this is `available on github `_, and so is `the analysis code itself `_. Uses of PyUFuncObject internals ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ There are no functions in the public API which return ``PyUFuncObject*`` values directly, so any code that access PyUFuncObject fields will have to mention that token in the course of defining a variable, performing a cast, setting up a typedef, etc. Therefore, I searched Github for all files written in C, C++, Objective C, Python, or Cython, which mentioned either "PyUFuncObject AND types" or "PyUFuncObject AND NOT types". (This is to work around limitations on how many results Github search is willing to return to a single query.) In addition, I searched for ``PyUFuncObject`` on searchcode.com. The full report on these searches is available here: https://rawgit.com/njsmith/ufunc-abi-analysis/master/reports/pyufuncobject-report.html The following were screened out as non-problems: - Copies of NumPy itself (an astonishing number of people have checked in copies of it to their own source tree) - NumPy forks / precursors / etc. (e.g. Numeric also had a type called PyUFuncObject, the "bohrium" project has a fork of numpy 1.6, etc.) - Cython-generated boilerplate used to generate the "object has changed size" warning (which we `unconditionally filter out anyway `_) - Lots of calls to ``PyUFunc_RegisterLoopForType`` and friends, which require casting the first argument to ``PyUFuncObject*`` - Misc. other unproblematic stuff (like Cython header declarations that never get used) There were also several cases that actually referenced PyUFuncObject internal fields: - The "rational" dtype from numpy-dtypes, which is used in a few projects, accesses ``ufunc->nargs`` as a safety check, but does not touch any other fields (`see here `_). - Numba: does some rather elaborate things to support the definition of on-the-fly JITted ufuncs. These seem to be clear deficiencies in the ufunc API (e.g., there's no good way to control the lifespan of the array of function pointers passed to ``PyUFunc_FromFuncAndData``), so we should work with them to provide the API they need to do this in a maintainable way. Some of the relevant code: https://github.com/numba/numba/tree/master/numba/npyufunc https://github.com/numba/numba/blob/98752647a95ac6c9d480e81ca5c8afcfa3ddfd18/numba/npyufunc/_internal.c - dynd-python: Contains some code that attempts to extract the inner loops from a numpy ufunc object and wrap them into the dynd 'ufunc' equivalent: https://github.com/libdynd/dynd-python/blob/c06f8fc4e72257abac589faf76f10df8c045159b/dynd/src/numpy_ufunc_kernel.cpp - gulinalg: I'm not sure if anyone is still using this code since most of it was merged into numpy itself, but it's not a big deal in any case: all it contains is a `debugging function `_ that dumps some internal fields from the PyUFuncObject. If you look, though, all calls to this function are already commented out :-). The full report is available here: https://rawgit.com/njsmith/ufunc-abi-analysis/master/reports/pyufuncobject-report.html In the course of this analysis, it was also noted that the standard Cython pxd files contain a wrapper for ufunc objects:: cdef class ufunc [object PyUFuncObject]: ... which means that Cython code can access internal struct fields via an object of type ``ufunc``, and thus escape our string-based search above. Therefore I also examined all Cython files on Github or searchcode.com that matched the query ``ufunc``, and searched for any lines matching any of the following regular expressions:: cdef\W+ufunc catches: 'cdef ufunc fn' cdef\W+.*\.\W*ufunc catches: 'cdef np.ufunc fn' <.*ufunc\W*> catches: '( fn).nargs', '(< np.ufunc > fn).nargs' cdef.*\(.*ufunc catches: 'cdef doit(np.ufunc fn, ...):' (I considered parsing the actual source and analysing it that way, but decided I was too lazy. This could be done if anyone is worried that the above regexes might miss things though.) There were zero files that contained matches for any of the above regexes: https://rawgit.com/njsmith/ufunc-abi-analysis/master/reports/ufunc-cython-report.html Uses of PyUFunc_ReplaceLoopBySignature ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Applying the same screening as above, the only code that was found that used this function is also in Numba: https://rawgit.com/njsmith/ufunc-abi-analysis/master/reports/PyUFunc_ReplaceLoopBySignature-report.html Uses of PyUFunc_SetUsesArraysAsData ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Aside from being semi-broken since 1.7 (it never got implemented for "masked" ufunc loops, i.e. those that use where=), there appear to be zero uses of this functionality either inside or outside NumPy: https://rawgit.com/njsmith/ufunc-abi-analysis/master/reports/PyUFunc_SetUsesArraysAsData-report.html Rationale --------- **Rationale for preserving the remaining API functions**:: PyUFunc_GenericFunction PyUFunc_FromFuncAndData PyUFunc_FromFuncAndDataAndSignature PyUFunc_RegisterLoopForDescr PyUFunc_RegisterLoopForType In addition to being widely used, these functions can easily be preserved even if we change how ufuncs work internally, because they only ingest loop function pointers, they never return them. So they can easily be modified to wrap whatever loop function(s) they receive inside an adapter function that calls them at the appropriate time, and then register that adapter function using whatever API we add in the future. **Rationale for preserving the particular fields that are preserved**: Preserving ``nargs`` lets us avoid a little bit of breakage with the random dtype, and it doesn't seem like preserving ``nin``, ``nout``, ``nargs`` fields will produce any undue burden on future changes to ufunc internals; even if we were to introduce variadic ufuncs we could always just stick a -1 in the appropriate fields or whatever. **Rationale for removing PyUFunc_ReplaceLoopBySignature**: this function *returns* the PyUFunc_GenericFunction that was replaced; if we stop representing all loops using the legacy PyUFunc_GenericFunction type, then this will not be possible to do in the future. **Rationale for removing PyUFunc_SetUsesArraysAsData**: If set as the ``innerloopdata`` on a ufunc loop, then this function acts as a sentinel value, and causes the ``innerloopdata`` to instead be set to a pointer to the passed-in PyArrayObjects. In principle we could preserve this function, but it has a number of deficiencies: - No-one appears to use it. - It's been buggy for several releases and no-one noticed. - AFAIK the only reason it was included in the first place is that it provides a backdoor for ufunc loops to get access to the dtypes -- but we are planning to fix this in a better way. - It can't be shimmed as easily as the loop registration functions, because we don't anticipate that the new-and-improved ufunc loop functions will *get* access to the array objects, only to the dtypes; so this would have to remain cluttering up the core dispatch path indefinitely. - We have good reason for *not* wanting to get ufunc loops get access to the actual array objects, because one of the goals on our roadmap is exactly to enable the use of ufuncs on non-ndarray objects. Giving ufuncs access to dtypes alone creates a clean boundary here: it guarantees that ufunc loops can work equally on all duck-array objects (so long as they have a dtype), and enforces the invariant that anything which affects the interpretation of data values should be attached to the dtype, not to the array object. Rejected alternatives --------------------- **Do nothing**: there's no way we'll ever be able to touch ufuncs at all if we don't hide those fields sooner or later. While any amount of breakage is regrettable, the costs of cleaning things up now are less than the costs of never improving numpy's APIs. **Somehow sneak the dtype information in via ``void *innerloopdata``**: This might let us preserve the signature of PyUFunc_GenericFunction, and thus preserve PyUFunc_ReplaceLoopBySignature. But we'd still have the problem of leaving way too much internal state exposed, and it's not even clear how this would work, given that we actually do want to preserve the use of ``innerloopdata`` for actual per-loop data. (This is where the PyUFunc_SetUsesArraysAsData hack fails.) -- Nathaniel J. Smith -- http://vorpus.org From njs at pobox.com Mon Sep 21 01:17:07 2015 From: njs at pobox.com (Nathaniel Smith) Date: Sun, 20 Sep 2015 22:17:07 -0700 Subject: [Numpy-discussion] Update on the recent mailing list outage Message-ID: Hi all, I'm sure you all noticed that numpy-discussion was down for 4-5 days this week. Apparently this was due to some unnoticed misconfigurations that were triggered when moving it from the ancient and unmaintainable server that it's been on for the last umpteen years. Some more details are here: https://github.com/numpy/numpy/issues/6325 There's some hope that this should reduce the frequency of outages going forward, though it's still not entirely clear how stable this new setup is or who is responsible for maintaining the list now or in the future -- if you know then please speak up :-). -n -- Nathaniel J. Smith -- http://vorpus.org From sebastian at sipsolutions.net Mon Sep 21 05:32:13 2015 From: sebastian at sipsolutions.net (Sebastian Berg) Date: Mon, 21 Sep 2015 11:32:13 +0200 Subject: [Numpy-discussion] Governance model request In-Reply-To: References: Message-ID: <1442827933.393.61.camel@sipsolutions.net> On So, 2015-09-20 at 11:20 -0700, Travis Oliphant wrote: > After long conversations at BIDS this weekend and after reading the > entire governance document, I realized that the steering council is > very large and I don't agree with the mechanism by which it is > chosen. > Hmmm, well I never had the impression that the steering council would be huge. But maybe you are right, and if it is, I could imagine something like option 2, but vote based (could possibly dual use those in charge of NumFOCUS relations, we had even discussed this possibility) which would have final say if necessary (could mean that the contributers definition could be broadened a bit). However, I am not sure this is what you suggested, because for me it should be a regular vote (if just because I am scared of having to make the right pick). And while I will not block this if others agree, I am currently not comfortable with either picking a BDFL (sorry guys :P) or very fond of an oligarchy for live. Anyway, I still don't claim to have a good grasp on these things, but without a vote, it seems a bit what Matthew warned about. One thing I could imagine is something like an "Advisory Board", without (much) formal power. If we had a voted Steering Council, it could be the former members + old time contributers which we would choose now. These could be invited to meetings at the very least. Just my current, probably not well thought out thoughts on the matter. But neither of your three options feel very obvious to me unfortunately. - Sebastian > A one year time frame is pretty short on the context of a two decades > old project and I believe the current council has too few people who > have been around the community long enough to help unstuck difficult > situations if that were necessary. > > I would recommend three possible adjustments to the steering council > concept. > > 1 - define a BDFL for the council. I would nominate chuck Harris > > 2 - limit the council to 3 people. I would nominate chuck, nathaniel, > and pauli. > > 3 - add me as a permanent member of the steering council. > > Writing NumPy was a significant amount of work. I have been working > indirectly or directly in support of NumPy continously since I wrote > it. While I don't actively participate all the time, I still have a > lot of knowledge, context, and experience in how NumPy is used, why it > is the way it is, and how things could be better. I also work with > people directly who have and will contribute regularly. > > I am formally requesting that the steering council concept be adjusted > in one of these three ways. > > Thanks, > > Travis > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > https://mail.scipy.org/mailman/listinfo/numpy-discussion -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 819 bytes Desc: This is a digitally signed message part URL: From sebastian at sipsolutions.net Mon Sep 21 05:39:54 2015 From: sebastian at sipsolutions.net (Sebastian Berg) Date: Mon, 21 Sep 2015 11:39:54 +0200 Subject: [Numpy-discussion] Governance model request In-Reply-To: <1442827933.393.61.camel@sipsolutions.net> References: <1442827933.393.61.camel@sipsolutions.net> Message-ID: <1442828394.393.64.camel@sipsolutions.net> On Mo, 2015-09-21 at 11:32 +0200, Sebastian Berg wrote: > On So, 2015-09-20 at 11:20 -0700, Travis Oliphant wrote: > > After long conversations at BIDS this weekend and after reading the > > entire governance document, I realized that the steering council is > > very large and I don't agree with the mechanism by which it is > > chosen. > > > > Hmmm, well I never had the impression that the steering council would be > huge. But maybe you are right, and if it is, I could imagine something > like option 2, but vote based (could possibly dual use those in charge > of NumFOCUS relations, we had even discussed this possibility) which > would have final say if necessary (could mean that the contributers > definition could be broadened a bit). > However, I am not sure this is what you suggested, because for me it > should be a regular vote (if just because I am scared of having to make > the right pick). And while I will not block this if others agree, I am > currently not comfortable with either picking a BDFL (sorry guys :P) or > very fond of an oligarchy for live. > > Anyway, I still don't claim to have a good grasp on these things, but > without a vote, it seems a bit what Matthew warned about. > > One thing I could imagine is something like an "Advisory Board", without > (much) formal power. If we had a voted Steering Council, it could be the > former members + old time contributers which we would choose now. These > could be invited to meetings at the very least. > > Just my current, probably not well thought out thoughts on the matter. > But neither of your three options feel very obvious to me unfortunately. > > - Sebastian > > > > A one year time frame is pretty short on the context of a two decades > > old project and I believe the current council has too few people who > > have been around the community long enough to help unstuck difficult > > situations if that were necessary. > > > > I would recommend three possible adjustments to the steering council > > concept. > > > > 1 - define a BDFL for the council. I would nominate chuck Harris > > > > 2 - limit the council to 3 people. I would nominate chuck, nathaniel, > > and pauli. > > > > 3 - add me as a permanent member of the steering council. > > Though, maybe you should be in the steering council in any case even by the current rules. Maybe you were not too active for a while, but I doubt you will quite stop doing stuff on numpy soon.... > > Writing NumPy was a significant amount of work. I have been working > > indirectly or directly in support of NumPy continously since I wrote > > it. While I don't actively participate all the time, I still have a > > lot of knowledge, context, and experience in how NumPy is used, why it > > is the way it is, and how things could be better. I also work with > > people directly who have and will contribute regularly. > > > > I am formally requesting that the steering council concept be adjusted > > in one of these three ways. > > > > Thanks, > > > > Travis > > > > _______________________________________________ > > NumPy-Discussion mailing list > > NumPy-Discussion at scipy.org > > https://mail.scipy.org/mailman/listinfo/numpy-discussion > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > https://mail.scipy.org/mailman/listinfo/numpy-discussion -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 819 bytes Desc: This is a digitally signed message part URL: From solipsis at pitrou.net Mon Sep 21 05:44:34 2015 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 21 Sep 2015 11:44:34 +0200 Subject: [Numpy-discussion] draft NEP for breaking ufunc ABI in a controlled way References: Message-ID: <20150921114434.6c4bf981@fsol> Hi Nathaniel, On Sun, 20 Sep 2015 21:13:30 -0700 Nathaniel Smith wrote: > Given this, I propose that for 1.11 we: > 1) go ahead and hide/disable the problematic parts of the ABI/API, > 2) coordinate with the known affected projects to minimize disruption > to their users (which is made easier since they are all projects that > are almost exclusively distributed via conda, which enforces strict > NumPy ABI versioning), > 3) publicize these changes widely so as to give any private code that > might be affected a chance to speak up or adapt, and > 4) leave the "ABI version tag" as it is, so as not to force rebuilds > of the vast majority of projects that will be unaffected by these > changes. Thanks for a detailed and clear explanation of the proposed changes. As far as Numba is concerned, making changes is ok for us provided Numpy provides APIs to do what we want. Regards Antoine. From jaime.frio at gmail.com Mon Sep 21 10:29:08 2015 From: jaime.frio at gmail.com (=?UTF-8?Q?Jaime_Fern=C3=A1ndez_del_R=C3=ADo?=) Date: Mon, 21 Sep 2015 07:29:08 -0700 Subject: [Numpy-discussion] draft NEP for breaking ufunc ABI in a controlled way In-Reply-To: References: Message-ID: We have the PyArrayObject vs PyArrayObject_fields definition in ndarraytypes.h that is used to enforce access to the members through inline functions rather than directly, which seems to me like the right way to go: don't leave stones unturned, hide everything and provide PyUFunc_NIN, PyUFunc_NOUT and friends to handle those too. On Sun, Sep 20, 2015 at 9:13 PM, Nathaniel Smith wrote: > Hi all, > > Here's a first draft NEP for comments. > > -- > > Synopsis > ======== > > Improving numpy's dtype system requires that ufunc loops start having > access to details of the specific dtype instance they are acting on: > e.g. an implementation of np.equal for strings needs access to the > dtype object in order to know what "n" to pass to strncmp. Similar > issues arise with variable length strings, missing values, categorical > data, unit support, datetime with timezone support, etc. -- this is a > major blocker for improving numpy. > > Unfortunately, the current ufunc inner loop function signature makes > it very difficult to provide this information. We might be able to > wedge it in there, but it'd be ugly. > > The other option would be to change the signature. What would happen > if we did this? For most common uses of the C API/ABI, we could do > this easily while maintaining backwards compatibility. But there are > also some rarely-used parts of the API/ABI that would be > prohibitively difficult to preserve. > > In addition, there are other potential changes to ufuncs on the > horizon (e.g. extensions of gufuncs to allow them to be used more > generally), and the current API exposure is so massive that any such > changes will be difficult to make in a fully compatible way. This NEP > thus considers the possibility of closing down the ufunc API to a > minimal, maintainable subset of the current API. > > To better understand the consequences of this potential change, I > performed an exhaustive analysis of all the code on Github, Bitbucket, > and Fedora, among others. The results make me highly confident that of > all the publically available projects in the world, the only ones > which touch the problematic parts of the ufunc API are: Numba, > dynd-python, and `gulinalg `_ > (with the latter's exposure being trivial). > > Given this, I propose that for 1.11 we: > 1) go ahead and hide/disable the problematic parts of the ABI/API, > 2) coordinate with the known affected projects to minimize disruption > to their users (which is made easier since they are all projects that > are almost exclusively distributed via conda, which enforces strict > NumPy ABI versioning), > 3) publicize these changes widely so as to give any private code that > might be affected a chance to speak up or adapt, and > 4) leave the "ABI version tag" as it is, so as not to force rebuilds > of the vast majority of projects that will be unaffected by these > changes. > > This NEP defers the question of exactly what the improved API should > be, since there's no point in trying to nail down the details until > we've decided whether it's even possible to change. > > > Details > ======= > > The problem > ----------- > > Currently, a ufunc inner loop implementation is called via the > following function prototype:: > > typedef void (*PyUFuncGenericFunction) > (char **args, > npy_intp *dimensions, > npy_intp *strides, > void *innerloopdata); > > Here ``args`` is an array of pointers to 1-d buffers of input/output > data, ``dimensions`` is a pointer to the number of entries in these > buffers, ``strides`` is an array of integers giving the strides for > each input/output array, and ``innerloopdata`` is an arbitrary void* > supplied by whoever registered the ufunc loop. (For gufuncs, extra > shape and stride information about the core dimensions also gets > packed into the ends of these arrays in a somewhat complicated way.) > > There are 4 key items that define a NumPy array: data, shape, strides, > dtype. Notice that this function only gets access to 3 of them. Our > goal is to fix that. For example, a better signature would be:: > > typedef void (*PyUFuncGenericFunction_NEW) > (char **data, > npy_intp *shapes, > npy_intp *strides, > PyArray_Descr *dtypes, /* NEW */ > void *innerloopdata); > > (In practice I suspect we might want to make some more changes as > well, like upgrading gufunc core shape/strides to proper arguments > instead of tacking it onto the existing arrays, and adding an "escape > valve" void* reserved for future extensions. But working out such > details is outside the scope of this NEP; the above will do for > illustration.) > > The goal of this NEP is to clear the ground so that we can start > supporting ufunc inner loops that take dtype arguments, and make other > enhancements to ufunc functionality going forward. > > > Proposal > -------- > > Currently, the public API/ABI for ufuncs consists of the functions:: > > PyUFunc_GenericFunction > > PyUFunc_FromFuncAndData > PyUFunc_FromFuncAndDataAndSignature > PyUFunc_RegisterLoopForDescr > PyUFunc_RegisterLoopForType > > PyUFunc_ReplaceLoopBySignature > PyUFunc_SetUsesArraysAsData > > together with direct access to PyUFuncObject's internal fields:: > > typedef struct { > PyObject_HEAD > int nin, nout, nargs; > int identity; > PyUFuncGenericFunction *functions; > void **data; > int ntypes; > int check_return; > const char *name; > char *types; > const char *doc; > void *ptr; > PyObject *obj; > PyObject *userloops; > int core_enabled; > int core_num_dim_ix; > int *core_num_dims; > int *core_dim_ixs; > int *core_offsets; > char *core_signature; > PyUFunc_TypeResolutionFunc *type_resolver; > PyUFunc_LegacyInnerLoopSelectionFunc *legacy_inner_loop_selector; > PyUFunc_InnerLoopSelectionFunc *inner_loop_selector; > PyUFunc_MaskedInnerLoopSelectionFunc *masked_inner_loop_selector; > npy_uint32 *op_flags; > npy_uint32 iter_flags; > } PyUFuncObject; > > Obviously almost any future changes to how ufuncs work internally will > involve touching some part of this public API/ABI. > > Concretely, the proposal here is that we avoid this by disabling the > following functions (i.e., any attempt to call them should simply > raise a ``NotImplementedError``):: > > PyUFunc_ReplaceLoopBySignature > PyUFunc_SetUsesArraysAsData > > and that we reduce the publicly visible portion of PyUFuncObject down to:: > > typedef struct { > PyObject_HEAD > int nin, nout, nargs; > } PyUFuncObject; > > > Data on current API/ABI usage > ----------------------------- > > In order to assess how much code would be affected by this proposal, I > used a combination of Github search and Searchcode.com to trawl > through the majority of all publicly available open source code. > Neither search tool provides a fine-grained enough query language to > directly tell us what we want to know, so I instead followed the > strategy of first, casting a wide net: picking a set of search terms > that are likely to catch all possibly-broken code (together with many > false positives), and second, using automated tools to sift out the > false positives and see what remained. Altogether, I reviewed 4464 > search results. > > The tool I wrote to do this is `available on github > `_, and so is `the analysis code > itself `_. > > > Uses of PyUFuncObject internals > ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ > > There are no functions in the public API which return > ``PyUFuncObject*`` values directly, so any code that access > PyUFuncObject fields will have to mention that token in the course of > defining a variable, performing a cast, setting up a typedef, etc. > > Therefore, I searched Github for all files written in C, C++, > Objective C, Python, or Cython, which mentioned either "PyUFuncObject > AND types" or "PyUFuncObject AND NOT types". (This is to work around > limitations on how many results Github search is willing to return to > a single query.) In addition, I searched for ``PyUFuncObject`` on > searchcode.com. > > The full report on these searches is available here: > > https://rawgit.com/njsmith/ufunc-abi-analysis/master/reports/pyufuncobject-report.html > > The following were screened out as non-problems: > > - Copies of NumPy itself (an astonishing number of people have checked > in copies of it to their own source tree) > - NumPy forks / precursors / etc. (e.g. Numeric also had a type called > PyUFuncObject, the "bohrium" project has a fork of numpy 1.6, etc.) > - Cython-generated boilerplate used to generate the "object has > changed size" warning (which we `unconditionally filter out anyway > `_) > - Lots of calls to ``PyUFunc_RegisterLoopForType`` and friends, which > require casting the first argument to ``PyUFuncObject*`` > - Misc. other unproblematic stuff (like Cython header declarations > that never get used) > > There were also several cases that actually referenced PyUFuncObject > internal fields: > > - The "rational" dtype from numpy-dtypes, which is used in a few > projects, accesses ``ufunc->nargs`` as a safety check, but does not > touch any other fields (`see here > < > https://github.com/numpy/numpy-dtypes/blob/c0175a6b1c5aa89b4520b29487f06d0e200e2a03/npytypes/rational/rational.c#L1140-L1151 > >`_). > > - Numba: does some rather elaborate things to support the definition > of on-the-fly JITted ufuncs. These seem to be clear deficiencies in > the ufunc API (e.g., there's no good way to control the lifespan of > the array of function pointers passed to ``PyUFunc_FromFuncAndData``), > so we should work with them to provide the API they need to do this in > a maintainable way. Some of the relevant code: > > https://github.com/numba/numba/tree/master/numba/npyufunc > > https://github.com/numba/numba/blob/98752647a95ac6c9d480e81ca5c8afcfa3ddfd18/numba/npyufunc/_internal.c > > - dynd-python: Contains some code that attempts to extract the inner > loops from a numpy ufunc object and wrap them into the dynd 'ufunc' > equivalent: > > https://github.com/libdynd/dynd-python/blob/c06f8fc4e72257abac589faf76f10df8c045159b/dynd/src/numpy_ufunc_kernel.cpp > > - gulinalg: I'm not sure if anyone is still using this code since most > of it was merged into numpy itself, but it's not a big deal in any > case: all it contains is a `debugging function > < > https://github.com/ContinuumIO/gulinalg/blob/2ef365c48427c026dab4f45dc6f8b1b9af184460/gulinalg/src/gulinalg.c.src#L527-L550 > >`_ > that dumps some internal fields from the PyUFuncObject. If you look, > though, all calls to this function are already commented out :-). > > The full report is available here: > > https://rawgit.com/njsmith/ufunc-abi-analysis/master/reports/pyufuncobject-report.html > > In the course of this analysis, it was also noted that the standard > Cython pxd files contain a wrapper for ufunc objects:: > > cdef class ufunc [object PyUFuncObject]: > ... > > which means that Cython code can access internal struct fields via an > object of type ``ufunc``, and thus escape our string-based search > above. Therefore I also examined all Cython files on Github or > searchcode.com that matched the query ``ufunc``, and searched for any > lines matching any of the following regular expressions:: > > cdef\W+ufunc > catches: 'cdef ufunc fn' > cdef\W+.*\.\W*ufunc > catches: 'cdef np.ufunc fn' > <.*ufunc\W*> > catches: '( fn).nargs', '(< np.ufunc > fn).nargs' > cdef.*\(.*ufunc > catches: 'cdef doit(np.ufunc fn, ...):' > > (I considered parsing the actual source and analysing it that way, but > decided I was too lazy. This could be done if anyone is worried that > the above regexes might miss things though.) > > There were zero files that contained matches for any of the above regexes: > > https://rawgit.com/njsmith/ufunc-abi-analysis/master/reports/ufunc-cython-report.html > > > Uses of PyUFunc_ReplaceLoopBySignature > ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ > > Applying the same screening as above, the only code that was found > that used this function is also in Numba: > > https://rawgit.com/njsmith/ufunc-abi-analysis/master/reports/PyUFunc_ReplaceLoopBySignature-report.html > > > Uses of PyUFunc_SetUsesArraysAsData > ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ > > Aside from being semi-broken since 1.7 (it never got implemented for > "masked" ufunc loops, i.e. those that use where=), there appear to be > zero uses of this functionality either inside or outside NumPy: > > https://rawgit.com/njsmith/ufunc-abi-analysis/master/reports/PyUFunc_SetUsesArraysAsData-report.html > > > Rationale > --------- > > **Rationale for preserving the remaining API functions**:: > > PyUFunc_GenericFunction > > PyUFunc_FromFuncAndData > PyUFunc_FromFuncAndDataAndSignature > PyUFunc_RegisterLoopForDescr > PyUFunc_RegisterLoopForType > > In addition to being widely used, these functions can easily be > preserved even if we change how ufuncs work internally, because they > only ingest loop function pointers, they never return them. So they > can easily be modified to wrap whatever loop function(s) they receive > inside an adapter function that calls them at the appropriate time, > and then register that adapter function using whatever API we add in > the future. > > **Rationale for preserving the particular fields that are preserved**: > Preserving ``nargs`` lets us avoid a little bit of breakage with the > random dtype, and it doesn't seem like preserving ``nin``, ``nout``, > ``nargs`` fields will produce any undue burden on future changes to > ufunc internals; even if we were to introduce variadic ufuncs we could > always just stick a -1 in the appropriate fields or whatever. > > **Rationale for removing PyUFunc_ReplaceLoopBySignature**: this > function *returns* the PyUFunc_GenericFunction that was replaced; if > we stop representing all loops using the legacy > PyUFunc_GenericFunction type, then this will not be possible to do in > the future. > > **Rationale for removing PyUFunc_SetUsesArraysAsData**: If set as the > ``innerloopdata`` on a ufunc loop, then this function acts as a > sentinel value, and causes the ``innerloopdata`` to instead be set to > a pointer to the passed-in PyArrayObjects. In principle we could > preserve this function, but it has a number of deficiencies: > - No-one appears to use it. > - It's been buggy for several releases and no-one noticed. > - AFAIK the only reason it was included in the first place is that it > provides a backdoor for ufunc loops to get access to the dtypes -- but > we are planning to fix this in a better way. > - It can't be shimmed as easily as the loop registration functions, > because we don't anticipate that the new-and-improved ufunc loop > functions will *get* access to the array objects, only to the dtypes; > so this would have to remain cluttering up the core dispatch path > indefinitely. > - We have good reason for *not* wanting to get ufunc loops get access > to the actual array objects, because one of the goals on our roadmap > is exactly to enable the use of ufuncs on non-ndarray objects. Giving > ufuncs access to dtypes alone creates a clean boundary here: it > guarantees that ufunc loops can work equally on all duck-array objects > (so long as they have a dtype), and enforces the invariant that > anything which affects the interpretation of data values should be > attached to the dtype, not to the array object. > > > Rejected alternatives > --------------------- > > **Do nothing**: there's no way we'll ever be able to touch ufuncs at > all if we don't hide those fields sooner or later. While any amount of > breakage is regrettable, the costs of cleaning things up now are less > than the costs of never improving numpy's APIs. > > **Somehow sneak the dtype information in via ``void > *innerloopdata``**: This might let us preserve the signature of > PyUFunc_GenericFunction, and thus preserve > PyUFunc_ReplaceLoopBySignature. But we'd still have the problem of > leaving way too much internal state exposed, and it's not even clear > how this would work, given that we actually do want to preserve the > use of ``innerloopdata`` for actual per-loop data. (This is where the > PyUFunc_SetUsesArraysAsData hack fails.) > > > -- > Nathaniel J. Smith -- http://vorpus.org > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > https://mail.scipy.org/mailman/listinfo/numpy-discussion > -- (\__/) ( O.o) ( > <) Este es Conejo. Copia a Conejo en tu firma y ay?dale en sus planes de dominaci?n mundial. -------------- next part -------------- An HTML attachment was scrubbed... URL: From travis at continuum.io Mon Sep 21 12:20:33 2015 From: travis at continuum.io (Travis Oliphant) Date: Mon, 21 Sep 2015 09:20:33 -0700 Subject: [Numpy-discussion] Governance model request In-Reply-To: <1442828394.393.64.camel@sipsolutions.net> References: <1442827933.393.61.camel@sipsolutions.net> <1442828394.393.64.camel@sipsolutions.net> Message-ID: I wrote my recommendations quickly before heading on a plane. I hope the spirit of them was caught correctly. I also want to re-emphasize that I completely understand that the Steering Council is not to be making decisions that often and almost all activity will be similar to it is now --- discussion, debate, proposals, and pull-requests --- that is a good thing. However, there is a need for leadership to help unstick things and move the project forward from time to time because quite often doing *something* can be better than trying to please everyone with a voice. My concerns about how to do this judgment have 2 major components: 1) The need for long-term consistency --- a one-year horizon on defining this group is too short in my mind for a decades-old project like NumPy. 2) The group that helps unstick things needs to be small (1, 3, or 5 at the most) We could call this group the "adjudication group" rather than the "Steering Council" as well. I could see that having a formal method of changing that "adjudication group" would be a good idea as well (and perhaps that formal vote could be made by a vote of a group of active contributors. In that case, I would define active as having a time-window of 5 years instead of just 1). Thanks, -Travis On Mon, Sep 21, 2015 at 2:39 AM, Sebastian Berg wrote: > On Mo, 2015-09-21 at 11:32 +0200, Sebastian Berg wrote: > > On So, 2015-09-20 at 11:20 -0700, Travis Oliphant wrote: > > > After long conversations at BIDS this weekend and after reading the > > > entire governance document, I realized that the steering council is > > > very large and I don't agree with the mechanism by which it is > > > chosen. > > > > > > > Hmmm, well I never had the impression that the steering council would be > > huge. But maybe you are right, and if it is, I could imagine something > > like option 2, but vote based (could possibly dual use those in charge > > of NumFOCUS relations, we had even discussed this possibility) which > > would have final say if necessary (could mean that the contributers > > definition could be broadened a bit). > > However, I am not sure this is what you suggested, because for me it > > should be a regular vote (if just because I am scared of having to make > > the right pick). And while I will not block this if others agree, I am > > currently not comfortable with either picking a BDFL (sorry guys :P) or > > very fond of an oligarchy for live. > > > > Anyway, I still don't claim to have a good grasp on these things, but > > without a vote, it seems a bit what Matthew warned about. > > > > One thing I could imagine is something like an "Advisory Board", without > > (much) formal power. If we had a voted Steering Council, it could be the > > former members + old time contributers which we would choose now. These > > could be invited to meetings at the very least. > > > > Just my current, probably not well thought out thoughts on the matter. > > But neither of your three options feel very obvious to me unfortunately. > > > > - Sebastian > > > > > > > A one year time frame is pretty short on the context of a two decades > > > old project and I believe the current council has too few people who > > > have been around the community long enough to help unstuck difficult > > > situations if that were necessary. > > > > > > I would recommend three possible adjustments to the steering council > > > concept. > > > > > > 1 - define a BDFL for the council. I would nominate chuck Harris > > > > > > 2 - limit the council to 3 people. I would nominate chuck, nathaniel, > > > and pauli. > > > > > > 3 - add me as a permanent member of the steering council. > > > > > Though, maybe you should be in the steering council in any case even by > the current rules. Maybe you were not too active for a while, but I > doubt you will quite stop doing stuff on numpy soon.... > > > > > Writing NumPy was a significant amount of work. I have been working > > > indirectly or directly in support of NumPy continously since I wrote > > > it. While I don't actively participate all the time, I still have a > > > lot of knowledge, context, and experience in how NumPy is used, why it > > > is the way it is, and how things could be better. I also work with > > > people directly who have and will contribute regularly. > > > > > > I am formally requesting that the steering council concept be adjusted > > > in one of these three ways. > > > > > > Thanks, > > > > > > Travis > > > > > > _______________________________________________ > > > NumPy-Discussion mailing list > > > NumPy-Discussion at scipy.org > > > https://mail.scipy.org/mailman/listinfo/numpy-discussion > > > > _______________________________________________ > > NumPy-Discussion mailing list > > NumPy-Discussion at scipy.org > > https://mail.scipy.org/mailman/listinfo/numpy-discussion > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > https://mail.scipy.org/mailman/listinfo/numpy-discussion > > -- *Travis Oliphant* *Co-founder and CEO* @teoliphant 512-222-5440 http://www.continuum.io -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.barker at noaa.gov Mon Sep 21 17:22:47 2015 From: chris.barker at noaa.gov (Chris Barker) Date: Mon, 21 Sep 2015 14:22:47 -0700 Subject: [Numpy-discussion] Governance model request In-Reply-To: References: Message-ID: On Sun, Sep 20, 2015 at 11:20 AM, Travis Oliphant wrote: > After long conversations at BIDS this weekend and after reading the entire > governance document, I realized that the steering council is very large > How large are we talking? I think there were 8 people named -- and I'm not sure all 8 want to do it. But 5 or seems like a fine number -- I wonder if defining the number is a good idea. > A one year time frame is pretty short on the context of a two decades old > project > I see: """ who have produced contributions that are substantial in quality and quantity, and sustained over at least one year """ that could be longer, but it looks like it DOESN'T mean "have contributed within the last year" -- so we could certainly, at this point, have a lot of folks that have been around a long time that could qualify. A certain guy named Travis come to mind... and I believe the current council has too few people who have been around > the community long enough to help unstuck difficult situations if that were > necessary. > That's actually a bad sign right there -- the suggested group are the folks that have actually been contributing lately -- too bad that apparently isn't anyone that has been around a long time :-( also see: """ This will include but is not limited to code, code review, infrastructure work, mailing list and chat participation, community help/building, education and outreach, design work, etc. """ so it's not only people doing the actual coding ( A really small number :-( ) But I note that the the suggestion to "seed the council" with "everyone who has reviewed/merged a pull request since Jan 1, 2014" - I'm not sure that gives us as diverse a council as we'd like -- but it is only the seed... > > 1 - define a BDFL for the council. > Well, that would make a number of things easier -- but I think there was a consensus that there simply was no obvious candidate -- and if the candidate is not obvious, then they can't really be a BDFL. > 2 - limit the council to 3 people. I would nominate chuck, nathaniel, and > pauli. > Good group -- but maybe 3 a bit too small. > 3 - add me as a permanent member of the steering council. > I'd love to see you on the council, and there is no question of your qualifications -- but I don't think anyone should be anymore permanent than any other. According to the docs so far, the only thing anyone needs to do to remain on the council is stay active -- and not piss everyone off so much as to get a consensus from everyone else that you need to be kicked off. And the end of this -- I think the big question still at hand is how big the council should be. My experience with consensus suggests that it not be very big :-) -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Mon Sep 21 18:19:44 2015 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 21 Sep 2015 16:19:44 -0600 Subject: [Numpy-discussion] 1.10.0rc1 coming tomorrow, 22 Sept. Message-ID: Hi All, Just a heads up. The lack of reported problems in 1.10.0b1 has been stunning. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From matthew.brett at gmail.com Mon Sep 21 19:54:46 2015 From: matthew.brett at gmail.com (Matthew Brett) Date: Mon, 21 Sep 2015 16:54:46 -0700 Subject: [Numpy-discussion] ANN: Numpy 1.9.3 bugfix release Message-ID: Hi, We have just released numpy 1.9.3, a small bugfix release to fix a bug on Python 3.5, as well as some build issues. You likely only need to upgrade from 1.9.2 if you are on Python 3.5. There are source and OSX wheels up on pypi. We currently have no plans to add 1.9.3 to the sourceforge site, because we cannot yet build a Python 3.5 installer with the standard mingw compiler. Issues fixed ============ * `#5866 `__: fix error finding Python headers when ``build_ext`` ``--include-dirs`` is set; * `#6016 `__: fix ``np.loadtxt`` error on Python 3.5 when reading from gzip files; * `#5555 `__: Replace deprecated options for ifort; * `#6096 `__: remove /GL for VS2015 in check_long_double_representation; * `#6141 `__: enable Visual Studio 2015 C99 features; * `#6171 `__: revert C99 complex for MSVC14. Cheers, Matthew Brett - for the numpy developers. From matthew.brett at gmail.com Mon Sep 21 21:24:16 2015 From: matthew.brett at gmail.com (Matthew Brett) Date: Mon, 21 Sep 2015 18:24:16 -0700 Subject: [Numpy-discussion] Governance model request In-Reply-To: References: Message-ID: Hi Travis, and all, You might have seen I was advocating for having someone who takes final responsibility for the project, partly to get discussions unstuck, as you said. I agree with Chris, that at this stage, there is no-one who could be Benevolent Dictator for the project. It seems to me that (nearly) everyone would agree that, if there is a leader, we would not want a person who dictates, but someone who is good at putting in the hard work and time it takes to hear all the arguments, and guide people to agreement, where that is possible. If there is a leader, I think we should select them for those skills. For the proposal that you join the steering committee, I see two problems. The first is, that the members of the steering committee have to be the people who are keeping in touch with the day to day work of the project. I am sure you would agree that, in the past, you have not had enough time for that. Your recent emails about the new work you want to do, also imply that you may be too busy to get involved in the detailed discussions needed for getting the code merged. In any case, I think you'd also agree that in the past you have hoped that you would have more time for numpy than you did. The second problem is that you have a potential conflict of interest, in that it is possible for the needs of Continuum to conflict with the needs of numpy. I believe, from previous emails on this list, that you don't think that is very important, but I continue to disagree about that. For example, see this interview with Linus Torvalds, where he talks about going out of his way to make sure that people can trust his decisions, despite the fact he is paid to work on Linux [1]. In practice, the most obvious step that I can think of, is to defer the decision as to whether you are on the steering committee for six months. I guess over that time you will be working with the other numpy developers more closely. I should say that Stefan and I are working on a governance proposal, in this case for scikit-image, where we split the governance into 1) the developers doing the work and making the day to day decisions and 2) the trustees, usually from other relevant projects, or no-longer-active developers, who make sure the project does not go off the rails. I think you'd be an excellent and obvious trustee, in that model. Cheers, Matthew [1] http://www.bbc.com/news/technology-18419231 From bryanv at continuum.io Mon Sep 21 21:33:31 2015 From: bryanv at continuum.io (Bryan Van de Ven) Date: Mon, 21 Sep 2015 21:33:31 -0400 Subject: [Numpy-discussion] Governance model request In-Reply-To: References: Message-ID: <8941C0D2-17EC-420D-A8F6-CA018144811C@continuum.io> > On Sep 21, 2015, at 9:24 PM, Matthew Brett wrote: > > The second problem is that you have a potential conflict of interest, > in that it is possible for the needs of Continuum to conflict with the > needs of numpy. I believe, from previous emails on this list, that > you don't think that is very important, but I continue to disagree > about that. For example, see this interview with Linus Torvalds, Can you actually spell out some particular, articulable, concrete points of concern? Otherwise this just seems like idle, non-constructive speculation (at best). Bryan From cournape at gmail.com Mon Sep 21 21:42:04 2015 From: cournape at gmail.com (David Cournapeau) Date: Mon, 21 Sep 2015 18:42:04 -0700 Subject: [Numpy-discussion] Governance model request In-Reply-To: <8941C0D2-17EC-420D-A8F6-CA018144811C@continuum.io> References: <8941C0D2-17EC-420D-A8F6-CA018144811C@continuum.io> Message-ID: On Mon, Sep 21, 2015 at 6:33 PM, Bryan Van de Ven wrote: > > > On Sep 21, 2015, at 9:24 PM, Matthew Brett > wrote: > > > > The second problem is that you have a potential conflict of interest, > > in that it is possible for the needs of Continuum to conflict with the > > needs of numpy. I believe, from previous emails on this list, that > > you don't think that is very important, but I continue to disagree > > about that. For example, see this interview with Linus Torvalds, > > Can you actually spell out some particular, articulable, concrete points > of concern? Otherwise this just seems like idle, non-constructive > speculation (at best). > There is ample history of such things happening in OSS history, so I think that's a fair concern, even if that has not happened for numpy yet. David -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Tue Sep 22 00:23:42 2015 From: njs at pobox.com (Nathaniel Smith) Date: Mon, 21 Sep 2015 21:23:42 -0700 Subject: [Numpy-discussion] draft NEP for breaking ufunc ABI in a controlled way In-Reply-To: References: Message-ID: On Mon, Sep 21, 2015 at 7:29 AM, Jaime Fern?ndez del R?o wrote: > We have the PyArrayObject vs PyArrayObject_fields definition in > ndarraytypes.h that is used to enforce access to the members through inline > functions rather than directly, which seems to me like the right way to go: > don't leave stones unturned, hide everything and provide PyUFunc_NIN, > PyUFunc_NOUT and friends to handle those too. The PyArrayObject vs PyArrayObject_fields distinction is only enabled if a downstream library explicitly requests it with #define NPY_NO_DEPRECATED_API, though -- the idea is that the changes in this NEP would be enabled unconditionally, even for old code. So the reason nin/nout/nargs are left exposed in this proposal is that there's some existing code out there that would break (until updated) if we hid them, and not much benefit to breaking it. If we're fine with breaking that code then we could just hide them unconditionally too. The only code I found in the wild that would be affected is the "rational" user-defined dtype, which would be trivially fixable since the only thing it does with ufunc->nargs is a quick consistency check: https://github.com/numpy/numpy-dtypes/blob/c0175a6b1c5aa89b4520b29487f06d0e200e2a03/npytypes/rational/rational.c#L1140-L1151 Also it's not 100% clear right now whether we even want to keep supporting the old user-defined dtype API that this particular code is based around. But if this code uses ufunc->nargs then perhaps other code does too? I'm open to opinions -- I doubt it matters that much either way. I just want to make sure that we can hide the other stuff :-). When it comes to evolving these APIs in general: one unfortunate thing about the PyArrayObject changes in 1.7 is that because they were implemented using *inline* functions (/macros) they haven't affected the a*B*i exposure at all, even in code that has upgraded to the new calling conventions. While user code no longer *names* the internal fields directly, we still have to implement exactly the same fields and put them in exactly the same place in memory or else break ABI. And the other unfortunate thing is that we don't really have a mechanism for saying "okay, we're dropping support for the old way of doing things in 1.xx" -- in particular the current NPY_NO_DEPRECATED_API mechanism doesn't give us any way to detect and error out if someone tries to use an old version of the APIs, so ABI breaks still mean segfaults. I'm thinking that if/when we figure out how to implement the "sliding window" API/ABI idea that we talked about at SciPy, then that will give us a strategy for cleanly transitioning to a world with a maintainable API+ABI and it becomes worth sitting down and making up a set of setters/getters for the attributes that we want to make public in a maintainable way. But until then our only real options are either hard breaks or nothing, so unless we want to do a hard break there's not much point talking about it. -n > On Sun, Sep 20, 2015 at 9:13 PM, Nathaniel Smith wrote: >> >> Hi all, >> >> Here's a first draft NEP for comments. >> >> -- >> >> Synopsis >> ======== >> >> Improving numpy's dtype system requires that ufunc loops start having >> access to details of the specific dtype instance they are acting on: >> e.g. an implementation of np.equal for strings needs access to the >> dtype object in order to know what "n" to pass to strncmp. Similar >> issues arise with variable length strings, missing values, categorical >> data, unit support, datetime with timezone support, etc. -- this is a >> major blocker for improving numpy. >> >> Unfortunately, the current ufunc inner loop function signature makes >> it very difficult to provide this information. We might be able to >> wedge it in there, but it'd be ugly. >> >> The other option would be to change the signature. What would happen >> if we did this? For most common uses of the C API/ABI, we could do >> this easily while maintaining backwards compatibility. But there are >> also some rarely-used parts of the API/ABI that would be >> prohibitively difficult to preserve. >> >> In addition, there are other potential changes to ufuncs on the >> horizon (e.g. extensions of gufuncs to allow them to be used more >> generally), and the current API exposure is so massive that any such >> changes will be difficult to make in a fully compatible way. This NEP >> thus considers the possibility of closing down the ufunc API to a >> minimal, maintainable subset of the current API. >> >> To better understand the consequences of this potential change, I >> performed an exhaustive analysis of all the code on Github, Bitbucket, >> and Fedora, among others. The results make me highly confident that of >> all the publically available projects in the world, the only ones >> which touch the problematic parts of the ufunc API are: Numba, >> dynd-python, and `gulinalg `_ >> (with the latter's exposure being trivial). >> >> Given this, I propose that for 1.11 we: >> 1) go ahead and hide/disable the problematic parts of the ABI/API, >> 2) coordinate with the known affected projects to minimize disruption >> to their users (which is made easier since they are all projects that >> are almost exclusively distributed via conda, which enforces strict >> NumPy ABI versioning), >> 3) publicize these changes widely so as to give any private code that >> might be affected a chance to speak up or adapt, and >> 4) leave the "ABI version tag" as it is, so as not to force rebuilds >> of the vast majority of projects that will be unaffected by these >> changes. >> >> This NEP defers the question of exactly what the improved API should >> be, since there's no point in trying to nail down the details until >> we've decided whether it's even possible to change. >> >> >> Details >> ======= >> >> The problem >> ----------- >> >> Currently, a ufunc inner loop implementation is called via the >> following function prototype:: >> >> typedef void (*PyUFuncGenericFunction) >> (char **args, >> npy_intp *dimensions, >> npy_intp *strides, >> void *innerloopdata); >> >> Here ``args`` is an array of pointers to 1-d buffers of input/output >> data, ``dimensions`` is a pointer to the number of entries in these >> buffers, ``strides`` is an array of integers giving the strides for >> each input/output array, and ``innerloopdata`` is an arbitrary void* >> supplied by whoever registered the ufunc loop. (For gufuncs, extra >> shape and stride information about the core dimensions also gets >> packed into the ends of these arrays in a somewhat complicated way.) >> >> There are 4 key items that define a NumPy array: data, shape, strides, >> dtype. Notice that this function only gets access to 3 of them. Our >> goal is to fix that. For example, a better signature would be:: >> >> typedef void (*PyUFuncGenericFunction_NEW) >> (char **data, >> npy_intp *shapes, >> npy_intp *strides, >> PyArray_Descr *dtypes, /* NEW */ >> void *innerloopdata); >> >> (In practice I suspect we might want to make some more changes as >> well, like upgrading gufunc core shape/strides to proper arguments >> instead of tacking it onto the existing arrays, and adding an "escape >> valve" void* reserved for future extensions. But working out such >> details is outside the scope of this NEP; the above will do for >> illustration.) >> >> The goal of this NEP is to clear the ground so that we can start >> supporting ufunc inner loops that take dtype arguments, and make other >> enhancements to ufunc functionality going forward. >> >> >> Proposal >> -------- >> >> Currently, the public API/ABI for ufuncs consists of the functions:: >> >> PyUFunc_GenericFunction >> >> PyUFunc_FromFuncAndData >> PyUFunc_FromFuncAndDataAndSignature >> PyUFunc_RegisterLoopForDescr >> PyUFunc_RegisterLoopForType >> >> PyUFunc_ReplaceLoopBySignature >> PyUFunc_SetUsesArraysAsData >> >> together with direct access to PyUFuncObject's internal fields:: >> >> typedef struct { >> PyObject_HEAD >> int nin, nout, nargs; >> int identity; >> PyUFuncGenericFunction *functions; >> void **data; >> int ntypes; >> int check_return; >> const char *name; >> char *types; >> const char *doc; >> void *ptr; >> PyObject *obj; >> PyObject *userloops; >> int core_enabled; >> int core_num_dim_ix; >> int *core_num_dims; >> int *core_dim_ixs; >> int *core_offsets; >> char *core_signature; >> PyUFunc_TypeResolutionFunc *type_resolver; >> PyUFunc_LegacyInnerLoopSelectionFunc *legacy_inner_loop_selector; >> PyUFunc_InnerLoopSelectionFunc *inner_loop_selector; >> PyUFunc_MaskedInnerLoopSelectionFunc *masked_inner_loop_selector; >> npy_uint32 *op_flags; >> npy_uint32 iter_flags; >> } PyUFuncObject; >> >> Obviously almost any future changes to how ufuncs work internally will >> involve touching some part of this public API/ABI. >> >> Concretely, the proposal here is that we avoid this by disabling the >> following functions (i.e., any attempt to call them should simply >> raise a ``NotImplementedError``):: >> >> PyUFunc_ReplaceLoopBySignature >> PyUFunc_SetUsesArraysAsData >> >> and that we reduce the publicly visible portion of PyUFuncObject down to:: >> >> typedef struct { >> PyObject_HEAD >> int nin, nout, nargs; >> } PyUFuncObject; >> >> >> Data on current API/ABI usage >> ----------------------------- >> >> In order to assess how much code would be affected by this proposal, I >> used a combination of Github search and Searchcode.com to trawl >> through the majority of all publicly available open source code. >> Neither search tool provides a fine-grained enough query language to >> directly tell us what we want to know, so I instead followed the >> strategy of first, casting a wide net: picking a set of search terms >> that are likely to catch all possibly-broken code (together with many >> false positives), and second, using automated tools to sift out the >> false positives and see what remained. Altogether, I reviewed 4464 >> search results. >> >> The tool I wrote to do this is `available on github >> `_, and so is `the analysis code >> itself `_. >> >> >> Uses of PyUFuncObject internals >> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ >> >> There are no functions in the public API which return >> ``PyUFuncObject*`` values directly, so any code that access >> PyUFuncObject fields will have to mention that token in the course of >> defining a variable, performing a cast, setting up a typedef, etc. >> >> Therefore, I searched Github for all files written in C, C++, >> Objective C, Python, or Cython, which mentioned either "PyUFuncObject >> AND types" or "PyUFuncObject AND NOT types". (This is to work around >> limitations on how many results Github search is willing to return to >> a single query.) In addition, I searched for ``PyUFuncObject`` on >> searchcode.com. >> >> The full report on these searches is available here: >> >> https://rawgit.com/njsmith/ufunc-abi-analysis/master/reports/pyufuncobject-report.html >> >> The following were screened out as non-problems: >> >> - Copies of NumPy itself (an astonishing number of people have checked >> in copies of it to their own source tree) >> - NumPy forks / precursors / etc. (e.g. Numeric also had a type called >> PyUFuncObject, the "bohrium" project has a fork of numpy 1.6, etc.) >> - Cython-generated boilerplate used to generate the "object has >> changed size" warning (which we `unconditionally filter out anyway >> `_) >> - Lots of calls to ``PyUFunc_RegisterLoopForType`` and friends, which >> require casting the first argument to ``PyUFuncObject*`` >> - Misc. other unproblematic stuff (like Cython header declarations >> that never get used) >> >> There were also several cases that actually referenced PyUFuncObject >> internal fields: >> >> - The "rational" dtype from numpy-dtypes, which is used in a few >> projects, accesses ``ufunc->nargs`` as a safety check, but does not >> touch any other fields (`see here >> >> `_). >> >> - Numba: does some rather elaborate things to support the definition >> of on-the-fly JITted ufuncs. These seem to be clear deficiencies in >> the ufunc API (e.g., there's no good way to control the lifespan of >> the array of function pointers passed to ``PyUFunc_FromFuncAndData``), >> so we should work with them to provide the API they need to do this in >> a maintainable way. Some of the relevant code: >> >> https://github.com/numba/numba/tree/master/numba/npyufunc >> >> https://github.com/numba/numba/blob/98752647a95ac6c9d480e81ca5c8afcfa3ddfd18/numba/npyufunc/_internal.c >> >> - dynd-python: Contains some code that attempts to extract the inner >> loops from a numpy ufunc object and wrap them into the dynd 'ufunc' >> equivalent: >> >> https://github.com/libdynd/dynd-python/blob/c06f8fc4e72257abac589faf76f10df8c045159b/dynd/src/numpy_ufunc_kernel.cpp >> >> - gulinalg: I'm not sure if anyone is still using this code since most >> of it was merged into numpy itself, but it's not a big deal in any >> case: all it contains is a `debugging function >> >> `_ >> that dumps some internal fields from the PyUFuncObject. If you look, >> though, all calls to this function are already commented out :-). >> >> The full report is available here: >> >> https://rawgit.com/njsmith/ufunc-abi-analysis/master/reports/pyufuncobject-report.html >> >> In the course of this analysis, it was also noted that the standard >> Cython pxd files contain a wrapper for ufunc objects:: >> >> cdef class ufunc [object PyUFuncObject]: >> ... >> >> which means that Cython code can access internal struct fields via an >> object of type ``ufunc``, and thus escape our string-based search >> above. Therefore I also examined all Cython files on Github or >> searchcode.com that matched the query ``ufunc``, and searched for any >> lines matching any of the following regular expressions:: >> >> cdef\W+ufunc >> catches: 'cdef ufunc fn' >> cdef\W+.*\.\W*ufunc >> catches: 'cdef np.ufunc fn' >> <.*ufunc\W*> >> catches: '( fn).nargs', '(< np.ufunc > fn).nargs' >> cdef.*\(.*ufunc >> catches: 'cdef doit(np.ufunc fn, ...):' >> >> (I considered parsing the actual source and analysing it that way, but >> decided I was too lazy. This could be done if anyone is worried that >> the above regexes might miss things though.) >> >> There were zero files that contained matches for any of the above regexes: >> >> https://rawgit.com/njsmith/ufunc-abi-analysis/master/reports/ufunc-cython-report.html >> >> >> Uses of PyUFunc_ReplaceLoopBySignature >> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ >> >> Applying the same screening as above, the only code that was found >> that used this function is also in Numba: >> >> https://rawgit.com/njsmith/ufunc-abi-analysis/master/reports/PyUFunc_ReplaceLoopBySignature-report.html >> >> >> Uses of PyUFunc_SetUsesArraysAsData >> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ >> >> Aside from being semi-broken since 1.7 (it never got implemented for >> "masked" ufunc loops, i.e. those that use where=), there appear to be >> zero uses of this functionality either inside or outside NumPy: >> >> https://rawgit.com/njsmith/ufunc-abi-analysis/master/reports/PyUFunc_SetUsesArraysAsData-report.html >> >> >> Rationale >> --------- >> >> **Rationale for preserving the remaining API functions**:: >> >> PyUFunc_GenericFunction >> >> PyUFunc_FromFuncAndData >> PyUFunc_FromFuncAndDataAndSignature >> PyUFunc_RegisterLoopForDescr >> PyUFunc_RegisterLoopForType >> >> In addition to being widely used, these functions can easily be >> preserved even if we change how ufuncs work internally, because they >> only ingest loop function pointers, they never return them. So they >> can easily be modified to wrap whatever loop function(s) they receive >> inside an adapter function that calls them at the appropriate time, >> and then register that adapter function using whatever API we add in >> the future. >> >> **Rationale for preserving the particular fields that are preserved**: >> Preserving ``nargs`` lets us avoid a little bit of breakage with the >> random dtype, and it doesn't seem like preserving ``nin``, ``nout``, >> ``nargs`` fields will produce any undue burden on future changes to >> ufunc internals; even if we were to introduce variadic ufuncs we could >> always just stick a -1 in the appropriate fields or whatever. >> >> **Rationale for removing PyUFunc_ReplaceLoopBySignature**: this >> function *returns* the PyUFunc_GenericFunction that was replaced; if >> we stop representing all loops using the legacy >> PyUFunc_GenericFunction type, then this will not be possible to do in >> the future. >> >> **Rationale for removing PyUFunc_SetUsesArraysAsData**: If set as the >> ``innerloopdata`` on a ufunc loop, then this function acts as a >> sentinel value, and causes the ``innerloopdata`` to instead be set to >> a pointer to the passed-in PyArrayObjects. In principle we could >> preserve this function, but it has a number of deficiencies: >> - No-one appears to use it. >> - It's been buggy for several releases and no-one noticed. >> - AFAIK the only reason it was included in the first place is that it >> provides a backdoor for ufunc loops to get access to the dtypes -- but >> we are planning to fix this in a better way. >> - It can't be shimmed as easily as the loop registration functions, >> because we don't anticipate that the new-and-improved ufunc loop >> functions will *get* access to the array objects, only to the dtypes; >> so this would have to remain cluttering up the core dispatch path >> indefinitely. >> - We have good reason for *not* wanting to get ufunc loops get access >> to the actual array objects, because one of the goals on our roadmap >> is exactly to enable the use of ufuncs on non-ndarray objects. Giving >> ufuncs access to dtypes alone creates a clean boundary here: it >> guarantees that ufunc loops can work equally on all duck-array objects >> (so long as they have a dtype), and enforces the invariant that >> anything which affects the interpretation of data values should be >> attached to the dtype, not to the array object. >> >> >> Rejected alternatives >> --------------------- >> >> **Do nothing**: there's no way we'll ever be able to touch ufuncs at >> all if we don't hide those fields sooner or later. While any amount of >> breakage is regrettable, the costs of cleaning things up now are less >> than the costs of never improving numpy's APIs. >> >> **Somehow sneak the dtype information in via ``void >> *innerloopdata``**: This might let us preserve the signature of >> PyUFunc_GenericFunction, and thus preserve >> PyUFunc_ReplaceLoopBySignature. But we'd still have the problem of >> leaving way too much internal state exposed, and it's not even clear >> how this would work, given that we actually do want to preserve the >> use of ``innerloopdata`` for actual per-loop data. (This is where the >> PyUFunc_SetUsesArraysAsData hack fails.) >> >> >> -- >> Nathaniel J. Smith -- http://vorpus.org >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> https://mail.scipy.org/mailman/listinfo/numpy-discussion > > > > > -- > (\__/) > ( O.o) > ( > <) Este es Conejo. Copia a Conejo en tu firma y ay?dale en sus planes de > dominaci?n mundial. > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > https://mail.scipy.org/mailman/listinfo/numpy-discussion > -- Nathaniel J. Smith -- http://vorpus.org From njs at pobox.com Tue Sep 22 00:38:36 2015 From: njs at pobox.com (Nathaniel Smith) Date: Mon, 21 Sep 2015 21:38:36 -0700 Subject: [Numpy-discussion] draft NEP for breaking ufunc ABI in a controlled way In-Reply-To: <20150921114434.6c4bf981@fsol> References: <20150921114434.6c4bf981@fsol> Message-ID: Hi Antoine, On Mon, Sep 21, 2015 at 2:44 AM, Antoine Pitrou wrote: > > Hi Nathaniel, > > On Sun, 20 Sep 2015 21:13:30 -0700 > Nathaniel Smith wrote: >> Given this, I propose that for 1.11 we: >> 1) go ahead and hide/disable the problematic parts of the ABI/API, >> 2) coordinate with the known affected projects to minimize disruption >> to their users (which is made easier since they are all projects that >> are almost exclusively distributed via conda, which enforces strict >> NumPy ABI versioning), >> 3) publicize these changes widely so as to give any private code that >> might be affected a chance to speak up or adapt, and >> 4) leave the "ABI version tag" as it is, so as not to force rebuilds >> of the vast majority of projects that will be unaffected by these >> changes. > > Thanks for a detailed and clear explanation of the proposed changes. > As far as Numba is concerned, making changes is ok for us provided > Numpy provides APIs to do what we want. Good to hear, thanks! Any interest in designing those new APIs that will do what you want? :-) A no-brainer is that PyUFuncObject should just take responsibility for managing the memory of its own internal arrays instead of assuming that they'll always be statically allocated and forcing elaborate workarounds when they're not, but there is a lot of complicated stuff going on in numba/npyufunc/_internal.c... I am even wondering whether we should go ahead and reify a first-class "ufunc loop" object, so it can have its own refcounting. -n -- Nathaniel J. Smith -- http://vorpus.org From bryanv at continuum.io Tue Sep 22 00:47:45 2015 From: bryanv at continuum.io (Bryan Van de Ven) Date: Tue, 22 Sep 2015 00:47:45 -0400 Subject: [Numpy-discussion] draft NEP for breaking ufunc ABI in a controlled way In-Reply-To: References: Message-ID: > > until then our only real options are either hard breaks or nothing, so > unless we want to do a hard break there's not much point talking about > it. I think this is the most important sentence from this thread. Thank you Nathaniel for you extremely thorough analysis of the impact on real-world projects. Bryan > On Sep 22, 2015, at 12:23 AM, Nathaniel Smith wrote: > > On Mon, Sep 21, 2015 at 7:29 AM, Jaime Fern?ndez del R?o > wrote: >> We have the PyArrayObject vs PyArrayObject_fields definition in >> ndarraytypes.h that is used to enforce access to the members through inline >> functions rather than directly, which seems to me like the right way to go: >> don't leave stones unturned, hide everything and provide PyUFunc_NIN, >> PyUFunc_NOUT and friends to handle those too. > > The PyArrayObject vs PyArrayObject_fields distinction is only enabled > if a downstream library explicitly requests it with #define > NPY_NO_DEPRECATED_API, though -- the idea is that the changes in this > NEP would be enabled unconditionally, even for old code. So the reason > nin/nout/nargs are left exposed in this proposal is that there's some > existing code out there that would break (until updated) if we hid > them, and not much benefit to breaking it. > > If we're fine with breaking that code then we could just hide them > unconditionally too. The only code I found in the wild that would be > affected is the "rational" user-defined dtype, which would be > trivially fixable since the only thing it does with ufunc->nargs is a > quick consistency check: > > https://github.com/numpy/numpy-dtypes/blob/c0175a6b1c5aa89b4520b29487f06d0e200e2a03/npytypes/rational/rational.c#L1140-L1151 > > Also it's not 100% clear right now whether we even want to keep > supporting the old user-defined dtype API that this particular code is > based around. But if this code uses ufunc->nargs then perhaps other > code does too? I'm open to opinions -- I doubt it matters that much > either way. I just want to make sure that we can hide the other stuff > :-). > > When it comes to evolving these APIs in general: one unfortunate thing > about the PyArrayObject changes in 1.7 is that because they were > implemented using *inline* functions (/macros) they haven't affected > the a*B*i exposure at all, even in code that has upgraded to the new > calling conventions. While user code no longer *names* the internal > fields directly, we still have to implement exactly the same fields > and put them in exactly the same place in memory or else break ABI. > And the other unfortunate thing is that we don't really have a > mechanism for saying "okay, we're dropping support for the old way of > doing things in 1.xx" -- in particular the current > NPY_NO_DEPRECATED_API mechanism doesn't give us any way to detect and > error out if someone tries to use an old version of the APIs, so ABI > breaks still mean segfaults. I'm thinking that if/when we figure out > how to implement the "sliding window" API/ABI idea that we talked > about at SciPy, then that will give us a strategy for cleanly > transitioning to a world with a maintainable API+ABI and it becomes > worth sitting down and making up a set of setters/getters for the > attributes that we want to make public in a maintainable way. But > until then our only real options are either hard breaks or nothing, so > unless we want to do a hard break there's not much point talking about > it. > > -n > >> On Sun, Sep 20, 2015 at 9:13 PM, Nathaniel Smith wrote: >>> >>> Hi all, >>> >>> Here's a first draft NEP for comments. >>> >>> -- >>> >>> Synopsis >>> ======== >>> >>> Improving numpy's dtype system requires that ufunc loops start having >>> access to details of the specific dtype instance they are acting on: >>> e.g. an implementation of np.equal for strings needs access to the >>> dtype object in order to know what "n" to pass to strncmp. Similar >>> issues arise with variable length strings, missing values, categorical >>> data, unit support, datetime with timezone support, etc. -- this is a >>> major blocker for improving numpy. >>> >>> Unfortunately, the current ufunc inner loop function signature makes >>> it very difficult to provide this information. We might be able to >>> wedge it in there, but it'd be ugly. >>> >>> The other option would be to change the signature. What would happen >>> if we did this? For most common uses of the C API/ABI, we could do >>> this easily while maintaining backwards compatibility. But there are >>> also some rarely-used parts of the API/ABI that would be >>> prohibitively difficult to preserve. >>> >>> In addition, there are other potential changes to ufuncs on the >>> horizon (e.g. extensions of gufuncs to allow them to be used more >>> generally), and the current API exposure is so massive that any such >>> changes will be difficult to make in a fully compatible way. This NEP >>> thus considers the possibility of closing down the ufunc API to a >>> minimal, maintainable subset of the current API. >>> >>> To better understand the consequences of this potential change, I >>> performed an exhaustive analysis of all the code on Github, Bitbucket, >>> and Fedora, among others. The results make me highly confident that of >>> all the publically available projects in the world, the only ones >>> which touch the problematic parts of the ufunc API are: Numba, >>> dynd-python, and `gulinalg `_ >>> (with the latter's exposure being trivial). >>> >>> Given this, I propose that for 1.11 we: >>> 1) go ahead and hide/disable the problematic parts of the ABI/API, >>> 2) coordinate with the known affected projects to minimize disruption >>> to their users (which is made easier since they are all projects that >>> are almost exclusively distributed via conda, which enforces strict >>> NumPy ABI versioning), >>> 3) publicize these changes widely so as to give any private code that >>> might be affected a chance to speak up or adapt, and >>> 4) leave the "ABI version tag" as it is, so as not to force rebuilds >>> of the vast majority of projects that will be unaffected by these >>> changes. >>> >>> This NEP defers the question of exactly what the improved API should >>> be, since there's no point in trying to nail down the details until >>> we've decided whether it's even possible to change. >>> >>> >>> Details >>> ======= >>> >>> The problem >>> ----------- >>> >>> Currently, a ufunc inner loop implementation is called via the >>> following function prototype:: >>> >>> typedef void (*PyUFuncGenericFunction) >>> (char **args, >>> npy_intp *dimensions, >>> npy_intp *strides, >>> void *innerloopdata); >>> >>> Here ``args`` is an array of pointers to 1-d buffers of input/output >>> data, ``dimensions`` is a pointer to the number of entries in these >>> buffers, ``strides`` is an array of integers giving the strides for >>> each input/output array, and ``innerloopdata`` is an arbitrary void* >>> supplied by whoever registered the ufunc loop. (For gufuncs, extra >>> shape and stride information about the core dimensions also gets >>> packed into the ends of these arrays in a somewhat complicated way.) >>> >>> There are 4 key items that define a NumPy array: data, shape, strides, >>> dtype. Notice that this function only gets access to 3 of them. Our >>> goal is to fix that. For example, a better signature would be:: >>> >>> typedef void (*PyUFuncGenericFunction_NEW) >>> (char **data, >>> npy_intp *shapes, >>> npy_intp *strides, >>> PyArray_Descr *dtypes, /* NEW */ >>> void *innerloopdata); >>> >>> (In practice I suspect we might want to make some more changes as >>> well, like upgrading gufunc core shape/strides to proper arguments >>> instead of tacking it onto the existing arrays, and adding an "escape >>> valve" void* reserved for future extensions. But working out such >>> details is outside the scope of this NEP; the above will do for >>> illustration.) >>> >>> The goal of this NEP is to clear the ground so that we can start >>> supporting ufunc inner loops that take dtype arguments, and make other >>> enhancements to ufunc functionality going forward. >>> >>> >>> Proposal >>> -------- >>> >>> Currently, the public API/ABI for ufuncs consists of the functions:: >>> >>> PyUFunc_GenericFunction >>> >>> PyUFunc_FromFuncAndData >>> PyUFunc_FromFuncAndDataAndSignature >>> PyUFunc_RegisterLoopForDescr >>> PyUFunc_RegisterLoopForType >>> >>> PyUFunc_ReplaceLoopBySignature >>> PyUFunc_SetUsesArraysAsData >>> >>> together with direct access to PyUFuncObject's internal fields:: >>> >>> typedef struct { >>> PyObject_HEAD >>> int nin, nout, nargs; >>> int identity; >>> PyUFuncGenericFunction *functions; >>> void **data; >>> int ntypes; >>> int check_return; >>> const char *name; >>> char *types; >>> const char *doc; >>> void *ptr; >>> PyObject *obj; >>> PyObject *userloops; >>> int core_enabled; >>> int core_num_dim_ix; >>> int *core_num_dims; >>> int *core_dim_ixs; >>> int *core_offsets; >>> char *core_signature; >>> PyUFunc_TypeResolutionFunc *type_resolver; >>> PyUFunc_LegacyInnerLoopSelectionFunc *legacy_inner_loop_selector; >>> PyUFunc_InnerLoopSelectionFunc *inner_loop_selector; >>> PyUFunc_MaskedInnerLoopSelectionFunc *masked_inner_loop_selector; >>> npy_uint32 *op_flags; >>> npy_uint32 iter_flags; >>> } PyUFuncObject; >>> >>> Obviously almost any future changes to how ufuncs work internally will >>> involve touching some part of this public API/ABI. >>> >>> Concretely, the proposal here is that we avoid this by disabling the >>> following functions (i.e., any attempt to call them should simply >>> raise a ``NotImplementedError``):: >>> >>> PyUFunc_ReplaceLoopBySignature >>> PyUFunc_SetUsesArraysAsData >>> >>> and that we reduce the publicly visible portion of PyUFuncObject down to:: >>> >>> typedef struct { >>> PyObject_HEAD >>> int nin, nout, nargs; >>> } PyUFuncObject; >>> >>> >>> Data on current API/ABI usage >>> ----------------------------- >>> >>> In order to assess how much code would be affected by this proposal, I >>> used a combination of Github search and Searchcode.com to trawl >>> through the majority of all publicly available open source code. >>> Neither search tool provides a fine-grained enough query language to >>> directly tell us what we want to know, so I instead followed the >>> strategy of first, casting a wide net: picking a set of search terms >>> that are likely to catch all possibly-broken code (together with many >>> false positives), and second, using automated tools to sift out the >>> false positives and see what remained. Altogether, I reviewed 4464 >>> search results. >>> >>> The tool I wrote to do this is `available on github >>> `_, and so is `the analysis code >>> itself `_. >>> >>> >>> Uses of PyUFuncObject internals >>> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ >>> >>> There are no functions in the public API which return >>> ``PyUFuncObject*`` values directly, so any code that access >>> PyUFuncObject fields will have to mention that token in the course of >>> defining a variable, performing a cast, setting up a typedef, etc. >>> >>> Therefore, I searched Github for all files written in C, C++, >>> Objective C, Python, or Cython, which mentioned either "PyUFuncObject >>> AND types" or "PyUFuncObject AND NOT types". (This is to work around >>> limitations on how many results Github search is willing to return to >>> a single query.) In addition, I searched for ``PyUFuncObject`` on >>> searchcode.com. >>> >>> The full report on these searches is available here: >>> >>> https://rawgit.com/njsmith/ufunc-abi-analysis/master/reports/pyufuncobject-report.html >>> >>> The following were screened out as non-problems: >>> >>> - Copies of NumPy itself (an astonishing number of people have checked >>> in copies of it to their own source tree) >>> - NumPy forks / precursors / etc. (e.g. Numeric also had a type called >>> PyUFuncObject, the "bohrium" project has a fork of numpy 1.6, etc.) >>> - Cython-generated boilerplate used to generate the "object has >>> changed size" warning (which we `unconditionally filter out anyway >>> `_) >>> - Lots of calls to ``PyUFunc_RegisterLoopForType`` and friends, which >>> require casting the first argument to ``PyUFuncObject*`` >>> - Misc. other unproblematic stuff (like Cython header declarations >>> that never get used) >>> >>> There were also several cases that actually referenced PyUFuncObject >>> internal fields: >>> >>> - The "rational" dtype from numpy-dtypes, which is used in a few >>> projects, accesses ``ufunc->nargs`` as a safety check, but does not >>> touch any other fields (`see here >>> >>> `_). >>> >>> - Numba: does some rather elaborate things to support the definition >>> of on-the-fly JITted ufuncs. These seem to be clear deficiencies in >>> the ufunc API (e.g., there's no good way to control the lifespan of >>> the array of function pointers passed to ``PyUFunc_FromFuncAndData``), >>> so we should work with them to provide the API they need to do this in >>> a maintainable way. Some of the relevant code: >>> >>> https://github.com/numba/numba/tree/master/numba/npyufunc >>> >>> https://github.com/numba/numba/blob/98752647a95ac6c9d480e81ca5c8afcfa3ddfd18/numba/npyufunc/_internal.c >>> >>> - dynd-python: Contains some code that attempts to extract the inner >>> loops from a numpy ufunc object and wrap them into the dynd 'ufunc' >>> equivalent: >>> >>> https://github.com/libdynd/dynd-python/blob/c06f8fc4e72257abac589faf76f10df8c045159b/dynd/src/numpy_ufunc_kernel.cpp >>> >>> - gulinalg: I'm not sure if anyone is still using this code since most >>> of it was merged into numpy itself, but it's not a big deal in any >>> case: all it contains is a `debugging function >>> >>> `_ >>> that dumps some internal fields from the PyUFuncObject. If you look, >>> though, all calls to this function are already commented out :-). >>> >>> The full report is available here: >>> >>> https://rawgit.com/njsmith/ufunc-abi-analysis/master/reports/pyufuncobject-report.html >>> >>> In the course of this analysis, it was also noted that the standard >>> Cython pxd files contain a wrapper for ufunc objects:: >>> >>> cdef class ufunc [object PyUFuncObject]: >>> ... >>> >>> which means that Cython code can access internal struct fields via an >>> object of type ``ufunc``, and thus escape our string-based search >>> above. Therefore I also examined all Cython files on Github or >>> searchcode.com that matched the query ``ufunc``, and searched for any >>> lines matching any of the following regular expressions:: >>> >>> cdef\W+ufunc >>> catches: 'cdef ufunc fn' >>> cdef\W+.*\.\W*ufunc >>> catches: 'cdef np.ufunc fn' >>> <.*ufunc\W*> >>> catches: '( fn).nargs', '(< np.ufunc > fn).nargs' >>> cdef.*\(.*ufunc >>> catches: 'cdef doit(np.ufunc fn, ...):' >>> >>> (I considered parsing the actual source and analysing it that way, but >>> decided I was too lazy. This could be done if anyone is worried that >>> the above regexes might miss things though.) >>> >>> There were zero files that contained matches for any of the above regexes: >>> >>> https://rawgit.com/njsmith/ufunc-abi-analysis/master/reports/ufunc-cython-report.html >>> >>> >>> Uses of PyUFunc_ReplaceLoopBySignature >>> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ >>> >>> Applying the same screening as above, the only code that was found >>> that used this function is also in Numba: >>> >>> https://rawgit.com/njsmith/ufunc-abi-analysis/master/reports/PyUFunc_ReplaceLoopBySignature-report.html >>> >>> >>> Uses of PyUFunc_SetUsesArraysAsData >>> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ >>> >>> Aside from being semi-broken since 1.7 (it never got implemented for >>> "masked" ufunc loops, i.e. those that use where=), there appear to be >>> zero uses of this functionality either inside or outside NumPy: >>> >>> https://rawgit.com/njsmith/ufunc-abi-analysis/master/reports/PyUFunc_SetUsesArraysAsData-report.html >>> >>> >>> Rationale >>> --------- >>> >>> **Rationale for preserving the remaining API functions**:: >>> >>> PyUFunc_GenericFunction >>> >>> PyUFunc_FromFuncAndData >>> PyUFunc_FromFuncAndDataAndSignature >>> PyUFunc_RegisterLoopForDescr >>> PyUFunc_RegisterLoopForType >>> >>> In addition to being widely used, these functions can easily be >>> preserved even if we change how ufuncs work internally, because they >>> only ingest loop function pointers, they never return them. So they >>> can easily be modified to wrap whatever loop function(s) they receive >>> inside an adapter function that calls them at the appropriate time, >>> and then register that adapter function using whatever API we add in >>> the future. >>> >>> **Rationale for preserving the particular fields that are preserved**: >>> Preserving ``nargs`` lets us avoid a little bit of breakage with the >>> random dtype, and it doesn't seem like preserving ``nin``, ``nout``, >>> ``nargs`` fields will produce any undue burden on future changes to >>> ufunc internals; even if we were to introduce variadic ufuncs we could >>> always just stick a -1 in the appropriate fields or whatever. >>> >>> **Rationale for removing PyUFunc_ReplaceLoopBySignature**: this >>> function *returns* the PyUFunc_GenericFunction that was replaced; if >>> we stop representing all loops using the legacy >>> PyUFunc_GenericFunction type, then this will not be possible to do in >>> the future. >>> >>> **Rationale for removing PyUFunc_SetUsesArraysAsData**: If set as the >>> ``innerloopdata`` on a ufunc loop, then this function acts as a >>> sentinel value, and causes the ``innerloopdata`` to instead be set to >>> a pointer to the passed-in PyArrayObjects. In principle we could >>> preserve this function, but it has a number of deficiencies: >>> - No-one appears to use it. >>> - It's been buggy for several releases and no-one noticed. >>> - AFAIK the only reason it was included in the first place is that it >>> provides a backdoor for ufunc loops to get access to the dtypes -- but >>> we are planning to fix this in a better way. >>> - It can't be shimmed as easily as the loop registration functions, >>> because we don't anticipate that the new-and-improved ufunc loop >>> functions will *get* access to the array objects, only to the dtypes; >>> so this would have to remain cluttering up the core dispatch path >>> indefinitely. >>> - We have good reason for *not* wanting to get ufunc loops get access >>> to the actual array objects, because one of the goals on our roadmap >>> is exactly to enable the use of ufuncs on non-ndarray objects. Giving >>> ufuncs access to dtypes alone creates a clean boundary here: it >>> guarantees that ufunc loops can work equally on all duck-array objects >>> (so long as they have a dtype), and enforces the invariant that >>> anything which affects the interpretation of data values should be >>> attached to the dtype, not to the array object. >>> >>> >>> Rejected alternatives >>> --------------------- >>> >>> **Do nothing**: there's no way we'll ever be able to touch ufuncs at >>> all if we don't hide those fields sooner or later. While any amount of >>> breakage is regrettable, the costs of cleaning things up now are less >>> than the costs of never improving numpy's APIs. >>> >>> **Somehow sneak the dtype information in via ``void >>> *innerloopdata``**: This might let us preserve the signature of >>> PyUFunc_GenericFunction, and thus preserve >>> PyUFunc_ReplaceLoopBySignature. But we'd still have the problem of >>> leaving way too much internal state exposed, and it's not even clear >>> how this would work, given that we actually do want to preserve the >>> use of ``innerloopdata`` for actual per-loop data. (This is where the >>> PyUFunc_SetUsesArraysAsData hack fails.) >>> >>> >>> -- >>> Nathaniel J. Smith -- http://vorpus.org >>> _______________________________________________ >>> NumPy-Discussion mailing list >>> NumPy-Discussion at scipy.org >>> https://mail.scipy.org/mailman/listinfo/numpy-discussion >> >> >> >> >> -- >> (\__/) >> ( O.o) >> ( > <) Este es Conejo. Copia a Conejo en tu firma y ay?dale en sus planes de >> dominaci?n mundial. >> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> https://mail.scipy.org/mailman/listinfo/numpy-discussion >> > > > > -- > Nathaniel J. Smith -- http://vorpus.org > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > https://mail.scipy.org/mailman/listinfo/numpy-discussion From travis at continuum.io Tue Sep 22 01:01:51 2015 From: travis at continuum.io (Travis Oliphant) Date: Tue, 22 Sep 2015 00:01:51 -0500 Subject: [Numpy-discussion] Governance model request In-Reply-To: References: Message-ID: On Mon, Sep 21, 2015 at 8:24 PM, Matthew Brett wrote: > Hi Travis, and all, > > You might have seen I was advocating for having someone who takes > final responsibility for the project, partly to get discussions > unstuck, as you said. > > I agree with Chris, that at this stage, there is no-one who could be > Benevolent Dictator for the project. It seems to me that (nearly) > everyone would agree that, if there is a leader, we would not want a > person who dictates, but someone who is good at putting in the hard > work and time it takes to hear all the arguments, and guide people to > agreement, where that is possible. If there is a leader, I think we > should select them for those skills. > I don't really agree that someone couldn't be found. I think the key is that 1) the person would be able to make a decision about what needs to be done if the community is stuck and 2) their is not really a "for life" clause in that their would be a way to call for a replacement. > For the proposal that you join the steering committee, I see two problems. > > The first is, that the members of the steering committee have to be > the people who are keeping in touch with the day to day work of the > project. I am sure you would agree that, in the past, you have not > had enough time for that. Your recent emails about the new work you > want to do, also imply that you may be too busy to get involved in the > detailed discussions needed for getting the code merged. In any case, > I think you'd also agree that in the past you have hoped that you > would have more time for numpy than you did. > I don't think this is true. The steering committee would only be called upon to unstick things and make decisions. At those times, it would not take long to come up to speed on the issues and make a decision. > The second problem is that you have a potential conflict of interest, > in that it is possible for the needs of Continuum to conflict with the > needs of numpy. I think this is a red-herring and should not be an issue --- except for obvious situations where I would have to not participate in a "vote" (such as whether or not Continuum would be an institutional sponsor or something like that). Everyone has associations and affiliations and goals and plans that could lead to conflicts of interest. This kind of requirement un-necessarily limits the governance to only certain kinds of people who have only "volunteer" time. This is actually quite damaging as it limits the ability for people to get paid to work on NumPy. This feels like a world-view debate that is actually best left to a different mailing list. > I believe, from previous emails on this list, that > you don't think that is very important, but I continue to disagree > about that. For example, see this interview with Linus Torvalds, > where he talks about going out of his way to make sure that people can > trust his decisions, despite the fact he is paid to work on Linux [1]. > But this is in a BDFL role and not a steering committee role with 9 other members. I don't think the situation compares or applies at all. The unwarranted fear this kind of reasoning can create in the mind of otherwise reasonable people is unfortunate. > > In practice, the most obvious step that I can think of, is to defer > the decision as to whether you are on the steering committee for six > months. I guess over that time you will be working with the other > numpy developers more closely. > Fundamentally the "steering committee" is too big. I think the committee should be much smaller (no more than 5 and really three is the right size for what it is supposed to actually do). I've had experience with small committees and large committees. Large councils make it very difficult to move things forward. If it is to remain that big, I think it needs people on it like me, Robert Kern, or David Cournapeau who have a longer history with the project and understand why some of the things are the way they are. I still hear far too many conversations that start from a lack of understanding of the early conversations that led to why NumPy is the way it is. This lack of context is not helpful and potentially dangerous. The advanced indexing discussions happening right now and the renewed array-scalar discussions for example are both examples of features that have had previous discussions and have a long history of back and forth between various people. > > I should say that Stefan and I are working on a governance proposal, > in this case for scikit-image, where we split the governance into 1) > the developers doing the work and making the day to day decisions and > 2) the trustees, usually from other relevant projects, or > no-longer-active developers, who make sure the project does not go off > the rails. I think you'd be an excellent and obvious trustee, in > that model. > I like the trustee model too and think such an addition to the NumPy concept would help alleviate my concerns about actually being on a "steering committee" but my preferred outcome is actually that the agreed upon steering council be smaller and that people who have a right to vote on things like the make-up of the steering committee be comprised of people who have been significantly involved in the past 3 years (not just the past one year). -Travis > > Cheers, > > Matthew > > > [1] http://www.bbc.com/news/technology-18419231 > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > https://mail.scipy.org/mailman/listinfo/numpy-discussion > -- *Travis Oliphant* *Co-founder and CEO* @teoliphant 512-222-5440 http://www.continuum.io -------------- next part -------------- An HTML attachment was scrubbed... URL: From bryanv at continuum.io Tue Sep 22 01:15:55 2015 From: bryanv at continuum.io (Bryan Van de Ven) Date: Tue, 22 Sep 2015 01:15:55 -0400 Subject: [Numpy-discussion] Governance model request In-Reply-To: References: <8941C0D2-17EC-420D-A8F6-CA018144811C@continuum.io> Message-ID: > On Sep 21, 2015, at 9:42 PM, David Cournapeau wrote: > > > > On Mon, Sep 21, 2015 at 6:33 PM, Bryan Van de Ven wrote: > > > On Sep 21, 2015, at 9:24 PM, Matthew Brett wrote: > > > > The second problem is that you have a potential conflict of interest, > > in that it is possible for the needs of Continuum to conflict with the > > needs of numpy. I believe, from previous emails on this list, that > > you don't think that is very important, but I continue to disagree > > about that. For example, see this interview with Linus Torvalds, > > Can you actually spell out some particular, articulable, concrete points of concern? Otherwise this just seems like idle, non-constructive speculation (at best). > > There is ample history of such things happening in OSS history, so I think that's a fair concern, even if that has not happened for numpy yet. Specific examples to support that claim would be appreciated. In particular, examples where an OSS project was corrupted (is that the word?) by a company specifically at the hand of the project's original creator would be especially relevant. Beyond that, what (even in a broad sense) is an example of a goal that "Continuum might need" that would conceivably do detriment to the NumPy community? That it be faster? Simpler to maintain? Easier to extend? Integrate better with more OS projects? Attract new active developers? Receive more financial support? Grow its user base even more? And then, even if there is some sliver of daylight to be uncovered between Travis and the community, what is the current organizational mechanism by which a problematic contribution is forced into NumPy against the will of all other core developers? Finally, is there any previous instance whatsoever that can be pointed to of Travis forcing some change into NumPy contrary to the interest of the community? If not, what actual basis is there to exclude him from a steering committee? I obviously have a point of view; but my opinion here is my own, which is: this sort of pre-emptive impugning of someone's integrity is speculation upon speculation upon speculation Bryan From stefanv at berkeley.edu Tue Sep 22 01:57:12 2015 From: stefanv at berkeley.edu (Stefan van der Walt) Date: Mon, 21 Sep 2015 22:57:12 -0700 Subject: [Numpy-discussion] Governance model request In-Reply-To: References: Message-ID: <87oagvt2rr.fsf@berkeley.edu> On 2015-09-20 11:20:28, Travis Oliphant wrote: > I would recommend three possible adjustments to the steering council > concept. > > 1 - define a BDFL for the council. I would nominate chuck Harris > > 2 - limit the council to 3 people. I would nominate chuck, nathaniel, and > pauli. > > 3 - add me as a permanent member of the steering council. I would split the above into two parts: a suggestion on how to change the governance model (first half of 1 and 2) and then some thoughts on what to do once those changes have been made (latter half of 1 and 2, as well as 3). For now, since those changes are not in place yet, it's probably best to focus on the governance model. I would agree that one person (or a very small group) is best suited to "getting things unstuck". And, personally, I believe it best for that person/persons to be elected by the community (whatever we define "the community" to be)---which is what I presume you suggested when you mentioned nominating candidates. Since Matthew mentioned the governance proposal we're working on, here is a very early draft: https://github.com/stefanv/skimage-org/blob/governance_proposal/governance.md As I said, this is still a work-in-progress--comments are welcome. E.g., the weighting element in the voting has to be fine tuned (but was put in place to prevent rapid take-overs). Essentially, we need: - a way for community members to express disagreement without being ousted, - protection against individuals who want to exert disproportional influence, - protection against those in leadership roles who cause the project long-term harm, - and a way for the community to change the direction of the project if they so wished. St?fan From stefanv at berkeley.edu Tue Sep 22 02:07:24 2015 From: stefanv at berkeley.edu (Stefan van der Walt) Date: Mon, 21 Sep 2015 23:07:24 -0700 Subject: [Numpy-discussion] Governance model request In-Reply-To: References: <8941C0D2-17EC-420D-A8F6-CA018144811C@continuum.io> Message-ID: <87mvwft2ar.fsf@berkeley.edu> On 2015-09-21 22:15:55, Bryan Van de Ven wrote: > Beyond that, what (even in a broad sense) is an example of a goal that > "Continuum might need" that would conceivably do detriment to the > NumPy community? That it be faster? Simpler to maintain? Easier to > extend? Integrate better with more OS projects? Attract new active > developers? Receive more financial support? Grow its user base even > more? I don't know how productive it is to dream up examples, but it's not very hard to do. Currently, e.g., the community is not ready to adopt numba as part of the ufunc core. But it's been stated by some that, with so many people running Conda, breaking the ABI is of little consequence. And then it wouldn't be much of a leap to think that numba is an acceptable dependency. There's a broad range of Continuum projects that intersect with what NumPy does: numba, DyND, dask and Odo to name a few. Integrating them into NumPy may make a lot more sense for someone from Continuum than for other members of the community. St?fan From travis at continuum.io Tue Sep 22 02:08:06 2015 From: travis at continuum.io (Travis Oliphant) Date: Tue, 22 Sep 2015 01:08:06 -0500 Subject: [Numpy-discussion] Governance model request In-Reply-To: <87oagvt2rr.fsf@berkeley.edu> References: <87oagvt2rr.fsf@berkeley.edu> Message-ID: Thank you for posting that draft as it is a useful comparison to borrow from. I think Nathaniel's original document is a great start. Perhaps some tweaks along the lines of what you and Matt have suggested could also be useful. I agree that my proposal is mostly about altering the governance model, mixed with some concern about being "automatically disqualified" from a council that can decide the future of NumPy if things don't move forward. -Travis On Tue, Sep 22, 2015 at 12:57 AM, Stefan van der Walt wrote: > On 2015-09-20 11:20:28, Travis Oliphant wrote: > > I would recommend three possible adjustments to the steering council > > concept. > > > > 1 - define a BDFL for the council. I would nominate chuck Harris > > > > 2 - limit the council to 3 people. I would nominate chuck, nathaniel, > and > > pauli. > > > > 3 - add me as a permanent member of the steering council. > > I would split the above into two parts: a suggestion on how to change > the governance model (first half of 1 and 2) and then some thoughts on > what to do once those changes have been made (latter half of 1 and 2, as > well as 3). > > For now, since those changes are not in place yet, it's probably best > to focus on the governance model. > > I would agree that one person (or a very small group) is best suited to > "getting things unstuck". And, personally, I believe it best for that > person/persons to be elected by the community (whatever we define "the > community" to be)---which is what I presume you suggested when you > mentioned nominating candidates. > > Since Matthew mentioned the governance proposal we're working on, here > is a very early draft: > > > https://github.com/stefanv/skimage-org/blob/governance_proposal/governance.md > > As I said, this is still a work-in-progress--comments are welcome. > E.g., the weighting element in the voting has to be fine tuned (but was > put in place to prevent rapid take-overs). > > Essentially, we need: > > - a way for community members to express disagreement without being > ousted, > - protection against individuals who want to exert disproportional > influence, > - protection against those in leadership roles who cause the project > long-term harm, > - and a way for the community to change the direction of the project if > they so wished. > > St?fan > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > https://mail.scipy.org/mailman/listinfo/numpy-discussion > -- *Travis Oliphant* *Co-founder and CEO* @teoliphant 512-222-5440 http://www.continuum.io -------------- next part -------------- An HTML attachment was scrubbed... URL: From bryanv at continuum.io Tue Sep 22 02:28:48 2015 From: bryanv at continuum.io (Bryan Van de Ven) Date: Tue, 22 Sep 2015 02:28:48 -0400 Subject: [Numpy-discussion] Governance model request In-Reply-To: <87mvwft2ar.fsf@berkeley.edu> References: <8941C0D2-17EC-420D-A8F6-CA018144811C@continuum.io> <87mvwft2ar.fsf@berkeley.edu> Message-ID: <370C591F-9711-43E6-BE12-8C3E96946803@continuum.io> > I don't know how productive it is to dream up examples, but it's not Well, agreed, to be honest. > very hard to do. Currently, e.g., the community is not ready to adopt > numba as part of the ufunc core. But it's been stated by some that, Who are you speaking for? The entire community? Under what mandate? > with so many people running Conda, breaking the ABI is of little > consequence. And then it wouldn't be much of a leap to think that numba > is an acceptable dependency. The current somewhat concrete proposal I am aware of involves funding cleaning up dtypes. Is there another concrete, credible proposal to make Numba a dependency of NumPy that you can refer to? If not, why are we mired in hypotheticals? > There's a broad range of Continuum projects that intersect with what > NumPy does: numba, DyND, dask and Odo to name a few. Integrating them > into NumPy may make a lot more sense for someone from Continuum than for > other members of the community. May? Can you elaborate? More speculation. My own position is that these projects want to integrate with NumPy, not the converse. Regardless of my opinion, can you actually make any specific arguements, one way or the otehr? What if if some integrations actually make more sense for the community? Is this simply a dogmatic ideological position that anything whatsoever that benefits both NumPy and Continuum simultaneously is bad, on principle? That's fine, as such, but let's make that position explicit if that's all it is. Bryan From travis at continuum.io Tue Sep 22 02:29:12 2015 From: travis at continuum.io (Travis Oliphant) Date: Tue, 22 Sep 2015 01:29:12 -0500 Subject: [Numpy-discussion] Governance model request In-Reply-To: <87mvwft2ar.fsf@berkeley.edu> References: <8941C0D2-17EC-420D-A8F6-CA018144811C@continuum.io> <87mvwft2ar.fsf@berkeley.edu> Message-ID: On Tue, Sep 22, 2015 at 1:07 AM, Stefan van der Walt wrote: > On 2015-09-21 22:15:55, Bryan Van de Ven wrote: > > Beyond that, what (even in a broad sense) is an example of a goal that > > "Continuum might need" that would conceivably do detriment to the > > NumPy community? That it be faster? Simpler to maintain? Easier to > > extend? Integrate better with more OS projects? Attract new active > > developers? Receive more financial support? Grow its user base even > > more? > > I don't know how productive it is to dream up examples, but it's not > very hard to do. Currently, e.g., the community is not ready to adopt > numba as part of the ufunc core. But it's been stated by some that, > with so many people running Conda, breaking the ABI is of little > consequence. And then it wouldn't be much of a leap to think that numba > is an acceptable dependency. > A couple of things to help clarify: 1) nobody believes that the community should be forced to adopt numba as part of ufunc core yet --- but this could happen someday just as Cython is now being adopted but was proposed 8 years ago that it "could be adopted" That's a red-hearing. 2) I have stated that breaking the ABI is of little consequence because of conda as well as other tools. I still believe that. This has nothing to do with any benefit Continuum might or might not receive because of conda. Everyone else who wants to make a conda-based distribution also benefits (Cloudera, Microsoft, Intel, ...) or use conda also benefits. I don't think the community realizes the damange that is done with FUD like this. There are real implications. It halts progress, creates confusion, and I think ultimately damages the community. Numba being an acceptable dependency means a lot more than conda --- it's dependent on LLVM compiled support which would have to be carefully tested --- first as only an optional dependency for many years. > > There's a broad range of Continuum projects that intersect with what > NumPy does: numba, DyND, dask and Odo to name a few. Integrating them > into NumPy may make a lot more sense for someone from Continuum than for > other members of the community. > I don't see how. None of these have been proposed for integrating into NumPy. I don't see how integrating numba into NumPy benefits Continuum at all. It's much easier for us to keep it separate. At this point Continuum doesn't have an opinion about integrating DyND into NumPy or not. These projects will all succeed or fail on their own based on users needs. Whether or not they every become a part of NumPy will depend on whether they are useful as such not because a person at Continuum is part of a steering committee (with other people on it). I know that you were responding to specific question by Brian as to how their could be a conflict of interest for Continuum and NumPy development. I don't think this is a useful conversation --- we could dream up all kinds of conflicts of interest for BIDS and NumPy too (e.g. perhaps BIDS really wants Spark to take over and for NumPy to have special connections to Spark). Are we to not allow anyone at BIDS to participate in the steering council because of their other interests? But remember, the original point is whether or not someone from Continuum (or I presume any company and not just singling out Continuum for special treatment) should be on the steering council. Are you really arguing that they shouldn't because there are other projects Continuum is working on that have some overlap with NumPy. I really hope you don't actually believe that. -Travis > St?fan > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > https://mail.scipy.org/mailman/listinfo/numpy-discussion > -- *Travis Oliphant* *Co-founder and CEO* @teoliphant 512-222-5440 http://www.continuum.io -------------- next part -------------- An HTML attachment was scrubbed... URL: From travis at continuum.io Tue Sep 22 02:51:33 2015 From: travis at continuum.io (Travis Oliphant) Date: Tue, 22 Sep 2015 01:51:33 -0500 Subject: [Numpy-discussion] 1.10.0rc1 coming tomorrow, 22 Sept. In-Reply-To: References: Message-ID: Of course it will be 1.10.0 final where all the problems will show up suddenly :-) Perhaps we can get to where we are testing Anaconda against beta releases better. -Travis On Mon, Sep 21, 2015 at 5:19 PM, Charles R Harris wrote: > Hi All, > > Just a heads up. The lack of reported problems in 1.10.0b1 has been > stunning. > > Chuck > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > https://mail.scipy.org/mailman/listinfo/numpy-discussion > > -- *Travis Oliphant* *Co-founder and CEO* @teoliphant 512-222-5440 http://www.continuum.io -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefanv at berkeley.edu Tue Sep 22 02:57:49 2015 From: stefanv at berkeley.edu (Stefan van der Walt) Date: Mon, 21 Sep 2015 23:57:49 -0700 Subject: [Numpy-discussion] Governance model request In-Reply-To: <370C591F-9711-43E6-BE12-8C3E96946803@continuum.io> References: <8941C0D2-17EC-420D-A8F6-CA018144811C@continuum.io> <87mvwft2ar.fsf@berkeley.edu> <370C591F-9711-43E6-BE12-8C3E96946803@continuum.io> Message-ID: <87eghrszyq.fsf@berkeley.edu> Hi Brian On 2015-09-21 23:28:48, Bryan Van de Ven wrote: >> very hard to do. Currently, e.g., the community is not ready to adopt >> numba as part of the ufunc core. But it's been stated by some that, > > Who are you speaking for? The entire community? Under what mandate? I am speaking on behalf of myself, under no mandate. > The current somewhat concrete proposal I am aware of involves funding > cleaning up dtypes. Is there another concrete, credible proposal to > make Numba a dependency of NumPy that you can refer to? If not, why > are we mired in hypotheticals? I'm sorry if I misunderstood, but I thought you wanted us to explore hypothetical confictual situations. > May? Can you elaborate? More speculation. My own position is that > these projects want to integrate with NumPy, not the > converse. Regardless of my opinion, can you actually make any specific > arguements, one way or the otehr? What if if some integrations > actually make more sense for the community? Is this simply a dogmatic > ideological position that anything whatsoever that benefits both NumPy > and Continuum simultaneously is bad, on principle? That's fine, as > such, but let's make that position explicit if that's all it is. No, I don't have such a dogmatic ideological position. I think, however, that it is somewhat unimaginative to propose that there are no potential conflicts whatsoever. I am happy if we can find solutions that benefit both numpy and any company out there. But in the end, I'm sure you'd agree that we want the decisions that lead to such solutions to be taken in the best interest of the project, and not be weighed by alterior motivations of any sorts. In the end, even the *perception* that that is not the case can be very harmful. St?fan From njs at pobox.com Tue Sep 22 03:11:33 2015 From: njs at pobox.com (Nathaniel Smith) Date: Tue, 22 Sep 2015 00:11:33 -0700 Subject: [Numpy-discussion] Governance model request In-Reply-To: References: <1442827933.393.61.camel@sipsolutions.net> <1442828394.393.64.camel@sipsolutions.net> Message-ID: On Mon, Sep 21, 2015 at 9:20 AM, Travis Oliphant wrote: > > I wrote my recommendations quickly before heading on a plane. I hope the spirit of them was caught correctly. I also want to re-emphasize that I completely understand that the Steering Council is not to be making decisions that often and almost all activity will be similar to it is now --- discussion, debate, proposals, and pull-requests --- that is a good thing. > > However, there is a need for leadership to help unstick things and move the project forward from time to time because quite often doing *something* can be better than trying to please everyone with a voice. My concerns about how to do this judgment have 2 major components: > > 1) The need for long-term consistency --- a one-year horizon on defining this group is too short in my mind for a decades-old project like NumPy. > 2) The group that helps unstick things needs to be small (1, 3, or 5 at the most) For reference, the rules for steering council membership were taken directly from those used by the Jupyter project, and their steering council currently has 10 people, making it larger than the "seed council" proposed in the numpy document: https://github.com/jupyter/governance/blob/master/people.md > We could call this group the "adjudication group" rather than the "Steering Council" as well. I could see that having a formal method of changing that "adjudication group" would be a good idea as well (and perhaps that formal vote could be made by a vote of a group of active contributors. In that case, I would define active as having a time-window of 5 years instead of just 1). I may be misreading things, but I'm getting the impression that the active "adjudication group" you envision is radically different from the "steering council" as envisioned by the current governance document. It also, I think, radically different from anything I've ever seen in a functioning community-run FOSS project and frankly it's something where if I saw a project using this model, it would make me extremely wary about contributing. The key point that I think differs is that you envision that this "adjudication group" will actually intervene into discussions and make formal decisions in situations other than true irreconcilable crises, which in my estimation happen approximately never. The only two kinds of F/OSS projects that I can think of that run like this are (a) projects that are not really community driven at all, but rather run as internal company projects that happen to have a public repository, (b) massive projects like Debian and Fedora that have to manage literally thousands of contributors, and thus have especially robust backstop procedures to handle the rare truly irreconcilable situation. E.g., the Debian CTTE acts as an "adjudication group" in the way it sounds like you envision it: on a regular basis, irreconcilable arguments in Debian get taken to them to decide, and they issue a ruling. By some back of the envelope calculations, it looks like they issue approximately ~0.002 rulings per debian-contributor-year [1][2]. If we assume crudely that irreconcilable differences scale linearly with the size of a project, this suggests that a ~20 person project like NumPy should require a ruling ~once every 20 years. Or quoting myself from the last thread about this [3]: ] Or on the other end of things, you have e.g. Subversion, which had an ] elaborate defined governance system with different levels of ] "core-ness", a voting system, etc. -- and they were 6 years into the ] project before they had their first vote. (The vote was on the crucial ] technical decision of whether to write function calls like "f ()" or ] "f()".) These are two real projects and how they really work. And even in projects that do have a BDFL, the successful ones almost never use this power to actually "unstick things" (i.e., use their formal power to resolve a discussion). Consider PEP 484, Guido's somewhat controversial type hints proposal: rather than use his power to move the debate along, he explicitly delegated his power to one of the idea's strongest critics [4]. Of course, things to get stuck. But the only time that getting them unstuck needs or even benefits from the existence of a formal "unsticking things" group is if the group is actually using some powers that they have. In 99.9% of cases, though, the correct way to get things unstuck is for a new idea to be introduced, or for someone respected to act as a mediator, or for someone to summarize the situation and isolate some core of agreement that allows for forward progress. But all of *these* things can and should be done by anyone and everyone who can contribute them -- restricting them to a small "adjudication group" makes no sense. In terms of real-world examples: From my point of view, the worst parts of the NA fiasco was caused by the decision to cut off debate with a "ruling". (Specifically, that instead of working on implementing bitpattern-NAs -- which did have consensus -- then Mark would go off and work on the masked-NA strategy, and eventually that it should be merged, despite it not having consensus. I note that dynd supports only bitpattern-NAs.) OTOH, probably the most difficult technical debate we've had recently is issue 5844, about the potential interactions between __binop__ methods and __numpy_ufunc__, and this is a debate that AFAICT certainly would not have benefited from the existence of an adjudication body. As it happens, all three of your proposed adjudication-body-members actually are in the debate anyway; if they hadn't been, then the very last thing that debate needs is someone wading in without any understanding of the complex technical issues and trying to force some resolution. I'm not saying that truly irreconcilable problems never arise. But given that they're so rare, and that realistically any system that could solve them would need to be backed by basically the same group of people that are in the current governance document's "steering council" (because in an OSS project it's the people doing the work that ultimately have the power), there's no point in building an elaborate voting system and method for defining the electorate just to handle these cases -- and quite a bit of harm. So the proposed system is basically the simplest one that could work. (I described this line of reasoning in more detail in [3].) I think it would really help if you could provide some examples of successful community-driven projects and/or specific debates that benefited from having an adjudication body like you envision? I also suspect this fundamental difference in how we view the role of a governance body explains why I am unbothered by the idea of steering council membership "timing out" after 2 years of inactivity. The steering council's job is to informally manage things day to day, and in extreme cases to make judgement calls. These two things both crucially require a broad and up-to-date understanding of the issues facing the project, ongoing debates, personalities, etc. But the council is never intended to make any judgement call on its own; the whole idea of the structure is to make sure that decisions are based on as broad a scope of expertise as possible. In particular we regularly get historical insight from Chuck, Ralf, Pauli, Robert Kern, David Cournapeau, Pearu Peterson, Anne Archibald, Matthew Brett, Stefan van der Walt... and it doesn't matter at all whether they're on the council or not. (If it did matter, that would be terrible, right? Why would you want to *not* listen to any of those people?) If you plan to become more active and give your perspective on things more then that's awesome and welcome, but AFAICT this particular point is pretty orthogonal to the composition of the steering council. See also this comment from Fernando about the Jupyter steering council, which happened by chance to cross my email this morning: https://github.com/jupyter/governance/pull/6#issuecomment-142036050 -n [1] https://www.debian.org/devel/tech-ctte#status [2] https://contributors.debian.org/ [3] https://mail.scipy.org/pipermail/numpy-discussion/2015-September/073537.html [4] See e.g. the bottom of this message from Nick Coghlan, where he talks about how PEP "pronouncements" are done -- emphasizing in particular (my paraphrase) that the point of a BDFL/BDFL-delegate is not to resolve any substantive issues, and that a common strategy is to choose the person who's most skeptical of the idea to be the BDFL-delegate: http://thread.gmane.org/gmane.comp.python.distutils.devel/23867 -- Nathaniel J. Smith -- http://vorpus.org From stefanv at berkeley.edu Tue Sep 22 03:16:03 2015 From: stefanv at berkeley.edu (Stefan van der Walt) Date: Tue, 22 Sep 2015 00:16:03 -0700 Subject: [Numpy-discussion] Governance model request In-Reply-To: References: <8941C0D2-17EC-420D-A8F6-CA018144811C@continuum.io> <87mvwft2ar.fsf@berkeley.edu> Message-ID: <87d1xaudos.fsf@berkeley.edu> Hi Travis On 2015-09-21 23:29:12, Travis Oliphant wrote: > 1) nobody believes that the community should be forced to adopt numba as > part of ufunc core yet --- but this could happen someday just as Cython is > now being adopted but was proposed 8 years ago that it "could be adopted" > That's a red-hearing. Yes, I'd like to clarify: I was not against including any specific technology in NumPy. I was highlighting that there may be different motivations for members of the general community and those working for, say, Continuum, to get certain features adopted. > 2) I have stated that breaking the ABI is of little consequence because > of conda as well as other tools. I still believe that. This has nothing > to do with any benefit Continuum might or might not receive because of > conda. Everyone else who wants to make a conda-based distribution also > benefits (Cloudera, Microsoft, Intel, ...) or use conda also benefits. > I don't think the community realizes the damange that is done with FUD like > this. There are real implications. It halts progress, creates confusion, > and I think ultimately damages the community. This is an old argument, and the reason why we have extensive measures in place to guard against ABI breakage. But, reading what you wrote above, I would like to understand better what FUD you are referring to, because I, rightly or wrongly, believe there is a real concern here that is being glossed over. > I don't see how. None of these have been proposed for integrating into > NumPy. I don't see how integrating numba into NumPy benefits Continuum > at all. It's much easier for us to keep it separate. At this point > Continuum doesn't have an opinion about integrating DyND into NumPy or > not. I think that touches, tangentially at least, on the problem. If an employee of Continuum were steering NumPy, and the company developed an opinion on those integrations, would such a person not feel compelled to toe the company line? (Whether the company is Continuum or another is besides the point?I am only trying to understand the dynamics of working for a company and leading an open source project that closely interacts with their producs.) > I know that you were responding to specific question by Brian as to how > their could be a conflict of interest for Continuum and NumPy development. > I don't think this is a useful conversation --- we could dream up all > kinds of conflicts of interest for BIDS and NumPy too (e.g. perhaps BIDS > really wants Spark to take over and for NumPy to have special connections > to Spark). Are we to not allow anyone at BIDS to participate in the > steering council because of their other interests? I guess that's an interesting example, but BIDS (which sits inside a university and is funded primarily by foundations) has no financial, and very few other, incentives to do so. > But remember, the original point is whether or not someone from Continuum > (or I presume any company and not just singling out Continuum for special > treatment) should be on the steering council. Are you really arguing > that they shouldn't because there are other projects Continuum is working > on that have some overlap with NumPy. I really hope you don't actually > believe that. Here's what I'm trying to say (and I apologise for ruffling feathers in the process): There are concerns amongst members of the community that (will) arise when strong players from industry try / hint at exerting (some) executive control over NumPy. We can say that these concerns amount to spreading FUD, that they are uninformed, unrealistic, etc., but ultimately they are still out there, and until they are discussed and addressed, I find it hard to see how we can move forward with ease. St?fan From njs at pobox.com Tue Sep 22 03:19:28 2015 From: njs at pobox.com (Nathaniel Smith) Date: Tue, 22 Sep 2015 00:19:28 -0700 Subject: [Numpy-discussion] 1.10.0rc1 coming tomorrow, 22 Sept. In-Reply-To: References: Message-ID: On Sep 21, 2015 11:51 PM, "Travis Oliphant" wrote: > > Of course it will be 1.10.0 final where all the problems will show up suddenly :-) > > Perhaps we can get to where we are testing Anaconda against beta releases better. The most useful thing would actually not even involve you doing any more testing, but just if you could make builds available so that end-users could easily conda install the prereleases and do their own testing against their own choice. In principle I guess we could provide our own binstar channel for this, but it's difficult given that AFAIK rebuilding numpy in conda requires also rebuilding the whole stack, and the numpy build recipes are still proprietary. -n -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Tue Sep 22 03:33:12 2015 From: njs at pobox.com (Nathaniel Smith) Date: Tue, 22 Sep 2015 00:33:12 -0700 Subject: [Numpy-discussion] Governance model request In-Reply-To: <370C591F-9711-43E6-BE12-8C3E96946803@continuum.io> References: <8941C0D2-17EC-420D-A8F6-CA018144811C@continuum.io> <87mvwft2ar.fsf@berkeley.edu> <370C591F-9711-43E6-BE12-8C3E96946803@continuum.io> Message-ID: Hi Bryan, I understand where you're coming from, but I'd appreciate it if we could keep the discussion on a less visceral level? Nobody's personal integrity is being impugned, but it's the nature of this kind of governance discussion that we have to consider unlikely-and-unpleasant hypotheticals. It's like when you talk to a lawyer about a contract or a will or whatever: they'll make you think about all kinds of horrible possibilities, not because any of them are likely, but because sooner or later *something* will go wrong, and the point of having a contract/will/governance document is to provide some framework to handle whichever unlikely edge case does arise. And the other purpose of this kind of framework is to avoid the *perception* (whether justified or not) of these kinds of conflicts of interest -- if not handled well then this can easily scare away volunteers, contributions from other companies, etc. Obviously you know Travis and Continuum well as an employee there, but to most people without that personal connection, Continuum is just a large corporate entity with unknown motives. Imagine if instead of Continuum we were talking about it was Google or RandomAggressiveStartup -- some company that you didn't have any personal connection or insight into. For someone in this position, it's not unreasonable to want more of a reassurance that things will work out then just being asked to trust that the CEO is personally committed to not being evil and they can trust him. Also, in these messages you seem to be working from a framework where people working in good faith will always agree, and so any suggestion of a possible conflict of interest can only arise through bad faith. But this isn't true. Is it really so difficult to imagine that, say, Continuum and Enthought might at some point have conflicting needs for numpy, or for Continuum's vision of the future of numpy could be less-than-perfectly-representative with every bit of numpy's entire giant userbase? Continuum is a company that has in the past submitted rather obscure patches to numpy that AFAICT are used exclusively by a particular contracting customer (e.g. [1]), and that is currently investing a substantial multiple of numpy's development budget on building a direct competitor to numpy. To emphasize: I personally am not concerned by these facts -- we did merge that patch I linked, and there's no animosity between the numpy and dynd teams -- but reasonable people could reasonably be concerned that tricky situations might emerge in the future, and I've talked to plenty of people who are nervous about Continuum's influence in general. And with my numpy developer hat on, I don't even care which "side" is right, that's irrelevant to me, because my concern is with providing a space where both "sides" feel comfortable working together. This is why it's crucial that numpy-the-project be clearly distinguishable as an independent entity that is beholden only to its own community: it's *exactly because* we *value* the contribution of companies like Continuum, and want to be able to freely foster those relationships without creating suspicion and bad blood. Also to emphasize: none of this means that Travis can't be on the steering council -- I think that's a more complex issue that I'll address separately. All I'm saying is that pretending that you aren't going to reassure people by pretending this elephant isn't in the room, or by taking a reasonable set of concerns and aggressively turning them into a referendum on individual people's integrity. Finally, can I point out... anyone who has some wariness around the possible influence of financial interests on the community (whether justified or not!) is in particular not going to be reassured if you keep aggressively attempting to shut down any perceived criticism of *your own employer*. I know that your paycheck is not dictating your opinions, and probably the hypothetical people I'm talking about are being totally unfair to you for even considering such a thing, but... strictly as a matter of practical rhetoric, I don't think this is the most effective way to accomplish your goals. -n [1] https://github.com/numpy/numpy/pull/359 On Mon, Sep 21, 2015 at 11:28 PM, Bryan Van de Ven wrote: > >> I don't know how productive it is to dream up examples, but it's not > > Well, agreed, to be honest. > >> very hard to do. Currently, e.g., the community is not ready to adopt >> numba as part of the ufunc core. But it's been stated by some that, > > Who are you speaking for? The entire community? Under what mandate? > >> with so many people running Conda, breaking the ABI is of little >> consequence. And then it wouldn't be much of a leap to think that numba >> is an acceptable dependency. > > The current somewhat concrete proposal I am aware of involves funding cleaning up dtypes. Is there another concrete, credible proposal to make Numba a dependency of NumPy that you can refer to? If not, why are we mired in hypotheticals? > >> There's a broad range of Continuum projects that intersect with what >> NumPy does: numba, DyND, dask and Odo to name a few. Integrating them >> into NumPy may make a lot more sense for someone from Continuum than for >> other members of the community. > > May? Can you elaborate? More speculation. My own position is that these projects want to integrate with NumPy, not the converse. Regardless of my opinion, can you actually make any specific arguements, one way or the otehr? What if if some integrations actually make more sense for the community? Is this simply a dogmatic ideological position that anything whatsoever that benefits both NumPy and Continuum simultaneously is bad, on principle? That's fine, as such, but let's make that position explicit if that's all it is. > > Bryan > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > https://mail.scipy.org/mailman/listinfo/numpy-discussion -- Nathaniel J. Smith -- http://vorpus.org From solipsis at pitrou.net Tue Sep 22 03:51:25 2015 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 22 Sep 2015 09:51:25 +0200 Subject: [Numpy-discussion] draft NEP for breaking ufunc ABI in a controlled way References: <20150921114434.6c4bf981@fsol> Message-ID: <20150922095125.6d449471@fsol> On Mon, 21 Sep 2015 21:38:36 -0700 Nathaniel Smith wrote: > Hi Antoine, > > On Mon, Sep 21, 2015 at 2:44 AM, Antoine Pitrou wrote: > > > > Hi Nathaniel, > > > > On Sun, 20 Sep 2015 21:13:30 -0700 > > Nathaniel Smith wrote: > >> Given this, I propose that for 1.11 we: > >> 1) go ahead and hide/disable the problematic parts of the ABI/API, > >> 2) coordinate with the known affected projects to minimize disruption > >> to their users (which is made easier since they are all projects that > >> are almost exclusively distributed via conda, which enforces strict > >> NumPy ABI versioning), > >> 3) publicize these changes widely so as to give any private code that > >> might be affected a chance to speak up or adapt, and > >> 4) leave the "ABI version tag" as it is, so as not to force rebuilds > >> of the vast majority of projects that will be unaffected by these > >> changes. > > > > Thanks for a detailed and clear explanation of the proposed changes. > > As far as Numba is concerned, making changes is ok for us provided > > Numpy provides APIs to do what we want. > > Good to hear, thanks! > > Any interest in designing those new APIs that will do what you want? > :-) I'll take a look. Regards Antoine. From njs at pobox.com Tue Sep 22 04:12:51 2015 From: njs at pobox.com (Nathaniel Smith) Date: Tue, 22 Sep 2015 01:12:51 -0700 Subject: [Numpy-discussion] Governance model request In-Reply-To: References: <8941C0D2-17EC-420D-A8F6-CA018144811C@continuum.io> Message-ID: On Mon, Sep 21, 2015 at 10:15 PM, Bryan Van de Ven wrote: > >> On Sep 21, 2015, at 9:42 PM, David Cournapeau wrote: >> There is ample history of such things happening in OSS history, so I think that's a fair concern, even if that has not happened for numpy yet. > > Specific examples to support that claim would be appreciated. In particular, examples where an OSS project was corrupted (is that the word?) by a company specifically at the hand of the project's original creator would be especially relevant. I have no expectation that continuum will follow any of these paths, and in most cases am not even sure what that would mean, BUT just because I think it is useful to have a wide variety of concrete examples to draw on -- data is good! -- there actually are *lots* of examples of "community revolts" wresting projects from their original founders, in a variety of corporate and non-corporate contexts. Some examples include the nodejs->iojs fork and merge (which was about wresting control of the project from the founding company), the gcc->egcs fork and merge (which removed RMS's control over day-to-day running of the project), the openoffice->libreoffice fork, the xfree86->x.org fork (where the original core team decided to change the license and all the developers left), the mambo->joomla fork, the xchat->hexchat fork (triggered partially by people's annoyance at the original developer for trying to monetize the project), ... Along somewhat similar lines, there's also the fraught history of Qt and Trolltech and the conflicts between the community and commercial interests there. -n -- Nathaniel J. Smith -- http://vorpus.org From bryanv at continuum.io Tue Sep 22 04:21:56 2015 From: bryanv at continuum.io (Bryan Van de Ven) Date: Tue, 22 Sep 2015 04:21:56 -0400 Subject: [Numpy-discussion] Governance model request In-Reply-To: References: <8941C0D2-17EC-420D-A8F6-CA018144811C@continuum.io> Message-ID: <4722446B-47BE-43F4-8CF4-14FDD211E252@continuum.io> > I have no expectation that continuum will follow any of these paths, > and in most cases am not even sure what that would mean, BUT just > because I think it is useful to have a wide variety of concrete > examples to draw on -- data is good! -- there actually are *lots* of > examples of "community revolts" wresting projects from their original > founders, in a variety of corporate and non-corporate contexts. Some > examples include the nodejs->iojs fork and merge (which was about > wresting control of the project from the founding company), the > gcc->egcs fork and merge (which removed RMS's control over day-to-day > running of the project), the openoffice->libreoffice fork, the > xfree86->x.org fork (where the original core team decided to change > the license and all the developers left), the mambo->joomla fork, the > xchat->hexchat fork (triggered partially by people's annoyance at the > original developer for trying to monetize the project), ... Along > somewhat similar lines, there's also the fraught history of Qt and > Trolltech and the conflicts between the community and commercial > interests there. All of there are exactly the opposite of what I asked about, and what was suggested as the threat: an original founder and corporate interest wresting control from the community. Bryan From travis at continuum.io Tue Sep 22 04:24:37 2015 From: travis at continuum.io (Travis Oliphant) Date: Tue, 22 Sep 2015 03:24:37 -0500 Subject: [Numpy-discussion] Governance model request In-Reply-To: References: <1442827933.393.61.camel@sipsolutions.net> <1442828394.393.64.camel@sipsolutions.net> Message-ID: I actually do agree with your view of the steering council as being usually not really being needed. You are creating a straw-man by indicating otherwise. I don't believe a small council should do anything *except* resolve disputes that cannot be resolved without one. Like you, I would expect that would almost never happen --- but I would argue that extrapolating from Debian's experience is not actually relevant here. So, if the steering council is not really needed then why have it at all? Let's just eliminate the concept entirely. But there are real questions that have to have an answer or an approach to making a decision. The answer to these questions cannot really be a vague notion of "lack of vigorous opposition by people who read the mailing list" which then gets parried about as "the community decided this." The NumPy user base is far, far larger than the number of people that read this list. For better or for worse, we will always be subject to the "tyranny of who has time to contribute lately". Fundamentally, I would argue that this kind of "tyranny" should at least be tempered by additional considerations from long-time contributors who may also be acting more indirectly than is measured by a simple git log. So, what are the questions that have to have an answer that even calls for some kind of governance? I know Nathaniel's document listed some of them (and perhaps all of them). Here are mine: 1) who gets commit rights to the repo (and who has them removed)? 2) who tags the release of NumPy? 3) how is it decided where money is spent if it's donated to the project (and not just to people directly)? 4) how is it decided if someone needs to be removed from participation in the group because they are not adding to the conversation (we have been fortunate that this hasn't happened in NumPy before --- but it could)? 5) how is it decided what goes on the NumPy website --- i.e. will advertisers be able to put their logos or book-covers there? If I understand what you are proposing, then basically the "steering council" decides these things. Perhaps rather than a steering council, though, we just need clear answers to questions like the above --- which might be handled differently for different questions. I don't think these questions have very easy answers. Ultimately NumPy has relied on and continues to rely on the mutual respect of all the people that have worked on the code and tried to make it better. We all have opinions about how things have gone in the past, and what has gone well and what hasn't. But, nothing you have said persuades me that you have a full picture of past history with respect to a lot of the difficult kinds of conversations that have happened and the different modes of activity that have tried to help move NumPy along. In fact, I think you mis-understand and mis-interpret that history quite often. I'm convinced you are well-intentioned and doing the very best you can, and I'm very grateful that you are passionate and eager about moving NumPy forward. Ultimately I hope it will help things. Here is my attempt at a proposal for how to answer the above questions: 1) who gets commit rights to the repo (and who has them removed)? * people who contribute regularly are granted commit rights by another committer with at least two additional nominations and the lack of a veto within 1 week of the proposal. * nobody has commit rights removed except by unanimous consent of all the other committers (with consent being implied if not responded to within 2 weeks). 2) who tags the release of NumPy? * whomever volunteers to be release manager and if there is no veto from committers. 3) how is it decided where money is spent if it's donated to the project (and not just to people directly)? * three people who self-select represent NumPy to Numfocus following the rules of Numfocus (that there is only one representative from any organization). * If 3 committers oppose one of those people and nominate another in place, then that person is replaced. 4) how is it decided if someone needs to be removed from participation in the group because they are not adding to the conversation (we have been fortunate that this hasn't happened in NumPy before --- but it could)? * unanimous consent of all committers (with a 2 week period given for consent to be given --- and it is assumed given if they are not heard from). 5) how is it decided what goes on the NumPy website --- i.e. will advertisers be able to put their logos or book-covers there? * only Numfocus can advertise and put their logo on the website Now, I'm sure one can poke holes in the above --- and I would welcome better answers to the above questions. Perhaps we should just decide how specific decisions get made and make a document that lists that and only talks about committers instead of inventing another "bit" to differentiate people in the community. -Travis On Tue, Sep 22, 2015 at 2:11 AM, Nathaniel Smith wrote: > On Mon, Sep 21, 2015 at 9:20 AM, Travis Oliphant > wrote: > > > > I wrote my recommendations quickly before heading on a plane. I hope > the spirit of them was caught correctly. I also want to re-emphasize > that I completely understand that the Steering Council is not to be making > decisions that often and almost all activity will be similar to it is now > --- discussion, debate, proposals, and pull-requests --- that is a good > thing. > > > > However, there is a need for leadership to help unstick things and move > the project forward from time to time because quite often doing *something* > can be better than trying to please everyone with a voice. My concerns > about how to do this judgment have 2 major components: > > > > 1) The need for long-term consistency --- a one-year horizon on defining > this group is too short in my mind for a decades-old project like NumPy. > > 2) The group that helps unstick things needs to be small (1, 3, or 5 at > the most) > > For reference, the rules for steering council membership were taken > directly from those used by the Jupyter project, and their steering > council currently has 10 people, making it larger than the "seed > council" proposed in the numpy document: > https://github.com/jupyter/governance/blob/master/people.md > > > We could call this group the "adjudication group" rather than the > "Steering Council" as well. I could see that having a formal method of > changing that "adjudication group" would be a good idea as well (and > perhaps that formal vote could be made by a vote of a group of active > contributors. In that case, I would define active as having a time-window > of 5 years instead of just 1). > > I may be misreading things, but I'm getting the impression that the > active "adjudication group" you envision is radically different from > the "steering council" as envisioned by the current governance > document. It also, I think, radically different from anything I've > ever seen in a functioning community-run FOSS project and frankly it's > something where if I saw a project using this model, it would make me > extremely wary about contributing. > > The key point that I think differs is that you envision that this > "adjudication group" will actually intervene into discussions and make > formal decisions in situations other than true irreconcilable crises, > which in my estimation happen approximately never. The only two kinds > of F/OSS projects that I can think of that run like this are (a) > projects that are not really community driven at all, but rather run > as internal company projects that happen to have a public repository, > (b) massive projects like Debian and Fedora that have to manage > literally thousands of contributors, and thus have especially robust > backstop procedures to handle the rare truly irreconcilable situation. > > E.g., the Debian CTTE acts as an "adjudication group" in the way it > sounds like you envision it: on a regular basis, irreconcilable > arguments in Debian get taken to them to decide, and they issue a > ruling. By some back of the envelope calculations, it looks like they > issue approximately ~0.002 rulings per debian-contributor-year [1][2]. > If we assume crudely that irreconcilable differences scale linearly > with the size of a project, this suggests that a ~20 person project > like NumPy should require a ruling ~once every 20 years. > > Or quoting myself from the last thread about this [3]: > ] Or on the other end of things, you have e.g. Subversion, which had an > ] elaborate defined governance system with different levels of > ] "core-ness", a voting system, etc. -- and they were 6 years into the > ] project before they had their first vote. (The vote was on the crucial > ] technical decision of whether to write function calls like "f ()" or > ] "f()".) > > These are two real projects and how they really work. And even in > projects that do have a BDFL, the successful ones almost never use > this power to actually "unstick things" (i.e., use their formal power > to resolve a discussion). Consider PEP 484, Guido's somewhat > controversial type hints proposal: rather than use his power to move > the debate along, he explicitly delegated his power to one of the > idea's strongest critics [4]. > > Of course, things to get stuck. But the only time that getting them > unstuck needs or even benefits from the existence of a formal > "unsticking things" group is if the group is actually using some > powers that they have. In 99.9% of cases, though, the correct way to > get things unstuck is for a new idea to be introduced, or for someone > respected to act as a mediator, or for someone to summarize the > situation and isolate some core of agreement that allows for forward > progress. But all of *these* things can and should be done by anyone > and everyone who can contribute them -- restricting them to a small > "adjudication group" makes no sense. > > In terms of real-world examples: From my point of view, the worst > parts of the NA fiasco was caused by the decision to cut off debate > with a "ruling". (Specifically, that instead of working on > implementing bitpattern-NAs -- which did have consensus -- then Mark > would go off and work on the masked-NA strategy, and eventually that > it should be merged, despite it not having consensus. I note that dynd > supports only bitpattern-NAs.) > > OTOH, probably the most difficult technical debate we've had recently > is issue 5844, about the potential interactions between __binop__ > methods and __numpy_ufunc__, and this is a debate that AFAICT > certainly would not have benefited from the existence of an > adjudication body. As it happens, all three of your proposed > adjudication-body-members actually are in the debate anyway; if they > hadn't been, then the very last thing that debate needs is someone > wading in without any understanding of the complex technical issues > and trying to force some resolution. > > I'm not saying that truly irreconcilable problems never arise. But > given that they're so rare, and that realistically any system that > could solve them would need to be backed by basically the same group > of people that are in the current governance document's "steering > council" (because in an OSS project it's the people doing the work > that ultimately have the power), there's no point in building an > elaborate voting system and method for defining the electorate just to > handle these cases -- and quite a bit of harm. So the proposed system > is basically the simplest one that could work. (I described this line > of reasoning in more detail in [3].) > > I think it would really help if you could provide some examples of > successful community-driven projects and/or specific debates that > benefited from having an adjudication body like you envision? > > I also suspect this fundamental difference in how we view the role of > a governance body explains why I am unbothered by the idea of steering > council membership "timing out" after 2 years of inactivity. The > steering council's job is to informally manage things day to day, and > in extreme cases to make judgement calls. These two things both > crucially require a broad and up-to-date understanding of the issues > facing the project, ongoing debates, personalities, etc. But the > council is never intended to make any judgement call on its own; the > whole idea of the structure is to make sure that decisions are based > on as broad a scope of expertise as possible. In particular we > regularly get historical insight from Chuck, Ralf, Pauli, Robert Kern, > David Cournapeau, Pearu Peterson, Anne Archibald, Matthew Brett, > Stefan van der Walt... > and it doesn't matter at all whether they're on the council or not. > (If it did matter, that would be terrible, right? Why would you want > to *not* listen to any of those people?) If you plan to become more > active and give your perspective on things more then that's awesome > and welcome, but AFAICT this particular point is pretty orthogonal to > the composition of the steering council. > > See also this comment from Fernando about the Jupyter steering > council, which happened by chance to cross my email this morning: > https://github.com/jupyter/governance/pull/6#issuecomment-142036050 > > -n > > [1] https://www.debian.org/devel/tech-ctte#status > [2] https://contributors.debian.org/ > [3] > https://mail.scipy.org/pipermail/numpy-discussion/2015-September/073537.html > [4] See e.g. the bottom of this message from Nick Coghlan, where he > talks about how PEP "pronouncements" are done -- emphasizing in > particular (my paraphrase) that the point of a BDFL/BDFL-delegate is > not to resolve any substantive issues, and that a common strategy is > to choose the person who's most skeptical of the idea to be the > BDFL-delegate: > http://thread.gmane.org/gmane.comp.python.distutils.devel/23867 > > -- > Nathaniel J. Smith -- http://vorpus.org > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > https://mail.scipy.org/mailman/listinfo/numpy-discussion > -- *Travis Oliphant* *Co-founder and CEO* @teoliphant 512-222-5440 http://www.continuum.io -------------- next part -------------- An HTML attachment was scrubbed... URL: From travis at continuum.io Tue Sep 22 04:50:41 2015 From: travis at continuum.io (Travis Oliphant) Date: Tue, 22 Sep 2015 03:50:41 -0500 Subject: [Numpy-discussion] Governance model request In-Reply-To: <87eghrszyq.fsf@berkeley.edu> References: <8941C0D2-17EC-420D-A8F6-CA018144811C@continuum.io> <87mvwft2ar.fsf@berkeley.edu> <370C591F-9711-43E6-BE12-8C3E96946803@continuum.io> <87eghrszyq.fsf@berkeley.edu> Message-ID: > > > > > May? Can you elaborate? More speculation. My own position is that > > these projects want to integrate with NumPy, not the > > converse. Regardless of my opinion, can you actually make any specific > > arguements, one way or the otehr? What if if some integrations > > actually make more sense for the community? Is this simply a dogmatic > > ideological position that anything whatsoever that benefits both NumPy > > and Continuum simultaneously is bad, on principle? That's fine, as > > such, but let's make that position explicit if that's all it is. > > No, I don't have such a dogmatic ideological position. I think, > however, that it is somewhat unimaginative to propose that there are no > potential conflicts whatsoever. > > I am happy if we can find solutions that benefit both numpy and any > company out there. But in the end, I'm sure you'd agree that we want > the decisions that lead to such solutions to be taken in the best > interest of the project, and not be weighed by alterior motivations of > any sorts. In the end, even the *perception* that that is not the case > can be very harmful. > I will only comment on the last point. I completely agree that the *perception* that this is not the case can be harmful. But, what concerns me is where this perception comes from --- from actual evidence of anything that is not in the best interests of the project --- or just ideological differences of opinion about the way the world works and the perceptions around open source and markets. It is quite easy for someone to spread FUD about companies that contribute to open source --- and it has the effect of discouraging companies from continuing to contribute to community projects. This removes a huge amount of potential support from projects. In NumPy's case in particular, this kind of attitude basically guarantees that I won't be able to contribute effectively and potentially even people I fund to contribute might not be accepted --- not because we can't faithfully participate in the same spirit that we have always contributed to SciPy and NumPy and other open source projects --- but because people are basically going to question things just because. What exactly do you need me to say to get you to believe that I have nothing but the best interests of array computing in Python at heart? The only thing that is different between me today and me 18 years ago is that 1) I have more resources now, 2) I have more knowledge about computer science and software architecture and 3) I have more experience with how NumPy gets used. All I can do is continue to try and make things better the best way I know how. -Travis -- *Travis Oliphant* *Co-founder and CEO* @teoliphant 512-222-5440 http://www.continuum.io -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Tue Sep 22 05:33:00 2015 From: njs at pobox.com (Nathaniel Smith) Date: Tue, 22 Sep 2015 02:33:00 -0700 Subject: [Numpy-discussion] Governance model request In-Reply-To: References: <1442827933.393.61.camel@sipsolutions.net> <1442828394.393.64.camel@sipsolutions.net> Message-ID: On Tue, Sep 22, 2015 at 1:24 AM, Travis Oliphant wrote: > I actually do agree with your view of the steering council as being > usually not really being needed. You are creating a straw-man by > indicating otherwise. I don't believe a small council should do anything > *except* resolve disputes that cannot be resolved without one. Like you, I > would expect that would almost never happen --- but I would argue that > extrapolating from Debian's experience is not actually relevant here. > To be clear, Debian was only one example -- what I'm extrapolating from is every community-driven F/OSS project that I'm aware of. It's entirely possible my data set is incomplete -- if you have some other examples that you think would be better to extrapolate from, then I'd be genuinely glad to hear them. You may have noticed that I'm a bit of an enthusiast on this topic :-). > > So, if the steering council is not really needed then why have it at all? > Let's just eliminate the concept entirely. > > In my view, the reasons for having such a council are: 1) The framework is useful even if you never use it, because it means people can run "what if" scenarios in their mind and make decisions on that basis. In the US legal system, only a vanishingly small fraction of cases go to the Supreme Court -- but the rules governing the Supreme Court have a huge effect on all cases, because people can reason about what would happen *if* they tried to appeal to the Supreme Court. 2) It provides a formal structure for interfacing with the outside world. E.g., one can't do anything with money or corporate contributions without having some kind of written-down and enforceable rules for making decisions (even if in practice you always stick to the "everyone is equal and we govern by consensus" part of the rules). 3) There are rare but important cases where discussions have to be had in private. The main one is "personnel decisions" like inviting people to join the council; another example Fernando has mentioned to me is that when they need to coordinate a press release between the project and a funding body, the steering council reviews the press release before it goes public. That's pretty much it, IMO. The framework we all worked out at the dev meeting in Austin seems to handle these cases well AFAICT. > But there are real questions that have to have an answer or an approach to > making a decision. The answer to these questions cannot really be a vague > notion of "lack of vigorous opposition by people who read the mailing list" > which then gets parried about as "the community decided this." The NumPy > user base is far, far larger than the number of people that read this list. > According to the dev meeting rules, no particularly "vigorous opposition" is required -- anyone who notices that something bad is happening can write a single email and stop an idea dead in its tracks, with only the steering council able to overrule. We expect this will rarely if ever happen, because the threat will be enough to keep everyone honest and listening, but about the only way we could possibly be *more* democratic is if we started phoning up random users at home to ask their opinion. This is actually explicitly designed to prevent the situation where whoever talks the loudest and longest wins, and to put those with more and less time available on an equal footing. > For better or for worse, we will always be subject to the "tyranny of who > has time to contribute lately". Fundamentally, I would argue that this > kind of "tyranny" should at least be tempered by additional considerations > from long-time contributors who may also be acting more indirectly than is > measured by a simple git log. > I guess I am missing something fundamental here. Who are these long-time contributors who will sit on your council of 1/3/5 but who don't even read the mailing list? How will they know when their special insight is necessary? > So, what are the questions that have to have an answer that even calls for > some kind of governance? I know Nathaniel's document listed some of them > (and perhaps all of them). Here are mine: > > 1) who gets commit rights to the repo (and who has them removed)? > 2) who tags the release of NumPy? > 3) how is it decided where money is spent if it's donated to the project > (and not just to people directly)? > 4) how is it decided if someone needs to be removed from participation in > the group because they are not adding to the conversation (we have been > fortunate that this hasn't happened in NumPy before --- but it could)? > 5) how is it decided what goes on the NumPy website --- i.e. will > advertisers be able to put their logos or book-covers there? > > If I understand what you are proposing, then basically the "steering > council" decides these things. > No, absolutely not. The proposal is that these issues are decided by open discussion on the mailing list. (With the possible exception of #4 -- I suspect that given an extreme situation like this, then once all other efforts to mitigate the situation had failed the steering council would probably feel compelled to talk things over to double-check they had consensus before acting, and likely some part of this would have to be done in private.) This is pretty explicit in the document (and again, this is text and ideas stolen directly from Jupyter/IPython): "During the everyday project activities, council members participate in all discussions, code review and other project activities as peers with all other Contributors and the Community. In these everyday activities, Council Members do not have any special power or privilege through their membership on the Council. [...] the Council may, if necessary [do pretty much anything, but] the Council's primary responsibility is to facilitate the ordinary community-based decision making procedure described above. If we ever have to step in and formally override the community for the health of the Project, then we will do so, but we will consider reaching this point to indicate a failure in our leadership." If this is unclear then suggestions for improvement would certainly be welcome. This is pretty much how we do things now, and it has worked pretty well so far -- people do get commit bits, releases get tagged, and everyone seems happy with this part. I'd actually like to see a more explicit set of rules around e.g. who gets commit bits, just to set expectations and provide a clearer onramp for new contributors, but that's something that we can easily discuss and make a decision on later. All of these questions are ones that are easy enough to solve given some framework for governance, but they do all require substantive discussion. We're not going to do them justice if we try to solve them off-the-cuff in this thread, and trying would just derail the underlying discussion about how we make decisions. So I think we should stay focused on that. > Perhaps rather than a steering council, though, we just need clear > answers to questions like the above --- which might be handled differently > for different questions. I don't think these questions have very easy > answers. > > Ultimately NumPy has relied on and continues to rely on the mutual respect > of all the people that have worked on the code and tried to make it better. > We all have opinions about how things have gone in the past, and what > has gone well and what hasn't. But, nothing you have said persuades me > that you have a full picture of past history with respect to a lot of the > difficult kinds of conversations that have happened and the different modes > of activity that have tried to help move NumPy along. In fact, I think > you mis-understand and mis-interpret that history quite often. > I'm honestly somewhat unclear on why having a "full picture of past history" is necessary to contribute effectively (surely if that were the case, then only Jim Hugunin could comment?), but if there are things I'm missing then I'd certainly like to know. You've made comments along these lines several times now, but unfortunately I can't do much to improve my understanding without more concrete examples. I'm convinced you are well-intentioned and doing the very best you can, and > I'm very grateful that you are passionate and eager about moving NumPy > forward. Ultimately I hope it will help things. > Thank you. I am also convinced that you're well-intentioned and doing the very best you can, and grateful that you are also passionate and eager about moving NumPy forward. -n -- Nathaniel J. Smith -- http://vorpus.org -------------- next part -------------- An HTML attachment was scrubbed... URL: From travis at continuum.io Tue Sep 22 05:33:24 2015 From: travis at continuum.io (Travis Oliphant) Date: Tue, 22 Sep 2015 04:33:24 -0500 Subject: [Numpy-discussion] Governance model request In-Reply-To: <87d1xaudos.fsf@berkeley.edu> References: <8941C0D2-17EC-420D-A8F6-CA018144811C@continuum.io> <87mvwft2ar.fsf@berkeley.edu> <87d1xaudos.fsf@berkeley.edu> Message-ID: On Tue, Sep 22, 2015 at 2:16 AM, Stefan van der Walt wrote: > Hi Travis > > On 2015-09-21 23:29:12, Travis Oliphant wrote: > > 1) nobody believes that the community should be forced to adopt numba > as > > part of ufunc core yet --- but this could happen someday just as Cython > is > > now being adopted but was proposed 8 years ago that it "could be adopted" > > That's a red-hearing. > > Yes, I'd like to clarify: I was not against including any specific > technology in NumPy. I was highlighting that there may be different > motivations for members of the general community and those working for, > say, Continuum, to get certain features adopted. > This is what I'm calling you out on. Why? I think that is an unfair statement and inaccurate. The general community includes Continuum, Enthought, Microsoft, Intel, various hedge funds, investment banks and companies large and small. Are you saying that people should not be upfront about their affiliations with a company? That if they are not academics, then they should not participate in the discussion? It is hard enough to be at a company and get time to contribute effort back to an open source project. We should not be questioning people's motives just *because* they are at a company. We should not assume people cannot think in terns of the success of the project, just because they are at a company. Their proposals and contributions can be evaluated on their merits and value --- so this whole discussion seems to be just revealing an anti-company paranoia rather than helping understand the actual concern. > > 2) I have stated that breaking the ABI is of little consequence because > > of conda as well as other tools. I still believe that. This has > nothing > > to do with any benefit Continuum might or might not receive because of > > conda. Everyone else who wants to make a conda-based distribution also > > benefits (Cloudera, Microsoft, Intel, ...) or use conda also benefits. > > I don't think the community realizes the damange that is done with FUD > like > > this. There are real implications. It halts progress, creates > confusion, > > and I think ultimately damages the community. > > This is an old argument, and the reason why we have extensive measures > in place to guard against ABI breakage. But, reading what you wrote > above, I would like to understand better what FUD you are referring to, > because I, rightly or wrongly, believe there is a real concern here that > is being glossed over. > I don't know which is the "old argument". Anyway, old arguments can still be right. The fact is that not breaking the ABI has caused real damage to the community. NumPy was never designed to not have it's ABI broken for over a decade. We have some attempts to guard against ABI breakage --- but they are not perfect. We have not moved the code-base forward for fear of breaking the ABI. When it was hard to update your Python installation that was a concern. There are very few cases where this is still the concern (conda is a big part of it but not the only part as other distros and approaches for easily updating the install exist) --- having this drive major architecture decisions is a serious mistake in my mind, and causes a lot more work than it should. The FUD I'm talking about is the anti-company FUD that has influenced discussions in the past. I really hope that we can move past this. > > > I don't see how. None of these have been proposed for integrating into > > NumPy. I don't see how integrating numba into NumPy benefits Continuum > > at all. It's much easier for us to keep it separate. At this point > > Continuum doesn't have an opinion about integrating DyND into NumPy or > > not. > > I think that touches, tangentially at least, on the problem. If an > employee of Continuum were steering NumPy, and the company developed an > opinion on those integrations, would such a person not feel compelled to > toe the company line? (Whether the company is Continuum or another is > besides the point?I am only trying to understand the dynamics of working > for a company and leading an open source project that closely interacts > with their producs.) > O.K. if you are honestly asking this question out of inexperience, then I can at least help you understand because perhaps that is the problem (creating a straw-man that doesn't exist). I have never seen a motivated open source developer at a company who "tows the company line" within a community project that is accepted long term. All that would do is drive the developer out of the company and be a sure-fire way to make sure their contributions are not accepted. I know that at Continuum, for example, if someone disagreed with me about an open source technology, didn't tell me and just "towed the company line" (whatever they thought that was) --- they would be fired. Most software companies that participate in open source are like that. The worst thing that happens is the company no longer participates in the discussion --- that is what happens if they can't get contributions. It really is no different than me being worried that an academic might push to get something into a library because it would "help their publication record" --- which I also think is a very real potential concern. Or perhaps you have a favor owed to another colleague who helped you along in your academic career and they "really want" some feature added. The actual conflict of interest that exists there has the same amount of weight in my mind as the ones you hypothesize about. > > > I know that you were responding to specific question by Brian as to how > > their could be a conflict of interest for Continuum and NumPy > development. > > I don't think this is a useful conversation --- we could dream up all > > kinds of conflicts of interest for BIDS and NumPy too (e.g. perhaps BIDS > > really wants Spark to take over and for NumPy to have special connections > > to Spark). Are we to not allow anyone at BIDS to participate in the > > steering council because of their other interests? > > I guess that's an interesting example, but BIDS (which sits inside a > university and is funded primarily by foundations) has no financial, and > very few other, incentives to do so. > I think you don't understand how academic finances work. I could see a lot of ways such interest could develop --- and could influence funding agencies. I don't think they are very likely --- but they are possible --- and to me the same degree of possibility that Continuum would try to "force anything" on the NumPy community. > > > But remember, the original point is whether or not someone from Continuum > > (or I presume any company and not just singling out Continuum for special > > treatment) should be on the steering council. Are you really arguing > > that they shouldn't because there are other projects Continuum is working > > on that have some overlap with NumPy. I really hope you don't actually > > believe that. > > Here's what I'm trying to say (and I apologise for ruffling feathers in > the process): > > There are concerns amongst members of the community that (will) arise > when strong players from industry try / hint at exerting (some) > executive control over NumPy. We can say that these concerns amount to > spreading FUD, that they are uninformed, unrealistic, etc., but > ultimately they are still out there, and until they are discussed and > addressed, I find it hard to see how we can move forward with ease. > I'm sorry you have these concerns. I don't think they are as warranted as you believe. NumPy will get strong players from industry coming in. In fact, one of the reasons we all felt it was important to establish Numfocus was to be an organization that could shield these players from the development of NumPy and other projects. I am not one of those "strong industry players" you speak of. I am a friend. I am a participant. I am one of the community. Perhaps you and others feel that *I* am that strong industry player trying to exert "executive control" over NumPy. Is that true? Is that what you feel? I'm starting to wonder if that is actually the problem here. -Travis > St?fan > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > https://mail.scipy.org/mailman/listinfo/numpy-discussion > -- *Travis Oliphant* *Co-founder and CEO* @teoliphant 512-222-5440 http://www.continuum.io -------------- next part -------------- An HTML attachment was scrubbed... URL: From travis at continuum.io Tue Sep 22 05:43:18 2015 From: travis at continuum.io (Travis Oliphant) Date: Tue, 22 Sep 2015 04:43:18 -0500 Subject: [Numpy-discussion] 1.10.0rc1 coming tomorrow, 22 Sept. In-Reply-To: References: Message-ID: Absolutely it would be good if others can test. All I was suggesting is that we do run a pretty decent set of tests upon build and that would be helpful. If the numpy build recipes are not available, it is only because they have not been updated to use conda-build yet. If somebody wants to volunteer to convert all of our internal recipes to conda-build recipes so they could be open source --- we would welcome the help. But, it's not just the numpy recipes, it's the downstream binaries and their test-suite as well that is useful to run. I am hoping we will have something automatic here in the next few months on anaconda.org that will make this easier -- but no promises at this point. -Travis On Tue, Sep 22, 2015 at 2:19 AM, Nathaniel Smith wrote: > On Sep 21, 2015 11:51 PM, "Travis Oliphant" wrote: > > > > Of course it will be 1.10.0 final where all the problems will show up > suddenly :-) > > > > Perhaps we can get to where we are testing Anaconda against beta > releases better. > > The most useful thing would actually not even involve you doing any more > testing, but just if you could make builds available so that end-users > could easily conda install the prereleases and do their own testing against > their own choice. In principle I guess we could provide our own binstar > channel for this, but it's difficult given that AFAIK rebuilding numpy in > conda requires also rebuilding the whole stack, and the numpy build recipes > are still proprietary. > > -n > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > https://mail.scipy.org/mailman/listinfo/numpy-discussion > > -- *Travis Oliphant* *Co-founder and CEO* @teoliphant 512-222-5440 http://www.continuum.io -------------- next part -------------- An HTML attachment was scrubbed... URL: From travis at continuum.io Tue Sep 22 06:08:09 2015 From: travis at continuum.io (Travis Oliphant) Date: Tue, 22 Sep 2015 05:08:09 -0500 Subject: [Numpy-discussion] Governance model request In-Reply-To: References: <1442827933.393.61.camel@sipsolutions.net> <1442828394.393.64.camel@sipsolutions.net> Message-ID: On Tue, Sep 22, 2015 at 4:33 AM, Nathaniel Smith wrote: > On Tue, Sep 22, 2015 at 1:24 AM, Travis Oliphant > wrote: > >> I actually do agree with your view of the steering council as being >> usually not really being needed. You are creating a straw-man by >> indicating otherwise. I don't believe a small council should do anything >> *except* resolve disputes that cannot be resolved without one. Like you, I >> would expect that would almost never happen --- but I would argue that >> extrapolating from Debian's experience is not actually relevant here. >> > > To be clear, Debian was only one example -- what I'm extrapolating from is > every community-driven F/OSS project that I'm aware of. > > It's entirely possible my data set is incomplete -- if you have some other > examples that you think would be better to extrapolate from, then I'd be > genuinely glad to hear them. You may have noticed that I'm a bit of an > enthusiast on this topic :-). > > Yes, you are much better at that than I am. I'm not even sure where I would look for this kind of data. > >> > So, if the steering council is not really needed then why have it at all? >> Let's just eliminate the concept entirely. >> >> > In my view, the reasons for having such a council are: > 1) The framework is useful even if you never use it, because it means > people can run "what if" scenarios in their mind and make decisions on that > basis. In the US legal system, only a vanishingly small fraction of cases > go to the Supreme Court -- but the rules governing the Supreme Court have a > huge effect on all cases, because people can reason about what would happen > *if* they tried to appeal to the Supreme Court. > O.K. That is a good point. I can see the value in that. > 2) It provides a formal structure for interfacing with the outside world. > E.g., one can't do anything with money or corporate contributions without > having some kind of written-down and enforceable rules for making decisions > (even if in practice you always stick to the "everyone is equal and we > govern by consensus" part of the rules). > O.K. > 3) There are rare but important cases where discussions have to be had in > private. The main one is "personnel decisions" like inviting people to join > the council; another example Fernando has mentioned to me is that when they > need to coordinate a press release between the project and a funding body, > the steering council reviews the press release before it goes public. > O.K. > That's pretty much it, IMO. > > The framework we all worked out at the dev meeting in Austin seems to > handle these cases well AFAICT. > How did we "all" work it out when not everyone was there? This is where I get lost. You talk about community decision making and yet any actual decision is always a subset of the community. I suppose you just rely on the "if nobody complains than it's o.k." rule? That really only works if the project is moving slowly. > But there are real questions that have to have an answer or an approach to >> making a decision. The answer to these questions cannot really be a vague >> notion of "lack of vigorous opposition by people who read the mailing list" >> which then gets parried about as "the community decided this." The NumPy >> user base is far, far larger than the number of people that read this list. >> > > According to the dev meeting rules, no particularly "vigorous opposition" > is required -- anyone who notices that something bad is happening can write > a single email and stop an idea dead in its tracks, with only the steering > council able to overrule. We expect this will rarely if ever happen, > because the threat will be enough to keep everyone honest and listening, > but about the only way we could possibly be *more* democratic is if we > started phoning up random users at home to ask their opinion. > O.K. so how long is the time allowed for this kind of opposition to be noted? > > This is actually explicitly designed to prevent the situation where > whoever talks the loudest and longest wins, and to put those with more and > less time available on an equal footing. > > >> For better or for worse, we will always be subject to the "tyranny of who >> has time to contribute lately". Fundamentally, I would argue that this >> kind of "tyranny" should at least be tempered by additional considerations >> from long-time contributors who may also be acting more indirectly than is >> measured by a simple git log. >> > > I guess I am missing something fundamental here. Who are these long-time > contributors who will sit on your council of 1/3/5 but who don't even read > the mailing list? How will they know when their special insight is > necessary? > The long-time contributors wouldn't necessarily sit on that council. But, I would support the idea of an advisory council that could act if it saw things going the wrong direction. This is where those people would sit. In the case of a 1 / 3 / 5 member council -- I would not argue to be on it at all (but I would argue that some care should be taken to be sure it has people with some years of experience if they are available). I'm only asking to be on a steering council that is larger than 5 people, and I don't actually prefer that the steering council be larger than 5 people. > No, absolutely not. The proposal is that these issues are decided by open > discussion on the mailing list. (With the possible exception of #4 -- I > suspect that given an extreme situation like this, then once all other > efforts to mitigate the situation had failed the steering council would > probably feel compelled to talk things over to double-check they had > consensus before acting, and likely some part of this would have to be done > in private.) > O.K. Then, I am misunderstanding. > > This is pretty explicit in the document (and again, this is text and ideas > stolen directly from Jupyter/IPython): > > "During the everyday project activities, council members participate in > all discussions, code review and other project activities as peers with all > other Contributors and the Community. In these everyday activities, Council > Members do not have any special power or privilege through their membership > on the Council. [...] the Council may, if necessary [do pretty much > anything, but] the Council's primary responsibility is to facilitate the > ordinary community-based decision making procedure described above. If we > ever have to step in and formally override the community for the health of > the Project, then we will do so, but we will consider reaching this point > to indicate a failure in our leadership." > Granting commit rights to the repo does not seem to me to be an "everyday project activity" --- so I suppose I was confused. I suppose that could happen with no serious objections on the list. It seems that the people who actually have the commit bit presently should be consulted more than the general public, though. > This is pretty much how we do things now, and it has worked pretty well so > far -- people do get commit bits, releases get tagged, and everyone seems > happy with this part. I'd actually like to see a more explicit set of rules > around e.g. who gets commit bits, just to set expectations and provide a > clearer onramp for new contributors, but that's something that we can > easily discuss and make a decision on later. > O.K. I think that the commit bit has not been that clear --- and was closer when I was more active to the commiters rule I gave before. > > All of these questions are ones that are easy enough to solve given some > framework for governance, but they do all require substantive discussion. > We're not going to do them justice if we try to solve them off-the-cuff in > this thread, and trying would just derail the underlying discussion about > how we make decisions. So I think we should stay focused on that. > > >> Perhaps rather than a steering council, though, we just need clear >> answers to questions like the above --- which might be handled differently >> for different questions. I don't think these questions have very easy >> answers. >> >> Ultimately NumPy has relied on and continues to rely on the mutual >> respect of all the people that have worked on the code and tried to make it >> better. We all have opinions about how things have gone in the past, >> and what has gone well and what hasn't. But, nothing you have said >> persuades me that you have a full picture of past history with respect to a >> lot of the difficult kinds of conversations that have happened and the >> different modes of activity that have tried to help move NumPy along. In >> fact, I think you mis-understand and mis-interpret that history quite >> often. >> > > I'm honestly somewhat unclear on why having a "full picture of past > history" is necessary to contribute effectively (surely if that were the > case, then only Jim Hugunin could comment?), but if there are things I'm > missing then I'd certainly like to know. You've made comments along these > lines several times now, but unfortunately I can't do much to improve my > understanding without more concrete examples. > The reason is that you in particular make very long arguments that try to persuade on the basis of your understanding of how things were done in the past (both here and in other projects). If you don't understand the history and the arguments that have been had before, you are throwing away information. It's not a show-stopper, for sure, but it does potentially make it less likely that a better solution will be found. I will have to respond to your incorrect characterizations in another thread. -Travis -------------- next part -------------- An HTML attachment was scrubbed... URL: From shoyer at gmail.com Tue Sep 22 06:43:07 2015 From: shoyer at gmail.com (Stephan Hoyer) Date: Tue, 22 Sep 2015 03:43:07 -0700 Subject: [Numpy-discussion] Governance model request In-Reply-To: References: <8941C0D2-17EC-420D-A8F6-CA018144811C@continuum.io> <87mvwft2ar.fsf@berkeley.edu> <87d1xaudos.fsf@berkeley.edu> Message-ID: On Tue, Sep 22, 2015 at 2:33 AM, Travis Oliphant wrote: > The FUD I'm talking about is the anti-company FUD that has influenced > discussions in the past. I really hope that we can move past this. > I have mostly stayed out of the governance discussion, in deference to how new I am in this community, but I do want to take a moment to speak up here to echo Travis's concern about anti-company FUD. Everyone invested in NumPy has their own projects, priorities and employers which shape their agenda. As far as I can tell, Travis and Continuum have only ever acted with the long term health of the scipy ecosystem in mind. Stephan -------------- next part -------------- An HTML attachment was scrubbed... URL: From travis at continuum.io Tue Sep 22 06:44:12 2015 From: travis at continuum.io (Travis Oliphant) Date: Tue, 22 Sep 2015 05:44:12 -0500 Subject: [Numpy-discussion] Governance model request In-Reply-To: References: <8941C0D2-17EC-420D-A8F6-CA018144811C@continuum.io> <87mvwft2ar.fsf@berkeley.edu> <370C591F-9711-43E6-BE12-8C3E96946803@continuum.io> Message-ID: On Tue, Sep 22, 2015 at 2:33 AM, Nathaniel Smith wrote: > Hi Bryan, > > I understand where you're coming from, but I'd appreciate it if we > could keep the discussion on a less visceral level? Nobody's personal > integrity is being impugned, but it's the nature of this kind of > governance discussion that we have to consider unlikely-and-unpleasant > hypotheticals. It's like when you talk to a lawyer about a contract or > a will or whatever: they'll make you think about all kinds of horrible > possibilities, not because any of them are likely, but because sooner > or later *something* will go wrong, and the point of having a > contract/will/governance document is to provide some framework to > handle whichever unlikely edge case does arise. > > And the other purpose of this kind of framework is to avoid the > *perception* (whether justified or not) of these kinds of conflicts of > interest -- if not handled well then this can easily scare away > volunteers, contributions from other companies, etc. Obviously you > know Travis and Continuum well as an employee there, but to most > people without that personal connection, Continuum is just a large > corporate entity with unknown motives. Imagine if instead of Continuum > we were talking about it was Google or RandomAggressiveStartup -- some > company that you didn't have any personal connection or insight into. > For someone in this position, it's not unreasonable to want more of a > reassurance that things will work out then just being asked to trust > that the CEO is personally committed to not being evil and they can > trust him. > Anybody who comes to NumPy should know that I wrote it and then gave it to the community -- creating an independent foundation at the same time as I created a Company to create a division between them so that my actions could be understood. I really don't know how to give more reassurance. I'm actually offended that so many at BIDS seem eager to crucify my intentions when I've done nothing but give away my time, my energy, my resources, and my sleep to NumPy for many, many years. I guess if your intent is to drive me away, then you are succeeding. > > Also, in these messages you seem to be working from a framework where > people working in good faith will always agree, and so any suggestion > of a possible conflict of interest can only arise through bad faith. > But this isn't true. Is it really so difficult to imagine that, say, > Continuum and Enthought might at some point have conflicting needs for > numpy, or for Continuum's vision of the future of numpy could be > less-than-perfectly-representative with every bit of numpy's entire > giant userbase? Of course not, but this is no different than anyone else and a company should not be singled out. All you are doing is forcing any contribution to be either only from a non-profit or have individuals hide their actual allegiances. > Continuum is a company that has in the past submitted > rather obscure patches to numpy that AFAICT are used exclusively by a > particular contracting customer (e.g. [1]), and that is currently > investing a substantial multiple of numpy's development budget on > building a direct competitor to numpy. > Good grief! These comments are so much bunk that I have to call you on it emphatically. You claim below that you are unconcerned but yet you insinuate some crazy, ulterior motivations for Continuum actually helping people upgrade their NumPy installation that broke their old code because of changes to NumPy. This kind of comment really upsets me. You dismiss real things and real problems that happen and brush it away because it's *commercial*. That patch you reference was actually an attempt to fix a problem that the community pushed on the world --- therefore breaking people's code (but good thing the ABI didn't change...). We were up front about it and worked with the community to get a change into NumPy to accommodate a user of the tool. It was hard work to figure out how to do that. To have you use that as some kind of argument against Continuum is not only unfair, it is exactly the kind of mis-characterization and mis-interpretation of events that I refer to in other posts. And to say that we are investing a substantial multiple of Numpy's development budget in a competitor is also incorrect. Continuum invests in competent people who want to do interesting things. We don't have a rule that says things are "off-limits" including NumPy. If competent people feel like an alternative to NumPy is a good idea, then a certain amount of exploration in that direction is allowed. DyND does not have to be a competitor to NumPy. It might compete with *your* vision of NumPy, but it doesn't have to compete with NumPy. > > To emphasize: I personally am not concerned by these facts -- we did > merge that patch I linked, and there's no animosity between the numpy > and dynd teams -- but reasonable people could reasonably be concerned > that tricky situations might emerge in the future, and I've talked to > plenty of people who are nervous about Continuum's influence in > general. Who are these people? How about they come forward and express what it is they are actually nervous about. Really? nervous? What kind of "tricky" situations are we talking about. Can't you see that this sounds as odd to me as me telling you that I'm concerned about BIDS influence? What about Dato, Databricks, Enthought, or Cloudera influence? What does this even mean? Is this just Matthew and Stefan or are there others as well with these feelings? These are the only actual people who have expressed in public what might be considered concern that I am aware of. I think this kind of anti-commercial commentary has no place in a community that also includes people that work at companies. I can't see how we can agree to *any* governance document with this kind of FUD being spread around. > And with my numpy developer hat on, I don't even care which > "side" is right, that's irrelevant to me, because my concern is with > providing a space where both "sides" feel comfortable working > together. This is why it's crucial that numpy-the-project be clearly > distinguishable as an independent entity that is beholden only to its > own community: it's *exactly because* we *value* the contribution of > companies like Continuum, and want to be able to freely foster those > relationships without creating suspicion and bad blood. > Of course we agree on this. I have no idea why anyone thinks we don't? That's the one thing we *do* agree on. That NumPy is an independent project which can be influenced by anyone in the community and should be developed based on technical discussions and not fear of hob-goblins and people that also work at companies that may benefit from the work that goes on here (there is a large list in this camp). I am deeply saddened by the insinuation and the implication of what these threads are telling me about how little my efforts have been valued by people I care about. > Also to emphasize: none of this means that Travis can't be on the > steering council -- I think that's a more complex issue that I'll > address separately. All I'm saying is that pretending that you aren't > going to reassure people by pretending this elephant isn't in the > room, or by taking a reasonable set of concerns and aggressively > turning them into a referendum on individual people's integrity. > We should call out the elephant in the room. But, I think we should understand who and what the elephant is. Perhaps there are too many off-list and back-channel conversations happening at BIDS and elsewhere that are serving to bias people against facts. Facts that otherwise would show that I and Continuum have always just been trying to ensure the success of NumPy as an independent project that is fully supported, backward compatible, maintained, available to the world in easy to install ways, and documented. > Finally, can I point out... anyone who has some wariness around the > possible influence of financial interests on the community (whether > justified or not!) is in particular not going to be reassured if you > keep aggressively attempting to shut down any perceived criticism of > *your own employer*. I know that your paycheck is not dictating your > opinions, and probably the hypothetical people I'm talking about are > being totally unfair to you for even considering such a thing, but... > strictly as a matter of practical rhetoric, I don't think this is the > most effective way to accomplish your goals. > > I agree with this. I certainly did not encourage Bryan. He was acting out of his own sense of injustice. But, I would add, that your insinuation and mis-characterization of my activities at Continuum and potentially elsewhere are unfair and incorrect and also not effective at getting governance documents approved and ratified. I will get over feeling offended and work to get over my frustration at academics for thrusting this anti-company and potentially anti-Travis rhetoric on the community. But, if you indeed have such hard feelings, then please air all of them so we can hopefully just get past this. > -n > > [1] https://github.com/numpy/numpy/pull/359 > > > On Mon, Sep 21, 2015 at 11:28 PM, Bryan Van de Ven > wrote: > > > >> I don't know how productive it is to dream up examples, but it's not > > > > Well, agreed, to be honest. > > > >> very hard to do. Currently, e.g., the community is not ready to adopt > >> numba as part of the ufunc core. But it's been stated by some that, > > > > Who are you speaking for? The entire community? Under what mandate? > > > >> with so many people running Conda, breaking the ABI is of little > >> consequence. And then it wouldn't be much of a leap to think that numba > >> is an acceptable dependency. > > > > The current somewhat concrete proposal I am aware of involves funding > cleaning up dtypes. Is there another concrete, credible proposal to make > Numba a dependency of NumPy that you can refer to? If not, why are we mired > in hypotheticals? > > > >> There's a broad range of Continuum projects that intersect with what > >> NumPy does: numba, DyND, dask and Odo to name a few. Integrating them > >> into NumPy may make a lot more sense for someone from Continuum than for > >> other members of the community. > > > > May? Can you elaborate? More speculation. My own position is that these > projects want to integrate with NumPy, not the converse. Regardless of my > opinion, can you actually make any specific arguements, one way or the > otehr? What if if some integrations actually make more sense for the > community? Is this simply a dogmatic ideological position that anything > whatsoever that benefits both NumPy and Continuum simultaneously is bad, on > principle? That's fine, as such, but let's make that position explicit if > that's all it is. > > > > Bryan > > _______________________________________________ > > NumPy-Discussion mailing list > > NumPy-Discussion at scipy.org > > https://mail.scipy.org/mailman/listinfo/numpy-discussion > > > > -- > Nathaniel J. Smith -- http://vorpus.org > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > https://mail.scipy.org/mailman/listinfo/numpy-discussion > -- *Travis Oliphant* *Co-founder and CEO* @teoliphant 512-222-5440 http://www.continuum.io -------------- next part -------------- An HTML attachment was scrubbed... URL: From sebastian at sipsolutions.net Tue Sep 22 07:41:55 2015 From: sebastian at sipsolutions.net (Sebastian Berg) Date: Tue, 22 Sep 2015 13:41:55 +0200 Subject: [Numpy-discussion] Governance model request In-Reply-To: References: <8941C0D2-17EC-420D-A8F6-CA018144811C@continuum.io> <87mvwft2ar.fsf@berkeley.edu> <370C591F-9711-43E6-BE12-8C3E96946803@continuum.io> Message-ID: <1442922115.4216.60.camel@sipsolutions.net> On Di, 2015-09-22 at 05:44 -0500, Travis Oliphant wrote: > > > On Tue, Sep 22, 2015 at 2:33 AM, Nathaniel Smith > wrote: > Hi Bryan, > > I understand where you're coming from, but I'd appreciate it > if we > could keep the discussion on a less visceral level? Nobody's > personal > integrity is being impugned, but it's the nature of this kind > of > governance discussion that we have to consider > unlikely-and-unpleasant > hypotheticals. It's like when you talk to a lawyer about a > contract or > a will or whatever: they'll make you think about all kinds of > horrible > possibilities, not because any of them are likely, but because > sooner > or later *something* will go wrong, and the point of having a > contract/will/governance document is to provide some framework > to > handle whichever unlikely edge case does arise. > > And the other purpose of this kind of framework is to avoid > the > *perception* (whether justified or not) of these kinds of > conflicts of > interest -- if not handled well then this can easily scare > away > volunteers, contributions from other companies, etc. Obviously > you > know Travis and Continuum well as an employee there, but to > most > people without that personal connection, Continuum is just a > large > corporate entity with unknown motives. Imagine if instead of > Continuum > we were talking about it was Google or RandomAggressiveStartup > -- some > company that you didn't have any personal connection or > insight into. > For someone in this position, it's not unreasonable to want > more of a > reassurance that things will work out then just being asked to > trust > that the CEO is personally committed to not being evil and > they can > trust him. > > > Anybody who comes to NumPy should know that I wrote it and then gave > it to the community -- creating an independent foundation at the same > time as I created a Company to create a division between them so that > my actions could be understood. I really don't know how to give more > reassurance. > > > I'm actually offended that so many at BIDS seem eager to crucify my > intentions when I've done nothing but give away my time, my energy, my > resources, and my sleep to NumPy for many, many years. I guess if > your intent is to drive me away, then you are succeeding. > > Frankly, I am bit surprised at how this is developing. Why?! Nobody, except being silly, really would doubt your intentions. That said, I will be honest with you. I do not feel I know you well enough (or the mode you operate) to for example accept you as BDFL of numpy [1] and I frankly do not think that -- just like anyone else -- you should have any special place in the governance document (which obviously does not mean you should be blocked from being on the steering council) [2]. I just think there should probably not be any specific name in the NumPy governance document at all. The thing is, NOBODY really seems suggests that [3]. I have dislike giving any special "power" to ANYONE. Frankly, I think if some people are nervous (not so much those active in the discussion), it is probably because they perceive that you have some direct power over numpy. As you have said, this is just not true. But *because* of the lack of a governance document, I would not be surprised if it *appears* to many like you could wield a lot of power if you so wished. Simply because few people really know how things are decided currently. And I think this wrong perception is what makes some nervous. If you say "maybe we should stop worrying about ABI" it may sound like "two years from now we will definitely break ABI", the curse being that it does not matter that you and those who know numpy's well know that it is just you stressing strongly that we should seriously discuss it. I have to admit, you sometimes sound a bit too definite in your suggestions given your former position ;). I hope I have not been rude here, Sebastian [1] I am sure you were *exactly* the right person to start numpy and be the de-facto BDFL then, but today this is not on the table anyway. [2] Also, lets be honest, you probably do have quite a bit of soft influence, just by knowing the community, being at the conferences, having NumFOCUS close by, etc. [3] I guess you have in some sense, but I now understand that as a suggestion to approach a different problem. And we can find another solution for that problem! > Also, in these messages you seem to be working from a > framework where > people working in good faith will always agree, and so any > suggestion > of a possible conflict of interest can only arise through bad > faith. > But this isn't true. Is it really so difficult to imagine > that, say, > Continuum and Enthought might at some point have conflicting > needs for > numpy, or for Continuum's vision of the future of numpy could > be > less-than-perfectly-representative with every bit of numpy's > entire > giant userbase? > > > Of course not, but this is no different than anyone else and a company > should not be singled out. All you are doing is forcing any > contribution to be either only from a non-profit or have individuals > hide their actual allegiances. > > Continuum is a company that has in the past submitted > rather obscure patches to numpy that AFAICT are used > exclusively by a > particular contracting customer (e.g. [1]), and that is > currently > investing a substantial multiple of numpy's development budget > on > building a direct competitor to numpy. > > > Good grief! These comments are so much bunk that I have to call you > on it emphatically. You claim below that you are unconcerned but yet > you insinuate some crazy, ulterior motivations for Continuum actually > helping people upgrade their NumPy installation that broke their old > code because of changes to NumPy. This kind of comment really > upsets me. You dismiss real things and real problems that happen and > brush it away because it's *commercial*. > > > That patch you reference was actually an attempt to fix a problem that > the community pushed on the world --- therefore breaking people's code > (but good thing the ABI didn't change...). We were up front about it > and worked with the community to get a change into NumPy to > accommodate a user of the tool. It was hard work to figure out how to > do that. To have you use that as some kind of argument against > Continuum is not only unfair, it is exactly the kind of > mis-characterization and mis-interpretation of events that I refer to > in other posts. > > > And to say that we are investing a substantial multiple of Numpy's > development budget in a competitor is also incorrect. Continuum > invests in competent people who want to do interesting things. We > don't have a rule that says things are "off-limits" including NumPy. > If competent people feel like an alternative to NumPy is a good idea, > then a certain amount of exploration in that direction is allowed. > DyND does not have to be a competitor to NumPy. It might compete > with *your* vision of NumPy, but it doesn't have to compete with > NumPy. > > > To emphasize: I personally am not concerned by these facts -- > we did > merge that patch I linked, and there's no animosity between > the numpy > and dynd teams -- but reasonable people could reasonably be > concerned > that tricky situations might emerge in the future, and I've > talked to > plenty of people who are nervous about Continuum's influence > in > general. > > > Who are these people? How about they come forward and express what > it is they are actually nervous about. Really? nervous? What kind > of "tricky" situations are we talking about. Can't you see that this > sounds as odd to me as me telling you that I'm concerned about BIDS > influence? What about Dato, Databricks, Enthought, or Cloudera > influence? What does this even mean? Is this just Matthew and > Stefan or are there others as well with these feelings? These are > the only actual people who have expressed in public what might be > considered concern that I am aware of. I think this kind of > anti-commercial commentary has no place in a community that also > includes people that work at companies. > > > I can't see how we can agree to *any* governance document with this > kind of FUD being spread around. > > And with my numpy developer hat on, I don't even care which > "side" is right, that's irrelevant to me, because my concern > is with > providing a space where both "sides" feel comfortable working > together. This is why it's crucial that numpy-the-project be > clearly > distinguishable as an independent entity that is beholden only > to its > own community: it's *exactly because* we *value* the > contribution of > companies like Continuum, and want to be able to freely foster > those > relationships without creating suspicion and bad blood. > > > Of course we agree on this. I have no idea why anyone thinks we > don't? That's the one thing we *do* agree on. That NumPy is an > independent project which can be influenced by anyone in the community > and should be developed based on technical discussions and not fear of > hob-goblins and people that also work at companies that may benefit > from the work that goes on here (there is a large list in this camp). > I am deeply saddened by the insinuation and the implication of what > these threads are telling me about how little my efforts have been > valued by people I care about. > > Also to emphasize: none of this means that Travis can't be on > the > steering council -- I think that's a more complex issue that > I'll > address separately. All I'm saying is that pretending that you > aren't > going to reassure people by pretending this elephant isn't in > the > room, or by taking a reasonable set of concerns and > aggressively > turning them into a referendum on individual people's > integrity. > > > We should call out the elephant in the room. But, I think we should > understand who and what the elephant is. Perhaps there are too many > off-list and back-channel conversations happening at BIDS and > elsewhere that are serving to bias people against facts. Facts that > otherwise would show that I and Continuum have always just been trying > to ensure the success of NumPy as an independent project that is fully > supported, backward compatible, maintained, available to the world in > easy to install ways, and documented. > > > > Finally, can I point out... anyone who has some wariness > around the > possible influence of financial interests on the community > (whether > justified or not!) is in particular not going to be reassured > if you > keep aggressively attempting to shut down any perceived > criticism of > *your own employer*. I know that your paycheck is not > dictating your > opinions, and probably the hypothetical people I'm talking > about are > being totally unfair to you for even considering such a thing, > but... > strictly as a matter of practical rhetoric, I don't think this > is the > most effective way to accomplish your goals. > > > > I agree with this. I certainly did not encourage Bryan. He was > acting out of his own sense of injustice. > > > But, I would add, that your insinuation and mis-characterization of my > activities at Continuum and potentially elsewhere are unfair and > incorrect and also not effective at getting governance documents > approved and ratified. I will get over feeling offended and work to > get over my frustration at academics for thrusting this anti-company > and potentially anti-Travis rhetoric on the community. > > > But, if you indeed have such hard feelings, then please air all of > them so we can hopefully just get past this. > > > > -n > > [1] https://github.com/numpy/numpy/pull/359 > > > On Mon, Sep 21, 2015 at 11:28 PM, Bryan Van de Ven > wrote: > > > >> I don't know how productive it is to dream up examples, but > it's not > > > > Well, agreed, to be honest. > > > >> very hard to do. Currently, e.g., the community is not > ready to adopt > >> numba as part of the ufunc core. But it's been stated by > some that, > > > > Who are you speaking for? The entire community? Under what > mandate? > > > >> with so many people running Conda, breaking the ABI is of > little > >> consequence. And then it wouldn't be much of a leap to > think that numba > >> is an acceptable dependency. > > > > The current somewhat concrete proposal I am aware of > involves funding cleaning up dtypes. Is there another > concrete, credible proposal to make Numba a dependency of > NumPy that you can refer to? If not, why are we mired in > hypotheticals? > > > >> There's a broad range of Continuum projects that intersect > with what > >> NumPy does: numba, DyND, dask and Odo to name a few. > Integrating them > >> into NumPy may make a lot more sense for someone from > Continuum than for > >> other members of the community. > > > > May? Can you elaborate? More speculation. My own position is > that these projects want to integrate with NumPy, not the > converse. Regardless of my opinion, can you actually make any > specific arguements, one way or the otehr? What if if some > integrations actually make more sense for the community? Is > this simply a dogmatic ideological position that anything > whatsoever that benefits both NumPy and Continuum > simultaneously is bad, on principle? That's fine, as such, but > let's make that position explicit if that's all it is. > > > > Bryan > > _______________________________________________ > > NumPy-Discussion mailing list > > NumPy-Discussion at scipy.org > > https://mail.scipy.org/mailman/listinfo/numpy-discussion > > > > > -- > Nathaniel J. Smith -- http://vorpus.org > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > https://mail.scipy.org/mailman/listinfo/numpy-discussion > > > > > > -- > > Travis Oliphant > Co-founder and CEO > > > > > > @teoliphant > 512-222-5440 > http://www.continuum.io > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > https://mail.scipy.org/mailman/listinfo/numpy-discussion -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 819 bytes Desc: This is a digitally signed message part URL: From sturla.molden at gmail.com Tue Sep 22 08:19:44 2015 From: sturla.molden at gmail.com (Sturla Molden) Date: Tue, 22 Sep 2015 14:19:44 +0200 Subject: [Numpy-discussion] Governance model request In-Reply-To: References: Message-ID: On 20/09/15 20:20, Travis Oliphant wrote: > 1 - define a BDFL for the council. I would nominate chuck Harris > > 2 - limit the council to 3 people. I would nominate chuck, nathaniel, > and pauli. > > 3 - add me as a permanent member of the steering council. I have stayed out of this governance debate until now. Personally I would prefer if you were BDFL. Sturla From stsci.perry at gmail.com Tue Sep 22 08:31:29 2015 From: stsci.perry at gmail.com (Perry Greenfield) Date: Tue, 22 Sep 2015 08:31:29 -0400 Subject: [Numpy-discussion] Governance model request In-Reply-To: References: Message-ID: I?ve also stayed out of this until now. I?m surprised and disheartened at the amount of suspicion and distrust directed towards Travis. I don?t think anyone has invested as much personal time and resources (e.g., money) towards supporting numpy, and not just in creating it but through efforts at Continuum and Numfocus. So much of this distrust appears based on what Continuum might do rather than what the actual record is. I agree with Travis that virtually everyone has their own interests. I don?t think non-profit institutions (academia or otherwise) are any more virtuous than for-profit companies when it comes to possible conflicts of interest. As long as everyone understands the interests involved, that shouldn?t bar anyone from participating in governance. If anyone deserves a special seat at the table it is Travis. I?m completely convinced he has the community?s greater interests at heart (not to say that he always understand all the interests and may need input to help in that). Perry From sturla.molden at gmail.com Tue Sep 22 08:51:38 2015 From: sturla.molden at gmail.com (Sturla Molden) Date: Tue, 22 Sep 2015 14:51:38 +0200 Subject: [Numpy-discussion] Governance model request In-Reply-To: References: Message-ID: On 22/09/15 14:31, Perry Greenfield wrote: > I?ve also stayed out of this until now. I?m surprised and disheartened at the amount of suspicion and distrust directed towards Travis. I have no idea where this distrust comes from. Nobody has invested so much of their time in NumPy. Without Travis there would not even be a NumPy. > So much of this distrust appears based on what Continuum might do rather than what the actual record is. Being CEO of Continuum should not disqualify him in any way. > I agree with Travis that virtually everyone has their own interests. Even Guido and Linus have employers. Should we distrust Guido as Python BDFL because some day Dropbox will be evil? It is just nonsense. Sturla From solipsis at pitrou.net Tue Sep 22 09:10:13 2015 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 22 Sep 2015 15:10:13 +0200 Subject: [Numpy-discussion] 1.10.0rc1 coming tomorrow, 22 Sept. References: Message-ID: <20150922151013.649b8a2b@fsol> On Tue, 22 Sep 2015 04:43:18 -0500 Travis Oliphant wrote: > Absolutely it would be good if others can test. All I was suggesting is > that we do run a pretty decent set of tests upon build and that would be > helpful. > > If the numpy build recipes are not available, it is only because they have > not been updated to use conda-build yet. If somebody wants to volunteer to > convert all of our internal recipes to conda-build recipes so they could be > open source --- we would welcome the help. Note there's a skeleton recipe for numpy here: https://github.com/conda/conda-recipes/tree/master/numpy-openblas If there's interest I could try to come up with a slightly better one, although I can only promise to make it work on Linux. Regards Antoine. From solipsis at pitrou.net Tue Sep 22 10:57:55 2015 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 22 Sep 2015 16:57:55 +0200 Subject: [Numpy-discussion] draft NEP for breaking ufunc ABI in a controlled way References: Message-ID: <20150922165755.492660b8@fsol> Hi, This e-mail is an attempt at proposing an API to solve Numba's needs. Attribute access ---------------- int PyUFunc_Nin(PyUFuncObject *) Replaces ufunc->nin. int PyUFunc_Nout(PyUFuncObject *) Replaces ufunc->nout. int PyUFunc_Nargs(PyUFuncObject *) Replaces ufunc->nargs. PyObject *PyUFunc_Name(PyUFuncObject *) Replaces ufunc->name, returns a unicode object. (alternative: return a const char *) For introspection, the following would be nice too: int PyUFunc_Identity(PyFuncObject *) Replaces ufunc->identity. const char *PyUFunc_Signature(PyUFuncObject *, int i) Gives a pointer to the types of the i'th signature. (equivalent today to &ufunc->ntypes[i * ufunc->nargs]) Lifetime control ---------------- PyObject *PyUFunc_SetObject(PyUFuncObject *, PyObject *) Sets the ufunc's "object" to the given object. The object has no special semantics except that it is DECREF'ed when the ufunc is deallocated (this is today's ufunc->obj). The DECREF should happen only after the ufunc has accessed any internal resources (since the DECREF could deallocate some of those resources). PyObject *PyUFunc_GetObject(PyUFuncObject *) Return the ufunc's current "object". Loop registration ----------------- int PyUFunc_RegisterLoopForSignature( PyUFuncObject* ufunc, PyUFuncGenericFunction function, int *arg_types, void *data, PyObject *obj) Register a loop implementation for the given arg_types (built-in types, presumably). This either appends the loop to the types and functions array (reallocating it if necessary), or replaces an existing one with the same signature. A copy of arg_types is done, such that the caller does not have to manage its lifetime. The optional "PyObject *obj" is an object which gets DECREF'ed when the loop is relinquished (for example when the ufunc is destroyed, or when the loop gets replaced with another by calling this function again). I cannot say I'm 100% sure this is sufficient, but this seems it should cover our current needs. Note this is a minimal proposal. For example, Numpy could instead decide to pass and return all argument types as PyArray_Descr pointers rather than raw integers, and that would probably work for us too. Regards Antoine. From m.h.vankerkwijk at gmail.com Tue Sep 22 12:32:24 2015 From: m.h.vankerkwijk at gmail.com (Marten van Kerkwijk) Date: Tue, 22 Sep 2015 12:32:24 -0400 Subject: [Numpy-discussion] Governance model request In-Reply-To: References: Message-ID: Hi All, I've been reading this thread with amazement and a bit of worry. It seems Nathaniel's proposal is clearly an improvement, even if it is not perfect. But it is in the end for a project where, at least as seen from the outside, the main challenge is not in governance, but rather in having only a small group of people who understand the code well enough that they are able to make and judge modifications. As such, this discussion doesn't seem worthy of the effort, and even less of needless heat and irritation, of the type that seems unlikely would have arisen if this conversation had been in person instead of per e-mail. Might it be an idea to accept the proposal provisionally, returning to it a year from now with practical experience? This certainly has the benefit of allowing to switch focus to the more pressing and fortunately also more interesting work to be done on interfacing numpy/ndarray nicely with other classes (i.e., __numpy_ufunc__ and/or similar, and the dtype generalisations, which have me quite intrigued -- either might be very interesting for the Quantity class in astropy, as well as for a work-in-progress Variable class [which propagates uncertainties including covariances]). ? All the best, Marten -- Prof. M. H. van Kerkwijk Dept. of Astronomy & Astroph., 50 St George St., Toronto, ON, M5S 3H4, Canada McLennan Labs, room 1203B, tel: +1(416)9467288, fax: +1(416)9467287 mhvk at astro.utoronto.ca, http://www.astro.utoronto.ca/~mhvk -------------- next part -------------- An HTML attachment was scrubbed... URL: From rmay31 at gmail.com Tue Sep 22 13:45:15 2015 From: rmay31 at gmail.com (Ryan May) Date: Tue, 22 Sep 2015 11:45:15 -0600 Subject: [Numpy-discussion] Governance model request In-Reply-To: References: Message-ID: This has to be one of the most bizarre threads I've ever read in my life. Somehow companies are lurking around like the boogeyman and academics are completely free of ulterior motives and conflicts of interest? This is just asinine--we're all people and have various motivations. (Having just gotten out of my university after 15 years, the idea that academics are somehow immune to ulterior motives and conflicts of interest makes me laugh hysterically.) The sad part is that this worry completely unnecessary. This is an open source project, not international politics, and the end goal is to produce software. Therefore, 99.9% of the time motives (ulterior, profit, or otherwise) are completely orthogonal to the question of: IS IT A GOOD TECHNICAL IDEA? It's really that simple: does the proposed change move the project in a direction that we want to go? Now, for the 0.01% of the time, where nobody can agree on that answer, or the question is non-technical, and there is concern about the motives of members of the "council" (or whatnot), it's again simple: RECUSAL. It's a simple concept that I learned in the godawful ethics class NSF forced grad students to take: if you have a conflict of interest, you don't vote. It's how the grownups from the Supreme Court to the college football playoff deal with the fact that people WILL have conflicts; potential conflicts don't disbar qualified individuals from being included in the group, just from weighing in when their decisions can be clouded. So how about we stop making up reasons to discourage participation by (over-)qualified individuals, and actually take advantage of the fact that people actually want to move numpy forward? Ryan -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefanv at berkeley.edu Tue Sep 22 14:20:50 2015 From: stefanv at berkeley.edu (Stefan van der Walt) Date: Tue, 22 Sep 2015 11:20:50 -0700 Subject: [Numpy-discussion] Governance model request In-Reply-To: References: <8941C0D2-17EC-420D-A8F6-CA018144811C@continuum.io> <87mvwft2ar.fsf@berkeley.edu> <370C591F-9711-43E6-BE12-8C3E96946803@continuum.io> Message-ID: <87bncutiwt.fsf@berkeley.edu> Hi Travis On 2015-09-22 03:44:12, Travis Oliphant wrote: > I'm actually offended that so many at BIDS seem eager to crucify my > intentions when I've done nothing but give away my time, my energy, my > resources, and my sleep to NumPy for many, many years. I guess if your > intent is to drive me away, then you are succeeding. I guess we've gone off the rails pretty far at this point, so let me at least take a step back, and make sure that you know that: - I have never doubted that your intensions for NumPy are anything but good (I know they are!), - I *want* the community to be a welcoming place for companies to contribute (otherwise, I guess I'd not be such a fervent supporter of the scientific eco-system using the BSD license), and - I love your enthusiasm for the project. After all, that is a big part of what inspired me to become involved in the first place. My goal is not to spread uncertainty, fear nor doubt?if that was the perception left, I apologize. I'll re-iterate that I wanted to highlight a concern about the interactions of a (somewhat weakly cohesive) community and strong, driven personalities such as yourself backed by a formidable amount of development power. No matter how good your intensions are, there are risks involved in this kind of interaction, and if we fail to even *admit* that, we are in trouble. Lest the above be read in a negative light again, let me state it up-front: *I don't think you will hijack the project, use it for your own gain, or attempt to do anything you don't believe to be in the best interest of NumPy.* What I'm saying is that we absolutely need to move forward in a way that brings everyone along, and makes everyone rest assured that their voice will be heard. Also, please know that I have not discussed these matters with Nathaniel behind the scenes, other than an informal hour-long discussion about his original governance proposal. There is no BIDS conspiracy or attempts at crucifixion. After all, you were an invited guest speaker at an event I organized this weekend, since I value your opinion and insights. Either way, let me again apologize if my suggested lack of insight hurt people's feelings. I can only hope that, in educating me, we all learn a few lessons. St?fan From ben.v.root at gmail.com Tue Sep 22 14:48:14 2015 From: ben.v.root at gmail.com (Benjamin Root) Date: Tue, 22 Sep 2015 14:48:14 -0400 Subject: [Numpy-discussion] Governance model request In-Reply-To: <87bncutiwt.fsf@berkeley.edu> References: <8941C0D2-17EC-420D-A8F6-CA018144811C@continuum.io> <87mvwft2ar.fsf@berkeley.edu> <370C591F-9711-43E6-BE12-8C3E96946803@continuum.io> <87bncutiwt.fsf@berkeley.edu> Message-ID: To expand on Ryan's point a bit about recusal... this is why we have a general policy against self-merging and why peer review is so valuable. A ban on self-merging is much like recusal, and I think it is a fantastic policy. As for a BDFL, I used to like that idea having seen it work well for Linux and Python, but I have found it at odds with the policy of recusal and no self-merging. That said, as I am sure Thomas Caswell can attest, a non-self-merging policy can result in a lot of ideas getting stalled, waiting for review that may or may not happen. I don't know what the solution is, but I am sympathetic to those who are apprehensive about a BDFL -- regardless of who is in that role. Ben Root On Tue, Sep 22, 2015 at 2:20 PM, Stefan van der Walt wrote: > Hi Travis > > On 2015-09-22 03:44:12, Travis Oliphant wrote: > > I'm actually offended that so many at BIDS seem eager to crucify my > > intentions when I've done nothing but give away my time, my energy, my > > resources, and my sleep to NumPy for many, many years. I guess if your > > intent is to drive me away, then you are succeeding. > > I guess we've gone off the rails pretty far at this point, so let me at > least take a step back, and make sure that you know that: > > - I have never doubted that your intensions for NumPy are anything but > good (I know they are!), > - I *want* the community to be a welcoming place for companies to > contribute (otherwise, I guess I'd not be such a fervent supporter of > the scientific eco-system using the BSD license), and > - I love your enthusiasm for the project. After all, that is a big part > of what inspired me to become involved in the first place. > > My goal is not to spread uncertainty, fear nor doubt?if that was the > perception left, I apologize. > > I'll re-iterate that I wanted to highlight a concern about the > interactions of a (somewhat weakly cohesive) community and strong, > driven personalities such as yourself backed by a formidable amount of > development power. No matter how good your intensions are, there are > risks involved in this kind of interaction, and if we fail to even > *admit* that, we are in trouble. > > Lest the above be read in a negative light again, let me state it > up-front: *I don't think you will hijack the project, use it for your > own gain, or attempt to do anything you don't believe to be in the best > interest of NumPy.* What I'm saying is that we absolutely need to move > forward in a way that brings everyone along, and makes everyone rest > assured that their voice will be heard. > > Also, please know that I have not discussed these matters with Nathaniel > behind the scenes, other than an informal hour-long discussion about his > original governance proposal. There is no BIDS conspiracy or attempts > at crucifixion. After all, you were an invited guest speaker at an > event I organized this weekend, since I value your opinion and insights. > > Either way, let me again apologize if my suggested lack of insight hurt > people's feelings. I can only hope that, in educating me, we all learn > a few lessons. > > St?fan > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > https://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From matthew.brett at gmail.com Tue Sep 22 14:48:57 2015 From: matthew.brett at gmail.com (Matthew Brett) Date: Tue, 22 Sep 2015 11:48:57 -0700 Subject: [Numpy-discussion] Governance model request In-Reply-To: <87bncutiwt.fsf@berkeley.edu> References: <8941C0D2-17EC-420D-A8F6-CA018144811C@continuum.io> <87mvwft2ar.fsf@berkeley.edu> <370C591F-9711-43E6-BE12-8C3E96946803@continuum.io> <87bncutiwt.fsf@berkeley.edu> Message-ID: Hi, On Tue, Sep 22, 2015 at 11:20 AM, Stefan van der Walt wrote: > Hi Travis > > On 2015-09-22 03:44:12, Travis Oliphant wrote: >> I'm actually offended that so many at BIDS seem eager to crucify my >> intentions when I've done nothing but give away my time, my energy, my >> resources, and my sleep to NumPy for many, many years. I guess if your >> intent is to drive me away, then you are succeeding. > > I guess we've gone off the rails pretty far at this point, so let me at > least take a step back, and make sure that you know that: > > - I have never doubted that your intensions for NumPy are anything but > good (I know they are!), > - I *want* the community to be a welcoming place for companies to > contribute (otherwise, I guess I'd not be such a fervent supporter of > the scientific eco-system using the BSD license), and > - I love your enthusiasm for the project. After all, that is a big part > of what inspired me to become involved in the first place. > > My goal is not to spread uncertainty, fear nor doubt?if that was the > perception left, I apologize. > > I'll re-iterate that I wanted to highlight a concern about the > interactions of a (somewhat weakly cohesive) community and strong, > driven personalities such as yourself backed by a formidable amount of > development power. No matter how good your intensions are, there are > risks involved in this kind of interaction, and if we fail to even > *admit* that, we are in trouble. > > Lest the above be read in a negative light again, let me state it > up-front: *I don't think you will hijack the project, use it for your > own gain, or attempt to do anything you don't believe to be in the best > interest of NumPy.* What I'm saying is that we absolutely need to move > forward in a way that brings everyone along, and makes everyone rest > assured that their voice will be heard. > > Also, please know that I have not discussed these matters with Nathaniel > behind the scenes, other than an informal hour-long discussion about his > original governance proposal. There is no BIDS conspiracy or attempts > at crucifixion. After all, you were an invited guest speaker at an > event I organized this weekend, since I value your opinion and insights. > > Either way, let me again apologize if my suggested lack of insight hurt > people's feelings. I can only hope that, in educating me, we all learn > a few lessons. I'm also in favor of taking a step back. The point is, that a sensible organization and a sensible leader has to take the possibility of conflict of interest into account. They also have to consider the perception of a conflict of interest. It is the opposite of sensible, to respond to this with 'how dare you" or by asserting that this could never happen or by saying that we shouldn't talk about that in case people get frightened. I point you again to Linus' interview [1]. He is not upset that he has been insulted by the implication of conflict of interest, he soberly accepts that this will always be an issue, with companies in particular, and goes out of his way to address that in an explicit and reasonable way. Cheers, Matthew [1] http://www.bbc.com/news/technology-18419231 From charlesr.harris at gmail.com Tue Sep 22 14:54:07 2015 From: charlesr.harris at gmail.com (Charles R Harris) Date: Tue, 22 Sep 2015 12:54:07 -0600 Subject: [Numpy-discussion] Commit rights for Allan Haldane Message-ID: Hi All, Allan Haldane has been given commit rights. Here's to the new member of the team. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From jaime.frio at gmail.com Tue Sep 22 15:10:05 2015 From: jaime.frio at gmail.com (=?UTF-8?Q?Jaime_Fern=C3=A1ndez_del_R=C3=ADo?=) Date: Tue, 22 Sep 2015 12:10:05 -0700 Subject: [Numpy-discussion] Commit rights for Allan Haldane In-Reply-To: References: Message-ID: Congrats Allan! Jaime On Tue, Sep 22, 2015 at 11:54 AM, Charles R Harris < charlesr.harris at gmail.com> wrote: > Hi All, > > Allan Haldane has been given commit rights. Here's to the new member of > the team. > > Chuck > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > https://mail.scipy.org/mailman/listinfo/numpy-discussion > > -- (\__/) ( O.o) ( > <) Este es Conejo. Copia a Conejo en tu firma y ay?dale en sus planes de dominaci?n mundial. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jehturner at gmail.com Tue Sep 22 16:03:33 2015 From: jehturner at gmail.com (James E.H. Turner) Date: Tue, 22 Sep 2015 17:03:33 -0300 Subject: [Numpy-discussion] Governance model request In-Reply-To: References: Message-ID: <5601B415.9000401@gmail.com> I don't think I've contributed code to NumPy itself, but as someone involved in the scientific python ecosystem for a while, I can't see why people would consider Continuum less of a legitimate participant or community member than individual contributors, especially if the person behind it has had the opportunity to control things previously and instead passed NumPy onto the community. I'd be wary of commercial interests dominating the agenda, but that's different from them having a proportionate (in this case minor) say when they have something to offer. And that's all true *even if* Travis were heavily biased to his own commercial ends, which is not consistent with my understanding of his wider efforts and sacrifices over most of a decade that I've been paying attention. Remember this is free/open source software and if enough people don't like the committee at some point, the project can be forked as an option of last resort. Nothing is set in stone, nor code lost. Just saying (I probably won't reply to any criticism or corrections, to avoid adding peripheral noise/heat to the thread). Cheers, James (from, but not on behalf of, a non-profit research facility). From travis at continuum.io Tue Sep 22 16:53:46 2015 From: travis at continuum.io (Travis Oliphant) Date: Tue, 22 Sep 2015 15:53:46 -0500 Subject: [Numpy-discussion] Commit rights for Allan Haldane In-Reply-To: References: Message-ID: Excellent news! Welcome Allan. -Travis On Tue, Sep 22, 2015 at 1:54 PM, Charles R Harris wrote: > Hi All, > > Allan Haldane has been given commit rights. Here's to the new member of > the team. > > Chuck > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > https://mail.scipy.org/mailman/listinfo/numpy-discussion > > -- *Travis Oliphant* *Co-founder and CEO* @teoliphant 512-222-5440 http://www.continuum.io -------------- next part -------------- An HTML attachment was scrubbed... URL: From allanhaldane at gmail.com Tue Sep 22 17:27:08 2015 From: allanhaldane at gmail.com (Allan Haldane) Date: Tue, 22 Sep 2015 17:27:08 -0400 Subject: [Numpy-discussion] Commit rights for Allan Haldane In-Reply-To: References: Message-ID: <5601C7AC.1020502@gmail.com> Thanks all. I'm very happy to contribute back to a project which has been so useful to me over many years! On 09/22/2015 04:53 PM, Travis Oliphant wrote: > Excellent news! Welcome Allan. > > -Travis > > > On Tue, Sep 22, 2015 at 1:54 PM, Charles R Harris > > wrote: > > Hi All, > > Allan Haldane has been given commit rights. Here's to the new member > of the team. > > Chuck > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > https://mail.scipy.org/mailman/listinfo/numpy-discussion > > > > > -- > * > Travis Oliphant* > /Co-founder and CEO/ > / > / > > @teoliphant > 512-222-5440 > http://www.continuum.io > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > https://mail.scipy.org/mailman/listinfo/numpy-discussion > From charlesr.harris at gmail.com Tue Sep 22 18:43:11 2015 From: charlesr.harris at gmail.com (Charles R Harris) Date: Tue, 22 Sep 2015 16:43:11 -0600 Subject: [Numpy-discussion] draft NEP for breaking ufunc ABI in a controlled way In-Reply-To: References: Message-ID: On Mon, Sep 21, 2015 at 10:23 PM, Nathaniel Smith wrote: > On Mon, Sep 21, 2015 at 7:29 AM, Jaime Fern?ndez del R?o > wrote: > > We have the PyArrayObject vs PyArrayObject_fields definition in > > ndarraytypes.h that is used to enforce access to the members through > inline > > functions rather than directly, which seems to me like the right way to > go: > > don't leave stones unturned, hide everything and provide PyUFunc_NIN, > > PyUFunc_NOUT and friends to handle those too. > > The PyArrayObject vs PyArrayObject_fields distinction is only enabled > if a downstream library explicitly requests it with #define > NPY_NO_DEPRECATED_API, though -- the idea is that the changes in this > NEP would be enabled unconditionally, even for old code. So the reason > nin/nout/nargs are left exposed in this proposal is that there's some > existing code out there that would break (until updated) if we hid > them, and not much benefit to breaking it. > > If we're fine with breaking that code then we could just hide them > unconditionally too. The only code I found in the wild that would be > affected is the "rational" user-defined dtype, which would be > trivially fixable since the only thing it does with ufunc->nargs is a > quick consistency check: > > > https://github.com/numpy/numpy-dtypes/blob/c0175a6b1c5aa89b4520b29487f06d0e200e2a03/npytypes/rational/rational.c#L1140-L1151 > > Also it's not 100% clear right now whether we even want to keep > supporting the old user-defined dtype API that this particular code is > based around. But if this code uses ufunc->nargs then perhaps other > code does too? I'm open to opinions -- I doubt it matters that much > either way. I just want to make sure that we can hide the other stuff > :-). > > When it comes to evolving these APIs in general: one unfortunate thing > about the PyArrayObject changes in 1.7 is that because they were > implemented using *inline* functions (/macros) they haven't affected > One thing we might consider along the way is separating numpy.multiarray and friends into an actual library plus a module. That way the new numpy api would be exposed in the library rather than by importing an array of pointers from the module. the a*B*i exposure at all, even in code that has upgraded to the new > calling conventions. While user code no longer *names* the internal > fields directly, we still have to implement exactly the same fields > and put them in exactly the same place in memory or else break ABI. > And the other unfortunate thing is that we don't really have a > mechanism for saying "okay, we're dropping support for the old way of > doing things in 1.xx" -- in particular the current > NPY_NO_DEPRECATED_API mechanism doesn't give us any way to detect and > error out if someone tries to use an old version of the APIs, so ABI > breaks still mean segfaults. I'm thinking that if/when we figure out > how to implement the "sliding window" API/ABI idea that we talked > about at SciPy, then that will give us a strategy for cleanly > transitioning to a world with a maintainable API+ABI and it becomes > worth sitting down and making up a set of setters/getters for the > attributes that we want to make public in a maintainable way. But > until then our only real options are either hard breaks or nothing, so > unless we want to do a hard break there's not much point talking about > it. > > -n > Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From cournape at gmail.com Tue Sep 22 18:52:01 2015 From: cournape at gmail.com (David Cournapeau) Date: Tue, 22 Sep 2015 15:52:01 -0700 Subject: [Numpy-discussion] draft NEP for breaking ufunc ABI in a controlled way In-Reply-To: References: Message-ID: On Tue, Sep 22, 2015 at 3:43 PM, Charles R Harris wrote: > > > On Mon, Sep 21, 2015 at 10:23 PM, Nathaniel Smith wrote: > >> On Mon, Sep 21, 2015 at 7:29 AM, Jaime Fern?ndez del R?o >> wrote: >> > We have the PyArrayObject vs PyArrayObject_fields definition in >> > ndarraytypes.h that is used to enforce access to the members through >> inline >> > functions rather than directly, which seems to me like the right way >> to go: >> > don't leave stones unturned, hide everything and provide PyUFunc_NIN, >> > PyUFunc_NOUT and friends to handle those too. >> >> The PyArrayObject vs PyArrayObject_fields distinction is only enabled >> if a downstream library explicitly requests it with #define >> NPY_NO_DEPRECATED_API, though -- the idea is that the changes in this >> NEP would be enabled unconditionally, even for old code. So the reason >> nin/nout/nargs are left exposed in this proposal is that there's some >> existing code out there that would break (until updated) if we hid >> them, and not much benefit to breaking it. >> >> If we're fine with breaking that code then we could just hide them >> unconditionally too. The only code I found in the wild that would be >> affected is the "rational" user-defined dtype, which would be >> trivially fixable since the only thing it does with ufunc->nargs is a >> quick consistency check: >> >> >> https://github.com/numpy/numpy-dtypes/blob/c0175a6b1c5aa89b4520b29487f06d0e200e2a03/npytypes/rational/rational.c#L1140-L1151 >> >> Also it's not 100% clear right now whether we even want to keep >> supporting the old user-defined dtype API that this particular code is >> based around. But if this code uses ufunc->nargs then perhaps other >> code does too? I'm open to opinions -- I doubt it matters that much >> either way. I just want to make sure that we can hide the other stuff >> :-). >> >> When it comes to evolving these APIs in general: one unfortunate thing >> about the PyArrayObject changes in 1.7 is that because they were >> implemented using *inline* functions (/macros) they haven't affected >> > > One thing we might consider along the way is separating numpy.multiarray > and friends into an actual library plus a module. That way the new numpy > api would be exposed in the library rather than by importing an array of > pointers from the module. > Agreed. This would help the cythonizing process as well (on which I will try to write more about in a separate thread later). David > > the a*B*i exposure at all, even in code that has upgraded to the new >> calling conventions. While user code no longer *names* the internal >> fields directly, we still have to implement exactly the same fields >> and put them in exactly the same place in memory or else break ABI. >> And the other unfortunate thing is that we don't really have a >> mechanism for saying "okay, we're dropping support for the old way of >> doing things in 1.xx" -- in particular the current >> NPY_NO_DEPRECATED_API mechanism doesn't give us any way to detect and >> error out if someone tries to use an old version of the APIs, so ABI >> breaks still mean segfaults. I'm thinking that if/when we figure out >> how to implement the "sliding window" API/ABI idea that we talked >> about at SciPy, then that will give us a strategy for cleanly >> transitioning to a world with a maintainable API+ABI and it becomes >> worth sitting down and making up a set of setters/getters for the >> attributes that we want to make public in a maintainable way. But >> until then our only real options are either hard breaks or nothing, so >> unless we want to do a hard break there's not much point talking about >> it. >> >> -n >> > > > > Chuck > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > https://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From travis at continuum.io Tue Sep 22 18:54:56 2015 From: travis at continuum.io (Travis Oliphant) Date: Tue, 22 Sep 2015 17:54:56 -0500 Subject: [Numpy-discussion] draft NEP for breaking ufunc ABI in a controlled way In-Reply-To: References: Message-ID: That sounds like a very good idea. I know that one of the original motivations for the odd import mechanism of NumPy was the AIX platform and it's lack of a shared library. I can't imagine that is still actually a problem. A simpler, library-based mechanism would be a welcome change from my perspective. -Travis On Tue, Sep 22, 2015 at 5:43 PM, Charles R Harris wrote: > > > On Mon, Sep 21, 2015 at 10:23 PM, Nathaniel Smith wrote: > >> On Mon, Sep 21, 2015 at 7:29 AM, Jaime Fern?ndez del R?o >> wrote: >> > We have the PyArrayObject vs PyArrayObject_fields definition in >> > ndarraytypes.h that is used to enforce access to the members through >> inline >> > functions rather than directly, which seems to me like the right way >> to go: >> > don't leave stones unturned, hide everything and provide PyUFunc_NIN, >> > PyUFunc_NOUT and friends to handle those too. >> >> The PyArrayObject vs PyArrayObject_fields distinction is only enabled >> if a downstream library explicitly requests it with #define >> NPY_NO_DEPRECATED_API, though -- the idea is that the changes in this >> NEP would be enabled unconditionally, even for old code. So the reason >> nin/nout/nargs are left exposed in this proposal is that there's some >> existing code out there that would break (until updated) if we hid >> them, and not much benefit to breaking it. >> >> If we're fine with breaking that code then we could just hide them >> unconditionally too. The only code I found in the wild that would be >> affected is the "rational" user-defined dtype, which would be >> trivially fixable since the only thing it does with ufunc->nargs is a >> quick consistency check: >> >> >> https://github.com/numpy/numpy-dtypes/blob/c0175a6b1c5aa89b4520b29487f06d0e200e2a03/npytypes/rational/rational.c#L1140-L1151 >> >> Also it's not 100% clear right now whether we even want to keep >> supporting the old user-defined dtype API that this particular code is >> based around. But if this code uses ufunc->nargs then perhaps other >> code does too? I'm open to opinions -- I doubt it matters that much >> either way. I just want to make sure that we can hide the other stuff >> :-). >> >> When it comes to evolving these APIs in general: one unfortunate thing >> about the PyArrayObject changes in 1.7 is that because they were >> implemented using *inline* functions (/macros) they haven't affected >> > > One thing we might consider along the way is separating numpy.multiarray > and friends into an actual library plus a module. That way the new numpy > api would be exposed in the library rather than by importing an array of > pointers from the module. > > the a*B*i exposure at all, even in code that has upgraded to the new >> calling conventions. While user code no longer *names* the internal >> fields directly, we still have to implement exactly the same fields >> and put them in exactly the same place in memory or else break ABI. >> And the other unfortunate thing is that we don't really have a >> mechanism for saying "okay, we're dropping support for the old way of >> doing things in 1.xx" -- in particular the current >> NPY_NO_DEPRECATED_API mechanism doesn't give us any way to detect and >> error out if someone tries to use an old version of the APIs, so ABI >> breaks still mean segfaults. I'm thinking that if/when we figure out >> how to implement the "sliding window" API/ABI idea that we talked >> about at SciPy, then that will give us a strategy for cleanly >> transitioning to a world with a maintainable API+ABI and it becomes >> worth sitting down and making up a set of setters/getters for the >> attributes that we want to make public in a maintainable way. But >> until then our only real options are either hard breaks or nothing, so >> unless we want to do a hard break there's not much point talking about >> it. >> >> -n >> > > > > Chuck > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > https://mail.scipy.org/mailman/listinfo/numpy-discussion > > -- *Travis Oliphant* *Co-founder and CEO* @teoliphant 512-222-5440 http://www.continuum.io -------------- next part -------------- An HTML attachment was scrubbed... URL: From travis at continuum.io Tue Sep 22 19:12:33 2015 From: travis at continuum.io (Travis Oliphant) Date: Tue, 22 Sep 2015 18:12:33 -0500 Subject: [Numpy-discussion] Governance model request In-Reply-To: <87bncutiwt.fsf@berkeley.edu> References: <8941C0D2-17EC-420D-A8F6-CA018144811C@continuum.io> <87mvwft2ar.fsf@berkeley.edu> <370C591F-9711-43E6-BE12-8C3E96946803@continuum.io> <87bncutiwt.fsf@berkeley.edu> Message-ID: On Tue, Sep 22, 2015 at 1:20 PM, Stefan van der Walt wrote: > > > I guess we've gone off the rails pretty far at this point, so let me at > least take a step back, and make sure that you know that: > > - I have never doubted that your intensions for NumPy are anything but > good (I know they are!), > - I *want* the community to be a welcoming place for companies to > contribute (otherwise, I guess I'd not be such a fervent supporter of > the scientific eco-system using the BSD license), and > - I love your enthusiasm for the project. After all, that is a big part > of what inspired me to become involved in the first place. > > My goal is not to spread uncertainty, fear nor doubt?if that was the > perception left, I apologize. > > I'll re-iterate that I wanted to highlight a concern about the > interactions of a (somewhat weakly cohesive) community and strong, > driven personalities such as yourself backed by a formidable amount of > development power. No matter how good your intensions are, there are > risks involved in this kind of interaction, and if we fail to even > *admit* that, we are in trouble. > > Lest the above be read in a negative light again, let me state it > up-front: *I don't think you will hijack the project, use it for your > own gain, or attempt to do anything you don't believe to be in the best > interest of NumPy.* What I'm saying is that we absolutely need to move > forward in a way that brings everyone along, and makes everyone rest > assured that their voice will be heard. > > Thank you for the clarification. I'm sorry that I started to question your intentions. I agree that everyone should rest assured that their voice will be heard. I have been and continue to be a staunch advocate for the voices that are not even on this mailing list. > Also, please know that I have not discussed these matters with Nathaniel > behind the scenes, other than an informal hour-long discussion about his > original governance proposal. There is no BIDS conspiracy or attempts > at crucifixion. After all, you were an invited guest speaker at an > event I organized this weekend, since I value your opinion and insights. > > Thank you. I'm sorry for implying otherwise. That was wrong of me. I know we are just trying to bring all the voices to the table. Best, -Travis -------------- next part -------------- An HTML attachment was scrubbed... URL: From travis at continuum.io Tue Sep 22 19:13:57 2015 From: travis at continuum.io (Travis Oliphant) Date: Tue, 22 Sep 2015 18:13:57 -0500 Subject: [Numpy-discussion] Governance model request In-Reply-To: References: <8941C0D2-17EC-420D-A8F6-CA018144811C@continuum.io> <87mvwft2ar.fsf@berkeley.edu> <370C591F-9711-43E6-BE12-8C3E96946803@continuum.io> <87bncutiwt.fsf@berkeley.edu> Message-ID: I completely agree. I don't think self-merging is a good idea. I had gotten used to self merging to SciPy and NumPy until roughly 2008 or 2009 when I was finally broken of that habit after stepping on a few people's toes and learning better habits. On Tue, Sep 22, 2015 at 1:48 PM, Benjamin Root wrote: > To expand on Ryan's point a bit about recusal... this is why we have a > general policy against self-merging and why peer review is so valuable. A > ban on self-merging is much like recusal, and I think it is a fantastic > policy. > > As for a BDFL, I used to like that idea having seen it work well for Linux > and Python, but I have found it at odds with the policy of recusal and no > self-merging. That said, as I am sure Thomas Caswell can attest, a > non-self-merging policy can result in a lot of ideas getting stalled, > waiting for review that may or may not happen. I don't know what the > solution is, but I am sympathetic to those who are apprehensive about a > BDFL -- regardless of who is in that role. > > Ben Root > > > On Tue, Sep 22, 2015 at 2:20 PM, Stefan van der Walt > wrote: > >> Hi Travis >> >> On 2015-09-22 03:44:12, Travis Oliphant wrote: >> > I'm actually offended that so many at BIDS seem eager to crucify my >> > intentions when I've done nothing but give away my time, my energy, my >> > resources, and my sleep to NumPy for many, many years. I guess if >> your >> > intent is to drive me away, then you are succeeding. >> >> I guess we've gone off the rails pretty far at this point, so let me at >> least take a step back, and make sure that you know that: >> >> - I have never doubted that your intensions for NumPy are anything but >> good (I know they are!), >> - I *want* the community to be a welcoming place for companies to >> contribute (otherwise, I guess I'd not be such a fervent supporter of >> the scientific eco-system using the BSD license), and >> - I love your enthusiasm for the project. After all, that is a big part >> of what inspired me to become involved in the first place. >> >> My goal is not to spread uncertainty, fear nor doubt?if that was the >> perception left, I apologize. >> >> I'll re-iterate that I wanted to highlight a concern about the >> interactions of a (somewhat weakly cohesive) community and strong, >> driven personalities such as yourself backed by a formidable amount of >> development power. No matter how good your intensions are, there are >> risks involved in this kind of interaction, and if we fail to even >> *admit* that, we are in trouble. >> >> Lest the above be read in a negative light again, let me state it >> up-front: *I don't think you will hijack the project, use it for your >> own gain, or attempt to do anything you don't believe to be in the best >> interest of NumPy.* What I'm saying is that we absolutely need to move >> forward in a way that brings everyone along, and makes everyone rest >> assured that their voice will be heard. >> >> Also, please know that I have not discussed these matters with Nathaniel >> behind the scenes, other than an informal hour-long discussion about his >> original governance proposal. There is no BIDS conspiracy or attempts >> at crucifixion. After all, you were an invited guest speaker at an >> event I organized this weekend, since I value your opinion and insights. >> >> Either way, let me again apologize if my suggested lack of insight hurt >> people's feelings. I can only hope that, in educating me, we all learn >> a few lessons. >> >> St?fan >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> https://mail.scipy.org/mailman/listinfo/numpy-discussion >> > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > https://mail.scipy.org/mailman/listinfo/numpy-discussion > > -- *Travis Oliphant* *Co-founder and CEO* @teoliphant 512-222-5440 http://www.continuum.io -------------- next part -------------- An HTML attachment was scrubbed... URL: From travis at continuum.io Tue Sep 22 19:16:39 2015 From: travis at continuum.io (Travis Oliphant) Date: Tue, 22 Sep 2015 18:16:39 -0500 Subject: [Numpy-discussion] Governance model request In-Reply-To: References: <8941C0D2-17EC-420D-A8F6-CA018144811C@continuum.io> <87mvwft2ar.fsf@berkeley.edu> <370C591F-9711-43E6-BE12-8C3E96946803@continuum.io> <87bncutiwt.fsf@berkeley.edu> Message-ID: I am not upset nor was I ever upset about discussing the possibility of conflict of interest. Of course it can be discussed --- but it should be discussed directly about specific things --- and as others have said it is generally easily handled when it actually could arise. The key is to understand affiliations. We should not do things in the community that actually encourage people to hide their affiliations for fear of backlash or bias. I was annoyed at the insinuation that conflict of interest is a company-only problem that academics are somehow immune to. I was upset about accusations and mis-interpretations of my activities and those of my colleagues in behalf of the community. On Tue, Sep 22, 2015 at 1:48 PM, Matthew Brett wrote: > Hi, > > On Tue, Sep 22, 2015 at 11:20 AM, Stefan van der Walt > wrote: > > Hi Travis > > > > On 2015-09-22 03:44:12, Travis Oliphant wrote: > >> I'm actually offended that so many at BIDS seem eager to crucify my > >> intentions when I've done nothing but give away my time, my energy, my > >> resources, and my sleep to NumPy for many, many years. I guess if > your > >> intent is to drive me away, then you are succeeding. > > > > I guess we've gone off the rails pretty far at this point, so let me at > > least take a step back, and make sure that you know that: > > > > - I have never doubted that your intensions for NumPy are anything but > > good (I know they are!), > > - I *want* the community to be a welcoming place for companies to > > contribute (otherwise, I guess I'd not be such a fervent supporter of > > the scientific eco-system using the BSD license), and > > - I love your enthusiasm for the project. After all, that is a big part > > of what inspired me to become involved in the first place. > > > > My goal is not to spread uncertainty, fear nor doubt?if that was the > > perception left, I apologize. > > > > I'll re-iterate that I wanted to highlight a concern about the > > interactions of a (somewhat weakly cohesive) community and strong, > > driven personalities such as yourself backed by a formidable amount of > > development power. No matter how good your intensions are, there are > > risks involved in this kind of interaction, and if we fail to even > > *admit* that, we are in trouble. > > > > Lest the above be read in a negative light again, let me state it > > up-front: *I don't think you will hijack the project, use it for your > > own gain, or attempt to do anything you don't believe to be in the best > > interest of NumPy.* What I'm saying is that we absolutely need to move > > forward in a way that brings everyone along, and makes everyone rest > > assured that their voice will be heard. > > > > Also, please know that I have not discussed these matters with Nathaniel > > behind the scenes, other than an informal hour-long discussion about his > > original governance proposal. There is no BIDS conspiracy or attempts > > at crucifixion. After all, you were an invited guest speaker at an > > event I organized this weekend, since I value your opinion and insights. > > > > Either way, let me again apologize if my suggested lack of insight hurt > > people's feelings. I can only hope that, in educating me, we all learn > > a few lessons. > > I'm also in favor of taking a step back. > > The point is, that a sensible organization and a sensible leader has > to take the possibility of conflict of interest into account. They > also have to consider the perception of a conflict of interest. > > It is the opposite of sensible, to respond to this with 'how dare you" > or by asserting that this could never happen or by saying that we > shouldn't talk about that in case people get frightened. I point you > again to Linus' interview [1]. He is not upset that he has been > insulted by the implication of conflict of interest, he soberly > accepts that this will always be an issue, with companies in > particular, and goes out of his way to address that in an explicit and > reasonable way. > > Cheers, > > Matthew > > [1] http://www.bbc.com/news/technology-18419231 > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > https://mail.scipy.org/mailman/listinfo/numpy-discussion > -- *Travis Oliphant* *Co-founder and CEO* @teoliphant 512-222-5440 http://www.continuum.io -------------- next part -------------- An HTML attachment was scrubbed... URL: From bryanv at continuum.io Tue Sep 22 22:55:52 2015 From: bryanv at continuum.io (Bryan Van de Ven) Date: Tue, 22 Sep 2015 21:55:52 -0500 Subject: [Numpy-discussion] Governance model request In-Reply-To: References: <8941C0D2-17EC-420D-A8F6-CA018144811C@continuum.io> <87mvwft2ar.fsf@berkeley.edu> <370C591F-9711-43E6-BE12-8C3E96946803@continuum.io> <87bncutiwt.fsf@berkeley.edu> Message-ID: > On Sep 22, 2015, at 1:48 PM, Matthew Brett wrote: > > The point is, that a sensible organization and a sensible leader has > to take the possibility of conflict of interest into account. They > also have to consider the perception of a conflict of interest. Of course, and the policies to deal with conflicts have deal with the possibility that *anyone* at all might have conflict. But that was not your suggestion. Your suggestion was to make one individual be subject to additional scrutiny that no one else would be subject to. Please explain why should one person be singled out for a special "six-month waiting period" when the exact same possibility for conflict exists with anyone who is ever on the committee? > It is the opposite of sensible, to respond to this with 'how dare you" > or by asserting that this could never happen or by saying that we > shouldn't talk about that in case people get frightened. I point you Red herring. The objection (as many people have now voiced) is the double standard you proposed. > again to Linus' interview [1]. He is not upset that he has been > insulted by the implication of conflict of interest, he soberly > accepts that this will always be an issue, with companies in > particular, and goes out of his way to address that in an explicit and > reasonable way. Your selective quotation is impressive. Also in that interview, Linus points out that his employment contract is probably "unique in the entire world". Which means in practical terms that the details of what he has does are fairly well irrelevant to any other situation. So what is the point in bringing it up, at all, except to try and diminish someone else by comparison? (I'm done.) Bryan From charlesr.harris at gmail.com Tue Sep 22 22:58:58 2015 From: charlesr.harris at gmail.com (Charles R Harris) Date: Tue, 22 Sep 2015 20:58:58 -0600 Subject: [Numpy-discussion] facebook, twitter, and g+ Message-ID: Hi All, Just posting to elicit thoughts about scipy.org having a presence in social media for announcements. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Tue Sep 22 23:12:52 2015 From: charlesr.harris at gmail.com (Charles R Harris) Date: Tue, 22 Sep 2015 21:12:52 -0600 Subject: [Numpy-discussion] ANN: Numpy 1.10.0rc1 released. Message-ID: Hi all, I'm pleased to announce the availability of Numpy 1.10.0rc1. Sources and 32 bit binary packages for Windows may be found at Sourceforge . Please test this out, as I would like to move to the final release as rapidly as possible and the lack of error reports from the beta has left me nervous. It's been quiet, too quiet. In the movies, we would all die in the next five minutes. Cheers Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From bryanv at continuum.io Tue Sep 22 23:14:03 2015 From: bryanv at continuum.io (Bryan Van de Ven) Date: Tue, 22 Sep 2015 22:14:03 -0500 Subject: [Numpy-discussion] facebook, twitter, and g+ In-Reply-To: References: Message-ID: <2FE6404F-D4E3-490F-9EC3-ED35434D7451@continuum.io> > On Sep 22, 2015, at 9:58 PM, Charles R Harris wrote: > > Hi All, > > Just posting to elicit thoughts about scipy.org having a presence in social media for announcements. Of the ones listed in the subject, I would suggest Twitter is the most valuable. It has been great for release and other announcements. I find Facebook and G+ a little odd for software projects personally, though I guess they could be useful if you wanted to have somewhat longer "posts". But of course that's also yet more for someone to have maintain. One additional consideration is that some people will inevitably attempt to elicit technical support over these channels. I generally have to steer people to the mailing list immediately. Bryan From jaime.frio at gmail.com Tue Sep 22 23:18:46 2015 From: jaime.frio at gmail.com (=?UTF-8?Q?Jaime_Fern=C3=A1ndez_del_R=C3=ADo?=) Date: Tue, 22 Sep 2015 20:18:46 -0700 Subject: [Numpy-discussion] facebook, twitter, and g+ In-Reply-To: <2FE6404F-D4E3-490F-9EC3-ED35434D7451@continuum.io> References: <2FE6404F-D4E3-490F-9EC3-ED35434D7451@continuum.io> Message-ID: +1 for twitter +0 for the others On Tue, Sep 22, 2015 at 8:14 PM, Bryan Van de Ven wrote: > > > On Sep 22, 2015, at 9:58 PM, Charles R Harris > wrote: > > > > Hi All, > > > > Just posting to elicit thoughts about scipy.org having a presence in > social media for announcements. > > Of the ones listed in the subject, I would suggest Twitter is the most > valuable. It has been great for release and other announcements. I find > Facebook and G+ a little odd for software projects personally, though I > guess they could be useful if you wanted to have somewhat longer "posts". > But of course that's also yet more for someone to have maintain. > > One additional consideration is that some people will inevitably attempt > to elicit technical support over these channels. I generally have to steer > people to the mailing list immediately. > > Bryan > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > https://mail.scipy.org/mailman/listinfo/numpy-discussion > -- (\__/) ( O.o) ( > <) Este es Conejo. Copia a Conejo en tu firma y ay?dale en sus planes de dominaci?n mundial. -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Wed Sep 23 00:07:36 2015 From: njs at pobox.com (Nathaniel Smith) Date: Tue, 22 Sep 2015 21:07:36 -0700 Subject: [Numpy-discussion] ANN: Numpy 1.10.0rc1 released. In-Reply-To: References: Message-ID: On Tue, Sep 22, 2015 at 8:12 PM, Charles R Harris wrote: > Hi all, > > I'm pleased to announce the availability of Numpy 1.10.0rc1. Sources and 32 > bit binary packages for Windows may be found at Sourceforge. Please test > this out, as I would like to move to the final release as rapidly as > possible and the lack of error reports from the beta has left me nervous. > It's been quiet, too quiet. In the movies, we would all die in the next five > minutes. The release was coming from INSIDE THE HOUSE! Thanks Chuck! -n -- Nathaniel J. Smith -- http://vorpus.org From njs at pobox.com Wed Sep 23 00:19:27 2015 From: njs at pobox.com (Nathaniel Smith) Date: Tue, 22 Sep 2015 21:19:27 -0700 Subject: [Numpy-discussion] draft NEP for breaking ufunc ABI in a controlled way In-Reply-To: References: Message-ID: On Tue, Sep 22, 2015 at 3:43 PM, Charles R Harris wrote: > > > On Mon, Sep 21, 2015 at 10:23 PM, Nathaniel Smith wrote: [...] >> When it comes to evolving these APIs in general: one unfortunate thing >> about the PyArrayObject changes in 1.7 is that because they were >> implemented using *inline* functions (/macros) they haven't affected > > > One thing we might consider along the way is separating numpy.multiarray and > friends into an actual library plus a module. That way the new numpy api > would be exposed in the library rather than by importing an array of > pointers from the module. > I'm not sure whether we'll be able to pull this off at the technical level? Partly because anything involving cross-platform linker behavior is a recipe for unpleasantness, but mostly because doing sliding-window API/ABI tracking requires that we have some way to check which of multiple APIs a given third-party package is requesting, and provide a nice error if the one they want isn't available, and I'm not certain how to accomplish that via the regular linker. But sure, something to look into when we reach that point :-) -n -- Nathaniel J. Smith -- http://vorpus.org From charlesr.harris at gmail.com Wed Sep 23 00:51:48 2015 From: charlesr.harris at gmail.com (Charles R Harris) Date: Tue, 22 Sep 2015 22:51:48 -0600 Subject: [Numpy-discussion] draft NEP for breaking ufunc ABI in a controlled way In-Reply-To: References: Message-ID: On Tue, Sep 22, 2015 at 10:19 PM, Nathaniel Smith wrote: > On Tue, Sep 22, 2015 at 3:43 PM, Charles R Harris > wrote: > > > > > > On Mon, Sep 21, 2015 at 10:23 PM, Nathaniel Smith wrote: > [...] > >> When it comes to evolving these APIs in general: one unfortunate thing > >> about the PyArrayObject changes in 1.7 is that because they were > >> implemented using *inline* functions (/macros) they haven't affected > > > > > > One thing we might consider along the way is separating numpy.multiarray > and > > friends into an actual library plus a module. That way the new numpy api > > would be exposed in the library rather than by importing an array of > > pointers from the module. > > > > I'm not sure whether we'll be able to pull this off at the technical > level? Partly because anything involving cross-platform linker > behavior is a recipe for unpleasantness, but mostly because doing > sliding-window API/ABI tracking requires that we have some way to > check which of multiple APIs a given third-party package is > requesting, and provide a nice error if the one they want isn't > available, and I'm not certain how to accomplish that via the regular > linker. But sure, something to look into when we reach that point :-) > I'd recommend the Henry Ford approach, "Any customer can have a car painted any color that he wants so long as it is *black*". Essentially, an ABI break split between a backward compatible layer on top, and a bare metal layer below, with the latter recommended. We would still need to solve the 'hide the structure" problem, but that is probably unavoidable whatever approach we take. In any case, it might be worthwhile making a list of the functions such a library would expose. I'm not sure how big a problem linking would be, likely Windows would continue to be the largest source of problems if we go the shared library route. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Wed Sep 23 01:01:44 2015 From: charlesr.harris at gmail.com (Charles R Harris) Date: Tue, 22 Sep 2015 23:01:44 -0600 Subject: [Numpy-discussion] Governance model request In-Reply-To: References: <8941C0D2-17EC-420D-A8F6-CA018144811C@continuum.io> <87mvwft2ar.fsf@berkeley.edu> <370C591F-9711-43E6-BE12-8C3E96946803@continuum.io> <87bncutiwt.fsf@berkeley.edu> Message-ID: On Tue, Sep 22, 2015 at 8:55 PM, Bryan Van de Ven wrote: > > > On Sep 22, 2015, at 1:48 PM, Matthew Brett > wrote: > > > > The point is, that a sensible organization and a sensible leader has > > to take the possibility of conflict of interest into account. They > > also have to consider the perception of a conflict of interest. > > Of course, and the policies to deal with conflicts have deal with the > possibility that *anyone* at all might have conflict. But that was not your > suggestion. Your suggestion was to make one individual be subject to > additional scrutiny that no one else would be subject to. Please explain > why should one person be singled out for a special "six-month waiting > period" when the exact same possibility for conflict exists with anyone who > is ever on the committee? > > > It is the opposite of sensible, to respond to this with 'how dare you" > > or by asserting that this could never happen or by saying that we > > shouldn't talk about that in case people get frightened. I point you > > Red herring. The objection (as many people have now voiced) is the double > standard you proposed. > > > again to Linus' interview [1]. He is not upset that he has been > > insulted by the implication of conflict of interest, he soberly > > accepts that this will always be an issue, with companies in > > particular, and goes out of his way to address that in an explicit and > > reasonable way. > > Your selective quotation is impressive. Also in that interview, Linus > points out that his employment contract is probably "unique in the entire > world". Which means in practical terms that the details of what he has does > are fairly well irrelevant to any other situation. So what is the point in > bringing it up, at all, except to try and diminish someone else by > comparison? > > (I'm done.) > Andrew Morton would be a prominent counter example in the Linux community. He is employed by Google. Many other Linux developers are employed by Red Hat, Intel, and other companies. At this point Linus relies on those people for decisions on what goes into the kernel; it is much too big for a single person to deal with even in review. I also expect the Linux community would be overjoyed if every company provided drivers for their hardware. Subject to review, of course. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From josef.pktd at gmail.com Wed Sep 23 01:32:40 2015 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Wed, 23 Sep 2015 01:32:40 -0400 Subject: [Numpy-discussion] Governance model request In-Reply-To: References: <8941C0D2-17EC-420D-A8F6-CA018144811C@continuum.io> <87mvwft2ar.fsf@berkeley.edu> <370C591F-9711-43E6-BE12-8C3E96946803@continuum.io> <87bncutiwt.fsf@berkeley.edu> Message-ID: On Tue, Sep 22, 2015 at 10:55 PM, Bryan Van de Ven wrote: > > > On Sep 22, 2015, at 1:48 PM, Matthew Brett > wrote: > > > > The point is, that a sensible organization and a sensible leader has > > to take the possibility of conflict of interest into account. They > > also have to consider the perception of a conflict of interest. > > Of course, and the policies to deal with conflicts have deal with the > possibility that *anyone* at all might have conflict. But that was not your > suggestion. Your suggestion was to make one individual be subject to > additional scrutiny that no one else would be subject to. Please explain > why should one person be singled out for a special "six-month waiting > period" when the exact same possibility for conflict exists with anyone who > is ever on the committee? > I don't quite understand where the discussion went. The question was not whether Travis is singled out but whether he is "singled in". >From my perspectives (7 to 8 years) the situation has changed a lot. Most of the discussion and consensus building happens on github issues, pull requests and mailing lists. Merge policy has changed a lot since the svn days. Based on all the comments, Travis doesn't have time for this. And I think the final (last resort) decisions about code should be left to the active developers that know and participate in the actual work. If Travis is treated as developer but doesn't have a special status, then there is also no reason to "single out" him and Continuum for possibly too much influence. This is already the current status quo as it developed over the last several years, AFAICS. In my view, in a narrow sense, Travis is a "hit and run" contributor. good ideas and providing good contributions, but somebody has to integrate it, maintain it and fit it into the development plan. (Given my experience I would compare it more with GSOC contributions that need the "core developers" to provide the continuity in the development to absorb the good work.) Travis has too many other obligations and interests to provide this day to day continuity. Travis is still very important for providing ideas, pushing projects forward and as one of the community leaders, but I would leave the final decisions for the development of numpy to the developers in the trenches. I pretty much agree completely with Nathanial, and Sebastian, (except that I don't know much about any other FOSS communities) And to repeat my point from another thread on this: I'm very skeptical about any committee or board that is based on "outside members" and that is involved in the decision about code development. Josef > > > It is the opposite of sensible, to respond to this with 'how dare you" > > or by asserting that this could never happen or by saying that we > > shouldn't talk about that in case people get frightened. I point you > > Red herring. The objection (as many people have now voiced) is the double > standard you proposed. > > > again to Linus' interview [1]. He is not upset that he has been > > insulted by the implication of conflict of interest, he soberly > > accepts that this will always be an issue, with companies in > > particular, and goes out of his way to address that in an explicit and > > reasonable way. > > Your selective quotation is impressive. Also in that interview, Linus > points out that his employment contract is probably "unique in the entire > world". Which means in practical terms that the details of what he has does > are fairly well irrelevant to any other situation. So what is the point in > bringing it up, at all, except to try and diminish someone else by > comparison? > > (I'm done.) > > Bryan > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > https://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ay at mntw.re Wed Sep 23 01:58:05 2015 From: ay at mntw.re (Andriy Yurchuk) Date: Wed, 23 Sep 2015 07:58:05 +0200 Subject: [Numpy-discussion] Tentative NumPy Tutorial inaccessible Message-ID: <9f3a2b5086c48fbe3a292fb0ac8ffdcc@minuteware.net> Hi! The Tentative NumPy Tutorial is no longer accessible by the URL http://wiki.scipy.org/Tentative_NumPy_Tutorial, it returns a 403. The link to this page is still on NumPy homepage though. Has the page been moved somewhere else? --- Regards, Andriy Yurchuk From njs at pobox.com Wed Sep 23 02:53:19 2015 From: njs at pobox.com (Nathaniel Smith) Date: Tue, 22 Sep 2015 23:53:19 -0700 Subject: [Numpy-discussion] Tentative NumPy Tutorial inaccessible In-Reply-To: <9f3a2b5086c48fbe3a292fb0ac8ffdcc@minuteware.net> References: <9f3a2b5086c48fbe3a292fb0ac8ffdcc@minuteware.net> Message-ID: On Tue, Sep 22, 2015 at 10:58 PM, Andriy Yurchuk wrote: > Hi! > > The Tentative NumPy Tutorial is no longer accessible by the URL > http://wiki.scipy.org/Tentative_NumPy_Tutorial, it returns a 403. The link > to this page is still on NumPy homepage though. Has the page been moved > somewhere else? No, our infrastructure is just in some disarray right now... there's some discussion here: https://github.com/numpy/numpy/issues/6325 (and I just pointed this out there as well since the relevant admins seem to be reading that thread). -n -- Nathaniel J. Smith -- http://vorpus.org From fperez.net at gmail.com Wed Sep 23 04:02:16 2015 From: fperez.net at gmail.com (Fernando Perez) Date: Wed, 23 Sep 2015 01:02:16 -0700 Subject: [Numpy-discussion] Governance model request In-Reply-To: References: <8941C0D2-17EC-420D-A8F6-CA018144811C@continuum.io> <87mvwft2ar.fsf@berkeley.edu> <370C591F-9711-43E6-BE12-8C3E96946803@continuum.io> Message-ID: Hi all, I would like to pitch in here, I am sorry that I didn't have the time before... First, I want to disclose that recently Continuum made a research gift to the Jupyter project; we were just now writing up a blog post to acknowledge this, but in light of this discussion, I feel that I should say this up front so folks can gauge any potential bias accordingly. On Tue, Sep 22, 2015 at 3:44 AM, Travis Oliphant wrote: > I'm actually offended that so many at BIDS seem eager to crucify my > intentions when I've done nothing but give away my time, my energy, my > resources, and my sleep to NumPy for many, many years. I guess if your > intent is to drive me away, then you are succeeding. Travis, first, I'd like to kindly ask you not to conflate BIDS, an institution where a large number of people work, with the personal opinions of some, who happen to work there but in this case are speaking only for themselves. You say "so many at BIDS", but as far as I know, your disagreements are with Stefan and Nathaniel (Matthew doesn't work at BIDS). You are painting with a very wide brush the work of many people, and in the process, unfairly impacting others who have nothing to do with this. If anything, the only things I'm aware BIDS has done in an official capacity regarding you or Continuum is to offer hosting for Continuum developers at the DS4DS workshop and beyond (after an explicit request by Matt Rocklin, and one we were delighted to honor), and our hosting of your lecture in our Friday Data Science Lecture Series last week. With that out of the way... 1. I hope the discussion can move past the suspicion and innuendo about Continuum and Travis. I haven't always agreed with how Travis communicates some of his ideas, and I've said it to him in such instances (e.g. this weekend, as I myself was surprised at how his last round of comments had landed on the list a few days back). But I also have worked closely with him for years because I know that he has proven, not in words, but in actions, that he has the best interests of our community at heart, and that he is willing to try and do everything in his power to help whenever he can. When we founded Numfocus back in 2012, it would have been impossible for it to really bootstrap without Travis' generosity, since he effectively footed the bill for resources that were critically needed at the start. And yet, he was always willing to take every step necessary to help Numfocus grow independent of Continuum, so that it could be a real community asset: today, there's not a single Continuum employee on the NF board (Travis and I both resigned from the board a while back to allow for some fresh blood). The creation and open-sourcing of conda has also been a critical contribution, that I know many of us have benefited from: we all carry the scars from the python packaging horror shows, and conda/anaconda has been a life-changer. The fact that conda itself is open, means we have a core tool that we can build upon. To put it bluntly, few people in the whole world have given more of their life, energy and resources to our community than Travis, and have done so as generously as he has. He may have made mistakes, and again, I often disagree with how he communicates. But accusations and innuendo like the ones in this thread are damaging, hurtful and useless. And one thing that I hope people will remember is that, famous and powerful as Travis may be, he's still our colleague, a member of our community, and *a human being*, so let's remember that as well... 2. Conflicts of interest are a fact of life, in fact, I would argue that every healthy and sufficiently interconnected community eventually *should* have conflicts of interest. They are a sign that there is activity across multiple centers of interest, and individuals with connections in multiple areas of the community. And we *want* folks who are engaged enough precisely to have such interests! For conflict of interest management, we don't need to reinvent the wheel, this is actually something where our beloved institutions, blessed be their bureaucratic souls, have tons of training materials that happen to be not completely useless. Most universities and the national labs have information on COIs that provides guidelines, and Numpy could include in its governance model more explicit language about COIs if desired. So, the issue is not to view COIs as something evil or undesirable, but rather as the very real consequence of operating in an interconnected set of institutions. And once you take that stance, you deal with that rationally and realistically. For example, you accept that companies aren't the only ones with potential COIs: *all* entities have them. As Ryan May aptly pointed out, the notion that academic institutions are somehow immune to hidden agendas or other interests is naive at best... And I say that as someone who has happily stayed in academia, resisting multiple overtures from industry over the years, but not out of some quaint notion that academia is a pristine haven of principled purity. Quite the opposite: in building large and complex projects, I've seen painfully close how the university/government research world has its own flavor of the same power, financial and political ugliness that we attribute to the commercial side. 3. Commercial actors. Following up on the last paragraph, we should accept that *all* institutions have agendas, not just companies. We live in a world with companies, and I think it's naive to take a knee-jerk anti-commercial stance: our community has had a productive and successful history of interaction with industry in the past, and hopefully that will continue in the future. What is true, however, is that community projects should maintain the "seat of power" in the community, and *not* in any single company. In fact, this is important even to ensure that many companies feel comfortable engaging the projects, precisely so they know that the technology is driven in an open and neutral way even if some of their competitors participate. That's why a governance model that is anchored in neutral ground is so important. We've worked hard to make Numfocus the legal entity that can play that role (that's why it's a 501(c)3), and that's why we've framed our governance model for Jupyter in a way that makes all the institutions (including Berkeley and Cal Poly) simply 'partners' that contribute by virtue of supporting employees. But the owners of the decisions are the *individuals* who do the work and form the community, not the companies/institutions. If we accept these premises, then hopefully we can have a rational conversation about how to build a community, where at any point in time, any of us should be judged on the merit of our actions, not the hypotheticals of our intentions or our affiliations (commercial, government, academic, etc). Sorry for the long wall of text, I rarely post on this list anymore. But I was saddened to see the turn of this thread, and I hope I can contribute some perspective (and not make things worse :) Cheers, -- Fernando Perez (@fperez_org; http://fperez.org) fperez.net-at-gmail: mailing lists only (I ignore this when swamped!) fernando.perez-at-berkeley: contact me here for any direct mail -------------- next part -------------- An HTML attachment was scrubbed... URL: From sebastian at sipsolutions.net Wed Sep 23 04:25:38 2015 From: sebastian at sipsolutions.net (Sebastian Berg) Date: Wed, 23 Sep 2015 08:25:38 +0000 Subject: [Numpy-discussion] Steering Committee Size Message-ID: Hi, Trying to figure out at least a bit from the discussions. While I am happy with the draft, I wonder if someone has some insights about some questions: 1. How large crowds have examples of working well with apache style voting? 2. How large do we expect numpy steering council to be (I have always thought about 10). 3. More on opinions, how large does the community feel is too large (so that we should maybe elect people). And to maybe more a discussion point, does the community feel that those who would be/are affectivly now in the Steering Council do not sufficiently represent old time contributers who were not active in the past year(s). I cannot form/change my opinion based on the previous discussion, because I would like to get an idea of how everyone feels about these points first. Then we can fight about details :) - Sebastian (sending from phone, so sorry about eventual weird typos) From Dan.Patterson at carleton.ca Wed Sep 23 05:25:33 2015 From: Dan.Patterson at carleton.ca (Dan Patterson) Date: Wed, 23 Sep 2015 05:25:33 -0400 Subject: [Numpy-discussion] Tentative numpy tutorial In-Reply-To: References: Message-ID: <4CC6F34628424747B6824880164673C369C2E4C271@CCSMBX10.CUNET.CARLETON.CA> Bryan is this the tutorial to which you refer? http://docs.scipy.org/doc/scipy/reference/tutorial/index.html From faltet at gmail.com Wed Sep 23 06:39:59 2015 From: faltet at gmail.com (Francesc Alted) Date: Wed, 23 Sep 2015 12:39:59 +0200 Subject: [Numpy-discussion] Governance model request In-Reply-To: References: <8941C0D2-17EC-420D-A8F6-CA018144811C@continuum.io> <87mvwft2ar.fsf@berkeley.edu> <370C591F-9711-43E6-BE12-8C3E96946803@continuum.io> Message-ID: Hi Fernando, I am happy that you decided to chime in here. This thread derailed in a bad way and I hope that your wise words will help to redress the situation. In fact, I would like to propose you having part of a future steering committee of NumPy. I know that you may never have been implied in the hard core development of NumPy, but my perception is that your opinions are highly respected by almost everybody in the NumPy ecosystem. More than that, you have this rare ability of being able to get both a donation from Microsoft and at the same time (same year?) being awarded by the FSF, which frankly, is not an easy thing to do ;) Just to clear, I am not saying that you should act as the person for deciding the roadmap for NumPy at all, but a person that should be in charge of acting as an independent referee in the foreseeable Conflicts of Interest in the NumPy roadmap. Sorry if that means more work for you Fernando, because I know that you have become a very busy person, but I also know how much do you care about the NumPy ecosystem, and IMHO the NumPy community needs a person like you *now*. Francesc 2015-09-23 10:02 GMT+02:00 Fernando Perez : > Hi all, > > I would like to pitch in here, I am sorry that I didn't have the time > before... > > First, I want to disclose that recently Continuum made a research gift to > the Jupyter project; we were just now writing up a blog post to acknowledge > this, but in light of this discussion, I feel that I should say this up > front so folks can gauge any potential bias accordingly. > > > On Tue, Sep 22, 2015 at 3:44 AM, Travis Oliphant > wrote: > >> I'm actually offended that so many at BIDS seem eager to crucify my >> intentions when I've done nothing but give away my time, my energy, my >> resources, and my sleep to NumPy for many, many years. I guess if your >> intent is to drive me away, then you are succeeding. > > > Travis, first, I'd like to kindly ask you not to conflate BIDS, an > institution where a large number of people work, with the personal opinions > of some, who happen to work there but in this case are speaking only for > themselves. You say "so many at BIDS", but as far as I know, your > disagreements are with Stefan and Nathaniel (Matthew doesn't work at > BIDS). You are painting with a very wide brush the work of many people, > and in the process, unfairly impacting others who have nothing to do with > this. > > If anything, the only things I'm aware BIDS has done in an official > capacity regarding you or Continuum is to offer hosting for Continuum > developers at the DS4DS workshop and beyond (after an explicit request by > Matt Rocklin, and one we were delighted to honor), and our hosting of your > lecture in our Friday Data Science Lecture Series last week. > > With that out of the way... > > > 1. I hope the discussion can move past the suspicion and innuendo about > Continuum and Travis. I haven't always agreed with how Travis communicates > some of his ideas, and I've said it to him in such instances (e.g. this > weekend, as I myself was surprised at how his last round of comments had > landed on the list a few days back). But I also have worked closely with > him for years because I know that he has proven, not in words, but in > actions, that he has the best interests of our community at heart, and that > he is willing to try and do everything in his power to help whenever he > can. > > When we founded Numfocus back in 2012, it would have been impossible for > it to really bootstrap without Travis' generosity, since he effectively > footed the bill for resources that were critically needed at the start. And > yet, he was always willing to take every step necessary to help Numfocus > grow independent of Continuum, so that it could be a real community asset: > today, there's not a single Continuum employee on the NF board (Travis and > I both resigned from the board a while back to allow for some fresh blood). > > The creation and open-sourcing of conda has also been a critical > contribution, that I know many of us have benefited from: we all carry the > scars from the python packaging horror shows, and conda/anaconda has been a > life-changer. The fact that conda itself is open, means we have a core tool > that we can build upon. > > To put it bluntly, few people in the whole world have given more of their > life, energy and resources to our community than Travis, and have done so > as generously as he has. He may have made mistakes, and again, I often > disagree with how he communicates. But accusations and innuendo like the > ones in this thread are damaging, hurtful and useless. And one thing that > I hope people will remember is that, famous and powerful as Travis may be, > he's still our colleague, a member of our community, and *a human being*, > so let's remember that as well... > > > 2. Conflicts of interest are a fact of life, in fact, I would argue that > every healthy and sufficiently interconnected community eventually *should* > have conflicts of interest. They are a sign that there is activity across > multiple centers of interest, and individuals with connections in multiple > areas of the community. And we *want* folks who are engaged enough > precisely to have such interests! > > For conflict of interest management, we don't need to reinvent the wheel, > this is actually something where our beloved institutions, blessed be their > bureaucratic souls, have tons of training materials that happen to be not > completely useless. Most universities and the national labs have > information on COIs that provides guidelines, and Numpy could include in > its governance model more explicit language about COIs if desired. > > So, the issue is not to view COIs as something evil or undesirable, but > rather as the very real consequence of operating in an interconnected set > of institutions. And once you take that stance, you deal with that > rationally and realistically. > > For example, you accept that companies aren't the only ones with potential > COIs: *all* entities have them. As Ryan May aptly pointed out, the notion > that academic institutions are somehow immune to hidden agendas or other > interests is naive at best... And I say that as someone who has happily > stayed in academia, resisting multiple overtures from industry over the > years, but not out of some quaint notion that academia is a pristine haven > of principled purity. Quite the opposite: in building large and complex > projects, I've seen painfully close how the university/government research > world has its own flavor of the same power, financial and political > ugliness that we attribute to the commercial side. > > > 3. Commercial actors. Following up on the last paragraph, we should > accept that *all* institutions have agendas, not just companies. We live > in a world with companies, and I think it's naive to take a knee-jerk > anti-commercial stance: our community has had a productive and successful > history of interaction with industry in the past, and hopefully that will > continue in the future. > > What is true, however, is that community projects should maintain the > "seat of power" in the community, and *not* in any single company. In > fact, this is important even to ensure that many companies feel comfortable > engaging the projects, precisely so they know that the technology is driven > in an open and neutral way even if some of their competitors participate. > > That's why a governance model that is anchored in neutral ground is so > important. We've worked hard to make Numfocus the legal entity that can > play that role (that's why it's a 501(c)3), and that's why we've framed our > governance model for Jupyter in a way that makes all the institutions > (including Berkeley and Cal Poly) simply 'partners' that contribute by > virtue of supporting employees. But the owners of the decisions are the > *individuals* who do the work and form the community, not the > companies/institutions. > > > If we accept these premises, then hopefully we can have a rational > conversation about how to build a community, where at any point in time, > any of us should be judged on the merit of our actions, not the > hypotheticals of our intentions or our affiliations (commercial, > government, academic, etc). > > > Sorry for the long wall of text, I rarely post on this list anymore. But > I was saddened to see the turn of this thread, and I hope I can contribute > some perspective (and not make things worse :) > > > Cheers, > > -- > Fernando Perez (@fperez_org; http://fperez.org) > fperez.net-at-gmail: mailing lists only (I ignore this when swamped!) > fernando.perez-at-berkeley: contact me here for any direct mail > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > https://mail.scipy.org/mailman/listinfo/numpy-discussion > > -- Francesc Alted -------------- next part -------------- An HTML attachment was scrubbed... URL: From cimrman3 at ntc.zcu.cz Wed Sep 23 07:10:50 2015 From: cimrman3 at ntc.zcu.cz (Robert Cimrman) Date: Wed, 23 Sep 2015 13:10:50 +0200 Subject: [Numpy-discussion] ANN: SfePy 2015.3 Message-ID: <560288BA.1040305@ntc.zcu.cz> I am pleased to announce release 2015.3 of SfePy. Description ----------- SfePy (simple finite elements in Python) is a software for solving systems of coupled partial differential equations by the finite element method or by the isogeometric analysis (preliminary support). It is distributed under the new BSD license. Home page: http://sfepy.org Mailing list: http://groups.google.com/group/sfepy-devel Git (source) repository, issue tracker, wiki: http://github.com/sfepy Highlights of this release -------------------------- - preliminary support for parallel computing - unified evaluation of basis functions (= isogeometric analysis fields can be evaluated in arbitrary points) - (mostly) fixed finding of reference element coordinates of physical points - several new or improved examples For full release notes see http://docs.sfepy.org/doc/release_notes.html#id1 (rather long and technical). Best regards, Robert Cimrman on behalf of the SfePy development team --- Contributors to this release in alphabetical order: Robert Cimrman Vladimir Lukes From sebastian at sipsolutions.net Wed Sep 23 09:55:48 2015 From: sebastian at sipsolutions.net (Sebastian Berg) Date: Wed, 23 Sep 2015 13:55:48 +0000 Subject: [Numpy-discussion] Governance model request In-Reply-To: References: <8941C0D2-17EC-420D-A8F6-CA018144811C@continuum.io> <87mvwft2ar.fsf@berkeley.edu> <370C591F-9711-43E6-BE12-8C3E96946803@continuum.io> Message-ID: On Wed Sep 23 12:39:59 2015 GMT+0200, Francesc Alted wrote: > Hi Fernando, > > I am happy that you decided to chime in here. This thread derailed in a > bad way and I hope that your wise words will help to redress the situation. > > In fact, I would like to propose you having part of a future steering > committee of NumPy. I know that you may never have been implied in the > hard core development of NumPy, but my perception is that your opinions are > highly respected by almost everybody in the NumPy ecosystem. More than > that, you have this rare ability of being able to get both a donation from > Microsoft and at the same time (same year?) being awarded by the FSF, which > frankly, is not an easy thing to do ;) > > Just to clear, I am not saying that you should act as the person for > deciding the roadmap for NumPy at all, but a person that should be in > charge of acting as an independent referee in the foreseeable Conflicts of > Interest in the NumPy roadmap. > Is it ok if I get a bit angry soon ;)? We can find many great community leaders. But please tell me that you all feel that numpy is so central or whatever that these community leaders should be in some sense in charge of numpy. I am ready to accept that and maybe it can be helpful, and a huge gain for numpy, but I would like to see a clear statement and reasons. I like Fernandos mail, too, nor do I doubt Travis achievements. But discussing who is great community leader, etc. is frankly not obvious to me related to numpy governance. Now if you say: Guys, you should have some community leader guidance in numpy, then we can discuss who. But to me it is not obvious that community leaders who are *not* directly active in numpy should be explicitely in governance. I am aware that everyone wants to help, but right now I do not feel helped at all :). - Sebastian > Sorry if that means more work for you Fernando, because I know that you > have become a very busy person, but I also know how much do you care about > the NumPy ecosystem, and IMHO the NumPy community needs a person like you > *now*. > > Francesc > > 2015-09-23 10:02 GMT+02:00 Fernando Perez : > > > Hi all, > > > > I would like to pitch in here, I am sorry that I didn't have the time > > before... > > > > First, I want to disclose that recently Continuum made a research gift to > > the Jupyter project; we were just now writing up a blog post to acknowledge > > this, but in light of this discussion, I feel that I should say this up > > front so folks can gauge any potential bias accordingly. > > > > > > On Tue, Sep 22, 2015 at 3:44 AM, Travis Oliphant > > wrote: > > > >> I'm actually offended that so many at BIDS seem eager to crucify my > >> intentions when I've done nothing but give away my time, my energy, my > >> resources, and my sleep to NumPy for many, many years. I guess if your > >> intent is to drive me away, then you are succeeding. > > > > > > Travis, first, I'd like to kindly ask you not to conflate BIDS, an > > institution where a large number of people work, with the personal opinions > > of some, who happen to work there but in this case are speaking only for > > themselves. You say "so many at BIDS", but as far as I know, your > > disagreements are with Stefan and Nathaniel (Matthew doesn't work at > > BIDS). You are painting with a very wide brush the work of many people, > > and in the process, unfairly impacting others who have nothing to do with > > this. > > > > If anything, the only things I'm aware BIDS has done in an official > > capacity regarding you or Continuum is to offer hosting for Continuum > > developers at the DS4DS workshop and beyond (after an explicit request by > > Matt Rocklin, and one we were delighted to honor), and our hosting of your > > lecture in our Friday Data Science Lecture Series last week. > > > > With that out of the way... > > > > > > 1. I hope the discussion can move past the suspicion and innuendo about > > Continuum and Travis. I haven't always agreed with how Travis communicates > > some of his ideas, and I've said it to him in such instances (e.g. this > > weekend, as I myself was surprised at how his last round of comments had > > landed on the list a few days back). But I also have worked closely with > > him for years because I know that he has proven, not in words, but in > > actions, that he has the best interests of our community at heart, and that > > he is willing to try and do everything in his power to help whenever he > > can. > > > > When we founded Numfocus back in 2012, it would have been impossible for > > it to really bootstrap without Travis' generosity, since he effectively > > footed the bill for resources that were critically needed at the start. And > > yet, he was always willing to take every step necessary to help Numfocus > > grow independent of Continuum, so that it could be a real community asset: > > today, there's not a single Continuum employee on the NF board (Travis and > > I both resigned from the board a while back to allow for some fresh blood). > > > > The creation and open-sourcing of conda has also been a critical > > contribution, that I know many of us have benefited from: we all carry the > > scars from the python packaging horror shows, and conda/anaconda has been a > > life-changer. The fact that conda itself is open, means we have a core tool > > that we can build upon. > > > > To put it bluntly, few people in the whole world have given more of their > > life, energy and resources to our community than Travis, and have done so > > as generously as he has. He may have made mistakes, and again, I often > > disagree with how he communicates. But accusations and innuendo like the > > ones in this thread are damaging, hurtful and useless. And one thing that > > I hope people will remember is that, famous and powerful as Travis may be, > > he's still our colleague, a member of our community, and *a human being*, > > so let's remember that as well... > > > > > > 2. Conflicts of interest are a fact of life, in fact, I would argue that > > every healthy and sufficiently interconnected community eventually *should* > > have conflicts of interest. They are a sign that there is activity across > > multiple centers of interest, and individuals with connections in multiple > > areas of the community. And we *want* folks who are engaged enough > > precisely to have such interests! > > > > For conflict of interest management, we don't need to reinvent the wheel, > > this is actually something where our beloved institutions, blessed be their > > bureaucratic souls, have tons of training materials that happen to be not > > completely useless. Most universities and the national labs have > > information on COIs that provides guidelines, and Numpy could include in > > its governance model more explicit language about COIs if desired. > > > > So, the issue is not to view COIs as something evil or undesirable, but > > rather as the very real consequence of operating in an interconnected set > > of institutions. And once you take that stance, you deal with that > > rationally and realistically. > > > > For example, you accept that companies aren't the only ones with potential > > COIs: *all* entities have them. As Ryan May aptly pointed out, the notion > > that academic institutions are somehow immune to hidden agendas or other > > interests is naive at best... And I say that as someone who has happily > > stayed in academia, resisting multiple overtures from industry over the > > years, but not out of some quaint notion that academia is a pristine haven > > of principled purity. Quite the opposite: in building large and complex > > projects, I've seen painfully close how the university/government research > > world has its own flavor of the same power, financial and political > > ugliness that we attribute to the commercial side. > > > > > > 3. Commercial actors. Following up on the last paragraph, we should > > accept that *all* institutions have agendas, not just companies. We live > > in a world with companies, and I think it's naive to take a knee-jerk > > anti-commercial stance: our community has had a productive and successful > > history of interaction with industry in the past, and hopefully that will > > continue in the future. > > > > What is true, however, is that community projects should maintain the > > "seat of power" in the community, and *not* in any single company. In > > fact, this is important even to ensure that many companies feel comfortable > > engaging the projects, precisely so they know that the technology is driven > > in an open and neutral way even if some of their competitors participate. > > > > That's why a governance model that is anchored in neutral ground is so > > important. We've worked hard to make Numfocus the legal entity that can > > play that role (that's why it's a 501(c)3), and that's why we've framed our > > governance model for Jupyter in a way that makes all the institutions > > (including Berkeley and Cal Poly) simply 'partners' that contribute by > > virtue of supporting employees. But the owners of the decisions are the > > *individuals* who do the work and form the community, not the > > companies/institutions. > > > > > > If we accept these premises, then hopefully we can have a rational > > conversation about how to build a community, where at any point in time, > > any of us should be judged on the merit of our actions, not the > > hypotheticals of our intentions or our affiliations (commercial, > > government, academic, etc). > > > > > > Sorry for the long wall of text, I rarely post on this list anymore. But > > I was saddened to see the turn of this thread, and I hope I can contribute > > some perspective (and not make things worse :) > > > > > > Cheers, > > > > -- > > Fernando Perez (@fperez_org; http://fperez.org) > > fperez.net-at-gmail: mailing lists only (I ignore this when swamped!) > > fernando.perez-at-berkeley: contact me here for any direct mail > > > > _______________________________________________ > > NumPy-Discussion mailing list > > NumPy-Discussion at scipy.org > > https://mail.scipy.org/mailman/listinfo/numpy-discussion > > > > > > > -- > Francesc Alted > From fperez.net at gmail.com Wed Sep 23 10:21:08 2015 From: fperez.net at gmail.com (Fernando Perez) Date: Wed, 23 Sep 2015 07:21:08 -0700 Subject: [Numpy-discussion] Governance model request In-Reply-To: References: <8941C0D2-17EC-420D-A8F6-CA018144811C@continuum.io> <87mvwft2ar.fsf@berkeley.edu> <370C591F-9711-43E6-BE12-8C3E96946803@continuum.io> Message-ID: On Wed, Sep 23, 2015 at 6:55 AM, Sebastian Berg wrote: > Is it ok if I get a bit angry soon ;)? Don't worry, Sebastian :) I appreciate Francesc's kind words, but I have no intention of imposing my presence anywhere, it's not like I'm looking for extra work at this point. The process of establishing governance has to come organically from within a community. Cheers -- Fernando Perez (@fperez_org; http://fperez.org) fperez.net-at-gmail: mailing lists only (I ignore this when swamped!) fernando.perez-at-berkeley: contact me here for any direct mail -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.barker at noaa.gov Wed Sep 23 11:47:21 2015 From: chris.barker at noaa.gov (Chris Barker - NOAA Federal) Date: Wed, 23 Sep 2015 08:47:21 -0700 Subject: [Numpy-discussion] Governance model request In-Reply-To: References: <8941C0D2-17EC-420D-A8F6-CA018144811C@continuum.io> <87mvwft2ar.fsf@berkeley.edu> <370C591F-9711-43E6-BE12-8C3E96946803@continuum.io> Message-ID: <2479925613954550807@unknownmsgid> > But discussing who is great community leader, etc. is frankly not obvious to me related to numpy governance. Thank you Sebastian. Could we please try to get back to the governance issues, without naming names? There are some specific questions on the table that need to get hashed out. Numpy does not have a BDFL. BDFLs are not selected, they are naturally produced, and there is no one that fits that bill now. We *could* decide to have a single individual executive leader, selected by some sort of democratic process. But that is not the same as a BDFL. And I think the almost-consensus now is to not have that. So here is what I think is on the table: We have the steering council. Which leaves two questions: -How big should it be? -Who will be on the original, "seed" council? Note that as I understand the current draft of the governance doc, once established, the council itself decides who is on the council. So at this point we really are ONLY deciding how it's going to start. It has to be bootstrapped somehow. However, that had been contentious enough that it would probably be a good idea to hash out some guidelines about the council membership. Personally, I have no idea how big the council should be. Too big, and there is no point, consensus is harder to reach the larger the group, and the main (only?) role of the council is to resolve issues where consensus has not been reached in the larger community. But what is too big? As for make-up of the council, I think we need to expand beyond people who have recently contributed core code. Yes, the council does need to have expertise to make technical decisions, but if you think about the likely contentious issues like ABI breakage, a core-code focused view is incomplete. So there should be representation by: Someone(s) with a long history of working with the code -- that institutional memory of why decisions were made the way they were could be key. Someone(s) that may not have worked on the core code, but is a major player in the community, perhaps as the leader of a Numpy-dependent project. This would provide representation for the broad community. I do want to note that the governance document as I understand it is consistent with these suggestions. As for conflict of interest issues, etc: Chill out people! Numpy is an open source project, if it gets hijacked, it can be forked. And the council is also democratic -- no one person can drive the project. If a council member is not acting in the interests of the community, s/he can be removed. NOTE: while x.org, and egcs, Xemacs, and ... may be examples of failures of governance, they are also examples of successes of open source. -Chris From matthew.brett at gmail.com Wed Sep 23 11:51:02 2015 From: matthew.brett at gmail.com (Matthew Brett) Date: Wed, 23 Sep 2015 08:51:02 -0700 Subject: [Numpy-discussion] Tentative numpy tutorial In-Reply-To: <4CC6F34628424747B6824880164673C369C2E4C271@CCSMBX10.CUNET.CARLETON.CA> References: <4CC6F34628424747B6824880164673C369C2E4C271@CCSMBX10.CUNET.CARLETON.CA> Message-ID: Hi, On Wed, Sep 23, 2015 at 2:25 AM, Dan Patterson wrote: > Bryan is this the tutorial to which you refer? http://docs.scipy.org/doc/scipy/reference/tutorial/index.html > _______________________________________________ I think it is the http://wiki.scipy.org/Tentative_NumPy_Tutorial Cheers, Matthew From thomas.robitaille at gmail.com Wed Sep 23 12:39:00 2015 From: thomas.robitaille at gmail.com (Thomas Robitaille) Date: Wed, 23 Sep 2015 18:39:00 +0200 Subject: [Numpy-discussion] ANN: numtraits v0.2 Message-ID: Hi everyone, We have released a small experimental package called numtraits that builds on top of the traitlets package and provides a NumericalTrait class that can be used to validate properties such as: * number of dimension (for arrays) * shape (for arrays) * domain (e.g. positive, negative, range of values) * units (with support for astropy.units, pint, and quantities) The idea is to be able to write a class like: class Sphere(HasTraits): radius = NumericalTrait(domain='strictly-positive', ndim=0) position = NumericalTrait(shape=(3,)) and all the validation will then be done automatically when the user sets 'radius' or 'position'. In addition, tuples and lists can get automatically converted to arrays, and default values can be specified. You can read more about the package and see examples of it in use here: https://github.com/astrofrog/numtraits and it can be easily installed with pip install numtraits The package supports both Python 3.3+ and Legacy Python (2.7) :) At this point, we would be very interested in feedback - the package is still very young and we can still change the API if needed. Please open issues with suggestions! Cheers, Tom and Francesco From charlesr.harris at gmail.com Wed Sep 23 13:12:31 2015 From: charlesr.harris at gmail.com (Charles R Harris) Date: Wed, 23 Sep 2015 11:12:31 -0600 Subject: [Numpy-discussion] Governance model request In-Reply-To: <2479925613954550807@unknownmsgid> References: <8941C0D2-17EC-420D-A8F6-CA018144811C@continuum.io> <87mvwft2ar.fsf@berkeley.edu> <370C591F-9711-43E6-BE12-8C3E96946803@continuum.io> <2479925613954550807@unknownmsgid> Message-ID: On Wed, Sep 23, 2015 at 9:47 AM, Chris Barker - NOAA Federal < chris.barker at noaa.gov> wrote: > > But discussing who is great community leader, etc. is frankly not > obvious to me related to numpy governance. > > Thank you Sebastian. > > Could we please try to get back to the governance issues, without > naming names? There are some specific questions on the table that need > to get hashed out. > > > Numpy does not have a BDFL. BDFLs are not selected, they are naturally > produced, and there is no one that fits that bill now. We *could* > decide to have a single individual executive leader, selected by some > sort of democratic process. But that is not the same as a BDFL. And I > think the almost-consensus now is to not have that. > > So here is what I think is on the table: > > We have the steering council. Which leaves two questions: > > -How big should it be? > -Who will be on the original, "seed" council? > > Note that as I understand the current draft of the governance doc, > once established, the council itself decides who is on the council. So > at this point we really are ONLY deciding how it's going to start. It > has to be bootstrapped somehow. > > However, that had been contentious enough that it would probably be a > good idea to hash out some guidelines about the council membership. > > Personally, I have no idea how big the council should be. Too big, and > there is no point, consensus is harder to reach the larger the group, > and the main (only?) role of the council is to resolve issues where > consensus has not been reached in the larger community. But what is > too big? > > As for make-up of the council, I think we need to expand beyond people > who have recently contributed core code. > > Yes, the council does need to have expertise to make technical > decisions, but if you think about the likely contentious issues like > ABI breakage, a core-code focused view is incomplete. So there should > be representation by: > > Someone(s) with a long history of working with the code -- that > institutional memory of why decisions were made the way they were > could be key. > > Someone(s) that may not have worked on the core code, but is a major > player in the community, perhaps as the leader of a Numpy-dependent > project. This would provide representation for the broad community. > > I do want to note that the governance document as I understand it is > consistent with these suggestions. > > As for conflict of interest issues, etc: > > Chill out people! > > Numpy is an open source project, if it gets hijacked, it can be forked. > > And the council is also democratic -- no one person can drive the > project. If a council member is not acting in the interests of the > community, s/he can be removed. > > Hear, hear. Well put Chris. I don't disagree with any of this. > NOTE: while x.org, and egcs, Xemacs, and ... may be examples of > failures of governance, they are also examples of successes of open > source. > Good point. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Wed Sep 23 14:04:12 2015 From: njs at pobox.com (Nathaniel Smith) Date: Wed, 23 Sep 2015 11:04:12 -0700 Subject: [Numpy-discussion] interpretation of the draft governance document (was Re: Governance model request) Message-ID: Hi Travis, On Tue, Sep 22, 2015 at 3:08 AM, Travis Oliphant wrote: > > > On Tue, Sep 22, 2015 at 4:33 AM, Nathaniel Smith wrote: >> >> On Tue, Sep 22, 2015 at 1:24 AM, Travis Oliphant >> wrote: >>> >>> I actually do agree with your view of the steering council as being >>> usually not really being needed. You are creating a straw-man by >>> indicating otherwise. I don't believe a small council should do anything >>> *except* resolve disputes that cannot be resolved without one. Like you, I >>> would expect that would almost never happen --- but I would argue that >>> extrapolating from Debian's experience is not actually relevant here. >> >> >> To be clear, Debian was only one example -- what I'm extrapolating from is >> every community-driven F/OSS project that I'm aware of. >> >> It's entirely possible my data set is incomplete -- if you have some other >> examples that you think would be better to extrapolate from, then I'd be >> genuinely glad to hear them. You may have noticed that I'm a bit of an >> enthusiast on this topic :-). >> > > > Yes, you are much better at that than I am. I'm not even sure where I > would look for this kind of data. > >>> >>> >>> >>> So, if the steering council is not really needed then why have it at all? >>> Let's just eliminate the concept entirely. >>> >> >> In my view, the reasons for having such a council are: >> 1) The framework is useful even if you never use it, because it means >> people can run "what if" scenarios in their mind and make decisions on that >> basis. In the US legal system, only a vanishingly small fraction of cases go >> to the Supreme Court -- but the rules governing the Supreme Court have a >> huge effect on all cases, because people can reason about what would happen >> *if* they tried to appeal to the Supreme Court. > > > O.K. That is a good point. I can see the value in that. > > >> >> 2) It provides a formal structure for interfacing with the outside world. >> E.g., one can't do anything with money or corporate contributions without >> having some kind of written-down and enforceable rules for making decisions >> (even if in practice you always stick to the "everyone is equal and we >> govern by consensus" part of the rules). > > > O.K. > >> >> 3) There are rare but important cases where discussions have to be had in >> private. The main one is "personnel decisions" like inviting people to join >> the council; another example Fernando has mentioned to me is that when they >> need to coordinate a press release between the project and a funding body, >> the steering council reviews the press release before it goes public. > > > O.K. > > >> >> That's pretty much it, IMO. >> >> The framework we all worked out at the dev meeting in Austin seems to >> handle these cases well AFAICT. > > > How did we "all" work it out when not everyone was there? This is where I > get lost. You talk about community decision making and yet any actual > decision is always a subset of the community. I suppose you just rely on > the "if nobody complains than it's o.k." rule? That really only works if > the project is moving slowly. By "all" I just meant "all of us who were there" (which was a majority of the active maintainers + a number of other interested parties -- the list of attendees is in the meeting notes if you're curious). In general I totally agree with your concern about only including a subset of the community. That's why we followed up by posting to the list a full set of notes on tentative-decisions-made, and the draft governance document in particular, for further discussion. We've already had multiple threads talking about it, even before this one. And it's pretty explicit in the document itself that no non-trivial decision can be considered final unless it's *at least* been posted on the mailing list. We didn't try to legislate the exact review requirements for every decision, because it's impossible to have a set of rules that scales from trivial typos (which just get merged, only github subscribers even know it happened) to foundational discussions like this one. This means that one of the things we trust contributors (esp. senior contributors) to do is to use their knowledge of the project to make judgement calls about how risky or controversial a given change will be, or if there's some particular expertise that should be consulted. (E.g. we might make sure to ping Robert Kern if there's some np.random change being discussed; I'm hesitant to finalize the PyUFunc ABI changes being discussed in the other thread until Ralf gets back, because I know that among the core maintainers he's particularly critical of the idea of breaking ABI.) And if we new information later comes to light then a decision can always be revisited -- people may get grumpy if you try to re-open an issue that's been considered settled for a year, but if you have a good reason and nothing irrevocable has happened (e.g. a veto obviously can't remove code from an existing release), then, well, it's annoying but what can you do, let's hear your reason. It's could certainly happen that sometimes the steering council + mailing list readers will all miss something important. But this is unavoidable in any system -- we're obviously not going to, like, institute a one month waiting period on every single decision or something. Ultimately you have to trust the core maintainers to have good judgement about which changes to accept, together with good meta-judgement about how controversial or broad-reaching any given change is likely to be, and then hope that the rest of the community will also supplement as they can. >>> But there are real questions that have to have an answer or an approach >>> to making a decision. The answer to these questions cannot really be a >>> vague notion of "lack of vigorous opposition by people who read the mailing >>> list" which then gets parried about as "the community decided this." The >>> NumPy user base is far, far larger than the number of people that read this >>> list. >> >> >> According to the dev meeting rules, no particularly "vigorous opposition" >> is required -- anyone who notices that something bad is happening can write >> a single email and stop an idea dead in its tracks, with only the steering >> council able to overrule. We expect this will rarely if ever happen, because >> the threat will be enough to keep everyone honest and listening, but about >> the only way we could possibly be *more* democratic is if we started phoning >> up random users at home to ask their opinion. > > > O.K. so how long is the time allowed for this kind of opposition to be > noted? See above. For regular discussions, there are some rough guidelines (uncontroversial bug fixes can just be merged; substantive API changes need at least a few days review on the mailing list). This governance discussion has been left open for a few weeks, and: "worst case, if a change is more controversial than expected, or a crucial critique is delayed because someone was on vacation, then it's no big deal: we apologize for misjudging the situation, [back up, and sort things out](http://producingoss.com/en/producingoss.html#version-control-relaxation)." (I think we all thought the governance discussion was done, actually. But then you posted, and so now we're talking about it some more. No worries; if there's an issue, we'd rather know, right?) For formal council votes (the ones we expect will rarely if every happen), then we do have a slightly more formal rule: that the vote "should be held open for long enough to give all interested Council Members a chance to respond -- at least one week." The words "at least" are in there to emphasize that the goal is to get an honest read of the council; e.g. it's not legitimate to play games by scheduling a vote when you know someone is on vacation. >> >> This is actually explicitly designed to prevent the situation where >> whoever talks the loudest and longest wins, and to put those with more and >> less time available on an equal footing. >> >>> >>> For better or for worse, we will always be subject to the "tyranny of who >>> has time to contribute lately". Fundamentally, I would argue that this >>> kind of "tyranny" should at least be tempered by additional considerations >>> from long-time contributors who may also be acting more indirectly than is >>> measured by a simple git log. >> >> >> I guess I am missing something fundamental here. Who are these long-time >> contributors who will sit on your council of 1/3/5 but who don't even read >> the mailing list? How will they know when their special insight is >> necessary? > > > The long-time contributors wouldn't necessarily sit on that council. But, > I would support the idea of an advisory council that could act if it saw > things going the wrong direction. This is where those people would sit. In the draft governance document, anyone who cares enough to pay attention effectively has a seat on this advisory council. I assume this is a superset of the people that you would nominate? > In the case of a 1 / 3 / 5 member council -- I would not argue to be on it > at all (but I would argue that some care should be taken to be sure it has > people with some years of experience if they are available). I'm only > asking to be on a steering council that is larger than 5 people, and I don't > actually prefer that the steering council be larger than 5 people. > >> >> No, absolutely not. The proposal is that these issues are decided by open >> discussion on the mailing list. (With the possible exception of #4 -- I >> suspect that given an extreme situation like this, then once all other >> efforts to mitigate the situation had failed the steering council would >> probably feel compelled to talk things over to double-check they had >> consensus before acting, and likely some part of this would have to be done >> in private.) > > > O.K. Then, I am misunderstanding. Oh good, I'm glad when things turn out to be misunderstandings, because those are (relatively) easy to solve :-). >> >> This is pretty explicit in the document (and again, this is text and ideas >> stolen directly from Jupyter/IPython): >> >> "During the everyday project activities, council members participate in >> all discussions, code review and other project activities as peers with all >> other Contributors and the Community. In these everyday activities, Council >> Members do not have any special power or privilege through their membership >> on the Council. [...] the Council may, if necessary [do pretty much >> anything, but] the Council's primary responsibility is to facilitate the >> ordinary community-based decision making procedure described above. If we >> ever have to step in and formally override the community for the health of >> the Project, then we will do so, but we will consider reaching this point to >> indicate a failure in our leadership." > > > Granting commit rights to the repo does not seem to me to be an "everyday > project activity" --- so I suppose I was confused. I suppose that could > happen with no serious objections on the list. It seems that the people > who actually have the commit bit presently should be consulted more than the > general public, though. I see, right, slight miscommunication here -- I was thinking about the decision of "what process do we [in general] use to decide who gets a commit bit", not the decision "should [this person] get a commit bit". Currently, yeah, the general process is that commit bits get given out by a somewhat informal private discussion among active committers (or, presumably, the "the steering council" if it does get formalized). But if someone wants to suggest that we switch to some other process instead, or formalize the current process, then anything like that would be proposed and debated on the mailing list. (One of my favorite wacky policies is that there are projects like Rubinius which automatically grant commit bits to everyone who submits a successful patch. I guess the theory is that worst case, they go and merge something they shouldn't have, more senior folks see it going by and they revert it again, no biggie! Usually the problem is just the opposite -- there are never enough reviewers. But I have not quite had the guts to seriously propose this for numpy ;-).) -n -- Nathaniel J. Smith -- http://vorpus.org From travis at continuum.io Wed Sep 23 14:59:42 2015 From: travis at continuum.io (Travis Oliphant) Date: Wed, 23 Sep 2015 13:59:42 -0500 Subject: [Numpy-discussion] interpretation of the draft governance document (was Re: Governance model request) In-Reply-To: References: Message-ID: Hi Nathaniel, Thanks for the clarifications. Is the governance document committed to the repository? I keep looking for it and have a hard time finding it --- I think I read it last in an email. In this way, I could make Pull Requests to the governance document if there are concrete suggestions for change, and then have them reviewed in the standard way. I'm hopeful that a few tweaks to the document would satisfy all my concerns. Thanks, -Travis On Wed, Sep 23, 2015 at 1:04 PM, Nathaniel Smith wrote: > Hi Travis, > > On Tue, Sep 22, 2015 at 3:08 AM, Travis Oliphant > wrote: > > > > > > On Tue, Sep 22, 2015 at 4:33 AM, Nathaniel Smith wrote: > >> > >> On Tue, Sep 22, 2015 at 1:24 AM, Travis Oliphant > >> wrote: > >>> > >>> I actually do agree with your view of the steering council as being > >>> usually not really being needed. You are creating a straw-man by > >>> indicating otherwise. I don't believe a small council should do > anything > >>> *except* resolve disputes that cannot be resolved without one. Like > you, I > >>> would expect that would almost never happen --- but I would argue that > >>> extrapolating from Debian's experience is not actually relevant here. > >> > >> > >> To be clear, Debian was only one example -- what I'm extrapolating from > is > >> every community-driven F/OSS project that I'm aware of. > >> > >> It's entirely possible my data set is incomplete -- if you have some > other > >> examples that you think would be better to extrapolate from, then I'd be > >> genuinely glad to hear them. You may have noticed that I'm a bit of an > >> enthusiast on this topic :-). > >> > > > > > > Yes, you are much better at that than I am. I'm not even sure where I > > would look for this kind of data. > > > >>> > >>> > >>> > >>> So, if the steering council is not really needed then why have it at > all? > >>> Let's just eliminate the concept entirely. > >>> > >> > >> In my view, the reasons for having such a council are: > >> 1) The framework is useful even if you never use it, because it means > >> people can run "what if" scenarios in their mind and make decisions on > that > >> basis. In the US legal system, only a vanishingly small fraction of > cases go > >> to the Supreme Court -- but the rules governing the Supreme Court have a > >> huge effect on all cases, because people can reason about what would > happen > >> *if* they tried to appeal to the Supreme Court. > > > > > > O.K. That is a good point. I can see the value in that. > > > > > >> > >> 2) It provides a formal structure for interfacing with the outside > world. > >> E.g., one can't do anything with money or corporate contributions > without > >> having some kind of written-down and enforceable rules for making > decisions > >> (even if in practice you always stick to the "everyone is equal and we > >> govern by consensus" part of the rules). > > > > > > O.K. > > > >> > >> 3) There are rare but important cases where discussions have to be had > in > >> private. The main one is "personnel decisions" like inviting people to > join > >> the council; another example Fernando has mentioned to me is that when > they > >> need to coordinate a press release between the project and a funding > body, > >> the steering council reviews the press release before it goes public. > > > > > > O.K. > > > > > >> > >> That's pretty much it, IMO. > >> > >> The framework we all worked out at the dev meeting in Austin seems to > >> handle these cases well AFAICT. > > > > > > How did we "all" work it out when not everyone was there? This is > where I > > get lost. You talk about community decision making and yet any actual > > decision is always a subset of the community. I suppose you just rely > on > > the "if nobody complains than it's o.k." rule? That really only works > if > > the project is moving slowly. > > By "all" I just meant "all of us who were there" (which was a majority > of the active maintainers + a number of other interested parties -- > the list of attendees is in the meeting notes if you're curious). > > In general I totally agree with your concern about only including a > subset of the community. That's why we followed up by posting to the > list a full set of notes on tentative-decisions-made, and the draft > governance document in particular, for further discussion. We've > already had multiple threads talking about it, even before this one. > And it's pretty explicit in the document itself that no non-trivial > decision can be considered final unless it's *at least* been posted on > the mailing list. > > We didn't try to legislate the exact review requirements for every > decision, because it's impossible to have a set of rules that scales > from trivial typos (which just get merged, only github subscribers > even know it happened) to foundational discussions like this one. This > means that one of the things we trust contributors (esp. senior > contributors) to do is to use their knowledge of the project to make > judgement calls about how risky or controversial a given change will > be, or if there's some particular expertise that should be consulted. > (E.g. we might make sure to ping Robert Kern if there's some np.random > change being discussed; I'm hesitant to finalize the PyUFunc ABI > changes being discussed in the other thread until Ralf gets back, > because I know that among the core maintainers he's particularly > critical of the idea of breaking ABI.) > > And if we new information later comes to light then a decision can > always be revisited -- people may get grumpy if you try to re-open an > issue that's been considered settled for a year, but if you have a > good reason and nothing irrevocable has happened (e.g. a veto > obviously can't remove code from an existing release), then, well, > it's annoying but what can you do, let's hear your reason. > > It's could certainly happen that sometimes the steering council + > mailing list readers will all miss something important. But this is > unavoidable in any system -- we're obviously not going to, like, > institute a one month waiting period on every single decision or > something. Ultimately you have to trust the core maintainers to have > good judgement about which changes to accept, together with good > meta-judgement about how controversial or broad-reaching any given > change is likely to be, and then hope that the rest of the community > will also supplement as they can. > > >>> But there are real questions that have to have an answer or an approach > >>> to making a decision. The answer to these questions cannot really be a > >>> vague notion of "lack of vigorous opposition by people who read the > mailing > >>> list" which then gets parried about as "the community decided this." > The > >>> NumPy user base is far, far larger than the number of people that read > this > >>> list. > >> > >> > >> According to the dev meeting rules, no particularly "vigorous > opposition" > >> is required -- anyone who notices that something bad is happening can > write > >> a single email and stop an idea dead in its tracks, with only the > steering > >> council able to overrule. We expect this will rarely if ever happen, > because > >> the threat will be enough to keep everyone honest and listening, but > about > >> the only way we could possibly be *more* democratic is if we started > phoning > >> up random users at home to ask their opinion. > > > > > > O.K. so how long is the time allowed for this kind of opposition to be > > noted? > > See above. For regular discussions, there are some rough guidelines > (uncontroversial bug fixes can just be merged; substantive API changes > need at least a few days review on the mailing list). This governance > discussion has been left open for a few weeks, and: "worst case, if a > change is more controversial than expected, or a crucial critique is > delayed because someone was on vacation, then it's no big deal: we > apologize for misjudging the situation, [back up, and sort things > out]( > http://producingoss.com/en/producingoss.html#version-control-relaxation)." > > (I think we all thought the governance discussion was done, actually. > But then you posted, and so now we're talking about it some more. No > worries; if there's an issue, we'd rather know, right?) > > For formal council votes (the ones we expect will rarely if every > happen), then we do have a slightly more formal rule: that the vote > "should be held open for long enough to give all interested Council > Members a chance to respond -- at least one week." The words "at > least" are in there to emphasize that the goal is to get an honest > read of the council; e.g. it's not legitimate to play games by > scheduling a vote when you know someone is on vacation. > > >> > >> This is actually explicitly designed to prevent the situation where > >> whoever talks the loudest and longest wins, and to put those with more > and > >> less time available on an equal footing. > >> > >>> > >>> For better or for worse, we will always be subject to the "tyranny of > who > >>> has time to contribute lately". Fundamentally, I would argue that > this > >>> kind of "tyranny" should at least be tempered by additional > considerations > >>> from long-time contributors who may also be acting more indirectly > than is > >>> measured by a simple git log. > >> > >> > >> I guess I am missing something fundamental here. Who are these long-time > >> contributors who will sit on your council of 1/3/5 but who don't even > read > >> the mailing list? How will they know when their special insight is > >> necessary? > > > > > > The long-time contributors wouldn't necessarily sit on that council. > But, > > I would support the idea of an advisory council that could act if it saw > > things going the wrong direction. This is where those people would sit. > > In the draft governance document, anyone who cares enough to pay > attention effectively has a seat on this advisory council. I assume > this is a superset of the people that you would nominate? > > > In the case of a 1 / 3 / 5 member council -- I would not argue to be on > it > > at all (but I would argue that some care should be taken to be sure it > has > > people with some years of experience if they are available). I'm only > > asking to be on a steering council that is larger than 5 people, and I > don't > > actually prefer that the steering council be larger than 5 people. > > > >> > >> No, absolutely not. The proposal is that these issues are decided by > open > >> discussion on the mailing list. (With the possible exception of #4 -- I > >> suspect that given an extreme situation like this, then once all other > >> efforts to mitigate the situation had failed the steering council would > >> probably feel compelled to talk things over to double-check they had > >> consensus before acting, and likely some part of this would have to be > done > >> in private.) > > > > > > O.K. Then, I am misunderstanding. > > Oh good, I'm glad when things turn out to be misunderstandings, > because those are (relatively) easy to solve :-). > > >> > >> This is pretty explicit in the document (and again, this is text and > ideas > >> stolen directly from Jupyter/IPython): > >> > >> "During the everyday project activities, council members participate in > >> all discussions, code review and other project activities as peers with > all > >> other Contributors and the Community. In these everyday activities, > Council > >> Members do not have any special power or privilege through their > membership > >> on the Council. [...] the Council may, if necessary [do pretty much > >> anything, but] the Council's primary responsibility is to facilitate the > >> ordinary community-based decision making procedure described above. If > we > >> ever have to step in and formally override the community for the health > of > >> the Project, then we will do so, but we will consider reaching this > point to > >> indicate a failure in our leadership." > > > > > > Granting commit rights to the repo does not seem to me to be an "everyday > > project activity" --- so I suppose I was confused. I suppose that > could > > happen with no serious objections on the list. It seems that the people > > who actually have the commit bit presently should be consulted more than > the > > general public, though. > > I see, right, slight miscommunication here -- I was thinking about the > decision of "what process do we [in general] use to decide who gets a > commit bit", not the decision "should [this person] get a commit bit". > Currently, yeah, the general process is that commit bits get given out > by a somewhat informal private discussion among active committers (or, > presumably, the "the steering council" if it does get formalized). But > if someone wants to suggest that we switch to some other process > instead, or formalize the current process, then anything like that > would be proposed and debated on the mailing list. > > (One of my favorite wacky policies is that there are projects like > Rubinius which automatically grant commit bits to everyone who submits > a successful patch. I guess the theory is that worst case, they go and > merge something they shouldn't have, more senior folks see it going by > and they revert it again, no biggie! Usually the problem is just the > opposite -- there are never enough reviewers. But I have not quite had > the guts to seriously propose this for numpy ;-).) > > -n > > -- > Nathaniel J. Smith -- http://vorpus.org > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > https://mail.scipy.org/mailman/listinfo/numpy-discussion > -- *Travis Oliphant* *Co-founder and CEO* @teoliphant 512-222-5440 http://www.continuum.io -------------- next part -------------- An HTML attachment was scrubbed... URL: From travis at continuum.io Wed Sep 23 15:18:39 2015 From: travis at continuum.io (Travis Oliphant) Date: Wed, 23 Sep 2015 14:18:39 -0500 Subject: [Numpy-discussion] Governance model request In-Reply-To: References: <8941C0D2-17EC-420D-A8F6-CA018144811C@continuum.io> <87mvwft2ar.fsf@berkeley.edu> <370C591F-9711-43E6-BE12-8C3E96946803@continuum.io> Message-ID: On Wed, Sep 23, 2015 at 3:02 AM, Fernando Perez wrote: > Hi all, > > I would like to pitch in here, I am sorry that I didn't have the time > before... > > First, I want to disclose that recently Continuum made a research gift to > the Jupyter project; we were just now writing up a blog post to acknowledge > this, but in light of this discussion, I feel that I should say this up > front so folks can gauge any potential bias accordingly. > > > On Tue, Sep 22, 2015 at 3:44 AM, Travis Oliphant > wrote: > >> I'm actually offended that so many at BIDS seem eager to crucify my >> intentions when I've done nothing but give away my time, my energy, my >> resources, and my sleep to NumPy for many, many years. I guess if your >> intent is to drive me away, then you are succeeding. > > > Travis, first, I'd like to kindly ask you not to conflate BIDS, an > institution where a large number of people work, with the personal opinions > of some, who happen to work there but in this case are speaking only for > themselves. You say "so many at BIDS", but as far as I know, your > disagreements are with Stefan and Nathaniel (Matthew doesn't work at > BIDS). You are painting with a very wide brush the work of many people, > and in the process, unfairly impacting others who have nothing to do with > this. > I accept that criticism and apologize for doing that. My *human* side was coming out, and I was not being fair. In my head, though I was also trying to illustrate how some seemed to be doing the same thing for Continuum or other companies. This did not come out very artfully in the early morning hours. I'm sorry. BIDS is doing a lot for the community --- the recent DS4DS workshop, for example, was a spectacularly useful summit --- I hope that many different write-ups and reports of the event make their way out into the world. > > > 1. I hope the discussion can move past the suspicion and innuendo about > Continuum and Travis. I haven't always agreed with how Travis communicates > some of his ideas, and I've said it to him in such instances (e.g. this > weekend, as I myself was surprised at how his last round of comments had > landed on the list a few days back). But I also have worked closely with > him for years because I know that he has proven, not in words, but in > actions, that he has the best interests of our community at heart, and that > he is willing to try and do everything in his power to help whenever he > can. > I really hope it's just a perception problem (perhaps on my end). There are challenges with working in the commercial world (there are a lot of things to do that have nothing to do with the technology creation) and communicating on open-source mailing lists. As many have noticed, despite my intentions to contribute, I really can't do the same level of contribution personally that I could when I was a student and a professor and had more time. However, I think that it is also under-appreciated (or mis-understood) how much time I have spent with training and helping people who have contributed instead. It's important to me to build a company that can sponsor people to work on open-source (in a community setting). We are still working on that, but it has been my intent. So, far it's actually easier to sponsor new projects than it is to sponsor people on old projects. I am quite sure that if Continuum had put 3 people full time on NumPy in 2012, there would have been a lot of back-lash and mis-understanding. That's why we didn't do it. The collateral effect of that was the creation of other tools that could be somewhat competitive with NumPy long term -- or not. I'd like to learn how to work with the community in an optimal way so that everyone benefits --- and progress happens. That's also why we created Numfocus --- though it is ironic that NumPy has been one of the last projects to actually sign up and be a formally sponsored project. 2. Conflicts of interest are a fact of life, in fact, I would argue that > every healthy and sufficiently interconnected community eventually *should* > have conflicts of interest. They are a sign that there is activity across > multiple centers of interest, and individuals with connections in multiple > areas of the community. And we *want* folks who are engaged enough > precisely to have such interests! > > For conflict of interest management, we don't need to reinvent the wheel, > this is actually something where our beloved institutions, blessed be their > bureaucratic souls, have tons of training materials that happen to be not > completely useless. Most universities and the national labs have > information on COIs that provides guidelines, and Numpy could include in > its governance model more explicit language about COIs if desired. > > So, the issue is not to view COIs as something evil or undesirable, but > rather as the very real consequence of operating in an interconnected set > of institutions. And once you take that stance, you deal with that > rationally and realistically. > > For example, you accept that companies aren't the only ones with potential > COIs: *all* entities have them. As Ryan May aptly pointed out, the notion > that academic institutions are somehow immune to hidden agendas or other > interests is naive at best... And I say that as someone who has happily > stayed in academia, resisting multiple overtures from industry over the > years, but not out of some quaint notion that academia is a pristine haven > of principled purity. Quite the opposite: in building large and complex > projects, I've seen painfully close how the university/government research > world has its own flavor of the same power, financial and political > ugliness that we attribute to the commercial side. > Thanks for re-emphasizing this. I agree it's O.K. and even necessary to talk about COIs. If I have not been upfront about any possible COIs it's not because of a desire to hide them -- I may not be aware of them myself as I'm still focused on what technically can be better, and how to create a company that can produce as much relevant open source software as possible. I appreciate your helping us get to a better conversational ground. Stefan's and Nathaniel's recent emails have done that as well. Thank you, -Travis -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Wed Sep 23 15:40:29 2015 From: njs at pobox.com (Nathaniel Smith) Date: Wed, 23 Sep 2015 12:40:29 -0700 Subject: [Numpy-discussion] composition of the steering council (was Re: Governance model request) Message-ID: [splitting this out from the thread-o-doom] On Wed, Sep 23, 2015 at 8:47 AM, Chris Barker - NOAA Federal wrote: [snip] > So here is what I think is on the table: > > We have the steering council. Which leaves two questions: > > -How big should it be? > -Who will be on the original, "seed" council? > > Note that as I understand the current draft of the governance doc, > once established, the council itself decides who is on the council. So > at this point we really are ONLY deciding how it's going to start. It > has to be bootstrapped somehow. > > However, that had been contentious enough that it would probably be a > good idea to hash out some guidelines about the council membership. We actually do have some of those already -- dunno if they're perfect, but they exist :-). To make sure everyone's on the same page, here's a condensed summary of what the draft currently says: Joining the council: contributor must have produced "substantial" and "sustained" contributions over at least one year + be voted on by the current council + be interested and willing to serve. (Then there's some language emphasising that "contributions" should *not* be read narrowly as a euphemism for "lines of code".) Leaving the council: Happens by choice of member, or if inactive for one year and can't be contacted, or if inactive for two years. Former members are listed as "emeritus" to recognize their past service. Rejoining the council: aside from their entry on the list of emeritus members, a former-member and a never-member are treated identically in general, and the rules for re-joining are the same as the rules for joining. Proposal for seed council: "everyone who's merged a pull request since Jan 1, 2014". (Actual text is here: http://thread.gmane.org/gmane.comp.python.numeric.general/61106 , see section "Council membership".) We didn't talk much about these -- I think mostly on the theory that the exact details really aren't going to matter much in the end. These specific rules are exactly the rules that Jupyter/IPython use, stolen without changes. My interpretation is that these rules were designed to produce a council consisting of a broad spectrum of contributors who are actively engaged, in tune and up-to-date with the issues currently facing the project, and broadly respected by the broader community. The rationale for doing things this way (if we keep it) would be that the steering council's "primary responsibility is to facilitate the ordinary community-based decision making procedure", so you need people who are actively engaged in community discussions; and, if things break down then the people best positioned to resolve it are the ones who have the best view of what went wrong, understand the personalities involved, and so forth. In practice, I'm sure any council interventions would involve most members deferring to whoever they judge has the most expertise, whether or not that person is on the council -- it's not like they'll ever be making a decision in a vacuum. Regarding the seed council, I just tried to pick an objective criterion and an arbitrary date that seemed generally in keeping with idea of "should be active in the last 1-to-2-years-ish". Fiddling with the exact date in particular makes very little difference -- between pushing it back to 2 years ago today or forward to 1 year ago today, the only thing that changes is whether Pauli makes the list or not. (And Pauli is obviously a great council candidate, though I don't know whether he even wants to be on it.) > Personally, I have no idea how big the council should be. Too big, and > there is no point, consensus is harder to reach the larger the group, > and the main (only?) role of the council is to resolve issues where > consensus has not been reached in the larger community. But what is > too big? I'm a little wary of the idea of capping the council size. Assuming you're pre-filtering for basic competence and good faith (as we are), then the only way making the council smaller helps with decision making is that it arbitrarily throws away the opinion of some dedicated and valuable contributors. Plus then we have to start making judgements like "well, person A has been around for a while but pretty inactive recently, and person B is doing awesome stuff, should we kick person A off the council to let person B on or...?" Judging whether someone is or isn't a "substantial contributor" is fine, we can do that. Having to make a relative judgement of which of two people is the *more* "substantial contributor", though -- that sounds awful. And given how conflict-adverse groups can be, I suspect that capping the council size would in practice just have the effect that it never takes in new blood. (The old effect where "science advances one retirement at a time".) I'd be interested to hear what Jupyter/IPython's experience with this is, though: they currently have a 10 (!) person steering council, I assume they'd love to have more. Having lots of contributors who are active and engaged enough to meet the steering council qualifications is a good problem to have :-). Technically their situation is slightly different because their council runs on regular voting rather than Apache-style consensus voting (a luxury they can afford because they have a BDFL to step in if regular voting ends up disenfranchising the minority), but I sort of get the impression that in practice they just don't vote unless they know it will be unanimous, and it's worked out for them so far. (To understate the case.) > As for make-up of the council, I think we need to expand beyond people > who have recently contributed core code. > > Yes, the council does need to have expertise to make technical > decisions, but if you think about the likely contentious issues like > ABI breakage, a core-code focused view is incomplete. So there should > be representation by: > > Someone(s) with a long history of working with the code -- that > institutional memory of why decisions were made the way they were > could be key. Sure -- though I can't really imagine any way of framing a rule like this that *wouldn't* be satisfied by Chuck + Ralf + Pauli, so my guess is that such a rule would not actually have any effect on the council membership in practice. > Someone(s) that may not have worked on the core code, but is a major > player in the community, perhaps as the leader of a Numpy-dependent > project. This would provide representation for the broad community. Pointing out features of the current draft again for reference: In the current text, the "NumFOCUS subcommittee" does have an external member to provide some oversight. (So mathematically speaking, this means that the subcommittee is not a subset -- go figure. I blame IPython.) But, this oversight is specifically for financial matters only, not technical ones: "This Subcommittee shall NOT make decisions about the direction, scope or technical direction of the Project." Thomas Caswell (one of the leaders of matplotlib) volunteered to be our external member to start. We certainly could ask him to sit on the steering council as well, but honestly my guess is that this would have no effect, either positive or negative. (I know if someone asked me to serve on a hypothetical matplotlib steering council, then I would just... never do anything, because who am I to second-guess matplotlib's actual developers.) It's not like we don't already hear from downstream projects on a regular basis. And if things have gone *so* pear-shaped that we have a situation where the steering council feels they need to issue a ruling, *and simultaneously* the members of the council are so out-of-touch that they don't know or care about the needs of downstream projects, then the situation is unsalvageable and we should fork and start over. But I mean, it probably wouldn't hurt either. I'm not super-wedded to the current text. I just think we should limit how much effort we spend trying to cover every possible situation ahead of time. If we're clever enough to solve a problem now hypothetically, then we're probably also clever enough to solve it later when it becomes concrete. -n -- Nathaniel J. Smith -- http://vorpus.org From matthew.brett at gmail.com Wed Sep 23 15:57:19 2015 From: matthew.brett at gmail.com (Matthew Brett) Date: Wed, 23 Sep 2015 12:57:19 -0700 Subject: [Numpy-discussion] Governance model request In-Reply-To: References: <8941C0D2-17EC-420D-A8F6-CA018144811C@continuum.io> <87mvwft2ar.fsf@berkeley.edu> <370C591F-9711-43E6-BE12-8C3E96946803@continuum.io> Message-ID: Hi, On Wed, Sep 23, 2015 at 1:02 AM, Fernando Perez wrote: [snip] > 1. I hope the discussion can move past the suspicion and innuendo about > Continuum and Travis I'm glad the discussion has become a little more calm now, but I find it difficult not to be annoyed by this statement above. I see a severe reaction to perceived 'suspicion and innuendo', but I see no 'suspicion and innuendo'. Unless you mean that any suggestion of potential conflict of interest is suspicion and innuendo. You imply rudeness and bad faith when you say this. I suppose this must be aimed at me or Stefan or both of us. In any case, I think the accusation is unhelpful and unfair. It makes this discussion more difficult to have in the future. See you, Matthew From fperez.net at gmail.com Wed Sep 23 16:36:35 2015 From: fperez.net at gmail.com (Fernando Perez) Date: Wed, 23 Sep 2015 13:36:35 -0700 Subject: [Numpy-discussion] Governance model request In-Reply-To: References: <8941C0D2-17EC-420D-A8F6-CA018144811C@continuum.io> <87mvwft2ar.fsf@berkeley.edu> <370C591F-9711-43E6-BE12-8C3E96946803@continuum.io> Message-ID: On Wed, Sep 23, 2015 at 12:57 PM, Matthew Brett wrote: > I see a severe reaction to perceived 'suspicion and innuendo', but I > see no 'suspicion and innuendo'. Unless you mean that any suggestion > of potential conflict of interest is suspicion and innuendo. > No, as I said, COIs are absolutely a fact of life, and *should* be talked about, openly and directly. I was referring generically about the tone of this thread, that Ryan described as "bizarre", others as "surpised", "disheartened", etc. > You imply rudeness and bad faith when you say this. I suppose this > must be aimed at me or Stefan or both of us. In any case, I think the > accusation is unhelpful and unfair. It makes this discussion more > difficult to have in the future. > It was sincerely my intention to frame my critique in a specific way that made the discussion *easier* to have in the future, precisely by acknowledging that COIs were part of life, and not only for commercial entities. I was NOT implying that Stefan and you were somehow guilty of anything, I only mentioned your names when I asked Travis not to paint Berkeley folks with a broad brush, that's all. If only the two of you took offense at my wording, in the interest of keeping this already contentious and fraught thread contained, I offer: a) a public apology for my choice of words, since it was certainly not my intent to offend you, and I was not aiming at you personally. b) a suggestion that we discuss it further personally, taking advantage of the fact that we happen to be physically close. If anyone else would like me to answer in public because they feel a slight on my part, I will do so. Best, f -- Fernando Perez (@fperez_org; http://fperez.org) fperez.net-at-gmail: mailing lists only (I ignore this when swamped!) fernando.perez-at-berkeley: contact me here for any direct mail -------------- next part -------------- An HTML attachment was scrubbed... URL: From fperez.net at gmail.com Wed Sep 23 16:38:31 2015 From: fperez.net at gmail.com (Fernando Perez) Date: Wed, 23 Sep 2015 13:38:31 -0700 Subject: [Numpy-discussion] Governance model request In-Reply-To: References: <8941C0D2-17EC-420D-A8F6-CA018144811C@continuum.io> <87mvwft2ar.fsf@berkeley.edu> <370C591F-9711-43E6-BE12-8C3E96946803@continuum.io> Message-ID: On Wed, Sep 23, 2015 at 12:18 PM, Travis Oliphant wrote: > I accept that criticism and apologize for doing that. My *human* side > was coming out, and I was not being fair. In my head, though I was > also trying to illustrate how some seemed to be doing the same thing for > Continuum or other companies. This did not come out very artfully in the > early morning hours. I'm sorry. > Thank you. > BIDS is doing a lot for the community --- the recent DS4DS workshop, > for example, was a spectacularly useful summit --- I hope that many > different write-ups and reports of the event make their way out into the > world. > >> We will put out a full public report, along with all the slides, which we're collecting, etc. It may take us a few days, as we have an upcoming event for BIDS and our two partner sites (UW and NYU), but we'll do it, as we hope this will be useful to the entire community. Cheers, f -- Fernando Perez (@fperez_org; http://fperez.org) fperez.net-at-gmail: mailing lists only (I ignore this when swamped!) fernando.perez-at-berkeley: contact me here for any direct mail -------------- next part -------------- An HTML attachment was scrubbed... URL: From tcaswell at gmail.com Wed Sep 23 16:46:33 2015 From: tcaswell at gmail.com (Thomas Caswell) Date: Wed, 23 Sep 2015 20:46:33 +0000 Subject: [Numpy-discussion] composition of the steering council (was Re: Governance model request) In-Reply-To: References: Message-ID: If I were to be included in the steering council I suspect my main contribution would be to occasionally remind everyone that we are all committed to the success of the open scientific stack. Tom On Wed, Sep 23, 2015 at 3:40 PM Nathaniel Smith wrote: > [splitting this out from the thread-o-doom] > > On Wed, Sep 23, 2015 at 8:47 AM, Chris Barker - NOAA Federal > wrote: > [snip] > > So here is what I think is on the table: > > > > We have the steering council. Which leaves two questions: > > > > -How big should it be? > > -Who will be on the original, "seed" council? > > > > Note that as I understand the current draft of the governance doc, > > once established, the council itself decides who is on the council. So > > at this point we really are ONLY deciding how it's going to start. It > > has to be bootstrapped somehow. > > > > However, that had been contentious enough that it would probably be a > > good idea to hash out some guidelines about the council membership. > > We actually do have some of those already -- dunno if they're perfect, > but they exist :-). To make sure everyone's on the same page, here's a > condensed summary of what the draft currently says: > > Joining the council: contributor must have produced "substantial" and > "sustained" contributions over at least one year + be voted on by the > current council + be interested and willing to serve. (Then there's > some language emphasising that "contributions" should *not* be read > narrowly as a euphemism for "lines of code".) > > Leaving the council: Happens by choice of member, or if inactive for > one year and can't be contacted, or if inactive for two years. Former > members are listed as "emeritus" to recognize their past service. > > Rejoining the council: aside from their entry on the list of emeritus > members, a former-member and a never-member are treated identically in > general, and the rules for re-joining are the same as the rules for > joining. > > Proposal for seed council: "everyone who's merged a pull request since > Jan 1, 2014". > > (Actual text is here: > http://thread.gmane.org/gmane.comp.python.numeric.general/61106 , see > section "Council membership".) > > We didn't talk much about these -- I think mostly on the theory that > the exact details really aren't going to matter much in the end. These > specific rules are exactly the rules that Jupyter/IPython use, stolen > without changes. > > My interpretation is that these rules were designed to produce a > council consisting of a broad spectrum of contributors who are > actively engaged, in tune and up-to-date with the issues currently > facing the project, and broadly respected by the broader community. > The rationale for doing things this way (if we keep it) would be that > the steering council's "primary responsibility is to facilitate the > ordinary community-based decision making procedure", so you need > people who are actively engaged in community discussions; and, if > things break down then the people best positioned to resolve it are > the ones who have the best view of what went wrong, understand the > personalities involved, and so forth. > > In practice, I'm sure any council interventions would involve most > members deferring to whoever they judge has the most expertise, > whether or not that person is on the council -- it's not like they'll > ever be making a decision in a vacuum. > > Regarding the seed council, I just tried to pick an objective > criterion and an arbitrary date that seemed generally in keeping with > idea of "should be active in the last 1-to-2-years-ish". Fiddling with > the exact date in particular makes very little difference -- between > pushing it back to 2 years ago today or forward to 1 year ago today, > the only thing that changes is whether Pauli makes the list or not. > (And Pauli is obviously a great council candidate, though I don't know > whether he even wants to be on it.) > > > Personally, I have no idea how big the council should be. Too big, and > > there is no point, consensus is harder to reach the larger the group, > > and the main (only?) role of the council is to resolve issues where > > consensus has not been reached in the larger community. But what is > > too big? > > I'm a little wary of the idea of capping the council size. Assuming > you're pre-filtering for basic competence and good faith (as we are), > then the only way making the council smaller helps with decision > making is that it arbitrarily throws away the opinion of some > dedicated and valuable contributors. Plus then we have to start making > judgements like "well, person A has been around for a while but pretty > inactive recently, and person B is doing awesome stuff, should we kick > person A off the council to let person B on or...?" Judging whether > someone is or isn't a "substantial contributor" is fine, we can do > that. Having to make a relative judgement of which of two people is > the *more* "substantial contributor", though -- that sounds awful. > > And given how conflict-adverse groups can be, I suspect that capping > the council size would in practice just have the effect that it never > takes in new blood. (The old effect where "science advances one > retirement at a time".) > > I'd be interested to hear what Jupyter/IPython's experience with this > is, though: they currently have a 10 (!) person steering council, I > assume they'd love to have more. Having lots of contributors who are > active and engaged enough to meet the steering council qualifications > is a good problem to have :-). Technically their situation is slightly > different because their council runs on regular voting rather than > Apache-style consensus voting (a luxury they can afford because they > have a BDFL to step in if regular voting ends up disenfranchising the > minority), but I sort of get the impression that in practice they just > don't vote unless they know it will be unanimous, and it's worked out > for them so far. (To understate the case.) > > > As for make-up of the council, I think we need to expand beyond people > > who have recently contributed core code. > > > > Yes, the council does need to have expertise to make technical > > decisions, but if you think about the likely contentious issues like > > ABI breakage, a core-code focused view is incomplete. So there should > > be representation by: > > > > Someone(s) with a long history of working with the code -- that > > institutional memory of why decisions were made the way they were > > could be key. > > Sure -- though I can't really imagine any way of framing a rule like > this that *wouldn't* be satisfied by Chuck + Ralf + Pauli, so my guess > is that such a rule would not actually have any effect on the council > membership in practice. > > > Someone(s) that may not have worked on the core code, but is a major > > player in the community, perhaps as the leader of a Numpy-dependent > > project. This would provide representation for the broad community. > > Pointing out features of the current draft again for reference: In the > current text, the "NumFOCUS subcommittee" does have an external member > to provide some oversight. (So mathematically speaking, this means > that the subcommittee is not a subset -- go figure. I blame IPython.) > But, this oversight is specifically for financial matters only, not > technical ones: "This Subcommittee shall NOT make decisions about the > direction, scope or technical direction of the Project." > > Thomas Caswell (one of the leaders of matplotlib) volunteered to be > our external member to start. We certainly could ask him to sit on the > steering council as well, but honestly my guess is that this would > have no effect, either positive or negative. (I know if someone asked > me to serve on a hypothetical matplotlib steering council, then I > would just... never do anything, because who am I to second-guess > matplotlib's actual developers.) > > It's not like we don't already hear from downstream projects on a > regular basis. And if things have gone *so* pear-shaped that we have a > situation where the steering council feels they need to issue a > ruling, *and simultaneously* the members of the council are so > out-of-touch that they don't know or care about the needs of > downstream projects, then the situation is unsalvageable and we should > fork and start over. > > But I mean, it probably wouldn't hurt either. I'm not super-wedded to > the current text. I just think we should limit how much effort we > spend trying to cover every possible situation ahead of time. If we're > clever enough to solve a problem now hypothetically, then we're > probably also clever enough to solve it later when it becomes > concrete. > > -n > > -- > Nathaniel J. Smith -- http://vorpus.org > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > https://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jaime.frio at gmail.com Wed Sep 23 17:06:08 2015 From: jaime.frio at gmail.com (=?UTF-8?Q?Jaime_Fern=C3=A1ndez_del_R=C3=ADo?=) Date: Wed, 23 Sep 2015 14:06:08 -0700 Subject: [Numpy-discussion] "Become an Open Source Contributor" workshop Message-ID: Apologies for the cross-posting. The Data Science Student Society of the University of California San Diego, or DS3 @ UCSD as they like to call themselves, will be holding biweekly Python themed workshops starting this fall. On the week of October 19th, they will be having yours truly doing a "Become an Open Source Contributor" piece. It will be a shortish event, 60-90 minutes, so my idea was to cover the following: 1. (15 min) An introduction to the Python data science landscape. 2. (30 min) An overview of the GitHub workflow that most (all?) of the projects follow. 3. (30-45 min) A hands on session, where we would make sure everyone gets set up in GitHub, and forks and clones their favorite project. Time and participant willingness permitting, I would like to take advantage of my commit bits, and have some of the participants submit a simple PR, e.g. fixing a documentation typo, to NumPy or SciPy, and hit the green button right there, so that they get to leave as knighted FOSS contributors. And this is what I am hoping to get from you, the community: 1. If anyone in the area would like to get involved, please contact me. I have recruited a couple of volunteers from PySanDiego, but could use more help. 2. I'm also hoping to get some help, especially with the introductory part. Given that the crowd will mostly be university students and some faculty, it would be great if someone who actually knew what they were talking about could deliver a short, 10 minute talk, on Python, data science, and academia. I'm sure we could arrange it to have someone join by video conference. 3. If you have organized anything similar in the past, and have material that I could use to, ahem, draw inspiration from, or recommendations to make, or whatever, I'd love to hear from you. Thanks for reading! Jaime -- (\__/) ( O.o) ( > <) Este es Conejo. Copia a Conejo en tu firma y ay?dale en sus planes de dominaci?n mundial. -------------- next part -------------- An HTML attachment was scrubbed... URL: From travis at continuum.io Wed Sep 23 17:21:27 2015 From: travis at continuum.io (Travis Oliphant) Date: Wed, 23 Sep 2015 16:21:27 -0500 Subject: [Numpy-discussion] composition of the steering council (was Re: Governance model request) In-Reply-To: References: Message-ID: > > Regarding the seed council, I just tried to pick an objective > criterion and an arbitrary date that seemed generally in keeping with > idea of "should be active in the last 1-to-2-years-ish". Fiddling with > the exact date in particular makes very little difference -- between > pushing it back to 2 years ago today or forward to 1 year ago today, > the only thing that changes is whether Pauli makes the list or not. > (And Pauli is obviously a great council candidate, though I don't know > whether he even wants to be on it.) > > > Personally, I have no idea how big the council should be. Too big, and > > there is no point, consensus is harder to reach the larger the group, > > and the main (only?) role of the council is to resolve issues where > > consensus has not been reached in the larger community. But what is > > too big? > > > > As for make-up of the council, I think we need to expand beyond people > > who have recently contributed core code. > > > > Yes, the council does need to have expertise to make technical > > decisions, but if you think about the likely contentious issues like > > ABI breakage, a core-code focused view is incomplete. So there should > > be representation by: > > > > Someone(s) with a long history of working with the code -- that > > institutional memory of why decisions were made the way they were > > could be key. > > Sure -- though I can't really imagine any way of framing a rule like > this that *wouldn't* be satisfied by Chuck + Ralf + Pauli, so my guess > is that such a rule would not actually have any effect on the council > membership in practice. > As the original author of NumPy, I would like to be on the seed council as long as it is larger than 7 people. That is my proposal. I don't need to be a permanent member, but I do believe I have enough history that I can understand issues even if I haven't been working on code directly. I think I do bring history and information that provides all of the history that could be helpful on occasion. In addition, if a matter is important enough to even be brought to the attention of this council, I would like to be involved in the discussion about it. It's a simple change to the text --- basically an explanation that Travis requested to be on the seed council. -Travis -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefanv at berkeley.edu Wed Sep 23 17:31:16 2015 From: stefanv at berkeley.edu (Stefan van der Walt) Date: Wed, 23 Sep 2015 14:31:16 -0700 Subject: [Numpy-discussion] Governance model request In-Reply-To: References: <8941C0D2-17EC-420D-A8F6-CA018144811C@continuum.io> <87mvwft2ar.fsf@berkeley.edu> <370C591F-9711-43E6-BE12-8C3E96946803@continuum.io> Message-ID: <876130u8kb.fsf@berkeley.edu> On 2015-09-23 13:36:35, Fernando Perez wrote: > On Wed, Sep 23, 2015 at 12:57 PM, Matthew Brett > wrote: > >> I see a severe reaction to perceived 'suspicion and innuendo', but I >> see no 'suspicion and innuendo'. Unless you mean that any suggestion >> of potential conflict of interest is suspicion and innuendo. >> > > No, as I said, COIs are absolutely a fact of life, and *should* be talked > about, openly and directly. I was referring generically about the tone of > this thread, that Ryan described as "bizarre", others as "surpised", > "disheartened", etc. What I thought Matthew was saying is that you writing "I hope the discussion can move past the suspicion and innuendo about Continuum and Travis" gives validity to the view that those things are real whereas, in my view, they were constructed in responses by others who didn't accurately summarize the intent of what was being said (and I'm not laying blame to anyone, I could certainly have worded myself more clearly). But, yes, it was disheartening?especially because we all know one another and have hacked together late into the night at SciPy conferences. My experience has always been that we are reasonable people who listen; but perhaps we forget that about one another from time to time. It is important to me that we work towards making this mailing list a safe place for discussion again. Part of that may be to do some maintenance on bridges of trust that seem to have eroded a bit over the years. Perhaps some kind of group bonding activity, such as working on a shared project, would help? ;) > b) a suggestion that we discuss it further personally, taking advantage of > the fact that we happen to be physically close. Sure, I'm happy to discuss the personal side of this offline. St?fan From matthew.brett at gmail.com Wed Sep 23 17:32:27 2015 From: matthew.brett at gmail.com (Matthew Brett) Date: Wed, 23 Sep 2015 14:32:27 -0700 Subject: [Numpy-discussion] Governance model request In-Reply-To: References: <8941C0D2-17EC-420D-A8F6-CA018144811C@continuum.io> <87mvwft2ar.fsf@berkeley.edu> <370C591F-9711-43E6-BE12-8C3E96946803@continuum.io> Message-ID: Hi, On Wed, Sep 23, 2015 at 1:36 PM, Fernando Perez wrote: > On Wed, Sep 23, 2015 at 12:57 PM, Matthew Brett > wrote: >> >> I see a severe reaction to perceived 'suspicion and innuendo', but I >> see no 'suspicion and innuendo'. Unless you mean that any suggestion >> of potential conflict of interest is suspicion and innuendo. > > > No, as I said, COIs are absolutely a fact of life, and *should* be talked > about, openly and directly. I was referring generically about the tone of > this thread, that Ryan described as "bizarre", others as "surpised", > "disheartened", etc. Sure, I said above, it is clearly true the some people reacted very strongly. Did you see any remark made by me or Stefan or anyone else that could reasonably be described as bizarre, surprising or disheartening? Cheers, Matthew From chris.barker at noaa.gov Wed Sep 23 17:38:26 2015 From: chris.barker at noaa.gov (Chris Barker) Date: Wed, 23 Sep 2015 14:38:26 -0700 Subject: [Numpy-discussion] composition of the steering council (was Re: Governance model request) In-Reply-To: References: Message-ID: On Wed, Sep 23, 2015 at 12:40 PM, Nathaniel Smith wrote: > > However, that had been contentious enough that it would probably be a > > good idea to hash out some guidelines about the council membership. > > We actually do have some of those already -- dunno if they're perfect, > but they exist :-). know -- I guess I meant "expand the guidelines..." .. but thanks for putting it all here. Also, I think there was a slight disconnect between the guidelines and the proposed "seed" council, as it was only the seed, no biggie, but I think folks got the wrong impression. > Then there's > some language emphasising that "contributions" should *not* be read > narrowly as a euphemism for "lines of code".) > yeah, this got lost a bit somehow... > Leaving the council: Happens by choice of member, or if inactive for > one year and can't be contacted, or if inactive for two years. somehow this seem to have been interpreted as "inactive in contributing to code", rather than "inactive in council communication/activity" -- but two years seems pleanty long to me. Proposal for seed council: "everyone who's merged a pull request since > Jan 1, 2014". > I think it matters little how the council is seeded, as it can grow and change from there -- but maybe folks will feel better if we define a few other parameters -- after all, this wouldn't get you anyone that had made large non-code contributions to the project. We didn't talk much about these -- I think mostly on the theory that > the exact details really aren't going to matter much in the end. agreed -- it's kind of a self-regulating Oligarchy... > My interpretation is that these rules were designed to produce a > council consisting of a broad spectrum of contributors who are > actively engaged, in tune and up-to-date with the issues currently > facing the project, and broadly respected by the broader community. > yup -- that's what we want :-) > I'm a little wary of the idea of capping the council size. ... Judging whether > someone is or isn't a "substantial contributor" is fine, we can do > that. Having to make a relative judgement of which of two people is > the *more* "substantial contributor", though -- that sounds awful. > indeed it does -- tough problem. > And given how conflict-adverse groups can be, I suspect that capping > the council size would in practice just have the effect that it never > takes in new blood. Another very valid concern. Started me thinking about term limits -- which is an even worse idea :-) > I'd be interested to hear what Jupyter/IPython's experience with this > is, though: they currently have a 10 (!) person steering council, me too. Though as you mention, not having to rely on consensus makes a larger group more tenable. > > Someone(s) with a long history of working with the code -- that > > institutional memory of why decisions were made the way they were > > could be key. > > Sure -- though I can't really imagine any way of framing a rule like > this that *wouldn't* be satisfied by Chuck + Ralf + Pauli, so my guess > is that such a rule would not actually have any effect on the council > membership in practice. > well, as someone that has been around the community, and relied on numpy (and numarray, and Numeric before that) for I think 17 years, maybe I have a different idea of what "history" is. Though I honestly don't remember when the folks you names joined up... To be specific, someone with a memory of the Numeric -> numarray semi-disaster would be nice to have around... Though, as you've stated, plenty of opportunity for wise old souls to be consulted and contribute even if not actually on the council. > > Someone(s) that may not have worked on the core code, but is a major > > player in the community, perhaps as the leader of a Numpy-dependent > > project. This would provide representation for the broad community. > > Pointing out features of the current draft again for reference: In the > current text, the "NumFOCUS subcommittee" does have an external member > to provide some oversight. (So mathematically speaking, this means > that the subcommittee is not a subset -- go figure. I blame IPython.) > But, this oversight is specifically for financial matters only, not > technical ones: "This Subcommittee shall NOT make decisions about the > direction, scope or technical direction of the Project." > right -- I was specifically thinking someone from an external project to help with the touch technical decisions, like breaking teh ABI, what kin do missing value implementation to use, etc. These issues are very, very important to the third party packages. But again, plenty of opportunity to contribute to the discussion anyway -- which I guess leads the the core issue: if all goes well, it will matter not a wit who is on the council! Thomas Caswell (one of the leaders of matplotlib) volunteered to be > our external member to start. We certainly could ask him to sit on the > steering council as well, but honestly my guess is that this would > have no effect, either positive or negative. He's a great guy -- so I"m going to go positive :-) > (I know if someone asked > me to serve on a hypothetical matplotlib steering council, then I > would just... never do anything, because who am I to second-guess > matplotlib's actual developers.) > That's an asymmetrical relationship -- MPL (and many others) depend on numpy -- decisions (particularly the contentious ones where this is all relevant) made about numpy drive changes on MPL L-- not the other way around. > But I mean, it probably wouldn't hurt either. I'm not super-wedded to > the current text. I just think we should limit how much effort we > spend trying to cover every possible situation ahead of time. If we're > clever enough to solve a problem now hypothetically, then we're > probably also clever enough to solve it later when it becomes > concrete. > and maybe it's easier -- there will be a council to do it :-) -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.barker at noaa.gov Wed Sep 23 17:42:30 2015 From: chris.barker at noaa.gov (Chris Barker) Date: Wed, 23 Sep 2015 14:42:30 -0700 Subject: [Numpy-discussion] composition of the steering council (was Re: Governance model request) In-Reply-To: References: Message-ID: On Wed, Sep 23, 2015 at 2:21 PM, Travis Oliphant wrote: > As the original author of NumPy, I would like to be on the seed council as > long as it is larger than 7 people. That is my proposal. > Or the seed council could invite Travis to join as its first order of business :-) Actually, maybe that's a way to handle it -- declare that the first order of business for teh seed council is to expand the council. I'd still like some guidelines (suggestions) for history and at least one major dependent-on-numpy rep. Travis would certainly meet the history requirement -- and maybe the other, too. :-) > It's a simple change to the text --- basically an explanation that Travis > requested to be on the seed council. > I'd rather the final draft of the document didn't name names, but no biggie. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From travis at continuum.io Wed Sep 23 17:43:27 2015 From: travis at continuum.io (Travis Oliphant) Date: Wed, 23 Sep 2015 16:43:27 -0500 Subject: [Numpy-discussion] Steering Committee Size In-Reply-To: References: Message-ID: On Wed, Sep 23, 2015 at 3:25 AM, Sebastian Berg wrote: > Hi, > > Trying to figure out at least a bit from the discussions. While I am > happy with the draft, I wonder if someone has some insights about some > questions: > > 1. How large crowds have examples of working well with apache style voting? > I don't have experience with this, other than Python mailing list where it is fine. > 2. How large do we expect numpy steering council to be (I have always > thought about 10). > I don't know. I don't think this is set as far as I'm aware. 3. More on opinions, how large does the community feel is too large (so > that we should maybe elect people). > > And to maybe more a discussion point, does the community feel that those > who would be/are affectivly now in the Steering Council do not sufficiently > represent old time contributers who were not active in the past year(s). > As I mentioned before, I am happy to serve on the initial seed council to help transition more fully to this style of governance. -Travis -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.barker at noaa.gov Wed Sep 23 17:44:29 2015 From: chris.barker at noaa.gov (Chris Barker) Date: Wed, 23 Sep 2015 14:44:29 -0700 Subject: [Numpy-discussion] Governance model request In-Reply-To: <876130u8kb.fsf@berkeley.edu> References: <8941C0D2-17EC-420D-A8F6-CA018144811C@continuum.io> <87mvwft2ar.fsf@berkeley.edu> <370C591F-9711-43E6-BE12-8C3E96946803@continuum.io> <876130u8kb.fsf@berkeley.edu> Message-ID: On Wed, Sep 23, 2015 at 2:31 PM, Stefan van der Walt wrote: > > b) a suggestion that we discuss it further personally, taking advantage > of > > the fact that we happen to be physically close. > > Sure, I'm happy to discuss the personal side of this offline. > Hey! I want to have a beer with you folks too! Maybe time for a trip back to the old stomping grounds.... -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Wed Sep 23 17:49:28 2015 From: njs at pobox.com (Nathaniel Smith) Date: Wed, 23 Sep 2015 14:49:28 -0700 Subject: [Numpy-discussion] interpretation of the draft governance document (was Re: Governance model request) In-Reply-To: References: Message-ID: On Wed, Sep 23, 2015 at 11:59 AM, Travis Oliphant wrote: > Hi Nathaniel, > > Thanks for the clarifications. Is the governance document committed to > the repository? I keep looking for it and have a hard time finding it --- > I think I read it last in an email. > Indeed, sorry -- getting it into the repo has been on my todo list, though somewhat on hold given the uncertainty the last few days. For reference in the mean time, the original posting is here: http://thread.gmane.org/gmane.comp.python.numeric.general/61106 and I'll paste it again at the bottom of this email. (The only difference between the version in this email and the previous should be the addition of Jaime to fill in the empty slot on the tentative NumFOCUS subcommittee.) > > In this way, I could make Pull Requests to the governance document if there > are concrete suggestions for change, and then have them reviewed in the > standard way. > > While this makes sense, I think the "standard way" for reviewing substantive changes (i.e., not just simple wording changes) here would be via discussion on the mailing list :-). We're still trying to figure out the best way to balance github versus mailing list... it sounds like things are at least converging somewhat, though, so I'll work on getting a PR together soon in any case. -n ------ The purpose of this document is to formalize the governance process used by the NumPy project in both ordinary and extraordinary situations, and to clarify how decisions are made and how the various elements of our community interact, including the relationship between open source collaborative development and work that may be funded by for-profit or non-profit entities. Summary ======= NumPy is a community-owned and community-run project. To the maximum extent possible, decisions about project direction are made by community consensus (but note that "consensus" here has a somewhat technical meaning that might not match everyone's expectations -- see below). Some members of the community additionally contribute by serving on the NumPy steering council, where they are responsible for facilitating the establishment of community consensus, for stewarding project resources, and -- in extreme cases -- for making project decisions if the normal community-based process breaks down. The Project =========== The NumPy Project (The Project) is an open source software project affiliated with the 501(c)3 NumFocus Foundation. The goal of The Project is to develop open source software for array-based computing in Python, and in particular the `numpy` package, along with related software such as `f2py` and the NumPy Sphinx extensions. The Software developed by The Project is released under the BSD (or similar) open source license, developed openly and hosted on public GitHub repositories under the `numpy` GitHub organization. The Project is developed by a team of distributed developers, called Contributors. Contributors are individuals who have contributed code, documentation, designs or other work to the Project. Anyone can be a Contributor. Contributors can be affiliated with any legal entity or none. Contributors participate in the project by submitting, reviewing and discussing GitHub Pull Requests and Issues and participating in open and public Project discussions on GitHub, mailing lists, and other channels. The foundation of Project participation is openness and transparency. Here is a list of the current Contributors to the main NumPy repository: [ https://github.com/numpy/numpy/graphs/contributors](https://github.com/numpy/numpy/graphs/contributors) The Project Community consists of all Contributors and Users of the Project. Contributors work on behalf of and are responsible to the larger Project Community and we strive to keep the barrier between Contributors and Users as low as possible. The Project is formally affiliated with the 501(c)3 NumFOCUS Foundation ([ http://numfocus.org](http://numfocus.org)), which serves as its fiscal sponsor, may hold project trademarks and other intellectual property, helps manage project donations and acts as a parent legal entity. NumFOCUS is the only legal entity that has a formal relationship with the project (see Institutional Partners section below). Governance ========== This section describes the governance and leadership model of The Project. The foundations of Project governance are: - Openness & Transparency - Active Contribution - Institutional Neutrality Consensus-based decision making by the community ------------------------------------------------ Normally, all project decisions will be made by consensus of all interested Contributors. The primary goal of this approach is to ensure that the people who are most affected by and involved in any given change can contribute their knowledge in the confidence that their voices will be heard, because thoughtful review from a broad community is the best mechanism we know of for creating high-quality software. The mechanism we use to accomplish this goal may be unfamiliar for those who are not experienced with the cultural norms around free/open-source software development. We provide a summary here, and highly recommend that all Contributors additionally read [Chapter 4: Social and Political Infrastructure]( http://producingoss.com/en/producingoss.html#social-infrastructure) of Karl Fogel's classic *Producing Open Source Software*, and in particular the section on [Consensus-based Democracy]( http://producingoss.com/en/producingoss.html#consensus-democracy), for a more detailed discussion. In this context, consensus does *not* require: - that we wait to solicit everybody's opinion on every change, - that we ever hold a vote on anything, - or that everybody is happy or agrees with every decision. For us, what consensus means is that we entrust *everyone* with the right to veto any change if they feel it necessary. While this may sound like a recipe for obstruction and pain, this is not what happens. Instead, we find that most people take this responsibility seriously, and only invoke their veto when they judge that a serious problem is being ignored, and that their veto is necessary to protect the project. And in practice, it turns out that such vetoes are almost never formally invoked, because their mere possibility ensures that Contributors are motivated from the start to find some solution that everyone can live with -- thus accomplishing our goal of ensuring that all interested perspectives are taken into account. How do we know when consensus has been achieved? In principle, this is rather difficult, since consensus is defined by the absence of vetos, which requires us to somehow prove a negative. In practice, we use a combination of our best judgement (e.g., a simple and uncontroversial bug fix posted on GitHub and reviewed by a core developer is probably fine) and best efforts (e.g., all substantive API changes must be posted to the mailing list in order to give the broader community a chance to catch any problems and suggest improvements; we assume that anyone who cares enough about NumPy to invoke their veto right should be on the mailing list). If no-one bothers to comment on the mailing list after a few days, then it's probably fine. And worst case, if a change is more controversial than expected, or a crucial critique is delayed because someone was on vacation, then it's no big deal: we apologize for misjudging the situation, [back up, and sort things out]( http://producingoss.com/en/producingoss.html#version-control-relaxation). If one does need to invoke a formal veto, then it should consist of: - an unambiguous statement that a veto is being invoked, - an explanation of why it is being invoked, and - a description of what conditions (if any) would convince the vetoer to withdraw their veto. If all proposals for resolving some issue are vetoed, then the status quo wins by default. In the worst case, if a Contributor is genuinely misusing their veto in an obstructive fashion to the detriment of the project, then they can be ejected from the project by consensus of the Steering Council -- see below. Steering Council ---------------- The Project will have a Steering Council that consists of Project Contributors who have produced contributions that are substantial in quality and quantity, and sustained over at least one year. The overall role of the Council is to ensure, with input from the Community, the long-term well-being of the project, both technically and as a community. During the everyday project activities, council members participate in all discussions, code review and other project activities as peers with all other Contributors and the Community. In these everyday activities, Council Members do not have any special power or privilege through their membership on the Council. However, it is expected that because of the quality and quantity of their contributions and their expert knowledge of the Project Software and Services that Council Members will provide useful guidance, both technical and in terms of project direction, to potentially less experienced contributors. The Steering Council and its Members play a special role in certain situations. In particular, the Council may, if necessary: - Make decisions about the overall scope, vision and direction of the project. - Make decisions about strategic collaborations with other organizations or individuals. - Make decisions about specific technical issues, features, bugs and pull requests. They are the primary mechanism of guiding the code review process and merging pull requests. - Make decisions about the Services that are run by The Project and manage those Services for the benefit of the Project and Community. - Update policy documents such as this one. - Make decisions when regular community discussion doesn?t produce consensus on an issue in a reasonable time frame. However, the Council's primary responsibility is to facilitate the ordinary community-based decision making procedure described above. If we ever have to step in and formally override the community for the health of the Project, then we will do so, but we will consider reaching this point to indicate a failure in our leadership. ### Council decision making If it becomes necessary for the Steering Council to produce a formal decision, then they will use a form of the [Apache Foundation voting process](https://www.apache.org/foundation/voting.html). This is a formalized version of consensus, in which +1 votes indicate agreement, -1 votes are vetoes (and must be accompanied with a rationale, as above), and one can also vote fractionally (e.g. -0.5, +0.5) if one wishes to express an opinion without registering a full veto. These numeric votes are also often used informally as a way of getting a general sense of people's feelings on some issue, and should not normally be taken as formal votes. A formal vote only occurs if explicitly declared, and if this does occur then the vote should be held open for long enough to give all interested Council Members a chance to respond -- at least one week. In practice, we anticipate that for most Steering Council decisions (e.g., voting in new members) a more informal process will suffice. ### Council membership To become eligible to join the Steering Council, an individual must be a Project Contributor who has produced contributions that are substantial in quality and quantity, and sustained over at least one year. Potential Council Members are nominated by existing Council members and voted upon by the existing Council after asking if the potential Member is interested and willing to serve in that capacity. The Council will be initially formed from the set of existing Core Developers who, as of late 2015, have been significantly active over the last year. When considering potential Members, the Council will look at candidates with a comprehensive view of their contributions. This will include but is not limited to code, code review, infrastructure work, mailing list and chat participation, community help/building, education and outreach, design work, etc. We are deliberately not setting arbitrary quantitative metrics (like ?100 commits in this repo?) to avoid encouraging behavior that plays to the metrics rather than the project?s overall well-being. We want to encourage a diverse array of backgrounds, viewpoints and talents in our team, which is why we explicitly do not define code as the sole metric on which council membership will be evaluated. If a Council member becomes inactive in the project for a period of one year, they will be considered for removal from the Council. Before removal, inactive Member will be approached to see if they plan on returning to active participation. If not they will be removed immediately upon a Council vote. If they plan on returning to active participation soon, they will be given a grace period of one year. If they don?t return to active participation within that time period they will be removed by vote of the Council without further grace period. All former Council members can be considered for membership again at any time in the future, like any other Project Contributor. Retired Council members will be listed on the project website, acknowledging the period during which they were active in the Council. The Council reserves the right to eject current Members, if they are deemed to be actively harmful to the project?s well-being, and attempts at communication and conflict resolution have failed. This requires the consensus of the remaining Members. [We also have to decide on the initial membership for the Council. While the above text makes pains to distinguish between "committer" and "Council Member", in the past we've pretty much treated them as the same. So to keep things simple and deterministic, I propose that we seed the Council with everyone who has reviewed/merged a pull request since Jan 1, 2014, and move those who haven't used their commit bit in >1.5 years to the emeritus list. Based on the output of git log --grep="^Merge pull request" --since 2014-01-01 | grep Author: | sort -u I believe this would give us an initial Steering Council of: Sebastian Berg, Jaime Fern?ndez del R?o, Ralf Gommers, Alex Griffing, Charles Harris, Nathaniel Smith, Julian Taylor, and Pauli Virtanen (assuming everyone on that list is interested/willing to serve).] ### Conflict of interest It is expected that the Council Members will be employed at a wide range of companies, universities and non-profit organizations. Because of this, it is possible that Members will have conflict of interests. Such conflict of interests include, but are not limited to: - Financial interests, such as investments, employment or contracting work, outside of The Project that may influence their work on The Project. - Access to proprietary information of their employer that could potentially leak into their work with the Project. All members of the Council shall disclose to the rest of the Council any conflict of interest they may have. Members with a conflict of interest in a particular issue may participate in Council discussions on that issue, but must recuse themselves from voting on the issue. ### Private communications of the Council Unless specifically required, all Council discussions and activities will be public and done in collaboration and discussion with the Project Contributors and Community. The Council will have a private mailing list that will be used sparingly and only when a specific matter requires privacy. When private communications and decisions are needed, the Council will do its best to summarize those to the Community after eliding personal/private/sensitive information that should not be posted to the public internet. ### Subcommittees The Council can create subcommittees that provide leadership and guidance for specific aspects of the project. Like the Council as a whole, subcommittees should conduct their business in an open and public manner unless privacy is specifically called for. Private subcommittee communications should happen on the main private mailing list of the Council unless specifically called for. ### NumFOCUS Subcommittee The Council will maintain one narrowly focused subcommittee to manage its interactions with NumFOCUS. - The NumFOCUS Subcommittee is comprised of 5 persons who manage project funding that comes through NumFOCUS. It is expected that these funds will be spent in a manner that is consistent with the non-profit mission of NumFOCUS and the direction of the Project as determined by the full Council. - This Subcommittee shall NOT make decisions about the direction, scope or technical direction of the Project. - This Subcommittee will have 5 members, 4 of whom will be current Council Members and 1 of whom will be external to the Steering Council. No more than 2 Subcommitee Members can report to one person through employment or contracting work (including the reportee, i.e. the reportee + 1 is the max). This avoids effective majorities resting on one person. [Initially, the NumFOCUS subcommittee will consist of: Chuck Harris, Ralf Gommers, Nathaniel Smith, Jaime Fern?ndez del R?o as internal members, and Thomas Caswell as external member.] Institutional Partners and Funding ================================== The Steering Council are the primary leadership for the project. No outside institution, individual or legal entity has the ability to own, control, usurp or influence the project other than by participating in the Project as Contributors and Council Members. However, because institutions can be an important funding mechanism for the project, it is important to formally acknowledge institutional participation in the project. These are Institutional Partners. An Institutional Contributor is any individual Project Contributor who contributes to the project as part of their official duties at an Institutional Partner. Likewise, an Institutional Council Member is any Project Steering Council Member who contributes to the project as part of their official duties at an Institutional Partner. With these definitions, an Institutional Partner is any recognized legal entity in the United States or elsewhere that employs at least 1 Institutional Contributor of Institutional Council Member. Institutional Partners can be for-profit or non-profit entities. Institutions become eligible to become an Institutional Partner by employing individuals who actively contribute to The Project as part of their official duties. To state this another way, the only way for a Partner to influence the project is by actively contributing to the open development of the project, in equal terms to any other member of the community of Contributors and Council Members. Merely using Project Software in institutional context does not allow an entity to become an Institutional Partner. Financial gifts do not enable an entity to become an Institutional Partner. Once an institution becomes eligible for Institutional Partnership, the Steering Council must nominate and approve the Partnership. If an existing Institutional Partner no longer has a contributing employee, they will be given a 1 year grace period for remaining employees to begin contributing. An Institutional Partner is free to pursue funding for their work on The Project through any legal means. This could involve a non-profit organization raising money from private foundations and donors or a for-profit company building proprietary products and services that leverage Project Software and Services. Funding acquired by Institutional Partners to work on The Project is called Institutional Funding. However, no funding obtained by an Institutional Partner can override the Steering Council. If a Partner has funding to do NumPy work and the Council decides to not pursue that work as a project, the Partner is free to pursue it on their own. However in this situation, that part of the Partner?s work will not be under the NumPy umbrella and cannot use the Project trademarks in a way that suggests a formal relationship. Institutional Partner benefits are: - Acknowledgement on the NumPy websites, in talks and T-shirts. - Ability to acknowledge their own funding sources on the NumPy websites, in talks and T-shirts. - Ability to influence the project through the participation of their Council Member. - Council Members invited to NumPy Developer Meetings. Existing Institutional Partners: - UC Berkeley (Nathaniel Smith) Acknowledgements ================ Substantial portions of this document were ~~inspired~~ stolen wholesale from the Jupyter/IPython project's governance document, [IPEP 29]( https://github.com/ipython/ipython/wiki/IPEP-29:-Project-Governance). -- Nathaniel J. Smith -- http://vorpus.org -------------- next part -------------- An HTML attachment was scrubbed... URL: From fperez.net at gmail.com Wed Sep 23 17:50:53 2015 From: fperez.net at gmail.com (Fernando Perez) Date: Wed, 23 Sep 2015 14:50:53 -0700 Subject: [Numpy-discussion] Governance model request In-Reply-To: <876130u8kb.fsf@berkeley.edu> References: <8941C0D2-17EC-420D-A8F6-CA018144811C@continuum.io> <87mvwft2ar.fsf@berkeley.edu> <370C591F-9711-43E6-BE12-8C3E96946803@continuum.io> <876130u8kb.fsf@berkeley.edu> Message-ID: On Wed, Sep 23, 2015 at 2:31 PM, Stefan van der Walt wrote: > Perhaps some kind of group bonding activity, such as working on a shared > project, would help? ;) > Indeed, there's a bunch of fresh 2x4s, screws and bolts with your name on them ;) Cheers, f -- Fernando Perez (@fperez_org; http://fperez.org) fperez.net-at-gmail: mailing lists only (I ignore this when swamped!) fernando.perez-at-berkeley: contact me here for any direct mail -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Wed Sep 23 17:51:50 2015 From: charlesr.harris at gmail.com (Charles R Harris) Date: Wed, 23 Sep 2015 15:51:50 -0600 Subject: [Numpy-discussion] Governance model request In-Reply-To: References: <8941C0D2-17EC-420D-A8F6-CA018144811C@continuum.io> <87mvwft2ar.fsf@berkeley.edu> <370C591F-9711-43E6-BE12-8C3E96946803@continuum.io> Message-ID: On Wed, Sep 23, 2015 at 3:32 PM, Matthew Brett wrote: > Hi, > > On Wed, Sep 23, 2015 at 1:36 PM, Fernando Perez > wrote: > > On Wed, Sep 23, 2015 at 12:57 PM, Matthew Brett > > > wrote: > >> > >> I see a severe reaction to perceived 'suspicion and innuendo', but I > >> see no 'suspicion and innuendo'. Unless you mean that any suggestion > >> of potential conflict of interest is suspicion and innuendo. > > > > > > No, as I said, COIs are absolutely a fact of life, and *should* be talked > > about, openly and directly. I was referring generically about the tone > of > > this thread, that Ryan described as "bizarre", others as "surpised", > > "disheartened", etc. > > Sure, I said above, it is clearly true the some people reacted very > strongly. > > Did you see any remark made by me or Stefan or anyone else that could > reasonably be described as bizarre, surprising or disheartening? > Stephan and Matthew, go to your rooms ;) Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefanv at berkeley.edu Wed Sep 23 17:57:53 2015 From: stefanv at berkeley.edu (Stefan van der Walt) Date: Wed, 23 Sep 2015 14:57:53 -0700 Subject: [Numpy-discussion] "Become an Open Source Contributor" workshop In-Reply-To: References: Message-ID: <8737y4u7by.fsf@berkeley.edu> Hi Jaime On 2015-09-23 14:06:08, Jaime Fern?ndez del R?o wrote: > 3. If you have organized anything similar in the past, and have material > that I could use to, ahem, draw inspiration from, or recommendations to > make, or whatever, I'd love to hear from you. Here's the new developer workflow page for scikit-image, I'm sure many other projects have similar ones: http://scikit-image.org/docs/stable/contribute.html Perhaps you can harvest some ideas. Also, a beginner's summary to git workflow: http://rogerdudler.github.io/git-guide/ It's a lot to teach in only an hour or two, so if I were teaching I'd keep it simple (basic) and clear (to make sure the students can "keep it in their heads"), and to make sure they have a clear avenue for questions when they get stuck after the class. St?fan From nathan12343 at gmail.com Wed Sep 23 18:04:44 2015 From: nathan12343 at gmail.com (Nathan Goldbaum) Date: Wed, 23 Sep 2015 17:04:44 -0500 Subject: [Numpy-discussion] "Become an Open Source Contributor" workshop In-Reply-To: <8737y4u7by.fsf@berkeley.edu> References: <8737y4u7by.fsf@berkeley.edu> Message-ID: If you are going to do work at a terminal, I'd suggest using a library like doitlive (http://doitlive.readthedocs.org/en/latest/) so you can't make mistakes while still making it look like you are actually typing everything at a terminal. You will also be able to share your exact terminal sessions with the students if they want to come back to it later. On Wed, Sep 23, 2015 at 4:57 PM, Stefan van der Walt wrote: > Hi Jaime > > On 2015-09-23 14:06:08, Jaime Fern?ndez del R?o > wrote: > > 3. If you have organized anything similar in the past, and have > material > > that I could use to, ahem, draw inspiration from, or recommendations > to > > make, or whatever, I'd love to hear from you. > > Here's the new developer workflow page for scikit-image, I'm sure many > other projects have similar ones: > > http://scikit-image.org/docs/stable/contribute.html > > Perhaps you can harvest some ideas. Also, a beginner's summary to git > workflow: > > http://rogerdudler.github.io/git-guide/ > > It's a lot to teach in only an hour or two, so if I were teaching I'd > keep it simple (basic) and clear (to make sure the students can "keep it > in their heads"), and to make sure they have a clear avenue for > questions when they get stuck after the class. > > St?fan > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > https://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From fperez.net at gmail.com Wed Sep 23 18:08:50 2015 From: fperez.net at gmail.com (Fernando Perez) Date: Wed, 23 Sep 2015 15:08:50 -0700 Subject: [Numpy-discussion] Governance model request In-Reply-To: References: <8941C0D2-17EC-420D-A8F6-CA018144811C@continuum.io> <87mvwft2ar.fsf@berkeley.edu> <370C591F-9711-43E6-BE12-8C3E96946803@continuum.io> Message-ID: On Wed, Sep 23, 2015 at 2:32 PM, Matthew Brett wrote: > Did you see any remark made by me or Stefan or anyone else that could > reasonably be described as bizarre, surprising or disheartening? > As I said already, I wasn't referring to you personally, but to the tone of the thread. It was the taste that the thread left in my mouth after reading tens of very long emails, and I guess I wasn't the only one, since multiple folks used such adjectives. But it was a poor choice of words on my part. And no, I didn't make a quote-by-quote analysis of the thread, I'm sorry. One last time, it was *not* a personal reference to you: the only reason I mentioned your names was because of the Berkeley clarification regarding BIDS that I asked of Travis, that's all. If that comment hadn't been made, I would not have made any mention whatsoever of anyone in particular. I apologize for not foreseeing that this would have made you feel singled out, in retrospect, I should have. In my mind, it was the opposite, as I felt that you had every right to express whatever opinions you have speaking for yourselves, independent of your affiliations, and I was simply asking Travis to separate individuals from institutions. But I should have realized that calling anyone out by name in a context like this is a bad idea regardless. Cheers, f -- Fernando Perez (@fperez_org; http://fperez.org) fperez.net-at-gmail: mailing lists only (I ignore this when swamped!) fernando.perez-at-berkeley: contact me here for any direct mail -------------- next part -------------- An HTML attachment was scrubbed... URL: From travis at continuum.io Wed Sep 23 18:12:03 2015 From: travis at continuum.io (Travis Oliphant) Date: Wed, 23 Sep 2015 17:12:03 -0500 Subject: [Numpy-discussion] interpretation of the draft governance document (was Re: Governance model request) In-Reply-To: References: Message-ID: > > > Here is a list of the current Contributors to the main NumPy repository: > > [ > https://github.com/numpy/numpy/graphs/contributors](https://github.com/numpy/numpy/graphs/contributors) > > One of the problems with this list is that my contributions to the project are extremely under-represented because the large majority of my commitment of code happened in 2005 to 2006 before github was used. So, using this as a list of the contributors is quite misleading --- and there are a lot of people now looking only at lists like this one and it might confuse them why I care so much. So, if you are going to make this list public in a governance document like this, then I think some acknowledgement of the source of the original code and the contributors to that needs to also be made --- or you could just also point to the THANKS document which lists people up to about 2008. Between 2008 and 2010 we will lose contributions, still and this can be acknowledged. > Consensus-based decision making by the community > ------------------------------------------------ > > Normally, all project decisions will be made by consensus of all > interested Contributors. The primary goal of this approach is to ensure > that the people who are most affected by and involved in any given change > can contribute their knowledge in the confidence that their voices will be > heard, because thoughtful review from a broad community is the best > mechanism we know of for creating high-quality software. > > The mechanism we use to accomplish this goal may be unfamiliar for those > who are not experienced with the cultural norms around free/open-source > software development. We provide a summary here, and highly recommend that > all Contributors additionally read [Chapter 4: Social and Political > Infrastructure]( > http://producingoss.com/en/producingoss.html#social-infrastructure) of > Karl Fogel's classic *Producing Open Source Software*, and in particular > the section on [Consensus-based Democracy]( > http://producingoss.com/en/producingoss.html#consensus-democracy), for a > more detailed discussion. > > In this context, consensus does *not* require: > > - that we wait to solicit everybody's opinion on every change, > - that we ever hold a vote on anything, > - or that everybody is happy or agrees with every decision. > > For us, what consensus means is that we entrust *everyone* with the right > to veto any change if they feel it necessary. While this may sound like a > recipe for obstruction and pain, this is not what happens. Instead, we find > that most people take this responsibility seriously, and only invoke their > veto when they judge that a serious problem is being ignored, and that > their veto is necessary to protect the project. And in practice, it turns > out that such vetoes are almost never formally invoked, because their mere > possibility ensures that Contributors are motivated from the start to find > some solution that everyone can live with -- thus accomplishing our goal of > ensuring that all interested perspectives are taken into account. > > How do we know when consensus has been achieved? In principle, this is > rather difficult, since consensus is defined by the absence of vetos, which > requires us to somehow prove a negative. In practice, we use a combination > of our best judgement (e.g., a simple and uncontroversial bug fix posted on > GitHub and reviewed by a core developer is probably fine) and best efforts > (e.g., all substantive API changes must be posted to the mailing list in > order to give the broader community a chance to catch any problems and > suggest improvements; we assume that anyone who cares enough about NumPy to > invoke their veto right should be on the mailing list). If no-one bothers > to comment on the mailing list after a few days, then it's probably fine. > And worst case, if a change is more controversial than expected, or a > crucial critique is delayed because someone was on vacation, then it's no > big deal: we apologize for misjudging the situation, [back up, and sort > things out]( > http://producingoss.com/en/producingoss.html#version-control-relaxation). > > If one does need to invoke a formal veto, then it should consist of: > > - an unambiguous statement that a veto is being invoked, > - an explanation of why it is being invoked, and > - a description of what conditions (if any) would convince the vetoer to > withdraw their veto. > > If all proposals for resolving some issue are vetoed, then the status quo > wins by default. > > In the worst case, if a Contributor is genuinely misusing their veto in an > obstructive fashion to the detriment of the project, then they can be > ejected from the project by consensus of the Steering Council -- see below. > > Steering Council > ---------------- > > The Project will have a Steering Council that consists of Project > Contributors who have produced contributions that are substantial in > quality and quantity, and sustained over at least one year. The overall > role of the Council is to ensure, with input from the Community, the > long-term well-being of the project, both technically and as a community. > > During the everyday project activities, council members participate in all > discussions, code review and other project activities as peers with all > other Contributors and the Community. In these everyday activities, Council > Members do not have any special power or privilege through their membership > on the Council. However, it is expected that because of the quality and > quantity of their contributions and their expert knowledge of the Project > Software and Services that Council Members will provide useful guidance, > both technical and in terms of project direction, to potentially less > experienced contributors. > > The Steering Council and its Members play a special role in certain > situations. In particular, the Council may, if necessary: > > - Make decisions about the overall scope, vision and direction of the > project. > - Make decisions about strategic collaborations with other organizations > or individuals. > - Make decisions about specific technical issues, features, bugs and > pull requests. They are the primary mechanism of guiding the code review > process and merging pull requests. > - Make decisions about the Services that are run by The Project and > manage those Services for the benefit of the Project and Community. > - Update policy documents such as this one. > - Make decisions when regular community discussion doesn?t produce > consensus on an issue in a reasonable time frame. > > However, the Council's primary responsibility is to facilitate the > ordinary community-based decision making procedure described above. If we > ever have to step in and formally override the community for the health of > the Project, then we will do so, but we will consider reaching this point > to indicate a failure in our leadership. > > ### Council decision making > > If it becomes necessary for the Steering Council to produce a formal > decision, then they will use a form of the [Apache Foundation voting > process](https://www.apache.org/foundation/voting.html). This is a > formalized version of consensus, in which +1 votes indicate agreement, -1 > votes are vetoes (and must be accompanied with a rationale, as above), and > one can also vote fractionally (e.g. -0.5, +0.5) if one wishes to express > an opinion without registering a full veto. These numeric votes are also > often used informally as a way of getting a general sense of people's > feelings on some issue, and should not normally be taken as formal votes. A > formal vote only occurs if explicitly declared, and if this does occur then > the vote should be held open for long enough to give all interested Council > Members a chance to respond -- at least one week. > > In practice, we anticipate that for most Steering Council decisions (e.g., > voting in new members) a more informal process will suffice. > > ### Council membership > > To become eligible to join the Steering Council, an individual must be a > Project Contributor who has produced contributions that are substantial in > quality and quantity, and sustained over at least one year. Potential > Council Members are nominated by existing Council members and voted upon by > the existing Council after asking if the potential Member is interested and > willing to serve in that capacity. The Council will be initially formed > from the set of existing Core Developers who, as of late 2015, have been > significantly active over the last year. > > Concretely, I'm asking to be included in this initial council so a simple "along with Travis Oliphant who is the original author of NumPy". If other long-time contributors to the code-base also want to be on this initial seed council, I think it would make sense as well. > When considering potential Members, the Council will look at candidates > with a comprehensive view of their contributions. This will include but is > not limited to code, code review, infrastructure work, mailing list and > chat participation, community help/building, education and outreach, design > work, etc. We are deliberately not setting arbitrary quantitative metrics > (like ?100 commits in this repo?) to avoid encouraging behavior that plays > to the metrics rather than the project?s overall well-being. We want to > encourage a diverse array of backgrounds, viewpoints and talents in our > team, which is why we explicitly do not define code as the sole metric on > which council membership will be evaluated. > > If a Council member becomes inactive in the project for a period of one > year, they will be considered for removal from the Council. Before removal, > inactive Member will be approached to see if they plan on returning to > active participation. If not they will be removed immediately upon a > Council vote. If they plan on returning to active participation soon, they > will be given a grace period of one year. If they don?t return to active > participation within that time period they will be removed by vote of the > Council without further grace period. All former Council members can be > considered for membership again at any time in the future, like any other > Project Contributor. Retired Council members will be listed on the project > website, acknowledging the period during which they were active in the > Council. > I would just remove quite a bit of this language and just say that members who are inactive on the Council may be asked to step down if they are not planning on returning to active participation. > The Council reserves the right to eject current Members, if they are > deemed to be actively harmful to the project?s well-being, and attempts at > communication and conflict resolution have failed. This requires the > consensus of the remaining Members. > > [We also have to decide on the initial membership for the Council. While > the above text makes pains to distinguish between "committer" and "Council > Member", in the past we've pretty much treated them as the same. So to keep > things simple and deterministic, I propose that we seed the Council with > everyone who has reviewed/merged a pull request since Jan 1, 2014, and move > those who haven't used their commit bit in >1.5 years to the emeritus list. > Based on the output of > > git log --grep="^Merge pull request" --since 2014-01-01 | grep Author: > | sort -u > > I believe this would give us an initial Steering Council of: Sebastian > Berg, Jaime Fern?ndez del R?o, Ralf Gommers, Alex Griffing, Charles Harris, > Nathaniel Smith, Julian Taylor, and Pauli Virtanen (assuming everyone on > that list is interested/willing to serve).] > I would ask to be on this initial council by having the rule include "original contributors of the code" which would basically include Robert Kern and Pearu Peterson in addition to Chuck Harris (though Pearu's main contribution was numpy.distutils and f2py and he may not be interested any longer in the rest of it). Thanks, -Travis -------------- next part -------------- An HTML attachment was scrubbed... URL: From travis at continuum.io Wed Sep 23 18:19:49 2015 From: travis at continuum.io (Travis Oliphant) Date: Wed, 23 Sep 2015 17:19:49 -0500 Subject: [Numpy-discussion] Governance model request In-Reply-To: References: <8941C0D2-17EC-420D-A8F6-CA018144811C@continuum.io> <87mvwft2ar.fsf@berkeley.edu> <370C591F-9711-43E6-BE12-8C3E96946803@continuum.io> Message-ID: > > > One last time, it was *not* a personal reference to you: the only reason I > mentioned your names was because of the Berkeley clarification regarding > BIDS that I asked of Travis, that's all. If that comment hadn't been made, > I would not have made any mention whatsoever of anyone in particular. I > apologize for not foreseeing that this would have made you feel singled > out, in retrospect, I should have. > > In my mind, it was the opposite, as I felt that you had every right to > express whatever opinions you have speaking for yourselves, independent of > your affiliations, and I was simply asking Travis to separate individuals > from institutions. But I should have realized that calling anyone out by > name in a context like this is a bad idea regardless. > > This was my fault for not being more careful in my words. I felt multiple things when I wrote my emails that led to incorrectly chosen words --- but mostly I was feeling unappreciated, attacked, and accused. I'm sure now that was not intended --- but there have been mis-understandings. I expect they will happen again. I know if we listen to each other and trust that while we may see the world differently and have different framings of solutions --- we can work to coordinate on an important technical activity together. In retrospect, my initial email requesting inclusion on the seed council could have been worded better (as there were multiple things conflated together). I am responding to the actual text of the governance document in the other thread so as to clarify what my proposal actually is in the context of that document. -Travis -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.barker at noaa.gov Wed Sep 23 18:23:46 2015 From: chris.barker at noaa.gov (Chris Barker) Date: Wed, 23 Sep 2015 15:23:46 -0700 Subject: [Numpy-discussion] "Become an Open Source Contributor" workshop In-Reply-To: References: <8737y4u7by.fsf@berkeley.edu> Message-ID: On Wed, Sep 23, 2015 at 3:04 PM, Nathan Goldbaum wrote: > If you are going to do work at a terminal, I'd suggest using a library > like doitlive (http://doitlive.readthedocs.org/en/latest/) so you can't > make mistakes while still making it look like you are actually typing > everything at a terminal. > This is pretty cool! And, I think perfect for a presentation or the like. But for teaching, I find that one of the valuable things I do is make mistakes at the command line (or iPython notebook, or editor...), and then show the students how I discover and recover from the mistake... just a thought.... -Chris > You will also be able to share your exact terminal sessions with the > students if they want to come back to it later. > > On Wed, Sep 23, 2015 at 4:57 PM, Stefan van der Walt > wrote: > >> Hi Jaime >> >> On 2015-09-23 14:06:08, Jaime Fern?ndez del R?o >> wrote: >> > 3. If you have organized anything similar in the past, and have >> material >> > that I could use to, ahem, draw inspiration from, or recommendations >> to >> > make, or whatever, I'd love to hear from you. >> >> Here's the new developer workflow page for scikit-image, I'm sure many >> other projects have similar ones: >> >> http://scikit-image.org/docs/stable/contribute.html >> >> Perhaps you can harvest some ideas. Also, a beginner's summary to git >> workflow: >> >> http://rogerdudler.github.io/git-guide/ >> >> It's a lot to teach in only an hour or two, so if I were teaching I'd >> keep it simple (basic) and clear (to make sure the students can "keep it >> in their heads"), and to make sure they have a clear avenue for >> questions when they get stuck after the class. >> >> St?fan >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> https://mail.scipy.org/mailman/listinfo/numpy-discussion >> > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > https://mail.scipy.org/mailman/listinfo/numpy-discussion > > -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From shoyer at gmail.com Wed Sep 23 18:24:30 2015 From: shoyer at gmail.com (Stephan Hoyer) Date: Wed, 23 Sep 2015 15:24:30 -0700 (PDT) Subject: [Numpy-discussion] interpretation of the draft governance document (was Re: Governance model request) In-Reply-To: References: Message-ID: <1443047070277.e04bfd96@Nodemailer> Travis -- have you included all your email addresses in your GitHub profile? When I type git shortlog -ne, I see 2063 commits from your Continuum address that seem to be missing from the contributors page on github. Generally speaking, the git logs tend to be more reliable for these counts. On Wed, Sep 23, 2015 at 3:12 PM, Travis Oliphant wrote: >> >> >> Here is a list of the current Contributors to the main NumPy repository: >> >> [ >> https://github.com/numpy/numpy/graphs/contributors](https://github.com/numpy/numpy/graphs/contributors) >> >> > One of the problems with this list is that my contributions to the project > are extremely under-represented because the large majority of my commitment > of code happened in 2005 to 2006 before github was used. So, using > this as a list of the contributors is quite misleading --- and there are a > lot of people now looking only at lists like this one and it might confuse > them why I care so much. So, if you are going to make this list public > in a governance document like this, then I think some acknowledgement of > the source of the original code and the contributors to that needs to also > be made --- or you could just also point to the THANKS document which lists > people up to about 2008. Between 2008 and 2010 we will lose > contributions, still and this can be acknowledged. >> Consensus-based decision making by the community >> ------------------------------------------------ >> >> Normally, all project decisions will be made by consensus of all >> interested Contributors. The primary goal of this approach is to ensure >> that the people who are most affected by and involved in any given change >> can contribute their knowledge in the confidence that their voices will be >> heard, because thoughtful review from a broad community is the best >> mechanism we know of for creating high-quality software. >> >> The mechanism we use to accomplish this goal may be unfamiliar for those >> who are not experienced with the cultural norms around free/open-source >> software development. We provide a summary here, and highly recommend that >> all Contributors additionally read [Chapter 4: Social and Political >> Infrastructure]( >> http://producingoss.com/en/producingoss.html#social-infrastructure) of >> Karl Fogel's classic *Producing Open Source Software*, and in particular >> the section on [Consensus-based Democracy]( >> http://producingoss.com/en/producingoss.html#consensus-democracy), for a >> more detailed discussion. >> >> In this context, consensus does *not* require: >> >> - that we wait to solicit everybody's opinion on every change, >> - that we ever hold a vote on anything, >> - or that everybody is happy or agrees with every decision. >> >> For us, what consensus means is that we entrust *everyone* with the right >> to veto any change if they feel it necessary. While this may sound like a >> recipe for obstruction and pain, this is not what happens. Instead, we find >> that most people take this responsibility seriously, and only invoke their >> veto when they judge that a serious problem is being ignored, and that >> their veto is necessary to protect the project. And in practice, it turns >> out that such vetoes are almost never formally invoked, because their mere >> possibility ensures that Contributors are motivated from the start to find >> some solution that everyone can live with -- thus accomplishing our goal of >> ensuring that all interested perspectives are taken into account. >> >> How do we know when consensus has been achieved? In principle, this is >> rather difficult, since consensus is defined by the absence of vetos, which >> requires us to somehow prove a negative. In practice, we use a combination >> of our best judgement (e.g., a simple and uncontroversial bug fix posted on >> GitHub and reviewed by a core developer is probably fine) and best efforts >> (e.g., all substantive API changes must be posted to the mailing list in >> order to give the broader community a chance to catch any problems and >> suggest improvements; we assume that anyone who cares enough about NumPy to >> invoke their veto right should be on the mailing list). If no-one bothers >> to comment on the mailing list after a few days, then it's probably fine. >> And worst case, if a change is more controversial than expected, or a >> crucial critique is delayed because someone was on vacation, then it's no >> big deal: we apologize for misjudging the situation, [back up, and sort >> things out]( >> http://producingoss.com/en/producingoss.html#version-control-relaxation). >> >> If one does need to invoke a formal veto, then it should consist of: >> >> - an unambiguous statement that a veto is being invoked, >> - an explanation of why it is being invoked, and >> - a description of what conditions (if any) would convince the vetoer to >> withdraw their veto. >> >> If all proposals for resolving some issue are vetoed, then the status quo >> wins by default. >> >> In the worst case, if a Contributor is genuinely misusing their veto in an >> obstructive fashion to the detriment of the project, then they can be >> ejected from the project by consensus of the Steering Council -- see below. >> >> Steering Council >> ---------------- >> >> The Project will have a Steering Council that consists of Project >> Contributors who have produced contributions that are substantial in >> quality and quantity, and sustained over at least one year. The overall >> role of the Council is to ensure, with input from the Community, the >> long-term well-being of the project, both technically and as a community. >> >> During the everyday project activities, council members participate in all >> discussions, code review and other project activities as peers with all >> other Contributors and the Community. In these everyday activities, Council >> Members do not have any special power or privilege through their membership >> on the Council. However, it is expected that because of the quality and >> quantity of their contributions and their expert knowledge of the Project >> Software and Services that Council Members will provide useful guidance, >> both technical and in terms of project direction, to potentially less >> experienced contributors. >> >> The Steering Council and its Members play a special role in certain >> situations. In particular, the Council may, if necessary: >> >> - Make decisions about the overall scope, vision and direction of the >> project. >> - Make decisions about strategic collaborations with other organizations >> or individuals. >> - Make decisions about specific technical issues, features, bugs and >> pull requests. They are the primary mechanism of guiding the code review >> process and merging pull requests. >> - Make decisions about the Services that are run by The Project and >> manage those Services for the benefit of the Project and Community. >> - Update policy documents such as this one. >> - Make decisions when regular community discussion doesn?t produce >> consensus on an issue in a reasonable time frame. >> >> However, the Council's primary responsibility is to facilitate the >> ordinary community-based decision making procedure described above. If we >> ever have to step in and formally override the community for the health of >> the Project, then we will do so, but we will consider reaching this point >> to indicate a failure in our leadership. >> >> ### Council decision making >> >> If it becomes necessary for the Steering Council to produce a formal >> decision, then they will use a form of the [Apache Foundation voting >> process](https://www.apache.org/foundation/voting.html). This is a >> formalized version of consensus, in which +1 votes indicate agreement, -1 >> votes are vetoes (and must be accompanied with a rationale, as above), and >> one can also vote fractionally (e.g. -0.5, +0.5) if one wishes to express >> an opinion without registering a full veto. These numeric votes are also >> often used informally as a way of getting a general sense of people's >> feelings on some issue, and should not normally be taken as formal votes. A >> formal vote only occurs if explicitly declared, and if this does occur then >> the vote should be held open for long enough to give all interested Council >> Members a chance to respond -- at least one week. >> >> In practice, we anticipate that for most Steering Council decisions (e.g., >> voting in new members) a more informal process will suffice. >> >> ### Council membership >> >> To become eligible to join the Steering Council, an individual must be a >> Project Contributor who has produced contributions that are substantial in >> quality and quantity, and sustained over at least one year. Potential >> Council Members are nominated by existing Council members and voted upon by >> the existing Council after asking if the potential Member is interested and >> willing to serve in that capacity. The Council will be initially formed >> from the set of existing Core Developers who, as of late 2015, have been >> significantly active over the last year. >> >> > Concretely, I'm asking to be included in this initial council so a simple > "along with Travis Oliphant who is the original author of NumPy". If > other long-time contributors to the code-base also want to be on this > initial seed council, I think it would make sense as well. >> When considering potential Members, the Council will look at candidates >> with a comprehensive view of their contributions. This will include but is >> not limited to code, code review, infrastructure work, mailing list and >> chat participation, community help/building, education and outreach, design >> work, etc. We are deliberately not setting arbitrary quantitative metrics >> (like ?100 commits in this repo?) to avoid encouraging behavior that plays >> to the metrics rather than the project?s overall well-being. We want to >> encourage a diverse array of backgrounds, viewpoints and talents in our >> team, which is why we explicitly do not define code as the sole metric on >> which council membership will be evaluated. >> >> If a Council member becomes inactive in the project for a period of one >> year, they will be considered for removal from the Council. Before removal, >> inactive Member will be approached to see if they plan on returning to >> active participation. If not they will be removed immediately upon a >> Council vote. If they plan on returning to active participation soon, they >> will be given a grace period of one year. If they don?t return to active >> participation within that time period they will be removed by vote of the >> Council without further grace period. All former Council members can be >> considered for membership again at any time in the future, like any other >> Project Contributor. Retired Council members will be listed on the project >> website, acknowledging the period during which they were active in the >> Council. >> > I would just remove quite a bit of this language and just say that members > who are inactive on the Council may be asked to step down if they are not > planning on returning to active participation. >> The Council reserves the right to eject current Members, if they are >> deemed to be actively harmful to the project?s well-being, and attempts at >> communication and conflict resolution have failed. This requires the >> consensus of the remaining Members. >> >> [We also have to decide on the initial membership for the Council. While >> the above text makes pains to distinguish between "committer" and "Council >> Member", in the past we've pretty much treated them as the same. So to keep >> things simple and deterministic, I propose that we seed the Council with >> everyone who has reviewed/merged a pull request since Jan 1, 2014, and move >> those who haven't used their commit bit in >1.5 years to the emeritus list. >> Based on the output of >> >> git log --grep="^Merge pull request" --since 2014-01-01 | grep Author: >> | sort -u >> >> I believe this would give us an initial Steering Council of: Sebastian >> Berg, Jaime Fern?ndez del R?o, Ralf Gommers, Alex Griffing, Charles Harris, >> Nathaniel Smith, Julian Taylor, and Pauli Virtanen (assuming everyone on >> that list is interested/willing to serve).] >> > I would ask to be on this initial council by having the rule include > "original contributors of the code" which would basically include Robert > Kern and Pearu Peterson in addition to Chuck Harris (though Pearu's main > contribution was numpy.distutils and f2py and he may not be interested any > longer in the rest of it). > Thanks, > -Travis -------------- next part -------------- An HTML attachment was scrubbed... URL: From matthew.brett at gmail.com Wed Sep 23 18:29:17 2015 From: matthew.brett at gmail.com (Matthew Brett) Date: Wed, 23 Sep 2015 15:29:17 -0700 Subject: [Numpy-discussion] Governance model request In-Reply-To: References: <8941C0D2-17EC-420D-A8F6-CA018144811C@continuum.io> <87mvwft2ar.fsf@berkeley.edu> <370C591F-9711-43E6-BE12-8C3E96946803@continuum.io> Message-ID: On Wed, Sep 23, 2015 at 3:08 PM, Fernando Perez wrote: > > On Wed, Sep 23, 2015 at 2:32 PM, Matthew Brett > wrote: >> >> Did you see any remark made by me or Stefan or anyone else that could >> reasonably be described as bizarre, surprising or disheartening? > > > As I said already, I wasn't referring to you personally, but to the tone of > the thread. It was the taste that the thread left in my mouth after reading > tens of very long emails, and I guess I wasn't the only one, since multiple > folks used such adjectives. But it was a poor choice of words on my part. > > One last time, it was *not* a personal reference to you: the only reason I > mentioned your names was because of the Berkeley clarification regarding > BIDS that I asked of Travis, that's all. You know me well, I not reacting to personal offense, otherwise I would email you privately. See you, Matthew From fperez.net at gmail.com Wed Sep 23 18:37:45 2015 From: fperez.net at gmail.com (Fernando Perez) Date: Wed, 23 Sep 2015 15:37:45 -0700 Subject: [Numpy-discussion] Governance model request In-Reply-To: References: <8941C0D2-17EC-420D-A8F6-CA018144811C@continuum.io> <87mvwft2ar.fsf@berkeley.edu> <370C591F-9711-43E6-BE12-8C3E96946803@continuum.io> Message-ID: On Wed, Sep 23, 2015 at 3:19 PM, Travis Oliphant wrote: > This was my fault for not being more careful in my words. I felt > multiple things when I wrote my emails that led to incorrectly chosen words > --- but mostly I was feeling unappreciated, attacked, and accused. I'm > sure now that was not intended --- but there have been mis-understandings. > I expect they will happen again. I know if we listen to each other and > trust that while we may see the world differently and have different > framings of solutions --- we can work to coordinate on an important > technical activity together. > > In retrospect, my initial email requesting inclusion on the seed council > could have been worded better (as there were multiple things conflated > together). I am responding to the actual text of the governance document > in the other thread so as to clarify what my proposal actually is in the > context of that document. > No worries, I wasn't digging it up further, really! Just clarifying for other reasons, not digging into you :) Glad to see the other thread proceeding further, let's let this one die as peacefully as it can... Best, f -- Fernando Perez (@fperez_org; http://fperez.org) fperez.net-at-gmail: mailing lists only (I ignore this when swamped!) fernando.perez-at-berkeley: contact me here for any direct mail -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Wed Sep 23 19:08:34 2015 From: charlesr.harris at gmail.com (Charles R Harris) Date: Wed, 23 Sep 2015 17:08:34 -0600 Subject: [Numpy-discussion] composition of the steering council (was Re: Governance model request) In-Reply-To: References: Message-ID: On Wed, Sep 23, 2015 at 3:21 PM, Travis Oliphant wrote: > > >> Regarding the seed council, I just tried to pick an objective >> criterion and an arbitrary date that seemed generally in keeping with >> idea of "should be active in the last 1-to-2-years-ish". Fiddling with >> the exact date in particular makes very little difference -- between >> pushing it back to 2 years ago today or forward to 1 year ago today, >> the only thing that changes is whether Pauli makes the list or not. >> (And Pauli is obviously a great council candidate, though I don't know >> whether he even wants to be on it.) >> >> > Personally, I have no idea how big the council should be. Too big, and >> > there is no point, consensus is harder to reach the larger the group, >> > and the main (only?) role of the council is to resolve issues where >> > consensus has not been reached in the larger community. But what is >> > too big? >> >> >> > As for make-up of the council, I think we need to expand beyond people >> > who have recently contributed core code. >> > >> > Yes, the council does need to have expertise to make technical >> > decisions, but if you think about the likely contentious issues like >> > ABI breakage, a core-code focused view is incomplete. So there should >> > be representation by: >> > >> > Someone(s) with a long history of working with the code -- that >> > institutional memory of why decisions were made the way they were >> > could be key. >> >> Sure -- though I can't really imagine any way of framing a rule like >> this that *wouldn't* be satisfied by Chuck + Ralf + Pauli, so my guess >> is that such a rule would not actually have any effect on the council >> membership in practice. >> > > As the original author of NumPy, I would like to be on the seed council as > long as it is larger than 7 people. That is my proposal. I don't need > to be a permanent member, but I do believe I have enough history that I can > understand issues even if I haven't been working on code directly. > > I think I do bring history and information that provides all of the > history that could be helpful on occasion. In addition, if a matter is > important enough to even be brought to the attention of this council, I > would like to be involved in the discussion about it. > > It's a simple change to the text --- basically an explanation that Travis > requested to be on the seed council. > I too would like you to be a member. We could either write it into the text in recognition of your status as the Numpy creator, or it could be the first order of business. I would only ask that you give yourself some time to become familiar with how things work and the people involved in the current community. It has been some years since you have been active in code development. Chuck > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Wed Sep 23 19:19:19 2015 From: charlesr.harris at gmail.com (Charles R Harris) Date: Wed, 23 Sep 2015 17:19:19 -0600 Subject: [Numpy-discussion] composition of the steering council (was Re: Governance model request) In-Reply-To: References: Message-ID: On Wed, Sep 23, 2015 at 3:42 PM, Chris Barker wrote: > On Wed, Sep 23, 2015 at 2:21 PM, Travis Oliphant > wrote: > > >> As the original author of NumPy, I would like to be on the seed council >> as long as it is larger than 7 people. That is my proposal. >> > > Or the seed council could invite Travis to join as its first order of > business :-) > > Actually, maybe that's a way to handle it -- declare that the first order > of business for teh seed council is to expand the council. > Perhaps we should specify a yearly meeting to review the past year and nominate people for commit rights and council membership. Long term, we might also want to start removing commit rights, perhaps by adding a team category on github with restricted rights -- committer emeritus, so to speak. > I'd still like some guidelines (suggestions) for history and at least one > major dependent-on-numpy rep. Travis would certainly meet the history > requirement -- and maybe the other, too. :-) > > >> It's a simple change to the text --- basically an explanation that Travis >> requested to be on the seed council. >> > > I'd rather the final draft of the document didn't name names, but no > biggie. > Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Wed Sep 23 19:30:09 2015 From: charlesr.harris at gmail.com (Charles R Harris) Date: Wed, 23 Sep 2015 17:30:09 -0600 Subject: [Numpy-discussion] "Become an Open Source Contributor" workshop In-Reply-To: References: Message-ID: On Wed, Sep 23, 2015 at 3:06 PM, Jaime Fern?ndez del R?o < jaime.frio at gmail.com> wrote: > Apologies for the cross-posting. > > The Data Science Student Society of the University of California San > Diego, or DS3 @ UCSD as they like to call themselves, will be holding > biweekly Python themed workshops starting this fall. On the week of > October 19th, they will be having yours truly doing a "Become an Open > Source Contributor" piece. It will be a shortish event, 60-90 minutes, so > my idea was to cover the following: > > 1. (15 min) An introduction to the Python data science landscape. > 2. (30 min) An overview of the GitHub workflow that most (all?) of the > projects follow. > 3. (30-45 min) A hands on session, where we would make sure everyone > gets set up in GitHub, and forks and clones their favorite project. Time > and participant willingness permitting, I would like to take advantage of > my commit bits, and have some of the participants submit a simple PR, e.g. > fixing a documentation typo, to NumPy or SciPy, and hit the green button > right there, so that they get to leave as knighted FOSS contributors. > > You could create a `foolscrap` repo in the numpy project on github and use that. That would probably be useful for other people as well. Chuck > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jaime.frio at gmail.com Wed Sep 23 19:41:47 2015 From: jaime.frio at gmail.com (=?UTF-8?Q?Jaime_Fern=C3=A1ndez_del_R=C3=ADo?=) Date: Wed, 23 Sep 2015 16:41:47 -0700 Subject: [Numpy-discussion] composition of the steering council (was Re: Governance model request) In-Reply-To: References: Message-ID: For what it is worth, I'm +1 on including any of the "founding fathers" (and uncles!) that wish to be included in the committee, or council, or however we ended up calling it. I'm also OK with singling out Travis as creator of NumPy in the wording of the document. Especially since the prerogative of members is not to **VOTE**, but to **VETO**, I don't see adding more people having any effect on the individual power of members, so my +1 stands regardless of what the total member count is. Jaime On Wed, Sep 23, 2015 at 4:08 PM, Charles R Harris wrote: > > > On Wed, Sep 23, 2015 at 3:21 PM, Travis Oliphant > wrote: > >> >> >>> Regarding the seed council, I just tried to pick an objective >>> criterion and an arbitrary date that seemed generally in keeping with >>> idea of "should be active in the last 1-to-2-years-ish". Fiddling with >>> the exact date in particular makes very little difference -- between >>> pushing it back to 2 years ago today or forward to 1 year ago today, >>> the only thing that changes is whether Pauli makes the list or not. >>> (And Pauli is obviously a great council candidate, though I don't know >>> whether he even wants to be on it.) >>> >>> > Personally, I have no idea how big the council should be. Too big, and >>> > there is no point, consensus is harder to reach the larger the group, >>> > and the main (only?) role of the council is to resolve issues where >>> > consensus has not been reached in the larger community. But what is >>> > too big? >>> >>> >>> > As for make-up of the council, I think we need to expand beyond people >>> > who have recently contributed core code. >>> > >>> > Yes, the council does need to have expertise to make technical >>> > decisions, but if you think about the likely contentious issues like >>> > ABI breakage, a core-code focused view is incomplete. So there should >>> > be representation by: >>> > >>> > Someone(s) with a long history of working with the code -- that >>> > institutional memory of why decisions were made the way they were >>> > could be key. >>> >>> Sure -- though I can't really imagine any way of framing a rule like >>> this that *wouldn't* be satisfied by Chuck + Ralf + Pauli, so my guess >>> is that such a rule would not actually have any effect on the council >>> membership in practice. >>> >> >> As the original author of NumPy, I would like to be on the seed council >> as long as it is larger than 7 people. That is my proposal. I don't >> need to be a permanent member, but I do believe I have enough history that >> I can understand issues even if I haven't been working on code directly. >> >> >> I think I do bring history and information that provides all of the >> history that could be helpful on occasion. In addition, if a matter is >> important enough to even be brought to the attention of this council, I >> would like to be involved in the discussion about it. >> >> It's a simple change to the text --- basically an explanation that Travis >> requested to be on the seed council. >> > > I too would like you to be a member. We could either write it into the > text in recognition of your status as the Numpy creator, or it could be the > first order of business. I would only ask that you give yourself some time > to become familiar with how things work and the people involved in the > current community. It has been some years since you have been active in > code development. > > Chuck > >> >> >> > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > https://mail.scipy.org/mailman/listinfo/numpy-discussion > > -- (\__/) ( O.o) ( > <) Este es Conejo. Copia a Conejo en tu firma y ay?dale en sus planes de dominaci?n mundial. -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Wed Sep 23 19:47:18 2015 From: charlesr.harris at gmail.com (Charles R Harris) Date: Wed, 23 Sep 2015 17:47:18 -0600 Subject: [Numpy-discussion] composition of the steering council (was Re: Governance model request) In-Reply-To: References: Message-ID: On Wed, Sep 23, 2015 at 5:41 PM, Jaime Fern?ndez del R?o < jaime.frio at gmail.com> wrote: > For what it is worth, I'm +1 on including any of the "founding fathers" > (and uncles!) that wish to be included in the committee, or council, or > however we ended up calling it. I'm also OK with singling out Travis as > creator of NumPy in the wording of the document. > I think "Founding Entities" would be the politically correct term ;) > > Especially since the prerogative of members is not to **VOTE**, but to > **VETO**, I don't see adding more people having any effect on the > individual power of members, so my +1 stands regardless of what the total > member count is. > > Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefanv at berkeley.edu Wed Sep 23 20:10:20 2015 From: stefanv at berkeley.edu (Stefan van der Walt) Date: Wed, 23 Sep 2015 17:10:20 -0700 Subject: [Numpy-discussion] ANN: numtraits v0.2 In-Reply-To: References: Message-ID: <87twqksmmr.fsf@berkeley.edu> Hi Thomas On 2015-09-23 09:39:00, Thomas Robitaille wrote: > We have released a small experimental package called numtraits that > builds on top of the traitlets package and provides a NumericalTrait > class that can be used to validate properties such as: This looks great! At the moment, a pip install tries to install a different version of NumPy, even though I already have the development version on my tree. In scikit-image, we use something like the following in setup.py to prevent that from happening: # Do not try and upgrade larger dependencies INSTALL_REQUIRES = ['numpy', 'traitlets'] try: __import__('numpy') INSTALL_REQUIRES = ['traitlets'] except ImportError: pass St?fan From njs at pobox.com Wed Sep 23 20:16:14 2015 From: njs at pobox.com (Nathaniel Smith) Date: Wed, 23 Sep 2015 17:16:14 -0700 Subject: [Numpy-discussion] ANN: numtraits v0.2 In-Reply-To: <87twqksmmr.fsf@berkeley.edu> References: <87twqksmmr.fsf@berkeley.edu> Message-ID: On Wed, Sep 23, 2015 at 5:10 PM, Stefan van der Walt wrote: > Hi Thomas > > On 2015-09-23 09:39:00, Thomas Robitaille wrote: >> We have released a small experimental package called numtraits that >> builds on top of the traitlets package and provides a NumericalTrait >> class that can be used to validate properties such as: > > This looks great! At the moment, a pip install tries to install a > different version of NumPy, even though I already have the development > version on my tree. FYI the alternative solution is to fix your local numpy install to give pip an accurate picture of what you have installed. The key thing is to make sure you have a .egg-info / .dist-info for your numpy -- that's what pip looks for to figure out what's installed. (python setupegg.py egg_info will generate one IIRC). -n -- Nathaniel J. Smith -- http://vorpus.org From matthew.brett at gmail.com Wed Sep 23 20:24:00 2015 From: matthew.brett at gmail.com (Matthew Brett) Date: Wed, 23 Sep 2015 17:24:00 -0700 Subject: [Numpy-discussion] Plan for 1.9.4 release Message-ID: Hi, I'm afraid our very helpful testers have found a problem with f2py and Numpy 1.9.3 (and earlier): https://github.com/numpy/numpy/issues/6348 I think we need a numpy 1.9.4 soon therefore. I started a branch / PR here : https://github.com/numpy/numpy/pull/6350 Any other fixes? Carl - do you want to get your mingw-w64 PR in for this? Cheers, Matthew From stefanv at berkeley.edu Wed Sep 23 20:28:25 2015 From: stefanv at berkeley.edu (Stefan van der Walt) Date: Wed, 23 Sep 2015 17:28:25 -0700 Subject: [Numpy-discussion] ANN: numtraits v0.2 In-Reply-To: References: <87twqksmmr.fsf@berkeley.edu> Message-ID: <87r3loslsm.fsf@berkeley.edu> On 2015-09-23 17:16:14, Nathaniel Smith wrote: >> This looks great! At the moment, a pip install tries to install a >> different version of NumPy, even though I already have the development >> version on my tree. > > FYI the alternative solution is to fix your local numpy install to > give pip an accurate picture of what you have installed. The key thing > is to make sure you have a .egg-info / .dist-info for your numpy -- > that's what pip looks for to figure out what's installed. (python > setupegg.py egg_info will generate one IIRC). Looks like 'pip install -e .' in the numpy directory also fixed it. Good to know, thanks. St?fan From fperez.net at gmail.com Wed Sep 23 20:30:50 2015 From: fperez.net at gmail.com (Fernando Perez) Date: Wed, 23 Sep 2015 17:30:50 -0700 Subject: [Numpy-discussion] Plan for 1.9.4 release In-Reply-To: References: Message-ID: Chance this will be merged into 1.10? I know there's an rc floating out there for 1.10, and this sounds like an actual bug... Would be great to have a solid 1.10 for the 3.5 crowd :) On Wed, Sep 23, 2015 at 5:24 PM, Matthew Brett wrote: > Hi, > > I'm afraid our very helpful testers have found a problem with f2py and > Numpy 1.9.3 (and earlier): > > https://github.com/numpy/numpy/issues/6348 > > I think we need a numpy 1.9.4 soon therefore. > > I started a branch / PR here : https://github.com/numpy/numpy/pull/6350 > > Any other fixes? Carl - do you want to get your mingw-w64 PR in for this? > > Cheers, > > Matthew > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > https://mail.scipy.org/mailman/listinfo/numpy-discussion > -- Fernando Perez (@fperez_org; http://fperez.org) fperez.net-at-gmail: mailing lists only (I ignore this when swamped!) fernando.perez-at-berkeley: contact me here for any direct mail -------------- next part -------------- An HTML attachment was scrubbed... URL: From matthew.brett at gmail.com Wed Sep 23 20:43:36 2015 From: matthew.brett at gmail.com (Matthew Brett) Date: Wed, 23 Sep 2015 17:43:36 -0700 Subject: [Numpy-discussion] Plan for 1.9.4 release In-Reply-To: References: Message-ID: Hi, On Wed, Sep 23, 2015 at 5:30 PM, Fernando Perez wrote: > Chance this will be merged into 1.10? I know there's an rc floating out > there for 1.10, and this sounds like an actual bug... Would be great to > have a solid 1.10 for the 3.5 crowd :) 1.10 already contains the f2py fixes, luckily... Matthew From travis at continuum.io Wed Sep 23 20:47:36 2015 From: travis at continuum.io (Travis Oliphant) Date: Wed, 23 Sep 2015 19:47:36 -0500 Subject: [Numpy-discussion] composition of the steering council (was Re: Governance model request) In-Reply-To: References: Message-ID: Thanks Chuck, I have been paying some attention, actually --- just not speaking up until there is a major difference of opinion (like the governance document...). I guess I don't feel like I've completely lost track of "how things work" --- while there are some new wonderful faces and contributors. It all feels pretty familiar to past experiences (just pleasantly bigger and more people). I don't expect to be the most active participant for sure, but I continue to hope to train others where possible and be a resource for others to ask questions of. -Travis On Wed, Sep 23, 2015 at 6:08 PM, Charles R Harris wrote: > > > On Wed, Sep 23, 2015 at 3:21 PM, Travis Oliphant > wrote: > >> >> >>> Regarding the seed council, I just tried to pick an objective >>> criterion and an arbitrary date that seemed generally in keeping with >>> idea of "should be active in the last 1-to-2-years-ish". Fiddling with >>> the exact date in particular makes very little difference -- between >>> pushing it back to 2 years ago today or forward to 1 year ago today, >>> the only thing that changes is whether Pauli makes the list or not. >>> (And Pauli is obviously a great council candidate, though I don't know >>> whether he even wants to be on it.) >>> >>> > Personally, I have no idea how big the council should be. Too big, and >>> > there is no point, consensus is harder to reach the larger the group, >>> > and the main (only?) role of the council is to resolve issues where >>> > consensus has not been reached in the larger community. But what is >>> > too big? >>> >>> >>> > As for make-up of the council, I think we need to expand beyond people >>> > who have recently contributed core code. >>> > >>> > Yes, the council does need to have expertise to make technical >>> > decisions, but if you think about the likely contentious issues like >>> > ABI breakage, a core-code focused view is incomplete. So there should >>> > be representation by: >>> > >>> > Someone(s) with a long history of working with the code -- that >>> > institutional memory of why decisions were made the way they were >>> > could be key. >>> >>> Sure -- though I can't really imagine any way of framing a rule like >>> this that *wouldn't* be satisfied by Chuck + Ralf + Pauli, so my guess >>> is that such a rule would not actually have any effect on the council >>> membership in practice. >>> >> >> As the original author of NumPy, I would like to be on the seed council >> as long as it is larger than 7 people. That is my proposal. I don't >> need to be a permanent member, but I do believe I have enough history that >> I can understand issues even if I haven't been working on code directly. >> >> >> I think I do bring history and information that provides all of the >> history that could be helpful on occasion. In addition, if a matter is >> important enough to even be brought to the attention of this council, I >> would like to be involved in the discussion about it. >> >> It's a simple change to the text --- basically an explanation that Travis >> requested to be on the seed council. >> > > I too would like you to be a member. We could either write it into the > text in recognition of your status as the Numpy creator, or it could be the > first order of business. I would only ask that you give yourself some time > to become familiar with how things work and the people involved in the > current community. It has been some years since you have been active in > code development. > > Chuck > >> >> >> > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > https://mail.scipy.org/mailman/listinfo/numpy-discussion > > -- *Travis Oliphant* *Co-founder and CEO* @teoliphant 512-222-5440 http://www.continuum.io -------------- next part -------------- An HTML attachment was scrubbed... URL: From travis at continuum.io Wed Sep 23 20:48:45 2015 From: travis at continuum.io (Travis Oliphant) Date: Wed, 23 Sep 2015 19:48:45 -0500 Subject: [Numpy-discussion] composition of the steering council (was Re: Governance model request) In-Reply-To: References: Message-ID: On Wed, Sep 23, 2015 at 6:19 PM, Charles R Harris wrote: > > > On Wed, Sep 23, 2015 at 3:42 PM, Chris Barker > wrote: > >> On Wed, Sep 23, 2015 at 2:21 PM, Travis Oliphant >> wrote: >> >> >>> As the original author of NumPy, I would like to be on the seed council >>> as long as it is larger than 7 people. That is my proposal. >>> >> >> Or the seed council could invite Travis to join as its first order of >> business :-) >> >> Actually, maybe that's a way to handle it -- declare that the first order >> of business for teh seed council is to expand the council. >> > > Perhaps we should specify a yearly meeting to review the past year and > nominate people for commit rights and council membership. Long term, we > might also want to start removing commit rights, perhaps by adding a team > category on github with restricted rights -- committer emeritus, so to > speak. > That's a pretty good idea, actually. > > >> I'd still like some guidelines (suggestions) for history and at least one >> major dependent-on-numpy rep. Travis would certainly meet the history >> requirement -- and maybe the other, too. :-) >> >> >>> It's a simple change to the text --- basically an explanation that >>> Travis requested to be on the seed council. >>> >> >> I'd rather the final draft of the document didn't name names, but no >> biggie. >> > I'm fine with that too --- except you will need to name the initial seed council. -Travis > > Chuck > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > https://mail.scipy.org/mailman/listinfo/numpy-discussion > > -- *Travis Oliphant* *Co-founder and CEO* @teoliphant 512-222-5440 http://www.continuum.io -------------- next part -------------- An HTML attachment was scrubbed... URL: From fperez.net at gmail.com Wed Sep 23 20:48:39 2015 From: fperez.net at gmail.com (Fernando Perez) Date: Wed, 23 Sep 2015 17:48:39 -0700 Subject: [Numpy-discussion] Plan for 1.9.4 release In-Reply-To: References: Message-ID: Ah, great, glad to hear that. Thx! On Wed, Sep 23, 2015 at 5:43 PM, Matthew Brett wrote: > Hi, > > On Wed, Sep 23, 2015 at 5:30 PM, Fernando Perez > wrote: > > Chance this will be merged into 1.10? I know there's an rc floating out > > there for 1.10, and this sounds like an actual bug... Would be great to > > have a solid 1.10 for the 3.5 crowd :) > > 1.10 already contains the f2py fixes, luckily... > > Matthew > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > https://mail.scipy.org/mailman/listinfo/numpy-discussion > -- Fernando Perez (@fperez_org; http://fperez.org) fperez.net-at-gmail: mailing lists only (I ignore this when swamped!) fernando.perez-at-berkeley: contact me here for any direct mail -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefanv at berkeley.edu Wed Sep 23 21:14:20 2015 From: stefanv at berkeley.edu (Stefan van der Walt) Date: Wed, 23 Sep 2015 18:14:20 -0700 Subject: [Numpy-discussion] Tentative NumPy Tutorial inaccessible In-Reply-To: References: <9f3a2b5086c48fbe3a292fb0ac8ffdcc@minuteware.net> Message-ID: <87mvwcsjo3.fsf@berkeley.edu> On 2015-09-22 23:53:19, Nathaniel Smith wrote: > On Tue, Sep 22, 2015 at 10:58 PM, Andriy Yurchuk wrote: >> Hi! >> >> The Tentative NumPy Tutorial is no longer accessible by the URL >> http://wiki.scipy.org/Tentative_NumPy_Tutorial, it returns a 403. The link >> to this page is still on NumPy homepage though. Has the page been moved >> somewhere else? > > No, our infrastructure is just in some disarray right now... there's > some discussion here: https://github.com/numpy/numpy/issues/6325 > (and I just pointed this out there as well since the relevant admins > seem to be reading that thread). FWIW, that tutorial is not that useful any longer. I'd refer to http://scipy-lectures.github.io/ instead. St?fan From sylvain.corlay at gmail.com Wed Sep 23 22:58:02 2015 From: sylvain.corlay at gmail.com (Sylvain Corlay) Date: Wed, 23 Sep 2015 22:58:02 -0400 Subject: [Numpy-discussion] ANN: numtraits v0.2 In-Reply-To: References: Message-ID: Hi Thomas, This is great news! FYI, the traitlets module has been undergoing significant refactoring lately, improving the API to favor a broader usage in the community. One reason for this is that several projects outside of the Jupyter organization are considering adopting traitlets. You can find a summary of the ongoing work and API changes here: https://github.com/ipython/traitlets/issues/48 One of the items in this discussion is about what would be the best place for a repository of trait types for standard data structures of the scipy stack (numpy array, pandas series and dataframes, etc...) It is unlikely that such trait types would be accepted in those libraries at this moment, and the main traitlets package might not be the right place for it either - hence the need for another repo. However, if we don't work on a centralized project, we will probably see a number of competing implementations in different libraries that are clients of traitlets. Hence the idea would be to propose a new project in the Jupyter incubator with a reference implementation. What would be cool would be to join forces and work on a specification or start a discussion of what the ideal implementation for such trait types would look like. Cheers, Sylvain On Wed, Sep 23, 2015 at 12:39 PM, Thomas Robitaille < thomas.robitaille at gmail.com> wrote: > Hi everyone, > > We have released a small experimental package called numtraits that > builds on top of the traitlets package and provides a NumericalTrait > class that can be used to validate properties such as: > > * number of dimension (for arrays) > * shape (for arrays) > * domain (e.g. positive, negative, range of values) > * units (with support for astropy.units, pint, and quantities) > > The idea is to be able to write a class like: > > class Sphere(HasTraits): > > radius = NumericalTrait(domain='strictly-positive', ndim=0) > position = NumericalTrait(shape=(3,)) > > and all the validation will then be done automatically when the user > sets 'radius' or 'position'. > > In addition, tuples and lists can get automatically converted to > arrays, and default values can be specified. You can read more about > the package and see examples of it in use here: > > https://github.com/astrofrog/numtraits > > and it can be easily installed with > > pip install numtraits > > The package supports both Python 3.3+ and Legacy Python (2.7) :) > > At this point, we would be very interested in feedback - the package > is still very young and we can still change the API if needed. Please > open issues with suggestions! > > Cheers, > > Tom and Francesco > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > https://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Thu Sep 24 00:36:40 2015 From: njs at pobox.com (Nathaniel Smith) Date: Wed, 23 Sep 2015 21:36:40 -0700 Subject: [Numpy-discussion] Fwd: [Matplotlib-devel] Fwd: [NumFOCUS Projects] grant opportunity In-Reply-To: References: Message-ID: ---------- Forwarded message ---------- From: Thomas Caswell Date: Wed, Sep 23, 2015 at 8:51 PM Subject: [Matplotlib-devel] Fwd: [NumFOCUS Projects] grant opportunity To: matplotlib development list If anyone has some downtime and would like to be paid to hack on open source. Submissions are due Sept 30 Tom ---------- Forwarded message --------- From: Gina Helfrich Date: Fri, Sep 18, 2015 at 12:10 PM Subject: [NumFOCUS Projects] grant opportunity To: Hi all, Wanted to bring this opportunity to your attention: Stripe is looking to host three to four developers at their San Francisco office for three months to work full-time on an open-source project. The grant is no-strings-attached: $7,500 per month in addition to desk space at our office and meals during the week. "We want to provide everything needed to focus and have a substantial impact on an open-source project." More info: https://stripe.com/blog/open-source-retreat-2016 The program will run from January 15th until April 15th, 2016 at Stripe HQ in the Mission District of San Francisco. Best, Gina -- Gina Helfrich, Ph.D. Communications Director NumFOCUS | Open Code, Better Science gina at numfocus.org | 512-222-5449 -- You received this message because you are subscribed to the Google Groups "Fiscally Sponsored Project Representatives" group. To unsubscribe from this group and stop receiving emails from it, send an email to projects+unsubscribe at numfocus.org. To post to this group, send email to projects at numfocus.org. Visit this group at http://groups.google.com/a/numfocus.org/group/projects/. To view this discussion on the web visit https://groups.google.com/a/numfocus.org/d/msgid/projects/CAFhTXRPxOsUubAHE%2BJuXzk%2B%3D4KjJmwXnAWGzNGw%3DPuXnQeiFhg%40mail.gmail.com. _______________________________________________ Matplotlib-devel mailing list Matplotlib-devel at python.org https://mail.python.org/mailman/listinfo/matplotlib-devel -- Nathaniel J. Smith -- http://vorpus.org From pi at berkeley.edu Thu Sep 24 01:06:53 2015 From: pi at berkeley.edu (Paul Ivanov) Date: Wed, 23 Sep 2015 22:06:53 -0700 Subject: [Numpy-discussion] Governance model request In-Reply-To: References: <8941C0D2-17EC-420D-A8F6-CA018144811C@continuum.io> <87mvwft2ar.fsf@berkeley.edu> <370C591F-9711-43E6-BE12-8C3E96946803@continuum.io> Message-ID: On Wed, Sep 23, 2015 at 3:37 PM, Fernando Perez wrote: > > Glad to see the other thread proceeding further, let's let this one die as > peacefully as it can... > > This Monty Python sketch seems relevant: https://www.youtube.com/watch?v=grbSQ6O6kbs (hope everyone's in good health and regains better spirits, back to lurking for me) -- _ / \ A* \^ - ,./ _.`\\ / \ / ,--.S \/ \ / `"~,_ \ \ __o ? _ \<,_ /:\ --(_)/-(_)----.../ | \ --------------.......J Paul Ivanov http://pirsquared.org | GPG/PGP key id: 0x0F3E28F7 -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Thu Sep 24 01:24:21 2015 From: charlesr.harris at gmail.com (Charles R Harris) Date: Wed, 23 Sep 2015 23:24:21 -0600 Subject: [Numpy-discussion] Python needs goto Message-ID: At last, goto for python ! Usage: from goto import with_goto @with_goto def range(start, stop): i = start result = [] label .begin if i == stop: goto .end result.append(i) i += 1 goto .begin label .end return result HT: LWN Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From andyfaff at gmail.com Thu Sep 24 03:17:12 2015 From: andyfaff at gmail.com (Andrew Nelson) Date: Thu, 24 Sep 2015 17:17:12 +1000 Subject: [Numpy-discussion] Problem while writing array with np.savetxt Message-ID: Dear list, whilst trying to write an array to disk I am coming across the following. What am I doing wrong? Surely f is a file handle? (python 3.4.3, numpy 1.9.2) import numpy as np a = np.arange(10.) with open('test.dat', 'w') as f: np.savetxt(f, a) --------------------------------------------------------------------------- TypeError Traceback (most recent call last) in () 2 a = np.arange(10.) 3 with open('test.dat', 'w') as f:----> 4 np.savetxt(f, a) /Users/anz/Documents/Andy/programming/dev3/lib/python3.4/site-packages/numpy/lib/npyio.py in savetxt(fname, X, fmt, delimiter, newline, header, footer, comments) 1085 else: 1086 for row in X:-> 1087 fh.write(asbytes(format % tuple(row) + newline)) 1088 if len(footer) > 0: 1089 footer = footer.replace('\n', '\n' + comments) TypeError: must be str, not bytes -- _____________________________________ Dr. Andrew Nelson _____________________________________ -------------- next part -------------- An HTML attachment was scrubbed... URL: From jensj at fysik.dtu.dk Thu Sep 24 03:17:33 2015 From: jensj at fysik.dtu.dk (=?windows-1252?Q?Jens_J=F8rgen_Mortensen?=) Date: Thu, 24 Sep 2015 09:17:33 +0200 Subject: [Numpy-discussion] ANN: Numpy 1.10.0rc1 released. In-Reply-To: References: Message-ID: <5603A38D.6090406@fysik.dtu.dk> On 09/23/2015 06:07 AM, Nathaniel Smith wrote: > On Tue, Sep 22, 2015 at 8:12 PM, Charles R Harris > wrote: >> Hi all, >> >> I'm pleased to announce the availability of Numpy 1.10.0rc1. Sources and 32 >> bit binary packages for Windows may be found at Sourceforge. Please test >> this out, as I would like to move to the final release as rapidly as >> possible and the lack of error reports from the beta has left me nervous. >> It's been quiet, too quiet. In the movies, we would all die in the next five >> minutes. Hi! I found this strange thing: jensj at jordan:~$ python Python 2.7.9 (default, Apr 2 2015, 15:33:21) [GCC 4.9.2] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import numpy as np >>> a = np.zeros((2, 2, 2)) >>> b = np.zeros((2, 2, 2)) >>> a[0, 0] = 7 >>> b[0, 0] = 6 >>> np.vdot(a[:, :, 0], b[:, :, 0]).real 84.0 >>> np.__version__ '1.10.0rc1' The result should be 42. This is on Ubuntu 15.04. Jens J?rgen From njs at pobox.com Thu Sep 24 03:20:23 2015 From: njs at pobox.com (Nathaniel Smith) Date: Thu, 24 Sep 2015 00:20:23 -0700 Subject: [Numpy-discussion] draft NEP for breaking ufunc ABI in a controlled way In-Reply-To: <20150922165755.492660b8@fsol> References: <20150922165755.492660b8@fsol> Message-ID: On Tue, Sep 22, 2015 at 7:57 AM, Antoine Pitrou wrote: > > Hi, > > This e-mail is an attempt at proposing an API to solve Numba's needs. Thanks! > Attribute access > ---------------- > > int PyUFunc_Nin(PyUFuncObject *) > > Replaces ufunc->nin. > > int PyUFunc_Nout(PyUFuncObject *) > > Replaces ufunc->nout. > > int PyUFunc_Nargs(PyUFuncObject *) > > Replaces ufunc->nargs. > > PyObject *PyUFunc_Name(PyUFuncObject *) > > Replaces ufunc->name, returns a unicode object. > (alternative: return a const char *) These all seem trivially supportable going forward. > For introspection, the following would be nice too: > > int PyUFunc_Identity(PyFuncObject *) > > Replaces ufunc->identity. Hmm, I can imagine cases where we might want to change how this works. (E.g. if np.dot were a ufunc then the existing identity settings wouldn't work very well... and I have some vague memory that there might already some delicate code in a few places because of difficulties in defining "zero" and "one" for arbitrary dtypes.) > const char *PyUFunc_Signature(PyUFuncObject *, int i) > > Gives a pointer to the types of the i'th signature. > (equivalent today to &ufunc->ntypes[i * ufunc->nargs]) I assume the 'i' part isn't actually interesting here (since there's no longer any parallel vector of function pointers accessible), and the high-level semantics that you're looking for are "please give me the set of signatures that have a loop defined"? [Edit: Also, see the discussion below about integer type pointers. The consequences here are that we can certainly provide an operation like this, but if we do then we might be abandoning it in a few releases (e.g. it might start telling you about only a subset of defined signatures). So can you expand a bit on what you mean by "would be nice" above?] > Lifetime control > ---------------- > > PyObject *PyUFunc_SetObject(PyUFuncObject *, PyObject *) > > Sets the ufunc's "object" to the given object. The object has no > special semantics except that it is DECREF'ed when the ufunc is > deallocated (this is today's ufunc->obj). The DECREF should happen > only after the ufunc has accessed any internal resources (since the > DECREF could deallocate some of those resources). I understand why you need a "base" object like this for individual loops, but if ufuncs start managing the ufunc-level memory buffers internally, then is this still useful? I guess I'm curious to see an example. > PyObject *PyUFunc_GetObject(PyUFuncObject *) > > Return the ufunc's current "object". Oh, are you planning to actually use this to attach some arbitrary metadata, not just attach deallocation callbacks? > Loop registration > ----------------- > > int PyUFunc_RegisterLoopForSignature( > PyUFuncObject* ufunc, > PyUFuncGenericFunction function, int *arg_types, > void *data, PyObject *obj) > > Register a loop implementation for the given arg_types (built-in > types, presumably). This either appends the loop to the types and > functions array (reallocating it if necessary), or replaces an > existing one with the same signature. > > A copy of arg_types is done, such that the caller does not have to > manage its lifetime. The optional "PyObject *obj" is an object which > gets DECREF'ed when the loop is relinquished (for example when the > ufunc is destroyed, or when the loop gets replaced with another by > calling this function again). > > > I cannot say I'm 100% sure this is sufficient, but this seems it should > cover our current needs. > > Note this is a minimal proposal. For example, Numpy could instead decide > to pass and return all argument types as PyArray_Descr pointers rather > than raw integers, and that would probably work for us too. Hmm, that's an interesting and tricky point, actually -- I think the way it will work eventually is that signatures will be specified in terms of "dtypetypes" (i.e., subclasses of dtype, rather than ints *or* instances of dtype = PyArray_Descrs). But I guess that's just a challenge we'll have to think about when implementing this stuff -- either it means that the new ufunc API will have to wait a bit for more of the new dtype machinery to be ready, or we'll have to temporarily bridge the gap with an loop registration API that takes new-style loop callbacks but uses int signatures (and then later turn it into a thin wrapper around the final API). -n -- Nathaniel J. Smith -- http://vorpus.org From njs at pobox.com Thu Sep 24 03:22:43 2015 From: njs at pobox.com (Nathaniel Smith) Date: Thu, 24 Sep 2015 00:22:43 -0700 Subject: [Numpy-discussion] interpretation of the draft governance document (was Re: Governance model request) In-Reply-To: References: Message-ID: On Wed, Sep 23, 2015 at 3:12 PM, Travis Oliphant wrote: >> >> Here is a list of the current Contributors to the main NumPy repository: >> >> >> [https://github.com/numpy/numpy/graphs/contributors](https://github.com/numpy/numpy/graphs/contributors) >> > > One of the problems with this list is that my contributions to the project > are extremely under-represented because the large majority of my commitment > of code happened in 2005 to 2006 before github was used. So, using this > as a list of the contributors is quite misleading --- and there are a lot of > people now looking only at lists like this one and it might confuse them why > I care so much. So, if you are going to make this list public in a > governance document like this, then I think some acknowledgement of the > source of the original code and the contributors to that needs to also be > made --- or you could just also point to the THANKS document which lists > people up to about 2008. Between 2008 and 2010 we will lose contributions, > still and this can be acknowledged. I'm actually not even sure why this link is even here -- it's another thing that just got copied from Jupyter/IPython :-) It looks like the reason your commits are massively undercounted on that page is that github uses emails to match up commits to github accounts. Unfortunately it looks like almost all your old commits used for your email address, and apparently the @teoliphant github account doesn't have that listed as one of its emails. I guess that's probably unfixable on your end, and while git has a standard way to solve these issues (.mailmap -- that's why Stephan got the right answer from git shortlog), github is not clever enough to use it: https://github.com/holman/ama/issues/551#issuecomment-46129792 So it sounds like that page is basically unfixable, and if it's misleading then we should just delete it from the doc. We definitely don't want to diminish anyone's contributions! [snip] >> ### Council membership >> >> To become eligible to join the Steering Council, an individual must be a >> Project Contributor who has produced contributions that are substantial in >> quality and quantity, and sustained over at least one year. Potential >> Council Members are nominated by existing Council members and voted upon by >> the existing Council after asking if the potential Member is interested and >> willing to serve in that capacity. The Council will be initially formed from >> the set of existing Core Developers who, as of late 2015, have been >> significantly active over the last year. >> > > Concretely, I'm asking to be included in this initial council so a simple > "along with Travis Oliphant who is the original author of NumPy". If other > long-time contributors to the code-base also want to be on this initial seed > council, I think it would make sense as well. I'll leave this discussion for the other thread that's specifically about the council membership, so as not to confuse matters. >> >> When considering potential Members, the Council will look at candidates >> with a comprehensive view of their contributions. This will include but is >> not limited to code, code review, infrastructure work, mailing list and chat >> participation, community help/building, education and outreach, design work, >> etc. We are deliberately not setting arbitrary quantitative metrics (like >> ?100 commits in this repo?) to avoid encouraging behavior that plays to the >> metrics rather than the project?s overall well-being. We want to encourage a >> diverse array of backgrounds, viewpoints and talents in our team, which is >> why we explicitly do not define code as the sole metric on which council >> membership will be evaluated. >> >> If a Council member becomes inactive in the project for a period of one >> year, they will be considered for removal from the Council. Before removal, >> inactive Member will be approached to see if they plan on returning to >> active participation. If not they will be removed immediately upon a Council >> vote. If they plan on returning to active participation soon, they will be >> given a grace period of one year. If they don?t return to active >> participation within that time period they will be removed by vote of the >> Council without further grace period. All former Council members can be >> considered for membership again at any time in the future, like any other >> Project Contributor. Retired Council members will be listed on the project >> website, acknowledging the period during which they were active in the >> Council. > > > I would just remove quite a bit of this language and just say that members > who are inactive on the Council may be asked to step down if they are not > planning on returning to active participation. I think it's important to have *some* kind of relatively objective/impartial criteria here, because if the rules force somebody to be the bad guy before anyone can get pruned, then the council will never get pruned, and over time we'll end up with an official council membership list that gets more and more out of touch with reality. Which is the kind of "social debt" we don't need IMO -- we already have enough technical debt to sort out :-). That doesn't mean it necessarily has to be these particular criteria though... probably the decision about what exactly they should be is another thing to defer to the other thread (or at least put on hold until we have some decision on the initial membership question). I'm feeling good -- it sounds like we're narrowing things down quite a bit now. -n -- Nathaniel J. Smith -- http://vorpus.org From efiring at hawaii.edu Thu Sep 24 03:24:33 2015 From: efiring at hawaii.edu (Eric Firing) Date: Wed, 23 Sep 2015 21:24:33 -1000 Subject: [Numpy-discussion] Problem while writing array with np.savetxt In-Reply-To: References: Message-ID: <5603A531.7030607@hawaii.edu> On 2015/09/23 9:17 PM, Andrew Nelson wrote: > Dear list, > whilst trying to write an array to disk I am coming across the > following. What am I doing wrong? Surely f is a file handle? > (python 3.4.3, numpy 1.9.2) > > import numpy as np > a = np.arange(10.) > with open('test.dat', 'w') as f: > np.savetxt(f, a) > It will work if you open with 'wb'. Yes, this seems like a bug; or at least, the anomaly should be noted in the docstring. Eric > --------------------------------------------------------------------------- > > TypeError Traceback (most recent call last) > in() 2 a = np.arange(10.)3 with open('test.dat', 'w') as f:----> > 4np.savetxt(f, > a)/Users/anz/Documents/Andy/programming/dev3/lib/python3.4/site-packages/numpy/lib/npyio.py > in savetxt(fname, X, fmt, delimiter, newline, header, footer, comments) > 1085 else:1086 for row in X:-> 1087fh.write(asbytes(format % tuple(row) > + newline))1088 if len(footer) > 0:1089 footer = footer.replace('\n', > '\n' + comments)TypeError: must be str, not bytes > > > > > -- > _____________________________________ > Dr. Andrew Nelson > > > _____________________________________ > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > https://mail.scipy.org/mailman/listinfo/numpy-discussion > From njs at pobox.com Thu Sep 24 03:28:47 2015 From: njs at pobox.com (Nathaniel Smith) Date: Thu, 24 Sep 2015 00:28:47 -0700 Subject: [Numpy-discussion] interpretation of the draft governance document (was Re: Governance model request) In-Reply-To: References: Message-ID: On Thu, Sep 24, 2015 at 12:22 AM, Nathaniel Smith wrote: > So it sounds like that page is basically unfixable, and if it's > misleading then we should just delete it from the doc. We definitely > don't want to diminish anyone's contributions! I just posted a pull request version of the governance document: https://github.com/numpy/numpy/pull/6352 It's definitely still a draft, but now that we're making changes I wanted to get it into a form where people can at least see what those are :-). In interests of making that easy, the PR is structured as a ReST-converted version of the document originally posted, plus clearly-separated commits on top of that. The only non-trivial changes are: - I split the lists of people out into a separate document: https://github.com/njsmith/numpy/blob/governance/doc/source/dev/governance/people.rst (currently filled with the original seed council proposal for the reasons mentioned above) - I removed the link to the inaccurate github "contributors" page -n -- Nathaniel J. Smith -- http://vorpus.org From thomas.robitaille at gmail.com Thu Sep 24 03:31:12 2015 From: thomas.robitaille at gmail.com (Thomas Robitaille) Date: Thu, 24 Sep 2015 09:31:12 +0200 Subject: [Numpy-discussion] ANN: numtraits v0.2 In-Reply-To: References: Message-ID: <5603A6C0.6080306@gmail.com> Hi Sylvain, Sylvain Corlay wrote: > Hi Thomas, > > This is great news! > > FYI, the traitlets module has been undergoing significant refactoring > lately, improving the API to favor a broader usage in the community. > One reason for this is that several projects outside of the Jupyter > organization are considering adopting traitlets. You can find a summary > of the ongoing work and API changes > here: https://github.com/ipython/traitlets/issues/48 > > One of the items in this discussion is about what would be the best > place for a repository of trait types for standard data structures of > the scipy stack (numpy array, pandas series and dataframes, etc...) It > is unlikely that such trait types would be accepted in those libraries > at this moment, and the main traitlets package might not be the right > place for it either - hence the need for another repo. However, if we > don't work on a centralized project, we will probably see a number of > competing implementations in different libraries that are clients of > traitlets. > > Hence the idea would be to propose a new project in the Jupyter > incubator with a reference implementation. What would be cool would be > to join forces and work on a specification or start a discussion of what > the ideal implementation for such trait types would look like. I'm very open to collaborating on centralizing these kind of scipy-stack traits. I'm not particularly attached to the idea of keeping our numtraits implementation separate, and would be very happy to merge it in to a larger effort or only re-use parts of it. Realistically I won't be able to lead/write a proposal for the incubator in the next few weeks, but if no one gets to it first, I can try and work on it later in the year. Cheers, Tom > > Cheers, > > Sylvain > > > On Wed, Sep 23, 2015 at 12:39 PM, Thomas Robitaille > > wrote: > > Hi everyone, > > We have released a small experimental package called numtraits that > builds on top of the traitlets package and provides a NumericalTrait > class that can be used to validate properties such as: > > * number of dimension (for arrays) > * shape (for arrays) > * domain (e.g. positive, negative, range of values) > * units (with support for astropy.units, pint, and quantities) > > The idea is to be able to write a class like: > > class Sphere(HasTraits): > > radius = NumericalTrait(domain='strictly-positive', ndim=0) > position = NumericalTrait(shape=(3,)) > > and all the validation will then be done automatically when the user > sets 'radius' or 'position'. > > In addition, tuples and lists can get automatically converted to > arrays, and default values can be specified. You can read more about > the package and see examples of it in use here: > > https://github.com/astrofrog/numtraits > > and it can be easily installed with > > pip install numtraits > > The package supports both Python 3.3+ and Legacy Python (2.7) :) > > At this point, we would be very interested in feedback - the package > is still very young and we can still change the API if needed. Please > open issues with suggestions! > > Cheers, > > Tom and Francesco > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > https://mail.scipy.org/mailman/listinfo/numpy-discussion > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > https://mail.scipy.org/mailman/listinfo/numpy-discussion From sebastian at sipsolutions.net Thu Sep 24 04:45:40 2015 From: sebastian at sipsolutions.net (Sebastian Berg) Date: Thu, 24 Sep 2015 10:45:40 +0200 Subject: [Numpy-discussion] interpretation of the draft governance document (was Re: Governance model request) In-Reply-To: References: Message-ID: <1443084340.1783.5.camel@sipsolutions.net> On Do, 2015-09-24 at 00:22 -0700, Nathaniel Smith wrote: > On Wed, Sep 23, 2015 at 3:12 PM, Travis Oliphant wrote: > >> > >> Here is a list of the current Contributors to the main NumPy repository: > >> > >> To become eligible to join the Steering Council, an individual must be a > >> Project Contributor who has produced contributions that are substantial in > >> quality and quantity, and sustained over at least one year. Potential > >> Council Members are nominated by existing Council members and voted upon by > >> the existing Council after asking if the potential Member is interested and > >> willing to serve in that capacity. The Council will be initially formed from > >> the set of existing Core Developers who, as of late 2015, have been > >> significantly active over the last year. > >> > > > > Concretely, I'm asking to be included in this initial council so a simple > > "along with Travis Oliphant who is the original author of NumPy". If other > > long-time contributors to the code-base also want to be on this initial seed > > council, I think it would make sense as well. > > I'll leave this discussion for the other thread that's specifically > about the council membership, so as not to confuse matters. Well, I think we can at least learn one thing from this directly. I think we should include some relation to historic contributions into the document. I think what might be nice to do would be for example to also "seed" the emeritus list (if it is not too difficult). It is somewhat unrelated to governance, but since we want it to be a prominent document, it should not overshadow the past. - Sebastian > > -n > -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 819 bytes Desc: This is a digitally signed message part URL: From jtaylor.debian at googlemail.com Thu Sep 24 04:59:19 2015 From: jtaylor.debian at googlemail.com (Julian Taylor) Date: Thu, 24 Sep 2015 10:59:19 +0200 Subject: [Numpy-discussion] Problem while writing array with np.savetxt In-Reply-To: <5603A531.7030607@hawaii.edu> References: <5603A531.7030607@hawaii.edu> Message-ID: <5603BB67.1070203@googlemail.com> numpy text io is fundamentally broken in python3, there are sometimes workarounds, but currently its probably best to stick to python2 or not use it. The workarounds usually just make actually fixing the functions harder. On 09/24/2015 09:24 AM, Eric Firing wrote: > On 2015/09/23 9:17 PM, Andrew Nelson wrote: >> Dear list, >> whilst trying to write an array to disk I am coming across the >> following. What am I doing wrong? Surely f is a file handle? >> (python 3.4.3, numpy 1.9.2) >> >> import numpy as np >> a = np.arange(10.) >> with open('test.dat', 'w') as f: >> np.savetxt(f, a) >> > > It will work if you open with 'wb'. Yes, this seems like a bug; or at > least, the anomaly should be noted in the docstring. > > Eric > >> --------------------------------------------------------------------------- >> >> >> TypeError Traceback (most recent call >> last) >> in() 2 a = np.arange(10.)3 >> with open('test.dat', 'w') as f:----> >> 4np.savetxt(f, >> a)/Users/anz/Documents/Andy/programming/dev3/lib/python3.4/site-packages/numpy/lib/npyio.py >> >> in savetxt(fname, X, fmt, delimiter, newline, header, footer, comments) >> 1085 else:1086 for row in X:-> 1087fh.write(asbytes(format % tuple(row) >> + newline))1088 if len(footer) > 0:1089 footer = footer.replace('\n', >> '\n' + comments)TypeError: must be str, not bytes >> >> >> >> >> -- >> _____________________________________ >> Dr. Andrew Nelson >> >> >> _____________________________________ >> >> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> https://mail.scipy.org/mailman/listinfo/numpy-discussion >> > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > https://mail.scipy.org/mailman/listinfo/numpy-discussion From robert.kern at gmail.com Thu Sep 24 05:19:50 2015 From: robert.kern at gmail.com (Robert Kern) Date: Thu, 24 Sep 2015 10:19:50 +0100 Subject: [Numpy-discussion] interpretation of the draft governance document (was Re: Governance model request) In-Reply-To: References: Message-ID: On Wed, Sep 23, 2015 at 11:12 PM, Travis Oliphant wrote: >> [We also have to decide on the initial membership for the Council. While the above text makes pains to distinguish between "committer" and "Council Member", in the past we've pretty much treated them as the same. So to keep things simple and deterministic, I propose that we seed the Council with everyone who has reviewed/merged a pull request since Jan 1, 2014, and move those who haven't used their commit bit in >1.5 years to the emeritus list. Based on the output of >> >> git log --grep="^Merge pull request" --since 2014-01-01 | grep Author: | sort -u >> >> I believe this would give us an initial Steering Council of: Sebastian Berg, Jaime Fern?ndez del R?o, Ralf Gommers, Alex Griffing, Charles Harris, Nathaniel Smith, Julian Taylor, and Pauli Virtanen (assuming everyone on that list is interested/willing to serve).] > > I would ask to be on this initial council by having the rule include "original contributors of the code" which would basically include Robert Kern and Pearu Peterson in addition to Chuck Harris (though Pearu's main contribution was numpy.distutils and f2py and he may not be interested any longer in the rest of it). Were I to want such a thing, I would simply review and merge one of the many languishing PRs. I have the power to add myself to this list without asking for special exemptions to the general rule, merely a token bit of effort on my part. -- Robert Kern -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Thu Sep 24 05:34:19 2015 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 24 Sep 2015 11:34:19 +0200 Subject: [Numpy-discussion] draft NEP for breaking ufunc ABI in a controlled way References: <20150922165755.492660b8@fsol> Message-ID: <20150924113419.60a1acb3@fsol> On Thu, 24 Sep 2015 00:20:23 -0700 Nathaniel Smith wrote: > > int PyUFunc_Identity(PyFuncObject *) > > > > Replaces ufunc->identity. > > Hmm, I can imagine cases where we might want to change how this works. > (E.g. if np.dot were a ufunc then the existing identity settings > wouldn't work very well... and I have some vague memory that there > might already some delicate code in a few places because of > difficulties in defining "zero" and "one" for arbitrary dtypes.) Yes... As long as there is a way for us to set the identity value (whatever the exact API) when constructing a ufunc, it should be ok. > I assume the 'i' part isn't actually interesting here (since there's > no longer any parallel vector of function pointers accessible), and > the high-level semantics that you're looking for are "please give me > the set of signatures that have a loop defined"? Indeed. > [Edit: Also, see the discussion below about integer type pointers. The > consequences here are that we can certainly provide an operation like > this, but if we do then we might be abandoning it in a few releases > (e.g. it might start telling you about only a subset of defined > signatures). So can you expand a bit on what you mean by "would be > nice" above?] "Would be nice" really means "we could make use of it" for letting the user access ufunc metadata. We don't *need* it currently. But generally being able to query the high-level properties of a ufunc, from C, sounds like a good thing, and perhaps other people would be interested. > > PyObject *PyUFunc_SetObject(PyUFuncObject *, PyObject *) > > > > Sets the ufunc's "object" to the given object. The object has no > > special semantics except that it is DECREF'ed when the ufunc is > > deallocated (this is today's ufunc->obj). The DECREF should happen > > only after the ufunc has accessed any internal resources (since the > > DECREF could deallocate some of those resources). > > I understand why you need a "base" object like this for individual > loops, but if ufuncs start managing the ufunc-level memory buffers > internally, then is this still useful? I guess I'm curious to see an > example. Well, for example, we dynamically allocate the ufunc's name (and possibly its docstring), so we need to deallocate it when the ufunc is destroyed. Actually, we should probably deallocate more stuff that we currently don't (such as the execution environment)... > > PyObject *PyUFunc_GetObject(PyUFuncObject *) > > > > Return the ufunc's current "object". > > Oh, are you planning to actually use this to attach some arbitrary > metadata, not just attach deallocation callbacks? No, just deallocation callbacks. I was including the GetObject function for completeness, I'm not sure we would need it (but it sounds trivial to provide and maintain). > Hmm, that's an interesting and tricky point, actually -- I think the > way it will work eventually is that signatures will be specified in > terms of "dtypetypes" (i.e., subclasses of dtype, rather than ints > *or* instances of dtype = PyArray_Descrs). Subclasses? I'm not sure what you mean by that, how would one specify e.g. an int64 vs. an int32? Are you referring to Travis' dtypes-as-classes project, or something similar? In that case though, a dtype would still be an instance of a "dtypetype" (metatype), not a subclass :-) > But I guess that's just a > challenge we'll have to think about when implementing this stuff -- > either it means that the new ufunc API will have to wait a bit for > more of the new dtype machinery to be ready, or we'll have to > temporarily bridge the gap with an loop registration API that takes > new-style loop callbacks but uses int signatures (and then later turn > it into a thin wrapper around the final API). Well, as long as you keep the int typecodes in Numpy (and I guess you'll do for quite some time, for compatibility), bridging should be easy indeed. Regards Antoine. From sebastian at sipsolutions.net Thu Sep 24 05:45:47 2015 From: sebastian at sipsolutions.net (Sebastian Berg) Date: Thu, 24 Sep 2015 11:45:47 +0200 Subject: [Numpy-discussion] composition of the steering council (was Re: Governance model request) In-Reply-To: References: Message-ID: <1443087947.1783.50.camel@sipsolutions.net> On Mi, 2015-09-23 at 17:08 -0600, Charles R Harris wrote: > > > On Wed, Sep 23, 2015 at 3:21 PM, Travis Oliphant > wrote: > > > Regarding the seed council, I just tried to pick an > objective > criterion and an arbitrary date that seemed generally > in keeping with > idea of "should be active in the last > 1-to-2-years-ish". Fiddling with > the exact date in particular makes very little > difference -- between > pushing it back to 2 years ago today or forward to 1 > year ago today, > the only thing that changes is whether Pauli makes the > list or not. > (And Pauli is obviously a great council candidate, > though I don't know > whether he even wants to be on it.) > > > Personally, I have no idea how big the council > should be. Too big, and > > there is no point, consensus is harder to reach the > larger the group, > > and the main (only?) role of the council is to > resolve issues where > > consensus has not been reached in the larger > community. But what is > > too big? > > > > As for make-up of the council, I think we need to > expand beyond people > > who have recently contributed core code. > > > > Yes, the council does need to have expertise to make > technical > > decisions, but if you think about the likely > contentious issues like > > ABI breakage, a core-code focused view is > incomplete. So there should > > be representation by: > > > > Someone(s) with a long history of working with the > code -- that > > institutional memory of why decisions were made the > way they were > > could be key. > > Sure -- though I can't really imagine any way of > framing a rule like > this that *wouldn't* be satisfied by Chuck + Ralf + > Pauli, so my guess > is that such a rule would not actually have any effect > on the council > membership in practice. > > > As the original author of NumPy, I would like to be on the > seed council as long as it is larger than 7 people. That is > my proposal. I don't need to be a permanent member, but I > do believe I have enough history that I can understand issues > even if I haven't been working on code directly. > > > I think I do bring history and information that provides all > of the history that could be helpful on occasion. In > addition, if a matter is important enough to even be brought > to the attention of this council, I would like to be involved > in the discussion about it. > > > > It's a simple change to the text --- basically an explanation > that Travis requested to be on the seed council. > > I too would like you to be a member. We could either write it into the > text in recognition of your status as the Numpy creator, or it could > be the first order of business. I would only ask that you give > yourself some time to become familiar with how things work and the > people involved in the current community. It has been some years since > you have been active in code development. > I think I can agree with that. On a serious note, I now realize that I am probably the one with the most objection, so for everyone, do not bother with trying to convince me, you probably cannot fully, nor do you have to, I will let it stand as is after this and let others take over from here (after this, probably whatever Chuck says is good). [1] More to the point of the actual members: So to say, I feel the council members have to try to be *directly* active and see being active as a necessary *commitment* (i.e. also try to travel to meetings). This will always be a difficult judgment of course, but there is no help to it. The current definitions imply this. And two years seems fine. It is not that short, at least unless someone stops contributing very abruptly which I do not think is that usual. I will weight in to keep the current times but do not feel very strongly. About using the commit log to seed, I think there are some old term contributers (David Cournapeau maybe?), who never stopped doing quite a bit but may not have merge commits. However, I think we can start of with what we had, then I would hope Chuck and maybe Ralf can fill in the blanks. About the size, I think if we get too many -- if that is possible -- we should just change the governance at that time to be not veto based anymore. This is something to keep in mind, but probably does not need to be formalized. - Sebastian [1] Sorry to "footnote" this, but I think I am probably rudely repeating myself and frankly do **not want this to be discussed**. It is just to try to be fully clear where I come from: Until SciPy 2015, I could list many people on this list who have shown more direct involvement in numpy then Travis since I joined and have no affiliation to numpy. If Travis had been new to the community at the time, I would be surprised if I would even recognize his name. I know this is only half the picture and Travis already mentioned another side, but this is what I mostly saw even if it may be a harsh and rude assessment. > > Chuck > > > > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > https://mail.scipy.org/mailman/listinfo/numpy-discussion -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 819 bytes Desc: This is a digitally signed message part URL: From davidmenhur at gmail.com Thu Sep 24 06:21:10 2015 From: davidmenhur at gmail.com (=?UTF-8?B?RGHPgGlk?=) Date: Thu, 24 Sep 2015 12:21:10 +0200 Subject: [Numpy-discussion] ANN: Numpy 1.10.0rc1 released. In-Reply-To: <5603A38D.6090406@fysik.dtu.dk> References: <5603A38D.6090406@fysik.dtu.dk> Message-ID: On 24 September 2015 at 09:17, Jens J?rgen Mortensen wrote: > > >>> np.vdot(a[:, :, 0], b[:, :, 0]).real > 84.0 > >>> np.__version__ > '1.10.0rc1' > > The result should be 42. This is on Ubuntu 15.04. I can reproduce on Fedora with Numpy linked against Openblas. Numpy 1.9 in the same configuration works. The full test suite raises one error, but only due to lack of space when saving a large array. -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefanv at berkeley.edu Thu Sep 24 06:26:14 2015 From: stefanv at berkeley.edu (Stefan van der Walt) Date: Thu, 24 Sep 2015 03:26:14 -0700 Subject: [Numpy-discussion] ANN: Numpy 1.10.0rc1 released. In-Reply-To: <5603A38D.6090406@fysik.dtu.dk> References: <5603A38D.6090406@fysik.dtu.dk> Message-ID: <87vbb0qfjt.fsf@berkeley.edu> On 2015-09-24 00:17:33, Jens J?rgen Mortensen wrote: > jensj at jordan:~$ python > Python 2.7.9 (default, Apr 2 2015, 15:33:21) > [GCC 4.9.2] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > >>> import numpy as np > >>> a = np.zeros((2, 2, 2)) > >>> b = np.zeros((2, 2, 2)) > >>> a[0, 0] = 7 > >>> b[0, 0] = 6 > >>> np.vdot(a[:, :, 0], b[:, :, 0]).real > 84.0 > >>> np.__version__ > '1.10.0rc1' > > The result should be 42. This is on Ubuntu 15.04. The input is not supposed to be matrices?but the docstring specifically states that those should then be ravelled, so this is indeed a bug. If I bisected correctly, the problematic change is this one: commit 44877b36870ec2a0505c536a30b9fbf06a414c17 Author: Sebastian Berg Date: Wed Sep 3 18:33:06 2014 +0200 ENH: Allow ravel to reshape in place when possible This fixes a lot of corner cases in reshape 'K' when the array is not contiguous in the first place, it was previously never ravelled in place. Closes gh-5033 St?fan From sebastian at sipsolutions.net Thu Sep 24 07:06:27 2015 From: sebastian at sipsolutions.net (Sebastian Berg) Date: Thu, 24 Sep 2015 13:06:27 +0200 Subject: [Numpy-discussion] ANN: Numpy 1.10.0rc1 released. In-Reply-To: <87vbb0qfjt.fsf@berkeley.edu> References: <5603A38D.6090406@fysik.dtu.dk> <87vbb0qfjt.fsf@berkeley.edu> Message-ID: <1443092787.3544.8.camel@sipsolutions.net> On Do, 2015-09-24 at 03:26 -0700, Stefan van der Walt wrote: > On 2015-09-24 00:17:33, Jens J?rgen Mortensen wrote: > > jensj at jordan:~$ python > > Python 2.7.9 (default, Apr 2 2015, 15:33:21) > > [GCC 4.9.2] on linux2 > > Type "help", "copyright", "credits" or "license" for more information. > > >>> import numpy as np > > >>> a = np.zeros((2, 2, 2)) > > >>> b = np.zeros((2, 2, 2)) > > >>> a[0, 0] = 7 > > >>> b[0, 0] = 6 > > >>> np.vdot(a[:, :, 0], b[:, :, 0]).real > > 84.0 > > >>> np.__version__ > > '1.10.0rc1' > > > > The result should be 42. This is on Ubuntu 15.04. > > The input is not supposed to be matrices?but the docstring specifically > states that those should then be ravelled, so this is indeed a bug. > > If I bisected correctly, the problematic change is this one: > Yeah, vdot uses `ravel`. That is correct, but it should only use it after making sure the input is C-order, or make sure the output of ravel is C-order (vdot was changed too in the mean time). So the fix will be adding the contiguity check in vdot. - Sebastian > > commit 44877b36870ec2a0505c536a30b9fbf06a414c17 > Author: Sebastian Berg > Date: Wed Sep 3 18:33:06 2014 +0200 > > ENH: Allow ravel to reshape in place when possible > > This fixes a lot of corner cases in reshape 'K' when the array > is not contiguous in the first place, it was previously never > ravelled in place. > > Closes gh-5033 > > > St?fan > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > https://mail.scipy.org/mailman/listinfo/numpy-discussion -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 819 bytes Desc: This is a digitally signed message part URL: From sebastian at sipsolutions.net Thu Sep 24 07:14:49 2015 From: sebastian at sipsolutions.net (Sebastian Berg) Date: Thu, 24 Sep 2015 13:14:49 +0200 Subject: [Numpy-discussion] ANN: Numpy 1.10.0rc1 released. In-Reply-To: <87vbb0qfjt.fsf@berkeley.edu> References: <5603A38D.6090406@fysik.dtu.dk> <87vbb0qfjt.fsf@berkeley.edu> Message-ID: <1443093289.3544.12.camel@sipsolutions.net> On Do, 2015-09-24 at 03:26 -0700, Stefan van der Walt wrote: > On 2015-09-24 00:17:33, Jens J?rgen Mortensen wrote: > > jensj at jordan:~$ python > > Python 2.7.9 (default, Apr 2 2015, 15:33:21) > > [GCC 4.9.2] on linux2 > > Type "help", "copyright", "credits" or "license" for more information. > > >>> import numpy as np > > >>> a = np.zeros((2, 2, 2)) > > >>> b = np.zeros((2, 2, 2)) > > >>> a[0, 0] = 7 > > >>> b[0, 0] = 6 > > >>> np.vdot(a[:, :, 0], b[:, :, 0]).real > > 84.0 > > >>> np.__version__ > > '1.10.0rc1' > > > > The result should be 42. This is on Ubuntu 15.04. > > The input is not supposed to be matrices?but the docstring specifically > states that those should then be ravelled, so this is indeed a bug. > > If I bisected correctly, the problematic change is this one: > Hmmm, unless we want to make sure that the output of ravel is always contiguous (which would be a difference from `.reshape(-1)`. It would be a bit safer, and not a useless feature maybe, since we can point to `.reshape(-1)` as well for those who do not care. As far as I can see a contiguous output was not guaranteed for "keeporder", but nobody probably ever used keeporder.... - Sebastian > > commit 44877b36870ec2a0505c536a30b9fbf06a414c17 > Author: Sebastian Berg > Date: Wed Sep 3 18:33:06 2014 +0200 > > ENH: Allow ravel to reshape in place when possible > > This fixes a lot of corner cases in reshape 'K' when the array > is not contiguous in the first place, it was previously never > ravelled in place. > > Closes gh-5033 > > > St?fan > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > https://mail.scipy.org/mailman/listinfo/numpy-discussion -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 819 bytes Desc: This is a digitally signed message part URL: From projetmbc at gmail.com Thu Sep 24 07:25:16 2015 From: projetmbc at gmail.com (Christophe Bal) Date: Thu, 24 Sep 2015 13:25:16 +0200 Subject: [Numpy-discussion] Python needs goto In-Reply-To: References: Message-ID: Hello. Can you give an example where GOTO is useful ? Le 24 sept. 2015 07:24, "Charles R Harris" a ?crit : > At last, goto for python ! > > Usage: > > from goto import with_goto > > @with_goto > def range(start, stop): > i = start > result = [] > > label .begin > if i == stop: > goto .end > > result.append(i) > i += 1 > goto .begin > > label .end > return result > > > HT: LWN > > Chuck > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > https://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From sebastian at sipsolutions.net Thu Sep 24 07:39:19 2015 From: sebastian at sipsolutions.net (Sebastian Berg) Date: Thu, 24 Sep 2015 13:39:19 +0200 Subject: [Numpy-discussion] ANN: Numpy 1.10.0rc1 released. In-Reply-To: <1443093289.3544.12.camel@sipsolutions.net> References: <5603A38D.6090406@fysik.dtu.dk> <87vbb0qfjt.fsf@berkeley.edu> <1443093289.3544.12.camel@sipsolutions.net> Message-ID: <1443094759.3544.15.camel@sipsolutions.net> On Do, 2015-09-24 at 13:14 +0200, Sebastian Berg wrote: > On Do, 2015-09-24 at 03:26 -0700, Stefan van der Walt wrote: > > On 2015-09-24 00:17:33, Jens J?rgen Mortensen wrote: > > > jensj at jordan:~$ python > > > Python 2.7.9 (default, Apr 2 2015, 15:33:21) > > > [GCC 4.9.2] on linux2 > > > Type "help", "copyright", "credits" or "license" for more information. > > > >>> import numpy as np > > > >>> a = np.zeros((2, 2, 2)) > > > >>> b = np.zeros((2, 2, 2)) > > > >>> a[0, 0] = 7 > > > >>> b[0, 0] = 6 > > > >>> np.vdot(a[:, :, 0], b[:, :, 0]).real > > > 84.0 > > > >>> np.__version__ > > > '1.10.0rc1' > > > > > > The result should be 42. This is on Ubuntu 15.04. > > > > The input is not supposed to be matrices?but the docstring specifically > > states that those should then be ravelled, so this is indeed a bug. > > > > If I bisected correctly, the problematic change is this one: > > > > Hmmm, unless we want to make sure that the output of ravel is always > contiguous (which would be a difference from `.reshape(-1)`. > It would be a bit safer, and not a useless feature maybe, since we can > point to `.reshape(-1)` as well for those who do not care. > So to say, I am not quite sure yet. But if anyone is even a tiny bit nervous about doing less copying/not guaranteeing a contiguous result, lets change the definition of ravel to return a contiguous array always (and document it pointing people to reshape if they have no need for it). > As far as I can see a contiguous output was not guaranteed for > "keeporder", but nobody probably ever used keeporder.... > > - Sebastian > > > > > > commit 44877b36870ec2a0505c536a30b9fbf06a414c17 > > Author: Sebastian Berg > > Date: Wed Sep 3 18:33:06 2014 +0200 > > > > ENH: Allow ravel to reshape in place when possible > > > > This fixes a lot of corner cases in reshape 'K' when the array > > is not contiguous in the first place, it was previously never > > ravelled in place. > > > > Closes gh-5033 > > > > > > St?fan > > _______________________________________________ > > NumPy-Discussion mailing list > > NumPy-Discussion at scipy.org > > https://mail.scipy.org/mailman/listinfo/numpy-discussion > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > https://mail.scipy.org/mailman/listinfo/numpy-discussion -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 819 bytes Desc: This is a digitally signed message part URL: From sebastian at sipsolutions.net Thu Sep 24 08:11:55 2015 From: sebastian at sipsolutions.net (Sebastian Berg) Date: Thu, 24 Sep 2015 14:11:55 +0200 Subject: [Numpy-discussion] interpretation of the draft governance document (was Re: Governance model request) In-Reply-To: <1443084340.1783.5.camel@sipsolutions.net> References: <1443084340.1783.5.camel@sipsolutions.net> Message-ID: <1443096715.7144.0.camel@sipsolutions.net> On Do, 2015-09-24 at 10:45 +0200, Sebastian Berg wrote: > On Do, 2015-09-24 at 00:22 -0700, Nathaniel Smith wrote: > > On Wed, Sep 23, 2015 at 3:12 PM, Travis Oliphant wrote: > > >> > > >> Here is a list of the current Contributors to the main NumPy repository: > > > >> > > >> To become eligible to join the Steering Council, an individual must be a > > >> Project Contributor who has produced contributions that are substantial in > > >> quality and quantity, and sustained over at least one year. Potential > > >> Council Members are nominated by existing Council members and voted upon by > > >> the existing Council after asking if the potential Member is interested and > > >> willing to serve in that capacity. The Council will be initially formed from > > >> the set of existing Core Developers who, as of late 2015, have been > > >> significantly active over the last year. > > >> > > > > > > Concretely, I'm asking to be included in this initial council so a simple > > > "along with Travis Oliphant who is the original author of NumPy". If other > > > long-time contributors to the code-base also want to be on this initial seed > > > council, I think it would make sense as well. > > > > I'll leave this discussion for the other thread that's specifically > > about the council membership, so as not to confuse matters. > > Well, I think we can at least learn one thing from this directly. I > think we should include some relation to historic contributions into the > document. I think what might be nice to do would be for example to also > "seed" the emeritus list (if it is not too difficult). > It is somewhat unrelated to governance, but since we want it to be a > prominent document, it should not overshadow the past. > Heh, sorry, I see this was already pretty much the case. Could maybe be extended a bit, but missed that.... > - Sebastian > > > > > > > -n > > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > https://mail.scipy.org/mailman/listinfo/numpy-discussion -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 819 bytes Desc: This is a digitally signed message part URL: From scollis.acrf at gmail.com Thu Sep 24 09:03:25 2015 From: scollis.acrf at gmail.com (Scott Collis) Date: Thu, 24 Sep 2015 08:03:25 -0500 Subject: [Numpy-discussion] "Become an Open Source Contributor" workshop In-Reply-To: References: Message-ID: <5603F49D.5090502@gmail.com> Hey Jamie, List, Having just come back from a conference where our toolkit, Py-ART [1] has picked up a nice following of people keen to contribute I was wondering if you will be opening this up via a google hangout or similar? I would love to advertise this to our users. We all want more contributors and a big roadblock is understanding the fork and pull request system of GitHub We did run a course that had some GitGub etc here: https://github.com/scollis/SusSoPrac You are welcome to use anything liberally! Cheers, Scott On 9/23/15 4:39 PM, numpy-discussion-request at scipy.org wrote: > Send NumPy-Discussion mailing list submissions to > numpy-discussion at scipy.org > > To subscribe or unsubscribe via the World Wide Web, visit > https://mail.scipy.org/mailman/listinfo/numpy-discussion > or, via email, send a message with subject or body 'help' to > numpy-discussion-request at scipy.org > > You can reach the person managing the list at > numpy-discussion-owner at scipy.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of NumPy-Discussion digest..." > > > Today's Topics: > > 1. "Become an Open Source Contributor" workshop > (Jaime Fern?ndez del R?o) > 2. Re: composition of the steering council (was Re: Governance > model request) (Travis Oliphant) > 3. Re: Governance model request (Stefan van der Walt) > 4. Re: Governance model request (Matthew Brett) > 5. Re: composition of the steering council (was Re: Governance > model request) (Chris Barker) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Wed, 23 Sep 2015 14:06:08 -0700 > From: Jaime Fern?ndez del R?o > To: SciPy Developers List , Discussion of > Numerical Python > Subject: [Numpy-discussion] "Become an Open Source Contributor" > workshop > Message-ID: > > Content-Type: text/plain; charset="utf-8" > > Apologies for the cross-posting. > > The Data Science Student Society of the University of California San Diego, > or DS3 @ UCSD as they like to call themselves, will be holding biweekly > Python themed workshops starting this fall. On the week of October 19th, > they will be having yours truly doing a "Become an Open Source Contributor" > piece. It will be a shortish event, 60-90 minutes, so my idea was to cover > the following: > > 1. (15 min) An introduction to the Python data science landscape. > 2. (30 min) An overview of the GitHub workflow that most (all?) of the > projects follow. > 3. (30-45 min) A hands on session, where we would make sure everyone > gets set up in GitHub, and forks and clones their favorite project. Time > and participant willingness permitting, I would like to take advantage of > my commit bits, and have some of the participants submit a simple PR, e.g. > fixing a documentation typo, to NumPy or SciPy, and hit the green button > right there, so that they get to leave as knighted FOSS contributors. > > And this is what I am hoping to get from you, the community: > > 1. If anyone in the area would like to get involved, please contact me. > I have recruited a couple of volunteers from PySanDiego, but could use more > help. > 2. I'm also hoping to get some help, especially with the introductory > part. Given that the crowd will mostly be university students and some > faculty, it would be great if someone who actually knew what they were > talking about could deliver a short, 10 minute talk, on Python, data > science, and academia. I'm sure we could arrange it to have someone join > by video conference. > 3. If you have organized anything similar in the past, and have material > that I could use to, ahem, draw inspiration from, or recommendations to > make, or whatever, I'd love to hear from you. > > Thanks for reading! > > Jaime > From rays at blue-cove.com Thu Sep 24 10:41:57 2015 From: rays at blue-cove.com (R Schumacher) Date: Thu, 24 Sep 2015 07:41:57 -0700 Subject: [Numpy-discussion] Python needs goto In-Reply-To: References: Message-ID: <201509241442.t8OEg3Hp020817@blue-cove.com> An older non-decorator implementation, with examples (slower) http://entrian.com/goto/ At 04:25 AM 9/24/2015, you wrote: >Hello. > >Can you give an example where GOTO is useful ? >Le? 24 sept. 2015 07:24, "Charles R Harris" ><charlesr.harris at gmail.com> a ??crit? : >At last, goto for python! > >Usage: >from goto import with_goto > > >@with_goto > >def range(start, stop): > > i = start > > result = [] > > > label .begin > > if i == stop: > > goto .end > > > result.append(i) > > i += 1 > > goto .begin > > > label .end > > return result > >HT: LWN > >Chuck > >_______________________________________________ >NumPy-Discussion mailing list >NumPy-Discussion at scipy.org >https://mail.scipy.org/mailman/listinfo/numpy-discussion > >_______________________________________________ >NumPy-Discussion mailing list >NumPy-Discussion at scipy.org >https://mail.scipy.org/mailman/listinfo/numpy-discussion -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Thu Sep 24 13:03:39 2015 From: njs at pobox.com (Nathaniel Smith) Date: Thu, 24 Sep 2015 10:03:39 -0700 Subject: [Numpy-discussion] ANN: Numpy 1.10.0rc1 released. In-Reply-To: <1443093289.3544.12.camel@sipsolutions.net> References: <5603A38D.6090406@fysik.dtu.dk> <87vbb0qfjt.fsf@berkeley.edu> <1443093289.3544.12.camel@sipsolutions.net> Message-ID: On Sep 24, 2015 4:14 AM, "Sebastian Berg" wrote: > > On Do, 2015-09-24 at 03:26 -0700, Stefan van der Walt wrote: > > On 2015-09-24 00:17:33, Jens J?rgen Mortensen wrote: > > > jensj at jordan:~$ python > > > Python 2.7.9 (default, Apr 2 2015, 15:33:21) > > > [GCC 4.9.2] on linux2 > > > Type "help", "copyright", "credits" or "license" for more information. > > > >>> import numpy as np > > > >>> a = np.zeros((2, 2, 2)) > > > >>> b = np.zeros((2, 2, 2)) > > > >>> a[0, 0] = 7 > > > >>> b[0, 0] = 6 > > > >>> np.vdot(a[:, :, 0], b[:, :, 0]).real > > > 84.0 > > > >>> np.__version__ > > > '1.10.0rc1' > > > > > > The result should be 42. This is on Ubuntu 15.04. > > > > The input is not supposed to be matrices?but the docstring specifically > > states that those should then be ravelled, so this is indeed a bug. > > > > If I bisected correctly, the problematic change is this one: > > > > Hmmm, unless we want to make sure that the output of ravel is always > contiguous (which would be a difference from `.reshape(-1)`. > It would be a bit safer, and not a useless feature maybe, since we can > point to `.reshape(-1)` as well for those who do not care. > > As far as I can see a contiguous output was not guaranteed for > "keeporder", but nobody probably ever used keeporder.... I don't understand what these have to do with each other. If vdot is going to ravel then yes, it certainly needs to control the order of the raveling. (And there are lots of functions like this that implicitly ravel in numpy, for better or worse... Do we need to audit any others?) But this is "virtual" order that matters, right, nothing to do with storage order? -n -------------- next part -------------- An HTML attachment was scrubbed... URL: From sebastian at sipsolutions.net Thu Sep 24 13:18:44 2015 From: sebastian at sipsolutions.net (Sebastian Berg) Date: Thu, 24 Sep 2015 19:18:44 +0200 Subject: [Numpy-discussion] ANN: Numpy 1.10.0rc1 released. In-Reply-To: References: <5603A38D.6090406@fysik.dtu.dk> <87vbb0qfjt.fsf@berkeley.edu> <1443093289.3544.12.camel@sipsolutions.net> Message-ID: <1443115124.12219.8.camel@sipsolutions.net> On Do, 2015-09-24 at 10:03 -0700, Nathaniel Smith wrote: > On Sep 24, 2015 4:14 AM, "Sebastian Berg" > wrote: > > > > On Do, 2015-09-24 at 03:26 -0700, Stefan van der Walt wrote: > > > On 2015-09-24 00:17:33, Jens J?rgen Mortensen > wrote: > > > > jensj at jordan:~$ python > > > > Python 2.7.9 (default, Apr 2 2015, 15:33:21) > > > > [GCC 4.9.2] on linux2 > > > > Type "help", "copyright", "credits" or "license" for more > information. > > > > >>> import numpy as np > > > > >>> a = np.zeros((2, 2, 2)) > > > > >>> b = np.zeros((2, 2, 2)) > > > > >>> a[0, 0] = 7 > > > > >>> b[0, 0] = 6 > > > > >>> np.vdot(a[:, :, 0], b[:, :, 0]).real > > > > 84.0 > > > > >>> np.__version__ > > > > '1.10.0rc1' > > > > > > > > The result should be 42. This is on Ubuntu 15.04. > > > > > > The input is not supposed to be matrices?but the docstring > specifically > > > states that those should then be ravelled, so this is indeed a > bug. > > > > > > If I bisected correctly, the problematic change is this one: > > > > > > > Hmmm, unless we want to make sure that the output of ravel is always > > contiguous (which would be a difference from `.reshape(-1)`. > > It would be a bit safer, and not a useless feature maybe, since we > can > > point to `.reshape(-1)` as well for those who do not care. > > > > As far as I can see a contiguous output was not guaranteed for > > "keeporder", but nobody probably ever used keeporder.... > > I don't understand what these have to do with each other. If vdot is > going to ravel then yes, it certainly needs to control the order of > the raveling. (And there are lots of functions like this that > implicitly ravel in numpy, for better or worse... Do we need to audit > any others?) But this is "virtual" order that matters, right, nothing > to do with storage order? > No, this change also potentially changed the contiguity of the output, which does matter to vdot. `.reshape(-1)` will be a view for arrays such as `np.zeros((4, 4, 4)[..., 0]`. However, before, `ravel` would always return a *contiguous* chunk of memory (except for "keeporder"), so for the above case a copy would be forced. I think I underestimated the potential risk at the time, while I always thought of `ravel` to create a view whenever possible (and the documentation reads like it does), it never did and probably we should be careful to avoid possible problems for C-interfacing code. Asking everyone to use `reshape(-1)` instead if they want to have views whenever possible, is maybe not as pretty sometimes, but safe. I doubt "keeporder" should be special though, so I think I would also add forcing of a copy there. - Sebastian > -n > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > https://mail.scipy.org/mailman/listinfo/numpy-discussion -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 819 bytes Desc: This is a digitally signed message part URL: From cournape at gmail.com Thu Sep 24 13:45:05 2015 From: cournape at gmail.com (David Cournapeau) Date: Thu, 24 Sep 2015 18:45:05 +0100 Subject: [Numpy-discussion] composition of the steering council (was Re: Governance model request) In-Reply-To: <1443087947.1783.50.camel@sipsolutions.net> References: <1443087947.1783.50.camel@sipsolutions.net> Message-ID: On Thu, Sep 24, 2015 at 10:45 AM, Sebastian Berg wrote: > On Mi, 2015-09-23 at 17:08 -0600, Charles R Harris wrote: > > > > > > On Wed, Sep 23, 2015 at 3:21 PM, Travis Oliphant > > wrote: > > > > > > Regarding the seed council, I just tried to pick an > > objective > > criterion and an arbitrary date that seemed generally > > in keeping with > > idea of "should be active in the last > > 1-to-2-years-ish". Fiddling with > > the exact date in particular makes very little > > difference -- between > > pushing it back to 2 years ago today or forward to 1 > > year ago today, > > the only thing that changes is whether Pauli makes the > > list or not. > > (And Pauli is obviously a great council candidate, > > though I don't know > > whether he even wants to be on it.) > > > > > Personally, I have no idea how big the council > > should be. Too big, and > > > there is no point, consensus is harder to reach the > > larger the group, > > > and the main (only?) role of the council is to > > resolve issues where > > > consensus has not been reached in the larger > > community. But what is > > > too big? > > > > > > > As for make-up of the council, I think we need to > > expand beyond people > > > who have recently contributed core code. > > > > > > Yes, the council does need to have expertise to make > > technical > > > decisions, but if you think about the likely > > contentious issues like > > > ABI breakage, a core-code focused view is > > incomplete. So there should > > > be representation by: > > > > > > Someone(s) with a long history of working with the > > code -- that > > > institutional memory of why decisions were made the > > way they were > > > could be key. > > > > Sure -- though I can't really imagine any way of > > framing a rule like > > this that *wouldn't* be satisfied by Chuck + Ralf + > > Pauli, so my guess > > is that such a rule would not actually have any effect > > on the council > > membership in practice. > > > > > > As the original author of NumPy, I would like to be on the > > seed council as long as it is larger than 7 people. That is > > my proposal. I don't need to be a permanent member, but I > > do believe I have enough history that I can understand issues > > even if I haven't been working on code directly. > > > > > > I think I do bring history and information that provides all > > of the history that could be helpful on occasion. In > > addition, if a matter is important enough to even be brought > > to the attention of this council, I would like to be involved > > in the discussion about it. > > > > > > > > It's a simple change to the text --- basically an explanation > > that Travis requested to be on the seed council. > > > > > I too would like you to be a member. We could either write it into the > > text in recognition of your status as the Numpy creator, or it could > > be the first order of business. I would only ask that you give > > yourself some time to become familiar with how things work and the > > people involved in the current community. It has been some years since > > you have been active in code development. > > > > I think I can agree with that. On a serious note, I now realize that I > am probably the one with the most objection, so for everyone, do not > bother with trying to convince me, you probably cannot fully, nor do you > have to, I will let it stand as is after this and let others take over > from here (after this, probably whatever Chuck says is good). [1] > > More to the point of the actual members: > > So to say, I feel the council members have to try to be *directly* > active and see being active as a necessary *commitment* (i.e. also try > to travel to meetings). This will always be a difficult judgment of > course, but there is no help to it. The current definitions imply this. > And two years seems fine. It is not that short, at least unless someone > stops contributing very abruptly which I do not think is that usual. I > will weight in to keep the current times but do not feel very strongly. > > About using the commit log to seed, I think there are some old term > contributers (David Cournapeau maybe?), who never stopped doing quite a > bit but may not have merge commits. However, I think we can start of > with what we had, then I would hope Chuck and maybe Ralf can fill in the > blanks. > AFAIK, I still have merge commits. I am actually doing a bit of numpy development ATM, so I would prefer keeping them, but I won't fight it either. David > > About the size, I think if we get too many -- if that is possible -- we > should just change the governance at that time to be not veto based > anymore. This is something to keep in mind, but probably does not need > to be formalized. > > - Sebastian > > > [1] Sorry to "footnote" this, but I think I am probably rudely repeating > myself and frankly do **not want this to be discussed**. It is just to > try to be fully clear where I come from: > Until SciPy 2015, I could list many people on this list who have shown > more direct involvement in numpy then Travis since I joined and have no > affiliation to numpy. If Travis had been new to the community at the > time, I would be surprised if I would even recognize his name. > I know this is only half the picture and Travis already mentioned > another side, but this is what I mostly saw even if it may be a harsh > and rude assessment. > > > > > > Chuck > > > > > > > > > > > > _______________________________________________ > > NumPy-Discussion mailing list > > NumPy-Discussion at scipy.org > > https://mail.scipy.org/mailman/listinfo/numpy-discussion > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > https://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From alex.eberspaecher at gmail.com Thu Sep 24 13:54:19 2015 From: alex.eberspaecher at gmail.com (=?UTF-8?Q?Alexander_Ebersp=c3=a4cher?=) Date: Thu, 24 Sep 2015 19:54:19 +0200 Subject: [Numpy-discussion] Python needs goto In-Reply-To: References: Message-ID: <560438CB.7020400@gmail.com> On 24.09.2015 13:25, Christophe Bal wrote: > Can you give an example where GOTO is useful ? I think those pieces of code are best understood with some humour.. However, basically I can think two main causes for using goto: 1. Stop whatever your code is doing and jump towards the end of the program. However, this is mainly something useful for languages without exception handling and garbage collection. 2. Get out of something deeply nested. Also, this probably isn't very useful in Python as there's exception handling. Best Alex -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 473 bytes Desc: OpenPGP digital signature URL: From yarkot1 at gmail.com Thu Sep 24 14:13:10 2015 From: yarkot1 at gmail.com (Yarko Tymciurak) Date: Thu, 24 Sep 2015 13:13:10 -0500 Subject: [Numpy-discussion] Python needs goto In-Reply-To: <560438CB.7020400@gmail.com> References: <560438CB.7020400@gmail.com> Message-ID: On Thu, Sep 24, 2015 at 12:54 PM, Alexander Ebersp?cher < alex.eberspaecher at gmail.com> wrote: > On 24.09.2015 13:25, Christophe Bal wrote: > > > Can you give an example where GOTO is useful ? > > I think those pieces of code are best understood with some humour.. > > However, basically I can think two main causes for using goto: > > 1. Stop whatever your code is doing and jump towards the end of the > program. However, this is mainly something useful for languages without > exception handling and garbage collection. > > 2. Get out of something deeply nested. Also, this probably isn't very > useful in Python as there's exception handling. > I think there are more valid uses - I've read that "goto" basically is what a state machine does. Have a read of the brief implementation notes for "goto" in golang, for example. Goto may not be unreasonable to use, just most people would abuse. Sort of like "everyone shouldn't write assembly, but if you understand the machine, you can make good things happen". Without compiler/interpreter checks, more responsibility rests on the coder to keep out of trouble. > > Best > > Alex > > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > https://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From nevion at gmail.com Thu Sep 24 14:39:37 2015 From: nevion at gmail.com (Jason Newton) Date: Thu, 24 Sep 2015 14:39:37 -0400 Subject: [Numpy-discussion] Python needs goto In-Reply-To: <560438CB.7020400@gmail.com> References: <560438CB.7020400@gmail.com> Message-ID: In my experience, it's also come up with finite-state-machines where there's lots of loops. You might consider something like a long-lived client-loop on some socket, where states like try-connect, connected, and while-connected-and-everythings-ok exist and each can have it's own never ending loops and each loop jumps to each others section. It actually is more straightforward than adding unneeded layers like callbacks which will force you to design and pass around data, use auxiliary classes/structs, and it's easier to shoot yourself in the foot with burden and context loss that way. -Jason On Thu, Sep 24, 2015 at 1:54 PM, Alexander Ebersp?cher < alex.eberspaecher at gmail.com> wrote: > On 24.09.2015 13:25, Christophe Bal wrote: > > > Can you give an example where GOTO is useful ? > > I think those pieces of code are best understood with some humour.. > > However, basically I can think two main causes for using goto: > > 1. Stop whatever your code is doing and jump towards the end of the > program. However, this is mainly something useful for languages without > exception handling and garbage collection. > > 2. Get out of something deeply nested. Also, this probably isn't very > useful in Python as there's exception handling. > > Best > > Alex > > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > https://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Thu Sep 24 14:41:57 2015 From: charlesr.harris at gmail.com (Charles R Harris) Date: Thu, 24 Sep 2015 12:41:57 -0600 Subject: [Numpy-discussion] Python needs goto In-Reply-To: References: <560438CB.7020400@gmail.com> Message-ID: On Thu, Sep 24, 2015 at 12:13 PM, Yarko Tymciurak wrote: > > > On Thu, Sep 24, 2015 at 12:54 PM, Alexander Ebersp?cher < > alex.eberspaecher at gmail.com> wrote: > >> On 24.09.2015 13:25, Christophe Bal wrote: >> >> > Can you give an example where GOTO is useful ? >> >> I think those pieces of code are best understood with some humour.. >> >> However, basically I can think two main causes for using goto: >> >> 1. Stop whatever your code is doing and jump towards the end of the >> program. However, this is mainly something useful for languages without >> exception handling and garbage collection. >> >> 2. Get out of something deeply nested. Also, this probably isn't very >> useful in Python as there's exception handling. >> > > I think there are more valid uses - I've read that "goto" basically is > what a state machine does. > Have a read of the brief implementation notes for "goto" in golang, for > example. Goto may not be unreasonable to use, just most people would > abuse. Sort of like "everyone shouldn't write assembly, but if you > understand the machine, you can make good things happen". Without > compiler/interpreter checks, more responsibility rests on the coder to keep > out of trouble. > I would agree about state machines. When implemented using the standard control flow constructs they always look a bit artificial. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From ben.v.root at gmail.com Thu Sep 24 14:50:10 2015 From: ben.v.root at gmail.com (Benjamin Root) Date: Thu, 24 Sep 2015 14:50:10 -0400 Subject: [Numpy-discussion] Python needs goto In-Reply-To: References: <560438CB.7020400@gmail.com> Message-ID: Most of the time when I wanted to use goto in my early days, I found that breaks and continues were better and easier to understand. I will admit that there are occasional nested if/elif/else code that get messy without a goto. But which smells worse? A "goto" package or a complex if/elif/else? Ben Root On Thu, Sep 24, 2015 at 2:41 PM, Charles R Harris wrote: > > > On Thu, Sep 24, 2015 at 12:13 PM, Yarko Tymciurak > wrote: > >> >> >> On Thu, Sep 24, 2015 at 12:54 PM, Alexander Ebersp?cher < >> alex.eberspaecher at gmail.com> wrote: >> >>> On 24.09.2015 13:25, Christophe Bal wrote: >>> >>> > Can you give an example where GOTO is useful ? >>> >>> I think those pieces of code are best understood with some humour.. >>> >>> However, basically I can think two main causes for using goto: >>> >>> 1. Stop whatever your code is doing and jump towards the end of the >>> program. However, this is mainly something useful for languages without >>> exception handling and garbage collection. >>> >>> 2. Get out of something deeply nested. Also, this probably isn't very >>> useful in Python as there's exception handling. >>> >> >> I think there are more valid uses - I've read that "goto" basically is >> what a state machine does. >> Have a read of the brief implementation notes for "goto" in golang, for >> example. Goto may not be unreasonable to use, just most people would >> abuse. Sort of like "everyone shouldn't write assembly, but if you >> understand the machine, you can make good things happen". Without >> compiler/interpreter checks, more responsibility rests on the coder to keep >> out of trouble. >> > > I would agree about state machines. When implemented using the standard > control flow constructs they always look a bit artificial. > > Chuck > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > https://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From projetmbc at gmail.com Thu Sep 24 14:56:00 2015 From: projetmbc at gmail.com (Christophe Bal) Date: Thu, 24 Sep 2015 20:56:00 +0200 Subject: [Numpy-discussion] Python needs goto In-Reply-To: References: <560438CB.7020400@gmail.com> Message-ID: To be honest, I am not very conviced. Does someone can give a pseudo example ? Le 24 sept. 2015 20:50, "Benjamin Root" a ?crit : > Most of the time when I wanted to use goto in my early days, I found that > breaks and continues were better and easier to understand. I will admit > that there are occasional nested if/elif/else code that get messy without a > goto. But which smells worse? A "goto" package or a complex if/elif/else? > > Ben Root > > On Thu, Sep 24, 2015 at 2:41 PM, Charles R Harris < > charlesr.harris at gmail.com> wrote: > >> >> >> On Thu, Sep 24, 2015 at 12:13 PM, Yarko Tymciurak >> wrote: >> >>> >>> >>> On Thu, Sep 24, 2015 at 12:54 PM, Alexander Ebersp?cher < >>> alex.eberspaecher at gmail.com> wrote: >>> >>>> On 24.09.2015 13:25, Christophe Bal wrote: >>>> >>>> > Can you give an example where GOTO is useful ? >>>> >>>> I think those pieces of code are best understood with some humour.. >>>> >>>> However, basically I can think two main causes for using goto: >>>> >>>> 1. Stop whatever your code is doing and jump towards the end of the >>>> program. However, this is mainly something useful for languages without >>>> exception handling and garbage collection. >>>> >>>> 2. Get out of something deeply nested. Also, this probably isn't very >>>> useful in Python as there's exception handling. >>>> >>> >>> I think there are more valid uses - I've read that "goto" basically is >>> what a state machine does. >>> Have a read of the brief implementation notes for "goto" in golang, for >>> example. Goto may not be unreasonable to use, just most people would >>> abuse. Sort of like "everyone shouldn't write assembly, but if you >>> understand the machine, you can make good things happen". Without >>> compiler/interpreter checks, more responsibility rests on the coder to keep >>> out of trouble. >>> >> >> I would agree about state machines. When implemented using the standard >> control flow constructs they always look a bit artificial. >> >> Chuck >> >> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> https://mail.scipy.org/mailman/listinfo/numpy-discussion >> >> > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > https://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From projetmbc at gmail.com Thu Sep 24 14:57:13 2015 From: projetmbc at gmail.com (Christophe Bal) Date: Thu, 24 Sep 2015 20:57:13 +0200 Subject: [Numpy-discussion] Python needs goto In-Reply-To: References: <560438CB.7020400@gmail.com> Message-ID: By pseudo example, I mean pseudo code. Le 24 sept. 2015 20:56, "Christophe Bal" a ?crit : > To be honest, I am not very conviced. Does someone can give a pseudo > example ? > Le 24 sept. 2015 20:50, "Benjamin Root" a ?crit : > >> Most of the time when I wanted to use goto in my early days, I found that >> breaks and continues were better and easier to understand. I will admit >> that there are occasional nested if/elif/else code that get messy without a >> goto. But which smells worse? A "goto" package or a complex if/elif/else? >> >> Ben Root >> >> On Thu, Sep 24, 2015 at 2:41 PM, Charles R Harris < >> charlesr.harris at gmail.com> wrote: >> >>> >>> >>> On Thu, Sep 24, 2015 at 12:13 PM, Yarko Tymciurak >>> wrote: >>> >>>> >>>> >>>> On Thu, Sep 24, 2015 at 12:54 PM, Alexander Ebersp?cher < >>>> alex.eberspaecher at gmail.com> wrote: >>>> >>>>> On 24.09.2015 13:25, Christophe Bal wrote: >>>>> >>>>> > Can you give an example where GOTO is useful ? >>>>> >>>>> I think those pieces of code are best understood with some humour.. >>>>> >>>>> However, basically I can think two main causes for using goto: >>>>> >>>>> 1. Stop whatever your code is doing and jump towards the end of the >>>>> program. However, this is mainly something useful for languages without >>>>> exception handling and garbage collection. >>>>> >>>>> 2. Get out of something deeply nested. Also, this probably isn't very >>>>> useful in Python as there's exception handling. >>>>> >>>> >>>> I think there are more valid uses - I've read that "goto" basically is >>>> what a state machine does. >>>> Have a read of the brief implementation notes for "goto" in golang, for >>>> example. Goto may not be unreasonable to use, just most people would >>>> abuse. Sort of like "everyone shouldn't write assembly, but if you >>>> understand the machine, you can make good things happen". Without >>>> compiler/interpreter checks, more responsibility rests on the coder to keep >>>> out of trouble. >>>> >>> >>> I would agree about state machines. When implemented using the standard >>> control flow constructs they always look a bit artificial. >>> >>> Chuck >>> >>> >>> _______________________________________________ >>> NumPy-Discussion mailing list >>> NumPy-Discussion at scipy.org >>> https://mail.scipy.org/mailman/listinfo/numpy-discussion >>> >>> >> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> https://mail.scipy.org/mailman/listinfo/numpy-discussion >> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Thu Sep 24 15:12:46 2015 From: charlesr.harris at gmail.com (Charles R Harris) Date: Thu, 24 Sep 2015 13:12:46 -0600 Subject: [Numpy-discussion] Python needs goto In-Reply-To: References: <560438CB.7020400@gmail.com> Message-ID: On Thu, Sep 24, 2015 at 12:57 PM, Christophe Bal wrote: > By pseudo example, I mean pseudo code. > I find Cleve Moler's old Fortran version of Brent's zero finding algorithm using gotos clearer than the structured versions you can find in Numerical Recipes. The operation of the algorithm is easiest to describe as a finite state machine. As others have said, goto is also useful for languages without exception handling, C for instance. You will find gotos in the Linux kernel and in the Numpy core used for that purpose. However, the post was for laughs, I didn't expect serious discussion. The current thread is a pleasant surprise. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From travis at continuum.io Thu Sep 24 20:30:23 2015 From: travis at continuum.io (Travis Oliphant) Date: Thu, 24 Sep 2015 19:30:23 -0500 Subject: [Numpy-discussion] composition of the steering council (was Re: Governance model request) In-Reply-To: <1443087947.1783.50.camel@sipsolutions.net> References: <1443087947.1783.50.camel@sipsolutions.net> Message-ID: > > > > [1] Sorry to "footnote" this, but I think I am probably rudely repeating > myself and frankly do **not want this to be discussed**. It is just to > try to be fully clear where I come from: > Until SciPy 2015, I could list many people on this list who have shown > more direct involvement in numpy then Travis since I joined and have no > affiliation to numpy. If Travis had been new to the community at the > time, I would be surprised if I would even recognize his name. > I know this is only half the picture and Travis already mentioned > another side, but this is what I mostly saw even if it may be a harsh > and rude assessment. > > I do understand this. That's actually why I'm speaking up, because I don't think my activity has been understood by many people who have joined this list only recently. I don't want to interfere with your activity or impede your progress, or to be asked permission for anything. In fact, I want to understand how to best use my limited time to support things. You in particular are interested in indexing and fixing it --- the current code is there for a reason and some of the issues being discussed today have been discussed before --- though we have the benefit of hindsight now. I have mostly been behind the scenes helping people since about 2010 --- but still thinking a lot about NumPy, the downstream community, integration with other libraries, and where things could go. I don't have the time to commit major code changes, but I do have the time to contribute perspective and even a design idea or two from time to time. Obviously, nobody has to listen. I understand and appreciate that there are a lot of people that have contributed code and discussion since 2009 and to them it probably seems I'm just popping in and out --- and if you only look at the contributor log you can wonder "who is this guy...". But, I did do *a lot* of work to get NumPy off the ground. Quite a bit of that work was very lonely with people interested in the merger but pretty skeptical until the work was nearly done (and then many people helped finish it and get it working and tested). I wish I had been a better architect at the time (I can see now many things that would have been done differently). But, I'm still proud of the work I did in creating a foundation many could build on --- at the time nobody else was stepping up to do the job. Since that time, I have remained very interested in the success of NumPy and supporting the many *users* of NumPy. What I most bring to the current community is having observed many, many uses of NumPy in the wild --- from people who would never post to this list and whose use-cases are absent from discussion or misunderstood. I also bring knowledge about the wider Python ecosystem and the broader world outside of NumPy alone. The group is free to take my ideas and/or contributions or leave them. And I am also free to just review pull requests and contribute if and when I might. Best, -Travis > > > > > Chuck > > > > > > > > > > > > _______________________________________________ > > NumPy-Discussion mailing list > > NumPy-Discussion at scipy.org > > https://mail.scipy.org/mailman/listinfo/numpy-discussion > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > https://mail.scipy.org/mailman/listinfo/numpy-discussion > > -- *Travis Oliphant* *Co-founder and CEO* @teoliphant 512-222-5440 http://www.continuum.io -------------- next part -------------- An HTML attachment was scrubbed... URL: From alex.eberspaecher at gmail.com Fri Sep 25 01:53:41 2015 From: alex.eberspaecher at gmail.com (=?UTF-8?Q?Alexander_Ebersp=c3=a4cher?=) Date: Fri, 25 Sep 2015 07:53:41 +0200 Subject: [Numpy-discussion] Python needs goto In-Reply-To: References: <560438CB.7020400@gmail.com> Message-ID: <5604E165.7050704@gmail.com> On 24.09.2015 21:12, Charles R Harris wrote: > I find Cleve Moler's old Fortran version of Brent's zero finding > algorithm using gotos clearer than the structured versions you can find > in Numerical Recipes. The operation of the algorithm is easiest to > describe as a finite state machine. I need to look into that piece of Fortran code. I've never (knowingly) implemented a state-machine myself. However I can say that specifically Fortran has named loops, which probably helps to avoid many uses of goto. > However, the post was for laughs, I didn't expect serious discussion. Well, then let me fill in the missing piece - the obligatory XKCD strip: https://www.xkcd.com/292/ Alex -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 473 bytes Desc: OpenPGP digital signature URL: From projetmbc at gmail.com Fri Sep 25 07:11:30 2015 From: projetmbc at gmail.com (Christophe Bal) Date: Fri, 25 Sep 2015 13:11:30 +0200 Subject: [Numpy-discussion] Python needs goto In-Reply-To: <5604E165.7050704@gmail.com> References: <560438CB.7020400@gmail.com> <5604E165.7050704@gmail.com> Message-ID: Very funny ! ? Le 25 sept. 2015 07:53, "Alexander Ebersp?cher" a ?crit : > On 24.09.2015 21:12, Charles R Harris wrote: > > > I find Cleve Moler's old Fortran version of Brent's zero finding > > algorithm using gotos clearer than the structured versions you can find > > in Numerical Recipes. The operation of the algorithm is easiest to > > describe as a finite state machine. > > I need to look into that piece of Fortran code. I've never (knowingly) > implemented a state-machine myself. However I can say that specifically > Fortran has named loops, which probably helps to avoid many uses of goto. > > > However, the post was for laughs, I didn't expect serious discussion. > > Well, then let me fill in the missing piece - the obligatory XKCD strip: > > https://www.xkcd.com/292/ > > Alex > > > > > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > https://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Fri Sep 25 08:27:55 2015 From: njs at pobox.com (Nathaniel Smith) Date: Fri, 25 Sep 2015 05:27:55 -0700 Subject: [Numpy-discussion] composition of the steering council (was Re: Governance model request) In-Reply-To: References: <1443087947.1783.50.camel@sipsolutions.net> Message-ID: On Thu, Sep 24, 2015 at 10:45 AM, David Cournapeau wrote: > > On Thu, Sep 24, 2015 at 10:45 AM, Sebastian Berg > wrote: >> [...] >> About using the commit log to seed, I think there are some old term >> contributers (David Cournapeau maybe?), who never stopped doing quite a >> bit but may not have merge commits. However, I think we can start of >> with what we had, then I would hope Chuck and maybe Ralf can fill in the >> blanks. > > > AFAIK, I still have merge commits. I am actually doing a bit of numpy > development ATM, so I would prefer keeping them, but I won't fight it > either. For the record -- Sebastian's "not have merge commits" here I think means that looking at git history shows that you haven't reviewed/merged any other people's pull requests in the time period we've been looking at for the steering council discussion. AFAIK you do still have write permission to the repository on github, and there hasn't been any proposal to change those permissions around. (I'd certainly be in favor of *not* restricting commit permission to just the "steering council". But we can talk about that later once we've figured out what the steering council actually is :-).) -n -- Nathaniel J. Smith -- http://vorpus.org From njs at pobox.com Fri Sep 25 08:40:23 2015 From: njs at pobox.com (Nathaniel Smith) Date: Fri, 25 Sep 2015 05:40:23 -0700 Subject: [Numpy-discussion] composition of the steering council (was Re: Governance model request) In-Reply-To: References: Message-ID: [Travis: sorry for writing all this referring to you in third person -- I guess it might feel a bit awkward to read! You're certainly part of the intended audience, but then it felt even more awkward trying to write in second person... this is clearly a bug in English.] Hi all, On Wed, Sep 23, 2015 at 2:21 PM, Travis Oliphant wrote: > As the original author of NumPy, I would like to be on the seed council as > long as it is larger than 7 people. That is my proposal. I don't need > to be a permanent member, but I do believe I have enough history that I can > understand issues even if I haven't been working on code directly. > > I think I do bring history and information that provides all of the history > that could be helpful on occasion. In addition, if a matter is important > enough to even be brought to the attention of this council, I would like to > be involved in the discussion about it. Regarding the question of Travis being on the council: My overall feeling on this pretty neutral: I think it won't make much of a difference to NumPy really either way, because the important thing will be Travis's insights, available time to contribute, etc., and these will (I assume) be pretty much the same regardless of whether he's on the council or not. (Any matter so intractable that it actually needs the council's emergency powers will presumably be heralded by an epic multi-month message thread of doom, plus Travis has plenty of friends on the council who know where to find him when historical insight is needed, so I'm not worried about anyone missing out on a chance to be involved.) I'm sure we can make it work either way. But, I want to play devil's advocate for a bit, because there are some connected issues that IMHO we should at least think through if we're going to do this, and I think the easiest way to do that is to try and articulate the case against. So, here's the case against Travis being on the steering council immediately + an alternative option for comparison: ----- begin devil's advocate ----- First, just as a procedural matter, it seems clear that putting Travis on the council now will require changing the document in ways that violate Sebastian/Chris's concept of a "no names" rule -- at least in spirit if not in letter. The problem isn't just the initial council seeding; it's that Travis formally stepped down from the project more than 2.5 years ago, was mostly inactive for some time before that, and IIRC has been almost completely unheard-from between then and a few weeks ago. (Of course please correct me if I'm forgetting something, and obviously I'm talking with respect to numpy only -- clearly Travis has been doing tons of great things, but while numpy certainly benefits from e.g. the existence of NumFOCUS, creating NumFOCUS doesn't really feel like what we mean by "project activity" :-).) This means that as currently written, it's pretty unambiguous: he doesn't qualify to be added by the normal nomination process (requires "sustained" activity over the last year), and if he were added anyway (e.g. as part of the seed council) then according to the rules he would then be immediately removed for inactivity (requires being active within the last 1 year, or else within the 2 years plus affirmation that they planned to "return to active participation soon" -- this post hoc analysis requires a bit of squinting to apply at all, but it's pretty hard to reconcile with >2 years inactivity + his "moving on from numpy to blaze" post). To avoid referencing him by name we could add some text about "founding developers" or something as a fig leaf, but judging from Robert's last email it sounds like Travis is the only person in question, so this really would just be a fig leaf. Of course we have the option of modifying the rules to make this work -- I'm not really sure how to do this, but it's our text and I'm sure we can make it do whatever we want somehow. But any kinds of special case modifications for a single person create two problems: 1) Given that whether or not Travis is listed on the steering council probably won't affect the project much or at all, it could easily appear to an outside observer that the purpose of these rules changes was not to benefit NumPy, but only to benefit Travis's ego. Not that there's anything wrong with massaging Travis's ego :-). BUT, it sends a clear message (whether we mean it that way or not): that we think being on the steering council should affect one's ego. And there *is* something very wrong with this *message*. It's crucial that serving on the steering council be understood to be a job, not a privilege -- sort of like being release manager. It's an important and valued job, we're glad people volunteer, but it's an absolutely fundamental rule that council members do *not* get any special treatment outside of specific limited circumstances. If being on the steering council becomes a trophy or judgement of worth, and being left off it becomes an insult that implies someone's contributions are less worthy, then we are starting down the slippery slope that Matthew worries about, where there's an unspoken class distinction between the "important contributors" and "everyone else". This exact problem has destroyed the community in many F/OSS projects (see: BSDs, XFree86, lots of modern corporate-controlled projects). Emotions get high in these discussions, but it's important to remember that at the end of the day all this "steering council" stuff is just some bureaucracy we need to get organized so that we can get back to the real work that matters -- no-one's worth is up for debate, and the steering council is just one part of the whole project. And not even the most important part. 2) If we change the rules in one case, then it's hard to prove to outside observers that the rules really are the rules and are really applied equally to everyone. Brian Granger was telling us at SciPy how this has been a major challenge for Jupyter/IPython when working with large companies -- they really want to have confidence in the governance structure before they get involved. We don't want to end up in a conversation like: We'd like to contribute, but first we'd like some evidence that you're really a robust independent project and not subject to corporate capture. Well, here's our governance document, it clearly lays out the rules for contributions, and in particular how corporate contributions get no special privileges... Sure, that's what it says, but the first time a well-connected CEO who employs the leaders of substantial portions of the numerical ecosystem asked, you changed the rules to give him special privileges. Well, yes, but that was an extremely special case that had nothing to do with the corporate stuff you just said -- it was because Travis is just that awesome. Well, we are a faceless corporate monolith and have no idea who this Travis guy is, but okay, fine, he's just that awesome. Is anyone else that awesome? Where's the line -- what other special cases are there that are special enough to break the rules? The next time one comes up, will you follow the rules that you wrote down or do something else? ...we're not sure? ...that would kind of suck. So those are two problems that would apply for any special cases added to the rules, regardless of who particularly they were for. There's also the further concern that the steering council's main job is to "provide useful guidance, both technical and in terms of project direction, to potentially less experienced contributors" and "facilitate the ordinary community-based decision making procedure", and in Travis's case in particular, it's sort of unclear whether he's the best person for these jobs right now. Between his limited time (e.g. "I don't have time myself to engage in the community process" [1]) and the way the project has changed since he was actively involved, interactions in recent years have tended to be a bit awkward -- not because of any wrong-doing on anyone's part, but just because we're generally out-of-sync, resulting in e.g. self-merged pull requests [2][3] [glad to hear you don't do this anymore though!], or struggles to contribute to discussions based on limited context [4], or emails [5][6] that scare Chris Barker [7]. And usually with these kinds of discussions (e.g. for commit rights) you get enthusiastic unanimity, so Sebastian's unusually frank concerns give me serious pause [8]. Everyone else on the proposed steering council list certainly doesn't agree about everything, but we do all have ongoing relationships from interacting on the tracker and mailing list, making day-to-day decisions, and it's not clear to me that Travis even wants/is able to participate at that level currently. So what's the alternative option? I'd suggest: Keep the Jupyer/IPython rules for council member eligibility, and add Travis in a year when he becomes eligible by those rules. (Assuming he remains active and interested -- personally in his position I'd be tempted to just stick with the much cushier "Founder / Leader Emeritus" job -- all the respect, none of the day-to-day hassle ;-).) In the mean time we still get his insight and other contributions, and it provides a clear period for him ease into how we do things these days, which addresses Chuck and Sebastian's concerns. And, more importantly, it takes advantage of a unique opportunity: it takes the above negatives and turns them into positives. If later on someone feels slighted at not being on the steering council, we can say "look, even *Travis* isn't (wasn't) on the steering council -- you've misunderstood what this means", or if IBM comes calling we can say: Look, if we were going to break the rules for anyone, we would have broken them for Travis. But he followed the same process as everyone else. That's how we do things around here. I've had several long conversations with Travis recently, and one thing that's been very clear is that he still sees NumPy as being his baby and he's hungry to find ways to help it -- but maybe struggling to work out how to do that most effectively. It's a thing about babies -- eventually they grow up, become rebellious teenagers, and then -- if you did a good job parenting, and are very lucky -- they move out, have their own lives, and maybe call occasionally to ask for advice and money ;-). I'm not a parent, but I have been a child, and I understand this is often a challenging transition... In the case of NumPy, I think the last few years have shown that the project can more-or-less grow and succeed even without Travis -- but pulling a George Washington http://deadpresidents.tumblr.com/post/24687113413/did-washington-stepping-down-after-2-terms-impact like this would make a contribution to NumPy's long-term stability in a way that only Travis can do. ----- end devil's advocate ----- Again, I want to emphasize that while I've tried to make as strong a case as I can above, my actual belief right now is that NumPy will be just fine either way -- especially if we can think of ways to address and minimize the two specific problems described above. But at the very least I wanted to make sure they were on the table as concerns. -n [1] https://mail.scipy.org/pipermail/numpy-discussion/2015-September/073564.html [2] https://github.com/numpy/numpy/pull/2940 [3] https://github.com/numpy/numpy/pull/2772 [4] https://github.com/numpy/numpy/issues/5844#issuecomment-141723799 [5] https://mail.scipy.org/pipermail/numpy-discussion/2015-September/073562.html [6] https://mail.python.org/pipermail/python-dev/2015-September/141609.html [7] https://mail.scipy.org/pipermail/numpy-discussion/2015-September/073582.html [8] https://mail.scipy.org/pipermail/numpy-discussion/2015-September/073748.html -- Nathaniel J. Smith -- http://vorpus.org From tcaswell at gmail.com Fri Sep 25 10:15:25 2015 From: tcaswell at gmail.com (Thomas Caswell) Date: Fri, 25 Sep 2015 14:15:25 +0000 Subject: [Numpy-discussion] composition of the steering council (was Re: Governance model request) In-Reply-To: References: Message-ID: To respond to the devils advocate: Creating this organizational framework is a one time boot-strapping event. You could use wording like "The initial council will include those who have made significant contributions to numpy in the past and want to be on it" or "The initial council will be constructed by invitation by Nathaniel and Chuck". More objective criteria should be used going forward, but in terms of getting things spun up quickly doing things by fiat is probably ok. I am not even sure that the method by which the initial group is formed needs to go into the governing document. I think this addresses most of the concerns, IBM is happy (enough) because this was done as part of a one-time boot strapping operation of standing the rules up and there are no explicit names in the governing documents. It also acknowledges that there is a discontinuity/singularity in the governance of the project which means you get to do singular thing :). Tom On Fri, Sep 25, 2015 at 8:40 AM Nathaniel Smith wrote: > [Travis: sorry for writing all this referring to you in third person > -- I guess it might feel a bit awkward to read! You're certainly part > of the intended audience, but then it felt even more awkward trying to > write in second person... this is clearly a bug in English.] > > Hi all, > > On Wed, Sep 23, 2015 at 2:21 PM, Travis Oliphant > wrote: > > As the original author of NumPy, I would like to be on the seed council > as > > long as it is larger than 7 people. That is my proposal. I don't > need > > to be a permanent member, but I do believe I have enough history that I > can > > understand issues even if I haven't been working on code directly. > > > > I think I do bring history and information that provides all of the > history > > that could be helpful on occasion. In addition, if a matter is > important > > enough to even be brought to the attention of this council, I would like > to > > be involved in the discussion about it. > > Regarding the question of Travis being on the council: > > My overall feeling on this pretty neutral: I think it won't make much > of a difference to NumPy really either way, because the important > thing will be Travis's insights, available time to contribute, etc., > and these will (I assume) be pretty much the same regardless of > whether he's on the council or not. (Any matter so intractable that it > actually needs the council's emergency powers will presumably be > heralded by an epic multi-month message thread of doom, plus Travis > has plenty of friends on the council who know where to find him when > historical insight is needed, so I'm not worried about anyone missing > out on a chance to be involved.) I'm sure we can make it work either > way. > > But, I want to play devil's advocate for a bit, because there are some > connected issues that IMHO we should at least think through if we're > going to do this, and I think the easiest way to do that is to try and > articulate the case against. So, here's the case against Travis being > on the steering council immediately + an alternative option for > comparison: > > ----- begin devil's advocate ----- > > First, just as a procedural matter, it seems clear that putting Travis > on the council now will require changing the document in ways that > violate Sebastian/Chris's concept of a "no names" rule -- at least in > spirit if not in letter. The problem isn't just the initial council > seeding; it's that Travis formally stepped down from the project more > than 2.5 years ago, was mostly inactive for some time before that, and > IIRC has been almost completely unheard-from between then and a few > weeks ago. (Of course please correct me if I'm forgetting something, > and obviously I'm talking with respect to numpy only -- clearly Travis > has been doing tons of great things, but while numpy certainly > benefits from e.g. the existence of NumFOCUS, creating NumFOCUS > doesn't really feel like what we mean by "project activity" :-).) > > This means that as currently written, it's pretty unambiguous: he > doesn't qualify to be added by the normal nomination process (requires > "sustained" activity over the last year), and if he were added anyway > (e.g. as part of the seed council) then according to the rules he > would then be immediately removed for inactivity (requires being > active within the last 1 year, or else within the 2 years plus > affirmation that they planned to "return to active participation soon" > -- this post hoc analysis requires a bit of squinting to apply at all, > but it's pretty hard to reconcile with >2 years inactivity + his > "moving on from numpy to blaze" post). To avoid referencing him by > name we could add some text about "founding developers" or something > as a fig leaf, but judging from Robert's last email it sounds like > Travis is the only person in question, so this really would just be a > fig leaf. > > Of course we have the option of modifying the rules to make this work > -- I'm not really sure how to do this, but it's our text and I'm sure > we can make it do whatever we want somehow. But any kinds of special > case modifications for a single person create two problems: > > 1) Given that whether or not Travis is listed on the steering council > probably won't affect the project much or at all, it could easily > appear to an outside observer that the purpose of these rules changes > was not to benefit NumPy, but only to benefit Travis's ego. Not that > there's anything wrong with massaging Travis's ego :-). BUT, it sends > a clear message (whether we mean it that way or not): that we think > being on the steering council should affect one's ego. And there *is* > something very wrong with this *message*. > > It's crucial that serving on the steering council be understood to be > a job, not a privilege -- sort of like being release manager. It's an > important and valued job, we're glad people volunteer, but it's an > absolutely fundamental rule that council members do *not* get any > special treatment outside of specific limited circumstances. If being > on the steering council becomes a trophy or judgement of worth, and > being left off it becomes an insult that implies someone's > contributions are less worthy, then we are starting down the slippery > slope that Matthew worries about, where there's an unspoken class > distinction between the "important contributors" and "everyone else". > This exact problem has destroyed the community in many F/OSS projects > (see: BSDs, XFree86, lots of modern corporate-controlled projects). > > Emotions get high in these discussions, but it's important to remember > that at the end of the day all this "steering council" stuff is just > some bureaucracy we need to get organized so that we can get back to > the real work that matters -- no-one's worth is up for debate, and the > steering council is just one part of the whole project. And not even > the most important part. > > 2) If we change the rules in one case, then it's hard to prove to > outside observers that the rules really are the rules and are really > applied equally to everyone. Brian Granger was telling us at SciPy how > this has been a major challenge for Jupyter/IPython when working with > large companies -- they really want to have confidence in the > governance structure before they get involved. We don't want to end up > in a conversation like: > > We'd like to contribute, but first we'd like some evidence that > you're really a robust independent project and not subject to > corporate capture. > Well, here's our governance document, it clearly lays out the > rules for contributions, and in particular how corporate contributions > get no special privileges... > Sure, that's what it says, but the first time a well-connected > CEO who employs the leaders of substantial portions of the numerical > ecosystem asked, you changed the rules to give him special privileges. > Well, yes, but that was an extremely special case that had > nothing to do with the corporate stuff you just said -- it was because > Travis is just that awesome. > Well, we are a faceless corporate monolith and have no idea who > this Travis guy is, but okay, fine, he's just that awesome. Is anyone > else that awesome? Where's the line -- what other special cases are > there that are special enough to break the rules? The next time one > comes up, will you follow the rules that you wrote down or do > something else? > ...we're not sure? > > ...that would kind of suck. > > So those are two problems that would apply for any special cases added > to the rules, regardless of who particularly they were for. > > There's also the further concern that the steering council's main job > is to "provide useful guidance, both technical and in terms of project > direction, to potentially less experienced contributors" and > "facilitate the ordinary community-based decision making procedure", > and in Travis's case in particular, it's sort of unclear whether he's > the best person for these jobs right now. Between his limited time > (e.g. "I don't have time myself to engage in the community process" > [1]) and the way the project has changed since he was actively > involved, interactions in recent years have tended to be a bit awkward > -- not because of any wrong-doing on anyone's part, but just because > we're generally out-of-sync, resulting in e.g. self-merged pull > requests [2][3] [glad to hear you don't do this anymore though!], or > struggles to contribute to discussions based on limited context [4], > or emails [5][6] that scare Chris Barker [7]. And usually with these > kinds of discussions (e.g. for commit rights) you get enthusiastic > unanimity, so Sebastian's unusually frank concerns give me serious > pause [8]. Everyone else on the proposed steering council list > certainly doesn't agree about everything, but we do all have ongoing > relationships from interacting on the tracker and mailing list, making > day-to-day decisions, and it's not clear to me that Travis even > wants/is able to participate at that level currently. > > So what's the alternative option? I'd suggest: Keep the Jupyer/IPython > rules for council member eligibility, and add Travis in a year when he > becomes eligible by those rules. (Assuming he remains active and > interested -- personally in his position I'd be tempted to just stick > with the much cushier "Founder / Leader Emeritus" job -- all the > respect, none of the day-to-day hassle ;-).) In the mean time we still > get his insight and other contributions, and it provides a clear > period for him ease into how we do things these days, which addresses > Chuck and Sebastian's concerns. > > And, more importantly, it takes advantage of a unique opportunity: it > takes the above negatives and turns them into positives. If later on > someone feels slighted at not being on the steering council, we can > say "look, even *Travis* isn't (wasn't) on the steering council -- > you've misunderstood what this means", or if IBM comes calling we can > say: > > Look, if we were going to break the rules for anyone, we would > have broken them for Travis. But he followed the same process as > everyone else. That's how we do things around here. > > I've had several long conversations with Travis recently, and one > thing that's been very clear is that he still sees NumPy as being his > baby and he's hungry to find ways to help it -- but maybe struggling > to work out how to do that most effectively. It's a thing about babies > -- eventually they grow up, become rebellious teenagers, and then -- > if you did a good job parenting, and are very lucky -- they move out, > have their own lives, and maybe call occasionally to ask for advice > and money ;-). I'm not a parent, but I have been a child, and I > understand this is often a challenging transition... > > In the case of NumPy, I think the last few years have shown that the > project can more-or-less grow and succeed even without Travis -- but > pulling a George Washington > > http://deadpresidents.tumblr.com/post/24687113413/did-washington-stepping-down-after-2-terms-impact > like this would make a contribution to NumPy's long-term stability in > a way that only Travis can do. > > ----- end devil's advocate ----- > > Again, I want to emphasize that while I've tried to make as strong a > case as I can above, my actual belief right now is that NumPy will be > just fine either way -- especially if we can think of ways to address > and minimize the two specific problems described above. But at the > very least I wanted to make sure they were on the table as concerns. > > -n > > [1] > https://mail.scipy.org/pipermail/numpy-discussion/2015-September/073564.html > [2] https://github.com/numpy/numpy/pull/2940 > [3] https://github.com/numpy/numpy/pull/2772 > [4] https://github.com/numpy/numpy/issues/5844#issuecomment-141723799 > [5] > https://mail.scipy.org/pipermail/numpy-discussion/2015-September/073562.html > [6] > https://mail.python.org/pipermail/python-dev/2015-September/141609.html > [7] > https://mail.scipy.org/pipermail/numpy-discussion/2015-September/073582.html > [8] > https://mail.scipy.org/pipermail/numpy-discussion/2015-September/073748.html > > -- > Nathaniel J. Smith -- http://vorpus.org > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > https://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From sebastian at sipsolutions.net Fri Sep 25 10:25:20 2015 From: sebastian at sipsolutions.net (Sebastian Berg) Date: Fri, 25 Sep 2015 16:25:20 +0200 Subject: [Numpy-discussion] composition of the steering council (was Re: Governance model request) In-Reply-To: References: Message-ID: <1443191120.13246.6.camel@sipsolutions.net> On Mi, 2015-09-23 at 19:48 -0500, Travis Oliphant wrote: > > > On Wed, Sep 23, 2015 at 6:19 PM, Charles R Harris > wrote: > > > On Wed, Sep 23, 2015 at 3:42 PM, Chris Barker > wrote: > On Wed, Sep 23, 2015 at 2:21 PM, Travis Oliphant > wrote: > > As the original author of NumPy, I would like > to be on the seed council as long as it is > larger than 7 people. That is my proposal. > > > Or the seed council could invite Travis to join as its > first order of business :-) > > > Actually, maybe that's a way to handle it -- declare > that the first order of business for teh seed council > is to expand the council. > > > Perhaps we should specify a yearly meeting to review the past > year and nominate people for commit rights and council > membership. Long term, we might also want to start removing > commit rights, perhaps by adding a team category on github > with restricted rights -- committer emeritus, so to speak. > > > > That's a pretty good idea, actually. > > +1, lets make sure that dynamics of getting new people in and inactive out won't stall too easily. > > > > > > I'd still like some guidelines (suggestions) for > history and at least one major dependent-on-numpy rep. > Travis would certainly meet the history requirement -- > and maybe the other, too. :-) > > It's a simple change to the text --- basically > an explanation that Travis requested to be on > the seed council. > > > I'd rather the final draft of the document didn't name > names, but no biggie. > > > I'm fine with that too --- except you will need to name the initial > seed council. > > > -Travis > > > > > > > > Chuck > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > https://mail.scipy.org/mailman/listinfo/numpy-discussion > > > > > > -- > > Travis Oliphant > Co-founder and CEO > > > > > > @teoliphant > 512-222-5440 > http://www.continuum.io > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > https://mail.scipy.org/mailman/listinfo/numpy-discussion -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 819 bytes Desc: This is a digitally signed message part URL: From m.h.vankerkwijk at gmail.com Fri Sep 25 10:25:32 2015 From: m.h.vankerkwijk at gmail.com (Marten van Kerkwijk) Date: Fri, 25 Sep 2015 10:25:32 -0400 Subject: [Numpy-discussion] composition of the steering council (was Re: Governance model request) In-Reply-To: References: Message-ID: Hi Nathaniel, Piping in again from the outside: being on council would certainly seem to be a job not a privilege, and I suspect it will be hard enough to find people willing to actually put in work to worry about restricting membership overly. Given this, my suggestion would be to have a general, no-name escape clause to the rules, stating that anybody who does not formally meet the rules is welcome to *volunteer* being on the council, with a suggestion on what s/he might contribute, and the existing council can then decide whether or not that contribution is welcome. My sense would be that any time this happens one will be most happy to accept. All the best, Marten ? p.s. Independently of rules, I don't see how Travis would not qualify even from current work, given that he has just committed to actively try to improve/generalize dtype. -------------- next part -------------- An HTML attachment was scrubbed... URL: From wrw at mac.com Fri Sep 25 11:21:52 2015 From: wrw at mac.com (William Ray Wing) Date: Fri, 25 Sep 2015 11:21:52 -0400 Subject: [Numpy-discussion] Python needs goto In-Reply-To: <5604E165.7050704@gmail.com> References: <560438CB.7020400@gmail.com> <5604E165.7050704@gmail.com> Message-ID: <2D2EEE27-08DA-46C7-A05A-E5A37A10BB16@mac.com> > On Sep 25, 2015, at 1:53 AM, Alexander Ebersp?cher wrote: > > On 24.09.2015 21:12, Charles R Harris wrote: > >> I find Cleve Moler's old Fortran version of Brent's zero finding >> algorithm using gotos clearer than the structured versions you can find >> in Numerical Recipes. The operation of the algorithm is easiest to >> describe as a finite state machine. > > I need to look into that piece of Fortran code. I've never (knowingly) > implemented a state-machine myself. However I can say that specifically > Fortran has named loops, which probably helps to avoid many uses of goto. > GOTOs really can be useful. COMPUTED GOTOs, on the other hand, are a debugging nightmare, literally!!! Bill >> However, the post was for laughs, I didn't expect serious discussion. > > Well, then let me fill in the missing piece - the obligatory XKCD strip: > > https://www.xkcd.com/292/ > > Alex > > > > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > https://mail.scipy.org/mailman/listinfo/numpy-discussion From archibald at astron.nl Fri Sep 25 12:27:50 2015 From: archibald at astron.nl (Anne Archibald) Date: Fri, 25 Sep 2015 16:27:50 +0000 Subject: [Numpy-discussion] Python needs goto In-Reply-To: References: <560438CB.7020400@gmail.com> Message-ID: goto! and comefrom! Together with exceptions, threads, lambda, super, generators, and coroutines, all we're lacking is call-with-current-continuation for the full list of impenetrable control-flow constructs. Oh, and lisp-style resumable exception handling. (Suggested syntax: drop(exception, value) to return control to where the exception was raised and make the raise statement return value.) On Thu, Sep 24, 2015 at 8:42 PM Charles R Harris wrote: > On Thu, Sep 24, 2015 at 12:13 PM, Yarko Tymciurak > wrote: > >> >> >> I think there are more valid uses - I've read that "goto" basically is >> what a state machine does. >> Have a read of the brief implementation notes for "goto" in golang, for >> example. Goto may not be unreasonable to use, just most people would >> abuse. Sort of like "everyone shouldn't write assembly, but if you >> understand the machine, you can make good things happen". Without >> compiler/interpreter checks, more responsibility rests on the coder to keep >> out of trouble. >> > > I would agree about state machines. When implemented using the standard > control flow constructs they always look a bit artificial. > > That depends what your "standard" control flow constructs are. Has anyone tried implementing a state machine using coroutines? They seem like a rather natural setup: each state is a coroutine that loops, doing the appropriate actions for the state and then handing control over to the coroutine for the next state. Anne -------------- next part -------------- An HTML attachment was scrubbed... URL: From ben.v.root at gmail.com Fri Sep 25 12:40:27 2015 From: ben.v.root at gmail.com (Benjamin Root) Date: Fri, 25 Sep 2015 12:40:27 -0400 Subject: [Numpy-discussion] Python needs goto In-Reply-To: References: <560438CB.7020400@gmail.com> Message-ID: Ow! Ow! Ow! I am just a meteorologist that has an obsession with looking up unfamiliar technology terms. I need a Tylenol... Ben Root On Fri, Sep 25, 2015 at 12:27 PM, Anne Archibald wrote: > goto! and comefrom! Together with exceptions, threads, lambda, super, > generators, and coroutines, all we're lacking is > call-with-current-continuation for the full list of impenetrable > control-flow constructs. Oh, and lisp-style resumable exception handling. > (Suggested syntax: drop(exception, value) to return control to where the > exception was raised and make the raise statement return value.) > > On Thu, Sep 24, 2015 at 8:42 PM Charles R Harris < > charlesr.harris at gmail.com> wrote: > >> On Thu, Sep 24, 2015 at 12:13 PM, Yarko Tymciurak >> wrote: >> >>> >>> >>> I think there are more valid uses - I've read that "goto" basically is >>> what a state machine does. >>> Have a read of the brief implementation notes for "goto" in golang, for >>> example. Goto may not be unreasonable to use, just most people would >>> abuse. Sort of like "everyone shouldn't write assembly, but if you >>> understand the machine, you can make good things happen". Without >>> compiler/interpreter checks, more responsibility rests on the coder to keep >>> out of trouble. >>> >> >> I would agree about state machines. When implemented using the standard >> control flow constructs they always look a bit artificial. >> >> > That depends what your "standard" control flow constructs are. Has anyone > tried implementing a state machine using coroutines? They seem like a > rather natural setup: each state is a coroutine that loops, doing the > appropriate actions for the state and then handing control over to the > coroutine for the next state. > > Anne > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > https://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Fri Sep 25 14:06:01 2015 From: charlesr.harris at gmail.com (Charles R Harris) Date: Fri, 25 Sep 2015 12:06:01 -0600 Subject: [Numpy-discussion] Python needs goto In-Reply-To: References: <560438CB.7020400@gmail.com> Message-ID: On Fri, Sep 25, 2015 at 10:27 AM, Anne Archibald wrote: > goto! and comefrom! Together with exceptions, threads, lambda, super, > generators, and coroutines, all we're lacking is > call-with-current-continuation for the full list of impenetrable > control-flow constructs. Oh, and lisp-style resumable exception handling. > (Suggested syntax: drop(exception, value) to return control to where the > exception was raised and make the raise statement return value.) > > On Thu, Sep 24, 2015 at 8:42 PM Charles R Harris < > charlesr.harris at gmail.com> wrote: > >> On Thu, Sep 24, 2015 at 12:13 PM, Yarko Tymciurak >> wrote: >> >>> >>> >>> I think there are more valid uses - I've read that "goto" basically is >>> what a state machine does. >>> Have a read of the brief implementation notes for "goto" in golang, for >>> example. Goto may not be unreasonable to use, just most people would >>> abuse. Sort of like "everyone shouldn't write assembly, but if you >>> understand the machine, you can make good things happen". Without >>> compiler/interpreter checks, more responsibility rests on the coder to keep >>> out of trouble. >>> >> >> I would agree about state machines. When implemented using the standard >> control flow constructs they always look a bit artificial. >> >> > That depends what your "standard" control flow constructs are. Has anyone > tried implementing a state machine using coroutines? They seem like a > rather natural setup: each state is a coroutine that loops, doing the > appropriate actions for the state and then handing control over to the > coroutine for the next state. > > Might well do. TAOCP has an example elevator, passenger simulation that, IIRC, used coroutines. I think I may have even once used a language that had them (Algol 68?). It will be interesting to see what their inclusion in Python 3.5 leads to. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From ehermes at chem.wisc.edu Fri Sep 25 15:04:40 2015 From: ehermes at chem.wisc.edu (Eric Hermes) Date: Fri, 25 Sep 2015 19:04:40 +0000 Subject: [Numpy-discussion] Including .f2py_f2cmap in numpy.distutils? Message-ID: Hello, I am attempting to set up a numpy.distutils setup.py for a small python program that uses a Fortran module. Currently, the setup is able to compile and install the program seemingly successfully, but the f2py script erroneously maps the data types I am using to float, rather than double. I have the proper mapping set up in a .f2py_f2cmap in the source directory, but it does not seem to be copied to the build directory at compile time, and I cannot figure out how to make it get copied. Is there a simple way to do what I am trying to do? Alternatively, is there a way to specify the mapping in my setup.py scripts? Here's a github repo with the project: https://github.com/ehermes/ased3 Thanks, Eric Hermes -------------- next part -------------- An HTML attachment was scrubbed... URL: From thomas.haslwanter at gmail.com Fri Sep 25 16:22:27 2015 From: thomas.haslwanter at gmail.com (Thomas Haslwanter) Date: Fri, 25 Sep 2015 22:22:27 +0200 Subject: [Numpy-discussion] "Numpy for Matlab Users" is missing Message-ID: Does anybody know what happened to the following links on the NumPy homepage (http://www.numpy.org/): - http://wiki.scipy.org/Wiki/NumPy_for_Matlab_Users - http://www.numpy.org/ - http://wiki.scipy.org/Numpy_Functions_by_Category They have not been available for some time now ... -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Fri Sep 25 17:02:41 2015 From: njs at pobox.com (Nathaniel Smith) Date: Fri, 25 Sep 2015 14:02:41 -0700 Subject: [Numpy-discussion] Python needs goto In-Reply-To: References: <560438CB.7020400@gmail.com> Message-ID: On Sep 25, 2015 11:06 AM, "Charles R Harris" wrote: > > > > On Fri, Sep 25, 2015 at 10:27 AM, Anne Archibald wrote: >> >> goto! and comefrom! Together with exceptions, threads, lambda, super, generators, and coroutines, all we're lacking is call-with-current-continuation for the full list of impenetrable control-flow constructs. Oh, and lisp-style resumable exception handling. (Suggested syntax: drop(exception, value) to return control to where the exception was raised and make the raise statement return value.) >> >> On Thu, Sep 24, 2015 at 8:42 PM Charles R Harris < charlesr.harris at gmail.com> wrote: >>> >>> On Thu, Sep 24, 2015 at 12:13 PM, Yarko Tymciurak wrote: >>>> >>>> >>>> >>>> I think there are more valid uses - I've read that "goto" basically is what a state machine does. >>>> Have a read of the brief implementation notes for "goto" in golang, for example. Goto may not be unreasonable to use, just most people would abuse. Sort of like "everyone shouldn't write assembly, but if you understand the machine, you can make good things happen". Without compiler/interpreter checks, more responsibility rests on the coder to keep out of trouble. >>> >>> >>> I would agree about state machines. When implemented using the standard control flow constructs they always look a bit artificial. >>> >> >> That depends what your "standard" control flow constructs are. Has anyone tried implementing a state machine using coroutines? They seem like a rather natural setup: each state is a coroutine that loops, doing the appropriate actions for the state and then handing control over to the coroutine for the next state. >> > > Might well do. TAOCP has an example elevator, passenger simulation that, IIRC, used coroutines. I think I may have even once used a language that had them (Algol 68?). It will be interesting to see what their inclusion in Python 3.5 leads to. The coroutines in 3.5 are just syntactic sugar around features that were added in *2*.5 (yield expressions and yield from), so no need to wait :-). They fall far short of arbitrary continuations, though. I don't think it's particularly cumbersome to implement a state machine with one function per state and the return value specifying the new state -- there's a little more typing to set up the local variables and such each time, but if you have complex states then this seems like even an advantage over goto (since with goto you may have to set up local state by hand each time, basically reimplementing your own scoping mechanism). Of course if all your states are trivial then this overhead is just annoying. ...I suspect you actually might be able to implement resumable exceptions in cpython as a (very evil) third party module. I'm going to stop thinking about that now before I get too curious and do something I regret. Actually, don't stackless / greenlet give you full continuations? -n -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Fri Sep 25 17:32:44 2015 From: njs at pobox.com (Nathaniel Smith) Date: Fri, 25 Sep 2015 14:32:44 -0700 Subject: [Numpy-discussion] "Numpy for Matlab Users" is missing In-Reply-To: References: Message-ID: The wiki is down, presumably as part of this: https://github.com/numpy/numpy/issues/6325 On Sep 25, 2015 1:22 PM, "Thomas Haslwanter" wrote: > Does anybody know what happened to the following links on the NumPy > homepage (http://www.numpy.org/): > > > - http://wiki.scipy.org/Wiki/NumPy_for_Matlab_Users > - http://www.numpy.org/ > - http://wiki.scipy.org/Numpy_Functions_by_Category > > They have not been available for some time now ... > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > https://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rmay31 at gmail.com Fri Sep 25 17:39:26 2015 From: rmay31 at gmail.com (Ryan May) Date: Fri, 25 Sep 2015 15:39:26 -0600 Subject: [Numpy-discussion] Python needs goto In-Reply-To: References: <560438CB.7020400@gmail.com> Message-ID: On Fri, Sep 25, 2015 at 3:02 PM, Nathaniel Smith wrote: > > The coroutines in 3.5 are just syntactic sugar around features that were > added in *2*.5 (yield expressions and yield from), so no need to wait :-). > They fall far short of arbitrary continuations, though. > Correction: Python 3.4 gained "yield from". Prior to that, you had a lot of work to properly delegate from one generator to another. But yes, async and await are just syntactic sugar (consistent with other languages) for python 3.4's coroutine functionality. Ryan -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Fri Sep 25 18:59:39 2015 From: njs at pobox.com (Nathaniel Smith) Date: Fri, 25 Sep 2015 15:59:39 -0700 Subject: [Numpy-discussion] Python needs goto In-Reply-To: References: <560438CB.7020400@gmail.com> Message-ID: On Fri, Sep 25, 2015 at 2:39 PM, Ryan May wrote: > > On Fri, Sep 25, 2015 at 3:02 PM, Nathaniel Smith wrote: >> >> The coroutines in 3.5 are just syntactic sugar around features that were >> added in *2*.5 (yield expressions and yield from), so no need to wait :-). >> They fall far short of arbitrary continuations, though. > > > Correction: Python 3.4 gained "yield from". Prior to that, you had a lot of > work to properly delegate from one generator to another. Ah, right, thanks for the correction. Not sure how much yield from matters for state machines, but certainly it's great for async IO. -n -- Nathaniel J. Smith -- http://vorpus.org From njs at pobox.com Fri Sep 25 19:57:19 2015 From: njs at pobox.com (Nathaniel Smith) Date: Fri, 25 Sep 2015 16:57:19 -0700 Subject: [Numpy-discussion] composition of the steering council (was Re: Governance model request) In-Reply-To: References: Message-ID: On Fri, Sep 25, 2015 at 7:25 AM, Marten van Kerkwijk wrote: [...] > > p.s. Independently of rules, I don't see how Travis would not qualify even > from current work, given that he has just committed to actively try to > improve/generalize dtype. Just to clarify what's happening here (at least as I understand it -- hopefully Travis / Continuum folks will be in communication with the community at some point), it sounds like the plan is not so much that Travis will be working on the dtype stuff (he has no time) as that he's hiring other people to work on it: "Travis (Continuum) is working with 3 part-time people (currently Kerry Oliphant, Christian Tismer and Yarko Tymciurak) with additional advisement from Irwin Zaid, Jeff Reback, and Antoine Pitrou. The target of this effort is two Python packages (currently called memtype and gufunc) which could serve as prototypes and/or scaffolding and/or implementations of the same ideas that get into NumPy. They will be working and communicating with the NumPy community. Travis's brother, Kerry Oliphant, is leading the group and tasked with communicating with the broader community about the activities of this team." -- https://bids.hackpad.com/Python-Stack-Refactoring-uU9RVWkMc0J So this seems orthogonal to the steering council discussion -- decisions about steering council membership are 100% about actual work done by actual individuals, without any regard to who is writing checks to whom. (This is a key part of the whole maintaining-independence, keeping-corporate-influence-at-arms-length aspect of the governance plan.) (It also sounds like Continuum's current plan is that instead of improving numpy directly, they will be off building a toy array library to explore the idea of implementing this stuff directly in the core interpreter as builtin types. And in parallel, those of us who are interested in this stuff in numpy will be getting to work on actually implementing new dtype stuff in numpy. And I guess at some point we'll compare notes and see how close our two implementations ended up being in retrospect? Right now I don't understand any path for their work to be useful to numpy, and it sounds like for now the numpy team should just proceed with our existing refactoring plans on the assumption that nothing will come of the Continuum work, and if that's wrong then it will be a pleasant surprise. But Travis is aware of my thoughts here and keen to do it this way anyway, and it's their money and time, so, up to them really...) -n -- Nathaniel J. Smith -- http://vorpus.org From sebastian at sipsolutions.net Sat Sep 26 16:03:49 2015 From: sebastian at sipsolutions.net (Sebastian Berg) Date: Sat, 26 Sep 2015 22:03:49 +0200 Subject: [Numpy-discussion] composition of the steering council (was Re: Governance model request) In-Reply-To: References: Message-ID: <1443297829.13246.109.camel@sipsolutions.net> On Fr, 2015-09-25 at 14:15 +0000, Thomas Caswell wrote: > To respond to the devils advocate: > > > Creating this organizational framework is a one time boot-strapping > event. You could use wording like "The initial council will include > those who have made significant contributions to numpy in the past and > want to be on it" or "The initial council will be constructed by > invitation by Nathaniel and Chuck". More objective criteria should be > used going forward, but in terms of getting things spun up quickly > doing things by fiat is probably ok. I am not even sure that the > method by which the initial group is formed needs to go into the > governing document. > I think you are probably right, we probably do not need to document how exactly people were picked to be in the seed. At least unless the final list creates a headache for someone in the community, in which case a formal definition may be easier for consensus. Maybe I can say that if we agree to the current proposal, and you, Travis, say that you plan to be more directly active/visible than the last few years, then I am happy if you are in the seed. Plus, I do have a feeling that this might come more easy now if we do plan more regular meetings and maybe discussions on some larger issues. However, I still do have a bit of a bad taste about doing an exception for you to stay should you not find the time. This is just because I like the current rules and do not like exceptions much. As I said, I will not get in the way of any consensus saying otherwise though and I am sure there are many ways to change the current draft that even I will like ;)! - Sebastian > > I think this addresses most of the concerns, IBM is happy (enough) > because this was done as part of a one-time boot strapping operation > of standing the rules up and there are no explicit names in the > governing documents. It also acknowledges that there is a > discontinuity/singularity in the governance of the project which means > you get to do singular thing :). > > > Tom > > > > On Fri, Sep 25, 2015 at 8:40 AM Nathaniel Smith wrote: > > [Travis: sorry for writing all this referring to you in third > person > -- I guess it might feel a bit awkward to read! You're > certainly part > of the intended audience, but then it felt even more awkward > trying to > write in second person... this is clearly a bug in English.] > > Hi all, > > On Wed, Sep 23, 2015 at 2:21 PM, Travis Oliphant > wrote: > > As the original author of NumPy, I would like to be on the > seed council as > > long as it is larger than 7 people. That is my proposal. > I don't need > > to be a permanent member, but I do believe I have enough > history that I can > > understand issues even if I haven't been working on code > directly. > > > > I think I do bring history and information that provides all > of the history > > that could be helpful on occasion. In addition, if a > matter is important > > enough to even be brought to the attention of this council, > I would like to > > be involved in the discussion about it. > > Regarding the question of Travis being on the council: > > My overall feeling on this pretty neutral: I think it won't > make much > of a difference to NumPy really either way, because the > important > thing will be Travis's insights, available time to contribute, > etc., > and these will (I assume) be pretty much the same regardless > of > whether he's on the council or not. (Any matter so intractable > that it > actually needs the council's emergency powers will presumably > be > heralded by an epic multi-month message thread of doom, plus > Travis > has plenty of friends on the council who know where to find > him when > historical insight is needed, so I'm not worried about anyone > missing > out on a chance to be involved.) I'm sure we can make it work > either > way. > > But, I want to play devil's advocate for a bit, because there > are some > connected issues that IMHO we should at least think through if > we're > going to do this, and I think the easiest way to do that is to > try and > articulate the case against. So, here's the case against > Travis being > on the steering council immediately + an alternative option > for > comparison: > > ----- begin devil's advocate ----- > > First, just as a procedural matter, it seems clear that > putting Travis > on the council now will require changing the document in ways > that > violate Sebastian/Chris's concept of a "no names" rule -- at > least in > spirit if not in letter. The problem isn't just the initial > council > seeding; it's that Travis formally stepped down from the > project more > than 2.5 years ago, was mostly inactive for some time before > that, and > IIRC has been almost completely unheard-from between then and > a few > weeks ago. (Of course please correct me if I'm forgetting > something, > and obviously I'm talking with respect to numpy only -- > clearly Travis > has been doing tons of great things, but while numpy certainly > benefits from e.g. the existence of NumFOCUS, creating > NumFOCUS > doesn't really feel like what we mean by "project > activity" :-).) > > This means that as currently written, it's pretty unambiguous: > he > doesn't qualify to be added by the normal nomination process > (requires > "sustained" activity over the last year), and if he were added > anyway > (e.g. as part of the seed council) then according to the rules > he > would then be immediately removed for inactivity (requires > being > active within the last 1 year, or else within the 2 years plus > affirmation that they planned to "return to active > participation soon" > -- this post hoc analysis requires a bit of squinting to apply > at all, > but it's pretty hard to reconcile with >2 years inactivity + > his > "moving on from numpy to blaze" post). To avoid referencing > him by > name we could add some text about "founding developers" or > something > as a fig leaf, but judging from Robert's last email it sounds > like > Travis is the only person in question, so this really would > just be a > fig leaf. > > Of course we have the option of modifying the rules to make > this work > -- I'm not really sure how to do this, but it's our text and > I'm sure > we can make it do whatever we want somehow. But any kinds of > special > case modifications for a single person create two problems: > > 1) Given that whether or not Travis is listed on the steering > council > probably won't affect the project much or at all, it could > easily > appear to an outside observer that the purpose of these rules > changes > was not to benefit NumPy, but only to benefit Travis's ego. > Not that > there's anything wrong with massaging Travis's ego :-). BUT, > it sends > a clear message (whether we mean it that way or not): that we > think > being on the steering council should affect one's ego. And > there *is* > something very wrong with this *message*. > > It's crucial that serving on the steering council be > understood to be > a job, not a privilege -- sort of like being release manager. > It's an > important and valued job, we're glad people volunteer, but > it's an > absolutely fundamental rule that council members do *not* get > any > special treatment outside of specific limited circumstances. > If being > on the steering council becomes a trophy or judgement of > worth, and > being left off it becomes an insult that implies someone's > contributions are less worthy, then we are starting down the > slippery > slope that Matthew worries about, where there's an unspoken > class > distinction between the "important contributors" and "everyone > else". > This exact problem has destroyed the community in many F/OSS > projects > (see: BSDs, XFree86, lots of modern corporate-controlled > projects). > > Emotions get high in these discussions, but it's important to > remember > that at the end of the day all this "steering council" stuff > is just > some bureaucracy we need to get organized so that we can get > back to > the real work that matters -- no-one's worth is up for debate, > and the > steering council is just one part of the whole project. And > not even > the most important part. > > 2) If we change the rules in one case, then it's hard to prove > to > outside observers that the rules really are the rules and are > really > applied equally to everyone. Brian Granger was telling us at > SciPy how > this has been a major challenge for Jupyter/IPython when > working with > large companies -- they really want to have confidence in the > governance structure before they get involved. We don't want > to end up > in a conversation like: > > We'd like to contribute, but first we'd like some > evidence that > you're really a robust independent project and not subject to > corporate capture. > Well, here's our governance document, it clearly lays > out the > rules for contributions, and in particular how corporate > contributions > get no special privileges... > Sure, that's what it says, but the first time a > well-connected > CEO who employs the leaders of substantial portions of the > numerical > ecosystem asked, you changed the rules to give him special > privileges. > Well, yes, but that was an extremely special case that > had > nothing to do with the corporate stuff you just said -- it was > because > Travis is just that awesome. > Well, we are a faceless corporate monolith and have no > idea who > this Travis guy is, but okay, fine, he's just that awesome. Is > anyone > else that awesome? Where's the line -- what other special > cases are > there that are special enough to break the rules? The next > time one > comes up, will you follow the rules that you wrote down or do > something else? > ...we're not sure? > > ...that would kind of suck. > > So those are two problems that would apply for any special > cases added > to the rules, regardless of who particularly they were for. > > There's also the further concern that the steering council's > main job > is to "provide useful guidance, both technical and in terms of > project > direction, to potentially less experienced contributors" and > "facilitate the ordinary community-based decision making > procedure", > and in Travis's case in particular, it's sort of unclear > whether he's > the best person for these jobs right now. Between his limited > time > (e.g. "I don't have time myself to engage in the community > process" > [1]) and the way the project has changed since he was actively > involved, interactions in recent years have tended to be a bit > awkward > -- not because of any wrong-doing on anyone's part, but just > because > we're generally out-of-sync, resulting in e.g. self-merged > pull > requests [2][3] [glad to hear you don't do this anymore > though!], or > struggles to contribute to discussions based on limited > context [4], > or emails [5][6] that scare Chris Barker [7]. And usually with > these > kinds of discussions (e.g. for commit rights) you get > enthusiastic > unanimity, so Sebastian's unusually frank concerns give me > serious > pause [8]. Everyone else on the proposed steering council list > certainly doesn't agree about everything, but we do all have > ongoing > relationships from interacting on the tracker and mailing > list, making > day-to-day decisions, and it's not clear to me that Travis > even > wants/is able to participate at that level currently. > > So what's the alternative option? I'd suggest: Keep the > Jupyer/IPython > rules for council member eligibility, and add Travis in a year > when he > becomes eligible by those rules. (Assuming he remains active > and > interested -- personally in his position I'd be tempted to > just stick > with the much cushier "Founder / Leader Emeritus" job -- all > the > respect, none of the day-to-day hassle ;-).) In the mean time > we still > get his insight and other contributions, and it provides a > clear > period for him ease into how we do things these days, which > addresses > Chuck and Sebastian's concerns. > > And, more importantly, it takes advantage of a unique > opportunity: it > takes the above negatives and turns them into positives. If > later on > someone feels slighted at not being on the steering council, > we can > say "look, even *Travis* isn't (wasn't) on the steering > council -- > you've misunderstood what this means", or if IBM comes calling > we can > say: > > Look, if we were going to break the rules for anyone, > we would > have broken them for Travis. But he followed the same process > as > everyone else. That's how we do things around here. > > I've had several long conversations with Travis recently, and > one > thing that's been very clear is that he still sees NumPy as > being his > baby and he's hungry to find ways to help it -- but maybe > struggling > to work out how to do that most effectively. It's a thing > about babies > -- eventually they grow up, become rebellious teenagers, and > then -- > if you did a good job parenting, and are very lucky -- they > move out, > have their own lives, and maybe call occasionally to ask for > advice > and money ;-). I'm not a parent, but I have been a child, and > I > understand this is often a challenging transition... > > In the case of NumPy, I think the last few years have shown > that the > project can more-or-less grow and succeed even without Travis > -- but > pulling a George Washington > http://deadpresidents.tumblr.com/post/24687113413/did-washington-stepping-down-after-2-terms-impact > like this would make a contribution to NumPy's long-term > stability in > a way that only Travis can do. > > ----- end devil's advocate ----- > > Again, I want to emphasize that while I've tried to make as > strong a > case as I can above, my actual belief right now is that NumPy > will be > just fine either way -- especially if we can think of ways to > address > and minimize the two specific problems described above. But at > the > very least I wanted to make sure they were on the table as > concerns. > > -n > > [1] > https://mail.scipy.org/pipermail/numpy-discussion/2015-September/073564.html > [2] https://github.com/numpy/numpy/pull/2940 > [3] https://github.com/numpy/numpy/pull/2772 > [4] > https://github.com/numpy/numpy/issues/5844#issuecomment-141723799 > [5] > https://mail.scipy.org/pipermail/numpy-discussion/2015-September/073562.html > [6] > https://mail.python.org/pipermail/python-dev/2015-September/141609.html > [7] > https://mail.scipy.org/pipermail/numpy-discussion/2015-September/073582.html > [8] > https://mail.scipy.org/pipermail/numpy-discussion/2015-September/073748.html > > -- > Nathaniel J. Smith -- http://vorpus.org > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > https://mail.scipy.org/mailman/listinfo/numpy-discussion > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > https://mail.scipy.org/mailman/listinfo/numpy-discussion -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 819 bytes Desc: This is a digitally signed message part URL: From charlesr.harris at gmail.com Sun Sep 27 14:24:33 2015 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sun, 27 Sep 2015 12:24:33 -0600 Subject: [Numpy-discussion] Numpy 1.10.0rc2 coming Monday, Sep 28. Message-ID: Hi All, Just a heads up. If you have encountered any unreported errors with rc1, please let us know. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From antony.lee at berkeley.edu Sun Sep 27 15:58:31 2015 From: antony.lee at berkeley.edu (Antony Lee) Date: Sun, 27 Sep 2015 12:58:31 -0700 Subject: [Numpy-discussion] Fixing the dtype of np.full's return value Message-ID: Hi all, The docstring of np.full indicates that the result of the dtype is `np.array(fill_value).dtype`, as long as the keyword argument `dtype` itself is not set. This is actually not the case: the current implementation always returns a float array when `dtype` is not set, see e.g. In [1]: np.full(1, 1) Out[1]: array([ 1.]) In [2]: np.full(1, None) Out[2]: array([ nan]) In [3]: np.full(1, None).dtype Out[3]: dtype('float64') In [4]: np.array(None) Out[4]: array(None, dtype=object) The note about return value of the dtype was actually explicitly discussed in https://github.com/numpy/numpy/pull/2875 but the tests failed to cover the case where the `dtype` argument is not passed. We could either change the docstring to match the current behavior, or fix the behavior to match what the docstring says (my preference). @njsmith mentioned in https://github.com/numpy/numpy/issues/6366 that this may be acceptable as a bug fix, as "it's a very new function so there probably aren't many people relying on it" (it was introduced in 1.8). I guess the options are: - Fix the behavior outright and squeeze this in 1.10 as a bugfix (my preference). - Emit a warning in 1.10, fix in 1.11. - Do nothing for 1.10, warn in 1.11, fix in 1.12 (at that point the argument of `np.full` being a very new function starts becoming invalid...). - Change the docstring. Thoughts? Antony -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Sun Sep 27 18:31:43 2015 From: njs at pobox.com (Nathaniel Smith) Date: Sun, 27 Sep 2015 15:31:43 -0700 Subject: [Numpy-discussion] Fixing the dtype of np.full's return value In-Reply-To: References: Message-ID: On Sep 27, 2015 12:58 PM, "Antony Lee" wrote: > > Hi all, > > The docstring of np.full indicates that the result of the dtype is `np.array(fill_value).dtype`, as long as the keyword argument `dtype` itself is not set. This is actually not the case: the current implementation always returns a float array when `dtype` is not set, see e.g. > > In [1]: np.full(1, 1) > Out[1]: array([ 1.]) > > In [2]: np.full(1, None) > Out[2]: array([ nan]) > > In [3]: np.full(1, None).dtype > Out[3]: dtype('float64') > > In [4]: np.array(None) > Out[4]: array(None, dtype=object) > > The note about return value of the dtype was actually explicitly discussed in https://github.com/numpy/numpy/pull/2875 but the tests failed to cover the case where the `dtype` argument is not passed. > > We could either change the docstring to match the current behavior, or fix the behavior to match what the docstring says (my preference). @njsmith mentioned in https://github.com/numpy/numpy/issues/6366 that this may be acceptable as a bug fix, as "it's a very new function so there probably aren't many people relying on it" (it was introduced in 1.8). > > I guess the options are: > - Fix the behavior outright and squeeze this in 1.10 as a bugfix (my preference). > - Emit a warning in 1.10, fix in 1.11. > - Do nothing for 1.10, warn in 1.11, fix in 1.12 (at that point the argument of `np.full` being a very new function starts becoming invalid...). > - Change the docstring. If this had been caught last month, I'd be in favor of making the change in the 1.10 prereleases and seeing if it caused any problems. At this point I think it's too risky for the 1.10 cycle, because we're very near the final release so there's little chance to notice if this does cause problems. Given that, my vote is for a FutureWarning in 1.10 and change it in 1.11 (assuming we don't get a chorus of complaints before then). -n -------------- next part -------------- An HTML attachment was scrubbed... URL: From pearu.peterson at gmail.com Mon Sep 28 09:28:09 2015 From: pearu.peterson at gmail.com (Pearu Peterson) Date: Mon, 28 Sep 2015 16:28:09 +0300 Subject: [Numpy-discussion] Including .f2py_f2cmap in numpy.distutils? In-Reply-To: References: Message-ID: Hi, Currently, .f2py_f2cmap must be located in a directory where setup.py or f2py.py is called (to be exact, where numpy.f2py.capi_maps is imported). This location is hardcoded and there is no way to specify the file location within setup.py scripts. However, you don't need to use .f2py_f2cmap file for specifying the correct mapping. Read the code in numpy/f2py/capi_maps.py and you'll find that inserting the following codelet to setup.py file might work (untested code): from numpy.f2py.capi_maps import f2c_map f2c_map['real'].update(sp='float', dp='double', qp='long_double') HTH, Pearu On Fri, Sep 25, 2015 at 10:04 PM, Eric Hermes wrote: > Hello, > > > > I am attempting to set up a numpy.distutils setup.py for a small python > program that uses a Fortran module. Currently, the setup is able to compile > and install the program seemingly successfully, but the f2py script > erroneously maps the data types I am using to float, rather than double. I > have the proper mapping set up in a .f2py_f2cmap in the source directory, > but it does not seem to be copied to the build directory at compile time, > and I cannot figure out how to make it get copied. Is there a simple way to > do what I am trying to do? Alternatively, is there a way to specify the > mapping in my setup.py scripts? > > > > Here's a github repo with the project: > > > > https://github.com/ehermes/ased3 > > > > Thanks, > > Eric Hermes > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > https://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From gael.varoquaux at normalesup.org Mon Sep 28 17:15:36 2015 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Mon, 28 Sep 2015 23:15:36 +0200 Subject: [Numpy-discussion] New version of "scipy lecture notes" Message-ID: <20150928211536.GI2445658@phare.normalesup.org> Dear Pythonistas, We have just released a new version of the "scipy lecture notes": http://www.scipy-lectures.org/ These are a consistent set of materials to learn the core aspects of the scientific Python ecosystem, from beginner to expert. They are written and maintained by a set of volunteers and published under a CC-BY license. Highlights of the latest version includes: * a chapter giving a introduction to statistics in Python * a new layout with emphasis on readability including on small devices * fully doctesting for Python 2 and 3 compatibility We hope that you will find these notes useful, for you, your colleagues, or your students. Ga?l From charlesr.harris at gmail.com Tue Sep 29 00:38:42 2015 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 28 Sep 2015 22:38:42 -0600 Subject: [Numpy-discussion] Numpy 1.10.0rc2 released Message-ID: Hi all, I'm pleased to announce the availability of Numpy 1.10.0rc12. Sources and 32 bit binary packages for Windows may be found at Sourceforge . There have been a few fixes since rc1. If there are no more problems I hope to release the final in a week or so. Cheers Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Tue Sep 29 04:06:57 2015 From: njs at pobox.com (Nathaniel Smith) Date: Tue, 29 Sep 2015 01:06:57 -0700 Subject: [Numpy-discussion] composition of the steering council (was Re: Governance model request) In-Reply-To: References: Message-ID: On Fri, Sep 25, 2015 at 7:15 AM, Thomas Caswell wrote: > To respond to the devils advocate: > > Creating this organizational framework is a one time boot-strapping event. > You could use wording like "The initial council will include those who have > made significant contributions to numpy in the past and want to be on it" or > "The initial council will be constructed by invitation by Nathaniel and > Chuck". More objective criteria should be used going forward, but in terms > of getting things spun up quickly doing things by fiat is probably ok. I am > not even sure that the method by which the initial group is formed needs to > go into the governing document. The problem is that according to the current text, not only is Travis ineligible to join the council (it's a little weird to put people on the seed council who wouldn't be eligible to join it normally, but okay, sure) -- he's not even eligible to stay on the council once he joins. So we would need to change the text no matter what. Which we can do, if we decide that that's what we need to do to accomplish what we want. It's our text, after all. I think it's extremely important though that what we actually do, and what we write down saying we will do, somehow match. Otherwise this whole exercise has no point. -n -- Nathaniel J. Smith -- http://vorpus.org From travis at continuum.io Tue Sep 29 10:32:30 2015 From: travis at continuum.io (Travis Oliphant) Date: Tue, 29 Sep 2015 10:32:30 -0400 Subject: [Numpy-discussion] composition of the steering council (was Re: Governance model request) In-Reply-To: References: Message-ID: Thanks for the candid discussion and for expressing concerns freely. I think Nathaniel's "parenting" characterization of NumPy from me is pretty accurate. I do feel a responsibility for the *stuff* that's out there, and that is what drives me. I do see the many contributions from others and really learn from them as well. I have seen conversations on this list and others have characterizations of history that I don't agree with which affects decisions that are being made --- and so I feel compelled to try and share my view. I'm in a situation now where at least for 6 months or so I can help with NumPy more than I have been able to for 7 years. Focusing on the initial governance text, my issues are that 1) 1 year of inactivity to be removed from the council is too little for a long-running project like NumPy --- somewhere between 2 and 4 years would be more appropriate. I suppose 1 year of inactivity is fine if that is defined only as "failure to vote on matters before the council" 2) The seed council should not just be recent contributors but should include as many people as are willing to help who have a long history with the project. 3) I think people who contribute significantly generally should be able to re-join the steering council more easily than "going through the 1-year vetting process" again --- they would have to be approved by the current steering council but it should not be automatically disallowed (thus requiring the equivalent of an amendment to change it). I applaud the fact that the steering council will not be and should not be used except when absolutely necessary and for limited functions. Thanks, -Travis On Tue, Sep 29, 2015 at 4:06 AM, Nathaniel Smith wrote: > On Fri, Sep 25, 2015 at 7:15 AM, Thomas Caswell > wrote: > > To respond to the devils advocate: > > > > Creating this organizational framework is a one time boot-strapping > event. > > You could use wording like "The initial council will include those who > have > > made significant contributions to numpy in the past and want to be on > it" or > > "The initial council will be constructed by invitation by Nathaniel and > > Chuck". More objective criteria should be used going forward, but in > terms > > of getting things spun up quickly doing things by fiat is probably ok. > I am > > not even sure that the method by which the initial group is formed needs > to > > go into the governing document. > > The problem is that according to the current text, not only is Travis > ineligible to join the council (it's a little weird to put people on > the seed council who wouldn't be eligible to join it normally, but > okay, sure) -- he's not even eligible to stay on the council once he > joins. So we would need to change the text no matter what. > > Which we can do, if we decide that that's what we need to do to > accomplish what we want. It's our text, after all. I think it's > extremely important though that what we actually do, and what we write > down saying we will do, somehow match. Otherwise this whole exercise > has no point. > > -n > > -- > Nathaniel J. Smith -- http://vorpus.org > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > https://mail.scipy.org/mailman/listinfo/numpy-discussion > -- *Travis Oliphant* *Co-founder and CEO* @teoliphant 512-222-5440 http://www.continuum.io -------------- next part -------------- An HTML attachment was scrubbed... URL: From juha.jeronen at jyu.fi Tue Sep 29 10:35:01 2015 From: juha.jeronen at jyu.fi (Juha Jeronen) Date: Tue, 29 Sep 2015 17:35:01 +0300 Subject: [Numpy-discussion] Cython-based OpenMP-accelerated quartic polynomial solver Message-ID: <560AA195.8030001@jyu.fi> Hi all, I recently developed a Cython-based, OpenMP-accelerated quartic (and cubic, quadratic) polynomial solver to address a personal research need for quickly solving a very large number of independent low-degree polynomial equations for both real and complex coefficients. For example, on an Intel i7-4710MQ, running with 8 threads this code solves approximately 1.2e7 quartic polynomial equations per second. (With OMP_NUM_THREADS=4 the solver gives approximately 8.9e6 solutions per second, so it seems HyperThreading in this case helps about 30%.) The algorithms for cubics and quadratics come from Numerical Recipes (3rd ed.), and the quartic problem is internally reduced to a cubic and two quadratics, using well-known standard tricks. Since to my understanding, for solving polynomial equations NumPy only provides roots(), which works one problem at a time, I'm writing to inquire whether there would be interest to include this functionality to NumPy, if I contribute the code (and clean it up for integration)? I have some previous open source contribution experience. I have authored the IVTC and Phosphor deinterlacers for VLC, and a modular postprocessing filter framework for the Panda3D game engine (posted at the forums on panda3d.org, projected for version 1.10). Currently the polynomial solver is available in a git repository hosted by our university, the relevant files being: https://yousource.it.jyu.fi/jjrandom2/miniprojects/blobs/master/misc/tworods/polysolve2.pyx https://yousource.it.jyu.fi/jjrandom2/miniprojects/blobs/master/misc/tworods/polysolve.py https://yousource.it.jyu.fi/jjrandom2/miniprojects/blobs/master/misc/tworods/polysolve2example.py https://yousource.it.jyu.fi/jjrandom2/miniprojects/blobs/master/misc/tworods/setup.py https://yousource.it.jyu.fi/jjrandom2/miniprojects/blobs/master/misc/tworods/compile.sh I'm working on a research grant, which allows me to hold the copyright; license is BSD. Polysolve2 is the fast Cython+OpenMP version, while polysolve is an earlier (slower) experimental version using only NumPy array operations. The example module contains a usage example, and setup (in the standard manner) instructs distutils and Cython as to how to build polysolve2 (including setting the relevant math flags and enabling OpenMP). (The setup script includes also some stuff specific to my solver for the Ziegler problem; basically, the "tworods" module can be ignored. I apologize for the inconvenience.) Thoughts? Best regards, -J ------------------------------------------------- Juha Jeronen, Ph.D. juha.jeronen at jyu.fi University of Jyv?skyl? Department of Mathematical Information Technology ------------------------------------------------- From charlesr.harris at gmail.com Tue Sep 29 11:13:15 2015 From: charlesr.harris at gmail.com (Charles R Harris) Date: Tue, 29 Sep 2015 09:13:15 -0600 Subject: [Numpy-discussion] Sign of NaN Message-ID: Hi All, Due to a recent commit, Numpy master now raises an error when applying the sign function to an object array containing NaN. Other options may be preferable, returning NaN for instance, so I would like to open the topic for discussion on the list. Thoughts? Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From freddyrietdijk at fridh.nl Tue Sep 29 11:17:58 2015 From: freddyrietdijk at fridh.nl (Freddy Rietdijk) Date: Tue, 29 Sep 2015 17:17:58 +0200 Subject: [Numpy-discussion] Sign of NaN In-Reply-To: References: Message-ID: I wouldn't know of any valid output when applying the sign function to NaN. Therefore, I think it is correct to return a ValueError. Furthermore, I would prefer such an error over just returning NaN since it helps you locating where NaN is generated. On Tue, Sep 29, 2015 at 5:13 PM, Charles R Harris wrote: > Hi All, > > Due to a recent commit, Numpy master now raises an error when applying the > sign function to an object array containing NaN. Other options may be > preferable, returning NaN for instance, so I would like to open the topic > for discussion on the list. > > Thoughts? > > Chuck > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > https://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From archibald at astron.nl Tue Sep 29 11:25:13 2015 From: archibald at astron.nl (Anne Archibald) Date: Tue, 29 Sep 2015 15:25:13 +0000 Subject: [Numpy-discussion] Sign of NaN In-Reply-To: References: Message-ID: IEEE 754 has signum(NaN)->NaN. So does np.sign on floating-point arrays. Why should it be different for object arrays? Anne P.S. If you want exceptions when NaNs appear, that's what np.seterr is for. -A On Tue, Sep 29, 2015 at 5:18 PM Freddy Rietdijk wrote: > I wouldn't know of any valid output when applying the sign function to > NaN. Therefore, I think it is correct to return a ValueError. Furthermore, > I would prefer such an error over just returning NaN since it helps you > locating where NaN is generated. > > On Tue, Sep 29, 2015 at 5:13 PM, Charles R Harris < > charlesr.harris at gmail.com> wrote: > >> Hi All, >> >> Due to a recent commit, Numpy master now raises an error when applying >> the sign function to an object array containing NaN. Other options may be >> preferable, returning NaN for instance, so I would like to open the topic >> for discussion on the list. >> >> Thoughts? >> >> Chuck >> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> https://mail.scipy.org/mailman/listinfo/numpy-discussion >> >> > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > https://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From josef.pktd at gmail.com Tue Sep 29 11:39:15 2015 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Tue, 29 Sep 2015 11:39:15 -0400 Subject: [Numpy-discussion] Sign of NaN In-Reply-To: References: Message-ID: On Tue, Sep 29, 2015 at 11:25 AM, Anne Archibald wrote: > IEEE 754 has signum(NaN)->NaN. So does np.sign on floating-point arrays. > Why should it be different for object arrays? > > Anne > > P.S. If you want exceptions when NaNs appear, that's what np.seterr is > for. -A > I also think NaN should be treated the same way as floating point numbers (whatever that is). Otherwise it is difficult to remember when nan is essentially a float dtype or another dtype. (given that float is the smallest dtype that can hold a nan) Josef > > On Tue, Sep 29, 2015 at 5:18 PM Freddy Rietdijk > wrote: > >> I wouldn't know of any valid output when applying the sign function to >> NaN. Therefore, I think it is correct to return a ValueError. Furthermore, >> I would prefer such an error over just returning NaN since it helps you >> locating where NaN is generated. >> >> On Tue, Sep 29, 2015 at 5:13 PM, Charles R Harris < >> charlesr.harris at gmail.com> wrote: >> >>> Hi All, >>> >>> Due to a recent commit, Numpy master now raises an error when applying >>> the sign function to an object array containing NaN. Other options may be >>> preferable, returning NaN for instance, so I would like to open the topic >>> for discussion on the list. >>> >>> Thoughts? >>> >>> Chuck >>> >>> _______________________________________________ >>> NumPy-Discussion mailing list >>> NumPy-Discussion at scipy.org >>> https://mail.scipy.org/mailman/listinfo/numpy-discussion >>> >>> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> https://mail.scipy.org/mailman/listinfo/numpy-discussion >> > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > https://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Tue Sep 29 11:43:53 2015 From: charlesr.harris at gmail.com (Charles R Harris) Date: Tue, 29 Sep 2015 09:43:53 -0600 Subject: [Numpy-discussion] Sign of NaN In-Reply-To: References: Message-ID: On Tue, Sep 29, 2015 at 9:25 AM, Anne Archibald wrote: > IEEE 754 has signum(NaN)->NaN. So does np.sign on floating-point arrays. > Why should it be different for object arrays? > What about non-numeric objects in general ? Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From allanhaldane at gmail.com Tue Sep 29 11:44:10 2015 From: allanhaldane at gmail.com (Allan Haldane) Date: Tue, 29 Sep 2015 11:44:10 -0400 Subject: [Numpy-discussion] Sign of NaN In-Reply-To: References: Message-ID: <560AB1CA.2040109@gmail.com> On 09/29/2015 11:39 AM, josef.pktd at gmail.com wrote: > > > On Tue, Sep 29, 2015 at 11:25 AM, Anne Archibald > wrote: > > IEEE 754 has signum(NaN)->NaN. So does np.sign on floating-point > arrays. Why should it be different for object arrays? > > Anne > > P.S. If you want exceptions when NaNs appear, that's what np.seterr > is for. -A > > > > I also think NaN should be treated the same way as floating point > numbers (whatever that is). Otherwise it is difficult to remember when > nan is essentially a float dtype or another dtype. > (given that float is the smallest dtype that can hold a nan) > Note that I've reimplemented np.sign for object arrays along these lines in this open PR: https://github.com/numpy/numpy/pull/6320 That PR recursively uses the np.sign ufunc to evaluate object arrays containing float and complex numbers. This way the behavior on object arrays is identical to float/complex arrays. Here is what the np.sign ufunc does (for arbitrary x): np.sign(np.nan) -> nan np.sign(complex(np.nan, x)) -> complex(nan, 0) np.sign(complex(x, np.nan)) -> complex(nan, 0) Allan From solipsis at pitrou.net Tue Sep 29 12:14:51 2015 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 29 Sep 2015 18:14:51 +0200 Subject: [Numpy-discussion] Sign of NaN References: Message-ID: <20150929181451.2bf09822@fsol> On Tue, 29 Sep 2015 09:13:15 -0600 Charles R Harris wrote: > > Due to a recent commit, Numpy master now raises an error when applying the > sign function to an object array containing NaN. Other options may be > preferable, returning NaN for instance, so I would like to open the topic > for discussion on the list. None for example? float('nan') may be a bit weird amongst e.g. an array of Decimals. Regards Antoine. From matti.picus at gmail.com Tue Sep 29 12:16:39 2015 From: matti.picus at gmail.com (Matti Picus) Date: Tue, 29 Sep 2015 19:16:39 +0300 Subject: [Numpy-discussion] Cython-based OpenMP-accelerated quartic, polynomial solver Message-ID: <560AB967.3000305@gmail.com> An HTML attachment was scrubbed... URL: From joferkington at gmail.com Tue Sep 29 12:40:35 2015 From: joferkington at gmail.com (Joe Kington) Date: Tue, 29 Sep 2015 11:40:35 -0500 Subject: [Numpy-discussion] Sign of NaN In-Reply-To: <20150929181451.2bf09822@fsol> References: <20150929181451.2bf09822@fsol> Message-ID: On Tue, Sep 29, 2015 at 11:14 AM, Antoine Pitrou wrote: > > None for example? float('nan') may be a bit weird amongst e.g. an array > of Decimals The downside to `None` is that it's one more thing to check for and makes object arrays an even weirder edge case. (Incidentally, Decimal does have its own non-float NaN which throws a whole different wrench in the works. ` np.sign(Decimal('NaN'))` is going to raise an error no matter what.) A float (or numpy) NaN makes more sense to return for mixed datatypes than None does, in my opinion. At least then one can use `isfinite`, etc to check while `np.isfinite(None)` will raise an error. Furthermore, if there's at least one floating point NaN in the object array, getting a float NaN out makes sense. Just my $0.02, anyway. -------------- next part -------------- An HTML attachment was scrubbed... URL: From juha.jeronen at jyu.fi Tue Sep 29 13:15:24 2015 From: juha.jeronen at jyu.fi (Juha Jeronen) Date: Tue, 29 Sep 2015 20:15:24 +0300 Subject: [Numpy-discussion] Cython-based OpenMP-accelerated quartic, polynomial solver In-Reply-To: <560AB967.3000305@gmail.com> References: <560AB967.3000305@gmail.com> Message-ID: <560AC72C.8000907@jyu.fi> Hi, On 29.09.2015 19:16, Matti Picus wrote: >> The algorithms for cubics and quadratics come from Numerical Recipes >> (3rd ed.), and the quartic problem is internally reduced to a cubic and >> two quadratics, using well-known standard tricks. > Nice, wll documented code. > Just to be sure you are on safe ground, you took the *algorithms* but no *code* from Numerical Recipes, right? They are pretty clear about their licensing, see > http://www.nr.com/licenses/redistribute.html > This bit from the site might be relevant: > "We never give permission for Numerical Recipes source code to be posted on any public server, or distributed with any freeware or shareware package." > Matti Yes, the code is original (written from scratch), only the algorithms come from the book. -J ------------------------------------------------- Juha Jeronen, Ph.D. juha.jeronen at jyu.fi University of Jyv?skyl? Department of Mathematical Information Technology ------------------------------------------------- -------------- next part -------------- An HTML attachment was scrubbed... URL: From shoyer at gmail.com Tue Sep 29 13:59:47 2015 From: shoyer at gmail.com (Stephan Hoyer) Date: Tue, 29 Sep 2015 10:59:47 -0700 Subject: [Numpy-discussion] Sign of NaN In-Reply-To: References: Message-ID: On Tue, Sep 29, 2015 at 8:13 AM, Charles R Harris wrote: > Due to a recent commit, Numpy master now raises an error when applying the > sign function to an object array containing NaN. Other options may be > preferable, returning NaN for instance, so I would like to open the topic > for discussion on the list. > We discussed this last month on the list and on GitHub: https://mail.scipy.org/pipermail/numpy-discussion/2015-August/073503.html https://github.com/numpy/numpy/issues/6265 https://github.com/numpy/numpy/pull/6269/files The discussion was focused on what to do in the generic fallback case. Now that I think about this more, I think it makes sense to explicitly check for NaN in the unorderable case, and return NaN is the input is NaN. I would not return NaN in general from unorderable objects, though -- in general we should raise an error. It sounds like Allan has already fixed this in his PR, but it also would not be hard to add that logic to the existing code. Is this code in the NumPy 1.10? Stephan -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Tue Sep 29 14:15:24 2015 From: charlesr.harris at gmail.com (Charles R Harris) Date: Tue, 29 Sep 2015 12:15:24 -0600 Subject: [Numpy-discussion] Sign of NaN In-Reply-To: References: Message-ID: On Tue, Sep 29, 2015 at 11:59 AM, Stephan Hoyer wrote: > On Tue, Sep 29, 2015 at 8:13 AM, Charles R Harris < > charlesr.harris at gmail.com> wrote: > >> Due to a recent commit, Numpy master now raises an error when applying >> the sign function to an object array containing NaN. Other options may be >> preferable, returning NaN for instance, so I would like to open the topic >> for discussion on the list. >> > > We discussed this last month on the list and on GitHub: > https://mail.scipy.org/pipermail/numpy-discussion/2015-August/073503.html > https://github.com/numpy/numpy/issues/6265 > https://github.com/numpy/numpy/pull/6269/files > > The discussion was focused on what to do in the generic fallback case. Now > that I think about this more, I think it makes sense to explicitly check > for NaN in the unorderable case, and return NaN is the input is NaN. I > would not return NaN in general from unorderable objects, though -- in > general we should raise an error. > > It sounds like Allan has already fixed this in his PR, but it also would > not be hard to add that logic to the existing code. Is this code in the > NumPy 1.10? > No. NumPy 1.10 also has differing behavior between python 2 and python 3. The reason I raise the question now is that current master has replace use of PyObject_Compare by PyObject_RichCompare for both python 2 and 3. I would be easy to extend it. I'm less sure of Allan's work, on a quick look it seems more complicated. charris at fc [~]$ python3 Python 3.4.2 (default, Jul 9 2015, 17:24:30) [GCC 5.1.1 20150618 (Red Hat 5.1.1-4)] on linux Type "help", "copyright", "credits" or "license" for more information. >>> import numpy as np >>> np.sign(np.array([float('nan')]*3, np.object)) array([None, None, None], dtype=object) >>> charris at fc [~]$ python2 Python 2.7.10 (default, Jul 5 2015, 14:15:43) [GCC 5.1.1 20150618 (Red Hat 5.1.1-4)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import numpy as np >>> np.sign(np.array([float('nan')]*3, np.object)) array([-1, -1, -1], dtype=object) Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Tue Sep 29 14:16:37 2015 From: njs at pobox.com (Nathaniel Smith) Date: Tue, 29 Sep 2015 11:16:37 -0700 Subject: [Numpy-discussion] Sign of NaN In-Reply-To: References: Message-ID: On Sep 29, 2015 8:25 AM, "Anne Archibald" wrote: > > IEEE 754 has signum(NaN)->NaN. So does np.sign on floating-point arrays. Why should it be different for object arrays? The argument for doing it this way would be that arbitrary python objects don't have a sign, and the natural way to implement something like np.sign's semantics using only the "object" interface is if obj < 0: return -1 elif obj > 0: return 1 elif obj == 0: return 0 else: raise In general I'm not a big fan of trying to do all kinds of guessing about how to handle random objects in object arrays, the kind that ends up with a big chain of type checks and fallback behaviors. Pretty soon we find ourselves trying to extend the language with our own generic dispatch system for arbitrary python types, just for object arrays. (The current hack where for object arrays np.log will try calling obj.log() is particularly horrible. There is no rule in python that "log" is a reserved method name for "logarithm" on arbitrary objects. Ditto for the other ufuncs that implement this hack.) Plus we hope that many use cases for object arrays will soon be supplanted by better dtype support, so now may not be the best time to invest heavily in making object arrays complicated and powerful. OTOH sometimes practicality beats purity, and at least object arrays are already kinda cordoned off from the rest of the system, so I don't feel as strongly as if we were talking about core functionality. ...is there a compelling reason to even support np.sign on object arrays? This seems pretty far into the weeds, and that tends to lead to poor intuition and decision making. -n -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Tue Sep 29 14:53:08 2015 From: charlesr.harris at gmail.com (Charles R Harris) Date: Tue, 29 Sep 2015 12:53:08 -0600 Subject: [Numpy-discussion] Sign of NaN In-Reply-To: References: Message-ID: On Tue, Sep 29, 2015 at 12:16 PM, Nathaniel Smith wrote: > On Sep 29, 2015 8:25 AM, "Anne Archibald" wrote: > > > > IEEE 754 has signum(NaN)->NaN. So does np.sign on floating-point arrays. > Why should it be different for object arrays? > > The argument for doing it this way would be that arbitrary python objects > don't have a sign, and the natural way to implement something like > np.sign's semantics using only the "object" interface is > > if obj < 0: > return -1 > elif obj > 0: > return 1 > elif obj == 0: > return 0 > else: > raise > That is what current master does, using PyObject_RichCompareBool for the comparisons. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From josef.pktd at gmail.com Tue Sep 29 14:58:24 2015 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Tue, 29 Sep 2015 14:58:24 -0400 Subject: [Numpy-discussion] Sign of NaN In-Reply-To: References: Message-ID: On Tue, Sep 29, 2015 at 2:16 PM, Nathaniel Smith wrote: > On Sep 29, 2015 8:25 AM, "Anne Archibald" wrote: > > > > IEEE 754 has signum(NaN)->NaN. So does np.sign on floating-point arrays. > Why should it be different for object arrays? > > The argument for doing it this way would be that arbitrary python objects > don't have a sign, and the natural way to implement something like > np.sign's semantics using only the "object" interface is > > if obj < 0: > return -1 > elif obj > 0: > return 1 > elif obj == 0: > return 0 > else: > raise > > In general I'm not a big fan of trying to do all kinds of guessing about > how to handle random objects in object arrays, the kind that ends up with a > big chain of type checks and fallback behaviors. Pretty soon we find > ourselves trying to extend the language with our own generic dispatch > system for arbitrary python types, just for object arrays. (The current > hack where for object arrays np.log will try calling obj.log() is > particularly horrible. There is no rule in python that "log" is a reserved > method name for "logarithm" on arbitrary objects. Ditto for the other > ufuncs that implement this hack.) > > Plus we hope that many use cases for object arrays will soon be supplanted > by better dtype support, so now may not be the best time to invest heavily > in making object arrays complicated and powerful. > > OTOH sometimes practicality beats purity, and at least object arrays are > already kinda cordoned off from the rest of the system, so I don't feel as > strongly as if we were talking about core functionality. > > ...is there a compelling reason to even support np.sign on object arrays? > This seems pretty far into the weeds, and that tends to lead to poor > intuition and decision making. > One of the usecases that has sneaked in during the last few numpy versions is that object arrays contain numerical arrays where the shapes don't add up to a rectangular array. In those cases being able to apply numerical operations might be useful. But I'm +0 since I don't work with object arrays. Josef > -n > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > https://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From sebastian at sipsolutions.net Tue Sep 29 17:07:18 2015 From: sebastian at sipsolutions.net (Sebastian Berg) Date: Tue, 29 Sep 2015 23:07:18 +0200 Subject: [Numpy-discussion] Sign of NaN In-Reply-To: References: Message-ID: <1443560838.20753.13.camel@sipsolutions.net> On Di, 2015-09-29 at 11:16 -0700, Nathaniel Smith wrote: > On Sep 29, 2015 8:25 AM, "Anne Archibald" wrote: > > > > IEEE 754 has signum(NaN)->NaN. So does np.sign on floating-point > arrays. Why should it be different for object arrays? > > The argument for doing it this way would be that arbitrary python > objects don't have a sign, and the natural way to implement something > like np.sign's semantics using only the "object" interface is > > if obj < 0: > return -1 > elif obj > 0: > return 1 > elif obj == 0: > return 0 > else: > raise > > In general I'm not a big fan of trying to do all kinds of guessing > about how to handle random objects in object arrays, the kind that > ends up with a big chain of type checks and fallback behaviors. Pretty > soon we find ourselves trying to extend the language with our own > generic dispatch system for arbitrary python types, just for object > arrays. (The current hack where for object arrays np.log will try > calling obj.log() is particularly horrible. There is no rule in python > that "log" is a reserved method name for "logarithm" on arbitrary > objects. Ditto for the other ufuncs that implement this hack.) > > Plus we hope that many use cases for object arrays will soon be > supplanted by better dtype support, so now may not be the best time to > invest heavily in making object arrays complicated and powerful. > I have the little dream here that what could happen is that we create a PyFloatDtype kind of thing (it is a bit different from our float because it would always convert back to a python float and maybe raises more errors), which "registers" with the dtype system in that it says "I know how to handle python floats and store them in an array and provide ufunc implementations for it". Then, the "object" dtype ufuncs would try to call the ufunc on each element, including "conversion". They would find a "float", since it is not an array-like container, they interpret it as a PyFloatDtype scalar and call the scalars ufunc (the PyFloatDtype scalar would be a python float). Of course likely I am thinking down the wrong road, but if you want e.g. an array of Decimals, you need some way to tell that numpy as a PyDecimalDtype. Now "object" would possibly be just a fallback to mean "figure out what to use for each element". It would be a bit slower, but it would work very generally, because numpy would not impose limits as such. - Sebastian > OTOH sometimes practicality beats purity, and at least object arrays > are already kinda cordoned off from the rest of the system, so I don't > feel as strongly as if we were talking about core functionality. > > ...is there a compelling reason to even support np.sign on object > arrays? This seems pretty far into the weeds, and that tends to lead > to poor intuition and decision making. > > -n > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > https://mail.scipy.org/mailman/listinfo/numpy-discussion -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 819 bytes Desc: This is a digitally signed message part URL: From allanhaldane at gmail.com Tue Sep 29 18:28:12 2015 From: allanhaldane at gmail.com (Allan Haldane) Date: Tue, 29 Sep 2015 18:28:12 -0400 Subject: [Numpy-discussion] Sign of NaN In-Reply-To: References: Message-ID: <560B107C.2030005@gmail.com> On 09/29/2015 02:16 PM, Nathaniel Smith wrote: > On Sep 29, 2015 8:25 AM, "Anne Archibald" > wrote: >> >> IEEE 754 has signum(NaN)->NaN. So does np.sign on floating-point > arrays. Why should it be different for object arrays? > > The argument for doing it this way would be that arbitrary python > objects don't have a sign, and the natural way to implement something > like np.sign's semantics using only the "object" interface is > > if obj < 0: > return -1 > elif obj > 0: > return 1 > elif obj == 0: > return 0 > else: > raise > > In general I'm not a big fan of trying to do all kinds of guessing about > how to handle random objects in object arrays, the kind that ends up > with a big chain of type checks and fallback behaviors. Pretty soon we > find ourselves trying to extend the language with our own generic > dispatch system for arbitrary python types, just for object arrays. (The > current hack where for object arrays np.log will try calling obj.log() > is particularly horrible. There is no rule in python that "log" is a > reserved method name for "logarithm" on arbitrary objects. Ditto for the > other ufuncs that implement this hack.) > > Plus we hope that many use cases for object arrays will soon be > supplanted by better dtype support, so now may not be the best time to > invest heavily in making object arrays complicated and powerful. Even though I submitted the PR to make object arrays more powerful, this makes a lot of sense to me. Let's say we finish a new dtype system, in which (I imagine) each dtype specifies how to calculate each ufunc elementwise for that type. What are the remaining use cases for generic object arrays? The only one I see is having an array with elements of different types, which seems like a dubious idea anyway. (Nested ndarrays of varying length could be implemented as a dtype, as could the PyFloatDtype Sebastian mentioned, without need for a generic 'object' dtype which has to figure out how to call ufuncs on individual objects of different type). Allan From chris.barker at noaa.gov Tue Sep 29 20:48:58 2015 From: chris.barker at noaa.gov (Chris Barker - NOAA Federal) Date: Tue, 29 Sep 2015 17:48:58 -0700 Subject: [Numpy-discussion] Cython-based OpenMP-accelerated quartic polynomial solver In-Reply-To: <560AA195.8030001@jyu.fi> References: <560AA195.8030001@jyu.fi> Message-ID: <-4907756911279440734@unknownmsgid> This sounds pretty cool -- and I've had a use case. So it would be nice to get into Numpy. But: I doubt we'd want OpenMP dependence in Numpy itself. But maybe a pure Cython non-MP version? Are we putting Cuthon in Numpy now? I lost track. -CHB Sent from my iPhone > On Sep 29, 2015, at 7:35 AM, Juha Jeronen wrote: > > Hi all, > > I recently developed a Cython-based, OpenMP-accelerated quartic (and cubic, quadratic) polynomial solver to address a personal research need for quickly solving a very large number of independent low-degree polynomial equations for both real and complex coefficients. > > For example, on an Intel i7-4710MQ, running with 8 threads this code solves approximately 1.2e7 quartic polynomial equations per second. (With OMP_NUM_THREADS=4 the solver gives approximately 8.9e6 solutions per second, so it seems HyperThreading in this case helps about 30%.) > > The algorithms for cubics and quadratics come from Numerical Recipes (3rd ed.), and the quartic problem is internally reduced to a cubic and two quadratics, using well-known standard tricks. > > > Since to my understanding, for solving polynomial equations NumPy only provides roots(), which works one problem at a time, I'm writing to inquire whether there would be interest to include this functionality to NumPy, if I contribute the code (and clean it up for integration)? > > > I have some previous open source contribution experience. I have authored the IVTC and Phosphor deinterlacers for VLC, and a modular postprocessing filter framework for the Panda3D game engine (posted at the forums on panda3d.org, projected for version 1.10). > > Currently the polynomial solver is available in a git repository hosted by our university, the relevant files being: > > https://yousource.it.jyu.fi/jjrandom2/miniprojects/blobs/master/misc/tworods/polysolve2.pyx > https://yousource.it.jyu.fi/jjrandom2/miniprojects/blobs/master/misc/tworods/polysolve.py > https://yousource.it.jyu.fi/jjrandom2/miniprojects/blobs/master/misc/tworods/polysolve2example.py > https://yousource.it.jyu.fi/jjrandom2/miniprojects/blobs/master/misc/tworods/setup.py > https://yousource.it.jyu.fi/jjrandom2/miniprojects/blobs/master/misc/tworods/compile.sh > > I'm working on a research grant, which allows me to hold the copyright; license is BSD. > > Polysolve2 is the fast Cython+OpenMP version, while polysolve is an earlier (slower) experimental version using only NumPy array operations. The example module contains a usage example, and setup (in the standard manner) instructs distutils and Cython as to how to build polysolve2 (including setting the relevant math flags and enabling OpenMP). > > (The setup script includes also some stuff specific to my solver for the Ziegler problem; basically, the "tworods" module can be ignored. I apologize for the inconvenience.) > > > Thoughts? > > > Best regards, > > -J > > ------------------------------------------------- > Juha Jeronen, Ph.D. > juha.jeronen at jyu.fi > University of Jyv?skyl? > Department of Mathematical Information Technology > ------------------------------------------------- > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > https://mail.scipy.org/mailman/listinfo/numpy-discussion From chris.barker at noaa.gov Tue Sep 29 20:58:26 2015 From: chris.barker at noaa.gov (Chris Barker - NOAA Federal) Date: Tue, 29 Sep 2015 17:58:26 -0700 Subject: [Numpy-discussion] Sign of NaN In-Reply-To: References: Message-ID: <7501048721861220575@unknownmsgid> One of the usecases that has sneaked in during the last few numpy versions is that object arrays contain numerical arrays where the shapes don't add up to a rectangular array. I think that's the wrong way to dve that problem -- we really should have a "proper" ragged array implementation. But is is the easiest way at this point. For this, and other use-cases, special casing Numpy arrays stored in object arrays does make sense: "If this is s a Numpy array, pass the operation through." -CHB In those cases being able to apply numerical operations might be useful. But I'm +0 since I don't work with object arrays. Josef > -n > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > https://mail.scipy.org/mailman/listinfo/numpy-discussion > > _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion at scipy.org https://mail.scipy.org/mailman/listinfo/numpy-discussion -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Tue Sep 29 21:31:56 2015 From: charlesr.harris at gmail.com (Charles R Harris) Date: Tue, 29 Sep 2015 19:31:56 -0600 Subject: [Numpy-discussion] Sign of NaN In-Reply-To: <7501048721861220575@unknownmsgid> References: <7501048721861220575@unknownmsgid> Message-ID: On Tue, Sep 29, 2015 at 6:58 PM, Chris Barker - NOAA Federal < chris.barker at noaa.gov> wrote: > > One of the usecases that has sneaked in during the last few numpy versions > is that object arrays contain numerical arrays where the shapes don't add > up to a rectangular array. > > > I think that's the wrong way to dve that problem -- we really should have > a "proper" ragged array implementation. But is is the easiest way at this > point. > > For this, and other use-cases, special casing Numpy arrays stored in > object arrays does make sense: > > "If this is s a Numpy array, pass the operation through." > Because we now (development) use rich compare, the result looks like In [1]: a = ones(3) In [2]: b = array([a, -a], object) In [3]: b Out[3]: array([[1.0, 1.0, 1.0], [-1.0, -1.0, -1.0]], dtype=object) In [4]: sign(b) Out[4]: array([[1L, 1L, 1L], [-1L, -1L, -1L]], dtype=object) The function returns long integers in order to not special case Python 3. Hmm, wonder if we might want to change that. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Tue Sep 29 21:35:12 2015 From: charlesr.harris at gmail.com (Charles R Harris) Date: Tue, 29 Sep 2015 19:35:12 -0600 Subject: [Numpy-discussion] Sign of NaN In-Reply-To: References: <7501048721861220575@unknownmsgid> Message-ID: On Tue, Sep 29, 2015 at 7:31 PM, Charles R Harris wrote: > > > On Tue, Sep 29, 2015 at 6:58 PM, Chris Barker - NOAA Federal < > chris.barker at noaa.gov> wrote: > >> >> One of the usecases that has sneaked in during the last few numpy >> versions is that object arrays contain numerical arrays where the shapes >> don't add up to a rectangular array. >> >> >> I think that's the wrong way to dve that problem -- we really should have >> a "proper" ragged array implementation. But is is the easiest way at this >> point. >> >> For this, and other use-cases, special casing Numpy arrays stored in >> object arrays does make sense: >> >> "If this is s a Numpy array, pass the operation through." >> > > Because we now (development) use rich compare, the result looks like > > In [1]: a = ones(3) > > In [2]: b = array([a, -a], object) > > In [3]: b > Out[3]: > array([[1.0, 1.0, 1.0], > [-1.0, -1.0, -1.0]], dtype=object) > > In [4]: sign(b) > Out[4]: > array([[1L, 1L, 1L], > [-1L, -1L, -1L]], dtype=object) > > The function returns long integers in order to not special case Python 3. > Hmm, wonder if we might want to change that. > Oops, not what was intended. In fact it raises an error In [7]: b Out[7]: array([array([ 1., 1., 1.]), array([-1., -1., -1.])], dtype=object) In [8]: sign(b) --------------------------------------------------------------------------- ValueError Traceback (most recent call last) in () ----> 1 sign(b) ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all() Chuck > > Chuck > -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Tue Sep 29 21:37:37 2015 From: charlesr.harris at gmail.com (Charles R Harris) Date: Tue, 29 Sep 2015 19:37:37 -0600 Subject: [Numpy-discussion] Cython-based OpenMP-accelerated quartic polynomial solver In-Reply-To: <-4907756911279440734@unknownmsgid> References: <560AA195.8030001@jyu.fi> <-4907756911279440734@unknownmsgid> Message-ID: On Tue, Sep 29, 2015 at 6:48 PM, Chris Barker - NOAA Federal < chris.barker at noaa.gov> wrote: > This sounds pretty cool -- and I've had a use case. So it would be > nice to get into Numpy. > > But: I doubt we'd want OpenMP dependence in Numpy itself. > > But maybe a pure Cython non-MP version? > > Are we putting Cuthon in Numpy now? I lost track. > Yes, but we need to be careful of Numeric Recipes. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Wed Sep 30 03:01:17 2015 From: njs at pobox.com (Nathaniel Smith) Date: Wed, 30 Sep 2015 00:01:17 -0700 Subject: [Numpy-discussion] Sign of NaN In-Reply-To: <1443560838.20753.13.camel@sipsolutions.net> References: <1443560838.20753.13.camel@sipsolutions.net> Message-ID: On Tue, Sep 29, 2015 at 2:07 PM, Sebastian Berg wrote: > On Di, 2015-09-29 at 11:16 -0700, Nathaniel Smith wrote: [...] >> In general I'm not a big fan of trying to do all kinds of guessing >> about how to handle random objects in object arrays, the kind that >> ends up with a big chain of type checks and fallback behaviors. Pretty >> soon we find ourselves trying to extend the language with our own >> generic dispatch system for arbitrary python types, just for object >> arrays. (The current hack where for object arrays np.log will try >> calling obj.log() is particularly horrible. There is no rule in python >> that "log" is a reserved method name for "logarithm" on arbitrary >> objects. Ditto for the other ufuncs that implement this hack.) >> >> Plus we hope that many use cases for object arrays will soon be >> supplanted by better dtype support, so now may not be the best time to >> invest heavily in making object arrays complicated and powerful. >> > > I have the little dream here that what could happen is that we create a > PyFloatDtype kind of thing (it is a bit different from our float because > it would always convert back to a python float and maybe raises more > errors), which "registers" with the dtype system in that it says "I know > how to handle python floats and store them in an array and provide ufunc > implementations for it". > > Then, the "object" dtype ufuncs would try to call the ufunc on each > element, including "conversion". They would find a "float", since it is > not an array-like container, they interpret it as a PyFloatDtype scalar > and call the scalars ufunc (the PyFloatDtype scalar would be a python > float). I'm not sure I understand this, but it did make me think of one possible approach -- in my notebook sketches for what the New and Improved ufunc API might look like, I was already pondering whether the inner loop should receive a pointer to the ufunc object itself. Not for any reason in particular, but just because hey they're sorta vaguely like methods and methods get pointers to the object. But now I know what this is useful for :-). If ufunc loops get a pointer to the ufunc object itself, then we can define a single inner loop function that looks like (sorta-Cython code): cdef generic_object_inner_loop(ufunc, args, strides, n, ...): for i in range(n): arg_objs = [] for i in range(ufunc.narg): args_objs.append( (args[j] + strides[j] * i)) ufunc(*arg_objs[:ufunc.nin], out=arg_objs[ufunc.nin:]) and register it by default in every ufunc with signature "{}->{}".format("O" * ufunc.nin, "O" * ufunc.nout). And this would in just a few lines of code provide a pretty sensible generic behavior for *all* object array ufuncs -- they recursively call the ufunc on their contents. As a prerequisite of course we would need to remove the auto-coercion of unknown objects to object arrays, otherwise this becomes an infinite recursion. But we already decided to do that. And for this to be really useful for arbitrary objects, not just the ones that asarray recognizes, then we need __numpy_ufunc__. But again, we already decided to do that :-). -n -- Nathaniel J. Smith -- http://vorpus.org From sebastian at sipsolutions.net Wed Sep 30 03:32:02 2015 From: sebastian at sipsolutions.net (Sebastian Berg) Date: Wed, 30 Sep 2015 09:32:02 +0200 Subject: [Numpy-discussion] Sign of NaN In-Reply-To: References: <1443560838.20753.13.camel@sipsolutions.net> Message-ID: <1443598322.20753.27.camel@sipsolutions.net> On Mi, 2015-09-30 at 00:01 -0700, Nathaniel Smith wrote: > On Tue, Sep 29, 2015 at 2:07 PM, Sebastian Berg > wrote: > > On Di, 2015-09-29 at 11:16 -0700, Nathaniel Smith wrote: > [...] > >> In general I'm not a big fan of trying to do all kinds of guessing > >> about how to handle random objects in object arrays, the kind that > >> ends up with a big chain of type checks and fallback behaviors. Pretty > >> soon we find ourselves trying to extend the language with our own > >> generic dispatch system for arbitrary python types, just for object > >> arrays. (The current hack where for object arrays np.log will try > >> calling obj.log() is particularly horrible. There is no rule in python > >> that "log" is a reserved method name for "logarithm" on arbitrary > >> objects. Ditto for the other ufuncs that implement this hack.) > >> > >> Plus we hope that many use cases for object arrays will soon be > >> supplanted by better dtype support, so now may not be the best time to > >> invest heavily in making object arrays complicated and powerful. > >> > > > > I have the little dream here that what could happen is that we create a > > PyFloatDtype kind of thing (it is a bit different from our float because > > it would always convert back to a python float and maybe raises more > > errors), which "registers" with the dtype system in that it says "I know > > how to handle python floats and store them in an array and provide ufunc > > implementations for it". > > > > Then, the "object" dtype ufuncs would try to call the ufunc on each > > element, including "conversion". They would find a "float", since it is > > not an array-like container, they interpret it as a PyFloatDtype scalar > > and call the scalars ufunc (the PyFloatDtype scalar would be a python > > float). > > I'm not sure I understand this, but it did make me think of one > possible approach -- > > in my notebook sketches for what the New and Improved ufunc API might > look like, I was already pondering whether the inner loop should > receive a pointer to the ufunc object itself. Not for any reason in > particular, but just because hey they're sorta vaguely like methods > and methods get pointers to the object. But now I know what this is > useful for :-). > > If ufunc loops get a pointer to the ufunc object itself, then we can > define a single inner loop function that looks like (sorta-Cython > code): > > cdef generic_object_inner_loop(ufunc, args, strides, n, ...): > for i in range(n): > arg_objs = [] > for i in range(ufunc.narg): > args_objs.append( (args[j] + strides[j] * i)) > ufunc(*arg_objs[:ufunc.nin], out=arg_objs[ufunc.nin:]) > > and register it by default in every ufunc with signature > "{}->{}".format("O" * ufunc.nin, "O" * ufunc.nout). And this would in > just a few lines of code provide a pretty sensible generic behavior > for *all* object array ufuncs -- they recursively call the ufunc on > their contents. > > As a prerequisite of course we would need to remove the auto-coercion > of unknown objects to object arrays, otherwise this becomes an > infinite recursion. But we already decided to do that. > > And for this to be really useful for arbitrary objects, not just the > ones that asarray recognizes, then we need __numpy_ufunc__. But again, > we already decided to do that :-). > Well, what I mean is. A `Decimal` will probably never know about numpy itself. So I was wondering if you should teach numpy the other way around about it. I.e. you would create an object which has all the information about ufuncs and casting for Decimal and register it with numpy. Then when numpy sees a Decimal (also in `asarray` it would know what to do with them, how to store them in an array, etc. The `Decimal` object would be the scalar version of an array of Decimals. By the way, in some way an array is a "Scalar" as well, it can be put into another array and if you apply the ufunc to it, it applies the ufunc to all its elements. This all is likely too complicated though, maybe it is better to just force the user to subclass the Decimal to achieve this. I am sure there are quite a few roads we could go and we just need to think about it some more about what we want and what we can do. :) - Sebastian > -n > -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 819 bytes Desc: This is a digitally signed message part URL: From juha.jeronen at jyu.fi Wed Sep 30 04:20:11 2015 From: juha.jeronen at jyu.fi (Juha Jeronen) Date: Wed, 30 Sep 2015 11:20:11 +0300 Subject: [Numpy-discussion] Cython-based OpenMP-accelerated quartic polynomial solver In-Reply-To: <-4907756911279440734@unknownmsgid> References: <560AA195.8030001@jyu.fi> <-4907756911279440734@unknownmsgid> Message-ID: <560B9B3B.7000106@jyu.fi> Hi, On 30.09.2015 03:48, Chris Barker - NOAA Federal wrote: > This sounds pretty cool -- and I've had a use case. So it would be > nice to get into Numpy. > > But: I doubt we'd want OpenMP dependence in Numpy itself. It is indeed a good point not to add new dependencies for a small feature such as this. From a software engineering point of view, I fully agree. Note however that with the current version of the code, not having OpenMP will severely limit the performance, especially on quad-core machines, since an important factor in the speed comes from the parallel processing of the independent problem instances. But I suppose there may be another technical option to support multiple cores (doesn't NumPy already do this in some of its routines?). OpenMP was just the most convenient solution from the implementation viewpoint. > But maybe a pure Cython non-MP version? That is of course possible. IIRC, changing the processing to occur on a single core should only require replacing Cython.parallel.prange() with range() in the array processing loops. (Note range(), not xrange(), even for Python 2.x - Cython compiles a loop using range() into an efficient C loop if some simple compile-time conditions are fulfilled.) > Are we putting Cuthon in Numpy now? I lost track. I thought so? But then again, I'm just a user :) -J ------------------------------------------------- Juha Jeronen, Ph.D. juha.jeronen at jyu.fi University of Jyv?skyl? Department of Mathematical Information Technology ------------------------------------------------- From juha.jeronen at jyu.fi Wed Sep 30 05:08:31 2015 From: juha.jeronen at jyu.fi (Juha Jeronen) Date: Wed, 30 Sep 2015 12:08:31 +0300 Subject: [Numpy-discussion] Cython-based OpenMP-accelerated quartic polynomial solver In-Reply-To: References: <560AA195.8030001@jyu.fi> <-4907756911279440734@unknownmsgid> Message-ID: <560BA68F.100@jyu.fi> Hi, On 30.09.2015 04:37, Charles R Harris wrote: > > > On Tue, Sep 29, 2015 at 6:48 PM, Chris Barker - NOAA Federal > > wrote: > > This sounds pretty cool -- and I've had a use case. So it would be > nice to get into Numpy. > > But: I doubt we'd want OpenMP dependence in Numpy itself. > > But maybe a pure Cython non-MP version? > > Are we putting Cuthon in Numpy now? I lost track. > > > Yes, but we need to be careful of Numeric Recipes. True. As I said, only the algorithms come from NR, and the code was written from scratch. I haven't even looked at any of their code, precisely due to the restrictive license. In this particular case, the referred pages (pp. 227-229, 3rd ed.) contain only a description of the solution process in English, with equations - there is no example code printed into the book in section 5.6, Quadratic and Cubic Equations, which in its entirety is contained on pp. 227-229. Furthermore, on p. 228, equation (5.6.12), which contains the solutions of the cubic equation, is attributed to Francois Viete. The reference given is the treatise "De emendatione", published in 1615. The same solution, also with attribution to Francois Viete, is given in Wikipedia, but without especially defining temporary variables to eliminate some common subexpressions: https://en.wikipedia.org/wiki/Cubic_function#Trigonometric_.28and_hyperbolic.29_method The solution to the quadratic equation is the standard one, but with special care taken to avoid catastrophic cancellation in computing the other root. Wikipedia mentions that in the standard formula, this issue exists: https://en.wikipedia.org/wiki/Quadratic_equation#Avoiding_loss_of_significance but does not give an explicit remedy. References given are Kahan, Willian (November 20, 2004), On the Cost of Floating-Point Computation Without Extra-Precise Arithmetic. ( http://www.cs.berkeley.edu/~wkahan/Qdrtcs.pdf , see esp. p. 8, which provides the same remedy as NR ) Higham, Nicholas (2002), Accuracy and Stability of Numerical Algorithms (2nd ed.), SIAM, p. 10, ISBN 978-0-89871-521-7. (Browsing through Kahan's text, I see now that I could improve the discriminant computation to handle marginal cases better. Might do this in a future version.) For both of these cases, I've used NR as a reference, because the authors have taken special care to choose only numerically stable approaches - which is absolutely critical, yet something that sadly not all mathematics texts care about. The quartic equation is not treated in NR. The reduction trick is reported in Wikipedia: https://en.wikipedia.org/wiki/Quartic_function#Solving_by_factoring_into_quadratics where the original reference given is Brookfield, G. (2007). "Factoring quartic polynomials: A lost art". Mathematics Magazine 80 (1): 67?70. (This seemed an elegant approach, given stable solvers for cubics and quadratics.) -J ------------------------------------------------- Juha Jeronen, Ph.D. juha.jeronen at jyu.fi University of Jyv?skyl? Department of Mathematical Information Technology ------------------------------------------------- -------------- next part -------------- An HTML attachment was scrubbed... URL: From davidmenhur at gmail.com Wed Sep 30 05:27:58 2015 From: davidmenhur at gmail.com (=?UTF-8?B?RGHPgGlk?=) Date: Wed, 30 Sep 2015 11:27:58 +0200 Subject: [Numpy-discussion] Cython-based OpenMP-accelerated quartic polynomial solver In-Reply-To: <560B9B3B.7000106@jyu.fi> References: <560AA195.8030001@jyu.fi> <-4907756911279440734@unknownmsgid> <560B9B3B.7000106@jyu.fi> Message-ID: On 30 September 2015 at 10:20, Juha Jeronen wrote: > > Are we putting Cuthon in Numpy now? I lost track. >> > > I thought so? But then again, I'm just a user :) As of this moment, there are three Cython modules in Numpy, so yes. Releases ship the C generated modules, so it is not a dependency. ./numpy/distutils/tests/pyrex_ext/primes.pyx ./numpy/random/mtrand/mtrand.pyx ./tools/allocation_tracking/alloc_hook.pyx In any case, we should always include a single threaded version because sometimes the computations are already parallelised at a higher level. Is there a nice way to ship both versions? After all, most implementations of BLAS and friends do spawn OpenMP threads, so I don't think it would be outrageous to take advantage of it in more places; provided there is a nice way to fallback to a serial version when it is not available. /David. -------------- next part -------------- An HTML attachment was scrubbed... URL: From matthieu.brucher at gmail.com Wed Sep 30 05:35:33 2015 From: matthieu.brucher at gmail.com (Matthieu Brucher) Date: Wed, 30 Sep 2015 10:35:33 +0100 Subject: [Numpy-discussion] Cython-based OpenMP-accelerated quartic polynomial solver In-Reply-To: References: <560AA195.8030001@jyu.fi> <-4907756911279440734@unknownmsgid> Message-ID: Yes, obviously, the code has NR parts, so it can't be licensed as BSD as it is... Matthieu 2015-09-30 2:37 GMT+01:00 Charles R Harris : > > > On Tue, Sep 29, 2015 at 6:48 PM, Chris Barker - NOAA Federal > wrote: >> >> This sounds pretty cool -- and I've had a use case. So it would be >> nice to get into Numpy. >> >> But: I doubt we'd want OpenMP dependence in Numpy itself. >> >> But maybe a pure Cython non-MP version? >> >> Are we putting Cuthon in Numpy now? I lost track. > > > Yes, but we need to be careful of Numeric Recipes. > > > > Chuck > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > https://mail.scipy.org/mailman/listinfo/numpy-discussion > -- Information System Engineer, Ph.D. Blog: http://matt.eifelle.com LinkedIn: http://www.linkedin.com/in/matthieubrucher From juha.jeronen at jyu.fi Wed Sep 30 05:49:56 2015 From: juha.jeronen at jyu.fi (Juha Jeronen) Date: Wed, 30 Sep 2015 12:49:56 +0300 Subject: [Numpy-discussion] Cython-based OpenMP-accelerated quartic polynomial solver In-Reply-To: References: <560AA195.8030001@jyu.fi> <-4907756911279440734@unknownmsgid> Message-ID: <560BB044.9060609@jyu.fi> Hi, What qualifies as an NR part? (See my previous message concerning the references; NR is not the only source which has these algorithms. Again, the code itself is original to this solver, I haven't even looked at the code provided with NR.) -J ------------------------------------------------- Juha Jeronen, Ph.D. juha.jeronen at jyu.fi University of Jyv?skyl? Department of Mathematical Information Technology ------------------------------------------------- On 30.09.2015 12:35, Matthieu Brucher wrote: > Yes, obviously, the code has NR parts, so it can't be licensed as BSD > as it is... > > Matthieu > > 2015-09-30 2:37 GMT+01:00 Charles R Harris : >> >> On Tue, Sep 29, 2015 at 6:48 PM, Chris Barker - NOAA Federal >> wrote: >>> This sounds pretty cool -- and I've had a use case. So it would be >>> nice to get into Numpy. >>> >>> But: I doubt we'd want OpenMP dependence in Numpy itself. >>> >>> But maybe a pure Cython non-MP version? >>> >>> Are we putting Cuthon in Numpy now? I lost track. >> >> Yes, but we need to be careful of Numeric Recipes. >> >> >> >> Chuck >> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> https://mail.scipy.org/mailman/listinfo/numpy-discussion >> > > From pav at iki.fi Wed Sep 30 06:21:03 2015 From: pav at iki.fi (Pauli Virtanen) Date: Wed, 30 Sep 2015 10:21:03 +0000 (UTC) Subject: [Numpy-discussion] =?utf-8?q?Cython-based_OpenMP-accelerated_quar?= =?utf-8?q?tic=09polynomial_solver?= References: <560AA195.8030001@jyu.fi> Message-ID: Juha Jeronen jyu.fi> writes: > I recently developed a Cython-based, OpenMP-accelerated quartic (and > cubic, quadratic) polynomial solver to address a personal research need > for quickly solving a very large number of independent low-degree > polynomial equations for both real and complex coefficients. My 2c in this context would be to also think how this best fits in how collections of polynomials are represented in Numpy. AFAICS, Numpy's polynomial module supports evaluation of polynomial collections (cf. numpy.polynomial.polynomial.polyval), but the corresponding root finding routine (numpy.polynomial.polynomial.polyroots) only deals with one polynomial at a time. The present code in principle could be used to speed up the latter routine after it is generalized to multiple polynomials. The general case is probably doable just by coding up the companion matrix approach using low-level routines (or maybe with the new vectorized np.linalg routines). Some relevant code elsewhere for similar purposes can be found in scipy.inteprolate.PPoly/BPoly (and in the future BSpline). https://docs.scipy.org/doc/scipy/reference/generated/scipy.interpolate.PPoly.html However, since it's piecewise, there's purposefully support only for real-valued roots. -- Pauli Virtanen From nathan12343 at gmail.com Wed Sep 30 09:57:37 2015 From: nathan12343 at gmail.com (Nathan Goldbaum) Date: Wed, 30 Sep 2015 08:57:37 -0500 Subject: [Numpy-discussion] Cython-based OpenMP-accelerated quartic polynomial solver In-Reply-To: <560B9B3B.7000106@jyu.fi> References: <560AA195.8030001@jyu.fi> <-4907756911279440734@unknownmsgid> <560B9B3B.7000106@jyu.fi> Message-ID: On Wed, Sep 30, 2015 at 3:20 AM, Juha Jeronen wrote: > Hi, > > On 30.09.2015 03:48, Chris Barker - NOAA Federal wrote: > >> This sounds pretty cool -- and I've had a use case. So it would be >> nice to get into Numpy. >> >> But: I doubt we'd want OpenMP dependence in Numpy itself. >> > > It is indeed a good point not to add new dependencies for a small feature > such as this. From a software engineering point of view, I fully agree. > > Note however that with the current version of the code, not having OpenMP > will severely limit the performance, especially on quad-core machines, > since an important factor in the speed comes from the parallel processing > of the independent problem instances. > > But I suppose there may be another technical option to support multiple > cores (doesn't NumPy already do this in some of its routines?). OpenMP was > just the most convenient solution from the implementation viewpoint. > > > But maybe a pure Cython non-MP version? >> > > That is of course possible. IIRC, changing the processing to occur on a > single core should only require replacing Cython.parallel.prange() with > range() in the array processing loops. > > (Note range(), not xrange(), even for Python 2.x - Cython compiles a loop > using range() into an efficient C loop if some simple compile-time > conditions are fulfilled.) > > > Are we putting Cuthon in Numpy now? I lost track. >> > > I thought so? But then again, I'm just a user :) For what it's worth (no idea what the order of magnitude of the technical risk is for something like this in a library like numpy), it's actually quite simple to dynamically test for OpenMP support at install time. Here's what we do in yt: https://bitbucket.org/yt_analysis/yt/src/f8934e13abaf349f58c52c076320d44ee4f61a7f/yt/utilities/lib/setup.py?at=yt&fileviewer=file-view-default#setup.py-8 Basically, just try to compile a simple OpenMP test program in a subprocess. If that succeeds, then great, we can add -fopenmp as a compilation flag. If not, don't do that. This was implemented when Apple switched from GCC 4.2 to LLVM/Clang in OS X 10.7. We've had exactly one bug report about this, and it wasn't even related to OpenMP, instead it was an issue with the way we were using subprocess which broke when someone had set "CC = ccache gcc". Hope that's helpful, Nathan Goldbaum > > > > -J > > ------------------------------------------------- > Juha Jeronen, Ph.D. > juha.jeronen at jyu.fi > University of Jyv?skyl? > Department of Mathematical Information Technology > ------------------------------------------------- > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > https://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.barker at noaa.gov Wed Sep 30 12:04:22 2015 From: chris.barker at noaa.gov (Chris Barker) Date: Wed, 30 Sep 2015 09:04:22 -0700 Subject: [Numpy-discussion] composition of the steering council (was Re: Governance model request) In-Reply-To: References: Message-ID: On Tue, Sep 29, 2015 at 7:32 AM, Travis Oliphant wrote: > I'm in a situation now where at least for 6 months or so I can help with > NumPy more than I have been able to for 7 years. > great news! > 1) 1 year of inactivity to be removed from the council is too little for a > long-running project like NumPy > I always read this as "activity in council matters", not "activity in commits to the code". If that's not what is meant, then we should re-write that. So no, Travis is not ineligible for the council, as there has been no council to be active on. And in that case, my vague memory is that Travis has popped his head into architecture discussions on this list at least once a year essentially forever, so there is every indication that he has been and will be active in that sense. --- somewhere between 2 and 4 years would be more appropriate. I suppose > 1 year of inactivity is fine if that is defined only as "failure to vote on > matters before the council" > exactly. -- I wld expand "activity" to mean more than voting, but that's the idea. More like active participation in discussions / issues, and/or attending meetings, if there are any. > 2) The seed council should not just be recent contributors but should > include as many people as are willing to help who have a long history with > the project. > again, I don't think we need so much emphasis in the "seed council" that is only needed to have SOME way to start. Though I guess as long as we're having the discussion, we should have an outline of the expectations for the council in general, which should indeed include a couple members with long history. > 3) I think people who contribute significantly generally should be able to > re-join the steering council more easily than "going through the 1-year > vetting process" again --- they would have to be approved by the current > steering council but it should not be automatically disallowed (thus > requiring the equivalent of an amendment to change it). > Sure -- give the Council Flexibility to keep the council current. However, the idea here is that we want some continuity -- not have people popping on and off the council on short time spans as their schedules allow. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.barker at noaa.gov Wed Sep 30 12:11:08 2015 From: chris.barker at noaa.gov (Chris Barker) Date: Wed, 30 Sep 2015 09:11:08 -0700 Subject: [Numpy-discussion] Sign of NaN In-Reply-To: References: <7501048721861220575@unknownmsgid> Message-ID: On Tue, Sep 29, 2015 at 6:35 PM, Charles R Harris wrote: > For this, and other use-cases, special casing Numpy arrays stored in >>> object arrays does make sense: >>> >>> "If this is s a Numpy array, pass the operation through." >>> >> >> Because we now (development) use rich compare, the result looks like >> >> Oops, not what was intended. In fact it raises an error > > In [7]: b > Out[7]: array([array([ 1., 1., 1.]), array([-1., -1., -1.])], > dtype=object) > > In [8]: sign(b) > --------------------------------------------------------------------------- > ValueError Traceback (most recent call last) > in () > ----> 1 sign(b) > > ValueError: The truth value of an array with more than one element is > ambiguous. Use a.any() or a.all() > exactly -- it seems to me that a special case for numpy arrays as objects in object arrays makes sense, so you'd get: In [6]: oa Out[6]: array([[1.0, 1.0, 1.0], [-1.0, -1.0, -1.0]], dtype=object) In [7]: np.sign(oa) Out[7]: array([[1, 1, 1], [-1, -1, -1]], dtype=object) (which you do now in the version I'm running). Though rather than the special case, maybe we really need dtype=ndarray arrays? oa = np.array([a1, a2], dtype=np.ndarray) Then we could count on everything in the array being an array..... -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.barker at noaa.gov Wed Sep 30 12:13:20 2015 From: chris.barker at noaa.gov (Chris Barker) Date: Wed, 30 Sep 2015 09:13:20 -0700 Subject: [Numpy-discussion] Sign of NaN In-Reply-To: <1443598322.20753.27.camel@sipsolutions.net> References: <1443560838.20753.13.camel@sipsolutions.net> <1443598322.20753.27.camel@sipsolutions.net> Message-ID: On Wed, Sep 30, 2015 at 12:32 AM, Sebastian Berg wrote: > > >> Plus we hope that many use cases for object arrays will soon be > > >> supplanted by better dtype support, so now may not be the best time to > > >> invest heavily in making object arrays complicated and powerful.Well, > what I mean is. A > > `Decimal` will probably never know about numpy > itself. So I was wondering if you should teach numpy the other way > around about it. > indeed -- but the way to do that is to create a Decimal dtype -- if we have the "better dtype support", then that shouldn't be hard to do. -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Wed Sep 30 12:20:43 2015 From: njs at pobox.com (Nathaniel Smith) Date: Wed, 30 Sep 2015 09:20:43 -0700 Subject: [Numpy-discussion] Cython-based OpenMP-accelerated quartic polynomial solver In-Reply-To: References: <560AA195.8030001@jyu.fi> <-4907756911279440734@unknownmsgid> <560B9B3B.7000106@jyu.fi> Message-ID: On Sep 30, 2015 2:28 AM, "Da?id" wrote: [...] > Is there a nice way to ship both versions? After all, most implementations of BLAS and friends do spawn OpenMP threads, so I don't think it would be outrageous to take advantage of it in more places; provided there is a nice way to fallback to a serial version when it is not available. This is incorrect -- the only common implementation of BLAS that uses *OpenMP* threads is OpenBLAS, and even then it's not the default -- it only happens if you run it in a special non-default configuration. The challenges to providing transparent multithreading in numpy generally are: - gcc + OpenMP on linux still breaks multiprocessing. There's a patch to fix this but they still haven't applied it; alternatively there's a workaround you can use in multiprocessing (not using fork mode), but this requires every user update their code and the workaround has other limitations. We're unlikely to use OpenMP while this is the case. - parallel code in general is not very composable. If someone is calling a numpy operation from one thread, great, transparently using multiple threads internally is a win. If they're exploiting some higher-level structure in their problem to break it into pieces and process each in parallel, and then using numpy on each piece, then numpy spawning threads internally will probably destroy performance. And numpy is too low-level to know which case it's in. This problem exists to some extent already with multi-threaded BLAS, so people use various BLAS-specific knobs to manage it in ad hoc ways, but this doesn't scale. (Ironically OpenMP is more composable then most approaches to threading, but only if everyone is using it and, as per above, not everyone is and we currently can't.) -n -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.barker at noaa.gov Wed Sep 30 12:20:34 2015 From: chris.barker at noaa.gov (Chris Barker) Date: Wed, 30 Sep 2015 09:20:34 -0700 Subject: [Numpy-discussion] Cython-based OpenMP-accelerated quartic polynomial solver In-Reply-To: References: <560AA195.8030001@jyu.fi> <-4907756911279440734@unknownmsgid> <560B9B3B.7000106@jyu.fi> Message-ID: On Wed, Sep 30, 2015 at 6:57 AM, Nathan Goldbaum wrote: > Note however that with the current version of the code, not having OpenMP >> will severely limit the performance, especially on quad-core machines, >> since an important factor in the speed comes from the parallel processing >> of the independent problem instances. >> > It seems you got much more than 4 times performance improvement -- which is the most you'll get from four cores, presumably :-). not that another 4 times or so isn't a good thing. But I suppose there may be another technical option to support multiple >> cores > > python threads with nogil? > For what it's worth (no idea what the order of magnitude of the technical > risk is for something like this in a library like numpy), it's actually > quite simple to dynamically test for OpenMP support at install time. > > > Basically, just try to compile a simple OpenMP test program in a > subprocess. If that succeeds, then great, we can add -fopenmp as a > compilation flag. If not, don't do that. > this sounds like an compilation-tiem tiem test, not isntall time. And install time isn't really helpful either, we really want plan old wheels, etc to work. We'd need a run-time check. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From sebastian at sipsolutions.net Wed Sep 30 12:25:15 2015 From: sebastian at sipsolutions.net (Sebastian Berg) Date: Wed, 30 Sep 2015 18:25:15 +0200 Subject: [Numpy-discussion] Sign of NaN In-Reply-To: References: <7501048721861220575@unknownmsgid> Message-ID: <1443630315.1524.5.camel@sipsolutions.net> On Mi, 2015-09-30 at 09:11 -0700, Chris Barker wrote: > On Tue, Sep 29, 2015 at 6:35 PM, Charles R Harris > wrote: > For this, and other use-cases, special casing > Numpy arrays stored in object arrays does make > sense: > > > "If this is s a Numpy array, pass the > operation through." > > > Because we now (development) use rich compare, the > result looks like > > > Oops, not what was intended. In fact it raises an error > > In [7]: b > Out[7]: array([array([ 1., 1., 1.]), array([-1., -1., > -1.])], dtype=object) > > In [8]: sign(b) > --------------------------------------------------------------------------- > ValueError Traceback (most > recent call last) > in () > ----> 1 sign(b) > > ValueError: The truth value of an array with more than one > element is ambiguous. Use a.any() or a.all() > > > exactly -- it seems to me that a special case for numpy arrays as > objects in object arrays makes sense, so you'd get: > > > In [6]: oa > Out[6]: > array([[1.0, 1.0, 1.0], > [-1.0, -1.0, -1.0]], dtype=object) > > > In [7]: np.sign(oa) > Out[7]: > array([[1, 1, 1], > [-1, -1, -1]], dtype=object) > > > (which you do now in the version I'm running). > > > Though rather than the special case, maybe we really need > dtype=ndarray arrays? > I think this (as a dtype) is an obvious solution. The other solution, I am not sure about in general to be honest. We may have to be more careful about creating a monster with new dtypes, rather than being careful to implement all possible features ;). It is not that I think we would not have consistent rules, etc. it is just that we *want* to force code to be obvious. If someone has arrays inside arrays, maybe he should be expected to specify that. It actually breaks some logic (or cannot be implemented for everything), because we have signatures such as `O->?`, which does not support array output. - Sebastian > > oa = np.array([a1, a2], dtype=np.ndarray) > > > > Then we could count on everything in the array being an array..... > > > -CHB > > > -- > > Christopher Barker, Ph.D. > Oceanographer > > Emergency Response Division > NOAA/NOS/OR&R (206) 526-6959 voice > 7600 Sand Point Way NE (206) 526-6329 fax > Seattle, WA 98115 (206) 526-6317 main reception > > Chris.Barker at noaa.gov > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > https://mail.scipy.org/mailman/listinfo/numpy-discussion -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 819 bytes Desc: This is a digitally signed message part URL: From robert.kern at gmail.com Wed Sep 30 13:14:00 2015 From: robert.kern at gmail.com (Robert Kern) Date: Wed, 30 Sep 2015 18:14:00 +0100 Subject: [Numpy-discussion] Cython-based OpenMP-accelerated quartic polynomial solver In-Reply-To: References: <560AA195.8030001@jyu.fi> <-4907756911279440734@unknownmsgid> Message-ID: On Wed, Sep 30, 2015 at 10:35 AM, Matthieu Brucher < matthieu.brucher at gmail.com> wrote: > > Yes, obviously, the code has NR parts, so it can't be licensed as BSD > as it is... It's not obvious to me, especially after Juha's further clarifications. -- Robert Kern -------------- next part -------------- An HTML attachment was scrubbed... URL: From matthieu.brucher at gmail.com Wed Sep 30 13:16:19 2015 From: matthieu.brucher at gmail.com (Matthieu Brucher) Date: Wed, 30 Sep 2015 18:16:19 +0100 Subject: [Numpy-discussion] Cython-based OpenMP-accelerated quartic polynomial solver In-Reply-To: References: <560AA195.8030001@jyu.fi> <-4907756911279440734@unknownmsgid> Message-ID: After, I agree with you. 2015-09-30 18:14 GMT+01:00 Robert Kern : > On Wed, Sep 30, 2015 at 10:35 AM, Matthieu Brucher > wrote: >> >> Yes, obviously, the code has NR parts, so it can't be licensed as BSD >> as it is... > > It's not obvious to me, especially after Juha's further clarifications. > > -- > Robert Kern > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > https://mail.scipy.org/mailman/listinfo/numpy-discussion > -- Information System Engineer, Ph.D. Blog: http://matt.eifelle.com LinkedIn: http://www.linkedin.com/in/matthieubrucher From sturla.molden at gmail.com Wed Sep 30 18:04:50 2015 From: sturla.molden at gmail.com (Sturla Molden) Date: Thu, 01 Oct 2015 00:04:50 +0200 Subject: [Numpy-discussion] Cython-based OpenMP-accelerated quartic polynomial solver In-Reply-To: References: <560AA195.8030001@jyu.fi> <-4907756911279440734@unknownmsgid> <560B9B3B.7000106@jyu.fi> Message-ID: On 30/09/15 11:27, Da?id wrote: > Is there a nice way to ship both versions? After all, most > implementations of BLAS and friends do spawn OpenMP threads, so I don't > think it would be outrageous to take advantage of it in more places; Some do, others don't. ACML uses OpenMP. GotoBLAS uses OpenMP. Intel MKL uses Intel TBB and OpenMP (both of them). OpenBLAS will by default use an internal threadpool. It can be configured to use OpenMP instead. ATLAS uses its own threadpool. Apple Accelerate Framework uses a kernel thread-pool called the Grand Central Dispatch (GCD). A library called libdispatch uses kqueue to access the GCD. There are two principal problems with using OpenMP in NumPy: One is that the GNU OpenMP threadpool is not fork-safe, and can break multiprocessing on some platforms (e.g. when using Python 2.7 on Linux). Anything that uses GCD has this nasty effect on Apple and FreeBSD as well. Note that the problem is actually in multiprocessing. It is not present on Windows (there is no fork system call) and it is avoidable even on Linux with Python 3.4 or later. Also the default builds of NumPy and SciPy on MacOSX uses GCD by default. The other problem is that NumPy with its iterator objects and gufuncs is not particularly fit for multithreading. There will be a lot of contention for the iterator object. Without a major redesign, one thread would do useful work while most of the others would be busy-waiting on a spinlock and fighting for iterator access. Not to mention that the iterator object would be false shared between the processors, which would trash the performance if you have more than one CPU, even when compared to using a single thread. This means that for multithreading to be useful in NumPy, the loops will have to be redesigned so that the work sharing between the threads is taken care of ahead of creating the iterator, i.e. allowing us to use one iterator per thread. Each iterator would then iterate over a slice of the original array. This is ok, but we cannot simply do this by adding OpenMP pragmas in the C code. Given the two problems mentioned above, it would likely be better to use a fork-safe threadpool instead of OpenMP. Sturla From sturla.molden at gmail.com Wed Sep 30 18:07:33 2015 From: sturla.molden at gmail.com (Sturla Molden) Date: Thu, 01 Oct 2015 00:07:33 +0200 Subject: [Numpy-discussion] Cython-based OpenMP-accelerated quartic polynomial solver In-Reply-To: References: <560AA195.8030001@jyu.fi> <-4907756911279440734@unknownmsgid> <560B9B3B.7000106@jyu.fi> Message-ID: On 30/09/15 18:20, Nathaniel Smith wrote: > This is incorrect -- the only common implementation of BLAS that uses > *OpenMP* threads is OpenBLAS, MKL and ACML also use OpenMP. Sturla From sturla.molden at gmail.com Wed Sep 30 18:30:42 2015 From: sturla.molden at gmail.com (Sturla Molden) Date: Thu, 01 Oct 2015 00:30:42 +0200 Subject: [Numpy-discussion] Cython-based OpenMP-accelerated quartic polynomial solver In-Reply-To: References: <560AA195.8030001@jyu.fi> <-4907756911279440734@unknownmsgid> <560B9B3B.7000106@jyu.fi> Message-ID: On 30/09/15 18:20, Chris Barker wrote: > We'd need a run-time check. We need to amend the compiler classes in numpy.distutils with OpenMP relevant information (compiler flags and libraries). The OpenMP support libraries must be statically linked. Sturla From sturla.molden at gmail.com Wed Sep 30 18:50:08 2015 From: sturla.molden at gmail.com (Sturla Molden) Date: Thu, 01 Oct 2015 00:50:08 +0200 Subject: [Numpy-discussion] Cython-based OpenMP-accelerated quartic polynomial solver In-Reply-To: References: <560AA195.8030001@jyu.fi> <-4907756911279440734@unknownmsgid> <560B9B3B.7000106@jyu.fi> Message-ID: On 30/09/15 15:57, Nathan Goldbaum wrote: > Basically, just try to compile a simple OpenMP test program in a > subprocess. If that succeeds, then great, we can add -fopenmp as a > compilation flag. If not, don't do that. Unless you use GCC on Linux, it will be more complex than that. I.e. do you need -fopenmp, -openmp or /openmp? And which libraries do you need to link, if any? Link GOMP? Link some other OpenMP runtime libraries? Link pthreads? There is three different pthread implementations to choose between on Windows, depending on GCC version (MinGW, MinGW-w64, and Cygwin need different pthread libs for OpenMP). It gets messy. The only clean way is to add the correct flags and libraries into the compiler classes in numpy.distutils, and then let distutils and bento query those. STurla From sturla.molden at gmail.com Wed Sep 30 19:02:02 2015 From: sturla.molden at gmail.com (Sturla Molden) Date: Thu, 01 Oct 2015 01:02:02 +0200 Subject: [Numpy-discussion] Cython-based OpenMP-accelerated quartic polynomial solver In-Reply-To: References: <560AA195.8030001@jyu.fi> <-4907756911279440734@unknownmsgid> <560B9B3B.7000106@jyu.fi> Message-ID: On 30/09/15 18:20, Chris Barker wrote: > python threads with nogil? This is often the easiest, particularly if we construct a fork-safe threadpool. Sturla From juha.jeronen at jyu.fi Wed Sep 30 19:53:52 2015 From: juha.jeronen at jyu.fi (Juha Jeronen) Date: Thu, 1 Oct 2015 02:53:52 +0300 Subject: [Numpy-discussion] Cython-based OpenMP-accelerated quartic polynomial solver In-Reply-To: References: <560AA195.8030001@jyu.fi> <-4907756911279440734@unknownmsgid> <560B9B3B.7000106@jyu.fi> Message-ID: <560C7610.1020108@jyu.fi> On 30.09.2015 19:20, Chris Barker wrote: > On Wed, Sep 30, 2015 at 6:57 AM, Nathan Goldbaum > > wrote: > > Note however that with the current version of the code, not > having OpenMP will severely limit the performance, especially > on quad-core machines, since an important factor in the speed > comes from the parallel processing of the independent problem > instances. > > > It seems you got much more than 4 times performance improvement True, good point. That's the power of a short explicit algorithm for a specific problem :) (And maybe some Cython, too.) > -- which is the most you'll get from four cores, presumably :-). not > that another 4 times or so isn't a good thing. True in general :) To be precise, in this case, HyperThreading seems to offer some extra performance. And there seems to be something wonky with the power management on my laptop. The number I posted (1.2e7 quartics per second) seemed slightly low compared to earlier informal benchmarking I had done, and indeed, benchmarking with 8 threads again I'm now getting 1.6e7 quartics per second (i7-4710MQ at 2.5 GHz, RAM at 1333 MHz, running on Linux Mint 17.2). Here are some more complete benchmark results. Because the polynomials to be solved are randomly generated during each run of the program, unless otherwise noted, I've rerun the benchmark three times for each number of threads and picked the lowest-performance result, rounded down. Number of threads vs. quartics solved per second: 1: 3.0e6 2: 6.1e6 4: 1.1e7 8: 1.6e7 (note: higher than previously!) Threads vs. cubics: 1: 8.4e6 2: 1.6e7 4: 3.0e7 8: 4.6e7 Threads vs. quadratics: 1: 2.5e7 2: 4.8e7 4: 8.5e7 (typical across 9 benchmarks; lowest 5.7e7) 8: 9.5e7 From these, in general the weak scaling seems pretty good, and for quadratics and cubics, HyperThreading offers some extra performance, namely about 50% over the 4-thread result in both cases. The total speedup for quartics is 5.3x (3.6x without HT), and for cubics, 5.4x (3.5x without HT). "About 4x, as expected" is still a good characterization, though :) For quadratics, 4 threads seems to be the useful maximum; improvement with HT is only 11%, with a total speedup of 3.8x (vs. 3.4x without HT). These require much fewer arithmetic operations than the higher-order cases, so a possible explanation is that this case is memory-bound. Running the numbers on a simplistic estimate, the total of input and output is far from hitting the DDR3 bandwidth limit, but of course that's ignoring (at least) latencies and the pattern of data accesses. Testing the completely silly "linear" case that is there for documenting code structure only, I get: 1: 1.9e8 2: 3.1e8 4: 3.6e8 8: 2.9e8 which should give a reasonable first approximation for the maximum practical throughput on this machine, when performing almost no arithmetic. Anyway, returning from the sidetrack... > But I suppose there may be another technical option to support > multiple cores > > > python threads with nogil? ...maybe? I hadn't thought about this option. I suppose that in pseudocode, the code structure would need to be something like this? ---8<---8<---8<--- from threading import Thread cdef fast_solver(...) nogil: # ...do stuff without touching Python objects... def my_thread_entry(j_start, j_end): cdef unsigned int j with nogil: for j in range(j_start, j_end): fast_solver(...) # process local slice t1 = Thread( target=my_thread_entry, args=(0, j_split) ) t2 = Thread( target=my_thread_entry, args=(j_split, j_end_global) ) t1.start() t2.start() t1.join() t2.join() ---8<---8<---8<--- -J -------------- next part -------------- An HTML attachment was scrubbed... URL: From juha.jeronen at jyu.fi Wed Sep 30 19:56:15 2015 From: juha.jeronen at jyu.fi (Juha Jeronen) Date: Thu, 1 Oct 2015 02:56:15 +0300 Subject: [Numpy-discussion] Cython-based OpenMP-accelerated quartic polynomial solver In-Reply-To: References: <560AA195.8030001@jyu.fi> Message-ID: <560C769F.3090900@jyu.fi> On 30.09.2015 13:21, Pauli Virtanen wrote: > Juha Jeronen jyu.fi> writes: >> I recently developed a Cython-based, OpenMP-accelerated quartic (and >> cubic, quadratic) polynomial solver to address a personal research need >> for quickly solving a very large number of independent low-degree >> polynomial equations for both real and complex coefficients. > My 2c in this context would be to also think how this best fits > in how collections of polynomials are represented in Numpy. > > AFAICS, Numpy's polynomial module supports evaluation of polynomial > collections (cf. numpy.polynomial.polynomial.polyval), > but the corresponding root finding routine > (numpy.polynomial.polynomial.polyroots) only deals with one > polynomial at a time. > > The present code in principle could be used to speed up the latter > routine after it is generalized to multiple polynomials. The general > case is probably doable just by coding up the companion matrix > approach using low-level routines (or maybe with the new vectorized > np.linalg routines). > > Some relevant code elsewhere for similar purposes can be > found in scipy.inteprolate.PPoly/BPoly (and in the future BSpline). > > https://docs.scipy.org/doc/scipy/reference/generated/scipy.interpolate.PPoly.html > > However, since it's piecewise, there's purposefully support only > for real-valued roots. Thanks. It sounds like numpy.polynomial.polynomial.polyroots() is a good candidate place for this, if as you suggest, it is first generalized to handle multiple polynomials. -J From juha.jeronen at jyu.fi Wed Sep 30 20:20:13 2015 From: juha.jeronen at jyu.fi (Juha Jeronen) Date: Thu, 1 Oct 2015 03:20:13 +0300 Subject: [Numpy-discussion] Cython-based OpenMP-accelerated quartic polynomial solver In-Reply-To: References: <560AA195.8030001@jyu.fi> <-4907756911279440734@unknownmsgid> <560B9B3B.7000106@jyu.fi> Message-ID: <560C7C3D.9080107@jyu.fi> On 30.09.2015 19:20, Nathaniel Smith wrote: > The challenges to providing transparent multithreading in numpy > generally are: > > - gcc + OpenMP on linux still breaks multiprocessing. There's a patch > to fix this but they still haven't applied it; alternatively there's a > workaround you can use in multiprocessing (not using fork mode), but > this requires every user update their code and the workaround has > other limitations. We're unlikely to use OpenMP while this is the case. > Ah, I didn't know this. Thanks. > - parallel code in general is not very composable. If someone is > calling a numpy operation from one thread, great, transparently using > multiple threads internally is a win. If they're exploiting some > higher-level structure in their problem to break it into pieces and > process each in parallel, and then using numpy on each piece, then > numpy spawning threads internally will probably destroy performance. > And numpy is too low-level to know which case it's in. This problem > exists to some extent already with multi-threaded BLAS, so people use > various BLAS-specific knobs to manage it in ad hoc ways, but this > doesn't scale. Very good point. I've had both kinds of use cases myself. It would be nice if there was some way to tell NumPy to either use additional threads or not, but that adds complexity. It's also not a good solution, considering that any higher-level code building on NumPy, if it is designed to be at all reusable, may find *itself* in either role. Only the code that, at any particular point of time in the development of a software project, happens to form the top level at that time, has the required context... Then again, the matter is further complicated by considering codes that run on a single machine, versus codes that run on a cluster. Threads being local to each node in a cluster, it may make sense in a solver targeted for a cluster to split the problem at the process level, distribute the processes across the network, and use the threading capability to accelerate computation on each node. A complex issue with probably no easy solutions :) -J From juha.jeronen at jyu.fi Wed Sep 30 20:32:31 2015 From: juha.jeronen at jyu.fi (Juha Jeronen) Date: Thu, 1 Oct 2015 03:32:31 +0300 Subject: [Numpy-discussion] Cython-based OpenMP-accelerated quartic polynomial solver In-Reply-To: References: <560AA195.8030001@jyu.fi> <-4907756911279440734@unknownmsgid> <560B9B3B.7000106@jyu.fi> Message-ID: <560C7F1F.9060106@jyu.fi> On 01.10.2015 01:04, Sturla Molden wrote: > There are two principal problems with using OpenMP in NumPy: > > One is that the GNU OpenMP threadpool is not fork-safe, and can break > multiprocessing on some platforms (e.g. when using Python 2.7 on > Linux). Anything that uses GCD has this nasty effect on Apple and > FreeBSD as well. Note that the problem is actually in multiprocessing. > It is not present on Windows (there is no fork system call) and it is > avoidable even on Linux with Python 3.4 or later. Also the default > builds of NumPy and SciPy on MacOSX uses GCD by default. > > The other problem is that NumPy with its iterator objects and gufuncs > is not particularly fit for multithreading. There will be a lot of > contention for the iterator object. Without a major redesign, one > thread would do useful work while most of the others would be > busy-waiting on a spinlock and fighting for iterator access. Not to > mention that the iterator object would be false shared between the > processors, which would trash the performance if you have more than > one CPU, even when compared to using a single thread. This means that > for multithreading to be useful in NumPy, the loops will have to be > redesigned so that the work sharing between the threads is taken care > of ahead of creating the iterator, i.e. allowing us to use one > iterator per thread. Each iterator would then iterate over a slice of > the original array. This is ok, but we cannot simply do this by adding > OpenMP pragmas in the C code. Thanks for the details. > Given the two problems mentioned above, it would likely be better to > use a fork-safe threadpool instead of OpenMP. Sounds good. Out of curiosity, are there any standard fork-safe threadpools, or would this imply rolling our own? Also, after Chris's, Nathaniel's and your replies, I'm no longer sure an attempt at transparent multithreading belongs in the polynomial solver. Although the additional performance would be (very) nice when running it from a single-threaded program - after all, quad-cores are common these days - it is not just this specific solver that encounters the issue, but instead the same considerations apply to basically all NumPy operations. So maybe it would be better, at least at first, to make a pure-Cython version with no attempt at multithreading? -J From sturla.molden at gmail.com Wed Sep 30 20:32:49 2015 From: sturla.molden at gmail.com (Sturla Molden) Date: Thu, 01 Oct 2015 02:32:49 +0200 Subject: [Numpy-discussion] Cython-based OpenMP-accelerated quartic polynomial solver In-Reply-To: <560C7C3D.9080107@jyu.fi> References: <560AA195.8030001@jyu.fi> <-4907756911279440734@unknownmsgid> <560B9B3B.7000106@jyu.fi> <560C7C3D.9080107@jyu.fi> Message-ID: On 01/10/15 02:20, Juha Jeronen wrote: > Then again, the matter is further complicated by considering codes that > run on a single machine, versus codes that run on a cluster.Threads > being local to each node in a cluster, You can run MPI programs on a single machine and you get OpenMP implementations for clusters. Just pick an API and stick with it. Sturla From sturla.molden at gmail.com Wed Sep 30 20:52:12 2015 From: sturla.molden at gmail.com (Sturla Molden) Date: Thu, 01 Oct 2015 02:52:12 +0200 Subject: [Numpy-discussion] Cython-based OpenMP-accelerated quartic polynomial solver In-Reply-To: <560C7F1F.9060106@jyu.fi> References: <560AA195.8030001@jyu.fi> <-4907756911279440734@unknownmsgid> <560B9B3B.7000106@jyu.fi> <560C7F1F.9060106@jyu.fi> Message-ID: On 01/10/15 02:32, Juha Jeronen wrote: > Sounds good. Out of curiosity, are there any standard fork-safe > threadpools, or would this imply rolling our own? You have to roll your own. Basically use pthreads_atfork to register a callback that shuts down the threadpool before a fork and another callback that restarts it. Python's threading module does not expose the pthreads_atfork function, so you must call it from Cython. I believe there is also a tiny atfork module in PyPI. > So maybe it would be better, at least at first, to make a pure-Cython > version with no attempt at multithreading? I would start by making a pure Cython version that works correctly. The next step would be to ensure that it releases the GIL. After that you can worry about parallel processing, or just tell the user to use threads or joblib. Sturla