From stefan at bytereef.org Sun Mar 1 20:45:31 2015 From: stefan at bytereef.org (s.krah) Date: Sun, 01 Mar 2015 19:45:31 +0000 Subject: [Python-Dev] Wg: Re: [Python-checkins] cpython (3.4): Issue #23446: Use PyMem_New instead of PyMem_Malloc to avoid possible integer In-Reply-To: <20150301185759.GA23016@bytereef.org> References: <20150216113449.60738.84196@psf.io> <20150216163424.GA3056@bytereef.org> <20150301185759.GA23016@bytereef.org> Message-ID: <14bd6df46d2.c371a38e217388.9204368102411012987@bytereef.org> == == == == == == Weitergeleitete Nachricht == == == == == == Absender : Stefan Krah<stefan at bytereef.org> Empf?nger : "Victor Stinner"<victor.stinner at gmail.com> Datum : So, 01 Mrz 2015 18:58:43 +0000 Betreff : Re: [Python-Dev] [Python-checkins] cpython (3.4): Issue #23446: Use PyMem_New instead of PyMem_Malloc to avoid possible integer == == == == == == Weitergeleitete Nachricht == == == == == == On Mon, Feb 16, 2015 at 10:14:59PM +0100, Victor Stinner wrote: > 2015-02-16 17:34 GMT+01:00 Stefan Krah <stefan at bytereef.org>: > > > > On Mon, Feb 16, 2015 at 11:35:52AM +0000, serhiy.storchaka wrote: > >> diff --git a/Modules/_testbuffer.c b/Modules/_testbuffer.c > >> --- a/Modules/_testbuffer.c > >> +++ b/Modules/_testbuffer.c > >> @@ -850,7 +850,7 @@ > >> Py_ssize_t *dest; > >> Py_ssize_t x, i; > >> > >> - dest = PyMem_Malloc(len * (sizeof *dest)); > >> + dest = PyMem_New(Py_ssize_t, len); > >> if (dest == NULL) { > >> PyErr_NoMemory(); > >> return NULL; > > > > This, too, was already protected by len == ndim <= 64. > > I don't understand why you don't want to use PyMem_New() even if it > cannot overflow. PyMem_New() is more readable no? It's readable, but I don't see a reason to change code that already has an overflow analysis, especially in 3.4. As I wrote in http://bugs.python.org/issue23446#msg235770 , people need to see that one *can* make certain assumptions about PEP-3118 buffers (otherwise one would go insane with overflow checks when doing arithmetic on the buffers. So, in a sense, this commit removes information for the reader. I would prefer an "assert(len <= 64)" for documentation purposes while keeping the multiplication. Stefan Krah -------------- next part -------------- An HTML attachment was scrubbed... URL: From mistersheik at gmail.com Mon Mar 2 19:35:52 2015 From: mistersheik at gmail.com (Neil Girdhar) Date: Mon, 2 Mar 2015 13:35:52 -0500 Subject: [Python-Dev] PEP 448 review In-Reply-To: References: Message-ID: Hi everyone, The patch is ready for review now, and I should have time this week to make changes and respond to comments. Best, Neil On Wed, Feb 25, 2015 at 2:42 PM, Guido van Rossum wrote: > I'm back, I've re-read the PEP, and I've re-read the long thread with "(no > subject)". > > I think Georg Brandl nailed it: > > """ > > > > > > > > > *I like the "sequence and dict flattening" part of the PEP, mostly because > itis consistent and should be easy to understand, but the comprehension > syntaxenhancements seem to be bad for readability and "comprehending" what > the codedoes.The call syntax part is a mixed bag on the one hand it is nice > to be consistent with the extended possibilities in literals (flattening), > but on the other hand there would be small but annoying inconsistencies > anyways (e.g. the duplicate kwarg case above).* > """ > > Greg Ewing followed up explaining that the inconsistency between dict > flattening and call syntax is inherent in the pre-existing different rules > for dicts vs. keyword args: {'a':1, 'a':2} results in {'a':2}, while f(a=1, > a=2) is an error. (This form is a SyntaxError; the dynamic case f(a=1, > **{'a': 1}) is a TypeError.) > > For me, allowing f(*a, *b) and f(**d, **e) and all the other combinations > for function calls proposed by the PEP is an easy +1 -- it's a > straightforward extension of the existing pattern, and anybody who knows > what f(x, *a) does will understand f(x, *a, y, *b). Guessing what f(**d, > **e) means shouldn't be hard either. Understanding the edge case for > duplicate keys with f(**d, **e) is a little harder, but the error messages > are pretty clear, and it is not a new edge case. > > The sequence and dict flattening syntax proposals are also clean and > logical -- we already have *-unpacking on the receiving side, so allowing > *x in tuple expressions reads pretty naturally (and the similarity with *a > in argument lists certainly helps). From here, having [a, *x, b, *y] is > also natural, and then the extension to other displays is natural: {a, *x, > b, *y} and {a:1, **d, b:2, **e}. This, too, gets a +1 from me. > > So that leaves comprehensions. IIRC, during the development of the patch > we realized that f(*x for x in xs) is sufficiently ambiguous that we > decided to disallow it -- note that f(x for x in xs) is already somewhat of > a special case because an argument can only be a "bare" generator > expression if it is the only argument. The same reasoning doesn't apply (in > that form) to list, set and dict comprehensions -- while f(x for x in xs) > is identical in meaning to f((x for x in xs)), [x for x in xs] is NOT the > same as [(x for x in xs)] (that's a list of one element, and the element is > a generator expression). > > The basic premise of this part of the proposal is that if you have a few > iterables, the new proposal (without comprehensions) lets you create a list > or generator expression that iterates over all of them, essentially > flattening them: > > >>> xs = [1, 2, 3] > >>> ys = ['abc', 'def'] > >>> zs = [99] > >>> [*xs, *ys, *zs] > [1, 2, 3, 'abc', 'def', 99] > >>> > > But now suppose you have a list of iterables: > > >>> xss = [[1, 2, 3], ['abc', 'def'], [99]] > >>> [*xss[0], *xss[1], *xss[2]] > [1, 2, 3, 'abc', 'def', 99] > >>> > > Wouldn't it be nice if you could write the latter using a comprehension? > > >>> xss = [[1, 2, 3], ['abc', 'def'], [99]] > >>> [*xs for xs in xss] > [1, 2, 3, 'abc', 'def', 99] > >>> > > This is somewhat seductive, and the following is even nicer: the *xs > position may be an expression, e.g.: > > >>> xss = [[1, 2, 3], ['abc', 'def'], [99]] > >>> [*xs[:2] for xs in xss] > [1, 2, 'abc', 'def', 99] > >>> > > On the other hand, I had to explore the possibilities here by > experimenting in the interpreter, and I discovered some odd edge cases > (e.g. you can parenthesize the starred expression, but that seems a > syntactic accident). > > All in all I am personally +0 on the comprehension part of the PEP, and I > like that it provides a way to "flatten" a sequence of sequences, but I > think very few people in the thread have supported this part. Therefore I > would like to ask Neil to update the PEP and the patch to take out the > comprehension part, so that the two "easy wins" can make it into Python 3.5 > (basically, I am accepting two-thirds of the PEP :-). There is some time > yet until alpha 2. > > I would also like code reviewers (Benjamin?) to start reviewing the patch > , taking into account that the > comprehension part needs to be removed. > > -- > --Guido van Rossum (python.org/~guido) > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From victor.stinner at gmail.com Mon Mar 2 21:17:52 2015 From: victor.stinner at gmail.com (Victor Stinner) Date: Mon, 2 Mar 2015 21:17:52 +0100 Subject: [Python-Dev] PEP 448 review In-Reply-To: References: Message-ID: Where is the patch? Victor Le lundi 2 mars 2015, Neil Girdhar a ?crit : > Hi everyone, > > The patch is ready for review now, and I should have time this week to > make changes and respond to comments. > > Best, > > Neil > > On Wed, Feb 25, 2015 at 2:42 PM, Guido van Rossum > wrote: > >> I'm back, I've re-read the PEP, and I've re-read the long thread with >> "(no subject)". >> >> I think Georg Brandl nailed it: >> >> """ >> >> >> >> >> >> >> >> >> *I like the "sequence and dict flattening" part of the PEP, mostly >> because itis consistent and should be easy to understand, but the >> comprehension syntaxenhancements seem to be bad for readability and >> "comprehending" what the codedoes.The call syntax part is a mixed bag on >> the one hand it is nice to be consistent with the extended possibilities in >> literals (flattening), but on the other hand there would be small but >> annoying inconsistencies anyways (e.g. the duplicate kwarg case above).* >> """ >> >> Greg Ewing followed up explaining that the inconsistency between dict >> flattening and call syntax is inherent in the pre-existing different rules >> for dicts vs. keyword args: {'a':1, 'a':2} results in {'a':2}, while f(a=1, >> a=2) is an error. (This form is a SyntaxError; the dynamic case f(a=1, >> **{'a': 1}) is a TypeError.) >> >> For me, allowing f(*a, *b) and f(**d, **e) and all the other combinations >> for function calls proposed by the PEP is an easy +1 -- it's a >> straightforward extension of the existing pattern, and anybody who knows >> what f(x, *a) does will understand f(x, *a, y, *b). Guessing what f(**d, >> **e) means shouldn't be hard either. Understanding the edge case for >> duplicate keys with f(**d, **e) is a little harder, but the error messages >> are pretty clear, and it is not a new edge case. >> >> The sequence and dict flattening syntax proposals are also clean and >> logical -- we already have *-unpacking on the receiving side, so allowing >> *x in tuple expressions reads pretty naturally (and the similarity with *a >> in argument lists certainly helps). From here, having [a, *x, b, *y] is >> also natural, and then the extension to other displays is natural: {a, *x, >> b, *y} and {a:1, **d, b:2, **e}. This, too, gets a +1 from me. >> >> So that leaves comprehensions. IIRC, during the development of the patch >> we realized that f(*x for x in xs) is sufficiently ambiguous that we >> decided to disallow it -- note that f(x for x in xs) is already somewhat of >> a special case because an argument can only be a "bare" generator >> expression if it is the only argument. The same reasoning doesn't apply (in >> that form) to list, set and dict comprehensions -- while f(x for x in xs) >> is identical in meaning to f((x for x in xs)), [x for x in xs] is NOT the >> same as [(x for x in xs)] (that's a list of one element, and the element is >> a generator expression). >> >> The basic premise of this part of the proposal is that if you have a few >> iterables, the new proposal (without comprehensions) lets you create a list >> or generator expression that iterates over all of them, essentially >> flattening them: >> >> >>> xs = [1, 2, 3] >> >>> ys = ['abc', 'def'] >> >>> zs = [99] >> >>> [*xs, *ys, *zs] >> [1, 2, 3, 'abc', 'def', 99] >> >>> >> >> But now suppose you have a list of iterables: >> >> >>> xss = [[1, 2, 3], ['abc', 'def'], [99]] >> >>> [*xss[0], *xss[1], *xss[2]] >> [1, 2, 3, 'abc', 'def', 99] >> >>> >> >> Wouldn't it be nice if you could write the latter using a comprehension? >> >> >>> xss = [[1, 2, 3], ['abc', 'def'], [99]] >> >>> [*xs for xs in xss] >> [1, 2, 3, 'abc', 'def', 99] >> >>> >> >> This is somewhat seductive, and the following is even nicer: the *xs >> position may be an expression, e.g.: >> >> >>> xss = [[1, 2, 3], ['abc', 'def'], [99]] >> >>> [*xs[:2] for xs in xss] >> [1, 2, 'abc', 'def', 99] >> >>> >> >> On the other hand, I had to explore the possibilities here by >> experimenting in the interpreter, and I discovered some odd edge cases >> (e.g. you can parenthesize the starred expression, but that seems a >> syntactic accident). >> >> All in all I am personally +0 on the comprehension part of the PEP, and I >> like that it provides a way to "flatten" a sequence of sequences, but I >> think very few people in the thread have supported this part. Therefore I >> would like to ask Neil to update the PEP and the patch to take out the >> comprehension part, so that the two "easy wins" can make it into Python 3.5 >> (basically, I am accepting two-thirds of the PEP :-). There is some time >> yet until alpha 2. >> >> I would also like code reviewers (Benjamin?) to start reviewing the patch >> , taking into account that the >> comprehension part needs to be removed. >> >> -- >> --Guido van Rossum (python.org/~guido) >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mistersheik at gmail.com Mon Mar 2 21:18:12 2015 From: mistersheik at gmail.com (Neil Girdhar) Date: Mon, 2 Mar 2015 15:18:12 -0500 Subject: [Python-Dev] PEP 448 review In-Reply-To: References: Message-ID: http://bugs.python.org/issue2292 On Mon, Mar 2, 2015 at 3:17 PM, Victor Stinner wrote: > Where is the patch? > > Victor > > Le lundi 2 mars 2015, Neil Girdhar a ?crit : > > Hi everyone, >> >> The patch is ready for review now, and I should have time this week to >> make changes and respond to comments. >> >> Best, >> >> Neil >> >> On Wed, Feb 25, 2015 at 2:42 PM, Guido van Rossum >> wrote: >> >>> I'm back, I've re-read the PEP, and I've re-read the long thread with >>> "(no subject)". >>> >>> I think Georg Brandl nailed it: >>> >>> """ >>> >>> >>> >>> >>> >>> >>> >>> >>> *I like the "sequence and dict flattening" part of the PEP, mostly >>> because itis consistent and should be easy to understand, but the >>> comprehension syntaxenhancements seem to be bad for readability and >>> "comprehending" what the codedoes.The call syntax part is a mixed bag on >>> the one hand it is nice to be consistent with the extended possibilities in >>> literals (flattening), but on the other hand there would be small but >>> annoying inconsistencies anyways (e.g. the duplicate kwarg case above).* >>> """ >>> >>> Greg Ewing followed up explaining that the inconsistency between dict >>> flattening and call syntax is inherent in the pre-existing different rules >>> for dicts vs. keyword args: {'a':1, 'a':2} results in {'a':2}, while f(a=1, >>> a=2) is an error. (This form is a SyntaxError; the dynamic case f(a=1, >>> **{'a': 1}) is a TypeError.) >>> >>> For me, allowing f(*a, *b) and f(**d, **e) and all the other >>> combinations for function calls proposed by the PEP is an easy +1 -- it's a >>> straightforward extension of the existing pattern, and anybody who knows >>> what f(x, *a) does will understand f(x, *a, y, *b). Guessing what f(**d, >>> **e) means shouldn't be hard either. Understanding the edge case for >>> duplicate keys with f(**d, **e) is a little harder, but the error messages >>> are pretty clear, and it is not a new edge case. >>> >>> The sequence and dict flattening syntax proposals are also clean and >>> logical -- we already have *-unpacking on the receiving side, so allowing >>> *x in tuple expressions reads pretty naturally (and the similarity with *a >>> in argument lists certainly helps). From here, having [a, *x, b, *y] is >>> also natural, and then the extension to other displays is natural: {a, *x, >>> b, *y} and {a:1, **d, b:2, **e}. This, too, gets a +1 from me. >>> >>> So that leaves comprehensions. IIRC, during the development of the patch >>> we realized that f(*x for x in xs) is sufficiently ambiguous that we >>> decided to disallow it -- note that f(x for x in xs) is already somewhat of >>> a special case because an argument can only be a "bare" generator >>> expression if it is the only argument. The same reasoning doesn't apply (in >>> that form) to list, set and dict comprehensions -- while f(x for x in xs) >>> is identical in meaning to f((x for x in xs)), [x for x in xs] is NOT the >>> same as [(x for x in xs)] (that's a list of one element, and the element is >>> a generator expression). >>> >>> The basic premise of this part of the proposal is that if you have a few >>> iterables, the new proposal (without comprehensions) lets you create a list >>> or generator expression that iterates over all of them, essentially >>> flattening them: >>> >>> >>> xs = [1, 2, 3] >>> >>> ys = ['abc', 'def'] >>> >>> zs = [99] >>> >>> [*xs, *ys, *zs] >>> [1, 2, 3, 'abc', 'def', 99] >>> >>> >>> >>> But now suppose you have a list of iterables: >>> >>> >>> xss = [[1, 2, 3], ['abc', 'def'], [99]] >>> >>> [*xss[0], *xss[1], *xss[2]] >>> [1, 2, 3, 'abc', 'def', 99] >>> >>> >>> >>> Wouldn't it be nice if you could write the latter using a comprehension? >>> >>> >>> xss = [[1, 2, 3], ['abc', 'def'], [99]] >>> >>> [*xs for xs in xss] >>> [1, 2, 3, 'abc', 'def', 99] >>> >>> >>> >>> This is somewhat seductive, and the following is even nicer: the *xs >>> position may be an expression, e.g.: >>> >>> >>> xss = [[1, 2, 3], ['abc', 'def'], [99]] >>> >>> [*xs[:2] for xs in xss] >>> [1, 2, 'abc', 'def', 99] >>> >>> >>> >>> On the other hand, I had to explore the possibilities here by >>> experimenting in the interpreter, and I discovered some odd edge cases >>> (e.g. you can parenthesize the starred expression, but that seems a >>> syntactic accident). >>> >>> All in all I am personally +0 on the comprehension part of the PEP, and >>> I like that it provides a way to "flatten" a sequence of sequences, but I >>> think very few people in the thread have supported this part. Therefore I >>> would like to ask Neil to update the PEP and the patch to take out the >>> comprehension part, so that the two "easy wins" can make it into Python 3.5 >>> (basically, I am accepting two-thirds of the PEP :-). There is some time >>> yet until alpha 2. >>> >>> I would also like code reviewers (Benjamin?) to start reviewing the >>> patch , taking into account that the >>> comprehension part needs to be removed. >>> >>> -- >>> --Guido van Rossum (python.org/~guido) >>> >>> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From ethan at stoneleaf.us Mon Mar 2 22:03:33 2015 From: ethan at stoneleaf.us (Ethan Furman) Date: Mon, 02 Mar 2015 13:03:33 -0800 Subject: [Python-Dev] PEP 448 review In-Reply-To: References: Message-ID: <54F4D025.3010000@stoneleaf.us> On 03/02/2015 12:18 PM, Neil Girdhar wrote: > On Mon, Mar 2, 2015 at 3:17 PM, Victor Stinner wrote: >> Le lundi 2 mars 2015, Neil Girdhar a ?crit : >>> >>> The patch is ready for review now, and I should have time this week to make changes and respond to comments. >> >> Where is the patch? > > http://bugs.python.org/issue2292 The latest file there is from Feb 26, while your message that the patch was ready for review is from today -- so is the patch from five days ago the most recent? -- ~Ethan~ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: OpenPGP digital signature URL: From mistersheik at gmail.com Mon Mar 2 22:54:38 2015 From: mistersheik at gmail.com (Neil Girdhar) Date: Mon, 2 Mar 2015 16:54:38 -0500 Subject: [Python-Dev] PEP 448 review In-Reply-To: References: Message-ID: It's from five days ago. I asked Joshua to take a look at something, but I guess he is busy. Best, Neil ? The latest file there is from Feb 26, while your message that the patch was ready for review is from today -- so is the patch from five days ago the most recent? -- ~Ethan~ On Mon, Mar 2, 2015 at 3:18 PM, Neil Girdhar wrote: > http://bugs.python.org/issue2292 > > On Mon, Mar 2, 2015 at 3:17 PM, Victor Stinner > wrote: > >> Where is the patch? >> >> Victor >> >> Le lundi 2 mars 2015, Neil Girdhar a ?crit : >> >> Hi everyone, >>> >>> The patch is ready for review now, and I should have time this week to >>> make changes and respond to comments. >>> >>> Best, >>> >>> Neil >>> >>> On Wed, Feb 25, 2015 at 2:42 PM, Guido van Rossum >>> wrote: >>> >>>> I'm back, I've re-read the PEP, and I've re-read the long thread with >>>> "(no subject)". >>>> >>>> I think Georg Brandl nailed it: >>>> >>>> """ >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> *I like the "sequence and dict flattening" part of the PEP, mostly >>>> because itis consistent and should be easy to understand, but the >>>> comprehension syntaxenhancements seem to be bad for readability and >>>> "comprehending" what the codedoes.The call syntax part is a mixed bag on >>>> the one hand it is nice to be consistent with the extended possibilities in >>>> literals (flattening), but on the other hand there would be small but >>>> annoying inconsistencies anyways (e.g. the duplicate kwarg case above).* >>>> """ >>>> >>>> Greg Ewing followed up explaining that the inconsistency between dict >>>> flattening and call syntax is inherent in the pre-existing different rules >>>> for dicts vs. keyword args: {'a':1, 'a':2} results in {'a':2}, while f(a=1, >>>> a=2) is an error. (This form is a SyntaxError; the dynamic case f(a=1, >>>> **{'a': 1}) is a TypeError.) >>>> >>>> For me, allowing f(*a, *b) and f(**d, **e) and all the other >>>> combinations for function calls proposed by the PEP is an easy +1 -- it's a >>>> straightforward extension of the existing pattern, and anybody who knows >>>> what f(x, *a) does will understand f(x, *a, y, *b). Guessing what f(**d, >>>> **e) means shouldn't be hard either. Understanding the edge case for >>>> duplicate keys with f(**d, **e) is a little harder, but the error messages >>>> are pretty clear, and it is not a new edge case. >>>> >>>> The sequence and dict flattening syntax proposals are also clean and >>>> logical -- we already have *-unpacking on the receiving side, so allowing >>>> *x in tuple expressions reads pretty naturally (and the similarity with *a >>>> in argument lists certainly helps). From here, having [a, *x, b, *y] is >>>> also natural, and then the extension to other displays is natural: {a, *x, >>>> b, *y} and {a:1, **d, b:2, **e}. This, too, gets a +1 from me. >>>> >>>> So that leaves comprehensions. IIRC, during the development of the >>>> patch we realized that f(*x for x in xs) is sufficiently ambiguous that we >>>> decided to disallow it -- note that f(x for x in xs) is already somewhat of >>>> a special case because an argument can only be a "bare" generator >>>> expression if it is the only argument. The same reasoning doesn't apply (in >>>> that form) to list, set and dict comprehensions -- while f(x for x in xs) >>>> is identical in meaning to f((x for x in xs)), [x for x in xs] is NOT the >>>> same as [(x for x in xs)] (that's a list of one element, and the element is >>>> a generator expression). >>>> >>>> The basic premise of this part of the proposal is that if you have a >>>> few iterables, the new proposal (without comprehensions) lets you create a >>>> list or generator expression that iterates over all of them, essentially >>>> flattening them: >>>> >>>> >>> xs = [1, 2, 3] >>>> >>> ys = ['abc', 'def'] >>>> >>> zs = [99] >>>> >>> [*xs, *ys, *zs] >>>> [1, 2, 3, 'abc', 'def', 99] >>>> >>> >>>> >>>> But now suppose you have a list of iterables: >>>> >>>> >>> xss = [[1, 2, 3], ['abc', 'def'], [99]] >>>> >>> [*xss[0], *xss[1], *xss[2]] >>>> [1, 2, 3, 'abc', 'def', 99] >>>> >>> >>>> >>>> Wouldn't it be nice if you could write the latter using a comprehension? >>>> >>>> >>> xss = [[1, 2, 3], ['abc', 'def'], [99]] >>>> >>> [*xs for xs in xss] >>>> [1, 2, 3, 'abc', 'def', 99] >>>> >>> >>>> >>>> This is somewhat seductive, and the following is even nicer: the *xs >>>> position may be an expression, e.g.: >>>> >>>> >>> xss = [[1, 2, 3], ['abc', 'def'], [99]] >>>> >>> [*xs[:2] for xs in xss] >>>> [1, 2, 'abc', 'def', 99] >>>> >>> >>>> >>>> On the other hand, I had to explore the possibilities here by >>>> experimenting in the interpreter, and I discovered some odd edge cases >>>> (e.g. you can parenthesize the starred expression, but that seems a >>>> syntactic accident). >>>> >>>> All in all I am personally +0 on the comprehension part of the PEP, and >>>> I like that it provides a way to "flatten" a sequence of sequences, but I >>>> think very few people in the thread have supported this part. Therefore I >>>> would like to ask Neil to update the PEP and the patch to take out the >>>> comprehension part, so that the two "easy wins" can make it into Python 3.5 >>>> (basically, I am accepting two-thirds of the PEP :-). There is some time >>>> yet until alpha 2. >>>> >>>> I would also like code reviewers (Benjamin?) to start reviewing the >>>> patch , taking into account that the >>>> comprehension part needs to be removed. >>>> >>>> -- >>>> --Guido van Rossum (python.org/~guido) >>>> >>>> >>> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.barker at noaa.gov Tue Mar 3 06:25:21 2015 From: chris.barker at noaa.gov (Chris Barker) Date: Mon, 2 Mar 2015 21:25:21 -0800 Subject: [Python-Dev] PEP 485 review (isclose()) In-Reply-To: References: Message-ID: On Fri, Feb 27, 2015 at 12:07 PM, Chris Barker wrote: > I'll edit the text as you suggest, > Done. > and then work on a patch -- I'm sure I'll have questions for Python-dev > when I actually do that, but I'll get started on my own and see how far I > get. > OK -- big question 1: As far as I can tell, the math module is entirely a C extension. So I can: 1) write isclose() in C -- I could probably do that (It's been a while -- I'm a big Cython fan these days...) and I'd probably punt on having it work with anything other than floats if I did that -- which would fit how most of the match module works anyway. 2) Write it in Python, and monkey-patch it in to the math module -- I honestly have no idea how to do that, but as I can add a new name to the math module after importing it, it should be doable --but I have no idea where the code would go. Suggestions?? Thanks, -Chris > -Chris > > > > > > On Fri, Feb 27, 2015 at 11:38 AM, Guido van Rossum > wrote: > >> I think it's time to accept PEP 485. I've re-read it once more, and it >> looks like the text is in great shape. (My only recommendation would be to >> update the Abstract to state that we're specifically adding math.isclose().) >> >> A wording question: "This implementation has a flag that lets the user >> select which relative tolerance test to apply -- this PEP does not suggest >> that that be retained, but rather than the weak test be selected." -- I >> think this was meant to say "... rather *that* the weak test be selected", >> right? (It would be nice if the sample implementation defaulted to the >> choice in the PEP.) >> >> However, those are just minor edits, and I hereby approve the PEP. Thanks >> Chris and everyone else for the fruitful discussion (and thanks especially >> to Chris for eventually ending the bikeshedding and writing a PEP that >> explains each of the choices). Congrats! >> >> -- >> --Guido van Rossum (python.org/~guido) >> > > > > -- > > Christopher Barker, Ph.D. > Oceanographer > > Emergency Response Division > NOAA/NOS/OR&R (206) 526-6959 voice > 7600 Sand Point Way NE (206) 526-6329 fax > Seattle, WA 98115 (206) 526-6317 main reception > > Chris.Barker at noaa.gov > -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Tue Mar 3 07:39:21 2015 From: guido at python.org (Guido van Rossum) Date: Mon, 2 Mar 2015 22:39:21 -0800 Subject: [Python-Dev] PEP 485 review (isclose()) In-Reply-To: References: Message-ID: It should be written in C. I thought you knew. :-) I guess it's okay to drop Decimal support (*). Complex support probably should be in cmath. But this is an amendmend to the PEP as accepted. If anyone cares about these changes they should speak up, otherwise the PEP should be updated. (*) Adding it to the decimal module would require a discussion with Raymond Hettinger, but Decimal users can probably copy and paste the formula from the PEP. On Monday, March 2, 2015, Chris Barker wrote: > On Fri, Feb 27, 2015 at 12:07 PM, Chris Barker > wrote: > >> I'll edit the text as you suggest, >> > > Done. > > >> and then work on a patch -- I'm sure I'll have questions for Python-dev >> when I actually do that, but I'll get started on my own and see how far I >> get. >> > > OK -- big question 1: > > As far as I can tell, the math module is entirely a C extension. So I can: > > 1) write isclose() in C -- I could probably do that (It's been a while -- > I'm a big Cython fan these days...) and I'd probably punt on having it work > with anything other than floats if I did that -- which would fit how most > of the match module works anyway. > > 2) Write it in Python, and monkey-patch it in to the math module -- I > honestly have no idea how to do that, but as I can add a new name to the > math module after importing it, it should be doable --but I have no idea > where the code would go. > > Suggestions?? > > Thanks, > -Chris > > > > > > > > >> -Chris >> >> >> >> >> >> On Fri, Feb 27, 2015 at 11:38 AM, Guido van Rossum > > wrote: >> >>> I think it's time to accept PEP 485. I've re-read it once more, and it >>> looks like the text is in great shape. (My only recommendation would be to >>> update the Abstract to state that we're specifically adding math.isclose().) >>> >>> A wording question: "This implementation has a flag that lets the user >>> select which relative tolerance test to apply -- this PEP does not suggest >>> that that be retained, but rather than the weak test be selected." -- I >>> think this was meant to say "... rather *that* the weak test be selected", >>> right? (It would be nice if the sample implementation defaulted to the >>> choice in the PEP.) >>> >>> However, those are just minor edits, and I hereby approve the PEP. >>> Thanks Chris and everyone else for the fruitful discussion (and thanks >>> especially to Chris for eventually ending the bikeshedding and writing a >>> PEP that explains each of the choices). Congrats! >>> >>> -- >>> --Guido van Rossum (python.org/~guido) >>> >> >> >> >> -- >> >> Christopher Barker, Ph.D. >> Oceanographer >> >> Emergency Response Division >> NOAA/NOS/OR&R (206) 526-6959 voice >> 7600 Sand Point Way NE (206) 526-6329 fax >> Seattle, WA 98115 (206) 526-6317 main reception >> >> Chris.Barker at noaa.gov >> >> > > > > -- > > Christopher Barker, Ph.D. > Oceanographer > > Emergency Response Division > NOAA/NOS/OR&R (206) 526-6959 voice > 7600 Sand Point Way NE (206) 526-6329 fax > Seattle, WA 98115 (206) 526-6317 main reception > > Chris.Barker at noaa.gov > > -- --Guido van Rossum (on iPad) -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.barker at noaa.gov Tue Mar 3 07:48:31 2015 From: chris.barker at noaa.gov (Chris Barker) Date: Mon, 2 Mar 2015 22:48:31 -0800 Subject: [Python-Dev] PEP 485 review (isclose()) In-Reply-To: References: Message-ID: On Mon, Mar 2, 2015 at 10:39 PM, Guido van Rossum wrote: > It should be written in C. I thought you knew. :-) > It crossed my mind when we talked about the math module -- but I didn't really think about it... I guess it's okay to drop Decimal support (*). Complex support probably > should be in cmath. > fair enough -- and easy enough to do if we do float-only in math, and complex-only in cmath. But this is an amendmend to the PEP as accepted. If anyone cares about > these changes they should speak up, otherwise the PEP should be updated. > will do. > (*) Adding it to the decimal module would require a discussion with > Raymond Hettinger, but Decimal users can probably copy and paste the > formula from the PEP. > > yup -- but maybe worth putting in there while we're at it. though as Decimal is arbitrary precision, maybe it's not needed.... -Chris > On Monday, March 2, 2015, Chris Barker wrote: > >> On Fri, Feb 27, 2015 at 12:07 PM, Chris Barker >> wrote: >> >>> I'll edit the text as you suggest, >>> >> >> Done. >> >> >>> and then work on a patch -- I'm sure I'll have questions for Python-dev >>> when I actually do that, but I'll get started on my own and see how far I >>> get. >>> >> >> OK -- big question 1: >> >> As far as I can tell, the math module is entirely a C extension. So I can: >> >> 1) write isclose() in C -- I could probably do that (It's been a while -- >> I'm a big Cython fan these days...) and I'd probably punt on having it work >> with anything other than floats if I did that -- which would fit how most >> of the match module works anyway. >> >> 2) Write it in Python, and monkey-patch it in to the math module -- I >> honestly have no idea how to do that, but as I can add a new name to the >> math module after importing it, it should be doable --but I have no idea >> where the code would go. >> >> Suggestions?? >> >> Thanks, >> -Chris >> >> >> >> >> >> >> >> >>> -Chris >>> >>> >>> >>> >>> >>> On Fri, Feb 27, 2015 at 11:38 AM, Guido van Rossum >>> wrote: >>> >>>> I think it's time to accept PEP 485. I've re-read it once more, and it >>>> looks like the text is in great shape. (My only recommendation would be to >>>> update the Abstract to state that we're specifically adding math.isclose().) >>>> >>>> A wording question: "This implementation has a flag that lets the user >>>> select which relative tolerance test to apply -- this PEP does not suggest >>>> that that be retained, but rather than the weak test be selected." -- I >>>> think this was meant to say "... rather *that* the weak test be selected", >>>> right? (It would be nice if the sample implementation defaulted to the >>>> choice in the PEP.) >>>> >>>> However, those are just minor edits, and I hereby approve the PEP. >>>> Thanks Chris and everyone else for the fruitful discussion (and thanks >>>> especially to Chris for eventually ending the bikeshedding and writing a >>>> PEP that explains each of the choices). Congrats! >>>> >>>> -- >>>> --Guido van Rossum (python.org/~guido) >>>> >>> >>> >>> >>> -- >>> >>> Christopher Barker, Ph.D. >>> Oceanographer >>> >>> Emergency Response Division >>> NOAA/NOS/OR&R (206) 526-6959 voice >>> 7600 Sand Point Way NE (206) 526-6329 fax >>> Seattle, WA 98115 (206) 526-6317 main reception >>> >>> Chris.Barker at noaa.gov >>> >> >> >> >> -- >> >> Christopher Barker, Ph.D. >> Oceanographer >> >> Emergency Response Division >> NOAA/NOS/OR&R (206) 526-6959 voice >> 7600 Sand Point Way NE (206) 526-6329 fax >> Seattle, WA 98115 (206) 526-6317 main reception >> >> Chris.Barker at noaa.gov >> > > > -- > --Guido van Rossum (on iPad) > -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From victor.stinner at gmail.com Tue Mar 3 10:17:40 2015 From: victor.stinner at gmail.com (Victor Stinner) Date: Tue, 3 Mar 2015 10:17:40 +0100 Subject: [Python-Dev] PEP 485 review (isclose()) In-Reply-To: References: Message-ID: 2015-03-03 6:25 GMT+01:00 Chris Barker : > As far as I can tell, the math module is entirely a C extension. So I can: > (...) > 2) Write it in Python, and monkey-patch it in to the math module -- I > honestly have no idea how to do that, but as I can add a new name to the > math module after importing it, it should be doable --but I have no idea > where the code would go. Maybe it's time to rename the math module to _math and create a math.py module, like _decimal/decimal? math.py should end with "from _math import *". It may be interesting to have Python implementation of math.fsum(), math.factorial(), math.degrees() and math.radians(). Extract of fsum() comment: Full precision summation of a sequence of floats. def msum(iterable): partials = [] # sorted, non-overlapping partial sums for x in iterable: i = 0 for y in partials: if abs(x) < abs(y): x, y = y, x hi = x + y lo = y - (hi - x) if lo: partials[i] = lo i += 1 x = hi partials[i:] = [x] return sum_exact(partials) The C implementation of factorial is not naive: "Divide-and-conquer factorial algorithm" (see the C code). Obvisouly, a expect lower performances from Python code manipulating numbers (because of the cost of boxing-unboxing, cost of functions calls, etc.). But it might help other Python implementations to implement the math module. Victor From guido at python.org Tue Mar 3 17:14:39 2015 From: guido at python.org (Guido van Rossum) Date: Tue, 3 Mar 2015 08:14:39 -0800 Subject: [Python-Dev] PEP 485 review (isclose()) In-Reply-To: References: Message-ID: On Mon, Mar 2, 2015 at 10:48 PM, Chris Barker wrote: > On Mon, Mar 2, 2015 at 10:39 PM, Guido van Rossum > wrote: > > (*) Adding it to the decimal module would require a discussion with >> Raymond Hettinger, but Decimal users can probably copy and paste the >> formula from the PEP. >> > > yup -- but maybe worth putting in there while we're at it. though as > Decimal is arbitrary precision, maybe it's not needed.... > It's not really arbitrary precision, it's decimal *floating* point with a large but finite precision, so all the same arguments apply. But my reasoning was more that (at least when I was last involved in the project) the decimal module tries to stick pretty close to the IEEE 754 standard (maybe nowadays a later version?) and random Python-only additions are controversial. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From ethan at stoneleaf.us Tue Mar 3 17:43:06 2015 From: ethan at stoneleaf.us (Ethan Furman) Date: Tue, 03 Mar 2015 08:43:06 -0800 Subject: [Python-Dev] PEP 485 review (isclose()) In-Reply-To: References: Message-ID: <54F5E49A.8080803@stoneleaf.us> On 03/03/2015 01:17 AM, Victor Stinner wrote: > 2015-03-03 6:25 GMT+01:00 Chris Barker : >> >> As far as I can tell, the math module is entirely a C extension. So I can: >> 2) Write it in Python, and monkey-patch it in to the math module -- I >> honestly have no idea how to do that, but as I can add a new name to the >> math module after importing it, it should be doable --but I have no idea >> where the code would go. > Maybe it's time to rename the math module to _math and create a > math.py module, like _decimal/decimal? math.py should end with "from > _math import *". +1 -- ~Ethan~ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: OpenPGP digital signature URL: From chris.barker at noaa.gov Wed Mar 4 21:12:34 2015 From: chris.barker at noaa.gov (Chris Barker) Date: Wed, 4 Mar 2015 12:12:34 -0800 Subject: [Python-Dev] PEP 485 review (isclose()) In-Reply-To: <54F5E49A.8080803@stoneleaf.us> References: <54F5E49A.8080803@stoneleaf.us> Message-ID: On Tue, Mar 3, 2015 at 8:43 AM, Ethan Furman wrote: > On 03/03/2015 01:17 AM, Victor Stinner wrote: > > > Maybe it's time to rename the math module to _math and create a > > math.py module, like _decimal/decimal? math.py should end with "from > > _math import *". > > +1 > What do folks think? If we're going to do this, I'll write isclose() in python. And I could do the work to split it out, too, I suppose. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Wed Mar 4 21:22:41 2015 From: guido at python.org (Guido van Rossum) Date: Wed, 4 Mar 2015 12:22:41 -0800 Subject: [Python-Dev] PEP 485 review (isclose()) In-Reply-To: References: <54F5E49A.8080803@stoneleaf.us> Message-ID: No. On Wed, Mar 4, 2015 at 12:12 PM, Chris Barker wrote: > On Tue, Mar 3, 2015 at 8:43 AM, Ethan Furman wrote: > >> On 03/03/2015 01:17 AM, Victor Stinner wrote: >> > > >> > Maybe it's time to rename the math module to _math and create a >> > math.py module, like _decimal/decimal? math.py should end with "from >> > _math import *". >> >> +1 >> > > What do folks think? If we're going to do this, I'll write isclose() in > python. And I could do the work to split it out, too, I suppose. > > -Chris > > > > > > -- > > Christopher Barker, Ph.D. > Oceanographer > > Emergency Response Division > NOAA/NOS/OR&R (206) 526-6959 voice > 7600 Sand Point Way NE (206) 526-6329 fax > Seattle, WA 98115 (206) 526-6317 main reception > > Chris.Barker at noaa.gov > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/guido%40python.org > > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Wed Mar 4 21:23:24 2015 From: brett at python.org (Brett Cannon) Date: Wed, 04 Mar 2015 20:23:24 +0000 Subject: [Python-Dev] PEP 485 review (isclose()) References: <54F5E49A.8080803@stoneleaf.us> Message-ID: On Wed, Mar 4, 2015 at 3:14 PM Chris Barker wrote: > On Tue, Mar 3, 2015 at 8:43 AM, Ethan Furman wrote: > >> On 03/03/2015 01:17 AM, Victor Stinner wrote: >> > > >> > Maybe it's time to rename the math module to _math and create a >> > math.py module, like _decimal/decimal? math.py should end with "from >> > _math import *". >> >> +1 >> > > What do folks think? If we're going to do this, I'll write isclose() in > python. And I could do the work to split it out, too, I suppose. > My vote -- as always -- is to do it in Python. If someone is sufficiently motivated to re-implement in C then that's great, but I don't think it should be required to be in C. -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Wed Mar 4 21:33:48 2015 From: guido at python.org (Guido van Rossum) Date: Wed, 4 Mar 2015 12:33:48 -0800 Subject: [Python-Dev] PEP 485 review (isclose()) In-Reply-To: References: <54F5E49A.8080803@stoneleaf.us> Message-ID: In this case I disagree. The math module mostly wraps the C math library and the bar should remain high for things to be added to it. Let this be someone's opportunity to learn C (I guess not Chris's :-). On Wed, Mar 4, 2015 at 12:23 PM, Brett Cannon wrote: > > > On Wed, Mar 4, 2015 at 3:14 PM Chris Barker wrote: > >> On Tue, Mar 3, 2015 at 8:43 AM, Ethan Furman wrote: >> >>> On 03/03/2015 01:17 AM, Victor Stinner wrote: >>> >> >> >>> > Maybe it's time to rename the math module to _math and create a >>> > math.py module, like _decimal/decimal? math.py should end with "from >>> > _math import *". >>> >>> +1 >>> >> >> What do folks think? If we're going to do this, I'll write isclose() in >> python. And I could do the work to split it out, too, I suppose. >> > > My vote -- as always -- is to do it in Python. If someone is sufficiently > motivated to re-implement in C then that's great, but I don't think it > should be required to be in C. > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/guido%40python.org > > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.barker at noaa.gov Thu Mar 5 00:26:40 2015 From: chris.barker at noaa.gov (Chris Barker) Date: Wed, 4 Mar 2015 15:26:40 -0800 Subject: [Python-Dev] PEP 485 review (isclose()) In-Reply-To: References: <54F5E49A.8080803@stoneleaf.us> Message-ID: On Wed, Mar 4, 2015 at 12:33 PM, Guido van Rossum wrote: > In this case I disagree. The math module mostly wraps the C math library > and the bar should remain high for things to be added to it. Let this be > someone's opportunity to learn C (I guess not Chris's :-). > As it happens, the first C code I ever wrote, beyond toy examples, was python extensions. I dropped that for Cython a good while ago, but I can remember enough for this simple function ;-) I agree that it's not worth going to python for the math module for just this -- but others were proposing other reasons to do it. Back to look at K&R ;-) -Chris > On Wed, Mar 4, 2015 at 12:23 PM, Brett Cannon wrote: > >> >> >> On Wed, Mar 4, 2015 at 3:14 PM Chris Barker >> wrote: >> >>> On Tue, Mar 3, 2015 at 8:43 AM, Ethan Furman wrote: >>> >>>> On 03/03/2015 01:17 AM, Victor Stinner wrote: >>>> >>> >>> >>>> > Maybe it's time to rename the math module to _math and create a >>>> > math.py module, like _decimal/decimal? math.py should end with "from >>>> > _math import *". >>>> >>>> +1 >>>> >>> >>> What do folks think? If we're going to do this, I'll write isclose() in >>> python. And I could do the work to split it out, too, I suppose. >>> >> >> My vote -- as always -- is to do it in Python. If someone is sufficiently >> motivated to re-implement in C then that's great, but I don't think it >> should be required to be in C. >> >> _______________________________________________ >> Python-Dev mailing list >> Python-Dev at python.org >> https://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe: >> https://mail.python.org/mailman/options/python-dev/guido%40python.org >> >> > > > -- > --Guido van Rossum (python.org/~guido) > -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Thu Mar 5 01:24:39 2015 From: guido at python.org (Guido van Rossum) Date: Wed, 4 Mar 2015 16:24:39 -0800 Subject: [Python-Dev] Any volunteers to implement PEP 479? In-Reply-To: References: Message-ID: Ping? Anyone? http://bugs.python.org/issue22906 On Fri, Feb 20, 2015 at 9:56 PM, Chris Angelico wrote: > On Wed, Jan 7, 2015 at 2:48 PM, Guido van Rossum wrote: > > There's a proof of concept patch in http://bugs.python.org/issue22906, > but > > it doesn't have the __future__ import and probably gets other details > wrong. > > > > Reference: > > PEP 479 -- Change StopIteration handling inside generators > > > > -- > > --Guido van Rossum (python.org/~guido) > > For the benefit of people who haven't been following the tracker > issue: There is now a patch which *does* create the __future__ > directive and so on. It applies cleanly to current tip (I just tested > it again today), and the test suite passes on Debian AMD64. Can other > people please try this, on other platforms, and see how it goes? > > ChrisA > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/guido%40python.org > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From ethan at stoneleaf.us Thu Mar 5 01:42:15 2015 From: ethan at stoneleaf.us (Ethan Furman) Date: Wed, 04 Mar 2015 16:42:15 -0800 Subject: [Python-Dev] Any volunteers to implement PEP 479? In-Reply-To: References: Message-ID: <54F7A667.4050205@stoneleaf.us> On 03/04/2015 04:24 PM, Guido van Rossum wrote: > > Ping? Anyone? http://bugs.python.org/issue22906 Running tests on an ubuntu system... -- ~Ethan~ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: OpenPGP digital signature URL: From shahrukhkhan201994 at gmail.com Thu Mar 5 12:13:39 2015 From: shahrukhkhan201994 at gmail.com (SHAHRUKH KHAN) Date: Thu, 5 Mar 2015 16:43:39 +0530 Subject: [Python-Dev] Python-Dev Digest, Vol 140, Issue 6 In-Reply-To: References: Message-ID: i think it's time to rename math module to math.py module On Thu, Mar 5, 2015 at 4:30 PM, wrote: > Send Python-Dev mailing list submissions to > python-dev at python.org > > To subscribe or unsubscribe via the World Wide Web, visit > https://mail.python.org/mailman/listinfo/python-dev > or, via email, send a message with subject or body 'help' to > python-dev-request at python.org > > You can reach the person managing the list at > python-dev-owner at python.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of Python-Dev digest..." > > > Today's Topics: > > 1. Re: PEP 485 review (isclose()) (Chris Barker) > 2. Re: PEP 485 review (isclose()) (Guido van Rossum) > 3. Re: PEP 485 review (isclose()) (Brett Cannon) > 4. Re: PEP 485 review (isclose()) (Guido van Rossum) > 5. Re: PEP 485 review (isclose()) (Chris Barker) > 6. Re: Any volunteers to implement PEP 479? (Guido van Rossum) > 7. Re: Any volunteers to implement PEP 479? (Ethan Furman) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Wed, 4 Mar 2015 12:12:34 -0800 > From: Chris Barker > To: Ethan Furman > Cc: Python Dev > Subject: Re: [Python-Dev] PEP 485 review (isclose()) > Message-ID: > 8i3jFA at mail.gmail.com> > Content-Type: text/plain; charset="utf-8" > > On Tue, Mar 3, 2015 at 8:43 AM, Ethan Furman wrote: > > > On 03/03/2015 01:17 AM, Victor Stinner wrote: > > > > > > > Maybe it's time to rename the math module to _math and create a > > > math.py module, like _decimal/decimal? math.py should end with "from > > > _math import *". > > > > +1 > > > > What do folks think? If we're going to do this, I'll write isclose() in > python. And I could do the work to split it out, too, I suppose. > > -Chris > > > > > > -- > > Christopher Barker, Ph.D. > Oceanographer > > Emergency Response Division > NOAA/NOS/OR&R (206) 526-6959 voice > 7600 Sand Point Way NE (206) 526-6329 fax > Seattle, WA 98115 (206) 526-6317 main reception > > Chris.Barker at noaa.gov > -------------- next part -------------- > An HTML attachment was scrubbed... > URL: < > http://mail.python.org/pipermail/python-dev/attachments/20150304/a6ef308b/attachment-0001.html > > > > ------------------------------ > > Message: 2 > Date: Wed, 4 Mar 2015 12:22:41 -0800 > From: Guido van Rossum > To: Chris Barker > Cc: Ethan Furman , Python Dev > > Subject: Re: [Python-Dev] PEP 485 review (isclose()) > Message-ID: > < > CAP7+vJJMxAqG2nuXAXnsFn4+m7N9-0aC4XDe6M07F2-fTNFifw at mail.gmail.com> > Content-Type: text/plain; charset="utf-8" > > No. > > On Wed, Mar 4, 2015 at 12:12 PM, Chris Barker > wrote: > > > On Tue, Mar 3, 2015 at 8:43 AM, Ethan Furman wrote: > > > >> On 03/03/2015 01:17 AM, Victor Stinner wrote: > >> > > > > > >> > Maybe it's time to rename the math module to _math and create a > >> > math.py module, like _decimal/decimal? math.py should end with "from > >> > _math import *". > >> > >> +1 > >> > > > > What do folks think? If we're going to do this, I'll write isclose() in > > python. And I could do the work to split it out, too, I suppose. > > > > -Chris > > > > > > > > > > > > -- > > > > Christopher Barker, Ph.D. > > Oceanographer > > > > Emergency Response Division > > NOAA/NOS/OR&R (206) 526-6959 voice > > 7600 Sand Point Way NE (206) 526-6329 fax > > Seattle, WA 98115 (206) 526-6317 main reception > > > > Chris.Barker at noaa.gov > > > > _______________________________________________ > > Python-Dev mailing list > > Python-Dev at python.org > > https://mail.python.org/mailman/listinfo/python-dev > > Unsubscribe: > > https://mail.python.org/mailman/options/python-dev/guido%40python.org > > > > > > > -- > --Guido van Rossum (python.org/~guido) > -------------- next part -------------- > An HTML attachment was scrubbed... > URL: < > http://mail.python.org/pipermail/python-dev/attachments/20150304/e108818c/attachment-0001.html > > > > ------------------------------ > > Message: 3 > Date: Wed, 04 Mar 2015 20:23:24 +0000 > From: Brett Cannon > To: Chris Barker , Ethan Furman > > Cc: Python Dev > Subject: Re: [Python-Dev] PEP 485 review (isclose()) > Message-ID: > 2W4PZo25prx+U7m4CZX83QD6Qw_K2QEZTkn-WJyaQHevFA at mail.gmail.com> > Content-Type: text/plain; charset="utf-8" > > On Wed, Mar 4, 2015 at 3:14 PM Chris Barker wrote: > > > On Tue, Mar 3, 2015 at 8:43 AM, Ethan Furman wrote: > > > >> On 03/03/2015 01:17 AM, Victor Stinner wrote: > >> > > > > > >> > Maybe it's time to rename the math module to _math and create a > >> > math.py module, like _decimal/decimal? math.py should end with "from > >> > _math import *". > >> > >> +1 > >> > > > > What do folks think? If we're going to do this, I'll write isclose() in > > python. And I could do the work to split it out, too, I suppose. > > > > My vote -- as always -- is to do it in Python. If someone is sufficiently > motivated to re-implement in C then that's great, but I don't think it > should be required to be in C. > -------------- next part -------------- > An HTML attachment was scrubbed... > URL: < > http://mail.python.org/pipermail/python-dev/attachments/20150304/d2643082/attachment-0001.html > > > > ------------------------------ > > Message: 4 > Date: Wed, 4 Mar 2015 12:33:48 -0800 > From: Guido van Rossum > To: Brett Cannon > Cc: Chris Barker , Ethan Furman > , Python Dev > Subject: Re: [Python-Dev] PEP 485 review (isclose()) > Message-ID: > jFc6h0E3-1WoUBeQH-3Qg at mail.gmail.com> > Content-Type: text/plain; charset="utf-8" > > In this case I disagree. The math module mostly wraps the C math library > and the bar should remain high for things to be added to it. Let this be > someone's opportunity to learn C (I guess not Chris's :-). > > On Wed, Mar 4, 2015 at 12:23 PM, Brett Cannon wrote: > > > > > > > On Wed, Mar 4, 2015 at 3:14 PM Chris Barker > wrote: > > > >> On Tue, Mar 3, 2015 at 8:43 AM, Ethan Furman > wrote: > >> > >>> On 03/03/2015 01:17 AM, Victor Stinner wrote: > >>> > >> > >> > >>> > Maybe it's time to rename the math module to _math and create a > >>> > math.py module, like _decimal/decimal? math.py should end with "from > >>> > _math import *". > >>> > >>> +1 > >>> > >> > >> What do folks think? If we're going to do this, I'll write isclose() in > >> python. And I could do the work to split it out, too, I suppose. > >> > > > > My vote -- as always -- is to do it in Python. If someone is sufficiently > > motivated to re-implement in C then that's great, but I don't think it > > should be required to be in C. > > > > _______________________________________________ > > Python-Dev mailing list > > Python-Dev at python.org > > https://mail.python.org/mailman/listinfo/python-dev > > Unsubscribe: > > https://mail.python.org/mailman/options/python-dev/guido%40python.org > > > > > > > -- > --Guido van Rossum (python.org/~guido) > -------------- next part -------------- > An HTML attachment was scrubbed... > URL: < > http://mail.python.org/pipermail/python-dev/attachments/20150304/8a086610/attachment-0001.html > > > > ------------------------------ > > Message: 5 > Date: Wed, 4 Mar 2015 15:26:40 -0800 > From: Chris Barker > To: Guido van Rossum > Cc: Brett Cannon , Ethan Furman > , Python Dev > Subject: Re: [Python-Dev] PEP 485 review (isclose()) > Message-ID: > < > CALGmxE+So8SnybKaEBRpZjos8szrBpsCa3N7HcRusweHsgE_2A at mail.gmail.com> > Content-Type: text/plain; charset="utf-8" > > On Wed, Mar 4, 2015 at 12:33 PM, Guido van Rossum > wrote: > > > In this case I disagree. The math module mostly wraps the C math library > > and the bar should remain high for things to be added to it. Let this be > > someone's opportunity to learn C (I guess not Chris's :-). > > > > As it happens, the first C code I ever wrote, beyond toy examples, was > python extensions. I dropped that for Cython a good while ago, but I can > remember enough for this simple function ;-) > > I agree that it's not worth going to python for the math module for just > this -- but others were proposing other reasons to do it. > > Back to look at K&R ;-) > > -Chris > > > > > > On Wed, Mar 4, 2015 at 12:23 PM, Brett Cannon wrote: > > > >> > >> > >> On Wed, Mar 4, 2015 at 3:14 PM Chris Barker > >> wrote: > >> > >>> On Tue, Mar 3, 2015 at 8:43 AM, Ethan Furman > wrote: > >>> > >>>> On 03/03/2015 01:17 AM, Victor Stinner wrote: > >>>> > >>> > >>> > >>>> > Maybe it's time to rename the math module to _math and create a > >>>> > math.py module, like _decimal/decimal? math.py should end with "from > >>>> > _math import *". > >>>> > >>>> +1 > >>>> > >>> > >>> What do folks think? If we're going to do this, I'll write isclose() in > >>> python. And I could do the work to split it out, too, I suppose. > >>> > >> > >> My vote -- as always -- is to do it in Python. If someone is > sufficiently > >> motivated to re-implement in C then that's great, but I don't think it > >> should be required to be in C. > >> > >> _______________________________________________ > >> Python-Dev mailing list > >> Python-Dev at python.org > >> https://mail.python.org/mailman/listinfo/python-dev > >> Unsubscribe: > >> https://mail.python.org/mailman/options/python-dev/guido%40python.org > >> > >> > > > > > > -- > > --Guido van Rossum (python.org/~guido) > > > > > > -- > > Christopher Barker, Ph.D. > Oceanographer > > Emergency Response Division > NOAA/NOS/OR&R (206) 526-6959 voice > 7600 Sand Point Way NE (206) 526-6329 fax > Seattle, WA 98115 (206) 526-6317 main reception > > Chris.Barker at noaa.gov > -------------- next part -------------- > An HTML attachment was scrubbed... > URL: < > http://mail.python.org/pipermail/python-dev/attachments/20150304/10af4471/attachment-0001.html > > > > ------------------------------ > > Message: 6 > Date: Wed, 4 Mar 2015 16:24:39 -0800 > From: Guido van Rossum > To: Chris Angelico > Cc: Python-Dev > Subject: Re: [Python-Dev] Any volunteers to implement PEP 479? > Message-ID: > ncUXA at mail.gmail.com> > Content-Type: text/plain; charset="utf-8" > > Ping? Anyone? http://bugs.python.org/issue22906 > > On Fri, Feb 20, 2015 at 9:56 PM, Chris Angelico wrote: > > > On Wed, Jan 7, 2015 at 2:48 PM, Guido van Rossum > wrote: > > > There's a proof of concept patch in http://bugs.python.org/issue22906, > > but > > > it doesn't have the __future__ import and probably gets other details > > wrong. > > > > > > Reference: > > > PEP 479 -- Change StopIteration handling inside generators > > > > > > -- > > > --Guido van Rossum (python.org/~guido) > > > > For the benefit of people who haven't been following the tracker > > issue: There is now a patch which *does* create the __future__ > > directive and so on. It applies cleanly to current tip (I just tested > > it again today), and the test suite passes on Debian AMD64. Can other > > people please try this, on other platforms, and see how it goes? > > > > ChrisA > > _______________________________________________ > > Python-Dev mailing list > > Python-Dev at python.org > > https://mail.python.org/mailman/listinfo/python-dev > > Unsubscribe: > > https://mail.python.org/mailman/options/python-dev/guido%40python.org > > > > > > -- > --Guido van Rossum (python.org/~guido) > -------------- next part -------------- > An HTML attachment was scrubbed... > URL: < > http://mail.python.org/pipermail/python-dev/attachments/20150304/57d420cc/attachment-0001.html > > > > ------------------------------ > > Message: 7 > Date: Wed, 04 Mar 2015 16:42:15 -0800 > From: Ethan Furman > To: python-dev at python.org > Subject: Re: [Python-Dev] Any volunteers to implement PEP 479? > Message-ID: <54F7A667.4050205 at stoneleaf.us> > Content-Type: text/plain; charset="iso-8859-1" > > On 03/04/2015 04:24 PM, Guido van Rossum wrote: > > > > Ping? Anyone? http://bugs.python.org/issue22906 > > Running tests on an ubuntu system... > > -- > ~Ethan~ > > -------------- next part -------------- > A non-text attachment was scrubbed... > Name: signature.asc > Type: application/pgp-signature > Size: 836 bytes > Desc: OpenPGP digital signature > URL: < > http://mail.python.org/pipermail/python-dev/attachments/20150304/96c32fd5/attachment-0001.sig > > > > ------------------------------ > > Subject: Digest Footer > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > > > ------------------------------ > > End of Python-Dev Digest, Vol 140, Issue 6 > ****************************************** > -------------- next part -------------- An HTML attachment was scrubbed... URL: From storchaka at gmail.com Thu Mar 5 23:36:08 2015 From: storchaka at gmail.com (Serhiy Storchaka) Date: Fri, 06 Mar 2015 00:36:08 +0200 Subject: [Python-Dev] Make review Message-ID: If you have ready patches that wait for review and committing, tell me. Send me no more than 5 links to issues per person (for first time) in private and I'll try to make reviews if I'm acquainted with affected modules or areas. From victor.stinner at gmail.com Fri Mar 6 00:03:48 2015 From: victor.stinner at gmail.com (Victor Stinner) Date: Fri, 6 Mar 2015 00:03:48 +0100 Subject: [Python-Dev] PEP 485 review (isclose()) In-Reply-To: References: Message-ID: 2015-03-03 10:17 GMT+01:00 Victor Stinner : > Maybe it's time to rename the math module to _math and create a > math.py module, like _decimal/decimal? math.py should end with "from > _math import *". Since the idea looks to be well accepted, I proposed a patch to implement it: http://bugs.python.org/issue23595? math.py contains 4 constants (e, pi, inf, nan; nan is optionnal) and 4 functions: degrees(), radians(), factorial(), fsum(). See the issue for details. Victor From victor.stinner at gmail.com Fri Mar 6 13:53:07 2015 From: victor.stinner at gmail.com (Victor Stinner) Date: Fri, 6 Mar 2015 13:53:07 +0100 Subject: [Python-Dev] subprocess, buffered files, pipes and broken pipe errors Message-ID: Hi, => I propose to ignore BrokenPipeError on Popen.__exit__(), what do you think? A recent issue fixed subprocess.Popen.__exit__() to read the exit status of the child process, even if stdin.close() raised a BrokenPipeError: http://bugs.python.org/issue21619 When I saw the issue, I was surprised that Popen.__exit__() doesn't ignore BrokenPipeError. Popen.communicate() is designed to handle corner cases and it ignores BrokenPipeError. Popen.__exit__() only returns when the process exited. The method starts by closing all pipes. Usually the child process notices that stdin was closed and exit (programs used as "pipes" like grep, cat, wc, etc.). The problem is that Popen uses a buffered writer for Popen.stdin with a default buffer size of 8 KB. We may buffer some data and so stdin.close() will call stdin.write(). If the child process exited or closed stdin, the pipe is broken, and the write in the parent process (stdin.close() in Popen.__exit__) will raise BrokenPipeError. I propose to ignore BrokenPipeError in Popen.__exit__, as done in communicate(), for convinience: http://bugs.python.org/issue23570 Serhiy wants to keep BrokenPipeError, he wrote that file.close() should not ignore write errors (read the issue for details). I consider that BrokenPipeError on a pipe is different than a write error on a regular file. EPIPE and SIGPIPE are designed to notify that the pipe is closed and that it's now inefficient to continue to write into this pipe. Ignoring BrokenPipeError in Popen.__exit__() respects this constrain because the method closes stdin and only returns when the process exited. So the caller will not write anything into stdin anymore. What do you think? Victor From storchaka at gmail.com Fri Mar 6 16:10:44 2015 From: storchaka at gmail.com (Serhiy Storchaka) Date: Fri, 06 Mar 2015 17:10:44 +0200 Subject: [Python-Dev] subprocess, buffered files, pipes and broken pipe errors In-Reply-To: References: Message-ID: On 06.03.15 14:53, Victor Stinner wrote: > I propose to ignore BrokenPipeError in Popen.__exit__, as done in > communicate(), for convinience: > http://bugs.python.org/issue23570 > > Serhiy wants to keep BrokenPipeError, he wrote that file.close() > should not ignore write errors (read the issue for details). I rather said about file.__exit__. > I consider that BrokenPipeError on a pipe is different than a write > error on a regular file. > > EPIPE and SIGPIPE are designed to notify that the pipe is closed and > that it's now inefficient to continue to write into this pipe. And into the file like open('/dev/stdout', 'wb'). > Ignoring BrokenPipeError in Popen.__exit__() respects this constrain > because the method closes stdin and only returns when the process > exited. So the caller will not write anything into stdin anymore. And the caller will not write anything into the file after calling file.__exit__. I don't see large difference between open('file', 'wb') and Popen('cat >file', stdin=PIPE), between sys.stdout with redirecting stdout and running external program with Pipe(). From bcannon at gmail.com Fri Mar 6 17:34:45 2015 From: bcannon at gmail.com (Brett Cannon) Date: Fri, 06 Mar 2015 16:34:45 +0000 Subject: [Python-Dev] PEP 488: elimination of PYO files Message-ID: Over on the import-sig I proposed eliminating the concept of .pyo files since they only signify that *some* optimization took place, not *what* optimizations took place. Everyone on the SIG was positive with the idea so I wrote a PEP, got positive feedback from the SIG again, and so now I present to you PEP 488 for discussion. There is no patch yet, but this is not a complicated change and I could get it done at the sprints at PyCon if necessary (I suspect updating the test suite will take the most work). There are currently two open issues, although one is purely a bikeshed topic on formatting of file names so I don't really consider it open for change from what is proposed in the PEP without Guido saying he hates my preference or someone having a really good argument for some alternative. The second open issue on the common case file name is something to reasonably debate and come to consensus on. ======================= PEP: 488 Title: Elimination of PYO files Version: $Revision$ Last-Modified: $Date$ Author: Brett Cannon Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 20-Feb-2015 Post-History: 2015-03-06 Abstract ======== This PEP proposes eliminating the concept of PYO files from Python. To continue the support of the separation of bytecode files based on their optimization level, this PEP proposes extending the PYC file name to include the optimization level in bytecode repository directory (i.e., the ``__pycache__`` directory). Rationale ========= As of today, bytecode files come in two flavours: PYC and PYO. A PYC file is the bytecode file generated and read from when no optimization level is specified at interpreter startup (i.e., ``-O`` is not specified). A PYO file represents the bytecode file that is read/written when **any** optimization level is specified (i.e., when ``-O`` is specified, including ``-OO``). This means that while PYC files clearly delineate the optimization level used when they were generated -- namely no optimizations beyond the peepholer -- the same is not true for PYO files. Put in terms of optimization levels and the file extension: - 0: ``.pyc`` - 1 (``-O``): ``.pyo`` - 2 (``-OO``): ``.pyo`` The reuse of the ``.pyo`` file extension for both level 1 and 2 optimizations means that there is no clear way to tell what optimization level was used to generate the bytecode file. In terms of reading PYO files, this can lead to an interpreter using a mixture of optimization levels with its code if the user was not careful to make sure all PYO files were generated using the same optimization level (typically done by blindly deleting all PYO files and then using the `compileall` module to compile all-new PYO files [1]_). This issue is only compounded when people optimize Python code beyond what the interpreter natively supports, e.g., using the astoptimizer project [2]_. In terms of writing PYO files, the need to delete all PYO files every time one either changes the optimization level they want to use or are unsure of what optimization was used the last time PYO files were generated leads to unnecessary file churn. The change proposed by this PEP also allows for **all** optimization levels to be pre-compiled for bytecode files ahead of time, something that is currently impossible thanks to the reuse of the ``.pyo`` file extension for multiple optimization levels. As for distributing bytecode-only modules, having to distribute both ``.pyc`` and ``.pyo`` files is unnecessary for the common use-case of code obfuscation and smaller file deployments. Proposal ======== To eliminate the ambiguity that PYO files present, this PEP proposes eliminating the concept of PYO files and their accompanying ``.pyo`` file extension. To allow for the optimization level to be unambiguous as well as to avoid having to regenerate optimized bytecode files needlessly in the `__pycache__` directory, the optimization level used to generate a PYC file will be incorporated into the bytecode file name. Currently bytecode file names are created by ``importlib.util.cache_from_source()``, approximately using the following expression defined by PEP 3147 [3]_, [4]_, [5]_:: '{name}.{cache_tag}.pyc'.format(name=module_name, cache_tag=sys.implementation.cache_tag) This PEP proposes to change the expression to:: '{name}.{cache_tag}.opt-{optimization}.pyc'.format( name=module_name, cache_tag=sys.implementation.cache_tag, optimization=str(sys.flags.optimize)) The "opt-" prefix was chosen so as to provide a visual separator from the cache tag. The placement of the optimization level after the cache tag was chosen to preserve lexicographic sort order of bytecode file names based on module name and cache tag which will not vary for a single interpreter. The "opt-" prefix was chosen over "o" so as to be somewhat self-documenting. The "opt-" prefix was chosen over "O" so as to not have any confusion with "0" while being so close to the interpreter version number. A period was chosen over a hyphen as a separator so as to distinguish clearly that the optimization level is not part of the interpreter version as specified by the cache tag. It also lends to the use of the period in the file name to delineate semantically different concepts. For example, the bytecode file name of ``importlib.cpython-35.pyc`` would become ``importlib.cpython-35.opt-0.pyc``. If ``-OO`` had been passed to the interpreter then instead of ``importlib.cpython-35.pyo`` the file name would be ``importlib.cpython-35.opt-2.pyc``. Implementation ============== importlib --------- As ``importlib.util.cache_from_source()`` is the API that exposes bytecode file paths as while as being directly used by importlib, it requires the most critical change. As of Python 3.4, the function's signature is:: importlib.util.cache_from_source(path, debug_override=None) This PEP proposes changing the signature in Python 3.5 to:: importlib.util.cache_from_source(path, debug_override=None, *, optimization=None) The introduced ``optimization`` keyword-only parameter will control what optimization level is specified in the file name. If the argument is ``None`` then the current optimization level of the interpreter will be assumed. Any argument given for ``optimization`` will be passed to ``str()`` and must have ``str.isalnum()`` be true, else ``ValueError`` will be raised (this prevents invalid characters being used in the file name). If the empty string is passed in for ``optimization`` then the addition of the optimization will be suppressed, reverting to the file name format which predates this PEP. It is expected that beyond Python's own 0-2 optimization levels, third-party code will use a hash of optimization names to specify the optimization level, e.g. ``hashlib.sha256(','.join(['dead code elimination', 'constant folding'])).hexdigest()``. While this might lead to long file names, it is assumed that most users never look at the contents of the __pycache__ directory and so this won't be an issue. The ``debug_override`` parameter will be deprecated. As the parameter expects a boolean, the integer value of the boolean will be used as if it had been provided as the argument to ``optimization`` (a ``None`` argument will mean the same as for ``optimization``). A deprecation warning will be raised when ``debug_override`` is given a value other than ``None``, but there are no plans for the complete removal of the parameter as this time (but removal will be no later than Python 4). The various module attributes for importlib.machinery which relate to bytecode file suffixes will be updated [7]_. The ``DEBUG_BYTECODE_SUFFIXES`` and ``OPTIMIZED_BYTECODE_SUFFIXES`` will both be documented as deprecated and set to the same value as ``BYTECODE_SUFFIXES`` (removal of ``DEBUG_BYTECODE_SUFFIXES`` and ``OPTIMIZED_BYTECODE_SUFFIXES`` is not currently planned, but will be not later than Python 4). All various finders and loaders will also be updated as necessary, but updating the previous mentioned parts of importlib should be all that is required. Rest of the standard library ---------------------------- The various functions exposed by the ``py_compile`` and ``compileall`` functions will be updated as necessary to make sure they follow the new bytecode file name semantics [6]_, [1]_. The CLI for the ``compileall`` module will not be directly affected (the ``-b`` flag will be implicitly as it will no longer generate ``.pyo`` files when ``-O`` is specified). Compatibility Considerations ============================ Any code directly manipulating bytecode files from Python 3.2 on will need to consider the impact of this change on their code (prior to Python 3.2 -- including all of Python 2 -- there was no __pycache__ which already necessitates bifurcating bytecode file handling support). If code was setting the ``debug_override`` argument to ``importlib.util.cache_from_source()`` then care will be needed if they want the path to a bytecode file with an optimization level of 2. Otherwise only code **not** using ``importlib.util.cache_from_source()`` will need updating. As for people who distribute bytecode-only modules (i.e., use a bytecode file instead of a source file), they will have to choose which optimization level they want their bytecode files to be since distributing a ``.pyo`` file with a ``.pyc`` file will no longer be of any use. Since people typically only distribute bytecode files for code obfuscation purposes or smaller distribution size then only having to distribute a single ``.pyc`` should actually be beneficial to these use-cases. And since the magic number for bytecode files changed in Python 3.5 to support PEP 465 there is no need to support pre-existing ``.pyo`` files [8]_. Rejected Ideas ============== N/A Open Issues =========== Formatting of the optimization level in the file name ----------------------------------------------------- Using the "opt-" prefix and placing the optimization level between the cache tag and file extension is not critical. All options which have been considered are: * ``importlib.cpython-35.opt-0.pyc`` * ``importlib.cpython-35.opt0.pyc`` * ``importlib.cpython-35.o0.pyc`` * ``importlib.cpython-35.O0.pyc`` * ``importlib.cpython-35.0.pyc`` * ``importlib.cpython-35-O0.pyc`` * ``importlib.O0.cpython-35.pyc`` * ``importlib.o0.cpython-35.pyc`` * ``importlib.0.cpython-35.pyc`` These were initially rejected either because they would change the sort order of bytecode files, possible ambiguity with the cache tag, or were not self-documenting enough. Not specifying the optimization level when it is at 0 ----------------------------------------------------- It has been suggested that for the common case of when the optimizations are at level 0 that the entire part of the file name relating to the optimization level be left out. This would allow for file names of ``.pyc`` files to go unchanged, potentially leading to less backwards-compatibility issues (although Python 3.5 introduces a new magic number for bytecode so all bytecode files will have to be regenerated regardless of the outcome of this PEP). It would also allow a potentially redundant bit of information to be left out of the file name if an implementation of Python did not allow for optimizing bytecode. This would only occur, though, if the interpreter didn't support ``-O`` **and** didn't implement the ast module, else users could implement their own optimizations. Arguments against allow this special case is "explicit is better than implicit" and "special cases aren't special enough to break the rules". There are also currently no Python 3 interpreters that don't support ``-O``, so a potential Python 3 implementation which doesn't allow bytecode optimization is entirely theoretical at the moment. References ========== .. [1] The compileall module (https://docs.python.org/3/library/compileall.html#module-compileall) .. [2] The astoptimizer project (https://pypi.python.org/pypi/astoptimizer) .. [3] ``importlib.util.cache_from_source()`` ( https://docs.python.org/3.5/library/importlib.html#importlib.util.cache_from_source ) .. [4] Implementation of ``importlib.util.cache_from_source()`` from CPython 3.4.3rc1 ( https://hg.python.org/cpython/file/038297948389/Lib/importlib/_bootstrap.py#l437 ) .. [5] PEP 3147, PYC Repository Directories, Warsaw (http://www.python.org/dev/peps/pep-3147) .. [6] The py_compile module (https://docs.python.org/3/library/compileall.html#module-compileall) .. [7] The importlib.machinery module ( https://docs.python.org/3/library/importlib.html#module-importlib.machinery) .. [8] ``importlib.util.MAGIC_NUMBER`` ( https://docs.python.org/3/library/importlib.html#importlib.util.MAGIC_NUMBER ) Copyright ========= This document has been placed in the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End: -------------- next part -------------- An HTML attachment was scrubbed... URL: From status at bugs.python.org Fri Mar 6 18:08:19 2015 From: status at bugs.python.org (Python tracker) Date: Fri, 6 Mar 2015 18:08:19 +0100 (CET) Subject: [Python-Dev] Summary of Python tracker Issues Message-ID: <20150306170819.68DE05663B@psf.upfronthosting.co.za> ACTIVITY SUMMARY (2015-02-27 - 2015-03-06) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 4804 (+16) closed 30555 (+45) total 35359 (+61) Open issues with patches: 2260 Issues opened (47) ================== #22079: Ensure in PyType_Ready() that base class of static type is sta http://bugs.python.org/issue22079 reopened by doko #23538: New Windows installer in 3.5.0a1 breaks compatibility with Win http://bugs.python.org/issue23538 opened by Link Mauve #23539: Content-length not set for HTTP methods expecting body when bo http://bugs.python.org/issue23539 opened by demian.brecht #23540: Proposal for asyncio: SubprocessTransport.detach() to detach a http://bugs.python.org/issue23540 opened by martius #23542: Update PEP 476 for using urllib2.build_opener() http://bugs.python.org/issue23542 opened by Shakeel Mohamed #23543: encoding error trying to save string to file http://bugs.python.org/issue23543 opened by Gravitania #23544: IDLE hangs when selecting Stack View with debug active http://bugs.python.org/issue23544 opened by Andrew.Harrington #23545: Turn on extra warnings on GCC http://bugs.python.org/issue23545 opened by serhiy.storchaka #23546: windows, IDLE and pep 397 http://bugs.python.org/issue23546 opened by Liam.Marsh #23549: heapq docs should be more precise about how to access the smal http://bugs.python.org/issue23549 opened by eli.bendersky #23550: Add to unicodedata a function to query the "Quick_Check" prope http://bugs.python.org/issue23550 opened by Hammerite #23551: IDLE to provide menu options for using PIP http://bugs.python.org/issue23551 opened by rhettinger #23552: Have timeit warn about runs that are not independent of each o http://bugs.python.org/issue23552 opened by rhettinger #23553: Reduce the number of comparisons for range checking. http://bugs.python.org/issue23553 opened by rhettinger #23554: EchoServerClientProtocol class should be called EchoServerProt http://bugs.python.org/issue23554 opened by gc #23555: behavioural change in argparse set_defaults in python 2.7.9 http://bugs.python.org/issue23555 opened by doko #23556: Scope for raise without argument is different in Python 2 and http://bugs.python.org/issue23556 opened by a3nm #23557: Misc/SpecialBuilds.txt contains outdated information about PYM http://bugs.python.org/issue23557 opened by Jakub Klama #23560: Group the docs of similar methods in stdtypes.rst http://bugs.python.org/issue23560 opened by ezio.melotti #23564: Patch fixing sanity check for ordered fd sequence in _posixsub http://bugs.python.org/issue23564 opened by Hisham Muhammad #23565: local_clear walks the list of threads without holding head_loc http://bugs.python.org/issue23565 opened by stutzbach #23566: RFE: faulthandler.register() should support file descriptors http://bugs.python.org/issue23566 opened by haypo #23567: os.stat() tuple access vs named attribute access int vs float http://bugs.python.org/issue23567 opened by gregory.p.smith #23568: unittest.mock.MagicMock doesn't support __rdivmod__ http://bugs.python.org/issue23568 opened by zkrynicki #23570: Change "with subprocess.Popen():" (context manager) to ignore http://bugs.python.org/issue23570 opened by haypo #23571: Raise SystemError if a function returns a result with an excep http://bugs.python.org/issue23571 opened by haypo #23572: functools.singledispatch fails when "not BaseClass" is True http://bugs.python.org/issue23572 opened by Sergio Pascual #23573: Avoid redundant allocations in str.find and like http://bugs.python.org/issue23573 opened by serhiy.storchaka #23574: datetime: support leap seconds http://bugs.python.org/issue23574 opened by haypo #23575: MIPS64 needs ffi's n32.S http://bugs.python.org/issue23575 opened by Simon Hoinkis #23577: Add tests for wsgiref.validate http://bugs.python.org/issue23577 opened by ashkop #23578: struct.pack error messages do not indicate which argument was http://bugs.python.org/issue23578 opened by alynn #23579: Amazon.com links http://bugs.python.org/issue23579 opened by Edrie Ddrie #23581: unittest.mock.MagicMock doesn't support matmul (@) operator http://bugs.python.org/issue23581 opened by zkrynicki #23583: IDLE: printing unicode subclasses broken (again) http://bugs.python.org/issue23583 opened by mjpieters #23584: test_doctest lineendings fails in verbose mode http://bugs.python.org/issue23584 opened by jbeck #23585: patchcheck doesn't depend on all http://bugs.python.org/issue23585 opened by rbcollins #23587: asyncio: use the new traceback.TracebackException class http://bugs.python.org/issue23587 opened by haypo #23588: Errno conflicts in ssl.SSLError http://bugs.python.org/issue23588 opened by Ben.Darnell #23589: Redundant sentence in FAQ http://bugs.python.org/issue23589 opened by fossilet #23591: Add IntFlags http://bugs.python.org/issue23591 opened by serhiy.storchaka #23592: SIGSEGV on interpreter shutdown, with daemon threads running w http://bugs.python.org/issue23592 opened by A. Skrobov #23593: Update Windows and OS X installer OpenSSL to 1.0.2 http://bugs.python.org/issue23593 opened by ned.deily #23595: Split the math module into _math (C) + math (py) http://bugs.python.org/issue23595 opened by haypo #23596: gzip argparse interface http://bugs.python.org/issue23596 opened by Antony.Lee #23597: Allow easy display of local variables in log messages? http://bugs.python.org/issue23597 opened by ncoghlan #23598: No backspace on curses on iTerm (mac) http://bugs.python.org/issue23598 opened by ragreener Most recent 15 issues with no replies (15) ========================================== #23598: No backspace on curses on iTerm (mac) http://bugs.python.org/issue23598 #23591: Add IntFlags http://bugs.python.org/issue23591 #23588: Errno conflicts in ssl.SSLError http://bugs.python.org/issue23588 #23585: patchcheck doesn't depend on all http://bugs.python.org/issue23585 #23584: test_doctest lineendings fails in verbose mode http://bugs.python.org/issue23584 #23581: unittest.mock.MagicMock doesn't support matmul (@) operator http://bugs.python.org/issue23581 #23578: struct.pack error messages do not indicate which argument was http://bugs.python.org/issue23578 #23577: Add tests for wsgiref.validate http://bugs.python.org/issue23577 #23575: MIPS64 needs ffi's n32.S http://bugs.python.org/issue23575 #23573: Avoid redundant allocations in str.find and like http://bugs.python.org/issue23573 #23572: functools.singledispatch fails when "not BaseClass" is True http://bugs.python.org/issue23572 #23568: unittest.mock.MagicMock doesn't support __rdivmod__ http://bugs.python.org/issue23568 #23565: local_clear walks the list of threads without holding head_loc http://bugs.python.org/issue23565 #23560: Group the docs of similar methods in stdtypes.rst http://bugs.python.org/issue23560 #23557: Misc/SpecialBuilds.txt contains outdated information about PYM http://bugs.python.org/issue23557 Most recent 15 issues waiting for review (15) ============================================= #23598: No backspace on curses on iTerm (mac) http://bugs.python.org/issue23598 #23596: gzip argparse interface http://bugs.python.org/issue23596 #23595: Split the math module into _math (C) + math (py) http://bugs.python.org/issue23595 #23593: Update Windows and OS X installer OpenSSL to 1.0.2 http://bugs.python.org/issue23593 #23591: Add IntFlags http://bugs.python.org/issue23591 #23589: Redundant sentence in FAQ http://bugs.python.org/issue23589 #23585: patchcheck doesn't depend on all http://bugs.python.org/issue23585 #23584: test_doctest lineendings fails in verbose mode http://bugs.python.org/issue23584 #23583: IDLE: printing unicode subclasses broken (again) http://bugs.python.org/issue23583 #23577: Add tests for wsgiref.validate http://bugs.python.org/issue23577 #23575: MIPS64 needs ffi's n32.S http://bugs.python.org/issue23575 #23574: datetime: support leap seconds http://bugs.python.org/issue23574 #23573: Avoid redundant allocations in str.find and like http://bugs.python.org/issue23573 #23571: Raise SystemError if a function returns a result with an excep http://bugs.python.org/issue23571 #23570: Change "with subprocess.Popen():" (context manager) to ignore http://bugs.python.org/issue23570 Top 10 most discussed issues (10) ================================= #23496: Steps for Android Native Build of Python 3.4.2 http://bugs.python.org/issue23496 26 msgs #23539: Content-length not set for HTTP methods expecting body when bo http://bugs.python.org/issue23539 26 msgs #23595: Split the math module into _math (C) + math (py) http://bugs.python.org/issue23595 14 msgs #17911: traceback: add a new thin class storing a traceback without st http://bugs.python.org/issue17911 13 msgs #23553: Reduce the number of comparisons for range checking. http://bugs.python.org/issue23553 12 msgs #23571: Raise SystemError if a function returns a result with an excep http://bugs.python.org/issue23571 12 msgs #22936: traceback module has no way to show locals http://bugs.python.org/issue22936 11 msgs #23512: The list of built-in functions is not alphabetical on https:// http://bugs.python.org/issue23512 10 msgs #23524: Use _set_thread_local_invalid_parameter_handler in posixmodule http://bugs.python.org/issue23524 10 msgs #23505: Urlparse insufficient validation leads to open redirect http://bugs.python.org/issue23505 9 msgs Issues closed (43) ================== #3111: multiprocessing ppc Debian/ ia64 Ubuntu compilation error http://bugs.python.org/issue3111 closed by davin #6461: multiprocessing: freezing apps on Windows http://bugs.python.org/issue6461 closed by davin #11097: MSI: Remove win32com dependency from installer generator http://bugs.python.org/issue11097 closed by steve.dower #13697: python RLock implementation unsafe with signals http://bugs.python.org/issue13697 closed by pitrou #18382: multiprocessing's overlapped PipeConnection issues on Windows http://bugs.python.org/issue18382 closed by steve.dower #19075: Add sorting algorithm visualization to turtledemo http://bugs.python.org/issue19075 closed by ethan.furman #19931: namedtuple docstrings are verbose for no added benefit http://bugs.python.org/issue19931 closed by rhettinger #19980: Improve help('non-topic') response http://bugs.python.org/issue19980 closed by serhiy.storchaka #20147: multiprocessing.Queue.get() raises queue.Empty exception if ev http://bugs.python.org/issue20147 closed by davin #20521: Cleanup for "dis" module documentation http://bugs.python.org/issue20521 closed by berker.peksag #21293: Remove "capsule hack" from object.c? http://bugs.python.org/issue21293 closed by larry #21619: Cleaning up a subprocess with a broken pipe http://bugs.python.org/issue21619 closed by serhiy.storchaka #22791: datetime.utcfromtimestamp() shoud have option for create tz aw http://bugs.python.org/issue22791 closed by belopolsky #22801: collections.Counter, when empty, doesn't raise an error with & http://bugs.python.org/issue22801 closed by rhettinger #22817: re.split fails with lookahead/behind http://bugs.python.org/issue22817 closed by serhiy.storchaka #22840: strpdate('20141110', '%Y%m%d%H%S') returns wrong date http://bugs.python.org/issue22840 closed by belopolsky #23263: Python 3 gives misleading errors when validating unicode ident http://bugs.python.org/issue23263 closed by haypo #23304: Unused Superclass in calendar.py http://bugs.python.org/issue23304 closed by serhiy.storchaka #23362: integer overflow in string translate http://bugs.python.org/issue23362 closed by python-dev #23367: integer overflow in unicodedata.normalize http://bugs.python.org/issue23367 closed by python-dev #23368: integer overflow in _PyUnicode_AsKind http://bugs.python.org/issue23368 closed by serhiy.storchaka #23387: test_urllib2 fails with HTTP Error 502: Bad Gateway http://bugs.python.org/issue23387 closed by berker.peksag #23471: 404 Not Found when downloading Python 3.4.3rc1 Documentation http://bugs.python.org/issue23471 closed by larry #23477: Increase coverage for wsgiref module http://bugs.python.org/issue23477 closed by berker.peksag #23494: adding timedelta to datetime object is not timezone aware http://bugs.python.org/issue23494 closed by belopolsky #23504: Add __all__ into types http://bugs.python.org/issue23504 closed by serhiy.storchaka #23521: OverflowError from timedelta * float in datetime.py http://bugs.python.org/issue23521 closed by belopolsky #23527: test_smtpnet uses incorrect port for STARTTLS http://bugs.python.org/issue23527 closed by berker.peksag #23541: Re module's match() fails to halt on a particular input http://bugs.python.org/issue23541 closed by serhiy.storchaka #23547: Misdirected http://bugs.python.org/issue23547 closed by benjamin.peterson #23548: TypeError in event loop finalizer, new in Python 3.4.3 http://bugs.python.org/issue23548 closed by haypo #23558: Unable to install Python 3.4.3 amd64 on Windows 8.1 http://bugs.python.org/issue23558 closed by steve.dower #23559: [doc] inconsistent range example output http://bugs.python.org/issue23559 closed by georg.brandl #23561: a little typo in dis.rst; need a non-trivial preceding dot http://bugs.python.org/issue23561 closed by python-dev #23562: file.read() followed by file.write() result in incorrect behav http://bugs.python.org/issue23562 closed by r.david.murray #23563: Faster urllib.urlparse utility functions http://bugs.python.org/issue23563 closed by serhiy.storchaka #23569: unittest.mock.MagicMock.__div__ works but __truediv__ doesn' http://bugs.python.org/issue23569 closed by ned.deily #23576: HTTPS reads can block when content length not available and ti http://bugs.python.org/issue23576 closed by pitrou #23580: Amazon.com links http://bugs.python.org/issue23580 closed by ethan.furman #23582: multiprocessing.Queue.get is not getting all the items in the http://bugs.python.org/issue23582 closed by r.david.murray #23586: Add classproperty to the builtin functions http://bugs.python.org/issue23586 closed by rhettinger #23590: Potential leak in PyFloat_AsDouble. Refcount error. http://bugs.python.org/issue23590 closed by python-dev #23594: Wrong variable name in traceback http://bugs.python.org/issue23594 closed by ned.deily From ericsnowcurrently at gmail.com Fri Mar 6 18:10:56 2015 From: ericsnowcurrently at gmail.com (Eric Snow) Date: Fri, 6 Mar 2015 10:10:56 -0700 Subject: [Python-Dev] PEP 488: elimination of PYO files In-Reply-To: References: Message-ID: On Fri, Mar 6, 2015 at 9:34 AM, Brett Cannon wrote: > Not specifying the optimization level when it is at 0 > ----------------------------------------------------- > > It has been suggested that for the common case of when the > optimizations are at level 0 that the entire part of the file name > relating to the optimization level be left out. This would allow for > file names of ``.pyc`` files to go unchanged, potentially leading to > less backwards-compatibility issues (although Python 3.5 introduces a > new magic number for bytecode so all bytecode files will have to be > regenerated regardless of the outcome of this PEP). > > It would also allow a potentially redundant bit of information to be > left out of the file name if an implementation of Python did not > allow for optimizing bytecode. This would only occur, though, if the > interpreter didn't support ``-O`` **and** didn't implement the ast > module, else users could implement their own optimizations. The presence of the "opt-0" part in the filename could imply that there are other supported optimization levels, even when there aren't. So leaving that out when optimizations aren't supported may be a good idea. Perhaps add sys.implementation.supports_optimization or something like that? Then only leave "opt-0" off if the implementation does not support any optimization. --eric From mark at hotpy.org Fri Mar 6 19:02:23 2015 From: mark at hotpy.org (Mark Shannon) Date: Fri, 06 Mar 2015 18:02:23 +0000 Subject: [Python-Dev] PEP 488: elimination of PYO files In-Reply-To: References: Message-ID: <54F9EBAF.1080400@hotpy.org> On 06/03/15 16:34, Brett Cannon wrote: > Over on the import-sig I proposed eliminating the concept of .pyo files > since they only signify that /some/ optimization took place, not > /what/ optimizations took place. Everyone on the SIG was positive with > the idea so I wrote a PEP, got positive feedback from the SIG again, and > so now I present to you PEP 488 for discussion. > [snip] Historically -O and -OO have been the antithesis of optimisation, they change the behaviour of the program with no noticeable effect on performance. If a change is to be made, why not just drop .pyo files and be done with it? Any worthwhile optimisation needs to be done at runtime or involve much more than tweaking bytecode. Cheers, Mark. From brett at python.org Fri Mar 6 19:11:19 2015 From: brett at python.org (Brett Cannon) Date: Fri, 06 Mar 2015 18:11:19 +0000 Subject: [Python-Dev] PEP 488: elimination of PYO files References: <54F9EBAF.1080400@hotpy.org> Message-ID: On Fri, Mar 6, 2015 at 1:03 PM Mark Shannon wrote: > > On 06/03/15 16:34, Brett Cannon wrote: > > Over on the import-sig I proposed eliminating the concept of .pyo files > > since they only signify that /some/ optimization took place, not > > /what/ optimizations took place. Everyone on the SIG was positive with > > the idea so I wrote a PEP, got positive feedback from the SIG again, and > > so now I present to you PEP 488 for discussion. > > > [snip] > > Historically -O and -OO have been the antithesis of optimisation, they > change the behaviour of the program with no noticeable effect on > performance. > If a change is to be made, why not just drop .pyo files and be done with > it? > I disagree with your premise that .pyo files don't have a noticeable effect on performance. If you don't use asserts a lot then there is no effect, but if you use them heavily or have them perform expensive calculations then there is an impact. And the dropping of docstrings does have an impact on memory usage when you use Python at scale. You're also assuming that we will never develop an AST optimizer that will go beyond what the peepholer can do based on raw bytecode, or something that involves a bit of calculation and thus something you wouldn't want to do at startup. > > Any worthwhile optimisation needs to be done at runtime or involve much > more than tweaking bytecode. > I disagree again. If you do something like whole program analysis and want to use that to optimize something, you will surface that through bytecode and not editing the source. So while you are doing "much more than tweaking bytecode" externally to Python, you still have to surface to the interpreter through bytecode. -------------- next part -------------- An HTML attachment was scrubbed... URL: From mistersheik at gmail.com Fri Mar 6 19:27:05 2015 From: mistersheik at gmail.com (Neil Girdhar) Date: Fri, 6 Mar 2015 13:27:05 -0500 Subject: [Python-Dev] PEP 488: elimination of PYO files In-Reply-To: References: <54F9EBAF.1080400@hotpy.org> Message-ID: On Fri, Mar 6, 2015 at 1:11 PM, Brett Cannon wrote: > > > On Fri, Mar 6, 2015 at 1:03 PM Mark Shannon wrote: > >> >> On 06/03/15 16:34, Brett Cannon wrote: >> > Over on the import-sig I proposed eliminating the concept of .pyo files >> > since they only signify that /some/ optimization took place, not >> > /what/ optimizations took place. Everyone on the SIG was positive with >> > the idea so I wrote a PEP, got positive feedback from the SIG again, and >> > so now I present to you PEP 488 for discussion. >> > >> [snip] >> >> Historically -O and -OO have been the antithesis of optimisation, they >> change the behaviour of the program with no noticeable effect on >> performance. >> If a change is to be made, why not just drop .pyo files and be done with >> it? >> > > I disagree with your premise that .pyo files don't have a noticeable > effect on performance. If you don't use asserts a lot then there is no > effect, but if you use them heavily or have them perform expensive > calculations then there is an impact. And the dropping of docstrings does > have an impact on memory usage when you use Python at scale. > > You're also assuming that we will never develop an AST optimizer that will > go beyond what the peepholer can do based on raw bytecode, or something > that involves a bit of calculation and thus something you wouldn't want to > do at startup. > I don't want to speak for him, but you're going to get the best results optimizing ASTs at runtime, which is what I thought he was suggesting. Trying to optimize Python at compile time is setting your sights really low. You have so little information then. > > >> >> Any worthwhile optimisation needs to be done at runtime or involve much >> more than tweaking bytecode. >> > > I disagree again. If you do something like whole program analysis and want > to use that to optimize something, you will surface that through bytecode > and not editing the source. So while you are doing "much more than tweaking > bytecode" externally to Python, you still have to surface to the > interpreter through bytecode. > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/mistersheik%40gmail.com > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Fri Mar 6 19:34:48 2015 From: brett at python.org (Brett Cannon) Date: Fri, 06 Mar 2015 18:34:48 +0000 Subject: [Python-Dev] PEP 488: elimination of PYO files References: <54F9EBAF.1080400@hotpy.org> Message-ID: On Fri, Mar 6, 2015 at 1:27 PM Neil Girdhar wrote: > On Fri, Mar 6, 2015 at 1:11 PM, Brett Cannon wrote: > >> >> >> On Fri, Mar 6, 2015 at 1:03 PM Mark Shannon wrote: >> >>> >>> On 06/03/15 16:34, Brett Cannon wrote: >>> > Over on the import-sig I proposed eliminating the concept of .pyo files >>> > since they only signify that /some/ optimization took place, not >>> > /what/ optimizations took place. Everyone on the SIG was positive with >>> > the idea so I wrote a PEP, got positive feedback from the SIG again, >>> and >>> > so now I present to you PEP 488 for discussion. >>> > >>> [snip] >>> >>> Historically -O and -OO have been the antithesis of optimisation, they >>> change the behaviour of the program with no noticeable effect on >>> performance. >>> If a change is to be made, why not just drop .pyo files and be done with >>> it? >>> >> >> I disagree with your premise that .pyo files don't have a noticeable >> effect on performance. If you don't use asserts a lot then there is no >> effect, but if you use them heavily or have them perform expensive >> calculations then there is an impact. And the dropping of docstrings does >> have an impact on memory usage when you use Python at scale. >> >> You're also assuming that we will never develop an AST optimizer that >> will go beyond what the peepholer can do based on raw bytecode, or >> something that involves a bit of calculation and thus something you >> wouldn't want to do at startup. >> > > I don't want to speak for him, but you're going to get the best results > optimizing ASTs at runtime, which is what I thought he was suggesting. > Trying to optimize Python at compile time is setting your sights really > low. You have so little information then. > OK, I don't want to derail the discussion of the PEP into one over how best to optimize CPython's performance relative to bytecode vs. runtime like PyPy. The point is that we have -O and -OO and people do have uses for those flags. People can also do custom optimizations thanks to the flexibility of loaders. All of this leads to wanting different bytecode files for different optimization levels to make sure you're actually executing your code with the optimizations you expect. If people think that optimizing code and surfacing it in bytecode files is a waste and want to suggest either dropping .pyo files entirely or dropping -O and only having -OO that's fine, but that is not what this PEP is proposing nor a PEP I want to bother writing. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ethan at stoneleaf.us Fri Mar 6 19:53:40 2015 From: ethan at stoneleaf.us (Ethan Furman) Date: Fri, 06 Mar 2015 10:53:40 -0800 Subject: [Python-Dev] PEP 488: elimination of PYO files In-Reply-To: References: Message-ID: <54F9F7B4.9050709@stoneleaf.us> On 03/06/2015 08:34 AM, Brett Cannon wrote: > Over on the import-sig I proposed eliminating the concept of .pyo files since they only signify that /some/ optimization > took place, not /what/ optimizations took place. Everyone on the SIG was positive with the idea so I wrote a PEP, got > positive feedback from the SIG again, and so now I present to you PEP 488 for discussion. +1 overall, comments in-line. > Implementation > ============== > > importlib > --------- > > As ``importlib.util.cache_from_source()`` is the API that exposes > bytecode file paths as while as being directly used by importlib, it > requires the most critical change. Not sure what that sentence is supposed to say -- maybe "as well as" and not "as while as" ? > The ``debug_override`` parameter will be deprecated. As the parameter > expects a boolean, the integer value of the boolean will be used as > if it had been provided as the argument to ``optimization`` (a > ``None`` argument will mean the same as for ``optimization``). A > deprecation warning will be raised when ``debug_override`` is given a > value other than ``None``, but there are no plans for the complete > removal of the parameter as this time (but removal will be no later > than Python 4). "at this time" not "as this time" > Rest of the standard library > ---------------------------- > > The various functions exposed by the ``py_compile`` and > ``compileall`` functions will be updated as necessary to make sure > they follow the new bytecode file name semantics [6]_, [1]_. The CLI > for the ``compileall`` module will not be directly affected (the > ``-b`` flag will be implicitly as it will no longer generate ``.pyo`` > files when ``-O`` is specified). "will be implicit" not "will be implicitly" -- ~Ethan~ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: OpenPGP digital signature URL: From brett at python.org Fri Mar 6 19:59:02 2015 From: brett at python.org (Brett Cannon) Date: Fri, 06 Mar 2015 18:59:02 +0000 Subject: [Python-Dev] PEP 488: elimination of PYO files References: <54F9F7B4.9050709@stoneleaf.us> Message-ID: Thanks! All suggestions applied to my local copy. On Fri, Mar 6, 2015 at 1:55 PM Ethan Furman wrote: > On 03/06/2015 08:34 AM, Brett Cannon wrote: > > Over on the import-sig I proposed eliminating the concept of .pyo files > since they only signify that /some/ optimization > > took place, not /what/ optimizations took place. Everyone on the SIG was > positive with the idea so I wrote a PEP, got > > positive feedback from the SIG again, and so now I present to you PEP > 488 for discussion. > > +1 overall, comments in-line. > > > Implementation > > ============== > > > > importlib > > --------- > > > > As ``importlib.util.cache_from_source()`` is the API that exposes > > bytecode file paths as while as being directly used by importlib, it > > requires the most critical change. > > Not sure what that sentence is supposed to say -- maybe "as well as" and > not "as while as" ? > > > > The ``debug_override`` parameter will be deprecated. As the parameter > > expects a boolean, the integer value of the boolean will be used as > > if it had been provided as the argument to ``optimization`` (a > > ``None`` argument will mean the same as for ``optimization``). A > > deprecation warning will be raised when ``debug_override`` is given a > > value other than ``None``, but there are no plans for the complete > > removal of the parameter as this time (but removal will be no later > > than Python 4). > > "at this time" not "as this time" > > > > Rest of the standard library > > ---------------------------- > > > > The various functions exposed by the ``py_compile`` and > > ``compileall`` functions will be updated as necessary to make sure > > they follow the new bytecode file name semantics [6]_, [1]_. The CLI > > for the ``compileall`` module will not be directly affected (the > > ``-b`` flag will be implicitly as it will no longer generate ``.pyo`` > > files when ``-O`` is specified). > > "will be implicit" not "will be implicitly" > > -- > ~Ethan~ > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ > brett%40python.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: From benjamin at python.org Fri Mar 6 20:09:15 2015 From: benjamin at python.org (Benjamin Peterson) Date: Fri, 06 Mar 2015 14:09:15 -0500 Subject: [Python-Dev] PEP 488: elimination of PYO files In-Reply-To: References: <54F9EBAF.1080400@hotpy.org> Message-ID: <1425668955.3364036.236612829.494092A7@webmail.messagingengine.com> On Fri, Mar 6, 2015, at 13:34, Brett Cannon wrote: > On Fri, Mar 6, 2015 at 1:27 PM Neil Girdhar > wrote: > > > On Fri, Mar 6, 2015 at 1:11 PM, Brett Cannon wrote: > > > >> > >> > >> On Fri, Mar 6, 2015 at 1:03 PM Mark Shannon wrote: > >> > >>> > >>> On 06/03/15 16:34, Brett Cannon wrote: > >>> > Over on the import-sig I proposed eliminating the concept of .pyo files > >>> > since they only signify that /some/ optimization took place, not > >>> > /what/ optimizations took place. Everyone on the SIG was positive with > >>> > the idea so I wrote a PEP, got positive feedback from the SIG again, > >>> and > >>> > so now I present to you PEP 488 for discussion. > >>> > > >>> [snip] > >>> > >>> Historically -O and -OO have been the antithesis of optimisation, they > >>> change the behaviour of the program with no noticeable effect on > >>> performance. > >>> If a change is to be made, why not just drop .pyo files and be done with > >>> it? > >>> > >> > >> I disagree with your premise that .pyo files don't have a noticeable > >> effect on performance. If you don't use asserts a lot then there is no > >> effect, but if you use them heavily or have them perform expensive > >> calculations then there is an impact. And the dropping of docstrings does > >> have an impact on memory usage when you use Python at scale. > >> > >> You're also assuming that we will never develop an AST optimizer that > >> will go beyond what the peepholer can do based on raw bytecode, or > >> something that involves a bit of calculation and thus something you > >> wouldn't want to do at startup. > >> > > > > I don't want to speak for him, but you're going to get the best results > > optimizing ASTs at runtime, which is what I thought he was suggesting. > > Trying to optimize Python at compile time is setting your sights really > > low. You have so little information then. > > > > OK, I don't want to derail the discussion of the PEP into one over how > best > to optimize CPython's performance relative to bytecode vs. runtime like > PyPy. The point is that we have -O and -OO and people do have uses for > those flags. People can also do custom optimizations thanks to the > flexibility of loaders. I think it would be preferable deprecate -O and -OO and replace them with flags like --no-docstrings or --no-asserts. Ideally, "optimization" levels shouldn't change program semantics. From rosuav at gmail.com Fri Mar 6 21:13:05 2015 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 7 Mar 2015 07:13:05 +1100 Subject: [Python-Dev] PEP 488: elimination of PYO files In-Reply-To: <1425668955.3364036.236612829.494092A7@webmail.messagingengine.com> References: <54F9EBAF.1080400@hotpy.org> <1425668955.3364036.236612829.494092A7@webmail.messagingengine.com> Message-ID: On Sat, Mar 7, 2015 at 6:09 AM, Benjamin Peterson wrote: > I think it would be preferable deprecate -O and -OO and replace them > with flags like --no-docstrings or --no-asserts. Ideally, "optimization" > levels shouldn't change program semantics. Plenty of C compilers have optimization levels that can change behaviour (replacing division with mult-by-reciprocal and such), so I don't see any reason for Python to object. The removal of docstrings will be a problem to only a handful of programs (eg [1]) which use them for more than introspection. And the removal of assert shouldn't be a semantic change to a well-written program. [1] https://github.com/Rosuav/LetMeKnow/blob/master/letmeknow.py ChrisA From brett at python.org Fri Mar 6 21:11:33 2015 From: brett at python.org (Brett Cannon) Date: Fri, 06 Mar 2015 20:11:33 +0000 Subject: [Python-Dev] PEP 488: elimination of PYO files In-Reply-To: <1425668955.3364036.236612829.494092A7@webmail.messagingengine.com> References: <54F9EBAF.1080400@hotpy.org> <1425668955.3364036.236612829.494092A7@webmail.messagingengine.com> Message-ID: On Fri, Mar 6, 2015 at 2:09 PM Benjamin Peterson wrote: > > > On Fri, Mar 6, 2015, at 13:34, Brett Cannon wrote: > > On Fri, Mar 6, 2015 at 1:27 PM Neil Girdhar > > wrote: > > > > > On Fri, Mar 6, 2015 at 1:11 PM, Brett Cannon wrote: > > > > > >> > > >> > > >> On Fri, Mar 6, 2015 at 1:03 PM Mark Shannon wrote: > > >> > > >>> > > >>> On 06/03/15 16:34, Brett Cannon wrote: > > >>> > Over on the import-sig I proposed eliminating the concept of .pyo > files > > >>> > since they only signify that /some/ optimization took place, not > > >>> > /what/ optimizations took place. Everyone on the SIG was positive > with > > >>> > the idea so I wrote a PEP, got positive feedback from the SIG > again, > > >>> and > > >>> > so now I present to you PEP 488 for discussion. > > >>> > > > >>> [snip] > > >>> > > >>> Historically -O and -OO have been the antithesis of optimisation, > they > > >>> change the behaviour of the program with no noticeable effect on > > >>> performance. > > >>> If a change is to be made, why not just drop .pyo files and be done > with > > >>> it? > > >>> > > >> > > >> I disagree with your premise that .pyo files don't have a noticeable > > >> effect on performance. If you don't use asserts a lot then there is no > > >> effect, but if you use them heavily or have them perform expensive > > >> calculations then there is an impact. And the dropping of docstrings > does > > >> have an impact on memory usage when you use Python at scale. > > >> > > >> You're also assuming that we will never develop an AST optimizer that > > >> will go beyond what the peepholer can do based on raw bytecode, or > > >> something that involves a bit of calculation and thus something you > > >> wouldn't want to do at startup. > > >> > > > > > > I don't want to speak for him, but you're going to get the best results > > > optimizing ASTs at runtime, which is what I thought he was suggesting. > > > Trying to optimize Python at compile time is setting your sights really > > > low. You have so little information then. > > > > > > > OK, I don't want to derail the discussion of the PEP into one over how > > best > > to optimize CPython's performance relative to bytecode vs. runtime like > > PyPy. The point is that we have -O and -OO and people do have uses for > > those flags. People can also do custom optimizations thanks to the > > flexibility of loaders. > > I think it would be preferable deprecate -O and -OO and replace them > with flags like --no-docstrings or --no-asserts. Ideally, "optimization" > levels shouldn't change program semantics. > OK, but that doesn't influence the PEP's goal of dropping .pyo files. Are you suggesting that the tag be changed to be less specific to optimizations and more free-form? Like `importlib.cpython-35.__no-asserts_no-docstrings__.pyc`? Otherwise stuff like this gets baked into the .pyc file itself instead of the file name, but I don't think we should just drop the ability to switch off asserts and docstrings like Mark seemed to be suggesting. -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Fri Mar 6 21:37:05 2015 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 6 Mar 2015 21:37:05 +0100 Subject: [Python-Dev] PEP 488: elimination of PYO files References: <54F9EBAF.1080400@hotpy.org> Message-ID: <20150306213705.5a9f2c9c@fsol> On Fri, 06 Mar 2015 18:11:19 +0000 Brett Cannon wrote: > And the dropping of docstrings does have an impact on > memory usage when you use Python at scale. What kind of "scale" are you talking about? Do you have any numbers about such impact? > You're also assuming that we will never develop an AST optimizer No, the assumption is that we don't have such an optimizer *right now*. Having command-line options because they might be useful some day is silly. Regards Antoine. From steve at pearwood.info Fri Mar 6 23:34:20 2015 From: steve at pearwood.info (Steven D'Aprano) Date: Sat, 7 Mar 2015 09:34:20 +1100 Subject: [Python-Dev] PEP 488: elimination of PYO files In-Reply-To: <20150306213705.5a9f2c9c@fsol> References: <54F9EBAF.1080400@hotpy.org> <20150306213705.5a9f2c9c@fsol> Message-ID: <20150306223419.GN7655@ando.pearwood.info> On Fri, Mar 06, 2015 at 09:37:05PM +0100, Antoine Pitrou wrote: > On Fri, 06 Mar 2015 18:11:19 +0000 > Brett Cannon wrote: > > And the dropping of docstrings does have an impact on > > memory usage when you use Python at scale. > > What kind of "scale" are you talking about? Do you have any numbers > about such impact? > > > You're also assuming that we will never develop an AST optimizer > > No, the assumption is that we don't have such an optimizer *right now*. > Having command-line options because they might be useful some day is > silly. Quoting the PEP: This issue is only compounded when people optimize Python code beyond what the interpreter natively supports, e.g., using the astoptimizer project [2]_. Brett, I'm a very strong +1 on the PEP. It's well-written and gives a good explanation for why such a thing is needed. The current behaviour of re-using the same .pyo file for two distinct sets of bytecode is out-and-out buggy: [steve at ando ~]$ python3.3 -O -c "import dis; print(dis.__doc__[:32])" Disassembler of Python byte code [steve at ando ~]$ python3.3 -OO -c "import dis; print(dis.__doc__[:32])" Disassembler of Python byte code The second should fail, since doc strings should be removed under -OO optimization, but because the .pyo file already exists it doesn't. Even if CPython drops -O and -OO altogether, this PEP should still be accepted to allow third party optimizers like astoptimizer to interact without getting in each other's way. (And for the record, I'm an equally strong -1 on dropping -O and -OO.) Thank you. -- Steve From solipsis at pitrou.net Fri Mar 6 23:47:08 2015 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 6 Mar 2015 23:47:08 +0100 Subject: [Python-Dev] PEP 488: elimination of PYO files References: <54F9EBAF.1080400@hotpy.org> <20150306213705.5a9f2c9c@fsol> <20150306223419.GN7655@ando.pearwood.info> Message-ID: <20150306234708.704a2fdf@fsol> On Sat, 7 Mar 2015 09:34:20 +1100 Steven D'Aprano wrote: > On Fri, Mar 06, 2015 at 09:37:05PM +0100, Antoine Pitrou wrote: > > On Fri, 06 Mar 2015 18:11:19 +0000 > > Brett Cannon wrote: > > > And the dropping of docstrings does have an impact on > > > memory usage when you use Python at scale. > > > > What kind of "scale" are you talking about? Do you have any numbers > > about such impact? > > > > > You're also assuming that we will never develop an AST optimizer > > > > No, the assumption is that we don't have such an optimizer *right now*. > > Having command-line options because they might be useful some day is > > silly. > > Quoting the PEP: > > This issue is only compounded when people optimize Python > code beyond what the interpreter natively supports, e.g., > using the astoptimizer project [2]_. The astoptimizer project is not part of Python. It's third-party software that has no relationship to .pyo files. Regards Antoine. From benjamin at python.org Sat Mar 7 00:48:03 2015 From: benjamin at python.org (Benjamin Peterson) Date: Fri, 06 Mar 2015 18:48:03 -0500 Subject: [Python-Dev] PEP 488: elimination of PYO files In-Reply-To: References: <54F9EBAF.1080400@hotpy.org> <1425668955.3364036.236612829.494092A7@webmail.messagingengine.com> Message-ID: <1425685683.3874448.236704113.06D01977@webmail.messagingengine.com> On Fri, Mar 6, 2015, at 15:13, Chris Angelico wrote: > On Sat, Mar 7, 2015 at 6:09 AM, Benjamin Peterson > wrote: > > I think it would be preferable deprecate -O and -OO and replace them > > with flags like --no-docstrings or --no-asserts. Ideally, "optimization" > > levels shouldn't change program semantics. > > Plenty of C compilers have optimization levels that can change > behaviour (replacing division with mult-by-reciprocal and such), so I > don't see any reason for Python to object. Yes, but in the C world, you have pass scary flags like -ffast-math. Just -O2 generally won't reach outside of standard semantics. From benjamin at python.org Sat Mar 7 00:49:11 2015 From: benjamin at python.org (Benjamin Peterson) Date: Fri, 06 Mar 2015 18:49:11 -0500 Subject: [Python-Dev] PEP 488: elimination of PYO files In-Reply-To: References: <54F9EBAF.1080400@hotpy.org> <1425668955.3364036.236612829.494092A7@webmail.messagingengine.com> Message-ID: <1425685751.3874542.236704353.11A94396@webmail.messagingengine.com> On Fri, Mar 6, 2015, at 15:11, Brett Cannon wrote: > > OK, but that doesn't influence the PEP's goal of dropping .pyo files. Correct. > > Are you suggesting that the tag be changed to be less specific to > optimizations and more free-form? Like > `importlib.cpython-35.__no-asserts_no-docstrings__.pyc`? Otherwise stuff > like this gets baked into the .pyc file itself instead of the file name, > but I don't think we should just drop the ability to switch off asserts > and > docstrings like Mark seemed to be suggesting. Basically, though the filename strings could perhaps be more compact. From brett at python.org Sat Mar 7 01:01:39 2015 From: brett at python.org (Brett Cannon) Date: Sat, 07 Mar 2015 00:01:39 +0000 Subject: [Python-Dev] PEP 488: elimination of PYO files In-Reply-To: <20150306213705.5a9f2c9c@fsol> References: <54F9EBAF.1080400@hotpy.org> <20150306213705.5a9f2c9c@fsol> Message-ID: On Fri, Mar 6, 2015 at 3:37 PM Antoine Pitrou wrote: > On Fri, 06 Mar 2015 18:11:19 +0000 > Brett Cannon wrote: > > And the dropping of docstrings does have an impact on > > memory usage when you use Python at scale. > > What kind of "scale" are you talking about? Do you have any numbers > about such impact? > I know YouTube at least uses -OO and I don't have numbers to share (numbers I were last shown were years ago and I wouldn't be authorized to share anyway, but I do know they still use -OO). > > > You're also assuming that we will never develop an AST optimizer > > No, the assumption is that we don't have such an optimizer *right now*. > Having command-line options because they might be useful some day is > silly. > I'm not talking about changing any command-line option in the PEP so I don't know what you're referring to. -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Sat Mar 7 01:03:14 2015 From: brett at python.org (Brett Cannon) Date: Sat, 07 Mar 2015 00:03:14 +0000 Subject: [Python-Dev] PEP 488: elimination of PYO files In-Reply-To: <20150306234708.704a2fdf@fsol> References: <54F9EBAF.1080400@hotpy.org> <20150306213705.5a9f2c9c@fsol> <20150306223419.GN7655@ando.pearwood.info> <20150306234708.704a2fdf@fsol> Message-ID: On Fri, Mar 6, 2015 at 5:47 PM Antoine Pitrou wrote: > On Sat, 7 Mar 2015 09:34:20 +1100 > Steven D'Aprano wrote: > > > On Fri, Mar 06, 2015 at 09:37:05PM +0100, Antoine Pitrou wrote: > > > On Fri, 06 Mar 2015 18:11:19 +0000 > > > Brett Cannon wrote: > > > > And the dropping of docstrings does have an impact on > > > > memory usage when you use Python at scale. > > > > > > What kind of "scale" are you talking about? Do you have any numbers > > > about such impact? > > > > > > > You're also assuming that we will never develop an AST optimizer > > > > > > No, the assumption is that we don't have such an optimizer *right now*. > > > Having command-line options because they might be useful some day is > > > silly. > > > > Quoting the PEP: > > > > This issue is only compounded when people optimize Python > > code beyond what the interpreter natively supports, e.g., > > using the astoptimizer project [2]_. > > The astoptimizer project is not part of Python. It's third-party > software that has no relationship to .pyo files. > Directly, no. But the point is that the PEP enables the astoptimizer project to write out .pyc files specifying different optimizations that won't clash with -O or -OO .pyc files. -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Sat Mar 7 01:04:52 2015 From: brett at python.org (Brett Cannon) Date: Sat, 07 Mar 2015 00:04:52 +0000 Subject: [Python-Dev] PEP 488: elimination of PYO files In-Reply-To: <1425685751.3874542.236704353.11A94396@webmail.messagingengine.com> References: <54F9EBAF.1080400@hotpy.org> <1425668955.3364036.236612829.494092A7@webmail.messagingengine.com> <1425685751.3874542.236704353.11A94396@webmail.messagingengine.com> Message-ID: On Fri, Mar 6, 2015 at 6:49 PM Benjamin Peterson wrote: > > > On Fri, Mar 6, 2015, at 15:11, Brett Cannon wrote: > > > > OK, but that doesn't influence the PEP's goal of dropping .pyo files. > > Correct. > > > > > Are you suggesting that the tag be changed to be less specific to > > optimizations and more free-form? Like > > `importlib.cpython-35.__no-asserts_no-docstrings__.pyc`? Otherwise stuff > > like this gets baked into the .pyc file itself instead of the file name, > > but I don't think we should just drop the ability to switch off asserts > > and > > docstrings like Mark seemed to be suggesting. > > Basically, though the filename strings could perhaps be more compact. > That's fine. Do you have a file name format you want to propose then instead of "opt-{}" (which is what I'm assuming your "basically" is referring to)? -------------- next part -------------- An HTML attachment was scrubbed... URL: From ron3200 at gmail.com Sat Mar 7 02:00:20 2015 From: ron3200 at gmail.com (Ron Adam) Date: Fri, 06 Mar 2015 20:00:20 -0500 Subject: [Python-Dev] PEP 488: elimination of PYO files In-Reply-To: References: Message-ID: On 03/06/2015 11:34 AM, Brett Cannon wrote: > There are currently two open issues, although one is purely a bikeshed > topic on formatting of file names so I don't really consider it open for > change from what is proposed in the PEP without Guido saying he hates my > preference or someone having a really good argument for some alternative. > The second open issue on the common case file name is something to > reasonably debate and come to consensus on. > > ======================= > > PEP: 488 > Title: Elimination of PYO files +1 And... Have you considered doing this by having different magic numbers in the .pyc file for standard, -O, and -O0 compiled bytecode files? Python already checks that number and recompiles the files if it's not what it's expected to be. And it wouldn't require any naming conventions or new cache directories. It seems to me it would be much easier to do as well. Cheers, Ron From steve at pearwood.info Sat Mar 7 10:58:59 2015 From: steve at pearwood.info (Steven D'Aprano) Date: Sat, 7 Mar 2015 20:58:59 +1100 Subject: [Python-Dev] PEP 488: elimination of PYO files In-Reply-To: References: Message-ID: <20150307095858.GO7655@ando.pearwood.info> On Fri, Mar 06, 2015 at 08:00:20PM -0500, Ron Adam wrote: > Have you considered doing this by having different magic numbers in the > .pyc file for standard, -O, and -O0 compiled bytecode files? Python > already checks that number and recompiles the files if it's not what it's > expected to be. And it wouldn't require any naming conventions or new > cache directories. It seems to me it would be much easier to do as well. And it would fail to solve the problem. The problem isn't just that the .pyo file can contain the wrong byte-code for the optimization level, that's only part of the problem. Another issue is that you cannot have pre-compiled byte-code for multiple different optimization levels. You can have a "no optimization" byte-code file, the .pyc file, but only one "optimized" byte-code file at the same time. Brett's proposal will allow -O optimized and -OO optimized byte-code files to co-exist, as well as setting up a clear naming convention for future optimizers in either the Python compiler or third-party optimizers. No new cache directories are needed. The __pycache__ directory has been used since Python 3.3 (or was it 3.2? I forget which). -- Steve From ron3200 at gmail.com Sat Mar 7 15:29:11 2015 From: ron3200 at gmail.com (Ron Adam) Date: Sat, 07 Mar 2015 09:29:11 -0500 Subject: [Python-Dev] PEP 488: elimination of PYO files In-Reply-To: <20150307095858.GO7655@ando.pearwood.info> References: <20150307095858.GO7655@ando.pearwood.info> Message-ID: On 03/07/2015 04:58 AM, Steven D'Aprano wrote: > On Fri, Mar 06, 2015 at 08:00:20PM -0500, Ron Adam wrote: > >> >Have you considered doing this by having different magic numbers in the >> >.pyc file for standard, -O, and -O0 compiled bytecode files? Python >> >already checks that number and recompiles the files if it's not what it's >> >expected to be. And it wouldn't require any naming conventions or new >> >cache directories. It seems to me it would be much easier to do as well. > And it would fail to solve the problem. The problem isn't just that the > .pyo file can contain the wrong byte-code for the optimization level, > that's only part of the problem. Another issue is that you cannot have > pre-compiled byte-code for multiple different optimization levels. You > can have a "no optimization" byte-code file, the .pyc file, but only one > "optimized" byte-code file at the same time. > > Brett's proposal will allow -O optimized and -OO optimized byte-code > files to co-exist, as well as setting up a clear naming convention for > future optimizers in either the Python compiler or third-party > optimizers. So all the different versions can be generated ahead of time. I think that is the main difference. My suggestion would cause a recompile of all dependent python files when different optimisation levels are used in different projects. Which may be worse than not generating bytecode files at all. OK A few questions... Can a submodule use an optimazation level that is different from the file that imports it? (Other than the case this is trying to solve.) Is there way to specify that an imported module not use any optimisation level, or to always use a specific optimisation level? Is there a way to run tests with all the different optimisation levels? Cheers, Ron From brett at python.org Sat Mar 7 15:55:11 2015 From: brett at python.org (Brett Cannon) Date: Sat, 07 Mar 2015 14:55:11 +0000 Subject: [Python-Dev] PEP 488: elimination of PYO files In-Reply-To: References: <20150307095858.GO7655@ando.pearwood.info> Message-ID: On Sat, Mar 7, 2015 at 9:29 AM Ron Adam wrote: > > > On 03/07/2015 04:58 AM, Steven D'Aprano wrote: > > On Fri, Mar 06, 2015 at 08:00:20PM -0500, Ron Adam wrote: > > > >> >Have you considered doing this by having different magic numbers in the > >> >.pyc file for standard, -O, and -O0 compiled bytecode files? Python > >> >already checks that number and recompiles the files if it's not what > it's > >> >expected to be. And it wouldn't require any naming conventions or new > >> >cache directories. It seems to me it would be much easier to do as > well. > > And it would fail to solve the problem. The problem isn't just that the > > .pyo file can contain the wrong byte-code for the optimization level, > > that's only part of the problem. Another issue is that you cannot have > > pre-compiled byte-code for multiple different optimization levels. You > > can have a "no optimization" byte-code file, the .pyc file, but only one > > "optimized" byte-code file at the same time. > > > > Brett's proposal will allow -O optimized and -OO optimized byte-code > > files to co-exist, as well as setting up a clear naming convention for > > future optimizers in either the Python compiler or third-party > > optimizers. > > So all the different versions can be generated ahead of time. I think that > is the main difference. > > My suggestion would cause a recompile of all dependent python files when > different optimisation levels are used in different projects. Which may be > worse than not generating bytecode files at all. OK > > > A few questions... > > Can a submodule use an optimazation level that is different from the file > that imports it? (Other than the case this is trying to solve.) > Currently yes, with this PEP no (without purposefully doing it with some custom loader). > > Is there way to specify that an imported module not use any optimisation > level, or to always use a specific optimisation level? > Not without a custom loader. > > Is there a way to run tests with all the different optimisation levels? > You have to remember you can't change the optimization levels of the interpreter once you have started it up. The change in semantics is handled deep in the AST compiler and there is no exposed way to flip-flop the setting once the interpreter starts. So to test the different optimization levels would require either (a) implementing the optimizations are part of some AST optimizer and doing the right thing in terms of reloading the modules, or (b) simply running the tests again by running the interpreter again with different flags (this is when something like tox is useful). -------------- next part -------------- An HTML attachment was scrubbed... URL: From scott+python-dev at scottdial.com Sat Mar 7 18:30:59 2015 From: scott+python-dev at scottdial.com (Scott Dial) Date: Sat, 07 Mar 2015 12:30:59 -0500 Subject: [Python-Dev] PEP 488: elimination of PYO files In-Reply-To: References: Message-ID: <54FB35D3.7070104@scottdial.com> On 2015-03-06 11:34 AM, Brett Cannon wrote: > This PEP proposes eliminating the concept of PYO files from Python. > To continue the support of the separation of bytecode files based on > their optimization level, this PEP proposes extending the PYC file > name to include the optimization level in bytecode repository > directory (i.e., the ``__pycache__`` directory). As a packager, this PEP is a bit silent on it's expectations about what will happen with (for instance) Debian and Fedora packages for Python. My familiarity is with Fedora, and on that platform, we ship .pyc and .pyo files (using -O for the .pyo). Is it your expectation that such platforms will still distribute -O only? Or also -OO? In my world, all of the __pycache__ directories are owned by root. -- Scott Dial scott at scottdial.com From pcmanticore at gmail.com Sat Mar 7 22:46:28 2015 From: pcmanticore at gmail.com (Claudiu Popa) Date: Sat, 7 Mar 2015 23:46:28 +0200 Subject: [Python-Dev] Asking for review for Windows issues 21518 and 22080 Message-ID: Hello, The winreg module has a function for loading a registry key under another registry key, called winreg.LoadKey. Unfortunately, the module doesn't provide a way to unload that key after the user finishes operating with it. There's a patch [1] for exporting the RegUnloadKey [2] API in winreg module as winreg.UnloadKey, similar to how RegLoadKey is exported as winreg.LoadKey. The patch is helped by another one [3], which provides a new module, test.support.windows_helper, for handling various issues on the Windows platform, such as acquiring or releasing a privilege. Unfortunately, it seems there's a dearth of reviewers for this platform. Could someone knowledgeable with Windows be so kind to review these patches? They could make a good addition for Python 3.5. Thank you very much. [1] http://bugs.python.org/issue21518 - Expose RegUnloadKey in winreg [2] https://msdn.microsoft.com/en-us/library/windows/desktop/ms724924%28v=vs.85%29.aspx [3] http://bugs.python.org/issue22080 - Add windows_helper module helper /Claudiu From phqnha at gmail.com Sat Mar 7 21:14:23 2015 From: phqnha at gmail.com (nha pham) Date: Sat, 7 Mar 2015 13:14:23 -0700 Subject: [Python-Dev] Optimize binary insertion sort algorithm in Timsort. Message-ID: This describes an optimization for "binary insertion sort" (BINSORT for short). BINSORT has been implemented in Python, CyThon, and Timsort (the default Array.sort() in JAVA SE 7 and JAVA SE 8) I have read the BINSORT in Timsort http://grepcode.com/file/repository.grepcode.com/java/root/jdk/openjdk/8-b132/java/util/TimSort.java#TimSort.binarySort%28java.lang.Object%5B%5D%2Cint%2Cint%2Cint%2Cjava.util.Comparator%29 and I think that I can make a little more optimization. ================= The old BINSORT: The basic idea is to use binary search on the sorted list to find a final position for a new element X, then insert X to the sorted list. [SORTED_LIST], [X in UNSORTED_LIST] // pick X in UNSORTED_LIST index = binarySeach([SORTED_LIST]) // use binary search to find appropriate location for X in SORTED_LIST [SORTED_LIST].add(index, X) // insert X to index location ================== New BINSORT: [SORTED_LIST], [A] // A is an UNSORTED_LIST j = compare(A[i], A[i+1]) // pick A[i], compare to next element index = binarySeach([SORTED_LIST], A[i]) // use binary search to find // appropriate location for A[i] in SORTED_LIST, and remember the index [SORTED_LIST].add(index, A[i]) // insert A[i] to index location // Now for A[+1], where already know where it should be, compare to A[i] if j >= 0: // A[i] > A[i+1], so A[i+1] will be in the right side of A[i] // We only have to search on a reduced Array: index = binarySearch(SORTED_LIST[index : length(SORTED_LIST)], A[i+1]) else: // we search on the left side of of A[i] index = binarySearch(SORTED_LIST[0 : index], A[i+1]) [SORTED_LIST].add(index, A[i+1]) // insert A[i+1] to index location //repeat the loop ================== Run test. Intuitively, new BINSORT will be better if the Array become bigger, because it reduce the array to search with the cost of only 1 more comparison. I only care about small array, with the length < 100 (we have known that in Timsort, the list is divided to chunks with length 64, then apply BINSORT to them). So I make a big Array, divide them, and apply new BINSORT in each chunk, then compare to OLD BINSORT. The source code is in the bottom of this document. Here is the results: cpuinfo: model name : Intel(R) Core(TM) i3 CPU M 350 @ 2.27GHz stepping : 2 microcode : 0xc cpu MHz : 933.000 cache size : 3072 KB ----- random array: ARRAY_SIZE: 1000000 CHUNK_SIZE: 100 DATA: randint(0, 1000000) OLD BINSORT: 81.45754 new BINSORT: 5.26754 RATIO: (OLD - new) / new = 14.464 --- incremental array: ARRAY_SIZE: 1000000 CHUNK_SIZE: 100 DATA: range(0, 1000000) OLD BINSORT: 81.87927 new BINSORT: 5.41651 RATIO: (OLD - new) / new = 14.11659 --- decremental array: ARRAY_SIZE: 1000000 CHUNK_SIZE: 100 DATA: range(0, 1000000) OLD BINSORT: 81.45723 new BINSORT: 5.09823 RATIO: (OLD - new) / new = 14.97753 ---- all equal array: ARRAY_SIZE: 1000000 CHUNK_SIZE: 100 DATA: 50000 OLD BINSORT: 40.46027 new BINSORT: 5.41221 RATIO: (OLD - new) / new = 6.47573 ==================== What should we do next: - Tuning my test code (I have just graphed it). - Test other cases, and bigger array (my laptop cannot handle array more than 10^6.) - Modifying Timsort in java.util. and test if it is better. ==================== My test code, written by Python. from timeit import Timer setup ="""\ import bisect from random import randint from timeit import Timer SIZE = 1000000 CHUNK = 100 NUM_CHUNK = SIZE/CHUNK data = [] data2 = [] data3 = [] for i in range(0,SIZE): data.append(randint(0,1000000)) #data.append(i) #data = data[::-1] """ sample ="""\ for j in range(0,NUM_CHUNK): low = CHUNK*j high= low + CHUNK data2.append(data[low]) index = low for i in range(low,high): x = data[i] index = bisect.bisect_right(data2[low:], x, low, len(data2) - low-1) data2.insert(index, x) """ new ="""\ for j in range(0,NUM_CHUNK): low = CHUNK*j high= low + CHUNK data3.append(data[low]) index = low for i in range(low,high): x = data[i] if x >= data[i-1]: index = bisect.bisect_right(data3[low:len(data3) - low-1], x, index, len(data3) - low-1) else: index = bisect.bisect_right(data3[low:index], x, low, index) data3.insert(index, x) """ t2 = Timer(stmt = sample, setup=setup) a = t2.timeit(1) print a t3 = Timer(stmt = new, setup=setup) b = t3.timeit(1) print b print (str((a - b)/b)) ==================================== Nha Pham Mar 07 2015 -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Sat Mar 7 23:03:18 2015 From: brett at python.org (Brett Cannon) Date: Sat, 07 Mar 2015 22:03:18 +0000 Subject: [Python-Dev] PEP 488: elimination of PYO files In-Reply-To: <54FB35D3.7070104@scottdial.com> References: <54FB35D3.7070104@scottdial.com> Message-ID: On Sat, Mar 7, 2015 at 12:39 PM Scott Dial wrote: > On 2015-03-06 11:34 AM, Brett Cannon wrote: > > This PEP proposes eliminating the concept of PYO files from Python. > > To continue the support of the separation of bytecode files based on > > their optimization level, this PEP proposes extending the PYC file > > name to include the optimization level in bytecode repository > > directory (i.e., the ``__pycache__`` directory). > > As a packager, this PEP is a bit silent on it's expectations about what > will happen with (for instance) Debian and Fedora packages for Python. > My familiarity is with Fedora, and on that platform, we ship .pyc and > .pyo files (using -O for the .pyo). Is it your expectation that such > platforms will still distribute -O only? Or also -OO? In my world, all > of the __pycache__ directories are owned by root. > I assume they will generate all .pyc files at all levels, but I don't if it's my place to dictate such a thing since bytecode files are an optimization to Python itself and do not influence how people interact with the interpreter like with PEP 394 (The "python" Command on Unix-Like Systems). -------------- next part -------------- An HTML attachment was scrubbed... URL: From victor.stinner at gmail.com Sun Mar 8 03:13:01 2015 From: victor.stinner at gmail.com (Victor Stinner) Date: Sun, 8 Mar 2015 03:13:01 +0100 Subject: [Python-Dev] PEP 471 Final: os.scandir() merged into Python 3.5 Message-ID: Hi, FYI I commited the implementation of os.scandir() written by Ben Hoyt. I hope that it will be part of Python 3.5 alpha 2 (Ben just sent the final patch today). Please test this new feature. You may benchmark here. http://bugs.python.org/issue22524 contains some benchmark tools and benchmark results of older versions of the patch. The implementation was tested on Windows and Linux. I'm now watching for buildbots to see how other platforms like os.scandir(). Bad news: OpenIndiana doesn't support d_type: the dirent structure has no d_type field. I already fixed the implementation to support this case. os.scandir() is still useful on OpenIndiana, because the stat result is cached in a DirEntry, so only one syscall is required, instead of multiple, when multiple DirEntry methods are called (ex: entry.is_dir() and not entry.is_symlink()). Victor From benhoyt at gmail.com Sun Mar 8 03:31:35 2015 From: benhoyt at gmail.com (Ben Hoyt) Date: Sat, 7 Mar 2015 21:31:35 -0500 Subject: [Python-Dev] PEP 471 Final: os.scandir() merged into Python 3.5 In-Reply-To: References: Message-ID: Thanks for committing this, Victor! And fixing the d_type issue on funky platforms. Others: if you want to benchmark this, the simplest way is to use my os.walk() benchmark.py test program here: https://github.com/benhoyt/scandir -- it compares the built-in os.walk() implemented with os.listdir() with a version of walk() implemented with os.scandir(). I see huge gains on Windows (12-50x) and modest gains on my Linux VM (3-5x). Note that the actual CPython version of os.walk() doesn't yet use os.scandir(). I intend to open a separate issue for that shortly (or Victor can). But that part should be fairly straight-forward, as I already have a version available in my GitHub project. -Ben On Sat, Mar 7, 2015 at 9:13 PM, Victor Stinner wrote: > Hi, > > FYI I commited the implementation of os.scandir() written by Ben Hoyt. > I hope that it will be part of Python 3.5 alpha 2 (Ben just sent the > final patch today). > > Please test this new feature. You may benchmark here. > http://bugs.python.org/issue22524 contains some benchmark tools and > benchmark results of older versions of the patch. > > The implementation was tested on Windows and Linux. I'm now watching > for buildbots to see how other platforms like os.scandir(). > > Bad news: OpenIndiana doesn't support d_type: the dirent structure has > no d_type field. I already fixed the implementation to support this > case. os.scandir() is still useful on OpenIndiana, because the stat > result is cached in a DirEntry, so only one syscall is required, > instead of multiple, when multiple DirEntry methods are called (ex: > entry.is_dir() and not entry.is_symlink()). > > Victor > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/benhoyt%40gmail.com > -------------- next part -------------- An HTML attachment was scrubbed... URL: From victor.stinner at gmail.com Sun Mar 8 03:39:03 2015 From: victor.stinner at gmail.com (Victor Stinner) Date: Sun, 8 Mar 2015 03:39:03 +0100 Subject: [Python-Dev] PEP 471 Final: os.scandir() merged into Python 3.5 In-Reply-To: References: Message-ID: 2015-03-08 3:31 GMT+01:00 Ben Hoyt : > Thanks for committing this, Victor! And fixing the d_type issue on funky > platforms. You're welcome. > Note that the actual CPython version of os.walk() doesn't yet use > os.scandir(). I intend to open a separate issue for that shortly (or Victor > can). But that part should be fairly straight-forward, as I already have a > version available in my GitHub project. Yes, I just opened an issue for os.walk(): http://bugs.python.org/issue23605 We need a patch and benchmarks on Linux and Windows for that (including benchmarks on a NFS share for the Linux case). I changed the status of the PEP 471 to Final even if os.walk() was not modified yet. IMO the most important part was os.scandir() since "os.scandir()" is in the title of the PEP 471. Victor From ryan.stuart.85 at gmail.com Sun Mar 8 04:48:44 2015 From: ryan.stuart.85 at gmail.com (Ryan Stuart) Date: Sun, 08 Mar 2015 03:48:44 +0000 Subject: [Python-Dev] PEP 471 Final: os.scandir() merged into Python 3.5 In-Reply-To: References: Message-ID: Hi, On Sun, 8 Mar 2015 at 12:33 Ben Hoyt wrote: > Others: if you want to benchmark this, the simplest way is to use my > os.walk() benchmark.py test program here: > https://github.com/benhoyt/scandir -- it compares the built-in os.walk() > implemented with os.listdir() with a version of walk() implemented with > os.scandir(). I see huge gains on Windows (12-50x) and modest gains on my > Linux VM (3-5x). > I have a MacBook Pro Laptop running OS X 10.10.2. I did the following: - hg update -r 8ef4f75a8018 - patch -p1 < scandir-8.patch - ./configure --with-pydebug && make -j7 I then ran ./python.exe ~/Workspace/python/scandir/benchmark.py and I got: *Creating tree at /Users/rstuart/Workspace/python/scandir/benchtree: depth=4, num_dirs=5, num_files=50* *Using slower ctypes version of scandir* *Comparing against builtin version of os.walk()* *Priming the system's cache...* *Benchmarking walks on /Users/rstuart/Workspace/python/scandir/benchtree, repeat 1/3...* *Benchmarking walks on /Users/rstuart/Workspace/python/scandir/benchtree, repeat 2/3...* *Benchmarking walks on /Users/rstuart/Workspace/python/scandir/benchtree, repeat 3/3...* *os.walk took 0.184s, scandir.walk took 0.158s -- 1.2x as fast* I then did ./python.exe ~/Workspace/python/scandir/benchmark.py -s and got: *Using slower ctypes version of scandir* *Comparing against builtin version of os.walk()* *Priming the system's cache...* *Benchmarking walks on /Users/rstuart/Workspace/python/scandir/benchtree, repeat 1/3...* *Benchmarking walks on /Users/rstuart/Workspace/python/scandir/benchtree, repeat 2/3...* *Benchmarking walks on /Users/rstuart/Workspace/python/scandir/benchtree, repeat 3/3...* *os.walk size 23400, scandir.walk size 23400 -- equal* *os.walk took 0.483s, scandir.walk took 0.463s -- 1.0x as fast* Hope this helps. Cheers Note that the actual CPython version of os.walk() doesn't yet use > os.scandir(). I intend to open a separate issue for that shortly (or Victor > can). But that part should be fairly straight-forward, as I already have a > version available in my GitHub project. > > -Ben > > > On Sat, Mar 7, 2015 at 9:13 PM, Victor Stinner > wrote: > >> Hi, >> >> FYI I commited the implementation of os.scandir() written by Ben Hoyt. >> I hope that it will be part of Python 3.5 alpha 2 (Ben just sent the >> final patch today). >> >> Please test this new feature. You may benchmark here. >> http://bugs.python.org/issue22524 contains some benchmark tools and >> benchmark results of older versions of the patch. >> >> The implementation was tested on Windows and Linux. I'm now watching >> for buildbots to see how other platforms like os.scandir(). >> >> Bad news: OpenIndiana doesn't support d_type: the dirent structure has >> no d_type field. I already fixed the implementation to support this >> case. os.scandir() is still useful on OpenIndiana, because the stat >> result is cached in a DirEntry, so only one syscall is required, >> instead of multiple, when multiple DirEntry methods are called (ex: >> entry.is_dir() and not entry.is_symlink()). >> >> Victor >> _______________________________________________ >> Python-Dev mailing list >> Python-Dev at python.org >> https://mail.python.org/mailman/listinfo/python-dev >> > Unsubscribe: >> https://mail.python.org/mailman/options/python-dev/benhoyt%40gmail.com >> > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ > ryan.stuart.85%40gmail.com > -------------- next part -------------- An HTML attachment was scrubbed... URL: From barry at python.org Sun Mar 8 12:54:26 2015 From: barry at python.org (Barry Warsaw) Date: Sun, 8 Mar 2015 07:54:26 -0400 Subject: [Python-Dev] PEP 488: elimination of PYO files In-Reply-To: <54FB35D3.7070104@scottdial.com> References: <54FB35D3.7070104@scottdial.com> Message-ID: <20150308075426.1019f87d@limelight.wooz.org> On Mar 07, 2015, at 12:30 PM, Scott Dial wrote: >As a packager, this PEP is a bit silent on it's expectations about what >will happen with (for instance) Debian and Fedora packages for Python. >My familiarity is with Fedora, and on that platform, we ship .pyc and >.pyo files (using -O for the .pyo). Is it your expectation that such >platforms will still distribute -O only? Or also -OO? In my world, all >of the __pycache__ directories are owned by root. On Debuntu, we don't create pyo files by default, although IIRC it's an option. The problem has been the collision between -O and -OO, so this PEP may open the door to us generating both optimization levels automatically. It's not something that's very high on my todo list, but if there's enough interest it probably wouldn't be difficult to add. Cheers, -Barry From ncoghlan at gmail.com Sun Mar 8 13:17:16 2015 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 8 Mar 2015 22:17:16 +1000 Subject: [Python-Dev] PEP 488: elimination of PYO files In-Reply-To: References: <54FB35D3.7070104@scottdial.com> Message-ID: For the record here: +1 on the PEP from me (the comments I made on import-sig have already incorporated into this version of the PEP) On 8 March 2015 at 08:03, Brett Cannon wrote: > > > On Sat, Mar 7, 2015 at 12:39 PM Scott Dial > wrote: >> >> On 2015-03-06 11:34 AM, Brett Cannon wrote: >> > This PEP proposes eliminating the concept of PYO files from Python. >> > To continue the support of the separation of bytecode files based on >> > their optimization level, this PEP proposes extending the PYC file >> > name to include the optimization level in bytecode repository >> > directory (i.e., the ``__pycache__`` directory). >> >> As a packager, this PEP is a bit silent on it's expectations about what >> will happen with (for instance) Debian and Fedora packages for Python. >> My familiarity is with Fedora, and on that platform, we ship .pyc and >> .pyo files (using -O for the .pyo). Is it your expectation that such >> platforms will still distribute -O only? Or also -OO? In my world, all >> of the __pycache__ directories are owned by root. > > > I assume they will generate all .pyc files at all levels, but I don't if > it's my place to dictate such a thing since bytecode files are an > optimization to Python itself and do not influence how people interact with > the interpreter like with PEP 394 (The "python" Command on Unix-Like > Systems). = Fedora near term = Nothing significant will change for RPMs with policy compliant (https://fedoraproject.org/wiki/Packaging:Python#Byte_compiling) spec files, even after switching to Python 3.5. The only different will be that the current .pyo files will be replaced with files using the new PEP 488 naming scheme. Folks using custom spec files with their own pattern matches that assume "pyo" may need to adjust them (e.g. to grab everything in __pycache__, as recommended in the packaging guidelines) = Fedora long term = After the switch to Python 3.5 (which, given the release schedule, I currently expect will be in Fedora 24 early next year), the RPM Python build macros *might* be changed to generate -OO files in addition to -O ones, making it easier to run Python applications in -OO mode to reduce memory usage. The PEP doesn't mandate such a change, but it does enable it. This would have the downside of making every Python package in the distro slightly larger (all 15k+ of them), so there'd need to be a clear and compelling pay-off to justify that cost. While folks may be tempted to say "disk space is cheap", VM and container provisioning latency are performance critical in a number of use cases, so making the Fedora Cloud and Atomic Host images even slightly larger isn't likely to win us any friends, so actually making such a change would require compelling benchmarks demonstrating reductions in runtime memory usage that justify the larger image size, as well as confirming that the change doesn't place undue pressure on the DVD ISO image contents. This isn't really the right list for more in-depth discussion of what Fedora *might* do though - any discussion will be entirely theoretical until after Python 3.5 comes out, and even then the appropriate list would be python-devel at lists.fedoraproject.org rather than here. Regards, Nick. P.S. For those that aren't already aware, I became a voting member of Fedora's Environments & Stacks Working Group several months ago and now work closely with Slavek Kabrda (the Fedora Python maintainer, amongst other things) on Python's integration into Fedora and its derivatives. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Sun Mar 8 13:20:28 2015 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 8 Mar 2015 22:20:28 +1000 Subject: [Python-Dev] PEP 471 Final: os.scandir() merged into Python 3.5 In-Reply-To: References: Message-ID: On 8 March 2015 at 12:13, Victor Stinner wrote: > Hi, > > FYI I commited the implementation of os.scandir() written by Ben Hoyt. > I hope that it will be part of Python 3.5 alpha 2 (Ben just sent the > final patch today). Thanks to everyone that worked on getting this PEP through to Final status! Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From p.f.moore at gmail.com Sun Mar 8 13:36:45 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Sun, 8 Mar 2015 12:36:45 +0000 Subject: [Python-Dev] PEP 471 Final: os.scandir() merged into Python 3.5 In-Reply-To: References: Message-ID: On 8 March 2015 at 02:13, Victor Stinner wrote: > FYI I commited the implementation of os.scandir() written by Ben Hoyt. > I hope that it will be part of Python 3.5 alpha 2 (Ben just sent the > final patch today). Yay! Great news. Paul From p.f.moore at gmail.com Sun Mar 8 13:39:57 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Sun, 8 Mar 2015 12:39:57 +0000 Subject: [Python-Dev] Request for Pronouncement: PEP 441 - Improving Python ZIP Application Support In-Reply-To: References: <54EB7DDE.9030405@stoneleaf.us> Message-ID: On 26 February 2015 at 21:48, Paul Moore wrote: > On 26 February 2015 at 21:34, Guido van Rossum wrote: >> Accepted! >> >> Thanks for your patience, Paul, and thanks everyone for their feedback. >> >> I know there are still a few small edits to the PEP, but those don't affect >> my acceptance. Congrats! > > Excellent, thanks to everyone for the helpful comments throughout. I'm still looking for someone who has the time to commit the final patch on http://bugs.python.org/issue23491. Python 3.5a2 is due out today (at least according to the PEP) so I've probably missed that, but it would be nice if it could be in for alpha 3 at least. Thanks, Paul From brett at python.org Sun Mar 8 15:35:12 2015 From: brett at python.org (Brett Cannon) Date: Sun, 08 Mar 2015 14:35:12 +0000 Subject: [Python-Dev] Request for Pronouncement: PEP 441 - Improving Python ZIP Application Support References: <54EB7DDE.9030405@stoneleaf.us> Message-ID: On Sun, Mar 8, 2015, 08:40 Paul Moore wrote: On 26 February 2015 at 21:48, Paul Moore wrote: > On 26 February 2015 at 21:34, Guido van Rossum wrote: >> Accepted! >> >> Thanks for your patience, Paul, and thanks everyone for their feedback. >> >> I know there are still a few small edits to the PEP, but those don't affect >> my acceptance. Congrats! > > Excellent, thanks to everyone for the helpful comments throughout. I'm still looking for someone who has the time to commit the final patch on http://bugs.python.org/issue23491. Python 3.5a2 is due out today (at least according to the PEP) so I've probably missed that, but it would be nice if it could be in for alpha 3 at least. If no one gets to it I can submit on Friday. -brett Thanks, Paul _______________________________________________ Python-Dev mailing list Python-Dev at python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/brett%40python.org -------------- next part -------------- An HTML attachment was scrubbed... URL: From mistersheik at gmail.com Sun Mar 8 21:38:44 2015 From: mistersheik at gmail.com (Neil Girdhar) Date: Sun, 8 Mar 2015 16:38:44 -0400 Subject: [Python-Dev] PEP 448 review In-Reply-To: References: Message-ID: Anyone have time to do a code review? http://bugs.python.org/issue2292 On Mon, Mar 2, 2015 at 4:54 PM, Neil Girdhar wrote: > It's from five days ago. I asked Joshua to take a look at something, but > I guess he is busy. > > Best, > > Neil > > ? > > The latest file there is from Feb 26, while your message that the patch > was ready for review is from today -- so is the > patch from five days ago the most recent? > > -- > ~Ethan~ > > On Mon, Mar 2, 2015 at 3:18 PM, Neil Girdhar > wrote: > >> http://bugs.python.org/issue2292 >> >> On Mon, Mar 2, 2015 at 3:17 PM, Victor Stinner >> wrote: >> >>> Where is the patch? >>> >>> Victor >>> >>> Le lundi 2 mars 2015, Neil Girdhar a ?crit : >>> >>> Hi everyone, >>>> >>>> The patch is ready for review now, and I should have time this week to >>>> make changes and respond to comments. >>>> >>>> Best, >>>> >>>> Neil >>>> >>>> On Wed, Feb 25, 2015 at 2:42 PM, Guido van Rossum >>>> wrote: >>>> >>>>> I'm back, I've re-read the PEP, and I've re-read the long thread with >>>>> "(no subject)". >>>>> >>>>> I think Georg Brandl nailed it: >>>>> >>>>> """ >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> *I like the "sequence and dict flattening" part of the PEP, mostly >>>>> because itis consistent and should be easy to understand, but the >>>>> comprehension syntaxenhancements seem to be bad for readability and >>>>> "comprehending" what the codedoes.The call syntax part is a mixed bag on >>>>> the one hand it is nice to be consistent with the extended possibilities in >>>>> literals (flattening), but on the other hand there would be small but >>>>> annoying inconsistencies anyways (e.g. the duplicate kwarg case above).* >>>>> """ >>>>> >>>>> Greg Ewing followed up explaining that the inconsistency between dict >>>>> flattening and call syntax is inherent in the pre-existing different rules >>>>> for dicts vs. keyword args: {'a':1, 'a':2} results in {'a':2}, while f(a=1, >>>>> a=2) is an error. (This form is a SyntaxError; the dynamic case f(a=1, >>>>> **{'a': 1}) is a TypeError.) >>>>> >>>>> For me, allowing f(*a, *b) and f(**d, **e) and all the other >>>>> combinations for function calls proposed by the PEP is an easy +1 -- it's a >>>>> straightforward extension of the existing pattern, and anybody who knows >>>>> what f(x, *a) does will understand f(x, *a, y, *b). Guessing what f(**d, >>>>> **e) means shouldn't be hard either. Understanding the edge case for >>>>> duplicate keys with f(**d, **e) is a little harder, but the error messages >>>>> are pretty clear, and it is not a new edge case. >>>>> >>>>> The sequence and dict flattening syntax proposals are also clean and >>>>> logical -- we already have *-unpacking on the receiving side, so allowing >>>>> *x in tuple expressions reads pretty naturally (and the similarity with *a >>>>> in argument lists certainly helps). From here, having [a, *x, b, *y] is >>>>> also natural, and then the extension to other displays is natural: {a, *x, >>>>> b, *y} and {a:1, **d, b:2, **e}. This, too, gets a +1 from me. >>>>> >>>>> So that leaves comprehensions. IIRC, during the development of the >>>>> patch we realized that f(*x for x in xs) is sufficiently ambiguous that we >>>>> decided to disallow it -- note that f(x for x in xs) is already somewhat of >>>>> a special case because an argument can only be a "bare" generator >>>>> expression if it is the only argument. The same reasoning doesn't apply (in >>>>> that form) to list, set and dict comprehensions -- while f(x for x in xs) >>>>> is identical in meaning to f((x for x in xs)), [x for x in xs] is NOT the >>>>> same as [(x for x in xs)] (that's a list of one element, and the element is >>>>> a generator expression). >>>>> >>>>> The basic premise of this part of the proposal is that if you have a >>>>> few iterables, the new proposal (without comprehensions) lets you create a >>>>> list or generator expression that iterates over all of them, essentially >>>>> flattening them: >>>>> >>>>> >>> xs = [1, 2, 3] >>>>> >>> ys = ['abc', 'def'] >>>>> >>> zs = [99] >>>>> >>> [*xs, *ys, *zs] >>>>> [1, 2, 3, 'abc', 'def', 99] >>>>> >>> >>>>> >>>>> But now suppose you have a list of iterables: >>>>> >>>>> >>> xss = [[1, 2, 3], ['abc', 'def'], [99]] >>>>> >>> [*xss[0], *xss[1], *xss[2]] >>>>> [1, 2, 3, 'abc', 'def', 99] >>>>> >>> >>>>> >>>>> Wouldn't it be nice if you could write the latter using a >>>>> comprehension? >>>>> >>>>> >>> xss = [[1, 2, 3], ['abc', 'def'], [99]] >>>>> >>> [*xs for xs in xss] >>>>> [1, 2, 3, 'abc', 'def', 99] >>>>> >>> >>>>> >>>>> This is somewhat seductive, and the following is even nicer: the *xs >>>>> position may be an expression, e.g.: >>>>> >>>>> >>> xss = [[1, 2, 3], ['abc', 'def'], [99]] >>>>> >>> [*xs[:2] for xs in xss] >>>>> [1, 2, 'abc', 'def', 99] >>>>> >>> >>>>> >>>>> On the other hand, I had to explore the possibilities here by >>>>> experimenting in the interpreter, and I discovered some odd edge cases >>>>> (e.g. you can parenthesize the starred expression, but that seems a >>>>> syntactic accident). >>>>> >>>>> All in all I am personally +0 on the comprehension part of the PEP, >>>>> and I like that it provides a way to "flatten" a sequence of sequences, but >>>>> I think very few people in the thread have supported this part. Therefore I >>>>> would like to ask Neil to update the PEP and the patch to take out the >>>>> comprehension part, so that the two "easy wins" can make it into Python 3.5 >>>>> (basically, I am accepting two-thirds of the PEP :-). There is some time >>>>> yet until alpha 2. >>>>> >>>>> I would also like code reviewers (Benjamin?) to start reviewing the >>>>> patch , taking into account that >>>>> the comprehension part needs to be removed. >>>>> >>>>> -- >>>>> --Guido van Rossum (python.org/~guido) >>>>> >>>>> >>>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From timothy.c.delaney at gmail.com Sun Mar 8 23:00:31 2015 From: timothy.c.delaney at gmail.com (Tim Delaney) Date: Mon, 9 Mar 2015 09:00:31 +1100 Subject: [Python-Dev] PEP 471 Final: os.scandir() merged into Python 3.5 In-Reply-To: References: Message-ID: On 8 March 2015 at 13:31, Ben Hoyt wrote: > Thanks for committing this, Victor! And fixing the d_type issue on funky > platforms. > > Others: if you want to benchmark this, the simplest way is to use my > os.walk() benchmark.py test program here: > https://github.com/benhoyt/scandir -- it compares the built-in os.walk() > implemented with os.listdir() with a version of walk() implemented with > os.scandir(). I see huge gains on Windows (12-50x) and modest gains on my > Linux VM (3-5x). > > Note that the actual CPython version of os.walk() doesn't yet use > os.scandir(). I intend to open a separate issue for that shortly (or Victor > can). But that part should be fairly straight-forward, as I already have a > version available in my GitHub project. > I think it would be a good idea to report the type of drive/mount along with the results. I imagine that there might be significant differences between solid state drives, hard drives and network mounts. Tim Delaney -------------- next part -------------- An HTML attachment was scrubbed... URL: From phqnha at gmail.com Mon Mar 9 01:17:16 2015 From: phqnha at gmail.com (nha pham) Date: Sun, 8 Mar 2015 17:17:16 -0700 Subject: [Python-Dev] Tunning binary insertion sort algorithm in Timsort. Message-ID: We can optimize the TimSort algorithm by optimizing its binary insertion sort. The current version of binary insertion sort use this idea: Use binary search to find a final position in sorted list for a new element X. Then insert X to that location. I suggest another idea: Use binary search to find a final postion in sorted list for a new element X. Before insert X to that location, compare X with its next element. For the next element, we already know if it is lower or bigger than X, so we can reduce the search area to the left side or on the right side of X in the sorted list. I have applied my idea on java.util. ComparableTimSort.sort() and testing. The execute time is reduced by 2%-6% with array of random integer. Here is detail about algorithm and testing: https://github.com/nhapq/Optimize_binary_insertion_sort Sincerely. phqnha -------------- next part -------------- An HTML attachment was scrubbed... URL: From ethan at stoneleaf.us Mon Mar 9 04:31:30 2015 From: ethan at stoneleaf.us (Ethan Furman) Date: Sun, 08 Mar 2015 20:31:30 -0700 Subject: [Python-Dev] boxing and unboxing data types Message-ID: <54FD1412.3020002@stoneleaf.us> When data is passed from Python to a native library (such as in an O/S call), how does the unboxing of data types occur? For a specific instance, os.open allows an integer whose various bits express desired behavior as `flags` -- if flags is 1, for example, the file is open write-only. If I pass an int-like object to os.open, __int__ is called; if I pass a thin wrapper over int to os.open, __int__ is not called. So the real question: anywhere in Python where an int is expected (for lower-level API work), but not directly received, should __int__ (or __index__) be called? and failure to do so is a bug? Here's my simple test code: class Wrap: def __init__(self, value): self.value = value def __int__(self): print('__int__') return self.value def __index__(self): print('__index__') return self.value class Thin(int): def __int__(self): print('__int__') return super().__int__() def __index__(self): print('__index__') return super().__index__() two = Wrap(2) [0, 1, 2][two] # __index__ # 2 import struct struct.pack('i', two) # __index__ # b'\x02\x00\x00\x00' t = Thin(1) huh = os.open('blah.txt', t) # Traceback (most recent call last): # File "", line 1, in # FileNotFoundError: [Errno 2] No such file or directory: 'blah.txt' -- ~Ethan~ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: OpenPGP digital signature URL: From steve at pearwood.info Mon Mar 9 05:12:44 2015 From: steve at pearwood.info (Steven D'Aprano) Date: Mon, 9 Mar 2015 15:12:44 +1100 Subject: [Python-Dev] boxing and unboxing data types In-Reply-To: <54FD1412.3020002@stoneleaf.us> References: <54FD1412.3020002@stoneleaf.us> Message-ID: <20150309041243.GS7655@ando.pearwood.info> On Sun, Mar 08, 2015 at 08:31:30PM -0700, Ethan Furman wrote: > When data is passed from Python to a native library (such as in an O/S > call), how does the unboxing of data types occur? [...] > So the real question: anywhere in Python where an int is expected (for > lower-level API work), but not directly received, should __int__ (or > __index__) be called? and failure to do so is a bug? I think the answer is in the docs: https://docs.python.org/3/reference/datamodel.html#object.__int__ Immediately below that __index__ is described, with this note: In order to have a coherent integer type class, when __index__() is defined __int__() should also be defined, and both should return the same value. The PEP adding __index__ is also useful: https://www.python.org/dev/peps/pep-0357/ My summary is as follows: __int__ is used as the special method for int(), and it should coerce the object to an integer. This may be lossy e.g. int(2.999) --> 2 or may involve a conversion from a non-numeric type to integer e.g. int("2"). __index__ is used when the object in question actually represents an integer of some kind, e.g. a fixed-with integer. Conversion should be lossless and conceptually may be thought of a way of telling Python "this value actually is an int, even though it doesn't inherit from int" (for some definition of "is an int"). There's no built-in way of calling __index__ that I know of (no equivalent to int(obj)), but slicing at the very least will call it, e.g. seq[a:] will call type(a).__index__. If you define __index__ for your class, you should also define __int__ and have the two return the same value. I would expect that an IntFlags object should inherit from int, and if that is not possible, practical or desirable for some reason, then it should define __index__ and __int__. Failure to call __index__ is not necessarily a bug. I think it is allowed for functions to insist on an actual int, as slicing did for many years, but it is an obvious enhancement to allow such functions to accept arbitrary int-like objects. Does that answer your questions? -- Steve From njs at pobox.com Mon Mar 9 05:17:52 2015 From: njs at pobox.com (Nathaniel Smith) Date: Sun, 8 Mar 2015 21:17:52 -0700 Subject: [Python-Dev] boxing and unboxing data types In-Reply-To: <20150309041243.GS7655@ando.pearwood.info> References: <54FD1412.3020002@stoneleaf.us> <20150309041243.GS7655@ando.pearwood.info> Message-ID: On Mar 8, 2015 9:13 PM, "Steven D'Aprano" wrote: > > There's no built-in way of calling __index__ that I know of (no > equivalent to int(obj)), There's operator.index(obj), at least. > but slicing at the very least will call it, > e.g. seq[a:] will call type(a).__index__. -n -------------- next part -------------- An HTML attachment was scrubbed... URL: From ethan at stoneleaf.us Mon Mar 9 05:33:00 2015 From: ethan at stoneleaf.us (Ethan Furman) Date: Sun, 08 Mar 2015 21:33:00 -0700 Subject: [Python-Dev] boxing and unboxing data types In-Reply-To: <20150309041243.GS7655@ando.pearwood.info> References: <54FD1412.3020002@stoneleaf.us> <20150309041243.GS7655@ando.pearwood.info> Message-ID: <54FD227C.5080807@stoneleaf.us> On 03/08/2015 09:12 PM, Steven D'Aprano wrote: > Does that answer your questions? No, unfortunately. You correctly guessed my question is motivated by the IntFlag discussion. I guess it could boil down to: if IntEnum was not based on 'int', but instead had the __int__ and __index__ methods (plus all the other __xxx__ methods that int has), would it still be a drop-in replacement for actual ints? Even when being used to talk to non-Python libs? -- ~Ethan~ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: OpenPGP digital signature URL: From storchaka at gmail.com Mon Mar 9 07:07:28 2015 From: storchaka at gmail.com (Serhiy Storchaka) Date: Mon, 09 Mar 2015 08:07:28 +0200 Subject: [Python-Dev] boxing and unboxing data types In-Reply-To: <54FD227C.5080807@stoneleaf.us> References: <54FD1412.3020002@stoneleaf.us> <20150309041243.GS7655@ando.pearwood.info> <54FD227C.5080807@stoneleaf.us> Message-ID: On 09.03.15 06:33, Ethan Furman wrote: > I guess it could boil down to: if IntEnum was not based on 'int', but instead had the __int__ and __index__ methods > (plus all the other __xxx__ methods that int has), would it still be a drop-in replacement for actual ints? Even when > being used to talk to non-Python libs? If you don't call isinstance(x, int) (PyLong_Check* in C). Most conversions from Python to C implicitly call __index__ or __int__, but unfortunately not all. >>> float(Thin(42)) 42.0 >>> float(Wrap(42)) Traceback (most recent call last): File "", line 1, in TypeError: float() argument must be a string or a number, not 'Wrap' >>> '%*s' % (Thin(5), 'x') ' x' >>> '%*s' % (Wrap(5), 'x') Traceback (most recent call last): File "", line 1, in TypeError: * wants int >>> OSError(Thin(2), 'No such file or directory') FileNotFoundError(2, 'No such file or directory') >>> OSError(Wrap(2), 'No such file or directory') OSError(<__main__.Wrap object at 0xb6fe81ac>, 'No such file or directory') >>> re.match('(x)', 'x').group(Thin(1)) 'x' >>> re.match('(x)', 'x').group(Wrap(1)) Traceback (most recent call last): File "", line 1, in IndexError: no such group And to be ideal drop-in replacement IntEnum should override such methods as __eq__ and __hash__ (so it could be used as mapping key). If all methods should be overridden to quack as int, why not take an int? From ethan at stoneleaf.us Mon Mar 9 07:12:02 2015 From: ethan at stoneleaf.us (Ethan Furman) Date: Sun, 08 Mar 2015 23:12:02 -0700 Subject: [Python-Dev] boxing and unboxing data types In-Reply-To: References: <54FD1412.3020002@stoneleaf.us> <20150309041243.GS7655@ando.pearwood.info> <54FD227C.5080807@stoneleaf.us> Message-ID: <54FD39B2.5000603@stoneleaf.us> On 03/08/2015 11:07 PM, Serhiy Storchaka wrote: > If you don't call isinstance(x, int) (PyLong_Check* in C). > > Most conversions from Python to C implicitly call __index__ or __int__, but unfortunately not all. [snip examples] Thanks, Serhiy, that's what I was looking for. -- ~Ethan~ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: OpenPGP digital signature URL: From tjreedy at udel.edu Mon Mar 9 07:49:09 2015 From: tjreedy at udel.edu (Terry Reedy) Date: Mon, 09 Mar 2015 02:49:09 -0400 Subject: [Python-Dev] Tunning binary insertion sort algorithm in Timsort. In-Reply-To: References: Message-ID: On 3/8/2015 8:17 PM, nha pham wrote: > We can optimize the TimSort algorithm by optimizing its binary insertion > sort. > > The current version of binary insertion sort use this idea: > > Use binary search to find a final position in sorted list for a new > element X. Then insert X to that location. > > I suggest another idea: > > Use binary search to find a final postion in sorted list for a new > element X. Before insert X to that location, compare X with its next > element. > For the next element, we already know if it is lower or bigger than X, > so we can reduce the search area to the left side or on the right side > of X in the sorted list. > > I have applied my idea on java.util. ComparableTimSort.sort() and > testing. The execute time is reduced by 2%-6% with array of random integer. > > Here is detail about algorithm and testing: > https://github.com/nhapq/Optimize_binary_insertion_sort Apply the idea to timsort in python, and try the different cases discussed in Tim's documentation. If it still looks good, open an issue with the patch and put tim.peters as nosy. -- Terry Jan Reedy From storchaka at gmail.com Mon Mar 9 08:13:57 2015 From: storchaka at gmail.com (Serhiy Storchaka) Date: Mon, 09 Mar 2015 09:13:57 +0200 Subject: [Python-Dev] boxing and unboxing data types In-Reply-To: <54FD39B2.5000603@stoneleaf.us> References: <54FD1412.3020002@stoneleaf.us> <20150309041243.GS7655@ando.pearwood.info> <54FD227C.5080807@stoneleaf.us> <54FD39B2.5000603@stoneleaf.us> Message-ID: On 09.03.15 08:12, Ethan Furman wrote: > On 03/08/2015 11:07 PM, Serhiy Storchaka wrote: > >> If you don't call isinstance(x, int) (PyLong_Check* in C). >> >> Most conversions from Python to C implicitly call __index__ or __int__, but unfortunately not all. > > [snip examples] > > Thanks, Serhiy, that's what I was looking for. May be most if not all of these examples can be considered as bugs and slowly fixed, but we can't control third-party code. From fijall at gmail.com Mon Mar 9 09:19:50 2015 From: fijall at gmail.com (Maciej Fijalkowski) Date: Mon, 9 Mar 2015 10:19:50 +0200 Subject: [Python-Dev] boxing and unboxing data types In-Reply-To: References: <54FD1412.3020002@stoneleaf.us> <20150309041243.GS7655@ando.pearwood.info> <54FD227C.5080807@stoneleaf.us> Message-ID: Not all your examples are good. * float(x) calls __float__ (not __int__) * re.group requires __eq__ (and __hash__) * I'm unsure about OSError * the % thing at the very least works on pypy On Mon, Mar 9, 2015 at 8:07 AM, Serhiy Storchaka wrote: > On 09.03.15 06:33, Ethan Furman wrote: >> >> I guess it could boil down to: if IntEnum was not based on 'int', but >> instead had the __int__ and __index__ methods >> (plus all the other __xxx__ methods that int has), would it still be a >> drop-in replacement for actual ints? Even when >> being used to talk to non-Python libs? > > > If you don't call isinstance(x, int) (PyLong_Check* in C). > > Most conversions from Python to C implicitly call __index__ or __int__, but > unfortunately not all. > >>>> float(Thin(42)) > 42.0 >>>> float(Wrap(42)) > Traceback (most recent call last): > File "", line 1, in > TypeError: float() argument must be a string or a number, not 'Wrap' > >>>> '%*s' % (Thin(5), 'x') > ' x' >>>> '%*s' % (Wrap(5), 'x') > Traceback (most recent call last): > File "", line 1, in > TypeError: * wants int > >>>> OSError(Thin(2), 'No such file or directory') > FileNotFoundError(2, 'No such file or directory') >>>> OSError(Wrap(2), 'No such file or directory') > OSError(<__main__.Wrap object at 0xb6fe81ac>, 'No such file or directory') > >>>> re.match('(x)', 'x').group(Thin(1)) > 'x' >>>> re.match('(x)', 'x').group(Wrap(1)) > Traceback (most recent call last): > File "", line 1, in > IndexError: no such group > > And to be ideal drop-in replacement IntEnum should override such methods as > __eq__ and __hash__ (so it could be used as mapping key). If all methods > should be overridden to quack as int, why not take an int? > > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/fijall%40gmail.com From solipsis at pitrou.net Mon Mar 9 09:45:27 2015 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 9 Mar 2015 09:45:27 +0100 Subject: [Python-Dev] boxing and unboxing data types References: <54FD1412.3020002@stoneleaf.us> <20150309041243.GS7655@ando.pearwood.info> Message-ID: <20150309094527.135aac2d@fsol> On Mon, 9 Mar 2015 15:12:44 +1100 Steven D'Aprano wrote: > > My summary is as follows: > > __int__ is used as the special method for int(), and it should coerce > the object to an integer. This may be lossy e.g. int(2.999) --> 2 or may > involve a conversion from a non-numeric type to integer e.g. int("2"). Your example is misleading. Strings don't have an __int__: >>> s = "3" >>> s.__int__() Traceback (most recent call last): File "", line 1, in AttributeError: 'str' object has no attribute '__int__' Only int-compatible or int-coercible types (e.g. float, Decimal) should have an __int__ method. Regards Antoine. From storchaka at gmail.com Mon Mar 9 10:10:09 2015 From: storchaka at gmail.com (Serhiy Storchaka) Date: Mon, 09 Mar 2015 11:10:09 +0200 Subject: [Python-Dev] boxing and unboxing data types In-Reply-To: References: <54FD1412.3020002@stoneleaf.us> <20150309041243.GS7655@ando.pearwood.info> <54FD227C.5080807@stoneleaf.us> Message-ID: On 09.03.15 10:19, Maciej Fijalkowski wrote: > Not all your examples are good. > > * float(x) calls __float__ (not __int__) > > * re.group requires __eq__ (and __hash__) > > * I'm unsure about OSError > > * the % thing at the very least works on pypy Yes, all these examples are implementation defined and can differ between CPython and PyPy. There is about a dozen of similar examples only in C part of CPython. Most of them have in common is that the behavior of the function depends on the argument type. For example in case of re.group an argument is either integer index or string group name. OSError constructor can produce OSError subtype if first argument is known integer errno. float either convert a number to float or parse a string (or bytes). Python functions can be more lenient (if they allows ducktyping) or more strict (if they explicitly check the type). They rarely call __index__ or __int__. From larry at hastings.org Mon Mar 9 10:31:06 2015 From: larry at hastings.org (Larry Hastings) Date: Mon, 09 Mar 2015 02:31:06 -0700 Subject: [Python-Dev] PEP 471 Final: os.scandir() merged into Python 3.5 In-Reply-To: References: Message-ID: <54FD685A.7030004@hastings.org> On 03/07/2015 06:13 PM, Victor Stinner wrote: > Hi, > > FYI I commited the implementation of os.scandir() written by Ben Hoyt. > I hope that it will be part of Python 3.5 alpha 2 It is. //arry/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From larry at hastings.org Mon Mar 9 10:34:50 2015 From: larry at hastings.org (Larry Hastings) Date: Mon, 09 Mar 2015 02:34:50 -0700 Subject: [Python-Dev] [RELEASED] Python 3.5.0a2 is now available Message-ID: <54FD693A.5010704@hastings.org> On behalf of the Python development community and the Python 3.5 release team, I'm thrilled to announce the availability of Python 3.5.0a2. Python 3.5.0a2 is the second alpha release of Python 3.5, which will be the next major release of Python. Python 3.5 is still under heavy development, and is far from complete. This is a preview release, and its use is not recommended for production settings. Two important notes for Windows users about Python 3.5.0a2: * If you have previously installed Python 3.5.0a1, you must manually uninstall it before installing Python 3.5.0a2 (issue23612). * If installing Python 3.5.0a2 as a non-privileged user, you may need to escalate to administrator privileges to install an update to your C runtime libraries. You can find Python 3.5.0a2 here: https://www.python.org/downloads/release/python-350a2/ Happy hacking, //arry/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Mon Mar 9 11:20:54 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Mon, 9 Mar 2015 10:20:54 +0000 Subject: [Python-Dev] [RELEASED] Python 3.5.0a2 is now available In-Reply-To: <54FD693A.5010704@hastings.org> References: <54FD693A.5010704@hastings.org> Message-ID: On 9 March 2015 at 09:34, Larry Hastings wrote: > On behalf of the Python development community and the Python 3.5 release > team, I'm thrilled to announce the availability of Python 3.5.0a2. Python > 3.5.0a2 is the second alpha release of Python 3.5, which will be the next > major release of Python. Python 3.5 is still under heavy development, and > is far from complete. Hmm, I just tried to install the 64-bit "full installer" version, for all users with the default options. This is on a PC that hasn't had 3.5 installed before, and doesn't have Visual Studio 2015 installed. When it got to the step "precompiling standard library" I got an error window pop up saying "python.exe - system error. The program can't start because api-ms-win-crt-runtime-l1-1-0.dll is missing from your computer. Try reinstalling the program to fix this problem." All there was was an "OK" button. Pressing that told me "Setup was successful" but then "py -3.5 -V" gives me nothing (no error, no version, just returns me to the command prompt). Same result if I do "& 'C:\Program Files\Python 3.5\python.exe' -V". Python 3.5.0a2 (64-bit) is showing in my "Add/Remove Programs". This is Windows 7, 64-bit. Paul From p.f.moore at gmail.com Mon Mar 9 11:23:58 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Mon, 9 Mar 2015 10:23:58 +0000 Subject: [Python-Dev] [RELEASED] Python 3.5.0a2 is now available In-Reply-To: References: <54FD693A.5010704@hastings.org> Message-ID: Submitted as http://bugs.python.org/issue23619 On 9 March 2015 at 10:20, Paul Moore wrote: > On 9 March 2015 at 09:34, Larry Hastings wrote: >> On behalf of the Python development community and the Python 3.5 release >> team, I'm thrilled to announce the availability of Python 3.5.0a2. Python >> 3.5.0a2 is the second alpha release of Python 3.5, which will be the next >> major release of Python. Python 3.5 is still under heavy development, and >> is far from complete. > > Hmm, I just tried to install the 64-bit "full installer" version, for > all users with the default options. This is on a PC that hasn't had > 3.5 installed before, and doesn't have Visual Studio 2015 installed. > When it got to the step "precompiling standard library" I got an error > window pop up saying "python.exe - system error. The program can't > start because api-ms-win-crt-runtime-l1-1-0.dll is missing from your > computer. Try reinstalling the program to fix this problem." All there > was was an "OK" button. Pressing that told me "Setup was successful" > but then "py -3.5 -V" gives me nothing (no error, no version, just > returns me to the command prompt). Same result if I do "& 'C:\Program > Files\Python 3.5\python.exe' -V". > > Python 3.5.0a2 (64-bit) is showing in my "Add/Remove Programs". > > This is Windows 7, 64-bit. > > Paul From p.f.moore at gmail.com Mon Mar 9 12:19:18 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Mon, 9 Mar 2015 11:19:18 +0000 Subject: [Python-Dev] Thoughts on running Python 3.5 on Windows (path, pip install --user, etc) Message-ID: I just thought I'd give installing Python 3.5 a go on my PC, now that 3.5a2 has come out. I didn't get very far (see earlier message) but it prompted me to think about how I'd use it, and what habits I'd need to change. I'd suggest that the "what's new in 3.5" document probably needs a section on the new installer that explains this stuff... First of all, I always use "all users" installs, so I have Python in "Program Files" now. As a result, doing "pip install foo" requires elevation. As that's a PITA, I probably need to switch to using "pip install --user". All that's fine, and from there "py -3.5" works fine, as does "py -3.5 -m foo". But even if it is, not every entry point has a corresponding "-m" invocation (pygments' pygmentize command doesn't seem to, for example) But suppose I want to put Python 3.5 on my PATH. The installer has an "add Python to PATH" checkbox, but (if I'm reading the WiX source right, I didn't select that on install) that doesn't add the user directory. So I need to add that to my PATH. Is that right? And of course, that means I need to *know* the user site directory ($env:LOCALAPPDATA\Python\Python35\Scripts), correct? It feels to me like this might be a frustrating step backwards for Windows users who have recently (with the arrival of ensurepip) got to the point where they can just run Python with "Add to path" and then simply do pip install pygments pygmentize --help Maybe the answer is that we simply start recommending that everyone on Windows uses per-user installs. It makes little difference to me (beyond the fact that when I want to look at the source of something in the stdlib, the location of the file is a lot harder to remember than C:\Apps\Python34\Lib\whatever.py) but I doubt it's what most people will expect. I'm completely OK with the idea that we move to "Program Files" as a default location. And I have no real issue with Steve's position that the "Add to Path" option has issues that can't really be solved because of the way Windows constructs the PATH. But I know there have been a lot of people frustrated by the complicated instructions needed to get something like pygmentize working, for whom getting pip in 2.7 and 3.4 was a major improvement. So I think we need *some* documentation helping them deal with what could well seem like a step backwards in Python 3.5... If I knew what the best (recommended) answer was, I'd be happy to write it up. But I'm not sure I do (after all, the target audience is people for whom "Add C:\PythonXY\Scripts to your PATH" was a problem in the pre-3.4 days). Should I raise this as a bug against the 3.5 documentation? If so, should it be a release blocker for the final release? Paul From benhoyt at gmail.com Mon Mar 9 12:58:26 2015 From: benhoyt at gmail.com (Ben Hoyt) Date: Mon, 9 Mar 2015 07:58:26 -0400 Subject: [Python-Dev] PEP 471 Final: os.scandir() merged into Python 3.5 In-Reply-To: References: Message-ID: Hi Ryan, > ./configure --with-pydebug && make -j7 > > I then ran ./python.exe ~/Workspace/python/scandir/benchmark.py and I got: > > Creating tree at /Users/rstuart/Workspace/python/scandir/benchtree: depth=4, num_dirs=5, num_files=50 > Using slower ctypes version of scandir > Comparing against builtin version of os.walk() > Priming the system's cache... > Benchmarking walks on /Users/rstuart/Workspace/python/scandir/benchtree, repeat 1/3... > Benchmarking walks on /Users/rstuart/Workspace/python/scandir/benchtree, repeat 2/3... > Benchmarking walks on /Users/rstuart/Workspace/python/scandir/benchtree, repeat 3/3... > os.walk took 0.184s, scandir.walk took 0.158s -- 1.2x as fast Note that this benchmark is invalid for a couple of reasons. First, you're compiling Python in debug mode (--with-pydebug), which produces significantly slower code in my tests -- for example, on Windows benchmark.py is about twice as slow when Python is compiled in debug mode. Second, as the output above shows, benchmark.py is "Using slower ctypes version of scandir" and not a C version at all. If os.scandir() is available, benchmark.py should use that, so there's something wrong here -- maybe the patch didn't apply correctly or maybe you're testing with a different version of Python than the one you built? In any case, the easiest way to test it now is to download Python 3.5 alpha 2 which just came out: https://www.python.org/downloads/release/python-350a2/ I just tried this on my Mac Mini (i5 2.3GHz, 2 GB RAM, HFS+ on rotational drive) and got the following results: Using Python 3.5's builtin os.scandir() Comparing against builtin version of os.walk() Priming the system's cache... Benchmarking walks on benchtree, repeat 1/3... Benchmarking walks on benchtree, repeat 2/3... Benchmarking walks on benchtree, repeat 3/3... os.walk took 0.074s, scandir.walk took 0.016s -- 4.7x as fast > I then did ./python.exe ~/Workspace/python/scandir/benchmark.py -s and got: Also note that "benchmark.py -s" tests the system os.walk() against a get_tree_size() function using scandir's DirEntry.stat().st_size, which provides huge gains on Windows (because stat().st_size doesn't require on OS call) but only modest gains on POSIX systems, which still require an OS stat call to get the size (though not the file type, so at least it's only one stat call). I get "2.2x as fast" on my Mac for "benchmark.py -s". -Ben From benhoyt at gmail.com Mon Mar 9 13:29:24 2015 From: benhoyt at gmail.com (Ben Hoyt) Date: Mon, 9 Mar 2015 08:29:24 -0400 Subject: [Python-Dev] [RELEASED] Python 3.5.0a2 is now available In-Reply-To: References: <54FD693A.5010704@hastings.org> Message-ID: I'm seeing the same issue (though I also get the missing-DLL error dialog when I run python.exe). -Ben On Mon, Mar 9, 2015 at 6:20 AM, Paul Moore wrote: > On 9 March 2015 at 09:34, Larry Hastings wrote: > > On behalf of the Python development community and the Python 3.5 release > > team, I'm thrilled to announce the availability of Python 3.5.0a2. > Python > > 3.5.0a2 is the second alpha release of Python 3.5, which will be the next > > major release of Python. Python 3.5 is still under heavy development, > and > > is far from complete. > > Hmm, I just tried to install the 64-bit "full installer" version, for > all users with the default options. This is on a PC that hasn't had > 3.5 installed before, and doesn't have Visual Studio 2015 installed. > When it got to the step "precompiling standard library" I got an error > window pop up saying "python.exe - system error. The program can't > start because api-ms-win-crt-runtime-l1-1-0.dll is missing from your > computer. Try reinstalling the program to fix this problem." All there > was was an "OK" button. Pressing that told me "Setup was successful" > but then "py -3.5 -V" gives me nothing (no error, no version, just > returns me to the command prompt). Same result if I do "& 'C:\Program > Files\Python 3.5\python.exe' -V". > > Python 3.5.0a2 (64-bit) is showing in my "Add/Remove Programs". > > This is Windows 7, 64-bit. > > Paul > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/benhoyt%40gmail.com > -------------- next part -------------- An HTML attachment was scrubbed... URL: From Steve.Dower at microsoft.com Mon Mar 9 14:14:01 2015 From: Steve.Dower at microsoft.com (Steve Dower) Date: Mon, 9 Mar 2015 13:14:01 +0000 Subject: [Python-Dev] [RELEASED] Python 3.5.0a2 is now available In-Reply-To: References: <54FD693A.5010704@hastings.org> , Message-ID: Thanks for finding this. I'm following up on the issue of anyone else is having the same issue. As an aside, it'd be great to hear if it's worked for anyone at all :) Cheers, Steve Top-posted from my Windows Phone ________________________________ From: Ben Hoyt Sent: ?3/?9/?2015 5:29 To: Paul Moore Cc: Python Dev Subject: Re: [Python-Dev] [RELEASED] Python 3.5.0a2 is now available I'm seeing the same issue (though I also get the missing-DLL error dialog when I run python.exe). -Ben On Mon, Mar 9, 2015 at 6:20 AM, Paul Moore > wrote: On 9 March 2015 at 09:34, Larry Hastings > wrote: > On behalf of the Python development community and the Python 3.5 release > team, I'm thrilled to announce the availability of Python 3.5.0a2. Python > 3.5.0a2 is the second alpha release of Python 3.5, which will be the next > major release of Python. Python 3.5 is still under heavy development, and > is far from complete. Hmm, I just tried to install the 64-bit "full installer" version, for all users with the default options. This is on a PC that hasn't had 3.5 installed before, and doesn't have Visual Studio 2015 installed. When it got to the step "precompiling standard library" I got an error window pop up saying "python.exe - system error. The program can't start because api-ms-win-crt-runtime-l1-1-0.dll is missing from your computer. Try reinstalling the program to fix this problem." All there was was an "OK" button. Pressing that told me "Setup was successful" but then "py -3.5 -V" gives me nothing (no error, no version, just returns me to the command prompt). Same result if I do "& 'C:\Program Files\Python 3.5\python.exe' -V". Python 3.5.0a2 (64-bit) is showing in my "Add/Remove Programs". This is Windows 7, 64-bit. Paul _______________________________________________ Python-Dev mailing list Python-Dev at python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/benhoyt%40gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Mon Mar 9 14:45:42 2015 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 9 Mar 2015 23:45:42 +1000 Subject: [Python-Dev] Thoughts on running Python 3.5 on Windows (path, pip install --user, etc) In-Reply-To: References: Message-ID: On 9 March 2015 at 21:19, Paul Moore wrote: > Should I raise this as a bug against the 3.5 documentation? If so, > should it be a release blocker for the final release? I'm happy to let the folks that use Windows for development regularly decide on the best answer from a user experience perspective, but I think a release blocker docs issue would be a good way to go for ensuring it gets resolved be the release goes out. Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From rmsr at lab.net Mon Mar 9 06:57:30 2015 From: rmsr at lab.net (Ryan Smith-Roberts) Date: Sun, 8 Mar 2015 22:57:30 -0700 Subject: [Python-Dev] Tunning binary insertion sort algorithm in Timsort. In-Reply-To: References: Message-ID: I suspect that you will find the Python community extremely conservative about any changes to its sorting algorithm, given that it took thirteen years and some really impressive automated verification software to find this bug: http://envisage-project.eu/proving-android-java-and-python-sorting-algorithm-is-broken-and-how-to-fix-it/ On Sun, Mar 8, 2015 at 5:17 PM, nha pham wrote: > We can optimize the TimSort algorithm by optimizing its binary insertion > sort. > > > > The current version of binary insertion sort use this idea: > > Use binary search to find a final position in sorted list for a new > element X. Then insert X to that location. > > > > I suggest another idea: > > Use binary search to find a final postion in sorted list for a new element > X. Before insert X to that location, compare X with its next element. > > For the next element, we already know if it is lower or bigger than X, so > we can reduce the search area to the left side or on the right side of X in > the sorted list. > > > > I have applied my idea on java.util. ComparableTimSort.sort() and testing. > The execute time is reduced by 2%-6% with array of random integer. > > > > Here is detail about algorithm and testing: > https://github.com/nhapq/Optimize_binary_insertion_sort > > > > Sincerely. > > phqnha > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/rmsr%40lab.net > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mistersheik at gmail.com Mon Mar 9 14:52:01 2015 From: mistersheik at gmail.com (Neil Girdhar) Date: Mon, 9 Mar 2015 09:52:01 -0400 Subject: [Python-Dev] boxing and unboxing data types In-Reply-To: References: <54FD1412.3020002@stoneleaf.us> <20150309041243.GS7655@ando.pearwood.info> <54FD227C.5080807@stoneleaf.us> Message-ID: On Mon, Mar 9, 2015 at 2:07 AM, Serhiy Storchaka wrote: > On 09.03.15 06:33, Ethan Furman wrote: > >> I guess it could boil down to: if IntEnum was not based on 'int', but >> instead had the __int__ and __index__ methods >> (plus all the other __xxx__ methods that int has), would it still be a >> drop-in replacement for actual ints? Even when >> being used to talk to non-Python libs? >> > > If you don't call isinstance(x, int) (PyLong_Check* in C). > > Most conversions from Python to C implicitly call __index__ or __int__, > but unfortunately not all. > > >>> float(Thin(42)) > 42.0 > >>> float(Wrap(42)) > Traceback (most recent call last): > File "", line 1, in > TypeError: float() argument must be a string or a number, not 'Wrap' > > >>> '%*s' % (Thin(5), 'x') > ' x' > >>> '%*s' % (Wrap(5), 'x') > Traceback (most recent call last): > File "", line 1, in > TypeError: * wants int > > >>> OSError(Thin(2), 'No such file or directory') > FileNotFoundError(2, 'No such file or directory') > >>> OSError(Wrap(2), 'No such file or directory') > OSError(<__main__.Wrap object at 0xb6fe81ac>, 'No such file or directory') > > >>> re.match('(x)', 'x').group(Thin(1)) > 'x' > >>> re.match('(x)', 'x').group(Wrap(1)) > Traceback (most recent call last): > File "", line 1, in > IndexError: no such group > > And to be ideal drop-in replacement IntEnum should override such methods > as __eq__ and __hash__ (so it could be used as mapping key). If all methods > should be overridden to quack as int, why not take an int? > > You're absolutely right that if *all the methods should be overrriden to quack as int, then you should subclass int (the Liskov substitution principle). But all methods should not be overridden ? mainly the methods you overrode in your patch should be exposed. Here is a list of methods on int that should not be on IntFlags in my opinion (give or take a couple): __abs__, __add__, __delattr__, __divmod__, __float__, __floor__, __floordiv__, __index__, __lshift__, __mod__, __mul__, __pos__, __pow__, __radd__, __rdivmod__, __rfloordiv__, __rlshift__, __rmod__, __rmul__, __round__, __rpow__, __rrshift__, __rshift__, __rsub__, __rtruediv__, __sub__, __truediv__, __trunc__, conjugate, denominator, imag, numerator, real. I don't think __index__ should be exposed either since are you really going to slice a list using IntFlags? Really? > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ > mistersheik%40gmail.com > -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Mon Mar 9 15:14:51 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Mon, 9 Mar 2015 14:14:51 +0000 Subject: [Python-Dev] Thoughts on running Python 3.5 on Windows (path, pip install --user, etc) In-Reply-To: References: Message-ID: On 9 March 2015 at 13:45, Nick Coghlan wrote: > I'm happy to let the folks that use Windows for development regularly > decide on the best answer from a user experience perspective, but I > think a release blocker docs issue would be a good way to go for > ensuring it gets resolved be the release goes out. Done. http://bugs.python.org/issue23623 From storchaka at gmail.com Mon Mar 9 16:11:50 2015 From: storchaka at gmail.com (Serhiy Storchaka) Date: Mon, 09 Mar 2015 17:11:50 +0200 Subject: [Python-Dev] boxing and unboxing data types In-Reply-To: References: <54FD1412.3020002@stoneleaf.us> <2847718.KSDy3zi7qt@raxxla> Message-ID: <3487932.K7gDH0h8FU@raxxla> ?????????, 09-???-2015 10:18:50 ?? ????????: > On Mon, Mar 9, 2015 at 10:10 AM, Serhiy Storchaka wrote: > > ?????????, 09-???-2015 09:52:01 ?? ????????: > > > On Mon, Mar 9, 2015 at 2:07 AM, Serhiy Storchaka > > > > And to be ideal drop-in replacement IntEnum should override such methods > > > > as __eq__ and __hash__ (so it could be used as mapping key). If all methods > > > > should be overridden to quack as int, why not take an int? > > > > > > You're absolutely right that if *all the methods should be overrriden to > > > quack as int, then you should subclass int (the Liskov substitution > > > principle). But all methods should not be overridden ? mainly the methods > > > you overrode in your patch should be exposed. Here is a list of methods on > > > int that should not be on IntFlags in my opinion (give or take a couple): > > > > > > __abs__, __add__, __delattr__, __divmod__, __float__, __floor__, > > > __floordiv__, __index__, __lshift__, __mod__, __mul__, __pos__, __pow__, > > > __radd__, __rdivmod__, __rfloordiv__, __rlshift__, __rmod__, __rmul__, > > > __round__, __rpow__, __rrshift__, __rshift__, __rsub__, __rtruediv__, > > > __sub__, __truediv__, __trunc__, conjugate, denominator, imag, numerator, > > > real. > > > > > > I don't think __index__ should be exposed either since are you really going > > > to slice a list using IntFlags? Really? > > > > Definitely __index__ should be exposed. __int__ is for lossy conversion to int > > (as in float). __index__ is for lossless conversion. > > Is it? __index__ promises lossless conversion, but __index__ is *for* > indexing. I spite of its name it is for any lossless conversion. > > __add__ should be exposed because some code can use + instead of | for > > combining flags. But it shouldn't preserve the type, because this is not > > recommended way. > > I think it should be blocked because it can lead to all kinds of weird > bugs. If the flag is already set and you add it a copy, it silently spills > over into other flags. This is a mistake that a good interface prevents. I think this is a case when backward compatibility has larger weight. > > For the same reason I think __lshift__, __rshift__, __sub__, > > __mul__, __divmod__, __floordiv__, __mod__, etc should be exposed too. So the > > majority of the methods should be exposed, and there is a risk that we loss > > something. > > I totally disagree with all of those. > > > For good compatibility with Python code IntFlags should expose also > > __subclasscheck__ or __subclasshook__. And when we are at this point, why not > > use int subclass? > > Here's another reason. What if someone wants to use an IntFlags object, > but wants to use a fixed width type for storage, say numpy.int32? Why > shouldn't they be able to do that? By using composition, you can easily > provide such an option. You can design abstract interface Flags that can be combined with int or other type. But why you want to use numpy.int32 as storage? This doesn't save much memory, because with composition the IntFlags class weighs more than int subclass. From Steve.Dower at microsoft.com Mon Mar 9 16:37:23 2015 From: Steve.Dower at microsoft.com (Steve Dower) Date: Mon, 9 Mar 2015 15:37:23 +0000 Subject: [Python-Dev] Thoughts on running Python 3.5 on Windows (path, pip install --user, etc) Message-ID: Paul Moore wrote: > I just thought I'd give installing Python 3.5 a go on my PC, now that > 3.5a2 has come out. I didn't get very far (see earlier message) but it prompted > me to think about how I'd use it, and what habits I'd need to change. > > I'd suggest that the "what's new in 3.5" document probably needs a section on > the new installer that explains this stuff... This is true. Right now I'm in experimentation mode, and being more aggressive about changing things than is probably a good idea (because it solicits feedback like this :) ). When things settle down I expect to end up closer to where we started, so there's not a huge amount of value in writing it all up right now. I'll get there. > First of all, I always use "all users" installs, so I have Python in "Program > Files" now. As a result, doing "pip install foo" requires elevation. As that's a > PITA, I probably need to switch to using "pip install --user". All that's fine, > and from there "py -3.5" works fine, as does "py -3.5 -m foo". But even if it > is, not every entry point has a corresponding "-m" invocation (pygments' > pygmentize command doesn't seem to, for example) I know you're already involved in this Paul, but for everyone else there's a big discussion going on at https://github.com/pypa/pip/issues/1668 about changing pip's default behaviour to handle falling back to --user automatically. > But suppose I want to put Python 3.5 on my PATH. The installer has an "add > Python to PATH" checkbox, but (if I'm reading the WiX source right, I didn't > select that on install) that doesn't add the user directory. So I need to add > that to my PATH. Is that right? And of course, that means I need to *know* the > user site directory ($env:LOCALAPPDATA\Python\Python35\Scripts), correct? Correct. There's no way to add a per-user directory to PATH from an all-users installation (except for a few approaches that I expect/hope would trigger malware detectors...) > Maybe the answer is that we simply start recommending that everyone on Windows > uses per-user installs. It makes little difference to me (beyond the fact that > when I want to look at the source of something in the stdlib, the location of > the file is a lot harder to remember than C:\Apps\Python34\Lib\whatever.py) but > I doubt it's what most people will expect. I'm okay with this. Installing for all users is really something that could be considered an advanced option rather than the default, especially since the aim (AIUI) of the all-users install is to pretend that Python was shipped with the OS. (I'd kind of like to take that further by splitting things more sensibly between Program Files, Common Files and System32, but there's very little gain from that and much MUCH pain as long as people are still expecting C:\PythonXY installs...) Cheers, Steve From steve at pearwood.info Mon Mar 9 16:46:59 2015 From: steve at pearwood.info (Steven D'Aprano) Date: Tue, 10 Mar 2015 02:46:59 +1100 Subject: [Python-Dev] boxing and unboxing data types In-Reply-To: References: <54FD1412.3020002@stoneleaf.us> <20150309041243.GS7655@ando.pearwood.info> <54FD227C.5080807@stoneleaf.us> Message-ID: <20150309154659.GU7655@ando.pearwood.info> On Mon, Mar 09, 2015 at 09:52:01AM -0400, Neil Girdhar wrote: > Here is a list of methods on > int that should not be on IntFlags in my opinion (give or take a couple): > > __abs__, __add__, __delattr__, __divmod__, __float__, __floor__, > __floordiv__, __index__, __lshift__, __mod__, __mul__, __pos__, __pow__, > __radd__, __rdivmod__, __rfloordiv__, __rlshift__, __rmod__, __rmul__, > __round__, __rpow__, __rrshift__, __rshift__, __rsub__, __rtruediv__, > __sub__, __truediv__, __trunc__, conjugate, denominator, imag, numerator, > real. > > I don't think __index__ should be exposed either since are you really going > to slice a list using IntFlags? Really? In what way is this an *Int*Flags object if it is nothing like an int? It sounds like what you want is a bunch of Enum inside a set with a custom __str__, not IntFlags. -- Steve From mistersheik at gmail.com Mon Mar 9 16:50:14 2015 From: mistersheik at gmail.com (Neil Girdhar) Date: Mon, 9 Mar 2015 11:50:14 -0400 Subject: [Python-Dev] boxing and unboxing data types In-Reply-To: <20150309154659.GU7655@ando.pearwood.info> References: <54FD1412.3020002@stoneleaf.us> <20150309041243.GS7655@ando.pearwood.info> <54FD227C.5080807@stoneleaf.us> <20150309154659.GU7655@ando.pearwood.info> Message-ID: On Mon, Mar 9, 2015 at 11:46 AM, Steven D'Aprano wrote: > On Mon, Mar 09, 2015 at 09:52:01AM -0400, Neil Girdhar wrote: > > > Here is a list of methods on > > int that should not be on IntFlags in my opinion (give or take a couple): > > > > __abs__, __add__, __delattr__, __divmod__, __float__, __floor__, > > __floordiv__, __index__, __lshift__, __mod__, __mul__, __pos__, __pow__, > > __radd__, __rdivmod__, __rfloordiv__, __rlshift__, __rmod__, __rmul__, > > __round__, __rpow__, __rrshift__, __rshift__, __rsub__, __rtruediv__, > > __sub__, __truediv__, __trunc__, conjugate, denominator, imag, numerator, > > real. > > > > I don't think __index__ should be exposed either since are you really > going > > to slice a list using IntFlags? Really? > > In what way is this an *Int*Flags object if it is nothing like an int? > It sounds like what you want is a bunch of Enum inside a set with a custom > __str__, not IntFlags. > > It doesn't matter what you call it. I believe the goal of this is to have a flags object with flags operations and pretty-printing. It makes more sense to me to decide the interface and then the implementation. > > -- > Steve > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/mistersheik%40gmail.com > -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Mon Mar 9 16:53:33 2015 From: steve at pearwood.info (Steven D'Aprano) Date: Tue, 10 Mar 2015 02:53:33 +1100 Subject: [Python-Dev] Tunning binary insertion sort algorithm in Timsort. In-Reply-To: References: Message-ID: <20150309155329.GV7655@ando.pearwood.info> On Sun, Mar 08, 2015 at 10:57:30PM -0700, Ryan Smith-Roberts wrote: > I suspect that you will find the Python community extremely conservative > about any changes to its sorting algorithm, given that it took thirteen > years and some really impressive automated verification software to find > this bug: On the other hand, the only person who really needs to be convinced is Tim Peters. It's really not up to the Python community. The bug tracker is the right place for discussing this. -- Steve From mistersheik at gmail.com Mon Mar 9 16:48:26 2015 From: mistersheik at gmail.com (Neil Girdhar) Date: Mon, 9 Mar 2015 11:48:26 -0400 Subject: [Python-Dev] boxing and unboxing data types In-Reply-To: <3487932.K7gDH0h8FU@raxxla> References: <54FD1412.3020002@stoneleaf.us> <2847718.KSDy3zi7qt@raxxla> <3487932.K7gDH0h8FU@raxxla> Message-ID: On Mon, Mar 9, 2015 at 11:11 AM, Serhiy Storchaka wrote: > ?????????, 09-???-2015 10:18:50 ?? ????????: > > On Mon, Mar 9, 2015 at 10:10 AM, Serhiy Storchaka > wrote: > > > ?????????, 09-???-2015 09:52:01 ?? ????????: > > > > On Mon, Mar 9, 2015 at 2:07 AM, Serhiy Storchaka < > storchaka at gmail.com> > > > > > And to be ideal drop-in replacement IntEnum should override such > methods > > > > > as __eq__ and __hash__ (so it could be used as mapping key). If > all methods > > > > > should be overridden to quack as int, why not take an int? > > > > > > > > You're absolutely right that if *all the methods should be > overrriden to > > > > quack as int, then you should subclass int (the Liskov substitution > > > > principle). But all methods should not be overridden ? mainly the > methods > > > > you overrode in your patch should be exposed. Here is a list of > methods on > > > > int that should not be on IntFlags in my opinion (give or take a > couple): > > > > > > > > __abs__, __add__, __delattr__, __divmod__, __float__, __floor__, > > > > __floordiv__, __index__, __lshift__, __mod__, __mul__, __pos__, > __pow__, > > > > __radd__, __rdivmod__, __rfloordiv__, __rlshift__, __rmod__, > __rmul__, > > > > __round__, __rpow__, __rrshift__, __rshift__, __rsub__, __rtruediv__, > > > > __sub__, __truediv__, __trunc__, conjugate, denominator, imag, > numerator, > > > > real. > > > > > > > > I don't think __index__ should be exposed either since are you > really going > > > > to slice a list using IntFlags? Really? > > > > > > Definitely __index__ should be exposed. __int__ is for lossy > conversion to int > > > (as in float). __index__ is for lossless conversion. > > > > Is it? __index__ promises lossless conversion, but __index__ is *for* > > indexing. > > I spite of its name it is for any lossless conversion. > You're right. > > > > __add__ should be exposed because some code can use + instead of | for > > > combining flags. But it shouldn't preserve the type, because this is > not > > > recommended way. > > > > I think it should be blocked because it can lead to all kinds of weird > > bugs. If the flag is already set and you add it a copy, it silently > spills > > over into other flags. This is a mistake that a good interface prevents. > > I think this is a case when backward compatibility has larger weight. > > So you agree that the ideal solution is composition, but you prefer inheritance in order to not break code? Then,I think the big question is how much code would actually break if you presented the ideal interface. I imagine that 99% of the code using flags only uses __or__ to compose and __and__, __invert__ to erase flags. > > > For the same reason I think __lshift__, __rshift__, __sub__, > > > __mul__, __divmod__, __floordiv__, __mod__, etc should be exposed too. > So the > > > majority of the methods should be exposed, and there is a risk that we > loss > > > something. > > > > I totally disagree with all of those. > > > > > For good compatibility with Python code IntFlags should expose also > > > __subclasscheck__ or __subclasshook__. And when we are at this point, > why not > > > use int subclass? > > > > Here's another reason. What if someone wants to use an IntFlags object, > > but wants to use a fixed width type for storage, say numpy.int32? Why > > shouldn't they be able to do that? By using composition, you can easily > > provide such an option. > > You can design abstract interface Flags that can be combined with int or > other type. But why you want to use numpy.int32 as storage? This doesn't > save much memory, because with composition the IntFlags class weighs more > than int subclass. > > Maybe you're storing a bunch of flags in a numpy array having dtype np.int32? It's contrived, I agree. -------------- next part -------------- An HTML attachment was scrubbed... URL: From skip.montanaro at gmail.com Mon Mar 9 17:28:55 2015 From: skip.montanaro at gmail.com (Skip Montanaro) Date: Mon, 9 Mar 2015 11:28:55 -0500 Subject: [Python-Dev] Tunning binary insertion sort algorithm in Timsort. In-Reply-To: <20150309155329.GV7655@ando.pearwood.info> References: <20150309155329.GV7655@ando.pearwood.info> Message-ID: On Mon, Mar 9, 2015 at 10:53 AM, Steven D'Aprano wrote: > On Sun, Mar 08, 2015 at 10:57:30PM -0700, Ryan Smith-Roberts wrote: >> I suspect that you will find the Python community extremely conservative >> about any changes to its sorting algorithm, given that it took thirteen >> years and some really impressive automated verification software to find >> this bug: > > On the other hand, the only person who really needs to be convinced is > Tim Peters. It's really not up to the Python community. Also, there's no sense discouraging people who have ideas to contribute. So what if nha pham's contribution isn't accepted? You never know when the next Tim Peters, Georg Brandl, Mark Dickinson, or Brett Cannon will turn up. (That list is practically endless. Don't feel slighted if I failed to mention you.) With seven billion people out there, a few of them are bound to be pretty smart. Skip From donald at stufft.io Mon Mar 9 17:35:44 2015 From: donald at stufft.io (Donald Stufft) Date: Mon, 9 Mar 2015 12:35:44 -0400 Subject: [Python-Dev] Thoughts on running Python 3.5 on Windows (path, pip install --user, etc) In-Reply-To: References: Message-ID: <4CD3C0B8-C9B2-4C7C-9257-C4F51276C8C2@stufft.io> > On Mar 9, 2015, at 11:37 AM, Steve Dower wrote: > > Paul Moore wrote: >> I just thought I'd give installing Python 3.5 a go on my PC, now that >> 3.5a2 has come out. I didn't get very far (see earlier message) but it prompted >> me to think about how I'd use it, and what habits I'd need to change. >> >> I'd suggest that the "what's new in 3.5" document probably needs a section on >> the new installer that explains this stuff... > > This is true. Right now I'm in experimentation mode, and being more aggressive about changing things than is probably a good idea (because it solicits feedback like this :) ). When things settle down I expect to end up closer to where we started, so there's not a huge amount of value in writing it all up right now. I'll get there. > >> First of all, I always use "all users" installs, so I have Python in "Program >> Files" now. As a result, doing "pip install foo" requires elevation. As that's a >> PITA, I probably need to switch to using "pip install --user". All that's fine, >> and from there "py -3.5" works fine, as does "py -3.5 -m foo". But even if it >> is, not every entry point has a corresponding "-m" invocation (pygments' >> pygmentize command doesn't seem to, for example) > > I know you're already involved in this Paul, but for everyone else there's a big discussion going on at https://github.com/pypa/pip/issues/1668 about changing pip's default behaviour to handle falling back to --user automatically. > >> But suppose I want to put Python 3.5 on my PATH. The installer has an "add >> Python to PATH" checkbox, but (if I'm reading the WiX source right, I didn't >> select that on install) that doesn't add the user directory. So I need to add >> that to my PATH. Is that right? And of course, that means I need to *know* the >> user site directory ($env:LOCALAPPDATA\Python\Python35\Scripts), correct? > > Correct. There's no way to add a per-user directory to PATH from an all-users installation (except for a few approaches that I expect/hope would trigger malware detectors...) > >> Maybe the answer is that we simply start recommending that everyone on Windows >> uses per-user installs. It makes little difference to me (beyond the fact that >> when I want to look at the source of something in the stdlib, the location of >> the file is a lot harder to remember than C:\Apps\Python34\Lib\whatever.py) but >> I doubt it's what most people will expect. > > I'm okay with this. Installing for all users is really something that could be considered an advanced option rather than the default, especially since the aim (AIUI) of the all-users install is to pretend that Python was shipped with the OS. (I'd kind of like to take that further by splitting things more sensibly between Program Files, Common Files and System32, but there's very little gain from that and much MUCH pain as long as people are still expecting C:\PythonXY installs?) Maybe the answer is to write up a PEP and standardize the idea of entry points, specifically the console_scripts and ui_scripts (or whatever it?s called) entrypoints and then give Python something like -m, but which executes a specific entry point name instead of a module name (or maybe -m can fall back to looking at entry points? I don?t know). I?ve given this like? 30s worth of thought, but maybe: pip install pygmentize # Implicit ?user py -e pygmetize Is an OK UX for people to have without needing to add the user site bin directory to their PATH. Maybe it?s a horrible idea and we should all forget I mentioned it :) --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From p.f.moore at gmail.com Mon Mar 9 17:50:41 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Mon, 9 Mar 2015 16:50:41 +0000 Subject: [Python-Dev] Thoughts on running Python 3.5 on Windows (path, pip install --user, etc) In-Reply-To: <4CD3C0B8-C9B2-4C7C-9257-C4F51276C8C2@stufft.io> References: <4CD3C0B8-C9B2-4C7C-9257-C4F51276C8C2@stufft.io> Message-ID: On 9 March 2015 at 16:35, Donald Stufft wrote: >> I'm okay with this. Installing for all users is really something that could be considered an advanced option rather than the default, especially since the aim (AIUI) of the all-users install is to pretend that Python was shipped with the OS. (I'd kind of like to take that further by splitting things more sensibly between Program Files, Common Files and System32, but there's very little gain from that and much MUCH pain as long as people are still expecting C:\PythonXY installs?) > > Maybe the answer is to write up a PEP and standardize the idea of entry points, specifically the console_scripts and ui_scripts (or whatever it?s called) entrypoints and then give Python something like -m, but which executes a specific entry point name instead of a module name (or maybe -m can fall back to looking at entry points? I don?t know). > > I?ve given this like? 30s worth of thought, but maybe: > > pip install pygmentize # Implicit ?user > py -e pygmetize > > Is an OK UX for people to have without needing to add the user site bin directory to their PATH. Maybe it?s a horrible idea and we should all forget I mentioned it :) That would be good. You can do it now using setuptools' entry point API, so making it part of the core is certainly not impossible. But is it practical on the 3.5 release timescales? It's worth taking into account the fact that pygmentize is exposed by pygments, so there's no way to deduce the package from the entry point name. Also, even if a core feature appears, nobody will be using it for ages. On 9 March 2015 at 15:37, Steve Dower wrote: >> Maybe the answer is that we simply start recommending that everyone on Windows >> uses per-user installs. It makes little difference to me (beyond the fact that >> when I want to look at the source of something in the stdlib, the location of >> the file is a lot harder to remember than C:\Apps\Python34\Lib\whatever.py) but >> I doubt it's what most people will expect. > > I'm okay with this. Installing for all users is really something that could be considered an advanced option rather than the default, especially since the aim (AIUI) of the all-users install is to pretend that Python was shipped with the OS. (I'd kind of like to take that further by splitting things more sensibly between Program Files, Common Files and System32, but there's very little gain from that and much MUCH pain as long as people are still expecting C:\PythonXY installs...) Just to be clear, I'm *not* okay with this in the form I proposed it. I think we're a long way yet from a clean and understandable proposal for having user installs be as usable as system installs. It's worth noting that (as far as I know) users don't typically use user installs even on Unix, where the issue of the system install being read-only has always been the norm. To me, that implies that there are still some pieces of the puzzle to be addressed. And again, I don't know if this is going to be solved in Python 3.5 timescales. Beta 1 is May 24th - would a major change like a version of pip defaulting to user installs be acceptable after that? More specifically, would core changes to install processes and recommended practices to support that be allowed after the beta? Maybe I'm being pessimistic. A good solution would be fantastic. But at the moment, I feel like 3.4 (and 2.7.9) with pip and the scripts directory on PATH was a huge usability step forward for naive users on Windows, and we're in danger here of reverting it. And getting "naive user" feedback until after the release is notoriously hard... A "practicality beats purity" solution of sticking to the old install scheme for one more version seems like it should be left as an option, at least as a fallback if we don't solve the issues in time. Paul From storchaka at gmail.com Mon Mar 9 17:54:37 2015 From: storchaka at gmail.com (Serhiy Storchaka) Date: Mon, 09 Mar 2015 18:54:37 +0200 Subject: [Python-Dev] boxing and unboxing data types In-Reply-To: References: <54FD1412.3020002@stoneleaf.us> <2847718.KSDy3zi7qt@raxxla> <3487932.K7gDH0h8FU@raxxla> Message-ID: On 09.03.15 17:48, Neil Girdhar wrote: > So you agree that the ideal solution is composition, but you prefer > inheritance in order to not break code? Yes, I agree. There is two advantages in the inheritance: larger backward compatibility and simpler implementation. > Then,I think the big question > is how much code would actually break if you presented the ideal > interface. I imagine that 99% of the code using flags only uses __or__ > to compose and __and__, __invert__ to erase flags. I don't know and don't want to guess. Let just follow the way of bool and IntEnum. When users will be encouraged to use IntEnum and IntFlags instead of plain ints we could consider the idea of dropping inheritance of bool, IntEnum and IntFlags from int. This is not near future. > > Here's another reason. What if someone wants to use an IntFlags object, > > but wants to use a fixed width type for storage, say numpy.int32? Why > > shouldn't they be able to do that? By using composition, you can easily > > provide such an option. > You can design abstract interface Flags that can be combined with > int or other type. But why you want to use numpy.int32 as storage? > This doesn't save much memory, because with composition the IntFlags > class weighs more than int subclass. > Maybe you're storing a bunch of flags in a numpy array having dtype > np.int32? It's contrived, I agree. I afraid that composition will not help you with this. Can numpy array pack int-like objects into fixed-width integer array and then restore original type on unboxing? From ischwabacher at wisc.edu Mon Mar 9 18:39:18 2015 From: ischwabacher at wisc.edu (Isaac Schwabacher) Date: Mon, 09 Mar 2015 12:39:18 -0500 Subject: [Python-Dev] Tunning binary insertion sort algorithm in Timsort. In-Reply-To: <7790de4c2be2c.54fdd645@wiscmail.wisc.edu> References: <7750938a2fea2.54fdd09e@wiscmail.wisc.edu> <76e0c70929f48.54fdd0da@wiscmail.wisc.edu> <76f094a32c8ca.54fdd119@wiscmail.wisc.edu> <7740f7e62cf4b.54fdd155@wiscmail.wisc.edu> <778088e82a599.54fdd192@wiscmail.wisc.edu> <7740ebbf29f9e.54fdd2be@wiscmail.wisc.edu> <76f0f50b2c800.54fdd2fb@wiscmail.wisc.edu> <7710a1122c5ba.54fdd337@wiscmail.wisc.edu> <7780f4192fd2c.54fdd373@wiscmail.wisc.edu> <7790d04c299a7.54fdd590@wiscmail.wisc.edu> <7710a1822e0a9.54fdd5cc@wiscmail.wisc.edu> <7720dc73289e8.54fdd609@wiscmail.wisc.edu> <7790de4c2be2c.54fdd645@wiscmail.wisc.edu> Message-ID: <76f0eced29ac4.54fd9476@wiscmail.wisc.edu> On 15-03-08, nha pham wrote: > > We can optimize the TimSort algorithm by optimizing its binary insertion sort. > > The current version of binary insertion sort use this idea: > > Use binary search to find a final position in sorted list for a new element X. Then insert X to that location. > > I suggest another idea: > > Use binary search to find a final postion in sorted list for a new element X. Before insert X to that location, compare X with its next element. > > For the next element, we already know if it is lower or bigger than X, so we can reduce the search area to the left side or on the right side of X in the sorted list. I don't understand how this is an improvement, since with binary search the idea is that each comparison cuts the remaining list to search in half; i.e., each comparison yields one bit of information. Here, you're spending a comparison to cut the list to search at the element you just inserted, which is probably not right in the middle. If you miss the middle, you're getting on average less than a full bit of information from your comparison, so you're not reducing the remaining search space by as much as you would be if you just compared to the element in the middle of the list. > I have applied my idea on java.util. ComparableTimSort.sort() and testing. The execute time is reduced by 2%-6% with array of random integer. For all that, though, experiment trumps theory... > Here is detail about algorithm and testing:?https://github.com/nhapq/Optimize_binary_insertion_sort > > Sincerely. > > phqnha From phqnha at gmail.com Mon Mar 9 19:49:46 2015 From: phqnha at gmail.com (nha pham) Date: Mon, 9 Mar 2015 11:49:46 -0700 Subject: [Python-Dev] Tunning binary insertion sort algorithm in Timsort. In-Reply-To: References: <7750938a2fea2.54fdd09e@wiscmail.wisc.edu> <76e0c70929f48.54fdd0da@wiscmail.wisc.edu> <76f094a32c8ca.54fdd119@wiscmail.wisc.edu> <7740f7e62cf4b.54fdd155@wiscmail.wisc.edu> <778088e82a599.54fdd192@wiscmail.wisc.edu> <7740ebbf29f9e.54fdd2be@wiscmail.wisc.edu> <76f0f50b2c800.54fdd2fb@wiscmail.wisc.edu> <7710a1122c5ba.54fdd337@wiscmail.wisc.edu> <7780f4192fd2c.54fdd373@wiscmail.wisc.edu> <7790d04c299a7.54fdd590@wiscmail.wisc.edu> <7710a1822e0a9.54fdd5cc@wiscmail.wisc.edu> <7720dc73289e8.54fdd609@wiscmail.wisc.edu> <7790de4c2be2c.54fdd645@wiscmail.wisc.edu> <76f0eced29ac4.54fd9476@wiscmail.wisc.edu> Message-ID: I do not know exactly, one thing I can imagine is: it turns the worst case of binary insertion sort to best case. With sorted array in range of 32 or 64 items, built from zero element. The new element you put into the sorted list has a high chance of being the smallest or the the highest of the sorted list (or nearly highest or nearly smallest) If that case happen, the old binary insertion sort will have the investigate all the list, while with my idea, it just have to compare more 1-2 times. I will try to run more test an more thinking to make sure though. On Mon, Mar 9, 2015 at 11:48 AM, nha pham wrote: > I do not know exactly, one thing I can imagine is: it turns the worst case > of binary insertion sort to best case. > With sorted array in range of 32 or 64 items, built from zero element. The > new element you put into the sorted list has a high chance of being the > smallest or the the highest of the sorted list (or nearly highest or nearly > smallest) > > If that case happen, the old binary insertion sort will have the > investigate all the list, while with my idea, it just have to compare more > 1-2 times. > I will try to run more test an more thinking to make sure though. > > > > On Mon, Mar 9, 2015 at 10:39 AM, Isaac Schwabacher > wrote: > >> On 15-03-08, nha pham >> wrote: >> > >> > We can optimize the TimSort algorithm by optimizing its binary >> insertion sort. >> > >> > The current version of binary insertion sort use this idea: >> > >> > Use binary search to find a final position in sorted list for a new >> element X. Then insert X to that location. >> > >> > I suggest another idea: >> > >> > Use binary search to find a final postion in sorted list for a new >> element X. Before insert X to that location, compare X with its next >> element. >> > >> > For the next element, we already know if it is lower or bigger than X, >> so we can reduce the search area to the left side or on the right side of X >> in the sorted list. >> >> I don't understand how this is an improvement, since with binary search >> the idea is that each comparison cuts the remaining list to search in half; >> i.e., each comparison yields one bit of information. Here, you're spending >> a comparison to cut the list to search at the element you just inserted, >> which is probably not right in the middle. If you miss the middle, you're >> getting on average less than a full bit of information from your >> comparison, so you're not reducing the remaining search space by as much as >> you would be if you just compared to the element in the middle of the list. >> >> > I have applied my idea on java.util. ComparableTimSort.sort() and >> testing. The execute time is reduced by 2%-6% with array of random integer. >> >> For all that, though, experiment trumps theory... >> >> > Here is detail about algorithm and testing: >> https://github.com/nhapq/Optimize_binary_insertion_sort >> > >> > Sincerely. >> > >> > phqnha >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mistersheik at gmail.com Mon Mar 9 21:42:48 2015 From: mistersheik at gmail.com (Neil Girdhar) Date: Mon, 9 Mar 2015 16:42:48 -0400 Subject: [Python-Dev] Tunning binary insertion sort algorithm in Timsort. In-Reply-To: References: <7750938a2fea2.54fdd09e@wiscmail.wisc.edu> <76e0c70929f48.54fdd0da@wiscmail.wisc.edu> <76f094a32c8ca.54fdd119@wiscmail.wisc.edu> <7740f7e62cf4b.54fdd155@wiscmail.wisc.edu> <778088e82a599.54fdd192@wiscmail.wisc.edu> <7740ebbf29f9e.54fdd2be@wiscmail.wisc.edu> <76f0f50b2c800.54fdd2fb@wiscmail.wisc.edu> <7710a1122c5ba.54fdd337@wiscmail.wisc.edu> <7780f4192fd2c.54fdd373@wiscmail.wisc.edu> <7790d04c299a7.54fdd590@wiscmail.wisc.edu> <7710a1822e0a9.54fdd5cc@wiscmail.wisc.edu> <7720dc73289e8.54fdd609@wiscmail.wisc.edu> <7790de4c2be2c.54fdd645@wiscmail.wisc.edu> <76f0eced29ac4.54fd9476@wiscmail.wisc.edu> Message-ID: It may be that the comparison that you do is between two elements that are almost always in the same cache line whereas the binary search might often incur a cache miss. On Mon, Mar 9, 2015 at 2:49 PM, nha pham wrote: > I do not know exactly, one thing I can imagine is: it turns the worst case > of binary insertion sort to best case. > With sorted array in range of 32 or 64 items, built from zero element. The > new element you put into the sorted list has a high chance of being the > smallest or the the highest of the sorted list (or nearly highest or nearly > smallest) > > If that case happen, the old binary insertion sort will have the > investigate all the list, while with my idea, it just have to compare more > 1-2 times. > I will try to run more test an more thinking to make sure though. > > On Mon, Mar 9, 2015 at 11:48 AM, nha pham wrote: > >> I do not know exactly, one thing I can imagine is: it turns the worst >> case of binary insertion sort to best case. >> With sorted array in range of 32 or 64 items, built from zero element. >> The new element you put into the sorted list has a high chance of being the >> smallest or the the highest of the sorted list (or nearly highest or nearly >> smallest) >> >> If that case happen, the old binary insertion sort will have the >> investigate all the list, while with my idea, it just have to compare more >> 1-2 times. >> I will try to run more test an more thinking to make sure though. >> >> >> >> On Mon, Mar 9, 2015 at 10:39 AM, Isaac Schwabacher > > wrote: >> >>> On 15-03-08, nha pham >>> wrote: >>> > >>> > We can optimize the TimSort algorithm by optimizing its binary >>> insertion sort. >>> > >>> > The current version of binary insertion sort use this idea: >>> > >>> > Use binary search to find a final position in sorted list for a new >>> element X. Then insert X to that location. >>> > >>> > I suggest another idea: >>> > >>> > Use binary search to find a final postion in sorted list for a new >>> element X. Before insert X to that location, compare X with its next >>> element. >>> > >>> > For the next element, we already know if it is lower or bigger than X, >>> so we can reduce the search area to the left side or on the right side of X >>> in the sorted list. >>> >>> I don't understand how this is an improvement, since with binary search >>> the idea is that each comparison cuts the remaining list to search in half; >>> i.e., each comparison yields one bit of information. Here, you're spending >>> a comparison to cut the list to search at the element you just inserted, >>> which is probably not right in the middle. If you miss the middle, you're >>> getting on average less than a full bit of information from your >>> comparison, so you're not reducing the remaining search space by as much as >>> you would be if you just compared to the element in the middle of the list. >>> >>> > I have applied my idea on java.util. ComparableTimSort.sort() and >>> testing. The execute time is reduced by 2%-6% with array of random integer. >>> >>> For all that, though, experiment trumps theory... >>> >>> > Here is detail about algorithm and testing: >>> https://github.com/nhapq/Optimize_binary_insertion_sort >>> > >>> > Sincerely. >>> > >>> > phqnha >>> >> >> > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/mistersheik%40gmail.com > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mistersheik at gmail.com Mon Mar 9 21:49:27 2015 From: mistersheik at gmail.com (Neil Girdhar) Date: Mon, 9 Mar 2015 16:49:27 -0400 Subject: [Python-Dev] boxing and unboxing data types In-Reply-To: References: <54FD1412.3020002@stoneleaf.us> <2847718.KSDy3zi7qt@raxxla> <3487932.K7gDH0h8FU@raxxla> Message-ID: On Mon, Mar 9, 2015 at 12:54 PM, Serhiy Storchaka wrote: > On 09.03.15 17:48, Neil Girdhar wrote: > >> So you agree that the ideal solution is composition, but you prefer >> inheritance in order to not break code? >> > > Yes, I agree. There is two advantages in the inheritance: larger backward > compatibility and simpler implementation. > > Inheritance might be more backwards compatible, but I believe that you should check how much code is genuine not restricted to the idealized flags interface. It's not worth talking about "simpler implementation" since the two solutions differ by only a couple dozen lines. On the other hand, composition is better design. It prevents you from making mistakes like adding to flags and having carries, or using flags in an unintended way. > Then,I think the big question >> is how much code would actually break if you presented the ideal >> interface. I imagine that 99% of the code using flags only uses __or__ >> to compose and __and__, __invert__ to erase flags. >> > > I don't know and don't want to guess. Let just follow the way of bool and > IntEnum. When users will be encouraged to use IntEnum and IntFlags instead > of plain ints we could consider the idea of dropping inheritance of bool, > IntEnum and IntFlags from int. This is not near future. I think it's the other way around. You should typically start with the modest interface and add methods as you need. If you start with full blown inheritance, you will find it only increasingly more difficult to remove methods in changing your solution. Using inheritance instead of composition is one of the most common errors in objected oriented programming, and I get the impression from your other paragraph that you're seduced by the slightly shorter code. I don't think it's worth giving in to that without proof that composition will actually break a significant amount of code. Regarding IntEnum ? that should inherit from int since they are truly just integer constants. It's too late for bool; that ship has sailed unfortunately. > > > > Here's another reason. What if someone wants to use an IntFlags >> object, >> > but wants to use a fixed width type for storage, say numpy.int32? >> Why >> > shouldn't they be able to do that? By using composition, you can >> easily >> > provide such an option. >> You can design abstract interface Flags that can be combined with >> int or other type. But why you want to use numpy.int32 as storage? >> This doesn't save much memory, because with composition the IntFlags >> class weighs more than int subclass. >> Maybe you're storing a bunch of flags in a numpy array having dtype >> np.int32? It's contrived, I agree. >> > > I afraid that composition will not help you with this. Can numpy array > pack int-like objects into fixed-width integer array and then restore > original type on unboxing? You're right. > > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ > mistersheik%40gmail.com > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Tue Mar 10 00:11:53 2015 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 10 Mar 2015 09:11:53 +1000 Subject: [Python-Dev] Thoughts on running Python 3.5 on Windows (path, pip install --user, etc) In-Reply-To: <4CD3C0B8-C9B2-4C7C-9257-C4F51276C8C2@stufft.io> References: <4CD3C0B8-C9B2-4C7C-9257-C4F51276C8C2@stufft.io> Message-ID: On 10 Mar 2015 02:37, "Donald Stufft" wrote: > > > > I'm okay with this. Installing for all users is really something that could be considered an advanced option rather than the default, especially since the aim (AIUI) of the all-users install is to pretend that Python was shipped with the OS. (I'd kind of like to take that further by splitting things more sensibly between Program Files, Common Files and System32, but there's very little gain from that and much MUCH pain as long as people are still expecting C:\PythonXY installs?) > > Maybe the answer is to write up a PEP and standardize the idea of entry points, specifically the console_scripts and ui_scripts (or whatever it?s called) entrypoints and then give Python something like -m, but which executes a specific entry point name instead of a module name (or maybe -m can fall back to looking at entry points? I don?t know). While I like the idea of offering something more "built in" in this space, my initial inclination is to prefer extending "-m" to accept the "module.name:function.name" format to let you invoke entry points by the name of the target function (Possible API name: runpy.run_cli_function), and then add a "runpy.call" that can be used to call an arbitrary function with positional and keyword string arguments based on sys.argv and (optionally?) print the repr of the result. It wouldn't be a universal panacea (and would need a PEP to work out the exact UX details), but would likely make quite a few libraries more command line accessible without needing to modify them. Cheers, Nick. > > I?ve given this like? 30s worth of thought, but maybe: > > pip install pygmentize # Implicit ?user > py -e pygmetize > > Is an OK UX for people to have without needing to add the user site bin directory to their PATH. Maybe it?s a horrible idea and we should all forget I mentioned it :) > > --- > Donald Stufft > PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com > -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Tue Mar 10 00:16:35 2015 From: donald at stufft.io (Donald Stufft) Date: Mon, 9 Mar 2015 19:16:35 -0400 Subject: [Python-Dev] Thoughts on running Python 3.5 on Windows (path, pip install --user, etc) In-Reply-To: References: <4CD3C0B8-C9B2-4C7C-9257-C4F51276C8C2@stufft.io> Message-ID: <284FD035-2925-4763-8685-CB5292104343@stufft.io> > On Mar 9, 2015, at 7:11 PM, Nick Coghlan wrote: > > > On 10 Mar 2015 02:37, "Donald Stufft" > wrote: > > > > > > I'm okay with this. Installing for all users is really something that could be considered an advanced option rather than the default, especially since the aim (AIUI) of the all-users install is to pretend that Python was shipped with the OS. (I'd kind of like to take that further by splitting things more sensibly between Program Files, Common Files and System32, but there's very little gain from that and much MUCH pain as long as people are still expecting C:\PythonXY installs?) > > > > Maybe the answer is to write up a PEP and standardize the idea of entry points, specifically the console_scripts and ui_scripts (or whatever it?s called) entrypoints and then give Python something like -m, but which executes a specific entry point name instead of a module name (or maybe -m can fall back to looking at entry points? I don?t know). > > While I like the idea of offering something more "built in" in this space, my initial inclination is to prefer extending "-m" to accept the "module.name:function.name " format to let you invoke entry points by the name of the target function (Possible API name: runpy.run_cli_function), and then add a "runpy.call" that can be used to call an arbitrary function with positional and keyword string arguments based on sys.argv and (optionally?) print the repr of the result. > > It wouldn't be a universal panacea (and would need a PEP to work out the exact UX details), but would likely make quite a few libraries more command line accessible without needing to modify them. > > If I understand this correctly, you?re suggesting that to run ``pygmentize`` without using the script wrapper, you?d need to do ``py -m pygments.cmdline:main`` instead of ``pygmentize``? I don?t think that actually solves the problem (except by making it so that the script wrappers can maybe just be exactly #!/usr/bin/python -m pygments.cmdline:main but that?s a different thing..). I?m not against it in general though, I just don?t know that it solves the problem Paul was mentioning. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From p.f.moore at gmail.com Tue Mar 10 00:18:53 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Mon, 9 Mar 2015 23:18:53 +0000 Subject: [Python-Dev] Thoughts on running Python 3.5 on Windows (path, pip install --user, etc) In-Reply-To: References: <4CD3C0B8-C9B2-4C7C-9257-C4F51276C8C2@stufft.io> Message-ID: On 9 March 2015 at 23:11, Nick Coghlan wrote: > While I like the idea of offering something more "built in" in this space, > my initial inclination is to prefer extending "-m" to accept the > "module.name:function.name" format to let you invoke entry points by the > name of the target function (Possible API name: runpy.run_cli_function), and > then add a "runpy.call" that can be used to call an arbitrary function with > positional and keyword string arguments based on sys.argv and (optionally?) > print the repr of the result. > > It wouldn't be a universal panacea (and would need a PEP to work out the > exact UX details), but would likely make quite a few libraries more command > line accessible without needing to modify them. Personally I doubt it would make much difference. If the docs say "pygmentize" I'm unlikely to dig around to find that the incantation "python -m pygments.somemodule:main" does the same thing using 3 times as many characters. I'd just add Python to my PATH and say stuff it. Paul From ncoghlan at gmail.com Tue Mar 10 00:22:06 2015 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 10 Mar 2015 09:22:06 +1000 Subject: [Python-Dev] boxing and unboxing data types In-Reply-To: References: <54FD1412.3020002@stoneleaf.us> <2847718.KSDy3zi7qt@raxxla> <3487932.K7gDH0h8FU@raxxla> Message-ID: On 10 Mar 2015 06:51, "Neil Girdhar" wrote: > > > > On Mon, Mar 9, 2015 at 12:54 PM, Serhiy Storchaka wrote: >> >> On 09.03.15 17:48, Neil Girdhar wrote: >>> >>> So you agree that the ideal solution is composition, but you prefer >>> inheritance in order to not break code? >> >> >> Yes, I agree. There is two advantages in the inheritance: larger backward compatibility and simpler implementation. >> > > Inheritance might be more backwards compatible, but I believe that you should check how much code is genuine not restricted to the idealized flags interface. It's not worth talking about "simpler implementation" since the two solutions differ by only a couple dozen lines. We literally can't do this, as the vast majority of Python code in the world is locked up behind institutional firewalls or has otherwise never been published. The open source stuff is merely the tip of a truly enormous iceberg. If we want to *use* IntFlags in the standard library (and that's the only pay-off significant enough to justify having it in the standard library), then it needs to inherit from int. However, cloning the full enum module architecture to create flags.FlagsMeta, flags.Flags and flags.IntFlags would make sense to me. It would also make sense to try that idea out on PyPI for a while before incorporating it into the stdlib. Regards, Nick. > > On the other hand, composition is better design. It prevents you from making mistakes like adding to flags and having carries, or using flags in an unintended way. > >>> >>> Then,I think the big question >>> is how much code would actually break if you presented the ideal >>> interface. I imagine that 99% of the code using flags only uses __or__ >>> to compose and __and__, __invert__ to erase flags. >> >> >> I don't know and don't want to guess. Let just follow the way of bool and IntEnum. When users will be encouraged to use IntEnum and IntFlags instead of plain ints we could consider the idea of dropping inheritance of bool, IntEnum and IntFlags from int. This is not near future. > > > I think it's the other way around. You should typically start with the modest interface and add methods as you need. If you start with full blown inheritance, you will find it only increasingly more difficult to remove methods in changing your solution. Using inheritance instead of composition is one of the most common errors in objected oriented programming, and I get the impression from your other paragraph that you're seduced by the slightly shorter code. I don't think it's worth giving in to that without proof that composition will actually break a significant amount of code. > > Regarding IntEnum ? that should inherit from int since they are truly just integer constants. It's too late for bool; that ship has sailed unfortunately. > >> >> >> >>> > Here's another reason. What if someone wants to use an IntFlags object, >>> > but wants to use a fixed width type for storage, say numpy.int32? Why >>> > shouldn't they be able to do that? By using composition, you can easily >>> > provide such an option. >>> You can design abstract interface Flags that can be combined with >>> int or other type. But why you want to use numpy.int32 as storage? >>> This doesn't save much memory, because with composition the IntFlags >>> class weighs more than int subclass. >>> Maybe you're storing a bunch of flags in a numpy array having dtype >>> np.int32? It's contrived, I agree. >> >> >> I afraid that composition will not help you with this. Can numpy array pack int-like objects into fixed-width integer array and then restore original type on unboxing? > > > You're right. >> >> >> >> >> _______________________________________________ >> Python-Dev mailing list >> Python-Dev at python.org >> https://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe: https://mail.python.org/mailman/options/python-dev/mistersheik%40gmail.com > > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ryan.stuart.85 at gmail.com Tue Mar 10 00:56:02 2015 From: ryan.stuart.85 at gmail.com (Ryan Stuart) Date: Mon, 09 Mar 2015 23:56:02 +0000 Subject: [Python-Dev] PEP 471 Final: os.scandir() merged into Python 3.5 In-Reply-To: References: Message-ID: Hi Ben, On Mon, 9 Mar 2015 at 21:58 Ben Hoyt wrote: > Note that this benchmark is invalid for a couple of reasons. (...) > Thanks a lot for the guidance Ben, greatly appreciated. Just starting to take an interest in the development of CPython and so something like running a benchmark seemed like a good a place as any to start. Since I want to get comfortable with compiling from source I tried this again. Instead of applying the patch, since the issue is now closed, I just compiled from the tip of the default branch which at the time was 94920:0469af231d22. I also didn't configure with --with-pydebug. Here are the new results: *Ryans-MacBook-Pro:cpython rstuart$ ./python.exe ~/Workspace/python/scandir/benchmark.py* Using Python 3.5's builtin os.scandir() Comparing against builtin version of os.walk() Priming the system's cache... Benchmarking walks on /Users/rstuart/Workspace/python/scandir/benchtree, repeat 1/3... Benchmarking walks on /Users/rstuart/Workspace/python/scandir/benchtree, repeat 2/3... Benchmarking walks on /Users/rstuart/Workspace/python/scandir/benchtree, repeat 3/3... os.walk took 0.061s, scandir.walk took 0.012s -- 5.2x as fast *Ryans-MacBook-Pro:cpython rstuart$ ./python.exe ~/Workspace/python/scandir/benchmark.py -s* Using Python 3.5's builtin os.scandir() Comparing against builtin version of os.walk() Priming the system's cache... Benchmarking walks on /Users/rstuart/Workspace/python/scandir/benchtree, repeat 1/3... Benchmarking walks on /Users/rstuart/Workspace/python/scandir/benchtree, repeat 2/3... Benchmarking walks on /Users/rstuart/Workspace/python/scandir/benchtree, repeat 3/3... os.walk size 23400, scandir.walk size 23400 -- equal os.walk took 0.109s, scandir.walk took 0.049s -- 2.2x as fast This is on a Retina Mid 2012 MacBook Pro with an SSD. Cheers > you're compiling Python in debug mode (--with-pydebug), which produces > significantly slower code in my tests -- for example, on Windows > benchmark.py is about twice as slow when Python is compiled in debug > mode. > > Second, as the output above shows, benchmark.py is "Using slower > ctypes version of scandir" and not a C version at all. If os.scandir() > is available, benchmark.py should use that, so there's something wrong > here -- maybe the patch didn't apply correctly or maybe you're testing > with a different version of Python than the one you built? > > In any case, the easiest way to test it now is to download Python 3.5 > alpha 2 which just came out: > https://www.python.org/downloads/release/python-350a2/ > > I just tried this on my Mac Mini (i5 2.3GHz, 2 GB RAM, HFS+ on > rotational drive) and got the following results: > > Using Python 3.5's builtin os.scandir() > Comparing against builtin version of os.walk() > Priming the system's cache... > Benchmarking walks on benchtree, repeat 1/3... > Benchmarking walks on benchtree, repeat 2/3... > Benchmarking walks on benchtree, repeat 3/3... > os.walk took 0.074s, scandir.walk took 0.016s -- 4.7x as fast > > > I then did ./python.exe ~/Workspace/python/scandir/benchmark.py -s and > got: > > Also note that "benchmark.py -s" tests the system os.walk() against a > get_tree_size() function using scandir's DirEntry.stat().st_size, > which provides huge gains on Windows (because stat().st_size doesn't > require on OS call) but only modest gains on POSIX systems, which > still require an OS stat call to get the size (though not the file > type, so at least it's only one stat call). I get "2.2x as fast" on my > Mac for "benchmark.py -s". > > -Ben > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mistersheik at gmail.com Tue Mar 10 01:23:18 2015 From: mistersheik at gmail.com (Neil Girdhar) Date: Mon, 9 Mar 2015 20:23:18 -0400 Subject: [Python-Dev] boxing and unboxing data types In-Reply-To: References: <54FD1412.3020002@stoneleaf.us> <2847718.KSDy3zi7qt@raxxla> <3487932.K7gDH0h8FU@raxxla> Message-ID: Totally agree On 9 Mar 2015 19:22, "Nick Coghlan" wrote: > > On 10 Mar 2015 06:51, "Neil Girdhar" wrote: > > > > > > > > On Mon, Mar 9, 2015 at 12:54 PM, Serhiy Storchaka > wrote: > >> > >> On 09.03.15 17:48, Neil Girdhar wrote: > >>> > >>> So you agree that the ideal solution is composition, but you prefer > >>> inheritance in order to not break code? > >> > >> > >> Yes, I agree. There is two advantages in the inheritance: larger > backward compatibility and simpler implementation. > >> > > > > Inheritance might be more backwards compatible, but I believe that you > should check how much code is genuine not restricted to the idealized flags > interface. It's not worth talking about "simpler implementation" since > the two solutions differ by only a couple dozen lines. > > We literally can't do this, as the vast majority of Python code in the > world is locked up behind institutional firewalls or has otherwise never > been published. The open source stuff is merely the tip of a truly enormous > iceberg. > > If we want to *use* IntFlags in the standard library (and that's the only > pay-off significant enough to justify having it in the standard library), > then it needs to inherit from int. > > However, cloning the full enum module architecture to create > flags.FlagsMeta, flags.Flags and flags.IntFlags would make sense to me. > > It would also make sense to try that idea out on PyPI for a while before > incorporating it into the stdlib. > > Regards, > Nick. > > > > > On the other hand, composition is better design. It prevents you from > making mistakes like adding to flags and having carries, or using flags in > an unintended way. > > > >>> > >>> Then,I think the big question > >>> is how much code would actually break if you presented the ideal > >>> interface. I imagine that 99% of the code using flags only uses __or__ > >>> to compose and __and__, __invert__ to erase flags. > >> > >> > >> I don't know and don't want to guess. Let just follow the way of bool > and IntEnum. When users will be encouraged to use IntEnum and IntFlags > instead of plain ints we could consider the idea of dropping inheritance of > bool, IntEnum and IntFlags from int. This is not near future. > > > > > > I think it's the other way around. You should typically start with the > modest interface and add methods as you need. If you start with full blown > inheritance, you will find it only increasingly more difficult to remove > methods in changing your solution. Using inheritance instead of > composition is one of the most common errors in objected oriented > programming, and I get the impression from your other paragraph that you're > seduced by the slightly shorter code. I don't think it's worth giving in > to that without proof that composition will actually break a significant > amount of code. > > > > Regarding IntEnum ? that should inherit from int since they are truly > just integer constants. It's too late for bool; that ship has sailed > unfortunately. > > > >> > >> > >> > >>> > Here's another reason. What if someone wants to use an IntFlags > object, > >>> > but wants to use a fixed width type for storage, say > numpy.int32? Why > >>> > shouldn't they be able to do that? By using composition, you > can easily > >>> > provide such an option. > >>> You can design abstract interface Flags that can be combined with > >>> int or other type. But why you want to use numpy.int32 as storage? > >>> This doesn't save much memory, because with composition the > IntFlags > >>> class weighs more than int subclass. > >>> Maybe you're storing a bunch of flags in a numpy array having dtype > >>> np.int32? It's contrived, I agree. > >> > >> > >> I afraid that composition will not help you with this. Can numpy array > pack int-like objects into fixed-width integer array and then restore > original type on unboxing? > > > > > > You're right. > >> > >> > >> > >> > >> _______________________________________________ > >> Python-Dev mailing list > >> Python-Dev at python.org > >> https://mail.python.org/mailman/listinfo/python-dev > >> Unsubscribe: > https://mail.python.org/mailman/options/python-dev/mistersheik%40gmail.com > > > > > > > > _______________________________________________ > > Python-Dev mailing list > > Python-Dev at python.org > > https://mail.python.org/mailman/listinfo/python-dev > > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From benhoyt at gmail.com Tue Mar 10 01:40:50 2015 From: benhoyt at gmail.com (Ben Hoyt) Date: Mon, 9 Mar 2015 20:40:50 -0400 Subject: [Python-Dev] PEP 471 Final: os.scandir() merged into Python 3.5 In-Reply-To: References: Message-ID: > > os.walk took 0.061s, scandir.walk took 0.012s -- 5.2x as fast > Great, looks much better. :-) Even a bit better than what I'm seeing -- possibly due to your SSD. -Ben -------------- next part -------------- An HTML attachment was scrubbed... URL: From phqnha at gmail.com Tue Mar 10 04:46:12 2015 From: phqnha at gmail.com (nha pham) Date: Mon, 9 Mar 2015 20:46:12 -0700 Subject: [Python-Dev] Tunning binary insertion sort algorithm in Timsort. In-Reply-To: <76f0eced29ac4.54fd9476@wiscmail.wisc.edu> References: <7750938a2fea2.54fdd09e@wiscmail.wisc.edu> <76e0c70929f48.54fdd0da@wiscmail.wisc.edu> <76f094a32c8ca.54fdd119@wiscmail.wisc.edu> <7740f7e62cf4b.54fdd155@wiscmail.wisc.edu> <778088e82a599.54fdd192@wiscmail.wisc.edu> <7740ebbf29f9e.54fdd2be@wiscmail.wisc.edu> <76f0f50b2c800.54fdd2fb@wiscmail.wisc.edu> <7710a1122c5ba.54fdd337@wiscmail.wisc.edu> <7780f4192fd2c.54fdd373@wiscmail.wisc.edu> <7790d04c299a7.54fdd590@wiscmail.wisc.edu> <7710a1822e0a9.54fdd5cc@wiscmail.wisc.edu> <7720dc73289e8.54fdd609@wiscmail.wisc.edu> <7790de4c2be2c.54fdd645@wiscmail.wisc.edu> <76f0eced29ac4.54fd9476@wiscmail.wisc.edu> Message-ID: Thank you for your comment. I admit that I did not thinking about the proof before. Here is my naive proof. I hope you can give me some comments: ============= # This proof is an empirical thinking and is not completed, it just gives us a closer look. I hope someone can make it more mathematically. In the proof, we assume we are dealing with unique values array (none of them are equal together). Because if they are equal, the "lucky search" can happen and it is obviously not fair. Statement_1: With an array of size N or less than N, we need at most log2(N) comparisons to find a value (or a position, incase the search miss), using the binary search algorithm. proof: This statement is trivia, and I believe, someone outthere already proved it. We can check again by counting manually. let assume we have array of 32 items: 32 => 16 => 8 => 4 => 2 => 1 (5 comparison) how about 24 items (24 < 32): 24 => 12 => 6 => 3 => 2 => 1 (5 comparison) ok, good enough. Let's just believe on it to move on. Statement_2: If we divide an array into two parts, the more unbalanced arrays we divide, the more benefit we get from the binary search algorithm. proof: Let's assume we have an array of 256 items. case1: If we divide in middle: 128 - 128 Now, if we search on the left, it costs log2(128) = 7 If we search on the right, it cost los2(128) = 7 case2: If we divide unbalanced: 32 - 224 Now, if we search on the left, it costs log2(32) = 5 If we search on the right, it cost at max 8 comparisons (based on the statement_1). You may not believe me, so let's count it by hand: 224 => 112 => 56 => 28 => 14 => 7 => 4 => 2 => 1 So, if we search on the left, we win 2 comparisons compare to case1. We search on the right, we lose 1 comparison compare to case1 I call this is a "benefit". case3: What if we divide more unbalanced: 8 - 248 Search on the left: log2(8) = 3 comparisons. Search on the right, it costs at max 8 comparisons. So if we search on the left, we win 4 comparisons. We search on the right, we lose 1 comparisons. It is "more benefit", isnt it? Statement3: Because we are using random array. There is a 50-50 chance that next_X will be bigger or smaller than X. Statement4: We call N is the size of the sorted list, "index" is the position of X in the sorted list. Because the array is random, index has an equal chance to exist in any position in the sorted list. Statement5: Now we build a model based on previous statements: My idea costs 1 comparison (between X and next_X) to devide the array into two unbalanced parts. The old idea costs 1 comparison to divide the array into two balanced parts. Now let's see which one can find position for next_X faster: If index belongs to [N/4 to 3N/4]: we may lose 1 comparison, or we may not lose. If index belongs to [N/8 to N/4] or [3N/4 to 7N/8]: We may lose 1 comparison, or we win 1 comparison. If index belongs to [N/16 to N/8] or [7N/8 to 15N/16]: We may lose 1 comparison, or we win 2 comparison. If index belongs to [N/32 to N/16] or [15N/16 to 31N/32]: We may lose 1 comparison, or we win 3 comparison. If index belongs to [N/64 to N/32] or [31N/32 to 64N/64]: We may lose 1 comparison, or we win 4 comparison. ... and so on. Statement6: Now we apply the model to a real example. Assume that we already has a sorted list with 16 items. And we already know about "index" of X. We can think of it as a gamble game with 16 slots. In every slot, we only can bid 1 dollar (statement4). From slot 5th to slot 12th, we may lose 1, or we may not lose, 50-50 chance. So after a huge play times, probability told us that we will lose (8 x 1)/2 = 4 dollars. For slot 3, slot 4, slot 13, slot 14, We may lose 1, or we win 1. So after a huge play times, We wont lose or win anything. For slot 2, slot 15. We may lose 1, or we win 2. So after a huge play times, we can win (2-1)x2 = 2 dollars. For slot 1, slot 16. We may lose 1, or we win 3. So after a huge play times, we can win 4 dollars. In total, after a huge play times, we win 4 + 2 + 0 -4 = 2 dollars !!!!! You can test with sorted list 32 or 64 items or any number you want, I believe the benefit is even more. Conclusion: The unbalanced model give us more benefit than the balanced model. That means with an array big enough, My idea give more benefit than the old idea. I think the lucky ticket companies is already know about this. It is a shame that I do not know mathematic principle about this problem. If I have something more, I will update my proof at: https://github.com/nhapq/Optimize_binary_insertion_sort/blob/master/proof.txt ====== Thank you. Nha Pham. On Mon, Mar 9, 2015 at 10:39 AM, Isaac Schwabacher wrote: > On 15-03-08, nha pham > wrote: > > > > We can optimize the TimSort algorithm by optimizing its binary insertion > sort. > > > > The current version of binary insertion sort use this idea: > > > > Use binary search to find a final position in sorted list for a new > element X. Then insert X to that location. > > > > I suggest another idea: > > > > Use binary search to find a final postion in sorted list for a new > element X. Before insert X to that location, compare X with its next > element. > > > > For the next element, we already know if it is lower or bigger than X, > so we can reduce the search area to the left side or on the right side of X > in the sorted list. > > I don't understand how this is an improvement, since with binary search > the idea is that each comparison cuts the remaining list to search in half; > i.e., each comparison yields one bit of information. Here, you're spending > a comparison to cut the list to search at the element you just inserted, > which is probably not right in the middle. If you miss the middle, you're > getting on average less than a full bit of information from your > comparison, so you're not reducing the remaining search space by as much as > you would be if you just compared to the element in the middle of the list. > > > I have applied my idea on java.util. ComparableTimSort.sort() and > testing. The execute time is reduced by 2%-6% with array of random integer. > > For all that, though, experiment trumps theory... > > > Here is detail about algorithm and testing: > https://github.com/nhapq/Optimize_binary_insertion_sort > > > > Sincerely. > > > > phqnha > -------------- next part -------------- An HTML attachment was scrubbed... URL: From tim.peters at gmail.com Tue Mar 10 05:27:02 2015 From: tim.peters at gmail.com (Tim Peters) Date: Mon, 9 Mar 2015 23:27:02 -0500 Subject: [Python-Dev] Tunning binary insertion sort algorithm in Timsort. In-Reply-To: References: <7750938a2fea2.54fdd09e@wiscmail.wisc.edu> <76e0c70929f48.54fdd0da@wiscmail.wisc.edu> <76f094a32c8ca.54fdd119@wiscmail.wisc.edu> <7740f7e62cf4b.54fdd155@wiscmail.wisc.edu> <778088e82a599.54fdd192@wiscmail.wisc.edu> <7740ebbf29f9e.54fdd2be@wiscmail.wisc.edu> <76f0f50b2c800.54fdd2fb@wiscmail.wisc.edu> <7710a1122c5ba.54fdd337@wiscmail.wisc.edu> <7780f4192fd2c.54fdd373@wiscmail.wisc.edu> <7790d04c299a7.54fdd590@wiscmail.wisc.edu> <7710a1822e0a9.54fdd5cc@wiscmail.wisc.edu> <7720dc73289e8.54fdd609@wiscmail.wisc.edu> <7790de4c2be2c.54fdd645@wiscmail.wisc.edu> <76f0eced29ac4.54fd9476@wiscmail.wisc.edu> Message-ID: [nha pham ] > Statement_1: With an array of size N or less than N, we need at most log2(N) > comparisons to find a value > (or a position, incase the search miss), using the binary search algorithm. > > proof: This statement is trivia, and I believe, someone outthere already > proved it. Sorry for the quick message here. It's just a simple point where it will pay not to get off on a wrong foot ;-) Correct: for an array of size N, binary search can require as many as ceiling(log2(N+1)) comparisons. That's because there are N+1 possible results for an array of size N. For example, for an array of size 3, [A, B, C], "the answer" may be "before A", "between A and B", "between B and C", or "after C". 3 elements, 3+1 = 4 possible results. log2(3) comparisons are not enough to distinguish among 4 results. Make it trivial, an array of length 1. Then 1 comparison is obviously necessary and sufficient in all cases. And, indeed, ceiling(log2(1+1)) = 1. log2(1) equals 0, too small. For the rest, I haven't been able to understand your description or your pseudo-code. I'll try harder. Some things clearly aren't doing what you _intend_ them to do. For example, in your Python code, each time through the outer loop you're apparently trying to sort the next CHUNK elements, but you end up appending CHUNK+1 values to data2 (or data3). Or in this part: for i in range(low,high): x = data[i] if x >= data[i-1]: the first time that loop is executed low == 0, and so i == 0 on the first iteration, and so the conditional is if x >= data[0-1] That's referencing data[-1], which is the very last element in data - which has nothing to do with the CHUNK you're trying to sort at the time. So there are a number of errors here, which makes it that much harder to sort out (pun intended ) what you're trying to do. It would help you to add some asserts to verify your code is doing what you _hope_ it's doing. For example, add assert data2[low: high] == sorted(data[low: high]) assert len(data2) == high to the end of your `sample` loop, and similarly for data3 in your `new` loop. Until those asserts all pass, you're not testing code that's actually sorting correctly. Repair the errors and you almost certainly won't find `new` running over 10 times faster than `sample` anymore. I don't know what you _will_ discover, though. If the code doesn't have to sort correctly, there are much simpler ways to make it run _very_ much faster ;-) From phqnha at gmail.com Tue Mar 10 06:05:39 2015 From: phqnha at gmail.com (nha pham) Date: Mon, 9 Mar 2015 22:05:39 -0700 Subject: [Python-Dev] Tunning binary insertion sort algorithm in Timsort. In-Reply-To: References: <7750938a2fea2.54fdd09e@wiscmail.wisc.edu> <76e0c70929f48.54fdd0da@wiscmail.wisc.edu> <76f094a32c8ca.54fdd119@wiscmail.wisc.edu> <7740f7e62cf4b.54fdd155@wiscmail.wisc.edu> <778088e82a599.54fdd192@wiscmail.wisc.edu> <7740ebbf29f9e.54fdd2be@wiscmail.wisc.edu> <76f0f50b2c800.54fdd2fb@wiscmail.wisc.edu> <7710a1122c5ba.54fdd337@wiscmail.wisc.edu> <7780f4192fd2c.54fdd373@wiscmail.wisc.edu> <7790d04c299a7.54fdd590@wiscmail.wisc.edu> <7710a1822e0a9.54fdd5cc@wiscmail.wisc.edu> <7720dc73289e8.54fdd609@wiscmail.wisc.edu> <7790de4c2be2c.54fdd645@wiscmail.wisc.edu> <76f0eced29ac4.54fd9476@wiscmail.wisc.edu> Message-ID: Thank you very much. I am very happy that I got a reply from Tim Peter. You are correct, my mistake. The python code should be: for i in range(low+1,high): //because we already add data[low] x = data[i] if x >= data[i-1]: After I fix it, here is the result: random array 10^6: Old binsort: 1.3322 New binsort: 1.0015 ratio: 0.33 You are right, it is not ten times faster anymore. I will update other results soon. I do check the result of two sorting methods many times to make sure they are the same. It is just because I do not know how to put assert into the timeit.Timer class. I am pretty sure about this. I will try to write the proof more clearly, sorry for inconvenience. Thank you very much. Nha Pham. On Mon, Mar 9, 2015 at 9:27 PM, Tim Peters wrote: > [nha pham ] > > Statement_1: With an array of size N or less than N, we need at most > log2(N) > > comparisons to find a value > > (or a position, incase the search miss), using the binary search > algorithm. > > > > proof: This statement is trivia, and I believe, someone outthere already > > proved it. > > Sorry for the quick message here. It's just a simple point where it > will pay not to get off on a wrong foot ;-) > > Correct: for an array of size N, binary search can require as many as > ceiling(log2(N+1)) comparisons. > > That's because there are N+1 possible results for an array of size N. > For example, for an array of size 3, [A, B, C], "the answer" may be > "before A", "between A and B", "between B and C", or "after C". 3 > elements, 3+1 = 4 possible results. log2(3) comparisons are not > enough to distinguish among 4 results. > > Make it trivial, an array of length 1. Then 1 comparison is obviously > necessary and sufficient in all cases. And, indeed, > ceiling(log2(1+1)) = 1. log2(1) equals 0, too small. > > For the rest, I haven't been able to understand your description or > your pseudo-code. I'll try harder. Some things clearly aren't doing > what you _intend_ them to do. For example, in your Python code, each > time through the outer loop you're apparently trying to sort the next > CHUNK elements, but you end up appending CHUNK+1 values to data2 (or > data3). > > Or in this part: > > for i in range(low,high): > x = data[i] > if x >= data[i-1]: > > the first time that loop is executed low == 0, and so i == 0 on the > first iteration, and so the conditional is > > if x >= data[0-1] > > That's referencing data[-1], which is the very last element in data - > which has nothing to do with the CHUNK you're trying to sort at the > time. > > So there are a number of errors here, which makes it that much harder > to sort out (pun intended ) what you're trying to do. It would > help you to add some asserts to verify your code is doing what you > _hope_ it's doing. For example, add > > assert data2[low: high] == sorted(data[low: high]) > assert len(data2) == high > > to the end of your `sample` loop, and similarly for data3 in your > `new` loop. Until those asserts all pass, you're not testing code > that's actually sorting correctly. Repair the errors and you almost > certainly won't find `new` running over 10 times faster than `sample` > anymore. I don't know what you _will_ discover, though. If the code > doesn't have to sort correctly, there are much simpler ways to make it > run _very_ much faster ;-) > -------------- next part -------------- An HTML attachment was scrubbed... URL: From tim.peters at gmail.com Tue Mar 10 06:28:22 2015 From: tim.peters at gmail.com (Tim Peters) Date: Tue, 10 Mar 2015 00:28:22 -0500 Subject: [Python-Dev] Tunning binary insertion sort algorithm in Timsort. In-Reply-To: References: <7750938a2fea2.54fdd09e@wiscmail.wisc.edu> <76e0c70929f48.54fdd0da@wiscmail.wisc.edu> <76f094a32c8ca.54fdd119@wiscmail.wisc.edu> <7740f7e62cf4b.54fdd155@wiscmail.wisc.edu> <778088e82a599.54fdd192@wiscmail.wisc.edu> <7740ebbf29f9e.54fdd2be@wiscmail.wisc.edu> <76f0f50b2c800.54fdd2fb@wiscmail.wisc.edu> <7710a1122c5ba.54fdd337@wiscmail.wisc.edu> <7780f4192fd2c.54fdd373@wiscmail.wisc.edu> <7790d04c299a7.54fdd590@wiscmail.wisc.edu> <7710a1822e0a9.54fdd5cc@wiscmail.wisc.edu> <7720dc73289e8.54fdd609@wiscmail.wisc.edu> <7790de4c2be2c.54fdd645@wiscmail.wisc.edu> <76f0eced29ac4.54fd9476@wiscmail.wisc.edu> Message-ID: [nha pham ] > Thank you very much. I am very happy that I got a reply from Tim Peter. My pleasure to speak with you too :-) > You are correct, my mistake. > > The python code should be: > for i in range(low+1,high): //because we already add > data[low] > x = data[i] > if x >= data[i-1]: > > After I fix it, here is the result: > > random array 10^6: > Old binsort: 1.3322 > New binsort: 1.0015 > ratio: 0.33 > > You are right, it is not ten times faster anymore. I will update other > results soon. > > I do check the result of two sorting methods many times to make sure they > are the same. It is just because I do not know how to put assert into the > timeit.Timer class. `assert` is just another Python statement. You simply add it to the code - there's nothing tricky about this. You could, e.g., simply copy and paste the `assert`s I suggested last time. Before you do, trying adding `print index` to your inner loops, and make SIZE much smaller (say, 1000) so you're not overwhelmed with output. You'll be surprised by what you see on the second (and following) CHUNKs. For example, in both `sample` and `new` it will print 900 ninety nine times in a row when doing the last CHUNK. The code still isn't doing what you intend. Until it does, timing it makes little sense :-) > I am pretty sure about this. Note that I'm talking about the Python code here, the code you run through timeit. You cannot have checked the results of running _that_ specific code, because it doesn't work at all. You may have checked _other_ code many times. We may get to that later, but since I speak Python, I'm not going to understand what you're doing until we have Python code that works ;-) From p.f.moore at gmail.com Tue Mar 10 11:35:06 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 10 Mar 2015 10:35:06 +0000 Subject: [Python-Dev] Windows installer - File associations in "per user" installs Message-ID: On 9 March 2015 at 15:37, Steve Dower wrote: >> Maybe the answer is that we simply start recommending that everyone on Windows >> uses per-user installs. It makes little difference to me (beyond the fact that >> when I want to look at the source of something in the stdlib, the location of >> the file is a lot harder to remember than C:\Apps\Python34\Lib\whatever.py) but >> I doubt it's what most people will expect. > > I'm okay with this. Installing for all users is really something that could be considered an advanced option rather than the default, especially since the aim (AIUI) of the all-users install is to pretend that Python was shipped with the OS. (I'd kind of like to take that further by splitting things more sensibly between Program Files, Common Files and System32, but there's very little gain from that and much MUCH pain as long as people are still expecting C:\PythonXY installs...) I've just tried a per-user install of Python 3.5a2. The machine in question previously had (and still has) a system install of 3.4, with "Make this Python the default" selected (so the .py extension is associated with that version and specifically the 3.4 launcher). I didn't get the option to associate .py files with 3.5 (there's *no way* I'd consider that to be advanced usage - if I'm installing Python, why wouldn't I want to associate it with .py files [1]) and I still seem to have .py associated with the 3.4 launcher, not the 3.5 one that's in my %APPDATA% folder. >cmd /c assoc .py .py=Python.File >cmd /c ftype python.file python.file="C:\WINDOWS\py.exe" "%1" %* I'm happy if a per-user install of 3.5 makes a per-user filetype association (assuming such a thing is possible, I've never tried it before) but it's absolutely not OK if we're planning on recommending an install type that doesn't create the right association. Paul [1] Given that I have 3.4 and am installing an experimental 3.5 version, it's not actually at all clear cut which version I want as my default. In all honesty, I don't think this decision is actually something that should be defaulted. Maybe the "don't make the user make any choices in the default selection" approach has gone a little too far here? From Steve.Dower at microsoft.com Tue Mar 10 14:09:55 2015 From: Steve.Dower at microsoft.com (Steve Dower) Date: Tue, 10 Mar 2015 13:09:55 +0000 Subject: [Python-Dev] Windows installer - File associations in "per user" installs In-Reply-To: References: Message-ID: It's a bug. File and assign to me please. Top-posted from my Windows Phone ________________________________ From: Paul Moore Sent: ?3/?10/?2015 3:35 To: Steve Dower Cc: Python Dev Subject: Windows installer - File associations in "per user" installs On 9 March 2015 at 15:37, Steve Dower wrote: >> Maybe the answer is that we simply start recommending that everyone on Windows >> uses per-user installs. It makes little difference to me (beyond the fact that >> when I want to look at the source of something in the stdlib, the location of >> the file is a lot harder to remember than C:\Apps\Python34\Lib\whatever.py) but >> I doubt it's what most people will expect. > > I'm okay with this. Installing for all users is really something that could be considered an advanced option rather than the default, especially since the aim (AIUI) of the all-users install is to pretend that Python was shipped with the OS. (I'd kind of like to take that further by splitting things more sensibly between Program Files, Common Files and System32, but there's very little gain from that and much MUCH pain as long as people are still expecting C:\PythonXY installs...) I've just tried a per-user install of Python 3.5a2. The machine in question previously had (and still has) a system install of 3.4, with "Make this Python the default" selected (so the .py extension is associated with that version and specifically the 3.4 launcher). I didn't get the option to associate .py files with 3.5 (there's *no way* I'd consider that to be advanced usage - if I'm installing Python, why wouldn't I want to associate it with .py files [1]) and I still seem to have .py associated with the 3.4 launcher, not the 3.5 one that's in my %APPDATA% folder. >cmd /c assoc .py .py=Python.File >cmd /c ftype python.file python.file="C:\WINDOWS\py.exe" "%1" %* I'm happy if a per-user install of 3.5 makes a per-user filetype association (assuming such a thing is possible, I've never tried it before) but it's absolutely not OK if we're planning on recommending an install type that doesn't create the right association. Paul [1] Given that I have 3.4 and am installing an experimental 3.5 version, it's not actually at all clear cut which version I want as my default. In all honesty, I don't think this decision is actually something that should be defaulted. Maybe the "don't make the user make any choices in the default selection" approach has gone a little too far here? -------------- next part -------------- An HTML attachment was scrubbed... URL: From jimjjewett at gmail.com Tue Mar 10 17:01:31 2015 From: jimjjewett at gmail.com (Jim J. Jewett) Date: Tue, 10 Mar 2015 09:01:31 -0700 (PDT) Subject: [Python-Dev] Thoughts on running Python 3.5 on Windows (path, pip install --user, etc) In-Reply-To: Message-ID: <54ff155b.c9298c0a.23d8.41a0@mx.google.com> On 10 March 2015 at slightly after midnight. Paul Moore wrote: > Personally I doubt it would make much difference. If the docs say > "pygmentize" I'm unlikely to dig around to find that the incantation > "python -m pygments.somemodule:main" does the same thing using 3 times > as many characters. I'd just add Python to my PATH and say stuff it. There is value in getting the incantation down to a single (preferably short) line, because then it can be used as a shortcut. That means it can be created as a shortcut at installation time, and that someone writing their own batch file can just cut and paste from the shortcut properties' target. Not as simple as just adding to the path, but simpler than adding several directories to the path, or modifying other environment variables, or fighting an existing but conflicting python installation already on the path. -jJ -- If there are still threading problems with my replies, please email me with details, so that I can try to resolve them. -jJ From tim.peters at gmail.com Tue Mar 10 18:22:58 2015 From: tim.peters at gmail.com (Tim Peters) Date: Tue, 10 Mar 2015 12:22:58 -0500 Subject: [Python-Dev] Tunning binary insertion sort algorithm in Timsort. In-Reply-To: References: <7750938a2fea2.54fdd09e@wiscmail.wisc.edu> <76e0c70929f48.54fdd0da@wiscmail.wisc.edu> <76f094a32c8ca.54fdd119@wiscmail.wisc.edu> <7740f7e62cf4b.54fdd155@wiscmail.wisc.edu> <778088e82a599.54fdd192@wiscmail.wisc.edu> <7740ebbf29f9e.54fdd2be@wiscmail.wisc.edu> <76f0f50b2c800.54fdd2fb@wiscmail.wisc.edu> <7710a1122c5ba.54fdd337@wiscmail.wisc.edu> <7780f4192fd2c.54fdd373@wiscmail.wisc.edu> <7790d04c299a7.54fdd590@wiscmail.wisc.edu> <7710a1822e0a9.54fdd5cc@wiscmail.wisc.edu> <7720dc73289e8.54fdd609@wiscmail.wisc.edu> <7790de4c2be2c.54fdd645@wiscmail.wisc.edu> <76f0eced29ac4.54fd9476@wiscmail.wisc.edu> Message-ID: OK - here's what the current binsort does, ignoring that it skips an already-sorted prefix (if any), and creating a new list instead of modifying in-place: def oldsort(a): from bisect import bisect_right as br assert a result = [a[0]] for i in range(1, len(a)): x = a[i] index = br(result, x) result.insert(index, x) return result And here's my best guess as to what you _intend_ the new version to do. Please confirm that, or, if I'm guessing wrong, please give a Python function that _does_ implement your intent: def newsort(a): from bisect import bisect_right as br assert a oldx = a[0] result = [oldx] index = 0 for i in range(1, len(a)): x = a[i] if x < oldx: index = br(result, x, 0, index) else: index = br(result, x, index + 1) result.insert(index, x) oldx = x return result Now assuming that's right, I don't care about timing it ;-) The only basic question to me is whether it in fact reduces the number of comparisons. So here's an integer wrapper that bumps a global counter whenever it's asked to compare: class IntWrap(object): def __init__(self, i): self.i = i def __cmp__(a, b): global gncmp gncmp += 1 return cmp(a.i, b.i) def __repr__(self): return repr(self.i) Now we can build lists containing that, and get exact comparison counts. To start, for a given length `n`, this counts the total number of comparisons needed to sort all possible permutations of a list of length n, under both the old and new ways: def drive(n): import itertools global gncmp base = [IntWrap(i) for i in range(n)] oldcount = newcount = 0 numperms = 0 for p in itertools.permutations(base): numperms += 1 gncmp = 0 oldresult = oldsort(p) oldcount += gncmp gncmp = 0 newresult = newsort(p) newcount += gncmp assert oldresult == newresult == base print 'n', n, 'perms', numperms print 'old compares', oldcount print 'new compares', newcount print 'diff %', (newcount - oldcount) * 100.0 / max(oldcount, 1) And, finally, a tiny loop to drive it: for n in range(1, 13): print drive(n) It's still running as I type this, but the results aren't promising so far - as soon as the list length gets non-trivial, the new way requires more comparisons than the old way so far: n 1 perms 1 old compares 0 new compares 0 diff % 0.0 n 2 perms 2 old compares 2 new compares 2 diff % 0.0 n 3 perms 6 old compares 16 new compares 16 diff % 0.0 n 4 perms 24 old compares 112 new compares 116 diff % 3.57142857143 n 5 perms 120 old compares 848 new compares 880 diff % 3.77358490566 n 6 perms 720 old compares 7008 new compares 7296 diff % 4.1095890411 n 7 perms 5040 old compares 63456 new compares 66432 diff % 4.68986384266 n 8 perms 40320 old compares 628608 new compares 662496 diff % 5.39095907147 n 9 perms 362880 old compares 6826752 new compares 7202304 diff % 5.50118123523 n 10 perms 3628800 old compares 80605440 new compares 85006080 diff % 5.45948263542 I believe it would be very difficult to analyze this rigorously - and even if I saw an analysis it would be hard to trust it. Raw counts from simple code are hard to argue with ;-) FYI, here are two ideas I had way back when, but didn't pursue: 1. Merge "2 at a time" instead of just 1. That is, first "sort" the next 2 elements to be merged (1 compare and a possible swap). Then binary search to find where the smaller belongs, and a shorter binary search to find where the larger belongs. Then shift both into place. This can win on two counts: A. Less data movement, since the already-sorted values after the larger element get moved only once instead of twice. B. A possible cut in number of compares. Merging a sorted list of N elements with a sorted list of 2 elements has been studied a lot (e.g., search for "optimal merging of 2 elements" and find the paper by Hwang and Lin). The minimum average theoretical number of compares needed is ceiling(log2((N+2)*(N+1)/2)). 2. Instead of binary insertion sort, do an ordinary (but optimized) bottom-up merge sort. That may not cut the number of compares, but would slash worst-case data movement cost from O(n**2) to O(n*log(n)). As to why your code is sometimes faster, for the Python code in your timing harness, well, that didn't actually sort anything, so wasn't measuring anything interesting (or even explainable ;-) ). For the Java code, I have no guess - I don't know enough about Java internals. Maybe "lucky" data, maybe cache effects, maybe a mistake - don't know, and can't guess. Or maybe my guess (above) at the intent of your code is all wrong. It _is_ an interesting idea, though! Thanks for bringing it up :-) From lox.xiao at gmail.com Tue Mar 10 18:27:21 2015 From: lox.xiao at gmail.com (lou xiao) Date: Wed, 11 Mar 2015 01:27:21 +0800 Subject: [Python-Dev] str.lstrip bug? Message-ID: I find a bug in str.lstrip, when i call str.lstrip, i get this result. tiny? ~ python Python 2.7.5+ (default, Feb 27 2014, 19:37:08) [GCC 4.8.1] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> a='device_info' >>> a.lstrip('device') '_info' >>> a.lstrip('device_') 'nfo' >>> tiny? ~ uname -a Linux tinymint 3.11.0-12-generic #19-Ubuntu SMP Wed Oct 9 16:20:46 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux tiny? ~ -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Tue Mar 10 18:33:44 2015 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 11 Mar 2015 04:33:44 +1100 Subject: [Python-Dev] str.lstrip bug? In-Reply-To: References: Message-ID: On Wed, Mar 11, 2015 at 4:27 AM, lou xiao wrote: > I find a bug in str.lstrip, when i call str.lstrip, i get this result. > > tiny? ~ python > Python 2.7.5+ (default, Feb 27 2014, 19:37:08) > [GCC 4.8.1] on linux2 > Type "help", "copyright", "credits" or "license" for more information. >>>> a='device_info' >>>> a.lstrip('device') > '_info' >>>> a.lstrip('device_') > 'nfo' >>>> > tiny? ~ uname -a > Linux tinymint 3.11.0-12-generic #19-Ubuntu SMP Wed Oct 9 16:20:46 UTC 2013 > x86_64 x86_64 x86_64 GNU/Linux > tiny? ~ It's not a bug, because it isn't doing what you think it is. It strips a *set of characters*, not a prefix string. https://docs.python.org/2/library/stdtypes.html#str.lstrip https://docs.python.org/3/library/stdtypes.html#str.lstrip ChrisA From facundobatista at gmail.com Tue Mar 10 18:34:26 2015 From: facundobatista at gmail.com (Facundo Batista) Date: Tue, 10 Mar 2015 14:34:26 -0300 Subject: [Python-Dev] str.lstrip bug? In-Reply-To: References: Message-ID: On Tue, Mar 10, 2015 at 2:27 PM, lou xiao wrote: > tiny? ~ python > Python 2.7.5+ (default, Feb 27 2014, 19:37:08) > [GCC 4.8.1] on linux2 > Type "help", "copyright", "credits" or "license" for more information. >>>> a='device_info' >>>> a.lstrip('device') > '_info' >>>> a.lstrip('device_') > 'nfo' On one hand, this is the "development of python itself" list; this issue was more aimed to the general python list, of you was sure that this is a real bug, to the issue tracker. On the other hand, this is not a bug! If you pass a parameter to lstrip it will (quoted from its help) "remove characters in chars instead.", so the moment you pass "device_", it removes all those characers from the left... note that the 'i' is removed twice. Regards, -- . Facundo Blog: http://www.taniquetil.com.ar/plog/ PyAr: http://www.python.org/ar/ Twitter: @facundobatista From lox.xiao at gmail.com Tue Mar 10 18:43:12 2015 From: lox.xiao at gmail.com (lou xiao) Date: Wed, 11 Mar 2015 01:43:12 +0800 Subject: [Python-Dev] str.lstrip bug? In-Reply-To: References: Message-ID: ths, i misunderstood the method 2015-03-11 1:33 GMT+08:00 Chris Angelico : > On Wed, Mar 11, 2015 at 4:27 AM, lou xiao wrote: > > I find a bug in str.lstrip, when i call str.lstrip, i get this result. > > > > tiny? ~ python > > Python 2.7.5+ (default, Feb 27 2014, 19:37:08) > > [GCC 4.8.1] on linux2 > > Type "help", "copyright", "credits" or "license" for more information. > >>>> a='device_info' > >>>> a.lstrip('device') > > '_info' > >>>> a.lstrip('device_') > > 'nfo' > >>>> > > tiny? ~ uname -a > > Linux tinymint 3.11.0-12-generic #19-Ubuntu SMP Wed Oct 9 16:20:46 UTC > 2013 > > x86_64 x86_64 x86_64 GNU/Linux > > tiny? ~ > > It's not a bug, because it isn't doing what you think it is. It strips > a *set of characters*, not a prefix string. > > https://docs.python.org/2/library/stdtypes.html#str.lstrip > https://docs.python.org/3/library/stdtypes.html#str.lstrip > > ChrisA > -------------- next part -------------- An HTML attachment was scrubbed... URL: From me at the-compiler.org Tue Mar 10 18:34:34 2015 From: me at the-compiler.org (Florian Bruhin) Date: Tue, 10 Mar 2015 18:34:34 +0100 Subject: [Python-Dev] str.lstrip bug? In-Reply-To: References: Message-ID: <20150310173434.GO11094@tonks> * lou xiao [2015-03-11 01:27:21 +0800]: > I find a bug in str.lstrip, when i call str.lstrip, i get this result. You're misunderstanding how str.strip works. It takes a set of characters and removes them all from the beginning: >>> "abababcd".lstrip('ab') >>> 'cd' Florian -- http://www.the-compiler.org | me at the-compiler.org (Mail/XMPP) GPG: 916E B0C8 FD55 A072 | http://the-compiler.org/pubkey.asc I love long mails! | http://email.is-not-s.ms/ -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 819 bytes Desc: not available URL: From stephen at xemacs.org Tue Mar 10 19:42:20 2015 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Wed, 11 Mar 2015 03:42:20 +0900 Subject: [Python-Dev] str.lstrip bug? In-Reply-To: References: Message-ID: <87ioe87kz7.fsf@uwakimon.sk.tsukuba.ac.jp> lou xiao writes: > I find a bug in str.lstrip, when i call str.lstrip, i get this result. > >>> a.lstrip('device_') > 'nfo' > >>> Try a.lstrip('_ecived') You'll see that you get the same result. I suspect that you misunderstand the meaning of the argument, which is not a sequence of characters, but a *set* of characters. From ischwabacher at wisc.edu Tue Mar 10 18:57:41 2015 From: ischwabacher at wisc.edu (Isaac Schwabacher) Date: Tue, 10 Mar 2015 12:57:41 -0500 Subject: [Python-Dev] str.lstrip bug? In-Reply-To: <776081621755c1.54ff3070@wiscmail.wisc.edu> References: <747088be171788.54ff2d5e@wiscmail.wisc.edu> <7760d6ad177272.54ff2d9b@wiscmail.wisc.edu> <7570adf21732e2.54ff2e4f@wiscmail.wisc.edu> <7640b7c6171335.54ff2e8c@wiscmail.wisc.edu> <7640fb6f174491.54ff2ec8@wiscmail.wisc.edu> <748099b41731c5.54ff2f04@wiscmail.wisc.edu> <7570b96e1706b0.54ff2f41@wiscmail.wisc.edu> <7480fc3f17687a.54ff2f7d@wiscmail.wisc.edu> <7750a881177e82.54ff2fba@wiscmail.wisc.edu> <75809c6c17326a.54ff2ff6@wiscmail.wisc.edu> <7760d124172a35.54ff3033@wiscmail.wisc.edu> <776081621755c1.54ff3070@wiscmail.wisc.edu> Message-ID: <7490a88c171d64.54feea45@wiscmail.wisc.edu> On 15-03-10, Facundo Batista wrote: > On Tue, Mar 10, 2015 at 2:27 PM, lou xiao wrote: > > > tiny? ~ python > > Python 2.7.5+ (default, Feb 27 2014, 19:37:08) > > [GCC 4.8.1] on linux2 > > Type "help", "copyright", "credits" or "license" for more information. > >>>> a='device_info' > >>>> a.lstrip('device') > > '_info' > >>>> a.lstrip('device_') > > 'nfo' > > On one hand, this is the "development of python itself" list; this > issue was more aimed to the general python list, of you was sure that > this is a real bug, to the issue tracker. > > On the other hand, this is not a bug! If you pass a parameter to > lstrip it will (quoted from its help) "remove characters in chars > instead.", so the moment you pass "device_", it removes all those > characers from the left... note that the 'i' is removed twice. That said, I bet this is the most common string-munging operations that *isn't* available as a single function in the stdlib. I know my bash code is full of ${filename%.suffix} and such, and the fact that in python I have to either import re or resort to some combination of (starts|ends)with, r?partition, slicing and an if-clause makes that sort of code much more verbose and harder to read. Pathlib's Path.with_suffix helps in some but not all of these cases. Maybe the stdlib should have a simple way to do this? It could even be added as a kwarg (exact=False) to str.[lr]strip to minimize the effect on the API; alternatively it could be str.strip(prefix|suffix). ijs > Regards, > > -- > . Facundo > > Blog: http://www.taniquetil.com.ar/plog/ > PyAr: http://www.python.org/ar/ > Twitter: @facundobatista > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ischwabacher%40wisc.edu From ethan at stoneleaf.us Tue Mar 10 20:12:40 2015 From: ethan at stoneleaf.us (Ethan Furman) Date: Tue, 10 Mar 2015 12:12:40 -0700 Subject: [Python-Dev] str.lstrip bug? In-Reply-To: <7490a88c171d64.54feea45@wiscmail.wisc.edu> References: <747088be171788.54ff2d5e@wiscmail.wisc.edu> <7760d6ad177272.54ff2d9b@wiscmail.wisc.edu> <7570adf21732e2.54ff2e4f@wiscmail.wisc.edu> <7640b7c6171335.54ff2e8c@wiscmail.wisc.edu> <7640fb6f174491.54ff2ec8@wiscmail.wisc.edu> <748099b41731c5.54ff2f04@wiscmail.wisc.edu> <7570b96e1706b0.54ff2f41@wiscmail.wisc.edu> <7480fc3f17687a.54ff2f7d@wiscmail.wisc.edu> <7750a881177e82.54ff2fba@wiscmail.wisc.edu> <75809c6c17326a.54ff2ff6@wiscmail.wisc.edu> <7760d124172a35.54ff3033@wiscmail.wisc.edu> <776081621755c1.54ff3070@wiscmail.wisc.edu> <7490a88c171d64.54feea45@wiscmail.wisc.edu> Message-ID: <54FF4228.8070802@stoneleaf.us> On 03/10/2015 10:57 AM, Isaac Schwabacher wrote: > On 15-03-10, Facundo Batista wrote: >> On Tue, Mar 10, 2015 at 2:27 PM, lou xiao wrote: >> >>> tiny? ~ python >>> Python 2.7.5+ (default, Feb 27 2014, 19:37:08) >>> [GCC 4.8.1] on linux2 >>> Type "help", "copyright", "credits" or "license" for more information. >>> >>> a='device_info' >>> >>> a.lstrip('device') >>> '_info' >>> >>> a.lstrip('device_') >>> 'nfo' >> >> On one hand, this is the "development of python itself" list; this >> issue was more aimed to the general python list, of you was sure that >> this is a real bug, to the issue tracker. >> >> On the other hand, this is not a bug! If you pass a parameter to >> lstrip it will (quoted from its help) "remove characters in chars >> instead.", so the moment you pass "device_", it removes all those >> characers from the left... note that the 'i' is removed twice. > > That said, I bet this is the most common string-munging operations that *isn't* available as a single function in the stdlib. I know my bash code is full of ${filename%.suffix} and such, and the fact that in python I have to either import re or resort to some combination of (starts|ends)with, r?partition, slicing and an if-clause makes that sort of code much more verbose and harder to read. Pathlib's Path.with_suffix helps in some but not all of these cases. Maybe the stdlib should have a simple way to do this? It could even be added as a kwarg (exact=False) to str.[lr]strip to minimize the effect on the API; alternatively it could be str.strip(prefix|suffix). Nice way to bring it on-topic! And yes, that would be nice. -- ~Ethan~ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: OpenPGP digital signature URL: From ischwabacher at wisc.edu Tue Mar 10 21:32:03 2015 From: ischwabacher at wisc.edu (Isaac Schwabacher) Date: Tue, 10 Mar 2015 15:32:03 -0500 Subject: [Python-Dev] str.lstrip bug? In-Reply-To: <75f0bdb5173bcb.54ff544b@wiscmail.wisc.edu> References: <747088be171788.54ff2d5e@wiscmail.wisc.edu> <7760d6ad177272.54ff2d9b@wiscmail.wisc.edu> <7570adf21732e2.54ff2e4f@wiscmail.wisc.edu> <7640b7c6171335.54ff2e8c@wiscmail.wisc.edu> <7640fb6f174491.54ff2ec8@wiscmail.wisc.edu> <748099b41731c5.54ff2f04@wiscmail.wisc.edu> <7570b96e1706b0.54ff2f41@wiscmail.wisc.edu> <7480fc3f17687a.54ff2f7d@wiscmail.wisc.edu> <7750a881177e82.54ff2fba@wiscmail.wisc.edu> <75809c6c17326a.54ff2ff6@wiscmail.wisc.edu> <7760d124172a35.54ff3033@wiscmail.wisc.edu> <776081621755c1.54ff3070@wiscmail.wisc.edu> <7490a88c171d64.54feea45@wiscmail.wisc.edu> <54FF4228.8070802@stoneleaf.us> <75f0bdb5173bcb.54ff544b@wiscmail.wisc.edu> Message-ID: <772082aa1711d1.54ff0e73@wiscmail.wisc.edu> On 15-03-10, Ethan Furman wrote: > On 03/10/2015 10:57 AM, Isaac Schwabacher wrote: > > On 15-03-10, Facundo Batista wrote: > >> On Tue, Mar 10, 2015 at 2:27 PM, lou xiao wrote: > >> > >>> tiny? ~ python > >>> Python 2.7.5+ (default, Feb 27 2014, 19:37:08) > >>> [GCC 4.8.1] on linux2 > >>> Type "help", "copyright", "credits" or "license" for more information. > >>> >>> a='device_info' > >>> >>> a.lstrip('device') > >>> '_info' > >>> >>> a.lstrip('device_') > >>> 'nfo' > >> > >> On one hand, this is the "development of python itself" list; this > >> issue was more aimed to the general python list, of you was sure that > >> this is a real bug, to the issue tracker. > >> > >> On the other hand, this is not a bug! If you pass a parameter to > >> lstrip it will (quoted from its help) "remove characters in chars > >> instead.", so the moment you pass "device_", it removes all those > >> characers from the left... note that the 'i' is removed twice. > > > > That said, I bet this is the most common string-munging operations that *isn't* available as a single function in the stdlib. I know my bash code is full of ${filename%.suffix} and such, and the fact that in python I have to either import re or resort to some combination of (starts|ends)with, r?partition, slicing and an if-clause makes that sort of code much more verbose and harder to read. Pathlib's Path.with_suffix helps in some but not all of these cases. Maybe the stdlib should have a simple way to do this? It could even be added as a kwarg (exact=False) to str.[lr]strip to minimize the effect on the API; alternatively it could be str.strip(prefix|suffix). > > Nice way to bring it on-topic! > > And yes, that would be nice. Yes. When you are converting a code base from bash to python, you *really* don't want python to push back. > -- > ~Ethan~ From arigo at tunes.org Wed Mar 11 09:26:12 2015 From: arigo at tunes.org (Armin Rigo) Date: Wed, 11 Mar 2015 09:26:12 +0100 Subject: [Python-Dev] Tunning binary insertion sort algorithm in Timsort. In-Reply-To: References: <7750938a2fea2.54fdd09e@wiscmail.wisc.edu> <76e0c70929f48.54fdd0da@wiscmail.wisc.edu> <76f094a32c8ca.54fdd119@wiscmail.wisc.edu> <7740f7e62cf4b.54fdd155@wiscmail.wisc.edu> <778088e82a599.54fdd192@wiscmail.wisc.edu> <7740ebbf29f9e.54fdd2be@wiscmail.wisc.edu> <76f0f50b2c800.54fdd2fb@wiscmail.wisc.edu> <7710a1122c5ba.54fdd337@wiscmail.wisc.edu> <7780f4192fd2c.54fdd373@wiscmail.wisc.edu> <7790d04c299a7.54fdd590@wiscmail.wisc.edu> <7710a1822e0a9.54fdd5cc@wiscmail.wisc.edu> <7720dc73289e8.54fdd609@wiscmail.wisc.edu> <7790de4c2be2c.54fdd645@wiscmail.wisc.edu> <76f0eced29ac4.54fd9476@wiscmail.wisc.edu> Message-ID: Hi Tim, On 10 March 2015 at 18:22, Tim Peters wrote: > 1. Merge "2 at a time" instead of just 1. That is, first "sort" the > next 2 elements to be merged (1 compare and a possible swap). Then > binary search to find where the smaller belongs, and a shorter binary > search to find where the larger belongs. Then shift both into place. Good idea, but when I tried that it seemed to increase the total number of comparisons (on random inputs with only up to 136 items). The increase is on the order of 5%. I'm not sure reduced data movement can compensate for that in Python. Test and code available here: https://bitbucket.org/arigo/arigo/src/default/hack/pypy-hack/list_sort/ The change to insert two items at a time is here: https://bitbucket.org/arigo/arigo/commits/68e04d143dc242cfd9e3934451321f685a68a8e2 (This is taken straight from PyPy's code.) A bient?t, Armin. From storchaka at gmail.com Wed Mar 11 12:25:39 2015 From: storchaka at gmail.com (Serhiy Storchaka) Date: Wed, 11 Mar 2015 13:25:39 +0200 Subject: [Python-Dev] Not documented special methods Message-ID: There are many special names used in Python core and the stdlib, but absent in the documentation index [1]. If you see names that are defined or used in the module or area maintained by you, please add references to these names to the index and document them if it is needed. Repeat the lists here. Module level names used in pydoc: __author__ __credits__ __date__ __version__ Module level name used in doctest: __test__ Other module level names: __about__ (heapq only) __copyright__ (many modules) __cvsid__ (tarfile only) __docformat__ (doctest only) __email__ (test_with and test_keywordonlyarg only) __libmpdec_version__ (decimal only) __status__ (logging only) type attributes (mostly used in tests): __abstractmethods__ (used in abc, functools) __base__ __basicsize__ __dictoffset__ __flags__ (used in inspect, copyreg) __itemsize__ __weakrefoffset__ super() attributes: __self_class__ __thisclass__ Used in sqlite: __adapt__ __conform__ Used in ctypes: __ctype_be__ __ctype_le__ __ctypes_from_outparam__ Used in unittest: __unittest_expecting_failure__ __unittest_skip__ __unittest_skip_why__ float methods, for testing: __getformat__ __setformat__ Used in IDLE RPC: __attributes__ __methods__ Others: __alloc__ (bytearray method) __args__ (used in bdb) __build_class__ (builtins function, used in eval loop) __builtins__ (module attribute) __decimal_context__ (used in decimal) __exception__ (used in pdb) __getinitargs__ (used in pickle, datetime) __initializing__ (used in importlib) __isabstractmethod__ (function/method/descriptor attribute, used in abc, functools, types) __ltrace__ (used in eval loop, never set) __members__ (Enum attribute, used in many modules) __mp_main__ (used in multiprocessing) __new_member__ (Enum attribute, used in enum internally) __newobj__ (copyreg function, used in pickle, object.__reduce_ex__) __newobj_ex__ (copyreg function, used in pickle, object.__reduce_ex__) __objclass__ (descriptor/enum attribute, used in inspect, pydoc, doctest, multiprocessing) __prepare__ (metaclass method, used in builtins.__build_class__, types) __pycache__ (cache directory name) __return__ (used in pdb) __signature__ (used in inspect, never set) __sizeof__ (standard method, used in sys.getsizeof) __slotnames__ (used in object.__getstate__ for caching) __text_signature__ (function/method/descriptor attribute, used in inspect) __trunc__ (used in math.trunc, int, etc) __warningregistry__ (used in warnings) __weakref__ (used in weakref) __wrapped__ (used in inspect, functools, contextlib, asyncio) [1] http://bugs.python.org/issue23639 From brett at python.org Wed Mar 11 18:27:58 2015 From: brett at python.org (Brett Cannon) Date: Wed, 11 Mar 2015 17:27:58 +0000 Subject: [Python-Dev] libffi embedded in CPython In-Reply-To: <549891df.c332e00a.54c7.34f8@mx.google.com> References: <549891df.c332e00a.54c7.34f8@mx.google.com> Message-ID: On Mon, Dec 22, 2014 at 4:49 PM Jim J. Jewett wrote: > > > On Thu, Dec 18, 2014, at 14:13, Maciej Fijalkowski wrote: > > ... http://bugs.python.org/issue23085 ... > > is there any reason any more for libffi being included in CPython? > > > Paul Moore wrote: > > Probably the easiest way of moving this forward would be for someone > > to identify the CPython-specific patches in the current version ... > > Christian Heimes wrote: > > That's easy. All patches are tracked in the diff file > > https://hg.python.org/cpython/file/3de678cd184d/Modules/_ > ctypes/libffi.diff > > That (200+ lines) doesn't seem to have all the C changes, such as the > win64 sizeof changes from issue 11835. > > Besides http://bugs.python.org/issue23085, there is at least > http://bugs.python.org/issue22733 > http://bugs.python.org/issue20160 > http://bugs.python.org/issue11835 > > which sort of drives home the point that making sure we have a > good merge isn't trivial, and this isn't an area where we should > just assume that tests will catch everything. I don't think it > is just a quicky waiting on permission. > > I've no doubt that upstream libffi is better in many ways, but > those are ways people have already learned to live with. > > That said, I haven't seen any objections in principle, except > perhaps from Steve Dower in the issues. (I *think* he was > just saying "not worth the time to me", but it was ambiguous.) > > I do believe that Christian or Maciej *could* sort things out well > enough; I have no insight into whether they have (or someone else > has) the time to actually do so. > Did anyone ever step forward to do this? I'm a bit worried about the long-term viability of ctypes if we don't have a maintainer or at least someone making sure we are staying up-to-date with upstream libffi. The ctypes module is a dangerous thing, so having a chunk of C code that isn't being properly maintained seems to me to make it even more dangerous. I'm going to propose a somewhat controversial idea: let's deprecate the ctypes module. We now have things like cffi and Cython for people who need to interface with C code. Both of those projects are maintained. And they are not overly difficult to work with. All of this seems to match the reasons we added ctypes in the first place while also being safer to use than ctypes. And I'm not saying we need to remove it in Python 3.6 or something. But I think it would be wise to deprecate the module to promote people to use third-party solutions to interfacing with C code and to eventually remove the maintenance burden at some point when we clear out all of our deprecated cruft (I call that Python 4, you can call it "some day way in the future" if you prefer). -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Wed Mar 11 18:34:10 2015 From: brett at python.org (Brett Cannon) Date: Wed, 11 Mar 2015 17:34:10 +0000 Subject: [Python-Dev] PEP 488: elimination of PYO files In-Reply-To: <1425685751.3874542.236704353.11A94396@webmail.messagingengine.com> References: <54F9EBAF.1080400@hotpy.org> <1425668955.3364036.236612829.494092A7@webmail.messagingengine.com> <1425685751.3874542.236704353.11A94396@webmail.messagingengine.com> Message-ID: On Fri, Mar 6, 2015 at 6:49 PM Benjamin Peterson wrote: > > > On Fri, Mar 6, 2015, at 15:11, Brett Cannon wrote: > > > > OK, but that doesn't influence the PEP's goal of dropping .pyo files. > > Correct. > > > > > Are you suggesting that the tag be changed to be less specific to > > optimizations and more free-form? Like > > `importlib.cpython-35.__no-asserts_no-docstrings__.pyc`? Otherwise stuff > > like this gets baked into the .pyc file itself instead of the file name, > > but I don't think we should just drop the ability to switch off asserts > > and > > docstrings like Mark seemed to be suggesting. > > Basically, though the filename strings could perhaps be more compact. > I have a poll going on G+ to see what people think of the various proposed file name formats at https://plus.google.com/u/0/+BrettCannon/posts/fZynLNwHWGm . Feel free to vote if you have an opinion. -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Wed Mar 11 18:50:28 2015 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 11 Mar 2015 18:50:28 +0100 Subject: [Python-Dev] libffi embedded in CPython References: <549891df.c332e00a.54c7.34f8@mx.google.com> Message-ID: <20150311185028.6f05973c@fsol> On Wed, 11 Mar 2015 17:27:58 +0000 Brett Cannon wrote: > > Did anyone ever step forward to do this? I'm a bit worried about the > long-term viability of ctypes if we don't have a maintainer or at least > someone making sure we are staying up-to-date with upstream libffi. The > ctypes module is a dangerous thing, so having a chunk of C code that isn't > being properly maintained seems to me to make it even more dangerous. Depends what you call "dangerous". C code doesn't rot quicker than pure Python code :-) Also, libffi really offers a wrapper around platform ABIs, which rarely change. > I'm going to propose a somewhat controversial idea: let's deprecate the > ctypes module. This is gratuitous. > We now have things like cffi and Cython for people who need > to interface with C code. Both of those projects are maintained. And they > are not overly difficult to work with. But they are not ctypes. For example, cffi wouldn't be obvious to use for interfacing with non-C code, since it requires you to write C-like declarations. > All of this seems to match the > reasons we added ctypes in the first place while also being safer to use > than ctypes. I don't understand why cffi would be safer than ctypes. At least not in the operation mode where it doesn't need to invoke a C compiler. Cython is a completely different beast, it requires a separate compilation pass which makes it useless in some situations. Regards Antoine. From fijall at gmail.com Wed Mar 11 18:54:58 2015 From: fijall at gmail.com (Maciej Fijalkowski) Date: Wed, 11 Mar 2015 19:54:58 +0200 Subject: [Python-Dev] libffi embedded in CPython In-Reply-To: <20150311185028.6f05973c@fsol> References: <549891df.c332e00a.54c7.34f8@mx.google.com> <20150311185028.6f05973c@fsol> Message-ID: On Wed, Mar 11, 2015 at 7:50 PM, Antoine Pitrou wrote: > On Wed, 11 Mar 2015 17:27:58 +0000 > Brett Cannon wrote: >> >> Did anyone ever step forward to do this? I'm a bit worried about the >> long-term viability of ctypes if we don't have a maintainer or at least >> someone making sure we are staying up-to-date with upstream libffi. The >> ctypes module is a dangerous thing, so having a chunk of C code that isn't >> being properly maintained seems to me to make it even more dangerous. > > Depends what you call "dangerous". C code doesn't rot quicker than pure > Python code :-) Also, libffi really offers a wrapper around platform > ABIs, which rarely change. And yet, lesser known ABIs in libffi contain bugs (as we discovered trying to work there with anything else than x86 really). Also there *are* ABI differencies that change slowly over time (e.g. requiring stack to be 16 byte aligned) > >> I'm going to propose a somewhat controversial idea: let's deprecate the >> ctypes module. > > This is gratuitous. I'm +1 on deprecating ctypes > >> We now have things like cffi and Cython for people who need >> to interface with C code. Both of those projects are maintained. And they >> are not overly difficult to work with. > > But they are not ctypes. For example, cffi wouldn't be obvious to use > for interfacing with non-C code, since it requires you to write C-like > declarations. You mean like Fortran? Or what precisely? > I don't understand why cffi would be safer than ctypes. At least not in > the operation mode where it doesn't need to invoke a C compiler. > Cython is a completely different beast, it requires a separate > compilation pass which makes it useless in some situations. > Our main motivation for "safer" is "comes with less magic and less gotchas", which also means "does less stuff". It's also smaller. From p.f.moore at gmail.com Wed Mar 11 18:56:14 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 11 Mar 2015 17:56:14 +0000 Subject: [Python-Dev] libffi embedded in CPython In-Reply-To: References: <549891df.c332e00a.54c7.34f8@mx.google.com> Message-ID: On 11 March 2015 at 17:27, Brett Cannon wrote: > I'm going to propose a somewhat controversial idea: let's deprecate the > ctypes module. We now have things like cffi and Cython for people who need > to interface with C code. Both of those projects are maintained. And they > are not overly difficult to work with. All of this seems to match the > reasons we added ctypes in the first place while also being safer to use > than ctypes. -1. On Windows a huge amount of code uses ctypes to interface with the Windows API. Many of those projects transitioned from pywin32 because that is a big dependency, and a binary build. Not having in-core access to the Windows API would be a huge problem for many projects. Who would rewrite projects like pyreadline or colorama, for example? How would pip (which has to vendor everything and can't include binary dependencies because of its nature) locate the correct windows config file folders without ctypes in core? Paul From solipsis at pitrou.net Wed Mar 11 19:05:57 2015 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 11 Mar 2015 19:05:57 +0100 Subject: [Python-Dev] libffi embedded in CPython In-Reply-To: References: <549891df.c332e00a.54c7.34f8@mx.google.com> <20150311185028.6f05973c@fsol> Message-ID: <20150311190557.25eebc16@fsol> On Wed, 11 Mar 2015 19:54:58 +0200 Maciej Fijalkowski wrote: > > > > Depends what you call "dangerous". C code doesn't rot quicker than pure > > Python code :-) Also, libffi really offers a wrapper around platform > > ABIs, which rarely change. > > And yet, lesser known ABIs in libffi contain bugs (as we discovered > trying to work there with anything else than x86 really). Also there > *are* ABI differencies that change slowly over time (e.g. requiring > stack to be 16 byte aligned) Well, sure. The point is, such bugs are unlikely to appear at a fast rate... Also, I don't understand why libffi issues would affect cffi any less than it affects ctypes, at least in the compiler-less mode of operation. > >> We now have things like cffi and Cython for people who need > >> to interface with C code. Both of those projects are maintained. And they > >> are not overly difficult to work with. > > > > But they are not ctypes. For example, cffi wouldn't be obvious to use > > for interfacing with non-C code, since it requires you to write C-like > > declarations. > > You mean like Fortran? Or what precisely? Any toolchain that can generate native code. It can be Fortran, but it can also be code generated at runtime without there being any external declaration. Having to generate "C declarations" for such code would be a distraction. Of course, if cffi gains the same ability as ctypes (namely to lookup a function and declare its signature without going through the FFI.cdef() interface), that issue disappears. As a side note, ctypes has a large number of users, so even if it were deprecated that wouldn't be a good reason to stop maintaining it. And calling cffi "simple" while it relies on a parser of the C language (which would then have to be bundled with Python) is a bit misleading IMO. Regards Antoine. From solipsis at pitrou.net Wed Mar 11 19:17:50 2015 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 11 Mar 2015 19:17:50 +0100 Subject: [Python-Dev] (ctypes example) libffi embedded in CPython References: <549891df.c332e00a.54c7.34f8@mx.google.com> <20150311185028.6f05973c@fsol> <20150311190557.25eebc16@fsol> Message-ID: <20150311191750.7cf23a40@fsol> On Wed, 11 Mar 2015 19:05:57 +0100 Antoine Pitrou wrote: > > > > > > But they are not ctypes. For example, cffi wouldn't be obvious to use > > > for interfacing with non-C code, since it requires you to write C-like > > > declarations. > > > > You mean like Fortran? Or what precisely? > > Any toolchain that can generate native code. It can be Fortran, but it > can also be code generated at runtime without there being any external > declaration. Having to generate "C declarations" for such code would be > a distraction. For instance, you can look at the compiler example that Eli wrote using llvmlite. It implements a JIT compiler for a toy language. The JIT-compiled function is then declared and called using a simple ctypes declaration: https://github.com/eliben/pykaleidoscope/blob/master/chapter7.py#L937 Regards Antoine. From wes.turner at gmail.com Wed Mar 11 19:31:23 2015 From: wes.turner at gmail.com (Wes Turner) Date: Wed, 11 Mar 2015 13:31:23 -0500 Subject: [Python-Dev] libffi embedded in CPython In-Reply-To: References: <549891df.c332e00a.54c7.34f8@mx.google.com> <20150311185028.6f05973c@fsol> Message-ID: On Mar 11, 2015 12:55 PM, "Maciej Fijalkowski" wrote: > > On Wed, Mar 11, 2015 at 7:50 PM, Antoine Pitrou wrote: > > On Wed, 11 Mar 2015 17:27:58 +0000 > > Brett Cannon wrote: > >> > >> Did anyone ever step forward to do this? I'm a bit worried about the > >> long-term viability of ctypes if we don't have a maintainer or at least > >> someone making sure we are staying up-to-date with upstream libffi. The > >> ctypes module is a dangerous thing, so having a chunk of C code that isn't > >> being properly maintained seems to me to make it even more dangerous. > > > > Depends what you call "dangerous". C code doesn't rot quicker than pure > > Python code :-) Also, libffi really offers a wrapper around platform > > ABIs, which rarely change. > > And yet, lesser known ABIs in libffi contain bugs (as we discovered > trying to work there with anything else than x86 really). Also there > *are* ABI differencies that change slowly over time (e.g. requiring > stack to be 16 byte aligned) Are there tests for this? > > > > >> I'm going to propose a somewhat controversial idea: let's deprecate the > >> ctypes module. > > > > This is gratuitous. > > I'm +1 on deprecating ctypes -1. These docs are helpful for understanding the pros and cons of different CPython <-> C interfaces. https://scipy-lectures.github.io/advanced/interfacing_with_c/interfacing_with_c.html (https://github.com/scipy-lectures/scipy-lecture-notes/issues/131 discusses documenting CFFI here as well) -------------- next part -------------- An HTML attachment was scrubbed... URL: From tim.peters at gmail.com Wed Mar 11 20:19:20 2015 From: tim.peters at gmail.com (Tim Peters) Date: Wed, 11 Mar 2015 14:19:20 -0500 Subject: [Python-Dev] Tunning binary insertion sort algorithm in Timsort. In-Reply-To: References: <7750938a2fea2.54fdd09e@wiscmail.wisc.edu> <76e0c70929f48.54fdd0da@wiscmail.wisc.edu> <76f094a32c8ca.54fdd119@wiscmail.wisc.edu> <7740f7e62cf4b.54fdd155@wiscmail.wisc.edu> <778088e82a599.54fdd192@wiscmail.wisc.edu> <7740ebbf29f9e.54fdd2be@wiscmail.wisc.edu> <76f0f50b2c800.54fdd2fb@wiscmail.wisc.edu> <7710a1122c5ba.54fdd337@wiscmail.wisc.edu> <7780f4192fd2c.54fdd373@wiscmail.wisc.edu> <7790d04c299a7.54fdd590@wiscmail.wisc.edu> <7710a1822e0a9.54fdd5cc@wiscmail.wisc.edu> <7720dc73289e8.54fdd609@wiscmail.wisc.edu> <7790de4c2be2c.54fdd645@wiscmail.wisc.edu> <76f0eced29ac4.54fd9476@wiscmail.wisc.edu> Message-ID: [Tim] >> 1. Merge "2 at a time" instead of just 1. That is, first "sort" the >> next 2 elements to be merged (1 compare and a possible swap). Then >> binary search to find where the smaller belongs, and a shorter binary >> search to find where the larger belongs. Then shift both into place. [Armin] > Good idea, but when I tried that it seemed to increase the total > number of comparisons (on random inputs with only up to 136 items). > The increase is on the order of 5%. I'm not sure reduced data > movement can compensate for that in Python. Which is another way of saying "bad idea" - that must be why I didn't pursue it to begin with ;-) Thanks for trying! I plugged a similar routine into the code I showed before to count the # of comparisons in Nha Pham's idea, and this "merge 2 at a time" thing has a higher average # of compares (over all permutations) than Nha's (which in turn has a higher average than the status quo). That makes some sense, thinking about what they do. Nha's algorithm has some supernaturally good cases (input array already ascending or already descending), but "merge 2 at a time" doesn't appear to have any. In any case, the information-theoretic minimum average number of comparisons for merging N sorted elements with 2 sorted elements is ("where do the 2 belong in the final list of N+2 elements?" = comb(N+2, 2)): log2((N+2)*(N+1)/2) = log2(N+2) + log2(N+1) - 1 Add a comparison to get the 2 elements in order to begin with, and we're up to log2(N+2) + log2(N+1) Two independent binary inserts (first to a list of size N, and then to a list of size N+1) comes out to the same. So even being supernaturally clever can't reduce the average number of compares this way. And since, in context, we're only looking at short arrays, a marginal saving in data movement costs (which are still O(N**2) worst case) are unlikely to be significant. Still, if anyone wants to go nuts ... ;-) From cournape at gmail.com Wed Mar 11 21:36:21 2015 From: cournape at gmail.com (David Cournapeau) Date: Wed, 11 Mar 2015 20:36:21 +0000 Subject: [Python-Dev] Why does python use relative instead of absolute path when calling LoadLibrary* Message-ID: Hi, While looking at the import code of python for C extensions, I was wondering why we pass a relative path instead of an absolute path to LoadLibraryEx (see bottom for some context). In python 2.7, the full path existence was even checked before calling into LoadLibraryEx ( https://github.com/python/cpython/blob/2.7/Python/dynload_win.c#L189), but it looks like this check was removed in python 3.x branch. Is there any defined behaviour that depends on this path to be relative ? Context ----------- The reason why I am interested in this is the potential use of SetDllDirectory to share dlls between multiple python extensions. Currently, the only solutions I am aware of are: 1. putting the dlls in the PATH 2. bundling the dlls side by side the .pyd 3. patching packages to use preloading (using e.g. ctypes) I am investigating a solution 4, where the dlls would be put in a separate "private" directory only known of python itself, without the need to modify PATH. Patching python to use SetDllDirectory("some private paths specific to a python interpreter") works perfectly, except that it slightly changes the semantics of LoadLibraryEx not to look for dlls in the current directory. This breaks importing extensions built in place, unless I modify the call in ;https://github.com/python/cpython/blob/2.7/Python/dynload_win.c#L195 from: hDLL = LoadLibraryEx(pathname, NULL LOAD_WITH_ALTERED_SEARCH_PATH) to hDLL = LoadLibraryEx(pathbuf, NULL LOAD_WITH_ALTERED_SEARCH_PATH) That seems to work, but I am quite worried about changing any import semantics by accident. David -------------- next part -------------- An HTML attachment was scrubbed... URL: From fijall at gmail.com Wed Mar 11 22:10:14 2015 From: fijall at gmail.com (Maciej Fijalkowski) Date: Wed, 11 Mar 2015 23:10:14 +0200 Subject: [Python-Dev] libffi embedded in CPython In-Reply-To: <20150311190557.25eebc16@fsol> References: <549891df.c332e00a.54c7.34f8@mx.google.com> <20150311185028.6f05973c@fsol> <20150311190557.25eebc16@fsol> Message-ID: On Wed, Mar 11, 2015 at 8:05 PM, Antoine Pitrou wrote: > On Wed, 11 Mar 2015 19:54:58 +0200 > Maciej Fijalkowski wrote: >> > >> > Depends what you call "dangerous". C code doesn't rot quicker than pure >> > Python code :-) Also, libffi really offers a wrapper around platform >> > ABIs, which rarely change. >> >> And yet, lesser known ABIs in libffi contain bugs (as we discovered >> trying to work there with anything else than x86 really). Also there >> *are* ABI differencies that change slowly over time (e.g. requiring >> stack to be 16 byte aligned) > > Well, sure. The point is, such bugs are unlikely to appear at a fast > rate... Also, I don't understand why libffi issues would affect cffi > any less than it affects ctypes, at least in the compiler-less mode of > operation. My point here was only about shipping own libffi vs using the system one (and it does affect cffi equally with or without compiler) > >> >> We now have things like cffi and Cython for people who need >> >> to interface with C code. Both of those projects are maintained. And they >> >> are not overly difficult to work with. >> > >> > But they are not ctypes. For example, cffi wouldn't be obvious to use >> > for interfacing with non-C code, since it requires you to write C-like >> > declarations. >> >> You mean like Fortran? Or what precisely? > > Any toolchain that can generate native code. It can be Fortran, but it > can also be code generated at runtime without there being any external > declaration. Having to generate "C declarations" for such code would be > a distraction. > > Of course, if cffi gains the same ability as ctypes (namely to lookup > a function and declare its signature without going through the > FFI.cdef() interface), that issue disappears. > > As a side note, ctypes has a large number of users, so even if it were > deprecated that wouldn't be a good reason to stop maintaining it. > > And calling cffi "simple" while it relies on a parser of the C language > (which would then have to be bundled with Python) is a bit misleading > IMO. > > Regards > > Antoine. > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/fijall%40gmail.com From fijall at gmail.com Wed Mar 11 22:11:41 2015 From: fijall at gmail.com (Maciej Fijalkowski) Date: Wed, 11 Mar 2015 23:11:41 +0200 Subject: [Python-Dev] libffi embedded in CPython In-Reply-To: References: <549891df.c332e00a.54c7.34f8@mx.google.com> <20150311185028.6f05973c@fsol> Message-ID: On Wed, Mar 11, 2015 at 8:31 PM, Wes Turner wrote: > > On Mar 11, 2015 12:55 PM, "Maciej Fijalkowski" wrote: >> >> On Wed, Mar 11, 2015 at 7:50 PM, Antoine Pitrou >> wrote: >> > On Wed, 11 Mar 2015 17:27:58 +0000 >> > Brett Cannon wrote: >> >> >> >> Did anyone ever step forward to do this? I'm a bit worried about the >> >> long-term viability of ctypes if we don't have a maintainer or at least >> >> someone making sure we are staying up-to-date with upstream libffi. The >> >> ctypes module is a dangerous thing, so having a chunk of C code that >> >> isn't >> >> being properly maintained seems to me to make it even more dangerous. >> > >> > Depends what you call "dangerous". C code doesn't rot quicker than pure >> > Python code :-) Also, libffi really offers a wrapper around platform >> > ABIs, which rarely change. >> >> And yet, lesser known ABIs in libffi contain bugs (as we discovered >> trying to work there with anything else than x86 really). Also there >> *are* ABI differencies that change slowly over time (e.g. requiring >> stack to be 16 byte aligned) > > Are there tests for this? > What do you mean? The usual failure mode is "will segfault every now and again if the moon is in the right position" (e.g. the stack alignment thing only happens if the underlaying function uses certain SSE instructions that compilers emit these days in certain circumstances) From fijall at gmail.com Wed Mar 11 22:13:29 2015 From: fijall at gmail.com (Maciej Fijalkowski) Date: Wed, 11 Mar 2015 23:13:29 +0200 Subject: [Python-Dev] (ctypes example) libffi embedded in CPython In-Reply-To: <20150311191750.7cf23a40@fsol> References: <549891df.c332e00a.54c7.34f8@mx.google.com> <20150311185028.6f05973c@fsol> <20150311190557.25eebc16@fsol> <20150311191750.7cf23a40@fsol> Message-ID: On Wed, Mar 11, 2015 at 8:17 PM, Antoine Pitrou wrote: > On Wed, 11 Mar 2015 19:05:57 +0100 > Antoine Pitrou wrote: >> > > >> > > But they are not ctypes. For example, cffi wouldn't be obvious to use >> > > for interfacing with non-C code, since it requires you to write C-like >> > > declarations. >> > >> > You mean like Fortran? Or what precisely? >> >> Any toolchain that can generate native code. It can be Fortran, but it >> can also be code generated at runtime without there being any external >> declaration. Having to generate "C declarations" for such code would be >> a distraction. > > For instance, you can look at the compiler example that Eli wrote using > llvmlite. It implements a JIT compiler for a toy language. The > JIT-compiled function is then declared and called using a simple ctypes > declaration: > > https://github.com/eliben/pykaleidoscope/blob/master/chapter7.py#L937 > > Regards > > Antoine. It might be a matter of taste, but I don't find declaring C functions any more awkward than using strange interface that ctypes comes with. the equivalent in cffi would be ffi.cast("double (*)()", x) From arigo at tunes.org Wed Mar 11 22:28:53 2015 From: arigo at tunes.org (Armin Rigo) Date: Wed, 11 Mar 2015 22:28:53 +0100 Subject: [Python-Dev] PEP 488: elimination of PYO files In-Reply-To: References: <54F9EBAF.1080400@hotpy.org> Message-ID: Hi Brett, On 6 March 2015 at 19:11, Brett Cannon wrote: > I disagree with your premise that .pyo files don't have a noticeable effect > on performance. If you don't use asserts a lot then there is no effect, but > if you use them heavily or have them perform expensive calculations then > there is an impact. Maybe you'd be interested to learn that PyPy (at least the 2.x branch) uses a new bytecode, JUMP_IF_NOT_DEBUG, to conditionally jump over the "assert" line. In "optimized" mode PyPy follows the jumps; in "non-optimized" mode it doesn't. This mode is initialized with the -O flag but can be changed dynamically, as the bytecode is the same. We introduced it as a simple solution to the mess of .pyc versus .pyo. (We didn't consider the case of -OO very closely because PyPy is much bigger than CPython as a binary to start with, so the demand for that is lower.) A bient?t, Armin. From victor.stinner at gmail.com Wed Mar 11 22:34:47 2015 From: victor.stinner at gmail.com (Victor Stinner) Date: Wed, 11 Mar 2015 22:34:47 +0100 Subject: [Python-Dev] libffi embedded in CPython In-Reply-To: References: <549891df.c332e00a.54c7.34f8@mx.google.com> Message-ID: Le 11 mars 2015 18:29, "Brett Cannon" a ?crit : > I'm going to propose a somewhat controversial idea: let's deprecate the ctypes module. In the past I tried to deprecate many functions or modules because they are rarely or never used. Many developers prefered to keep them. By the way, I still want to remove plat-xxx modules like IN or CDROM :-) Getopt was deprecated when optparse was added to the stdlib. Then optparse was deprecated when argparse was added to the stdlib. Cython and cffi are not part of the stdlib and can be hard to install on some platforms. Ctypes is cool because it doesn't require C headers nor a C compiler. Is it possible to use cffi without a C compiler/headers as easily than ctypes? If you want to move forward, you should help to integrate cffi into the stdlib. What's the status? Victor -------------- next part -------------- An HTML attachment was scrubbed... URL: From fijall at gmail.com Wed Mar 11 22:45:15 2015 From: fijall at gmail.com (Maciej Fijalkowski) Date: Wed, 11 Mar 2015 23:45:15 +0200 Subject: [Python-Dev] libffi embedded in CPython In-Reply-To: References: <549891df.c332e00a.54c7.34f8@mx.google.com> Message-ID: On Wed, Mar 11, 2015 at 11:34 PM, Victor Stinner wrote: > > Le 11 mars 2015 18:29, "Brett Cannon" a ?crit : >> I'm going to propose a somewhat controversial idea: let's deprecate the >> ctypes module. > > In the past I tried to deprecate many functions or modules because they are > rarely or never used. Many developers prefered to keep them. By the way, I > still want to remove plat-xxx modules like IN or CDROM :-) > > Getopt was deprecated when optparse was added to the stdlib. Then optparse > was deprecated when argparse was added to the stdlib. > > Cython and cffi are not part of the stdlib and can be hard to install on > some platforms. Ctypes is cool because it doesn't require C headers nor a C > compiler. > > Is it possible to use cffi without a C compiler/headers as easily than > ctypes? yes, it has two modes, one that does that and the other that does extra safety at the cost of a C compiler From p.f.moore at gmail.com Wed Mar 11 23:03:33 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 11 Mar 2015 22:03:33 +0000 Subject: [Python-Dev] libffi embedded in CPython In-Reply-To: References: <549891df.c332e00a.54c7.34f8@mx.google.com> Message-ID: On 11 March 2015 at 21:45, Maciej Fijalkowski wrote: >> Is it possible to use cffi without a C compiler/headers as easily than >> ctypes? > > yes, it has two modes, one that does that and the other that does > extra safety at the cost of a C compiler So if someone were to propose a practical approach to including cffi into the stdlib, *and* assisting the many Windows projects using ctypes for access to the Windows API [1], then there may be a reasonable argument for deprecating ctypes. But nobody seems to be doing that, rather the suggestion appears to be just to deprecate a widely used part of the stdlib offering no migration path :-( Paul [1] There may be Unix users of ctypes, but AFAIK, the vast majority of use cases are on Windows. From brett at python.org Wed Mar 11 23:06:23 2015 From: brett at python.org (Brett Cannon) Date: Wed, 11 Mar 2015 22:06:23 +0000 Subject: [Python-Dev] Why does python use relative instead of absolute path when calling LoadLibrary* In-Reply-To: References: Message-ID: On Wed, Mar 11, 2015 at 4:37 PM David Cournapeau wrote: > Hi, > > While looking at the import code of python for C extensions, I was > wondering why we pass a relative path instead of an absolute path to > LoadLibraryEx (see bottom for some context). > > In python 2.7, the full path existence was even checked before calling > into LoadLibraryEx ( > https://github.com/python/cpython/blob/2.7/Python/dynload_win.c#L189), > but it looks like this check was removed in python 3.x branch. > I personally don't know of any specific reason. -Brett > > Is there any defined behaviour that depends on this path to be relative ? > > Context > ----------- > > The reason why I am interested in this is the potential use of > SetDllDirectory to share dlls between multiple python extensions. > Currently, the only solutions I am aware of are: > > 1. putting the dlls in the PATH > 2. bundling the dlls side by side the .pyd > 3. patching packages to use preloading (using e.g. ctypes) > > I am investigating a solution 4, where the dlls would be put in a separate > "private" directory only known of python itself, without the need to modify > PATH. > > Patching python to use SetDllDirectory("some private paths specific to a > python interpreter") works perfectly, except that it slightly changes the > semantics of LoadLibraryEx not to look for dlls in the current directory. > This breaks importing extensions built in place, unless I modify the call > in ;https://github.com/python/cpython/blob/2.7/Python/dynload_win.c#L195 > from: > > hDLL = LoadLibraryEx(pathname, NULL LOAD_WITH_ALTERED_SEARCH_PATH) > > to > > hDLL = LoadLibraryEx(pathbuf, NULL LOAD_WITH_ALTERED_SEARCH_PATH) > > That seems to work, but I am quite worried about changing any import > semantics by accident. > > David > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ > brett%40python.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Wed Mar 11 23:08:21 2015 From: brett at python.org (Brett Cannon) Date: Wed, 11 Mar 2015 22:08:21 +0000 Subject: [Python-Dev] PEP 488: elimination of PYO files In-Reply-To: References: <54F9EBAF.1080400@hotpy.org> Message-ID: On Wed, Mar 11, 2015 at 5:29 PM Armin Rigo wrote: > Hi Brett, > > On 6 March 2015 at 19:11, Brett Cannon wrote: > > I disagree with your premise that .pyo files don't have a noticeable > effect > > on performance. If you don't use asserts a lot then there is no effect, > but > > if you use them heavily or have them perform expensive calculations then > > there is an impact. > > Maybe you'd be interested to learn that PyPy (at least the 2.x branch) > uses a new bytecode, JUMP_IF_NOT_DEBUG, to conditionally jump over the > "assert" line. In "optimized" mode PyPy follows the jumps; in > "non-optimized" mode it doesn't. This mode is initialized with the -O > flag but can be changed dynamically, as the bytecode is the same. We > introduced it as a simple solution to the mess of .pyc versus .pyo. > (We didn't consider the case of -OO very closely because PyPy is much > bigger than CPython as a binary to start with, so the demand for that > is lower.) > Interesting, so you simply merged the optimization levels 0 and 1 in the bytecode and basically drop .pyo files thanks to it. That might be some motivation to support the default file name not having any specified optimization level at all. -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Wed Mar 11 23:20:58 2015 From: brett at python.org (Brett Cannon) Date: Wed, 11 Mar 2015 22:20:58 +0000 Subject: [Python-Dev] libffi embedded in CPython In-Reply-To: References: <549891df.c332e00a.54c7.34f8@mx.google.com> Message-ID: On Wed, Mar 11, 2015 at 6:03 PM Paul Moore wrote: > On 11 March 2015 at 21:45, Maciej Fijalkowski wrote: > >> Is it possible to use cffi without a C compiler/headers as easily than > >> ctypes? > > > > yes, it has two modes, one that does that and the other that does > > extra safety at the cost of a C compiler > > So if someone were to propose a practical approach to including cffi > into the stdlib, *and* assisting the many Windows projects using > ctypes for access to the Windows API [1], then there may be a > reasonable argument for deprecating ctypes. But nobody seems to be > doing that, rather the suggestion appears to be just to deprecate a > widely used part of the stdlib offering no migration path :-( > You're ignoring that it's not maintained, which is the entire reason I brought this up. No one seems to want to touch the code. Who knows what improvements, bugfixes, etc. exist upstream in libffi that we lack because no one wants to go through and figure it out. If someone would come forward and help maintain it then I have no issue with it sticking around. -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Wed Mar 11 23:28:18 2015 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 11 Mar 2015 23:28:18 +0100 Subject: [Python-Dev] libffi embedded in CPython References: <549891df.c332e00a.54c7.34f8@mx.google.com> Message-ID: <20150311232818.5997d7cc@fsol> On Wed, 11 Mar 2015 22:20:58 +0000 Brett Cannon wrote: > > You're ignoring that it's not maintained, According to which metric/criterion? changeset: 94932:86c9ef950288 user: Steve Dower date: Tue Mar 10 09:56:38 2015 -0700 summary: Issue #23606: Disable ctypes.util.find_library("c") on Windows so tests are skipped while we figure out how best to approach the CRT change changeset: 94756:7c6e3358221a user: Serhiy Storchaka date: Thu Feb 26 15:27:57 2015 +0200 summary: Silenced minor GCC warnings. changeset: 94653:a84ae2ccd220 user: Serhiy Storchaka date: Mon Feb 16 20:52:17 2015 +0200 summary: Issue #23450: Fixed possible integer overflows. [etc.] > Who knows what > improvements, bugfixes, etc. exist upstream in libffi that we lack because > no one wants to go through and figure it out. Well, who knows? How about you enlighten us about that? And why do you think cffi, which *also* requires libffi, would be better served in that regard? Regards Antoine. From solipsis at pitrou.net Wed Mar 11 23:31:45 2015 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 11 Mar 2015 23:31:45 +0100 Subject: [Python-Dev] libffi embedded in CPython In-Reply-To: References: <549891df.c332e00a.54c7.34f8@mx.google.com> <20150311185028.6f05973c@fsol> <20150311190557.25eebc16@fsol> Message-ID: <20150311233145.5a6cd376@fsol> On Wed, 11 Mar 2015 23:10:14 +0200 Maciej Fijalkowski wrote: > > > > Well, sure. The point is, such bugs are unlikely to appear at a fast > > rate... Also, I don't understand why libffi issues would affect cffi > > any less than it affects ctypes, at least in the compiler-less mode of > > operation. > > My point here was only about shipping own libffi vs using the system > one (and it does affect cffi equally with or without compiler) So what? If ctypes used the system libffi as cffi does, it would by construction be at least portable as cffi is. The only reason the bundled libffi was patched at some point was to be *more* portable than vanilla libffi is. So, really, I don't see how switching from ctypes to cffi solves any of this. Regards Antoine. From fijall at gmail.com Wed Mar 11 23:31:55 2015 From: fijall at gmail.com (Maciej Fijalkowski) Date: Thu, 12 Mar 2015 00:31:55 +0200 Subject: [Python-Dev] libffi embedded in CPython In-Reply-To: References: <549891df.c332e00a.54c7.34f8@mx.google.com> Message-ID: On Thu, Mar 12, 2015 at 12:20 AM, Brett Cannon wrote: > > > On Wed, Mar 11, 2015 at 6:03 PM Paul Moore wrote: >> >> On 11 March 2015 at 21:45, Maciej Fijalkowski wrote: >> >> Is it possible to use cffi without a C compiler/headers as easily than >> >> ctypes? >> > >> > yes, it has two modes, one that does that and the other that does >> > extra safety at the cost of a C compiler >> >> So if someone were to propose a practical approach to including cffi >> into the stdlib, *and* assisting the many Windows projects using >> ctypes for access to the Windows API [1], then there may be a >> reasonable argument for deprecating ctypes. But nobody seems to be >> doing that, rather the suggestion appears to be just to deprecate a >> widely used part of the stdlib offering no migration path :-( > > > You're ignoring that it's not maintained, which is the entire reason I > brought this up. No one seems to want to touch the code. Who knows what > improvements, bugfixes, etc. exist upstream in libffi that we lack because > no one wants to go through and figure it out. If someone would come forward > and help maintain it then I have no issue with it sticking around. It's a bit worse than that. Each time someone wants to touch the code (e.g. push back the upstream libffi fixes), there is "we need to review it, but there is noone to do it", "noone knows how it works, don't touch it" kind of feedback, which leads to disincentives to potential maintainers. I would be likely willing to rip off the libffi from CPython as it is for example (and just use the upstream one) From fijall at gmail.com Wed Mar 11 23:33:23 2015 From: fijall at gmail.com (Maciej Fijalkowski) Date: Thu, 12 Mar 2015 00:33:23 +0200 Subject: [Python-Dev] libffi embedded in CPython In-Reply-To: <20150311233145.5a6cd376@fsol> References: <549891df.c332e00a.54c7.34f8@mx.google.com> <20150311185028.6f05973c@fsol> <20150311190557.25eebc16@fsol> <20150311233145.5a6cd376@fsol> Message-ID: On Thu, Mar 12, 2015 at 12:31 AM, Antoine Pitrou wrote: > On Wed, 11 Mar 2015 23:10:14 +0200 > Maciej Fijalkowski wrote: >> > >> > Well, sure. The point is, such bugs are unlikely to appear at a fast >> > rate... Also, I don't understand why libffi issues would affect cffi >> > any less than it affects ctypes, at least in the compiler-less mode of >> > operation. >> >> My point here was only about shipping own libffi vs using the system >> one (and it does affect cffi equally with or without compiler) > > So what? If ctypes used the system libffi as cffi does, it would by > construction be at least portable as cffi is. The only reason the > bundled libffi was patched at some point was to be *more* portable than > vanilla libffi is. > > So, really, I don't see how switching from ctypes to cffi solves any of > this. You're missing my point. Ripping off the libffi from CPython is a good idea to start with. Maybe deprecating ctypes is *also* a good idea, but it's a separate discussion point. It certainly does not solve the libffi problem. From p.f.moore at gmail.com Wed Mar 11 23:55:33 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 11 Mar 2015 22:55:33 +0000 Subject: [Python-Dev] libffi embedded in CPython In-Reply-To: References: <549891df.c332e00a.54c7.34f8@mx.google.com> <20150311185028.6f05973c@fsol> <20150311190557.25eebc16@fsol> <20150311233145.5a6cd376@fsol> Message-ID: On 11 March 2015 at 22:33, Maciej Fijalkowski wrote: > You're missing my point. Ripping off the libffi from CPython is a good > idea to start with. Maybe deprecating ctypes is *also* a good idea, > but it's a separate discussion point. It certainly does not solve the > libffi problem. OK, so let's focus on the libffi side of things and ignore deprecating or replacing ctypes. I guess I don't see a problem with a proof-of-concept patch to upgrade the libffi (obviously it's not possible to rely on a "system" libffi on Windows, but treating it as an external might work). If it passes all the tests on all platforms, maybe it could be considered. I don't see how anyone can say "yes, we'll do it" without seeing evidence that it'll work. But equally, I don't think there's any reason to say it absolutely wouldn't be accepted regardless of evidence. So why not prepare a patch for 3.6 (I assume it's too late to make such a big change for 3.5) and we'll see what happens. Or even better, prepare a test build of 3.5 or even 3.4 that switches libffi - people can replace an existing install (I'd be willing to) and test it in live situations. But unless someone provides a patch, the status quo will win. At least until an actual bug that affects live code forces the issue. Paul From steve at pearwood.info Thu Mar 12 00:03:53 2015 From: steve at pearwood.info (Steven D'Aprano) Date: Thu, 12 Mar 2015 10:03:53 +1100 Subject: [Python-Dev] PEP 488: elimination of PYO files In-Reply-To: References: <54F9EBAF.1080400@hotpy.org> <1425668955.3364036.236612829.494092A7@webmail.messagingengine.com> <1425685751.3874542.236704353.11A94396@webmail.messagingengine.com> Message-ID: <20150311230352.GB7655@ando.pearwood.info> On Wed, Mar 11, 2015 at 05:34:10PM +0000, Brett Cannon wrote: > I have a poll going on G+ to see what people think of the various proposed > file name formats at > https://plus.google.com/u/0/+BrettCannon/posts/fZynLNwHWGm . Feel free to > vote if you have an opinion. G+ hates my browser and won't let me vote. I click on the button and nothing happens. I have Javascript enabled and I'm not using any ad blockers. For the record, I think only the first two options importlib.cpython-35.opt-0.pyc importlib.cpython-35.opt0.pyc are sane, and I prefer the first. I'm mildly inclined to leave out the opt* part for default, unoptimized code. In other words, the file name holds two or three '.' delimited fields, plus the extension: .-.[opt-].pyc where [...] is optional and the optimization codes for CPython will be 1 for -O and 2 for -OO. And 0 for unoptimized, if you decide that it should be mandatory. Thank you for moving forward on this, I think it is a good plan. -- Steve From solipsis at pitrou.net Thu Mar 12 00:11:40 2015 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 12 Mar 2015 00:11:40 +0100 Subject: [Python-Dev] libffi embedded in CPython In-Reply-To: References: <549891df.c332e00a.54c7.34f8@mx.google.com> <20150311185028.6f05973c@fsol> <20150311190557.25eebc16@fsol> <20150311233145.5a6cd376@fsol> Message-ID: <20150312001140.64419595@fsol> On Wed, 11 Mar 2015 22:55:33 +0000 Paul Moore wrote: > On 11 March 2015 at 22:33, Maciej Fijalkowski wrote: > > You're missing my point. Ripping off the libffi from CPython is a good > > idea to start with. Maybe deprecating ctypes is *also* a good idea, > > but it's a separate discussion point. It certainly does not solve the > > libffi problem. > > OK, so let's focus on the libffi side of things and ignore deprecating > or replacing ctypes. > > I guess I don't see a problem with a proof-of-concept patch to upgrade > the libffi (obviously it's not possible to rely on a "system" libffi > on Windows, but treating it as an external might work). If it passes > all the tests on all platforms, maybe it could be considered. Agreed. We have enough tests and enough buildbots that it can be attempted. (besides, the only more-or-less officially supported platforms are Linux, Windows, OS X; we also try to not to be too broken on the BSDs, but that's it) Regards Antoine. From guido at python.org Thu Mar 12 01:20:33 2015 From: guido at python.org (Guido van Rossum) Date: Wed, 11 Mar 2015 17:20:33 -0700 Subject: [Python-Dev] libffi embedded in CPython In-Reply-To: References: <549891df.c332e00a.54c7.34f8@mx.google.com> Message-ID: Can we please decouple the ctypes deprecation discussion from efforts to upgrade cffi? They can coexist just fine, and they don't even really solve the same problem. On Wed, Mar 11, 2015 at 3:20 PM, Brett Cannon wrote: > > > On Wed, Mar 11, 2015 at 6:03 PM Paul Moore wrote: > >> On 11 March 2015 at 21:45, Maciej Fijalkowski wrote: >> >> Is it possible to use cffi without a C compiler/headers as easily than >> >> ctypes? >> > >> > yes, it has two modes, one that does that and the other that does >> > extra safety at the cost of a C compiler >> >> So if someone were to propose a practical approach to including cffi >> into the stdlib, *and* assisting the many Windows projects using >> ctypes for access to the Windows API [1], then there may be a >> reasonable argument for deprecating ctypes. But nobody seems to be >> doing that, rather the suggestion appears to be just to deprecate a >> widely used part of the stdlib offering no migration path :-( >> > > You're ignoring that it's not maintained, which is the entire reason I > brought this up. No one seems to want to touch the code. Who knows what > improvements, bugfixes, etc. exist upstream in libffi that we lack because > no one wants to go through and figure it out. If someone would come forward > and help maintain it then I have no issue with it sticking around. > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/guido%40python.org > > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From wes.turner at gmail.com Thu Mar 12 03:41:37 2015 From: wes.turner at gmail.com (Wes Turner) Date: Wed, 11 Mar 2015 21:41:37 -0500 Subject: [Python-Dev] Why does python use relative instead of absolute path when calling LoadLibrary* In-Reply-To: References: Message-ID: On Mar 11, 2015 3:36 PM, "David Cournapeau" wrote: > > Hi, > > While looking at the import code of python for C extensions, I was wondering why we pass a relative path instead of an absolute path to LoadLibraryEx (see bottom for some context). > > In python 2.7, the full path existence was even checked before calling into LoadLibraryEx ( https://github.com/python/cpython/blob/2.7/Python/dynload_win.c#L189), but it looks like this check was removed in python 3.x branch. > > Is there any defined behaviour that depends on this path to be relative ? Just a guess: does it have to do with resolving symlinks (w/ POSIX filesystems)? -------------- next part -------------- An HTML attachment was scrubbed... URL: From soeren.j.loevborg at siemens.com Thu Mar 12 13:37:16 2015 From: soeren.j.loevborg at siemens.com (Loevborg, Soeren Jakob) Date: Thu, 12 Mar 2015 12:37:16 +0000 Subject: [Python-Dev] Request for comments: [issue22941] IPv4Interface arithmetic changes subnet mask Message-ID: <927E0DF62C47C940A006AFF297D7F512120C11@DEFTHW99EK0MSX.ww902.siemens.net> Hi, I'm looking for feedback on issue 22941, "IPv4Interface arithmetic changes subnet mask". As stated in the bug, I'd be happy to write a patch, if anyone would comment on my proposal. http://bugs.python.org/issue22941 Thank you, S?ren L?vborg --- Addition and subtraction of integers are documented for ipaddress.IPv4Address and ipaddress.IPv6Address, but also work for IPv4Interface and IPv6Interface (though the only documentation of this is a brief mention that the Interface classes inherit from the respective Address classes). That's good. The problem is that adding/subtracting an integer to an Interface object changes the subnet mask (to max_prefixlen), something which is 1) not documented and 2) not the least surprising result. >>> import ipaddress >>> ipaddress.IPv4Interface('10.0.0.1/8') + 1 IPv4Interface('10.0.0.2/32') >>> ipaddress.IPv6Interface('fd00::1/64') + 1 IPv6Interface('fd00::2/128') Proposed behavior: >>> import ipaddress >>> ipaddress.IPv4Interface('10.0.0.1/8') + 1 IPv4Interface('10.0.0.2/8') >>> ipaddress.IPv6Interface('fd00::1/64') + 1 IPv6Interface('fd00::2/64') It all comes down to a judgment call: is this a bug, or expected (but undocumented) behavior? In PEP 387 lingo: Is this a "reasonable bug fix"? Or is it a "design mistake", thus calling for a FutureWarning first, and the actual change later? >>> ipaddress.IPv4Interface('1.2.3.4/16') + 1 __main__:1: FutureWarning: Arithmetic on IPv4Interface objects will change in Python 3.6. If you rely on the current behavior, change 'interface + n' to 'IPv4Interface(interface.ip + n, 32)' From solipsis at pitrou.net Thu Mar 12 15:00:23 2015 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 12 Mar 2015 15:00:23 +0100 Subject: [Python-Dev] Request for comments: [issue22941] IPv4Interface arithmetic changes subnet mask References: <927E0DF62C47C940A006AFF297D7F512120C11@DEFTHW99EK0MSX.ww902.siemens.net> Message-ID: <20150312150023.04051b02@fsol> On Thu, 12 Mar 2015 12:37:16 +0000 "Loevborg, Soeren Jakob" wrote: > > >>> import ipaddress > >>> ipaddress.IPv4Interface('10.0.0.1/8') + 1 > IPv4Interface('10.0.0.2/32') > >>> ipaddress.IPv6Interface('fd00::1/64') + 1 > IPv6Interface('fd00::2/128') Given that the behaviour is unlikely to match any use case, I'd say it's a bug. Related issue: >>> ipaddress.IPv4Interface('10.0.0.255/8') + 1 IPv4Interface('10.0.1.0/32') Should the result be IPv4Interface('10.0.0.0/8') (i.e. wrap around)? Should OverflowError be raised? Regards Antoine. From brett at python.org Thu Mar 12 15:05:15 2015 From: brett at python.org (Brett Cannon) Date: Thu, 12 Mar 2015 14:05:15 +0000 Subject: [Python-Dev] libffi embedded in CPython In-Reply-To: References: <549891df.c332e00a.54c7.34f8@mx.google.com> Message-ID: On Wed, Mar 11, 2015 at 8:20 PM Guido van Rossum wrote: > Can we please decouple the ctypes deprecation discussion from efforts to > upgrade cffi? They can coexist just fine, and they don't even really solve > the same problem. > I mostly proposed deprecating ctypes because we were not keeping up with libffi upstream. If we solve the latter I'm not bothered enough to personally pursue the former. -Brett > > On Wed, Mar 11, 2015 at 3:20 PM, Brett Cannon wrote: > >> >> >> On Wed, Mar 11, 2015 at 6:03 PM Paul Moore wrote: >> >>> On 11 March 2015 at 21:45, Maciej Fijalkowski wrote: >>> >> Is it possible to use cffi without a C compiler/headers as easily than >>> >> ctypes? >>> > >>> > yes, it has two modes, one that does that and the other that does >>> > extra safety at the cost of a C compiler >>> >>> So if someone were to propose a practical approach to including cffi >>> into the stdlib, *and* assisting the many Windows projects using >>> ctypes for access to the Windows API [1], then there may be a >>> reasonable argument for deprecating ctypes. But nobody seems to be >>> doing that, rather the suggestion appears to be just to deprecate a >>> widely used part of the stdlib offering no migration path :-( >>> >> >> You're ignoring that it's not maintained, which is the entire reason I >> brought this up. No one seems to want to touch the code. Who knows what >> improvements, bugfixes, etc. exist upstream in libffi that we lack because >> no one wants to go through and figure it out. If someone would come forward >> and help maintain it then I have no issue with it sticking around. >> >> _______________________________________________ >> Python-Dev mailing list >> Python-Dev at python.org >> https://mail.python.org/mailman/listinfo/python-dev >> > Unsubscribe: >> https://mail.python.org/mailman/options/python-dev/guido%40python.org >> >> > > > -- > --Guido van Rossum (python.org/~guido) > -------------- next part -------------- An HTML attachment was scrubbed... URL: From thomas at python.org Thu Mar 12 15:25:18 2015 From: thomas at python.org (Thomas Wouters) Date: Thu, 12 Mar 2015 15:25:18 +0100 Subject: [Python-Dev] libffi embedded in CPython In-Reply-To: References: <549891df.c332e00a.54c7.34f8@mx.google.com> Message-ID: On Thu, Mar 12, 2015 at 1:20 AM, Guido van Rossum wrote: > Can we please decouple the ctypes deprecation discussion from efforts to > upgrade cffi? They can coexist just fine, and they don't even really solve > the same problem. > ctypes and cffi do actually solve the same problem, just not in the same way. > > On Wed, Mar 11, 2015 at 3:20 PM, Brett Cannon wrote: > >> >> >> On Wed, Mar 11, 2015 at 6:03 PM Paul Moore wrote: >> >>> On 11 March 2015 at 21:45, Maciej Fijalkowski wrote: >>> >> Is it possible to use cffi without a C compiler/headers as easily than >>> >> ctypes? >>> > >>> > yes, it has two modes, one that does that and the other that does >>> > extra safety at the cost of a C compiler >>> >>> So if someone were to propose a practical approach to including cffi >>> into the stdlib, *and* assisting the many Windows projects using >>> ctypes for access to the Windows API [1], then there may be a >>> reasonable argument for deprecating ctypes. But nobody seems to be >>> doing that, rather the suggestion appears to be just to deprecate a >>> widely used part of the stdlib offering no migration path :-( >>> >> >> You're ignoring that it's not maintained, which is the entire reason I >> brought this up. No one seems to want to touch the code. Who knows what >> improvements, bugfixes, etc. exist upstream in libffi that we lack because >> no one wants to go through and figure it out. If someone would come forward >> and help maintain it then I have no issue with it sticking around. >> >> _______________________________________________ >> Python-Dev mailing list >> Python-Dev at python.org >> https://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe: >> https://mail.python.org/mailman/options/python-dev/guido%40python.org >> >> > > > -- > --Guido van Rossum (python.org/~guido) > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/thomas%40python.org > > -- Thomas Wouters Hi! I'm an email virus! Think twice before sending your email to help me spread! -------------- next part -------------- An HTML attachment was scrubbed... URL: From zachary.ware+pydev at gmail.com Thu Mar 12 15:39:14 2015 From: zachary.ware+pydev at gmail.com (Zachary Ware) Date: Thu, 12 Mar 2015 09:39:14 -0500 Subject: [Python-Dev] libffi embedded in CPython In-Reply-To: <549891df.c332e00a.54c7.34f8@mx.google.com> References: <549891df.c332e00a.54c7.34f8@mx.google.com> Message-ID: I started this message about 3 months ago; at this point I'm just getting it posted so it stops rotting in my Drafts folder. On Mon, Dec 22, 2014 at 3:49 PM, Jim J. Jewett wrote: > On Thu, Dec 18, 2014, at 14:13, Maciej Fijalkowski wrote: >> ... http://bugs.python.org/issue23085 ... >> is there any reason any more for libffi being included in CPython? > > > Paul Moore wrote: >> Probably the easiest way of moving this forward would be for someone >> to identify the CPython-specific patches in the current version ... > > Christian Heimes wrote: >> That's easy. All patches are tracked in the diff file >> https://hg.python.org/cpython/file/3de678cd184d/Modules/_ctypes/libffi.diff > > That (200+ lines) doesn't seem to have all the C changes, such as the > win64 sizeof changes from issue 11835. I took a little bit of time to look at this a while ago. Something that quickly became clear that I'm not sure everybody is on the same page about (I, for one, had no idea previously) is that libffi for Windows and libffi for everything else are entirely separate beasts: the Windows build pulls all of its libffi source from Modules/_ctypes/libffi_msvc and doesn't even touch Modules/_ctypes/libffi (which is libffi for nearly everything else). Most of the files in libffi_msvc haven't been touched in about 4 years, except for a couple fixes by Steve Dower recently. README.ctypes in that folder states that the files were taken from the libffi CVS tree in 2004 and there have been some custom modifications since then, so there is really very little resemblance between libffi_msvc and current libffi. There's also 'libffi_arm_wince' and 'libffi_osx'. I would be fairly amazed if Python itself built on any version of Windows CE; personally I would be for ripping out any supposed support that we have for it (though if somebody is actually using it and it works, it's not hurting anything and should stay). That's a completely different issue, though. I'm all for ditching our 'libffi_msvc' in favor of adding libffi as another 'external' for the Windows build. I have managed to get _ctypes to build on Windows using vanilla libffi sources, prepared using their configure script from within Git Bash and built with our usual Windows build system (properly patched). Unfortunately, making things usable will take some work on ctypes itself, which I'm not qualified to do. I'm happy to pass on my procedure and patches for getting to the point of successful compilation to anyone who feels up to fixing the things that are broken. As for the lightly-patched version of libffi that we keep around for Linux et. al., looking at the diff file I don't see that there's a whole lot of reason to keep it around. -- Zach From eric at trueblade.com Thu Mar 12 15:46:34 2015 From: eric at trueblade.com (Eric V. Smith) Date: Thu, 12 Mar 2015 10:46:34 -0400 Subject: [Python-Dev] Request for comments: [issue22941] IPv4Interface arithmetic changes subnet mask In-Reply-To: <20150312150023.04051b02@fsol> References: <927E0DF62C47C940A006AFF297D7F512120C11@DEFTHW99EK0MSX.ww902.siemens.net> <20150312150023.04051b02@fsol> Message-ID: <5501A6CA.2080301@trueblade.com> On 3/12/2015 10:00 AM, Antoine Pitrou wrote: > On Thu, 12 Mar 2015 12:37:16 +0000 > "Loevborg, Soeren Jakob" wrote: >> >>>>> import ipaddress >>>>> ipaddress.IPv4Interface('10.0.0.1/8') + 1 >> IPv4Interface('10.0.0.2/32') >>>>> ipaddress.IPv6Interface('fd00::1/64') + 1 >> IPv6Interface('fd00::2/128') > > Given that the behaviour is unlikely to match any use case, I'd say > it's a bug. Agreed. > Related issue: > >>>> ipaddress.IPv4Interface('10.0.0.255/8') + 1 > IPv4Interface('10.0.1.0/32') > > Should the result be IPv4Interface('10.0.0.0/8') > (i.e. wrap around)? Should OverflowError be raised? I think it should be an OverflowError. 10.0.1.0/8 makes no sense, and 10.0.0.0/8 is unlikely to be desirable. -- Eric. From brett at python.org Thu Mar 12 18:26:02 2015 From: brett at python.org (Brett Cannon) Date: Thu, 12 Mar 2015 17:26:02 +0000 Subject: [Python-Dev] libffi embedded in CPython In-Reply-To: References: <549891df.c332e00a.54c7.34f8@mx.google.com> Message-ID: On Thu, Mar 12, 2015 at 10:39 AM Zachary Ware wrote: > I started this message about 3 months ago; at this point I'm just > getting it posted so it stops rotting in my Drafts folder. > Thanks for looking into this! > > On Mon, Dec 22, 2014 at 3:49 PM, Jim J. Jewett > wrote: > > On Thu, Dec 18, 2014, at 14:13, Maciej Fijalkowski wrote: > >> ... http://bugs.python.org/issue23085 ... > >> is there any reason any more for libffi being included in CPython? > > > > > > Paul Moore wrote: > >> Probably the easiest way of moving this forward would be for someone > >> to identify the CPython-specific patches in the current version ... > > > > Christian Heimes wrote: > >> That's easy. All patches are tracked in the diff file > >> https://hg.python.org/cpython/file/3de678cd184d/Modules/_ > ctypes/libffi.diff > > > > That (200+ lines) doesn't seem to have all the C changes, such as the > > win64 sizeof changes from issue 11835. > > I took a little bit of time to look at this a while ago. Something > that quickly became clear that I'm not sure everybody is on the same > page about (I, for one, had no idea previously) is that libffi for > Windows and libffi for everything else are entirely separate beasts: > the Windows build pulls all of its libffi source from > Modules/_ctypes/libffi_msvc and doesn't even touch > Modules/_ctypes/libffi (which is libffi for nearly everything else). > Most of the files in libffi_msvc haven't been touched in about 4 > years, except for a couple fixes by Steve Dower recently. > README.ctypes in that folder states that the files were taken from the > libffi CVS tree in 2004 and there have been some custom modifications > since then, so there is really very little resemblance between > libffi_msvc and current libffi. > That's what I was afraid of. > > There's also 'libffi_arm_wince' and 'libffi_osx'. I would be fairly > amazed if Python itself built on any version of Windows CE; personally > I would be for ripping out any supposed support that we have for it > (though if somebody is actually using it and it works, it's not > hurting anything and should stay). That's a completely different > issue, though. > > I'm all for ditching our 'libffi_msvc' in favor of adding libffi as > another 'external' for the Windows build. I have managed to get > _ctypes to build on Windows using vanilla libffi sources, prepared > using their configure script from within Git Bash and built with our > usual Windows build system (properly patched). Unfortunately, making > things usable will take some work on ctypes itself, which I'm not > qualified to do. I'm happy to pass on my procedure and patches for > getting to the point of successful compilation to anyone who feels up > to fixing the things that are broken. > So it seems possible to use upstream libffi but will require some work. > > As for the lightly-patched version of libffi that we keep around for > Linux et. al., looking at the diff file I don't see that there's a > whole lot of reason to keep it around. > For UNIX OSs we could probably rely on the system libffi then. What's the situation on OS X? Anyone know if it has libffi, or would be need to be pulled in to be used like on Windows? -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Thu Mar 12 18:44:27 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 12 Mar 2015 17:44:27 +0000 Subject: [Python-Dev] libffi embedded in CPython In-Reply-To: References: <549891df.c332e00a.54c7.34f8@mx.google.com> Message-ID: On 12 March 2015 at 17:26, Brett Cannon wrote: >> I'm all for ditching our 'libffi_msvc' in favor of adding libffi as >> another 'external' for the Windows build. I have managed to get >> _ctypes to build on Windows using vanilla libffi sources, prepared >> using their configure script from within Git Bash and built with our >> usual Windows build system (properly patched). Unfortunately, making >> things usable will take some work on ctypes itself, which I'm not >> qualified to do. I'm happy to pass on my procedure and patches for >> getting to the point of successful compilation to anyone who feels up >> to fixing the things that are broken. > > > So it seems possible to use upstream libffi but will require some work. I'd be willing to contemplate helping out on the Windows side of things, if nobody else steps up (with the proviso that I have little free time, and I'm saying this without much idea of what's involved :-)) If Zachary can give a bit more detail on what the work on ctypes is, and/or put what he has somewhere that I could have a look at, that might help. Paul. From scott+python-dev at scottdial.com Thu Mar 12 19:08:01 2015 From: scott+python-dev at scottdial.com (Scott Dial) Date: Thu, 12 Mar 2015 14:08:01 -0400 Subject: [Python-Dev] Request for comments: [issue22941] IPv4Interface arithmetic changes subnet mask In-Reply-To: <5501A6CA.2080301@trueblade.com> References: <927E0DF62C47C940A006AFF297D7F512120C11@DEFTHW99EK0MSX.ww902.siemens.net> <20150312150023.04051b02@fsol> <5501A6CA.2080301@trueblade.com> Message-ID: <5501D601.9090105@scottdial.com> On 2015-03-12 10:46 AM, Eric V. Smith wrote: > On 3/12/2015 10:00 AM, Antoine Pitrou wrote: >> Related issue: >> >>>>> ipaddress.IPv4Interface('10.0.0.255/8') + 1 >> IPv4Interface('10.0.1.0/32') >> >> Should the result be IPv4Interface('10.0.0.0/8') >> (i.e. wrap around)? Should OverflowError be raised? > > I think it should be an OverflowError. 10.0.1.0/8 makes no sense, and > 10.0.0.0/8 is unlikely to be desirable. > No, you are both imbuing meaning to the dot-notation that is not there. A.B.C.D is just an ugly way to represent a single 32-bit unsigned integer. The only contentious part would be when your addition would overlfow the host-portion of the address: >>> ipaddress.IPv4Interface('10.255.255.255/8') + 1 IPv4Interface('11.0.0.0/8') However, if you take the perspective that the address is just a 32-bit unsigned integer, then it makes perfect sense. I would argue it's likely to be a source of bugs, but I can't say either way because I never adopted using this library due to issues that are cataloged in the mailing list archives. (In the end, I wrote my own and have never found the need to support such operands.) -- Scott Dial scott at scottdial.com From solipsis at pitrou.net Thu Mar 12 19:17:17 2015 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 12 Mar 2015 19:17:17 +0100 Subject: [Python-Dev] Request for comments: [issue22941] IPv4Interface arithmetic changes subnet mask References: <927E0DF62C47C940A006AFF297D7F512120C11@DEFTHW99EK0MSX.ww902.siemens.net> <20150312150023.04051b02@fsol> <5501A6CA.2080301@trueblade.com> <5501D601.9090105@scottdial.com> Message-ID: <20150312191717.30acbd7b@fsol> On Thu, 12 Mar 2015 14:08:01 -0400 Scott Dial wrote: > On 2015-03-12 10:46 AM, Eric V. Smith wrote: > > On 3/12/2015 10:00 AM, Antoine Pitrou wrote: > >> Related issue: > >> > >>>>> ipaddress.IPv4Interface('10.0.0.255/8') + 1 > >> IPv4Interface('10.0.1.0/32') > >> > >> Should the result be IPv4Interface('10.0.0.0/8') > >> (i.e. wrap around)? Should OverflowError be raised? > > > > I think it should be an OverflowError. 10.0.1.0/8 makes no sense, and > > 10.0.0.0/8 is unlikely to be desirable. > > > > No, you are both imbuing meaning to the dot-notation that is not there. > A.B.C.D is just an ugly way to represent a single 32-bit unsigned > integer. The only contentious part would be when your addition would > overlfow the host-portion of the address: > > >>> ipaddress.IPv4Interface('10.255.255.255/8') + 1 > IPv4Interface('11.0.0.0/8') Oh, well, you're right, I borked my subnet. So, for the record, I meant "10.0.0.255/24" where the issue actually appears. Regards Antoine. From nad at acm.org Thu Mar 12 19:35:55 2015 From: nad at acm.org (Ned Deily) Date: Thu, 12 Mar 2015 11:35:55 -0700 Subject: [Python-Dev] libffi embedded in CPython References: <549891df.c332e00a.54c7.34f8@mx.google.com> Message-ID: In article , > For UNIX OSs we could probably rely on the system libffi then. What's the > situation on OS X? Anyone know if it has libffi, or would be need to be > pulled in to be used like on Windows? Ronald (in http://bugs.python.org/issue23534): "On OSX the internal copy of libffi that's used is based on the one in PyObjC, which in turn is based on the version of libffi on opensource.apple.com (IIRC with some small patches that fix minor issues found by the PyObjC testsuite)." -- Ned Deily, nad at acm.org From zachary.ware+pydev at gmail.com Thu Mar 12 19:54:16 2015 From: zachary.ware+pydev at gmail.com (Zachary Ware) Date: Thu, 12 Mar 2015 13:54:16 -0500 Subject: [Python-Dev] libffi embedded in CPython In-Reply-To: References: <549891df.c332e00a.54c7.34f8@mx.google.com> Message-ID: On Thu, Mar 12, 2015 at 12:44 PM, Paul Moore wrote: > I'd be willing to contemplate helping out on the Windows side of > things, if nobody else steps up (with the proviso that I have little > free time, and I'm saying this without much idea of what's involved > :-)) If Zachary can give a bit more detail on what the work on ctypes > is, and/or put what he has somewhere that I could have a look at, that > might help. I'll send you everything I've got next chance I have, but I don't know when that will be. -- Zach From eric at trueblade.com Thu Mar 12 19:55:05 2015 From: eric at trueblade.com (Eric V. Smith) Date: Thu, 12 Mar 2015 14:55:05 -0400 Subject: [Python-Dev] Request for comments: [issue22941] IPv4Interface arithmetic changes subnet mask In-Reply-To: <20150312191717.30acbd7b@fsol> References: <927E0DF62C47C940A006AFF297D7F512120C11@DEFTHW99EK0MSX.ww902.siemens.net> <20150312150023.04051b02@fsol> <5501A6CA.2080301@trueblade.com> <5501D601.9090105@scottdial.com> <20150312191717.30acbd7b@fsol> Message-ID: <5501E109.7090408@trueblade.com> On 3/12/2015 2:17 PM, Antoine Pitrou wrote: > On Thu, 12 Mar 2015 14:08:01 -0400 > Scott Dial wrote: > >> On 2015-03-12 10:46 AM, Eric V. Smith wrote: >>> On 3/12/2015 10:00 AM, Antoine Pitrou wrote: >>>> Related issue: >>>> >>>>>>> ipaddress.IPv4Interface('10.0.0.255/8') + 1 >>>> IPv4Interface('10.0.1.0/32') >>>> >>>> Should the result be IPv4Interface('10.0.0.0/8') >>>> (i.e. wrap around)? Should OverflowError be raised? >>> >>> I think it should be an OverflowError. 10.0.1.0/8 makes no sense, and >>> 10.0.0.0/8 is unlikely to be desirable. >>> >> >> No, you are both imbuing meaning to the dot-notation that is not there. >> A.B.C.D is just an ugly way to represent a single 32-bit unsigned >> integer. The only contentious part would be when your addition would >> overlfow the host-portion of the address: >> >>>>> ipaddress.IPv4Interface('10.255.255.255/8') + 1 >> IPv4Interface('11.0.0.0/8') > > Oh, well, you're right, I borked my subnet. So, for the record, I meant > "10.0.0.255/24" where the issue actually appears. I, too, was thinking /24. I think that overflowing the host portion should raise OverflowError. -- Eric. From p.f.moore at gmail.com Thu Mar 12 20:13:08 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 12 Mar 2015 19:13:08 +0000 Subject: [Python-Dev] libffi embedded in CPython In-Reply-To: References: <549891df.c332e00a.54c7.34f8@mx.google.com> Message-ID: On 12 March 2015 at 18:54, Zachary Ware wrote: > On Thu, Mar 12, 2015 at 12:44 PM, Paul Moore wrote: >> I'd be willing to contemplate helping out on the Windows side of >> things, if nobody else steps up (with the proviso that I have little >> free time, and I'm saying this without much idea of what's involved >> :-)) If Zachary can give a bit more detail on what the work on ctypes >> is, and/or put what he has somewhere that I could have a look at, that >> might help. > > I'll send you everything I've got next chance I have, but I don't know > when that will be. Thanks. No rush :-) Paul From fijall at gmail.com Thu Mar 12 20:36:45 2015 From: fijall at gmail.com (Maciej Fijalkowski) Date: Thu, 12 Mar 2015 21:36:45 +0200 Subject: [Python-Dev] libffi embedded in CPython In-Reply-To: References: <549891df.c332e00a.54c7.34f8@mx.google.com> Message-ID: On Thu, Mar 12, 2015 at 8:35 PM, Ned Deily wrote: > In article > , >> For UNIX OSs we could probably rely on the system libffi then. What's the >> situation on OS X? Anyone know if it has libffi, or would be need to be >> pulled in to be used like on Windows? > > Ronald (in http://bugs.python.org/issue23534): > "On OSX the internal copy of libffi that's used is based on the one in > PyObjC, which in turn is based on the version of libffi on > opensource.apple.com (IIRC with some small patches that fix minor issues > found by the PyObjC testsuite)." > > -- > Ned Deily, > nad at acm.org >From pypy experience, libffi installed on OS X tends to just work (we never had any issues with those) From thomas at python.org Thu Mar 12 20:58:32 2015 From: thomas at python.org (Thomas Wouters) Date: Thu, 12 Mar 2015 20:58:32 +0100 Subject: [Python-Dev] libffi embedded in CPython In-Reply-To: References: <549891df.c332e00a.54c7.34f8@mx.google.com> Message-ID: On Thu, Mar 12, 2015 at 6:26 PM, Brett Cannon wrote: > > > For UNIX OSs we could probably rely on the system libffi then. > Yes, it's possible to use the system libffi -- that's what most linux distros already do -- but only if you use dynamic linking. If you want to statically link libffi (useful when embedding, making installations that are a little more portable across systems, or when you're a company like Google that wants to statically link all the things) it's a little more confusing. (The reason Brett started this thread is that I mentioned these issues in an internal Google meeting that he happened to be in ;-P) _ctypes currently contains a bunch of duplicate definitions of the ffi_type_* structs (in Modules/_ctypes/cfield.c.) It also contains a separate implementation of ffi_closure_alloc and ffi_closure_free (in Modules/_ctypes/malloc_closure.c) which is only used on OSX. I guess those redefinitions are to cope with (much) older libffis, but that's just a guess. The redefinitions aren't a problem when using dynamic linking, as long as the new definitions aren't incompatible, but if libffi were to change the ABI of the ffi_type structs, yuck. It can be a problem when statically linking, depending on how you link -- and the only reason it works with the bundled libffi is that ctypes *does not build libffi the normal way*. Instead it runs libffi's configure and then selectively builds a few source files from it, using distutils. It would be a really, really good idea to clean up that mess if we're not going to bundle libffi anymore :-P Screaming-and-stabbing-at-setup.py'ly y'rs, -- Thomas Wouters Hi! I'm an email virus! Think twice before sending your email to help me spread! -------------- next part -------------- An HTML attachment was scrubbed... URL: From cournape at gmail.com Fri Mar 13 09:00:41 2015 From: cournape at gmail.com (David Cournapeau) Date: Fri, 13 Mar 2015 08:00:41 +0000 Subject: [Python-Dev] Why does python use relative instead of absolute path when calling LoadLibrary* In-Reply-To: References: Message-ID: Thank you both for your answers. I will go away with this modification, and see how it goes. David On Thu, Mar 12, 2015 at 2:41 AM, Wes Turner wrote: > > On Mar 11, 2015 3:36 PM, "David Cournapeau" wrote: > > > > Hi, > > > > While looking at the import code of python for C extensions, I was > wondering why we pass a relative path instead of an absolute path to > LoadLibraryEx (see bottom for some context). > > > > In python 2.7, the full path existence was even checked before calling > into LoadLibraryEx ( > https://github.com/python/cpython/blob/2.7/Python/dynload_win.c#L189), > but it looks like this check was removed in python 3.x branch. > > > > Is there any defined behaviour that depends on this path to be relative ? > > Just a guess: does it have to do with resolving symlinks (w/ POSIX > filesystems)? > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bcannon at gmail.com Fri Mar 13 14:39:59 2015 From: bcannon at gmail.com (Brett Cannon) Date: Fri, 13 Mar 2015 13:39:59 +0000 Subject: [Python-Dev] PEP 488: elimination of PYO files In-Reply-To: References: Message-ID: Here is the latest draft of the PEP. I have closed the issue of file name formatting thanks to the informal poll results being very clear on the preferred format and also closed the idea of embedding the optimization level in the bytecode file metadata (that can be another PEP if someone cares to write that one). The only remaining open issue is whether to special-case non-optimized bytecode file names and completely leave out the optimization level in that instance. I think this PEP is close enough to be finished to ask whether Guido wants to wield his BDFL stick for this PEP or if he wants to delegate to a BDFAP. ================================================ PEP: 488 Title: Elimination of PYO files Version: $Revision$ Last-Modified: $Date$ Author: Brett Cannon Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 20-Feb-2015 Post-History: 2015-03-06 2015-03-13 Abstract ======== This PEP proposes eliminating the concept of PYO files from Python. To continue the support of the separation of bytecode files based on their optimization level, this PEP proposes extending the PYC file name to include the optimization level in bytecode repository directory (i.e., the ``__pycache__`` directory). Rationale ========= As of today, bytecode files come in two flavours: PYC and PYO. A PYC file is the bytecode file generated and read from when no optimization level is specified at interpreter startup (i.e., ``-O`` is not specified). A PYO file represents the bytecode file that is read/written when **any** optimization level is specified (i.e., when ``-O`` is specified, including ``-OO``). This means that while PYC files clearly delineate the optimization level used when they were generated -- namely no optimizations beyond the peepholer -- the same is not true for PYO files. Put in terms of optimization levels and the file extension: - 0: ``.pyc`` - 1 (``-O``): ``.pyo`` - 2 (``-OO``): ``.pyo`` The reuse of the ``.pyo`` file extension for both level 1 and 2 optimizations means that there is no clear way to tell what optimization level was used to generate the bytecode file. In terms of reading PYO files, this can lead to an interpreter using a mixture of optimization levels with its code if the user was not careful to make sure all PYO files were generated using the same optimization level (typically done by blindly deleting all PYO files and then using the `compileall` module to compile all-new PYO files [1]_). This issue is only compounded when people optimize Python code beyond what the interpreter natively supports, e.g., using the astoptimizer project [2]_. In terms of writing PYO files, the need to delete all PYO files every time one either changes the optimization level they want to use or are unsure of what optimization was used the last time PYO files were generated leads to unnecessary file churn. The change proposed by this PEP also allows for **all** optimization levels to be pre-compiled for bytecode files ahead of time, something that is currently impossible thanks to the reuse of the ``.pyo`` file extension for multiple optimization levels. As for distributing bytecode-only modules, having to distribute both ``.pyc`` and ``.pyo`` files is unnecessary for the common use-case of code obfuscation and smaller file deployments. Proposal ======== To eliminate the ambiguity that PYO files present, this PEP proposes eliminating the concept of PYO files and their accompanying ``.pyo`` file extension. To allow for the optimization level to be unambiguous as well as to avoid having to regenerate optimized bytecode files needlessly in the `__pycache__` directory, the optimization level used to generate a PYC file will be incorporated into the bytecode file name. Currently bytecode file names are created by ``importlib.util.cache_from_source()``, approximately using the following expression defined by PEP 3147 [3]_, [4]_, [5]_:: '{name}.{cache_tag}.pyc'.format(name=module_name, cache_tag=sys.implementation.cache_tag) This PEP proposes to change the expression to:: '{name}.{cache_tag}.opt-{optimization}.pyc'.format( name=module_name, cache_tag=sys.implementation.cache_tag, optimization=str(sys.flags.optimize)) The "opt-" prefix was chosen so as to provide a visual separator from the cache tag. The placement of the optimization level after the cache tag was chosen to preserve lexicographic sort order of bytecode file names based on module name and cache tag which will not vary for a single interpreter. The "opt-" prefix was chosen over "o" so as to be somewhat self-documenting. The "opt-" prefix was chosen over "O" so as to not have any confusion with "0" while being so close to the interpreter version number. A period was chosen over a hyphen as a separator so as to distinguish clearly that the optimization level is not part of the interpreter version as specified by the cache tag. It also lends to the use of the period in the file name to delineate semantically different concepts. For example, the bytecode file name of ``importlib.cpython-35.pyc`` would become ``importlib.cpython-35.opt-0.pyc``. If ``-OO`` had been passed to the interpreter then instead of ``importlib.cpython-35.pyo`` the file name would be ``importlib.cpython-35.opt-2.pyc``. It should be noted that this change in no way affects the performance of import. Since the import system looks for a single bytecode file based on the optimization level of the interpreter already and generates a new bytecode file if it doesn't exist, the introduction of potentially more bytecode files in the ``__pycache__`` directory has no effect. The interpreter will continue to look for only a single bytecode file based on the optimization level and thus no increase in stat calls will occur. Implementation ============== importlib --------- As ``importlib.util.cache_from_source()`` is the API that exposes bytecode file paths as well as being directly used by importlib, it requires the most critical change. As of Python 3.4, the function's signature is:: importlib.util.cache_from_source(path, debug_override=None) This PEP proposes changing the signature in Python 3.5 to:: importlib.util.cache_from_source(path, debug_override=None, *, optimization=None) The introduced ``optimization`` keyword-only parameter will control what optimization level is specified in the file name. If the argument is ``None`` then the current optimization level of the interpreter will be assumed. Any argument given for ``optimization`` will be passed to ``str()`` and must have ``str.isalnum()`` be true, else ``ValueError`` will be raised (this prevents invalid characters being used in the file name). If the empty string is passed in for ``optimization`` then the addition of the optimization will be suppressed, reverting to the file name format which predates this PEP. It is expected that beyond Python's own 0-2 optimization levels, third-party code will use a hash of optimization names to specify the optimization level, e.g. ``hashlib.sha256(','.join(['dead code elimination', 'constant folding'])).hexdigest()``. While this might lead to long file names, it is assumed that most users never look at the contents of the __pycache__ directory and so this won't be an issue. The ``debug_override`` parameter will be deprecated. As the parameter expects a boolean, the integer value of the boolean will be used as if it had been provided as the argument to ``optimization`` (a ``None`` argument will mean the same as for ``optimization``). A deprecation warning will be raised when ``debug_override`` is given a value other than ``None``, but there are no plans for the complete removal of the parameter at this time (but removal will be no later than Python 4). The various module attributes for importlib.machinery which relate to bytecode file suffixes will be updated [7]_. The ``DEBUG_BYTECODE_SUFFIXES`` and ``OPTIMIZED_BYTECODE_SUFFIXES`` will both be documented as deprecated and set to the same value as ``BYTECODE_SUFFIXES`` (removal of ``DEBUG_BYTECODE_SUFFIXES`` and ``OPTIMIZED_BYTECODE_SUFFIXES`` is not currently planned, but will be not later than Python 4). All various finders and loaders will also be updated as necessary, but updating the previous mentioned parts of importlib should be all that is required. Rest of the standard library ---------------------------- The various functions exposed by the ``py_compile`` and ``compileall`` functions will be updated as necessary to make sure they follow the new bytecode file name semantics [6]_, [1]_. The CLI for the ``compileall`` module will not be directly affected (the ``-b`` flag will be implicit as it will no longer generate ``.pyo`` files when ``-O`` is specified). Compatibility Considerations ============================ Any code directly manipulating bytecode files from Python 3.2 on will need to consider the impact of this change on their code (prior to Python 3.2 -- including all of Python 2 -- there was no __pycache__ which already necessitates bifurcating bytecode file handling support). If code was setting the ``debug_override`` argument to ``importlib.util.cache_from_source()`` then care will be needed if they want the path to a bytecode file with an optimization level of 2. Otherwise only code **not** using ``importlib.util.cache_from_source()`` will need updating. As for people who distribute bytecode-only modules (i.e., use a bytecode file instead of a source file), they will have to choose which optimization level they want their bytecode files to be since distributing a ``.pyo`` file with a ``.pyc`` file will no longer be of any use. Since people typically only distribute bytecode files for code obfuscation purposes or smaller distribution size then only having to distribute a single ``.pyc`` should actually be beneficial to these use-cases. And since the magic number for bytecode files changed in Python 3.5 to support PEP 465 there is no need to support pre-existing ``.pyo`` files [8]_. Rejected Ideas ============== Completely dropping optimization levels from CPython ---------------------------------------------------- Some have suggested that instead of accommodating the various optimization levels in CPython, we should instead drop them entirely. The argument is that significant performance gains would occur from runtime optimizations through something like a JIT and not through pre-execution bytecode optimizations. This idea is rejected for this PEP as that ignores the fact that there are people who do find the pre-existing optimization levels for CPython useful. It also assumes that no other Python interpreter would find what this PEP proposes useful. Alternative formatting of the optimization level in the file name ----------------------------------------------------------------- Using the "opt-" prefix and placing the optimization level between the cache tag and file extension is not critical. All options which have been considered are: * ``importlib.cpython-35.opt-0.pyc`` * ``importlib.cpython-35.opt0.pyc`` * ``importlib.cpython-35.o0.pyc`` * ``importlib.cpython-35.O0.pyc`` * ``importlib.cpython-35.0.pyc`` * ``importlib.cpython-35-O0.pyc`` * ``importlib.O0.cpython-35.pyc`` * ``importlib.o0.cpython-35.pyc`` * ``importlib.0.cpython-35.pyc`` These were initially rejected either because they would change the sort order of bytecode files, possible ambiguity with the cache tag, or were not self-documenting enough. An informal poll was taken and people clearly preferred the formatting proposed by the PEP [9]_. Since this topic is non-technical and of personal choice, the issue is considered solved. Embedding the optimization level in the bytecode metadata --------------------------------------------------------- Some have suggested that rather than embedding the optimization level of bytecode in the file name that it be included in the file's metadata instead. This would mean every interpreter had a single copy of bytecode at any time. Changing the optimization level would thus require rewriting the bytecode, but there would also only be a single file to care about. This has been rejected due to the fact that Python is often installed as a root-level application and thus modifying the bytecode file for modules in the standard library are always possible. In this situation integrators would need to guess at what a reasonable optimization level was for users for any/all situations. By allowing multiple optimization levels to co-exist simultaneously it frees integrators from having to guess what users want and allows users to utilize the optimization level they want. Open Issues =========== Not specifying the optimization level when it is at 0 ----------------------------------------------------- It has been suggested that for the common case of when the optimizations are at level 0 that the entire part of the file name relating to the optimization level be left out. This would allow for file names of ``.pyc`` files to go unchanged, potentially leading to less backwards-compatibility issues (although Python 3.5 introduces a new magic number for bytecode so all bytecode files will have to be regenerated regardless of the outcome of this PEP). It would also allow a potentially redundant bit of information to be left out of the file name if an implementation of Python did not allow for optimizing bytecode. This would only occur, though, if the interpreter didn't support ``-O`` **and** didn't implement the ast module, else users could implement their own optimizations. Arguments against allowing this special case is "explicit is better than implicit" and "special cases aren't special enough to break the rules". At this people have weakly supporting this idea while no one has explicitly come out against it. References ========== .. [1] The compileall module (https://docs.python.org/3/library/compileall.html#module-compileall) .. [2] The astoptimizer project (https://pypi.python.org/pypi/astoptimizer) .. [3] ``importlib.util.cache_from_source()`` ( https://docs.python.org/3.5/library/importlib.html#importlib.util.cache_from_source ) .. [4] Implementation of ``importlib.util.cache_from_source()`` from CPython 3.4.3rc1 ( https://hg.python.org/cpython/file/038297948389/Lib/importlib/_bootstrap.py#l437 ) .. [5] PEP 3147, PYC Repository Directories, Warsaw (http://www.python.org/dev/peps/pep-3147) .. [6] The py_compile module (https://docs.python.org/3/library/compileall.html#module-compileall) .. [7] The importlib.machinery module ( https://docs.python.org/3/library/importlib.html#module-importlib.machinery) .. [8] ``importlib.util.MAGIC_NUMBER`` ( https://docs.python.org/3/library/importlib.html#importlib.util.MAGIC_NUMBER ) .. [9] Informal poll of file name format options on Google+ (https://plus.google.com/u/0/+BrettCannon/posts/fZynLNwHWGm) Copyright ========= This document has been placed in the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End: -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Fri Mar 13 15:50:52 2015 From: brett at python.org (Brett Cannon) Date: Fri, 13 Mar 2015 14:50:52 +0000 Subject: [Python-Dev] Request for Pronouncement: PEP 441 - Improving Python ZIP Application Support In-Reply-To: References: <54EB7DDE.9030405@stoneleaf.us> Message-ID: On Sun, Mar 8, 2015 at 10:35 AM Brett Cannon wrote: > > On Sun, Mar 8, 2015, 08:40 Paul Moore wrote: > > On 26 February 2015 at 21:48, Paul Moore wrote: > > On 26 February 2015 at 21:34, Guido van Rossum wrote: > >> Accepted! > >> > >> Thanks for your patience, Paul, and thanks everyone for their feedback. > >> > >> I know there are still a few small edits to the PEP, but those don't > affect > >> my acceptance. Congrats! > > > > Excellent, thanks to everyone for the helpful comments throughout. > > I'm still looking for someone who has the time to commit the final > patch on http://bugs.python.org/issue23491. Python 3.5a2 is due out > today (at least according to the PEP) so I've probably missed that, > but it would be nice if it could be in for alpha 3 at least. > > If no one gets to it I can submit on Friday. > Submitted in https://hg.python.org/cpython/rev/d1e9f337fea1 . Thanks for the hard work, Paul! -------------- next part -------------- An HTML attachment was scrubbed... URL: From status at bugs.python.org Fri Mar 13 18:08:21 2015 From: status at bugs.python.org (Python tracker) Date: Fri, 13 Mar 2015 18:08:21 +0100 (CET) Subject: [Python-Dev] Summary of Python tracker Issues Message-ID: <20150313170821.CF063560D1@psf.upfronthosting.co.za> ACTIVITY SUMMARY (2015-03-06 - 2015-03-13) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 4817 (+13) closed 30604 (+49) total 35421 (+62) Open issues with patches: 2266 Issues opened (49) ================== #11726: clarify that linecache only works on files that can be decoded http://bugs.python.org/issue11726 reopened by r.david.murray #23566: RFE: faulthandler.register() should support file descriptors http://bugs.python.org/issue23566 reopened by haypo #23599: single and double quotes stripped upon paste with MacPorts lib http://bugs.python.org/issue23599 opened by Jeff Doak #23600: tizinfo.fromutc changed for tzinfo wih StdOffset=0, DstOffset= http://bugs.python.org/issue23600 opened by peterjclaw #23601: use small object allocator for dict key storage http://bugs.python.org/issue23601 opened by jtaylor #23602: Implement __format__ for Fraction http://bugs.python.org/issue23602 opened by tuomas.suutari #23603: Embedding Python3.4 - PyUnicode_Check fails (MinGW-W64) http://bugs.python.org/issue23603 opened by Ashish Sadanandan #23605: Use the new os.scandir() function in os.walk() http://bugs.python.org/issue23605 opened by haypo #23606: ctypes.util.find_library("c") no longer makes sense http://bugs.python.org/issue23606 opened by steve.dower #23607: Inconsistency in datetime.utcfromtimestamp(Decimal) http://bugs.python.org/issue23607 opened by serhiy.storchaka #23609: Export PyModuleObject in moduleobject.h http://bugs.python.org/issue23609 opened by h.venev #23611: Pickle nested names with protocols < 4 http://bugs.python.org/issue23611 opened by serhiy.storchaka #23612: 3.5.0a2 Windows installer does not remove 3.5.0a1 http://bugs.python.org/issue23612 opened by steve.dower #23614: Opaque error message on UTF-8 decoding to surrogates http://bugs.python.org/issue23614 opened by Rosuav #23616: Idle: conflict between loop execution and undo shortcut. http://bugs.python.org/issue23616 opened by Davide Okami #23618: PEP 475: handle EINTR in the socket module http://bugs.python.org/issue23618 opened by haypo #23619: Python 3.5.0a2 installer fails on Windows http://bugs.python.org/issue23619 opened by pmoore #23621: Uninstalling Python 3.5 removes a "py.exe" that was installed http://bugs.python.org/issue23621 opened by pmoore #23622: Deprecate unrecognized backslash+letter escapes http://bugs.python.org/issue23622 opened by serhiy.storchaka #23623: Python 3.5 docs need to clarify how to set PATH, etc http://bugs.python.org/issue23623 opened by pmoore #23624: str.center inconsistent with format "^" http://bugs.python.org/issue23624 opened by Vedran.??a??i?? #23626: Windows per-user install of 3.5a2 doesn't associate .py files http://bugs.python.org/issue23626 opened by pmoore #23628: See if os.scandir() could help speed up importlib http://bugs.python.org/issue23628 opened by brett.cannon #23630: support multiple hosts in create_server/start_server http://bugs.python.org/issue23630 opened by sebastien.bourdeauducq #23631: 3.5 (a2) traceback regression snarls Idle http://bugs.python.org/issue23631 opened by terry.reedy #23632: memoryview doesn't allow tuple-indexing http://bugs.python.org/issue23632 opened by pitrou #23633: Improve py launcher help, index, and doc http://bugs.python.org/issue23633 opened by terry.reedy #23634: os.fdopen reopening a read-only fd on windows http://bugs.python.org/issue23634 opened by mattip #23636: Add scgi to urllib.parse.uses_netloc http://bugs.python.org/issue23636 opened by anthonyryan1 #23637: Warnings error with non-ascii chars. http://bugs.python.org/issue23637 opened by Luk????.N??mec #23639: Not documented special names http://bugs.python.org/issue23639 opened by serhiy.storchaka #23640: Enum.from_bytes() is broken http://bugs.python.org/issue23640 opened by serhiy.storchaka #23641: Got rid of bad dunder names http://bugs.python.org/issue23641 opened by serhiy.storchaka #23642: Interaction of ModuleSpec and C Extension Modules http://bugs.python.org/issue23642 opened by dabeaz #23644: g++ module compile fails with ???_Atomic??? does not name a ty http://bugs.python.org/issue23644 opened by Joshua.J.Cogliati #23645: Incorrect doc for __getslice__ http://bugs.python.org/issue23645 opened by gdementen #23646: PEP 475: handle EINTR in the time module, retry sleep() http://bugs.python.org/issue23646 opened by haypo #23647: imaplib.py MAXLINE value is too low for gmail http://bugs.python.org/issue23647 opened by arnt #23648: PEP 475 meta issue http://bugs.python.org/issue23648 opened by haypo #23649: tarfile not re-entrant for multi-threading http://bugs.python.org/issue23649 opened by sgnn7 #23652: ifdef uses of EPOLLxxx macros so we can compile on systems tha http://bugs.python.org/issue23652 opened by WanderingLogic #23653: Make inspect._empty test to False http://bugs.python.org/issue23653 opened by Lucretiel #23654: infinite loop in faulthandler._stack_overflow http://bugs.python.org/issue23654 opened by WanderingLogic #23655: Memory corruption using pickle over pipe to subprocess http://bugs.python.org/issue23655 opened by nagle #23656: shutil.copyfile (or built-in open) releases sqlite3's transact http://bugs.python.org/issue23656 opened by h-miyajima #23657: Don't do isinstance checks in zipapp http://bugs.python.org/issue23657 opened by brett.cannon #23658: multiprocessing: string arg to SystemExit http://bugs.python.org/issue23658 opened by Dan Nawrocki #23659: csv.register_dialect doc string http://bugs.python.org/issue23659 opened by Vladimir Ulupov #23660: Turtle left/right inverted when using different coordinates or http://bugs.python.org/issue23660 opened by aroberge Most recent 15 issues with no replies (15) ========================================== #23660: Turtle left/right inverted when using different coordinates or http://bugs.python.org/issue23660 #23658: multiprocessing: string arg to SystemExit http://bugs.python.org/issue23658 #23656: shutil.copyfile (or built-in open) releases sqlite3's transact http://bugs.python.org/issue23656 #23653: Make inspect._empty test to False http://bugs.python.org/issue23653 #23652: ifdef uses of EPOLLxxx macros so we can compile on systems tha http://bugs.python.org/issue23652 #23648: PEP 475 meta issue http://bugs.python.org/issue23648 #23628: See if os.scandir() could help speed up importlib http://bugs.python.org/issue23628 #23626: Windows per-user install of 3.5a2 doesn't associate .py files http://bugs.python.org/issue23626 #23622: Deprecate unrecognized backslash+letter escapes http://bugs.python.org/issue23622 #23621: Uninstalling Python 3.5 removes a "py.exe" that was installed http://bugs.python.org/issue23621 #23609: Export PyModuleObject in moduleobject.h http://bugs.python.org/issue23609 #23585: patchcheck doesn't depend on all http://bugs.python.org/issue23585 #23584: test_doctest lineendings fails in verbose mode http://bugs.python.org/issue23584 #23578: struct.pack error messages do not indicate which argument was http://bugs.python.org/issue23578 #23577: Add tests for wsgiref.validate http://bugs.python.org/issue23577 Most recent 15 issues waiting for review (15) ============================================= #23657: Don't do isinstance checks in zipapp http://bugs.python.org/issue23657 #23654: infinite loop in faulthandler._stack_overflow http://bugs.python.org/issue23654 #23652: ifdef uses of EPOLLxxx macros so we can compile on systems tha http://bugs.python.org/issue23652 #23649: tarfile not re-entrant for multi-threading http://bugs.python.org/issue23649 #23647: imaplib.py MAXLINE value is too low for gmail http://bugs.python.org/issue23647 #23646: PEP 475: handle EINTR in the time module, retry sleep() http://bugs.python.org/issue23646 #23645: Incorrect doc for __getslice__ http://bugs.python.org/issue23645 #23641: Got rid of bad dunder names http://bugs.python.org/issue23641 #23640: Enum.from_bytes() is broken http://bugs.python.org/issue23640 #23632: memoryview doesn't allow tuple-indexing http://bugs.python.org/issue23632 #23630: support multiple hosts in create_server/start_server http://bugs.python.org/issue23630 #23622: Deprecate unrecognized backslash+letter escapes http://bugs.python.org/issue23622 #23618: PEP 475: handle EINTR in the socket module http://bugs.python.org/issue23618 #23611: Pickle nested names with protocols < 4 http://bugs.python.org/issue23611 #23607: Inconsistency in datetime.utcfromtimestamp(Decimal) http://bugs.python.org/issue23607 Top 10 most discussed issues (10) ================================= #23605: Use the new os.scandir() function in os.walk() http://bugs.python.org/issue23605 24 msgs #23619: Python 3.5.0a2 installer fails on Windows http://bugs.python.org/issue23619 23 msgs #23496: Steps for Android Native Build of Python 3.4.2 http://bugs.python.org/issue23496 14 msgs #23246: distutils fails to locate vcvarsall with Visual C++ Compiler f http://bugs.python.org/issue23246 11 msgs #23566: RFE: faulthandler.register() should support file descriptors http://bugs.python.org/issue23566 11 msgs #23602: Implement __format__ for Fraction http://bugs.python.org/issue23602 10 msgs #23632: memoryview doesn't allow tuple-indexing http://bugs.python.org/issue23632 10 msgs #23524: Use _set_thread_local_invalid_parameter_handler in posixmodule http://bugs.python.org/issue23524 9 msgs #23633: Improve py launcher help, index, and doc http://bugs.python.org/issue23633 9 msgs #12916: Add inspect.splitdoc http://bugs.python.org/issue12916 8 msgs Issues closed (46) ================== #5527: multiprocessing won't work with Tkinter (under Linux) http://bugs.python.org/issue5527 closed by davin #7503: multiprocessing AuthenticationError "digest sent was rejected" http://bugs.python.org/issue7503 closed by davin #9191: winreg.c:Reg2Py() may leak memory (in unusual circumstances) http://bugs.python.org/issue9191 closed by Claudiu.Popa #10805: traceback.print_exception throws AttributeError when exception http://bugs.python.org/issue10805 closed by Claudiu.Popa #11705: sys.excepthook doesn't work in imported modules http://bugs.python.org/issue11705 closed by Claudiu.Popa #17403: Robotparser fails to parse some robots.txt http://bugs.python.org/issue17403 closed by berker.peksag #19909: Best practice docs for new SSL features http://bugs.python.org/issue19909 closed by christian.heimes #20617: test_ssl does not use mock http://bugs.python.org/issue20617 closed by berker.peksag #20659: Want to make a class method a property by combining decorators http://bugs.python.org/issue20659 closed by ncoghlan #20876: python -m test test_pathlib fails http://bugs.python.org/issue20876 closed by pitrou #21610: load_module not closing opened files http://bugs.python.org/issue21610 closed by python-dev #22522: sys.excepthook doesn't receive the traceback when called from http://bugs.python.org/issue22522 closed by Claudiu.Popa #22524: PEP 471 implementation: os.scandir() directory scanning functi http://bugs.python.org/issue22524 closed by haypo #22853: Multiprocessing.Queue._feed deadlocks on import http://bugs.python.org/issue22853 closed by serhiy.storchaka #22928: HTTP header injection in urrlib2/urllib/httplib/http.client http://bugs.python.org/issue22928 closed by serhiy.storchaka #23003: traceback.{print_exc,print_exception,format_exc,format_excepti http://bugs.python.org/issue23003 closed by berker.peksag #23051: multiprocessing.pool methods imap()[_unordered()] deadlock http://bugs.python.org/issue23051 closed by serhiy.storchaka #23081: Document PySequence_List(o) as equivalent to list(o) http://bugs.python.org/issue23081 closed by berker.peksag #23138: cookiejar parses cookie value as int with empty name-value pai http://bugs.python.org/issue23138 closed by serhiy.storchaka #23192: Generator return value ignored in lambda function http://bugs.python.org/issue23192 closed by serhiy.storchaka #23279: test_site/test_startup_imports fails when mpl_toolkit or logil http://bugs.python.org/issue23279 closed by brett.cannon #23432: Duplicate content in SystemExit documentation http://bugs.python.org/issue23432 closed by berker.peksag #23467: Improve byte formatting compatibility between Py2 and Py3 http://bugs.python.org/issue23467 closed by python-dev #23486: Enum member lookup is 20x slower than normal class attribute l http://bugs.python.org/issue23486 closed by python-dev #23491: PEP 441 - Improving Python Zip Application Support http://bugs.python.org/issue23491 closed by brett.cannon #23526: Silence resource warnings in test_httplib http://bugs.python.org/issue23526 closed by vadmium #23571: Raise SystemError if a function returns a result with an excep http://bugs.python.org/issue23571 closed by haypo #23579: Amazon.com links http://bugs.python.org/issue23579 closed by python-dev #23581: unittest.mock.MagicMock doesn't support matmul (@) operator http://bugs.python.org/issue23581 closed by berker.peksag #23595: Split the math module into _math (C) + math (py) http://bugs.python.org/issue23595 closed by haypo #23598: No backspace on curses on iTerm (mac) http://bugs.python.org/issue23598 closed by ned.deily #23604: Python 3.4 and 2.7 installation no Script folder and no pip in http://bugs.python.org/issue23604 closed by Daiyue Weng #23608: ssl.VERIFY_X509_TRUSTED_FIRST: Incorrect value for versionadde http://bugs.python.org/issue23608 closed by python-dev #23610: -m arg behavior http://bugs.python.org/issue23610 closed by zach.ware #23613: searchindex.js is annoying http://bugs.python.org/issue23613 closed by georg.brandl #23615: Reloading tokenize breaks tokenize.open() http://bugs.python.org/issue23615 closed by serhiy.storchaka #23617: Incorrect plural "includes" on Python Standard Library front p http://bugs.python.org/issue23617 closed by ned.deily #23620: cross-type comparison is different in python minor versions http://bugs.python.org/issue23620 closed by r.david.murray #23625: load_module() docs: zipped eggs are not loaded. http://bugs.python.org/issue23625 closed by brett.cannon #23627: Incorrect string formatting of float values in Python 2.7 http://bugs.python.org/issue23627 closed by eric.smith #23629: Default __sizeof__ is wrong for var-sized objects http://bugs.python.org/issue23629 closed by pitrou #23635: Python won't install correctly. http://bugs.python.org/issue23635 closed by Player72 #23638: shutil.copytree makedirs exception http://bugs.python.org/issue23638 closed by r.david.murray #23643: Segmentation Fault with Large Simple Function http://bugs.python.org/issue23643 closed by Michael Klein #23650: xml.etree.ElementTreee.write can't parse its own output http://bugs.python.org/issue23650 closed by ned.deily #23651: Typo in argparse allow_abrev http://bugs.python.org/issue23651 closed by berker.peksag From stefano.borini at ferrara.linux.it Fri Mar 13 19:25:36 2015 From: stefano.borini at ferrara.linux.it (Stefano Borini) Date: Fri, 13 Mar 2015 19:25:36 +0100 Subject: [Python-Dev] Why does python use relative instead of absolute path when calling LoadLibrary* In-Reply-To: References: Message-ID: <55032BA0.2030706@ferrara.linux.it> On 3/11/15 9:36 PM, David Cournapeau wrote: > Hi, > > While looking at the import code of python for C extensions, I was > wondering why we pass a relative path instead of an absolute path to > LoadLibraryEx (see bottom for some context). > > In python 2.7, the full path existence was even checked before calling > into LoadLibraryEx > (https://github.com/python/cpython/blob/2.7/Python/dynload_win.c#L189), > but it looks like this check was removed in python 3.x branch. > > Is there any defined behaviour that depends on this path to be relative ? My two cents. According to the MSDN documentation, passing a relative path is actually undefined https://msdn.microsoft.com/en-us/library/windows/desktop/ms684179%28v=vs.85%29.aspx ---------------------------- LOAD_WITH_ALTERED_SEARCH_PATH 0x00000008 If this value is used and lpFileName specifies an absolute path, the system uses the alternate file search strategy discussed in the Remarks section to find associated executable modules that the specified module causes to be loaded. If this value is used and lpFileName specifies a relative path, the behavior is undefined. --------------------------- meaning that wpathname _must_ be absolute. https://github.com/python/cpython/blob/master/Python/dynload_win.c#L222 From francismb at email.de Sat Mar 14 12:04:57 2015 From: francismb at email.de (francis) Date: Sat, 14 Mar 2015 12:04:57 +0100 Subject: [Python-Dev] Request for comments: [issue22941] IPv4Interface arithmetic changes subnet mask In-Reply-To: <5501E109.7090408@trueblade.com> References: <927E0DF62C47C940A006AFF297D7F512120C11@DEFTHW99EK0MSX.ww902.siemens.net> <20150312150023.04051b02@fsol> <5501A6CA.2080301@trueblade.com> <5501D601.9090105@scottdial.com> <20150312191717.30acbd7b@fsol> <5501E109.7090408@trueblade.com> Message-ID: <550415D9.8000100@email.de> Hi, > I, too, was thinking /24. I think that overflowing the host portion > should raise OverflowError. > Just curiosity, why not a modulo calculation on the subnet instead of raising the error? Thanks in advance! francis From francismb at email.de Sat Mar 14 12:34:48 2015 From: francismb at email.de (francis) Date: Sat, 14 Mar 2015 12:34:48 +0100 Subject: [Python-Dev] (ctypes example) libffi embedded in CPython In-Reply-To: References: <549891df.c332e00a.54c7.34f8@mx.google.com> <20150311185028.6f05973c@fsol> <20150311190557.25eebc16@fsol> <20150311191750.7cf23a40@fsol> Message-ID: <55041CD8.4010608@email.de> Hi > > It might be a matter of taste, but I don't find declaring C functions > any more awkward than using strange interface that ctypes comes with. > the equivalent in cffi would be ffi.cast("double (*)()", x) - Could you please elaborate on "using strange interface"? - Is there an easy way to convert the current code using ctypes to cffi? Thanks in advance! francis From doko at ubuntu.com Sat Mar 14 15:42:45 2015 From: doko at ubuntu.com (Matthias Klose) Date: Sat, 14 Mar 2015 15:42:45 +0100 Subject: [Python-Dev] libffi embedded in CPython In-Reply-To: References: <549891df.c332e00a.54c7.34f8@mx.google.com> Message-ID: <550448E5.8030405@ubuntu.com> On 03/11/2015 06:27 PM, Brett Cannon wrote: > On Mon, Dec 22, 2014 at 4:49 PM Jim J. Jewett wrote: > >> >> >> On Thu, Dec 18, 2014, at 14:13, Maciej Fijalkowski wrote: >>> ... http://bugs.python.org/issue23085 ... >>> is there any reason any more for libffi being included in CPython? >> >> >> Paul Moore wrote: >>> Probably the easiest way of moving this forward would be for someone >>> to identify the CPython-specific patches in the current version ... >> >> Christian Heimes wrote: >>> That's easy. All patches are tracked in the diff file >>> https://hg.python.org/cpython/file/3de678cd184d/Modules/_ >> ctypes/libffi.diff >> >> That (200+ lines) doesn't seem to have all the C changes, such as the >> win64 sizeof changes from issue 11835. >> >> Besides http://bugs.python.org/issue23085, there is at least >> http://bugs.python.org/issue22733 >> http://bugs.python.org/issue20160 >> http://bugs.python.org/issue11835 >> >> which sort of drives home the point that making sure we have a >> good merge isn't trivial, and this isn't an area where we should >> just assume that tests will catch everything. I don't think it >> is just a quicky waiting on permission. >> >> I've no doubt that upstream libffi is better in many ways, but >> those are ways people have already learned to live with. >> >> That said, I haven't seen any objections in principle, except >> perhaps from Steve Dower in the issues. (I *think* he was >> just saying "not worth the time to me", but it was ambiguous.) >> >> I do believe that Christian or Maciej *could* sort things out well >> enough; I have no insight into whether they have (or someone else >> has) the time to actually do so. >> > > Did anyone ever step forward to do this? I'm a bit worried about the > long-term viability of ctypes if we don't have a maintainer or at least > someone making sure we are staying up-to-date with upstream libffi. my plan was to merge the next libffi release, something along the lines which is released with GCC 5. But this release isn't yet done. From eric at trueblade.com Sat Mar 14 21:52:27 2015 From: eric at trueblade.com (Eric V. Smith) Date: Sat, 14 Mar 2015 16:52:27 -0400 Subject: [Python-Dev] Request for comments: [issue22941] IPv4Interface arithmetic changes subnet mask In-Reply-To: <550415D9.8000100@email.de> References: <927E0DF62C47C940A006AFF297D7F512120C11@DEFTHW99EK0MSX.ww902.siemens.net> <20150312150023.04051b02@fsol> <5501A6CA.2080301@trueblade.com> <5501D601.9090105@scottdial.com> <20150312191717.30acbd7b@fsol> <5501E109.7090408@trueblade.com> <550415D9.8000100@email.de> Message-ID: <55049F8B.9000504@trueblade.com> On 3/14/2015 7:04 AM, francis wrote: > Hi, >> I, too, was thinking /24. I think that overflowing the host portion >> should raise OverflowError. >> > Just curiosity, why not a modulo calculation on the subnet instead > of raising the error? Personally, I can't imaging wanting that behavior. I can't say I've ever needed additional at all with an IP address, but if I did, it would only be to stay within the host portion. To wrap around seems odd. If you had a /24 address and added 1000, should that really be the same as adding 232 (= 1000 % 256)? It seems more likely it's an error. I'm +0 on adding addition with an OverflowError for the host portion, and -1 on addition with modulo arithmetic. -- Eric. From eric at trueblade.com Sat Mar 14 22:15:22 2015 From: eric at trueblade.com (Eric V. Smith) Date: Sat, 14 Mar 2015 17:15:22 -0400 Subject: [Python-Dev] Request for comments: [issue22941] IPv4Interface arithmetic changes subnet mask In-Reply-To: <55049F8B.9000504@trueblade.com> References: <927E0DF62C47C940A006AFF297D7F512120C11@DEFTHW99EK0MSX.ww902.siemens.net> <20150312150023.04051b02@fsol> <5501A6CA.2080301@trueblade.com> <5501D601.9090105@scottdial.com> <20150312191717.30acbd7b@fsol> <5501E109.7090408@trueblade.com> <550415D9.8000100@email.de> <55049F8B.9000504@trueblade.com> Message-ID: <5504A4EA.7080509@trueblade.com> On 3/14/2015 4:52 PM, Eric V. Smith wrote: > On 3/14/2015 7:04 AM, francis wrote: >> Hi, >>> I, too, was thinking /24. I think that overflowing the host portion >>> should raise OverflowError. >>> >> Just curiosity, why not a modulo calculation on the subnet instead >> of raising the error? > > Personally, I can't imaging wanting that behavior. I can't say I've ever > needed additional at all with an IP address, but if I did, it would only That should be "I can't say I've ever needed addition at all...". > be to stay within the host portion. To wrap around seems odd. If you had > a /24 address and added 1000, should that really be the same as adding > 232 (= 1000 % 256)? It seems more likely it's an error. > > I'm +0 on adding addition with an OverflowError for the host portion, > and -1 on addition with modulo arithmetic. -- Eric. From larry at hastings.org Sun Mar 15 03:48:05 2015 From: larry at hastings.org (Larry Hastings) Date: Sat, 14 Mar 2015 19:48:05 -0700 Subject: [Python-Dev] Minor update to Python 3.5 release schedule Message-ID: <5504F2E5.3030707@hastings.org> I always intended all my releases to be on Sundays--that all the release engineering work is done on weekends, which is generally easier for everybody. But I goofed up the 3.5 release schedule and had proposed 3.5.0a3 to be released Saturday March 28th. With the assent of the team I bumped it forward a day to Sunday March 29th. I apologize for my goof. Cheers, //arry/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From stephen at xemacs.org Sun Mar 15 06:22:06 2015 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Sun, 15 Mar 2015 14:22:06 +0900 Subject: [Python-Dev] Request for comments: [issue22941] IPv4Interface arithmetic changes subnet mask In-Reply-To: <55049F8B.9000504@trueblade.com> References: <927E0DF62C47C940A006AFF297D7F512120C11@DEFTHW99EK0MSX.ww902.siemens.net> <20150312150023.04051b02@fsol> <5501A6CA.2080301@trueblade.com> <5501D601.9090105@scottdial.com> <20150312191717.30acbd7b@fsol> <5501E109.7090408@trueblade.com> <550415D9.8000100@email.de> <55049F8B.9000504@trueblade.com> Message-ID: <87oanuesxt.fsf@uwakimon.sk.tsukuba.ac.jp> Eric V. Smith writes: > Personally, I can't imaging wanting that behavior. I can't say I've ever > needed additional at all with an IP address, but if I did, it would only > be to stay within the host portion. To wrap around seems odd. It's worse than odd. I've occasionally had use for iterating something like a "ping" over a network, and if I make an off-by-one error "boom" I hit the network address or (probably worse) the broadcast address. The ipaddress module appears to have an __iter__ method on IPv4Network that dtrts. I don't see a real need for an addition operator given that. From ethan at stoneleaf.us Sun Mar 15 06:52:17 2015 From: ethan at stoneleaf.us (Ethan Furman) Date: Sat, 14 Mar 2015 22:52:17 -0700 Subject: [Python-Dev] backwards and forwards compatibility, the exact contents of pickle files, and IntEnum Message-ID: <55051E11.8030603@stoneleaf.us> I'm not sure exactly how to phrase this inquiry, so please bear with me. What exactly does backwards compatibility mean as far as pickle goes? We have the various protocols, we have the contents of pickle files created at those protocols, and we have different versions of Python. Should a pickle file created in Python 3.4 with protocol 3 and the contents of socket.AF_INET be unpicklable with a Python 3.3 system? Because currently it cannot be as socket.AF_INET became an IntEnum in 3.4. I'm thinking the answer is yes, it should be. So how do we fix it? There are a couple different options: - modify IntEnum pickle methods to return the name only - modify IntEnum pickle methods to pickle just like the int they represent The first option has the advantage that in 3.4 and above, you'll get back the IntEnum version. The second option has the advantage that the actual pickle contents are the same [1] as in previous versions. So, the final question: Do the contents of a pickle file at a certain protocol have to be the some between versions, or is it enough if unpickling them returns the correct object? -- ~Ethan~ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: OpenPGP digital signature URL: From ethan at stoneleaf.us Sun Mar 15 07:05:20 2015 From: ethan at stoneleaf.us (Ethan Furman) Date: Sat, 14 Mar 2015 23:05:20 -0700 Subject: [Python-Dev] backwards and forwards compatibility, the exact contents of pickle files, and IntEnum In-Reply-To: <55051E11.8030603@stoneleaf.us> References: <55051E11.8030603@stoneleaf.us> Message-ID: <55052120.2030705@stoneleaf.us> [1] The second option hasn't been coded yet, but the first one has. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: OpenPGP digital signature URL: From v+python at g.nevcal.com Sun Mar 15 07:46:04 2015 From: v+python at g.nevcal.com (Glenn Linderman) Date: Sat, 14 Mar 2015 23:46:04 -0700 Subject: [Python-Dev] backwards and forwards compatibility, the exact contents of pickle files, and IntEnum In-Reply-To: <55051E11.8030603@stoneleaf.us> References: <55051E11.8030603@stoneleaf.us> Message-ID: <55052AAC.6090907@g.nevcal.com> On 3/14/2015 10:52 PM, Ethan Furman wrote: > I'm not sure exactly how to phrase this inquiry, so please bear with me. > > What exactly does backwards compatibility mean as far as pickle goes? We have the various protocols, we have the > contents of pickle files created at those protocols, and we have different versions of Python. > > Should a pickle file created in Python 3.4 with protocol 3 and the contents of socket.AF_INET be unpicklable with a > Python 3.3 system? Because currently it cannot be as socket.AF_INET became an IntEnum in 3.4. > > I'm thinking the answer is yes, it should be. > > So how do we fix it? There are a couple different options: > > - modify IntEnum pickle methods to return the name only > > - modify IntEnum pickle methods to pickle just like the int they represent > > The first option has the advantage that in 3.4 and above, you'll get back the IntEnum version. > > The second option has the advantage that the actual pickle contents are the same [1] as in previous versions. > > So, the final question: Do the contents of a pickle file at a certain protocol have to be the some between versions, or > is it enough if unpickling them returns the correct object? In general, it would seem to me that each version of Python could add object types that are foreign to prior versions, and so as long as version N can unpickle anything produced by versions <=N, all would be well. While I am _not_ an expert in pickle, this would just be my expectation based on years of experience with compatibility issues in non-Python non-pickle systems. Pickle may have stronger compatibility goals than I would expect, and so must you, if you are thinking the answer to the above is "yes, it should be". :) -------------- next part -------------- An HTML attachment was scrubbed... URL: From ethan at stoneleaf.us Sun Mar 15 07:58:23 2015 From: ethan at stoneleaf.us (Ethan Furman) Date: Sat, 14 Mar 2015 23:58:23 -0700 Subject: [Python-Dev] backwards and forwards compatibility, the exact contents of pickle files, and IntEnum In-Reply-To: <55052AAC.6090907@g.nevcal.com> References: <55051E11.8030603@stoneleaf.us> <55052AAC.6090907@g.nevcal.com> Message-ID: <55052D8F.8080601@stoneleaf.us> On 03/14/2015 11:46 PM, Glenn Linderman wrote: > In general, it would seem to me that each version of Python could add object types that are foreign to prior versions, > and so as long as version N can unpickle anything produced by versions <=N, all would be well. The question to me is because we not so much adding something brand-new (socket.AF_INET has been there for ages), as changing the type of something already existing -- I'm just not sure if we need "end-result" compatibility (achieved by the first option) or if we need "bug for bug" compatibility (achieved by the second option). -- ~Ethan~ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: OpenPGP digital signature URL: From storchaka at gmail.com Sun Mar 15 08:13:55 2015 From: storchaka at gmail.com (Serhiy Storchaka) Date: Sun, 15 Mar 2015 09:13:55 +0200 Subject: [Python-Dev] backwards and forwards compatibility, the exact contents of pickle files, and IntEnum In-Reply-To: <55051E11.8030603@stoneleaf.us> References: <55051E11.8030603@stoneleaf.us> Message-ID: On 15.03.15 07:52, Ethan Furman wrote: > So how do we fix it? There are a couple different options: > > - modify IntEnum pickle methods to return the name only > > - modify IntEnum pickle methods to pickle just like the int they represent > > The first option has the advantage that in 3.4 and above, you'll get back the IntEnum version. > > The second option has the advantage that the actual pickle contents are the same [1] as in previous versions. > > So, the final question: Do the contents of a pickle file at a certain protocol have to be the some between versions, or > is it enough if unpickling them returns the correct object? With the second option you lost the type even for 3.5+. This is a step back. From ethan at stoneleaf.us Sun Mar 15 15:20:00 2015 From: ethan at stoneleaf.us (Ethan Furman) Date: Sun, 15 Mar 2015 07:20:00 -0700 Subject: [Python-Dev] backwards and forwards compatibility, the exact contents of pickle files, and IntEnum In-Reply-To: <5505324E.3090205@g.nevcal.com> References: <55051E11.8030603@stoneleaf.us> <55052AAC.6090907@g.nevcal.com> <55052D8F.8080601@stoneleaf.us> <5505324E.3090205@g.nevcal.com> Message-ID: <55059510.2020806@stoneleaf.us> Bringing back on list. :) On 03/15/2015 12:18 AM, Glenn Linderman wrote: > On 3/14/2015 11:58 PM, Ethan Furman wrote: >> On 03/14/2015 11:46 PM, Glenn Linderman wrote: >> >>> In general, it would seem to me that each version of Python could >>> add object types that are foreign to prior versions, and so as long >>> as version N can unpickle anything produced by versions <=N, all >>> would be well. >> >> The question to me is because we are not so much adding something >> brand-new (socket.AF_INET has been there for ages), as changing the >> type of something already existing -- I'm just not sure if we need >> "end-result" compatibility (achieved by the first option) or if we >> need "bug for bug" compatibility (achieved by the second option). > > Yes, the fact that it is not brand-new does add a mind-twist to the compatibility expectations. But while it is not > brand-new, it is changed. Exactly. IntEnum was advertised as being a "drop-in replacement" with the only change being better reprs and strs; so being unable to un-pickle in prior versions invalidates that claim. Part 2 of my question (Part 1 of which is "which method do we use to fix the IntEnum being used in the stdlib") is: do we just change the IntEnum subclasses (such as socket.AddressFamily) to have the fixed behavior, or do we change IntEnum itself? Having thought about it for a few minutes, we cannot make that change in IntEnum itself as it only works if the names are exported to the global namespace -- so this fix will work for the stdlib (since we export the names), but won't work otherwise. Unless someone steps forward and says that the actual pickle contents must be the same, I'm going to go with pickling the name: this is both unpicklable in prior versions, and doesn't lose the type in present versions. Just to be clear, there are no issue unpickling in the same version, only in prior versions. -- ~Ethan~ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: OpenPGP digital signature URL: From solipsis at pitrou.net Sun Mar 15 16:10:51 2015 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sun, 15 Mar 2015 16:10:51 +0100 Subject: [Python-Dev] backwards and forwards compatibility, the exact contents of pickle files, and IntEnum References: <55051E11.8030603@stoneleaf.us> Message-ID: <20150315161051.77d5e7ca@fsol> On Sat, 14 Mar 2015 22:52:17 -0700 Ethan Furman wrote: > I'm not sure exactly how to phrase this inquiry, so please bear with me. > > What exactly does backwards compatibility mean as far as pickle goes? We have the various protocols, we have the > contents of pickle files created at those protocols, and we have different versions of Python. > > Should a pickle file created in Python 3.4 with protocol 3 and the contents of socket.AF_INET be unpicklable with a > Python 3.3 system? Because currently it cannot be as socket.AF_INET became an IntEnum in 3.4. > > I'm thinking the answer is yes, it should be. I agree with you that this should not break. > So, the final question: Do the contents of a pickle file at a certain protocol have to be the some between versions, or > is it enough if unpickling them returns the correct object? It's certainly enough if unpickling returns the correct object. Exact serialization is an implementation detail. Regards Antoine. From ncoghlan at gmail.com Mon Mar 16 00:01:57 2015 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 16 Mar 2015 09:01:57 +1000 Subject: [Python-Dev] Request for Pronouncement: PEP 441 - Improving Python ZIP Application Support In-Reply-To: References: <54EB7DDE.9030405@stoneleaf.us> Message-ID: Huzzah! Thanks for working on this one folks, it should make the zipfile execution support much easier for current and future Python users to discover :) Cheers, Nick. -------------- next part -------------- An HTML attachment was scrubbed... URL: From encukou at gmail.com Mon Mar 16 13:38:13 2015 From: encukou at gmail.com (Petr Viktorin) Date: Mon, 16 Mar 2015 13:38:13 +0100 Subject: [Python-Dev] PEP 489: Redesigning extension module loading Message-ID: <5506CEB5.7050105@gmail.com> Hello, On import-sig, I've agreed to continue Nick Coghlan's work on making extension modules act more like Python ones, work well with PEP 451 (ModuleSpec), and encourage proper subinterpreter and reloading support. Here is the resulting PEP. I don't have a patch yet, but I'm working on it. There's a remaining open issue: providing a tool that can be run in test suites to check if a module behaves well with subinterpreters/reloading. I believe it's out of scope for this PEP but speak out if you disagree. Please discuss on import-sig. ======================= PEP: 489 Title: Redesigning extension module loading Version: $Revision$ Last-Modified: $Date$ Author: Petr Viktorin , Stefan Behnel , Nick Coghlan Discussions-To: import-sig at python.org Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 11-Aug-2013 Python-Version: 3.5 Post-History: 23-Aug-2013, 20-Feb-2015 Resolution: Abstract ======== This PEP proposes a redesign of the way in which extension modules interact with the import machinery. This was last revised for Python 3.0 in PEP 3121, but did not solve all problems at the time. The goal is to solve them by bringing extension modules closer to the way Python modules behave; specifically to hook into the ModuleSpec-based loading mechanism introduced in PEP 451. Extensions that do not require custom memory layout for their module objects may be executed in arbitrary pre-defined namespaces, paving the way for extension modules being runnable with Python's ``-m`` switch. Other extensions can use custom types for their module implementation. Module types are no longer restricted to types.ModuleType. This proposal makes it easy to support properties at the module level and to safely store arbitrary global state in the module that is covered by normal garbage collection and supports reloading and sub-interpreters. Extension authors are encouraged to take these issues into account when using the new API. Motivation ========== Python modules and extension modules are not being set up in the same way. For Python modules, the module is created and set up first, then the module code is being executed (PEP 302). A ModuleSpec object (PEP 451) is used to hold information about the module, and passed to the relevant hooks. For extensions, i.e. shared libraries, the module init function is executed straight away and does both the creation and initialisation. The initialisation function is not passed ModuleSpec information about the loaded module, such as the __file__ or fully-qualified name. This hinders relative imports and resource loading. This is specifically a problem for Cython generated modules, for which it's not uncommon that the module init code has the same level of complexity as that of any 'regular' Python module. Also, the lack of __file__ and __name__ information hinders the compilation of __init__.py modules, i.e. packages, especially when relative imports are being used at module init time. The other disadvantage of the discrepancy is that existing Python programmers learning C cannot effectively map concepts between the two domains. As long as extension modules are fundamentally different from pure Python ones in the way they're initialised, they are harder for people to pick up without relying on something like cffi, SWIG or Cython to handle the actual extension module creation. Currently, extension modules are also not added to sys.modules until they are fully initialized, which means that a (potentially transitive) re-import of the module will really try to reimport it and thus run into an infinite loop when it executes the module init function again. Without the fully qualified module name, it is not trivial to correctly add the module to sys.modules either. Furthermore, the majority of currently existing extension modules has problems with sub-interpreter support and/or reloading, and, while it is possible with the current infrastructure to support these features, it is neither easy nor efficient. Addressing these issues was the goal of PEP 3121, but many extensions, including some in the standard library, took the least-effort approach to porting to Python 3, leaving these issues unresolved. This PEP keeps the backwards-compatible behavior, which should reduce pressure and give extension authors adequate time to consider these issues when porting. The current process =================== Currently, extension modules export an initialisation function named "PyInit_modulename", named after the file name of the shared library. This function is executed by the import machinery and must return either NULL in the case of an exception, or a fully initialised module object. The function receives no arguments, so it has no way of knowing about its import context. During its execution, the module init function creates a module object based on a PyModuleDef struct. It then continues to initialise it by adding attributes to the module dict, creating types, etc. In the back, the shared library loader keeps a note of the fully qualified module name of the last module that it loaded, and when a module gets created that has a matching name, this global variable is used to determine the fully qualified name of the module object. This is not entirely safe as it relies on the module init function creating its own module object first, but this assumption usually holds in practice. The proposal ============ The current extension module initialisation will be deprecated in favour of a new initialisation scheme. Since the current scheme will continue to be available, existing code will continue to work unchanged, including binary compatibility. Extension modules that support the new initialisation scheme must export the public symbol "PyModuleExec_modulename", and optionally "PyModuleCreate_modulename", where "modulename" is the name of the module. This mimics the previous naming convention for the "PyInit_modulename" function. If defined, these symbols must resolve to C functions with the following signatures, respectively:: int (*PyModuleExecFunction)(PyObject* module) PyObject* (*PyModuleCreateFunction)(PyObject* module_spec) The PyModuleExec function ------------------------- The PyModuleExec function is used to implement "loader.exec_module" defined in PEP 451. It function will be called to initialize a module. (Usually, this amounts to setting the module's initial attributes.) This happens in two situations: when the module is first initialized for a given (sub-)interpreter, and possibly later when the module is reloaded. When PyModuleExec is called, the module has already been added to sys.modules, and import-related attributes specified in PEP 451 [#pep-0451-attributes]_) have been set on the module. The "module" argument receives the module object to initialize. If PyModuleCreate is defined, "module" will generally be the the object returned by it. It is possible for a custom loader to pass any object to PyModuleExec, so this function should check and fail with TypeError if the module's type is unsupported. Any other assumptions should also be checked. If PyModuleCreate is not defined, PyModuleExec is expected to operate on any Python object for which attributes can be added by PyObject_GetAttr* and retrieved by PyObject_SetAttr*. This allows loading an extension into a pre-created module, making it possible to run it as __main__ in the future, participate in certain lazy-loading schemes [#lazy_import_concerns]_, or enable other creative uses. If PyModuleExec replaces the module's entry in sys.modules, the new object will be used and returned by importlib machinery. (This mirrors the behavior of Python modules. Note that for extensions, implementing PyModuleCreate is usually a better solution for the use cases this serves.) The function must return ``0`` on success, or, on error, set an exception and return ``-1``. The PyModuleCreate function --------------------------- The optional PyModuleCreate function is used to implement "loader.create_module" defined in PEP 451. By exporting it, an extension module indicates that it uses a custom module object. This prevents loading the extension in a pre-created module, but gives greater flexibility in allowing a custom C-level layout of the module object. Most extensions will not need to implement this function. The "module_spec" argument receives a "ModuleSpec" instance, as defined in PEP 451. When called, this function must create and return a module object, or set an exception and return NULL. There is no requirement for the returned object to be an instance of types.ModuleType. Any type can be used, as long as it supports setting and getting attributes, including at least the import-related attributes specified in PEP 451 [#pep-0451-attributes]_. This follows the current support for allowing arbitrary objects in sys.modules and makes it easier for extension modules to define a type that exactly matches their needs for holding module state. Note that when this function is called, the module's entry in sys.modules is not populated yet. Attempting to import the same module again (possibly transitively), may lead to an infinite loop. Extension authors are advised to keep PyModuleCreate minimal, an in particular to not call user code from it. If PyModuleCreate is not defined, the default loader will construct a module object as if with PyModule_New. Initialization helper functions ------------------------------- For two initialization tasks previously done by PyModule_Create, two functions are introduced:: int PyModule_SetDocString(PyObject *m, const char *doc) int PyModule_AddFunctions(PyObject *m, PyMethodDef *functions) These set the module docstring, and add the module functions, respectively. Both will work on any Python object that supports setting attributes. They return ``0`` on success, and on failure, they set the exception and return ``-1``. PyCapsule convenience functions ------------------------------- Instead of custom module objects, PyCapsule will become the preferred mechanism for storing per-module C data. Two new convenience functions will be added to help with this. * :: PyObject *PyModule_AddCapsule( PyObject *module, const char *module_name, const char *attribute_name, void *pointer, PyCapsule_Destructor destructor) Add a new PyCapsule to *module* as *attribute_name*. The capsule name is formed by joining *module_name* and *attribute_name* by a dot. This convenience function can be used from a module initialization function instead of separate calls to PyCapsule_New and PyModule_AddObject. Returns a borrowed reference to the new capsule, or NULL (with exception set) on failure. * :: void *PyModule_GetCapsulePointer( PyObject *module, const char *module_name, const char *attribute_name) Returns the pointer stored in *module* as *attribute_name*, or NULL (with an exception set) on failure. The capsule name is formed by joining *module_name* and *attribute_name* by a dot. This convenience function can be used instead of separate calls to PyObject_GetAttr and PyCapsule_GetPointer. Extension authors are encouraged to define a macro to call PyModule_GetCapsulePointer and cast the result to an appropriate type. Generalizing PyModule_* functions --------------------------------- The following functions and macros will be modified to work on any object that supports attribute access: * PyModule_GetNameObject * PyModule_GetName * PyModule_GetFilenameObject * PyModule_GetFilename * PyModule_AddIntConstant * PyModule_AddStringConstant * PyModule_AddIntMacro * PyModule_AddStringMacro * PyModule_AddObject The PyModule_GetDict function will continue to only work on true module objects. This means that it should not be used on extension modules that only define PyModuleExec. Legacy Init ----------- If PyModuleExec is not defined, the import machinery will try to initialize the module using the PyModuleInit hook, as described in PEP 3121. If PyModuleExec is defined, PyModuleInit will be ignored. Modules requiring compatibility with previous versions of CPython may implement PyModuleInit in addition to the new hook. Subinterpreters and Interpreter Reloading ----------------------------------------- Extensions using the new initialization scheme are expected to support subinterpreters and multiple Py_Initialize/Py_Finalize cycles correctly. The mechanism is designed to make this easy, but care is still required on the part of the extension author. No user-defined functions, methods, or instances may leak to different interpreters. To achieve this, all module-level state should be kept in either the module dict, or in the module object. A simple rule of thumb is: Do not define any static data, except built-in types with no mutable or user-settable class attributes. Module Reloading ---------------- Reloading an extension module will re-execute its PyModuleInit function. Similar caveats apply to reloading an extension module as to reloading a Python module. Notably, attributes or any other state of the module are not reset before reloading. Additionally, due to limitations in shared library loading (both dlopen on POSIX and LoadModuleEx on Windows), it is not generally possible to load a modified library after it has changed on disk. Therefore, reloading extension modules is of limited use. Multiple modules in one library ------------------------------- To support multiple Python modules in one shared library, the library must export appropriate PyModuleExec_ or PyModuleCreate_ hooks for each exported module. The modules are loaded using a ModuleSpec with origin set to the name of the library file, and name set to the module name. Note that this mechanism can currently only be used to *load* such modules, not to *find* them. XXX: This is an existing issue; either fix it/wait for a fix or provide an example of how to load such modules. Implementation ============== XXX - not started Open issues =========== We should expose some kind of API in importlib.util (or a better place?) that can be used to check that a module works with reloading and subinterpreters. Related issues ============== The runpy module will need to be modified to take advantage of PEP 451 and this PEP. This is out of scope for this PEP. Previous Approaches =================== Stefan Behnel's initial proto-PEP [#stefans_protopep]_ had a "PyInit_modulename" hook that would create a module class, whose ``__init__`` would be then called to create the module. This proposal did not correspond to the (then nonexistent) PEP 451, where module creation and initialization is broken into distinct steps. It also did not support loading an extension into pre-existing module objects. Nick Coghlan proposed the Create annd Exec hooks, and wrote a prototype implementation [#nicks-prototype]_. At this time PEP 451 was still not implemented, so the prototype does not use ModuleSpec. References ========== .. [#lazy_import_concerns] https://mail.python.org/pipermail/python-dev/2013-August/128129.html .. [#pep-0451-attributes] https://www.python.org/dev/peps/pep-0451/#attributes .. [#stefans_protopep] https://mail.python.org/pipermail/python-dev/2013-August/128087.html .. [#nicks-prototype] https://mail.python.org/pipermail/python-dev/2013-August/128101.html Copyright ========= This document has been placed in the public domain. From jimjjewett at gmail.com Mon Mar 16 16:42:26 2015 From: jimjjewett at gmail.com (Jim J. Jewett) Date: Mon, 16 Mar 2015 08:42:26 -0700 (PDT) Subject: [Python-Dev] PEP 489: Redesigning extension module loading In-Reply-To: <5506CEB5.7050105@gmail.com> Message-ID: <5506f9e2.15148c0a.2361.ffff8a31@mx.google.com> On 16 March 2015 Petr Viktorin wrote: > If PyModuleCreate is not defined, PyModuleExec is expected to operate > on any Python object for which attributes can be added by PyObject_GetAttr* > and retrieved by PyObject_SetAttr*. I assume it is the other way around (add with Set and retrieve with Get), rather than a description of the required form of magic. > PyObject *PyModule_AddCapsule( > PyObject *module, > const char *module_name, > const char *attribute_name, > void *pointer, > PyCapsule_Destructor destructor) What happens if module_name doesn't match the module's __name__? Does it become a hidden attribute? A dotted attribute? Is the result undefined? Later, there is > void *PyModule_GetCapsulePointer( > PyObject *module, > const char *module_name, > const char *attribute_name) with the same apparently redundant arguments, but not a PyModule_SetCapsulePointer. Are capsule pointers read-only, or can they be replaced with another call to PyModule_AddCapsule, or by a simple PyObject_SetAttr? > Subinterpreters and Interpreter Reloading ... > No user-defined functions, methods, or instances may leak to different > interpreters. By "user-defined" do you mean "defined in python, as opposed to in the extension itself"? If so, what is the recommendation for modules that do want to support, say, callbacks? A dual-layer mapping that uses the interpreter as the first key? Naming it _module and only using it indirectly through module.py, which is not shared across interpreters? Not using this API at all? > To achieve this, all module-level state should be kept in either the module > dict, or in the module object. I don't see how that is related to leakage. > A simple rule of thumb is: Do not define any static data, except > built-in types > with no mutable or user-settable class attributes. What about singleton instances? Should they be per-interpreter? What about constants, such as PI? Where should configuration variables (e.g., MAX_SEARCH_DEPTH) be kept? What happens if this no-leakage rule is violated? Does the module not load, or does it just maybe lead to a crash down the road? -jJ -- If there are still threading problems with my replies, please email me with details, so that I can try to resolve them. -jJ From encukou at gmail.com Mon Mar 16 18:42:44 2015 From: encukou at gmail.com (Petr Viktorin) Date: Mon, 16 Mar 2015 18:42:44 +0100 Subject: [Python-Dev] PEP 489: Redesigning extension module loading In-Reply-To: <5506f9e2.15148c0a.2361.ffff8a31@mx.google.com> References: <5506CEB5.7050105@gmail.com> <5506f9e2.15148c0a.2361.ffff8a31@mx.google.com> Message-ID: On Mon, Mar 16, 2015 at 4:42 PM, Jim J. Jewett wrote: > > On 16 March 2015 Petr Viktorin wrote: > >> If PyModuleCreate is not defined, PyModuleExec is expected to operate >> on any Python object for which attributes can be added by PyObject_GetAttr* >> and retrieved by PyObject_SetAttr*. > > I assume it is the other way around (add with Set and retrieve with Get), > rather than a description of the required form of magic. Right you are, I mixed that up. >> PyObject *PyModule_AddCapsule( >> PyObject *module, >> const char *module_name, >> const char *attribute_name, >> void *pointer, >> PyCapsule_Destructor destructor) > > What happens if module_name doesn't match the module's __name__? > Does it become a hidden attribute? A dotted attribute? Is the > result undefined? The module_name is used to name the capsule, following the convention from PyCapsule_Import. The "module.__name__" is not used or checked. The function would do this: capsule_name = module_name + '.' + attribute_name capsule = PyCapsule_New(pointer, capsule_name, destructor) PyModule_AddObject(module, attribute_name, capsule) just with error handling, and suitable C code for the "+". I will add the pseudocode to the PEP. > Later, there is > >> void *PyModule_GetCapsulePointer( >> PyObject *module, >> const char *module_name, >> const char *attribute_name) > > with the same apparently redundant arguments, Here the behavior would be: capsule_name = module_name + '.' + attribute_name capsule = PyObject_GetAttr(module, attribute_name) return PyCapsule_GetPointer(capsule, capsule_name) > but not a > PyModule_SetCapsulePointer. Are capsule pointers read-only, or can > they be replaced with another call to PyModule_AddCapsule, or by a > simple PyObject_SetAttr? You can replace the capsule using any of those two, or set the pointer using PyCapsule_SetPointer, or (most likely) change the data the pointer points to. The added functions are just simple helpers for common operations, meant to encourage keeping per-module state. >> Subinterpreters and Interpreter Reloading > ... >> No user-defined functions, methods, or instances may leak to different >> interpreters. > > By "user-defined" do you mean "defined in python, as opposed to in > the extension itself"? Yes. > If so, what is the recommendation for modules that do want to support, > say, callbacks? A dual-layer mapping that uses the interpreter as the > first key? Naming it _module and only using it indirectly through > module.py, which is not shared across interpreters? Not using this > API at all? There is a separate module object, with its own dict, for each subinterpreter (as when creating the module with "PyModuleDef.m_size == 0" today). Callbacks should be stored on the appropriate module instance. Does that answer your question? I'm not sure how you meant "callbacks". >> To achieve this, all module-level state should be kept in either the module >> dict, or in the module object. > > I don't see how that is related to leakage. > >> A simple rule of thumb is: Do not define any static data, except >> built-in types >> with no mutable or user-settable class attributes. > > What about singleton instances? Should they be per-interpreter? Yes, definitely. > What about constants, such as PI? In PyModuleExec, create the constant using PyFloat_FromDouble, and add it using PyModule_FromObject. That will do the right thing. (Float constants can be shared, since they cannot refer to user-defined code. But this PEP shields you from needing to know this for every type.) > Where should configuration variables (e.g., MAX_SEARCH_DEPTH) be > kept? On the module object. > What happens if this no-leakage rule is violated? Does the module > not load, or does it just maybe lead to a crash down the road? It may, as today, lead to unexpected behavior down the road. This is explained here: https://docs.python.org/3/c-api/init.html#sub-interpreter-support Unfortunately, there's no good way to detect such leakage. This PEP adds the tools, documentation, and guidelines to make it easy to do the right thing, but won't prevent you from shooting yourself in the foot in C code. Thank you for sharing your concerns! I will keep them in mind when writing the docs for this. From mistersheik at gmail.com Tue Mar 17 00:10:50 2015 From: mistersheik at gmail.com (Neil Girdhar) Date: Mon, 16 Mar 2015 19:10:50 -0400 Subject: [Python-Dev] PEP 448 review In-Reply-To: References: Message-ID: Hi everyone, I was wondering what is left with the PEP 448 ( http://bugs.python.org/issue2292) code review? Big thanks to Benjamin, Ethan, and Serhiy for reviewing some (all?) of the code. What is the next step of this process? Thanks, Neil On Sun, Mar 8, 2015 at 4:38 PM, Neil Girdhar wrote: > Anyone have time to do a code review? > > http://bugs.python.org/issue2292 > > > On Mon, Mar 2, 2015 at 4:54 PM, Neil Girdhar > wrote: > >> It's from five days ago. I asked Joshua to take a look at something, but >> I guess he is busy. >> >> Best, >> >> Neil >> >> ? >> >> The latest file there is from Feb 26, while your message that the patch >> was ready for review is from today -- so is the >> patch from five days ago the most recent? >> >> -- >> ~Ethan~ >> >> On Mon, Mar 2, 2015 at 3:18 PM, Neil Girdhar >> wrote: >> >>> http://bugs.python.org/issue2292 >>> >>> On Mon, Mar 2, 2015 at 3:17 PM, Victor Stinner >> > wrote: >>> >>>> Where is the patch? >>>> >>>> Victor >>>> >>>> Le lundi 2 mars 2015, Neil Girdhar a ?crit : >>>> >>>> Hi everyone, >>>>> >>>>> The patch is ready for review now, and I should have time this week to >>>>> make changes and respond to comments. >>>>> >>>>> Best, >>>>> >>>>> Neil >>>>> >>>>> On Wed, Feb 25, 2015 at 2:42 PM, Guido van Rossum >>>>> wrote: >>>>> >>>>>> I'm back, I've re-read the PEP, and I've re-read the long thread with >>>>>> "(no subject)". >>>>>> >>>>>> I think Georg Brandl nailed it: >>>>>> >>>>>> """ >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> *I like the "sequence and dict flattening" part of the PEP, mostly >>>>>> because itis consistent and should be easy to understand, but the >>>>>> comprehension syntaxenhancements seem to be bad for readability and >>>>>> "comprehending" what the codedoes.The call syntax part is a mixed bag on >>>>>> the one hand it is nice to be consistent with the extended possibilities in >>>>>> literals (flattening), but on the other hand there would be small but >>>>>> annoying inconsistencies anyways (e.g. the duplicate kwarg case above).* >>>>>> """ >>>>>> >>>>>> Greg Ewing followed up explaining that the inconsistency between dict >>>>>> flattening and call syntax is inherent in the pre-existing different rules >>>>>> for dicts vs. keyword args: {'a':1, 'a':2} results in {'a':2}, while f(a=1, >>>>>> a=2) is an error. (This form is a SyntaxError; the dynamic case f(a=1, >>>>>> **{'a': 1}) is a TypeError.) >>>>>> >>>>>> For me, allowing f(*a, *b) and f(**d, **e) and all the other >>>>>> combinations for function calls proposed by the PEP is an easy +1 -- it's a >>>>>> straightforward extension of the existing pattern, and anybody who knows >>>>>> what f(x, *a) does will understand f(x, *a, y, *b). Guessing what f(**d, >>>>>> **e) means shouldn't be hard either. Understanding the edge case for >>>>>> duplicate keys with f(**d, **e) is a little harder, but the error messages >>>>>> are pretty clear, and it is not a new edge case. >>>>>> >>>>>> The sequence and dict flattening syntax proposals are also clean and >>>>>> logical -- we already have *-unpacking on the receiving side, so allowing >>>>>> *x in tuple expressions reads pretty naturally (and the similarity with *a >>>>>> in argument lists certainly helps). From here, having [a, *x, b, *y] is >>>>>> also natural, and then the extension to other displays is natural: {a, *x, >>>>>> b, *y} and {a:1, **d, b:2, **e}. This, too, gets a +1 from me. >>>>>> >>>>>> So that leaves comprehensions. IIRC, during the development of the >>>>>> patch we realized that f(*x for x in xs) is sufficiently ambiguous that we >>>>>> decided to disallow it -- note that f(x for x in xs) is already somewhat of >>>>>> a special case because an argument can only be a "bare" generator >>>>>> expression if it is the only argument. The same reasoning doesn't apply (in >>>>>> that form) to list, set and dict comprehensions -- while f(x for x in xs) >>>>>> is identical in meaning to f((x for x in xs)), [x for x in xs] is NOT the >>>>>> same as [(x for x in xs)] (that's a list of one element, and the element is >>>>>> a generator expression). >>>>>> >>>>>> The basic premise of this part of the proposal is that if you have a >>>>>> few iterables, the new proposal (without comprehensions) lets you create a >>>>>> list or generator expression that iterates over all of them, essentially >>>>>> flattening them: >>>>>> >>>>>> >>> xs = [1, 2, 3] >>>>>> >>> ys = ['abc', 'def'] >>>>>> >>> zs = [99] >>>>>> >>> [*xs, *ys, *zs] >>>>>> [1, 2, 3, 'abc', 'def', 99] >>>>>> >>> >>>>>> >>>>>> But now suppose you have a list of iterables: >>>>>> >>>>>> >>> xss = [[1, 2, 3], ['abc', 'def'], [99]] >>>>>> >>> [*xss[0], *xss[1], *xss[2]] >>>>>> [1, 2, 3, 'abc', 'def', 99] >>>>>> >>> >>>>>> >>>>>> Wouldn't it be nice if you could write the latter using a >>>>>> comprehension? >>>>>> >>>>>> >>> xss = [[1, 2, 3], ['abc', 'def'], [99]] >>>>>> >>> [*xs for xs in xss] >>>>>> [1, 2, 3, 'abc', 'def', 99] >>>>>> >>> >>>>>> >>>>>> This is somewhat seductive, and the following is even nicer: the *xs >>>>>> position may be an expression, e.g.: >>>>>> >>>>>> >>> xss = [[1, 2, 3], ['abc', 'def'], [99]] >>>>>> >>> [*xs[:2] for xs in xss] >>>>>> [1, 2, 'abc', 'def', 99] >>>>>> >>> >>>>>> >>>>>> On the other hand, I had to explore the possibilities here by >>>>>> experimenting in the interpreter, and I discovered some odd edge cases >>>>>> (e.g. you can parenthesize the starred expression, but that seems a >>>>>> syntactic accident). >>>>>> >>>>>> All in all I am personally +0 on the comprehension part of the PEP, >>>>>> and I like that it provides a way to "flatten" a sequence of sequences, but >>>>>> I think very few people in the thread have supported this part. Therefore I >>>>>> would like to ask Neil to update the PEP and the patch to take out the >>>>>> comprehension part, so that the two "easy wins" can make it into Python 3.5 >>>>>> (basically, I am accepting two-thirds of the PEP :-). There is some time >>>>>> yet until alpha 2. >>>>>> >>>>>> I would also like code reviewers (Benjamin?) to start reviewing the >>>>>> patch , taking into account that >>>>>> the comprehension part needs to be removed. >>>>>> >>>>>> -- >>>>>> --Guido van Rossum (python.org/~guido) >>>>>> >>>>>> >>>>> >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Tue Mar 17 14:49:58 2015 From: brett at python.org (Brett Cannon) Date: Tue, 17 Mar 2015 13:49:58 +0000 Subject: [Python-Dev] PEP 448 review In-Reply-To: References: Message-ID: On Mon, Mar 16, 2015 at 7:11 PM Neil Girdhar wrote: > Hi everyone, > > I was wondering what is left with the PEP 448 ( > http://bugs.python.org/issue2292) code review? Big thanks to Benjamin, > Ethan, and Serhiy for reviewing some (all?) of the code. What is the next > step of this process? > My suspicion is that if no one steps up between now and PyCon to do a complete code review of the final patch, we as a group will try to get it done at the PyCon sprints. I have made the issue a release blocker to help make sure it gets reviewed and doesn't accidentally get left behind. -------------- next part -------------- An HTML attachment was scrubbed... URL: From szport at gmail.com Tue Mar 17 20:05:16 2015 From: szport at gmail.com (Zaur Shibzukhov) Date: Tue, 17 Mar 2015 22:05:16 +0300 Subject: [Python-Dev] Possible wrong behavior of the dict? Message-ID: Hello! In order to explain, let define subclass of dict: class Pair: def __init__(self, key, val): self.key = key self.val = val class MyDict(dict): # def __init__(self, *args, **kwds): if len(args) > 1: raise TypeError('Expected at most 1 arguments, but got %d' % len(args)) for key, val in args[0]: self[key] = val for key, val in kwds.items(): self[key] = val def __getitem__(self, key): pair = dict.__getitem__(key) return pair.value def __setitem__(self, key, val): if key in self: pair = dict.__getitem__(key) pair.value = value else: pair = Pair(key, val) dict.__setitem__(self, key, pair) def values(self): for key in self: p = dict.__getitem__(self, key) yield p.value def items(self): for key, p in dict.__iter__(self): yield p.key, p.value The simple test give me strange result: >>> d = MyDict([('a', 1), ('b', 2), ('c', 3)]) >>> dict(d) {'a': <__main__.Pair at 0x104ca9e48>, 'b': <__main__.Pair at 0x104ca9e80>, 'c': <__main__.Pair at 0x104ca9eb8>} instead of {'a':1, 'b':2, 'c':3}. Is this right behavior of the dict? --- Zaur Shibzukhov -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Tue Mar 17 20:12:36 2015 From: brett at python.org (Brett Cannon) Date: Tue, 17 Mar 2015 19:12:36 +0000 Subject: [Python-Dev] Possible wrong behavior of the dict? In-Reply-To: References: Message-ID: On Tue, Mar 17, 2015 at 3:05 PM Zaur Shibzukhov wrote: > Hello! > > In order to explain, let define subclass of dict: > > class Pair: > def __init__(self, key, val): > self.key = key > self.val = val > > class MyDict(dict): > # > def __init__(self, *args, **kwds): > if len(args) > 1: > raise TypeError('Expected at most 1 arguments, but got %d' % > len(args)) > > for key, val in args[0]: > self[key] = val > > for key, val in kwds.items(): > self[key] = val > > def __getitem__(self, key): > pair = dict.__getitem__(key) > return pair.value > > def __setitem__(self, key, val): > if key in self: > pair = dict.__getitem__(key) > pair.value = value > else: > pair = Pair(key, val) > dict.__setitem__(self, key, pair) > > def values(self): > for key in self: > p = dict.__getitem__(self, key) > yield p.value > > def items(self): > for key, p in dict.__iter__(self): > yield p.key, p.value > > > The simple test give me strange result: > > >>> d = MyDict([('a', 1), ('b', 2), ('c', 3)]) > >>> dict(d) > {'a': <__main__.Pair at 0x104ca9e48>, > 'b': <__main__.Pair at 0x104ca9e80>, > 'c': <__main__.Pair at 0x104ca9eb8>} > > instead of {'a':1, 'b':2, 'c':3}. > > > Is this right behavior of the dict? > Yes because in your __setitem__ call you are storing the value as the Pair. So when dict prints its repr it prints the key and value, and in this case the value is a Pair. -------------- next part -------------- An HTML attachment was scrubbed... URL: From szport at gmail.com Tue Mar 17 20:29:07 2015 From: szport at gmail.com (Zaur Shibzukhov) Date: Tue, 17 Mar 2015 22:29:07 +0300 Subject: [Python-Dev] Possible wrong behavior of the dict? In-Reply-To: References: Message-ID: Yes... But I expected that dict constructor will use `__getitem__` or `items` method of MyDict instance in order to retrieve items of the MyDict instance during construction of the dict instance... Instead it interpreted MyDict instance as the dict instance during construction of new dict.This exactly caused my confusion. --- *Zaur Shibzukhov* 2015-03-17 22:12 GMT+03:00 Brett Cannon : > > > On Tue, Mar 17, 2015 at 3:05 PM Zaur Shibzukhov wrote: > >> Hello! >> >> In order to explain, let define subclass of dict: >> >> class Pair: >> def __init__(self, key, val): >> self.key = key >> self.val = val >> >> class MyDict(dict): >> # >> def __init__(self, *args, **kwds): >> if len(args) > 1: >> raise TypeError('Expected at most 1 arguments, but got %d' % >> len(args)) >> >> for key, val in args[0]: >> self[key] = val >> >> for key, val in kwds.items(): >> self[key] = val >> >> def __getitem__(self, key): >> pair = dict.__getitem__(key) >> return pair.value >> >> def __setitem__(self, key, val): >> if key in self: >> pair = dict.__getitem__(key) >> pair.value = value >> else: >> pair = Pair(key, val) >> dict.__setitem__(self, key, pair) >> >> def values(self): >> for key in self: >> p = dict.__getitem__(self, key) >> yield p.value >> >> def items(self): >> for key, p in dict.__iter__(self): >> yield p.key, p.value >> >> >> The simple test give me strange result: >> >> >>> d = MyDict([('a', 1), ('b', 2), ('c', 3)]) >> >>> dict(d) >> {'a': <__main__.Pair at 0x104ca9e48>, >> 'b': <__main__.Pair at 0x104ca9e80>, >> 'c': <__main__.Pair at 0x104ca9eb8>} >> >> instead of {'a':1, 'b':2, 'c':3}. >> >> >> Is this right behavior of the dict? >> > > Yes because in your __setitem__ call you are storing the value as the > Pair. So when dict prints its repr it prints the key and value, and in this > case the value is a Pair. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Tue Mar 17 20:38:59 2015 From: brett at python.org (Brett Cannon) Date: Tue, 17 Mar 2015 19:38:59 +0000 Subject: [Python-Dev] Possible wrong behavior of the dict? In-Reply-To: References: Message-ID: On Tue, Mar 17, 2015 at 3:29 PM Zaur Shibzukhov wrote: > Yes... But I expected that dict constructor will use `__getitem__` or > `items` method of MyDict instance in order to retrieve items of the MyDict > instance during construction of the dict instance... Instead it interpreted > MyDict instance as the dict instance during construction of new dict.This > exactly caused my confusion. > It's because you subclassed dict. Copying is optimized to skip over using the methods you listed when the object is a dict and so we know the structure of the object at the C level. You can look at https://hg.python.org/cpython/file/22a0c925a7c2/Objects/dictobject.c#l1997 to see the actual code. -Brett > > --- > *Zaur Shibzukhov* > > > 2015-03-17 22:12 GMT+03:00 Brett Cannon : > >> >> >> On Tue, Mar 17, 2015 at 3:05 PM Zaur Shibzukhov wrote: >> >>> Hello! >>> >>> In order to explain, let define subclass of dict: >>> >>> class Pair: >>> def __init__(self, key, val): >>> self.key = key >>> self.val = val >>> >>> class MyDict(dict): >>> # >>> def __init__(self, *args, **kwds): >>> if len(args) > 1: >>> raise TypeError('Expected at most 1 arguments, but got %d' % >>> len(args)) >>> >>> for key, val in args[0]: >>> self[key] = val >>> >>> for key, val in kwds.items(): >>> self[key] = val >>> >>> def __getitem__(self, key): >>> pair = dict.__getitem__(key) >>> return pair.value >>> >>> def __setitem__(self, key, val): >>> if key in self: >>> pair = dict.__getitem__(key) >>> pair.value = value >>> else: >>> pair = Pair(key, val) >>> dict.__setitem__(self, key, pair) >>> >>> def values(self): >>> for key in self: >>> p = dict.__getitem__(self, key) >>> yield p.value >>> >>> def items(self): >>> for key, p in dict.__iter__(self): >>> yield p.key, p.value >>> >>> >>> The simple test give me strange result: >>> >>> >>> d = MyDict([('a', 1), ('b', 2), ('c', 3)]) >>> >>> dict(d) >>> {'a': <__main__.Pair at 0x104ca9e48>, >>> 'b': <__main__.Pair at 0x104ca9e80>, >>> 'c': <__main__.Pair at 0x104ca9eb8>} >>> >>> instead of {'a':1, 'b':2, 'c':3}. >>> >>> >>> Is this right behavior of the dict? >>> >> >> Yes because in your __setitem__ call you are storing the value as the >> Pair. So when dict prints its repr it prints the key and value, and in this >> case the value is a Pair. >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From szport at gmail.com Tue Mar 17 20:45:50 2015 From: szport at gmail.com (Zaur Shibzukhov) Date: Tue, 17 Mar 2015 22:45:50 +0300 Subject: [Python-Dev] Possible wrong behavior of the dict? In-Reply-To: References: Message-ID: So in such cases it should not subclassed `dict`, but `collections.MutableMapping`, for example? --- *Zaur Shibzukhov* 2015-03-17 22:38 GMT+03:00 Brett Cannon : > > > On Tue, Mar 17, 2015 at 3:29 PM Zaur Shibzukhov wrote: > >> Yes... But I expected that dict constructor will use `__getitem__` or >> `items` method of MyDict instance in order to retrieve items of the MyDict >> instance during construction of the dict instance... Instead it interpreted >> MyDict instance as the dict instance during construction of new dict.This >> exactly caused my confusion. >> > > It's because you subclassed dict. Copying is optimized to skip over using > the methods you listed when the object is a dict and so we know the > structure of the object at the C level. You can look at > https://hg.python.org/cpython/file/22a0c925a7c2/Objects/dictobject.c#l1997 > to see the actual code. > > -Brett > > >> >> --- >> *Zaur Shibzukhov* >> >> >> 2015-03-17 22:12 GMT+03:00 Brett Cannon : >> >>> >>> >>> On Tue, Mar 17, 2015 at 3:05 PM Zaur Shibzukhov >>> wrote: >>> >>>> Hello! >>>> >>>> In order to explain, let define subclass of dict: >>>> >>>> class Pair: >>>> def __init__(self, key, val): >>>> self.key = key >>>> self.val = val >>>> >>>> class MyDict(dict): >>>> # >>>> def __init__(self, *args, **kwds): >>>> if len(args) > 1: >>>> raise TypeError('Expected at most 1 arguments, but got %d' >>>> % len(args)) >>>> >>>> for key, val in args[0]: >>>> self[key] = val >>>> >>>> for key, val in kwds.items(): >>>> self[key] = val >>>> >>>> def __getitem__(self, key): >>>> pair = dict.__getitem__(key) >>>> return pair.value >>>> >>>> def __setitem__(self, key, val): >>>> if key in self: >>>> pair = dict.__getitem__(key) >>>> pair.value = value >>>> else: >>>> pair = Pair(key, val) >>>> dict.__setitem__(self, key, pair) >>>> >>>> def values(self): >>>> for key in self: >>>> p = dict.__getitem__(self, key) >>>> yield p.value >>>> >>>> def items(self): >>>> for key, p in dict.__iter__(self): >>>> yield p.key, p.value >>>> >>>> >>>> The simple test give me strange result: >>>> >>>> >>> d = MyDict([('a', 1), ('b', 2), ('c', 3)]) >>>> >>> dict(d) >>>> {'a': <__main__.Pair at 0x104ca9e48>, >>>> 'b': <__main__.Pair at 0x104ca9e80>, >>>> 'c': <__main__.Pair at 0x104ca9eb8>} >>>> >>>> instead of {'a':1, 'b':2, 'c':3}. >>>> >>>> >>>> Is this right behavior of the dict? >>>> >>> >>> Yes because in your __setitem__ call you are storing the value as the >>> Pair. So when dict prints its repr it prints the key and value, and in this >>> case the value is a Pair. >>> >> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Tue Mar 17 20:48:14 2015 From: brett at python.org (Brett Cannon) Date: Tue, 17 Mar 2015 19:48:14 +0000 Subject: [Python-Dev] Possible wrong behavior of the dict? In-Reply-To: References: Message-ID: On Tue, Mar 17, 2015 at 3:46 PM Zaur Shibzukhov wrote: > So in such cases it should not subclassed `dict`, but > `collections.MutableMapping`, for example? > Yes (see the comment at https://hg.python.org/cpython/file/22a0c925a7c2/Objects/dictobject.c#l2003 ). -Brett > > --- > *Zaur Shibzukhov* > > > 2015-03-17 22:38 GMT+03:00 Brett Cannon : > >> >> >> On Tue, Mar 17, 2015 at 3:29 PM Zaur Shibzukhov wrote: >> >>> Yes... But I expected that dict constructor will use `__getitem__` or >>> `items` method of MyDict instance in order to retrieve items of the MyDict >>> instance during construction of the dict instance... Instead it interpreted >>> MyDict instance as the dict instance during construction of new dict.This >>> exactly caused my confusion. >>> >> >> It's because you subclassed dict. Copying is optimized to skip over using >> the methods you listed when the object is a dict and so we know the >> structure of the object at the C level. You can look at >> https://hg.python.org/cpython/file/22a0c925a7c2/Objects/dictobject.c#l1997 >> to see the actual code. >> >> -Brett >> >> >>> >>> --- >>> *Zaur Shibzukhov* >>> >>> >>> 2015-03-17 22:12 GMT+03:00 Brett Cannon : >>> >>>> >>>> >>>> On Tue, Mar 17, 2015 at 3:05 PM Zaur Shibzukhov >>>> wrote: >>>> >>>>> Hello! >>>>> >>>>> In order to explain, let define subclass of dict: >>>>> >>>>> class Pair: >>>>> def __init__(self, key, val): >>>>> self.key = key >>>>> self.val = val >>>>> >>>>> class MyDict(dict): >>>>> # >>>>> def __init__(self, *args, **kwds): >>>>> if len(args) > 1: >>>>> raise TypeError('Expected at most 1 arguments, but got %d' >>>>> % len(args)) >>>>> >>>>> for key, val in args[0]: >>>>> self[key] = val >>>>> >>>>> for key, val in kwds.items(): >>>>> self[key] = val >>>>> >>>>> def __getitem__(self, key): >>>>> pair = dict.__getitem__(key) >>>>> return pair.value >>>>> >>>>> def __setitem__(self, key, val): >>>>> if key in self: >>>>> pair = dict.__getitem__(key) >>>>> pair.value = value >>>>> else: >>>>> pair = Pair(key, val) >>>>> dict.__setitem__(self, key, pair) >>>>> >>>>> def values(self): >>>>> for key in self: >>>>> p = dict.__getitem__(self, key) >>>>> yield p.value >>>>> >>>>> def items(self): >>>>> for key, p in dict.__iter__(self): >>>>> yield p.key, p.value >>>>> >>>>> >>>>> The simple test give me strange result: >>>>> >>>>> >>> d = MyDict([('a', 1), ('b', 2), ('c', 3)]) >>>>> >>> dict(d) >>>>> {'a': <__main__.Pair at 0x104ca9e48>, >>>>> 'b': <__main__.Pair at 0x104ca9e80>, >>>>> 'c': <__main__.Pair at 0x104ca9eb8>} >>>>> >>>>> instead of {'a':1, 'b':2, 'c':3}. >>>>> >>>>> >>>>> Is this right behavior of the dict? >>>>> >>>> >>>> Yes because in your __setitem__ call you are storing the value as the >>>> Pair. So when dict prints its repr it prints the key and value, and in this >>>> case the value is a Pair. >>>> >>> >>> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From szport at gmail.com Tue Mar 17 20:50:47 2015 From: szport at gmail.com (Zaur Shibzukhov) Date: Tue, 17 Mar 2015 22:50:47 +0300 Subject: [Python-Dev] Possible wrong behavior of the dict? In-Reply-To: References: Message-ID: Thanks. --- *Zaur Shibzukhov* 2015-03-17 22:48 GMT+03:00 Brett Cannon : > > > On Tue, Mar 17, 2015 at 3:46 PM Zaur Shibzukhov wrote: > >> So in such cases it should not subclassed `dict`, but >> `collections.MutableMapping`, for example? >> > > Yes (see the comment at > https://hg.python.org/cpython/file/22a0c925a7c2/Objects/dictobject.c#l2003 > ). > > -Brett > > >> >> --- >> *Zaur Shibzukhov* >> >> >> 2015-03-17 22:38 GMT+03:00 Brett Cannon : >> >>> >>> >>> On Tue, Mar 17, 2015 at 3:29 PM Zaur Shibzukhov >>> wrote: >>> >>>> Yes... But I expected that dict constructor will use `__getitem__` or >>>> `items` method of MyDict instance in order to retrieve items of the MyDict >>>> instance during construction of the dict instance... Instead it interpreted >>>> MyDict instance as the dict instance during construction of new dict.This >>>> exactly caused my confusion. >>>> >>> >>> It's because you subclassed dict. Copying is optimized to skip over >>> using the methods you listed when the object is a dict and so we know the >>> structure of the object at the C level. You can look at >>> https://hg.python.org/cpython/file/22a0c925a7c2/Objects/dictobject.c#l1997 >>> to see the actual code. >>> >>> -Brett >>> >>> >>>> >>>> --- >>>> *Zaur Shibzukhov* >>>> >>>> >>>> 2015-03-17 22:12 GMT+03:00 Brett Cannon : >>>> >>>>> >>>>> >>>>> On Tue, Mar 17, 2015 at 3:05 PM Zaur Shibzukhov >>>>> wrote: >>>>> >>>>>> Hello! >>>>>> >>>>>> In order to explain, let define subclass of dict: >>>>>> >>>>>> class Pair: >>>>>> def __init__(self, key, val): >>>>>> self.key = key >>>>>> self.val = val >>>>>> >>>>>> class MyDict(dict): >>>>>> # >>>>>> def __init__(self, *args, **kwds): >>>>>> if len(args) > 1: >>>>>> raise TypeError('Expected at most 1 arguments, but got >>>>>> %d' % len(args)) >>>>>> >>>>>> for key, val in args[0]: >>>>>> self[key] = val >>>>>> >>>>>> for key, val in kwds.items(): >>>>>> self[key] = val >>>>>> >>>>>> def __getitem__(self, key): >>>>>> pair = dict.__getitem__(key) >>>>>> return pair.value >>>>>> >>>>>> def __setitem__(self, key, val): >>>>>> if key in self: >>>>>> pair = dict.__getitem__(key) >>>>>> pair.value = value >>>>>> else: >>>>>> pair = Pair(key, val) >>>>>> dict.__setitem__(self, key, pair) >>>>>> >>>>>> def values(self): >>>>>> for key in self: >>>>>> p = dict.__getitem__(self, key) >>>>>> yield p.value >>>>>> >>>>>> def items(self): >>>>>> for key, p in dict.__iter__(self): >>>>>> yield p.key, p.value >>>>>> >>>>>> >>>>>> The simple test give me strange result: >>>>>> >>>>>> >>> d = MyDict([('a', 1), ('b', 2), ('c', 3)]) >>>>>> >>> dict(d) >>>>>> {'a': <__main__.Pair at 0x104ca9e48>, >>>>>> 'b': <__main__.Pair at 0x104ca9e80>, >>>>>> 'c': <__main__.Pair at 0x104ca9eb8>} >>>>>> >>>>>> instead of {'a':1, 'b':2, 'c':3}. >>>>>> >>>>>> >>>>>> Is this right behavior of the dict? >>>>>> >>>>> >>>>> Yes because in your __setitem__ call you are storing the value as the >>>>> Pair. So when dict prints its repr it prints the key and value, and in this >>>>> case the value is a Pair. >>>>> >>>> >>>> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From me at the-compiler.org Tue Mar 17 20:33:46 2015 From: me at the-compiler.org (Florian Bruhin) Date: Tue, 17 Mar 2015 20:33:46 +0100 Subject: [Python-Dev] Possible wrong behavior of the dict? In-Reply-To: References: Message-ID: <20150317193345.GI10871@tonks> * Zaur Shibzukhov [2015-03-17 22:29:07 +0300]: > Yes... But I expected that dict constructor will use `__getitem__` or > `items` method of MyDict instance in order to retrieve items of the MyDict > instance during construction of the dict instance... Instead it interpreted > MyDict instance as the dict instance during construction of new dict.This > exactly caused my confusion. Subclassing builtins is always a recipe for trouble, because the C implementation doesn't necessarily call your Python methods. You should probably use collections.UserDict or collections.abc.(Mutable)Mapping instead: https://docs.python.org/3/library/collections.html#collections.UserDict https://docs.python.org/3/library/collections.abc.html#collections.abc.Mapping Florian -- http://www.the-compiler.org | me at the-compiler.org (Mail/XMPP) GPG: 916E B0C8 FD55 A072 | http://the-compiler.org/pubkey.asc I love long mails! | http://email.is-not-s.ms/ -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 819 bytes Desc: not available URL: From breamoreboy at yahoo.co.uk Tue Mar 17 21:41:28 2015 From: breamoreboy at yahoo.co.uk (Mark Lawrence) Date: Tue, 17 Mar 2015 20:41:28 +0000 Subject: [Python-Dev] PEP 489: Redesigning extension module loading In-Reply-To: <5506CEB5.7050105@gmail.com> References: <5506CEB5.7050105@gmail.com> Message-ID: On 16/03/2015 12:38, Petr Viktorin wrote: > Hello, Can you use anything from the meta issue http://bugs.python.org/issue15787 for PEP 3121 and PEP 384 or will the work that you are doing render everything done previously redundant? -- My fellow Pythonistas, ask not what our language can do for you, ask what you can do for our language. Mark Lawrence From ncoghlan at gmail.com Wed Mar 18 15:52:05 2015 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 19 Mar 2015 00:52:05 +1000 Subject: [Python-Dev] PEP 448 review In-Reply-To: References: Message-ID: On 17 March 2015 at 23:49, Brett Cannon wrote: > > > On Mon, Mar 16, 2015 at 7:11 PM Neil Girdhar wrote: >> >> Hi everyone, >> >> I was wondering what is left with the PEP 448 >> (http://bugs.python.org/issue2292) code review? Big thanks to Benjamin, >> Ethan, and Serhiy for reviewing some (all?) of the code. What is the next >> step of this process? > > > My suspicion is that if no one steps up between now and PyCon to do a > complete code review of the final patch, we as a group will try to get it > done at the PyCon sprints. I have made the issue a release blocker to help > make sure it gets reviewed and doesn't accidentally get left behind. Good idea - I just bumped the PEP 479 issue (http://bugs.python.org/issue22906) similarly, as well as giving it an initial review (Neil had already noted it needed tests for the new behaviour, and the language reference doc updates look surprisingly minimal to me). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Wed Mar 18 16:01:14 2015 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 19 Mar 2015 01:01:14 +1000 Subject: [Python-Dev] PEP 489: Redesigning extension module loading In-Reply-To: References: <5506CEB5.7050105@gmail.com> Message-ID: On 18 March 2015 at 06:41, Mark Lawrence wrote: > On 16/03/2015 12:38, Petr Viktorin wrote: >> >> Hello, > > > Can you use anything from the meta issue http://bugs.python.org/issue15787 > for PEP 3121 and PEP 384 or will the work that you are doing render > everything done previously redundant? Nothing should break in relation to PEP 3121 or 384, so I think that determination would still need to be made on a case by case basis. Alternatively, it may be possible to update the abitype.py converter to also switch to the new module initialisation hooks (if we can figure out a good way of automating that). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From orion at cora.nwra.com Wed Mar 18 16:46:12 2015 From: orion at cora.nwra.com (Orion Poplawski) Date: Wed, 18 Mar 2015 15:46:12 +0000 (UTC) Subject: [Python-Dev] Use ptyhon -s as default shbang for system python executables/daemons Message-ID: We're starting a discussion in Fedora about setting the default shbang for system python executables and/or daemons to python -s or python -Es (or ?). See also: https://bugzilla.redhat.com/show_bug.cgi?id=1202151 https://fedorahosted.org/fpc/ticket/513 Basically we're wanting to avoid locally installed items causing security issues or other bad behavior, without too adversely affecting users' abilities to work around issues or intentionally alter behavior. It would be good to get some feedback from the broader python community before implementing anything, so I'm asking for feedback here. Thank you, Orion From barry at python.org Wed Mar 18 17:22:03 2015 From: barry at python.org (Barry Warsaw) Date: Wed, 18 Mar 2015 12:22:03 -0400 Subject: [Python-Dev] Use ptyhon -s as default shbang for system python executables/daemons In-Reply-To: References: Message-ID: <20150318122203.12ff1ee3@limelight.wooz.org> On Mar 18, 2015, at 03:46 PM, Orion Poplawski wrote: >We're starting a discussion in Fedora about setting the default shbang for >system python executables and/or daemons to python -s or python -Es (or ?). We've talked about this in Debian/Ubuntu land and the general consensus is that for Python 2, use -Es and for Python 3 use -I (which implies -Es). I'm not sure we're consistent yet in making sure our build tools install these switches in our shebangs, but I'm hoping after Jessie we can make some progress on that. Cheers, -Barry From victor.stinner at gmail.com Wed Mar 18 17:31:09 2015 From: victor.stinner at gmail.com (Victor Stinner) Date: Wed, 18 Mar 2015 17:31:09 +0100 Subject: [Python-Dev] Use ptyhon -s as default shbang for system python executables/daemons In-Reply-To: References: Message-ID: 2015-03-18 16:46 GMT+01:00 Orion Poplawski : > We're starting a discussion in Fedora about setting the default shbang for > system python executables and/or daemons to python -s or python -Es (or ?). Python 3.4 has -I which is more strict than -Es. It remembers me "Perl suid", /usr/bin/sperl. Maybe we should have a "spython" program which is like "python -I" (so it means adding spython, spython3, spython3.5). Does it work to pass command line options to Python in the shebang? > Basically we're wanting to avoid locally installed items causing security > issues or other bad behavior, without too adversely affecting users' > abilities to work around issues or intentionally alter behavior. > > It would be good to get some feedback from the broader python community > before implementing anything, so I'm asking for feedback here. Personally, instead of having to enable a switch to have a safe Python, I would prefer to have a safe Python by default and enable an option to enter the unsafe mode. But it may break backward compatibility depending on changes made in Python if we take this way. Victor From barry at python.org Wed Mar 18 17:48:25 2015 From: barry at python.org (Barry Warsaw) Date: Wed, 18 Mar 2015 12:48:25 -0400 Subject: [Python-Dev] Use ptyhon -s as default shbang for system python executables/daemons In-Reply-To: References: Message-ID: <20150318124825.337dc1c6@anarchist.wooz.org> On Mar 18, 2015, at 05:31 PM, Victor Stinner wrote: >Does it work to pass command line options to Python in the shebang? Yes, but only one "word", thus -Es or -I. We've often mused about whether it makes sense to have two Pythons on the system. One for system scripts and another for users. System Python ('/usr/bin/spython' perhaps) would be locked down and only extensible by system packages. On Debuntu that might mean no /usr/local on sys.path. It would also have a much more limited set of installed packages, i.e. only those needed to support system functionality. /usr/bin/python2 and /usr/bin/python3 then would be user tools, with all the goodness they currently have. It's never gotten much farther than musings, but protecting the system against the weird things people install would be a good thing. OTOH, this feels a lot like virtual environments so maybe there's something useful to be mined there. Cheers, -Barry From a.badger at gmail.com Wed Mar 18 22:44:31 2015 From: a.badger at gmail.com (Toshio Kuratomi) Date: Wed, 18 Mar 2015 14:44:31 -0700 Subject: [Python-Dev] Use ptyhon -s as default shbang for system python executables/daemons In-Reply-To: <20150318122203.12ff1ee3@limelight.wooz.org> References: <20150318122203.12ff1ee3@limelight.wooz.org> Message-ID: <20150318214431.GA4050@roan.lan> On Wed, Mar 18, 2015 at 12:22:03PM -0400, Barry Warsaw wrote: > On Mar 18, 2015, at 03:46 PM, Orion Poplawski wrote: > > >We're starting a discussion in Fedora about setting the default shbang for > >system python executables and/or daemons to python -s or python -Es (or ?). > > We've talked about this in Debian/Ubuntu land and the general consensus is > that for Python 2, use -Es and for Python 3 use -I (which implies -Es). I'm > not sure we're consistent yet in making sure our build tools install these > switches in our shebangs, but I'm hoping after Jessie we can make some > progress on that. > Interesting, I've cautiously in favor of -s in Fedora but the more I've thought about it the less I've liked -E. It just seems like PYTHONPATH is analagous to LD_LIBRARY_PATH for C programs and PATH for shell scripting. We leave both of those for local admins and users to affect the behaviour of programs if they needed to. Was there some discussion of -E specifically in Debian where it was consciously decided that PYTHONPATH was not analagous or that the benefit risk was different than for those other env vars? -Toshio -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 181 bytes Desc: not available URL: From donald at stufft.io Wed Mar 18 22:51:20 2015 From: donald at stufft.io (Donald Stufft) Date: Wed, 18 Mar 2015 17:51:20 -0400 Subject: [Python-Dev] Use ptyhon -s as default shbang for system python executables/daemons In-Reply-To: <20150318124825.337dc1c6@anarchist.wooz.org> References: <20150318124825.337dc1c6@anarchist.wooz.org> Message-ID: <654D3177-05D2-403F-9531-136FC98ED73F@stufft.io> > On Mar 18, 2015, at 12:48 PM, Barry Warsaw wrote: > > On Mar 18, 2015, at 05:31 PM, Victor Stinner wrote: > >> Does it work to pass command line options to Python in the shebang? > > Yes, but only one "word", thus -Es or -I. > > We've often mused about whether it makes sense to have two Pythons on the > system. One for system scripts and another for users. System Python > ('/usr/bin/spython' perhaps) would be locked down and only extensible by > system packages. On Debuntu that might mean no /usr/local on sys.path. It > would also have a much more limited set of installed packages, i.e. only those > needed to support system functionality. > > /usr/bin/python2 and /usr/bin/python3 then would be user tools, with all the > goodness they currently have. > > It's never gotten much farther than musings, but protecting the system against > the weird things people install would be a good thing. OTOH, this feels a lot > like virtual environments so maybe there's something useful to be mined there. > > Cheers, > -Barry > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/donald%40stufft.io I?ve long wished that the OS had it?s own virtual environment. A lot of problems seems to come from trying to cram the things the OS wants with the things that the user wants into the same namespace. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From barry at python.org Wed Mar 18 22:56:07 2015 From: barry at python.org (Barry Warsaw) Date: Wed, 18 Mar 2015 17:56:07 -0400 Subject: [Python-Dev] Use ptyhon -s as default shbang for system python executables/daemons In-Reply-To: <20150318214431.GA4050@roan.lan> References: <20150318122203.12ff1ee3@limelight.wooz.org> <20150318214431.GA4050@roan.lan> Message-ID: <20150318175607.66a25ad3@limelight.wooz.org> On Mar 18, 2015, at 02:44 PM, Toshio Kuratomi wrote: >Interesting, I've cautiously in favor of -s in Fedora but the more I've >thought about it the less I've liked -E. It just seems like PYTHONPATH is >analagous to LD_LIBRARY_PATH for C programs and PATH for shell scripting. >We leave both of those for local admins and users to affect the behaviour of >programs if they needed to. It is, and it isn't. It's different because you can always explicitly override the shebang line if needed. So if a local admin really needed to override $PYTHONPATH (though I can't come up with a use case right now), they could just: $ python3 -s /usr/bin/foo >Was there some discussion of -E specifically in Debian where it was >consciously decided that PYTHONPATH was not analagous or that the benefit >risk was different than for those other env vars? I'd have to go digging around the archives. It wasn't a recent discussion IIRC. Cheers, -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 819 bytes Desc: OpenPGP digital signature URL: From barry at python.org Wed Mar 18 22:58:35 2015 From: barry at python.org (Barry Warsaw) Date: Wed, 18 Mar 2015 17:58:35 -0400 Subject: [Python-Dev] Use ptyhon -s as default shbang for system python executables/daemons In-Reply-To: <654D3177-05D2-403F-9531-136FC98ED73F@stufft.io> References: <20150318124825.337dc1c6@anarchist.wooz.org> <654D3177-05D2-403F-9531-136FC98ED73F@stufft.io> Message-ID: <20150318175835.0fc5e10b@limelight.wooz.org> On Mar 18, 2015, at 05:51 PM, Donald Stufft wrote: >I?ve long wished that the OS had it?s own virtual environment. A lot of >problems seems to come from trying to cram the things the OS wants with the >things that the user wants into the same namespace. Yep, and those breakages can be difficult to debug. Comment #47 for Bug 123456: "Oh wait, you installed *what?!*" Cheers, -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 819 bytes Desc: OpenPGP digital signature URL: From storchaka at gmail.com Thu Mar 19 10:28:37 2015 From: storchaka at gmail.com (Serhiy Storchaka) Date: Thu, 19 Mar 2015 11:28:37 +0200 Subject: [Python-Dev] Needed reviews Message-ID: Here is list of my ready for review patches. It is incomplete and contains only patches for which I don't expect objections or long discussion. Most of them are relative easy and need only formal review. Most of them wait for a review many months. https://bugs.python.org/issue23681 Have -b warn when directly comparing ints and bytes https://bugs.python.org/issue23676 Add support of UnicodeTranslateError in standard error handlers https://bugs.python.org/issue23671 string.Template doesn't work with the self keyword argument https://bugs.python.org/issue23637 Outputting warnings fails when file patch is not ASCII and message is unicode on 2.7. https://bugs.python.org/issue23622 Deprecate unrecognized backslash+letter escapes in re https://bugs.python.org/issue23611 Pickle nested names (e.g. unbound methods) with protocols < 4 https://bugs.python.org/issue23583 IDLE: printing unicode subclasses broken (again) https://bugs.python.org/issue23573 Avoid redundant memory allocations in str.find and like https://bugs.python.org/issue23509 Speed up Counter operators https://bugs.python.org/issue23502 Tkinter doesn't support large integers (out of 32-bit range) https://bugs.python.org/issue23488 Random objects twice as big as necessary on 64-bit builds https://bugs.python.org/issue23466 PEP 461: Inconsistency between str and bytes formatting of integers https://bugs.python.org/issue23419 Faster default __reduce__ for classes without __init__ https://bugs.python.org/issue23290 Faster set copying https://bugs.python.org/issue23252 Add support of writing to unseekable file (e.g. socket) in zipfile https://bugs.python.org/issue23502 pprint: added support for mapping proxy https://bugs.python.org/issue23001 Accept mutable bytes-like objects in builtins that for now support only read-only bytes-like objects https://bugs.python.org/issue22995 Restrict default pickleability. Fail earlier for some types instead of producing incorrect data. https://bugs.python.org/issue22958 Constructors of weakref mapping classes don't accept "self" and "dict" keyword arguments https://bugs.python.org/issue22831 Use "with" to avoid possible fd leaks. Large patch with many simple changes. https://bugs.python.org/issue22826 Support context management protocol in bkfile and simplify Tools/freeze/bkfile.py https://bugs.python.org/issue22721 pprint output for sets and dicts is not stable https://bugs.python.org/issue22687 horrible performance of textwrap.wrap() with a long word https://bugs.python.org/issue22682 Add support of KZ1048 (RK1048) encoding https://bugs.python.org/issue22681 Add support of KOI8-T encoding https://bugs.python.org/issue23671 string.Template doesn't work with the self keyword argument https://bugs.python.org/issue23171 Accept arbitrary iterables in cvs.writerow() https://bugs.python.org/issue23136 Fix inconsistency in handling week 0 in _strptime() https://bugs.python.org/issue22557 Speed up local import https://bugs.python.org/issue22493 Deprecate the use of flags not at the start of regular expression https://bugs.python.org/issue22390 test.regrtest should complain if a test doesn't remove temporary files https://bugs.python.org/issue22364 Improve some re error messages using regex for hints https://bugs.python.org/issue22115 Add new methods to trace Tkinter variables https://bugs.python.org/issue22035 Fatal error in dbm.gdbm https://bugs.python.org/issue21802 Reader of BufferedRWPair is not closed if writer's close() fails https://bugs.python.org/issue21859 Add Python implementation of FileIO https://bugs.python.org/issue21717 Exclusive mode for ZipFile https://bugs.python.org/issue21708 Deprecate nonstandard behavior of a dumbdbm database https://bugs.python.org/issue21526 Support new booleans in Tkinter https://bugs.python.org/issue20168 Derby: Convert the _tkinter module to use Argument Clinic https://bugs.python.org/issue20159 Derby: Convert the ElementTree module to use Argument Clinic https://bugs.python.org/issue20148 Derby: Convert the _sre module to use Argument Clinic https://bugs.python.org/issue19930 os.makedirs('dir1/dir2', 0) always fails https://bugs.python.org/issue18684 Pointers point out of array bound in _sre.c https://bugs.python.org/issue18473 Some objects pickled by Python 3.x are not unpicklable in Python 2.x https://bugs.python.org/issue17711 Persistent id in pickle with protocol version 0 https://bugs.python.org/issue17530 pprint could use line continuation for long bytes literals https://bugs.python.org/issue16314 Support xz compression in distutils https://bugs.python.org/issue15490 Correct __sizeof__ support for StringIO https://bugs.python.org/issue15133 Make tkinter.getboolean() and BooleanVar.get() support Tcl_Obj and always return bool. https://bugs.python.org/issue14904 test_unicode_repr_oflw (in test_bigmem) crashes https://bugs.python.org/issue14260 Make re.groupindex non-modifiable. https://bugs.python.org/issue13583 Add support of slice indexes in sqlite3.Row https://bugs.python.org/issue11344 Add os.path.splitpath(path) function https://bugs.python.org/issue10803 Better support of bytearray objects in ctypes https://bugs.python.org/issue6598 calling email.utils.make_msgid frequently has a non-trivial probability of generating colliding ids https://bugs.python.org/issue4727 Use pickle protocol 4 in the copy module for support more types https://bugs.python.org/issue2175 Add support of character streams of InputSource object in SAX parsers https://bugs.python.org/issue433028 Add support if local flags in regular expressions. (?flag:...) From a.badger at gmail.com Thu Mar 19 19:15:39 2015 From: a.badger at gmail.com (Toshio Kuratomi) Date: Thu, 19 Mar 2015 11:15:39 -0700 Subject: [Python-Dev] Use ptyhon -s as default shbang for system python executables/daemons In-Reply-To: <20150318175607.66a25ad3@limelight.wooz.org> References: <20150318122203.12ff1ee3@limelight.wooz.org> <20150318214431.GA4050@roan.lan> <20150318175607.66a25ad3@limelight.wooz.org> Message-ID: On Wed, Mar 18, 2015 at 2:56 PM, Barry Warsaw wrote: > On Mar 18, 2015, at 02:44 PM, Toshio Kuratomi wrote: > >>Interesting, I've cautiously in favor of -s in Fedora but the more I've >>thought about it the less I've liked -E. It just seems like PYTHONPATH is >>analagous to LD_LIBRARY_PATH for C programs and PATH for shell scripting. >>We leave both of those for local admins and users to affect the behaviour of >>programs if they needed to. > > It is, and it isn't. It's different because you can always explicitly > override the shebang line if needed. So if a local admin really needed to > override $PYTHONPATH (though I can't come up with a use case right now), they > could just: > > $ python3 -s /usr/bin/foo > I could see that as a difference. However, the environment variables give users the ability to change things globally whereas overriding the shebang line is case-by-case so it's not a complete replacement of the functionality. LD_LIBRARY_PATH can be used for things like logging all calls to a specific function, applying a bugfix to a library when you don't have root on the box, evaluating how a potential replacement for a system library will affect the whole system, and other things that are supposed to affect a large number of the files on the OS. PYTHONPATH can be used for the same purposes as long as -E is not embedded into the shebang lines. (I suppose you could write a "python" wrapper script that discards -E... but you'd need root on the box to install your wrapper [since system packages are encouraged to use the full path to python rather than env python] and the change would affect everyone on the box rather than just the person setting the env var). Using -E by default for all system applications would prevent that. The benefit of -E is that it isolates the effects of PYTHONPATH to only specific programs. However, that benefit doesn't seem as striking as it first appears (or at least, as it first appeared to me :-). Unix env vars have their own method of isolation: if the env var is marked for export then it is sent to child processes. If it is not marked for export then it only affects the current process. So it seems like -E isn't adding something new; it's just protecting users from themselves. That seems contrary to "consenting adults" (although distributions are separate entities from python-dev ;-). What makes -s different from -E? If you think about it in the context of users being able to set PYTHONPATH to add libraries that can override system packages then I think specifying -s for system packages establishes a default behaviour: The user can override the defaults but only if they change the environment. Without -s, this expectation is violated for libraries in the user site directory. With -s, the user would have to add the user site directory to PYTHONPATH if they want the libraries there to override system packages. So I guess I'm still leaning towards -E being the wrong choice for Fedora but Fedora lives within a broader ecosystem of python-providing distributions. So I'm interested in seeing whether Debian thought about these aspects when they decided on using -E or if that would change anyone's minds and also what other distributions think about adding -s and/or -E to their packaged applications' shebang lines. -Toshio From barry at python.org Thu Mar 19 20:27:34 2015 From: barry at python.org (Barry Warsaw) Date: Thu, 19 Mar 2015 15:27:34 -0400 Subject: [Python-Dev] Use ptyhon -s as default shbang for system python executables/daemons In-Reply-To: References: <20150318122203.12ff1ee3@limelight.wooz.org> <20150318214431.GA4050@roan.lan> <20150318175607.66a25ad3@limelight.wooz.org> Message-ID: <20150319152734.5bb6cd79@limelight.wooz.org> On Mar 19, 2015, at 11:15 AM, Toshio Kuratomi wrote: >I could see that as a difference. However, the environment variables >give users the ability to change things globally whereas overriding >the shebang line is case-by-case so it's not a complete replacement >of the functionality. You make some good points. I guess it's a trade-off between flexibility and a known secure execution environment. I'm not sure there's a right answer; different admins might have valid different opinions. Cheers, -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 819 bytes Desc: OpenPGP digital signature URL: From sturla.molden at gmail.com Thu Mar 19 20:57:04 2015 From: sturla.molden at gmail.com (Sturla Molden) Date: Thu, 19 Mar 2015 19:57:04 +0000 (UTC) Subject: [Python-Dev] Use ptyhon -s as default shbang for system python executables/daemons References: Message-ID: <772804275448487310.697539sturla.molden-gmail.com@news.gmane.org> Orion Poplawski wrote: > It would be good to get some feedback from the broader python community > before implementing anything, so I'm asking for feedback here. On my systems I have /use/bin/python for the system and ~/anaconda/python for me. Apple and Ubuntu can do whatever they want with their Python's, I am not touching them. The main thing is to keep the system Python and the user Python separate. That is, both the executable and the folder where packages are installed. Whatever I install for myself can fuck up ~/anaconda, but is not allowed to mess with the system files or system folders. Sturla From a.badger at gmail.com Thu Mar 19 21:47:51 2015 From: a.badger at gmail.com (Toshio Kuratomi) Date: Thu, 19 Mar 2015 13:47:51 -0700 Subject: [Python-Dev] Use ptyhon -s as default shbang for system python executables/daemons In-Reply-To: <20150319152734.5bb6cd79@limelight.wooz.org> References: <20150318122203.12ff1ee3@limelight.wooz.org> <20150318214431.GA4050@roan.lan> <20150318175607.66a25ad3@limelight.wooz.org> <20150319152734.5bb6cd79@limelight.wooz.org> Message-ID: I think I've found the Debian discussion (October 2012): http://comments.gmane.org/gmane.linux.debian.devel.python/8188 Lack of PYTHONWARNINGS was brought up late in the discussion thread but I think the understanding that when a particular user sets an environment variable they want it to apply to all scripts they run was kind of lost in the followups (It wasn't directly addressed or mentioned again.) -Toshio On Thu, Mar 19, 2015 at 12:27 PM, Barry Warsaw wrote: > On Mar 19, 2015, at 11:15 AM, Toshio Kuratomi wrote: > >>I could see that as a difference. However, the environment variables >>give users the ability to change things globally whereas overriding >>the shebang line is case-by-case so it's not a complete replacement >>of the functionality. > > You make some good points. I guess it's a trade-off between flexibility and a > known secure execution environment. I'm not sure there's a right answer; > different admins might have valid different opinions. > > Cheers, > -Barry > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/a.badger%40gmail.com > From cs at zip.com.au Thu Mar 19 22:46:19 2015 From: cs at zip.com.au (Cameron Simpson) Date: Fri, 20 Mar 2015 08:46:19 +1100 Subject: [Python-Dev] Use ptyhon -s as default shbang for system python executables/daemons In-Reply-To: <772804275448487310.697539sturla.molden-gmail.com@news.gmane.org> References: <772804275448487310.697539sturla.molden-gmail.com@news.gmane.org> Message-ID: <20150319214619.GA12976@cskk.homeip.net> On 19Mar2015 19:57, Sturla Molden wrote: >Orion Poplawski wrote: >> It would be good to get some feedback from the broader python community >> before implementing anything, so I'm asking for feedback here. > >On my systems I have /use/bin/python for the system and ~/anaconda/python >for me. Apple and Ubuntu can do whatever they want with their Python's, I >am not touching them. The main thing is to keep the system Python and the >user Python separate. That is, both the executable and the folder where >packages are installed. Whatever I install for myself can fuck up >~/anaconda, but is not allowed to mess with the system files or system >folders. Me too. I keep my default virtualenvs in ~/var/venv, and have personal "py2" and "py3" scripts that invoke via the venvs. Having a similar separation in the system would be a good thing, for the same reasons. Also, it would let the OS supplier keep a _much_ smaller package/addon list for the "core" admin python instance, making QA etc easier and more reliable. Popular OSes let the local sysadmin (== "the user") pull in all sorts of stuff to the "system" Python, even from the supplier's repositories. Having a walled off "core" admin python as well seems very prudent. Cheers, Cameron Simpson conclude that this language as a tool is an open invitation for clever tricks; and while exactly this may be the explanation for some of its appeal, /viz./ to those who like to show how clever they are. I am sorry, but I must regard this as one of the most damning things that can be said about a programming language. - Edsger Dijkstra, _The Humble Programmer_ (CACM, Ocotber, 1972) From victor.stinner at gmail.com Thu Mar 19 23:27:26 2015 From: victor.stinner at gmail.com (Victor Stinner) Date: Thu, 19 Mar 2015 23:27:26 +0100 Subject: [Python-Dev] Use ptyhon -s as default shbang for system python executables/daemons In-Reply-To: References: <20150318122203.12ff1ee3@limelight.wooz.org> <20150318214431.GA4050@roan.lan> <20150318175607.66a25ad3@limelight.wooz.org> <20150319152734.5bb6cd79@limelight.wooz.org> Message-ID: 2015-03-19 21:47 GMT+01:00 Toshio Kuratomi : > I think I've found the Debian discussion (October 2012): > > http://comments.gmane.org/gmane.linux.debian.devel.python/8188 > > Lack of PYTHONWARNINGS was brought up late in the discussion thread Maybe we need to modify -E or add a new option to only ignore PYTHONPATH. Victor From techtonik at gmail.com Fri Mar 20 14:31:29 2015 From: techtonik at gmail.com (anatoly techtonik) Date: Fri, 20 Mar 2015 16:31:29 +0300 Subject: [Python-Dev] 2.7.9 regression in argparse Message-ID: Just a pointer for possible regression http://bugs.python.org/issue23058 -- anatoly t. From lkb.teichmann at gmail.com Fri Mar 20 15:03:05 2015 From: lkb.teichmann at gmail.com (Martin Teichmann) Date: Fri, 20 Mar 2015 15:03:05 +0100 Subject: [Python-Dev] super() does not work during class initialization Message-ID: Hi list, while a class is being initialized in a metaclass, it is not always possible to call classmethods of the class, as they might use super(), which in turn uses __class__, which is not initialized yet. I know that this is a known issue, but well, sometimes it even makes sense to fix already known issues... so I wrote a patch that moves the initialization of __class__ into type.__new__, so that one may use super() in a class once it starts existing. It's issue 23722 on the issue tracker. To illustrate what the problem is, the following code raises a RuntimeError: class Meta(type): def __init__(self, name, bases, dict): super().__init__(name, bases, dict) self.f() class A(metaclass=Meta): @classmethod def f(self): super() it works fine with my patch applied. Technically, my patch slightly changes the semantics of python if a metaclass returns something different in its __new__ than what type.__new__ returns. But I could not find any documentation of the current behavior, and also the tests don't test for it, and I consider the current behavior actually buggy. As an example let me give the following simple Singleton metaclass: class Singleton(type): def __new__(cls, name, bases, dict): return super().__new__(cls, name, bases, dict)() class A(metaclass=Singleton): def test(self): assert(isinstance(__class__, type)) A.test() The current python fails the assertion, while with my patch everything is fine, and I personally think __class__ should always actually refer to the class being defined, which means at the minimum that it is actually, well, a class. Greetings Martin From status at bugs.python.org Fri Mar 20 18:08:19 2015 From: status at bugs.python.org (Python tracker) Date: Fri, 20 Mar 2015 18:08:19 +0100 (CET) Subject: [Python-Dev] Summary of Python tracker Issues Message-ID: <20150320170819.CEDD65666D@psf.upfronthosting.co.za> ACTIVITY SUMMARY (2015-03-13 - 2015-03-20) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 4798 (-19) closed 30685 (+81) total 35483 (+62) Open issues with patches: 2258 Issues opened (49) ================== #2211: Cookie.Morsel interface needs update http://bugs.python.org/issue2211 reopened by haypo #22181: os.urandom() should use Linux 3.17 getrandom() syscall http://bugs.python.org/issue22181 reopened by haypo #22359: Remove incorrect uses of recursive make http://bugs.python.org/issue22359 reopened by doko #22516: Windows Installer won't - even when using "just for me"option http://bugs.python.org/issue22516 reopened by NaCl #23187: Segmentation fault, possibly asyncio related http://bugs.python.org/issue23187 reopened by haypo #23637: Warnings error with non-ascii chars. http://bugs.python.org/issue23637 reopened by serhiy.storchaka #23661: Setting a exception side_effect on a mock from create_autospec http://bugs.python.org/issue23661 opened by Ignacio Rossi #23662: Cookie.domain is undocumented http://bugs.python.org/issue23662 opened by Malcolm Smith #23663: Crash on failure in ctypes on Cygwin http://bugs.python.org/issue23663 opened by elieux #23664: Modernize HTML output of difflib.HtmlDiff.make_file() http://bugs.python.org/issue23664 opened by berker.peksag #23665: Provide IDLE menu option to set command-line arguments http://bugs.python.org/issue23665 opened by rhettinger #23666: Add shell session logging option to IDLE http://bugs.python.org/issue23666 opened by rhettinger #23667: IDLE to provide option for making trailing whitespace visible http://bugs.python.org/issue23667 opened by rhettinger #23668: Support os.[f]truncate on Windows http://bugs.python.org/issue23668 opened by steve.dower #23669: test_socket.NonBlockingTCPTests failing due to race condition http://bugs.python.org/issue23669 opened by steve.dower #23670: Modifications to support iOS as a cross-compilation target http://bugs.python.org/issue23670 opened by freakboy3742 #23671: string.Template doesn't work with the self keyword argument http://bugs.python.org/issue23671 opened by serhiy.storchaka #23672: IDLE can crash if file name contains non-BMP Unicode character http://bugs.python.org/issue23672 opened by kamisky #23674: super() documentation isn't very clear http://bugs.python.org/issue23674 opened by Tapani Kiiskinen #23675: glossary entry for 'method resolution order' links to somethin http://bugs.python.org/issue23675 opened by r.david.murray #23676: Add support of UnicodeTranslateError in standard error handler http://bugs.python.org/issue23676 opened by serhiy.storchaka #23677: Mention dict and set comps in library reference http://bugs.python.org/issue23677 opened by FrankMillman #23680: Sporadic freeze in test_interrupted_write_retry_text http://bugs.python.org/issue23680 opened by pitrou #23684: urlparse() documentation does not account for default scheme http://bugs.python.org/issue23684 opened by vadmium #23686: Update Windows and OS X installer OpenSSL to 1.0.2a http://bugs.python.org/issue23686 opened by alex #23688: unnecessary copying of memoryview in gzip.GzipFile.write? http://bugs.python.org/issue23688 opened by wolma #23689: Memory leak in Modules/sre_lib.h http://bugs.python.org/issue23689 opened by abacabadabacaba #23690: re functions never release GIL http://bugs.python.org/issue23690 opened by abacabadabacaba #23691: re.finditer iterator is not reentrant, but doesn't protect aga http://bugs.python.org/issue23691 opened by abacabadabacaba #23692: Undocumented feature prevents re module from finding certain m http://bugs.python.org/issue23692 opened by abacabadabacaba #23693: timeit accuracy could be better http://bugs.python.org/issue23693 opened by rbcollins #23695: idiom for clustering a data series into n-length groups http://bugs.python.org/issue23695 opened by Paddy McCarthy #23697: Module level map & submit for concurrent.futures http://bugs.python.org/issue23697 opened by ncoghlan #23698: Fix inconsistencies in behaviour and document it correctly for http://bugs.python.org/issue23698 opened by pythonhacker #23699: Add a macro to ease writing rich comparisons http://bugs.python.org/issue23699 opened by encukou #23701: Drop extraneous comment from winreg.QueryValue's docstring http://bugs.python.org/issue23701 opened by Claudiu.Popa #23702: docs.python.org/3/howto/descriptor.html still refers to "unbou http://bugs.python.org/issue23702 opened by pfalcon #23703: urljoin() with no directory segments duplicates filename http://bugs.python.org/issue23703 opened by vadmium #23704: Make deques a full MutableSequence by adding index(), insert() http://bugs.python.org/issue23704 opened by rhettinger #23705: Speed-up deque.__contains__ http://bugs.python.org/issue23705 opened by rhettinger #23706: pathlib.Path.write_text should include a newline argument http://bugs.python.org/issue23706 opened by daniel.ugra #23710: C API doc for PyObject_HEAD is outdated http://bugs.python.org/issue23710 opened by tpievila #23712: Experiment: Assume that exact unicode hashes are perfect disc http://bugs.python.org/issue23712 opened by rhettinger #23713: intermittent failure of multiprocessing unit test test_imap_un http://bugs.python.org/issue23713 opened by davin #23717: strptime() with year-weekday pair can produce invalid data http://bugs.python.org/issue23717 opened by serhiy.storchaka #23718: strptime() can produce invalid date with negative year day http://bugs.python.org/issue23718 opened by serhiy.storchaka #23719: PEP 475: port test_eintr to Windows http://bugs.python.org/issue23719 opened by haypo #23721: Set up a daily test coverage run http://bugs.python.org/issue23721 opened by brett.cannon #23722: During metaclass.__init__, super() of the constructed class do http://bugs.python.org/issue23722 opened by Martin.Teichmann Most recent 15 issues with no replies (15) ========================================== #23722: During metaclass.__init__, super() of the constructed class do http://bugs.python.org/issue23722 #23721: Set up a daily test coverage run http://bugs.python.org/issue23721 #23719: PEP 475: port test_eintr to Windows http://bugs.python.org/issue23719 #23718: strptime() can produce invalid date with negative year day http://bugs.python.org/issue23718 #23710: C API doc for PyObject_HEAD is outdated http://bugs.python.org/issue23710 #23706: pathlib.Path.write_text should include a newline argument http://bugs.python.org/issue23706 #23705: Speed-up deque.__contains__ http://bugs.python.org/issue23705 #23704: Make deques a full MutableSequence by adding index(), insert() http://bugs.python.org/issue23704 #23702: docs.python.org/3/howto/descriptor.html still refers to "unbou http://bugs.python.org/issue23702 #23701: Drop extraneous comment from winreg.QueryValue's docstring http://bugs.python.org/issue23701 #23697: Module level map & submit for concurrent.futures http://bugs.python.org/issue23697 #23692: Undocumented feature prevents re module from finding certain m http://bugs.python.org/issue23692 #23691: re.finditer iterator is not reentrant, but doesn't protect aga http://bugs.python.org/issue23691 #23675: glossary entry for 'method resolution order' links to somethin http://bugs.python.org/issue23675 #23669: test_socket.NonBlockingTCPTests failing due to race condition http://bugs.python.org/issue23669 Most recent 15 issues waiting for review (15) ============================================= #23713: intermittent failure of multiprocessing unit test test_imap_un http://bugs.python.org/issue23713 #23712: Experiment: Assume that exact unicode hashes are perfect disc http://bugs.python.org/issue23712 #23705: Speed-up deque.__contains__ http://bugs.python.org/issue23705 #23704: Make deques a full MutableSequence by adding index(), insert() http://bugs.python.org/issue23704 #23703: urljoin() with no directory segments duplicates filename http://bugs.python.org/issue23703 #23701: Drop extraneous comment from winreg.QueryValue's docstring http://bugs.python.org/issue23701 #23699: Add a macro to ease writing rich comparisons http://bugs.python.org/issue23699 #23689: Memory leak in Modules/sre_lib.h http://bugs.python.org/issue23689 #23688: unnecessary copying of memoryview in gzip.GzipFile.write? http://bugs.python.org/issue23688 #23677: Mention dict and set comps in library reference http://bugs.python.org/issue23677 #23676: Add support of UnicodeTranslateError in standard error handler http://bugs.python.org/issue23676 #23671: string.Template doesn't work with the self keyword argument http://bugs.python.org/issue23671 #23670: Modifications to support iOS as a cross-compilation target http://bugs.python.org/issue23670 #23668: Support os.[f]truncate on Windows http://bugs.python.org/issue23668 #23663: Crash on failure in ctypes on Cygwin http://bugs.python.org/issue23663 Top 10 most discussed issues (10) ================================= #23688: unnecessary copying of memoryview in gzip.GzipFile.write? http://bugs.python.org/issue23688 14 msgs #23187: Segmentation fault, possibly asyncio related http://bugs.python.org/issue23187 13 msgs #2211: Cookie.Morsel interface needs update http://bugs.python.org/issue2211 12 msgs #23648: PEP 475 meta issue http://bugs.python.org/issue23648 11 msgs #23670: Modifications to support iOS as a cross-compilation target http://bugs.python.org/issue23670 11 msgs #23441: rlcompleter: tab on empty prefix => insert spaces http://bugs.python.org/issue23441 9 msgs #23546: Windows, 'Edit withIDLE', and multplie installed versions http://bugs.python.org/issue23546 9 msgs #23680: Sporadic freeze in test_interrupted_write_retry_text http://bugs.python.org/issue23680 9 msgs #8027: distutils fail to determine c++ linker with unixcompiler if us http://bugs.python.org/issue8027 8 msgs #23042: ctypes module doesn't build on FreeBSD x86 http://bugs.python.org/issue23042 8 msgs Issues closed (78) ================== #2052: Allow changing difflib._file_template character encoding. http://bugs.python.org/issue2052 closed by berker.peksag #12155: queue example doesn't stop worker threads http://bugs.python.org/issue12155 closed by haypo #12304: expose signalfd(2) in the signal module http://bugs.python.org/issue12304 closed by haypo #12500: Skip test_ssl.test_connect_ex() on connection error http://bugs.python.org/issue12500 closed by haypo #14555: clock_gettime/settime/getres: Add more clock identifiers http://bugs.python.org/issue14555 closed by haypo #15612: Rewrite StringIO to use the _PyUnicodeWriter API http://bugs.python.org/issue15612 closed by haypo #17515: Add sys.setasthook() to allow to use a custom AST optimizer http://bugs.python.org/issue17515 closed by haypo #17628: str==str: compare the first character before calling memcmp() http://bugs.python.org/issue17628 closed by haypo #17911: traceback: add a new thin class storing a traceback without st http://bugs.python.org/issue17911 closed by rbcollins #18507: import_init() should not use Py_FatalError() but return an err http://bugs.python.org/issue18507 closed by haypo #18983: Specify time unit for timeit CLI http://bugs.python.org/issue18983 closed by rbcollins #19233: test_io.test_interrupted_write_retry_text() hangs on Solaris 1 http://bugs.python.org/issue19233 closed by haypo #19428: marshal: error cases are not documented http://bugs.python.org/issue19428 closed by haypo #19564: test_context() of test_multiprocessing_spawn hangs on "x86 Gen http://bugs.python.org/issue19564 closed by haypo #19654: test_tkinter sporadic skipped on "x86 Tiger 3.x" buildbot http://bugs.python.org/issue19654 closed by haypo #19748: test_time failures on AIX http://bugs.python.org/issue19748 closed by haypo #19816: test.regrtest: use tracemalloc to detect memory leaks? http://bugs.python.org/issue19816 closed by haypo #20114: Sporadic failure of test_semaphore_tracker() of test_multiproc http://bugs.python.org/issue20114 closed by haypo #20204: pydocs fails for some C implemented classes http://bugs.python.org/issue20204 closed by serhiy.storchaka #20600: test_create_server_ssl_verify_failed() failure on "PPC64 AIX 3 http://bugs.python.org/issue20600 closed by haypo #20614: test.script_helper should copy SYSTEMROOT environment variable http://bugs.python.org/issue20614 closed by haypo #20680: Pickle socket enums by names http://bugs.python.org/issue20680 closed by ethan.furman #20717: test_4_daemon_threads() fails randomly on "x86 Windows Server http://bugs.python.org/issue20717 closed by haypo #20719: testPythonOrg() of test_robotparser fails with the new www.pyt http://bugs.python.org/issue20719 closed by haypo #21644: Optimize bytearray(int) constructor to use calloc() http://bugs.python.org/issue21644 closed by haypo #21734: compilation of the _ctypes module fails on OpenIndiana: ffi_pr http://bugs.python.org/issue21734 closed by haypo #21735: test_threading.test_main_thread_after_fork_from_nonmain_thread http://bugs.python.org/issue21735 closed by haypo #21788: Rework Python finalization http://bugs.python.org/issue21788 closed by haypo #22333: test_threaded_import.test_parallel_meta_path() failed on x86 W http://bugs.python.org/issue22333 closed by haypo #22521: ctypes compilation fails on FreeBSD: Undefined symbol "ffi_cal http://bugs.python.org/issue22521 closed by haypo #22634: importing _ctypes failed: undefined symbol: ffi_call_win32 http://bugs.python.org/issue22634 closed by berker.peksag #22826: Support context management protocol in bkfile http://bugs.python.org/issue22826 closed by serhiy.storchaka #22861: [2.7] ssl._dnsname_match() and unicode http://bugs.python.org/issue22861 closed by haypo #22903: unittest creates non-picklable errors http://bugs.python.org/issue22903 closed by pitrou #22937: unittest errors can't show locals http://bugs.python.org/issue22937 closed by rbcollins #22971: test_pickle: "Fatal Python error: Cannot recover from stack ov http://bugs.python.org/issue22971 closed by haypo #22984: test_json.test_endless_recursion(): "Fatal Python error: Canno http://bugs.python.org/issue22984 closed by haypo #23001: Accept mutable bytes-like objects http://bugs.python.org/issue23001 closed by serhiy.storchaka #23136: BUG in how _strptime() handles week 0 http://bugs.python.org/issue23136 closed by serhiy.storchaka #23285: PEP 475 - EINTR handling http://bugs.python.org/issue23285 closed by haypo #23303: test_license_exists_at_url() of test_site fails on "x86 XP-4 3 http://bugs.python.org/issue23303 closed by haypo #23333: asyncio: add a new Protocol.connection_failed() method http://bugs.python.org/issue23333 closed by haypo #23353: generator bug with exception: tstate->exc_value is not cleared http://bugs.python.org/issue23353 closed by pitrou #23419: Faster default __reduce__ for classes without __init__ http://bugs.python.org/issue23419 closed by serhiy.storchaka #23456: asyncio: add missing @coroutine decorators http://bugs.python.org/issue23456 closed by haypo #23549: heapq docs should be more precise about how to access the smal http://bugs.python.org/issue23549 closed by eli.bendersky #23568: unittest.mock.MagicMock doesn't support __rdivmod__ http://bugs.python.org/issue23568 closed by berker.peksag #23585: patchcheck doesn't depend on all http://bugs.python.org/issue23585 closed by rbcollins #23593: Update Windows and OS X installer OpenSSL to 1.0.2 http://bugs.python.org/issue23593 closed by ned.deily #23621: Uninstalling Python 3.5 removes a "py.exe" that was installed http://bugs.python.org/issue23621 closed by steve.dower #23624: str.center inconsistent with format "^" http://bugs.python.org/issue23624 closed by rhettinger #23628: See if os.scandir() could help speed up importlib http://bugs.python.org/issue23628 closed by brett.cannon #23631: 3.5 (a2) traceback regression snarls Idle http://bugs.python.org/issue23631 closed by rbcollins #23632: memoryview doesn't allow tuple-indexing http://bugs.python.org/issue23632 closed by pitrou #23644: g++ module compile fails with ???_Atomic??? does not name a ty http://bugs.python.org/issue23644 closed by haypo #23645: Incorrect doc for __getslice__ http://bugs.python.org/issue23645 closed by python-dev #23646: PEP 475: handle EINTR in the time module, retry sleep() http://bugs.python.org/issue23646 closed by haypo #23656: shutil.copyfile (or built-in open) releases sqlite3's transact http://bugs.python.org/issue23656 closed by h-miyajima #23658: multiprocessing: string arg to SystemExit http://bugs.python.org/issue23658 closed by Dan Nawrocki #23673: IntEnum is unpicklable by previous Python versions http://bugs.python.org/issue23673 closed by ethan.furman #23678: imaplib status command cannot handle folder name containing wh http://bugs.python.org/issue23678 closed by r.david.murray #23679: SSL Ciphers RC4 http://bugs.python.org/issue23679 closed by benjamin.peterson #23681: Have -b warn when directly comparing ints and bytes http://bugs.python.org/issue23681 closed by serhiy.storchaka #23682: distutils docs still talk about compatibility with Python 2.2 http://bugs.python.org/issue23682 closed by berker.peksag #23683: allow timeit to run expensive reset code per repeats http://bugs.python.org/issue23683 closed by rbcollins #23685: Fix usage of PyMODINIT_FUNC http://bugs.python.org/issue23685 closed by haypo #23687: Stacktrace identifies wrong line in multiline list comprehensi http://bugs.python.org/issue23687 closed by r.david.murray #23694: PEP 475: handle EINTR in fileutils.c http://bugs.python.org/issue23694 closed by haypo #23696: zipimport: chain ImportError to OSError http://bugs.python.org/issue23696 closed by haypo #23700: tempfile.NamedTemporaryFile can close too early if used as ite http://bugs.python.org/issue23700 closed by serhiy.storchaka #23707: PEP 475: os.urandom() doesn't handle EINTR correctly http://bugs.python.org/issue23707 closed by haypo #23708: PEP 475: Add _Py_read() and _Py_write() functions http://bugs.python.org/issue23708 closed by haypo #23709: Refactor ossaudiodev: use _Py_read and _Py_write with the Py_b http://bugs.python.org/issue23709 closed by haypo #23711: ConfigParser module blames on section less ini file http://bugs.python.org/issue23711 closed by vadmium #23714: Set comprehension + eval doesn't work inside function definiti http://bugs.python.org/issue23714 closed by ned.deily #23715: PEP 475: handle EINTR in the signal module http://bugs.python.org/issue23715 closed by haypo #23716: test_multiprocessing_spawn hangs on x86 Ubuntu Shared 3.x http://bugs.python.org/issue23716 closed by haypo #23720: __del__() order is broken since 3.4.0 http://bugs.python.org/issue23720 closed by pitrou From bcannon at gmail.com Fri Mar 20 19:34:38 2015 From: bcannon at gmail.com (Brett Cannon) Date: Fri, 20 Mar 2015 18:34:38 +0000 Subject: [Python-Dev] Final call for PEP 488: eliminating PYO files Message-ID: I have decided to have the default case of no optimization levels mean that the .pyc file name will have *no* optimization level specified in the name and thus be just as it is today. I made this decision due to potential backwards-compatibility issues -- although I expect them to be minutes -- and to not force other implementations like PyPy to have some bogus value set since they don't have .pyo files to begin with (PyPy actually uses bytecode for -O and don't bother with -OO since PyPy already uses a bunch of memory when running). Since this closes out the last open issue, I need either a BDFL decision or a BDFAP to be assigned to make a decision. Guido? ====================================== PEP: 488 Title: Elimination of PYO files Version: $Revision$ Last-Modified: $Date$ Author: Brett Cannon Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 20-Feb-2015 Post-History: 2015-03-06 2015-03-13 2015-03-20 Abstract ======== This PEP proposes eliminating the concept of PYO files from Python. To continue the support of the separation of bytecode files based on their optimization level, this PEP proposes extending the PYC file name to include the optimization level in the bytecode repository directory when it's called for (i.e., the ``__pycache__`` directory). Rationale ========= As of today, bytecode files come in two flavours: PYC and PYO. A PYC file is the bytecode file generated and read from when no optimization level is specified at interpreter startup (i.e., ``-O`` is not specified). A PYO file represents the bytecode file that is read/written when **any** optimization level is specified (i.e., when ``-O`` **or** ``-OO`` is specified). This means that while PYC files clearly delineate the optimization level used when they were generated -- namely no optimizations beyond the peepholer -- the same is not true for PYO files. To put this in terms of optimization levels and the file extension: - 0: ``.pyc`` - 1 (``-O``): ``.pyo`` - 2 (``-OO``): ``.pyo`` The reuse of the ``.pyo`` file extension for both level 1 and 2 optimizations means that there is no clear way to tell what optimization level was used to generate the bytecode file. In terms of reading PYO files, this can lead to an interpreter using a mixture of optimization levels with its code if the user was not careful to make sure all PYO files were generated using the same optimization level (typically done by blindly deleting all PYO files and then using the `compileall` module to compile all-new PYO files [1]_). This issue is only compounded when people optimize Python code beyond what the interpreter natively supports, e.g., using the astoptimizer project [2]_. In terms of writing PYO files, the need to delete all PYO files every time one either changes the optimization level they want to use or are unsure of what optimization was used the last time PYO files were generated leads to unnecessary file churn. The change proposed by this PEP also allows for **all** optimization levels to be pre-compiled for bytecode files ahead of time, something that is currently impossible thanks to the reuse of the ``.pyo`` file extension for multiple optimization levels. As for distributing bytecode-only modules, having to distribute both ``.pyc`` and ``.pyo`` files is unnecessary for the common use-case of code obfuscation and smaller file deployments. This means that bytecode-only modules will only load from their non-optimized ``.pyc`` file name. Proposal ======== To eliminate the ambiguity that PYO files present, this PEP proposes eliminating the concept of PYO files and their accompanying ``.pyo`` file extension. To allow for the optimization level to be unambiguous as well as to avoid having to regenerate optimized bytecode files needlessly in the `__pycache__` directory, the optimization level used to generate the bytecode file will be incorporated into the bytecode file name. When no optimization level is specified, the pre-PEP ``.pyc`` file name will be used (i.e., no change in file name semantics). This increases backwards-compatibility while also being more understanding of Python implementations which have no use for optimization levels (e.g., PyPy[10]_). Currently bytecode file names are created by ``importlib.util.cache_from_source()``, approximately using the following expression defined by PEP 3147 [3]_, [4]_, [5]_:: '{name}.{cache_tag}.pyc'.format(name=module_name, cache_tag=sys.implementation.cache_tag) This PEP proposes to change the expression when an optimization level is specified to:: '{name}.{cache_tag}.opt-{optimization}.pyc'.format( name=module_name, cache_tag=sys.implementation.cache_tag, optimization=str(sys.flags.optimize)) The "opt-" prefix was chosen so as to provide a visual separator from the cache tag. The placement of the optimization level after the cache tag was chosen to preserve lexicographic sort order of bytecode file names based on module name and cache tag which will not vary for a single interpreter. The "opt-" prefix was chosen over "o" so as to be somewhat self-documenting. The "opt-" prefix was chosen over "O" so as to not have any confusion in case "0" was the leading prefix of the optimization level. A period was chosen over a hyphen as a separator so as to distinguish clearly that the optimization level is not part of the interpreter version as specified by the cache tag. It also lends to the use of the period in the file name to delineate semantically different concepts. For example, if ``-OO`` had been passed to the interpreter then instead of ``importlib.cpython-35.pyo`` the file name would be ``importlib.cpython-35.opt-2.pyc``. It should be noted that this change in no way affects the performance of import. Since the import system looks for a single bytecode file based on the optimization level of the interpreter already and generates a new bytecode file if it doesn't exist, the introduction of potentially more bytecode files in the ``__pycache__`` directory has no effect in terms of stat calls. The interpreter will continue to look for only a single bytecode file based on the optimization level and thus no increase in stat calls will occur. The only potentially negative result of this PEP is the probable increase in the number of ``.pyc`` files and thus increase in storage use. But for platforms where this is an issue, ``sys.dont_write_bytecode`` exists to turn off bytecode generation so that it can be controlled offline. Implementation ============== importlib --------- As ``importlib.util.cache_from_source()`` is the API that exposes bytecode file paths as well as being directly used by importlib, it requires the most critical change. As of Python 3.4, the function's signature is:: importlib.util.cache_from_source(path, debug_override=None) This PEP proposes changing the signature in Python 3.5 to:: importlib.util.cache_from_source(path, debug_override=None, *, optimization=None) The introduced ``optimization`` keyword-only parameter will control what optimization level is specified in the file name. If the argument is ``None`` then the current optimization level of the interpreter will be assumed (including no optimization). Any argument given for ``optimization`` will be passed to ``str()`` and must have ``str.isalnum()`` be true, else ``ValueError`` will be raised (this prevents invalid characters being used in the file name). If the empty string is passed in for ``optimization`` then the addition of the optimization will be suppressed, reverting to the file name format which predates this PEP. It is expected that beyond Python's own two optimization levels, third-party code will use a hash of optimization names to specify the optimization level, e.g. ``hashlib.sha256(','.join(['no dead code', 'const folding'])).hexdigest()``. While this might lead to long file names, it is assumed that most users never look at the contents of the __pycache__ directory and so this won't be an issue. The ``debug_override`` parameter will be deprecated. As the parameter expects a boolean, the integer value of the boolean will be used as if it had been provided as the argument to ``optimization`` (a ``None`` argument will mean the same as for ``optimization``). A deprecation warning will be raised when ``debug_override`` is given a value other than ``None``, but there are no plans for the complete removal of the parameter at this time (but removal will be no later than Python 4). The various module attributes for importlib.machinery which relate to bytecode file suffixes will be updated [7]_. The ``DEBUG_BYTECODE_SUFFIXES`` and ``OPTIMIZED_BYTECODE_SUFFIXES`` will both be documented as deprecated and set to the same value as ``BYTECODE_SUFFIXES`` (removal of ``DEBUG_BYTECODE_SUFFIXES`` and ``OPTIMIZED_BYTECODE_SUFFIXES`` is not currently planned, but will be not later than Python 4). All various finders and loaders will also be updated as necessary, but updating the previous mentioned parts of importlib should be all that is required. Rest of the standard library ---------------------------- The various functions exposed by the ``py_compile`` and ``compileall`` functions will be updated as necessary to make sure they follow the new bytecode file name semantics [6]_, [1]_. The CLI for the ``compileall`` module will not be directly affected (the ``-b`` flag will be implicit as it will no longer generate ``.pyo`` files when ``-O`` is specified). Compatibility Considerations ============================ Any code directly manipulating bytecode files from Python 3.2 on will need to consider the impact of this change on their code (prior to Python 3.2 -- including all of Python 2 -- there was no __pycache__ which already necessitates bifurcating bytecode file handling support). If code was setting the ``debug_override`` argument to ``importlib.util.cache_from_source()`` then care will be needed if they want the path to a bytecode file with an optimization level of 2. Otherwise only code **not** using ``importlib.util.cache_from_source()`` will need updating. As for people who distribute bytecode-only modules (i.e., use a bytecode file instead of a source file), they will have to choose which optimization level they want their bytecode files to be since distributing a ``.pyo`` file with a ``.pyc`` file will no longer be of any use. Since people typically only distribute bytecode files for code obfuscation purposes or smaller distribution size then only having to distribute a single ``.pyc`` should actually be beneficial to these use-cases. And since the magic number for bytecode files changed in Python 3.5 to support PEP 465 there is no need to support pre-existing ``.pyo`` files [8]_. Rejected Ideas ============== Completely dropping optimization levels from CPython ---------------------------------------------------- Some have suggested that instead of accommodating the various optimization levels in CPython, we should instead drop them entirely. The argument is that significant performance gains would occur from runtime optimizations through something like a JIT and not through pre-execution bytecode optimizations. This idea is rejected for this PEP as that ignores the fact that there are people who do find the pre-existing optimization levels for CPython useful. It also assumes that no other Python interpreter would find what this PEP proposes useful. Alternative formatting of the optimization level in the file name ----------------------------------------------------------------- Using the "opt-" prefix and placing the optimization level between the cache tag and file extension is not critical. All options which have been considered are: * ``importlib.cpython-35.opt-1.pyc`` * ``importlib.cpython-35.opt1.pyc`` * ``importlib.cpython-35.o1.pyc`` * ``importlib.cpython-35.O1.pyc`` * ``importlib.cpython-35.1.pyc`` * ``importlib.cpython-35-O1.pyc`` * ``importlib.O1.cpython-35.pyc`` * ``importlib.o1.cpython-35.pyc`` * ``importlib.1.cpython-35.pyc`` These were initially rejected either because they would change the sort order of bytecode files, possible ambiguity with the cache tag, or were not self-documenting enough. An informal poll was taken and people clearly preferred the formatting proposed by the PEP [9]_. Since this topic is non-technical and of personal choice, the issue is considered solved. Embedding the optimization level in the bytecode metadata --------------------------------------------------------- Some have suggested that rather than embedding the optimization level of bytecode in the file name that it be included in the file's metadata instead. This would mean every interpreter had a single copy of bytecode at any time. Changing the optimization level would thus require rewriting the bytecode, but there would also only be a single file to care about. This has been rejected due to the fact that Python is often installed as a root-level application and thus modifying the bytecode file for modules in the standard library are always possible. In this situation integrators would need to guess at what a reasonable optimization level was for users for any/all situations. By allowing multiple optimization levels to co-exist simultaneously it frees integrators from having to guess what users want and allows users to utilize the optimization level they want. References ========== .. [1] The compileall module (https://docs.python.org/3/library/compileall.html#module-compileall) .. [2] The astoptimizer project (https://pypi.python.org/pypi/astoptimizer) .. [3] ``importlib.util.cache_from_source()`` ( https://docs.python.org/3.5/library/importlib.html#importlib.util.cache_from_source ) .. [4] Implementation of ``importlib.util.cache_from_source()`` from CPython 3.4.3rc1 ( https://hg.python.org/cpython/file/038297948389/Lib/importlib/_bootstrap.py#l437 ) .. [5] PEP 3147, PYC Repository Directories, Warsaw (http://www.python.org/dev/peps/pep-3147) .. [6] The py_compile module (https://docs.python.org/3/library/compileall.html#module-compileall) .. [7] The importlib.machinery module ( https://docs.python.org/3/library/importlib.html#module-importlib.machinery) .. [8] ``importlib.util.MAGIC_NUMBER`` ( https://docs.python.org/3/library/importlib.html#importlib.util.MAGIC_NUMBER ) .. [9] Informal poll of file name format options on Google+ (https://plus.google.com/u/0/+BrettCannon/posts/fZynLNwHWGm) .. [10] The PyPy Project (http://pypy.org/) Copyright ========= This document has been placed in the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End: -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Fri Mar 20 21:41:30 2015 From: guido at python.org (Guido van Rossum) Date: Fri, 20 Mar 2015 13:41:30 -0700 Subject: [Python-Dev] Final call for PEP 488: eliminating PYO files In-Reply-To: References: Message-ID: I am willing to be the BDFL for this PEP. I have tried to skim the recent discussion (only python-dev) and I don't see much remaining controversy. HOWEVER... The PEP is not clear (or at least too subtle) about the actual name for optimization level 0. If I have foo.py, and I compile it three times with three different optimization levels (no optimization; -O; -OO), and then I look in __pycache__, would I see this: # (1) foo.cpython-35.pyc foo.cpython-35.opt-1.pyc foo.cpython-35.opt-2.pyc Or would I see this? # (2) foo.cpython-35.opt-0.pyc foo.cpython-35.opt-1.pyc foo.cpython-35.opt-2.pyc Your lead-in ("I have decided to have the default case of no optimization levels mean that the .pyc file name will have *no* optimization level specified in the name and thus be just as it is today.") makes me think I should expect (1), but I can't actually pinpoint where the language of the PEP says this. On Fri, Mar 20, 2015 at 11:34 AM, Brett Cannon wrote: > I have decided to have the default case of no optimization levels mean > that the .pyc file name will have *no* optimization level specified in > the name and thus be just as it is today. I made this decision due to > potential backwards-compatibility issues -- although I expect them to be > minutes -- and to not force other implementations like PyPy to have some > bogus value set since they don't have .pyo files to begin with (PyPy > actually uses bytecode for -O and don't bother with -OO since PyPy already > uses a bunch of memory when running). > > Since this closes out the last open issue, I need either a BDFL decision > or a BDFAP to be assigned to make a decision. Guido? > > ====================================== > > PEP: 488 > Title: Elimination of PYO files > Version: $Revision$ > Last-Modified: $Date$ > Author: Brett Cannon > Status: Draft > Type: Standards Track > Content-Type: text/x-rst > Created: 20-Feb-2015 > Post-History: > 2015-03-06 > 2015-03-13 > 2015-03-20 > > Abstract > ======== > > This PEP proposes eliminating the concept of PYO files from Python. > To continue the support of the separation of bytecode files based on > their optimization level, this PEP proposes extending the PYC file > name to include the optimization level in the bytecode repository > directory when it's called for (i.e., the ``__pycache__`` directory). > > > Rationale > ========= > > As of today, bytecode files come in two flavours: PYC and PYO. A PYC > file is the bytecode file generated and read from when no > optimization level is specified at interpreter startup (i.e., ``-O`` > is not specified). A PYO file represents the bytecode file that is > read/written when **any** optimization level is specified (i.e., when > ``-O`` **or** ``-OO`` is specified). This means that while PYC > files clearly delineate the optimization level used when they were > generated -- namely no optimizations beyond the peepholer -- the same > is not true for PYO files. To put this in terms of optimization > levels and the file extension: > > - 0: ``.pyc`` > - 1 (``-O``): ``.pyo`` > - 2 (``-OO``): ``.pyo`` > > The reuse of the ``.pyo`` file extension for both level 1 and 2 > optimizations means that there is no clear way to tell what > optimization level was used to generate the bytecode file. In terms > of reading PYO files, this can lead to an interpreter using a mixture > of optimization levels with its code if the user was not careful to > make sure all PYO files were generated using the same optimization > level (typically done by blindly deleting all PYO files and then > using the `compileall` module to compile all-new PYO files [1]_). > This issue is only compounded when people optimize Python code beyond > what the interpreter natively supports, e.g., using the astoptimizer > project [2]_. > > In terms of writing PYO files, the need to delete all PYO files > every time one either changes the optimization level they want to use > or are unsure of what optimization was used the last time PYO files > were generated leads to unnecessary file churn. The change proposed > by this PEP also allows for **all** optimization levels to be > pre-compiled for bytecode files ahead of time, something that is > currently impossible thanks to the reuse of the ``.pyo`` file > extension for multiple optimization levels. > > As for distributing bytecode-only modules, having to distribute both > ``.pyc`` and ``.pyo`` files is unnecessary for the common use-case > of code obfuscation and smaller file deployments. This means that > bytecode-only modules will only load from their non-optimized > ``.pyc`` file name. > > > Proposal > ======== > > To eliminate the ambiguity that PYO files present, this PEP proposes > eliminating the concept of PYO files and their accompanying ``.pyo`` > file extension. To allow for the optimization level to be unambiguous > as well as to avoid having to regenerate optimized bytecode files > needlessly in the `__pycache__` directory, the optimization level > used to generate the bytecode file will be incorporated into the > bytecode file name. When no optimization level is specified, the > pre-PEP ``.pyc`` file name will be used (i.e., no change in file name > semantics). This increases backwards-compatibility while also being > more understanding of Python implementations which have no use for > optimization levels (e.g., PyPy[10]_). > > Currently bytecode file names are created by > ``importlib.util.cache_from_source()``, approximately using the > following expression defined by PEP 3147 [3]_, [4]_, [5]_:: > > '{name}.{cache_tag}.pyc'.format(name=module_name, > cache_tag=sys.implementation.cache_tag) > > This PEP proposes to change the expression when an optimization > level is specified to:: > > '{name}.{cache_tag}.opt-{optimization}.pyc'.format( > name=module_name, > cache_tag=sys.implementation.cache_tag, > optimization=str(sys.flags.optimize)) > > The "opt-" prefix was chosen so as to provide a visual separator > from the cache tag. The placement of the optimization level after > the cache tag was chosen to preserve lexicographic sort order of > bytecode file names based on module name and cache tag which will > not vary for a single interpreter. The "opt-" prefix was chosen over > "o" so as to be somewhat self-documenting. The "opt-" prefix was > chosen over "O" so as to not have any confusion in case "0" was the > leading prefix of the optimization level. > > A period was chosen over a hyphen as a separator so as to distinguish > clearly that the optimization level is not part of the interpreter > version as specified by the cache tag. It also lends to the use of > the period in the file name to delineate semantically different > concepts. > > For example, if ``-OO`` had been passed to the interpreter then instead > of ``importlib.cpython-35.pyo`` the file name would be > ``importlib.cpython-35.opt-2.pyc``. > > It should be noted that this change in no way affects the performance > of import. Since the import system looks for a single bytecode file > based on the optimization level of the interpreter already and > generates a new bytecode file if it doesn't exist, the introduction > of potentially more bytecode files in the ``__pycache__`` directory > has no effect in terms of stat calls. The interpreter will continue > to look for only a single bytecode file based on the optimization > level and thus no increase in stat calls will occur. > > The only potentially negative result of this PEP is the probable > increase in the number of ``.pyc`` files and thus increase in storage > use. But for platforms where this is an issue, > ``sys.dont_write_bytecode`` exists to turn off bytecode generation so > that it can be controlled offline. > > > Implementation > ============== > > importlib > --------- > > As ``importlib.util.cache_from_source()`` is the API that exposes > bytecode file paths as well as being directly used by importlib, it > requires the most critical change. As of Python 3.4, the function's > signature is:: > > importlib.util.cache_from_source(path, debug_override=None) > > This PEP proposes changing the signature in Python 3.5 to:: > > importlib.util.cache_from_source(path, debug_override=None, *, > optimization=None) > > The introduced ``optimization`` keyword-only parameter will control > what optimization level is specified in the file name. If the > argument is ``None`` then the current optimization level of the > interpreter will be assumed (including no optimization). Any argument > given for ``optimization`` will be passed to ``str()`` and must have > ``str.isalnum()`` be true, else ``ValueError`` will be raised (this > prevents invalid characters being used in the file name). If the > empty string is passed in for ``optimization`` then the addition of > the optimization will be suppressed, reverting to the file name > format which predates this PEP. > > It is expected that beyond Python's own two optimization levels, > third-party code will use a hash of optimization names to specify the > optimization level, e.g. > ``hashlib.sha256(','.join(['no dead code', 'const > folding'])).hexdigest()``. > While this might lead to long file names, it is assumed that most > users never look at the contents of the __pycache__ directory and so > this won't be an issue. > > The ``debug_override`` parameter will be deprecated. As the parameter > expects a boolean, the integer value of the boolean will be used as > if it had been provided as the argument to ``optimization`` (a > ``None`` argument will mean the same as for ``optimization``). A > deprecation warning will be raised when ``debug_override`` is given a > value other than ``None``, but there are no plans for the complete > removal of the parameter at this time (but removal will be no later > than Python 4). > > The various module attributes for importlib.machinery which relate to > bytecode file suffixes will be updated [7]_. The > ``DEBUG_BYTECODE_SUFFIXES`` and ``OPTIMIZED_BYTECODE_SUFFIXES`` will > both be documented as deprecated and set to the same value as > ``BYTECODE_SUFFIXES`` (removal of ``DEBUG_BYTECODE_SUFFIXES`` and > ``OPTIMIZED_BYTECODE_SUFFIXES`` is not currently planned, but will be > not later than Python 4). > > All various finders and loaders will also be updated as necessary, > but updating the previous mentioned parts of importlib should be all > that is required. > > > Rest of the standard library > ---------------------------- > > The various functions exposed by the ``py_compile`` and > ``compileall`` functions will be updated as necessary to make sure > they follow the new bytecode file name semantics [6]_, [1]_. The CLI > for the ``compileall`` module will not be directly affected (the > ``-b`` flag will be implicit as it will no longer generate ``.pyo`` > files when ``-O`` is specified). > > > Compatibility Considerations > ============================ > > Any code directly manipulating bytecode files from Python 3.2 on > will need to consider the impact of this change on their code (prior > to Python 3.2 -- including all of Python 2 -- there was no > __pycache__ which already necessitates bifurcating bytecode file > handling support). If code was setting the ``debug_override`` > argument to ``importlib.util.cache_from_source()`` then care will be > needed if they want the path to a bytecode file with an optimization > level of 2. Otherwise only code **not** using > ``importlib.util.cache_from_source()`` will need updating. > > As for people who distribute bytecode-only modules (i.e., use a > bytecode file instead of a source file), they will have to choose > which optimization level they want their bytecode files to be since > distributing a ``.pyo`` file with a ``.pyc`` file will no longer be > of any use. Since people typically only distribute bytecode files for > code obfuscation purposes or smaller distribution size then only > having to distribute a single ``.pyc`` should actually be beneficial > to these use-cases. And since the magic number for bytecode files > changed in Python 3.5 to support PEP 465 there is no need to support > pre-existing ``.pyo`` files [8]_. > > > Rejected Ideas > ============== > > Completely dropping optimization levels from CPython > ---------------------------------------------------- > > Some have suggested that instead of accommodating the various > optimization levels in CPython, we should instead drop them > entirely. The argument is that significant performance gains would > occur from runtime optimizations through something like a JIT and not > through pre-execution bytecode optimizations. > > This idea is rejected for this PEP as that ignores the fact that > there are people who do find the pre-existing optimization levels for > CPython useful. It also assumes that no other Python interpreter > would find what this PEP proposes useful. > > > Alternative formatting of the optimization level in the file name > ----------------------------------------------------------------- > > Using the "opt-" prefix and placing the optimization level between > the cache tag and file extension is not critical. All options which > have been considered are: > > * ``importlib.cpython-35.opt-1.pyc`` > * ``importlib.cpython-35.opt1.pyc`` > * ``importlib.cpython-35.o1.pyc`` > * ``importlib.cpython-35.O1.pyc`` > * ``importlib.cpython-35.1.pyc`` > * ``importlib.cpython-35-O1.pyc`` > * ``importlib.O1.cpython-35.pyc`` > * ``importlib.o1.cpython-35.pyc`` > * ``importlib.1.cpython-35.pyc`` > > These were initially rejected either because they would change the > sort order of bytecode files, possible ambiguity with the cache tag, > or were not self-documenting enough. An informal poll was taken and > people clearly preferred the formatting proposed by the PEP [9]_. > Since this topic is non-technical and of personal choice, the issue > is considered solved. > > > Embedding the optimization level in the bytecode metadata > --------------------------------------------------------- > > Some have suggested that rather than embedding the optimization level > of bytecode in the file name that it be included in the file's > metadata instead. This would mean every interpreter had a single copy > of bytecode at any time. Changing the optimization level would thus > require rewriting the bytecode, but there would also only be a single > file to care about. > > This has been rejected due to the fact that Python is often installed > as a root-level application and thus modifying the bytecode file for > modules in the standard library are always possible. In this > situation integrators would need to guess at what a reasonable > optimization level was for users for any/all situations. By > allowing multiple optimization levels to co-exist simultaneously it > frees integrators from having to guess what users want and allows > users to utilize the optimization level they want. > > > References > ========== > > .. [1] The compileall module > (https://docs.python.org/3/library/compileall.html#module-compileall) > > .. [2] The astoptimizer project > (https://pypi.python.org/pypi/astoptimizer) > > .. [3] ``importlib.util.cache_from_source()`` > ( > https://docs.python.org/3.5/library/importlib.html#importlib.util.cache_from_source > ) > > .. [4] Implementation of ``importlib.util.cache_from_source()`` from > CPython 3.4.3rc1 > ( > https://hg.python.org/cpython/file/038297948389/Lib/importlib/_bootstrap.py#l437 > ) > > .. [5] PEP 3147, PYC Repository Directories, Warsaw > (http://www.python.org/dev/peps/pep-3147) > > .. [6] The py_compile module > (https://docs.python.org/3/library/compileall.html#module-compileall) > > .. [7] The importlib.machinery module > ( > https://docs.python.org/3/library/importlib.html#module-importlib.machinery > ) > > .. [8] ``importlib.util.MAGIC_NUMBER`` > ( > https://docs.python.org/3/library/importlib.html#importlib.util.MAGIC_NUMBER > ) > > .. [9] Informal poll of file name format options on Google+ > (https://plus.google.com/u/0/+BrettCannon/posts/fZynLNwHWGm) > > .. [10] The PyPy Project > (http://pypy.org/) > > > Copyright > ========= > > This document has been placed in the public domain. > > > .. > Local Variables: > mode: indented-text > indent-tabs-mode: nil > sentence-end-double-space: t > fill-column: 70 > coding: utf-8 > End: > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/guido%40python.org > > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From tjreedy at udel.edu Fri Mar 20 22:35:52 2015 From: tjreedy at udel.edu (Terry Reedy) Date: Fri, 20 Mar 2015 17:35:52 -0400 Subject: [Python-Dev] 2.7.9 regression in argparse In-Reply-To: References: Message-ID: On 3/20/2015 9:31 AM, anatoly techtonik wrote: > Just a pointer for possible regression http://bugs.python.org/issue23058 I just added the argparse maintainer to the nosy list -- Terry Jan Reedy From bcannon at gmail.com Sat Mar 21 00:00:24 2015 From: bcannon at gmail.com (Brett Cannon) Date: Fri, 20 Mar 2015 23:00:24 +0000 Subject: [Python-Dev] Final call for PEP 488: eliminating PYO files In-Reply-To: References: Message-ID: On Fri, Mar 20, 2015 at 4:41 PM Guido van Rossum wrote: > I am willing to be the BDFL for this PEP. I have tried to skim the recent > discussion (only python-dev) and I don't see much remaining controversy. > HOWEVER... The PEP is not clear (or at least too subtle) about the actual > name for optimization level 0. If I have foo.py, and I compile it three > times with three different optimization levels (no optimization; -O; -OO), > and then I look in __pycache__, would I see this: > > # (1) > foo.cpython-35.pyc > foo.cpython-35.opt-1.pyc > foo.cpython-35.opt-2.pyc > > Or would I see this? > > # (2) > foo.cpython-35.opt-0.pyc > foo.cpython-35.opt-1.pyc > foo.cpython-35.opt-2.pyc > #1 > > Your lead-in ("I have decided to have the default case of no optimization > levels mean that the .pyc file name will have *no* optimization level > specified in the name and thus be just as it is today.") makes me think I > should expect (1), but I can't actually pinpoint where the language of the > PEP says this. > It was meant to be explained by "When no optimization level is specified, the pre-PEP ``.pyc`` file name will be used (i.e., no change in file name semantics)", but obviously it's a bit too subtle. I just updated the PEP with an explicit list of bytecode file name examples based on no -O, -O, and -OO. -Brett > > > On Fri, Mar 20, 2015 at 11:34 AM, Brett Cannon wrote: > >> I have decided to have the default case of no optimization levels mean >> that the .pyc file name will have *no* optimization level specified in >> the name and thus be just as it is today. I made this decision due to >> potential backwards-compatibility issues -- although I expect them to be >> minutes -- and to not force other implementations like PyPy to have some >> bogus value set since they don't have .pyo files to begin with (PyPy >> actually uses bytecode for -O and don't bother with -OO since PyPy already >> uses a bunch of memory when running). >> >> Since this closes out the last open issue, I need either a BDFL decision >> or a BDFAP to be assigned to make a decision. Guido? >> >> ====================================== >> >> PEP: 488 >> Title: Elimination of PYO files >> Version: $Revision$ >> Last-Modified: $Date$ >> Author: Brett Cannon >> Status: Draft >> Type: Standards Track >> Content-Type: text/x-rst >> Created: 20-Feb-2015 >> Post-History: >> 2015-03-06 >> 2015-03-13 >> 2015-03-20 >> >> Abstract >> ======== >> >> This PEP proposes eliminating the concept of PYO files from Python. >> To continue the support of the separation of bytecode files based on >> their optimization level, this PEP proposes extending the PYC file >> name to include the optimization level in the bytecode repository >> directory when it's called for (i.e., the ``__pycache__`` directory). >> >> >> Rationale >> ========= >> >> As of today, bytecode files come in two flavours: PYC and PYO. A PYC >> file is the bytecode file generated and read from when no >> optimization level is specified at interpreter startup (i.e., ``-O`` >> is not specified). A PYO file represents the bytecode file that is >> read/written when **any** optimization level is specified (i.e., when >> ``-O`` **or** ``-OO`` is specified). This means that while PYC >> files clearly delineate the optimization level used when they were >> generated -- namely no optimizations beyond the peepholer -- the same >> is not true for PYO files. To put this in terms of optimization >> levels and the file extension: >> >> - 0: ``.pyc`` >> - 1 (``-O``): ``.pyo`` >> - 2 (``-OO``): ``.pyo`` >> >> The reuse of the ``.pyo`` file extension for both level 1 and 2 >> optimizations means that there is no clear way to tell what >> optimization level was used to generate the bytecode file. In terms >> of reading PYO files, this can lead to an interpreter using a mixture >> of optimization levels with its code if the user was not careful to >> make sure all PYO files were generated using the same optimization >> level (typically done by blindly deleting all PYO files and then >> using the `compileall` module to compile all-new PYO files [1]_). >> This issue is only compounded when people optimize Python code beyond >> what the interpreter natively supports, e.g., using the astoptimizer >> project [2]_. >> >> In terms of writing PYO files, the need to delete all PYO files >> every time one either changes the optimization level they want to use >> or are unsure of what optimization was used the last time PYO files >> were generated leads to unnecessary file churn. The change proposed >> by this PEP also allows for **all** optimization levels to be >> pre-compiled for bytecode files ahead of time, something that is >> currently impossible thanks to the reuse of the ``.pyo`` file >> extension for multiple optimization levels. >> >> As for distributing bytecode-only modules, having to distribute both >> ``.pyc`` and ``.pyo`` files is unnecessary for the common use-case >> of code obfuscation and smaller file deployments. This means that >> bytecode-only modules will only load from their non-optimized >> ``.pyc`` file name. >> >> >> Proposal >> ======== >> >> To eliminate the ambiguity that PYO files present, this PEP proposes >> eliminating the concept of PYO files and their accompanying ``.pyo`` >> file extension. To allow for the optimization level to be unambiguous >> as well as to avoid having to regenerate optimized bytecode files >> needlessly in the `__pycache__` directory, the optimization level >> used to generate the bytecode file will be incorporated into the >> bytecode file name. When no optimization level is specified, the >> pre-PEP ``.pyc`` file name will be used (i.e., no change in file name >> semantics). This increases backwards-compatibility while also being >> more understanding of Python implementations which have no use for >> optimization levels (e.g., PyPy[10]_). >> >> Currently bytecode file names are created by >> ``importlib.util.cache_from_source()``, approximately using the >> following expression defined by PEP 3147 [3]_, [4]_, [5]_:: >> >> '{name}.{cache_tag}.pyc'.format(name=module_name, >> >> cache_tag=sys.implementation.cache_tag) >> >> This PEP proposes to change the expression when an optimization >> level is specified to:: >> >> '{name}.{cache_tag}.opt-{optimization}.pyc'.format( >> name=module_name, >> cache_tag=sys.implementation.cache_tag, >> optimization=str(sys.flags.optimize)) >> >> The "opt-" prefix was chosen so as to provide a visual separator >> from the cache tag. The placement of the optimization level after >> the cache tag was chosen to preserve lexicographic sort order of >> bytecode file names based on module name and cache tag which will >> not vary for a single interpreter. The "opt-" prefix was chosen over >> "o" so as to be somewhat self-documenting. The "opt-" prefix was >> chosen over "O" so as to not have any confusion in case "0" was the >> leading prefix of the optimization level. >> >> A period was chosen over a hyphen as a separator so as to distinguish >> clearly that the optimization level is not part of the interpreter >> version as specified by the cache tag. It also lends to the use of >> the period in the file name to delineate semantically different >> concepts. >> >> For example, if ``-OO`` had been passed to the interpreter then instead >> of ``importlib.cpython-35.pyo`` the file name would be >> ``importlib.cpython-35.opt-2.pyc``. >> >> It should be noted that this change in no way affects the performance >> of import. Since the import system looks for a single bytecode file >> based on the optimization level of the interpreter already and >> generates a new bytecode file if it doesn't exist, the introduction >> of potentially more bytecode files in the ``__pycache__`` directory >> has no effect in terms of stat calls. The interpreter will continue >> to look for only a single bytecode file based on the optimization >> level and thus no increase in stat calls will occur. >> >> The only potentially negative result of this PEP is the probable >> increase in the number of ``.pyc`` files and thus increase in storage >> use. But for platforms where this is an issue, >> ``sys.dont_write_bytecode`` exists to turn off bytecode generation so >> that it can be controlled offline. >> >> >> Implementation >> ============== >> >> importlib >> --------- >> >> As ``importlib.util.cache_from_source()`` is the API that exposes >> bytecode file paths as well as being directly used by importlib, it >> requires the most critical change. As of Python 3.4, the function's >> signature is:: >> >> importlib.util.cache_from_source(path, debug_override=None) >> >> This PEP proposes changing the signature in Python 3.5 to:: >> >> importlib.util.cache_from_source(path, debug_override=None, *, >> optimization=None) >> >> The introduced ``optimization`` keyword-only parameter will control >> what optimization level is specified in the file name. If the >> argument is ``None`` then the current optimization level of the >> interpreter will be assumed (including no optimization). Any argument >> given for ``optimization`` will be passed to ``str()`` and must have >> ``str.isalnum()`` be true, else ``ValueError`` will be raised (this >> prevents invalid characters being used in the file name). If the >> empty string is passed in for ``optimization`` then the addition of >> the optimization will be suppressed, reverting to the file name >> format which predates this PEP. >> >> It is expected that beyond Python's own two optimization levels, >> third-party code will use a hash of optimization names to specify the >> optimization level, e.g. >> ``hashlib.sha256(','.join(['no dead code', 'const >> folding'])).hexdigest()``. >> While this might lead to long file names, it is assumed that most >> users never look at the contents of the __pycache__ directory and so >> this won't be an issue. >> >> The ``debug_override`` parameter will be deprecated. As the parameter >> expects a boolean, the integer value of the boolean will be used as >> if it had been provided as the argument to ``optimization`` (a >> ``None`` argument will mean the same as for ``optimization``). A >> deprecation warning will be raised when ``debug_override`` is given a >> value other than ``None``, but there are no plans for the complete >> removal of the parameter at this time (but removal will be no later >> than Python 4). >> >> The various module attributes for importlib.machinery which relate to >> bytecode file suffixes will be updated [7]_. The >> ``DEBUG_BYTECODE_SUFFIXES`` and ``OPTIMIZED_BYTECODE_SUFFIXES`` will >> both be documented as deprecated and set to the same value as >> ``BYTECODE_SUFFIXES`` (removal of ``DEBUG_BYTECODE_SUFFIXES`` and >> ``OPTIMIZED_BYTECODE_SUFFIXES`` is not currently planned, but will be >> not later than Python 4). >> >> All various finders and loaders will also be updated as necessary, >> but updating the previous mentioned parts of importlib should be all >> that is required. >> >> >> Rest of the standard library >> ---------------------------- >> >> The various functions exposed by the ``py_compile`` and >> ``compileall`` functions will be updated as necessary to make sure >> they follow the new bytecode file name semantics [6]_, [1]_. The CLI >> for the ``compileall`` module will not be directly affected (the >> ``-b`` flag will be implicit as it will no longer generate ``.pyo`` >> files when ``-O`` is specified). >> >> >> Compatibility Considerations >> ============================ >> >> Any code directly manipulating bytecode files from Python 3.2 on >> will need to consider the impact of this change on their code (prior >> to Python 3.2 -- including all of Python 2 -- there was no >> __pycache__ which already necessitates bifurcating bytecode file >> handling support). If code was setting the ``debug_override`` >> argument to ``importlib.util.cache_from_source()`` then care will be >> needed if they want the path to a bytecode file with an optimization >> level of 2. Otherwise only code **not** using >> ``importlib.util.cache_from_source()`` will need updating. >> >> As for people who distribute bytecode-only modules (i.e., use a >> bytecode file instead of a source file), they will have to choose >> which optimization level they want their bytecode files to be since >> distributing a ``.pyo`` file with a ``.pyc`` file will no longer be >> of any use. Since people typically only distribute bytecode files for >> code obfuscation purposes or smaller distribution size then only >> having to distribute a single ``.pyc`` should actually be beneficial >> to these use-cases. And since the magic number for bytecode files >> changed in Python 3.5 to support PEP 465 there is no need to support >> pre-existing ``.pyo`` files [8]_. >> >> >> Rejected Ideas >> ============== >> >> Completely dropping optimization levels from CPython >> ---------------------------------------------------- >> >> Some have suggested that instead of accommodating the various >> optimization levels in CPython, we should instead drop them >> entirely. The argument is that significant performance gains would >> occur from runtime optimizations through something like a JIT and not >> through pre-execution bytecode optimizations. >> >> This idea is rejected for this PEP as that ignores the fact that >> there are people who do find the pre-existing optimization levels for >> CPython useful. It also assumes that no other Python interpreter >> would find what this PEP proposes useful. >> >> >> Alternative formatting of the optimization level in the file name >> ----------------------------------------------------------------- >> >> Using the "opt-" prefix and placing the optimization level between >> the cache tag and file extension is not critical. All options which >> have been considered are: >> >> * ``importlib.cpython-35.opt-1.pyc`` >> * ``importlib.cpython-35.opt1.pyc`` >> * ``importlib.cpython-35.o1.pyc`` >> * ``importlib.cpython-35.O1.pyc`` >> * ``importlib.cpython-35.1.pyc`` >> * ``importlib.cpython-35-O1.pyc`` >> * ``importlib.O1.cpython-35.pyc`` >> * ``importlib.o1.cpython-35.pyc`` >> * ``importlib.1.cpython-35.pyc`` >> >> These were initially rejected either because they would change the >> sort order of bytecode files, possible ambiguity with the cache tag, >> or were not self-documenting enough. An informal poll was taken and >> people clearly preferred the formatting proposed by the PEP [9]_. >> Since this topic is non-technical and of personal choice, the issue >> is considered solved. >> >> >> Embedding the optimization level in the bytecode metadata >> --------------------------------------------------------- >> >> Some have suggested that rather than embedding the optimization level >> of bytecode in the file name that it be included in the file's >> metadata instead. This would mean every interpreter had a single copy >> of bytecode at any time. Changing the optimization level would thus >> require rewriting the bytecode, but there would also only be a single >> file to care about. >> >> This has been rejected due to the fact that Python is often installed >> as a root-level application and thus modifying the bytecode file for >> modules in the standard library are always possible. In this >> situation integrators would need to guess at what a reasonable >> optimization level was for users for any/all situations. By >> allowing multiple optimization levels to co-exist simultaneously it >> frees integrators from having to guess what users want and allows >> users to utilize the optimization level they want. >> >> >> References >> ========== >> >> .. [1] The compileall module >> (https://docs.python.org/3/library/compileall.html#module-compileall) >> >> .. [2] The astoptimizer project >> (https://pypi.python.org/pypi/astoptimizer) >> >> .. [3] ``importlib.util.cache_from_source()`` >> ( >> https://docs.python.org/3.5/library/importlib.html#importlib.util.cache_from_source >> ) >> >> .. [4] Implementation of ``importlib.util.cache_from_source()`` from >> CPython 3.4.3rc1 >> ( >> https://hg.python.org/cpython/file/038297948389/Lib/importlib/_bootstrap.py#l437 >> ) >> >> .. [5] PEP 3147, PYC Repository Directories, Warsaw >> (http://www.python.org/dev/peps/pep-3147) >> >> .. [6] The py_compile module >> (https://docs.python.org/3/library/compileall.html#module-compileall) >> >> .. [7] The importlib.machinery module >> ( >> https://docs.python.org/3/library/importlib.html#module-importlib.machinery >> ) >> >> .. [8] ``importlib.util.MAGIC_NUMBER`` >> ( >> https://docs.python.org/3/library/importlib.html#importlib.util.MAGIC_NUMBER >> ) >> >> .. [9] Informal poll of file name format options on Google+ >> (https://plus.google.com/u/0/+BrettCannon/posts/fZynLNwHWGm) >> >> .. [10] The PyPy Project >> (http://pypy.org/) >> >> >> Copyright >> ========= >> >> This document has been placed in the public domain. >> >> >> .. >> Local Variables: >> mode: indented-text >> indent-tabs-mode: nil >> sentence-end-double-space: t >> fill-column: 70 >> coding: utf-8 >> End: >> >> >> _______________________________________________ >> Python-Dev mailing list >> Python-Dev at python.org >> https://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe: >> https://mail.python.org/mailman/options/python-dev/guido%40python.org >> >> > > > -- > --Guido van Rossum (python.org/~guido) > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mistersheik at gmail.com Sat Mar 21 00:16:07 2015 From: mistersheik at gmail.com (Neil Girdhar) Date: Fri, 20 Mar 2015 19:16:07 -0400 Subject: [Python-Dev] Missing *-unpacking generalizations (issue 2292) In-Reply-To: <20150319175353.26159.19083@psf.upfronthosting.co.za> References: <20150319175353.26159.19083@psf.upfronthosting.co.za> Message-ID: Wow, this is an excellent review. Thank you. My only question is with respect to this: I think there ought to be two opcodes; one for unpacking maps in function calls and another for literals. The whole function location thing is rather hideous. What are the two opcodes? BUILD_MAP_UNPACK and BUILD_MAP_UNPACK_WITH_CALL? The first takes (n) a number of maps that it will merge, and the second does the same but also accepts (function_call_location) for the purpose of error reporting? Thanks, Neil On Thu, Mar 19, 2015 at 1:53 PM, wrote: > It's a start. > > There need to be documentation updates. > > There are still unrelated style changes in compile.c that should be > reverted. > > > http://bugs.python.org/review/2292/diff/14152/Include/opcode.h > File Include/opcode.h (right): > > http://bugs.python.org/review/2292/diff/14152/Include/opcode.h#newcode71 > Include/opcode.h:71: #define DELETE_GLOBAL 98 > This file should not be manually changed. Rather, > Tools/scripts/generate_opcode_h.py should be run. > > http://bugs.python.org/review/2292/diff/14152/Lib/importlib/_bootstrap.py > File Lib/importlib/_bootstrap.py (right): > > > http://bugs.python.org/review/2292/diff/14152/Lib/importlib/_bootstrap.py#newcode428 > Lib/importlib/_bootstrap.py:428: MAGIC_NUMBER = (3321).to_bytes(2, > 'little') + b'\r\n' > As the comment above indicates, the magic value should be incremented by > 10 not 1. Also, the comment needs to be updated. > > http://bugs.python.org/review/2292/diff/14152/Lib/test/test_ast.py > File Lib/test/test_ast.py (right): > > > http://bugs.python.org/review/2292/diff/14152/Lib/test/test_ast.py#newcode937 > Lib/test/test_ast.py:937: main() > Why is this here? > > http://bugs.python.org/review/2292/diff/14152/Lib/test/test_parser.py > File Lib/test/test_parser.py (right): > > > http://bugs.python.org/review/2292/diff/14152/Lib/test/test_parser.py#newcode334 > Lib/test/test_parser.py:334: self.check_expr('{**{}, 3:4, **{5:6, > 7:8}}') > Should there be tests for the new function call syntax, too? > > http://bugs.python.org/review/2292/diff/14152/Python/ast.c > File Python/ast.c (right): > > http://bugs.python.org/review/2292/diff/14152/Python/ast.c#newcode1746 > Python/ast.c:1746: ast_error(c, n, "iterable unpacking cannot be used in > comprehension"); > |n| should probably be |ch| > > http://bugs.python.org/review/2292/diff/14152/Python/ast.c#newcode2022 > Python/ast.c:2022: if (TYPE(CHILD(ch, 0)) == DOUBLESTAR) > int is_dict = TYPE(CHILD(ch, 0)) == DOUBLESTAR; > > http://bugs.python.org/review/2292/diff/14152/Python/ast.c#newcode2026 > Python/ast.c:2026: && TYPE(CHILD(ch, 1)) == COMMA)) { > boolean operators should be on the previous line > > http://bugs.python.org/review/2292/diff/14152/Python/ast.c#newcode2031 > Python/ast.c:2031: && TYPE(CHILD(ch, 1)) == comp_for) { > ditto > > http://bugs.python.org/review/2292/diff/14152/Python/ast.c#newcode2036 > Python/ast.c:2036: && TYPE(CHILD(ch, 3 - is_dict)) == comp_for) { > ditto > > http://bugs.python.org/review/2292/diff/14152/Python/ceval.c > File Python/ceval.c (right): > > http://bugs.python.org/review/2292/diff/14152/Python/ceval.c#newcode2403 > Python/ceval.c:2403: as_tuple = PyObject_CallFunctionObjArgs( > Use PyList_AsTuple. > > http://bugs.python.org/review/2292/diff/14152/Python/ceval.c#newcode2498 > Python/ceval.c:2498: TARGET(BUILD_MAP_UNPACK) { > I think there ought to be two opcodes; one for unpacking maps in > function calls and another for literals. The whole function location > thing is rather hideous. > > http://bugs.python.org/review/2292/diff/14152/Python/ceval.c#newcode2526 > Python/ceval.c:2526: if (PySet_Size(intersection)) { > Probably this would be faster with PySet_GET_SIZE(so). > > http://bugs.python.org/review/2292/diff/14152/Python/compile.c > File Python/compile.c (right): > > http://bugs.python.org/review/2292/diff/14152/Python/compile.c#newcode2931 > Python/compile.c:2931: asdl_seq_SET(elts, i, elt->v.Starred.value); > The compiler should not be mutating the AST. > > http://bugs.python.org/review/2292/diff/14152/Python/compile.c#newcode3088 > Python/compile.c:3088: int i, nseen, nkw = 0; > Many of these should probably be Py_ssize_t. > > http://bugs.python.org/review/2292/diff/14152/Python/symtable.c > File Python/symtable.c (right): > > http://bugs.python.org/review/2292/diff/14152/Python/symtable.c#newcode1372 > Python/symtable.c:1372: if (e != NULL) { > Please fix the callers, so they don't pass NULL here. > > http://bugs.python.org/review/2292/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Sat Mar 21 01:06:03 2015 From: guido at python.org (Guido van Rossum) Date: Fri, 20 Mar 2015 17:06:03 -0700 Subject: [Python-Dev] Final call for PEP 488: eliminating PYO files In-Reply-To: References: Message-ID: Awesome, that's what I was hoping. Accepted! Congrats and thank you very much for writing the PEP and guiding the discussion. On Fri, Mar 20, 2015 at 4:00 PM, Brett Cannon wrote: > > > On Fri, Mar 20, 2015 at 4:41 PM Guido van Rossum wrote: > >> I am willing to be the BDFL for this PEP. I have tried to skim the recent >> discussion (only python-dev) and I don't see much remaining controversy. >> HOWEVER... The PEP is not clear (or at least too subtle) about the actual >> name for optimization level 0. If I have foo.py, and I compile it three >> times with three different optimization levels (no optimization; -O; -OO), >> and then I look in __pycache__, would I see this: >> >> # (1) >> foo.cpython-35.pyc >> foo.cpython-35.opt-1.pyc >> foo.cpython-35.opt-2.pyc >> >> Or would I see this? >> >> # (2) >> foo.cpython-35.opt-0.pyc >> foo.cpython-35.opt-1.pyc >> foo.cpython-35.opt-2.pyc >> > > #1 > > >> >> Your lead-in ("I have decided to have the default case of no optimization >> levels mean that the .pyc file name will have *no* optimization level >> specified in the name and thus be just as it is today.") makes me think I >> should expect (1), but I can't actually pinpoint where the language of the >> PEP says this. >> > > It was meant to be explained by "When no optimization level is specified, > the pre-PEP ``.pyc`` file name will be used (i.e., no change in file name > semantics)", but obviously it's a bit too subtle. I just updated the PEP > with an explicit list of bytecode file name examples based on no -O, -O, > and -OO. > > -Brett > > >> >> >> On Fri, Mar 20, 2015 at 11:34 AM, Brett Cannon wrote: >> >>> I have decided to have the default case of no optimization levels mean >>> that the .pyc file name will have *no* optimization level specified in >>> the name and thus be just as it is today. I made this decision due to >>> potential backwards-compatibility issues -- although I expect them to be >>> minutes -- and to not force other implementations like PyPy to have some >>> bogus value set since they don't have .pyo files to begin with (PyPy >>> actually uses bytecode for -O and don't bother with -OO since PyPy already >>> uses a bunch of memory when running). >>> >>> Since this closes out the last open issue, I need either a BDFL decision >>> or a BDFAP to be assigned to make a decision. Guido? >>> >>> ====================================== >>> >>> PEP: 488 >>> Title: Elimination of PYO files >>> Version: $Revision$ >>> Last-Modified: $Date$ >>> Author: Brett Cannon >>> Status: Draft >>> Type: Standards Track >>> Content-Type: text/x-rst >>> Created: 20-Feb-2015 >>> Post-History: >>> 2015-03-06 >>> 2015-03-13 >>> 2015-03-20 >>> >>> Abstract >>> ======== >>> >>> This PEP proposes eliminating the concept of PYO files from Python. >>> To continue the support of the separation of bytecode files based on >>> their optimization level, this PEP proposes extending the PYC file >>> name to include the optimization level in the bytecode repository >>> directory when it's called for (i.e., the ``__pycache__`` directory). >>> >>> >>> Rationale >>> ========= >>> >>> As of today, bytecode files come in two flavours: PYC and PYO. A PYC >>> file is the bytecode file generated and read from when no >>> optimization level is specified at interpreter startup (i.e., ``-O`` >>> is not specified). A PYO file represents the bytecode file that is >>> read/written when **any** optimization level is specified (i.e., when >>> ``-O`` **or** ``-OO`` is specified). This means that while PYC >>> files clearly delineate the optimization level used when they were >>> generated -- namely no optimizations beyond the peepholer -- the same >>> is not true for PYO files. To put this in terms of optimization >>> levels and the file extension: >>> >>> - 0: ``.pyc`` >>> - 1 (``-O``): ``.pyo`` >>> - 2 (``-OO``): ``.pyo`` >>> >>> The reuse of the ``.pyo`` file extension for both level 1 and 2 >>> optimizations means that there is no clear way to tell what >>> optimization level was used to generate the bytecode file. In terms >>> of reading PYO files, this can lead to an interpreter using a mixture >>> of optimization levels with its code if the user was not careful to >>> make sure all PYO files were generated using the same optimization >>> level (typically done by blindly deleting all PYO files and then >>> using the `compileall` module to compile all-new PYO files [1]_). >>> This issue is only compounded when people optimize Python code beyond >>> what the interpreter natively supports, e.g., using the astoptimizer >>> project [2]_. >>> >>> In terms of writing PYO files, the need to delete all PYO files >>> every time one either changes the optimization level they want to use >>> or are unsure of what optimization was used the last time PYO files >>> were generated leads to unnecessary file churn. The change proposed >>> by this PEP also allows for **all** optimization levels to be >>> pre-compiled for bytecode files ahead of time, something that is >>> currently impossible thanks to the reuse of the ``.pyo`` file >>> extension for multiple optimization levels. >>> >>> As for distributing bytecode-only modules, having to distribute both >>> ``.pyc`` and ``.pyo`` files is unnecessary for the common use-case >>> of code obfuscation and smaller file deployments. This means that >>> bytecode-only modules will only load from their non-optimized >>> ``.pyc`` file name. >>> >>> >>> Proposal >>> ======== >>> >>> To eliminate the ambiguity that PYO files present, this PEP proposes >>> eliminating the concept of PYO files and their accompanying ``.pyo`` >>> file extension. To allow for the optimization level to be unambiguous >>> as well as to avoid having to regenerate optimized bytecode files >>> needlessly in the `__pycache__` directory, the optimization level >>> used to generate the bytecode file will be incorporated into the >>> bytecode file name. When no optimization level is specified, the >>> pre-PEP ``.pyc`` file name will be used (i.e., no change in file name >>> semantics). This increases backwards-compatibility while also being >>> more understanding of Python implementations which have no use for >>> optimization levels (e.g., PyPy[10]_). >>> >>> Currently bytecode file names are created by >>> ``importlib.util.cache_from_source()``, approximately using the >>> following expression defined by PEP 3147 [3]_, [4]_, [5]_:: >>> >>> '{name}.{cache_tag}.pyc'.format(name=module_name, >>> >>> cache_tag=sys.implementation.cache_tag) >>> >>> This PEP proposes to change the expression when an optimization >>> level is specified to:: >>> >>> '{name}.{cache_tag}.opt-{optimization}.pyc'.format( >>> name=module_name, >>> cache_tag=sys.implementation.cache_tag, >>> optimization=str(sys.flags.optimize)) >>> >>> The "opt-" prefix was chosen so as to provide a visual separator >>> from the cache tag. The placement of the optimization level after >>> the cache tag was chosen to preserve lexicographic sort order of >>> bytecode file names based on module name and cache tag which will >>> not vary for a single interpreter. The "opt-" prefix was chosen over >>> "o" so as to be somewhat self-documenting. The "opt-" prefix was >>> chosen over "O" so as to not have any confusion in case "0" was the >>> leading prefix of the optimization level. >>> >>> A period was chosen over a hyphen as a separator so as to distinguish >>> clearly that the optimization level is not part of the interpreter >>> version as specified by the cache tag. It also lends to the use of >>> the period in the file name to delineate semantically different >>> concepts. >>> >>> For example, if ``-OO`` had been passed to the interpreter then instead >>> of ``importlib.cpython-35.pyo`` the file name would be >>> ``importlib.cpython-35.opt-2.pyc``. >>> >>> It should be noted that this change in no way affects the performance >>> of import. Since the import system looks for a single bytecode file >>> based on the optimization level of the interpreter already and >>> generates a new bytecode file if it doesn't exist, the introduction >>> of potentially more bytecode files in the ``__pycache__`` directory >>> has no effect in terms of stat calls. The interpreter will continue >>> to look for only a single bytecode file based on the optimization >>> level and thus no increase in stat calls will occur. >>> >>> The only potentially negative result of this PEP is the probable >>> increase in the number of ``.pyc`` files and thus increase in storage >>> use. But for platforms where this is an issue, >>> ``sys.dont_write_bytecode`` exists to turn off bytecode generation so >>> that it can be controlled offline. >>> >>> >>> Implementation >>> ============== >>> >>> importlib >>> --------- >>> >>> As ``importlib.util.cache_from_source()`` is the API that exposes >>> bytecode file paths as well as being directly used by importlib, it >>> requires the most critical change. As of Python 3.4, the function's >>> signature is:: >>> >>> importlib.util.cache_from_source(path, debug_override=None) >>> >>> This PEP proposes changing the signature in Python 3.5 to:: >>> >>> importlib.util.cache_from_source(path, debug_override=None, *, >>> optimization=None) >>> >>> The introduced ``optimization`` keyword-only parameter will control >>> what optimization level is specified in the file name. If the >>> argument is ``None`` then the current optimization level of the >>> interpreter will be assumed (including no optimization). Any argument >>> given for ``optimization`` will be passed to ``str()`` and must have >>> ``str.isalnum()`` be true, else ``ValueError`` will be raised (this >>> prevents invalid characters being used in the file name). If the >>> empty string is passed in for ``optimization`` then the addition of >>> the optimization will be suppressed, reverting to the file name >>> format which predates this PEP. >>> >>> It is expected that beyond Python's own two optimization levels, >>> third-party code will use a hash of optimization names to specify the >>> optimization level, e.g. >>> ``hashlib.sha256(','.join(['no dead code', 'const >>> folding'])).hexdigest()``. >>> While this might lead to long file names, it is assumed that most >>> users never look at the contents of the __pycache__ directory and so >>> this won't be an issue. >>> >>> The ``debug_override`` parameter will be deprecated. As the parameter >>> expects a boolean, the integer value of the boolean will be used as >>> if it had been provided as the argument to ``optimization`` (a >>> ``None`` argument will mean the same as for ``optimization``). A >>> deprecation warning will be raised when ``debug_override`` is given a >>> value other than ``None``, but there are no plans for the complete >>> removal of the parameter at this time (but removal will be no later >>> than Python 4). >>> >>> The various module attributes for importlib.machinery which relate to >>> bytecode file suffixes will be updated [7]_. The >>> ``DEBUG_BYTECODE_SUFFIXES`` and ``OPTIMIZED_BYTECODE_SUFFIXES`` will >>> both be documented as deprecated and set to the same value as >>> ``BYTECODE_SUFFIXES`` (removal of ``DEBUG_BYTECODE_SUFFIXES`` and >>> ``OPTIMIZED_BYTECODE_SUFFIXES`` is not currently planned, but will be >>> not later than Python 4). >>> >>> All various finders and loaders will also be updated as necessary, >>> but updating the previous mentioned parts of importlib should be all >>> that is required. >>> >>> >>> Rest of the standard library >>> ---------------------------- >>> >>> The various functions exposed by the ``py_compile`` and >>> ``compileall`` functions will be updated as necessary to make sure >>> they follow the new bytecode file name semantics [6]_, [1]_. The CLI >>> for the ``compileall`` module will not be directly affected (the >>> ``-b`` flag will be implicit as it will no longer generate ``.pyo`` >>> files when ``-O`` is specified). >>> >>> >>> Compatibility Considerations >>> ============================ >>> >>> Any code directly manipulating bytecode files from Python 3.2 on >>> will need to consider the impact of this change on their code (prior >>> to Python 3.2 -- including all of Python 2 -- there was no >>> __pycache__ which already necessitates bifurcating bytecode file >>> handling support). If code was setting the ``debug_override`` >>> argument to ``importlib.util.cache_from_source()`` then care will be >>> needed if they want the path to a bytecode file with an optimization >>> level of 2. Otherwise only code **not** using >>> ``importlib.util.cache_from_source()`` will need updating. >>> >>> As for people who distribute bytecode-only modules (i.e., use a >>> bytecode file instead of a source file), they will have to choose >>> which optimization level they want their bytecode files to be since >>> distributing a ``.pyo`` file with a ``.pyc`` file will no longer be >>> of any use. Since people typically only distribute bytecode files for >>> code obfuscation purposes or smaller distribution size then only >>> having to distribute a single ``.pyc`` should actually be beneficial >>> to these use-cases. And since the magic number for bytecode files >>> changed in Python 3.5 to support PEP 465 there is no need to support >>> pre-existing ``.pyo`` files [8]_. >>> >>> >>> Rejected Ideas >>> ============== >>> >>> Completely dropping optimization levels from CPython >>> ---------------------------------------------------- >>> >>> Some have suggested that instead of accommodating the various >>> optimization levels in CPython, we should instead drop them >>> entirely. The argument is that significant performance gains would >>> occur from runtime optimizations through something like a JIT and not >>> through pre-execution bytecode optimizations. >>> >>> This idea is rejected for this PEP as that ignores the fact that >>> there are people who do find the pre-existing optimization levels for >>> CPython useful. It also assumes that no other Python interpreter >>> would find what this PEP proposes useful. >>> >>> >>> Alternative formatting of the optimization level in the file name >>> ----------------------------------------------------------------- >>> >>> Using the "opt-" prefix and placing the optimization level between >>> the cache tag and file extension is not critical. All options which >>> have been considered are: >>> >>> * ``importlib.cpython-35.opt-1.pyc`` >>> * ``importlib.cpython-35.opt1.pyc`` >>> * ``importlib.cpython-35.o1.pyc`` >>> * ``importlib.cpython-35.O1.pyc`` >>> * ``importlib.cpython-35.1.pyc`` >>> * ``importlib.cpython-35-O1.pyc`` >>> * ``importlib.O1.cpython-35.pyc`` >>> * ``importlib.o1.cpython-35.pyc`` >>> * ``importlib.1.cpython-35.pyc`` >>> >>> These were initially rejected either because they would change the >>> sort order of bytecode files, possible ambiguity with the cache tag, >>> or were not self-documenting enough. An informal poll was taken and >>> people clearly preferred the formatting proposed by the PEP [9]_. >>> Since this topic is non-technical and of personal choice, the issue >>> is considered solved. >>> >>> >>> Embedding the optimization level in the bytecode metadata >>> --------------------------------------------------------- >>> >>> Some have suggested that rather than embedding the optimization level >>> of bytecode in the file name that it be included in the file's >>> metadata instead. This would mean every interpreter had a single copy >>> of bytecode at any time. Changing the optimization level would thus >>> require rewriting the bytecode, but there would also only be a single >>> file to care about. >>> >>> This has been rejected due to the fact that Python is often installed >>> as a root-level application and thus modifying the bytecode file for >>> modules in the standard library are always possible. In this >>> situation integrators would need to guess at what a reasonable >>> optimization level was for users for any/all situations. By >>> allowing multiple optimization levels to co-exist simultaneously it >>> frees integrators from having to guess what users want and allows >>> users to utilize the optimization level they want. >>> >>> >>> References >>> ========== >>> >>> .. [1] The compileall module >>> (https://docs.python.org/3/library/compileall.html#module-compileall) >>> >>> .. [2] The astoptimizer project >>> (https://pypi.python.org/pypi/astoptimizer) >>> >>> .. [3] ``importlib.util.cache_from_source()`` >>> ( >>> https://docs.python.org/3.5/library/importlib.html#importlib.util.cache_from_source >>> ) >>> >>> .. [4] Implementation of ``importlib.util.cache_from_source()`` from >>> CPython 3.4.3rc1 >>> ( >>> https://hg.python.org/cpython/file/038297948389/Lib/importlib/_bootstrap.py#l437 >>> ) >>> >>> .. [5] PEP 3147, PYC Repository Directories, Warsaw >>> (http://www.python.org/dev/peps/pep-3147) >>> >>> .. [6] The py_compile module >>> (https://docs.python.org/3/library/compileall.html#module-compileall) >>> >>> .. [7] The importlib.machinery module >>> ( >>> https://docs.python.org/3/library/importlib.html#module-importlib.machinery >>> ) >>> >>> .. [8] ``importlib.util.MAGIC_NUMBER`` >>> ( >>> https://docs.python.org/3/library/importlib.html#importlib.util.MAGIC_NUMBER >>> ) >>> >>> .. [9] Informal poll of file name format options on Google+ >>> (https://plus.google.com/u/0/+BrettCannon/posts/fZynLNwHWGm) >>> >>> .. [10] The PyPy Project >>> (http://pypy.org/) >>> >>> >>> Copyright >>> ========= >>> >>> This document has been placed in the public domain. >>> >>> >>> .. >>> Local Variables: >>> mode: indented-text >>> indent-tabs-mode: nil >>> sentence-end-double-space: t >>> fill-column: 70 >>> coding: utf-8 >>> End: >>> >>> >>> _______________________________________________ >>> Python-Dev mailing list >>> Python-Dev at python.org >>> https://mail.python.org/mailman/listinfo/python-dev >>> Unsubscribe: >>> https://mail.python.org/mailman/options/python-dev/guido%40python.org >>> >>> >> >> >> -- >> --Guido van Rossum (python.org/~guido) >> > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From luoyonggang at gmail.com Sat Mar 21 01:15:29 2015 From: luoyonggang at gmail.com (=?UTF-8?B?572X5YuH5YiaKFlvbmdnYW5nIEx1bykg?=) Date: Sat, 21 Mar 2015 08:15:29 +0800 Subject: [Python-Dev] Fwd: class os.DirEntry is confusing, In-Reply-To: References: Message-ID: ---------- Forwarded message ---------- From: ???(Yonggang Luo) Date: 2015-03-21 2:24 GMT+08:00 Subject: class os.DirEntry is confusing, To: Python Should be replaced with os.FsEntry or os.ScanEntry -- ?? ? ??? Yours sincerely, Yonggang Luo -- ?? ? ??? Yours sincerely, Yonggang Luo From mistersheik at gmail.com Sat Mar 21 01:54:39 2015 From: mistersheik at gmail.com (Neil Girdhar) Date: Fri, 20 Mar 2015 20:54:39 -0400 Subject: [Python-Dev] pep 7 Message-ID: If ever someone wants to clean up the repository to conform to PEP 7, I wrote a program that catches a couple hundred PEP 7 violations in ./Python alone (1400 in the whole codebase): import os import re def grep(path, regex): reg_obj = re.compile(regex, re.M) res = [] for root, dirs, fnames in os.walk(path): for fname in fnames: if fname.endswith('.c'): path = os.path.join(root, fname) with open(path) as f: data = f.read() for m in reg_obj.finditer(data): line_number = sum(c == '\n' for c in data[:m.start()]) + 1 res.append("{}: {}".format(path, line_number)) return res for pattern in [ r'^\s*\|\|', r'^\s*\&\&', r'} else {', r'\ From lewisc at pdx.edu Sat Mar 21 02:30:06 2015 From: lewisc at pdx.edu (Lewis Coates) Date: Fri, 20 Mar 2015 18:30:06 -0700 Subject: [Python-Dev] some minor questions about pep8 Message-ID: In pep8 there are two conflicting statements, both https://www.python.org/dev/peps/pep-0008/#version-bookkeeping https://www.python.org/dev/peps/pep-0008/#imports Stipulate that they should be "at the top of the file after any module comments and docstrings." Which of these takes precedence? Secondly, we also have an "__author__", and "__project__" variables, I assume these would be put with the version information as well? -------------- next part -------------- An HTML attachment was scrubbed... URL: From ben+python at benfinney.id.au Sat Mar 21 02:38:55 2015 From: ben+python at benfinney.id.au (Ben Finney) Date: Sat, 21 Mar 2015 12:38:55 +1100 Subject: [Python-Dev] some minor questions about pep8 References: Message-ID: <85lhir6seo.fsf@benfinney.id.au> Lewis Coates writes: > In pep8 there are two conflicting statements, both > > https://www.python.org/dev/peps/pep-0008/#version-bookkeeping > https://www.python.org/dev/peps/pep-0008/#imports > > Stipulate that they should be "at the top of the file after any module > comments and docstrings." Which of these takes precedence? I don't know an official answer. The convention I've observed is overwhelmingly in one direction: import statements come before any assignment statements. > Secondly, we also have an "__author__", and "__project__" variables, I > assume these would be put with the version information as well? Yes. -- \ ?Welchen Teil von ?Gestalt? verstehen Sie nicht? [What part of | `\ ?gestalt? don't you understand?]? ?Karsten M. Self | _o__) | Ben Finney From brian at python.org Sat Mar 21 03:35:47 2015 From: brian at python.org (Brian Curtin) Date: Fri, 20 Mar 2015 21:35:47 -0500 Subject: [Python-Dev] pep 7 In-Reply-To: References: Message-ID: On Fri, Mar 20, 2015 at 7:54 PM, Neil Girdhar wrote: > If ever someone wants to clean up the repository to conform to PEP 7, I > wrote a program that catches a couple hundred PEP 7 violations in ./Python > alone (1400 in the whole codebase): > > import os > import re > > def grep(path, regex): > reg_obj = re.compile(regex, re.M) > res = [] > for root, dirs, fnames in os.walk(path): > for fname in fnames: > if fname.endswith('.c'): > path = os.path.join(root, fname) > with open(path) as f: > data = f.read() > for m in reg_obj.finditer(data): > line_number = sum(c == '\n' > for c in data[:m.start()]) + 1 > res.append("{}: {}".format(path, line_number)) > return res > > for pattern in [ > r'^\s*\|\|', > r'^\s*\&\&', > r'} else {', > r'\ ]: > print("Searching for", pattern) > print("\n".join(grep('.', pattern))) > > In my experience, it was hard to write PEP 7 conforming code when the > surrounding code is inconsistent. You can usually change surrounding code within reason if you want to add conforming code of your own, but there's little value and high risk in any mass change just to apply the style guidelines. From mistersheik at gmail.com Sat Mar 21 04:02:34 2015 From: mistersheik at gmail.com (Neil Girdhar) Date: Fri, 20 Mar 2015 23:02:34 -0400 Subject: [Python-Dev] pep 7 In-Reply-To: References: Message-ID: The code reviews I got asked me to revert PEP 7 changes. I can understand that, but then logically someone should go ahead and clean up the code. It's not "high risk" if you just check for whitespace equivalence of the source code and binary equivalence of the compiled code. The value is for people who are new to the codebase. Best, Neil On Fri, Mar 20, 2015 at 10:35 PM, Brian Curtin wrote: > On Fri, Mar 20, 2015 at 7:54 PM, Neil Girdhar > wrote: > > If ever someone wants to clean up the repository to conform to PEP 7, I > > wrote a program that catches a couple hundred PEP 7 violations in > ./Python > > alone (1400 in the whole codebase): > > > > import os > > import re > > > > def grep(path, regex): > > reg_obj = re.compile(regex, re.M) > > res = [] > > for root, dirs, fnames in os.walk(path): > > for fname in fnames: > > if fname.endswith('.c'): > > path = os.path.join(root, fname) > > with open(path) as f: > > data = f.read() > > for m in reg_obj.finditer(data): > > line_number = sum(c == '\n' > > for c in data[:m.start()]) + 1 > > res.append("{}: {}".format(path, line_number)) > > return res > > > > for pattern in [ > > r'^\s*\|\|', > > r'^\s*\&\&', > > r'} else {', > > r'\ > ]: > > print("Searching for", pattern) > > print("\n".join(grep('.', pattern))) > > > > In my experience, it was hard to write PEP 7 conforming code when the > > surrounding code is inconsistent. > > You can usually change surrounding code within reason if you want to > add conforming code of your own, but there's little value and high > risk in any mass change just to apply the style guidelines. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From brian at python.org Sat Mar 21 04:35:49 2015 From: brian at python.org (Brian Curtin) Date: Fri, 20 Mar 2015 22:35:49 -0500 Subject: [Python-Dev] pep 7 In-Reply-To: References: Message-ID: On Fri, Mar 20, 2015 at 10:02 PM, Neil Girdhar wrote: > The code reviews I got asked me to revert PEP 7 changes. I can understand > that, but then logically someone should go ahead and clean up the code. > It's not "high risk" if you just check for whitespace equivalence of the > source code and binary equivalence of the compiled code. The value is for > people who are new to the codebase. There are a lot of areas of the C code that aren't explicitly or directly tested, so yes, a lot of changes are high risk, especially in bulk. While a one time change while checking binary equivalence would do it, it's also a huge amount of churn just to follow a guideline. Without an automated checker for the guidelines, if things get in they just get in, and sometimes you can modify them while making improvements to the code, but sometimes it depends on what exactly you're doing as well. On top of this, we already disallow mass PEP 8 changes to avoid the churn there as well, and it took a good bit of convincing for another semi-recent mass change (although I can't remember the topic, but it was deemed safe enough to make). Another common issue with mass code churn like this is that it affects tooling, such as `hg blame` From guido at python.org Sat Mar 21 04:53:10 2015 From: guido at python.org (Guido van Rossum) Date: Fri, 20 Mar 2015 20:53:10 -0700 Subject: [Python-Dev] some minor questions about pep8 In-Reply-To: <85lhir6seo.fsf@benfinney.id.au> References: <85lhir6seo.fsf@benfinney.id.au> Message-ID: FWIW, I think __version__, __author__ etc. were bad ideas. Almost nobody manages these correctly. Note that the PEP 8 section starts with less than an endorsement: "If you *have* to have Subversion, CVS, or RCS crud in your source file, do it as follows." That said, if an official answer is required, common sense would suggest that __version__ should go before the imports. (I would put it before the docstring too, except then the docstring wouldn't be a docstring any more. Go figure.) On Fri, Mar 20, 2015 at 6:38 PM, Ben Finney wrote: > Lewis Coates writes: > > > In pep8 there are two conflicting statements, both > > > > https://www.python.org/dev/peps/pep-0008/#version-bookkeeping > > https://www.python.org/dev/peps/pep-0008/#imports > > > > Stipulate that they should be "at the top of the file after any module > > comments and docstrings." Which of these takes precedence? > > I don't know an official answer. The convention I've observed is > overwhelmingly in one direction: import statements come before any > assignment statements. > > > Secondly, we also have an "__author__", and "__project__" variables, I > > assume these would be put with the version information as well? > > Yes. > > -- > \ ?Welchen Teil von ?Gestalt? verstehen Sie nicht? [What part of | > `\ ?gestalt? don't you understand?]? ?Karsten M. Self | > _o__) | > Ben Finney > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/guido%40python.org > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Sat Mar 21 04:57:02 2015 From: guido at python.org (Guido van Rossum) Date: Fri, 20 Mar 2015 20:57:02 -0700 Subject: [Python-Dev] pep 7 In-Reply-To: References: Message-ID: Neil, you have no idea. Please back off. On Fri, Mar 20, 2015 at 8:02 PM, Neil Girdhar wrote: > The code reviews I got asked me to revert PEP 7 changes. I can understand > that, but then logically someone should go ahead and clean up the code. > It's not "high risk" if you just check for whitespace equivalence of the > source code and binary equivalence of the compiled code. The value is for > people who are new to the codebase. > > Best, > > Neil > > On Fri, Mar 20, 2015 at 10:35 PM, Brian Curtin wrote: > >> On Fri, Mar 20, 2015 at 7:54 PM, Neil Girdhar >> wrote: >> > If ever someone wants to clean up the repository to conform to PEP 7, I >> > wrote a program that catches a couple hundred PEP 7 violations in >> ./Python >> > alone (1400 in the whole codebase): >> > >> > import os >> > import re >> > >> > def grep(path, regex): >> > reg_obj = re.compile(regex, re.M) >> > res = [] >> > for root, dirs, fnames in os.walk(path): >> > for fname in fnames: >> > if fname.endswith('.c'): >> > path = os.path.join(root, fname) >> > with open(path) as f: >> > data = f.read() >> > for m in reg_obj.finditer(data): >> > line_number = sum(c == '\n' >> > for c in data[:m.start()]) + 1 >> > res.append("{}: {}".format(path, line_number)) >> > return res >> > >> > for pattern in [ >> > r'^\s*\|\|', >> > r'^\s*\&\&', >> > r'} else {', >> > r'\> > ]: >> > print("Searching for", pattern) >> > print("\n".join(grep('.', pattern))) >> > >> > In my experience, it was hard to write PEP 7 conforming code when the >> > surrounding code is inconsistent. >> >> You can usually change surrounding code within reason if you want to >> add conforming code of your own, but there's little value and high >> risk in any mass change just to apply the style guidelines. >> > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/guido%40python.org > > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From ianlee1521 at gmail.com Sat Mar 21 05:24:28 2015 From: ianlee1521 at gmail.com (Ian Lee) Date: Fri, 20 Mar 2015 21:24:28 -0700 Subject: [Python-Dev] some minor questions about pep8 In-Reply-To: References: <85lhir6seo.fsf@benfinney.id.au> Message-ID: Guido, In that case would you be open to a patch to update the PEP accordingly? Additionally, does that official statement cover other dunder assignments (e.g. "__author__"?). If so I'll update the PEP8 tool accordingly. Thanks, ~ Ian Lee On Mar 20, 2015 8:55 PM, "Guido van Rossum" wrote: > FWIW, I think __version__, __author__ etc. were bad ideas. Almost nobody > manages these correctly. Note that the PEP 8 section starts with less than > an endorsement: "If you *have* to have Subversion, CVS, or RCS crud in > your source file, do it as follows." > > That said, if an official answer is required, common sense would suggest > that __version__ should go before the imports. (I would put it before the > docstring too, except then the docstring wouldn't be a docstring any more. > Go figure.) > > On Fri, Mar 20, 2015 at 6:38 PM, Ben Finney > wrote: > >> Lewis Coates writes: >> >> > In pep8 there are two conflicting statements, both >> > >> > https://www.python.org/dev/peps/pep-0008/#version-bookkeeping >> > https://www.python.org/dev/peps/pep-0008/#imports >> > >> > Stipulate that they should be "at the top of the file after any module >> > comments and docstrings." Which of these takes precedence? >> >> I don't know an official answer. The convention I've observed is >> overwhelmingly in one direction: import statements come before any >> assignment statements. >> >> > Secondly, we also have an "__author__", and "__project__" variables, I >> > assume these would be put with the version information as well? >> >> Yes. >> >> -- >> \ ?Welchen Teil von ?Gestalt? verstehen Sie nicht? [What part of | >> `\ ?gestalt? don't you understand?]? ?Karsten M. Self | >> _o__) | >> Ben Finney >> >> _______________________________________________ >> Python-Dev mailing list >> Python-Dev at python.org >> https://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe: >> https://mail.python.org/mailman/options/python-dev/guido%40python.org >> > > > > -- > --Guido van Rossum (python.org/~guido) > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/ianlee1521%40gmail.com > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From brian at python.org Sat Mar 21 05:29:51 2015 From: brian at python.org (Brian Curtin) Date: Fri, 20 Mar 2015 23:29:51 -0500 Subject: [Python-Dev] pep 7 In-Reply-To: References: Message-ID: On Fri, Mar 20, 2015 at 10:57 PM, Guido van Rossum wrote: > Neil, you have no idea. Please back off. I wouldn't go that far. Wanting a quality code base certainly isn't a bad thing, but there's a lot more progress to be made by working with what's there and being as mindful as possible of the guidelines moving forward. After all, they're guidelines, not rules. From storchaka at gmail.com Sat Mar 21 05:41:18 2015 From: storchaka at gmail.com (Serhiy Storchaka) Date: Sat, 21 Mar 2015 06:41:18 +0200 Subject: [Python-Dev] How to document functions with optional positional parameters? Message-ID: How to document functions with optional positional parameters? For example binascii.crc32(). It has two positional parameters, one is mandatory, and one is optional with default value 0. With Argument Clinic its signature is crc32(data, crc=0, /). In the documentation it is written as crc32(data[, crc]) (and noted that the default value of the second parameter is 0). Both are not valid Python syntax. Can the documentation be change to crc32(data, crc=0)? Related issues: https://bugs.python.org/issue21488 (changed encode(obj, encoding='ascii', errors='strict') to encode(obj, [encoding[, errors]])) https://bugs.python.org/issue22832 (changed ioctl(fd, op[, arg[, mutate_flag]]) to ioctl(fd, request, arg=0, mutate_flag=True)) https://bugs.python.org/issue22341 (discussed changing crc32(data[, crc]) to crc32(data, crc=0)) From ianlee1521 at gmail.com Sat Mar 21 05:59:27 2015 From: ianlee1521 at gmail.com (Ian Lee) Date: Fri, 20 Mar 2015 21:59:27 -0700 Subject: [Python-Dev] some minor questions about pep8 In-Reply-To: References: <85lhir6seo.fsf@benfinney.id.au> Message-ID: FWIW, I would vote for the "__version__", "__author__", etc assignments being after the imports. Reason being cases where the "__version__" is not from VCS, but is calculated from pkg_resources: from pkg_resources import get_distribution __version__ = get_distribution('mypackage').version Also, then more useful things like "__all__" (which can very reasonably rely on imports), can be together with "__version__" and "__author__" assignments. I would vote: shebang docstring imports dunder assignments other code... ~ Ian Lee On Fri, Mar 20, 2015 at 9:24 PM, Ian Lee wrote: > Guido, > > In that case would you be open to a patch to update the PEP accordingly? > > Additionally, does that official statement cover other dunder assignments > (e.g. "__author__"?). If so I'll update the PEP8 tool accordingly. > > Thanks, > > ~ Ian Lee > On Mar 20, 2015 8:55 PM, "Guido van Rossum" wrote: > >> FWIW, I think __version__, __author__ etc. were bad ideas. Almost nobody >> manages these correctly. Note that the PEP 8 section starts with less than >> an endorsement: "If you *have* to have Subversion, CVS, or RCS crud in >> your source file, do it as follows." >> >> That said, if an official answer is required, common sense would suggest >> that __version__ should go before the imports. (I would put it before the >> docstring too, except then the docstring wouldn't be a docstring any more. >> Go figure.) >> >> On Fri, Mar 20, 2015 at 6:38 PM, Ben Finney >> wrote: >> >>> Lewis Coates writes: >>> >>> > In pep8 there are two conflicting statements, both >>> > >>> > https://www.python.org/dev/peps/pep-0008/#version-bookkeeping >>> > https://www.python.org/dev/peps/pep-0008/#imports >>> > >>> > Stipulate that they should be "at the top of the file after any module >>> > comments and docstrings." Which of these takes precedence? >>> >>> I don't know an official answer. The convention I've observed is >>> overwhelmingly in one direction: import statements come before any >>> assignment statements. >>> >>> > Secondly, we also have an "__author__", and "__project__" variables, I >>> > assume these would be put with the version information as well? >>> >>> Yes. >>> >>> -- >>> \ ?Welchen Teil von ?Gestalt? verstehen Sie nicht? [What part of | >>> `\ ?gestalt? don't you understand?]? ?Karsten M. Self | >>> _o__) | >>> Ben Finney >>> >>> _______________________________________________ >>> Python-Dev mailing list >>> Python-Dev at python.org >>> https://mail.python.org/mailman/listinfo/python-dev >>> Unsubscribe: >>> https://mail.python.org/mailman/options/python-dev/guido%40python.org >>> >> >> >> >> -- >> --Guido van Rossum (python.org/~guido) >> >> _______________________________________________ >> Python-Dev mailing list >> Python-Dev at python.org >> https://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe: >> https://mail.python.org/mailman/options/python-dev/ianlee1521%40gmail.com >> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From victor.stinner at gmail.com Sat Mar 21 12:03:02 2015 From: victor.stinner at gmail.com (Victor Stinner) Date: Sat, 21 Mar 2015 12:03:02 +0100 Subject: [Python-Dev] How to document functions with optional positional parameters? In-Reply-To: References: Message-ID: Le samedi 21 mars 2015, Serhiy Storchaka a ?crit : > > For example binascii.crc32(). It has two positional parameters, one is mandatory, and one is optional with default value 0. With Argument Clinic its signature is crc32(data, crc=0, /). The \ is useful, it indicates that you cannot use keywords. If you want to drop \, modify the function to accept keywords. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Sat Mar 21 12:46:14 2015 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 21 Mar 2015 21:46:14 +1000 Subject: [Python-Dev] Needed reviews In-Reply-To: References: Message-ID: On 19 March 2015 at 19:28, Serhiy Storchaka wrote: > Here is list of my ready for review patches. It is incomplete and contains > only patches for which I don't expect objections or long discussion. Most > of them are relative easy and need only formal review. Most of them wait > for a review many months. It's worth noting that If there are changes you feel are genuinely low risk, you can go ahead and merge them based on your own commit review (even if you also wrote the original patch). It's far from ideal, but also a pragmatic recognition of how low the available amount of core developer time we have is relative to the amount of work that *could* be done :( The only time that guideline doesn't apply is if the release manager declares mandatory commit reviews while preparing a release, and I think Larry uses a release branch for that these days instead. Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Sat Mar 21 12:50:04 2015 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 21 Mar 2015 21:50:04 +1000 Subject: [Python-Dev] Use ptyhon -s as default shbang for system python executables/daemons In-Reply-To: <20150318124825.337dc1c6@anarchist.wooz.org> References: <20150318124825.337dc1c6@anarchist.wooz.org> Message-ID: On 19 March 2015 at 02:48, Barry Warsaw wrote: > It's never gotten much farther than musings, but protecting the system against > the weird things people install would be a good thing. OTOH, this feels a lot > like virtual environments so maybe there's something useful to be mined there. I took it a fair bit further than musings: https://www.python.org/dev/peps/pep-0432/#a-system-python-executable Unfortunately PEP 432's a big project with very high barriers to entry (you need to come to grips with the current arcane startup sequence first), so I've never managed to hand it over after running out of time to pursue it myself. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Sat Mar 21 12:52:34 2015 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 21 Mar 2015 21:52:34 +1000 Subject: [Python-Dev] Use ptyhon -s as default shbang for system python executables/daemons In-Reply-To: <654D3177-05D2-403F-9531-136FC98ED73F@stufft.io> References: <20150318124825.337dc1c6@anarchist.wooz.org> <654D3177-05D2-403F-9531-136FC98ED73F@stufft.io> Message-ID: On 19 March 2015 at 07:51, Donald Stufft wrote: > I?ve long wished that the OS had it?s own virtual environment. A lot of problems > seems to come from trying to cram the things the OS wants with the things that > the user wants into the same namespace. I'm more wanting to go in the other direction and suggest to folks that if they're not *scripting the OS*, then the system Python isn't what they want, and they should be using at least a virtual environment, preferably an entirely separate software collection that they can upgrade on their own timeline, independently of what they system does. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Sat Mar 21 13:11:41 2015 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 21 Mar 2015 22:11:41 +1000 Subject: [Python-Dev] super() does not work during class initialization In-Reply-To: References: Message-ID: On 21 March 2015 at 00:03, Martin Teichmann wrote: > The current python fails the assertion, while with my patch everything is fine, > and I personally think __class__ should always actually refer to the class being > defined, which means at the minimum that it is actually, well, a class. For folks that haven't looked at the tracker issue: I personally like the change, but it does involve storing the cell object in a dunder-variable in the class namespace while it's being defined (until type.__new__ executes and both populates it and removes it from the class namespace). Since it introduces new behaviour that's visible to Python level code, I've suggested that Martin roll the suggestion into his current PEP 487 (which adds __init_subclass__ to similarly tidy up a few challenges with the way classes are currently initialised). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Sat Mar 21 13:14:45 2015 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 21 Mar 2015 22:14:45 +1000 Subject: [Python-Dev] Fwd: class os.DirEntry is confusing, In-Reply-To: References: Message-ID: On 21 March 2015 at 10:15, ???(Yonggang Luo) wrote: > ---------- Forwarded message ---------- > From: ???(Yonggang Luo) > Date: 2015-03-21 2:24 GMT+08:00 > Subject: class os.DirEntry is confusing, > To: Python > > > Should be replaced with os.FsEntry or os.ScanEntry The name comes from the underlying C/POSIX concept (dirent in this case), which is fairly common for the low level APIs in the os module. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From solipsis at pitrou.net Sat Mar 21 13:19:40 2015 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sat, 21 Mar 2015 13:19:40 +0100 Subject: [Python-Dev] Use ptyhon -s as default shbang for system python executables/daemons References: <20150318124825.337dc1c6@anarchist.wooz.org> <654D3177-05D2-403F-9531-136FC98ED73F@stufft.io> Message-ID: <20150321131940.6291fdaa@fsol> On Sat, 21 Mar 2015 21:52:34 +1000 Nick Coghlan wrote: > On 19 March 2015 at 07:51, Donald Stufft wrote: > > I?ve long wished that the OS had it?s own virtual environment. A lot of problems > > seems to come from trying to cram the things the OS wants with the things that > > the user wants into the same namespace. > > I'm more wanting to go in the other direction and suggest to folks > that if they're not *scripting the OS*, then the system Python isn't > what they want, and they should be using at least a virtual > environment, So that defeats the whole point of having a system distribution. I don't have any stakes in Debian / RedHat etc., but that sounds a little bit ridiculous to me ;-) On the other hand, perhaps it would allow ditching any requests to keep code compatible with old versions such as 2.6. Regards Antoine. From bcannon at gmail.com Sat Mar 21 14:34:19 2015 From: bcannon at gmail.com (Brett Cannon) Date: Sat, 21 Mar 2015 13:34:19 +0000 Subject: [Python-Dev] Final call for PEP 488: eliminating PYO files In-Reply-To: References: Message-ID: Thanks! PEP 488 is now marked as accepted. I expect I will have PEP 488 implemented before the PyCon sprints are over (work will be tracked in http://bugs.python.org/issue23731). On Fri, Mar 20, 2015 at 8:06 PM Guido van Rossum wrote: > Awesome, that's what I was hoping. Accepted! Congrats and thank you very > much for writing the PEP and guiding the discussion. > > On Fri, Mar 20, 2015 at 4:00 PM, Brett Cannon wrote: > >> >> >> On Fri, Mar 20, 2015 at 4:41 PM Guido van Rossum >> wrote: >> >>> I am willing to be the BDFL for this PEP. I have tried to skim the >>> recent discussion (only python-dev) and I don't see much remaining >>> controversy. HOWEVER... The PEP is not clear (or at least too subtle) about >>> the actual name for optimization level 0. If I have foo.py, and I compile >>> it three times with three different optimization levels (no optimization; >>> -O; -OO), and then I look in __pycache__, would I see this: >>> >>> # (1) >>> foo.cpython-35.pyc >>> foo.cpython-35.opt-1.pyc >>> foo.cpython-35.opt-2.pyc >>> >>> Or would I see this? >>> >>> # (2) >>> foo.cpython-35.opt-0.pyc >>> foo.cpython-35.opt-1.pyc >>> foo.cpython-35.opt-2.pyc >>> >> >> #1 >> >> >>> >>> Your lead-in ("I have decided to have the default case of no >>> optimization levels mean that the .pyc file name will have *no* optimization >>> level specified in the name and thus be just as it is today.") makes me >>> think I should expect (1), but I can't actually pinpoint where the language >>> of the PEP says this. >>> >> >> It was meant to be explained by "When no optimization level is >> specified, the pre-PEP ``.pyc`` file name will be used (i.e., no change >> in file name >> semantics)", but obviously it's a bit too subtle. I just updated the PEP >> with an explicit list of bytecode file name examples based on no -O, -O, >> and -OO. >> >> -Brett >> >> >>> >>> >>> On Fri, Mar 20, 2015 at 11:34 AM, Brett Cannon >>> wrote: >>> >>>> I have decided to have the default case of no optimization levels mean >>>> that the .pyc file name will have *no* optimization level specified in >>>> the name and thus be just as it is today. I made this decision due to >>>> potential backwards-compatibility issues -- although I expect them to be >>>> minutes -- and to not force other implementations like PyPy to have some >>>> bogus value set since they don't have .pyo files to begin with (PyPy >>>> actually uses bytecode for -O and don't bother with -OO since PyPy already >>>> uses a bunch of memory when running). >>>> >>>> Since this closes out the last open issue, I need either a BDFL >>>> decision or a BDFAP to be assigned to make a decision. Guido? >>>> >>>> ====================================== >>>> >>>> PEP: 488 >>>> Title: Elimination of PYO files >>>> Version: $Revision$ >>>> Last-Modified: $Date$ >>>> Author: Brett Cannon >>>> Status: Draft >>>> Type: Standards Track >>>> Content-Type: text/x-rst >>>> Created: 20-Feb-2015 >>>> Post-History: >>>> 2015-03-06 >>>> 2015-03-13 >>>> 2015-03-20 >>>> >>>> Abstract >>>> ======== >>>> >>>> This PEP proposes eliminating the concept of PYO files from Python. >>>> To continue the support of the separation of bytecode files based on >>>> their optimization level, this PEP proposes extending the PYC file >>>> name to include the optimization level in the bytecode repository >>>> directory when it's called for (i.e., the ``__pycache__`` directory). >>>> >>>> >>>> Rationale >>>> ========= >>>> >>>> As of today, bytecode files come in two flavours: PYC and PYO. A PYC >>>> file is the bytecode file generated and read from when no >>>> optimization level is specified at interpreter startup (i.e., ``-O`` >>>> is not specified). A PYO file represents the bytecode file that is >>>> read/written when **any** optimization level is specified (i.e., when >>>> ``-O`` **or** ``-OO`` is specified). This means that while PYC >>>> files clearly delineate the optimization level used when they were >>>> generated -- namely no optimizations beyond the peepholer -- the same >>>> is not true for PYO files. To put this in terms of optimization >>>> levels and the file extension: >>>> >>>> - 0: ``.pyc`` >>>> - 1 (``-O``): ``.pyo`` >>>> - 2 (``-OO``): ``.pyo`` >>>> >>>> The reuse of the ``.pyo`` file extension for both level 1 and 2 >>>> optimizations means that there is no clear way to tell what >>>> optimization level was used to generate the bytecode file. In terms >>>> of reading PYO files, this can lead to an interpreter using a mixture >>>> of optimization levels with its code if the user was not careful to >>>> make sure all PYO files were generated using the same optimization >>>> level (typically done by blindly deleting all PYO files and then >>>> using the `compileall` module to compile all-new PYO files [1]_). >>>> This issue is only compounded when people optimize Python code beyond >>>> what the interpreter natively supports, e.g., using the astoptimizer >>>> project [2]_. >>>> >>>> In terms of writing PYO files, the need to delete all PYO files >>>> every time one either changes the optimization level they want to use >>>> or are unsure of what optimization was used the last time PYO files >>>> were generated leads to unnecessary file churn. The change proposed >>>> by this PEP also allows for **all** optimization levels to be >>>> pre-compiled for bytecode files ahead of time, something that is >>>> currently impossible thanks to the reuse of the ``.pyo`` file >>>> extension for multiple optimization levels. >>>> >>>> As for distributing bytecode-only modules, having to distribute both >>>> ``.pyc`` and ``.pyo`` files is unnecessary for the common use-case >>>> of code obfuscation and smaller file deployments. This means that >>>> bytecode-only modules will only load from their non-optimized >>>> ``.pyc`` file name. >>>> >>>> >>>> Proposal >>>> ======== >>>> >>>> To eliminate the ambiguity that PYO files present, this PEP proposes >>>> eliminating the concept of PYO files and their accompanying ``.pyo`` >>>> file extension. To allow for the optimization level to be unambiguous >>>> as well as to avoid having to regenerate optimized bytecode files >>>> needlessly in the `__pycache__` directory, the optimization level >>>> used to generate the bytecode file will be incorporated into the >>>> bytecode file name. When no optimization level is specified, the >>>> pre-PEP ``.pyc`` file name will be used (i.e., no change in file name >>>> semantics). This increases backwards-compatibility while also being >>>> more understanding of Python implementations which have no use for >>>> optimization levels (e.g., PyPy[10]_). >>>> >>>> Currently bytecode file names are created by >>>> ``importlib.util.cache_from_source()``, approximately using the >>>> following expression defined by PEP 3147 [3]_, [4]_, [5]_:: >>>> >>>> '{name}.{cache_tag}.pyc'.format(name=module_name, >>>> >>>> cache_tag=sys.implementation.cache_tag) >>>> >>>> This PEP proposes to change the expression when an optimization >>>> level is specified to:: >>>> >>>> '{name}.{cache_tag}.opt-{optimization}.pyc'.format( >>>> name=module_name, >>>> cache_tag=sys.implementation.cache_tag, >>>> optimization=str(sys.flags.optimize)) >>>> >>>> The "opt-" prefix was chosen so as to provide a visual separator >>>> from the cache tag. The placement of the optimization level after >>>> the cache tag was chosen to preserve lexicographic sort order of >>>> bytecode file names based on module name and cache tag which will >>>> not vary for a single interpreter. The "opt-" prefix was chosen over >>>> "o" so as to be somewhat self-documenting. The "opt-" prefix was >>>> chosen over "O" so as to not have any confusion in case "0" was the >>>> leading prefix of the optimization level. >>>> >>>> A period was chosen over a hyphen as a separator so as to distinguish >>>> clearly that the optimization level is not part of the interpreter >>>> version as specified by the cache tag. It also lends to the use of >>>> the period in the file name to delineate semantically different >>>> concepts. >>>> >>>> For example, if ``-OO`` had been passed to the interpreter then instead >>>> of ``importlib.cpython-35.pyo`` the file name would be >>>> ``importlib.cpython-35.opt-2.pyc``. >>>> >>>> It should be noted that this change in no way affects the performance >>>> of import. Since the import system looks for a single bytecode file >>>> based on the optimization level of the interpreter already and >>>> generates a new bytecode file if it doesn't exist, the introduction >>>> of potentially more bytecode files in the ``__pycache__`` directory >>>> has no effect in terms of stat calls. The interpreter will continue >>>> to look for only a single bytecode file based on the optimization >>>> level and thus no increase in stat calls will occur. >>>> >>>> The only potentially negative result of this PEP is the probable >>>> increase in the number of ``.pyc`` files and thus increase in storage >>>> use. But for platforms where this is an issue, >>>> ``sys.dont_write_bytecode`` exists to turn off bytecode generation so >>>> that it can be controlled offline. >>>> >>>> >>>> Implementation >>>> ============== >>>> >>>> importlib >>>> --------- >>>> >>>> As ``importlib.util.cache_from_source()`` is the API that exposes >>>> bytecode file paths as well as being directly used by importlib, it >>>> requires the most critical change. As of Python 3.4, the function's >>>> signature is:: >>>> >>>> importlib.util.cache_from_source(path, debug_override=None) >>>> >>>> This PEP proposes changing the signature in Python 3.5 to:: >>>> >>>> importlib.util.cache_from_source(path, debug_override=None, *, >>>> optimization=None) >>>> >>>> The introduced ``optimization`` keyword-only parameter will control >>>> what optimization level is specified in the file name. If the >>>> argument is ``None`` then the current optimization level of the >>>> interpreter will be assumed (including no optimization). Any argument >>>> given for ``optimization`` will be passed to ``str()`` and must have >>>> ``str.isalnum()`` be true, else ``ValueError`` will be raised (this >>>> prevents invalid characters being used in the file name). If the >>>> empty string is passed in for ``optimization`` then the addition of >>>> the optimization will be suppressed, reverting to the file name >>>> format which predates this PEP. >>>> >>>> It is expected that beyond Python's own two optimization levels, >>>> third-party code will use a hash of optimization names to specify the >>>> optimization level, e.g. >>>> ``hashlib.sha256(','.join(['no dead code', 'const >>>> folding'])).hexdigest()``. >>>> While this might lead to long file names, it is assumed that most >>>> users never look at the contents of the __pycache__ directory and so >>>> this won't be an issue. >>>> >>>> The ``debug_override`` parameter will be deprecated. As the parameter >>>> expects a boolean, the integer value of the boolean will be used as >>>> if it had been provided as the argument to ``optimization`` (a >>>> ``None`` argument will mean the same as for ``optimization``). A >>>> deprecation warning will be raised when ``debug_override`` is given a >>>> value other than ``None``, but there are no plans for the complete >>>> removal of the parameter at this time (but removal will be no later >>>> than Python 4). >>>> >>>> The various module attributes for importlib.machinery which relate to >>>> bytecode file suffixes will be updated [7]_. The >>>> ``DEBUG_BYTECODE_SUFFIXES`` and ``OPTIMIZED_BYTECODE_SUFFIXES`` will >>>> both be documented as deprecated and set to the same value as >>>> ``BYTECODE_SUFFIXES`` (removal of ``DEBUG_BYTECODE_SUFFIXES`` and >>>> ``OPTIMIZED_BYTECODE_SUFFIXES`` is not currently planned, but will be >>>> not later than Python 4). >>>> >>>> All various finders and loaders will also be updated as necessary, >>>> but updating the previous mentioned parts of importlib should be all >>>> that is required. >>>> >>>> >>>> Rest of the standard library >>>> ---------------------------- >>>> >>>> The various functions exposed by the ``py_compile`` and >>>> ``compileall`` functions will be updated as necessary to make sure >>>> they follow the new bytecode file name semantics [6]_, [1]_. The CLI >>>> for the ``compileall`` module will not be directly affected (the >>>> ``-b`` flag will be implicit as it will no longer generate ``.pyo`` >>>> files when ``-O`` is specified). >>>> >>>> >>>> Compatibility Considerations >>>> ============================ >>>> >>>> Any code directly manipulating bytecode files from Python 3.2 on >>>> will need to consider the impact of this change on their code (prior >>>> to Python 3.2 -- including all of Python 2 -- there was no >>>> __pycache__ which already necessitates bifurcating bytecode file >>>> handling support). If code was setting the ``debug_override`` >>>> argument to ``importlib.util.cache_from_source()`` then care will be >>>> needed if they want the path to a bytecode file with an optimization >>>> level of 2. Otherwise only code **not** using >>>> ``importlib.util.cache_from_source()`` will need updating. >>>> >>>> As for people who distribute bytecode-only modules (i.e., use a >>>> bytecode file instead of a source file), they will have to choose >>>> which optimization level they want their bytecode files to be since >>>> distributing a ``.pyo`` file with a ``.pyc`` file will no longer be >>>> of any use. Since people typically only distribute bytecode files for >>>> code obfuscation purposes or smaller distribution size then only >>>> having to distribute a single ``.pyc`` should actually be beneficial >>>> to these use-cases. And since the magic number for bytecode files >>>> changed in Python 3.5 to support PEP 465 there is no need to support >>>> pre-existing ``.pyo`` files [8]_. >>>> >>>> >>>> Rejected Ideas >>>> ============== >>>> >>>> Completely dropping optimization levels from CPython >>>> ---------------------------------------------------- >>>> >>>> Some have suggested that instead of accommodating the various >>>> optimization levels in CPython, we should instead drop them >>>> entirely. The argument is that significant performance gains would >>>> occur from runtime optimizations through something like a JIT and not >>>> through pre-execution bytecode optimizations. >>>> >>>> This idea is rejected for this PEP as that ignores the fact that >>>> there are people who do find the pre-existing optimization levels for >>>> CPython useful. It also assumes that no other Python interpreter >>>> would find what this PEP proposes useful. >>>> >>>> >>>> Alternative formatting of the optimization level in the file name >>>> ----------------------------------------------------------------- >>>> >>>> Using the "opt-" prefix and placing the optimization level between >>>> the cache tag and file extension is not critical. All options which >>>> have been considered are: >>>> >>>> * ``importlib.cpython-35.opt-1.pyc`` >>>> * ``importlib.cpython-35.opt1.pyc`` >>>> * ``importlib.cpython-35.o1.pyc`` >>>> * ``importlib.cpython-35.O1.pyc`` >>>> * ``importlib.cpython-35.1.pyc`` >>>> * ``importlib.cpython-35-O1.pyc`` >>>> * ``importlib.O1.cpython-35.pyc`` >>>> * ``importlib.o1.cpython-35.pyc`` >>>> * ``importlib.1.cpython-35.pyc`` >>>> >>>> These were initially rejected either because they would change the >>>> sort order of bytecode files, possible ambiguity with the cache tag, >>>> or were not self-documenting enough. An informal poll was taken and >>>> people clearly preferred the formatting proposed by the PEP [9]_. >>>> Since this topic is non-technical and of personal choice, the issue >>>> is considered solved. >>>> >>>> >>>> Embedding the optimization level in the bytecode metadata >>>> --------------------------------------------------------- >>>> >>>> Some have suggested that rather than embedding the optimization level >>>> of bytecode in the file name that it be included in the file's >>>> metadata instead. This would mean every interpreter had a single copy >>>> of bytecode at any time. Changing the optimization level would thus >>>> require rewriting the bytecode, but there would also only be a single >>>> file to care about. >>>> >>>> This has been rejected due to the fact that Python is often installed >>>> as a root-level application and thus modifying the bytecode file for >>>> modules in the standard library are always possible. In this >>>> situation integrators would need to guess at what a reasonable >>>> optimization level was for users for any/all situations. By >>>> allowing multiple optimization levels to co-exist simultaneously it >>>> frees integrators from having to guess what users want and allows >>>> users to utilize the optimization level they want. >>>> >>>> >>>> References >>>> ========== >>>> >>>> .. [1] The compileall module >>>> (https://docs.python.org/3/library/compileall.html#module-compileall >>>> ) >>>> >>>> .. [2] The astoptimizer project >>>> (https://pypi.python.org/pypi/astoptimizer) >>>> >>>> .. [3] ``importlib.util.cache_from_source()`` >>>> ( >>>> https://docs.python.org/3.5/library/importlib.html#importlib.util.cache_from_source >>>> ) >>>> >>>> .. [4] Implementation of ``importlib.util.cache_from_source()`` from >>>> CPython 3.4.3rc1 >>>> ( >>>> https://hg.python.org/cpython/file/038297948389/Lib/importlib/_bootstrap.py#l437 >>>> ) >>>> >>>> .. [5] PEP 3147, PYC Repository Directories, Warsaw >>>> (http://www.python.org/dev/peps/pep-3147) >>>> >>>> .. [6] The py_compile module >>>> (https://docs.python.org/3/library/compileall.html#module-compileall >>>> ) >>>> >>>> .. [7] The importlib.machinery module >>>> ( >>>> https://docs.python.org/3/library/importlib.html#module-importlib.machinery >>>> ) >>>> >>>> .. [8] ``importlib.util.MAGIC_NUMBER`` >>>> ( >>>> https://docs.python.org/3/library/importlib.html#importlib.util.MAGIC_NUMBER >>>> ) >>>> >>>> .. [9] Informal poll of file name format options on Google+ >>>> (https://plus.google.com/u/0/+BrettCannon/posts/fZynLNwHWGm) >>>> >>>> .. [10] The PyPy Project >>>> (http://pypy.org/) >>>> >>>> >>>> Copyright >>>> ========= >>>> >>>> This document has been placed in the public domain. >>>> >>>> >>>> .. >>>> Local Variables: >>>> mode: indented-text >>>> indent-tabs-mode: nil >>>> sentence-end-double-space: t >>>> fill-column: 70 >>>> coding: utf-8 >>>> End: >>>> >>>> >>>> _______________________________________________ >>>> Python-Dev mailing list >>>> Python-Dev at python.org >>>> https://mail.python.org/mailman/listinfo/python-dev >>>> Unsubscribe: >>>> https://mail.python.org/mailman/options/python-dev/guido%40python.org >>>> >>>> >>> >>> >>> -- >>> --Guido van Rossum (python.org/~guido) >>> >> > > > -- > --Guido van Rossum (python.org/~guido) > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Sat Mar 21 15:16:21 2015 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 22 Mar 2015 00:16:21 +1000 Subject: [Python-Dev] Use ptyhon -s as default shbang for system python executables/daemons In-Reply-To: <20150321131940.6291fdaa@fsol> References: <20150318124825.337dc1c6@anarchist.wooz.org> <654D3177-05D2-403F-9531-136FC98ED73F@stufft.io> <20150321131940.6291fdaa@fsol> Message-ID: On 21 March 2015 at 22:19, Antoine Pitrou wrote: > On Sat, 21 Mar 2015 21:52:34 +1000 > Nick Coghlan wrote: >> On 19 March 2015 at 07:51, Donald Stufft wrote: >> > I?ve long wished that the OS had it?s own virtual environment. A lot of problems >> > seems to come from trying to cram the things the OS wants with the things that >> > the user wants into the same namespace. >> >> I'm more wanting to go in the other direction and suggest to folks >> that if they're not *scripting the OS*, then the system Python isn't >> what they want, and they should be using at least a virtual >> environment, > > So that defeats the whole point of having a system distribution. Given how many of the operating system packages are written in Python, it really doesn't - system administrators getting to use Python to script the OS is a secondary purpose, the system Python installation is mostly there to make the OS itself work (if it wasn't for that, migrating the system Python to Python 3 would have been a lot simpler!). In my view, running system independent applications in the system Python is at best a tertiary purpose - many of those apps (especially web services) *don't want* to be using system modules at all, which is why virtualenv switched to disabling the global site-packages by default. > I don't have any stakes in Debian / RedHat etc., but that sounds > a little bit ridiculous to me ;-) On the other hand, perhaps it would > allow ditching any requests to keep code compatible with old versions > such as 2.6. That's exactly the reason for the preference - unless they're specifically wanting to script RHEL/CentOS 6 or a similar long term support version at this point, ideally folks should be able to use Python 2.7 as their baseline version by now. RHEL/CentOS 7 are both on 2.7, while Python 2.7 has been available for RHEL/CentOS 6 by way of software collections for a couple of years now. When Python 2 goes EOL upstream in 2020, the standard 10 years of support on RHEL 7 will still have 4 years left to run - I'd prefer to see most folks migrated to a Python 3 software collection by then (or failing that, at least to the Python 3 stack Slavek is aiming to get into EPEL 7). Unfortunately, as far as I am aware, Debian Stable and Ubuntu LTS don't yet have a distro-endorsed equivalent of softwarecollections.org, and it's not yet clear if running on a faster moving base distro in a Docker or Rocket container will be an adequate alternative. I believe SCLs moved over to using Linux environment modules as the backend tech in 2.0 (for compatibility with existing HPC environments), so it should likely be possible to adapt them to work with APT in addition to RPM, but that requires finding folks both interested in doing the technical work, as well as in making the case for SCLs as a feature in the relevant distros (they're a large enough departure from the traditional mode of operation of a Linux distro that it took a long time to find a stable upstream home for the ongoing maintenance of SCLs for the Fedora/RHEL/CentOS ecosystem - primary maintenance responsibility finally landed earlier this year as a CentOS SIG) Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ericsnowcurrently at gmail.com Sat Mar 21 18:18:12 2015 From: ericsnowcurrently at gmail.com (Eric Snow) Date: Sat, 21 Mar 2015 11:18:12 -0600 Subject: [Python-Dev] Final call for PEP 488: eliminating PYO files In-Reply-To: References: Message-ID: On Mar 21, 2015 7:44 AM, "Brett Cannon" wrote: > > Thanks! PEP 488 is now marked as accepted. I expect I will have PEP 488 implemented before the PyCon sprints are over (work will be tracked in http://bugs.python.org/issue23731). > > On Fri, Mar 20, 2015 at 8:06 PM Guido van Rossum wrote: >> >> Awesome, that's what I was hoping. Accepted! Congrats and thank you very much for writing the PEP and guiding the discussion. Congratulations Brett! This is a welcome change. I'll be sure to give you a review. -eric -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Sat Mar 21 19:29:13 2015 From: donald at stufft.io (Donald Stufft) Date: Sat, 21 Mar 2015 14:29:13 -0400 Subject: [Python-Dev] Use ptyhon -s as default shbang for system python executables/daemons In-Reply-To: References: <20150318124825.337dc1c6@anarchist.wooz.org> <654D3177-05D2-403F-9531-136FC98ED73F@stufft.io> Message-ID: > On Mar 21, 2015, at 7:52 AM, Nick Coghlan wrote: > > On 19 March 2015 at 07:51, Donald Stufft wrote: >> I?ve long wished that the OS had it?s own virtual environment. A lot of problems >> seems to come from trying to cram the things the OS wants with the things that >> the user wants into the same namespace. > > I'm more wanting to go in the other direction and suggest to folks > that if they're not *scripting the OS*, then the system Python isn't > what they want, and they should be using at least a virtual > environment, preferably an entirely separate software collection that > they can upgrade on their own timeline, independently of what they > system does. > It?s likely easier to get the OS to move it?s own things to a virtual environment than it is to convince every single person who uses an OS to never install globally. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From tjreedy at udel.edu Sat Mar 21 19:47:42 2015 From: tjreedy at udel.edu (Terry Reedy) Date: Sat, 21 Mar 2015 14:47:42 -0400 Subject: [Python-Dev] How to document functions with optional positional parameters? In-Reply-To: References: Message-ID: On 3/21/2015 12:41 AM, Serhiy Storchaka wrote: > How to document functions with optional positional parameters? > > For example binascii.crc32(). It has two positional parameters, one is > mandatory, and one is optional with default value 0. With Argument > Clinic its signature is crc32(data, crc=0, /). In the documentation it > is written as crc32(data[, crc]) (and noted that the default value of > the second parameter is 0). Both are not valid Python syntax. Can the > documentation be change to crc32(data, crc=0)? I think the docs should be using / as well. > Related issues: > > https://bugs.python.org/issue21488 (changed encode(obj, > encoding='ascii', errors='strict') to encode(obj, [encoding[, errors]])) > > https://bugs.python.org/issue22832 (changed ioctl(fd, op[, arg[, > mutate_flag]]) to ioctl(fd, request, arg=0, mutate_flag=True)) > > https://bugs.python.org/issue22341 (discussed changing crc32(data[, > crc]) to crc32(data, crc=0)) > -- Terry Jan Reedy From barry at python.org Sat Mar 21 21:53:50 2015 From: barry at python.org (Barry Warsaw) Date: Sat, 21 Mar 2015 16:53:50 -0400 Subject: [Python-Dev] some minor questions about pep8 In-Reply-To: References: <85lhir6seo.fsf@benfinney.id.au> Message-ID: <20150321165350.726db103@anarchist.wooz.org> On Mar 20, 2015, at 08:53 PM, Guido van Rossum wrote: >FWIW, I think __version__, __author__ etc. were bad ideas. Almost nobody >manages these correctly. Note that the PEP 8 section starts with less than >an endorsement: "If you *have* to have Subversion, CVS, or RCS crud in your >source file, do it as follows." I tend to agree. Individual module version numbers are mostly useless, especially with today's hash-based vcses. That's different than package versions, and for which I really need to resurrect and update PEP 396. >That said, if an official answer is required, common sense would suggest >that __version__ should go before the imports. (I would put it before the >docstring too, except then the docstring wouldn't be a docstring any more. >Go figure.) And after __future__ imports too, right? Cheers, -Barry From jim.baker at python.org Sat Mar 21 21:14:03 2015 From: jim.baker at python.org (Jim Baker) Date: Sat, 21 Mar 2015 14:14:03 -0600 Subject: [Python-Dev] Google Summer of Code Message-ID: Jython plans to participate in the Google Summer of Code for 2015. If you are interested, I have outlined a number of projects on our ideas page that students could work on: - Work on JyNI, which adds C extension API support to Jython - Performance optimizations, including startup time - Python bytecode compiler for Python source (must be written in Java!) See https://github.com/jythontools/gsoc2015 for the ideas page. Pull requests and comments gladly accepted to expand this list of ideas. The ideal student must be a very strong Java developer, with an existing track record of projects on such sites as GitHub and BitBucket. Knowledge of C and especially of the CPython source code invariably helps. Ideally you will also become a Jython committer as a result of your intense participation this summer. Note that the deadline for student proposals is very soon, Friday March 27. I apologize we are just getting our participation for GSOC together, but we have also been busy finalizing the Jython 2.7.0 release candidate. -- - Jim jim.baker@{colorado.edu|python.org|rackspace.com|zyasoft.com} twitter.com/jimbaker github.com/jimbaker bitbucket.com/jimbaker -------------- next part -------------- An HTML attachment was scrubbed... URL: From barry at python.org Sat Mar 21 21:56:30 2015 From: barry at python.org (Barry Warsaw) Date: Sat, 21 Mar 2015 16:56:30 -0400 Subject: [Python-Dev] some minor questions about pep8 In-Reply-To: References: <85lhir6seo.fsf@benfinney.id.au> Message-ID: <20150321165630.3d6bd1a7@anarchist.wooz.org> On Mar 20, 2015, at 09:59 PM, Ian Lee wrote: >shebang >docstring >imports >dunder assignments >other code... I generally put imports after dunders, except for __future__. I want the first screenful of code to contain them, and imports can be a few dozen lines in some modules. -Barry From benjamin at python.org Sat Mar 21 21:57:22 2015 From: benjamin at python.org (Benjamin Peterson) Date: Sat, 21 Mar 2015 16:57:22 -0400 Subject: [Python-Dev] Missing *-unpacking generalizations (issue 2292) In-Reply-To: References: <20150319175353.26159.19083@psf.upfronthosting.co.za> Message-ID: <1426971442.2005452.243475606.2CF0A0A3@webmail.messagingengine.com> On Fri, Mar 20, 2015, at 19:16, Neil Girdhar wrote: > Wow, this is an excellent review. Thank you. > > My only question is with respect to this: > > I think there ought to be two opcodes; one for unpacking maps in > function calls and another for literals. The whole function location > thing is rather hideous. > > What are the two opcodes? BUILD_MAP_UNPACK and > BUILD_MAP_UNPACK_WITH_CALL? > > The first takes (n) a number of maps that it will merge, and the second > does the same but also accepts (function_call_location) for the purpose > of > error reporting? Exactly. From donald at stufft.io Sat Mar 21 22:00:11 2015 From: donald at stufft.io (Donald Stufft) Date: Sat, 21 Mar 2015 17:00:11 -0400 Subject: [Python-Dev] some minor questions about pep8 In-Reply-To: <20150321165350.726db103@anarchist.wooz.org> References: <85lhir6seo.fsf@benfinney.id.au> <20150321165350.726db103@anarchist.wooz.org> Message-ID: > On Mar 21, 2015, at 4:53 PM, Barry Warsaw wrote: > > On Mar 20, 2015, at 08:53 PM, Guido van Rossum wrote: > >> FWIW, I think __version__, __author__ etc. were bad ideas. Almost nobody >> manages these correctly. Note that the PEP 8 section starts with less than >> an endorsement: "If you *have* to have Subversion, CVS, or RCS crud in your >> source file, do it as follows." > > I tend to agree. Individual module version numbers are mostly useless, > especially with today's hash-based vcses. > > That's different than package versions, and for which I really need to > resurrect and update PEP 396. I sort of think (though I haven?t completely convinced myself) that adding something like __version__ to a package is a work around to the fact that we don?t have an API that lets someone ask ?what is the distribution and version that this module/import-package came from?. Something like: >>> import theorticalapi >>> import mymodule >>> >>> d = theorticalapi.module_dist(mymodule) >>> print(d.name) >>> mycustommodules >>> print(d.version) >>> 1.0 Since we (theoretically) should have all of this information already inside of the packaging metadata, we just don?t have a way to say ?given a particular import, give me the information I want). > >> That said, if an official answer is required, common sense would suggest >> that __version__ should go before the imports. (I would put it before the >> docstring too, except then the docstring wouldn't be a docstring any more. >> Go figure.) > > And after __future__ imports too, right? > > Cheers, > -Barry > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/donald%40stufft.io --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From python at mrabarnett.plus.com Sat Mar 21 22:17:26 2015 From: python at mrabarnett.plus.com (MRAB) Date: Sat, 21 Mar 2015 21:17:26 +0000 Subject: [Python-Dev] some minor questions about pep8 In-Reply-To: References: <85lhir6seo.fsf@benfinney.id.au> <20150321165350.726db103@anarchist.wooz.org> Message-ID: <550DDFE6.2020801@mrabarnett.plus.com> On 2015-03-21 21:00, Donald Stufft wrote: > >> On Mar 21, 2015, at 4:53 PM, Barry Warsaw wrote: >> >> On Mar 20, 2015, at 08:53 PM, Guido van Rossum wrote: >> >>> FWIW, I think __version__, __author__ etc. were bad ideas. Almost nobody >>> manages these correctly. Note that the PEP 8 section starts with less than >>> an endorsement: "If you *have* to have Subversion, CVS, or RCS crud in your >>> source file, do it as follows." >> >> I tend to agree. Individual module version numbers are mostly useless, >> especially with today's hash-based vcses. >> >> That's different than package versions, and for which I really need to >> resurrect and update PEP 396. > > I sort of think (though I haven?t completely convinced myself) that adding > something like __version__ to a package is a work around to the fact that > we don?t have an API that lets someone ask ?what is the distribution and > version that this module/import-package came from?. > > Something like: > > >>> import theorticalapi > >>> import mymodule > >>> > >>> d = theorticalapi.module_dist(mymodule) > >>> print(d.name) > >>> mycustommodules > >>> print(d.version) > >>> 1.0 > > Since we (theoretically) should have all of this information already inside > of the packaging metadata, we just don?t have a way to say ?given a particular > import, give me the information I want). > A bit OT (and should probably be in python-ideas), but I've come to think that it might be more helpful to have double version numbers. What do I mean by that? Well, an "addition" version that's increased when something is added, and a "deletion" version that's increased when something is removed. Suppose there's a new release of a module. If it has a higher "addition version", then something has been added, which won't affect existing code. If it has a higher "deletion version", then something has been removed, which _might_ affect existing code, so you'll need to check. If the code doesn't use what was removed, then you can accept the higher deletion version. >> >>> That said, if an official answer is required, common sense would suggest >>> that __version__ should go before the imports. (I would put it before the >>> docstring too, except then the docstring wouldn't be a docstring any more. >>> Go figure.) >> >> And after __future__ imports too, right? >> From donald at stufft.io Sat Mar 21 22:21:13 2015 From: donald at stufft.io (Donald Stufft) Date: Sat, 21 Mar 2015 17:21:13 -0400 Subject: [Python-Dev] some minor questions about pep8 In-Reply-To: <550DDFE6.2020801@mrabarnett.plus.com> References: <85lhir6seo.fsf@benfinney.id.au> <20150321165350.726db103@anarchist.wooz.org> <550DDFE6.2020801@mrabarnett.plus.com> Message-ID: <70DB1BB2-201D-4C89-BF4D-53368877D2A8@stufft.io> > On Mar 21, 2015, at 5:17 PM, MRAB wrote: > > On 2015-03-21 21:00, Donald Stufft wrote: >> >>> On Mar 21, 2015, at 4:53 PM, Barry Warsaw wrote: >>> >>> On Mar 20, 2015, at 08:53 PM, Guido van Rossum wrote: >>> >>>> FWIW, I think __version__, __author__ etc. were bad ideas. Almost nobody >>>> manages these correctly. Note that the PEP 8 section starts with less than >>>> an endorsement: "If you *have* to have Subversion, CVS, or RCS crud in your >>>> source file, do it as follows." >>> >>> I tend to agree. Individual module version numbers are mostly useless, >>> especially with today's hash-based vcses. >>> >>> That's different than package versions, and for which I really need to >>> resurrect and update PEP 396. >> >> I sort of think (though I haven?t completely convinced myself) that adding >> something like __version__ to a package is a work around to the fact that >> we don?t have an API that lets someone ask ?what is the distribution and >> version that this module/import-package came from?. >> >> Something like: >> >> >>> import theorticalapi >> >>> import mymodule >> >>> >> >>> d = theorticalapi.module_dist(mymodule) >> >>> print(d.name) >> >>> mycustommodules >> >>> print(d.version) >> >>> 1.0 >> >> Since we (theoretically) should have all of this information already inside >> of the packaging metadata, we just don?t have a way to say ?given a particular >> import, give me the information I want). >> > A bit OT (and should probably be in python-ideas), but I've come to think that it might be more helpful to have double version numbers. > > What do I mean by that? > > Well, an "addition" version that's increased when something is added, and a "deletion" version that's increased when something is removed. > > Suppose there's a new release of a module. > > If it has a higher "addition version", then something has been added, which won't affect existing code. > > If it has a higher "deletion version", then something has been removed, which _might_ affect existing code, so you'll need to check. If the code doesn't use what was removed, then you can accept the higher deletion version. This is likely better solved by something like https://semver.org/ which is similar to what many people do already: X.Y.Z+1 == Bug fixes only, should be backwards compatible. X.Y+1.0 == New features, should be backwards compatible but comes with new things. X+1.0.0 == Backwards Compatibility Break (Something deleted or changed) > >>> >>>> That said, if an official answer is required, common sense would suggest >>>> that __version__ should go before the imports. (I would put it before the >>>> docstring too, except then the docstring wouldn't be a docstring any more. >>>> Go figure.) >>> >>> And after __future__ imports too, right? >>> > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/donald%40stufft.io --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From barry at python.org Sat Mar 21 22:46:10 2015 From: barry at python.org (Barry Warsaw) Date: Sat, 21 Mar 2015 17:46:10 -0400 Subject: [Python-Dev] some minor questions about pep8 In-Reply-To: References: <85lhir6seo.fsf@benfinney.id.au> <20150321165350.726db103@anarchist.wooz.org> Message-ID: <20150321174610.1e310a80@limelight.wooz.org> On Mar 21, 2015, at 05:00 PM, Donald Stufft wrote: >I sort of think (though I haven?t completely convinced myself) that adding >something like __version__ to a package is a work around to the fact that >we don?t have an API that lets someone ask ?what is the distribution and >version that this module/import-package came from?. I tend to agree. Having that would solve one of the big problems that lead to PEP 394 thinking, that of having to update version numbers in more than one place. ISTM the best place to do it is once in setup.py and let the metadata flow. The only downside is for doing in-tree development without 'installing' the package (e.g. absence of venv, tox, or similar). Cheers, -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 819 bytes Desc: OpenPGP digital signature URL: From probiner at gmail.com Sun Mar 22 06:46:43 2015 From: probiner at gmail.com (pedro santos) Date: Sun, 22 Mar 2015 05:46:43 +0000 Subject: [Python-Dev] 0-base and 1-base indexed iterables? Custom slicing rules? Message-ID: Hi, I'm an Art and CG student learning Python and today's exercise was about positions in a tiled room. The fact that I had to check if a position was inside the room and given that in a 1x1 room, 0.0 was considered in and 1.0 was considered out, it kept me thinking about 0-base indexing iterables and slicing. Read some articles and discussions, some pros and cons to each 0-base and 1-base, concerns about slicing, etc. But ultimately the question that got stuck in me and didn't found an answer was: Why can't both 0-base and 1-base indexing exist in the same language, and why can't slicing be customized? If I'm indexing the ruler marks, intervals, boundaries, dots, makes sense to start of at 0; rul=[0,1,2,3,4,5,6] would index every mark on my ruler so that accordingly rul[0]=0, rul[5]=5. If I'm indexing the blue circles, natural number quantities, objects, spans, makes sense to start at 1; cir= [1,2,3,4,5] so that cir[1]=1 and cir[5]=5. Now, a lot of the discussion was to do with slicing coupled with the indexing and I don't totally understand why. a ? x < b is not so intuitive when dealing with objects ("I want balls 1 up to the the one before 3"), so on one side, you put the finger on what you want and on the other, on what you don't want. But this method does have the neat property of producing neighbor selections that border perfectly, as in [:a][a:b][b:c]. Although in inverse order(-1), the results can be unexpected as it returns values off-by-one from its counterpart like; L=[0,1,2,3,4,5] so that L[1:3]=[1,2] and L[3:1:-1]=[3:2]. So it's consistent with the rule a ? x < b, grabbing the lower limit item, but it can feel strange by not producing the same selection with inverse order. a ? x ? b is a natural way to select objets ("I want the balls 1 up to 3"), so you're putting the finger on the things you want. If you inverse the order(-1) it's still very easy to grasp what are you picking because whatever you select it's included like: L=[0,1,2,3,4,5] so that L[1:3]=[1,2,3] and L[3:1:-1]=[3,2,1]. Problems seem to arrive though, when trying to do neighbor selections, where one would have to do [:a][a+1:b][b+1:c] to have the border perfectly. That terrible? Even though one could see a ? x < b to be more adept to 0-base, and a ? x ? b to be more adept to 1-base, the way I see it, index base and slicing rules could be somehow independent. And one would index and slice the way it would fit the rationale of the problem or the data, because even slicing a 1-base indexed array with a ? x < b, would still produce an expected outcome, as in cir=[1,2,3,4,5] so that cir[1:3]=[1,2] or cir[:3]=[1,2]. Same thing applying a ? x ? b to a 0-base indexed array, as in rul[0,1,2,3,4,5] so that rul[:2]=[0,1,2] or rul[0:2]=[0,1,2]. Given that python is an example of human friendly code language, emphasizing readability, wouldn't having 0 and 1 base indexing and custom slicing methods, improve the thought process when writing and reading the code, by fitting them better to specific contexts or data? Is there some language that provides both or each language picks only one? Cheers -- *------------------------------[image: http://i153.photobucket.com/albums/s202/animatics/probiner-sig.gif]Pedro Alpiar?a dos Santos Animator 3DModeler Illustrator >> http://probiner.x10.mx/ * -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Sun Mar 22 07:12:51 2015 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 22 Mar 2015 16:12:51 +1000 Subject: [Python-Dev] How to document functions with optional positional parameters? In-Reply-To: References: Message-ID: On 22 March 2015 at 04:47, Terry Reedy wrote: > On 3/21/2015 12:41 AM, Serhiy Storchaka wrote: >> >> How to document functions with optional positional parameters? >> >> For example binascii.crc32(). It has two positional parameters, one is >> mandatory, and one is optional with default value 0. With Argument >> Clinic its signature is crc32(data, crc=0, /). In the documentation it >> is written as crc32(data[, crc]) (and noted that the default value of >> the second parameter is 0). Both are not valid Python syntax. Can the >> documentation be change to crc32(data, crc=0)? > > > I think the docs should be using / as well. Right, even though the Python level parser doesn't allow this syntax, it's the one used by Argument Clinic to express positional-only parameters for functions defined in C, and as a result the inspect module uses it as well: >>> import inspect >>> print(inspect.signature(hex)) (number, /) This is the specific problem Larry was aiming to address in https://www.python.org/dev/peps/pep-0457/ - getting the docs, Argument Clinic and the inspect module to align in their notation, even though he wasn't proposing to change Python itself to allow positional-only parameter definitions. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From v+python at g.nevcal.com Sun Mar 22 07:17:09 2015 From: v+python at g.nevcal.com (Glenn Linderman) Date: Sat, 21 Mar 2015 23:17:09 -0700 Subject: [Python-Dev] 0-base and 1-base indexed iterables? Custom slicing rules? In-Reply-To: References: Message-ID: <550E5E65.2020404@g.nevcal.com> On 3/21/2015 10:46 PM, pedro santos wrote: > > Hi, I'm an Art and CG student learning Python and today's exercise was > about positions in a tiled room. The fact that I had to check if a > position was inside the room and given that in a 1x1 room, 0.0 was > considered in and 1.0 was considered out, it kept me thinking about > 0-base indexing iterables and slicing. > This sort of discussion is better held on python-list at python.org which is for learning and using the current and past versions of Python. This mailing list is for coordinating the development of future versions of Python. If your idea is to start a discussion to attempt to affect future versions of Python, that sort of discussion belongs on python-ideas at python.org, and ideas that are appropriate for implementation in future versions of Python eventually come to python-dev when they are accepted for implementation. There are lots of friendly people on all these lists, but it is best to use the correct one, so that people can focus on the intended topics of the lists when reading each of them. Cheers! -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Sun Mar 22 07:26:48 2015 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 22 Mar 2015 16:26:48 +1000 Subject: [Python-Dev] some minor questions about pep8 In-Reply-To: <20150321174610.1e310a80@limelight.wooz.org> References: <85lhir6seo.fsf@benfinney.id.au> <20150321165350.726db103@anarchist.wooz.org> <20150321174610.1e310a80@limelight.wooz.org> Message-ID: On 22 March 2015 at 07:46, Barry Warsaw wrote: > On Mar 21, 2015, at 05:00 PM, Donald Stufft wrote: > >>I sort of think (though I haven?t completely convinced myself) that adding >>something like __version__ to a package is a work around to the fact that >>we don?t have an API that lets someone ask ?what is the distribution and >>version that this module/import-package came from?. > > I tend to agree. Having that would solve one of the big problems that lead to > PEP 394 thinking, that of having to update version numbers in more than one > place. ISTM the best place to do it is once in setup.py and let the metadata > flow. The only downside is for doing in-tree development without 'installing' > the package (e.g. absence of venv, tox, or similar). We don't *quite* track enough distribution metadata currently to reliably build the reverse mapping from module names to the packages that installed them. The draft "python.exports" extension in PEP 459 is supposed to provide that missing piece: https://www.python.org/dev/peps/pep-0459/#the-python-exports-extension Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From storchaka at gmail.com Sun Mar 22 09:41:27 2015 From: storchaka at gmail.com (Serhiy Storchaka) Date: Sun, 22 Mar 2015 10:41:27 +0200 Subject: [Python-Dev] How to document functions with optional positional parameters? In-Reply-To: References: Message-ID: On 21.03.15 13:03, Victor Stinner wrote: > The \ is useful, it indicates that you cannot use keywords. Wouldn't it confuse users? > If you want to drop \, modify the function to accept keywords. Yes, this is a solution. But parsing keyword arguments is slower than parsing positional arguments. And I'm working on patches that optimizes parsing code generated by Argument Clinic. First my patches will handle only positional parameters, with keywords it is harder. From storchaka at gmail.com Sun Mar 22 10:20:13 2015 From: storchaka at gmail.com (Serhiy Storchaka) Date: Sun, 22 Mar 2015 11:20:13 +0200 Subject: [Python-Dev] Needed reviews In-Reply-To: References: Message-ID: On 21.03.15 13:46, Nick Coghlan wrote: > On 19 March 2015 at 19:28, Serhiy Storchaka wrote: >> Here is list of my ready for review patches. It is incomplete and contains >> only patches for which I don't expect objections or long discussion. Most >> of them are relative easy and need only formal review. Most of them wait >> for a review many months. > > It's worth noting that If there are changes you feel are genuinely low > risk, you can go ahead and merge them based on your own commit review > (even if you also wrote the original patch). Yes, but four eyes are better than two eyes. I make mistakes. In some issues I hesitate about documentation part. In some issues (issue14260 and issue22721) I provided two alternative solutions and need a tip to choose from them. While I am mainly sure about the correctness of the patch, I'm often hesitate about the direction. Is the bug worth fixing? Is the new feature worth to be added to Python? Thanks Alexander, Amaury, Benjamin, Berker, Demian, ?ric, Ethan, Martin, Paul, Victor and others that responded on my request. From Stefan.Richthofer at gmx.de Sun Mar 22 11:44:59 2015 From: Stefan.Richthofer at gmx.de (Stefan Richthofer) Date: Sun, 22 Mar 2015 11:44:59 +0100 Subject: [Python-Dev] 0-base and 1-base indexed iterables? Custom slicing rules? In-Reply-To: References: Message-ID: An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Sun Mar 22 15:44:12 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Sun, 22 Mar 2015 14:44:12 +0000 Subject: [Python-Dev] Installing Python to non-ASCII paths Message-ID: Something that hit me today, which might become a more common issue when the Windows installers move towards installing to the user directory, is that there appear to be some bugs in handling of non-ASCII paths. Two that I spotted are a failure of the "script wrappers" installed by pip to work with a non-ASCII interpreter path (reported to distlib) and a possible issue with the py.exe launcher when a script has non-ASCII in the shebang line (not reported yet because I'm not clear on what's going on). I've only seen Windows-specific issues - I don't know how common non-ASCII paths for the python interpreter are on Unix or OSX, or whether the more or less universal use of UTF-8 on Unix makes such issues less common. But if anyone has an environment that makes testing on non-ASCII install paths easy, it might be worth doing some checks just so we can catch any major ones before 3.5 is released. On which note, I'm assuming neither of the issues I've found are major blockers. "pip.exe doesn't work if Python is installed in a directory with non-ASCII characters in the name" can be worked around by using python -m pip, and the launcher issue by using a generic shebang like #!/usr/bin/python3.5. Paul From mail at timgolden.me.uk Sun Mar 22 16:12:43 2015 From: mail at timgolden.me.uk (Tim Golden) Date: Sun, 22 Mar 2015 15:12:43 +0000 Subject: [Python-Dev] Installing Python to non-ASCII paths In-Reply-To: References: Message-ID: <550EDBEB.3010004@timgolden.me.uk> On 22/03/2015 14:44, Paul Moore wrote: > On which note, I'm assuming neither of the issues I've found are major > blockers. "pip.exe doesn't work if Python is installed in a directory > with non-ASCII characters in the name" can be worked around by using > python -m pip, and the launcher issue by using a generic shebang like > #!/usr/bin/python3.5. That can become a more major blocker if "pip doesn't work" == "ensurepip doesn't work and blocks thus the installer crashes" as was the case with a mimetypes issue a little while back. I'll create a ??? user (which is the easiest non-ASCII name to create on a UK keyboard) to see how cleanly the latest installer works. TJG From mail at timgolden.me.uk Sun Mar 22 16:29:54 2015 From: mail at timgolden.me.uk (Tim Golden) Date: Sun, 22 Mar 2015 15:29:54 +0000 Subject: [Python-Dev] Installing Python to non-ASCII paths In-Reply-To: <550EDBEB.3010004@timgolden.me.uk> References: <550EDBEB.3010004@timgolden.me.uk> Message-ID: <550EDFF2.3090601@timgolden.me.uk> On 22/03/2015 15:12, Tim Golden wrote: > On 22/03/2015 14:44, Paul Moore wrote: >> On which note, I'm assuming neither of the issues I've found are major >> blockers. "pip.exe doesn't work if Python is installed in a directory >> with non-ASCII characters in the name" can be worked around by using >> python -m pip, and the launcher issue by using a generic shebang like >> #!/usr/bin/python3.5. > > That can become a more major blocker if "pip doesn't work" == "ensurepip > doesn't work and blocks thus the installer crashes" as was the case with > a mimetypes issue a little while back. > > I'll create a ??? user (which is the easiest non-ASCII name to create on > a UK keyboard) to see how cleanly the latest installer works. Tried with "Mr ???". The installer's fine but the installed pip.exe fails while "py -3 -mpip" succeeeds as Paul notes. TJG From kwi at kwi.dk Sun Mar 22 18:56:09 2015 From: kwi at kwi.dk (=?UTF-8?B?U8O4cmVuIEzDuHZib3Jn?=) Date: Sun, 22 Mar 2015 18:56:09 +0100 Subject: [Python-Dev] Request for comments: [issue22941] IPv4Interface arithmetic changes subnet mask Message-ID: <550F0239.4040609@kwi.dk> Thanks for the feedback, and apologies for my late reply. I have to say, I'm not entirely sold on the argument for raising an exception on subnet "overflow". First, I'll note what happens if we overflow an IPv4Address: >>> ipaddress.IPv4Address('255.255.255.255') + 1 Traceback (most recent call last): ... ipaddress.AddressValueError: 4294967296 (>= 2**32) is not permitted as an IPv4 address Now, I used "IPv4Interface() + 1" to mean "Give me the IP next to the current one in the current subnet", knowing from the context that the address would be valid and available. >>> host = ipaddress.IPv4Interface('10.0.0.2/24') >>> peer = host + 1 In this context, I would welcome an exception, as it would certainly be an error if I overflowed the subnet. However, there are also situations in which overflowing would be valid and expected, e.g. as a way to skip to the "same" IP in the next subnet: >>> ip = ipaddress.IPv4Interface('10.0.0.42/24') >>> ip + ip.network.num_addresses IPv4Interface('10.0.1.42/24') It's not even a hypothetical example; I've been working on a distributed embedded system where all the hosts have two (redundant) addresses differing only by their subnet; this could be a convenient way calculate one address from the other. There's an additional issue with raising an exception, and that is that it still won't catch overflow errors in my example use case: >>> host = ipaddress.IPv4Interface('10.0.0.254/24') >>> peer = host + 1 >>> peer IPv4Interface('10.0.0.255/24') This doesn't overflow and does not trigger an exception, but the resulting peer address is still invalid (it's the subnet broadcast address, not a host address). As such, the exception isn't even useful as an error detection tool. (I'll not suggest raising an exception when hitting the broadcast or network address; that way lies madness.) As for consistency with IPv4Address, I can argue either way: Pro-exception: "Overflowing an IPv4Interface raises AddressValueError just like with IPv4Address." Contra-exception: "An IPv4Interface behaves exactly like an IPv4Address, except that it also has an associated subnet mask." (This is essentially how the type is currently documented). Thoughts? Best regards, S?ren L?vborg From victor.stinner at gmail.com Sun Mar 22 20:34:51 2015 From: victor.stinner at gmail.com (Victor Stinner) Date: Sun, 22 Mar 2015 20:34:51 +0100 Subject: [Python-Dev] Installing Python to non-ASCII paths In-Reply-To: References: Message-ID: Hi Paul, Please open an issue, I can take a look. Please describe a scenario to reproduce the issue. Victor 2015-03-22 15:44 GMT+01:00 Paul Moore : > Something that hit me today, which might become a more common issue > when the Windows installers move towards installing to the user > directory, is that there appear to be some bugs in handling of > non-ASCII paths. > > Two that I spotted are a failure of the "script wrappers" installed by > pip to work with a non-ASCII interpreter path (reported to distlib) > and a possible issue with the py.exe launcher when a script has > non-ASCII in the shebang line (not reported yet because I'm not clear > on what's going on). > > I've only seen Windows-specific issues - I don't know how common > non-ASCII paths for the python interpreter are on Unix or OSX, or > whether the more or less universal use of UTF-8 on Unix makes such > issues less common. But if anyone has an environment that makes > testing on non-ASCII install paths easy, it might be worth doing some > checks just so we can catch any major ones before 3.5 is released. > > On which note, I'm assuming neither of the issues I've found are major > blockers. "pip.exe doesn't work if Python is installed in a directory > with non-ASCII characters in the name" can be worked around by using > python -m pip, and the launcher issue by using a generic shebang like > #!/usr/bin/python3.5. > > Paul > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/victor.stinner%40gmail.com From p.f.moore at gmail.com Sun Mar 22 20:43:51 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Sun, 22 Mar 2015 19:43:51 +0000 Subject: [Python-Dev] Installing Python to non-ASCII paths In-Reply-To: References: Message-ID: On 22 March 2015 at 19:34, Victor Stinner wrote: > Please open an issue, I can take a look. Please describe a scenario to > reproduce the issue. The "issue" with the launcher seems to have bee a red herring. When I set up a test case properly, it worked. I suspect I messed up writing the shebang line in UTF8. Thanks anyway. (And the pip.exe issue is a distlib issue, which I have reported). Paul From ncoghlan at gmail.com Sun Mar 22 23:25:07 2015 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 23 Mar 2015 08:25:07 +1000 Subject: [Python-Dev] Needed reviews In-Reply-To: References: Message-ID: On 22 Mar 2015 19:22, "Serhiy Storchaka" wrote: > > On 21.03.15 13:46, Nick Coghlan wrote: >> >> On 19 March 2015 at 19:28, Serhiy Storchaka wrote: >>> >>> Here is list of my ready for review patches. It is incomplete and contains >>> only patches for which I don't expect objections or long discussion. Most >>> of them are relative easy and need only formal review. Most of them wait >>> for a review many months. >> >> >> It's worth noting that If there are changes you feel are genuinely low >> risk, you can go ahead and merge them based on your own commit review >> (even if you also wrote the original patch). > > > Yes, but four eyes are better than two eyes. I make mistakes. In some issues I hesitate about documentation part. In some issues (issue14260 and issue22721) I provided two alternative solutions and need a tip to choose from them. While I am mainly sure about the correctness of the patch, I'm often hesitate about the direction. Is the bug worth fixing? Is the new feature worth to be added to Python? Aye, agreed - those are the kinds of cases where I'd nudge folks for a review as well. Committing directly is something I'd only do where I'm already entirely confident the change is an improvement over the status quo. However, now that I think about it further, it's very rare for me to be completely confident in a change without another core dev at least giving a +1 on the general idea. > Thanks Alexander, Amaury, Benjamin, Berker, Demian, ?ric, Ethan, Martin, Paul, Victor and others that responded on my request. Indeed! One of the things I'm actually hoping to achieve through PEPs like 474 & 462 is to get to a point where we can better acknowledge and thank folks for all the work that goes into patch reviews :) Cheers, Nick. > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Sun Mar 22 23:42:05 2015 From: guido at python.org (Guido van Rossum) Date: Sun, 22 Mar 2015 15:42:05 -0700 Subject: [Python-Dev] some minor questions about pep8 In-Reply-To: References: <85lhir6seo.fsf@benfinney.id.au> Message-ID: On Fri, Mar 20, 2015 at 9:24 PM, Ian Lee wrote: > Guido, > > In that case would you be open to a patch to update the PEP accordingly? > Only if it totally eradicate __version__ and similar from the PEP. > Additionally, does that official statement cover other dunder assignments > (e.g. "__author__"?). If so I'll update the PEP8 tool accordingly. What official statement? I'm not the Pope. :-) --Guido > Thanks, > > ~ Ian Lee > On Mar 20, 2015 8:55 PM, "Guido van Rossum" wrote: > >> FWIW, I think __version__, __author__ etc. were bad ideas. Almost nobody >> manages these correctly. Note that the PEP 8 section starts with less than >> an endorsement: "If you *have* to have Subversion, CVS, or RCS crud in >> your source file, do it as follows." >> >> That said, if an official answer is required, common sense would suggest >> that __version__ should go before the imports. (I would put it before the >> docstring too, except then the docstring wouldn't be a docstring any more. >> Go figure.) >> >> On Fri, Mar 20, 2015 at 6:38 PM, Ben Finney >> wrote: >> >>> Lewis Coates writes: >>> >>> > In pep8 there are two conflicting statements, both >>> > >>> > https://www.python.org/dev/peps/pep-0008/#version-bookkeeping >>> > https://www.python.org/dev/peps/pep-0008/#imports >>> > >>> > Stipulate that they should be "at the top of the file after any module >>> > comments and docstrings." Which of these takes precedence? >>> >>> I don't know an official answer. The convention I've observed is >>> overwhelmingly in one direction: import statements come before any >>> assignment statements. >>> >>> > Secondly, we also have an "__author__", and "__project__" variables, I >>> > assume these would be put with the version information as well? >>> >>> Yes. >>> >>> -- >>> \ ?Welchen Teil von ?Gestalt? verstehen Sie nicht? [What part of | >>> `\ ?gestalt? don't you understand?]? ?Karsten M. Self | >>> _o__) | >>> Ben Finney >>> >>> _______________________________________________ >>> Python-Dev mailing list >>> Python-Dev at python.org >>> https://mail.python.org/mailman/listinfo/python-dev >>> Unsubscribe: >>> https://mail.python.org/mailman/options/python-dev/guido%40python.org >>> >> >> >> >> -- >> --Guido van Rossum (python.org/~guido) >> >> _______________________________________________ >> Python-Dev mailing list >> Python-Dev at python.org >> https://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe: >> https://mail.python.org/mailman/options/python-dev/ianlee1521%40gmail.com >> >> -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Sun Mar 22 23:51:23 2015 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 23 Mar 2015 08:51:23 +1000 Subject: [Python-Dev] Installing Python to non-ASCII paths In-Reply-To: References: Message-ID: On 23 Mar 2015 00:45, "Paul Moore" wrote: > > Something that hit me today, which might become a more common issue > when the Windows installers move towards installing to the user > directory, is that there appear to be some bugs in handling of > non-ASCII paths. > > Two that I spotted are a failure of the "script wrappers" installed by > pip to work with a non-ASCII interpreter path (reported to distlib) > and a possible issue with the py.exe launcher when a script has > non-ASCII in the shebang line (not reported yet because I'm not clear > on what's going on). > > I've only seen Windows-specific issues - I don't know how common > non-ASCII paths for the python interpreter are on Unix or OSX, or > whether the more or less universal use of UTF-8 on Unix makes such > issues less common. POSIX is fine if the locale encoding is correct, but can go fairly wrong if it isn't. Last major complaints I heard related to upstart sometimes getting it wrong in cron and for some daemonized setups (systemd appears to be more robust in setting it correctly as it pulls the expected setting from a system wide config file). "LANG=C" also doesn't work well, as that tells CPython to use ASCII instead of UTF-8 or whatever the actual system encoding is. Armin Ronacher pointed out "LANG=C.UTF-8" as a good alternative, but whether that's available or not is currently distro-specific. I filed an upstream bug with the glibc devs asking for that to be made standard, and they seemed amenable to the idea, but I haven't checked back in on its progress recently. > But if anyone has an environment that makes > testing on non-ASCII install paths easy, it might be worth doing some > checks just so we can catch any major ones before 3.5 is released. I'd suggest looking at the venv tests and using them as inspiration to create a separate "test_venv_nonascii" test file that checks: * creating a venv containing non-ASCII characters * copying the Python binary to a temporary directory with non-ASCII characters in the name and using that to create a venv More generally, we should likely enhance the venv tests to actually *run* the installed pip binary to list the installed packages. That will automatically test the distlib script wrappers, as well as checking the installed package set matches what we're currently bundling. With those changes, the buildbots would go a long way towards ensuring that non-ASCII installation paths always work correctly, as well as making it relatively straightforward for other implementations to adopt the same checks. Cheers, Nick. > > On which note, I'm assuming neither of the issues I've found are major > blockers. "pip.exe doesn't work if Python is installed in a directory > with non-ASCII characters in the name" can be worked around by using > python -m pip, and the launcher issue by using a generic shebang like > #!/usr/bin/python3.5. > > Paul > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From v+python at g.nevcal.com Mon Mar 23 02:46:15 2015 From: v+python at g.nevcal.com (Glenn Linderman) Date: Sun, 22 Mar 2015 18:46:15 -0700 Subject: [Python-Dev] Installing Python to non-ASCII paths In-Reply-To: <550EDBEB.3010004@timgolden.me.uk> References: <550EDBEB.3010004@timgolden.me.uk> Message-ID: <550F7067.4020902@g.nevcal.com> On 3/22/2015 8:12 AM, Tim Golden wrote: > I'll create a ??? user (which is the easiest non-ASCII name to create > on a UK keyboard) to see how cleanly the latest installer works. You can also copy/paste. A path with a Cyrillic, Greek, Chinese, Tibetan, Japanese, Armenian, and Romanian character, none of which are in the "Windows ANSI" character set, should suffice... Here ya go... ???????? In my work with Windows, I've certainly seen that ? is much more acceptable to more programs than ? or these other ones. -------------- next part -------------- An HTML attachment was scrubbed... URL: From cs at zip.com.au Mon Mar 23 07:10:42 2015 From: cs at zip.com.au (Cameron Simpson) Date: Mon, 23 Mar 2015 17:10:42 +1100 Subject: [Python-Dev] Use ptyhon -s as default shbang for system python executables/daemons In-Reply-To: References: Message-ID: <20150323061042.GA71955@cskk.homeip.net> On 21Mar2015 14:29, Donald Stufft wrote: >> On Mar 21, 2015, at 7:52 AM, Nick Coghlan wrote: >> On 19 March 2015 at 07:51, Donald Stufft wrote: >>> I?ve long wished that the OS had it?s own virtual environment. A lot of problems >>> seems to come from trying to cram the things the OS wants with the things that >>> the user wants into the same namespace. >> >> I'm more wanting to go in the other direction and suggest to folks >> that if they're not *scripting the OS*, then the system Python isn't >> what they want, and they should be using at least a virtual >> environment, preferably an entirely separate software collection that >> they can upgrade on their own timeline, independently of what they >> system does. > >It?s likely easier to get the OS to move it?s own things to a virtual >environment than it is to convince every single person who uses an OS >to never install globally. I agree. And just as a data point, this cropped up on the Fedora list yesterday: I broke Yum (by messing with Python libs) http://www.spinics.net/linux/fedora/fedora-users/msg458069.html TL;DR: OP used pip on his system python. Yum broke. Probably hampered his attempts to repair, too. Cheers, Cameron Simpson The mind reigns, but does not govern. - Paul Valery From mail at timgolden.me.uk Mon Mar 23 09:25:04 2015 From: mail at timgolden.me.uk (Tim Golden) Date: Mon, 23 Mar 2015 08:25:04 +0000 Subject: [Python-Dev] Installing Python to non-ASCII paths In-Reply-To: <550F7067.4020902@g.nevcal.com> References: <550EDBEB.3010004@timgolden.me.uk> <550F7067.4020902@g.nevcal.com> Message-ID: <550FCDE0.8030601@timgolden.me.uk> On 23/03/2015 01:46, Glenn Linderman wrote: > On 3/22/2015 8:12 AM, Tim Golden wrote: >> I'll create a ??? user (which is the easiest non-ASCII name to create >> on a UK keyboard) to see how cleanly the latest installer works. > > You can also copy/paste. A path with a Cyrillic, Greek, Chinese, > Tibetan, Japanese, Armenian, and Romanian character, none of which are > in the "Windows ANSI" character set, should suffice... Here ya go... > > ???????? > > In my work with Windows, I've certainly seen that ? is much more > acceptable to more programs than ? or these other ones. > Thanks, Glenn. Good point. TJG From a.badger at gmail.com Mon Mar 23 15:22:56 2015 From: a.badger at gmail.com (Toshio Kuratomi) Date: Mon, 23 Mar 2015 07:22:56 -0700 Subject: [Python-Dev] Use ptyhon -s as default shbang for system python executables/daemons In-Reply-To: References: <20150318122203.12ff1ee3@limelight.wooz.org> <20150318214431.GA4050@roan.lan> <20150318175607.66a25ad3@limelight.wooz.org> <20150319152734.5bb6cd79@limelight.wooz.org> Message-ID: -Toshio On Mar 19, 2015 3:27 PM, "Victor Stinner" wrote: > > 2015-03-19 21:47 GMT+01:00 Toshio Kuratomi : > > I think I've found the Debian discussion (October 2012): > > > > http://comments.gmane.org/gmane.linux.debian.devel.python/8188 > > > > Lack of PYTHONWARNINGS was brought up late in the discussion thread > > Maybe we need to modify -E or add a new option to only ignore PYTHONPATH. > I think pythonpath is still useful on its own. Building off Nick's idea of a system python vs a python for users to use, I would see a more useful modification to be able to specify SPYTHONPATH (and other env vars) to go along with /usr/bin/spython . That way the user maintains the capability to override specific libraries globally just like with LD_LIBRARY_PATH, PATH, and similar but you won't accidentally configure your own python to use one set of paths for your five python apps and then have that leak over and affect system tools. -Toshio -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Mon Mar 23 15:30:23 2015 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 23 Mar 2015 15:30:23 +0100 Subject: [Python-Dev] Use ptyhon -s as default shbang for system python executables/daemons References: <20150318122203.12ff1ee3@limelight.wooz.org> <20150318214431.GA4050@roan.lan> <20150318175607.66a25ad3@limelight.wooz.org> <20150319152734.5bb6cd79@limelight.wooz.org> Message-ID: <20150323153023.6fa0466e@fsol> On Mon, 23 Mar 2015 07:22:56 -0700 Toshio Kuratomi wrote: > > Building off Nick's idea of a system python vs a python for users to use, I > would see a more useful modification to be able to specify SPYTHONPATH (and > other env vars) to go along with /usr/bin/spython . That way the user > maintains the capability to override specific libraries globally just like > with LD_LIBRARY_PATH, PATH, and similar but you won't accidentally > configure your own python to use one set of paths for your five python apps > and then have that leak over and affect system tools. I really think Donald has a good point when he suggests a specific virtualenv for system programs using Python. Regards Antoine. From lkb.teichmann at gmail.com Mon Mar 23 10:38:30 2015 From: lkb.teichmann at gmail.com (Martin Teichmann) Date: Mon, 23 Mar 2015 10:38:30 +0100 Subject: [Python-Dev] super() does not work during class initialization In-Reply-To: References: Message-ID: > For folks that haven't looked at the tracker issue: I personally like > the change, but it does involve storing the cell object in a > dunder-variable in the class namespace while it's being defined (until > type.__new__ executes and both populates it and removes it from the > class namespace). I think I had the same weird feelings like you when writing the code. It feels a bit dodgy to abuse the class namespace to transport information from the compiler to type.__new__. Only when I saw that __qualname__ already does it, I decided it was probably not such a bad idea. The current implementation, setting the __class__ cell at the end of __build_class__ doesn't feel dodgy to me, it simply feels wrong. Nothing of real importance should happen there, as it is just an implementation detail. If we are fearing that we clutter namespace too much, we might call the entries @qualname and @cell, then we are sure they will never mask a user's class member. Most of my bad feelings actually only come from the fact that we don't know what we actually put our entries into, as __prepare__ might return something weird that doesn't do what we expect. Given Eric Snow's comment that class namespaces will be ordered by default soon anyways, we might deprecate __prepare__ altogether, eliminating most of the bad feelings. Speaking of which, instead of having OrderedDict re-implemented in C, maybe we could just change the compiler to leave the order in which things are defined in a class in the class namespace, say as a member __order__? Then we could use plain-old dicts for the class namespace, and we would not slow down class creation (not that it matters much), as determining the order would happen at compile time. > Since it introduces new behaviour that's visible to Python level code, > I've suggested that Martin roll the suggestion into his current PEP > 487 (which adds __init_subclass__ to similarly tidy up a few > challenges with the way classes are currently initialised). I thought about that, but while I stumbled over this issue while working on PEP 487, it is actually unrelated. I didn't want people to think that this change is needed to implement PEP 487. And while "and now for something completely different" might be a nice Monty Python reference, it is mostly frowned upon in PEPs as per PEP 1. Alternatively, I could write a new PEP. But I actually think that the change is not worth a PEP, because the changes will move the CPython implementation actually closer to what is documented, so I feel it is more a bugfix than an extension. I have no problem to pursue any of the above paths, how do other people feel about it? Greetings Martin From a.badger at gmail.com Mon Mar 23 16:06:13 2015 From: a.badger at gmail.com (Toshio Kuratomi) Date: Mon, 23 Mar 2015 08:06:13 -0700 Subject: [Python-Dev] Use ptyhon -s as default shbang for system python executables/daemons In-Reply-To: <20150323153023.6fa0466e@fsol> References: <20150318122203.12ff1ee3@limelight.wooz.org> <20150318214431.GA4050@roan.lan> <20150318175607.66a25ad3@limelight.wooz.org> <20150319152734.5bb6cd79@limelight.wooz.org> <20150323153023.6fa0466e@fsol> Message-ID: <20150323150613.GB4050@roan.lan> On Mon, Mar 23, 2015 at 03:30:23PM +0100, Antoine Pitrou wrote: > On Mon, 23 Mar 2015 07:22:56 -0700 > Toshio Kuratomi wrote: > > > > Building off Nick's idea of a system python vs a python for users to use, I > > would see a more useful modification to be able to specify SPYTHONPATH (and > > other env vars) to go along with /usr/bin/spython . That way the user > > maintains the capability to override specific libraries globally just like > > with LD_LIBRARY_PATH, PATH, and similar but you won't accidentally > > configure your own python to use one set of paths for your five python apps > > and then have that leak over and affect system tools. > > I really think Donald has a good point when he suggests a specific > virtualenv for system programs using Python. > The isolation is what we're seeking but I think the amount of work required and the added complexity for the distributions will make that hard to get distributions to sign up for. If someone had the time to write a front end to install packages into a single "system-wide isolation unit" whose backend was a virtualenv we might be able to get distributions on-board with using that. The front end would need to install software so that you can still invoke /usr/bin/system-application and "system-application" would take care of activating the virtualenv. It would need to be about as simple to build as the present python2 setup.py build/install with the flexibility in options that the distros need to install into FHS approved paths. Some things like man pages, locale files, config files, and possibly other data files might need to be installed outside of the virtualenv directory. Many setup.py's already punt on some of those, though, letting the user choose to install them manually. So this might be similar. It would need to be able to handle 32bit and 64bit versions of the same library installed on the same system. It would need to be able to handle different versions of the same library installed on the same system (as few of those as possible but it's an unfortunate cornercase that can't be entirely ignored even for just system packages). It would need a mode where it doesn't use the network at all; only operates with the packages and sources that are present on the box. And remember these two things: (1) we'd be asking the distros to do a tremendous amount of work changing their packages to install into a virtualenv instead of the python setup.py way that is well documented and everyone's been using for ages. it'll be a tough sell even with good tooling. (2) this theoretical front-end would have to appeal to the distro maintainers so there would have to be a lot of talk to understand what capabilities the distro maintainers would need from it. -Toshio -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 181 bytes Desc: not available URL: From solipsis at pitrou.net Mon Mar 23 16:14:52 2015 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 23 Mar 2015 16:14:52 +0100 Subject: [Python-Dev] Use ptyhon -s as default shbang for system python executables/daemons In-Reply-To: <20150323150613.GB4050@roan.lan> References: <20150318122203.12ff1ee3@limelight.wooz.org> <20150318214431.GA4050@roan.lan> <20150318175607.66a25ad3@limelight.wooz.org> <20150319152734.5bb6cd79@limelight.wooz.org> <20150323153023.6fa0466e@fsol> <20150323150613.GB4050@roan.lan> Message-ID: <20150323161452.01c7189c@fsol> On Mon, 23 Mar 2015 08:06:13 -0700 Toshio Kuratomi wrote: > > > > I really think Donald has a good point when he suggests a specific > > virtualenv for system programs using Python. > > > The isolation is what we're seeking but I think the amount of work required > and the added complexity for the distributions will make that hard to get > distributions to sign up for. > > If someone had the time to write a front end to install packages into > a single "system-wide isolation unit" whose backend was a virtualenv we > might be able to get distributions on-board with using that. I don't think we're asking distributions anything. We're suggesting a possible path, but it's not python-dev's job to dictate distributions how they should package Python. The virtualenv solution has the virtue that any improvement we might put in it to help system packagers would automatically benefit everyone. A specific "system Python" would not. > The front end would need to install software so that you can still invoke > /usr/bin/system-application and "system-application" would take care of > activating the virtualenv. It would need to be about as simple to build > as the present python2 setup.py build/install with the flexibility in > options that the distros need to install into FHS approved paths. Some > things like man pages, locale files, config files, and possibly other data > files might need to be installed outside of the virtualenv directory. Well, I don't understand what difference a virtualenv would make. Using a virtualenv amounts to invoking a different interpreter path. The rest of the filesystem (man pages locations, etc.) is still accessible in the same way. But I may miss something :-) Regards Antoine. From agriff at tin.it Mon Mar 23 18:59:10 2015 From: agriff at tin.it (Andrea Griffini) Date: Mon, 23 Mar 2015 18:59:10 +0100 Subject: [Python-Dev] --with-valgrind and --enable-shared Message-ID: Hello, does it have any sense for a linux distribution (arch to be specific) to provide default Python package compiled with valgrind support? I thought this flag was just about silencing false positives generated by valgrind (in other words a workaround for "bugs" of another software) and useful only when developing Python itself or C extensions. The same distribution also compiles by default to a shared library and this has a quite noticeable impact on performance on x64 (surprisingly for me) for CPU bound processing; in a few test cases I measured valgrind+shared Python running at 66% the speed of plain ./configure && make Python on my system. Is this setting reasonable for general users? If they are good defaults, why aren't them the default? Andrea Griffini -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Mon Mar 23 19:08:23 2015 From: brett at python.org (Brett Cannon) Date: Mon, 23 Mar 2015 18:08:23 +0000 Subject: [Python-Dev] --with-valgrind and --enable-shared In-Reply-To: References: Message-ID: On Mon, Mar 23, 2015 at 1:59 PM Andrea Griffini wrote: > Hello, > > does it have any sense for a linux distribution (arch to be specific) to > provide default Python package compiled with valgrind support? I thought > this flag was just about silencing false positives generated by valgrind > (in other words a workaround for "bugs" of another software) and useful > only when developing Python itself or C extensions. > It's not really our place to say if it makes sense for Arch to compile with valgrind flags turned on. It really depends on how they use Python in their Linux distribution and what their own goals are. > > The same distribution also compiles by default to a shared library and > this has a quite noticeable impact on performance on x64 (surprisingly for > me) for CPU bound processing; in a few test cases I measured > valgrind+shared Python running at 66% the speed of plain ./configure && > make Python on my system. Is this setting reasonable for general users? > Once again, it all depends on how Arch uses Python. > > If they are good defaults, why aren't them the default? > That's a question for Arch Linux and not us. -------------- next part -------------- An HTML attachment was scrubbed... URL: From agriff at tin.it Mon Mar 23 19:16:17 2015 From: agriff at tin.it (Andrea Griffini) Date: Mon, 23 Mar 2015 19:16:17 +0100 Subject: [Python-Dev] --with-valgrind and --enable-shared In-Reply-To: References: Message-ID: On Mon, Mar 23, 2015 at 7:08 PM, Brett Cannon wrote: > > > > It's not really our place to say if it makes sense for Arch to compile > with valgrind flags turned on. It really depends on how they use Python in > their Linux distribution and what their own goals are. > I already asked the package maintainers about this, I just wanted to know if my understanding about what --with-valgrind means is correct or if there are good reason to turn it on (except debugging Python). > > >> >> If they are good defaults, why aren't them the default? >> > > That's a question for Arch Linux and not us. > I probably didn't explain myself correctly: I was asking why they're not the default values for Python configure script... Andrea -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Mon Mar 23 19:23:51 2015 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 23 Mar 2015 19:23:51 +0100 Subject: [Python-Dev] --with-valgrind and --enable-shared References: Message-ID: <20150323192351.7ae5ce29@fsol> On Mon, 23 Mar 2015 19:16:17 +0100 Andrea Griffini wrote: > On Mon, Mar 23, 2015 at 7:08 PM, Brett Cannon wrote: > > > > It's not really our place to say if it makes sense for Arch to compile > > with valgrind flags turned on. It really depends on how they use Python in > > their Linux distribution and what their own goals are. > > > > I already asked the package maintainers about this, I just wanted to know > if my understanding about what --with-valgrind means is correct or if there > are good reason to turn it on (except debugging Python). I think your understanding is correct. It's the first time I hear of Python being compiled --with-valgrind in a Linux distribution. As for --enable-shared, it's more of a distro policy and isn't unheard of. The RedHat world (Mageia, etc.) enables it as well. Perhaps it makes updates easier if there are many installed programs embedding the Python interpreter. Regards Antoine. From njs at pobox.com Mon Mar 23 20:39:56 2015 From: njs at pobox.com (Nathaniel Smith) Date: Mon, 23 Mar 2015 12:39:56 -0700 Subject: [Python-Dev] Use ptyhon -s as default shbang for system python executables/daemons In-Reply-To: <20150323161452.01c7189c@fsol> References: <20150318122203.12ff1ee3@limelight.wooz.org> <20150318214431.GA4050@roan.lan> <20150318175607.66a25ad3@limelight.wooz.org> <20150319152734.5bb6cd79@limelight.wooz.org> <20150323153023.6fa0466e@fsol> <20150323150613.GB4050@roan.lan> <20150323161452.01c7189c@fsol> Message-ID: On Mar 23, 2015 8:15 AM, "Antoine Pitrou" wrote: > > On Mon, 23 Mar 2015 08:06:13 -0700 > Toshio Kuratomi wrote: > > > > > > I really think Donald has a good point when he suggests a specific > > > virtualenv for system programs using Python. > > > > > The isolation is what we're seeking but I think the amount of work required > > and the added complexity for the distributions will make that hard to get > > distributions to sign up for. > > > > If someone had the time to write a front end to install packages into > > a single "system-wide isolation unit" whose backend was a virtualenv we > > might be able to get distributions on-board with using that. > > I don't think we're asking distributions anything. We're suggesting a > possible path, but it's not python-dev's job to dictate distributions > how they should package Python. > > The virtualenv solution has the virtue that any improvement we might > put in it to help system packagers would automatically benefit everyone. > A specific "system Python" would not. > > > The front end would need to install software so that you can still invoke > > /usr/bin/system-application and "system-application" would take care of > > activating the virtualenv. It would need to be about as simple to build > > as the present python2 setup.py build/install with the flexibility in > > options that the distros need to install into FHS approved paths. Some > > things like man pages, locale files, config files, and possibly other data > > files might need to be installed outside of the virtualenv directory. > > Well, I don't understand what difference a virtualenv would make. > Using a virtualenv amounts to invoking a different interpreter path. > The rest of the filesystem (man pages locations, etc.) is still > accessible in the same way. But I may miss something :-) The main issue that jumps to my mind is that 'yum/apt-get install some-python-package' should install it into both the base python interpreter and the system virtualenv, but that 'sudo pip install some-python-package' should install into only the base interpreter but not the system virtualenv. (Even if those two commands are run in sequence with different versions of some-python-package.) This seems potentially complex. -n -------------- next part -------------- An HTML attachment was scrubbed... URL: From a.badger at gmail.com Mon Mar 23 20:47:07 2015 From: a.badger at gmail.com (Toshio Kuratomi) Date: Mon, 23 Mar 2015 12:47:07 -0700 Subject: [Python-Dev] Use ptyhon -s as default shbang for system python executables/daemons In-Reply-To: <20150323161452.01c7189c@fsol> References: <20150318214431.GA4050@roan.lan> <20150318175607.66a25ad3@limelight.wooz.org> <20150319152734.5bb6cd79@limelight.wooz.org> <20150323153023.6fa0466e@fsol> <20150323150613.GB4050@roan.lan> <20150323161452.01c7189c@fsol> Message-ID: <20150323194707.GC4050@roan.lan> On Mon, Mar 23, 2015 at 04:14:52PM +0100, Antoine Pitrou wrote: > On Mon, 23 Mar 2015 08:06:13 -0700 > Toshio Kuratomi wrote: > > > > > > I really think Donald has a good point when he suggests a specific > > > virtualenv for system programs using Python. > > > > > The isolation is what we're seeking but I think the amount of work required > > and the added complexity for the distributions will make that hard to get > > distributions to sign up for. > > > > If someone had the time to write a front end to install packages into > > a single "system-wide isolation unit" whose backend was a virtualenv we > > might be able to get distributions on-board with using that. > > I don't think we're asking distributions anything. We're suggesting a > possible path, but it's not python-dev's job to dictate distributions > how they should package Python. > > The virtualenv solution has the virtue that any improvement we might > put in it to help system packagers would automatically benefit everyone. > A specific "system Python" would not. > > > The front end would need to install software so that you can still invoke > > /usr/bin/system-application and "system-application" would take care of > > activating the virtualenv. It would need to be about as simple to build > > as the present python2 setup.py build/install with the flexibility in > > options that the distros need to install into FHS approved paths. Some > > things like man pages, locale files, config files, and possibly other data > > files might need to be installed outside of the virtualenv directory. > > Well, I don't understand what difference a virtualenv would make. > Using a virtualenv amounts to invoking a different interpreter path. > The rest of the filesystem (man pages locations, etc.) is still > accessible in the same way. But I may miss something :-) > I think people who are saying "The system should just use virtualenv" aren't realizing all of the reasons that's not as simple as it sounds for distributions to implement. thus the work required to implement alternate solutions like a system python may seem less to the distros unless those issues are partially addressed at the virtualenv and python-packaging level. -Toshio -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 181 bytes Desc: not available URL: From greg at krypto.org Mon Mar 23 22:58:06 2015 From: greg at krypto.org (Gregory P. Smith) Date: Mon, 23 Mar 2015 21:58:06 +0000 Subject: [Python-Dev] Use ptyhon -s as default shbang for system python executables/daemons In-Reply-To: <20150318124825.337dc1c6@anarchist.wooz.org> References: <20150318124825.337dc1c6@anarchist.wooz.org> Message-ID: On Wed, Mar 18, 2015 at 9:48 AM Barry Warsaw wrote: > On Mar 18, 2015, at 05:31 PM, Victor Stinner wrote: > > >Does it work to pass command line options to Python in the shebang? > > Yes, but only one "word", thus -Es or -I. > > We've often mused about whether it makes sense to have two Pythons on the > system. One for system scripts and another for users. System Python > ('/usr/bin/spython' perhaps) would be locked down and only extensible by > system packages. On Debuntu that might mean no /usr/local on sys.path. It > would also have a much more limited set of installed packages, i.e. only > those > needed to support system functionality. > I recommend this. It'd be nice to see a common OS distro actually do it. For a system-only Python lockdown you should remove distutils and pip to prevent anyone from easily installing anything that isn't a distro package into it. Possibly even removing its site-packages directory and sys.path entry all together (things your distro needs to install could mix directly with the stdlib packages) > /usr/bin/python2 and /usr/bin/python3 then would be user tools, with all > the > goodness they currently have. > > It's never gotten much farther than musings, but protecting the system > against > the weird things people install would be a good thing. OTOH, this feels a > lot > like virtual environments so maybe there's something useful to be mined > there. > While people sometimes suggest virtualenv as a solution for this. It isn't really the same thing. It isn't a hermetic clone of the original interpreter. It copies the main binary but symlinks back to much of the stdlib. So any existing virtualenv can be an out of date unsupported mix of both after the original interpreter is updated. This has caused users problems in the past with some minor version updates where an internal non-public API used between some binary and a stdlib module was updated as part of a bugfix. Suddenly they didn't match up for existing virtualenvs. virtualenv is an amazing hack that I promote to most anyone for their own applications use with the occasional known caveats (solvable by regurly rebuilding your virtualenvs)... But I wouldn't want to see it used for the core OS itself even though it sounds better at first glance. -gps -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Mon Mar 23 23:06:57 2015 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 23 Mar 2015 23:06:57 +0100 Subject: [Python-Dev] Use ptyhon -s as default shbang for system python executables/daemons References: <20150318124825.337dc1c6@anarchist.wooz.org> Message-ID: <20150323230657.294e3b6a@fsol> On Mon, 23 Mar 2015 21:58:06 +0000 "Gregory P. Smith" wrote: > > virtualenv is an amazing hack that I promote to most anyone for their own > applications use with the occasional known caveats (solvable by regurly > rebuilding your virtualenvs)... But I wouldn't want to see it used for the > core OS itself even though it sounds better at first glance. pyvenv is not a hack, it mostly creates a couple symlinks and adds a pip install into the environment. Regards Antoine. From cs at zip.com.au Mon Mar 23 23:10:30 2015 From: cs at zip.com.au (Cameron Simpson) Date: Tue, 24 Mar 2015 09:10:30 +1100 Subject: [Python-Dev] Use ptyhon -s as default shbang for system python executables/daemons In-Reply-To: References: Message-ID: <20150323221030.GA76846@cskk.homeip.net> On 23Mar2015 21:58, Gregory P. Smith wrote: >While people sometimes suggest virtualenv as a solution for this. It isn't >really the same thing. It isn't a hermetic clone of the original >interpreter. It copies the main binary but symlinks back to much of the >stdlib. Oh. I had thought a non-standalone venv arranged sys.path to fall back to the source interpreter. Clearly I have not paid attention. Cheers, Cameron Simpson Yes, sometimes Perl looks like line-noise to the uninitiated, but to the seasoned Perl programmer, it looks like checksummed line-noise with a mission in life. - The Llama Book From storchaka at gmail.com Mon Mar 23 23:29:38 2015 From: storchaka at gmail.com (Serhiy Storchaka) Date: Tue, 24 Mar 2015 00:29:38 +0200 Subject: [Python-Dev] cpython: #23657 Don't explicitly do an isinstance check for str in zipapp In-Reply-To: <20150322153314.70038.25460@psf.io> References: <20150322153314.70038.25460@psf.io> Message-ID: On 22.03.15 17:33, paul.moore wrote: > https://hg.python.org/cpython/rev/0b2993742650 > changeset: 95126:0b2993742650 > user: Paul Moore > date: Sun Mar 22 15:32:36 2015 +0000 > summary: > #23657 Don't explicitly do an isinstance check for str in zipapp > > As a result, explicitly support pathlib.Path objects as arguments. > Also added tests for the CLI interface. Congratulate with your first commit Paul! From p.f.moore at gmail.com Mon Mar 23 23:33:53 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Mon, 23 Mar 2015 22:33:53 +0000 Subject: [Python-Dev] cpython: #23657 Don't explicitly do an isinstance check for str in zipapp In-Reply-To: References: <20150322153314.70038.25460@psf.io> Message-ID: On 23 March 2015 at 22:29, Serhiy Storchaka wrote: >> As a result, explicitly support pathlib.Path objects as arguments. >> Also added tests for the CLI interface. > > Congratulate with your first commit Paul! Thanks :-) Paul From greg.ewing at canterbury.ac.nz Mon Mar 23 23:22:08 2015 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Tue, 24 Mar 2015 11:22:08 +1300 Subject: [Python-Dev] super() does not work during class initialization In-Reply-To: References: Message-ID: <55109210.9020901@canterbury.ac.nz> Martin Teichmann wrote: > maybe > we could just change the compiler to leave the order in which things are defined > in a class in the class namespace, say as a member __order__? Then we could > use plain-old dicts for the class namespace, and we would not slow down class > creation (not that it matters much), as determining the order would happen at > compile time. I don't think the compiler can determine the order in all cases. Consider: class Spam: if moon_is_full: alpha = 1 beta = 2 else: beta = 2 alpha = 1 -- Greg From kjp at uchicago.edu Tue Mar 24 15:28:52 2015 From: kjp at uchicago.edu (Karl Pickett) Date: Tue, 24 Mar 2015 09:28:52 -0500 Subject: [Python-Dev] How is obmalloc safe with "Invalid read of size 4" ? Message-ID: We are having random, rare, nonreproducible segfaults/hangs with python2 on ubuntu 14.04 in EC2. I've managed to attach GDB to some hung ones and there looks like clear memory corruption in the 'interned' hash table, causing lookdict_string() to spin forever because all remaining slots have a garbage 'key' pointer. This happens just loading the 'site' module dependencies, like 're' or 'codecs', before any of our code even gets run. So we then tried running it under valgrind, and we got a lot of nasty errors. Even after reading the Misc/README.valgrind, which talks about *uninitialized* reads being ok, I still don't see how reading from *freed* memory would ever be safe, and why the suppression file thinks thats ok: $ valgrind ./pymd79/bin/python -c "" ==14651== Memcheck, a memory error detector ==14651== Copyright (C) 2002-2013, and GNU GPL'd, by Julian Seward et al. ==14651== Using Valgrind-3.9.0 and LibVEX; rerun with -h for copyright info ==14651== Command: ./pymd79/bin/python -c ==14651== ==14651== Invalid read of size 4 ==14651== at 0x461E40: Py_ADDRESS_IN_RANGE (obmalloc.c:1911) ==14651== by 0x461EA3: PyObject_Free (obmalloc.c:994) ==14651== by 0x4789AB: tupledealloc (tupleobject.c:235) ==14651== by 0x5225BA: code_dealloc (codeobject.c:309) ==14651== by 0x4CFFC3: load_source_module (import.c:1100) ==14651== by 0x4D0E16: import_submodule (import.c:2700) ==14651== by 0x4D1E19: PyImport_ImportModuleLevel (import.c:2515) ==14651== by 0x4AE49A: builtin___import__ (bltinmodule.c:49) ==14651== by 0x422C89: PyObject_Call (abstract.c:2529) ==14651== by 0x4B12E5: PyEval_EvalFrameEx (ceval.c:3902) ==14651== by 0x4B6A47: PyEval_EvalCodeEx (ceval.c:3265) ==14651== by 0x4B6B71: PyEval_EvalCode (ceval.c:667) ==14651== Address 0x5bcd020 is 2,256 bytes inside a block of size 2,801 free'd ==14651== at 0x4C28577: free (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so) ==14651== by 0x4DB2B0: PyMarshal_ReadLastObjectFromFile (marshal.c:1145) ==14651== by 0x4CFE71: load_source_module (import.c:801) - Karl -------------- next part -------------- An HTML attachment was scrubbed... URL: From hrvoje.niksic at avl.com Tue Mar 24 16:22:31 2015 From: hrvoje.niksic at avl.com (Hrvoje Niksic) Date: Tue, 24 Mar 2015 16:22:31 +0100 Subject: [Python-Dev] How is obmalloc safe with "Invalid read of size 4" ? In-Reply-To: References: Message-ID: <55118137.60108@avl.com> On 03/24/2015 03:28 PM, Karl Pickett wrote: > So we then tried running it under valgrind, and we got a lot of nasty > errors. Even after reading the Misc/README.valgrind, which talks about > *uninitialized* reads being ok, I still don't see how reading from > *freed* memory would ever be safe, and why the suppression file thinks > thats ok: PyObject_Free() is not reading *freed* memory, it is reading memory outside (right before) the allocated range. This is, of course, undefined behavior as far as C is concerned and an invalid read in the eyes of valgrind. Still, it's the kind of thing you can get away with if you are writing a heavily optimized allocator (and if your name starts with "Tim" and ends with "Peters"). README.valgrind explains in quite some detail why this is done. In short, it allows for a very fast check whether the memory passed to PyObject_Free() was originally allocated by system malloc or by Python's pool allocator. From sunfish7 at gmail.com Tue Mar 24 16:53:18 2015 From: sunfish7 at gmail.com (=?utf-8?B?z4A=?=) Date: Tue, 24 Mar 2015 15:53:18 +0000 Subject: [Python-Dev] PiCxx Message-ID: <2E1837FF-9A40-40E6-BB27-FB57BA9D8E9C@gmail.com> Hello PythonDevvers, I apologise in advance. This is slightly off topic. This will be my only post on the subject. PyCXX (http://sourceforge.net/projects/cxx/ ) accomplishes roughly the same as Boost::Python (C++ wrapper for CPython), only without requiring Boost. The original code has become tangled and bloated. Over the winter I've rewritten it using C++11. New C++11 features have allowed for more direct/compact/concise/readable code. I've called my rewrite PiCxx and put it up here: https://github.com/p-i-/PiCxx PiCxx only supports Python3 currently, as that is all I need for my own use. It wouldn't be too much work to add 2.x support. Also I've only tested it on my OSX system (although it is designed to be cross-platform). It offers an alternative to Boost::Python that doesn't require Boost. Improvements, suggestions, bug fixes, etc are all welcome. Well, that's all. I originally subscribed to this list because I thought I might need some help navigating some of the dark corners of the CPython API, but thanks to a handful of Samurai on SO and IRC I seem to have scraped through. I will unsubscribe in due course; it's been charming to spend awhile in the belly of the snake, and humbling to witness how hard you guys work. Happy spring everyone, ? -------------- next part -------------- An HTML attachment was scrubbed... URL: From tjreedy at udel.edu Tue Mar 24 20:17:17 2015 From: tjreedy at udel.edu (Terry Reedy) Date: Tue, 24 Mar 2015 15:17:17 -0400 Subject: [Python-Dev] PiCxx In-Reply-To: <2E1837FF-9A40-40E6-BB27-FB57BA9D8E9C@gmail.com> References: <2E1837FF-9A40-40E6-BB27-FB57BA9D8E9C@gmail.com> Message-ID: On 3/24/2015 11:53 AM, ? wrote: > Hello PythonDevvers, > > I apologise in advance. This is slightly off topic. This will be my only > post on the subject. pydev is about development *of* Python the language and CPython the implementation. python-list is about development *with* Python. This post would be on-topic for python-list and would have fit better there. > PyCXX (http://sourceforge.net/projects/cxx/) accomplishes roughly the > same as Boost::Python (C++ wrapper for CPython), only without requiring > Boost. > > The original code has become tangled and bloated. > Over the winter I've rewritten it using C++11. New C++11 features have > allowed for more direct/compact/concise/readable code. > > I've called my rewrite PiCxx and put it up here: > https://github.com/p-i-/PiCxx > Improvements, suggestions, bug fixes, etc are all welcome. Announcements and requests like this are fairly routine on python-list. Sometimes they get responses and discussion on the list, sometimes not. One can participate on python-list via news/web mirror gmane.comp.python.general at news.gmane.org. -- Terry Jan Reedy From p.f.moore at gmail.com Tue Mar 24 22:31:29 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 24 Mar 2015 21:31:29 +0000 Subject: [Python-Dev] libffi embedded in CPython In-Reply-To: References: <549891df.c332e00a.54c7.34f8@mx.google.com> Message-ID: On 12 March 2015 at 17:44, Paul Moore wrote: > On 12 March 2015 at 17:26, Brett Cannon wrote: >>> I'm all for ditching our 'libffi_msvc' in favor of adding libffi as >>> another 'external' for the Windows build. I have managed to get >>> _ctypes to build on Windows using vanilla libffi sources, prepared >>> using their configure script from within Git Bash and built with our >>> usual Windows build system (properly patched). Unfortunately, making >>> things usable will take some work on ctypes itself, which I'm not >>> qualified to do. I'm happy to pass on my procedure and patches for >>> getting to the point of successful compilation to anyone who feels up >>> to fixing the things that are broken. >> >> >> So it seems possible to use upstream libffi but will require some work. > > I'd be willing to contemplate helping out on the Windows side of > things, if nobody else steps up (with the proviso that I have little > free time, and I'm saying this without much idea of what's involved > :-)) If Zachary can give a bit more detail on what the work on ctypes > is, and/or put what he has somewhere that I could have a look at, that > might help. One thing that seems to be an issue. On Windows, ctypes detects if the FFI call used the wrong number of arguments off the stack, and raises a ValueError if it does. The tests rely on that behaviour. But it's based on ffi_call() returning a value, which upstream libffi doesn't do. As far as I can tell (not that the libffi docs are exactly comprehensive...) there's no way of getting that information from upstream libffi. What does Unix ctypes do when faced with a call being made with the wrong number of arguments? On Windows, using upstream libffi and omitting the existing check, it seems to crash the Python process, which obviously isn't good. But the test that fails is Windows-specific, and short of going through all the tests looking for one that checks passing the wrong number of arguments and isn't platform-specific, I don't know how Unix handles this. Can anyone on Unix tell me if a ctypes call with the wrong number of arguments returns ValueError on Unix? Something like strcmp() (with no args) should do as a test, I guess... If there's a way Unix handles this, I can see about replicating it on Windows. But if there isn't, I fear we could always need a patched libffi to maintain the interface we currently have... Thanks, Paul From fijall at gmail.com Wed Mar 25 09:05:52 2015 From: fijall at gmail.com (Maciej Fijalkowski) Date: Wed, 25 Mar 2015 10:05:52 +0200 Subject: [Python-Dev] libffi embedded in CPython In-Reply-To: References: <549891df.c332e00a.54c7.34f8@mx.google.com> Message-ID: On Tue, Mar 24, 2015 at 11:31 PM, Paul Moore wrote: > On 12 March 2015 at 17:44, Paul Moore wrote: >> On 12 March 2015 at 17:26, Brett Cannon wrote: >>>> I'm all for ditching our 'libffi_msvc' in favor of adding libffi as >>>> another 'external' for the Windows build. I have managed to get >>>> _ctypes to build on Windows using vanilla libffi sources, prepared >>>> using their configure script from within Git Bash and built with our >>>> usual Windows build system (properly patched). Unfortunately, making >>>> things usable will take some work on ctypes itself, which I'm not >>>> qualified to do. I'm happy to pass on my procedure and patches for >>>> getting to the point of successful compilation to anyone who feels up >>>> to fixing the things that are broken. >>> >>> >>> So it seems possible to use upstream libffi but will require some work. >> >> I'd be willing to contemplate helping out on the Windows side of >> things, if nobody else steps up (with the proviso that I have little >> free time, and I'm saying this without much idea of what's involved >> :-)) If Zachary can give a bit more detail on what the work on ctypes >> is, and/or put what he has somewhere that I could have a look at, that >> might help. > > One thing that seems to be an issue. On Windows, ctypes detects if the > FFI call used the wrong number of arguments off the stack, and raises > a ValueError if it does. The tests rely on that behaviour. But it's > based on ffi_call() returning a value, which upstream libffi doesn't > do. As far as I can tell (not that the libffi docs are exactly > comprehensive...) there's no way of getting that information from > upstream libffi. > > What does Unix ctypes do when faced with a call being made with the > wrong number of arguments? On Windows, using upstream libffi and > omitting the existing check, it seems to crash the Python process, > which obviously isn't good. But the test that fails is > Windows-specific, and short of going through all the tests looking for > one that checks passing the wrong number of arguments and isn't > platform-specific, I don't know how Unix handles this. > > Can anyone on Unix tell me if a ctypes call with the wrong number of > arguments returns ValueError on Unix? Something like strcmp() (with no > args) should do as a test, I guess... > > If there's a way Unix handles this, I can see about replicating it on > Windows. But if there isn't, I fear we could always need a patched > libffi to maintain the interface we currently have... > > Thanks, > Paul Linux crashes. The mechanism for detecting the number of arguments is only available on windows (note that this is a band-aid anyway, since if your arguments are of the wrong kind you segfault anyway). We do have two copies of libffi one for windows one for unix anyway, don't we? From p.f.moore at gmail.com Wed Mar 25 09:15:18 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 25 Mar 2015 08:15:18 +0000 Subject: [Python-Dev] libffi embedded in CPython In-Reply-To: References: <549891df.c332e00a.54c7.34f8@mx.google.com> Message-ID: On 25 March 2015 at 08:05, Maciej Fijalkowski wrote: > Linux crashes. The mechanism for detecting the number of arguments is > only available on windows (note that this is a band-aid anyway, since > if your arguments are of the wrong kind you segfault anyway). We do > have two copies of libffi one for windows one for unix anyway, don't > we? Yes, this is in relation to looking at whether we can just use the upstream libffi directly on Windows. It looks like we guarantee a ValueError for the wrong number of arguments on Windows and upstream libffi doesn't give a way to detect that, so there may be a backward compatibility issue in doing so. Thanks for checking for me. Paul From ncoghlan at gmail.com Wed Mar 25 09:54:28 2015 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 25 Mar 2015 18:54:28 +1000 Subject: [Python-Dev] Use ptyhon -s as default shbang for system python executables/daemons In-Reply-To: References: <20150318124825.337dc1c6@anarchist.wooz.org> Message-ID: On 24 Mar 2015 07:59, "Gregory P. Smith" wrote: > > > On Wed, Mar 18, 2015 at 9:48 AM Barry Warsaw wrote: >> >> On Mar 18, 2015, at 05:31 PM, Victor Stinner wrote: >> >> >Does it work to pass command line options to Python in the shebang? >> >> Yes, but only one "word", thus -Es or -I. >> >> We've often mused about whether it makes sense to have two Pythons on the >> system. One for system scripts and another for users. System Python >> ('/usr/bin/spython' perhaps) would be locked down and only extensible by >> system packages. On Debuntu that might mean no /usr/local on sys.path. It >> would also have a much more limited set of installed packages, i.e. only those >> needed to support system functionality. > > > I recommend this. It'd be nice to see a common OS distro actually do it. > > For a system-only Python lockdown you should remove distutils and pip to prevent anyone from easily installing anything that isn't a distro package into it. Possibly even removing its site-packages directory and sys.path entry all together (things your distro needs to install could mix directly with the stdlib packages) The hard part of this is the complexity in getpath.c and the current difficulty of testing the interaction of all the different configuration settings properly. Completely overriding the standard path setup isn't too bad, it's *partially* overriding it that's hard. That's really one of the main goals of the proposed startup refactoring in PEP 432 - separating things into distinct phases so the path configuration is easier to customise and so the current behaviour becomes easier to test. That's a fairly big project with high barriers to entry though, as even just wrapping your head around the current accumulated complexity in the way the default sys.path gets constructed is a time consuming task. That approach has the virtue of not being a "system Python" specific solution, it's a "make embedding the CPython runtime much easier" solution, with "system Python with different default settings" as the reference use case. >> /usr/bin/python2 and /usr/bin/python3 then would be user tools, with all the >> goodness they currently have. >> >> It's never gotten much farther than musings, but protecting the system against >> the weird things people install would be a good thing. OTOH, this feels a lot >> like virtual environments so maybe there's something useful to be mined there. For distro level work it would be closer to the environment modules used in high performance computing work (which are also the basis of softwarecollections.org) Where things get really messy is on the package management side. Either we end up with yet another copy of modules that are often already duplicated between the Python 2 & 3 stacks, or else the relevant user Python needs to be customised to also be able to see the system level libraries for things like Kerberos, interfacing with the package manager, dbus, etc. The former approach = "ugh, no, just no" as far as I'm concerned, while the latter would be a messy patch to try to carry given the current difficulty of customising and testing the startup sequence. > > > While people sometimes suggest virtualenv as a solution for this. It isn't really the same thing. It isn't a hermetic clone of the original interpreter. It copies the main binary but symlinks back to much of the stdlib. So any existing virtualenv can be an out of date unsupported mix of both after the original interpreter is updated. This has caused users problems in the past with some minor version updates where an internal non-public API used between some binary and a stdlib module was updated as part of a bugfix. Suddenly they didn't match up for existing virtualenvs. > > virtualenv is an amazing hack that I promote to most anyone for their own applications use with the occasional known caveats (solvable by regurly rebuilding your virtualenvs)... But I wouldn't want to see it used for the core OS itself even though it sounds better at first glance. Right, this is why Fedora/RHEL/CentOS have software collections instead, and those have their own "gotchas" that currently make them unsuitable for use as the main system Python (for example: the Fedora installer & package manager are both written in Python, so having the main system Python be an SCL would bring SCL support into the minimal package set, which we wouldn't want to do. The typical aim is to find ways to get things *out* of that set, not add new ones) Cheers, Nick. -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Wed Mar 25 10:09:10 2015 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 25 Mar 2015 10:09:10 +0100 Subject: [Python-Dev] libffi embedded in CPython References: <549891df.c332e00a.54c7.34f8@mx.google.com> Message-ID: <20150325100910.10e29a5c@fsol> On Wed, 25 Mar 2015 08:15:18 +0000 Paul Moore wrote: > On 25 March 2015 at 08:05, Maciej Fijalkowski wrote: > > Linux crashes. The mechanism for detecting the number of arguments is > > only available on windows (note that this is a band-aid anyway, since > > if your arguments are of the wrong kind you segfault anyway). We do > > have two copies of libffi one for windows one for unix anyway, don't > > we? > > Yes, this is in relation to looking at whether we can just use the > upstream libffi directly on Windows. It looks like we guarantee a > ValueError for the wrong number of arguments on Windows and upstream > libffi doesn't give a way to detect that, so there may be a backward > compatibility issue in doing so. I'm not sure we guarantee anything. In any case, it's only a small proportion of the kind of crashes you can get by messing the signature. Regards Antoine. From p.f.moore at gmail.com Wed Mar 25 10:22:01 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 25 Mar 2015 09:22:01 +0000 Subject: [Python-Dev] libffi embedded in CPython In-Reply-To: <20150325100910.10e29a5c@fsol> References: <549891df.c332e00a.54c7.34f8@mx.google.com> <20150325100910.10e29a5c@fsol> Message-ID: On 25 March 2015 at 09:09, Antoine Pitrou wrote: > I'm not sure we guarantee anything. In any case, it's only a small > proportion of the kind of crashes you can get by messing the signature. Fair point. I guess what I'm asking is, would it be OK to remove the code that checks for a stack size discrepancy and raises ValueError, and the tests that verify this behaviour, as part of switching to using upstream libffi directly? On a related note, is there any information available on how the "externals" repo is maintained? In particular, should things in there be exact copies of upstream, or is it OK to include extra data (in this case, the results of running "configure" for the Windows build)? It works for me either way, it's just a matter of how the build process would be structured and maintained. Thanks, Paul Paul From solipsis at pitrou.net Wed Mar 25 10:30:30 2015 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 25 Mar 2015 10:30:30 +0100 Subject: [Python-Dev] libffi embedded in CPython In-Reply-To: References: <549891df.c332e00a.54c7.34f8@mx.google.com> <20150325100910.10e29a5c@fsol> Message-ID: <20150325103030.22842700@fsol> On Wed, 25 Mar 2015 09:22:01 +0000 Paul Moore wrote: > On 25 March 2015 at 09:09, Antoine Pitrou wrote: > > I'm not sure we guarantee anything. In any case, it's only a small > > proportion of the kind of crashes you can get by messing the signature. > > Fair point. I guess what I'm asking is, would it be OK to remove the > code that checks for a stack size discrepancy and raises ValueError, > and the tests that verify this behaviour, as part of switching to > using upstream libffi directly? IMHO yes. Of course it's still a behaviour change, but relying on it sounds seriously broken. > On a related note, is there any information available on how the > "externals" repo is maintained? In particular, should things in there > be exact copies of upstream, or is it OK to include extra data (in > this case, the results of running "configure" for the Windows build)? > It works for me either way, it's just a matter of how the build > process would be structured and maintained. Zachary or Steve could probably answer you on that one. Regards Antoine. From ncoghlan at gmail.com Wed Mar 25 13:32:03 2015 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 25 Mar 2015 22:32:03 +1000 Subject: [Python-Dev] super() does not work during class initialization In-Reply-To: <55109210.9020901@canterbury.ac.nz> References: <55109210.9020901@canterbury.ac.nz> Message-ID: On 24 March 2015 at 08:22, Greg Ewing wrote: > Martin Teichmann wrote: >> >> maybe >> we could just change the compiler to leave the order in which things are >> defined >> in a class in the class namespace, say as a member __order__? Then we >> could >> use plain-old dicts for the class namespace, and we would not slow down >> class >> creation (not that it matters much), as determining the order would happen >> at >> compile time. > > > I don't think the compiler can determine the order in > all cases. Consider: > > class Spam: > > if moon_is_full: > alpha = 1 > beta = 2 > else: > beta = 2 > alpha = 1 This is also expected to work in class namespaces: locals()["alpha"] = 1 The language reference suggests it isn't, there's an open tracker issue I filed some time ago to suggest clarifying it but haven't found the time to actually sit down and come up with readable wording: http://bugs.python.org/issue17960 Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From zachary.ware at gmail.com Wed Mar 25 14:36:35 2015 From: zachary.ware at gmail.com (Zachary Ware) Date: Wed, 25 Mar 2015 08:36:35 -0500 Subject: [Python-Dev] libffi embedded in CPython In-Reply-To: References: <549891df.c332e00a.54c7.34f8@mx.google.com> <20150325100910.10e29a5c@fsol> Message-ID: On Mar 25, 2015 4:22 AM, "Paul Moore" wrote: > > On 25 March 2015 at 09:09, Antoine Pitrou wrote: > > I'm not sure we guarantee anything. In any case, it's only a small > > proportion of the kind of crashes you can get by messing the signature. > > Fair point. I guess what I'm asking is, would it be OK to remove the > code that checks for a stack size discrepancy and raises ValueError, > and the tests that verify this behaviour, as part of switching to > using upstream libffi directly? > > On a related note, is there any information available on how the > "externals" repo is maintained? In particular, should things in there > be exact copies of upstream, or is it OK to include extra data (in > this case, the results of running "configure" for the Windows build)? > It works for me either way, it's just a matter of how the build > process would be structured and maintained. Its not extremely clear how it's "supposed to be" done; look at the differences between how we handle OpenSSL and Tcl/Tk, for example. One way or the other, though, we will store the configure output so that our build doesn't depend on any more than it absolutely has to. -- Zach On a phone -------------- next part -------------- An HTML attachment was scrubbed... URL: From lkb.teichmann at gmail.com Wed Mar 25 15:29:24 2015 From: lkb.teichmann at gmail.com (Martin Teichmann) Date: Wed, 25 Mar 2015 15:29:24 +0100 Subject: [Python-Dev] super() does not work during class initialization In-Reply-To: References: <55109210.9020901@canterbury.ac.nz> Message-ID: >> I don't think the compiler can determine the order in >> all cases. Consider: >> >> class Spam: >> >> if moon_is_full: >> alpha = 1 >> beta = 2 >> else: >> beta = 2 >> alpha = 1 > > This is also expected to work in class namespaces: > > locals()["alpha"] = 1 > > The language reference suggests it isn't, there's an open tracker > issue I filed some time ago to suggest clarifying it but haven't found > the time to actually sit down and come up with readable wording: > http://bugs.python.org/issue17960 Well, for sure the compiler cannot deduce things happening at runtime, but it still has an order in which things appear at compile time. And for nearly all use cases that's by far enough. We cannot stop people from doing weird things, but unless there is a use case for those weird things, we don't need to support them. And I think that tampering with locals() is not really a good idea. In your issue you mention that you want that so that you can create enums programatically. This is already doable by writing: from enum import Enum from types import new_class def cb(ns): ns.update({"a{}".format(i): i for i in range(100)}) return ns Calc = new_class("Calc", (Enum,), None, cb) I think this is simple enough that we don't need another way of doing it. Btw, going on-topic again, what should I do to get my patch to make super() work during class initialization into python? Greetings Martin From Steve.Dower at microsoft.com Wed Mar 25 16:57:53 2015 From: Steve.Dower at microsoft.com (Steve Dower) Date: Wed, 25 Mar 2015 15:57:53 +0000 Subject: [Python-Dev] libffi embedded in CPython In-Reply-To: References: <549891df.c332e00a.54c7.34f8@mx.google.com> <20150325100910.10e29a5c@fsol> Message-ID: Zachary Ware wrote: > On Mar 25, 2015 4:22 AM, "Paul Moore" wrote: >> On a related note, is there any information available on how the >> "externals" repo is maintained? In particular, should things in there >> be exact copies of upstream, or is it OK to include extra data (in >> this case, the results of running "configure" for the Windows build)? >> It works for me either way, it's just a matter of how the build >> process would be structured and maintained. > > Its not extremely clear how it's "supposed to be" done; look at the differences > between how we handle OpenSSL and Tcl/Tk, for example. One way or the other, > though, we will store the configure output so that our build doesn't depend on > any more than it absolutely has to. It would be nice for at least one other person to know how to do it. I can't see the SVN logs, but I feel like you're the only person who's touched it in a while :) As well as the configure output, we sometimes include very light patching (for example, Tcl/Tk currently has a patch to their makefile to work around a VC14 bug). I'm not sure whether that's considered good practice or not, but I prefer only needing to do it once vs. building it into the build system. Cheers, Steve From ncoghlan at gmail.com Thu Mar 26 03:28:15 2015 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 26 Mar 2015 12:28:15 +1000 Subject: [Python-Dev] libffi embedded in CPython In-Reply-To: References: <549891df.c332e00a.54c7.34f8@mx.google.com> <20150325100910.10e29a5c@fsol> Message-ID: On 26 March 2015 at 01:57, Steve Dower wrote: > Zachary Ware wrote: >> On Mar 25, 2015 4:22 AM, "Paul Moore" wrote: >>> On a related note, is there any information available on how the >>> "externals" repo is maintained? In particular, should things in there >>> be exact copies of upstream, or is it OK to include extra data (in >>> this case, the results of running "configure" for the Windows build)? >>> It works for me either way, it's just a matter of how the build >>> process would be structured and maintained. >> >> Its not extremely clear how it's "supposed to be" done; look at the differences >> between how we handle OpenSSL and Tcl/Tk, for example. One way or the other, >> though, we will store the configure output so that our build doesn't depend on >> any more than it absolutely has to. > > It would be nice for at least one other person to know how to do it. I can't see the SVN logs, but I feel like you're the only person who's touched it in a while :) > > As well as the configure output, we sometimes include very light patching (for example, Tcl/Tk currently has a patch to their makefile to work around a VC14 bug). I'm not sure whether that's considered good practice or not, but I prefer only needing to do it once vs. building it into the build system. It likely makes sense to try to at least write down the status quo regarding the externals repo as a new top level section in the Developer's Guide. That way you'll at least have a shared understanding of the starting point for any further improvements. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From zachary.ware+pydev at gmail.com Thu Mar 26 03:40:08 2015 From: zachary.ware+pydev at gmail.com (Zachary Ware) Date: Wed, 25 Mar 2015 21:40:08 -0500 Subject: [Python-Dev] libffi embedded in CPython In-Reply-To: References: <549891df.c332e00a.54c7.34f8@mx.google.com> <20150325100910.10e29a5c@fsol> Message-ID: On Mar 25, 2015 9:28 PM, "Nick Coghlan" wrote: > > On 26 March 2015 at 01:57, Steve Dower wrote: > > Zachary Ware wrote: > >> On Mar 25, 2015 4:22 AM, "Paul Moore" wrote: > >>> On a related note, is there any information available on how the > >>> "externals" repo is maintained? In particular, should things in there > >>> be exact copies of upstream, or is it OK to include extra data (in > >>> this case, the results of running "configure" for the Windows build)? > >>> It works for me either way, it's just a matter of how the build > >>> process would be structured and maintained. > >> > >> Its not extremely clear how it's "supposed to be" done; look at the differences > >> between how we handle OpenSSL and Tcl/Tk, for example. One way or the other, > >> though, we will store the configure output so that our build doesn't depend on > >> any more than it absolutely has to. > > > > It would be nice for at least one other person to know how to do it. I can't see the SVN logs, but I feel like you're the only person who's touched it in a while :) > > > > As well as the configure output, we sometimes include very light patching (for example, Tcl/Tk currently has a patch to their makefile to work around a VC14 bug). I'm not sure whether that's considered good practice or not, but I prefer only needing to do it once vs. building it into the build system. > > It likely makes sense to try to at least write down the status quo > regarding the externals repo as a new top level section in the > Developer's Guide. That way you'll at least have a shared > understanding of the starting point for any further improvements. I'll try to get something thrown together at least by the time I leave PyCon. Unfortunately I can't guarantee anything sooner than that. -- Zach On a phone -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Thu Mar 26 04:18:53 2015 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 26 Mar 2015 13:18:53 +1000 Subject: [Python-Dev] libffi embedded in CPython In-Reply-To: References: <549891df.c332e00a.54c7.34f8@mx.google.com> <20150325100910.10e29a5c@fsol> Message-ID: On 26 March 2015 at 12:40, Zachary Ware wrote: > On Mar 25, 2015 9:28 PM, "Nick Coghlan" wrote: >> >> On 26 March 2015 at 01:57, Steve Dower wrote: >> > Zachary Ware wrote: >> >> On Mar 25, 2015 4:22 AM, "Paul Moore" wrote: >> >>> On a related note, is there any information available on how the >> >>> "externals" repo is maintained? In particular, should things in there >> >>> be exact copies of upstream, or is it OK to include extra data (in >> >>> this case, the results of running "configure" for the Windows build)? >> >>> It works for me either way, it's just a matter of how the build >> >>> process would be structured and maintained. >> >> >> >> Its not extremely clear how it's "supposed to be" done; look at the >> >> differences >> >> between how we handle OpenSSL and Tcl/Tk, for example. One way or the >> >> other, >> >> though, we will store the configure output so that our build doesn't >> >> depend on >> >> any more than it absolutely has to. >> > >> > It would be nice for at least one other person to know how to do it. I >> > can't see the SVN logs, but I feel like you're the only person who's touched >> > it in a while :) >> > >> > As well as the configure output, we sometimes include very light >> > patching (for example, Tcl/Tk currently has a patch to their makefile to >> > work around a VC14 bug). I'm not sure whether that's considered good >> > practice or not, but I prefer only needing to do it once vs. building it >> > into the build system. >> >> It likely makes sense to try to at least write down the status quo >> regarding the externals repo as a new top level section in the >> Developer's Guide. That way you'll at least have a shared >> understanding of the starting point for any further improvements. > > I'll try to get something thrown together at least by the time I leave > PyCon. Unfortunately I can't guarantee anything sooner than that. That sounds plenty soon enough to me - my own list of "I should really do something about that some day" items stretches back years, as the issue tracker occasionally reminds me :) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From storchaka at gmail.com Thu Mar 26 10:11:51 2015 From: storchaka at gmail.com (Serhiy Storchaka) Date: Thu, 26 Mar 2015 11:11:51 +0200 Subject: [Python-Dev] peps: New PEP 490: Chain exceptions at C level In-Reply-To: <20150326080820.24234.33840@psf.io> References: <20150326080820.24234.33840@psf.io> Message-ID: On 26.03.15 10:08, victor.stinner wrote: > https://hg.python.org/peps/rev/7daf3bfd9586 > changeset: 5741:7daf3bfd9586 > user: Victor Stinner > date: Thu Mar 26 09:08:08 2015 +0100 > summary: > New PEP 490: Chain exceptions at C level > +Python 3.5 introduces a new private ``_PyErr_ChainExceptions()`` function which > +is enough to chain manually exceptions. It also was added in Python 3.4.3. I meditar about adding _PyErr_ReplaceException() in 2.7 for simpler backporting patches from 3.x. > +Functions like ``PyErr_SetString()`` don't chain automatically exceptions. To > +make usage of ``_PyErr_ChainExceptions()`` easier, new functions are added: > + > +* PyErr_SetStringChain(exc_type, message) > +* PyErr_FormatChaine(exc_type, format, ...) Typo. > +* PyErr_SetNoneChain(exc_type) > +* PyErr_SetObjectChain(exc_type, exc_value) I would first make these functions private, as _PyErr_ChainExceptions(). After proofing their usefulness in the stdlib, they can be made public. From andrew.svetlov at gmail.com Thu Mar 26 11:52:14 2015 From: andrew.svetlov at gmail.com (Andrew Svetlov) Date: Thu, 26 Mar 2015 12:52:14 +0200 Subject: [Python-Dev] peps: New PEP 490: Chain exceptions at C level In-Reply-To: References: <20150326080820.24234.33840@psf.io> Message-ID: There is another issue: exception chain is not set up on exception creation in python code, only on throwing. Thus I have to assign `__context__` and `__cause__` attributes manually if I want to call `future.set_exception(exc)` instead of `raise exc`. See aiohttp code for usage example: https://github.com/KeepSafe/aiohttp/blob/931efbd518d0d99522d1cd36b43620657cd07304/aiohttp/client.py#L522 On Thu, Mar 26, 2015 at 11:11 AM, Serhiy Storchaka wrote: > On 26.03.15 10:08, victor.stinner wrote: >> >> https://hg.python.org/peps/rev/7daf3bfd9586 >> changeset: 5741:7daf3bfd9586 >> user: Victor Stinner >> date: Thu Mar 26 09:08:08 2015 +0100 >> summary: >> New PEP 490: Chain exceptions at C level > > >> +Python 3.5 introduces a new private ``_PyErr_ChainExceptions()`` function >> which >> +is enough to chain manually exceptions. > > > It also was added in Python 3.4.3. > > I meditar about adding _PyErr_ReplaceException() in 2.7 for simpler > backporting patches from 3.x. > >> +Functions like ``PyErr_SetString()`` don't chain automatically >> exceptions. To >> +make usage of ``_PyErr_ChainExceptions()`` easier, new functions are >> added: >> + >> +* PyErr_SetStringChain(exc_type, message) >> +* PyErr_FormatChaine(exc_type, format, ...) > > > Typo. > >> +* PyErr_SetNoneChain(exc_type) >> +* PyErr_SetObjectChain(exc_type, exc_value) > > > I would first make these functions private, as _PyErr_ChainExceptions(). > After proofing their usefulness in the stdlib, they can be made public. > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/andrew.svetlov%40gmail.com -- Thanks, Andrew Svetlov From victor.stinner at gmail.com Thu Mar 26 12:36:07 2015 From: victor.stinner at gmail.com (Victor Stinner) Date: Thu, 26 Mar 2015 12:36:07 +0100 Subject: [Python-Dev] peps: New PEP 490: Chain exceptions at C level In-Reply-To: References: <20150326080820.24234.33840@psf.io> Message-ID: 2015-03-26 11:52 GMT+01:00 Andrew Svetlov : > There is another issue: exception chain is not set up on exception > creation in python code, only on throwing. I'm not suprised of that. > Thus I have to assign `__context__` and `__cause__` attributes > manually if I want to call `future.set_exception(exc)` instead of > `raise exc`. Do you mean that we need an helper to make this task even simpler? Or do you suggest to set them automatically in the constructor? Victor From victor.stinner at gmail.com Thu Mar 26 13:30:18 2015 From: victor.stinner at gmail.com (Victor Stinner) Date: Thu, 26 Mar 2015 13:30:18 +0100 Subject: [Python-Dev] RFC: PEP 490 - Chain exceptions at C level Message-ID: Hi, Here is the first draft of my PEP to chain automatically exceptions at C level in Python 3.5. Plain text version of the PEP below. HTML version of the PEP: https://www.python.org/dev/peps/pep-0490/ It has already an implementation, the pyerr_chain.patch file attached to http://bugs.python.org/issue23763 The patch is very short. We may modify more functions to hide the previous exception when the new exception contains as much or more information. Keeping the previous exception waste memory, make reference cycles more likely and is useless when displaying the full exception chain. I wrote a huge patch modifying a lot of functions to explicitly *not* chain exceptions: pyerr_match_clear-2.patch of the issue #23763. I wrote the patch to show where exceptions will be chained with pyerr_chain.patch applied. But we might apply it if we want to keep exactly the same behaviour between Python 3.4 and 3.5 for C modules the stdlib. pyerr_match_clear-2.patch adds also checks on the type of the previous exception to not replace an "unexpected" exception. For example, we can replace an expected TypeError with a ValueError (without chaining them), but we should not raise a ValueError if we got a KeyboardInterrupt, a MemoryError or other unexpected exceptions. But such changes are unrelated to the PEP and may be done in a separated issue. The starting point of my PEP is my work on CPython done with the pyfailmalloc module. This module injects MemoryError. I fixed a ton of issues in the code handling errors in CPython (see the issue #18408). I noticed that Python didn't complain when exceptions are lost. I added many assertions to detect these mistakes. Recently, I modified Python 3.5 to raise an exception even in release mode (issue #23571) if a C function returns NULL without an exception set, or if a C function returns a result with an exception set. Victor PEP: 490 Title: Chain exceptions at C level Version: $Revision$ Last-Modified: $Date$ Author: Victor Stinner Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 25-March-2015 Python-Version: 3.5 Abstract ======== Chain exceptions at C level, as already done at Python level. Rationale ========= Python 3 introduced a new killer feature: exceptions are chained by default, PEP 3134. Example:: try: raise TypeError("err1") except TypeError: raise ValueError("err2") Output:: Traceback (most recent call last): File "test.py", line 2, in raise TypeError("err1") TypeError: err1 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "test.py", line 4, in raise ValueError("err2") ValueError: err2 Exceptions are chained by default in Python code, but not in extensions written in C. A new private ``_PyErr_ChainExceptions()`` function was introduced in Python 3.4.3 and 3.5 to chain exceptions. Currently, it must be called explicitly to chain exceptions and its usage is not trivial. Example of ``_PyErr_ChainExceptions()`` usage from the ``zipimport`` module to chain the previous ``OSError`` to a new ``ZipImportError`` exception:: PyObject *exc, *val, *tb; PyErr_Fetch(&exc, &val, &tb); PyErr_Format(ZipImportError, "can't open Zip file: %R", archive); _PyErr_ChainExceptions(exc, val, tb); This PEP proposes to also chain exceptions automatically at C level to stay consistent and give more information on failures to help debugging. The previous example becomes simply:: PyErr_Format(ZipImportError, "can't open Zip file: %R", archive); Proposal ======== Modify PyErr_*() functions to chain exceptions ---------------------------------------------- Modify C functions raising exceptions of the Python C API to automatically chain exceptions: modify ``PyErr_SetString()``, ``PyErr_Format()``, ``PyErr_SetNone()``, etc. Modify functions to not chain exceptions ---------------------------------------- Keeping the previous exception is not always interesting when the new exception contains information of the previous exception or even more information, especially when the two exceptions have the same type. Example of an useless exception chain with ``int(str)``:: TypeError: a bytes-like object is required, not 'type' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "", line 1, in TypeError: int() argument must be a string, a bytes-like object or a number, not 'type' The new ``TypeError`` exception contains more information than the previous exception. The previous exception should be hidden. The ``PyErr_Clear()`` function can be called to clear the current exception before raising a new exception, to not chain the current exception with a new exception. Modify functions to chain exceptions ------------------------------------ Some functions save and then restore the current exception. If a new exception is raised, the exception is currently displayed into sys.stderr or ignored depending on the function. Some of these functions should be modified to chain exceptions instead. Examples of function ignoring the new exception(s): * ``ptrace_enter_call()``: ignore exception * ``subprocess_fork_exec()``: ignore exception raised by enable_gc() * ``t_bootstrap()`` of the ``_thread`` module: ignore exception raised by trying to display the bootstrap function to ``sys.stderr`` * ``PyDict_GetItem()``, ``_PyDict_GetItem_KnownHash()``: ignore exception raised by looking for a key in the dictionary * ``_PyErr_TrySetFromCause()``: ignore exception * ``PyFrame_LocalsToFast()``: ignore exception raised by ``dict_to_map()`` * ``_PyObject_Dump()``: ignore exception. ``_PyObject_Dump()`` is used to debug, to inspect a running process, it should not modify the Python state. * ``Py_ReprLeave()``: ignore exception "because there is no way to report them" * ``type_dealloc()``: ignore exception raised by ``remove_all_subclasses()`` * ``PyObject_ClearWeakRefs()``: ignore exception? * ``call_exc_trace()``, ``call_trace_protected()``: ignore exception * ``remove_importlib_frames()``: ignore exception * ``do_mktuple()``, helper used by ``Py_BuildValue()`` for example: ignore exception? * ``flush_io()``: ignore exception * ``sys_write()``, ``sys_format()``: ignore exception * ``_PyTraceback_Add()``: ignore exception * ``PyTraceBack_Print()``: ignore exception Examples of function displaying the new exception to ``sys.stderr``: * ``atexit_callfuncs()``: display exceptions with ``PyErr_Display()`` and return the latest exception, the function calls multiple callbacks and only returns the latest exception * ``sock_dealloc()``: log the ``ResourceWarning`` exception with ``PyErr_WriteUnraisable()`` * ``slot_tp_del()``: display exception with ``PyErr_WriteUnraisable()`` * ``_PyGen_Finalize()``: display ``gen_close()`` exception with ``PyErr_WriteUnraisable()`` * ``slot_tp_finalize()``: display exception raised by the ``__del__()`` method with ``PyErr_WriteUnraisable()`` * ``PyErr_GivenExceptionMatches()``: display exception raised by ``PyType_IsSubtype()`` with ``PyErr_WriteUnraisable()`` Backward compatibility ====================== A side effect of chaining exceptions is that exceptions store traceback objects which store frame objects which store local variables. Local variables are kept alive by exceptions. A common issue is a reference cycle between local variables and exceptions: an exception is stored in a local variable and the frame indirectly stored in the exception. The cycle only impacts applications storing exceptions. The reference cycle can now be fixed with the new ``traceback.TracebackException`` object introduced in Python 3.5. It stores informations required to format a full textual traceback without storing local variables. The ``asyncio`` is impacted by the reference cycle issue. This module is also maintained outside Python standard library to release a version for Python 3.3. ``traceback.TracebackException`` will maybe be backported in a private ``asyncio`` module to fix reference cycle issues. Alternatives ============ No change --------- A new private ``_PyErr_ChainExceptions()`` function is enough to chain manually exceptions. Exceptions will only be chained explicitly where it makes sense. New helpers to chain exceptions ------------------------------- Functions like ``PyErr_SetString()`` don't chain automatically exceptions. To make the usage of ``_PyErr_ChainExceptions()`` easier, new private functions are added: * ``_PyErr_SetStringChain(exc_type, message)`` * ``_PyErr_FormatChain(exc_type, format, ...)`` * ``_PyErr_SetNoneChain(exc_type)`` * ``_PyErr_SetObjectChain(exc_type, exc_value)`` Helper functions to raise specific exceptions like ``_PyErr_SetKeyError(key)`` or ``PyErr_SetImportError(message, name, path)`` don't chain exceptions. The generic ``_PyErr_ChainExceptions(exc_type, exc_value, exc_tb)`` should be used to chain exceptions with these helper functions. Appendix ======== PEPs ---- * `PEP 3134 -- Exception Chaining and Embedded Tracebacks `_ (Python 3.0): new ``__context__`` and ``__cause__`` attributes for exceptions * `PEP 415 - Implement context suppression with exception attributes `_ (Python 3.3): ``raise exc from None`` * `PEP 409 - Suppressing exception context `_ (superseded by the PEP 415) Python C API ------------ The header file ``Include/pyerror.h`` declares functions related to exceptions. Functions raising exceptions: * ``PyErr_SetNone(exc_type)`` * ``PyErr_SetObject(exc_type, exc_value)`` * ``PyErr_SetString(exc_type, message)`` * ``PyErr_Format(exc, format, ...)`` Helpers to raise specific exceptions: * ``PyErr_BadArgument()`` * ``PyErr_BadInternalCall()`` * ``PyErr_NoMemory()`` * ``PyErr_SetFromErrno(exc)`` * ``PyErr_SetFromWindowsErr(err)`` * ``PyErr_SetImportError(message, name, path)`` * ``_PyErr_SetKeyError(key)`` * ``_PyErr_TrySetFromCause(prefix_format, ...)`` Manage the current exception: * ``PyErr_Clear()``: clear the current exception, like ``except: pass`` * ``PyErr_Fetch(exc_type, exc_value, exc_tb)`` * ``PyErr_Restore(exc_type, exc_value, exc_tb)`` * ``PyErr_GetExcInfo(exc_type, exc_value, exc_tb)`` * ``PyErr_SetExcInfo(exc_type, exc_value, exc_tb)`` Others function to handle exceptions: * ``PyErr_ExceptionMatches(exc)``: check to implement ``except exc: ...`` * ``PyErr_GivenExceptionMatches(exc1, exc2)`` * ``PyErr_NormalizeException(exc_type, exc_value, exc_tb)`` * ``_PyErr_ChainExceptions(exc_type, exc_value, exc_tb)`` Python Issues ------------- Chain exceptions: * `Issue #23763: Chain exceptions in C `_ * `Issue #23696: zipimport: chain ImportError to OSError `_ * `Issue #21715: Chaining exceptions at C level `_: added ``_PyErr_ChainExceptions()`` * `Issue #18488: sqlite: finalize() method of user function may be called with an exception set if a call to step() method failed `_ * `Issue #23781: Add private _PyErr_ReplaceException() in 2.7 `_ * `Issue #23782: Leak in _PyTraceback_Add `_ Changes preventing to loose exceptions: * `Issue #23571: Raise SystemError if a function returns a result with an exception set `_ * `Issue #18408: Fixes crashes found by pyfailmalloc `_ Copyright ========= This document has been placed in the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End: From victor.stinner at gmail.com Thu Mar 26 13:31:56 2015 From: victor.stinner at gmail.com (Victor Stinner) Date: Thu, 26 Mar 2015 13:31:56 +0100 Subject: [Python-Dev] peps: New PEP 490: Chain exceptions at C level In-Reply-To: References: <20150326080820.24234.33840@psf.io> Message-ID: Thanks Serhiy for your review, I modified the PEP. I just sent a first draft (including your suggestions) to python-dev. Victor From andrew.svetlov at gmail.com Thu Mar 26 13:45:54 2015 From: andrew.svetlov at gmail.com (Andrew Svetlov) Date: Thu, 26 Mar 2015 14:45:54 +0200 Subject: [Python-Dev] peps: New PEP 490: Chain exceptions at C level In-Reply-To: References: <20150326080820.24234.33840@psf.io> Message-ID: I think setting context in exception constructor would be fine. On Thu, Mar 26, 2015 at 1:36 PM, Victor Stinner wrote: > 2015-03-26 11:52 GMT+01:00 Andrew Svetlov : >> There is another issue: exception chain is not set up on exception >> creation in python code, only on throwing. > > I'm not suprised of that. > >> Thus I have to assign `__context__` and `__cause__` attributes >> manually if I want to call `future.set_exception(exc)` instead of >> `raise exc`. > > Do you mean that we need an helper to make this task even simpler? Or > do you suggest to set them automatically in the constructor? > > Victor -- Thanks, Andrew Svetlov From victor.stinner at gmail.com Fri Mar 27 01:16:37 2015 From: victor.stinner at gmail.com (Victor Stinner) Date: Fri, 27 Mar 2015 01:16:37 +0100 Subject: [Python-Dev] Needed reviews In-Reply-To: References: Message-ID: 2015-03-19 10:28 GMT+01:00 Serhiy Storchaka : > https://bugs.python.org/issue23681 > Have -b warn when directly comparing ints and bytes closed > https://bugs.python.org/issue23676 > Add support of UnicodeTranslateError in standard error handlers commented > https://bugs.python.org/issue23671 > string.Template doesn't work with the self keyword argument closed > https://bugs.python.org/issue23637 > Outputting warnings fails when file patch is not ASCII and message is > unicode on 2.7. commented > https://bugs.python.org/issue23622 > Deprecate unrecognized backslash+letter escapes in re closed > https://bugs.python.org/issue23611 > Pickle nested names (e.g. unbound methods) with protocols < 4 commented > https://bugs.python.org/issue23583 > IDLE: printing unicode subclasses broken (again) closed > https://bugs.python.org/issue23573 > Avoid redundant memory allocations in str.find and like closed and reopened, already commented > https://bugs.python.org/issue23509 > Speed up Counter operators reviewed > https://bugs.python.org/issue23502 > Tkinter doesn't support large integers (out of 32-bit range) closed (note: the title was different, "pprint: added support for mapping proxy") > https://bugs.python.org/issue23488 > Random objects twice as big as necessary on 64-bit builds reviewed > https://bugs.python.org/issue23466 > PEP 461: Inconsistency between str and bytes formatting of integers already reviewed > https://bugs.python.org/issue23419 > Faster default __reduce__ for classes without __init__ closed (rejected) > https://bugs.python.org/issue23290 > Faster set copying commented and reviewed I stop here for tonight. Victor From storchaka at gmail.com Fri Mar 27 09:56:40 2015 From: storchaka at gmail.com (Serhiy Storchaka) Date: Fri, 27 Mar 2015 10:56:40 +0200 Subject: [Python-Dev] Needed reviews In-Reply-To: References: Message-ID: On 27.03.15 02:16, Victor Stinner wrote: > 2015-03-19 10:28 GMT+01:00 Serhiy Storchaka : >> https://bugs.python.org/issue23502 >> Tkinter doesn't support large integers (out of 32-bit range) > > closed > (note: the title was different, "pprint: added support for mapping proxy") My fault. The correct issue is https://bugs.python.org/issue16840. > I stop here for tonight. Many thanks Victor! From status at bugs.python.org Fri Mar 27 18:08:22 2015 From: status at bugs.python.org (Python tracker) Date: Fri, 27 Mar 2015 18:08:22 +0100 (CET) Subject: [Python-Dev] Summary of Python tracker Issues Message-ID: <20150327170822.4130A566AE@psf.upfronthosting.co.za> ACTIVITY SUMMARY (2015-03-20 - 2015-03-27) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 4820 (+22) closed 30732 (+47) total 35552 (+69) Open issues with patches: 2248 Issues opened (55) ================== #23571: Raise SystemError if a function returns a result with an excep http://bugs.python.org/issue23571 reopened by berker.peksag #23573: Avoid redundant allocations in str.find and like http://bugs.python.org/issue23573 reopened by serhiy.storchaka #23720: __del__() order is broken since 3.4.0 http://bugs.python.org/issue23720 reopened by Alexey Kazantsev #23723: Provide a way to disable bytecode staleness checks http://bugs.python.org/issue23723 opened by brett.cannon #23725: update tempfile docs to say that TemporaryFile is secure http://bugs.python.org/issue23725 opened by zbysz #23726: Don't enable GC for classes that don't add new fields http://bugs.python.org/issue23726 opened by eltoder #23727: rfc2231 line continuations result in an additional newline http://bugs.python.org/issue23727 opened by Brian Peterson #23728: binascii.crc_hqx() can return negative integer http://bugs.python.org/issue23728 opened by serhiy.storchaka #23730: Document return value for ZipFile.extract() http://bugs.python.org/issue23730 opened by mjpieters #23731: Implement PEP 488 http://bugs.python.org/issue23731 opened by brett.cannon #23732: Update porting HOWTO about new -b semantics http://bugs.python.org/issue23732 opened by brett.cannon #23733: Update porting HOWTO for bytes interpolation http://bugs.python.org/issue23733 opened by brett.cannon #23734: zipimport should not check pyc timestamps against zipped py fi http://bugs.python.org/issue23734 opened by gregory.p.smith #23735: Readline not adjusting width after resize with 6.3 http://bugs.python.org/issue23735 opened by Carlos Pita #23738: Clarify documentation of positional-only default values http://bugs.python.org/issue23738 opened by vadmium #23739: locale.setlocale checks locale name for string incorrectly http://bugs.python.org/issue23739 opened by Saulius ??emaitaitis #23740: http.client request and send method have some datatype issues http://bugs.python.org/issue23740 opened by r.david.murray #23743: Python crashes upon exit if importing g++ compiled mod after i http://bugs.python.org/issue23743 opened by matham #23745: Exception when parsing an email using email.parser.BytesParser http://bugs.python.org/issue23745 opened by Elmer #23746: sysconfg.is_python_build() is buggy http://bugs.python.org/issue23746 opened by pythonhacker #23747: platform module exposes win32_ver function on posix systems http://bugs.python.org/issue23747 opened by pythonhacker #23749: asyncio missing wrap_socket http://bugs.python.org/issue23749 opened by gc #23750: Clarify difference between os.system/subprocess.call in sectio http://bugs.python.org/issue23750 opened by Andreas Sommer #23752: Cleanup io.FileIO http://bugs.python.org/issue23752 opened by haypo #23754: Add a new os.read_into() function to avoid memory copies http://bugs.python.org/issue23754 opened by haypo #23755: tempfile.NamedTemporaryFile should be able to toggle "delete" http://bugs.python.org/issue23755 opened by Stephen Gallagher #23756: Tighten definition of bytes-like objects http://bugs.python.org/issue23756 opened by vadmium #23758: Improve documenation about num_params in sqlite3 create_functi http://bugs.python.org/issue23758 opened by ced #23759: urllib.parse: make coap:// known http://bugs.python.org/issue23759 opened by chrysn #23760: Tkinter in Python 3.4 on Windows don't post internal clipboard http://bugs.python.org/issue23760 opened by Victor Korolkevich #23761: test_socket fails on Mac OSX 10.9.5 http://bugs.python.org/issue23761 opened by willingc #23762: python.org page on new-style classes should be updated http://bugs.python.org/issue23762 opened by joelthelion #23763: Chain exceptions in C http://bugs.python.org/issue23763 opened by haypo #23764: Reference inspect.Signature.bind from functools.wraps document http://bugs.python.org/issue23764 opened by productivememberofsociety666 #23765: Remove IsBadStringPtr calls in ctypes http://bugs.python.org/issue23765 opened by steve.dower #23767: Library and include paths not added when cross compiling on lo http://bugs.python.org/issue23767 opened by kernevil #23768: assert on getting the representation of a thread in atexit fun http://bugs.python.org/issue23768 opened by xdegaye #23769: valgrind reports leaks for test_zipimport http://bugs.python.org/issue23769 opened by rkuska #23770: Rework of exceptions are handled in the parser module (in vali http://bugs.python.org/issue23770 opened by haypo #23771: Timeouts on "x86 Ubuntu Shared 3.x" buildbot http://bugs.python.org/issue23771 opened by haypo #23772: pymonotonic() gone backward on "AMD64 Debian root 3.x" buildbo http://bugs.python.org/issue23772 opened by haypo #23773: importlib does not properly remove frames when invoking extern http://bugs.python.org/issue23773 opened by shiz #23774: Test failures when unable to write to install location http://bugs.python.org/issue23774 opened by steve.dower #23779: imaplib authenticate raises TypeError if authenticator tries t http://bugs.python.org/issue23779 opened by craigh #23780: Surprising behaviour when passing list to os.path.join. http://bugs.python.org/issue23780 opened by The Compiler #23781: Add private _PyErr_ReplaceException() in 2.7 http://bugs.python.org/issue23781 opened by serhiy.storchaka #23782: Leak in _PyTraceback_Add http://bugs.python.org/issue23782 opened by serhiy.storchaka #23783: Leak in PyObject_ClearWeakRefs http://bugs.python.org/issue23783 opened by serhiy.storchaka #23785: Leak in TextIOWrapper.tell() http://bugs.python.org/issue23785 opened by serhiy.storchaka #23786: test_unaligned_buffers (test.test_hash.HashEqualityTestCase) . http://bugs.python.org/issue23786 opened by petriborg #23787: sum() function docstring lists arguments incorrectly http://bugs.python.org/issue23787 opened by vsinitsyn #23788: test_urllib2_localnet.test_bad_address fails: OSError not rais http://bugs.python.org/issue23788 opened by raniere #23789: decimal.Decimal: __format__ behaviour http://bugs.python.org/issue23789 opened by MonsieurPaul #23790: When xdrlib.Packer().pack_string() fails, the Packer is corrup http://bugs.python.org/issue23790 opened by ostcar #23791: Identify Moved Lines with difflib http://bugs.python.org/issue23791 opened by bkiefer Most recent 15 issues with no replies (15) ========================================== #23791: Identify Moved Lines with difflib http://bugs.python.org/issue23791 #23779: imaplib authenticate raises TypeError if authenticator tries t http://bugs.python.org/issue23779 #23773: importlib does not properly remove frames when invoking extern http://bugs.python.org/issue23773 #23770: Rework of exceptions are handled in the parser module (in vali http://bugs.python.org/issue23770 #23768: assert on getting the representation of a thread in atexit fun http://bugs.python.org/issue23768 #23767: Library and include paths not added when cross compiling on lo http://bugs.python.org/issue23767 #23762: python.org page on new-style classes should be updated http://bugs.python.org/issue23762 #23760: Tkinter in Python 3.4 on Windows don't post internal clipboard http://bugs.python.org/issue23760 #23758: Improve documenation about num_params in sqlite3 create_functi http://bugs.python.org/issue23758 #23745: Exception when parsing an email using email.parser.BytesParser http://bugs.python.org/issue23745 #23734: zipimport should not check pyc timestamps against zipped py fi http://bugs.python.org/issue23734 #23733: Update porting HOWTO for bytes interpolation http://bugs.python.org/issue23733 #23732: Update porting HOWTO about new -b semantics http://bugs.python.org/issue23732 #23730: Document return value for ZipFile.extract() http://bugs.python.org/issue23730 #23728: binascii.crc_hqx() can return negative integer http://bugs.python.org/issue23728 Most recent 15 issues waiting for review (15) ============================================= #23790: When xdrlib.Packer().pack_string() fails, the Packer is corrup http://bugs.python.org/issue23790 #23786: test_unaligned_buffers (test.test_hash.HashEqualityTestCase) . http://bugs.python.org/issue23786 #23785: Leak in TextIOWrapper.tell() http://bugs.python.org/issue23785 #23783: Leak in PyObject_ClearWeakRefs http://bugs.python.org/issue23783 #23781: Add private _PyErr_ReplaceException() in 2.7 http://bugs.python.org/issue23781 #23780: Surprising behaviour when passing list to os.path.join. http://bugs.python.org/issue23780 #23779: imaplib authenticate raises TypeError if authenticator tries t http://bugs.python.org/issue23779 #23773: importlib does not properly remove frames when invoking extern http://bugs.python.org/issue23773 #23770: Rework of exceptions are handled in the parser module (in vali http://bugs.python.org/issue23770 #23767: Library and include paths not added when cross compiling on lo http://bugs.python.org/issue23767 #23763: Chain exceptions in C http://bugs.python.org/issue23763 #23758: Improve documenation about num_params in sqlite3 create_functi http://bugs.python.org/issue23758 #23752: Cleanup io.FileIO http://bugs.python.org/issue23752 #23738: Clarify documentation of positional-only default values http://bugs.python.org/issue23738 #23726: Don't enable GC for classes that don't add new fields http://bugs.python.org/issue23726 Top 10 most discussed issues (10) ================================= #22516: Windows Installer won't - even when using "just for me"option http://bugs.python.org/issue22516 32 msgs #23571: Raise SystemError if a function returns a result with an excep http://bugs.python.org/issue23571 19 msgs #23764: Reference inspect.Signature.bind from functools.wraps document http://bugs.python.org/issue23764 15 msgs #16328: win_add2path.py sets wrong user path http://bugs.python.org/issue16328 13 msgs #23786: test_unaligned_buffers (test.test_hash.HashEqualityTestCase) . http://bugs.python.org/issue23786 13 msgs #23763: Chain exceptions in C http://bugs.python.org/issue23763 12 msgs #23546: Windows, 'Edit withIDLE', and multiple installed versions http://bugs.python.org/issue23546 10 msgs #2211: Cookie.Morsel interface needs update http://bugs.python.org/issue2211 9 msgs #23529: Limit decompressed data when reading from LZMAFile and BZ2File http://bugs.python.org/issue23529 8 msgs #23720: __del__() order is broken since 3.4.0 http://bugs.python.org/issue23720 8 msgs Issues closed (48) ================== #4727: copyreg doesn't support keyword only arguments in __new__ http://bugs.python.org/issue4727 closed by serhiy.storchaka #11468: Improve unittest basic example in the doc http://bugs.python.org/issue11468 closed by ezio.melotti #17530: pprint could use line continuation for long bytes literals http://bugs.python.org/issue17530 closed by serhiy.storchaka #21526: Support new booleans in Tkinter http://bugs.python.org/issue21526 closed by serhiy.storchaka #21560: gzip.write changes trailer ISIZE field before type checking - http://bugs.python.org/issue21560 closed by serhiy.storchaka #21717: Exclusive mode for ZipFile and TarFile http://bugs.python.org/issue21717 closed by serhiy.storchaka #21802: Reader of BufferedRWPair is not closed if writer's close() fai http://bugs.python.org/issue21802 closed by serhiy.storchaka #22163: max_wbits set incorrectly to -zlib.MAX_WBITS in tarfile, shoul http://bugs.python.org/issue22163 closed by vadmium #22289: support.transient_internet() doesn't catch timeout on FTP test http://bugs.python.org/issue22289 closed by ned.deily #22351: NNTP constructor exception leaves socket for garbage collector http://bugs.python.org/issue22351 closed by serhiy.storchaka #22364: Improve some re error messages using regex for hints http://bugs.python.org/issue22364 closed by serhiy.storchaka #22687: horrible performance of textwrap.wrap() with a long word http://bugs.python.org/issue22687 closed by serhiy.storchaka #22832: Tweak parameter names for fcntl module http://bugs.python.org/issue22832 closed by serhiy.storchaka #22933: Misleading sentence in doc for shutil.move http://bugs.python.org/issue22933 closed by python-dev #23075: Mock backport in 2.7 relies on implementation defined behavior http://bugs.python.org/issue23075 closed by serhiy.storchaka #23252: Add support of writing to unseekable file in zipfile http://bugs.python.org/issue23252 closed by serhiy.storchaka #23314: Disabling CRT asserts in debug build http://bugs.python.org/issue23314 closed by steve.dower #23465: Implement PEP 486 - Make the Python Launcher aware of virtual http://bugs.python.org/issue23465 closed by steve.dower #23502: pprint: added support for mapping proxy http://bugs.python.org/issue23502 closed by serhiy.storchaka #23512: The list of built-in functions is not alphabetical on https:// http://bugs.python.org/issue23512 closed by ezio.melotti #23539: Content-length not set for HTTP methods expecting body when bo http://bugs.python.org/issue23539 closed by r.david.murray #23583: IDLE: printing unicode subclasses broken (again) http://bugs.python.org/issue23583 closed by serhiy.storchaka #23619: Python 3.5.0a2 installer fails on Windows http://bugs.python.org/issue23619 closed by steve.dower #23622: Deprecate unrecognized backslash+letter escapes in re http://bugs.python.org/issue23622 closed by serhiy.storchaka #23654: infinite loop in faulthandler._stack_overflow http://bugs.python.org/issue23654 closed by haypo #23657: Don't do isinstance checks in zipapp http://bugs.python.org/issue23657 closed by paul.moore #23665: Provide IDLE menu option to set command-line arguments http://bugs.python.org/issue23665 closed by terry.reedy #23671: string.Template doesn't work with the self keyword argument http://bugs.python.org/issue23671 closed by serhiy.storchaka #23688: unnecessary copying of memoryview in gzip.GzipFile.write? http://bugs.python.org/issue23688 closed by serhiy.storchaka #23704: Make deques a full MutableSequence by adding index(), insert() http://bugs.python.org/issue23704 closed by rhettinger #23705: Speed-up deque.__contains__ http://bugs.python.org/issue23705 closed by rhettinger #23724: Stack Trace should show argument value passed into a function, http://bugs.python.org/issue23724 closed by r.david.murray #23729: Import ElementTree documentation for namespaces and XPath http://bugs.python.org/issue23729 closed by rhettinger #23736: "make test" on clean py3 install on CentOS 6.2 - 2 tests fail http://bugs.python.org/issue23736 closed by r.david.murray #23737: spam http://bugs.python.org/issue23737 closed by ezio.melotti #23741: Small pprint refactoring http://bugs.python.org/issue23741 closed by serhiy.storchaka #23742: expandvars removes single quotes ( ' ) http://bugs.python.org/issue23742 closed by serhiy.storchaka #23744: Speed-up deque.__bool__ http://bugs.python.org/issue23744 closed by rhettinger #23748: platform._uname_cache is writeable http://bugs.python.org/issue23748 closed by haypo #23751: Allow reverse lookup (value-to-member) for Enum sublcasses dec http://bugs.python.org/issue23751 closed by mmuddasani #23753: Drop HAVE_FSTAT: require fstat() to compile/use Python http://bugs.python.org/issue23753 closed by haypo #23757: tuple function gives wrong answer when called on list subclass http://bugs.python.org/issue23757 closed by r.david.murray #23766: json.dumps: solidus ("/") should be escaped http://bugs.python.org/issue23766 closed by r.david.murray #23775: Fix pprint of OrderedDict http://bugs.python.org/issue23775 closed by serhiy.storchaka #23776: Don't use assert for checking arguments in pprint http://bugs.python.org/issue23776 closed by serhiy.storchaka #23777: argparse subcommand action can't be specified http://bugs.python.org/issue23777 closed by berker.peksag #23778: Add a visual distinction to multiline versionchanged blocks http://bugs.python.org/issue23778 closed by r.david.murray #23784: Reloading tokenize module leads to error http://bugs.python.org/issue23784 closed by serhiy.storchaka From victor.stinner at gmail.com Sat Mar 28 10:39:30 2015 From: victor.stinner at gmail.com (Victor Stinner) Date: Sat, 28 Mar 2015 10:39:30 +0100 Subject: [Python-Dev] Sporadic failures of test_subprocess and test_multiprocessing_spawn Message-ID: Hi, Can you please take a look at the following issue and try to reproduce it? http://bugs.python.org/issue23771 The following tests sometimes hang on "x86 Ubuntu Shared 3.x" and "AMD64 Debian root 3.x" buildbots: - test_notify_all() of test_multiprocessing_spawn - test_double_close_on_error() of test_subprocess - other sporadic failures of test_subprocess I'm quite sure that they are regressions, maybe related to the implementation of the PEP 475. In the middle of all PEP 475 changes, I changed some functions to release the GIL on I/O, it wasn't the case before. I may be related. Are you able to reproduce these issues? I'm unable to reproduce them on Fedora 21. Maybe they are more likely on Debian-like operating systems? Victor From victor.stinner at gmail.com Sat Mar 28 10:48:32 2015 From: victor.stinner at gmail.com (Victor Stinner) Date: Sat, 28 Mar 2015 10:48:32 +0100 Subject: [Python-Dev] MemoryError and other bugs on AMD64 OpenIndiana 3.x Message-ID: Hi, The buildbot AMD64 OpenIndiana 3.x is always red since at least 6 months. Almost always, tests fail because the buildbot has not enough memory. Well, it's fun to see how Python handles MemoryError (it's pretty good, there is no hard crash! my work on failmalloc was maybe useful ;-)), but it doesn't help to detect regressions. Would it be possible to enhance this buildbot? I copy this email to the owner of buildbot. But I also sent it to python-dev because MemoryError is not the only error, please take care of our buildbots! Thanks, Victor From victor.stinner at gmail.com Sat Mar 28 10:51:09 2015 From: victor.stinner at gmail.com (Victor Stinner) Date: Sat, 28 Mar 2015 10:51:09 +0100 Subject: [Python-Dev] Buildbot x86 XP-4 3.x doesn't compile anymore: drop it? Message-ID: Hi, The buildbot x86 XP-4 3.x doesn't compile anymore since 3 months or more (maybe when Steve upgraded the Visual Studio project to VS 2015? I don't know). Would it be possible to fix this buildbot, or to turn it off? By the way, do we seriously want to support Windows XP? I mean, *who* will maintain it (no me sorry!). I saw recent changes to explicitly *drop* support for Windows older than Visa (stop using GetTickCount, always call GetTickCount64, for time.monotonic). Victor From victor.stinner at gmail.com Sat Mar 28 10:53:50 2015 From: victor.stinner at gmail.com (Victor Stinner) Date: Sat, 28 Mar 2015 10:53:50 +0100 Subject: [Python-Dev] OpenBSD buildbot has many failures Message-ID: Hi, The OpenBSD buildbot always fail with the same failures since many months (ex: test_socket). Is someone interested to fix them? If no, would it be possible to turn it off to get a better view of regressions? Currently, they are too many red buildbots to quickly see regressions introduced by recent commits. http://buildbot.python.org/all/builders/x86%20OpenBSD%205.5%203.x Victor From victor.stinner at gmail.com Sat Mar 28 10:57:40 2015 From: victor.stinner at gmail.com (Victor Stinner) Date: Sat, 28 Mar 2015 10:57:40 +0100 Subject: [Python-Dev] Buildbot PPC64 AIX 3.x failures Message-ID: Hi, There are many failures on the AIX buildbot. Can someone try to fix them? Or would it be possible to turn off the buildbot to quickly see regressions? http://buildbot.python.org/all/builders/PPC64%20AIX%203.x/builds/3426/steps/test/logs/stdio The buildbot has not enough memory, and some tests are failing since more than 3 months. Victor From mail at timgolden.me.uk Sat Mar 28 11:19:16 2015 From: mail at timgolden.me.uk (Tim Golden) Date: Sat, 28 Mar 2015 10:19:16 +0000 Subject: [Python-Dev] Buildbot x86 XP-4 3.x doesn't compile anymore: drop it? In-Reply-To: References: Message-ID: <55168024.7030705@timgolden.me.uk> On 28/03/2015 09:51, Victor Stinner wrote: > Hi, > > The buildbot x86 XP-4 3.x doesn't compile anymore since 3 months or > more (maybe when Steve upgraded the Visual Studio project to VS 2015? > I don't know). > > Would it be possible to fix this buildbot, or to turn it off? > > By the way, do we seriously want to support Windows XP? I mean, *who* > will maintain it (no me sorry!). I saw recent changes to explicitly > *drop* support for Windows older than Visa (stop using GetTickCount, > always call GetTickCount64, for time.monotonic). Per PEP 11, we track Microsoft lifecycle support, so we no longer support WinXP on 3.5 and newer. We do support XP on 3.4 and older, including 2.7. At least, that's my understanding. Given that, the only XP buildbot which makes sense is one running VS 2008 (2.7, 3.1, 3.2) or VS 2010 (3.3, 3.4). I think -- and I'm responding here to the wider set of emails Victor has just posted re failing buildbots -- that the difficulty with red buildbots is finding someone at the intersection of understanding the buildbot platform and understanding the code which has just broken it. (Or at least, having the wherewithal to come to understand both). And that someone might well not be the person who's hosting the buildbot: at its minimum they run up a VM and install the OS and the required tools. It's can be a way of offering help to the Python community without even being involved in core development at all. I concur with what Victor's touched on: that buildbots are not useful if they're always red. But finding the right person to turn them green is not necessarily easy. For myself I'm quite willing for someone to ping me with: "there's a Windows buildbot gone red; can you have a look?" But I have precious little time, and if the failure requires me to garner an understanding of a deep-level code change, a version-specific issue with the VS compiler, and some odd interaction with VM on which the buildbot is running, I'm unlikely to have enough time or energy to invest in it. Willing-but-not-always-able-ly-yours, TJG From rosuav at gmail.com Sat Mar 28 11:10:17 2015 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 28 Mar 2015 21:10:17 +1100 Subject: [Python-Dev] Sporadic failures of test_subprocess and test_multiprocessing_spawn In-Reply-To: References: Message-ID: On Sat, Mar 28, 2015 at 8:39 PM, Victor Stinner wrote: > Are you able to reproduce these issues? I'm unable to reproduce them > on Fedora 21. Maybe they are more likely on Debian-like operating > systems? I just tried it on my Debian Wheezy AMD64, not running as root. (The same computer the buildbot runs on, but the host OS rather than the virtual machine.) Unable to reproduce. Ran a full 'make test' and still no problems. Am now trying it on the buildbot itself, to see if I can recreate it. ChrisA From victor.stinner at gmail.com Sat Mar 28 11:36:57 2015 From: victor.stinner at gmail.com (Victor Stinner) Date: Sat, 28 Mar 2015 11:36:57 +0100 Subject: [Python-Dev] Buildbot x86 XP-4 3.x doesn't compile anymore: drop it? In-Reply-To: <55168024.7030705@timgolden.me.uk> References: <55168024.7030705@timgolden.me.uk> Message-ID: Le samedi 28 mars 2015, Tim Golden a ?crit : > > For myself I'm quite willing for someone to ping me with: "there's a > Windows buildbot gone red; can you have a look?" But I have precious little > time, and if the failure requires me to garner an understanding of a > deep-level code change, a version-specific issue with the VS compiler, and > some odd interaction with VM on which the buildbot is running, I'm unlikely > to have enough time or energy to invest in it. > You don't need to monitor buildobts everydays. Last days I checked buildbots because I made chnages which behave differently on each platform. Usually, I try to check buildbots each month and I only try to fix obvious bugs and a few of the most annoying issues. The problem is more on platforms which less interesged developers like AIX, OpenBSD, old FreeBSD, etc. Victor -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Sat Mar 28 12:26:08 2015 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 28 Mar 2015 22:26:08 +1100 Subject: [Python-Dev] Sporadic failures of test_subprocess and test_multiprocessing_spawn In-Reply-To: References: Message-ID: On Sat, Mar 28, 2015 at 9:10 PM, Chris Angelico wrote: > On Sat, Mar 28, 2015 at 8:39 PM, Victor Stinner > wrote: >> Are you able to reproduce these issues? I'm unable to reproduce them >> on Fedora 21. Maybe they are more likely on Debian-like operating >> systems? > > I just tried it on my Debian Wheezy AMD64, not running as root. (The > same computer the buildbot runs on, but the host OS rather than the > virtual machine.) Unable to reproduce. Ran a full 'make test' and > still no problems. > > Am now trying it on the buildbot itself, to see if I can recreate it. It seems to be stalling out. I'm not sure exactly what's happening here. Running just that one file doesn't do it, but running the full test suite results in a stall. ChrisA From victor.stinner at gmail.com Sat Mar 28 13:50:07 2015 From: victor.stinner at gmail.com (Victor Stinner) Date: Sat, 28 Mar 2015 13:50:07 +0100 Subject: [Python-Dev] Sporadic failures of test_subprocess and test_multiprocessing_spawn In-Reply-To: References: Message-ID: Good, you are able to reproduce it. The next step is to identify the sequence of test to reproduce it. How do you run the test suite? Are you using -j1? Victor Le samedi 28 mars 2015, Chris Angelico a ?crit : > On Sat, Mar 28, 2015 at 9:10 PM, Chris Angelico > wrote: > > On Sat, Mar 28, 2015 at 8:39 PM, Victor Stinner > > > wrote: > >> Are you able to reproduce these issues? I'm unable to reproduce them > >> on Fedora 21. Maybe they are more likely on Debian-like operating > >> systems? > > > > I just tried it on my Debian Wheezy AMD64, not running as root. (The > > same computer the buildbot runs on, but the host OS rather than the > > virtual machine.) Unable to reproduce. Ran a full 'make test' and > > still no problems. > > > > Am now trying it on the buildbot itself, to see if I can recreate it. > > It seems to be stalling out. I'm not sure exactly what's happening > here. Running just that one file doesn't do it, but running the full > test suite results in a stall. > > ChrisA > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/victor.stinner%40gmail.com > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Sat Mar 28 14:21:34 2015 From: rosuav at gmail.com (Chris Angelico) Date: Sun, 29 Mar 2015 00:21:34 +1100 Subject: [Python-Dev] Sporadic failures of test_subprocess and test_multiprocessing_spawn In-Reply-To: References: Message-ID: On Sat, Mar 28, 2015 at 11:50 PM, Victor Stinner wrote: > Good, you are able to reproduce it. The next step is to identify the > sequence of test to reproduce it. How do you run the test suite? Are you > using -j1? I just ran 'make test'. Early in the output are these lines: ./python ./Tools/scripts/run_tests.py /root/build/python -W default -bb -E -R -W error::BytesWarning -m test -r -w -j 0 -u all,-largefile,-audio,-gui (The build was done in /root/build, fwiw) So it seems to be -j0. ChrisA From dje.gcc at gmail.com Sat Mar 28 16:28:54 2015 From: dje.gcc at gmail.com (David Edelsohn) Date: Sat, 28 Mar 2015 11:28:54 -0400 Subject: [Python-Dev] Buildbot PPC64 AIX 3.x failures In-Reply-To: References: Message-ID: On Sat, Mar 28, 2015 at 5:57 AM, Victor Stinner wrote: > Hi, > > There are many failures on the AIX buildbot. Can someone try to fix > them? Or would it be possible to turn off the buildbot to quickly see > regressions? > > http://buildbot.python.org/all/builders/PPC64%20AIX%203.x/builds/3426/steps/test/logs/stdio > > The buildbot has not enough memory, and some tests are failing since > more than 3 months. I have cleaned out the build directories and restarted the buildslave. There is more than enough memory and most memory process limits are unlimited. One of the tests presumably crashed in a bad way that prevented normal cleanup by the buildslave. - David From storchaka at gmail.com Sat Mar 28 17:40:37 2015 From: storchaka at gmail.com (Serhiy Storchaka) Date: Sat, 28 Mar 2015 18:40:37 +0200 Subject: [Python-Dev] Sporadic failures of test_subprocess and test_multiprocessing_spawn In-Reply-To: References: Message-ID: On 28.03.15 11:39, Victor Stinner wrote: > Can you please take a look at the following issue and try to reproduce it? > http://bugs.python.org/issue23771 > > The following tests sometimes hang on "x86 Ubuntu Shared 3.x" and > "AMD64 Debian root 3.x" buildbots: > > - test_notify_all() of test_multiprocessing_spawn > - test_double_close_on_error() of test_subprocess > - other sporadic failures of test_subprocess > > I'm quite sure that they are regressions, maybe related to the > implementation of the PEP 475. In the middle of all PEP 475 changes, I > changed some functions to release the GIL on I/O, it wasn't the case > before. I may be related. > > Are you able to reproduce these issues? I'm unable to reproduce them > on Fedora 21. Maybe they are more likely on Debian-like operating > systems? Just run tests with low memory limit. (ulimit -v 60000; ./python -m test.regrtest -uall -v test_multiprocessing_spawn;) test_io also hangs. From db3l.net at gmail.com Sat Mar 28 21:14:29 2015 From: db3l.net at gmail.com (David Bolen) Date: Sat, 28 Mar 2015 16:14:29 -0400 Subject: [Python-Dev] Buildbot x86 XP-4 3.x doesn't compile anymore: drop it? References: Message-ID: Victor Stinner writes: > Would it be possible to fix this buildbot, or to turn it off? (...) > By the way, do we seriously want to support Windows XP? I mean, *who* > will maintain it (no me sorry!). I saw recent changes to explicitly > *drop* support for Windows older than Visa (stop using GetTickCount, > always call GetTickCount64, for time.monotonic). Disabling the 3.x branch (or 3.5+ going forward) on the XP buildbot seems plausible to me, assuming the issue is unlikely to be fixed (or someone found with time to look into it). If, as Tim suggests in his response, 3.5+ need not be officially supported on XP anyway, it may not be worth much effort to address. At some point (such as switching to VS 2015), we aren't going to be able to compile newer releases under XP anyway. I'm assuming you aren't suggesting turning off the XP buildbot entirely, correct? As long as any fixes are still getting applied to 2.x or earlier 3.x releases, I do think having a buildbot covering those branches for XP would be useful, if only to catch possible issues in back-ported fixes. There's still an amazing number of XP systems in service. While I doubt very many of those running XP also need the latest 3.x release of Python, I expect there are quite a few active 2.x users. -- David From victor.stinner at gmail.com Sun Mar 29 00:13:55 2015 From: victor.stinner at gmail.com (Victor Stinner) Date: Sun, 29 Mar 2015 00:13:55 +0100 Subject: [Python-Dev] Buildbot x86 XP-4 3.x doesn't compile anymore: drop it? In-Reply-To: References: Message-ID: Le 28 mars 2015 21:15, "David Bolen" a ?crit : > I'm assuming you aren't suggesting turning off the XP buildbot > entirely, correct? I'm talking about the 3.x slave, so only python 3.5. Would it be possible to specify in What's New In Python 3.5 which Windows versions are no more supported? Victor -------------- next part -------------- An HTML attachment was scrubbed... URL: From Steve.Dower at microsoft.com Sun Mar 29 05:16:19 2015 From: Steve.Dower at microsoft.com (Steve Dower) Date: Sun, 29 Mar 2015 03:16:19 +0000 Subject: [Python-Dev] Buildbot x86 XP-4 3.x doesn't compile anymore: drop it? In-Reply-To: References: , Message-ID: We discussed that a while back and decided the PEP was sufficient, but I'm not entirely opposed to a reminder. Probably best to repurpose this buildbot for 2.7. David has a Windows 8.1 one that should be marked as stable for 3.x (though it's not quite stable yet... AFAIK it's the only machine with VS 2015 other than mine). Cheers, Steve Top-posted from my Windows Phone ________________________________ From: Victor Stinner Sent: ?3/?28/?2015 16:14 To: David Bolen Cc: python-dev at python.org Subject: Re: [Python-Dev] Buildbot x86 XP-4 3.x doesn't compile anymore: drop it? Le 28 mars 2015 21:15, "David Bolen" > a ?crit : > I'm assuming you aren't suggesting turning off the XP buildbot > entirely, correct? I'm talking about the 3.x slave, so only python 3.5. Would it be possible to specify in What's New In Python 3.5 which Windows versions are no more supported? Victor -------------- next part -------------- An HTML attachment was scrubbed... URL: From pmiscml at gmail.com Sun Mar 29 20:16:08 2015 From: pmiscml at gmail.com (Paul Sokolovsky) Date: Sun, 29 Mar 2015 21:16:08 +0300 Subject: [Python-Dev] Builtin functions are magically static methods? Message-ID: <20150329211608.13d3f2da@x230> Hello, I looked into porting Python3 codecs module to MicroPython and saw rather strange behavior, which is best illustrated with following testcase: ========== def foo(a): print("func:", a) import _codecs fun = _codecs.utf_8_encode #fun = hash #fun = str.upper #fun = foo class Bar: meth = fun print(fun) print(fun("foo")) b = Bar() print(b.meth("bar")) ========== Uncommenting either _codecs.utf_8_encode or hash (both builtin functions) produces 2 similar output lines, which in particular means that its possible to call a native function as (normal) object method, which then behaves as if it was a staticmethod - self is not passed to a native function. Using native object method in this manner produces error of self type mismatch (TypeError: descriptor 'upper' for 'str' objects doesn't apply to 'Bar' object). And using standard Python function expectedly produces error about argument number mismatch, because used as a method, function gets extra self argument. So the questions are: 1. How so, the native functions exhibit such magic behavior? Is it documented somewhere - I never read or heard about that (cannot say I read each and every word in Python reference docs, but read enough. As an example, https://docs.python.org/3/library/stdtypes.html#functions is rather short and mentions difference in implementation, not in meta-behavior). 2. The main question: how to easily and cleanly achieve the same behavior for standard Python functions? I'd think it's staticmethod(), but: >>> staticmethod(lambda:1)() Traceback (most recent call last): File "", line 1, in TypeError: 'staticmethod' object is not callable Surprise. (By "easily and cleanly" I mean without meta-programming tricks, like instead of real arguments accept "*args, **kwargs" and then munge args). Thanks, Paul mailto:pmiscml at gmail.com From jcea at jcea.es Sun Mar 29 21:03:46 2015 From: jcea at jcea.es (Jesus Cea) Date: Sun, 29 Mar 2015 21:03:46 +0200 Subject: [Python-Dev] MemoryError and other bugs on AMD64 OpenIndiana 3.x In-Reply-To: References: Message-ID: <55184C92.9050103@jcea.es> Sorry, I am currently traveling. Away from real computer until April 7th. On 28/03/15 10:48, Victor Stinner wrote: > Hi, > > The buildbot AMD64 OpenIndiana 3.x is always red since at least 6 > months. Almost always, tests fail because the buildbot has not enough > memory. Well, it's fun to see how Python handles MemoryError (it's > pretty good, there is no hard crash! my work on failmalloc was maybe > useful ;-)), but it doesn't help to detect regressions. > > Would it be possible to enhance this buildbot? Those buildbots are hosted in OpenIndiana infraestructure. I take care of the Solaris Zone (similar to docker but better :-)) but I can't interact with the "real" machine. Sometimes changes in the "real" machine had impacted us and nobody realized until some time later, when nobody remembers what was changed or who did it. I have contacted the machine manager and he has said to me that 16GB were taken in the "/tmp" directory by buildbot user. He has deleted them and everything should be fine now. Lets see. Anyway if buildbot is leaving garbage behind we should take care of it. Doing a daily cron for manual cleaning up seems a bit hacky :). I would be interested, too, in getting an email when one of my buildbots is red for more than, lets say, 4 days in a row. An email per day would be fine. BTW, buildbot pages are painfully slow. The manager had to "insist" and be patient to see , etc. How can that be improved?. > I copy this email to the owner of buildbot. But I also sent it to > python-dev because MemoryError is not the only error, please take care > of our buildbots! Thanks for keeping me in the loop. I hope everything is going green for a while now :) -- Jes?s Cea Avi?n _/_/ _/_/_/ _/_/_/ jcea at jcea.es - http://www.jcea.es/ _/_/ _/_/ _/_/ _/_/ _/_/ Twitter: @jcea _/_/ _/_/ _/_/_/_/_/ jabber / xmpp:jcea at jabber.org _/_/ _/_/ _/_/ _/_/ _/_/ "Things are not so easy" _/_/ _/_/ _/_/ _/_/ _/_/ _/_/ "My name is Dump, Core Dump" _/_/_/ _/_/_/ _/_/ _/_/ "El amor es poner tu felicidad en la felicidad de otro" - Leibniz -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 473 bytes Desc: OpenPGP digital signature URL: From mark at hotpy.org Sun Mar 29 22:35:53 2015 From: mark at hotpy.org (Mark Shannon) Date: Sun, 29 Mar 2015 21:35:53 +0100 Subject: [Python-Dev] Builtin functions are magically static methods? In-Reply-To: <20150329211608.13d3f2da@x230> References: <20150329211608.13d3f2da@x230> Message-ID: <55186229.70603@hotpy.org> On 29/03/15 19:16, Paul Sokolovsky wrote: > Hello, > > I looked into porting Python3 codecs module to MicroPython and saw > rather strange behavior, which is best illustrated with following > testcase: > > ========== > def foo(a): > print("func:", a) > > import _codecs > fun = _codecs.utf_8_encode > #fun = hash > #fun = str.upper > #fun = foo > > > class Bar: > meth = fun > > print(fun) > print(fun("foo")) > b = Bar() > print(b.meth("bar")) > ========== > > Uncommenting either _codecs.utf_8_encode or hash (both builtin > functions) produces 2 similar output lines, which in particular means > that its possible to call a native function as (normal) object method, > which then behaves as if it was a staticmethod - self is not passed to > a native function. > > Using native object method in this manner produces error of self type > mismatch (TypeError: descriptor 'upper' for 'str' objects doesn't apply > to 'Bar' object). > > And using standard Python function expectedly produces error about > argument number mismatch, because used as a method, function gets extra > self argument. > > So the questions are: > > 1. How so, the native functions exhibit such magic behavior? Is it > documented somewhere - I never read or heard about that (cannot say I > read each and every word in Python reference docs, but read enough. As > an example, https://docs.python.org/3/library/stdtypes.html#functions > is rather short and mentions difference in implementation, not in > meta-behavior). In fact the "magic" is exhibited by Python functions, not by builtin ones. Python functions are descriptors, builtin functions are not. > > 2. The main question: how to easily and cleanly achieve the same > behavior for standard Python functions? I'd think it's staticmethod(), > but: > Write your own "BuiltinFunction" class which has the desired properties, ie. it would be callable, but not a descriptor. Then write a "builtin_function" decorator to produce such an object from a function. The class and decorator could be the same object. Personally, I think that such a class (plus a builtin function type that behaved like a Python function) would be a useful addition to the standard library. Modules do get converted from Python to C and vice-versa. >>>> staticmethod(lambda:1)() > Traceback (most recent call last): > File "", line 1, in > TypeError: 'staticmethod' object is not callable > > Surprise. > > (By "easily and cleanly" I mean without meta-programming tricks, like > instead of real arguments accept "*args, **kwargs" and then munge args). > > > Thanks, > Paul mailto:pmiscml at gmail.com Cheers, Mark From victor.stinner at gmail.com Mon Mar 30 02:26:07 2015 From: victor.stinner at gmail.com (Victor Stinner) Date: Mon, 30 Mar 2015 02:26:07 +0200 Subject: [Python-Dev] Sporadic failures of test_subprocess and test_multiprocessing_spawn In-Reply-To: References: Message-ID: Hi, 2015-03-28 12:26 GMT+01:00 Chris Angelico : > It seems to be stalling out. I'm not sure exactly what's happening > here. Running just that one file doesn't do it, but running the full > test suite results in a stall. Ok, I reproduced the issue on David's buildbot. A (child) process was stuck in _Py_open(), function called from _close_open_fds_safe() which is called to run a child process. Calling _Py_open() is not safe here because the GIL may or may not be held. After fork, the status of the GIL is unclear. I fixed the issue by replacing _Py_open() with _Py_open_noraise(), this version doesn't use the GIL (it doesn't release the GIL to call open() and it doesn't raise an exception). https://hg.python.org/cpython/rev/2e1234208bab Thanks David for the buildbot. Victor From victor.stinner at gmail.com Mon Mar 30 02:31:45 2015 From: victor.stinner at gmail.com (Victor Stinner) Date: Mon, 30 Mar 2015 02:31:45 +0200 Subject: [Python-Dev] Sporadic failures of test_subprocess and test_multiprocessing_spawn In-Reply-To: References: Message-ID: Hi Serhiy, 2015-03-28 17:40 GMT+01:00 Serhiy Storchaka : > Just run tests with low memory limit. > > (ulimit -v 60000; ./python -m test.regrtest -uall -v > test_multiprocessing_spawn;) > > test_io also hangs. I confirm that some tests using threads hang under very low memory condition. At bootstrap, Python doesn't handle correctly all exceptions when starting a new thread. The "parent thread" waits until the "child thread" completes, which never occurs because the thread already completed. I don't think that it was my regression, it probably exists since Python 2 and maybe before. I'm not interested to touch this fragile part of Python. It's maybe easy to fix the issue, I don't know. Victor From storchaka at gmail.com Mon Mar 30 08:06:51 2015 From: storchaka at gmail.com (Serhiy Storchaka) Date: Mon, 30 Mar 2015 09:06:51 +0300 Subject: [Python-Dev] cpython: Issue #23752: When built from an existing file descriptor, io.FileIO() now only In-Reply-To: <20150330012250.1925.16638@psf.io> References: <20150330012250.1925.16638@psf.io> Message-ID: On 30.03.15 04:22, victor.stinner wrote: > https://hg.python.org/cpython/rev/bc2a22eaa0af > changeset: 95269:bc2a22eaa0af > user: Victor Stinner > date: Mon Mar 30 03:21:06 2015 +0200 > summary: > Issue #23752: When built from an existing file descriptor, io.FileIO() now only > calls fstat() once. Before fstat() was called twice, which was not necessary. > > files: > Misc/NEWS | 26 ++++++++++++++++++++++++++ > Modules/_io/fileio.c | 24 ------------------------ > 2 files changed, 26 insertions(+), 24 deletions(-) > > > diff --git a/Misc/NEWS b/Misc/NEWS > --- a/Misc/NEWS > +++ b/Misc/NEWS > @@ -2,6 +2,32 @@ > Python News > +++++++++++ > > +What's New in Python 3.5.0 alpha 4? > +=================================== Return time machine back Victor. Current version is 3.5.0a2+. From victor.stinner at gmail.com Mon Mar 30 09:30:02 2015 From: victor.stinner at gmail.com (Victor Stinner) Date: Mon, 30 Mar 2015 09:30:02 +0200 Subject: [Python-Dev] cpython: Issue #23752: When built from an existing file descriptor, io.FileIO() now only In-Reply-To: References: <20150330012250.1925.16638@psf.io> Message-ID: Hi, 2015-03-30 8:06 GMT+02:00 Serhiy Storchaka : >> +What's New in Python 3.5.0 alpha 4? >> +=================================== > > Return time machine back Victor. Current version is 3.5.0a2+. Larry Hastings and Ned Deily told me that Larry is releasing Python 3.5 alpha 3. You can see their repository at: https://hg.python.org/releasing/3.5/ According to the PEP 478 - Python 3.5 Release Schedule, we are in time: 3.5.0 alpha 3: March 29, 2015 Maybe Misc/NEWS was not updated when the alpha 2 was released? It's always a mess to maintain this file... We should generate it from Mercurial changelogs instead. So is it correct to use "alpha 4" in Misc/NEWS? Victor From larry at hastings.org Mon Mar 30 10:44:10 2015 From: larry at hastings.org (Larry Hastings) Date: Mon, 30 Mar 2015 01:44:10 -0700 Subject: [Python-Dev] cpython: Issue #23752: When built from an existing file descriptor, io.FileIO() now only In-Reply-To: References: <20150330012250.1925.16638@psf.io> Message-ID: <55190CDA.4070203@hastings.org> On 03/30/2015 12:30 AM, Victor Stinner wrote: > So is it correct to use "alpha 4" in Misc/NEWS? It's the release manager's responsibility to update the version number in Misc/NEWS. As I finish up 3.5.0a3, I will be changing it to say "What's New in Python 3.5.0 alpha 4?". And I will change the release number in patchlevel.h to say "3.5.0a3+". I don't know why it is this way, but that's the way it is, and I assume there's a good reason for it. Since you don't have the 3.5.0a3 changes yet, no, it's not correct for you to put "alpha 4" into Misc/NEWS. Doing so would trip me up a little. The release process is mostly automated, although there are manual steps where the RM is guided through the process. Fixing up the heading in Misc/NEWS is one such manual step. FWIW, I have in the past bungled a release so badly that I missed making this change. I was happy to discover someone else had fixed it for me. So I'm not saying "nobody but the RM should ever touch it". But maybe you should wait a day or two after the release comes out before you start up your text editor. Cheers, //arry/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From larry at hastings.org Mon Mar 30 10:46:38 2015 From: larry at hastings.org (Larry Hastings) Date: Mon, 30 Mar 2015 01:46:38 -0700 Subject: [Python-Dev] [RELEASED] Python 3.5.0a3 is now available Message-ID: <55190D6E.8030300@hastings.org> On behalf of the Python development community and the Python 3.5 release team, I'm thrilled to announce the availability of Python 3.5.0a3. Python 3.5.0a3 is the third alpha release of Python 3.5, which will be the next major release of Python. Python 3.5 is still under heavy development, and is far from complete. This is a preview release, and its use is not recommended for production settings. Two important notes for Windows users about Python 3.5.0a3: * If you have previously installed Python 3.5.0a1, you may need to manually uninstall it before installing Python 3.5.0a3 (issue23612). * If installing Python 3.5.0a3 as a non-privileged user, you may need to escalate to administrator privileges to install an update to your C runtime libraries. You can find Python 3.5.0a3 here: https://www.python.org/downloads/release/python-350a3/ Happy hacking, //arry/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From victor.stinner at gmail.com Mon Mar 30 10:57:50 2015 From: victor.stinner at gmail.com (Victor Stinner) Date: Mon, 30 Mar 2015 10:57:50 +0200 Subject: [Python-Dev] [python-committers] [RELEASED] Python 3.5.0a3 is now available In-Reply-To: <55190D6E.8030300@hastings.org> References: <55190D6E.8030300@hastings.org> Message-ID: Hi, 2015-03-30 10:46 GMT+02:00 Larry Hastings : > On behalf of the Python development community and the Python 3.5 release > team, I'm thrilled to announce the availability of Python 3.5.0a3. Python > 3.5.0a3 is the third alpha release of Python 3.5, which will be the next > major release of Python. Python 3.5 is still under heavy development, and > is far from complete. Between alpha2 and alpha3, the implementation of the PEP 471 (os.scandir) and 475 (EINTR) made progress. For scandir, os.walk() now uses os.scandir() internally instead of os.listdir() and so should be (much) faster. These PEPs are very dependent on the platform, so I'm interested of feedback if you get any bug related to these PEPs. For example, Robert Collins told me that fixtures test started to hang. The reason is that time.sleep() was modified to restart on EINTR and fixtures test used an exception handler which expected time.sleep() to return on signal. The signal handler only modified a flag without raising an exception. Robert quickly fixed his test. I started to document EINTR changes in the documentation of each function, and I added a note in the "Porting to Python 3.5" section of What's New in Python 3.5: https://docs.python.org/dev/whatsnew/3.5.html#porting-to-python-3-5 By the way, Python now always requires a 64-bit signed integer type to work. Python internal functions now store timestamps as a number of nanoseconds in a 64-bit integer. It looks to be the case on all buildbots. Victor From ncoghlan at gmail.com Mon Mar 30 15:11:42 2015 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 30 Mar 2015 23:11:42 +1000 Subject: [Python-Dev] RFC: PEP 490 - Chain exceptions at C level In-Reply-To: References: Message-ID: On 26 March 2015 at 22:30, Victor Stinner wrote: > Hi, > > Here is the first draft of my PEP to chain automatically exceptions at > C level in Python 3.5. Plain text version of the PEP below. HTML > version of the PEP: > > https://www.python.org/dev/peps/pep-0490/ > > It has already an implementation, the pyerr_chain.patch file attached to > http://bugs.python.org/issue23763 > > The patch is very short. I like the idea, but I think it would raise the priority of the proposed change to the display of chained exceptions in http://bugs.python.org/issue12535 That proposal suggests adding the line "" to the start of earlier tracebacks in the chain so you get an up front hint that a chained exception will be shown after the initial traceback. > Backward compatibility > ====================== > > A side effect of chaining exceptions is that exceptions store > traceback objects which store frame objects which store local > variables. Local variables are kept alive by exceptions. A common > issue is a reference cycle between local variables and exceptions: an > exception is stored in a local variable and the frame indirectly > stored in the exception. The cycle only impacts applications storing > exceptions. > > The reference cycle can now be fixed with the new > ``traceback.TracebackException`` object introduced in Python 3.5. It > stores informations required to format a full textual traceback without > storing local variables. Also potentially worth noting here is "traceback.clear_frames()", introduced in 3.4 to explicitly clear local variables from a traceback: https://docs.python.org/3/library/traceback.html#traceback.clear_frames PEP 442 also makes it more likely that cycles will be cleared properly, even if some of the objects involved have __del__ methods: https://www.python.org/dev/peps/pep-0442/ However, I'll grant that neither of those can be backported the way that traceback.TracebackException can be. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From aarmour at cipmail.org Tue Mar 31 17:13:25 2015 From: aarmour at cipmail.org (Alan Armour) Date: Tue, 31 Mar 2015 11:13:25 -0400 Subject: [Python-Dev] importante!! Message-ID: its french! lol I just wanted to see if you could, as well as making python able to have assembly written, you should totally use blender as your main IDE it would be so epic and do pyjackd, and i think pyaudio will write some stuff, and like pycoreaudio and like directpy for windows direct x oh having blender as the main ide would just be epic -------------- next part -------------- An HTML attachment was scrubbed... URL: From marky1991 at gmail.com Tue Mar 31 17:36:45 2015 From: marky1991 at gmail.com (Mark Young) Date: Tue, 31 Mar 2015 11:36:45 -0400 Subject: [Python-Dev] importante!! In-Reply-To: References: Message-ID: Features of blender are relevant on blender mailing lists, not here . (I don't understand why you would want a 3d modeling program to be an IDE, but whatever floats your boat) Also, python-dev isn't really the place for feature requests. If you want something added, add it yourself. : ) -------------- next part -------------- An HTML attachment was scrubbed... URL: From rymg19 at gmail.com Tue Mar 31 17:31:21 2015 From: rymg19 at gmail.com (Ryan Gonzalez) Date: Tue, 31 Mar 2015 10:31:21 -0500 Subject: [Python-Dev] importante!! In-Reply-To: References: Message-ID: ???????????? I think this was meant for python-list. On Tue, Mar 31, 2015 at 10:13 AM, Alan Armour wrote: > its french! lol > > I just wanted to see if you could, as well as making python able to have > assembly written, you should totally use blender as your main IDE it would > be so epic > > and do pyjackd, and i think pyaudio will write some stuff, and like > pycoreaudio and like directpy for windows direct x > > oh having blender as the main ide would just be epic > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/rymg19%40gmail.com > > -- Ryan [ERROR]: Your autotools build scripts are 200 lines longer than your program. Something?s wrong. http://kirbyfan64.github.io/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From demianbrecht at gmail.com Tue Mar 31 18:55:32 2015 From: demianbrecht at gmail.com (Demian Brecht) Date: Tue, 31 Mar 2015 09:55:32 -0700 Subject: [Python-Dev] http.client: Determining Content-Length Message-ID: <18A92DFA-B232-405C-B894-C5464BF777D3@gmail.com> Hi all, I'm not sure whether this should be python-list or here, but given it's a premature code review for http.client, I figured I'd post here first. I'm in the process of adding support for chunked transfer encoding to http.client (issue #12319). One of the bits of functionality that I'm working on in is ironing out some of the kinks out in determining the content length of the request bodies. Given the number of data types allowed as bodies it does get a little messy, so I wanted to get some feedback here and see if anyone can shoot holes through it prior to updating the current patch. The tests are passing, but there may be use cases not accounted for in the new implementation. https://gist.github.com/demianbrecht/f94be5a51e32bb9c81e1 The above is intended to replace _set_content_length (current state of the patch can be seen here: http://bugs.python.org/review/12319/diff/14331/Lib/http/client.py). There is a comprehensive list of data types currently supported can be found here: http://bugs.python.org/issue23740. Comments and feedback are much appreciated. Thanks, Demian -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: