From benjamin at python.org Sat Nov 1 00:46:23 2014 From: benjamin at python.org (Benjamin Peterson) Date: Fri, 31 Oct 2014 19:46:23 -0400 Subject: [Python-Dev] 2.7.9 schedule Message-ID: <1414799183.1861359.185728873.4D1BA0D6@webmail.messagingengine.com> Multiple RGCs are not unlikely given. From benjamin at python.org Sat Nov 1 00:46:54 2014 From: benjamin at python.org (Benjamin Peterson) Date: Fri, 31 Oct 2014 19:46:54 -0400 Subject: [Python-Dev] 2.7.9 schedule In-Reply-To: <1414799183.1861359.185728873.4D1BA0D6@webmail.messagingengine.com> References: <1414799183.1861359.185728873.4D1BA0D6@webmail.messagingengine.com> Message-ID: <1414799214.1861359.185729009.49E7F25B@webmail.messagingengine.com> Oops, I hit "send" a bit early on that... On Fri, Oct 31, 2014, at 19:46, Benjamin Peterson wrote: > Multiple RGCs are not unlikely given. From benjamin at python.org Sat Nov 1 00:54:17 2014 From: benjamin at python.org (Benjamin Peterson) Date: Fri, 31 Oct 2014 19:54:17 -0400 Subject: [Python-Dev] 2.7.9 schedule redux Message-ID: <1414799657.1863548.185729737.587D6FE6@webmail.messagingengine.com> I'm updating the 2.7 release PEP with the following release dates 2.7.9rc1 Nov 22 2.7.9 final December 5 I expect PEPs 476 and 477 can be done by then? Given that 2.7.9 has had some large changes, having multiple rcs is not unlikely. Hopefully, we can get a final out by 2015, though. From donald at stufft.io Sat Nov 1 00:59:00 2014 From: donald at stufft.io (Donald Stufft) Date: Fri, 31 Oct 2014 19:59:00 -0400 Subject: [Python-Dev] 2.7.9 schedule redux In-Reply-To: <1414799657.1863548.185729737.587D6FE6@webmail.messagingengine.com> References: <1414799657.1863548.185729737.587D6FE6@webmail.messagingengine.com> Message-ID: <74D7D8DC-0E78-4DBA-8E17-5619B330D59D@stufft.io> > On Oct 31, 2014, at 7:54 PM, Benjamin Peterson wrote: > > I'm updating the 2.7 release PEP with the following release dates > > 2.7.9rc1 Nov 22 > 2.7.9 final December 5 > > I expect PEPs 476 and 477 can be done by then? > > Given that 2.7.9 has had some large changes, having multiple rcs is not > unlikely. Hopefully, we can get a final out by 2015, though. Yes on PEP 477, the actual back porting is done, I?m just slacking on doing the tests because we used unittest.mock in 3.4. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA From jeremy.kloth at gmail.com Sat Nov 1 01:12:35 2014 From: jeremy.kloth at gmail.com (Jeremy Kloth) Date: Fri, 31 Oct 2014 18:12:35 -0600 Subject: [Python-Dev] Buildslave kloth-win64 maintenance Message-ID: All, Just letting python-dev know that my buildslave will be down for a peroid of time due to a failed PSU. Currently awaiting RMA confirmation and I will post an update when the machine will be able to be back online. Sorry for the inconvenience. The builders hosted are the AMD64 Windows 7 ones. -- Jeremy Kloth From ncoghlan at gmail.com Sat Nov 1 13:05:14 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 1 Nov 2014 22:05:14 +1000 Subject: [Python-Dev] 2.7.9 schedule redux In-Reply-To: <74D7D8DC-0E78-4DBA-8E17-5619B330D59D@stufft.io> References: <1414799657.1863548.185729737.587D6FE6@webmail.messagingengine.com> <74D7D8DC-0E78-4DBA-8E17-5619B330D59D@stufft.io> Message-ID: On 1 November 2014 09:59, Donald Stufft wrote: > >> On Oct 31, 2014, at 7:54 PM, Benjamin Peterson wrote: >> >> I'm updating the 2.7 release PEP with the following release dates >> >> 2.7.9rc1 Nov 22 >> 2.7.9 final December 5 >> >> I expect PEPs 476 and 477 can be done by then? >> >> Given that 2.7.9 has had some large changes, having multiple rcs is not >> unlikely. Hopefully, we can get a final out by 2015, though. > > Yes on PEP 477, the actual back porting is done, I?m just slacking on doing > the tests because we used unittest.mock in 3.4. mock is only a single file, so making it available to the Python 2.7 regression test suite as something like test._mock_backport sounds tolerable to me. Another challenge is that the functional tests (i.e. the ones that actually run pip from the bundled wheel file, rather than mocking it out) are in test_venv, since they rely on venv to provide a degree of isolation from the Python instance running the test suite. However, given that pip's own tests cover compatibility with the 2.x series, the Python 3 test suite covers the functional integration with ensurepip, and the mock-based tests for ensurepip cover the 2.7 backport itself, I think we still have an acceptable level of regression testing coverage, even without being able to backport the venv based functional tests. Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From donald at stufft.io Sat Nov 1 15:40:30 2014 From: donald at stufft.io (Donald Stufft) Date: Sat, 1 Nov 2014 10:40:30 -0400 Subject: [Python-Dev] 2.7.9 schedule redux In-Reply-To: References: <1414799657.1863548.185729737.587D6FE6@webmail.messagingengine.com> <74D7D8DC-0E78-4DBA-8E17-5619B330D59D@stufft.io> Message-ID: <7D615717-BB86-49B4-88D7-9AC1261BCB71@stufft.io> > On Nov 1, 2014, at 8:05 AM, Nick Coghlan wrote: > > On 1 November 2014 09:59, Donald Stufft wrote: >> >>> On Oct 31, 2014, at 7:54 PM, Benjamin Peterson wrote: >>> >>> I'm updating the 2.7 release PEP with the following release dates >>> >>> 2.7.9rc1 Nov 22 >>> 2.7.9 final December 5 >>> >>> I expect PEPs 476 and 477 can be done by then? >>> >>> Given that 2.7.9 has had some large changes, having multiple rcs is not >>> unlikely. Hopefully, we can get a final out by 2015, though. >> >> Yes on PEP 477, the actual back porting is done, I?m just slacking on doing >> the tests because we used unittest.mock in 3.4. > > mock is only a single file, so making it available to the Python 2.7 > regression test suite as something like test._mock_backport sounds > tolerable to me. > > Another challenge is that the functional tests (i.e. the ones that > actually run pip from the bundled wheel file, rather than mocking it > out) are in test_venv, since they rely on venv to provide a degree of > isolation from the Python instance running the test suite. > > However, given that pip's own tests cover compatibility with the 2.x > series, the Python 3 test suite covers the functional integration with > ensurepip, and the mock-based tests for ensurepip cover the 2.7 > backport itself, I think we still have an acceptable level of > regression testing coverage, even without being able to backport the > venv based functional tests. Ok. The actual delta between ensurepip in 3.4 and in 2.7.9 is quite small too. It?s really just removing things like def foo(*, bar=None) and tempfile.TemporaryDirectory. No logic changes, just syntax/helper stuff. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA From tjreedy at udel.edu Sun Nov 2 03:37:10 2014 From: tjreedy at udel.edu (Terry Reedy) Date: Sat, 01 Nov 2014 22:37:10 -0400 Subject: [Python-Dev] problem with hg.python.org? Message-ID: My normally dependable pull.bat script has 3 times given me this. F:\Python\dev>hg --repository F:\Python\dev\5\py35 pull --verbose --config ui.merge=internal:merge ssh://hg at hg.python.org/cpython pulling from ssh://hg at hg.python.org/cpython searching for changes all local heads known remotely adding changesets adding manifests adding file changes transaction abort! rollback completed abort: data/Doc/howto/webservers.rst.i at 5f4ad429ae9f: unknown parent! remote: Unable to write to standard output: The pipe is being closed. Since I have no 'data/Doc/....i at 5f....9f, I am guessing that the problem is not local. -- Terry Jan Reedy From donald at stufft.io Sun Nov 2 04:03:15 2014 From: donald at stufft.io (Donald Stufft) Date: Sat, 1 Nov 2014 23:03:15 -0400 Subject: [Python-Dev] problem with hg.python.org? In-Reply-To: References: Message-ID: <43FF9550-87A6-4618-A3FC-6A61D86580D9@stufft.io> > On Nov 1, 2014, at 10:37 PM, Terry Reedy wrote: > > My normally dependable pull.bat script has 3 times given me this. > > F:\Python\dev>hg --repository F:\Python\dev\5\py35 pull --verbose --config ui.merge=internal:merge ssh://hg at hg.python.org/cpython > pulling from ssh://hg at hg.python.org/cpython > searching for changes > all local heads known remotely > adding changesets > adding manifests > adding file changes > transaction abort! > rollback completed > abort: data/Doc/howto/webservers.rst.i at 5f4ad429ae9f: unknown parent! > remote: Unable to write to standard output: The pipe is being closed. > > Since I have no 'data/Doc/....i at 5f....9f, I am guessing that the problem is not local. > > -- > Terry Jan Reedy > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/donald%40stufft.io The internet suggests trying hg verify, does that do anything? --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA From tjreedy at udel.edu Sun Nov 2 07:44:29 2014 From: tjreedy at udel.edu (Terry Reedy) Date: Sun, 02 Nov 2014 01:44:29 -0500 Subject: [Python-Dev] problem with hg.python.org? In-Reply-To: <43FF9550-87A6-4618-A3FC-6A61D86580D9@stufft.io> References: <43FF9550-87A6-4618-A3FC-6A61D86580D9@stufft.io> Message-ID: On 11/1/2014 11:03 PM, Donald Stufft wrote: > >> On Nov 1, 2014, at 10:37 PM, Terry Reedy wrote: > >> abort: data/Doc/howto/webservers.rst.i at 5f4ad429ae9f: unknown parent! >> remote: Unable to write to standard output: The pipe is being closed. > The internet suggests trying hg verify, does that do anything? I should have though of that. 60+ integrity errors. I am recloning. -- Terry Jan Reedy From ethan at stoneleaf.us Mon Nov 3 11:30:53 2014 From: ethan at stoneleaf.us (Ethan Furman) Date: Mon, 03 Nov 2014 02:30:53 -0800 Subject: [Python-Dev] The role of NotImplemented: What is it for and when should it be used? Message-ID: <5457595D.4030107@stoneleaf.us> Just to be clear, this is about NotImplemented, not NotImplementedError. tl;dr When a binary operation fails, should an exception be raised or NotImplemented returned? When a binary operation in Python is attempted, there are two possibilities: - it can work - it can't work The main reason [1] that it can't work is that the two operands are of different types, and the first type does not know how to deal with the second type. The question then becomes: how does the first type tell Python that it cannot perform the requested operation? The most obvious answer is to raise an exception, and TypeError is a good candidate. The problem with the exception raising approach is that once an exception is raised, Python doesn't try anything else to make the operation work. What's wrong with that? Well, the second type might know how to perform the operation, and in fact that is why we have the reflected special methods, such as __radd__ and __rmod__ -- but if the first type raises an exception the __rxxx__ methods will not be tried. Okay, how can the first type tell Python that it cannot do what is requested, but to go ahead and check with the second type to see if it does? That is where NotImplemented comes in -- if a special method (and only a special method) returns NotImplemented then Python will check to see if there is anything else it can do to make the operation succeed; if all attempts return NotImplemented, then Python itself will raise an appropriate exception [2]. In an effort to see how often NotImplemented is currently being returned I crafted a test script [3] to test the types bytes, bytearray, str, dict, list, tuple, Enum, Counter, defaultdict, deque, and OrderedDict with the operations for __add__, __and__, __floordiv__, __iadd__, __iand__, __ifloordiv__, __ilshift__, __imod__, __imul__, __ior__, __ipow__, __irshift__, __isub__, __itruediv__, __ixor__, __lshift__, __mod__, __mul__, __or__, __pow__, __rshift__, __sub__, __truediv__, and __xor__. Here are the results of the 275 tests: -------------------------------------------------------------------------------- testing control... ipow -- Exception raised errors in Control -- misunderstanding or bug? testing types against a foreign class iadd(Counter()) -- Exception <'SomeOtherClass' object has no attribute 'items'> raised instead of TypeError iand(Counter()) -- NotImplemented not returned, TypeError not raised ior(Counter()) -- Exception <'SomeOtherClass' object has no attribute 'items'> raised instead of TypeError isub(Counter()) -- Exception <'SomeOtherClass' object has no attribute 'items'> raised instead of TypeError testing types against a subclass mod(str()) -- NotImplemented not returned, TypeError not raised iadd(Counter()) -- Exception <'subtype' object has no attribute 'items'> raised (should have worked) iand(Counter()) -- NotImplemented not returned, TypeError not raised ior(Counter()) -- Exception <'subtype' object has no attribute 'items'> raised (should have worked) isub(Counter()) -- Exception <'subtype' object has no attribute 'items'> raised (should have worked) -------------------------------------------------------------------------------- Two observations: - __ipow__ doesn't seem to behave properly in the 3.x line (that error doesn't show up when testing against 2.7) - Counter should be returning NotImplemented instead of raising an AttributeError, for three reasons [4]: - a TypeError is more appropriate - subclasses /cannot/ work with the current implementation - __iand__ is currently a silent failure if the Counter is empty, and the other operand should trigger a failure Back to the main point... So, if my understanding is correct: - NotImplemented is used to signal Python that the requested operation could not be performed - it should be used by the binary special methods to signal type mismatch failure, so any subclass gets a chance to work. Is my understanding correct? Is this already in the docs somewhere, and I just missed it? -- ~Ethan~ [1] at least, it's the main reason in my code [2] usually a TypeError, stating either that the operation is not supported, or the types are unorderable [3] test script at the end [4] https://bugs.python.org/issue22766 [returning NotImplemented was rejected] -- 8< ---------------------------------------------------------------------------- from collections import Counter, defaultdict, deque, OrderedDict from fractions import Fraction from decimal import Decimal from enum import Enum import operator import sys py_ver = sys.version_info[:2] types = ( bytes, bytearray, str, dict, list, tuple, Enum, Counter, defaultdict, deque, OrderedDict, ) numeric_types = int, float, Decimal, Fraction operators = ( '__add__', '__and__', '__floordiv__', '__iadd__', '__iand__', '__ifloordiv__', '__ilshift__', '__imod__', '__imul__', '__ior__', '__ipow__', '__irshift__', '__isub__', '__itruediv__', '__ixor__', '__lshift__', '__mod__', '__mul__', '__or__', '__pow__', '__rshift__', '__sub__', '__truediv__', '__xor__', ) if py_ver >= (3, 0): operators += ('__gt__', '__ge__', '__le__','__lt__') ordered_reflections = { '__le__': '__ge__', '__lt__': '__gt__', '__ge__': '__le__', '__gt__': '__lt__', } # helpers class SomeOtherClass: """" used to test behavior when a different type is passed in to the special methods """ def __repr__(self): return 'SomeOtherClass' some_other_class = SomeOtherClass() class MainClassHandled(Exception): """ called by base class if both operands are of type base class """ class SubClassCalled(Exception): """ called by reflected operations for testing """ def create_control(test_op): def _any(self, other): if not type(other) is self.__class__: return NotImplemented raise MainClassHandled class Control: "returns NotImplemented when other object is not supported" _any.__name__ = op setattr(Control, test_op, _any) return Control() def create_subtype(test_op, base_class=object): def _any(*a): global subclass_called subclass_called = True raise SubClassCalled class subtype(base_class): __add__ = __sub__ = __mul__ = __truediv__ = __floordiv__ = _any __mod__ = __divmod__ = __pow__ = __lshift__ = __rshift__ = _any __and__ = __xor__ = __or__ = _any __radd__ = __rsub__ = __rmul__ = __rtruediv__ = __rfloordiv__ = _any __rmod__ = __rdivmod__ = __rpow__ = __rlshift__ = __rrshift__ = _any __rand__ = __rxor__ = __ror__ = _any __le__ = __lt__ = __gt__ = __ge__ = _any if issubclass(subtype, (bytes, bytearray)): value = b'hello' elif issubclass(subtype, str): value = 'goodbye' elif issubclass(subtype, (list, tuple)): value = (1, 2, 3) elif issubclass(subtype, (int, float, Decimal, Fraction)): value = 42 else: # ignore value return subtype() return subtype(value) # test exceptions # control against some other class print('testing control...\n') errors = False for op in operators: control = create_control(op) op = getattr(operator, op) try: op(control, some_other_class) except TypeError: # the end result of no method existing, or each method called returning # NotImplemented because it does not know how to perform the requested # operation between the two types pass except Exception as exc: errors = True print('%s(%s()) -- Exception <%s> raised instead of TypeError' % (op.__name__, test_type.__name__, exc)) else: errors = True print('Control -- TypeError not raised for op %r' % op) if errors: print('errors in Control -- misunderstanding or bug?\n') # control against a subclass errors = False for op in operators: subclass_called = False control = create_control(op) subtype = create_subtype(op, control.__class__) op = getattr(operator, op) try: op(control, subtype) except SubClassCalled: # if the control class properly signals that it doesn't know how to # perform the operation, of if Python notices that a reflected # operation exists, we get here (which is good) pass except MainClassHandled: errors = True print('Control did not yield to subclass for op %r' % op) except Exception as exc: if subclass_called: # exception was subverted to something more appropriate (like # unorderable types) pass errors = True print('%s -- Exception <%s> raised' % (op.__name__, exc)) else: errors = True print('Control -- op %r appears to have succeeded (it should not have)' % op) if errors: print('errors in Control -- misunderstanding or bug?\n') # tests print('testing types against a foreign class\n') for test_type in types + numeric_types: errors = False for op in operators: op = getattr(operator, op) try: op(test_type(), some_other_class) except TypeError: pass except Exception as exc: errors = True print('%s(%s()) -- Exception <%s> raised instead of TypeError' % (op.__name__, test_type.__name__, exc)) else: print('%s(%s()) -- NotImplemented not returned, TypeError not raised' % (op.__name__, test_type.__name__)) if errors: print() print() # test subclasses print('testing types against a subclass\n') for test_type in types: errors = False for op in operators: subclass_called = False if not test_type.__dict__.get(op): continue subclass = create_subtype(op, test_type) op = getattr(operator, op) try: if test_type is str: op('%s', subtype) else: op(test_type(), subtype) except SubClassCalled: # expected, ignore pass except Exception as exc: if subclass_called: # exception raised by subclass was changed pass errors = True print('%s(%s()) -- Exception <%s> raised (should have worked)' % (op.__name__, test_type.__name__, exc)) else: errors = True print('%s(%s()) -- NotImplemented not returned, TypeError not raised' % (op.__name__, test_type.__name__)) if errors: print() for test_type in numeric_types: errors = False for op in operators: subclass_called = False if not test_type.__dict__.get(op): continue subtype = create_subtype(op, test_type) op = getattr(operator, op) try: op(test_type(), subtype) except SubClassCalled: # expected, ignore pass except Exception as exc: if subclass_called: # exception raised by subclass was changed pass errors = True print('%s(%s()) -- Exception <%s> raised (should have worked)' % (op.__name__, test_type.__name__, exc)) else: errors = True print('%s(%s)) -- NotImplemented not returned' % (op.__name__, test_type.__name__)) if errors: print() -- 8< ---------------------------------------------------------------------------- From rdmurray at bitdance.com Mon Nov 3 14:32:38 2014 From: rdmurray at bitdance.com (R. David Murray) Date: Mon, 03 Nov 2014 08:32:38 -0500 Subject: [Python-Dev] The role of NotImplemented: What is it for and when should it be used? In-Reply-To: <5457595D.4030107@stoneleaf.us> References: <5457595D.4030107@stoneleaf.us> Message-ID: <20141103133239.58D89250230@webabinitio.net> See issue 22766 for some background on this question. On Mon, 03 Nov 2014 02:30:53 -0800, Ethan Furman wrote: > Just to be clear, this is about NotImplemented, not NotImplementedError. > > tl;dr When a binary operation fails, should an exception be raised or NotImplemented returned? > > > When a binary operation in Python is attempted, there are two possibilities: > > - it can work > - it can't work > > The main reason [1] that it can't work is that the two operands are of different types, and the first type does not know > how to deal with the second type. > > The question then becomes: how does the first type tell Python that it cannot perform the requested operation? The most > obvious answer is to raise an exception, and TypeError is a good candidate. The problem with the exception raising > approach is that once an exception is raised, Python doesn't try anything else to make the operation work. > > What's wrong with that? Well, the second type might know how to perform the operation, and in fact that is why we have > the reflected special methods, such as __radd__ and __rmod__ -- but if the first type raises an exception the __rxxx__ > methods will not be tried. > > Okay, how can the first type tell Python that it cannot do what is requested, but to go ahead and check with the second > type to see if it does? That is where NotImplemented comes in -- if a special method (and only a special method) > returns NotImplemented then Python will check to see if there is anything else it can do to make the operation succeed; > if all attempts return NotImplemented, then Python itself will raise an appropriate exception [2]. > > In an effort to see how often NotImplemented is currently being returned I crafted a test script [3] to test the types > bytes, bytearray, str, dict, list, tuple, Enum, Counter, defaultdict, deque, and OrderedDict with the operations for > __add__, __and__, __floordiv__, __iadd__, __iand__, __ifloordiv__, __ilshift__, __imod__, __imul__, __ior__, __ipow__, > __irshift__, __isub__, __itruediv__, __ixor__, __lshift__, __mod__, __mul__, __or__, __pow__, __rshift__, __sub__, > __truediv__, and __xor__. > > Here are the results of the 275 tests: > -------------------------------------------------------------------------------- > testing control... > > ipow -- Exception raised > errors in Control -- misunderstanding or bug? > > testing types against a foreign class > > iadd(Counter()) -- Exception <'SomeOtherClass' object has no attribute 'items'> raised instead of TypeError > iand(Counter()) -- NotImplemented not returned, TypeError not raised > ior(Counter()) -- Exception <'SomeOtherClass' object has no attribute 'items'> raised instead of TypeError > isub(Counter()) -- Exception <'SomeOtherClass' object has no attribute 'items'> raised instead of TypeError > > > testing types against a subclass > > mod(str()) -- NotImplemented not returned, TypeError not raised > > iadd(Counter()) -- Exception <'subtype' object has no attribute 'items'> raised (should have worked) > iand(Counter()) -- NotImplemented not returned, TypeError not raised > ior(Counter()) -- Exception <'subtype' object has no attribute 'items'> raised (should have worked) > isub(Counter()) -- Exception <'subtype' object has no attribute 'items'> raised (should have worked) > -------------------------------------------------------------------------------- > > Two observations: > > - __ipow__ doesn't seem to behave properly in the 3.x line (that error doesn't show up when testing against 2.7) > > - Counter should be returning NotImplemented instead of raising an AttributeError, for three reasons [4]: > - a TypeError is more appropriate > - subclasses /cannot/ work with the current implementation > - __iand__ is currently a silent failure if the Counter is empty, and the other operand should trigger a failure > > Back to the main point... > > So, if my understanding is correct: > > - NotImplemented is used to signal Python that the requested operation could not be performed > - it should be used by the binary special methods to signal type mismatch failure, so any subclass gets a chance to work. > > Is my understanding correct? Is this already in the docs somewhere, and I just missed it? > > -- > ~Ethan~ > > [1] at least, it's the main reason in my code > [2] usually a TypeError, stating either that the operation is not supported, or the types are unorderable > [3] test script at the end > [4] https://bugs.python.org/issue22766 [returning NotImplemented was rejected] > > -- 8< ---------------------------------------------------------------------------- > from collections import Counter, defaultdict, deque, OrderedDict > from fractions import Fraction > from decimal import Decimal > from enum import Enum > import operator > import sys > > py_ver = sys.version_info[:2] > > types = ( > bytes, bytearray, str, dict, list, tuple, > Enum, Counter, defaultdict, deque, OrderedDict, > ) > numeric_types = int, float, Decimal, Fraction > > operators = ( > '__add__', '__and__', '__floordiv__', > '__iadd__', '__iand__', '__ifloordiv__', '__ilshift__', > '__imod__', '__imul__', '__ior__', '__ipow__', > '__irshift__', '__isub__', '__itruediv__', '__ixor__', > '__lshift__', '__mod__', '__mul__', > '__or__', '__pow__', '__rshift__', '__sub__', '__truediv__', > '__xor__', > ) > > if py_ver >= (3, 0): > operators += ('__gt__', '__ge__', '__le__','__lt__') > > ordered_reflections = { > '__le__': '__ge__', > '__lt__': '__gt__', > '__ge__': '__le__', > '__gt__': '__lt__', > } > > > # helpers > > class SomeOtherClass: > """" > used to test behavior when a different type is passed in to the > special methods > """ > def __repr__(self): > return 'SomeOtherClass' > some_other_class = SomeOtherClass() > > class MainClassHandled(Exception): > """ > called by base class if both operands are of type base class > """ > > class SubClassCalled(Exception): > """ > called by reflected operations for testing > """ > > def create_control(test_op): > def _any(self, other): > if not type(other) is self.__class__: > return NotImplemented > raise MainClassHandled > class Control: > "returns NotImplemented when other object is not supported" > _any.__name__ = op > setattr(Control, test_op, _any) > return Control() > > def create_subtype(test_op, base_class=object): > def _any(*a): > global subclass_called > subclass_called = True > raise SubClassCalled > class subtype(base_class): > __add__ = __sub__ = __mul__ = __truediv__ = __floordiv__ = _any > __mod__ = __divmod__ = __pow__ = __lshift__ = __rshift__ = _any > __and__ = __xor__ = __or__ = _any > __radd__ = __rsub__ = __rmul__ = __rtruediv__ = __rfloordiv__ = _any > __rmod__ = __rdivmod__ = __rpow__ = __rlshift__ = __rrshift__ = _any > __rand__ = __rxor__ = __ror__ = _any > __le__ = __lt__ = __gt__ = __ge__ = _any > if issubclass(subtype, (bytes, bytearray)): > value = b'hello' > elif issubclass(subtype, str): > value = 'goodbye' > elif issubclass(subtype, (list, tuple)): > value = (1, 2, 3) > elif issubclass(subtype, (int, float, Decimal, Fraction)): > value = 42 > else: > # ignore value > return subtype() > return subtype(value) > > > # test exceptions > > # control against some other class > print('testing control...\n') > errors = False > for op in operators: > control = create_control(op) > op = getattr(operator, op) > try: > op(control, some_other_class) > except TypeError: > # the end result of no method existing, or each method called returning > # NotImplemented because it does not know how to perform the requested > # operation between the two types > pass > except Exception as exc: > errors = True > print('%s(%s()) -- Exception <%s> raised instead of TypeError' % > (op.__name__, test_type.__name__, exc)) > else: > errors = True > print('Control -- TypeError not raised for op %r' % op) > if errors: > print('errors in Control -- misunderstanding or bug?\n') > > # control against a subclass > errors = False > for op in operators: > subclass_called = False > control = create_control(op) > subtype = create_subtype(op, control.__class__) > op = getattr(operator, op) > try: > op(control, subtype) > except SubClassCalled: > # if the control class properly signals that it doesn't know how to > # perform the operation, of if Python notices that a reflected > # operation exists, we get here (which is good) > pass > except MainClassHandled: > errors = True > print('Control did not yield to subclass for op %r' % op) > except Exception as exc: > if subclass_called: > # exception was subverted to something more appropriate (like > # unorderable types) > pass > errors = True > print('%s -- Exception <%s> raised' % > (op.__name__, exc)) > else: > errors = True > print('Control -- op %r appears to have succeeded (it should not have)' % op) > if errors: > print('errors in Control -- misunderstanding or bug?\n') > > > # tests > print('testing types against a foreign class\n') > for test_type in types + numeric_types: > errors = False > for op in operators: > op = getattr(operator, op) > try: > op(test_type(), some_other_class) > except TypeError: > pass > except Exception as exc: > errors = True > print('%s(%s()) -- Exception <%s> raised instead of TypeError' % > (op.__name__, test_type.__name__, exc)) > else: > print('%s(%s()) -- NotImplemented not returned, TypeError not raised' % > (op.__name__, test_type.__name__)) > if errors: > print() > > print() > > # test subclasses > print('testing types against a subclass\n') > for test_type in types: > errors = False > for op in operators: > subclass_called = False > if not test_type.__dict__.get(op): > continue > subclass = create_subtype(op, test_type) > op = getattr(operator, op) > try: > if test_type is str: > op('%s', subtype) > else: > op(test_type(), subtype) > except SubClassCalled: > # expected, ignore > pass > except Exception as exc: > if subclass_called: > # exception raised by subclass was changed > pass > errors = True > print('%s(%s()) -- Exception <%s> raised (should have worked)' % > (op.__name__, test_type.__name__, exc)) > else: > errors = True > print('%s(%s()) -- NotImplemented not returned, TypeError not raised' % > (op.__name__, test_type.__name__)) > if errors: > print() > for test_type in numeric_types: > errors = False > for op in operators: > subclass_called = False > if not test_type.__dict__.get(op): > continue > subtype = create_subtype(op, test_type) > op = getattr(operator, op) > try: > op(test_type(), subtype) > except SubClassCalled: > # expected, ignore > pass > except Exception as exc: > if subclass_called: > # exception raised by subclass was changed > pass > errors = True > print('%s(%s()) -- Exception <%s> raised (should have worked)' % > (op.__name__, test_type.__name__, exc)) > else: > errors = True > print('%s(%s)) -- NotImplemented not returned' % > (op.__name__, test_type.__name__)) > if errors: > print() > -- 8< ---------------------------------------------------------------------------- > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/rdmurray%40bitdance.com From brett at python.org Mon Nov 3 16:05:31 2014 From: brett at python.org (Brett Cannon) Date: Mon, 03 Nov 2014 15:05:31 +0000 Subject: [Python-Dev] The role of NotImplemented: What is it for and when should it be used? References: <5457595D.4030107@stoneleaf.us> Message-ID: On Mon Nov 03 2014 at 5:31:21 AM Ethan Furman wrote: > Just to be clear, this is about NotImplemented, not NotImplementedError. > > tl;dr When a binary operation fails, should an exception be raised or > NotImplemented returned? > The docs for NotImplemented suggest it's only for rich comparison methods and not all binary operators: https://docs.python.org/3/library/constants.html#NotImplemented . But then had I not read that I would have said all binary operator methods should return NotImplemented when the types are incompatible. -Brett > > > When a binary operation in Python is attempted, there are two > possibilities: > > - it can work > - it can't work > > The main reason [1] that it can't work is that the two operands are of > different types, and the first type does not know > how to deal with the second type. > > The question then becomes: how does the first type tell Python that it > cannot perform the requested operation? The most > obvious answer is to raise an exception, and TypeError is a good > candidate. The problem with the exception raising > approach is that once an exception is raised, Python doesn't try anything > else to make the operation work. > > What's wrong with that? Well, the second type might know how to perform > the operation, and in fact that is why we have > the reflected special methods, such as __radd__ and __rmod__ -- but if the > first type raises an exception the __rxxx__ > methods will not be tried. > > Okay, how can the first type tell Python that it cannot do what is > requested, but to go ahead and check with the second > type to see if it does? That is where NotImplemented comes in -- if a > special method (and only a special method) > returns NotImplemented then Python will check to see if there is anything > else it can do to make the operation succeed; > if all attempts return NotImplemented, then Python itself will raise an > appropriate exception [2]. > > In an effort to see how often NotImplemented is currently being returned I > crafted a test script [3] to test the types > bytes, bytearray, str, dict, list, tuple, Enum, Counter, defaultdict, > deque, and OrderedDict with the operations for > __add__, __and__, __floordiv__, __iadd__, __iand__, __ifloordiv__, > __ilshift__, __imod__, __imul__, __ior__, __ipow__, > __irshift__, __isub__, __itruediv__, __ixor__, __lshift__, __mod__, > __mul__, __or__, __pow__, __rshift__, __sub__, > __truediv__, and __xor__. > > Here are the results of the 275 tests: > ------------------------------------------------------------ > -------------------- > testing control... > > ipow -- Exception and 'subtype'> raised > errors in Control -- misunderstanding or bug? > > testing types against a foreign class > > iadd(Counter()) -- Exception <'SomeOtherClass' object has no attribute > 'items'> raised instead of TypeError > iand(Counter()) -- NotImplemented not returned, TypeError not raised > ior(Counter()) -- Exception <'SomeOtherClass' object has no attribute > 'items'> raised instead of TypeError > isub(Counter()) -- Exception <'SomeOtherClass' object has no attribute > 'items'> raised instead of TypeError > > > testing types against a subclass > > mod(str()) -- NotImplemented not returned, TypeError not raised > > iadd(Counter()) -- Exception <'subtype' object has no attribute 'items'> > raised (should have worked) > iand(Counter()) -- NotImplemented not returned, TypeError not raised > ior(Counter()) -- Exception <'subtype' object has no attribute 'items'> > raised (should have worked) > isub(Counter()) -- Exception <'subtype' object has no attribute 'items'> > raised (should have worked) > ------------------------------------------------------------ > -------------------- > > Two observations: > > - __ipow__ doesn't seem to behave properly in the 3.x line (that error > doesn't show up when testing against 2.7) > > - Counter should be returning NotImplemented instead of raising an > AttributeError, for three reasons [4]: > - a TypeError is more appropriate > - subclasses /cannot/ work with the current implementation > - __iand__ is currently a silent failure if the Counter is empty, and > the other operand should trigger a failure > > Back to the main point... > > So, if my understanding is correct: > > - NotImplemented is used to signal Python that the requested operation > could not be performed > - it should be used by the binary special methods to signal type > mismatch failure, so any subclass gets a chance to work. > > Is my understanding correct? Is this already in the docs somewhere, and I > just missed it? > > -- > ~Ethan~ > > [1] at least, it's the main reason in my code > [2] usually a TypeError, stating either that the operation is not > supported, or the types are unorderable > [3] test script at the end > [4] https://bugs.python.org/issue22766 [returning NotImplemented was > rejected] > > -- 8< ------------------------------------------------------------ > ---------------- > from collections import Counter, defaultdict, deque, OrderedDict > from fractions import Fraction > from decimal import Decimal > from enum import Enum > import operator > import sys > > py_ver = sys.version_info[:2] > > types = ( > bytes, bytearray, str, dict, list, tuple, > Enum, Counter, defaultdict, deque, OrderedDict, > ) > numeric_types = int, float, Decimal, Fraction > > operators = ( > '__add__', '__and__', '__floordiv__', > '__iadd__', '__iand__', '__ifloordiv__', '__ilshift__', > '__imod__', '__imul__', '__ior__', '__ipow__', > '__irshift__', '__isub__', '__itruediv__', '__ixor__', > '__lshift__', '__mod__', '__mul__', > '__or__', '__pow__', '__rshift__', '__sub__', '__truediv__', > '__xor__', > ) > > if py_ver >= (3, 0): > operators += ('__gt__', '__ge__', '__le__','__lt__') > > ordered_reflections = { > '__le__': '__ge__', > '__lt__': '__gt__', > '__ge__': '__le__', > '__gt__': '__lt__', > } > > > # helpers > > class SomeOtherClass: > """" > used to test behavior when a different type is passed in to the > special methods > """ > def __repr__(self): > return 'SomeOtherClass' > some_other_class = SomeOtherClass() > > class MainClassHandled(Exception): > """ > called by base class if both operands are of type base class > """ > > class SubClassCalled(Exception): > """ > called by reflected operations for testing > """ > > def create_control(test_op): > def _any(self, other): > if not type(other) is self.__class__: > return NotImplemented > raise MainClassHandled > class Control: > "returns NotImplemented when other object is not supported" > _any.__name__ = op > setattr(Control, test_op, _any) > return Control() > > def create_subtype(test_op, base_class=object): > def _any(*a): > global subclass_called > subclass_called = True > raise SubClassCalled > class subtype(base_class): > __add__ = __sub__ = __mul__ = __truediv__ = __floordiv__ = _any > __mod__ = __divmod__ = __pow__ = __lshift__ = __rshift__ = _any > __and__ = __xor__ = __or__ = _any > __radd__ = __rsub__ = __rmul__ = __rtruediv__ = __rfloordiv__ = > _any > __rmod__ = __rdivmod__ = __rpow__ = __rlshift__ = __rrshift__ = > _any > __rand__ = __rxor__ = __ror__ = _any > __le__ = __lt__ = __gt__ = __ge__ = _any > if issubclass(subtype, (bytes, bytearray)): > value = b'hello' > elif issubclass(subtype, str): > value = 'goodbye' > elif issubclass(subtype, (list, tuple)): > value = (1, 2, 3) > elif issubclass(subtype, (int, float, Decimal, Fraction)): > value = 42 > else: > # ignore value > return subtype() > return subtype(value) > > > # test exceptions > > # control against some other class > print('testing control...\n') > errors = False > for op in operators: > control = create_control(op) > op = getattr(operator, op) > try: > op(control, some_other_class) > except TypeError: > # the end result of no method existing, or each method called > returning > # NotImplemented because it does not know how to perform the > requested > # operation between the two types > pass > except Exception as exc: > errors = True > print('%s(%s()) -- Exception <%s> raised instead of TypeError' % > (op.__name__, test_type.__name__, exc)) > else: > errors = True > print('Control -- TypeError not raised for op %r' % op) > if errors: > print('errors in Control -- misunderstanding or bug?\n') > > # control against a subclass > errors = False > for op in operators: > subclass_called = False > control = create_control(op) > subtype = create_subtype(op, control.__class__) > op = getattr(operator, op) > try: > op(control, subtype) > except SubClassCalled: > # if the control class properly signals that it doesn't know how > to > # perform the operation, of if Python notices that a reflected > # operation exists, we get here (which is good) > pass > except MainClassHandled: > errors = True > print('Control did not yield to subclass for op %r' % op) > except Exception as exc: > if subclass_called: > # exception was subverted to something more appropriate (like > # unorderable types) > pass > errors = True > print('%s -- Exception <%s> raised' % > (op.__name__, exc)) > else: > errors = True > print('Control -- op %r appears to have succeeded (it should not > have)' % op) > if errors: > print('errors in Control -- misunderstanding or bug?\n') > > > # tests > print('testing types against a foreign class\n') > for test_type in types + numeric_types: > errors = False > for op in operators: > op = getattr(operator, op) > try: > op(test_type(), some_other_class) > except TypeError: > pass > except Exception as exc: > errors = True > print('%s(%s()) -- Exception <%s> raised instead of > TypeError' % > (op.__name__, test_type.__name__, exc)) > else: > print('%s(%s()) -- NotImplemented not returned, TypeError not > raised' % > (op.__name__, test_type.__name__)) > if errors: > print() > > print() > > # test subclasses > print('testing types against a subclass\n') > for test_type in types: > errors = False > for op in operators: > subclass_called = False > if not test_type.__dict__.get(op): > continue > subclass = create_subtype(op, test_type) > op = getattr(operator, op) > try: > if test_type is str: > op('%s', subtype) > else: > op(test_type(), subtype) > except SubClassCalled: > # expected, ignore > pass > except Exception as exc: > if subclass_called: > # exception raised by subclass was changed > pass > errors = True > print('%s(%s()) -- Exception <%s> raised (should have > worked)' % > (op.__name__, test_type.__name__, exc)) > else: > errors = True > print('%s(%s()) -- NotImplemented not returned, TypeError not > raised' % > (op.__name__, test_type.__name__)) > if errors: > print() > for test_type in numeric_types: > errors = False > for op in operators: > subclass_called = False > if not test_type.__dict__.get(op): > continue > subtype = create_subtype(op, test_type) > op = getattr(operator, op) > try: > op(test_type(), subtype) > except SubClassCalled: > # expected, ignore > pass > except Exception as exc: > if subclass_called: > # exception raised by subclass was changed > pass > errors = True > print('%s(%s()) -- Exception <%s> raised (should have > worked)' % > (op.__name__, test_type.__name__, exc)) > else: > errors = True > print('%s(%s)) -- NotImplemented not returned' % > (op.__name__, test_type.__name__)) > if errors: > print() > -- 8< ------------------------------------------------------------ > ---------------- > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ > brett%40python.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rdmurray at bitdance.com Mon Nov 3 17:12:13 2014 From: rdmurray at bitdance.com (R. David Murray) Date: Mon, 03 Nov 2014 11:12:13 -0500 Subject: [Python-Dev] The role of NotImplemented: What is it for and when should it be used? In-Reply-To: References: <5457595D.4030107@stoneleaf.us> Message-ID: <20141103161213.82B7B250230@webabinitio.net> On Mon, 03 Nov 2014 15:05:31 +0000, Brett Cannon wrote: > On Mon Nov 03 2014 at 5:31:21 AM Ethan Furman wrote: > > > Just to be clear, this is about NotImplemented, not NotImplementedError. > > > > tl;dr When a binary operation fails, should an exception be raised or > > NotImplemented returned? > > > > The docs for NotImplemented suggest it's only for rich comparison methods > and not all binary operators: > https://docs.python.org/3/library/constants.html#NotImplemented . But then > had I not read that I would have said all binary operator methods should > return NotImplemented when the types are incompatible. Ethan opened an issue and then changed those docs, but I now believe that the docs should be changed back (see the discussion in issue 22766). --David From ethan at stoneleaf.us Mon Nov 3 17:22:30 2014 From: ethan at stoneleaf.us (Ethan Furman) Date: Mon, 03 Nov 2014 08:22:30 -0800 Subject: [Python-Dev] The role of NotImplemented: What is it for and when should it be used? In-Reply-To: <20141103161213.82B7B250230@webabinitio.net> References: <5457595D.4030107@stoneleaf.us> <20141103161213.82B7B250230@webabinitio.net> Message-ID: <5457ABC6.9090908@stoneleaf.us> On 11/03/2014 08:12 AM, R. David Murray wrote: > On Mon, 03 Nov 2014 15:05:31 +0000, Brett Cannon wrote: >> On Mon Nov 03 2014 at 5:31:21 AM Ethan Furman wrote: >> >>> Just to be clear, this is about NotImplemented, not NotImplementedError. >>> >>> tl;dr When a binary operation fails, should an exception be raised or >>> NotImplemented returned? >>> >> >> The docs for NotImplemented suggest it's only for rich comparison methods >> and not all binary operators: >> https://docs.python.org/3/library/constants.html#NotImplemented . But then >> had I not read that I would have said all binary operator methods should >> return NotImplemented when the types are incompatible. > > Ethan opened an issue and then changed those docs, but I now believe > that the docs should be changed back (see the discussion in issue > 22766). I was wondering myself, which is why I started this thread. -- ~Ethan~ From guido at python.org Mon Nov 3 17:48:07 2014 From: guido at python.org (Guido van Rossum) Date: Mon, 3 Nov 2014 08:48:07 -0800 Subject: [Python-Dev] The role of NotImplemented: What is it for and when should it be used? In-Reply-To: References: <5457595D.4030107@stoneleaf.us> Message-ID: Gotta be brief, but NotImplemented is for all binary ops. Power may be an exception because it's ternary? On Nov 3, 2014 8:08 AM, "Brett Cannon" wrote: > > > On Mon Nov 03 2014 at 5:31:21 AM Ethan Furman wrote: > >> Just to be clear, this is about NotImplemented, not NotImplementedError. >> >> tl;dr When a binary operation fails, should an exception be raised or >> NotImplemented returned? >> > > The docs for NotImplemented suggest it's only for rich comparison methods > and not all binary operators: > https://docs.python.org/3/library/constants.html#NotImplemented . But > then had I not read that I would have said all binary operator methods > should return NotImplemented when the types are incompatible. > > -Brett > > >> >> >> When a binary operation in Python is attempted, there are two >> possibilities: >> >> - it can work >> - it can't work >> >> The main reason [1] that it can't work is that the two operands are of >> different types, and the first type does not know >> how to deal with the second type. >> >> The question then becomes: how does the first type tell Python that it >> cannot perform the requested operation? The most >> obvious answer is to raise an exception, and TypeError is a good >> candidate. The problem with the exception raising >> approach is that once an exception is raised, Python doesn't try anything >> else to make the operation work. >> >> What's wrong with that? Well, the second type might know how to perform >> the operation, and in fact that is why we have >> the reflected special methods, such as __radd__ and __rmod__ -- but if >> the first type raises an exception the __rxxx__ >> methods will not be tried. >> >> Okay, how can the first type tell Python that it cannot do what is >> requested, but to go ahead and check with the second >> type to see if it does? That is where NotImplemented comes in -- if a >> special method (and only a special method) >> returns NotImplemented then Python will check to see if there is anything >> else it can do to make the operation succeed; >> if all attempts return NotImplemented, then Python itself will raise an >> appropriate exception [2]. >> >> In an effort to see how often NotImplemented is currently being returned >> I crafted a test script [3] to test the types >> bytes, bytearray, str, dict, list, tuple, Enum, Counter, defaultdict, >> deque, and OrderedDict with the operations for >> __add__, __and__, __floordiv__, __iadd__, __iand__, __ifloordiv__, >> __ilshift__, __imod__, __imul__, __ior__, __ipow__, >> __irshift__, __isub__, __itruediv__, __ixor__, __lshift__, __mod__, >> __mul__, __or__, __pow__, __rshift__, __sub__, >> __truediv__, and __xor__. >> >> Here are the results of the 275 tests: >> ------------------------------------------------------------ >> -------------------- >> testing control... >> >> ipow -- Exception > and 'subtype'> raised >> errors in Control -- misunderstanding or bug? >> >> testing types against a foreign class >> >> iadd(Counter()) -- Exception <'SomeOtherClass' object has no attribute >> 'items'> raised instead of TypeError >> iand(Counter()) -- NotImplemented not returned, TypeError not raised >> ior(Counter()) -- Exception <'SomeOtherClass' object has no attribute >> 'items'> raised instead of TypeError >> isub(Counter()) -- Exception <'SomeOtherClass' object has no attribute >> 'items'> raised instead of TypeError >> >> >> testing types against a subclass >> >> mod(str()) -- NotImplemented not returned, TypeError not raised >> >> iadd(Counter()) -- Exception <'subtype' object has no attribute 'items'> >> raised (should have worked) >> iand(Counter()) -- NotImplemented not returned, TypeError not raised >> ior(Counter()) -- Exception <'subtype' object has no attribute 'items'> >> raised (should have worked) >> isub(Counter()) -- Exception <'subtype' object has no attribute 'items'> >> raised (should have worked) >> ------------------------------------------------------------ >> -------------------- >> >> Two observations: >> >> - __ipow__ doesn't seem to behave properly in the 3.x line (that error >> doesn't show up when testing against 2.7) >> >> - Counter should be returning NotImplemented instead of raising an >> AttributeError, for three reasons [4]: >> - a TypeError is more appropriate >> - subclasses /cannot/ work with the current implementation >> - __iand__ is currently a silent failure if the Counter is empty, >> and the other operand should trigger a failure >> >> Back to the main point... >> >> So, if my understanding is correct: >> >> - NotImplemented is used to signal Python that the requested operation >> could not be performed >> - it should be used by the binary special methods to signal type >> mismatch failure, so any subclass gets a chance to work. >> >> Is my understanding correct? Is this already in the docs somewhere, and >> I just missed it? >> >> -- >> ~Ethan~ >> >> [1] at least, it's the main reason in my code >> [2] usually a TypeError, stating either that the operation is not >> supported, or the types are unorderable >> [3] test script at the end >> [4] https://bugs.python.org/issue22766 [returning NotImplemented was >> rejected] >> >> -- 8< ------------------------------------------------------------ >> ---------------- >> from collections import Counter, defaultdict, deque, OrderedDict >> from fractions import Fraction >> from decimal import Decimal >> from enum import Enum >> import operator >> import sys >> >> py_ver = sys.version_info[:2] >> >> types = ( >> bytes, bytearray, str, dict, list, tuple, >> Enum, Counter, defaultdict, deque, OrderedDict, >> ) >> numeric_types = int, float, Decimal, Fraction >> >> operators = ( >> '__add__', '__and__', '__floordiv__', >> '__iadd__', '__iand__', '__ifloordiv__', '__ilshift__', >> '__imod__', '__imul__', '__ior__', '__ipow__', >> '__irshift__', '__isub__', '__itruediv__', '__ixor__', >> '__lshift__', '__mod__', '__mul__', >> '__or__', '__pow__', '__rshift__', '__sub__', '__truediv__', >> '__xor__', >> ) >> >> if py_ver >= (3, 0): >> operators += ('__gt__', '__ge__', '__le__','__lt__') >> >> ordered_reflections = { >> '__le__': '__ge__', >> '__lt__': '__gt__', >> '__ge__': '__le__', >> '__gt__': '__lt__', >> } >> >> >> # helpers >> >> class SomeOtherClass: >> """" >> used to test behavior when a different type is passed in to the >> special methods >> """ >> def __repr__(self): >> return 'SomeOtherClass' >> some_other_class = SomeOtherClass() >> >> class MainClassHandled(Exception): >> """ >> called by base class if both operands are of type base class >> """ >> >> class SubClassCalled(Exception): >> """ >> called by reflected operations for testing >> """ >> >> def create_control(test_op): >> def _any(self, other): >> if not type(other) is self.__class__: >> return NotImplemented >> raise MainClassHandled >> class Control: >> "returns NotImplemented when other object is not supported" >> _any.__name__ = op >> setattr(Control, test_op, _any) >> return Control() >> >> def create_subtype(test_op, base_class=object): >> def _any(*a): >> global subclass_called >> subclass_called = True >> raise SubClassCalled >> class subtype(base_class): >> __add__ = __sub__ = __mul__ = __truediv__ = __floordiv__ = _any >> __mod__ = __divmod__ = __pow__ = __lshift__ = __rshift__ = _any >> __and__ = __xor__ = __or__ = _any >> __radd__ = __rsub__ = __rmul__ = __rtruediv__ = __rfloordiv__ = >> _any >> __rmod__ = __rdivmod__ = __rpow__ = __rlshift__ = __rrshift__ = >> _any >> __rand__ = __rxor__ = __ror__ = _any >> __le__ = __lt__ = __gt__ = __ge__ = _any >> if issubclass(subtype, (bytes, bytearray)): >> value = b'hello' >> elif issubclass(subtype, str): >> value = 'goodbye' >> elif issubclass(subtype, (list, tuple)): >> value = (1, 2, 3) >> elif issubclass(subtype, (int, float, Decimal, Fraction)): >> value = 42 >> else: >> # ignore value >> return subtype() >> return subtype(value) >> >> >> # test exceptions >> >> # control against some other class >> print('testing control...\n') >> errors = False >> for op in operators: >> control = create_control(op) >> op = getattr(operator, op) >> try: >> op(control, some_other_class) >> except TypeError: >> # the end result of no method existing, or each method called >> returning >> # NotImplemented because it does not know how to perform the >> requested >> # operation between the two types >> pass >> except Exception as exc: >> errors = True >> print('%s(%s()) -- Exception <%s> raised instead of TypeError' % >> (op.__name__, test_type.__name__, exc)) >> else: >> errors = True >> print('Control -- TypeError not raised for op %r' % op) >> if errors: >> print('errors in Control -- misunderstanding or bug?\n') >> >> # control against a subclass >> errors = False >> for op in operators: >> subclass_called = False >> control = create_control(op) >> subtype = create_subtype(op, control.__class__) >> op = getattr(operator, op) >> try: >> op(control, subtype) >> except SubClassCalled: >> # if the control class properly signals that it doesn't know how >> to >> # perform the operation, of if Python notices that a reflected >> # operation exists, we get here (which is good) >> pass >> except MainClassHandled: >> errors = True >> print('Control did not yield to subclass for op %r' % op) >> except Exception as exc: >> if subclass_called: >> # exception was subverted to something more appropriate (like >> # unorderable types) >> pass >> errors = True >> print('%s -- Exception <%s> raised' % >> (op.__name__, exc)) >> else: >> errors = True >> print('Control -- op %r appears to have succeeded (it should not >> have)' % op) >> if errors: >> print('errors in Control -- misunderstanding or bug?\n') >> >> >> # tests >> print('testing types against a foreign class\n') >> for test_type in types + numeric_types: >> errors = False >> for op in operators: >> op = getattr(operator, op) >> try: >> op(test_type(), some_other_class) >> except TypeError: >> pass >> except Exception as exc: >> errors = True >> print('%s(%s()) -- Exception <%s> raised instead of >> TypeError' % >> (op.__name__, test_type.__name__, exc)) >> else: >> print('%s(%s()) -- NotImplemented not returned, TypeError >> not raised' % >> (op.__name__, test_type.__name__)) >> if errors: >> print() >> >> print() >> >> # test subclasses >> print('testing types against a subclass\n') >> for test_type in types: >> errors = False >> for op in operators: >> subclass_called = False >> if not test_type.__dict__.get(op): >> continue >> subclass = create_subtype(op, test_type) >> op = getattr(operator, op) >> try: >> if test_type is str: >> op('%s', subtype) >> else: >> op(test_type(), subtype) >> except SubClassCalled: >> # expected, ignore >> pass >> except Exception as exc: >> if subclass_called: >> # exception raised by subclass was changed >> pass >> errors = True >> print('%s(%s()) -- Exception <%s> raised (should have >> worked)' % >> (op.__name__, test_type.__name__, exc)) >> else: >> errors = True >> print('%s(%s()) -- NotImplemented not returned, TypeError >> not raised' % >> (op.__name__, test_type.__name__)) >> if errors: >> print() >> for test_type in numeric_types: >> errors = False >> for op in operators: >> subclass_called = False >> if not test_type.__dict__.get(op): >> continue >> subtype = create_subtype(op, test_type) >> op = getattr(operator, op) >> try: >> op(test_type(), subtype) >> except SubClassCalled: >> # expected, ignore >> pass >> except Exception as exc: >> if subclass_called: >> # exception raised by subclass was changed >> pass >> errors = True >> print('%s(%s()) -- Exception <%s> raised (should have >> worked)' % >> (op.__name__, test_type.__name__, exc)) >> else: >> errors = True >> print('%s(%s)) -- NotImplemented not returned' % >> (op.__name__, test_type.__name__)) >> if errors: >> print() >> -- 8< ------------------------------------------------------------ >> ---------------- >> _______________________________________________ >> Python-Dev mailing list >> Python-Dev at python.org >> https://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe: https://mail.python.org/mailman/options/python-dev/ >> brett%40python.org >> > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/guido%40python.org > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Mon Nov 3 17:55:10 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 3 Nov 2014 17:55:10 +0100 Subject: [Python-Dev] The role of NotImplemented: What is it for and when should it be used? References: <5457595D.4030107@stoneleaf.us> Message-ID: <20141103175510.6059f838@fsol> On Mon, 3 Nov 2014 08:48:07 -0800 Guido van Rossum wrote: > Gotta be brief, but NotImplemented is for all binary ops. Even in-place ops? Regards Antoine. > Power may be an > exception because it's ternary? > On Nov 3, 2014 8:08 AM, "Brett Cannon" wrote: > > > > > > > On Mon Nov 03 2014 at 5:31:21 AM Ethan Furman wrote: > > > >> Just to be clear, this is about NotImplemented, not NotImplementedError. > >> > >> tl;dr When a binary operation fails, should an exception be raised or > >> NotImplemented returned? > >> > > > > The docs for NotImplemented suggest it's only for rich comparison methods > > and not all binary operators: > > https://docs.python.org/3/library/constants.html#NotImplemented . But > > then had I not read that I would have said all binary operator methods > > should return NotImplemented when the types are incompatible. > > > > -Brett > > > > > >> > >> > >> When a binary operation in Python is attempted, there are two > >> possibilities: > >> > >> - it can work > >> - it can't work > >> > >> The main reason [1] that it can't work is that the two operands are of > >> different types, and the first type does not know > >> how to deal with the second type. > >> > >> The question then becomes: how does the first type tell Python that it > >> cannot perform the requested operation? The most > >> obvious answer is to raise an exception, and TypeError is a good > >> candidate. The problem with the exception raising > >> approach is that once an exception is raised, Python doesn't try anything > >> else to make the operation work. > >> > >> What's wrong with that? Well, the second type might know how to perform > >> the operation, and in fact that is why we have > >> the reflected special methods, such as __radd__ and __rmod__ -- but if > >> the first type raises an exception the __rxxx__ > >> methods will not be tried. > >> > >> Okay, how can the first type tell Python that it cannot do what is > >> requested, but to go ahead and check with the second > >> type to see if it does? That is where NotImplemented comes in -- if a > >> special method (and only a special method) > >> returns NotImplemented then Python will check to see if there is anything > >> else it can do to make the operation succeed; > >> if all attempts return NotImplemented, then Python itself will raise an > >> appropriate exception [2]. > >> > >> In an effort to see how often NotImplemented is currently being returned > >> I crafted a test script [3] to test the types > >> bytes, bytearray, str, dict, list, tuple, Enum, Counter, defaultdict, > >> deque, and OrderedDict with the operations for > >> __add__, __and__, __floordiv__, __iadd__, __iand__, __ifloordiv__, > >> __ilshift__, __imod__, __imul__, __ior__, __ipow__, > >> __irshift__, __isub__, __itruediv__, __ixor__, __lshift__, __mod__, > >> __mul__, __or__, __pow__, __rshift__, __sub__, > >> __truediv__, and __xor__. > >> > >> Here are the results of the 275 tests: > >> ------------------------------------------------------------ > >> -------------------- > >> testing control... > >> > >> ipow -- Exception >> and 'subtype'> raised > >> errors in Control -- misunderstanding or bug? > >> > >> testing types against a foreign class > >> > >> iadd(Counter()) -- Exception <'SomeOtherClass' object has no attribute > >> 'items'> raised instead of TypeError > >> iand(Counter()) -- NotImplemented not returned, TypeError not raised > >> ior(Counter()) -- Exception <'SomeOtherClass' object has no attribute > >> 'items'> raised instead of TypeError > >> isub(Counter()) -- Exception <'SomeOtherClass' object has no attribute > >> 'items'> raised instead of TypeError > >> > >> > >> testing types against a subclass > >> > >> mod(str()) -- NotImplemented not returned, TypeError not raised > >> > >> iadd(Counter()) -- Exception <'subtype' object has no attribute 'items'> > >> raised (should have worked) > >> iand(Counter()) -- NotImplemented not returned, TypeError not raised > >> ior(Counter()) -- Exception <'subtype' object has no attribute 'items'> > >> raised (should have worked) > >> isub(Counter()) -- Exception <'subtype' object has no attribute 'items'> > >> raised (should have worked) > >> ------------------------------------------------------------ > >> -------------------- > >> > >> Two observations: > >> > >> - __ipow__ doesn't seem to behave properly in the 3.x line (that error > >> doesn't show up when testing against 2.7) > >> > >> - Counter should be returning NotImplemented instead of raising an > >> AttributeError, for three reasons [4]: > >> - a TypeError is more appropriate > >> - subclasses /cannot/ work with the current implementation > >> - __iand__ is currently a silent failure if the Counter is empty, > >> and the other operand should trigger a failure > >> > >> Back to the main point... > >> > >> So, if my understanding is correct: > >> > >> - NotImplemented is used to signal Python that the requested operation > >> could not be performed > >> - it should be used by the binary special methods to signal type > >> mismatch failure, so any subclass gets a chance to work. > >> > >> Is my understanding correct? Is this already in the docs somewhere, and > >> I just missed it? > >> > >> -- > >> ~Ethan~ > >> > >> [1] at least, it's the main reason in my code > >> [2] usually a TypeError, stating either that the operation is not > >> supported, or the types are unorderable > >> [3] test script at the end > >> [4] https://bugs.python.org/issue22766 [returning NotImplemented was > >> rejected] > >> > >> -- 8< ------------------------------------------------------------ > >> ---------------- > >> from collections import Counter, defaultdict, deque, OrderedDict > >> from fractions import Fraction > >> from decimal import Decimal > >> from enum import Enum > >> import operator > >> import sys > >> > >> py_ver = sys.version_info[:2] > >> > >> types = ( > >> bytes, bytearray, str, dict, list, tuple, > >> Enum, Counter, defaultdict, deque, OrderedDict, > >> ) > >> numeric_types = int, float, Decimal, Fraction > >> > >> operators = ( > >> '__add__', '__and__', '__floordiv__', > >> '__iadd__', '__iand__', '__ifloordiv__', '__ilshift__', > >> '__imod__', '__imul__', '__ior__', '__ipow__', > >> '__irshift__', '__isub__', '__itruediv__', '__ixor__', > >> '__lshift__', '__mod__', '__mul__', > >> '__or__', '__pow__', '__rshift__', '__sub__', '__truediv__', > >> '__xor__', > >> ) > >> > >> if py_ver >= (3, 0): > >> operators += ('__gt__', '__ge__', '__le__','__lt__') > >> > >> ordered_reflections = { > >> '__le__': '__ge__', > >> '__lt__': '__gt__', > >> '__ge__': '__le__', > >> '__gt__': '__lt__', > >> } > >> > >> > >> # helpers > >> > >> class SomeOtherClass: > >> """" > >> used to test behavior when a different type is passed in to the > >> special methods > >> """ > >> def __repr__(self): > >> return 'SomeOtherClass' > >> some_other_class = SomeOtherClass() > >> > >> class MainClassHandled(Exception): > >> """ > >> called by base class if both operands are of type base class > >> """ > >> > >> class SubClassCalled(Exception): > >> """ > >> called by reflected operations for testing > >> """ > >> > >> def create_control(test_op): > >> def _any(self, other): > >> if not type(other) is self.__class__: > >> return NotImplemented > >> raise MainClassHandled > >> class Control: > >> "returns NotImplemented when other object is not supported" > >> _any.__name__ = op > >> setattr(Control, test_op, _any) > >> return Control() > >> > >> def create_subtype(test_op, base_class=object): > >> def _any(*a): > >> global subclass_called > >> subclass_called = True > >> raise SubClassCalled > >> class subtype(base_class): > >> __add__ = __sub__ = __mul__ = __truediv__ = __floordiv__ = _any > >> __mod__ = __divmod__ = __pow__ = __lshift__ = __rshift__ = _any > >> __and__ = __xor__ = __or__ = _any > >> __radd__ = __rsub__ = __rmul__ = __rtruediv__ = __rfloordiv__ = > >> _any > >> __rmod__ = __rdivmod__ = __rpow__ = __rlshift__ = __rrshift__ = > >> _any > >> __rand__ = __rxor__ = __ror__ = _any > >> __le__ = __lt__ = __gt__ = __ge__ = _any > >> if issubclass(subtype, (bytes, bytearray)): > >> value = b'hello' > >> elif issubclass(subtype, str): > >> value = 'goodbye' > >> elif issubclass(subtype, (list, tuple)): > >> value = (1, 2, 3) > >> elif issubclass(subtype, (int, float, Decimal, Fraction)): > >> value = 42 > >> else: > >> # ignore value > >> return subtype() > >> return subtype(value) > >> > >> > >> # test exceptions > >> > >> # control against some other class > >> print('testing control...\n') > >> errors = False > >> for op in operators: > >> control = create_control(op) > >> op = getattr(operator, op) > >> try: > >> op(control, some_other_class) > >> except TypeError: > >> # the end result of no method existing, or each method called > >> returning > >> # NotImplemented because it does not know how to perform the > >> requested > >> # operation between the two types > >> pass > >> except Exception as exc: > >> errors = True > >> print('%s(%s()) -- Exception <%s> raised instead of TypeError' % > >> (op.__name__, test_type.__name__, exc)) > >> else: > >> errors = True > >> print('Control -- TypeError not raised for op %r' % op) > >> if errors: > >> print('errors in Control -- misunderstanding or bug?\n') > >> > >> # control against a subclass > >> errors = False > >> for op in operators: > >> subclass_called = False > >> control = create_control(op) > >> subtype = create_subtype(op, control.__class__) > >> op = getattr(operator, op) > >> try: > >> op(control, subtype) > >> except SubClassCalled: > >> # if the control class properly signals that it doesn't know how > >> to > >> # perform the operation, of if Python notices that a reflected > >> # operation exists, we get here (which is good) > >> pass > >> except MainClassHandled: > >> errors = True > >> print('Control did not yield to subclass for op %r' % op) > >> except Exception as exc: > >> if subclass_called: > >> # exception was subverted to something more appropriate (like > >> # unorderable types) > >> pass > >> errors = True > >> print('%s -- Exception <%s> raised' % > >> (op.__name__, exc)) > >> else: > >> errors = True > >> print('Control -- op %r appears to have succeeded (it should not > >> have)' % op) > >> if errors: > >> print('errors in Control -- misunderstanding or bug?\n') > >> > >> > >> # tests > >> print('testing types against a foreign class\n') > >> for test_type in types + numeric_types: > >> errors = False > >> for op in operators: > >> op = getattr(operator, op) > >> try: > >> op(test_type(), some_other_class) > >> except TypeError: > >> pass > >> except Exception as exc: > >> errors = True > >> print('%s(%s()) -- Exception <%s> raised instead of > >> TypeError' % > >> (op.__name__, test_type.__name__, exc)) > >> else: > >> print('%s(%s()) -- NotImplemented not returned, TypeError > >> not raised' % > >> (op.__name__, test_type.__name__)) > >> if errors: > >> print() > >> > >> print() > >> > >> # test subclasses > >> print('testing types against a subclass\n') > >> for test_type in types: > >> errors = False > >> for op in operators: > >> subclass_called = False > >> if not test_type.__dict__.get(op): > >> continue > >> subclass = create_subtype(op, test_type) > >> op = getattr(operator, op) > >> try: > >> if test_type is str: > >> op('%s', subtype) > >> else: > >> op(test_type(), subtype) > >> except SubClassCalled: > >> # expected, ignore > >> pass > >> except Exception as exc: > >> if subclass_called: > >> # exception raised by subclass was changed > >> pass > >> errors = True > >> print('%s(%s()) -- Exception <%s> raised (should have > >> worked)' % > >> (op.__name__, test_type.__name__, exc)) > >> else: > >> errors = True > >> print('%s(%s()) -- NotImplemented not returned, TypeError > >> not raised' % > >> (op.__name__, test_type.__name__)) > >> if errors: > >> print() > >> for test_type in numeric_types: > >> errors = False > >> for op in operators: > >> subclass_called = False > >> if not test_type.__dict__.get(op): > >> continue > >> subtype = create_subtype(op, test_type) > >> op = getattr(operator, op) > >> try: > >> op(test_type(), subtype) > >> except SubClassCalled: > >> # expected, ignore > >> pass > >> except Exception as exc: > >> if subclass_called: > >> # exception raised by subclass was changed > >> pass > >> errors = True > >> print('%s(%s()) -- Exception <%s> raised (should have > >> worked)' % > >> (op.__name__, test_type.__name__, exc)) > >> else: > >> errors = True > >> print('%s(%s)) -- NotImplemented not returned' % > >> (op.__name__, test_type.__name__)) > >> if errors: > >> print() > >> -- 8< ------------------------------------------------------------ > >> ---------------- > >> _______________________________________________ > >> Python-Dev mailing list > >> Python-Dev at python.org > >> https://mail.python.org/mailman/listinfo/python-dev > >> Unsubscribe: https://mail.python.org/mailman/options/python-dev/ > >> brett%40python.org > >> > > > > _______________________________________________ > > Python-Dev mailing list > > Python-Dev at python.org > > https://mail.python.org/mailman/listinfo/python-dev > > Unsubscribe: > > https://mail.python.org/mailman/options/python-dev/guido%40python.org > > > > > From guido at python.org Mon Nov 3 18:00:06 2014 From: guido at python.org (Guido van Rossum) Date: Mon, 3 Nov 2014 09:00:06 -0800 Subject: [Python-Dev] The role of NotImplemented: What is it for and when should it be used? In-Reply-To: <20141103175510.6059f838@fsol> References: <5457595D.4030107@stoneleaf.us> <20141103175510.6059f838@fsol> Message-ID: Not those. On Nov 3, 2014 8:56 AM, "Antoine Pitrou" wrote: > On Mon, 3 Nov 2014 08:48:07 -0800 > Guido van Rossum wrote: > > Gotta be brief, but NotImplemented is for all binary ops. > > Even in-place ops? > > Regards > > Antoine. > > > > Power may be an > > exception because it's ternary? > > On Nov 3, 2014 8:08 AM, "Brett Cannon" wrote: > > > > > > > > > > > On Mon Nov 03 2014 at 5:31:21 AM Ethan Furman > wrote: > > > > > >> Just to be clear, this is about NotImplemented, not > NotImplementedError. > > >> > > >> tl;dr When a binary operation fails, should an exception be raised or > > >> NotImplemented returned? > > >> > > > > > > The docs for NotImplemented suggest it's only for rich comparison > methods > > > and not all binary operators: > > > https://docs.python.org/3/library/constants.html#NotImplemented . But > > > then had I not read that I would have said all binary operator methods > > > should return NotImplemented when the types are incompatible. > > > > > > -Brett > > > > > > > > >> > > >> > > >> When a binary operation in Python is attempted, there are two > > >> possibilities: > > >> > > >> - it can work > > >> - it can't work > > >> > > >> The main reason [1] that it can't work is that the two operands are of > > >> different types, and the first type does not know > > >> how to deal with the second type. > > >> > > >> The question then becomes: how does the first type tell Python that it > > >> cannot perform the requested operation? The most > > >> obvious answer is to raise an exception, and TypeError is a good > > >> candidate. The problem with the exception raising > > >> approach is that once an exception is raised, Python doesn't try > anything > > >> else to make the operation work. > > >> > > >> What's wrong with that? Well, the second type might know how to > perform > > >> the operation, and in fact that is why we have > > >> the reflected special methods, such as __radd__ and __rmod__ -- but if > > >> the first type raises an exception the __rxxx__ > > >> methods will not be tried. > > >> > > >> Okay, how can the first type tell Python that it cannot do what is > > >> requested, but to go ahead and check with the second > > >> type to see if it does? That is where NotImplemented comes in -- if a > > >> special method (and only a special method) > > >> returns NotImplemented then Python will check to see if there is > anything > > >> else it can do to make the operation succeed; > > >> if all attempts return NotImplemented, then Python itself will raise > an > > >> appropriate exception [2]. > > >> > > >> In an effort to see how often NotImplemented is currently being > returned > > >> I crafted a test script [3] to test the types > > >> bytes, bytearray, str, dict, list, tuple, Enum, Counter, defaultdict, > > >> deque, and OrderedDict with the operations for > > >> __add__, __and__, __floordiv__, __iadd__, __iand__, __ifloordiv__, > > >> __ilshift__, __imod__, __imul__, __ior__, __ipow__, > > >> __irshift__, __isub__, __itruediv__, __ixor__, __lshift__, __mod__, > > >> __mul__, __or__, __pow__, __rshift__, __sub__, > > >> __truediv__, and __xor__. > > >> > > >> Here are the results of the 275 tests: > > >> ------------------------------------------------------------ > > >> -------------------- > > >> testing control... > > >> > > >> ipow -- Exception 'Control' > > >> and 'subtype'> raised > > >> errors in Control -- misunderstanding or bug? > > >> > > >> testing types against a foreign class > > >> > > >> iadd(Counter()) -- Exception <'SomeOtherClass' object has no attribute > > >> 'items'> raised instead of TypeError > > >> iand(Counter()) -- NotImplemented not returned, TypeError not raised > > >> ior(Counter()) -- Exception <'SomeOtherClass' object has no attribute > > >> 'items'> raised instead of TypeError > > >> isub(Counter()) -- Exception <'SomeOtherClass' object has no attribute > > >> 'items'> raised instead of TypeError > > >> > > >> > > >> testing types against a subclass > > >> > > >> mod(str()) -- NotImplemented not returned, TypeError not raised > > >> > > >> iadd(Counter()) -- Exception <'subtype' object has no attribute > 'items'> > > >> raised (should have worked) > > >> iand(Counter()) -- NotImplemented not returned, TypeError not raised > > >> ior(Counter()) -- Exception <'subtype' object has no attribute > 'items'> > > >> raised (should have worked) > > >> isub(Counter()) -- Exception <'subtype' object has no attribute > 'items'> > > >> raised (should have worked) > > >> ------------------------------------------------------------ > > >> -------------------- > > >> > > >> Two observations: > > >> > > >> - __ipow__ doesn't seem to behave properly in the 3.x line (that > error > > >> doesn't show up when testing against 2.7) > > >> > > >> - Counter should be returning NotImplemented instead of raising an > > >> AttributeError, for three reasons [4]: > > >> - a TypeError is more appropriate > > >> - subclasses /cannot/ work with the current implementation > > >> - __iand__ is currently a silent failure if the Counter is empty, > > >> and the other operand should trigger a failure > > >> > > >> Back to the main point... > > >> > > >> So, if my understanding is correct: > > >> > > >> - NotImplemented is used to signal Python that the requested > operation > > >> could not be performed > > >> - it should be used by the binary special methods to signal type > > >> mismatch failure, so any subclass gets a chance to work. > > >> > > >> Is my understanding correct? Is this already in the docs somewhere, > and > > >> I just missed it? > > >> > > >> -- > > >> ~Ethan~ > > >> > > >> [1] at least, it's the main reason in my code > > >> [2] usually a TypeError, stating either that the operation is not > > >> supported, or the types are unorderable > > >> [3] test script at the end > > >> [4] https://bugs.python.org/issue22766 [returning NotImplemented was > > >> rejected] > > >> > > >> -- 8< ------------------------------------------------------------ > > >> ---------------- > > >> from collections import Counter, defaultdict, deque, OrderedDict > > >> from fractions import Fraction > > >> from decimal import Decimal > > >> from enum import Enum > > >> import operator > > >> import sys > > >> > > >> py_ver = sys.version_info[:2] > > >> > > >> types = ( > > >> bytes, bytearray, str, dict, list, tuple, > > >> Enum, Counter, defaultdict, deque, OrderedDict, > > >> ) > > >> numeric_types = int, float, Decimal, Fraction > > >> > > >> operators = ( > > >> '__add__', '__and__', '__floordiv__', > > >> '__iadd__', '__iand__', '__ifloordiv__', '__ilshift__', > > >> '__imod__', '__imul__', '__ior__', '__ipow__', > > >> '__irshift__', '__isub__', '__itruediv__', '__ixor__', > > >> '__lshift__', '__mod__', '__mul__', > > >> '__or__', '__pow__', '__rshift__', '__sub__', '__truediv__', > > >> '__xor__', > > >> ) > > >> > > >> if py_ver >= (3, 0): > > >> operators += ('__gt__', '__ge__', '__le__','__lt__') > > >> > > >> ordered_reflections = { > > >> '__le__': '__ge__', > > >> '__lt__': '__gt__', > > >> '__ge__': '__le__', > > >> '__gt__': '__lt__', > > >> } > > >> > > >> > > >> # helpers > > >> > > >> class SomeOtherClass: > > >> """" > > >> used to test behavior when a different type is passed in to the > > >> special methods > > >> """ > > >> def __repr__(self): > > >> return 'SomeOtherClass' > > >> some_other_class = SomeOtherClass() > > >> > > >> class MainClassHandled(Exception): > > >> """ > > >> called by base class if both operands are of type base class > > >> """ > > >> > > >> class SubClassCalled(Exception): > > >> """ > > >> called by reflected operations for testing > > >> """ > > >> > > >> def create_control(test_op): > > >> def _any(self, other): > > >> if not type(other) is self.__class__: > > >> return NotImplemented > > >> raise MainClassHandled > > >> class Control: > > >> "returns NotImplemented when other object is not supported" > > >> _any.__name__ = op > > >> setattr(Control, test_op, _any) > > >> return Control() > > >> > > >> def create_subtype(test_op, base_class=object): > > >> def _any(*a): > > >> global subclass_called > > >> subclass_called = True > > >> raise SubClassCalled > > >> class subtype(base_class): > > >> __add__ = __sub__ = __mul__ = __truediv__ = __floordiv__ = > _any > > >> __mod__ = __divmod__ = __pow__ = __lshift__ = __rshift__ = > _any > > >> __and__ = __xor__ = __or__ = _any > > >> __radd__ = __rsub__ = __rmul__ = __rtruediv__ = > __rfloordiv__ = > > >> _any > > >> __rmod__ = __rdivmod__ = __rpow__ = __rlshift__ = > __rrshift__ = > > >> _any > > >> __rand__ = __rxor__ = __ror__ = _any > > >> __le__ = __lt__ = __gt__ = __ge__ = _any > > >> if issubclass(subtype, (bytes, bytearray)): > > >> value = b'hello' > > >> elif issubclass(subtype, str): > > >> value = 'goodbye' > > >> elif issubclass(subtype, (list, tuple)): > > >> value = (1, 2, 3) > > >> elif issubclass(subtype, (int, float, Decimal, Fraction)): > > >> value = 42 > > >> else: > > >> # ignore value > > >> return subtype() > > >> return subtype(value) > > >> > > >> > > >> # test exceptions > > >> > > >> # control against some other class > > >> print('testing control...\n') > > >> errors = False > > >> for op in operators: > > >> control = create_control(op) > > >> op = getattr(operator, op) > > >> try: > > >> op(control, some_other_class) > > >> except TypeError: > > >> # the end result of no method existing, or each method called > > >> returning > > >> # NotImplemented because it does not know how to perform the > > >> requested > > >> # operation between the two types > > >> pass > > >> except Exception as exc: > > >> errors = True > > >> print('%s(%s()) -- Exception <%s> raised instead of > TypeError' % > > >> (op.__name__, test_type.__name__, exc)) > > >> else: > > >> errors = True > > >> print('Control -- TypeError not raised for op %r' % op) > > >> if errors: > > >> print('errors in Control -- misunderstanding or bug?\n') > > >> > > >> # control against a subclass > > >> errors = False > > >> for op in operators: > > >> subclass_called = False > > >> control = create_control(op) > > >> subtype = create_subtype(op, control.__class__) > > >> op = getattr(operator, op) > > >> try: > > >> op(control, subtype) > > >> except SubClassCalled: > > >> # if the control class properly signals that it doesn't know > how > > >> to > > >> # perform the operation, of if Python notices that a > reflected > > >> # operation exists, we get here (which is good) > > >> pass > > >> except MainClassHandled: > > >> errors = True > > >> print('Control did not yield to subclass for op %r' % op) > > >> except Exception as exc: > > >> if subclass_called: > > >> # exception was subverted to something more appropriate > (like > > >> # unorderable types) > > >> pass > > >> errors = True > > >> print('%s -- Exception <%s> raised' % > > >> (op.__name__, exc)) > > >> else: > > >> errors = True > > >> print('Control -- op %r appears to have succeeded (it should > not > > >> have)' % op) > > >> if errors: > > >> print('errors in Control -- misunderstanding or bug?\n') > > >> > > >> > > >> # tests > > >> print('testing types against a foreign class\n') > > >> for test_type in types + numeric_types: > > >> errors = False > > >> for op in operators: > > >> op = getattr(operator, op) > > >> try: > > >> op(test_type(), some_other_class) > > >> except TypeError: > > >> pass > > >> except Exception as exc: > > >> errors = True > > >> print('%s(%s()) -- Exception <%s> raised instead of > > >> TypeError' % > > >> (op.__name__, test_type.__name__, exc)) > > >> else: > > >> print('%s(%s()) -- NotImplemented not returned, TypeError > > >> not raised' % > > >> (op.__name__, test_type.__name__)) > > >> if errors: > > >> print() > > >> > > >> print() > > >> > > >> # test subclasses > > >> print('testing types against a subclass\n') > > >> for test_type in types: > > >> errors = False > > >> for op in operators: > > >> subclass_called = False > > >> if not test_type.__dict__.get(op): > > >> continue > > >> subclass = create_subtype(op, test_type) > > >> op = getattr(operator, op) > > >> try: > > >> if test_type is str: > > >> op('%s', subtype) > > >> else: > > >> op(test_type(), subtype) > > >> except SubClassCalled: > > >> # expected, ignore > > >> pass > > >> except Exception as exc: > > >> if subclass_called: > > >> # exception raised by subclass was changed > > >> pass > > >> errors = True > > >> print('%s(%s()) -- Exception <%s> raised (should have > > >> worked)' % > > >> (op.__name__, test_type.__name__, exc)) > > >> else: > > >> errors = True > > >> print('%s(%s()) -- NotImplemented not returned, TypeError > > >> not raised' % > > >> (op.__name__, test_type.__name__)) > > >> if errors: > > >> print() > > >> for test_type in numeric_types: > > >> errors = False > > >> for op in operators: > > >> subclass_called = False > > >> if not test_type.__dict__.get(op): > > >> continue > > >> subtype = create_subtype(op, test_type) > > >> op = getattr(operator, op) > > >> try: > > >> op(test_type(), subtype) > > >> except SubClassCalled: > > >> # expected, ignore > > >> pass > > >> except Exception as exc: > > >> if subclass_called: > > >> # exception raised by subclass was changed > > >> pass > > >> errors = True > > >> print('%s(%s()) -- Exception <%s> raised (should have > > >> worked)' % > > >> (op.__name__, test_type.__name__, exc)) > > >> else: > > >> errors = True > > >> print('%s(%s)) -- NotImplemented not returned' % > > >> (op.__name__, test_type.__name__)) > > >> if errors: > > >> print() > > >> -- 8< ------------------------------------------------------------ > > >> ---------------- > > >> _______________________________________________ > > >> Python-Dev mailing list > > >> Python-Dev at python.org > > >> https://mail.python.org/mailman/listinfo/python-dev > > >> Unsubscribe: https://mail.python.org/mailman/options/python-dev/ > > >> brett%40python.org > > >> > > > > > > _______________________________________________ > > > Python-Dev mailing list > > > Python-Dev at python.org > > > https://mail.python.org/mailman/listinfo/python-dev > > > Unsubscribe: > > > https://mail.python.org/mailman/options/python-dev/guido%40python.org > > > > > > > > > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/guido%40python.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Mon Nov 3 18:05:43 2014 From: guido at python.org (Guido van Rossum) Date: Mon, 3 Nov 2014 09:05:43 -0800 Subject: [Python-Dev] The role of NotImplemented: What is it for and when should it be used? In-Reply-To: References: <5457595D.4030107@stoneleaf.us> <20141103175510.6059f838@fsol> Message-ID: Sorry, was too quick. For immutable types __iop__ may not exist and then the fallback machinery should work normally using NotImplemented. But if __iop__ exists it can choose not to allow __rop__, because the type would presumably change. This is probably more predictable. I don't even know if the byte code interpreter looks for Not implemented from __iop__. On Nov 3, 2014 9:00 AM, "Guido van Rossum" wrote: > Not those. > On Nov 3, 2014 8:56 AM, "Antoine Pitrou" wrote: > >> On Mon, 3 Nov 2014 08:48:07 -0800 >> Guido van Rossum wrote: >> > Gotta be brief, but NotImplemented is for all binary ops. >> >> Even in-place ops? >> >> Regards >> >> Antoine. >> >> >> > Power may be an >> > exception because it's ternary? >> > On Nov 3, 2014 8:08 AM, "Brett Cannon" wrote: >> > >> > > >> > > >> > > On Mon Nov 03 2014 at 5:31:21 AM Ethan Furman >> wrote: >> > > >> > >> Just to be clear, this is about NotImplemented, not >> NotImplementedError. >> > >> >> > >> tl;dr When a binary operation fails, should an exception be raised >> or >> > >> NotImplemented returned? >> > >> >> > > >> > > The docs for NotImplemented suggest it's only for rich comparison >> methods >> > > and not all binary operators: >> > > https://docs.python.org/3/library/constants.html#NotImplemented . But >> > > then had I not read that I would have said all binary operator methods >> > > should return NotImplemented when the types are incompatible. >> > > >> > > -Brett >> > > >> > > >> > >> >> > >> >> > >> When a binary operation in Python is attempted, there are two >> > >> possibilities: >> > >> >> > >> - it can work >> > >> - it can't work >> > >> >> > >> The main reason [1] that it can't work is that the two operands are >> of >> > >> different types, and the first type does not know >> > >> how to deal with the second type. >> > >> >> > >> The question then becomes: how does the first type tell Python that >> it >> > >> cannot perform the requested operation? The most >> > >> obvious answer is to raise an exception, and TypeError is a good >> > >> candidate. The problem with the exception raising >> > >> approach is that once an exception is raised, Python doesn't try >> anything >> > >> else to make the operation work. >> > >> >> > >> What's wrong with that? Well, the second type might know how to >> perform >> > >> the operation, and in fact that is why we have >> > >> the reflected special methods, such as __radd__ and __rmod__ -- but >> if >> > >> the first type raises an exception the __rxxx__ >> > >> methods will not be tried. >> > >> >> > >> Okay, how can the first type tell Python that it cannot do what is >> > >> requested, but to go ahead and check with the second >> > >> type to see if it does? That is where NotImplemented comes in -- if >> a >> > >> special method (and only a special method) >> > >> returns NotImplemented then Python will check to see if there is >> anything >> > >> else it can do to make the operation succeed; >> > >> if all attempts return NotImplemented, then Python itself will raise >> an >> > >> appropriate exception [2]. >> > >> >> > >> In an effort to see how often NotImplemented is currently being >> returned >> > >> I crafted a test script [3] to test the types >> > >> bytes, bytearray, str, dict, list, tuple, Enum, Counter, defaultdict, >> > >> deque, and OrderedDict with the operations for >> > >> __add__, __and__, __floordiv__, __iadd__, __iand__, __ifloordiv__, >> > >> __ilshift__, __imod__, __imul__, __ior__, __ipow__, >> > >> __irshift__, __isub__, __itruediv__, __ixor__, __lshift__, __mod__, >> > >> __mul__, __or__, __pow__, __rshift__, __sub__, >> > >> __truediv__, and __xor__. >> > >> >> > >> Here are the results of the 275 tests: >> > >> ------------------------------------------------------------ >> > >> -------------------- >> > >> testing control... >> > >> >> > >> ipow -- Exception > 'Control' >> > >> and 'subtype'> raised >> > >> errors in Control -- misunderstanding or bug? >> > >> >> > >> testing types against a foreign class >> > >> >> > >> iadd(Counter()) -- Exception <'SomeOtherClass' object has no >> attribute >> > >> 'items'> raised instead of TypeError >> > >> iand(Counter()) -- NotImplemented not returned, TypeError not raised >> > >> ior(Counter()) -- Exception <'SomeOtherClass' object has no attribute >> > >> 'items'> raised instead of TypeError >> > >> isub(Counter()) -- Exception <'SomeOtherClass' object has no >> attribute >> > >> 'items'> raised instead of TypeError >> > >> >> > >> >> > >> testing types against a subclass >> > >> >> > >> mod(str()) -- NotImplemented not returned, TypeError not raised >> > >> >> > >> iadd(Counter()) -- Exception <'subtype' object has no attribute >> 'items'> >> > >> raised (should have worked) >> > >> iand(Counter()) -- NotImplemented not returned, TypeError not raised >> > >> ior(Counter()) -- Exception <'subtype' object has no attribute >> 'items'> >> > >> raised (should have worked) >> > >> isub(Counter()) -- Exception <'subtype' object has no attribute >> 'items'> >> > >> raised (should have worked) >> > >> ------------------------------------------------------------ >> > >> -------------------- >> > >> >> > >> Two observations: >> > >> >> > >> - __ipow__ doesn't seem to behave properly in the 3.x line (that >> error >> > >> doesn't show up when testing against 2.7) >> > >> >> > >> - Counter should be returning NotImplemented instead of raising an >> > >> AttributeError, for three reasons [4]: >> > >> - a TypeError is more appropriate >> > >> - subclasses /cannot/ work with the current implementation >> > >> - __iand__ is currently a silent failure if the Counter is >> empty, >> > >> and the other operand should trigger a failure >> > >> >> > >> Back to the main point... >> > >> >> > >> So, if my understanding is correct: >> > >> >> > >> - NotImplemented is used to signal Python that the requested >> operation >> > >> could not be performed >> > >> - it should be used by the binary special methods to signal type >> > >> mismatch failure, so any subclass gets a chance to work. >> > >> >> > >> Is my understanding correct? Is this already in the docs somewhere, >> and >> > >> I just missed it? >> > >> >> > >> -- >> > >> ~Ethan~ >> > >> >> > >> [1] at least, it's the main reason in my code >> > >> [2] usually a TypeError, stating either that the operation is not >> > >> supported, or the types are unorderable >> > >> [3] test script at the end >> > >> [4] https://bugs.python.org/issue22766 [returning NotImplemented was >> > >> rejected] >> > >> >> > >> -- 8< ------------------------------------------------------------ >> > >> ---------------- >> > >> from collections import Counter, defaultdict, deque, OrderedDict >> > >> from fractions import Fraction >> > >> from decimal import Decimal >> > >> from enum import Enum >> > >> import operator >> > >> import sys >> > >> >> > >> py_ver = sys.version_info[:2] >> > >> >> > >> types = ( >> > >> bytes, bytearray, str, dict, list, tuple, >> > >> Enum, Counter, defaultdict, deque, OrderedDict, >> > >> ) >> > >> numeric_types = int, float, Decimal, Fraction >> > >> >> > >> operators = ( >> > >> '__add__', '__and__', '__floordiv__', >> > >> '__iadd__', '__iand__', '__ifloordiv__', '__ilshift__', >> > >> '__imod__', '__imul__', '__ior__', '__ipow__', >> > >> '__irshift__', '__isub__', '__itruediv__', '__ixor__', >> > >> '__lshift__', '__mod__', '__mul__', >> > >> '__or__', '__pow__', '__rshift__', '__sub__', '__truediv__', >> > >> '__xor__', >> > >> ) >> > >> >> > >> if py_ver >= (3, 0): >> > >> operators += ('__gt__', '__ge__', '__le__','__lt__') >> > >> >> > >> ordered_reflections = { >> > >> '__le__': '__ge__', >> > >> '__lt__': '__gt__', >> > >> '__ge__': '__le__', >> > >> '__gt__': '__lt__', >> > >> } >> > >> >> > >> >> > >> # helpers >> > >> >> > >> class SomeOtherClass: >> > >> """" >> > >> used to test behavior when a different type is passed in to the >> > >> special methods >> > >> """ >> > >> def __repr__(self): >> > >> return 'SomeOtherClass' >> > >> some_other_class = SomeOtherClass() >> > >> >> > >> class MainClassHandled(Exception): >> > >> """ >> > >> called by base class if both operands are of type base class >> > >> """ >> > >> >> > >> class SubClassCalled(Exception): >> > >> """ >> > >> called by reflected operations for testing >> > >> """ >> > >> >> > >> def create_control(test_op): >> > >> def _any(self, other): >> > >> if not type(other) is self.__class__: >> > >> return NotImplemented >> > >> raise MainClassHandled >> > >> class Control: >> > >> "returns NotImplemented when other object is not supported" >> > >> _any.__name__ = op >> > >> setattr(Control, test_op, _any) >> > >> return Control() >> > >> >> > >> def create_subtype(test_op, base_class=object): >> > >> def _any(*a): >> > >> global subclass_called >> > >> subclass_called = True >> > >> raise SubClassCalled >> > >> class subtype(base_class): >> > >> __add__ = __sub__ = __mul__ = __truediv__ = __floordiv__ = >> _any >> > >> __mod__ = __divmod__ = __pow__ = __lshift__ = __rshift__ = >> _any >> > >> __and__ = __xor__ = __or__ = _any >> > >> __radd__ = __rsub__ = __rmul__ = __rtruediv__ = >> __rfloordiv__ = >> > >> _any >> > >> __rmod__ = __rdivmod__ = __rpow__ = __rlshift__ = >> __rrshift__ = >> > >> _any >> > >> __rand__ = __rxor__ = __ror__ = _any >> > >> __le__ = __lt__ = __gt__ = __ge__ = _any >> > >> if issubclass(subtype, (bytes, bytearray)): >> > >> value = b'hello' >> > >> elif issubclass(subtype, str): >> > >> value = 'goodbye' >> > >> elif issubclass(subtype, (list, tuple)): >> > >> value = (1, 2, 3) >> > >> elif issubclass(subtype, (int, float, Decimal, Fraction)): >> > >> value = 42 >> > >> else: >> > >> # ignore value >> > >> return subtype() >> > >> return subtype(value) >> > >> >> > >> >> > >> # test exceptions >> > >> >> > >> # control against some other class >> > >> print('testing control...\n') >> > >> errors = False >> > >> for op in operators: >> > >> control = create_control(op) >> > >> op = getattr(operator, op) >> > >> try: >> > >> op(control, some_other_class) >> > >> except TypeError: >> > >> # the end result of no method existing, or each method >> called >> > >> returning >> > >> # NotImplemented because it does not know how to perform the >> > >> requested >> > >> # operation between the two types >> > >> pass >> > >> except Exception as exc: >> > >> errors = True >> > >> print('%s(%s()) -- Exception <%s> raised instead of >> TypeError' % >> > >> (op.__name__, test_type.__name__, exc)) >> > >> else: >> > >> errors = True >> > >> print('Control -- TypeError not raised for op %r' % op) >> > >> if errors: >> > >> print('errors in Control -- misunderstanding or bug?\n') >> > >> >> > >> # control against a subclass >> > >> errors = False >> > >> for op in operators: >> > >> subclass_called = False >> > >> control = create_control(op) >> > >> subtype = create_subtype(op, control.__class__) >> > >> op = getattr(operator, op) >> > >> try: >> > >> op(control, subtype) >> > >> except SubClassCalled: >> > >> # if the control class properly signals that it doesn't >> know how >> > >> to >> > >> # perform the operation, of if Python notices that a >> reflected >> > >> # operation exists, we get here (which is good) >> > >> pass >> > >> except MainClassHandled: >> > >> errors = True >> > >> print('Control did not yield to subclass for op %r' % op) >> > >> except Exception as exc: >> > >> if subclass_called: >> > >> # exception was subverted to something more appropriate >> (like >> > >> # unorderable types) >> > >> pass >> > >> errors = True >> > >> print('%s -- Exception <%s> raised' % >> > >> (op.__name__, exc)) >> > >> else: >> > >> errors = True >> > >> print('Control -- op %r appears to have succeeded (it >> should not >> > >> have)' % op) >> > >> if errors: >> > >> print('errors in Control -- misunderstanding or bug?\n') >> > >> >> > >> >> > >> # tests >> > >> print('testing types against a foreign class\n') >> > >> for test_type in types + numeric_types: >> > >> errors = False >> > >> for op in operators: >> > >> op = getattr(operator, op) >> > >> try: >> > >> op(test_type(), some_other_class) >> > >> except TypeError: >> > >> pass >> > >> except Exception as exc: >> > >> errors = True >> > >> print('%s(%s()) -- Exception <%s> raised instead of >> > >> TypeError' % >> > >> (op.__name__, test_type.__name__, exc)) >> > >> else: >> > >> print('%s(%s()) -- NotImplemented not returned, >> TypeError >> > >> not raised' % >> > >> (op.__name__, test_type.__name__)) >> > >> if errors: >> > >> print() >> > >> >> > >> print() >> > >> >> > >> # test subclasses >> > >> print('testing types against a subclass\n') >> > >> for test_type in types: >> > >> errors = False >> > >> for op in operators: >> > >> subclass_called = False >> > >> if not test_type.__dict__.get(op): >> > >> continue >> > >> subclass = create_subtype(op, test_type) >> > >> op = getattr(operator, op) >> > >> try: >> > >> if test_type is str: >> > >> op('%s', subtype) >> > >> else: >> > >> op(test_type(), subtype) >> > >> except SubClassCalled: >> > >> # expected, ignore >> > >> pass >> > >> except Exception as exc: >> > >> if subclass_called: >> > >> # exception raised by subclass was changed >> > >> pass >> > >> errors = True >> > >> print('%s(%s()) -- Exception <%s> raised (should have >> > >> worked)' % >> > >> (op.__name__, test_type.__name__, exc)) >> > >> else: >> > >> errors = True >> > >> print('%s(%s()) -- NotImplemented not returned, >> TypeError >> > >> not raised' % >> > >> (op.__name__, test_type.__name__)) >> > >> if errors: >> > >> print() >> > >> for test_type in numeric_types: >> > >> errors = False >> > >> for op in operators: >> > >> subclass_called = False >> > >> if not test_type.__dict__.get(op): >> > >> continue >> > >> subtype = create_subtype(op, test_type) >> > >> op = getattr(operator, op) >> > >> try: >> > >> op(test_type(), subtype) >> > >> except SubClassCalled: >> > >> # expected, ignore >> > >> pass >> > >> except Exception as exc: >> > >> if subclass_called: >> > >> # exception raised by subclass was changed >> > >> pass >> > >> errors = True >> > >> print('%s(%s()) -- Exception <%s> raised (should have >> > >> worked)' % >> > >> (op.__name__, test_type.__name__, exc)) >> > >> else: >> > >> errors = True >> > >> print('%s(%s)) -- NotImplemented not returned' % >> > >> (op.__name__, test_type.__name__)) >> > >> if errors: >> > >> print() >> > >> -- 8< ------------------------------------------------------------ >> > >> ---------------- >> > >> _______________________________________________ >> > >> Python-Dev mailing list >> > >> Python-Dev at python.org >> > >> https://mail.python.org/mailman/listinfo/python-dev >> > >> Unsubscribe: https://mail.python.org/mailman/options/python-dev/ >> > >> brett%40python.org >> > >> >> > > >> > > _______________________________________________ >> > > Python-Dev mailing list >> > > Python-Dev at python.org >> > > https://mail.python.org/mailman/listinfo/python-dev >> > > Unsubscribe: >> > > https://mail.python.org/mailman/options/python-dev/guido%40python.org >> > > >> > > >> > >> >> >> _______________________________________________ >> Python-Dev mailing list >> Python-Dev at python.org >> https://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe: >> https://mail.python.org/mailman/options/python-dev/guido%40python.org >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Mon Nov 3 18:10:59 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 3 Nov 2014 18:10:59 +0100 Subject: [Python-Dev] The role of NotImplemented: What is it for and when should it be used? In-Reply-To: References: <5457595D.4030107@stoneleaf.us> <20141103175510.6059f838@fsol> Message-ID: <20141103181059.7a1bb0d9@fsol> On Mon, 3 Nov 2014 09:05:43 -0800 Guido van Rossum wrote: > Sorry, was too quick. For immutable types __iop__ may not exist and then > the fallback machinery should work normally using NotImplemented. But if > __iop__ exists it can choose not to allow __rop__, because the type would > presumably change. This is probably more predictable. I don't even know if > the byte code interpreter looks for Not implemented from __iop__. Apparently it can tell it to fallback on __op__: >>> class C(list): ... def __iadd__(self, other): ... print("here") ... return NotImplemented ... >>> c = C() >>> c += [1] here >>> c [1] >>> type(c) Regards Antoine. From guido at python.org Mon Nov 3 19:00:24 2014 From: guido at python.org (Guido van Rossum) Date: Mon, 3 Nov 2014 10:00:24 -0800 Subject: [Python-Dev] The role of NotImplemented: What is it for and when should it be used? In-Reply-To: <20141103181059.7a1bb0d9@fsol> References: <5457595D.4030107@stoneleaf.us> <20141103175510.6059f838@fsol> <20141103181059.7a1bb0d9@fsol> Message-ID: That must be so that an immutable type can still implement __iop__ as an optimization. On Mon, Nov 3, 2014 at 9:10 AM, Antoine Pitrou wrote: > On Mon, 3 Nov 2014 09:05:43 -0800 > Guido van Rossum wrote: > > Sorry, was too quick. For immutable types __iop__ may not exist and then > > the fallback machinery should work normally using NotImplemented. But if > > __iop__ exists it can choose not to allow __rop__, because the type would > > presumably change. This is probably more predictable. I don't even know > if > > the byte code interpreter looks for Not implemented from __iop__. > > Apparently it can tell it to fallback on __op__: > > >>> class C(list): > ... def __iadd__(self, other): > ... print("here") > ... return NotImplemented > ... > >>> c = C() > >>> c += [1] > here > >>> c > [1] > >>> type(c) > > > > Regards > > Antoine. > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/guido%40python.org > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From ethan at stoneleaf.us Mon Nov 3 20:33:34 2014 From: ethan at stoneleaf.us (Ethan Furman) Date: Mon, 03 Nov 2014 11:33:34 -0800 Subject: [Python-Dev] The role of NotImplemented: What is it for and when should it be used? In-Reply-To: References: <5457595D.4030107@stoneleaf.us> <20141103175510.6059f838@fsol> <20141103181059.7a1bb0d9@fsol> Message-ID: <5457D88E.3040408@stoneleaf.us> Summary: NotImplemented _should_ be used by the normal and reflected binary methods (__lt__, __add__, __xor__, __rsub__, etc.) NotImplemented _may_ be used by the in-place binary methods (__iadd__, __ixor__, etc.), but the in-place methods are also free to raise an exception. Correct? -- ~Ethan~ From guido at python.org Mon Nov 3 21:06:08 2014 From: guido at python.org (Guido van Rossum) Date: Mon, 3 Nov 2014 12:06:08 -0800 Subject: [Python-Dev] The role of NotImplemented: What is it for and when should it be used? In-Reply-To: <5457D88E.3040408@stoneleaf.us> References: <5457595D.4030107@stoneleaf.us> <20141103175510.6059f838@fsol> <20141103181059.7a1bb0d9@fsol> <5457D88E.3040408@stoneleaf.us> Message-ID: Sounds good! On Mon, Nov 3, 2014 at 11:33 AM, Ethan Furman wrote: > Summary: > > NotImplemented _should_ be used by the normal and reflected binary methods > (__lt__, __add__, __xor__, __rsub__, etc.) > > NotImplemented _may_ be used by the in-place binary methods (__iadd__, > __ixor__, etc.), but the in-place methods are also free to raise an > exception. > > Correct? > > -- > ~Ethan~ > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ > guido%40python.org > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From romapera15 at gmail.com Tue Nov 4 00:19:46 2014 From: romapera15 at gmail.com (=?UTF-8?Q?fran=C3=A7ai_s?=) Date: Mon, 3 Nov 2014 21:19:46 -0200 Subject: [Python-Dev] [OFF-TOPIC] It is true that is impossible write in binary code, the lowest level of programming that you can write is in hex code? Message-ID: I intend to write in lowest level of computer programming as a hobby. It is true that is impossible write in binary code, the lowest level of programming that you can write is in hex code? What is the lowest level of programming computers that you can write ? Is binary code? Is hex code? Is another machine code? Honestly do not know if it is true that there is another machine code beyond the binary and hex code. Is Assembly? -------------- next part -------------- An HTML attachment was scrubbed... URL: From rymg19 at gmail.com Tue Nov 4 01:05:01 2014 From: rymg19 at gmail.com (Ryan Gonzalez) Date: Mon, 3 Nov 2014 18:05:01 -0600 Subject: [Python-Dev] [OFF-TOPIC] It is true that is impossible write in binary code, the lowest level of programming that you can write is in hex code? In-Reply-To: References: Message-ID: Wait...you sent this to 9fans AND Python Dev??? Why not ask on StackExchange Programmers or something? On Mon, Nov 3, 2014 at 5:19 PM, fran?ai s wrote: > I intend to write in lowest level of computer programming as a hobby. > > It is true that is impossible write in binary code, the lowest level of > programming that you can write is in hex code? > > What is the lowest level of programming computers that you can write ? > > Is binary code? > > Is hex code? > > Is another machine code? Honestly do not know if it is true that there is > another machine code beyond the binary and hex code. > > Is Assembly? > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/rymg19%40gmail.com > > -- Ryan If anybody ever asks me why I prefer C++ to C, my answer will be simple: "It's becauseslejfp23(@#Q*(E*EIdc-SEGFAULT. Wait, I don't think that was nul-terminated." Personal reality distortion fields are immune to contradictory evidence. - srean Check out my website: http://kirbyfan64.github.io/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Tue Nov 4 01:08:21 2014 From: steve at pearwood.info (Steven D'Aprano) Date: Tue, 4 Nov 2014 11:08:21 +1100 Subject: [Python-Dev] [OFF-TOPIC] It is true that is impossible write in binary code, the lowest level of programming that you can write is in hex code? In-Reply-To: References: Message-ID: <20141104000821.GA17635@ando.pearwood.info> This is off-topic for this mailing list, as you know. There are some mailing lists which approve of off-topic conversations, but this is not one of those. You could ask on the python-list at python.org mailing list, where it will still be off-topic, but the people there are more likely to answer. But even better would be to look for a mailing list or forum for assembly programming, machine code, or micro-code. On Mon, Nov 03, 2014 at 09:19:46PM -0200, fran?ai s wrote: > I intend to write in lowest level of computer programming as a hobby. [...] -- Steven From ncoghlan at gmail.com Tue Nov 4 12:15:02 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 4 Nov 2014 21:15:02 +1000 Subject: [Python-Dev] The role of NotImplemented: What is it for and when should it be used? Message-ID: It's worth noting that as far as I am aware, all the cases where CPython currently raises TypeError directly rather than returning NotImplemented are due to a longstanding bug in the handling concatenation and repetition of sequences implemented entirely in C (which includes the builtins): http://bugs.python.org/issue11477 The coercion dance in abstract.c currently gets the operand precedence wrong for sq_concat and sq_repeat, and also doesn't check their return values for NotImplemented. Types implemented in Python work correctly (including respecting NotImplemented return values), as those fill in both the nb_* and the sq_* slots. That's why this bug is in the "annoying implementation quirk" category rather than the "major correctness flaw" category. I tried to fix it ages ago directly in abstract.c, but the result was an unmaintainable mess. (The test changes in the draft patch will hopefully still prove useful some day). There's another possible implementation strategy, which is to change type creation to populate nb_add and nb_multiply when sq_concat and sq_repeat are defined, and then never call those two methods directly from abstract.c. I've never found the time myself to go back and try that version of the fix, but it's definitely an issue I'd love to see fixed at some point. Regards, Nick. -------------- next part -------------- An HTML attachment was scrubbed... URL: From benjamin at python.org Tue Nov 4 14:58:29 2014 From: benjamin at python.org (Benjamin Peterson) Date: Tue, 04 Nov 2014 09:58:29 -0400 Subject: [Python-Dev] [Python-checkins] cpython (2.7): Issue #22773: fix failing test with old readline versions due to issue #19884. In-Reply-To: <20141104135456.108393.94114@psf.io> References: <20141104135456.108393.94114@psf.io> Message-ID: <1415109509.2950700.186901193.57C2BE56@webmail.messagingengine.com> On Tue, Nov 4, 2014, at 09:55, antoine.pitrou wrote: > https://hg.python.org/cpython/rev/eba6e68e818c > changeset: 93382:eba6e68e818c > branch: 2.7 > parent: 93379:e54d0b197c82 > user: Antoine Pitrou > date: Tue Nov 04 14:52:10 2014 +0100 > summary: > Issue #22773: fix failing test with old readline versions due to issue > #19884. > > files: > Lib/test/test_readline.py | 4 ++++ > Modules/readline.c | 3 +++ > 2 files changed, 7 insertions(+), 0 deletions(-) > > > diff --git a/Lib/test/test_readline.py b/Lib/test/test_readline.py > --- a/Lib/test/test_readline.py > +++ b/Lib/test/test_readline.py > @@ -43,6 +43,10 @@ > > > class TestReadline(unittest.TestCase): > + > + @unittest.skipIf(readline._READLINE_VERSION < 0x0600 Shouldn't this use the runtime version? From solipsis at pitrou.net Tue Nov 4 15:12:36 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 4 Nov 2014 15:12:36 +0100 Subject: [Python-Dev] [Python-checkins] cpython (2.7): Issue #22773: fix failing test with old readline versions due to issue #19884. References: <20141104135456.108393.94114@psf.io> <1415109509.2950700.186901193.57C2BE56@webmail.messagingengine.com> Message-ID: <20141104151236.6be0645d@fsol> On Tue, 04 Nov 2014 09:58:29 -0400 Benjamin Peterson wrote: > > On Tue, Nov 4, 2014, at 09:55, antoine.pitrou wrote: > > https://hg.python.org/cpython/rev/eba6e68e818c > > changeset: 93382:eba6e68e818c > > branch: 2.7 > > parent: 93379:e54d0b197c82 > > user: Antoine Pitrou > > date: Tue Nov 04 14:52:10 2014 +0100 > > summary: > > Issue #22773: fix failing test with old readline versions due to issue > > #19884. > > > > files: > > Lib/test/test_readline.py | 4 ++++ > > Modules/readline.c | 3 +++ > > 2 files changed, 7 insertions(+), 0 deletions(-) > > > > > > diff --git a/Lib/test/test_readline.py b/Lib/test/test_readline.py > > --- a/Lib/test/test_readline.py > > +++ b/Lib/test/test_readline.py > > @@ -43,6 +43,10 @@ > > > > > > class TestReadline(unittest.TestCase): > > + > > + @unittest.skipIf(readline._READLINE_VERSION < 0x0600 > > Shouldn't this use the runtime version? I suppose in most libreadline deployments it shouldn't make a difference (you usually get it through your Linux distro; other OSes use non-GNU replacements, if any). Regards Antoine. From robertomartinezp at gmail.com Tue Nov 4 17:52:39 2014 From: robertomartinezp at gmail.com (=?ISO-8859-1?Q?Roberto_Mart=EDnez?=) Date: Tue, 4 Nov 2014 17:52:39 +0100 Subject: [Python-Dev] Dinamically set __call__ method Message-ID: Hi folks, I am trying to replace dinamically the __call__ method of an object using setattr. Example: $ cat testcall.py class A: def __init__(self): setattr(self, '__call__', self.newcall) def __call__(self): print("OLD") def newcall(self): print("NEW") a=A() a() I expect to get "NEW" instead of "OLD", but in Python 3.4 I get "OLD". $ python2.7 testcall.py NEW $ python3.4 testcall.py OLD I have a few questions: - Is this an expected behavior? - Is possible to replace __call__ dinamically in Python 3? How? Best regards, Roberto -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Tue Nov 4 19:22:12 2014 From: njs at pobox.com (Nathaniel Smith) Date: Tue, 4 Nov 2014 18:22:12 +0000 Subject: [Python-Dev] Dinamically set __call__ method In-Reply-To: References: Message-ID: On Tue, Nov 4, 2014 at 4:52 PM, Roberto Mart?nez wrote: > Hi folks, > > I am trying to replace dinamically the __call__ method of an object using > setattr. > Example: > > $ cat testcall.py > class A: > def __init__(self): > setattr(self, '__call__', self.newcall) > > def __call__(self): > print("OLD") > > def newcall(self): > print("NEW") > > a=A() > a() > > I expect to get "NEW" instead of "OLD", but in Python 3.4 I get "OLD". > > $ python2.7 testcall.py > NEW > $ python3.4 testcall.py > OLD > > I have a few questions: > > - Is this an expected behavior? > - Is possible to replace __call__ dinamically in Python 3? How? For new-style classes, special methods like __call__ are looked up directly on the class, not on the object itself. If you want to change the result of doing a(), then you need to reassign A.__call__, not a.__call__. In python 3, all classes are new-style classes. In python 2, only classes that inherit from 'object' are new-style classes. (If you replace 'class A:' with 'class A(object):' then you'll see the same behaviour on both py2 and py3.) Easy workaround: def __call__(self, *args, **kwargs): return self._my_call(*args, **kwargs) Now you can assign a._my_call to be whatever you want. -n -- Nathaniel J. Smith Postdoctoral researcher - Informatics - University of Edinburgh http://vorpus.org From ethan at stoneleaf.us Tue Nov 4 19:23:50 2014 From: ethan at stoneleaf.us (Ethan Furman) Date: Tue, 04 Nov 2014 10:23:50 -0800 Subject: [Python-Dev] Dinamically set __call__ method In-Reply-To: References: Message-ID: <545919B6.7040007@stoneleaf.us> This list is for the development _of_ Python, not development _with_ Python. Try asking on Python List. (forwarding...) On 11/04/2014 08:52 AM, Roberto Mart?nez wrote: > > I am trying to replace dinamically the __call__ method of an object using setattr. > Example: > > $ cat testcall.py > class A: > def __init__(self): > setattr(self, '__call__', self.newcall) > > def __call__(self): > print("OLD") > > def newcall(self): > print("NEW") > > a=A() > a() > > I expect to get "NEW" instead of "OLD", but in Python 3.4 I get "OLD". > > $ python2.7 testcall.py > NEW > $ python3.4 testcall.py > OLD > > I have a few questions: > > - Is this an expected behavior? > - Is possible to replace __call__ dinamically in Python 3? How? In 2.7 that would be a classic class, about which I know little. In 3.x you have a new class, one which inherits from 'object'. When you replace __call__ you need to replace it the class, not on the instance: setattr(__self__.__class__, self.newcall) -- ~Ethan~ From skip.montanaro at gmail.com Tue Nov 4 19:25:09 2014 From: skip.montanaro at gmail.com (Skip Montanaro) Date: Tue, 4 Nov 2014 12:25:09 -0600 Subject: [Python-Dev] Dinamically set __call__ method In-Reply-To: References: Message-ID: On Tue, Nov 4, 2014 at 10:52 AM, Roberto Mart?nez < robertomartinezp at gmail.com> wrote: > $ cat testcall.py > class A: > You are using old-style classes in Python 2.7 unless you explicitly inherit from object. If I vary the class line to be "class A(object):" I get the same behavior with 2.7 as you see with 3.4. My guess is this causes the different behavior between versions. You might also find it useful to "print A.__call__" and "print a.__call__" with different class statements. Skip -------------- next part -------------- An HTML attachment was scrubbed... URL: From ethan at stoneleaf.us Wed Nov 5 17:33:42 2014 From: ethan at stoneleaf.us (Ethan Furman) Date: Wed, 05 Nov 2014 08:33:42 -0800 Subject: [Python-Dev] Real-world use of Counter Message-ID: <545A5166.2050601@stoneleaf.us> I'm looking for real-world uses of collections.Counter, specifically to see if anyone has been surprised by, or had to spend extra-time debugging, issues with the in-place operators. If sufficient and/or compelling use-cases are uncovered, the behavior of Counter may be slightly modified. Background: Most Python data types will cause a TypeError to be raised if unusable types are passed in: --> {'a': 0}.update(5) TypeError: 'int' object is not iterable --> [1, 2, 3].extend(3.14) TypeError: 'float' object is not iterable --> from collections import Counter --> Counter() + [1, 2, 3] TypeError: unsupported operand type(s) for +: 'Counter' and 'list' Most Counter in-place methods also behave this way: --> c /= [1, 2, 3] TypeError: unsupported operand type(s) for /=: 'Counter' and 'list' However, in the case of a handful of Counter in-place methods (+=, -=, &=, and |=), this is what happens instead: --> c += [1, 2, 3] AttributeError: 'list' object has no attribute 'items' Even worse (in my opinion) is the case of an empty Counter `and`ed with an incompatible type: --> c &= [1, 2, 3] --> No error is raised at all. In order to avoid unnecessary code churn (the fix itself is quite simple), the maintainer of the collections module wants to know if anybody has actually been affected by these inconsistencies, and if so, whether it was a minor inconvenience, or a compelling use-case. So, if this has bitten you, now is the time to speak up! :) -- ~Ethan~ From python at mrabarnett.plus.com Wed Nov 5 19:09:50 2014 From: python at mrabarnett.plus.com (MRAB) Date: Wed, 05 Nov 2014 18:09:50 +0000 Subject: [Python-Dev] Real-world use of Counter In-Reply-To: <545A5166.2050601@stoneleaf.us> References: <545A5166.2050601@stoneleaf.us> Message-ID: <545A67EE.7090101@mrabarnett.plus.com> On 2014-11-05 16:33, Ethan Furman wrote: > I'm looking for real-world uses of collections.Counter, specifically to see if anyone has been surprised by, or had to > spend extra-time debugging, issues with the in-place operators. > > If sufficient and/or compelling use-cases are uncovered, the behavior of Counter may be slightly modified. > > Background: > > Most Python data types will cause a TypeError to be raised if unusable types are passed in: > > --> {'a': 0}.update(5) > TypeError: 'int' object is not iterable > > --> [1, 2, 3].extend(3.14) > TypeError: 'float' object is not iterable > > --> from collections import Counter > --> Counter() + [1, 2, 3] > TypeError: unsupported operand type(s) for +: 'Counter' and 'list' > > Most Counter in-place methods also behave this way: > > --> c /= [1, 2, 3] > TypeError: unsupported operand type(s) for /=: 'Counter' and 'list' > > However, in the case of a handful of Counter in-place methods (+=, -=, &=, and |=), this is what happens instead: > > --> c += [1, 2, 3] > AttributeError: 'list' object has no attribute 'items' > > Even worse (in my opinion) is the case of an empty Counter `and`ed with an incompatible type: > > --> c &= [1, 2, 3] > --> > > No error is raised at all. > > In order to avoid unnecessary code churn (the fix itself is quite simple), the maintainer of the collections module > wants to know if anybody has actually been affected by these inconsistencies, and if so, whether it was a minor > inconvenience, or a compelling use-case. > > So, if this has bitten you, now is the time to speak up! :) > Whenever I've used the Counter class, I've known it was a counter, and not tried to do any of the above, but at least, if I did, I'd get an exception (or hope I do). The final example, however, is odd. I think that one should be fixed. From raymond.hettinger at gmail.com Wed Nov 5 19:56:12 2014 From: raymond.hettinger at gmail.com (Raymond Hettinger) Date: Wed, 5 Nov 2014 10:56:12 -0800 Subject: [Python-Dev] Real-world use of Counter In-Reply-To: <545A5166.2050601@stoneleaf.us> References: <545A5166.2050601@stoneleaf.us> Message-ID: <9D3B9C1D-B2C0-4D73-B7DF-C4177C22A79D@gmail.com> > On Nov 5, 2014, at 8:33 AM, Ethan Furman wrote: > > I'm looking for real-world uses of collections.Counter, specifically to see if anyone has been surprised by, or had to spend extra-time debugging, issues with the in-place operators. Please stop using the mailing lists as way to make an end-run around discussions on the tracker. http://bugs.python.org/issue22766 Also, as asked the question is a bit loaded. Effectively, it asks "has anyone ever been surprised by an exception raised by a duck-typed function or method"? The in-place operations on counters are duck-typed. They are intended (by design) to work with ANY type that has an items() method. The exception raised if doesn't have on is an AttributeError saying that the operand needs to have an items() method. I do not want to change API for already deployed code just because you would rather see a TypeError instead. Minor API changes (switching exception types) creates unnecessary consternation for users. Please let this one die. It seems to have become your pet project even after I've made a decision and explained my rationale. Raymond -------------- next part -------------- An HTML attachment was scrubbed... URL: From ethan at stoneleaf.us Wed Nov 5 19:58:04 2014 From: ethan at stoneleaf.us (Ethan Furman) Date: Wed, 05 Nov 2014 10:58:04 -0800 Subject: [Python-Dev] Real-world use of Counter In-Reply-To: <545A67EE.7090101@mrabarnett.plus.com> References: <545A5166.2050601@stoneleaf.us> <545A67EE.7090101@mrabarnett.plus.com> Message-ID: <545A733C.3070301@stoneleaf.us> On 11/05/2014 10:09 AM, MRAB wrote: > On 2014-11-05 16:33, Ethan Furman wrote: >> >> Even worse (in my opinion) is the case of an empty Counter `and`ed with an incompatible type: >> >> --> c &= [1, 2, 3] >> --> >> >> No error is raised at all. > > The final example, however, is odd. I think that one should be fixed. https://bugs.python.org/issue22801 -- ~Ethan~ From ethan at stoneleaf.us Wed Nov 5 20:13:37 2014 From: ethan at stoneleaf.us (Ethan Furman) Date: Wed, 05 Nov 2014 11:13:37 -0800 Subject: [Python-Dev] Real-world use of Counter In-Reply-To: <9D3B9C1D-B2C0-4D73-B7DF-C4177C22A79D@gmail.com> References: <545A5166.2050601@stoneleaf.us> <9D3B9C1D-B2C0-4D73-B7DF-C4177C22A79D@gmail.com> Message-ID: <545A76E1.3070602@stoneleaf.us> On 11/05/2014 10:56 AM, Raymond Hettinger wrote: > > Please stop using the mailing lists as way to make an end-run around discussions on the tracker. > http://bugs.python.org/issue22766 You said that without compelling, real-world use cases you don't like to make changes. The tracker has a very limited audience, while the mailing lists have a much greater chance of reaching the users who may actually have the use-cases you would consider. You call it an end-run, I call it research. > Also, as asked the question is a bit loaded. Effectively, it asks "has anyone ever been surprised by an exception > raised by a duck-typed function or method"? Actually, it's asking, "Most other duck-typed methods will still raise a TypeError, but these few don't. Has that ever been a problem for you?" > The in-place operations on counters are duck-typed. They are intended (by design) to work with ANY type that has an > items() method. The exception raised if doesn't have on is an AttributeError saying that the operand needs to have an > items() method. It would still be duck-typed with a `hasattr` call on the second operand checking for the necessary method, a TypeError could just as easily state the problem is a missing `items()` method, and then those three [*] in-place methods would raise the same error as the 20(?) other Counter methods under similar conditions. > Please let this one die. It seems to have become your pet project even after I've made a decision and explained my > rationale. You indicated you might make a change with sufficient real-world use-cases, so I'm trying to find some. If none show up, I'll let the matter drop. -- ~Ethan~ [*] Amusingly enough, there are four overridden in-place methods: three raise an AttributeError, but one (&=) still raises a TypeError. From rdmurray at bitdance.com Wed Nov 5 20:47:56 2014 From: rdmurray at bitdance.com (R. David Murray) Date: Wed, 05 Nov 2014 14:47:56 -0500 Subject: [Python-Dev] Real-world use of Counter In-Reply-To: <545A76E1.3070602@stoneleaf.us> References: <545A5166.2050601@stoneleaf.us> <9D3B9C1D-B2C0-4D73-B7DF-C4177C22A79D@gmail.com> <545A76E1.3070602@stoneleaf.us> Message-ID: <20141105194757.4C8D2250E1C@webabinitio.net> On Wed, 05 Nov 2014 11:13:37 -0800, Ethan Furman wrote: > On 11/05/2014 10:56 AM, Raymond Hettinger wrote: > > The in-place operations on counters are duck-typed. They are intended (by design) to work with ANY type that has an > > items() method. The exception raised if doesn't have on is an AttributeError saying that the operand needs to have an > > items() method. > > It would still be duck-typed with a `hasattr` call on the second operand checking for the necessary method, a TypeError > could just as easily state the problem is a missing `items()` method, and then those three [*] in-place methods would > raise the same error as the 20(?) other Counter methods under similar conditions. The technique of duck typing as used in Python is you just let errors bubble up when the input doesn't work. Adding special case checks should in general only be done if the resulting error message is otherwise really confusing, which is not the case here. As I said on the issue, there is no reason I can see to add extra code just to turn an AttributeError into a TypeError. The AttributeError works just fine in letting you know your input type didn't work. --David From greg.ewing at canterbury.ac.nz Wed Nov 5 21:28:49 2014 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 06 Nov 2014 09:28:49 +1300 Subject: [Python-Dev] Real-world use of Counter In-Reply-To: <545A76E1.3070602@stoneleaf.us> References: <545A5166.2050601@stoneleaf.us> <9D3B9C1D-B2C0-4D73-B7DF-C4177C22A79D@gmail.com> <545A76E1.3070602@stoneleaf.us> Message-ID: <545A8881.8020006@canterbury.ac.nz> Ethan Furman wrote: > Actually, it's asking, "Most other duck-typed methods will still raise a > TypeError, but these few don't. Has that ever been a problem for you?" I don't think I've *ever* been bothered by getting an AttributeError instead of a TypeError or vice versa. Both indicate bugs in my code, and I debug it by looking at the code and traceback; I don't try to guess the problem based solely on the exception type. In this case the code would have to go out of its way to turn an AttributeError into a TypeError. I don't think the cost of that is worth whatever small benefit there might be, if any. Summary: Looks fine to me as it is. -- Greg From alexander.belopolsky at gmail.com Wed Nov 5 21:51:19 2014 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Wed, 5 Nov 2014 15:51:19 -0500 Subject: [Python-Dev] Real-world use of Counter In-Reply-To: <20141105194757.4C8D2250E1C@webabinitio.net> References: <545A5166.2050601@stoneleaf.us> <9D3B9C1D-B2C0-4D73-B7DF-C4177C22A79D@gmail.com> <545A76E1.3070602@stoneleaf.us> <20141105194757.4C8D2250E1C@webabinitio.net> Message-ID: On Wed, Nov 5, 2014 at 2:47 PM, R. David Murray wrote: > As I said on the issue, there is no reason I can see to add extra code > just to turn an AttributeError into a TypeError. The AttributeError > works just fine in letting you know your input type didn't work. > +1 Unlike ValueError or LookupError, TypeError and AttributeError indicate a logical problem with the code rather than an issue with the user input. >From the programmer perspective, any code that catches and mutates exceptions is a nuisance. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Thu Nov 6 10:10:35 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 6 Nov 2014 19:10:35 +1000 Subject: [Python-Dev] Real-world use of Counter In-Reply-To: References: <545A5166.2050601@stoneleaf.us> <9D3B9C1D-B2C0-4D73-B7DF-C4177C22A79D@gmail.com> <545A76E1.3070602@stoneleaf.us> <20141105194757.4C8D2250E1C@webabinitio.net> Message-ID: On 6 Nov 2014 06:53, "Alexander Belopolsky" wrote: > > > On Wed, Nov 5, 2014 at 2:47 PM, R. David Murray wrote: >> >> As I said on the issue, there is no reason I can see to add extra code >> just to turn an AttributeError into a TypeError. The AttributeError >> works just fine in letting you know your input type didn't work. > > > +1 > > Unlike ValueError or LookupError, TypeError and AttributeError indicate a logical problem with the code rather than an issue with the user input. >From the programmer perspective, any code that catches and mutates exceptions is a nuisance. Right. Especially in a ducktyping context, AttributeError and TypeError are often functionally equivalent - it usually isn't worthwhile adding code specifically to turn one into the other. The case that doesn't throw an exception at all seems a little strange, but I haven't looked into the details. Regards, Nick. > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bcannon at gmail.com Thu Nov 6 15:39:03 2014 From: bcannon at gmail.com (Brett Cannon) Date: Thu, 06 Nov 2014 14:39:03 +0000 Subject: [Python-Dev] [Python-checkins] cpython (2.7): #22650: test suite: load Unicode test data files from www.pythontest.net References: <20141106135652.108375.20457@psf.io> Message-ID: What is pythontest.net? Is it something we control, and if so how do we add things to it for tests? Did I miss an email on python-dev or python-committers about this? On Thu Nov 06 2014 at 8:57:22 AM georg.brandl wrote: > https://hg.python.org/cpython/rev/0af36ea1d010 > changeset: 93417:0af36ea1d010 > branch: 2.7 > parent: 93401:3e8d3c4bc17e > user: Georg Brandl > date: Thu Nov 06 14:37:49 2014 +0100 > summary: > #22650: test suite: load Unicode test data files from www.pythontest.net > > files: > Lib/test/test_codecmaps_cn.py | 8 +++----- > Lib/test/test_codecmaps_hk.py | 2 +- > Lib/test/test_codecmaps_jp.py | 12 +++++------- > Lib/test/test_codecmaps_kr.py | 8 +++----- > Lib/test/test_codecmaps_tw.py | 6 ++---- > Lib/test/test_normalization.py | 2 +- > 6 files changed, 15 insertions(+), 23 deletions(-) > > > diff --git a/Lib/test/test_codecmaps_cn.py b/Lib/test/test_codecmaps_cn.py > --- a/Lib/test/test_codecmaps_cn.py > +++ b/Lib/test/test_codecmaps_cn.py > @@ -10,19 +10,17 @@ > class TestGB2312Map(test_multibytecodec_support.TestBase_Mapping, > unittest.TestCase): > encoding = 'gb2312' > - mapfileurl = 'http://people.freebsd.org/~perky/i18n/EUC-CN.TXT' > + mapfileurl = 'http://www.pythontest.net/unicode/EUC-CN.TXT' > > class TestGBKMap(test_multibytecodec_support.TestBase_Mapping, > unittest.TestCase): > encoding = 'gbk' > - mapfileurl = 'http://www.unicode.org/Public/MAPPINGS/VENDORS/' \ > - 'MICSFT/WINDOWS/CP936.TXT' > + mapfileurl = 'http://www.pythontest.net/unicode/CP936.TXT' > > class TestGB18030Map(test_multibytecodec_support.TestBase_Mapping, > unittest.TestCase): > encoding = 'gb18030' > - mapfileurl = 'http://source.icu-project.org/repos/icu/data/' \ > - 'trunk/charset/data/xml/gb-18030-2000.xml' > + mapfileurl = 'http://www.pythontest.net/unicode/gb-18030-2000.xml' > > > def test_main(): > diff --git a/Lib/test/test_codecmaps_hk.py b/Lib/test/test_codecmaps_hk.py > --- a/Lib/test/test_codecmaps_hk.py > +++ b/Lib/test/test_codecmaps_hk.py > @@ -10,7 +10,7 @@ > class TestBig5HKSCSMap(test_multibytecodec_support.TestBase_Mapping, > unittest.TestCase): > encoding = 'big5hkscs' > - mapfileurl = 'http://people.freebsd.org/~ > perky/i18n/BIG5HKSCS-2004.TXT' > + mapfileurl = 'http://www.pythontest.net/unicode/BIG5HKSCS-2004.TXT' > > def test_main(): > test_support.run_unittest(__name__) > diff --git a/Lib/test/test_codecmaps_jp.py b/Lib/test/test_codecmaps_jp.py > --- a/Lib/test/test_codecmaps_jp.py > +++ b/Lib/test/test_codecmaps_jp.py > @@ -10,8 +10,7 @@ > class TestCP932Map(test_multibytecodec_support.TestBase_Mapping, > unittest.TestCase): > encoding = 'cp932' > - mapfileurl = 'http://www.unicode.org/Public/MAPPINGS/VENDORS/MICSFT/' > \ > - 'WINDOWS/CP932.TXT' > + mapfileurl = 'http://www.pythontest.net/unicode/CP932.TXT' > supmaps = [ > ('\x80', u'\u0080'), > ('\xa0', u'\uf8f0'), > @@ -27,15 +26,14 @@ > unittest.TestCase): > encoding = 'euc_jp' > mapfilename = 'EUC-JP.TXT' > - mapfileurl = 'http://people.freebsd.org/~perky/i18n/EUC-JP.TXT' > + mapfileurl = 'http://www.pythontest.net/unicode/EUC-JP.TXT' > > > class TestSJISCOMPATMap(test_multibytecodec_support.TestBase_Mapping, > unittest.TestCase): > encoding = 'shift_jis' > mapfilename = 'SHIFTJIS.TXT' > - mapfileurl = 'http://www.unicode.org/Public/MAPPINGS/OBSOLETE' \ > - '/EASTASIA/JIS/SHIFTJIS.TXT' > + mapfileurl = 'http://www.pythontest.net/unicode/SHIFTJIS.TXT' > pass_enctest = [ > ('\x81_', u'\\'), > ] > @@ -49,14 +47,14 @@ > unittest.TestCase): > encoding = 'euc_jisx0213' > mapfilename = 'EUC-JISX0213.TXT' > - mapfileurl = 'http://people.freebsd.org/~perky/i18n/EUC-JISX0213.TXT' > + mapfileurl = 'http://www.pythontest.net/unicode/EUC-JISX0213.TXT' > > > class TestSJISX0213Map(test_multibytecodec_support.TestBase_Mapping, > unittest.TestCase): > encoding = 'shift_jisx0213' > mapfilename = 'SHIFT_JISX0213.TXT' > - mapfileurl = 'http://people.freebsd.org/~ > perky/i18n/SHIFT_JISX0213.TXT' > + mapfileurl = 'http://www.pythontest.net/unicode/SHIFT_JISX0213.TXT' > > > def test_main(): > diff --git a/Lib/test/test_codecmaps_kr.py b/Lib/test/test_codecmaps_kr.py > --- a/Lib/test/test_codecmaps_kr.py > +++ b/Lib/test/test_codecmaps_kr.py > @@ -10,14 +10,13 @@ > class TestCP949Map(test_multibytecodec_support.TestBase_Mapping, > unittest.TestCase): > encoding = 'cp949' > - mapfileurl = 'http://www.unicode.org/Public/MAPPINGS/VENDORS/MICSFT' > \ > - '/WINDOWS/CP949.TXT' > + mapfileurl = 'http://www.pythontest.net/unicode/CP949.TXT' > > > class TestEUCKRMap(test_multibytecodec_support.TestBase_Mapping, > unittest.TestCase): > encoding = 'euc_kr' > - mapfileurl = 'http://people.freebsd.org/~perky/i18n/EUC-KR.TXT' > + mapfileurl = 'http://www.pythontest.net/unicode/EUC-KR.TXT' > > # A4D4 HANGUL FILLER indicates the begin of 8-bytes make-up sequence. > pass_enctest = [('\xa4\xd4', u'\u3164')] > @@ -27,8 +26,7 @@ > class TestJOHABMap(test_multibytecodec_support.TestBase_Mapping, > unittest.TestCase): > encoding = 'johab' > - mapfileurl = 'http://www.unicode.org/Public/MAPPINGS/OBSOLETE/ > EASTASIA/' \ > - 'KSC/JOHAB.TXT' > + mapfileurl = 'http://www.pythontest.net/unicode/JOHAB.TXT' > # KS X 1001 standard assigned 0x5c as WON SIGN. > # but, in early 90s that is the only era used johab widely, > # the most softwares implements it as REVERSE SOLIDUS. > diff --git a/Lib/test/test_codecmaps_tw.py b/Lib/test/test_codecmaps_tw.py > --- a/Lib/test/test_codecmaps_tw.py > +++ b/Lib/test/test_codecmaps_tw.py > @@ -10,14 +10,12 @@ > class TestBIG5Map(test_multibytecodec_support.TestBase_Mapping, > unittest.TestCase): > encoding = 'big5' > - mapfileurl = 'http://www.unicode.org/Public/MAPPINGS/OBSOLETE/' \ > - 'EASTASIA/OTHER/BIG5.TXT' > + mapfileurl = 'http://www.pythontest.net/unicode/BIG5.TXT' > > class TestCP950Map(test_multibytecodec_support.TestBase_Mapping, > unittest.TestCase): > encoding = 'cp950' > - mapfileurl = 'http://www.unicode.org/Public/MAPPINGS/VENDORS/MICSFT/' > \ > - 'WINDOWS/CP950.TXT' > + mapfileurl = 'http://www.pythontest.net/unicode/CP950.TXT' > pass_enctest = [ > ('\xa2\xcc', u'\u5341'), > ('\xa2\xce', u'\u5345'), > diff --git a/Lib/test/test_normalization.py b/Lib/test/test_normalization. > py > --- a/Lib/test/test_normalization.py > +++ b/Lib/test/test_normalization.py > @@ -7,7 +7,7 @@ > from unicodedata import normalize, unidata_version > > TESTDATAFILE = "NormalizationTest.txt" > -TESTDATAURL = "http://www.unicode.org/Public/" + unidata_version + > "/ucd/" + TESTDATAFILE > +TESTDATAURL = "http://www.pythontest.net/unicode/" + unidata_version + > "/" + TESTDATAFILE > > def check_version(testfile): > hdr = testfile.readline() > > -- > Repository URL: https://hg.python.org/cpython > _______________________________________________ > Python-checkins mailing list > Python-checkins at python.org > https://mail.python.org/mailman/listinfo/python-checkins > -------------- next part -------------- An HTML attachment was scrubbed... URL: From benjamin at python.org Thu Nov 6 15:41:50 2014 From: benjamin at python.org (Benjamin Peterson) Date: Thu, 06 Nov 2014 09:41:50 -0500 Subject: [Python-Dev] [Python-checkins] cpython (2.7): #22650: test suite: load Unicode test data files from www.pythontest.net In-Reply-To: References: <20141106135652.108375.20457@psf.io> Message-ID: <1415284910.3978287.187837097.131F9A4E@webmail.messagingengine.com> On Thu, Nov 6, 2014, at 09:39, Brett Cannon wrote: > What is pythontest.net? Is it something we control, and if so how do we > add > things to it for tests? Did I miss an email on python-dev or > python-committers about this? See https://bugs.python.org/issue22650 > > On Thu Nov 06 2014 at 8:57:22 AM georg.brandl > > wrote: > > > https://hg.python.org/cpython/rev/0af36ea1d010 > > changeset: 93417:0af36ea1d010 > > branch: 2.7 > > parent: 93401:3e8d3c4bc17e > > user: Georg Brandl > > date: Thu Nov 06 14:37:49 2014 +0100 > > summary: > > #22650: test suite: load Unicode test data files from www.pythontest.net > > > > files: > > Lib/test/test_codecmaps_cn.py | 8 +++----- > > Lib/test/test_codecmaps_hk.py | 2 +- > > Lib/test/test_codecmaps_jp.py | 12 +++++------- > > Lib/test/test_codecmaps_kr.py | 8 +++----- > > Lib/test/test_codecmaps_tw.py | 6 ++---- > > Lib/test/test_normalization.py | 2 +- > > 6 files changed, 15 insertions(+), 23 deletions(-) > > > > > > diff --git a/Lib/test/test_codecmaps_cn.py b/Lib/test/test_codecmaps_cn.py > > --- a/Lib/test/test_codecmaps_cn.py > > +++ b/Lib/test/test_codecmaps_cn.py > > @@ -10,19 +10,17 @@ > > class TestGB2312Map(test_multibytecodec_support.TestBase_Mapping, > > unittest.TestCase): > > encoding = 'gb2312' > > - mapfileurl = 'http://people.freebsd.org/~perky/i18n/EUC-CN.TXT' > > + mapfileurl = 'http://www.pythontest.net/unicode/EUC-CN.TXT' > > > > class TestGBKMap(test_multibytecodec_support.TestBase_Mapping, > > unittest.TestCase): > > encoding = 'gbk' > > - mapfileurl = 'http://www.unicode.org/Public/MAPPINGS/VENDORS/' \ > > - 'MICSFT/WINDOWS/CP936.TXT' > > + mapfileurl = 'http://www.pythontest.net/unicode/CP936.TXT' > > > > class TestGB18030Map(test_multibytecodec_support.TestBase_Mapping, > > unittest.TestCase): > > encoding = 'gb18030' > > - mapfileurl = 'http://source.icu-project.org/repos/icu/data/' \ > > - 'trunk/charset/data/xml/gb-18030-2000.xml' > > + mapfileurl = 'http://www.pythontest.net/unicode/gb-18030-2000.xml' > > > > > > def test_main(): > > diff --git a/Lib/test/test_codecmaps_hk.py b/Lib/test/test_codecmaps_hk.py > > --- a/Lib/test/test_codecmaps_hk.py > > +++ b/Lib/test/test_codecmaps_hk.py > > @@ -10,7 +10,7 @@ > > class TestBig5HKSCSMap(test_multibytecodec_support.TestBase_Mapping, > > unittest.TestCase): > > encoding = 'big5hkscs' > > - mapfileurl = 'http://people.freebsd.org/~ > > perky/i18n/BIG5HKSCS-2004.TXT' > > + mapfileurl = 'http://www.pythontest.net/unicode/BIG5HKSCS-2004.TXT' > > > > def test_main(): > > test_support.run_unittest(__name__) > > diff --git a/Lib/test/test_codecmaps_jp.py b/Lib/test/test_codecmaps_jp.py > > --- a/Lib/test/test_codecmaps_jp.py > > +++ b/Lib/test/test_codecmaps_jp.py > > @@ -10,8 +10,7 @@ > > class TestCP932Map(test_multibytecodec_support.TestBase_Mapping, > > unittest.TestCase): > > encoding = 'cp932' > > - mapfileurl = 'http://www.unicode.org/Public/MAPPINGS/VENDORS/MICSFT/' > > \ > > - 'WINDOWS/CP932.TXT' > > + mapfileurl = 'http://www.pythontest.net/unicode/CP932.TXT' > > supmaps = [ > > ('\x80', u'\u0080'), > > ('\xa0', u'\uf8f0'), > > @@ -27,15 +26,14 @@ > > unittest.TestCase): > > encoding = 'euc_jp' > > mapfilename = 'EUC-JP.TXT' > > - mapfileurl = 'http://people.freebsd.org/~perky/i18n/EUC-JP.TXT' > > + mapfileurl = 'http://www.pythontest.net/unicode/EUC-JP.TXT' > > > > > > class TestSJISCOMPATMap(test_multibytecodec_support.TestBase_Mapping, > > unittest.TestCase): > > encoding = 'shift_jis' > > mapfilename = 'SHIFTJIS.TXT' > > - mapfileurl = 'http://www.unicode.org/Public/MAPPINGS/OBSOLETE' \ > > - '/EASTASIA/JIS/SHIFTJIS.TXT' > > + mapfileurl = 'http://www.pythontest.net/unicode/SHIFTJIS.TXT' > > pass_enctest = [ > > ('\x81_', u'\\'), > > ] > > @@ -49,14 +47,14 @@ > > unittest.TestCase): > > encoding = 'euc_jisx0213' > > mapfilename = 'EUC-JISX0213.TXT' > > - mapfileurl = 'http://people.freebsd.org/~perky/i18n/EUC-JISX0213.TXT' > > + mapfileurl = 'http://www.pythontest.net/unicode/EUC-JISX0213.TXT' > > > > > > class TestSJISX0213Map(test_multibytecodec_support.TestBase_Mapping, > > unittest.TestCase): > > encoding = 'shift_jisx0213' > > mapfilename = 'SHIFT_JISX0213.TXT' > > - mapfileurl = 'http://people.freebsd.org/~ > > perky/i18n/SHIFT_JISX0213.TXT' > > + mapfileurl = 'http://www.pythontest.net/unicode/SHIFT_JISX0213.TXT' > > > > > > def test_main(): > > diff --git a/Lib/test/test_codecmaps_kr.py b/Lib/test/test_codecmaps_kr.py > > --- a/Lib/test/test_codecmaps_kr.py > > +++ b/Lib/test/test_codecmaps_kr.py > > @@ -10,14 +10,13 @@ > > class TestCP949Map(test_multibytecodec_support.TestBase_Mapping, > > unittest.TestCase): > > encoding = 'cp949' > > - mapfileurl = 'http://www.unicode.org/Public/MAPPINGS/VENDORS/MICSFT' > > \ > > - '/WINDOWS/CP949.TXT' > > + mapfileurl = 'http://www.pythontest.net/unicode/CP949.TXT' > > > > > > class TestEUCKRMap(test_multibytecodec_support.TestBase_Mapping, > > unittest.TestCase): > > encoding = 'euc_kr' > > - mapfileurl = 'http://people.freebsd.org/~perky/i18n/EUC-KR.TXT' > > + mapfileurl = 'http://www.pythontest.net/unicode/EUC-KR.TXT' > > > > # A4D4 HANGUL FILLER indicates the begin of 8-bytes make-up sequence. > > pass_enctest = [('\xa4\xd4', u'\u3164')] > > @@ -27,8 +26,7 @@ > > class TestJOHABMap(test_multibytecodec_support.TestBase_Mapping, > > unittest.TestCase): > > encoding = 'johab' > > - mapfileurl = 'http://www.unicode.org/Public/MAPPINGS/OBSOLETE/ > > EASTASIA/' \ > > - 'KSC/JOHAB.TXT' > > + mapfileurl = 'http://www.pythontest.net/unicode/JOHAB.TXT' > > # KS X 1001 standard assigned 0x5c as WON SIGN. > > # but, in early 90s that is the only era used johab widely, > > # the most softwares implements it as REVERSE SOLIDUS. > > diff --git a/Lib/test/test_codecmaps_tw.py b/Lib/test/test_codecmaps_tw.py > > --- a/Lib/test/test_codecmaps_tw.py > > +++ b/Lib/test/test_codecmaps_tw.py > > @@ -10,14 +10,12 @@ > > class TestBIG5Map(test_multibytecodec_support.TestBase_Mapping, > > unittest.TestCase): > > encoding = 'big5' > > - mapfileurl = 'http://www.unicode.org/Public/MAPPINGS/OBSOLETE/' \ > > - 'EASTASIA/OTHER/BIG5.TXT' > > + mapfileurl = 'http://www.pythontest.net/unicode/BIG5.TXT' > > > > class TestCP950Map(test_multibytecodec_support.TestBase_Mapping, > > unittest.TestCase): > > encoding = 'cp950' > > - mapfileurl = 'http://www.unicode.org/Public/MAPPINGS/VENDORS/MICSFT/' > > \ > > - 'WINDOWS/CP950.TXT' > > + mapfileurl = 'http://www.pythontest.net/unicode/CP950.TXT' > > pass_enctest = [ > > ('\xa2\xcc', u'\u5341'), > > ('\xa2\xce', u'\u5345'), > > diff --git a/Lib/test/test_normalization.py b/Lib/test/test_normalization. > > py > > --- a/Lib/test/test_normalization.py > > +++ b/Lib/test/test_normalization.py > > @@ -7,7 +7,7 @@ > > from unicodedata import normalize, unidata_version > > > > TESTDATAFILE = "NormalizationTest.txt" > > -TESTDATAURL = "http://www.unicode.org/Public/" + unidata_version + > > "/ucd/" + TESTDATAFILE > > +TESTDATAURL = "http://www.pythontest.net/unicode/" + unidata_version + > > "/" + TESTDATAFILE > > > > def check_version(testfile): > > hdr = testfile.readline() > > > > -- > > Repository URL: https://hg.python.org/cpython > > _______________________________________________ > > Python-checkins mailing list > > Python-checkins at python.org > > https://mail.python.org/mailman/listinfo/python-checkins > > > _______________________________________________ > Python-checkins mailing list > Python-checkins at python.org > https://mail.python.org/mailman/listinfo/python-checkins From mal at egenix.com Thu Nov 6 16:09:54 2014 From: mal at egenix.com (M.-A. Lemburg) Date: Thu, 06 Nov 2014 16:09:54 +0100 Subject: [Python-Dev] [Python-checkins] cpython (2.7): #22650: test suite: load Unicode test data files from www.pythontest.net In-Reply-To: References: <20141106135652.108375.20457@psf.io> Message-ID: <545B8F42.10000@egenix.com> On 06.11.2014 15:39, Brett Cannon wrote: > What is pythontest.net? Is it something we control, and if so how do we add > things to it for tests? Did I miss an email on python-dev or > python-committers about this? pythontest.net is a domain owned by the PSF and run by Donald Stufft and Benjamin (I believe). https://bugs.python.org/issue22650 has the details. > On Thu Nov 06 2014 at 8:57:22 AM georg.brandl > wrote: > >> https://hg.python.org/cpython/rev/0af36ea1d010 >> changeset: 93417:0af36ea1d010 >> branch: 2.7 >> parent: 93401:3e8d3c4bc17e >> user: Georg Brandl >> date: Thu Nov 06 14:37:49 2014 +0100 >> summary: >> #22650: test suite: load Unicode test data files from www.pythontest.net -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Nov 06 2014) >>> Python Projects, Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ 2014-10-24: Released eGenix pyOpenSSL 0.13.5 ... http://egenix.com/go63 ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From guido at python.org Thu Nov 6 16:46:39 2014 From: guido at python.org (Guido van Rossum) Date: Thu, 6 Nov 2014 07:46:39 -0800 Subject: [Python-Dev] Real-world use of Counter In-Reply-To: References: <545A5166.2050601@stoneleaf.us> <9D3B9C1D-B2C0-4D73-B7DF-C4177C22A79D@gmail.com> <545A76E1.3070602@stoneleaf.us> <20141105194757.4C8D2250E1C@webabinitio.net> Message-ID: On Thu, Nov 6, 2014 at 1:10 AM, Nick Coghlan wrote: > Right. Especially in a ducktyping context, AttributeError and TypeError > are often functionally equivalent - it usually isn't worthwhile adding code > specifically to turn one into the other. > Yeah, these are so often interchangeable that I wish they had a common ancestor. Then again when you are catching these you might as well be catching all exceptions. > The case that doesn't throw an exception at all seems a little strange, > but I haven't looked into the details. > It comes from a simple approach to creating an intersection; paraphrasing, the code does this: def intesection(a, b): result = set() for x in a: if x in b: result.add(x) return result If a is empty this never looks at b. I think it's okay not to raise in this case (though it would also be okay if it *did* raise). -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From bcannon at gmail.com Thu Nov 6 16:41:09 2014 From: bcannon at gmail.com (Brett Cannon) Date: Thu, 06 Nov 2014 15:41:09 +0000 Subject: [Python-Dev] [Python-checkins] cpython (2.7): #22650: test suite: load Unicode test data files from www.pythontest.net References: <20141106135652.108375.20457@psf.io> <1415284910.3978287.187837097.131F9A4E@webmail.messagingengine.com> Message-ID: Ah, cool! Just an FYI, the index.html file is not being served for me. -Brett On Thu Nov 06 2014 at 9:41:59 AM Benjamin Peterson wrote: > > > On Thu, Nov 6, 2014, at 09:39, Brett Cannon wrote: > > What is pythontest.net? Is it something we control, and if so how do we > > add > > things to it for tests? Did I miss an email on python-dev or > > python-committers about this? > > See https://bugs.python.org/issue22650 > > > > > On Thu Nov 06 2014 at 8:57:22 AM georg.brandl > > > > wrote: > > > > > https://hg.python.org/cpython/rev/0af36ea1d010 > > > changeset: 93417:0af36ea1d010 > > > branch: 2.7 > > > parent: 93401:3e8d3c4bc17e > > > user: Georg Brandl > > > date: Thu Nov 06 14:37:49 2014 +0100 > > > summary: > > > #22650: test suite: load Unicode test data files from > www.pythontest.net > > > > > > files: > > > Lib/test/test_codecmaps_cn.py | 8 +++----- > > > Lib/test/test_codecmaps_hk.py | 2 +- > > > Lib/test/test_codecmaps_jp.py | 12 +++++------- > > > Lib/test/test_codecmaps_kr.py | 8 +++----- > > > Lib/test/test_codecmaps_tw.py | 6 ++---- > > > Lib/test/test_normalization.py | 2 +- > > > 6 files changed, 15 insertions(+), 23 deletions(-) > > > > > > > > > diff --git a/Lib/test/test_codecmaps_cn.py > b/Lib/test/test_codecmaps_cn.py > > > --- a/Lib/test/test_codecmaps_cn.py > > > +++ b/Lib/test/test_codecmaps_cn.py > > > @@ -10,19 +10,17 @@ > > > class TestGB2312Map(test_multibytecodec_support.TestBase_Mapping, > > > unittest.TestCase): > > > encoding = 'gb2312' > > > - mapfileurl = 'http://people.freebsd.org/~perky/i18n/EUC-CN.TXT' > > > + mapfileurl = 'http://www.pythontest.net/unicode/EUC-CN.TXT' > > > > > > class TestGBKMap(test_multibytecodec_support.TestBase_Mapping, > > > unittest.TestCase): > > > encoding = 'gbk' > > > - mapfileurl = 'http://www.unicode.org/Public/MAPPINGS/VENDORS/' \ > > > - 'MICSFT/WINDOWS/CP936.TXT' > > > + mapfileurl = 'http://www.pythontest.net/unicode/CP936.TXT' > > > > > > class TestGB18030Map(test_multibytecodec_support.TestBase_Mapping, > > > unittest.TestCase): > > > encoding = 'gb18030' > > > - mapfileurl = 'http://source.icu-project.org/repos/icu/data/' \ > > > - 'trunk/charset/data/xml/gb-18030-2000.xml' > > > + mapfileurl = 'http://www.pythontest.net/unicode/gb-18030-2000.xml > ' > > > > > > > > > def test_main(): > > > diff --git a/Lib/test/test_codecmaps_hk.py > b/Lib/test/test_codecmaps_hk.py > > > --- a/Lib/test/test_codecmaps_hk.py > > > +++ b/Lib/test/test_codecmaps_hk.py > > > @@ -10,7 +10,7 @@ > > > class TestBig5HKSCSMap(test_multibytecodec_support.TestBase_Mapping, > > > unittest.TestCase): > > > encoding = 'big5hkscs' > > > - mapfileurl = 'http://people.freebsd.org/~ > > > perky/i18n/BIG5HKSCS-2004.TXT' > > > + mapfileurl = 'http://www.pythontest.net/uni > code/BIG5HKSCS-2004.TXT' > > > > > > def test_main(): > > > test_support.run_unittest(__name__) > > > diff --git a/Lib/test/test_codecmaps_jp.py > b/Lib/test/test_codecmaps_jp.py > > > --- a/Lib/test/test_codecmaps_jp.py > > > +++ b/Lib/test/test_codecmaps_jp.py > > > @@ -10,8 +10,7 @@ > > > class TestCP932Map(test_multibytecodec_support.TestBase_Mapping, > > > unittest.TestCase): > > > encoding = 'cp932' > > > - mapfileurl = 'http://www.unicode.org/Public/MAPPINGS/VENDORS/ > MICSFT/' > > > \ > > > - 'WINDOWS/CP932.TXT' > > > + mapfileurl = 'http://www.pythontest.net/unicode/CP932.TXT' > > > supmaps = [ > > > ('\x80', u'\u0080'), > > > ('\xa0', u'\uf8f0'), > > > @@ -27,15 +26,14 @@ > > > unittest.TestCase): > > > encoding = 'euc_jp' > > > mapfilename = 'EUC-JP.TXT' > > > - mapfileurl = 'http://people.freebsd.org/~perky/i18n/EUC-JP.TXT' > > > + mapfileurl = 'http://www.pythontest.net/unicode/EUC-JP.TXT' > > > > > > > > > class TestSJISCOMPATMap(test_multibytecodec_support.TestBase_Mapping, > > > unittest.TestCase): > > > encoding = 'shift_jis' > > > mapfilename = 'SHIFTJIS.TXT' > > > - mapfileurl = 'http://www.unicode.org/Public/MAPPINGS/OBSOLETE' \ > > > - '/EASTASIA/JIS/SHIFTJIS.TXT' > > > + mapfileurl = 'http://www.pythontest.net/unicode/SHIFTJIS.TXT' > > > pass_enctest = [ > > > ('\x81_', u'\\'), > > > ] > > > @@ -49,14 +47,14 @@ > > > unittest.TestCase): > > > encoding = 'euc_jisx0213' > > > mapfilename = 'EUC-JISX0213.TXT' > > > - mapfileurl = 'http://people.freebsd.org/~pe > rky/i18n/EUC-JISX0213.TXT' > > > + mapfileurl = 'http://www.pythontest.net/unicode/EUC-JISX0213.TXT' > > > > > > > > > class TestSJISX0213Map(test_multibytecodec_support.TestBase_Mapping, > > > unittest.TestCase): > > > encoding = 'shift_jisx0213' > > > mapfilename = 'SHIFT_JISX0213.TXT' > > > - mapfileurl = 'http://people.freebsd.org/~ > > > perky/i18n/SHIFT_JISX0213.TXT' > > > + mapfileurl = 'http://www.pythontest.net/uni > code/SHIFT_JISX0213.TXT' > > > > > > > > > def test_main(): > > > diff --git a/Lib/test/test_codecmaps_kr.py > b/Lib/test/test_codecmaps_kr.py > > > --- a/Lib/test/test_codecmaps_kr.py > > > +++ b/Lib/test/test_codecmaps_kr.py > > > @@ -10,14 +10,13 @@ > > > class TestCP949Map(test_multibytecodec_support.TestBase_Mapping, > > > unittest.TestCase): > > > encoding = 'cp949' > > > - mapfileurl = 'http://www.unicode.org/Public > /MAPPINGS/VENDORS/MICSFT' > > > \ > > > - '/WINDOWS/CP949.TXT' > > > + mapfileurl = 'http://www.pythontest.net/unicode/CP949.TXT' > > > > > > > > > class TestEUCKRMap(test_multibytecodec_support.TestBase_Mapping, > > > unittest.TestCase): > > > encoding = 'euc_kr' > > > - mapfileurl = 'http://people.freebsd.org/~perky/i18n/EUC-KR.TXT' > > > + mapfileurl = 'http://www.pythontest.net/unicode/EUC-KR.TXT' > > > > > > # A4D4 HANGUL FILLER indicates the begin of 8-bytes make-up > sequence. > > > pass_enctest = [('\xa4\xd4', u'\u3164')] > > > @@ -27,8 +26,7 @@ > > > class TestJOHABMap(test_multibytecodec_support.TestBase_Mapping, > > > unittest.TestCase): > > > encoding = 'johab' > > > - mapfileurl = 'http://www.unicode.org/Public/MAPPINGS/OBSOLETE/ > > > EASTASIA/' \ > > > - 'KSC/JOHAB.TXT' > > > + mapfileurl = 'http://www.pythontest.net/unicode/JOHAB.TXT' > > > # KS X 1001 standard assigned 0x5c as WON SIGN. > > > # but, in early 90s that is the only era used johab widely, > > > # the most softwares implements it as REVERSE SOLIDUS. > > > diff --git a/Lib/test/test_codecmaps_tw.py > b/Lib/test/test_codecmaps_tw.py > > > --- a/Lib/test/test_codecmaps_tw.py > > > +++ b/Lib/test/test_codecmaps_tw.py > > > @@ -10,14 +10,12 @@ > > > class TestBIG5Map(test_multibytecodec_support.TestBase_Mapping, > > > unittest.TestCase): > > > encoding = 'big5' > > > - mapfileurl = 'http://www.unicode.org/Public/MAPPINGS/OBSOLETE/' \ > > > - 'EASTASIA/OTHER/BIG5.TXT' > > > + mapfileurl = 'http://www.pythontest.net/unicode/BIG5.TXT' > > > > > > class TestCP950Map(test_multibytecodec_support.TestBase_Mapping, > > > unittest.TestCase): > > > encoding = 'cp950' > > > - mapfileurl = 'http://www.unicode.org/Public/MAPPINGS/VENDORS/ > MICSFT/' > > > \ > > > - 'WINDOWS/CP950.TXT' > > > + mapfileurl = 'http://www.pythontest.net/unicode/CP950.TXT' > > > pass_enctest = [ > > > ('\xa2\xcc', u'\u5341'), > > > ('\xa2\xce', u'\u5345'), > > > diff --git a/Lib/test/test_normalization.py > b/Lib/test/test_normalization. > > > py > > > --- a/Lib/test/test_normalization.py > > > +++ b/Lib/test/test_normalization.py > > > @@ -7,7 +7,7 @@ > > > from unicodedata import normalize, unidata_version > > > > > > TESTDATAFILE = "NormalizationTest.txt" > > > -TESTDATAURL = "http://www.unicode.org/Public/" + unidata_version + > > > "/ucd/" + TESTDATAFILE > > > +TESTDATAURL = "http://www.pythontest.net/unicode/" + unidata_version > + > > > "/" + TESTDATAFILE > > > > > > def check_version(testfile): > > > hdr = testfile.readline() > > > > > > -- > > > Repository URL: https://hg.python.org/cpython > > > _______________________________________________ > > > Python-checkins mailing list > > > Python-checkins at python.org > > > https://mail.python.org/mailman/listinfo/python-checkins > > > > > _______________________________________________ > > Python-checkins mailing list > > Python-checkins at python.org > > https://mail.python.org/mailman/listinfo/python-checkins > _______________________________________________ > Python-checkins mailing list > Python-checkins at python.org > https://mail.python.org/mailman/listinfo/python-checkins > -------------- next part -------------- An HTML attachment was scrubbed... URL: From benjamin at python.org Thu Nov 6 17:56:31 2014 From: benjamin at python.org (Benjamin Peterson) Date: Thu, 06 Nov 2014 11:56:31 -0500 Subject: [Python-Dev] [Python-checkins] cpython (2.7): #22650: test suite: load Unicode test data files from www.pythontest.net In-Reply-To: References: <20141106135652.108375.20457@psf.io> <1415284910.3978287.187837097.131F9A4E@webmail.messagingengine.com> Message-ID: <1415292991.4007266.187895233.3A8B1517@webmail.messagingengine.com> On Thu, Nov 6, 2014, at 10:41, Brett Cannon wrote: > Ah, cool! Just an FYI, the index.html file is not being served for me. At the moment it's only served on www. From mal at egenix.com Fri Nov 7 13:21:48 2014 From: mal at egenix.com (M.-A. Lemburg) Date: Fri, 07 Nov 2014 13:21:48 +0100 Subject: [Python-Dev] hg.python.org cloning troubles after Sep 13 changes Message-ID: <545CB95C.3080205@egenix.com> Just FYI (and for the archives), to perhaps save someone a few minutes: I've been hitting a problem with hg pull and hg clone on a box recently and after staring at it for a while, finally found the cause. Here's what hg printed: HG-Python/cpython> hg pull -u -b 2.7 abort: no suitable response from remote hg! Not much help. Using --debug: orig/HG-Python> hg --debug clone ssh://hg at hg.python.org/cpython cpython-2.7 running ssh -C -q -l marc-andre.lemburg -i /home/lemburg/.ssh/id_rsa hg at hg.python.org "hg -R cpython serve --stdio" sending hello command sending between command abort: no suitable response from remote hg! Still no help. Then I remembered that the box was newly setup on Sept 13 and checked Benjamins email. The server host key had also changed. After removing the offending key and accepting the new one, things started working again. The hg on the box is 1.7.5, so newer version may include a better error message. Cheers, -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Nov 07 2014) >>> Python Projects, Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From ethan at stoneleaf.us Fri Nov 7 15:26:17 2014 From: ethan at stoneleaf.us (Ethan Furman) Date: Fri, 07 Nov 2014 06:26:17 -0800 Subject: [Python-Dev] Real-world use of Counter In-Reply-To: <545A5166.2050601@stoneleaf.us> References: <545A5166.2050601@stoneleaf.us> Message-ID: <545CD689.2050303@stoneleaf.us> Thank you everyone for the discussion, it has been, as always, most educational. :) -- ~Ethan~ From andrew.svetlov at gmail.com Fri Nov 7 16:41:06 2014 From: andrew.svetlov at gmail.com (Andrew Svetlov) Date: Fri, 7 Nov 2014 17:41:06 +0200 Subject: [Python-Dev] hg.python.org cloning troubles after Sep 13 changes In-Reply-To: <545CB95C.3080205@egenix.com> References: <545CB95C.3080205@egenix.com> Message-ID: hg 3.1.1 prints comprehensive message that guess to call `ssh-keygen -R hg.python.org` On Fri, Nov 7, 2014 at 2:21 PM, M.-A. Lemburg wrote: > Just FYI (and for the archives), to perhaps save someone a few minutes: > > I've been hitting a problem with hg pull and hg clone on a box recently > and after staring at it for a while, finally found the cause. > > Here's what hg printed: > > HG-Python/cpython> hg pull -u -b 2.7 > abort: no suitable response from remote hg! > > Not much help. > > Using --debug: > > orig/HG-Python> hg --debug clone ssh://hg at hg.python.org/cpython cpython-2.7 > running ssh -C -q -l marc-andre.lemburg -i /home/lemburg/.ssh/id_rsa hg at hg.python.org "hg -R cpython > serve --stdio" > sending hello command > sending between command > abort: no suitable response from remote hg! > > Still no help. > > Then I remembered that the box was newly setup on Sept 13 and checked > Benjamins email. The server host key had also changed. After removing > the offending key and accepting the new one, things started working > again. > > The hg on the box is 1.7.5, so newer version may include a better > error message. > > Cheers, > -- > Marc-Andre Lemburg > eGenix.com > > Professional Python Services directly from the Source (#1, Nov 07 2014) >>>> Python Projects, Consulting and Support ... http://www.egenix.com/ >>>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ >>>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ > ________________________________________________________________________ > > ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: > > eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 > D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg > Registered at Amtsgericht Duesseldorf: HRB 46611 > http://www.egenix.com/company/contact/ > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/andrew.svetlov%40gmail.com -- Thanks, Andrew Svetlov From status at bugs.python.org Fri Nov 7 18:08:12 2014 From: status at bugs.python.org (Python tracker) Date: Fri, 7 Nov 2014 18:08:12 +0100 (CET) Subject: [Python-Dev] Summary of Python tracker Issues Message-ID: <20141107170812.651D2560C6@psf.upfronthosting.co.za> ACTIVITY SUMMARY (2014-10-31 - 2014-11-07) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 4622 ( -4) closed 29955 (+44) total 34577 (+40) Open issues with patches: 2158 Issues opened (32) ================== #17900: Recursive OrderedDict pickling http://bugs.python.org/issue17900 reopened by rhettinger #20160: broken ctypes calling convention on MSVC / 64-bit Windows (lar http://bugs.python.org/issue20160 reopened by doko #21274: define PATH_MAX for GNU/Hurd in Python/pythonrun.c http://bugs.python.org/issue21274 reopened by loewis #22695: open() declared deprecated in python 3 docs http://bugs.python.org/issue22695 reopened by serhiy.storchaka #22731: test_capi test fails because of mismatched newlines http://bugs.python.org/issue22731 reopened by steve.dower #22777: Test pickling with all protocols http://bugs.python.org/issue22777 opened by serhiy.storchaka #22780: NotImplemented doc section needs update http://bugs.python.org/issue22780 opened by ethan.furman #22781: ctypes: Differing results between Python and C. http://bugs.python.org/issue22781 opened by Geoff.Clements #22782: Python 3.4.2 Windows installer - cannot install 32 bit then 64 http://bugs.python.org/issue22782 opened by pmoore #22783: Pickle: use NEWOBJ instead of NEWOBJ_EX if possible http://bugs.python.org/issue22783 opened by serhiy.storchaka #22785: range docstring is less useful than in python 2 http://bugs.python.org/issue22785 opened by nedbat #22788: allow logging.handlers.HTTPHandler to take an SSLContext http://bugs.python.org/issue22788 opened by benjamin.peterson #22789: Compress the marshalled data in PYC files http://bugs.python.org/issue22789 opened by rhettinger #22790: some class attributes missing from dir(Class) http://bugs.python.org/issue22790 opened by techdragon #22791: datetime.utcfromtimestamp() shoud have option for create tz aw http://bugs.python.org/issue22791 opened by naoki #22794: missing test for issue 22457 regression commit http://bugs.python.org/issue22794 opened by rbcollins #22796: Support for httponly/secure cookies reintroduced lax parsing b http://bugs.python.org/issue22796 opened by Tim.Graham #22797: urllib.request.urlopen documentation falsely guarantees that a http://bugs.python.org/issue22797 opened by Joshua.Chin #22798: time.mktime doesn't update time.tzname http://bugs.python.org/issue22798 opened by akira #22799: wrong time.timezone http://bugs.python.org/issue22799 opened by akira #22800: IPv6Network constructor sometimes does not recognize legitimat http://bugs.python.org/issue22800 opened by pebenito #22801: collections.Counter, when empty, doesn't raise an error with & http://bugs.python.org/issue22801 opened by ethan.furman #22804: Can't run Idle in Windows 8 or windows 64 http://bugs.python.org/issue22804 opened by nx.u #22806: regrtest: add switch -c to run only modified tests http://bugs.python.org/issue22806 opened by georg.brandl #22807: uuid.uuid1() should use uuid_generate_time_safe() if available http://bugs.python.org/issue22807 opened by barry #22808: Typo in asyncio-eventloop.rst, time() link is wrong http://bugs.python.org/issue22808 opened by markgrandi #22810: tkinter: "alloc: invalid block:" after askopenfilename http://bugs.python.org/issue22810 opened by lccat #22811: _top_level_dir state leaks on defaultTestLoader http://bugs.python.org/issue22811 opened by rbcollins #22812: Documentation of unittest -p usage wrong on windows. http://bugs.python.org/issue22812 opened by rbcollins #22813: No facility for test randomisation http://bugs.python.org/issue22813 opened by rbcollins #22814: TestProgram loading fails when a script is used http://bugs.python.org/issue22814 opened by rbcollins #22815: unexpected successes are not output http://bugs.python.org/issue22815 opened by rbcollins Most recent 15 issues with no replies (15) ========================================== #22815: unexpected successes are not output http://bugs.python.org/issue22815 #22814: TestProgram loading fails when a script is used http://bugs.python.org/issue22814 #22813: No facility for test randomisation http://bugs.python.org/issue22813 #22811: _top_level_dir state leaks on defaultTestLoader http://bugs.python.org/issue22811 #22801: collections.Counter, when empty, doesn't raise an error with & http://bugs.python.org/issue22801 #22800: IPv6Network constructor sometimes does not recognize legitimat http://bugs.python.org/issue22800 #22799: wrong time.timezone http://bugs.python.org/issue22799 #22794: missing test for issue 22457 regression commit http://bugs.python.org/issue22794 #22783: Pickle: use NEWOBJ instead of NEWOBJ_EX if possible http://bugs.python.org/issue22783 #22765: Fixes for test_gdb (first frame address, entry values) http://bugs.python.org/issue22765 #22743: Specify supported XML version http://bugs.python.org/issue22743 #22737: Provide a rejected execution model and implementations for fut http://bugs.python.org/issue22737 #22733: MSVC ffi_prep_args doesn't handle 64-bit arguments properly http://bugs.python.org/issue22733 #22729: `wait` and `as_completed` depend on private api http://bugs.python.org/issue22729 #22726: Idle: add help to config dialogs http://bugs.python.org/issue22726 Most recent 15 issues waiting for review (15) ============================================= #22808: Typo in asyncio-eventloop.rst, time() link is wrong http://bugs.python.org/issue22808 #22806: regrtest: add switch -c to run only modified tests http://bugs.python.org/issue22806 #22801: collections.Counter, when empty, doesn't raise an error with & http://bugs.python.org/issue22801 #22798: time.mktime doesn't update time.tzname http://bugs.python.org/issue22798 #22797: urllib.request.urlopen documentation falsely guarantees that a http://bugs.python.org/issue22797 #22796: Support for httponly/secure cookies reintroduced lax parsing b http://bugs.python.org/issue22796 #22790: some class attributes missing from dir(Class) http://bugs.python.org/issue22790 #22788: allow logging.handlers.HTTPHandler to take an SSLContext http://bugs.python.org/issue22788 #22783: Pickle: use NEWOBJ instead of NEWOBJ_EX if possible http://bugs.python.org/issue22783 #22777: Test pickling with all protocols http://bugs.python.org/issue22777 #22768: Add a way to get the peer certificate of a SSL Transport http://bugs.python.org/issue22768 #22765: Fixes for test_gdb (first frame address, entry values) http://bugs.python.org/issue22765 #22764: object lifetime fragility in unittest tests http://bugs.python.org/issue22764 #22758: Regression in Python 3.2 cookie parsing http://bugs.python.org/issue22758 #22753: urllib2 localnet Changed test to lookup IP-address of localhos http://bugs.python.org/issue22753 Top 10 most discussed issues (10) ================================= #22790: some class attributes missing from dir(Class) http://bugs.python.org/issue22790 20 msgs #22780: NotImplemented doc section needs update http://bugs.python.org/issue22780 14 msgs #22796: Support for httponly/secure cookies reintroduced lax parsing b http://bugs.python.org/issue22796 13 msgs #22650: set up and use VM for net access in the test suite http://bugs.python.org/issue22650 9 msgs #22789: Compress the marshalled data in PYC files http://bugs.python.org/issue22789 9 msgs #22417: PEP 476: verify HTTPS certificates by default http://bugs.python.org/issue22417 8 msgs #22680: Blacklist FunctionTestCase from test discovery http://bugs.python.org/issue22680 8 msgs #20160: broken ctypes calling convention on MSVC / 64-bit Windows (lar http://bugs.python.org/issue20160 7 msgs #20220: TarFile.list() outputs wrong time http://bugs.python.org/issue20220 7 msgs #16353: add function to os module for getting path to default shell http://bugs.python.org/issue16353 6 msgs Issues closed (44) ================== #6623: Lib/ftplib.py Netrc class should be removed. http://bugs.python.org/issue6623 closed by berker.peksag #14460: In re's positive lookbehind assertion repetition works http://bugs.python.org/issue14460 closed by serhiy.storchaka #15279: Spurious unittest warnings http://bugs.python.org/issue15279 closed by ezio.melotti #17293: uuid.getnode() MAC address on AIX http://bugs.python.org/issue17293 closed by serhiy.storchaka #17717: Set up nasm from external.bat http://bugs.python.org/issue17717 closed by zach.ware #17896: Move Windows external libs from \..\ to \externals http://bugs.python.org/issue17896 closed by zach.ware #19031: Make help() enum aware http://bugs.python.org/issue19031 closed by ethan.furman #19046: SystemError: ..\Objects\weakrefobject.c:903: bad argument to i http://bugs.python.org/issue19046 closed by serhiy.storchaka #19753: test_gdb failure on SystemZ buildbot http://bugs.python.org/issue19753 closed by haypo #20597: PATH_MAX already defined on some Windows compilers http://bugs.python.org/issue20597 closed by Jeffrey.Armstrong #22242: Doc fix in the Import section in language reference. http://bugs.python.org/issue22242 closed by brett.cannon #22335: Python 3: Segfault instead of MemoryError when bytearray too b http://bugs.python.org/issue22335 closed by pitrou #22388: Unify style of "Contributed by" notes http://bugs.python.org/issue22388 closed by serhiy.storchaka #22406: uu-codec trailing garbage workaround is Python 2 code http://bugs.python.org/issue22406 closed by serhiy.storchaka #22457: load_tests not invoked in root __init__.py when start=package http://bugs.python.org/issue22457 closed by python-dev #22525: ast.literal_eval() doesn't do what the documentation says http://bugs.python.org/issue22525 closed by python-dev #22665: shutil.__all__ incomplete http://bugs.python.org/issue22665 closed by berker.peksag #22683: bisect index out of bounds issue http://bugs.python.org/issue22683 closed by rhettinger #22697: Deadlock with writing to stderr from forked process http://bugs.python.org/issue22697 closed by serhiy.storchaka #22718: pprint not handline uncomparable dictionary keys, set members http://bugs.python.org/issue22718 closed by rhettinger #22725: improve documentation for enumerate() (built-in function) http://bugs.python.org/issue22725 closed by rhettinger #22732: ctypes tests don't set correct restype for intptr_t functions http://bugs.python.org/issue22732 closed by steve.dower #22734: marshal needs a lower stack depth for debug builds on Windows http://bugs.python.org/issue22734 closed by steve.dower #22739: "There is no disk in the drive" error http://bugs.python.org/issue22739 closed by terry.reedy #22751: Fix test___all__ warning about modified environment http://bugs.python.org/issue22751 closed by ezio.melotti #22761: Catching StopIteraion inside generator expression. http://bugs.python.org/issue22761 closed by terry.reedy #22766: collections.Counter's in-place operators should return NotImpl http://bugs.python.org/issue22766 closed by rhettinger #22769: Tttk tag_has() throws TypeError when called without item http://bugs.python.org/issue22769 closed by serhiy.storchaka #22770: test_ttk_guionly and test_tk can cause Tk segfaults on OS X wh http://bugs.python.org/issue22770 closed by ned.deily #22773: Export Readline version and expect ANSI sequence for version < http://bugs.python.org/issue22773 closed by pitrou #22775: SimpleCookie not unpicklable with protocol 2+ http://bugs.python.org/issue22775 closed by serhiy.storchaka #22776: SyslogHandler's record formatting during emit(..) not covered http://bugs.python.org/issue22776 closed by python-dev #22778: list's in-place add doesn't return NotImplemented when appropr http://bugs.python.org/issue22778 closed by rhettinger #22779: deque in-place addition does not return NotImplemented when ap http://bugs.python.org/issue22779 closed by rhettinger #22784: test_asyncio fails without the ssl module http://bugs.python.org/issue22784 closed by python-dev #22786: Message could not be delivered http://bugs.python.org/issue22786 closed by ezio.melotti #22787: ssl.SSLContext.load_cert_chain() backport regression with None http://bugs.python.org/issue22787 closed by python-dev #22792: string replace() not documented http://bugs.python.org/issue22792 closed by georg.brandl #22793: test_uuid failure on OpenIndiana http://bugs.python.org/issue22793 closed by haypo #22795: Intermittent test_tarfile failures on zLinux http://bugs.python.org/issue22795 closed by serhiy.storchaka #22802: On Windows, if you try and use ccs=UTF-8 (or other variants) t http://bugs.python.org/issue22802 closed by georg.brandl #22803: textwrap.indent version added not documented http://bugs.python.org/issue22803 closed by rhettinger #22805: Having pythontest.net index.html link to hg repo http://bugs.python.org/issue22805 closed by brett.cannon #22809: Include/graminit.h and Python/graminit.c always rebuilt (break http://bugs.python.org/issue22809 closed by georg.brandl From contact at ionelmc.ro Sat Nov 8 16:46:08 2014 From: contact at ionelmc.ro (=?UTF-8?Q?Ionel_Cristian_M=C4=83rie=C8=99?=) Date: Sat, 8 Nov 2014 17:46:08 +0200 Subject: [Python-Dev] Improvements for Pathlib Message-ID: Hello, In the current incarnation Pathlib is missing some key features I need in my usecases. I want to contribute them but i'd like a bit of feedback on the new api before jumping to implementation. The four things I need are: #1. A context manager for temporary files and dirs (that cleans everything up on exit). Eg: with pathlib.TmpDir(suffix='', prefix='') as tmp: assert isinstance(tmp, pathlib.Path) with pathlib.TmpFile(suffix='', prefix='') as tmp: assert isinstance(tmp, pathlib.Path) #2. A context manager for changing working directory (that switches to the old cwd on exit). Eg: with path.cd(): assert os.getcwd() == path # ?3? . Shutil-like features: - copy_file - copy_dir (or copy_tree?) - copy (does copy_dir or copy_file depending on what the source is. How to handle symlinks?) - rm_dir (or rm_tree? Also, this would be required for the TmpDir path). #4. Zip files support. A specialised Path subclass that works inside a zip file. I'm ok having this as an external package but some discussion would be useful as I'd have to rely on pathlib internals. After some brief looking in the pathlib code it would appear the wise thing would be to have the zip path as the "drive" for the ZipPaths that refer to files inside the zip. I'm not sure where the best place is to store the internal zipfile object. Perhaps the accessor? Ideally the ZipFiles would work with the shtuil-like api just fine (or at least the readonly operations). Thanks, -- Ionel M. -------------- next part -------------- An HTML attachment was scrubbed... URL: From rymg19 at gmail.com Sat Nov 8 19:14:27 2014 From: rymg19 at gmail.com (Ryan Gonzalez) Date: Sat, 8 Nov 2014 12:14:27 -0600 Subject: [Python-Dev] Improvements for Pathlib In-Reply-To: References: Message-ID: +1 for the context manager ideas. Anything that is like a resource should be available as a context manager. On Sat, Nov 8, 2014 at 9:46 AM, Ionel Cristian M?rie? wrote: > Hello, > > In the current incarnation Pathlib is missing some key features I need in > my usecases. I want to contribute them but i'd like a bit of feedback on > the new api before jumping to implementation. > > The four things I need are: > > #1. A context manager for temporary files and dirs (that cleans everything > up on exit). Eg: > > with pathlib.TmpDir(suffix='', prefix='') as tmp: > assert isinstance(tmp, pathlib.Path) > > with pathlib.TmpFile(suffix='', prefix='') as tmp: > assert isinstance(tmp, pathlib.Path) > > #2. A context manager for changing working directory (that switches to the > old cwd on exit). Eg: > > with path.cd(): > assert os.getcwd() == path > > # > ?3? > . Shutil-like features: > > - copy_file > - copy_dir (or copy_tree?) > - copy (does copy_dir or copy_file depending on what the source is. How to > handle symlinks?) > - rm_dir (or rm_tree? Also, this would be required for the TmpDir path). > > #4. Zip files support. A specialised Path subclass that works inside a zip > file. I'm ok having this as an external package but some discussion would > be useful as I'd have to rely on pathlib internals. > > After some brief looking in the pathlib code it would appear the wise > thing would be to have the zip path as the "drive" for the ZipPaths that > refer to files inside the zip. > > I'm not sure where the best place is to store the internal zipfile object. > Perhaps the accessor? > > Ideally the ZipFiles would work with the shtuil-like api just fine (or at > least the readonly operations). > > Thanks, > -- Ionel M. > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/rymg19%40gmail.com > > -- Ryan If anybody ever asks me why I prefer C++ to C, my answer will be simple: "It's becauseslejfp23(@#Q*(E*EIdc-SEGFAULT. Wait, I don't think that was nul-terminated." Personal reality distortion fields are immune to contradictory evidence. - srean Check out my website: http://kirbyfan64.github.io/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From python-dev at masklinn.net Sat Nov 8 19:46:03 2014 From: python-dev at masklinn.net (Xavier Morel) Date: Sat, 8 Nov 2014 19:46:03 +0100 Subject: [Python-Dev] Improvements for Pathlib In-Reply-To: References: Message-ID: <60A2A14D-DCE4-4DF4-9F28-031F341F4EFC@masklinn.net> On 2014-11-08, at 16:46 , Ionel Cristian M?rie? wrote: > Hello, > > In the current incarnation Pathlib is missing some key features I need in my usecases. I want to contribute them but i'd like a bit of feedback on the new api before jumping to implementation. > > The four things I need are: > > #1. A context manager for temporary files and dirs (that cleans everything up on exit). Eg: > > with pathlib.TmpDir(suffix='', prefix='') as tmp: > assert isinstance(tmp, pathlib.Path) > > with pathlib.TmpFile(suffix='', prefix='') as tmp: > assert isinstance(tmp, pathlib.Path) > Why would pathlib need to provide this when tempfile already does? with tempfile.NamedTemporaryFile(prefix='') as f: tmp = pathlib.Path(f.name) with tempfile.TemporaryDirectoryDirectory(prefix='') as d: tmp = pathlib.Path(d.name) On 2014-11-08, at 19:14 , Ryan Gonzalez wrote: > +1 for the context manager ideas. Anything that is like a resource should be available as a context manager. The current working directory is not "a resource" though it's a big piece of global state. I've been bitten by context-manager-unexpectedly-being-global-state in the past (with warnings.catch_warnings, I had not noticed the note about thread-safety). All in all, not a fan. From ethan at stoneleaf.us Sat Nov 8 21:41:03 2014 From: ethan at stoneleaf.us (Ethan Furman) Date: Sat, 08 Nov 2014 12:41:03 -0800 Subject: [Python-Dev] Improvements for Pathlib In-Reply-To: <60A2A14D-DCE4-4DF4-9F28-031F341F4EFC@masklinn.net> References: <60A2A14D-DCE4-4DF4-9F28-031F341F4EFC@masklinn.net> Message-ID: <545E7FDF.8080106@stoneleaf.us> On 11/08/2014 10:46 AM, Xavier Morel wrote: > On 2014-11-08, at 16:46 , Ionel Cristian M?rie? wrote: >> >> In the current incarnation Pathlib is missing some key features I need in >> my usecases. I want to contribute them but i'd like a bit of feedback on >> the new api before jumping to implementation. >> #1. A context manager for temporary files and dirs (that cleans everything >> up on exit). > > Why would pathlib need to provide this when tempfile already does? Because tempfile doesn't accept PathLib instances? -- ~Ethan~ From python-dev at masklinn.net Sat Nov 8 23:06:20 2014 From: python-dev at masklinn.net (Xavier Morel) Date: Sat, 8 Nov 2014 23:06:20 +0100 Subject: [Python-Dev] Improvements for Pathlib In-Reply-To: References: <60A2A14D-DCE4-4DF4-9F28-031F341F4EFC@masklinn.net> Message-ID: <611A317D-3513-4B3B-92D7-19F7345FE4DA@masklinn.net> On 2014-11-08, at 20:02 , Ionel Cristian M?rie? wrote: > On Saturday, November 8, 2014, Xavier Morel wrote: > > Why would pathlib need to provide this when tempfile already does? > > with tempfile.NamedTemporaryFile(prefix='') as f: > tmp = pathlib.Path(f.name) > > with tempfile.TemporaryDirectoryDirectory(prefix='') as d: > tmp = pathlib.Path(d.name) > > For the same reason pathlib provides path manipulation functionality while os.path already does: a cohesive and more convenient api. In other words, I want to have less calls and variables around (less room for errors, less visual bloat), and this is such a common pattern it's worth supporting it in pathlib. But now you've got two different places for tempfiles depending whether you want them to have an fs path or not. And the API isn't even more convenient when it comes to tempfiles, only to tempdir. Which could be fixed just as easily by adding a `path` attribute to tempfile.TemporaryDirectory. On 2014-11-08, at 21:41 , Ethan Furman wrote: > On 11/08/2014 10:46 AM, Xavier Morel wrote: >> On 2014-11-08, at 16:46 , Ionel Cristian M?rie? wrote: >>> >>> In the current incarnation Pathlib is missing some key features I need in >>> my usecases. I want to contribute them but i'd like a bit of feedback on >>> the new api before jumping to implementation. > >>> #1. A context manager for temporary files and dirs (that cleans everything >>> up on exit). >> >> Why would pathlib need to provide this when tempfile already does? > > Because tempfile doesn't accept PathLib instances? Which can be fixed by adding support for `dir` being a Path in mkstemp, mktemp and mkdtemp. From cs at zip.com.au Sun Nov 9 04:18:23 2014 From: cs at zip.com.au (Cameron Simpson) Date: Sun, 9 Nov 2014 14:18:23 +1100 Subject: [Python-Dev] Improvements for Pathlib In-Reply-To: References: Message-ID: <20141109031823.GA87988@cskk.homeip.net> On 08Nov2014 17:46, Ionel Cristian M?rie? wrote: >#2. A context manager for changing working directory (that switches to the >old cwd on exit). Eg: > >with path.cd(): > assert os.getcwd() == path As elsewhere remarked, the current working directory is process global state. Changing it can easily have unwanted (and invisible until breakage becomes glaring) side effects. -1 on this element from me I'm afraid. Cheers, Cameron Simpson I couldn't think of anything else to do with it, so I put it on the web. From ncoghlan at gmail.com Sun Nov 9 13:44:27 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 9 Nov 2014 22:44:27 +1000 Subject: [Python-Dev] [Python-checkins] cpython (2.7): #22650: test suite: load Unicode test data files from www.pythontest.net In-Reply-To: References: <20141106135652.108375.20457@psf.io> Message-ID: On 7 Nov 2014 02:44, "Georg Brandl" wrote: > > On 11/06/2014 03:39 PM, Brett Cannon wrote: > > What is pythontest.net ? Is it something we control, and > > if so how do we add things to it for tests? Did I miss an email on python-dev or > > python-committers about this? > > Benjamin already gave the link to the related issue. The idea is to make the > networking tests depend on as few external hosts as possible, so that we can > reduce spurious skips or failures when one of them is not reachable or changes > its configuration. > > The pythontest.net domain was chosen to be able to test things like cookies or > certificates without a potential of compromising python.org related services. > > There is a repository at hg.python.org/pythontestdotnet which can be used by > developers; server configuration is maintained together with other Python > services on https://github.com/python/psf-salt. Thanks for the info! Could we capture that somewhere a bit more permanent? The developer guide would probably be a good place, although it may involve creating a new subsection. Cheers, Nick. > > cheers, > Georg > > _______________________________________________ > Python-checkins mailing list > Python-checkins at python.org > https://mail.python.org/mailman/listinfo/python-checkins -------------- next part -------------- An HTML attachment was scrubbed... URL: From tds333+pydev at gmail.com Mon Nov 10 09:38:12 2014 From: tds333+pydev at gmail.com (Wolfgang Langner) Date: Mon, 10 Nov 2014 09:38:12 +0100 Subject: [Python-Dev] Improvements for Pathlib In-Reply-To: References: Message-ID: <54607974.4050100@gmail.com> Hi, some background about my pathlib usage. I use the backported version of pathlib and tried to replace the usage of path.py (also widely used library). Some features part of path.py missing in pathlib: - expanduser, expandvars, I have to use the os functions to do this. It is possible but better if this is part of pathlib. - shutil stuff, as noted below - dirs, and files from path.py. I often had to traverse a directory. I know this is possible with a generator. But It is so often needed that I missed this. Simple dirs(pattern) and files(pattern) is enough. With support for "**" as recursion. Pathlib is a great library and it was good to add it to the standard lib. Now it is time to improve it further. On 08.11.2014 16:46, Ionel Cristian M?rie? wrote: > Hello, > > In the current incarnation Pathlib is missing some key features I need in my > usecases. I want to contribute them but i'd like a bit of feedback on the new > api before jumping to implementation. > > The four things I need are: > > #1. A context manager for temporary files and dirs (that cleans everything up > on exit). Eg: > > with pathlib.TmpDir(suffix='', prefix='') as tmp: > assert isinstance(tmp, pathlib.Path) > > with pathlib.TmpFile(suffix='', prefix='') as tmp: > assert isinstance(tmp, pathlib.Path) Good idea, but we should better add path support to tempfile module. > #2. A context manager for changing working directory (that switches to the old > cwd on exit). Eg: > > with path.cd (): > assert os.getcwd() == path Also useful. Even if the state is global. But has to be noted in documentation. > # > ?3? > . Shutil-like features: > > - copy_file > - copy_dir (or copy_tree?) > - copy (does copy_dir or copy_file depending on what the source is. How to > handle symlinks?) > - rm_dir (or rm_tree? Also, this would be required for the TmpDir path). I also often needed this. Thanks for bringing this up. Regards, Wolfgang From contact at ionelmc.ro Sat Nov 8 20:02:23 2014 From: contact at ionelmc.ro (=?UTF-8?Q?Ionel_Cristian_M=C4=83rie=C8=99?=) Date: Sat, 8 Nov 2014 21:02:23 +0200 Subject: [Python-Dev] Improvements for Pathlib In-Reply-To: <60A2A14D-DCE4-4DF4-9F28-031F341F4EFC@masklinn.net> References: <60A2A14D-DCE4-4DF4-9F28-031F341F4EFC@masklinn.net> Message-ID: On Saturday, November 8, 2014, Xavier Morel wrote: > > > Why would pathlib need to provide this when tempfile already does? > > with tempfile.NamedTemporaryFile(prefix='') as f: > tmp = pathlib.Path(f.name) > > with tempfile.TemporaryDirectoryDirectory(prefix='') as d: > tmp = pathlib.Path(d.name) > > For the same reason pathlib provides path manipulation functionality while os.path already does: a cohesive and more convenient api. In other words, I want to have less calls and variables around (less room for errors, less visual bloat), and this is such a common pattern it's worth supporting it in pathlib. -- Thanks, -- Ionel M. -------------- next part -------------- An HTML attachment was scrubbed... URL: From contact at ionelmc.ro Mon Nov 10 12:45:57 2014 From: contact at ionelmc.ro (=?UTF-8?Q?Ionel_Cristian_M=C4=83rie=C8=99?=) Date: Mon, 10 Nov 2014 13:45:57 +0200 Subject: [Python-Dev] pathlib difference to the backport Message-ID: Hey, It appears there's a peculiar difference between the pathlib in the 3.4 branch and the one on bitbucket: cpython's pathlib.Path implements a no-op context manager interface. What's the purpose of that? It's also inconsistent, stat and all the methods that depend on stat do not implement the "is closed check" (raise ValueError if __exit__ was already called). Can anyone explain? Thanks, -- Ionel M. -------------- next part -------------- An HTML attachment was scrubbed... URL: From contact at ionelmc.ro Mon Nov 10 12:51:45 2014 From: contact at ionelmc.ro (=?UTF-8?Q?Ionel_Cristian_M=C4=83rie=C8=99?=) Date: Mon, 10 Nov 2014 13:51:45 +0200 Subject: [Python-Dev] Improvements for Pathlib In-Reply-To: <611A317D-3513-4B3B-92D7-19F7345FE4DA@masklinn.net> References: <60A2A14D-DCE4-4DF4-9F28-031F341F4EFC@masklinn.net> <611A317D-3513-4B3B-92D7-19F7345FE4DA@masklinn.net> Message-ID: I disagree, having pathlib support in just one place in stdlib is in fact what's really inconsistent. Also, I'm pretty sure that sprinkling pathlib support all over the stdlib isn't going to be pretty. Thanks, -- Ionel M. On Sun, Nov 9, 2014 at 12:06 AM, Xavier Morel wrote: > On 2014-11-08, at 20:02 , Ionel Cristian M?rie? > wrote: > > On Saturday, November 8, 2014, Xavier Morel > wrote: > > > > Why would pathlib need to provide this when tempfile already does? > > > > with tempfile.NamedTemporaryFile(prefix='') as f: > > tmp = pathlib.Path(f.name) > > > > with tempfile.TemporaryDirectoryDirectory(prefix='') as d: > > tmp = pathlib.Path(d.name) > > > > For the same reason pathlib provides path manipulation functionality > while os.path already does: a cohesive and more convenient api. In other > words, I want to have less calls and variables around (less room for > errors, less visual bloat), and this is such a common pattern it's worth > supporting it in pathlib. > > But now you've got two different places for tempfiles depending whether > you want them to have an fs path or not. And the API isn't even more > convenient when it comes to tempfiles, only to tempdir. Which could be > fixed just as easily by adding a `path` attribute to > tempfile.TemporaryDirectory. > > On 2014-11-08, at 21:41 , Ethan Furman wrote: > > On 11/08/2014 10:46 AM, Xavier Morel wrote: > >> On 2014-11-08, at 16:46 , Ionel Cristian M?rie? wrote: > >>> > >>> In the current incarnation Pathlib is missing some key features I need > in > >>> my usecases. I want to contribute them but i'd like a bit of feedback > on > >>> the new api before jumping to implementation. > > > >>> #1. A context manager for temporary files and dirs (that cleans > everything > >>> up on exit). > >> > >> Why would pathlib need to provide this when tempfile already does? > > > > Because tempfile doesn't accept PathLib instances? > > Which can be fixed by adding support for `dir` being a Path in mkstemp, > mktemp and mkdtemp. > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/contact%40ionelmc.ro > -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Mon Nov 10 17:53:19 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 10 Nov 2014 17:53:19 +0100 Subject: [Python-Dev] pathlib difference to the backport References: Message-ID: <20141110175319.02926c34@fsol> On Mon, 10 Nov 2014 13:45:57 +0200 Ionel Cristian M?rie? wrote: > Hey, > > It appears there's a peculiar difference between the pathlib in the 3.4 > branch and the one on bitbucket: cpython's pathlib.Path implements a no-op > context manager interface. What's the purpose of that? Hum... That's a remnant of the various pathlib prototypes, where this was used for the openat()-based implementation. It should probably be removed. Regards Antoine. From contact at ionelmc.ro Mon Nov 10 18:10:14 2014 From: contact at ionelmc.ro (=?UTF-8?Q?Ionel_Cristian_M=C4=83rie=C8=99?=) Date: Mon, 10 Nov 2014 19:10:14 +0200 Subject: [Python-Dev] pathlib difference to the backport In-Reply-To: <20141110175319.02926c34@fsol> References: <20141110175319.02926c34@fsol> Message-ID: Speaking of that, shouldn't pathlib have support for dir_fd? Thanks, -- Ionel M. On Mon, Nov 10, 2014 at 6:53 PM, Antoine Pitrou wrote: > On Mon, 10 Nov 2014 13:45:57 +0200 > Ionel Cristian M?rie? wrote: > > Hey, > > > > It appears there's a peculiar difference between the pathlib in the 3.4 > > branch and the one on bitbucket: cpython's pathlib.Path implements a > no-op > > context manager interface. What's the purpose of that? > > Hum... That's a remnant of the various pathlib prototypes, where this > was used for the openat()-based implementation. It should probably be > removed. > > Regards > > Antoine. > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/contact%40ionelmc.ro > -------------- next part -------------- An HTML attachment was scrubbed... URL: From buck.2019 at gmail.com Wed Nov 12 23:16:50 2014 From: buck.2019 at gmail.com (Buck Golemon) Date: Wed, 12 Nov 2014 14:16:50 -0800 Subject: [Python-Dev] py34 makes it harder to read all of a pty Message-ID: This code works as expected in python <3.4 (printing 'ok' then an error), but only gives an error in python3.4. ``` import os read_fd, write_fd = os.openpty() os.write(write_fd, b'ok\n') os.close(write_fd) read_file = os.fdopen(read_fd, 'rb', 0) print("read: %r" % read_file.read()) print("read: %r" % read_file.read()) ``` This is due to the fix for issue21090 , which aimed to un-silence errors which previously went unheard. The fix is for me, as a user, to write a loop that uses os.read and interpretes EIO as EOF. This is what I had hoped file.read() would do for me, however, and what it used to do in previous pythons. -------------- next part -------------- An HTML attachment was scrubbed... URL: From cf.natali at gmail.com Thu Nov 13 08:34:58 2014 From: cf.natali at gmail.com (=?ISO-8859-1?Q?Charles=2DFran=E7ois_Natali?=) Date: Thu, 13 Nov 2014 07:34:58 +0000 Subject: [Python-Dev] py34 makes it harder to read all of a pty In-Reply-To: References: Message-ID: 2014-11-12 22:16 GMT+00:00 Buck Golemon : > This is due to the fix for issue21090, which aimed to un-silence errors > which previously went unheard. The fix is for me, as a user, to write a loop > that uses os.read and interpretes EIO as EOF. This is what I had hoped > file.read() would do for me, however, and what it used to do in previous > pythons. There's no reason for read() to interpret EIO as EOF in the general case: it was masked in previous versions because of a mere bug. The behavior is now correct, although being able to retrieve the data read so far in case of a buffered read could be useful. From rosuav at gmail.com Thu Nov 13 09:05:32 2014 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 13 Nov 2014 19:05:32 +1100 Subject: [Python-Dev] py34 makes it harder to read all of a pty In-Reply-To: References: Message-ID: On Thu, Nov 13, 2014 at 6:34 PM, Charles-Fran?ois Natali wrote: > 2014-11-12 22:16 GMT+00:00 Buck Golemon : >> This is due to the fix for issue21090, which aimed to un-silence errors >> which previously went unheard. The fix is for me, as a user, to write a loop >> that uses os.read and interpretes EIO as EOF. This is what I had hoped >> file.read() would do for me, however, and what it used to do in previous >> pythons. > > > There's no reason for read() to interpret EIO as EOF in the general > case: it was masked in previous versions because of a mere bug. The > behavior is now correct, although being able to retrieve the data read > so far in case of a buffered read could be useful. Every change breaks someone's workflow. http://xkcd.com/1172/ ChrisA From Steve.Dower at microsoft.com Thu Nov 13 20:47:42 2014 From: Steve.Dower at microsoft.com (Steve Dower) Date: Thu, 13 Nov 2014 19:47:42 +0000 Subject: [Python-Dev] Who's using VS/Windows to work on Python? Message-ID: <873699b83b804cbb9ff76730da2448ad@DM2PR0301MB0734.namprd03.prod.outlook.com> Just wondering who is regularly/occasionally using VS 2010 to work on Python? Also, who currently owns the Windows buildbots and are you willing/able to add a VS 2015 Preview installation (or give me access so I can do it)? I can also set up a VS 2015 buildbot, but the old ones will break if the projects change (I should do this anyway to at least become familiar with how the buildbots work...) I want to get a feel for who will be disrupted when I merge in the changes to build with VS 2015, which will help influence when that merge occurs. VS 2015 Preview was just released yesterday, and while it's not perfect yet, it's isolated enough from VS 2010 that there won't be any issues installing them both (I can't say the same for VS 2013...). If people are okay with this, then we can consider merging the changes sooner. Once VS 2015 RC is released, I'll be looking to merge asap., but I don't see a strong need to do it before then other than a longer bake time. (In any case, the old project files will still be there, but right now it's easier to handle when those change. Once they are in their own directory, changes to the VS 2010 projects will need to be duplicated by hand in the VS 2015 projects, and while the converse is true, it's not as critical.) Cheers, Steve -------------- next part -------------- An HTML attachment was scrubbed... URL: From mail at timgolden.me.uk Thu Nov 13 20:54:41 2014 From: mail at timgolden.me.uk (Tim Golden) Date: Thu, 13 Nov 2014 19:54:41 +0000 Subject: [Python-Dev] Who's using VS/Windows to work on Python? In-Reply-To: <873699b83b804cbb9ff76730da2448ad@DM2PR0301MB0734.namprd03.prod.outlook.com> References: <873699b83b804cbb9ff76730da2448ad@DM2PR0301MB0734.namprd03.prod.outlook.com> Message-ID: <54650C81.4060104@timgolden.me.uk> On 13/11/2014 19:47, Steve Dower wrote: > Just wondering who is regularly/occasionally using VS 2010 to work on > Python? I use VS 2010 (and VS 2008 if I'm building 2.7). Frankly my time has been so scarce this last year, I don't manage to use them from one month to the next. But I do use them. > I want to get a feel for who will be disrupted when I merge in the > changes to build with VS 2015, which will help influence when that merge > occurs. VS 2015 Preview was just released yesterday, and while it?s not > perfect yet, it?s isolated enough from VS 2010 that there won?t be any > issues installing them both (I can?t say the same for VS 2013?). If > people are okay with this, then we can consider merging the changes > sooner. Once VS 2015 RC is released, I?ll be looking to merge asap., but > I don?t see a strong need to do it before then other than a longer bake > time. Are the changes for VS2015 likely to hamper a build against VS2010? TJG From ethan at stoneleaf.us Thu Nov 13 21:50:56 2014 From: ethan at stoneleaf.us (Ethan Furman) Date: Thu, 13 Nov 2014 12:50:56 -0800 Subject: [Python-Dev] Who's using VS/Windows to work on Python? In-Reply-To: <873699b83b804cbb9ff76730da2448ad@DM2PR0301MB0734.namprd03.prod.outlook.com> References: <873699b83b804cbb9ff76730da2448ad@DM2PR0301MB0734.namprd03.prod.outlook.com> Message-ID: <546519B0.409@stoneleaf.us> On 11/13/2014 11:47 AM, Steve Dower wrote: > > Just wondering who is regularly/occasionally using VS 2010 to work on Python? Very occasional. In fact, my MSDS subscription expired and I missed the last call for renewals. :( If it will help (and I can get a renewed subscription), I can built/test on my Win7 machine. -- ~Ethan~ From Steve.Dower at microsoft.com Thu Nov 13 22:17:18 2014 From: Steve.Dower at microsoft.com (Steve Dower) Date: Thu, 13 Nov 2014 21:17:18 +0000 Subject: [Python-Dev] Who's using VS/Windows to work on Python? In-Reply-To: <54650C81.4060104@timgolden.me.uk> References: <873699b83b804cbb9ff76730da2448ad@DM2PR0301MB0734.namprd03.prod.outlook.com> <54650C81.4060104@timgolden.me.uk> Message-ID: <695285dac1cd4455add2434912ff12ac@DM2PR0301MB0734.namprd03.prod.outlook.com> Tim Golden wrote: > Are the changes for VS2015 likely to hamper a build against VS2010? I hadn't tried it, but it looks like it works fine with VS2010 (with a few extra warnings, but nothing fatal as far as the build is concerned). I'm a little surprised, tbh, but not overly upset :) Guess I can go ahead and start merging the changes (via issues w/patches) whenever I like then. The most painful change will probably be moving the x86 outputs from PCBuild to PCBuild\win32, but I'll get that into the buildbot scripts and it should be fine. Cheers, Steve From db3l.net at gmail.com Thu Nov 13 23:01:49 2014 From: db3l.net at gmail.com (David Bolen) Date: Thu, 13 Nov 2014 17:01:49 -0500 Subject: [Python-Dev] Who's using VS/Windows to work on Python? References: <873699b83b804cbb9ff76730da2448ad@DM2PR0301MB0734.namprd03.prod.outlook.com> Message-ID: Steve Dower writes: > Also, who currently owns the Windows buildbots and are you > willing/able to add a VS 2015 Preview installation (or give me > access so I can do it)? (...) I've got several of the Windows buildbots, and could add this. Is there benefit to just starting with one (I've got XP, Win7 and Win8) first to save some time? My only real concern is to verify that this will co-exist properly with the existing VS installations and buildbot build process. I note that the download page says not to use on production computers. That's probably just a CYA, and I don't mind if VS itself has issues, but if it risks compromising other aspects of the buildbot, then I think testing on a different machine would be preferred. Is anything aside from VS 2015 itself (e.g., the Ultimate 2015 download on the preview page) needed? I'm assuming not. -- David From Steve.Dower at microsoft.com Thu Nov 13 23:40:47 2014 From: Steve.Dower at microsoft.com (Steve Dower) Date: Thu, 13 Nov 2014 22:40:47 +0000 Subject: [Python-Dev] Who's using VS/Windows to work on Python? In-Reply-To: References: <873699b83b804cbb9ff76730da2448ad@DM2PR0301MB0734.namprd03.prod.outlook.com> Message-ID: <8d3d5e82a3c64126ab1717513a572d81@DM2PR0301MB0734.namprd03.prod.outlook.com> David Bolen wrote: > Steve Dower writes: > >> Also, who currently owns the Windows buildbots and are you >> willing/able to add a VS 2015 Preview installation (or give me access >> so I can do it)? (...) > > I've got several of the Windows buildbots, and could add this. Is there benefit > to just starting with one (I've got XP, Win7 and Win8) first to save some time? Starting with just the Win7 or Win8 one would be fine. Python 3.5 won't support XP, and VS 2015 doesn't support XP (though I believe it will still be able to build for XP, just not *on* XP). > My only real concern is to verify that this will co-exist properly with the > existing VS installations and buildbot build process. I note that the download > page says not to use on production computers. That's probably just a CYA, and I > don't mind if VS itself has issues, but if it risks compromising other aspects > of the buildbot, then I think testing on a different machine would be preferred. VS 2010 should be fine. The most likely issues are with VS 2013 (for teams that haven't updated their setup authoring yet), but these should have been ironed out already. > Is anything aside from VS 2015 itself (e.g., the Ultimate 2015 download on the > preview page) needed? I'm assuming not. Nope, that's it. I'll make sure there's a free version that will work eventually, but right now only Ultimate is available. I will make sure to let you know when the merge happens, but if VS 2015 is already there then that should be smooth anyway. Thanks, Steve > -- David From db3l.net at gmail.com Thu Nov 13 23:58:47 2014 From: db3l.net at gmail.com (David Bolen) Date: Thu, 13 Nov 2014 17:58:47 -0500 Subject: [Python-Dev] Who's using VS/Windows to work on Python? References: <873699b83b804cbb9ff76730da2448ad@DM2PR0301MB0734.namprd03.prod.outlook.com> <8d3d5e82a3c64126ab1717513a572d81@DM2PR0301MB0734.namprd03.prod.outlook.com> Message-ID: Steve Dower writes: > Starting with just the Win7 or Win8 one would be fine. Python 3.5 > won't support XP, and VS 2015 doesn't support XP (though I believe > it will still be able to build for XP, just not *on* XP). Ok, I'll probably try the Win8 buildbot first then. I'll let you know when it's available. FYI, the VS 2015 requirements page seems to include XP as a supported client OS (if I'm reading it correctly), and says the requirements are the same as for VS 2013. Of course, as you say it won't matter if we're dropping XP for Python 3.5. > VS 2010 should be fine. The most likely issues are with VS 2013 (for > teams that haven't updated their setup authoring yet), but these > should have been ironed out already. Ok, the buildbots are just VS 2008 and VS 2010, so no 2013 to worry about. > Nope, that's it. (...) Sounds good. -- David From ncoghlan at gmail.com Fri Nov 14 03:18:49 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 14 Nov 2014 12:18:49 +1000 Subject: [Python-Dev] Who's using VS/Windows to work on Python? In-Reply-To: <873699b83b804cbb9ff76730da2448ad@DM2PR0301MB0734.namprd03.prod.outlook.com> References: <873699b83b804cbb9ff76730da2448ad@DM2PR0301MB0734.namprd03.prod.outlook.com> Message-ID: On 14 November 2014 05:47, Steve Dower wrote: > Just wondering who is regularly/occasionally using VS 2010 to work on > Python? > > > > Also, who currently owns the Windows buildbots and are you willing/able to > add a VS 2015 Preview installation (or give me access so I can do it)? I > can also set up a VS 2015 buildbot, but the old ones will break if the > projects change (I should do this anyway to at least become familiar with > how the buildbots work?) > > > > > > I want to get a feel for who will be disrupted when I merge in the changes > to build with VS 2015, which will help influence when that merge occurs. VS > 2015 Preview was just released yesterday, and while it?s not perfect yet, > it?s isolated enough from VS 2010 that there won?t be any issues installing > them both (I can?t say the same for VS 2013?). If people are okay with > this, then we can consider merging the changes sooner. Once VS 2015 RC is > released, I?ll be looking to merge asap., but I don?t see a strong need to > do it before then other than a longer bake time. > With the recent (excellent) announcements around switching from the current Visual Studio Express model to the far more full featured Visual Studio Community approach, what will the situation be in relation to using Visual Studio Community edition to work on CPython? This thread, and Brian's post regarding renewing MSDN subscriptions made me realise we're going to need to be able to explain that to new Windows based contributors. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia -------------- next part -------------- An HTML attachment was scrubbed... URL: From rajshorya at gmail.com Fri Nov 14 09:34:49 2014 From: rajshorya at gmail.com (Shorya Raj) Date: Fri, 14 Nov 2014 21:34:49 +1300 Subject: [Python-Dev] Windows Dedicated Mailing List Message-ID: Hello I have been following the lists for a while, and have noticed that there do not seem to be any sort of specifics for Windows. Now, as someone developing python applications (and hoping to develop python itself on Windows), I think it may be prudent to consider adding a Windows specific mailing list regarding development on Windows, whether it is development of Python on windows, or developing Python applications on Windows. Please let me know what you guys think - if there is consensus, it would be great to get it up and running, and I would be more than willing to take steps from my side as required. Thanks SbSpider -------------- next part -------------- An HTML attachment was scrubbed... URL: From mail at timgolden.me.uk Fri Nov 14 16:10:59 2014 From: mail at timgolden.me.uk (Tim Golden) Date: Fri, 14 Nov 2014 15:10:59 +0000 Subject: [Python-Dev] Windows Dedicated Mailing List In-Reply-To: References: Message-ID: <54661B83.6080504@timgolden.me.uk> On 14/11/2014 08:34, Shorya Raj wrote: > Hello > I have been following the lists for a while, and have noticed that there > do not seem to be any sort of specifics for Windows. Now, as someone > developing python applications (and hoping to develop python itself on > Windows), I think it may be prudent to consider adding a Windows > specific mailing list regarding development on Windows, whether it is > development of Python on windows, or developing Python applications on > Windows. Please let me know what you guys think - if there is consensus, > it would be great to get it up and running, and I would be more than > willing to take steps from my side as required. That would be this: https://mail.python.org/mailman/listinfo/python-win32 TJG From Steve.Dower at microsoft.com Fri Nov 14 17:35:12 2014 From: Steve.Dower at microsoft.com (Steve Dower) Date: Fri, 14 Nov 2014 16:35:12 +0000 Subject: [Python-Dev] Windows Dedicated Mailing List In-Reply-To: <54661B83.6080504@timgolden.me.uk> References: , <54661B83.6080504@timgolden.me.uk> Message-ID: I'd like to keep development *of* Python here, regardless of platform. Otherwise all the Linux and Mac people might forget about us :) Top-posted from my Windows Phone ________________________________ From: Tim Golden Sent: ?11/?14/?2014 7:11 To: Shorya Raj; Python Dev Subject: Re: [Python-Dev] Windows Dedicated Mailing List On 14/11/2014 08:34, Shorya Raj wrote: > Hello > I have been following the lists for a while, and have noticed that there > do not seem to be any sort of specifics for Windows. Now, as someone > developing python applications (and hoping to develop python itself on > Windows), I think it may be prudent to consider adding a Windows > specific mailing list regarding development on Windows, whether it is > development of Python on windows, or developing Python applications on > Windows. Please let me know what you guys think - if there is consensus, > it would be great to get it up and running, and I would be more than > willing to take steps from my side as required. That would be this: https://mail.python.org/mailman/listinfo/python-win32 TJG _______________________________________________ Python-Dev mailing list Python-Dev at python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/steve.dower%40microsoft.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From status at bugs.python.org Fri Nov 14 18:08:15 2014 From: status at bugs.python.org (Python tracker) Date: Fri, 14 Nov 2014 18:08:15 +0100 (CET) Subject: [Python-Dev] Summary of Python tracker Issues Message-ID: <20141114170815.E2E06560D1@psf.upfronthosting.co.za> ACTIVITY SUMMARY (2014-11-07 - 2014-11-14) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 4647 (+25) closed 29986 (+31) total 34633 (+56) Open issues with patches: 2168 Issues opened (42) ================== #17293: uuid.getnode() MAC address on AIX http://bugs.python.org/issue17293 reopened by koobs #22817: re.split fails with lookahead/behind http://bugs.python.org/issue22817 opened by rexdwyer #22818: Deprecate splitting on possible zero-width re patterns http://bugs.python.org/issue22818 opened by serhiy.storchaka #22819: Python3.4: xml.sax.saxutils.XMLGenerator.__init__ fails with p http://bugs.python.org/issue22819 opened by Edward.K..Ream #22820: RESTART line with no output http://bugs.python.org/issue22820 opened by sukari #22822: IPv6Network constructor docs incorrect about valid input http://bugs.python.org/issue22822 opened by pebenito #22823: Use set literals instead of creating a set from a list http://bugs.python.org/issue22823 opened by rhettinger #22825: Modernize TextFile http://bugs.python.org/issue22825 opened by serhiy.storchaka #22826: Support context management protocol in bkfile http://bugs.python.org/issue22826 opened by serhiy.storchaka #22827: Backport ensurepip to 2.7 (PEP 477) http://bugs.python.org/issue22827 opened by dstufft #22829: Add --prompt option to venv http://bugs.python.org/issue22829 opened by ??ukasz.Balcerzak #22831: Use "with" to avoid possible fd leaks http://bugs.python.org/issue22831 opened by serhiy.storchaka #22832: Tweak parameter names for fcntl module http://bugs.python.org/issue22832 opened by brett.cannon #22833: The decode_header() function decodes raw part to bytes or str, http://bugs.python.org/issue22833 opened by py.user #22834: Unexpected FileNotFoundError when current directory is removed http://bugs.python.org/issue22834 opened by vadmium #22836: Broken "Exception ignored in:" message on exceptions in __repr http://bugs.python.org/issue22836 opened by The Compiler #22837: getpass returns garbage when typing tilde on Windows with dead http://bugs.python.org/issue22837 opened by The Compiler #22838: Convert re tests to unittest http://bugs.python.org/issue22838 opened by serhiy.storchaka #22840: strpdate('20141110', '%Y%m%d%H%S') returns wrong date http://bugs.python.org/issue22840 opened by dgorley #22841: Avoid to use coroutine with add_signal_handler() http://bugs.python.org/issue22841 opened by Ludovic.Gasc #22843: doc error: 6.2.4. Match Objects http://bugs.python.org/issue22843 opened by crkirkwood #22844: test_gdb failure on Debian Wheezy for Z http://bugs.python.org/issue22844 opened by David.Edelsohn #22847: Improve method cache efficiency http://bugs.python.org/issue22847 opened by pitrou #22848: Subparser help does not respect SUPPRESS argument http://bugs.python.org/issue22848 opened by Brett.Hannigan #22850: Backport ensurepip Windows installer changes to 2.7 http://bugs.python.org/issue22850 opened by steve.dower #22851: core crashes http://bugs.python.org/issue22851 opened by doko #22852: urllib.parse wrongly strips empty #fragment http://bugs.python.org/issue22852 opened by soilandreyes #22853: Multiprocessing.Queue._feed deadlocks on import http://bugs.python.org/issue22853 opened by ffinkernagel #22854: Documentation/implementation out of sync for IO http://bugs.python.org/issue22854 opened by viraptor #22855: csv writer with blank lineterminator breaks quoting http://bugs.python.org/issue22855 opened by Eric.Haszlakiewicz #22858: unittest.__init__:main shadows unittest.main http://bugs.python.org/issue22858 opened by rbcollins #22859: unittest.TestProgram.usageExit no longer invoked http://bugs.python.org/issue22859 opened by rbcollins #22860: unittest TestProgram hard to extend http://bugs.python.org/issue22860 opened by rbcollins #22861: [2.7] ssl._dnsname_match() and unicode http://bugs.python.org/issue22861 opened by haypo #22863: https://docs.python.org/ should make a true 2.7.8 version avai http://bugs.python.org/issue22863 opened by lemburg #22864: Add filter to multiprocessing.Pool http://bugs.python.org/issue22864 opened by Mike.Drob #22865: Allow pty.spawn to ignore data to copy http://bugs.python.org/issue22865 opened by RadicalZephyr #22867: document behavior of calling atexit.register() while atexit._r http://bugs.python.org/issue22867 opened by skip.montanaro #22869: Split pylifecycle.c out from pythonrun.c http://bugs.python.org/issue22869 opened by ncoghlan #22870: urlopen timeout failed with SSL socket http://bugs.python.org/issue22870 opened by daveti #22871: datetime documentation incomplete http://bugs.python.org/issue22871 opened by spalac24 #433030: SRE: Atomic Grouping (?>...) is not supported http://bugs.python.org/issue433030 reopened by serhiy.storchaka Most recent 15 issues with no replies (15) ========================================== #22871: datetime documentation incomplete http://bugs.python.org/issue22871 #22870: urlopen timeout failed with SSL socket http://bugs.python.org/issue22870 #22867: document behavior of calling atexit.register() while atexit._r http://bugs.python.org/issue22867 #22865: Allow pty.spawn to ignore data to copy http://bugs.python.org/issue22865 #22864: Add filter to multiprocessing.Pool http://bugs.python.org/issue22864 #22859: unittest.TestProgram.usageExit no longer invoked http://bugs.python.org/issue22859 #22853: Multiprocessing.Queue._feed deadlocks on import http://bugs.python.org/issue22853 #22844: test_gdb failure on Debian Wheezy for Z http://bugs.python.org/issue22844 #22841: Avoid to use coroutine with add_signal_handler() http://bugs.python.org/issue22841 #22838: Convert re tests to unittest http://bugs.python.org/issue22838 #22832: Tweak parameter names for fcntl module http://bugs.python.org/issue22832 #22831: Use "with" to avoid possible fd leaks http://bugs.python.org/issue22831 #22829: Add --prompt option to venv http://bugs.python.org/issue22829 #22825: Modernize TextFile http://bugs.python.org/issue22825 #22822: IPv6Network constructor docs incorrect about valid input http://bugs.python.org/issue22822 Most recent 15 issues waiting for review (15) ============================================= #22869: Split pylifecycle.c out from pythonrun.c http://bugs.python.org/issue22869 #22865: Allow pty.spawn to ignore data to copy http://bugs.python.org/issue22865 #22850: Backport ensurepip Windows installer changes to 2.7 http://bugs.python.org/issue22850 #22848: Subparser help does not respect SUPPRESS argument http://bugs.python.org/issue22848 #22847: Improve method cache efficiency http://bugs.python.org/issue22847 #22841: Avoid to use coroutine with add_signal_handler() http://bugs.python.org/issue22841 #22838: Convert re tests to unittest http://bugs.python.org/issue22838 #22831: Use "with" to avoid possible fd leaks http://bugs.python.org/issue22831 #22829: Add --prompt option to venv http://bugs.python.org/issue22829 #22827: Backport ensurepip to 2.7 (PEP 477) http://bugs.python.org/issue22827 #22826: Support context management protocol in bkfile http://bugs.python.org/issue22826 #22825: Modernize TextFile http://bugs.python.org/issue22825 #22823: Use set literals instead of creating a set from a list http://bugs.python.org/issue22823 #22818: Deprecate splitting on possible zero-width re patterns http://bugs.python.org/issue22818 #22817: re.split fails with lookahead/behind http://bugs.python.org/issue22817 Top 10 most discussed issues (10) ================================= #22687: horrible performance of textwrap.wrap() with a long word http://bugs.python.org/issue22687 25 msgs #22823: Use set literals instead of creating a set from a list http://bugs.python.org/issue22823 21 msgs #2636: Adding a new regex module (compatible with re) http://bugs.python.org/issue2636 11 msgs #22827: Backport ensurepip to 2.7 (PEP 477) http://bugs.python.org/issue22827 11 msgs #22840: strpdate('20141110', '%Y%m%d%H%S') returns wrong date http://bugs.python.org/issue22840 10 msgs #22850: Backport ensurepip Windows installer changes to 2.7 http://bugs.python.org/issue22850 7 msgs #22817: re.split fails with lookahead/behind http://bugs.python.org/issue22817 6 msgs #22813: No facility for test randomisation http://bugs.python.org/issue22813 5 msgs #22847: Improve method cache efficiency http://bugs.python.org/issue22847 5 msgs #22791: datetime.utcfromtimestamp() shoud have option for create tz aw http://bugs.python.org/issue22791 4 msgs Issues closed (29) ================== #9179: Lookback with group references incorrect (two issues?) http://bugs.python.org/issue9179 closed by serhiy.storchaka #12728: Python re lib fails case insensitive matches on Unicode data http://bugs.python.org/issue12728 closed by serhiy.storchaka #13444: closed stdout causes error on stderr when the interpreter unco http://bugs.python.org/issue13444 closed by berker.peksag #19796: urllib2.HTTPError.reason is not documented as "Added in 2.7" http://bugs.python.org/issue19796 closed by berker.peksag #21650: add json.tool option to avoid alphabetic sort of fields http://bugs.python.org/issue21650 closed by berker.peksag #22300: PEP 446 What's New Updates for 2.7.9 http://bugs.python.org/issue22300 closed by python-dev #22434: Use named constants internally in the re module http://bugs.python.org/issue22434 closed by serhiy.storchaka #22578: Add additional attributes to re.error http://bugs.python.org/issue22578 closed by serhiy.storchaka #22600: In Multiprocessing Queue() doesn't work with list : __.put(lis http://bugs.python.org/issue22600 closed by amaury.forgeotdarc #22719: os.path.isfile & os.path.exists bug in while loop http://bugs.python.org/issue22719 closed by zach.ware #22800: IPv6Network constructor sometimes does not recognize legitimat http://bugs.python.org/issue22800 closed by pmoody #22808: Typo in asyncio-eventloop.rst, time() link is wrong http://bugs.python.org/issue22808 closed by berker.peksag #22816: spam http://bugs.python.org/issue22816 closed by serhiy.storchaka #22821: Argument of wrong type is passed to fcntl() http://bugs.python.org/issue22821 closed by serhiy.storchaka #22824: Update reprlib to use set literals http://bugs.python.org/issue22824 closed by rhettinger #22828: Backport ensurepip to 2.7 (PEP 477) http://bugs.python.org/issue22828 closed by dstufft #22830: functools.cmp_to_key: misleading key function description http://bugs.python.org/issue22830 closed by rhettinger #22835: urllib2/httplib is rendering 400s for every authenticated-SSL http://bugs.python.org/issue22835 closed by pitrou #22839: Incorrect link to statistics in tracemalloc documentation http://bugs.python.org/issue22839 closed by berker.peksag #22842: zipfile simultaneous open broken and/or needlessly(?) consumes http://bugs.python.org/issue22842 closed by r.david.murray #22845: Minor tweaks dis documentation http://bugs.python.org/issue22845 closed by serhiy.storchaka #22846: distutils needlessly fails to build apsw http://bugs.python.org/issue22846 closed by pitrou #22849: Double DECREF in TextIOWrapper http://bugs.python.org/issue22849 closed by python-dev #22856: Function Summons http://bugs.python.org/issue22856 closed by r.david.murray #22857: strftime should support %f to print milliseconds http://bugs.python.org/issue22857 closed by r.david.murray #22862: os.walk fails on undecodable filenames http://bugs.python.org/issue22862 closed by r.david.murray #22866: ssl module in 2.7.9 should provide a way to configure default http://bugs.python.org/issue22866 closed by lemburg #22868: Minor error in the example of filter() http://bugs.python.org/issue22868 closed by python-dev #814253: Grouprefs in lookbehind assertions http://bugs.python.org/issue814253 closed by serhiy.storchaka From Steve.Dower at microsoft.com Fri Nov 14 17:57:47 2014 From: Steve.Dower at microsoft.com (Steve Dower) Date: Fri, 14 Nov 2014 16:57:47 +0000 Subject: [Python-Dev] Who's using VS/Windows to work on Python? In-Reply-To: References: <873699b83b804cbb9ff76730da2448ad@DM2PR0301MB0734.namprd03.prod.outlook.com> Message-ID: <09e6af3c36344e7eaef106f2531d378e@DM2PR0301MB0734.namprd03.prod.outlook.com> Nick Coghlan wrote: > With the recent (excellent) announcements around switching from the current > Visual Studio Express model to the far more full featured Visual Studio > Community approach, what will the situation be in relation to using Visual > Studio Community edition to work on CPython? It "just works" :) VS Community is indistinguishable from VS Pro apart from some of the branding and a few features so obscure that I hadn't heard of or used them until I was told they were missing. Also the license, which allows for free use on open-source projects regardless of the company where you work (the limits are for commercial use at companies that can afford the product). > This thread, and Brian's post regarding renewing MSDN subscriptions made me > realise we're going to need to be able to explain that to new Windows based > contributors. Replace "Express" with "Community" everywhere (once the project migrations go in, since VS Community *2013* isn't a lot of help right now). I don't know exactly what the situation will be with VS 2015, but there will either be a free Community or a free Express for Desktop (without the 64-bit bug that was in VS 2010 Express), either of which will be fine for working on Python. Cheers, Steve > Cheers, > Nick. From solipsis at pitrou.net Fri Nov 14 18:46:59 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 14 Nov 2014 18:46:59 +0100 Subject: [Python-Dev] Windows Dedicated Mailing List References: <54661B83.6080504@timgolden.me.uk> Message-ID: <20141114184659.6eb13d73@fsol> On Fri, 14 Nov 2014 16:35:12 +0000 Steve Dower wrote: > I'd like to keep development *of* Python here, regardless of platform. Otherwise all the Linux and Mac people might forget about us :) +1 from a Linux developer. I find it useful to know what happens on other platforms (also I occasionally fire a Windows VM to do some development). Regards Antoine. From rymg19 at gmail.com Fri Nov 14 21:16:53 2014 From: rymg19 at gmail.com (Ryan Gonzalez) Date: Fri, 14 Nov 2014 14:16:53 -0600 Subject: [Python-Dev] Windows Dedicated Mailing List In-Reply-To: <20141114184659.6eb13d73@fsol> References: <54661B83.6080504@timgolden.me.uk> <20141114184659.6eb13d73@fsol> Message-ID: I agree completely (although I use multibooting instead of a VM). On Fri, Nov 14, 2014 at 11:46 AM, Antoine Pitrou wrote: > On Fri, 14 Nov 2014 16:35:12 +0000 > Steve Dower wrote: > > I'd like to keep development *of* Python here, regardless of platform. > Otherwise all the Linux and Mac people might forget about us :) > > +1 from a Linux developer. I find it useful to know what happens on > other platforms (also I occasionally fire a Windows VM to do some > development). > > Regards > > Antoine. > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/rymg19%40gmail.com > -- Ryan If anybody ever asks me why I prefer C++ to C, my answer will be simple: "It's becauseslejfp23(@#Q*(E*EIdc-SEGFAULT. Wait, I don't think that was nul-terminated." Personal reality distortion fields are immune to contradictory evidence. - srean Check out my website: http://kirbyfan64.github.io/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From madewokherd at gmail.com Sat Nov 15 01:12:59 2014 From: madewokherd at gmail.com (Vincent Povirk) Date: Fri, 14 Nov 2014 18:12:59 -0600 Subject: [Python-Dev] OneGet provider for Python Message-ID: I have been following OneGet development very closely, and I have volunteered to create a OneGet provider for installing Python packages. Normally, I would keep quiet about this sort of thing until I have actual code that works, but since no one else on the OneGet seems as interested in doing it, and the Python community (if this mailing list is anything to go by) doesn't know what to make of this stuff, I figured I should try to open lines of communication before I start essentially making decisions on this community's behalf. So, the main advantage of a Python provider is that you would be able to do things like: > Install-Package -Provider Python -Name Pygments or > Install-Package Pygments-2.0.1-py3-none-any.whl and OneGet would be able to bootstrap (if necessary) the Python provider itself, Python 3.x, and pip, and use pip to install the .whl. (I'm unclear at the moment on whether pip is necessary to install a .whl, but currently it looks to be the best thing to lean on for the job of installing/uninstalling things). My end goal is to be able to package a Python application such that an end-user on Windows (who doesn't know anything about Python) can easily install it, without either of us having to think about how all the dependencies are going to get there. I think that the best approach for the moment is to lean heavily on pip for installing/uninstalling things, while duplicating other tasks (such as listing installed packages, querying information about package files, searching PyPI) in C# so that they do not require a Python environment. If anyone has questions or concerns about this, please let me know. Keep in mind that I am not subscribed to the mailing list and will have to be CC'd. If anyone has questions about OneGet generally, you should probably ask them directly (see https://github.com/OneGet/oneget), as I am not a definitive source of information on the project. Incidentally, it would be really useful if python.org provided stable url's that always redirected to the latest .msi installers, for bootstrapping purposes. I'd prefer to not rely on chocolatey (or on scraping the web site) for this. (Also, I don't think this is ready yet, but in the future it will be possible to add tags to websites, such that one can do things like "Install-Package https://www.python.org" and have the right things happen.) From madewokherd at gmail.com Sat Nov 15 02:44:55 2014 From: madewokherd at gmail.com (Vincent Povirk) Date: Fri, 14 Nov 2014 19:44:55 -0600 Subject: [Python-Dev] OneGet provider for Python In-Reply-To: References: Message-ID: Someone pointed out off-list that I didn't provide any explanation of what OneGet is. I saw it had been brought up on this list earlier, and incorrectly assumed that meant everyone would know what I was talking about. OneGet is a new unified interface for using tools that work with software, which is being built and will ship with future versions of Windows (Linux support is planned but seems to be a long way off currently). It's intended to help solve the problems that a package manager would solve on Linux, without enforcing any particular ideas about how software should be packaged, obtained, or installed. It relies on external components (called "providers") to talk to existing packaging systems and do any real work. Currently, the only UI for OneGet is a set of PowerShell commands, like Install-Package (which can take a filename, url, or name of a package as long as some provider can make sense of it), but there is an API available for use by installers, packaging tools that want to define external dependencies in some generic way, or other UI's. So, having a Python provider would make it easier for Windows users to install Python software, and cut down on the work required to package software written in Python for Windows. (Ideally, in this case developers would just have to package it in PyPI, or provide a source archive or .whl, and OneGet should be able to automate installation of that and everything it needs, in this case probably by installing Python and pip, and asking pip to do the real work. I am planning to write the component that figures out all the Python-specific parts of this.) Or, more succinctly, the plan is to do everything pip already does (on Windows only for now), but in a way that also supports all kinds of other packagers and doesn't require the user to manually install Python and pip. From tjreedy at udel.edu Sat Nov 15 03:15:43 2014 From: tjreedy at udel.edu (Terry Reedy) Date: Fri, 14 Nov 2014 21:15:43 -0500 Subject: [Python-Dev] OneGet provider for Python In-Reply-To: References: Message-ID: <5466B74F.80601@udel.edu> On 11/14/2014 7:12 PM, Vincent Povirk wrote: > If anyone has questions about OneGet generally, you should probably > ask them directly (see https://github.com/OneGet/oneget), as I am not > a definitive source of information on the project. No subscription needed to follow a thread. Python-dev is mirrored at news.gmane.net. One can either 'subscribe' to the newsgroup interface in a mai/news reader, as I do (OutlookExpress years ago, Thunderbird now), or use the webgroup interface. -- Terry Jan Reedy From stephen at xemacs.org Sat Nov 15 06:36:11 2014 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Sat, 15 Nov 2014 14:36:11 +0900 Subject: [Python-Dev] Windows Dedicated Mailing List In-Reply-To: References: Message-ID: <87bno9vxdg.fsf@uwakimon.sk.tsukuba.ac.jp> Shorya Raj writes: > I think it may be prudent to consider adding a Windows specific > mailing list regarding development on Windows, One possibly-relevant anecdote. We had such a list for XEmacs, where the biggest contributor 1998-2005 was Windows-based. *He* preferred communicating via the main developers' list (the reasons given by others in this thread were among his reasons), and the "xemacs-winnt" list devolved into a forum for teaching newbies how to escape spaces in filenames, and the occasional flame from a Windows-oriented user who wanted XEmacs to change into a "modern GUI application" (like, say, Word, except for writing programs). It died a natural death (long before XEmacs development went dormant). N.B. The problem with modern GUI, of course, was that all the Windows- based developers who contributed any code to speak of liked the mouse- free pinky-welded-to-the-control-key UI (that's *why* they used Emacsen). This might not be such a stark contrast in Python, which doesn't pretend to be a UI. On the other hand, the Windows-oriented Python developers (several of whom are currently inactive for reasons unrelated to their preferred platform) have never seemed uncomfortable here. From solipsis at pitrou.net Sat Nov 15 11:03:55 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sat, 15 Nov 2014 11:03:55 +0100 Subject: [Python-Dev] OneGet provider for Python References: Message-ID: <20141115110355.0334af35@fsol> On Fri, 14 Nov 2014 18:12:59 -0600 Vincent Povirk wrote: > I have been following OneGet development very closely, and I have > volunteered to create a OneGet provider for installing Python > packages. Normally, I would keep quiet about this sort of thing until > I have actual code that works, but since no one else on the OneGet > seems as interested in doing it, and the Python community (if this > mailing list is anything to go by) doesn't know what to make of this > stuff, I figured I should try to open lines of communication before I > start essentially making decisions on this community's behalf. > > So, the main advantage of a Python provider is that you would be able > to do things like: > > Install-Package -Provider Python -Name Pygments > or > > Install-Package Pygments-2.0.1-py3-none-any.whl > > and OneGet would be able to bootstrap (if necessary) the Python > provider itself, Python 3.x, and pip, and use pip to install the .whl. > (I'm unclear at the moment on whether pip is necessary to install a > .whl, but currently it looks to be the best thing to lean on for the > job of installing/uninstalling things). > > My end goal is to be able to package a Python application such that an > end-user on Windows (who doesn't know anything about Python) can > easily install it, without either of us having to think about how all > the dependencies are going to get there. > > I think that the best approach for the moment is to lean heavily on > pip for installing/uninstalling things, while duplicating other tasks > (such as listing installed packages, querying information about > package files, searching PyPI) in C# so that they do not require a > Python environment. > > If anyone has questions or concerns about this, please let me know. > Keep in mind that I am not subscribed to the mailing list and will > have to be CC'd. If you don't get any answers here, I would suggest posting on distutils-sig: https://mail.python.org/mailman/listinfo/distutils-sig There's a gmane mirror as well: http://news.gmane.org/gmane.comp.python.distutils.devel/ Regards Antoine. From p.f.moore at gmail.com Sat Nov 15 11:10:14 2014 From: p.f.moore at gmail.com (Paul Moore) Date: Sat, 15 Nov 2014 10:10:14 +0000 Subject: [Python-Dev] OneGet provider for Python In-Reply-To: References: Message-ID: On 15 November 2014 00:12, Vincent Povirk wrote: > My end goal is to be able to package a Python application such that an > end-user on Windows (who doesn't know anything about Python) can > easily install it, without either of us having to think about how all > the dependencies are going to get there. That sounds awesome. > If anyone has questions or concerns about this, please let me know. > Keep in mind that I am not subscribed to the mailing list and will > have to be CC'd. You should probably discuss this on distutils-sig, as that is the main list for anything to do with packaging on Python. As Terry pointed out, you can monitor the lists without subscribing, but it would probably be better to subscribe, as people can tend to forget to cc the original author if a thread gets long. > If anyone has questions about OneGet generally, you should probably > ask them directly (see https://github.com/OneGet/oneget), as I am not > a definitive source of information on the project. Unless I'm misreading their install instructions, OneGet needs Windows 8.1 or Windows Server 2012, as it's based on WMF 5. That would exclude people using Windows 7 or earlier (or even the original Windows 8), which is a big proportion of Windows users. While that's not something you can do much about, it is something that's going to impact the goal of making it easy for Windows users to handle managing Python. > Incidentally, it would be really useful if python.org provided stable > url's that always redirected to the latest .msi installers, for > bootstrapping purposes. I'd prefer to not rely on chocolatey (or on > scraping the web site) for this. https://www.python.org/ftp/python/$ver/python-$ver.msi https://www.python.org/ftp/python/$ver/python-$ver.amd64.msi As far as I know these are stable, supported URLs. Although I don't think they are documented anywhere :-( Paul From njs at pobox.com Sat Nov 15 11:54:52 2014 From: njs at pobox.com (Nathaniel Smith) Date: Sat, 15 Nov 2014 10:54:52 +0000 Subject: [Python-Dev] OneGet provider for Python In-Reply-To: References: Message-ID: On 15 Nov 2014 10:10, "Paul Moore" wrote: > > > Incidentally, it would be really useful if python.org provided stable > > url's that always redirected to the latest .msi installers, for > > bootstrapping purposes. I'd prefer to not rely on chocolatey (or on > > scraping the web site) for this. > > https://www.python.org/ftp/python/$ver/python-$ver.msi > https://www.python.org/ftp/python/$ver/python-$ver.amd64.msi Right, but what's the URL for "the latest 2.7.x release" or "the latest 3.x.x release"? -n -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Sat Nov 15 12:14:29 2014 From: p.f.moore at gmail.com (Paul Moore) Date: Sat, 15 Nov 2014 11:14:29 +0000 Subject: [Python-Dev] OneGet provider for Python In-Reply-To: References: Message-ID: On 15 November 2014 10:54, Nathaniel Smith wrote: > On 15 Nov 2014 10:10, "Paul Moore" wrote: >> >> > Incidentally, it would be really useful if python.org provided stable >> > url's that always redirected to the latest .msi installers, for >> > bootstrapping purposes. I'd prefer to not rely on chocolatey (or on >> > scraping the web site) for this. >> >> https://www.python.org/ftp/python/$ver/python-$ver.msi >> https://www.python.org/ftp/python/$ver/python-$ver.amd64.msi > > Right, but what's the URL for "the latest 2.7.x release" or "the latest > 3.x.x release"? I don't know. As I said, it would be nice if these URLs (and "latest" ones as you mentioned) were documented. Paul From benjamin at python.org Sat Nov 15 16:17:17 2014 From: benjamin at python.org (Benjamin Peterson) Date: Sat, 15 Nov 2014 10:17:17 -0500 Subject: [Python-Dev] OneGet provider for Python In-Reply-To: References: Message-ID: <1416064637.3227068.191359373.4A7379F7@webmail.messagingengine.com> On Sat, Nov 15, 2014, at 05:54, Nathaniel Smith wrote: > On 15 Nov 2014 10:10, "Paul Moore" wrote: > > > > > Incidentally, it would be really useful if python.org provided stable > > > url's that always redirected to the latest .msi installers, for > > > bootstrapping purposes. I'd prefer to not rely on chocolatey (or on > > > scraping the web site) for this. > > > > https://www.python.org/ftp/python/$ver/python-$ver.msi > > https://www.python.org/ftp/python/$ver/python-$ver.amd64.msi > > Right, but what's the URL for "the latest 2.7.x release" or "the latest > 3.x.x release"? The website has an API you know. From ncoghlan at gmail.com Sat Nov 15 16:41:40 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 16 Nov 2014 01:41:40 +1000 Subject: [Python-Dev] OneGet provider for Python In-Reply-To: References: Message-ID: On 15 November 2014 20:10, Paul Moore wrote: > On 15 November 2014 00:12, Vincent Povirk wrote: > > My end goal is to be able to package a Python application such that an > > end-user on Windows (who doesn't know anything about Python) can > > easily install it, without either of us having to think about how all > > the dependencies are going to get there. > > That sounds awesome. > > > If anyone has questions or concerns about this, please let me know. > > Keep in mind that I am not subscribed to the mailing list and will > > have to be CC'd. > > You should probably discuss this on distutils-sig, as that is the main > list for anything to do with packaging on Python. As Terry pointed > out, you can monitor the lists without subscribing, but it would > probably be better to subscribe, as people can tend to forget to cc > the original author if a thread gets long. > It's an interesting cross-over - the pip/pypi integration is definitely distutils-sig's domain, managing the CPython MSI packages is more python-dev. However, distutils-sig has a naturally higher proportion of folks interested in the general problem of software distribution (whether with Python specific tools or platform native ones). It's also the home of packaging.python.org maintenance, and as the Python ecosystem provider for OneGet becomes usable, it should likely be covered somewhere in there (perhaps in https://packaging.python.org/en/latest/deployment.html#os-packaging-installers although that section is currently still just a stub) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Sat Nov 15 16:40:17 2014 From: p.f.moore at gmail.com (Paul Moore) Date: Sat, 15 Nov 2014 15:40:17 +0000 Subject: [Python-Dev] OneGet provider for Python In-Reply-To: <1416064637.3227068.191359373.4A7379F7@webmail.messagingengine.com> References: <1416064637.3227068.191359373.4A7379F7@webmail.messagingengine.com> Message-ID: On 15 November 2014 15:17, Benjamin Peterson wrote: > On Sat, Nov 15, 2014, at 05:54, Nathaniel Smith wrote: >> On 15 Nov 2014 10:10, "Paul Moore" wrote: >> > >> > > Incidentally, it would be really useful if python.org provided stable >> > > url's that always redirected to the latest .msi installers, for >> > > bootstrapping purposes. I'd prefer to not rely on chocolatey (or on >> > > scraping the web site) for this. >> > >> > https://www.python.org/ftp/python/$ver/python-$ver.msi >> > https://www.python.org/ftp/python/$ver/python-$ver.amd64.msi >> >> Right, but what's the URL for "the latest 2.7.x release" or "the latest >> 3.x.x release"? > > The website has an API you know. Um, no. Where can I find out about it? Paul From rosuav at gmail.com Sat Nov 15 16:59:11 2014 From: rosuav at gmail.com (Chris Angelico) Date: Sun, 16 Nov 2014 02:59:11 +1100 Subject: [Python-Dev] OneGet provider for Python In-Reply-To: References: <1416064637.3227068.191359373.4A7379F7@webmail.messagingengine.com> Message-ID: On Sun, Nov 16, 2014 at 2:40 AM, Paul Moore wrote: > On 15 November 2014 15:17, Benjamin Peterson wrote: >> On Sat, Nov 15, 2014, at 05:54, Nathaniel Smith wrote: >>> On 15 Nov 2014 10:10, "Paul Moore" wrote: >>> > >>> > > Incidentally, it would be really useful if python.org provided stable >>> > > url's that always redirected to the latest .msi installers, for >>> > > bootstrapping purposes. I'd prefer to not rely on chocolatey (or on >>> > > scraping the web site) for this. >>> > >>> > https://www.python.org/ftp/python/$ver/python-$ver.msi >>> > https://www.python.org/ftp/python/$ver/python-$ver.amd64.msi >>> >>> Right, but what's the URL for "the latest 2.7.x release" or "the latest >>> 3.x.x release"? >> >> The website has an API you know. > > Um, no. Where can I find out about it? I didn't know either, and I've been pointing people to the .msi installers periodically - just by manually hunting down the latest link and recommending it. If there's an easy way to define the two tokens Nathaniel described, I'd like to hear it too. ChrisA From rajshorya at gmail.com Sat Nov 15 09:01:29 2014 From: rajshorya at gmail.com (Shorya Raj) Date: Sat, 15 Nov 2014 21:01:29 +1300 Subject: [Python-Dev] Windows Dedicated Mailing List In-Reply-To: <87bno9vxdg.fsf@uwakimon.sk.tsukuba.ac.jp> References: <87bno9vxdg.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: Fair enough. Like I said, I wanted to see what the consensus was, and it seems to be shifting towards not making another list. On Sat, Nov 15, 2014 at 6:36 PM, Stephen J. Turnbull wrote: > Shorya Raj writes: > > > I think it may be prudent to consider adding a Windows specific > > mailing list regarding development on Windows, > > One possibly-relevant anecdote. We had such a list for XEmacs, where > the biggest contributor 1998-2005 was Windows-based. *He* preferred > communicating via the main developers' list (the reasons given by > others in this thread were among his reasons), and the "xemacs-winnt" > list devolved into a forum for teaching newbies how to escape spaces > in filenames, and the occasional flame from a Windows-oriented user > who wanted XEmacs to change into a "modern GUI application" (like, > say, Word, except for writing programs). It died a natural death > (long before XEmacs development went dormant). > > N.B. The problem with modern GUI, of course, was that all the Windows- > based developers who contributed any code to speak of liked the mouse- > free pinky-welded-to-the-control-key UI (that's *why* they used > Emacsen). This might not be such a stark contrast in Python, which > doesn't pretend to be a UI. > > On the other hand, the Windows-oriented Python developers (several of > whom are currently inactive for reasons unrelated to their preferred > platform) have never seemed uncomfortable here. > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From madewokherd at gmail.com Sat Nov 15 17:50:19 2014 From: madewokherd at gmail.com (Vincent Povirk) Date: Sat, 15 Nov 2014 10:50:19 -0600 Subject: [Python-Dev] OneGet provider for Python In-Reply-To: References: <1416064637.3227068.191359373.4A7379F7@webmail.messagingengine.com> Message-ID: >> The website has an API you know. > > Um, no. Where can I find out about it? Ooh, I didn't expect that. I couldn't find any documentation, but the source code of urls_api.py at https://github.com/python/pythondotorg/tree/master/pydotorg is certainly interesting. Also, distutils-sig does seem more appropriate, and I'm interested in packaging generally so I'll probably subscribe there. Sorry I missed it and ended up in the wrong place first. From tjreedy at udel.edu Sat Nov 15 20:38:28 2014 From: tjreedy at udel.edu (Terry Reedy) Date: Sat, 15 Nov 2014 14:38:28 -0500 Subject: [Python-Dev] Windows Dedicated Mailing List In-Reply-To: References: <87bno9vxdg.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: On 11/15/2014 3:01 AM, Shorya Raj wrote: > Fair enough. Like I said, I wanted to see what the consensus was, and it > seems to be shifting towards not making another list. Agreed. The traffic here is not high enough to need a split. Most OS specific issues that need a patch end up as a specific issue on the tracker with a limited subset of us as nosy. -- Terry Jan Reedy From victor.stinner at gmail.com Sat Nov 15 22:53:09 2014 From: victor.stinner at gmail.com (Victor Stinner) Date: Sat, 15 Nov 2014 22:53:09 +0100 Subject: [Python-Dev] OneGet provider for Python In-Reply-To: References: <1416064637.3227068.191359373.4A7379F7@webmail.messagingengine.com> Message-ID: > Also, distutils-sig does seem more appropriate, IMO we need a new mailing to discuss which mailing list is the most appropriate (which includes another new mailing list). Victor -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Sun Nov 16 01:02:33 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sun, 16 Nov 2014 01:02:33 +0100 Subject: [Python-Dev] OneGet provider for Python References: <1416064637.3227068.191359373.4A7379F7@webmail.messagingengine.com> Message-ID: <20141116010233.65b8c078@fsol> On Sat, 15 Nov 2014 22:53:09 +0100 Victor Stinner wrote: > > Also, distutils-sig does seem more appropriate, > > IMO we need a new mailing to discuss which mailing list is the most > appropriate (which includes another new mailing list). I suggest you first write a PEP, and find a BDFL-delegate. Regards Antoine. From stefano.borini at ferrara.linux.it Sun Nov 16 11:23:41 2014 From: stefano.borini at ferrara.linux.it (Stefano Borini) Date: Sun, 16 Nov 2014 11:23:41 +0100 Subject: [Python-Dev] Problem displaying the warning symbol Message-ID: <54687B2D.2000803@ferrara.linux.it> Hi, I report a finding (bug?) about the warning unicode symbol, as reported here http://stackoverflow.com/questions/26919799/cannot-make-warning-sign-visible-on-osx-terminal In brief, python fails to correctly encode \u26a0 (warning sign) to utf-8, instead writing a space. It has no problems with other symbols. I m using python 3.4.0. From rdmurray at bitdance.com Sun Nov 16 17:49:37 2014 From: rdmurray at bitdance.com (R. David Murray) Date: Sun, 16 Nov 2014 11:49:37 -0500 Subject: [Python-Dev] Problem displaying the warning symbol In-Reply-To: <54687B2D.2000803@ferrara.linux.it> References: <54687B2D.2000803@ferrara.linux.it> Message-ID: <20141116164938.01E8D250EEC@webabinitio.net> On Sun, 16 Nov 2014 11:23:41 +0100, Stefano Borini wrote: > I report a finding (bug?) about the warning unicode symbol, as reported here > > http://stackoverflow.com/questions/26919799/cannot-make-warning-sign-visible-on-osx-terminal Bugs should be reported to bugs.python.org. They just get lost here. I'm doubtful this is a Python problem though. More likely it is an issue with the curses library on OSX. --David From tjreedy at udel.edu Sun Nov 16 20:23:01 2014 From: tjreedy at udel.edu (Terry Reedy) Date: Sun, 16 Nov 2014 14:23:01 -0500 Subject: [Python-Dev] Problem displaying the warning symbol In-Reply-To: <20141116164938.01E8D250EEC@webabinitio.net> References: <54687B2D.2000803@ferrara.linux.it> <20141116164938.01E8D250EEC@webabinitio.net> Message-ID: On 11/16/2014 11:49 AM, R. David Murray wrote: > On Sun, 16 Nov 2014 11:23:41 +0100, Stefano Borini wrote: >> I report a finding (bug?) about the warning unicode symbol, as reported here >> >> http://stackoverflow.com/questions/26919799/cannot-make-warning-sign-visible-on-osx-terminal Stefano, the right place to send an enquiry about a bug is python-list, not py-dev. > Bugs should be reported to bugs.python.org. But only if one is fairly sure there is a bug. The tracker is not for 'is this a bug' questions. > I'm doubtful this is a Python problem though. More likely it is an > issue with the curses library on OSX. Warning signs display fine in Idle's tkinter Text widget with Lucida Console font, and python utf-8 encodes and decodes the char. >>> s='\u26a0' >>> s '?' # up-pointing triangle /_\ with ! inside >>> b=s.encode('utf-8') >>> b b'\xe2\x9a\xa0' >>> b.decode('utf-8') '?' Presuming that the bytes are correct, there is no bug here. -- Terry Jan Reedy From buck.2019 at gmail.com Sun Nov 16 23:23:21 2014 From: buck.2019 at gmail.com (Buck Evan) Date: Sun, 16 Nov 2014 14:23:21 -0800 Subject: [Python-Dev] understanding ubuntu's patches to cpython Message-ID: This particular patch caused virtualenv issue #118 and a subsequent workaround: http://bazaar.launchpad.net/~ubuntu-branches/ubuntu/trusty/python2.7/trusty/view/head:/debian/patches/distutils-install-layout.diff#L281 Although this patch sits on the trusty branch of cpython patckaging, in point of fact on that line i have `return os.path.join(get_config_var('LIBPL'), "Makefile")`. I see that other patches in the same project also touch this line, but none of them have this LIBPL code that I find in my actual installation. Can any of you all help me understand what's going on here? I'm trying to pinpoint exactly when/how the problematic code (.replace("/usr/local","/usr") went away. I'm hoping "doko" is on this mailing list. Of course feel free to advise if there's a more appropriate forum for this discussion. -------------- next part -------------- An HTML attachment was scrubbed... URL: From buck.2019 at gmail.com Sun Nov 16 23:26:39 2014 From: buck.2019 at gmail.com (Buck Evan) Date: Sun, 16 Nov 2014 14:26:39 -0800 Subject: [Python-Dev] py34 makes it harder to read all of a pty In-Reply-To: References: Message-ID: Note that this is a fully unbuffered read. On Wed, Nov 12, 2014 at 11:34 PM, Charles-Fran?ois Natali < cf.natali at gmail.com> wrote: > 2014-11-12 22:16 GMT+00:00 Buck Golemon : > > This is due to the fix for issue21090, which aimed to un-silence errors > > which previously went unheard. The fix is for me, as a user, to write a > loop > > that uses os.read and interpretes EIO as EOF. This is what I had hoped > > file.read() would do for me, however, and what it used to do in previous > > pythons. > > > There's no reason for read() to interpret EIO as EOF in the general > case: it was masked in previous versions because of a mere bug. The > behavior is now correct, although being able to retrieve the data read > so far in case of a buffered read could be useful. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefano.borini at ferrara.linux.it Sun Nov 16 23:27:56 2014 From: stefano.borini at ferrara.linux.it (Stefano Borini) Date: Sun, 16 Nov 2014 23:27:56 +0100 Subject: [Python-Dev] Problem displaying the warning symbol In-Reply-To: References: <54687B2D.2000803@ferrara.linux.it> <20141116164938.01E8D250EEC@webabinitio.net> Message-ID: <546924EC.6040803@ferrara.linux.it> On 11/16/14 8:23 PM, Terry Reedy wrote: > Stefano, the right place to send an enquiry about a bug is python-list, > not py-dev. > ok, sorry, but I wasn't even sure if it was a bug, and what was the root cause. >> I'm doubtful this is a Python problem though. More likely it is an >> issue with the curses library on OSX. Exactly. That's why I posted it here. I'll try to write it in C and see what happens. > Presuming that the bytes are correct, there is no bug here. The bytes are correct, but if there's a problem in the osx curses lib, maybe a workaround is possible. I honestly don't understand the root cause at this point. However, let's close the conversation here, I will try to investigate a bit (if time allows) and if I find something conclusive I will either submit a patch or an investigation report. Thanks for the feedback. From nad at acm.org Mon Nov 17 01:18:36 2014 From: nad at acm.org (Ned Deily) Date: Sun, 16 Nov 2014 16:18:36 -0800 Subject: [Python-Dev] understanding ubuntu's patches to cpython References: Message-ID: In article , Buck Evan wrote: > This particular patch caused virtualenv issue #118 and a subsequent > workaround: > http://bazaar.launchpad.net/~ubuntu-branches/ubuntu/trusty/python2.7/trusty/vi > ew/head:/debian/patches/distutils-install-layout.diff#L281 > roposed/view/head:/debian/patches/distutils-install-layout.diff#L281> > > Although this patch sits on the trusty branch of cpython patckaging, in > point of fact on that line i have `return > os.path.join(get_config_var('LIBPL'), "Makefile")`. I see that other > patches in the same project also touch this line, but none of them have > this LIBPL code that I find in my actual installation. > > Can any of you all help me understand what's going on here? I'm trying to > pinpoint exactly when/how the problematic code (.replace("/usr/local","/usr") > went away. > > I'm hoping "doko" is on this mailing list. Of course feel free to advise if > there's a more appropriate forum for this discussion. This isn't really the place to ask about Ubuntu- (or Debian-) specific downstream modifications to CPython. Yes, there are people here who contribute to CPython, Debian, and Ubuntu but I think the best place to bring up questions like this, based on the Ubuntu project page, might be on the Ubuntu Core Developers mailing list: http://packages.ubuntu.com/source/utopic/python-defaults https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel-discuss Perhaps someone more familiar with the Ubuntu dev process will have a better suggestion. -- Ned Deily, nad at acm.org From stefan.bucur at gmail.com Mon Nov 17 15:49:36 2014 From: stefan.bucur at gmail.com (Stefan Bucur) Date: Mon, 17 Nov 2014 14:49:36 +0000 Subject: [Python-Dev] Static checker for common Python programming errors Message-ID: I'm developing a Python static analysis tool that flags common programming errors in Python programs. The tool is meant to complement other tools like Pylint (which perform checks at lexical and syntactic level) by going deeper with the code analysis and keeping track of the possible control flow paths in the program (path-sensitive analysis). For instance, a path-sensitive analysis detects that the following snippet of code would raise an AttributeError exception: if object is None: # If the True branch is taken, we know the object is None object.doSomething() # ... so this statement would always fail I'm writing first to the Python developers themselves to ask, in their experience, what common pitfalls in the language & its standard library such a static checker should look for. For instance, here [1] is a list of static checks for the C++ language, as part of the Clang static analyzer project. My preliminary list of Python checks is quite rudimentary, but maybe could serve as a discussion starter: * Proper Unicode handling (for 2.x) - encode() is not called on str object - decode() is not called on unicode object * Check for integer division by zero * Check for None object dereferences Thanks a lot, Stefan Bucur [1] http://clang-analyzer.llvm.org/available_checks.html -------------- next part -------------- An HTML attachment was scrubbed... URL: From mark at hotpy.org Mon Nov 17 16:04:41 2014 From: mark at hotpy.org (Mark Shannon) Date: Mon, 17 Nov 2014 15:04:41 +0000 Subject: [Python-Dev] Static checker for common Python programming errors In-Reply-To: References: Message-ID: <546A0E89.1080205@hotpy.org> Hi, I think this might be a bit off-topic for this mailing list, code-quality at python.org is the place for discussing static analysis tools. Although if anyone does have any comments on any particular checks they would like, I would be interested as well. Cheers, Mark. On 17/11/14 14:49, Stefan Bucur wrote: > I'm developing a Python static analysis tool that flags common > programming errors in Python programs. The tool is meant to complement > other tools like Pylint (which perform checks at lexical and syntactic > level) by going deeper with the code analysis and keeping track of the > possible control flow paths in the program (path-sensitive analysis). > > For instance, a path-sensitive analysis detects that the following > snippet of code would raise an AttributeError exception: > > if object is None: # If the True branch is taken, we know the object is None > object.doSomething() # ... so this statement would always fail > > I'm writing first to the Python developers themselves to ask, in their > experience, what common pitfalls in the language & its standard library > such a static checker should look for. For instance, here [1] is a list > of static checks for the C++ language, as part of the Clang static > analyzer project. > > My preliminary list of Python checks is quite rudimentary, but maybe > could serve as a discussion starter: > > * Proper Unicode handling (for 2.x) > - encode() is not called on str object > - decode() is not called on unicode object > * Check for integer division by zero > * Check for None object dereferences > > Thanks a lot, > Stefan Bucur > > [1] http://clang-analyzer.llvm.org/available_checks.html > > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/mark%40hotpy.org > From stefan.bucur at gmail.com Mon Nov 17 17:58:18 2014 From: stefan.bucur at gmail.com (Stefan Bucur) Date: Mon, 17 Nov 2014 16:58:18 +0000 Subject: [Python-Dev] Static checker for common Python programming errors References: <546A0E89.1080205@hotpy.org> Message-ID: Mark, thank you for the pointer! I will re-send my message there. Should I include both mailing lists in a single thread if I end up receiving replies from both? Cheers, Stefan On Mon Nov 17 2014 at 4:04:45 PM Mark Shannon wrote: > Hi, > > I think this might be a bit off-topic for this mailing list, > code-quality at python.org is the place for discussing static analysis tools. > > Although if anyone does have any comments on any particular checks > they would like, I would be interested as well. > > Cheers, > Mark. > > > On 17/11/14 14:49, Stefan Bucur wrote: > > I'm developing a Python static analysis tool that flags common > > programming errors in Python programs. The tool is meant to complement > > other tools like Pylint (which perform checks at lexical and syntactic > > level) by going deeper with the code analysis and keeping track of the > > possible control flow paths in the program (path-sensitive analysis). > > > > For instance, a path-sensitive analysis detects that the following > > snippet of code would raise an AttributeError exception: > > > > if object is None: # If the True branch is taken, we know the object is > None > > object.doSomething() # ... so this statement would always fail > > > > I'm writing first to the Python developers themselves to ask, in their > > experience, what common pitfalls in the language & its standard library > > such a static checker should look for. For instance, here [1] is a list > > of static checks for the C++ language, as part of the Clang static > > analyzer project. > > > > My preliminary list of Python checks is quite rudimentary, but maybe > > could serve as a discussion starter: > > > > * Proper Unicode handling (for 2.x) > > - encode() is not called on str object > > - decode() is not called on unicode object > > * Check for integer division by zero > > * Check for None object dereferences > > > > Thanks a lot, > > Stefan Bucur > > > > [1] http://clang-analyzer.llvm.org/available_checks.html > > > > > > > > _______________________________________________ > > Python-Dev mailing list > > Python-Dev at python.org > > https://mail.python.org/mailman/listinfo/python-dev > > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ > mark%40hotpy.org > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Mon Nov 17 18:25:50 2014 From: guido at python.org (Guido van Rossum) Date: Mon, 17 Nov 2014 09:25:50 -0800 Subject: [Python-Dev] Static checker for common Python programming errors In-Reply-To: References: Message-ID: Also, I should mention mypy (mypy-lang.org), which is a much more ambitious project that uses type annotations. I am trying to find time to work on a PEP that standardizes type annotations to match mypy's syntax (with probably some improvements and caveats). It's too early to post the PEP draft but if you're designing a type checker or IDE that could use help from type annotations, email me. On Mon, Nov 17, 2014 at 6:49 AM, Stefan Bucur wrote: > I'm developing a Python static analysis tool that flags common programming > errors in Python programs. The tool is meant to complement other tools like > Pylint (which perform checks at lexical and syntactic level) by going > deeper with the code analysis and keeping track of the possible control > flow paths in the program (path-sensitive analysis). > > For instance, a path-sensitive analysis detects that the following snippet > of code would raise an AttributeError exception: > > if object is None: # If the True branch is taken, we know the object is > None > object.doSomething() # ... so this statement would always fail > > I'm writing first to the Python developers themselves to ask, in their > experience, what common pitfalls in the language & its standard library > such a static checker should look for. For instance, here [1] is a list of > static checks for the C++ language, as part of the Clang static analyzer > project. > > My preliminary list of Python checks is quite rudimentary, but maybe could > serve as a discussion starter: > > * Proper Unicode handling (for 2.x) > - encode() is not called on str object > - decode() is not called on unicode object > * Check for integer division by zero > * Check for None object dereferences > > Thanks a lot, > Stefan Bucur > > [1] http://clang-analyzer.llvm.org/available_checks.html > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/guido%40python.org > > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Mon Nov 17 18:23:29 2014 From: brett at python.org (Brett Cannon) Date: Mon, 17 Nov 2014 17:23:29 +0000 Subject: [Python-Dev] Static checker for common Python programming errors References: <546A0E89.1080205@hotpy.org> Message-ID: On Mon Nov 17 2014 at 12:06:15 PM Stefan Bucur wrote: > Mark, thank you for the pointer! I will re-send my message there. Should I > include both mailing lists in a single thread if I end up receiving replies > from both? No as cross-posting becomes just a nightmare of moderation when someone is not on both lists; please only post to a single mailing list. -Brett > > Cheers, > Stefan > > > On Mon Nov 17 2014 at 4:04:45 PM Mark Shannon wrote: > >> Hi, >> >> I think this might be a bit off-topic for this mailing list, >> code-quality at python.org is the place for discussing static analysis >> tools. >> >> Although if anyone does have any comments on any particular checks >> they would like, I would be interested as well. >> >> Cheers, >> Mark. >> >> >> On 17/11/14 14:49, Stefan Bucur wrote: >> > I'm developing a Python static analysis tool that flags common >> > programming errors in Python programs. The tool is meant to complement >> > other tools like Pylint (which perform checks at lexical and syntactic >> > level) by going deeper with the code analysis and keeping track of the >> > possible control flow paths in the program (path-sensitive analysis). >> > >> > For instance, a path-sensitive analysis detects that the following >> > snippet of code would raise an AttributeError exception: >> > >> > if object is None: # If the True branch is taken, we know the object is >> None >> > object.doSomething() # ... so this statement would always fail >> > >> > I'm writing first to the Python developers themselves to ask, in their >> > experience, what common pitfalls in the language & its standard library >> > such a static checker should look for. For instance, here [1] is a list >> > of static checks for the C++ language, as part of the Clang static >> > analyzer project. >> > >> > My preliminary list of Python checks is quite rudimentary, but maybe >> > could serve as a discussion starter: >> > >> > * Proper Unicode handling (for 2.x) >> > - encode() is not called on str object >> > - decode() is not called on unicode object >> > * Check for integer division by zero >> > * Check for None object dereferences >> > >> > Thanks a lot, >> > Stefan Bucur >> > >> > [1] http://clang-analyzer.llvm.org/available_checks.html >> > >> > >> > >> > _______________________________________________ >> > Python-Dev mailing list >> > Python-Dev at python.org >> > https://mail.python.org/mailman/listinfo/python-dev >> > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ >> mark%40hotpy.org >> > >> > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ > brett%40python.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Mon Nov 17 19:02:55 2014 From: p.f.moore at gmail.com (Paul Moore) Date: Mon, 17 Nov 2014 18:02:55 +0000 Subject: [Python-Dev] OneGet provider for Python In-Reply-To: References: <1416064637.3227068.191359373.4A7379F7@webmail.messagingengine.com> Message-ID: On 15 November 2014 15:40, Paul Moore wrote: > On 15 November 2014 15:17, Benjamin Peterson wrote: >> On Sat, Nov 15, 2014, at 05:54, Nathaniel Smith wrote: >>> On 15 Nov 2014 10:10, "Paul Moore" wrote: >>> > >>> > > Incidentally, it would be really useful if python.org provided stable >>> > > url's that always redirected to the latest .msi installers, for >>> > > bootstrapping purposes. I'd prefer to not rely on chocolatey (or on >>> > > scraping the web site) for this. >>> > >>> > https://www.python.org/ftp/python/$ver/python-$ver.msi >>> > https://www.python.org/ftp/python/$ver/python-$ver.amd64.msi >>> >>> Right, but what's the URL for "the latest 2.7.x release" or "the latest >>> 3.x.x release"? >> >> The website has an API you know. > > Um, no. Where can I find out about it? I don't know if this got lost in the other messages in this thread, but *is* there a stable URL for "the latest Python 3.4 MSI for Windows amd64" (or similar)? Paul From nad at acm.org Mon Nov 17 20:23:24 2014 From: nad at acm.org (Ned Deily) Date: Mon, 17 Nov 2014 11:23:24 -0800 Subject: [Python-Dev] OneGet provider for Python References: <1416064637.3227068.191359373.4A7379F7@webmail.messagingengine.com> Message-ID: In article , Paul Moore wrote: > I don't know if this got lost in the other messages in this thread, > but *is* there a stable URL for "the latest Python 3.4 MSI for Windows > amd64" (or similar)? AFAIK, no, there is no such stable URL that directly downloads the latest installer(s) for a platform; the closest is probably https://www.python.org/downloads/windows/ which would require scraping. I'm not sure we would want to encourage such a thing; we want downloaders to read the web page information for each release and make an informed choice. And the number of installer variants may change from release to release for a platform, as was recently the case with the OS X installers. For testing purposes, scraping the web pages or using the (undocumented, see the code base on github) website JSON API are probably the best options now. You could open an issue on the website github issue tracker. -- Ned Deily, nad at acm.org From francis.giraldeau at gmail.com Mon Nov 17 20:50:21 2014 From: francis.giraldeau at gmail.com (Francis Giraldeau) Date: Mon, 17 Nov 2014 14:50:21 -0500 Subject: [Python-Dev] Static checker for common Python programming errors In-Reply-To: References: Message-ID: If I may, there are prior work on JavaScript that may be worth investigating. Formal verification of dynamically typed software is a challenging endeavour, but it is very valuable to avoid errors at runtime, providing benefits from strongly type language without the rigidity. http://cs.au.dk/~amoeller/papers/tajs/ Good luck! Francis 2014-11-17 9:49 GMT-05:00 Stefan Bucur : > I'm developing a Python static analysis tool that flags common programming > errors in Python programs. The tool is meant to complement other tools like > Pylint (which perform checks at lexical and syntactic level) by going > deeper with the code analysis and keeping track of the possible control > flow paths in the program (path-sensitive analysis). > > For instance, a path-sensitive analysis detects that the following snippet > of code would raise an AttributeError exception: > > if object is None: # If the True branch is taken, we know the object is > None > object.doSomething() # ... so this statement would always fail > > I'm writing first to the Python developers themselves to ask, in their > experience, what common pitfalls in the language & its standard library > such a static checker should look for. For instance, here [1] is a list of > static checks for the C++ language, as part of the Clang static analyzer > project. > > My preliminary list of Python checks is quite rudimentary, but maybe could > serve as a discussion starter: > > * Proper Unicode handling (for 2.x) > - encode() is not called on str object > - decode() is not called on unicode object > * Check for integer division by zero > * Check for None object dereferences > > Thanks a lot, > Stefan Bucur > > [1] http://clang-analyzer.llvm.org/available_checks.html > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/francis.giraldeau%40gmail.com > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Mon Nov 17 21:03:25 2014 From: p.f.moore at gmail.com (Paul Moore) Date: Mon, 17 Nov 2014 20:03:25 +0000 Subject: [Python-Dev] OneGet provider for Python In-Reply-To: References: <1416064637.3227068.191359373.4A7379F7@webmail.messagingengine.com> Message-ID: On 17 November 2014 19:23, Ned Deily wrote: > Paul Moore wrote: >> I don't know if this got lost in the other messages in this thread, >> but *is* there a stable URL for "the latest Python 3.4 MSI for Windows >> amd64" (or similar)? > > AFAIK, no, there is no such stable URL that directly downloads the > latest installer(s) for a platform; the closest is probably > https://www.python.org/downloads/windows/ which would require scraping. > I'm not sure we would want to encourage such a thing; I'm happy enough with just the direct links to the exact versions (3.4.1 etc). I have to update my automatic build script whenever a new minor version comes out, which is a bit of a pain but as you say, having to deliberately decide to upgrade the version it installs is not a bad thing. Paul From francis.giraldeau at gmail.com Mon Nov 17 23:09:57 2014 From: francis.giraldeau at gmail.com (Francis Giraldeau) Date: Mon, 17 Nov 2014 17:09:57 -0500 Subject: [Python-Dev] Support for Linux perf Message-ID: Hi, The PEP-418 is about performance counters, but there is no mention of performance management unit (PMU) counters, such as cache misses and instruction counts. The Linux perf tool aims at recording these samples at the system level. I ran linux perf on CPython for profiling. The resulting callstack is inside libpython.so, mostly recursive calls to PyEval_EvalFrameEx(), because the tool works at the ELF level. Here is an example with a dummy program (linux-tools on Ubuntu 14.04): $ perf record python crunch.py $ perf report --stdio # Overhead Command Shared Object Symbol # ........ ....... .................. ................................ # 32.37% python python2.7 [.] PyEval_EvalFrameEx 13.70% python libm-2.19.so [.] __sin_avx 5.25% python python2.7 [.] binary_op1.5010 4.82% python python2.7 [.] PyObject_GetAttr While this may be insightful for the interpreter developers, it it not so for the average Python developer. The report should display Python code instead. It seems obvious, still I haven't found the feature for that. When a performance counter reaches a given value, a sample is recorded. The most basic sample only records a timestamps, thread ID and the program counter (%rip). In addition, all executable memory maps of libraries are recorded. For the callstack, frame pointers are traversed, but most of the time, they are optimized on x86, so there is a fall back to unwind, which requires saving register values and a chunk of the stack. The memory space of the process is reconstructed offline. CPython seems to allocates code and frames on mmap() pages. If the data is outside about 1k from the top of stack, it is not available offline in the trace. We need some way to reconstitute this memory space of the interpreter to resolve the symbols, probably by dumping the data on disk. In Java, there is a small HotSpot agent that spits out the symbols of JIT code: https://github.com/jrudolph/perf-map-agent The problem is that CPython does not JIT code, and executed code is the ELF library itself. The executed frames are parameters of functions of the interpreter. I don't think the same approach can be used (maybe this can be applied to PyPy?). I looked at how Python frames are handled in GDB (file cpython/Tools/gdb/libpython.py). A python frame is detected in Frame(gdbframe).is_evalframeex() by a C call to PyEval_EvalFrameEx(). However, the traceback accesses PyFrameObject on the heap (at least for f->f_back = 0xa57460), which is possible in GDB when the program is paused and the whole memory space is available, but is not recorded for offline use in perf. Here is an example of callstack from GDB: #0 PyEval_EvalFrameEx (f=Frame 0x7ffff7f1b060, for file crunch.py, line 7, in bar (num=466829), throwflag=0) at ../Python/ceval.c:1039 #1 0x0000000000527877 in fast_function (func=, pp_stack=0x7fffffffd280, n=1, na=1, nk=0) at ../Python/ceval.c:4106 #2 0x0000000000527582 in call_function (pp_stack=0x7fffffffd280, oparg=1) at ../Python/ceval.c:4041 We could add a kernel module that "knows" how to make samples of CPython, but it means python structures becomes sort of ABI, and kernel devs won't allow a python interpreter in kernel mode ;-). What we really want is f_code data and related objects: (gdb) print (void *)(f->f_code) $8 = (void *) 0x7ffff7e370f0 Maybe we could save these pages every time some code is loaded from the interpreter? (the memory range is about 1.7MB, but ) Anyway, I think we must change CPython to support tools such as perf. Any thoughts? Cheers, Francis -------------- next part -------------- An HTML attachment was scrubbed... URL: From tjreedy at udel.edu Tue Nov 18 02:21:13 2014 From: tjreedy at udel.edu (Terry Reedy) Date: Mon, 17 Nov 2014 20:21:13 -0500 Subject: [Python-Dev] Static checker for common Python programming errors In-Reply-To: References: Message-ID: On 11/17/2014 9:49 AM, Stefan Bucur wrote: > I'm developing a Python static analysis tool that flags common > programming errors in Python programs. The tool is meant to complement > other tools like Pylint (which perform checks at lexical and syntactic > level) by going deeper with the code analysis and keeping track of the > possible control flow paths in the program (path-sensitive analysis). > > For instance, a path-sensitive analysis detects that the following > snippet of code would raise an AttributeError exception: > > if object is None: # If the True branch is taken, we know the object is None > object.doSomething() # ... so this statement would always fail > > I'm writing first to the Python developers themselves to ask, in their > experience, what common pitfalls in the language & its standard library > such a static checker should look for. For instance, here [1] is a list > of static checks for the C++ language, as part of the Clang static > analyzer project. You could also a) ask on python-list (new thread), or scan python questions on StackOverflow. Todays's example: "Why does my function return None?" Because there is no return statement. Perhaps current checkers can note that, but what about if some branches have a return and others do not? That is a likely bug. -- Terry Jan Reedy From python at mrabarnett.plus.com Tue Nov 18 03:22:38 2014 From: python at mrabarnett.plus.com (MRAB) Date: Tue, 18 Nov 2014 02:22:38 +0000 Subject: [Python-Dev] Static checker for common Python programming errors In-Reply-To: References: Message-ID: <546AAD6E.2000907@mrabarnett.plus.com> On 2014-11-18 01:21, Terry Reedy wrote: > On 11/17/2014 9:49 AM, Stefan Bucur wrote: >> I'm developing a Python static analysis tool that flags common >> programming errors in Python programs. The tool is meant to complement >> other tools like Pylint (which perform checks at lexical and syntactic >> level) by going deeper with the code analysis and keeping track of the >> possible control flow paths in the program (path-sensitive analysis). >> >> For instance, a path-sensitive analysis detects that the following >> snippet of code would raise an AttributeError exception: >> >> if object is None: # If the True branch is taken, we know the object is None >> object.doSomething() # ... so this statement would always fail >> >> I'm writing first to the Python developers themselves to ask, in their >> experience, what common pitfalls in the language & its standard library >> such a static checker should look for. For instance, here [1] is a list >> of static checks for the C++ language, as part of the Clang static >> analyzer project. > > You could also a) ask on python-list (new thread), or scan python > questions on StackOverflow. Todays's example: "Why does my function > return None?" Because there is no return statement. Perhaps current > checkers can note that, but what about if some branches have a return > and others do not? That is a likely bug. > Mutable default parameters comes up occasionally. From stefan.bucur at gmail.com Tue Nov 18 12:01:19 2014 From: stefan.bucur at gmail.com (Stefan Bucur) Date: Tue, 18 Nov 2014 11:01:19 +0000 Subject: [Python-Dev] Static checker for common Python programming errors References: Message-ID: Thanks for the pointer! There seem indeed to be more formal analysis tools for JavaScript than for Python (e.g., the most recent one for JS I know of is the Jalangi framework [1]). I assume the main reason is that JavaScript is standardized and somewhat simpler, so it's easier to construct formal specs for all language features than it is for Python, which is also evolving faster and relies on a lot of hard-to-model native functionality. That's why I'm planning to reuse as much as possible the "implicit specs" of the interpreter implementation, instead of re-stating them in an explicit model. We already have an execution engine that uses the interpreter to automatically explore multiple paths through a piece of Python code (you can read here [2] the academic paper, with case studies for Python and Lua). In turn, we could use that engine to discover paths, while checking program properties along each path. Guido's suggestion for a type checker raises some interesting applications of this multi-path analysis. For instance, we could examine the type of the objects assigned to a static variable across all discovered execution paths and determine its consistency. This analysis could either start with no type annotations and output suggested types, or take existing annotations and check them against the actual types. Thanks again, Stefan [1] https://github.com/SRA-SiliconValley/jalangi [2] http://dslab.epfl.ch/pubs/chef.pdf On Mon Nov 17 2014 at 8:50:21 PM Francis Giraldeau < francis.giraldeau at gmail.com> wrote: > If I may, there are prior work on JavaScript that may be worth > investigating. Formal verification of dynamically typed software is a > challenging endeavour, but it is very valuable to avoid errors at runtime, > providing benefits from strongly type language without the rigidity. > > http://cs.au.dk/~amoeller/papers/tajs/ > > Good luck! > > Francis > > 2014-11-17 9:49 GMT-05:00 Stefan Bucur : > >> I'm developing a Python static analysis tool that flags common >> programming errors in Python programs. The tool is meant to complement >> other tools like Pylint (which perform checks at lexical and syntactic >> level) by going deeper with the code analysis and keeping track of the >> possible control flow paths in the program (path-sensitive analysis). >> >> For instance, a path-sensitive analysis detects that the following >> snippet of code would raise an AttributeError exception: >> >> if object is None: # If the True branch is taken, we know the object is >> None >> object.doSomething() # ... so this statement would always fail >> >> I'm writing first to the Python developers themselves to ask, in their >> experience, what common pitfalls in the language & its standard library >> such a static checker should look for. For instance, here [1] is a list of >> static checks for the C++ language, as part of the Clang static analyzer >> project. >> >> My preliminary list of Python checks is quite rudimentary, but maybe >> could serve as a discussion starter: >> >> * Proper Unicode handling (for 2.x) >> - encode() is not called on str object >> - decode() is not called on unicode object >> * Check for integer division by zero >> * Check for None object dereferences >> >> Thanks a lot, >> Stefan Bucur >> >> [1] http://clang-analyzer.llvm.org/available_checks.html >> >> >> _______________________________________________ >> Python-Dev mailing list >> Python-Dev at python.org >> https://mail.python.org/mailman/listinfo/python-dev >> > Unsubscribe: >> https://mail.python.org/mailman/options/python-dev/francis.giraldeau%40gmail.com >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From doko at ubuntu.com Tue Nov 18 11:56:20 2014 From: doko at ubuntu.com (Matthias Klose) Date: Tue, 18 Nov 2014 11:56:20 +0100 Subject: [Python-Dev] understanding ubuntu's patches to cpython In-Reply-To: References: Message-ID: <546B25D4.5070109@ubuntu.com> On 11/16/2014 11:23 PM, Buck Evan wrote: > This particular patch caused virtualenv issue #118 and a subsequent > workaround: > http://bazaar.launchpad.net/~ubuntu-branches/ubuntu/trusty/python2.7/trusty/view/head:/debian/patches/distutils-install-layout.diff#L281 > > > Although this patch sits on the trusty branch of cpython patckaging, in > point of fact on that line i have `return > os.path.join(get_config_var('LIBPL'), "Makefile")`. I see that other > patches in the same project also touch this line, but none of them have > this LIBPL code that I find in my actual installation. I'm not sure what issue you have here. Indeed, as Ned suggested, best thing would be to open a bug report in Launchpad [1]. > Can any of you all help me understand what's going on here? I'm trying to > pinpoint exactly when/how the problematic code (.replace("/usr/local","/usr") > went away. See the README.Debian. In the past people installed packages into the system path using "sudo python setup.py --install", and breaking applications shipped with Ubuntu. This avoids accidental installation of such packages. Unfortunately you can do this again with pip, because pip acting on the system python silently overwrites system installed python modules. Matthias [1] https://bugs.launchpad.net/ubuntu/+source/python2.7/+filebug From guido at python.org Tue Nov 18 17:05:53 2014 From: guido at python.org (Guido van Rossum) Date: Tue, 18 Nov 2014 08:05:53 -0800 Subject: [Python-Dev] Static checker for common Python programming errors In-Reply-To: References: Message-ID: Please also check python-static-type-checking at googlegroups.com. On Nov 18, 2014 3:06 AM, "Stefan Bucur" wrote: > Thanks for the pointer! There seem indeed to be more formal analysis tools > for JavaScript than for Python (e.g., the most recent one for JS I know of > is the Jalangi framework [1]). I assume the main reason is that JavaScript > is standardized and somewhat simpler, so it's easier to construct formal > specs for all language features than it is for Python, which is also > evolving faster and relies on a lot of hard-to-model native functionality. > > That's why I'm planning to reuse as much as possible the "implicit specs" > of the interpreter implementation, instead of re-stating them in an > explicit model. > > We already have an execution engine that uses the interpreter to > automatically explore multiple paths through a piece of Python code (you > can read here [2] the academic paper, with case studies for Python and > Lua). In turn, we could use that engine to discover paths, while checking > program properties along each path. > > Guido's suggestion for a type checker raises some interesting applications > of this multi-path analysis. For instance, we could examine the type of the > objects assigned to a static variable across all discovered execution paths > and determine its consistency. This analysis could either start with no > type annotations and output suggested types, or take existing annotations > and check them against the actual types. > > Thanks again, > Stefan > > [1] https://github.com/SRA-SiliconValley/jalangi > [2] http://dslab.epfl.ch/pubs/chef.pdf > > On Mon Nov 17 2014 at 8:50:21 PM Francis Giraldeau < > francis.giraldeau at gmail.com> wrote: > >> If I may, there are prior work on JavaScript that may be worth >> investigating. Formal verification of dynamically typed software is a >> challenging endeavour, but it is very valuable to avoid errors at runtime, >> providing benefits from strongly type language without the rigidity. >> >> http://cs.au.dk/~amoeller/papers/tajs/ >> >> Good luck! >> >> Francis >> >> 2014-11-17 9:49 GMT-05:00 Stefan Bucur : >> >>> I'm developing a Python static analysis tool that flags common >>> programming errors in Python programs. The tool is meant to complement >>> other tools like Pylint (which perform checks at lexical and syntactic >>> level) by going deeper with the code analysis and keeping track of the >>> possible control flow paths in the program (path-sensitive analysis). >>> >>> For instance, a path-sensitive analysis detects that the following >>> snippet of code would raise an AttributeError exception: >>> >>> if object is None: # If the True branch is taken, we know the object is >>> None >>> object.doSomething() # ... so this statement would always fail >>> >>> I'm writing first to the Python developers themselves to ask, in their >>> experience, what common pitfalls in the language & its standard library >>> such a static checker should look for. For instance, here [1] is a list of >>> static checks for the C++ language, as part of the Clang static analyzer >>> project. >>> >>> My preliminary list of Python checks is quite rudimentary, but maybe >>> could serve as a discussion starter: >>> >>> * Proper Unicode handling (for 2.x) >>> - encode() is not called on str object >>> - decode() is not called on unicode object >>> * Check for integer division by zero >>> * Check for None object dereferences >>> >>> Thanks a lot, >>> Stefan Bucur >>> >>> [1] http://clang-analyzer.llvm.org/available_checks.html >>> >>> >>> _______________________________________________ >>> Python-Dev mailing list >>> Python-Dev at python.org >>> https://mail.python.org/mailman/listinfo/python-dev >>> >> Unsubscribe: >>> https://mail.python.org/mailman/options/python-dev/francis.giraldeau%40gmail.com >>> >>> >> > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/guido%40python.org > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From buck.2019 at gmail.com Tue Nov 18 18:10:18 2014 From: buck.2019 at gmail.com (Buck Evan) Date: Tue, 18 Nov 2014 09:10:18 -0800 Subject: [Python-Dev] understanding ubuntu's patches to cpython In-Reply-To: <546B25D4.5070109@ubuntu.com> References: <546B25D4.5070109@ubuntu.com> Message-ID: I feel that you've misunderstood my request entirely. I'm not trying to report an issue, just requesting help understanding how the resulting package relates to version-controlled source. It's possible that I have the wrong source entirely. Your comments seem to indicate that you believe I've clobbered my installation with something other than the standard ubuntu package. I attempt to show otherwise below. I'm pretty confident that the source on disk ('LIBPL') represents the package that all trusty users have. My aim is to track down what (version-controlled) change caused this function to change between precise and trusty. $ ls -l /usr/lib/python2.7/sysconfig.py -rw-r--r-- 1 root root 25048 Mar 22 2014 /usr/lib/python2.7/sysconfig.py $ sudo touch !$ sudo touch /usr/lib/python2.7/sysconfig.py $ ls -l /usr/lib/python2.7/sysconfig.py -rw-r--r-- 1 root root 25048 Nov 18 08:57 /usr/lib/python2.7/sysconfig.py $ sudo apt-get install --reinstall libpython2.7-minimal Preparing to unpack .../libpython2.7-minimal_2.7.6-8_amd64.deb ... Unpacking libpython2.7-minimal:amd64 (2.7.6-8) over (2.7.6-8) ... Setting up libpython2.7-minimal:amd64 (2.7.6-8) ... $ ls -l /usr/lib/python2.7/sysconfig.py -rw-r--r-- 1 root root 25048 Mar 22 2014 /usr/lib/python2.7/sysconfig.py $ grep -A3 def\ _get_makefile_filename /usr/lib/python2.7/sysconfig.py def _get_makefile_filename(): if _PYTHON_BUILD: return os.path.join(_PROJECT_BASE, "Makefile") return os.path.join(get_config_var('LIBPL'), "Makefile") $ apt-cache show libpython2.7-dev Package: libpython2.7-dev Priority: optional Section: libdevel Installed-Size: 32956 Maintainer: Ubuntu Developers Original-Maintainer: Matthias Klose Architecture: amd64 Source: python2.7 Version: 2.7.6-8 Replaces: python2.7 (<< 2.7-3), python2.7-dev (<< 2.7.3-10), python2.7-minimal (<< 2.7.3-10) Depends: libpython2.7-stdlib (= 2.7.6-8), libpython2.7 (= 2.7.6-8), libexpat1-dev Pre-Depends: multiarch-support Recommends: libc6-dev | libc-dev Filename: pool/main/p/python2.7/libpython2.7-dev_2.7.6-8_amd64.deb Size: 22005790 MD5sum: 6b088fe5ee9bdb69296447a0b51f926e SHA1: 6356820a6b7fce13e1539dbe88b4292a0c8f2a8d SHA256: 646170fe0b0976d38252e474a8ac4ef5d52af93158b5a489da92c2225fbd1640 Description-en: Header files and a static library for Python (v2.7) Header files, a static library and development tools for building Python (v2.7) modules, extending the Python interpreter or embedding Python (v2.7) in applications. . Maintainers of Python packages should read README.maintainers. . This package contains development files. It is normally not used on it's[sic] own, but as a dependency of python2.7-dev. Description-md5: 2ef5e0390998d9ab1872bc0f6b694ff7 Multi-Arch: same Bugs: https://bugs.launchpad.net/ubuntu/+filebug Origin: Ubuntu Supported: 5y On Tue, Nov 18, 2014 at 2:56 AM, Matthias Klose wrote: > On 11/16/2014 11:23 PM, Buck Evan wrote: > >> This particular patch caused virtualenv issue #118 and a subsequent >> workaround: >> http://bazaar.launchpad.net/~ubuntu-branches/ubuntu/trusty/ >> python2.7/trusty/view/head:/debian/patches/distutils- >> install-layout.diff#L281 >> > python2.7/trusty-proposed/view/head:/debian/patches/ >> distutils-install-layout.diff#L281> >> >> Although this patch sits on the trusty branch of cpython patckaging, in >> point of fact on that line i have `return >> os.path.join(get_config_var('LIBPL'), "Makefile")`. I see that other >> patches in the same project also touch this line, but none of them have >> this LIBPL code that I find in my actual installation. >> > > I'm not sure what issue you have here. Indeed, as Ned suggested, best > thing would be to open a bug report in Launchpad [1]. > > Can any of you all help me understand what's going on here? I'm trying to >> pinpoint exactly when/how the problematic code >> (.replace("/usr/local","/usr") >> went away. >> > > See the README.Debian. In the past people installed packages into the > system path using "sudo python setup.py --install", and breaking > applications shipped with Ubuntu. This avoids accidental installation of > such packages. Unfortunately you can do this again with pip, because pip > acting on the system python silently overwrites system installed python > modules. > > Matthias > > [1] https://bugs.launchpad.net/ubuntu/+source/python2.7/+filebug > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Wed Nov 19 21:10:09 2014 From: guido at python.org (Guido van Rossum) Date: Wed, 19 Nov 2014 12:10:09 -0800 Subject: [Python-Dev] PEP 479: Change StopIteration handling inside generators Message-ID: There's a new PEP proposing to change how to treat StopIteration bubbling up out of a generator frame (not caused by a return from the frame). The proposal is to replace such a StopIteration with a RuntimeError (chained to the original StopIteration), so that only *returning* from a generator (or falling off the end) causes the iteration to terminate. The proposal unifies the behavior of list comprehensions and generator expressions along the lines I had originally in mind when they were introduced. It renders useless/illegal certain hacks that have crept into some folks' arsenal of obfuscated Python tools. In Python 3.5 the proposed change is conditional on: from __future__ import replace_stopiteration_in_generators This would affect all generators (including generator expressions) compiled under its influence. The feature would become standard in Python 3.6 or 3.7. The PEP is here: https://www.python.org/dev/peps/pep-0479/ To avoid a lot of requests for clarification you may also want to read up on the python-ideas discussion, e.g. here: https://groups.google.com/forum/#!topic/python-ideas/yJi1gRot9yY I am leaning towards approving this PEP, but not until we've had a review here at python-dev. I would like to thank Chris Angelico for writing the original PEP draft. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From benjamin at python.org Wed Nov 19 21:15:27 2014 From: benjamin at python.org (Benjamin Peterson) Date: Wed, 19 Nov 2014 15:15:27 -0500 Subject: [Python-Dev] PEP 479: Change StopIteration handling inside generators In-Reply-To: References: Message-ID: <1416428127.707439.193050285.1F5F417E@webmail.messagingengine.com> On Wed, Nov 19, 2014, at 15:10, Guido van Rossum wrote: > There's a new PEP proposing to change how to treat StopIteration bubbling > up out of a generator frame (not caused by a return from the frame). The > proposal is to replace such a StopIteration with a RuntimeError (chained > to > the original StopIteration), so that only *returning* from a generator > (or > falling off the end) causes the iteration to terminate. > > The proposal unifies the behavior of list comprehensions and generator > expressions along the lines I had originally in mind when they were > introduced. It renders useless/illegal certain hacks that have crept into > some folks' arsenal of obfuscated Python tools. > > In Python 3.5 the proposed change is conditional on: > > from __future__ import replace_stopiteration_in_generators Drive-by comment: This seems like a terribly awkward name. Could a shorter and sweeter name not be found? From python at mrabarnett.plus.com Wed Nov 19 21:48:39 2014 From: python at mrabarnett.plus.com (MRAB) Date: Wed, 19 Nov 2014 20:48:39 +0000 Subject: [Python-Dev] PEP 479: Change StopIteration handling inside generators In-Reply-To: References: Message-ID: <546D0227.1030608@mrabarnett.plus.com> On 2014-11-19 20:10, Guido van Rossum wrote: > There's a new PEP proposing to change how to treat StopIteration > bubbling up out of a generator frame (not caused by a return from > the frame). The proposal is to replace such a StopIteration with a > RuntimeError (chained to the original StopIteration), so that only > *returning* from a generator (or falling off the end) causes the > iteration to terminate. > The PEP says """any generator that depends on an implicitly-raised StopIteration to terminate it will have to be rewritten to either catch that exception or use a for-loop""" Shouldn't that be "... explicitly-raised ...", because returning raises StopIteration implicitly? ("raise StopIteration" is explicit) From rosuav at gmail.com Wed Nov 19 22:29:49 2014 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 20 Nov 2014 08:29:49 +1100 Subject: [Python-Dev] PEP 479: Change StopIteration handling inside generators In-Reply-To: <546D0227.1030608@mrabarnett.plus.com> References: <546D0227.1030608@mrabarnett.plus.com> Message-ID: On Thu, Nov 20, 2014 at 7:48 AM, MRAB wrote: > The PEP says """any generator that depends on an implicitly-raised > StopIteration to terminate it will have to be rewritten to either catch > that exception or use a for-loop""" > > Shouldn't that be "... explicitly-raised ...", because returning raises > StopIteration implicitly? ("raise StopIteration" is explicit) The point here is primarily about some other function (maybe a next(iter), or maybe something else entirely) raising StopIteration. (If it explicitly raises StopIteration right there in the generator, it can be trivially converted into a return statement, anyway.) The return statement is an explicit indication that the generator should now return; permitting a StopIteration to bubble up through and out is the implicit option; but the 'plicitness' isn't necessarily obvious. ChrisA From rosuav at gmail.com Wed Nov 19 22:44:33 2014 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 20 Nov 2014 08:44:33 +1100 Subject: [Python-Dev] [Python-ideas] PEP 479: Change StopIteration handling inside generators In-Reply-To: References: <546A3DEE.5040206@stoneleaf.us> <20141117235641.GW2748@ando.pearwood.info> Message-ID: On Thu, Nov 20, 2014 at 3:45 AM, Nick Coghlan wrote: > The part I found most compelling was when you pointed out that in the > special method implementations, the normal return path was always spelled > with "return", while the "value missing" result was indicated with a special > kind of exception (StopIteration, AttributeError, IndexError or KeyError), > and then any other exception was consider unexpected. > > Generators add the third notion of being able to suspend execution via > "yield", which then left them with two different ways of spelling > termination inside the frame: "return" OR "raise StopIteration". The second > spelling ("raise StopIteration") is then inherently surprising, as it's > entirely redundant, *except* in that it allows you to effectively have a > "hidden return" in a generator frame that can't be done anywhere else. (The above was said on -ideas, but discussion is now moving to -dev, so I hope it's okay to send the response there.) Added a paragraph to the PEP draft: https://github.com/Rosuav/GenStopIter/commit/695961 Should today's date be added to the Post-History? ChrisA From ncoghlan at gmail.com Thu Nov 20 10:35:32 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 20 Nov 2014 19:35:32 +1000 Subject: [Python-Dev] PEP 479: Change StopIteration handling inside generators In-Reply-To: <546D0227.1030608@mrabarnett.plus.com> References: <546D0227.1030608@mrabarnett.plus.com> Message-ID: On 20 November 2014 06:48, MRAB wrote: > On 2014-11-19 20:10, Guido van Rossum wrote: > >> There's a new PEP proposing to change how to treat StopIteration >> bubbling up out of a generator frame (not caused by a return from >> the frame). The proposal is to replace such a StopIteration with a >> RuntimeError (chained to the original StopIteration), so that only >> *returning* from a generator (or falling off the end) causes the >> iteration to terminate. >> >> The PEP says """any generator that depends on an implicitly-raised > StopIteration to terminate it will have to be rewritten to either catch > that exception or use a for-loop""" > > Shouldn't that be "... explicitly-raised ...", because returning raises > StopIteration implicitly? ("raise StopIteration" is explicit) The ways a generator can currently be terminated: return [value] raise StopIteration[([value])] The first case is unchanged, and (as Chris noted), the second case can be trivially converted to the first case. It's the third implicit case which would no longer be allowed, which means no more "or stop()" tricks to get generator expressions to terminate early. (If folks wanted to restore that capability, they'd instead need to propose new syntax that works the same way in both comprehensions and generator expressions, rather than relying on the lazy evaluation of generator expressions) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Thu Nov 20 11:05:27 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 20 Nov 2014 20:05:27 +1000 Subject: [Python-Dev] PEP 479: Change StopIteration handling inside generators In-Reply-To: <1416428127.707439.193050285.1F5F417E@webmail.messagingengine.com> References: <1416428127.707439.193050285.1F5F417E@webmail.messagingengine.com> Message-ID: On 20 November 2014 06:15, Benjamin Peterson wrote: > > On Wed, Nov 19, 2014, at 15:10, Guido van Rossum wrote: > > There's a new PEP proposing to change how to treat StopIteration bubbling > > up out of a generator frame (not caused by a return from the frame). The > > proposal is to replace such a StopIteration with a RuntimeError (chained > > to > > the original StopIteration), so that only *returning* from a generator > > (or > > falling off the end) causes the iteration to terminate. > > > > The proposal unifies the behavior of list comprehensions and generator > > expressions along the lines I had originally in mind when they were > > introduced. It renders useless/illegal certain hacks that have crept into > > some folks' arsenal of obfuscated Python tools. > > > > In Python 3.5 the proposed change is conditional on: > > > > from __future__ import replace_stopiteration_in_generators > > Drive-by comment: This seems like a terribly awkward name. Could a > shorter and sweeter name not be found? > I think my suggestion was something like "from __future__ import generator_return". I saw that style as somewhat similar to "from __future__ import division" - it just tells you what the change affects (in this case, returning from generators), while requiring folks to look up the documentation to find out the exact details of the old behaviour and the new behaviour. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Thu Nov 20 20:36:54 2014 From: guido at python.org (Guido van Rossum) Date: Thu, 20 Nov 2014 11:36:54 -0800 Subject: [Python-Dev] PEP 479: Change StopIteration handling inside generators In-Reply-To: References: <1416428127.707439.193050285.1F5F417E@webmail.messagingengine.com> Message-ID: I've made some updates to the PEP: - added 19-Nov-2014 to Post-History - removed "implicitly-raised" from the abstract - changed the __future__ thing to generator_return - added a clarifying paragraph that Chris added to his own draft - added a link to http://bugs.python.org/issue22906 which has a proof-of-concept patch There's still a lively discussion on python-ideas; Steven D'Aprano has dug up quite a bit of evidence that StopIteration is used quite a bit in ways that will break under the new behavior, and there also seems to be quite a bit of third-party information that recommends StopIteration over return to terminate a generator early. However I don't see much evidence that the current behavior is *better* than the proposal -- I see the current behavior as a definite wart, and I have definitely seen people struggle to debug silent early loop termination due to an "escaped" StopIteration. That said, I think for most people the change won't matter, some people will have to apply one of a few simple fixes, and a rare few will have to rewrite their code in a non-trivial way (sometimes this will affect "clever" libraries). I wonder if the PEP needs a better transition plan, e.g. - right now, start an education campaign - with Python 3.5, introduce "from __future__ import generator_return", and silent deprecation warnings - with Python 3.6, start issuing non-silent deprecation warnings - with Python 3.7, make the new behavior the default (subject to some kind of review) It would also be useful if we could extend the PEP with some examples of the various categories of fixes that can be applied easily, e.g. a few examples of "raise StopIteration" directly in a generator that can be replaced with "return" (or omitted, if it's at the end); a few examples of situations where "yield from" can supply an elegant fix (and an alternative for code that needs to be backward compatible with Python 3.2 or 2.7); and finally (to be honest) an example of code that will require being made more complicated. Oh, and it would also be nice if the PEP included some suggested words that 3rd party educators can use to explain the relationship between StopIteration and generators in a healthier way (preferably a way that also applies to older versions). Chris, are you up to drafting these additions? On Thu, Nov 20, 2014 at 2:05 AM, Nick Coghlan wrote: > On 20 November 2014 06:15, Benjamin Peterson wrote: > >> >> On Wed, Nov 19, 2014, at 15:10, Guido van Rossum wrote: >> > There's a new PEP proposing to change how to treat StopIteration >> bubbling >> > up out of a generator frame (not caused by a return from the frame). The >> > proposal is to replace such a StopIteration with a RuntimeError (chained >> > to >> > the original StopIteration), so that only *returning* from a generator >> > (or >> > falling off the end) causes the iteration to terminate. >> > >> > The proposal unifies the behavior of list comprehensions and generator >> > expressions along the lines I had originally in mind when they were >> > introduced. It renders useless/illegal certain hacks that have crept >> into >> > some folks' arsenal of obfuscated Python tools. >> > >> > In Python 3.5 the proposed change is conditional on: >> > >> > from __future__ import replace_stopiteration_in_generators >> >> Drive-by comment: This seems like a terribly awkward name. Could a >> shorter and sweeter name not be found? >> > > I think my suggestion was something like "from __future__ import > generator_return". > > I saw that style as somewhat similar to "from __future__ import division" > - it just tells you what the change affects (in this case, returning from > generators), while requiring folks to look up the documentation to find out > the exact details of the old behaviour and the new behaviour. > > Cheers, > Nick. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Thu Nov 20 20:58:43 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 20 Nov 2014 20:58:43 +0100 Subject: [Python-Dev] PEP 479: Change StopIteration handling inside generators References: <1416428127.707439.193050285.1F5F417E@webmail.messagingengine.com> Message-ID: <20141120205843.0c59c6d2@fsol> On Thu, 20 Nov 2014 11:36:54 -0800 Guido van Rossum wrote: > I've made some updates to the PEP: > > - added 19-Nov-2014 to Post-History > - removed "implicitly-raised" from the abstract > - changed the __future__ thing to generator_return To me "generator_return" sounds like the addition to generator syntax allowing for return statements (which was done as part of the "yield from" PEP). How about "generate_escape"? Regards Antoine. From storchaka at gmail.com Thu Nov 20 21:13:48 2014 From: storchaka at gmail.com (Serhiy Storchaka) Date: Thu, 20 Nov 2014 22:13:48 +0200 Subject: [Python-Dev] PEP 479: Change StopIteration handling inside generators In-Reply-To: <20141120205843.0c59c6d2@fsol> References: <1416428127.707439.193050285.1F5F417E@webmail.messagingengine.com> <20141120205843.0c59c6d2@fsol> Message-ID: On 20.11.14 21:58, Antoine Pitrou wrote: > To me "generator_return" sounds like the addition to generator syntax > allowing for return statements (which was done as part of the "yield > from" PEP). How about "generate_escape"? Or may be "generator_stop_iteration"? From guido at python.org Thu Nov 20 23:04:24 2014 From: guido at python.org (Guido van Rossum) Date: Thu, 20 Nov 2014 14:04:24 -0800 Subject: [Python-Dev] PEP 479: Change StopIteration handling inside generators In-Reply-To: References: <1416428127.707439.193050285.1F5F417E@webmail.messagingengine.com> <20141120205843.0c59c6d2@fsol> Message-ID: On Thu, Nov 20, 2014 at 12:13 PM, Serhiy Storchaka wrote: > On 20.11.14 21:58, Antoine Pitrou wrote: > >> To me "generator_return" sounds like the addition to generator syntax >> allowing for return statements (which was done as part of the "yield >> from" PEP). How about "generate_escape"? >> > > Or may be "generator_stop_iteration"? > Or just "generator_stop"? -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From tjreedy at udel.edu Fri Nov 21 00:02:00 2014 From: tjreedy at udel.edu (Terry Reedy) Date: Thu, 20 Nov 2014 18:02:00 -0500 Subject: [Python-Dev] PEP 479: Change StopIteration handling inside generators In-Reply-To: References: <1416428127.707439.193050285.1F5F417E@webmail.messagingengine.com> <20141120205843.0c59c6d2@fsol> Message-ID: On 11/20/2014 5:04 PM, Guido van Rossum wrote: > On Thu, Nov 20, 2014 at 12:13 PM, Serhiy Storchaka > wrote: > > On 20.11.14 21:58, Antoine Pitrou wrote: > > To me "generator_return" sounds like the addition to generator > syntax > allowing for return statements (which was done as part of the "yield > from" PEP). How about "generate_escape"? > > Or may be "generator_stop_iteration"? > > Or just "generator_stop"? +1 About as sort as we could get for the informed user reader and neutrally cryptic so a naive reader will know to look it up rather than guess (wrongly) -- Terry Jan Reedy From solipsis at pitrou.net Fri Nov 21 00:13:40 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 21 Nov 2014 00:13:40 +0100 Subject: [Python-Dev] PEP 479: Change StopIteration handling inside generators References: <1416428127.707439.193050285.1F5F417E@webmail.messagingengine.com> <20141120205843.0c59c6d2@fsol> Message-ID: <20141121001340.408b5ea1@fsol> On Thu, 20 Nov 2014 14:04:24 -0800 Guido van Rossum wrote: > On Thu, Nov 20, 2014 at 12:13 PM, Serhiy Storchaka > wrote: > > > On 20.11.14 21:58, Antoine Pitrou wrote: > > > >> To me "generator_return" sounds like the addition to generator syntax > >> allowing for return statements (which was done as part of the "yield > >> from" PEP). How about "generate_escape"? > >> > > > > Or may be "generator_stop_iteration"? > > > > Or just "generator_stop"? That sounds good. Regards Antoine. From tjreedy at udel.edu Fri Nov 21 00:19:15 2014 From: tjreedy at udel.edu (Terry Reedy) Date: Thu, 20 Nov 2014 18:19:15 -0500 Subject: [Python-Dev] PEP 479: Change StopIteration handling inside generators In-Reply-To: References: <1416428127.707439.193050285.1F5F417E@webmail.messagingengine.com> Message-ID: On 11/20/2014 2:36 PM, Guido van Rossum wrote: > There's still a lively discussion on python-ideas; Steven D'Aprano has > dug up quite a bit of evidence that StopIteration is used quite a bit in > ways that will break under the new behavior, and there also seems to be > quite a bit of third-party information that recommends StopIteration > over return to terminate a generator early. > > However I don't see much evidence that the current behavior is *better* > than the proposal -- I see the current behavior as a definite wart, and > I have definitely seen people struggle to debug silent early loop > termination due to an "escaped" StopIteration. > > That said, I think for most people the change won't matter, some people > will have to apply one of a few simple fixes, and a rare few will have > to rewrite their code in a non-trivial way (sometimes this will affect > "clever" libraries). > > I wonder if the PEP needs a better transition plan, e.g. > > - right now, start an education campaign For StackOverflow, someone with a high-enough rep (you, Guido?) should add a tag for StopIteration with text like "The Python `StopIteration` exception." (Copied from the current AttributeError tag, typeerror and NameError also exist.). If and when the PEP is accepted, I would be willing to add comments and the new tag to existing and future questions/answers. > - with Python 3.5, introduce "from __future__ import generator_return", > and silent deprecation warnings > - with Python 3.6, start issuing non-silent deprecation warnings I agree that two deprecation periods are needed. > - with Python 3.7, make the new behavior the default (subject to some > kind of review) > > It would also be useful if we could extend the PEP with some examples of > the various categories of fixes that can be applied easily, e.g. a few > examples of "raise StopIteration" directly in a generator that can be > replaced with "return" (or omitted, if it's at the end); a few examples > of situations where "yield from" can supply an elegant fix (and an > alternative for code that needs to be backward compatible with Python > 3.2 or 2.7); and finally (to be honest) an example of code that will > require being made more complicated. Agree that explicit fixes are needed. Deprecation message could refer to section of PEP. > Oh, and it would also be nice if the PEP included some suggested words > that 3rd party educators can use to explain the relationship between > StopIteration and generators in a healthier way (preferably a way that > also applies to older versions). > > Chris, are you up to drafting these additions? -- Terry Jan Reedy From rosuav at gmail.com Fri Nov 21 00:34:06 2014 From: rosuav at gmail.com (Chris Angelico) Date: Fri, 21 Nov 2014 10:34:06 +1100 Subject: [Python-Dev] PEP 479: Change StopIteration handling inside generators In-Reply-To: References: <1416428127.707439.193050285.1F5F417E@webmail.messagingengine.com> Message-ID: On Fri, Nov 21, 2014 at 6:36 AM, Guido van Rossum wrote: > It would also be useful if we could extend the PEP with some examples of the > various categories of fixes that can be applied easily, e.g. a few examples > of "raise StopIteration" directly in a generator that can be replaced with > "return" (or omitted, if it's at the end); a few examples of situations > where "yield from" can supply an elegant fix (and an alternative for code > that needs to be backward compatible with Python 3.2 or 2.7); and finally > (to be honest) an example of code that will require being made more > complicated. > > Oh, and it would also be nice if the PEP included some suggested words that > 3rd party educators can use to explain the relationship between > StopIteration and generators in a healthier way (preferably a way that also > applies to older versions). > > Chris, are you up to drafting these additions? Sure, no problem. Will knock something up shortly. ChrisA From rosuav at gmail.com Fri Nov 21 00:35:25 2014 From: rosuav at gmail.com (Chris Angelico) Date: Fri, 21 Nov 2014 10:35:25 +1100 Subject: [Python-Dev] PEP 479: Change StopIteration handling inside generators In-Reply-To: References: <1416428127.707439.193050285.1F5F417E@webmail.messagingengine.com> <20141120205843.0c59c6d2@fsol> Message-ID: On Fri, Nov 21, 2014 at 9:04 AM, Guido van Rossum wrote: > On Thu, Nov 20, 2014 at 12:13 PM, Serhiy Storchaka > wrote: >> >> On 20.11.14 21:58, Antoine Pitrou wrote: >>> >>> To me "generator_return" sounds like the addition to generator syntax >>> allowing for return statements (which was done as part of the "yield >>> from" PEP). How about "generate_escape"? >> >> >> Or may be "generator_stop_iteration"? > > > Or just "generator_stop"? Unrelated to the GeneratorExit exception. I don't think that'll be a problem though. ChrisA From guido at python.org Fri Nov 21 00:44:29 2014 From: guido at python.org (Guido van Rossum) Date: Thu, 20 Nov 2014 15:44:29 -0800 Subject: [Python-Dev] PEP 479: Change StopIteration handling inside generators In-Reply-To: <20141121001340.408b5ea1@fsol> References: <1416428127.707439.193050285.1F5F417E@webmail.messagingengine.com> <20141120205843.0c59c6d2@fsol> <20141121001340.408b5ea1@fsol> Message-ID: On Thu, Nov 20, 2014 at 3:13 PM, Antoine Pitrou wrote: > On Thu, 20 Nov 2014 14:04:24 -0800 > Guido van Rossum wrote: > > > On Thu, Nov 20, 2014 at 12:13 PM, Serhiy Storchaka > > wrote: > > > > > On 20.11.14 21:58, Antoine Pitrou wrote: > > > > > >> To me "generator_return" sounds like the addition to generator syntax > > >> allowing for return statements (which was done as part of the "yield > > >> from" PEP). How about "generate_escape"? > > >> > > > > > > Or may be "generator_stop_iteration"? > > > > > > > Or just "generator_stop"? > > That sounds good. > OK, updated the PEP. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Fri Nov 21 03:33:15 2014 From: rosuav at gmail.com (Chris Angelico) Date: Fri, 21 Nov 2014 13:33:15 +1100 Subject: [Python-Dev] PEP 479: Change StopIteration handling inside generators In-Reply-To: References: <1416428127.707439.193050285.1F5F417E@webmail.messagingengine.com> Message-ID: On Fri, Nov 21, 2014 at 10:34 AM, Chris Angelico wrote: > On Fri, Nov 21, 2014 at 6:36 AM, Guido van Rossum wrote: >> It would also be useful if we could extend the PEP with some examples of the >> various categories of fixes that can be applied easily, e.g. a few examples >> of "raise StopIteration" directly in a generator that can be replaced with >> "return" (or omitted, if it's at the end); a few examples of situations >> where "yield from" can supply an elegant fix (and an alternative for code >> that needs to be backward compatible with Python 3.2 or 2.7); and finally >> (to be honest) an example of code that will require being made more >> complicated. >> >> Oh, and it would also be nice if the PEP included some suggested words that >> 3rd party educators can use to explain the relationship between >> StopIteration and generators in a healthier way (preferably a way that also >> applies to older versions). >> >> Chris, are you up to drafting these additions? > > Sure, no problem. Will knock something up shortly. Examples and explanatory text added. https://raw.githubusercontent.com/Rosuav/GenStopIter/master/pep-0479.txt ChrisA From ncoghlan at gmail.com Fri Nov 21 13:36:26 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 21 Nov 2014 22:36:26 +1000 Subject: [Python-Dev] Move selected documentation repos to PSF BitBucket account? Message-ID: For those that aren't aware, PEP 474 is a PEP I wrote a while back suggesting we set up a "forge.python.org" service that provides easier management of Mercurial repos that don't have the complex branching requirements of the main CPython repo. Think repos like the PEPs repo, or the developer guide repo. The primary objective of the PEP is to enable an online editing + pull request style workflow for these pure documentation projects that only have a single active branch. I'd been taking "must be hosted in PSF infrastructure" as a hard requirement, but MAL pointed out earlier this evening that in the age of DVCS's, that requirement may not make sense: if you avoid tightly coupling your automation to a particular DVCS host's infrastructure, then reverting back to self-hosting (if that becomes necessary for some reason) is mostly just a matter of "hg push". If that "must be self-hosted" constraint is removed, then the obvious candidate for Mercurial hosting that supports online editing + pull requests is the PSF's BitBucket account. There'd still be some work in such a change to make sure we didn't break automated regeneration of associated site elements, but that's still a lot simpler than adding an entirely new piece of infrastructure. If folks are amenable to that variant of the idea, I'll undefer PEP 474 and revise it accordingly, with the developer guide and the PEP's repo as the initially proposed candidates for transfer. Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From brett at python.org Fri Nov 21 14:29:11 2014 From: brett at python.org (Brett Cannon) Date: Fri, 21 Nov 2014 13:29:11 +0000 Subject: [Python-Dev] Move selected documentation repos to PSF BitBucket account? References: Message-ID: On Fri Nov 21 2014 at 7:37:13 AM Nick Coghlan wrote: > For those that aren't aware, PEP 474 is a PEP I wrote a while back > suggesting we set up a "forge.python.org" service that provides easier > management of Mercurial repos that don't have the complex branching > requirements of the main CPython repo. Think repos like the PEPs repo, > or the developer guide repo. > > The primary objective of the PEP is to enable an online editing + pull > request style workflow for these pure documentation projects that only > have a single active branch. > > I'd been taking "must be hosted in PSF infrastructure" as a hard > requirement, but MAL pointed out earlier this evening that in the age > of DVCS's, that requirement may not make sense: if you avoid tightly > coupling your automation to a particular DVCS host's infrastructure, > then reverting back to self-hosting (if that becomes necessary for > some reason) is mostly just a matter of "hg push". > I don't view self-hosting as ever being a requirement. We hosted ourselves mainly to fully control commit messages (we do like to be very explicit after all =). Because we didn't want to pollute our message log with people's own messages which didn't follow our commit log guidelines or were of high enough caliber, we chose to fully control the hosting so as to not give people a false hope that we would accept a pull request. > > If that "must be self-hosted" constraint is removed, then the obvious > candidate for Mercurial hosting that supports online editing + pull > requests is the PSF's BitBucket account. > There's also CodePlex and (ironically) SourceForge for open-source hg hosting. > > There'd still be some work in such a change to make sure we didn't > break automated regeneration of associated site elements, but that's > still a lot simpler than adding an entirely new piece of > infrastructure. > > If folks are amenable to that variant of the idea, I'll undefer PEP > 474 and revise it accordingly, with the developer guide and the PEP's > repo as the initially proposed candidates for transfer. > I think showing us how to ignore PR comments and only show those from merges and direct commits on a branch (e.g. blame, reading log output, etc.) would help, i.e. how to work with Mercurial so I only see commit messages from core developers or ones they directly could edit? -------------- next part -------------- An HTML attachment was scrubbed... URL: From raymond.hettinger at gmail.com Fri Nov 21 14:47:58 2014 From: raymond.hettinger at gmail.com (Raymond Hettinger) Date: Fri, 21 Nov 2014 05:47:58 -0800 Subject: [Python-Dev] PEP 479: Change StopIteration handling inside generators In-Reply-To: References: Message-ID: > On Nov 19, 2014, at 12:10 PM, Guido van Rossum wrote: > > There's a new PEP proposing to change how to treat StopIteration bubbling up out of a generator frame (not caused by a return from the frame). The proposal is to replace such a StopIteration with a RuntimeError (chained to the original StopIteration), so that only *returning* from a generator (or falling off the end) causes the iteration to terminate. > > The proposal unifies the behavior of list comprehensions and generator expressions along the lines I had originally in mind when they were introduced. It renders useless/illegal certain hacks that have crept into some folks' arsenal of obfuscated Python tools. I strongly recommend against accepting this PEP. The PEP itself eloquently articulates an important criticism, "Unofficial and apocryphal statistics suggest that this is seldom, if ever, a problem. [4]? Code does exist which relies on the current behaviour (e.g. [2]? , [5]? , [6]? ), and there is the concern that this would be unnecessary code churn to achieve little or no gain.". Another issue is that it breaks the way I and others have taught for years that generators are a kind of iterator (an object implementing the iterator protocol) and that a primary motivation for generators is to provide a simpler and more direct way of creating iterators. However, Chris explained that, "This proposal causes a separation of generators and iterators, so it's no longer possible to pretend that they're the same thing." That is a major and worrisome conceptual shift. Also, the proposal breaks a reasonably useful pattern of calling next(subiterator) inside a generator and letting the generator terminate when the data stream ends. Here is an example that I have taught for years: def izip(iterable1, iterable2): it1 = iter(iterable1) it2 = iter(iterable2) while True: v1 = next(it1) v2 = next(it2) yield v1, v2 The above code is not atypical. Several of the pure python equivalents in the itertools docs have documented this pattern to the world for over a decade. I have seen it other people's code bases as well (in several contexts including producer/consumer chains, generators that use next() to fetch initial values from a stream, and generators that have multiple subiterators). This behavior was guaranteed from day one in PEP 255, so we would be breaking a long-standing, published rule. Adding a try/except to catch the StopIteration make the above code compliant with the new PEP, but it wouldn't make the code better in any way. And after that fix, the code would be less beautiful that it is now, and I think that matters. Lastly, as I mentioned on python-ideas, if we really want people to migrate to Python 3, there should be a strong aversion to further increasing the semantic difference between Python 2 and Python 3 without a really good reason. Raymond P.S. The PEP 255 promise was also announced as a practice in the WhatsNew 2.2 document (where most people first learned what generators are and how to use them), "The end of the generator?s results can also be indicated by raising StopIteration manually, or by just letting the flow of execution fall off the bottom of the function." The technique was also used in the generator tutorial (which was tested Lib/test/test_generators.py). One other thought: A number of other languages have added generators modeled on the Python implementation. It would be worthwhile to check to see what the prevailing wisdom is regarding whether it should be illegal to raise StopIteration inside a generator. -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Fri Nov 21 14:53:41 2014 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 22 Nov 2014 00:53:41 +1100 Subject: [Python-Dev] PEP 479: Change StopIteration handling inside generators In-Reply-To: References: Message-ID: On Sat, Nov 22, 2014 at 12:47 AM, Raymond Hettinger wrote: > Also, the proposal breaks a reasonably useful pattern of calling > next(subiterator) inside a generator and letting the generator terminate > when the data stream ends. Here is an example that I have taught for > years: > > def izip(iterable1, iterable2): > it1 = iter(iterable1) > it2 = iter(iterable2) > while True: > v1 = next(it1) > v2 = next(it2) > yield v1, v2 Is it obvious to every user that this will consume an element from it1, then silently terminate if it2 no longer has any content? ChrisA From ncoghlan at gmail.com Fri Nov 21 14:57:07 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 21 Nov 2014 23:57:07 +1000 Subject: [Python-Dev] Move selected documentation repos to PSF BitBucket account? In-Reply-To: References: Message-ID: On 21 November 2014 23:29, Brett Cannon wrote: > On Fri Nov 21 2014 at 7:37:13 AM Nick Coghlan wrote: >> If that "must be self-hosted" constraint is removed, then the obvious >> candidate for Mercurial hosting that supports online editing + pull >> requests is the PSF's BitBucket account. > > There's also CodePlex and (ironically) SourceForge for open-source hg > hosting. Did SF end up actually integrating Hg hosting properly? They hadn't the last time I looked - it was still a third party addon to Allura. I'll spell this out in the PEP, but the reason I suggest BitBucket in particular is: - it's written in Python - the PSF already has an organisational account set up there - I have admin access, so I can bootstrap other folks as administrators (Christian Heimes & Brian Curtin are also admins) - I know the online editing works reliably, since I maintain the PyPI metadata PEP drafts there - having used both it and GitHub extensively, I'm confident the workflows are similar enough that anyone familiar with GitHub will be able to easily pick up the BitBucket UI As far as ignoring PR noise goes, we can still request that folks squash any commits (keep in mind that the proposal is only to move pure documentation repos, so long complex PR chains seem unlikely). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From brett at python.org Fri Nov 21 15:00:10 2014 From: brett at python.org (Brett Cannon) Date: Fri, 21 Nov 2014 14:00:10 +0000 Subject: [Python-Dev] Move selected documentation repos to PSF BitBucket account? References: Message-ID: On Fri Nov 21 2014 at 8:57:15 AM Nick Coghlan wrote: > On 21 November 2014 23:29, Brett Cannon wrote: > > On Fri Nov 21 2014 at 7:37:13 AM Nick Coghlan > wrote: > >> If that "must be self-hosted" constraint is removed, then the obvious > >> candidate for Mercurial hosting that supports online editing + pull > >> requests is the PSF's BitBucket account. > > > > There's also CodePlex and (ironically) SourceForge for open-source hg > > hosting. > > Did SF end up actually integrating Hg hosting properly? They hadn't > the last time I looked - it was still a third party addon to Allura. > > I'll spell this out in the PEP, but the reason I suggest BitBucket in > particular is: > > - it's written in Python > - the PSF already has an organisational account set up there > - I have admin access, so I can bootstrap other folks as > administrators (Christian Heimes & Brian Curtin are also admins) > - I know the online editing works reliably, since I maintain the PyPI > metadata PEP drafts there > - having used both it and GitHub extensively, I'm confident the > workflows are similar enough that anyone familiar with GitHub will be > able to easily pick up the BitBucket UI > You're putting more thought into the response than I did in the suggestion. =) I just know they claim to host hg repos. > > As far as ignoring PR noise goes, we can still request that folks > squash any commits (keep in mind that the proposal is only to move > pure documentation repos, so long complex PR chains seem unlikely). > Well, requesting that and actually getting it are two different things, especially when I don't know of any way to rewrite a commit message after the fact if we go back to someone and say "your commit message is bad, please fix it". -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Fri Nov 21 15:03:00 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 22 Nov 2014 00:03:00 +1000 Subject: [Python-Dev] Move selected documentation repos to PSF BitBucket account? In-Reply-To: References: Message-ID: On 22 November 2014 00:00, Brett Cannon wrote: >> As far as ignoring PR noise goes, we can still request that folks >> squash any commits (keep in mind that the proposal is only to move >> pure documentation repos, so long complex PR chains seem unlikely). > > Well, requesting that and actually getting it are two different things, > especially when I don't know of any way to rewrite a commit message after > the fact if we go back to someone and say "your commit message is bad, > please fix it". Worst case? Ask them to submit a new pull request. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Fri Nov 21 15:04:47 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 22 Nov 2014 00:04:47 +1000 Subject: [Python-Dev] Move selected documentation repos to PSF BitBucket account? In-Reply-To: References: Message-ID: On 22 November 2014 00:03, Nick Coghlan wrote: > On 22 November 2014 00:00, Brett Cannon wrote: >>> As far as ignoring PR noise goes, we can still request that folks >>> squash any commits (keep in mind that the proposal is only to move >>> pure documentation repos, so long complex PR chains seem unlikely). >> >> Well, requesting that and actually getting it are two different things, >> especially when I don't know of any way to rewrite a commit message after >> the fact if we go back to someone and say "your commit message is bad, >> please fix it". > > Worst case? Ask them to submit a new pull request. Oh, the other option is to pull the patch from the PR, apply it locally, and then use "hg commit --amend". It's still Mercurial under the hood, it's just got a streamlined single-click workflow for the cases where the patch doesn't need amending. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From p.f.moore at gmail.com Fri Nov 21 15:32:19 2014 From: p.f.moore at gmail.com (Paul Moore) Date: Fri, 21 Nov 2014 14:32:19 +0000 Subject: [Python-Dev] PEP 479: Change StopIteration handling inside generators In-Reply-To: References: Message-ID: On 21 November 2014 13:53, Chris Angelico wrote: > On Sat, Nov 22, 2014 at 12:47 AM, Raymond Hettinger > wrote: >> Also, the proposal breaks a reasonably useful pattern of calling >> next(subiterator) inside a generator and letting the generator terminate >> when the data stream ends. Here is an example that I have taught for >> years: >> >> def izip(iterable1, iterable2): >> it1 = iter(iterable1) >> it2 = iter(iterable2) >> while True: >> v1 = next(it1) >> v2 = next(it2) >> yield v1, v2 > > Is it obvious to every user that this will consume an element from > it1, then silently terminate if it2 no longer has any content? It is to me, certainly. I'm mostly with Raymond on this. The main exception is that there does seem to be a trend towards more "tricky" uses of this feature, typically around ideas such as a stop() function to break out of generator expressions. I do think this is bad practice and shouldn't be allowed. But given that it's possible to write bad code in plenty of ways, and changing the language "to protect people from themselves" is not a good idea, I don't think the benefits justify the change [1]. Paul [1] Probably simply telling people not to try to cram over-complex code into a genexp would address the worst aspects of the problem... From barry at python.org Fri Nov 21 16:26:47 2014 From: barry at python.org (Barry Warsaw) Date: Fri, 21 Nov 2014 10:26:47 -0500 Subject: [Python-Dev] Move selected documentation repos to PSF BitBucket account? In-Reply-To: References: Message-ID: <20141121102647.46e97f04@limelight.wooz.org> On Nov 21, 2014, at 10:36 PM, Nick Coghlan wrote: >I'd been taking "must be hosted in PSF infrastructure" as a hard >requirement, but MAL pointed out earlier this evening that in the age >of DVCS's, that requirement may not make sense: if you avoid tightly >coupling your automation to a particular DVCS host's infrastructure, >then reverting back to self-hosting (if that becomes necessary for >some reason) is mostly just a matter of "hg push". > >If that "must be self-hosted" constraint is removed, then the obvious >candidate for Mercurial hosting that supports online editing + pull >requests is the PSF's BitBucket account. For the record, I object to moving *official* PSF resources to proprietary, closed-source infrastructure that we do not control or have access to[*]. As nice and friendly as BitBucket or any other code hosting source is today, there are many reasons why I think this is a bad idea for *official* branches. We are beholden to their policies and operations, which may not align with PSF policies or principles today or in the future. We will not be able to customize the experience for our own needs. We will not have access to the underlying resources should we need them for any purpose. We cannot take action ourselves if some problem occurs, e.g. banning an offensive user. You're right that in a world of dvcs, branches can be mirrored anywhere. For that reason, I have no problem allowing developers to use non-PSF controlled resources *unofficially* if it makes their work easier and doesn't conflict with their own principles. However, in such cases, I still believe that the official, master, blessed repositories remain on PSF controlled infrastructure. Surely that too is possible in the world of dvcs, right? Cheers, -Barry [*] Please note that I am not objecting to our use of lower-level resources donated by our generous sponsors. It's a fine line perhaps, but I have no problem with a wiki running on a VM hosted on some donated hardware, since we still have full access to the machine, the OS, and the application software. From steve at pearwood.info Fri Nov 21 16:58:00 2014 From: steve at pearwood.info (Steven D'Aprano) Date: Sat, 22 Nov 2014 02:58:00 +1100 Subject: [Python-Dev] PEP 479: Change StopIteration handling inside generators In-Reply-To: References: Message-ID: <20141121155759.GM2748@ando.pearwood.info> On Sat, Nov 22, 2014 at 12:53:41AM +1100, Chris Angelico wrote: > On Sat, Nov 22, 2014 at 12:47 AM, Raymond Hettinger > wrote: > > Also, the proposal breaks a reasonably useful pattern of calling > > next(subiterator) inside a generator and letting the generator terminate > > when the data stream ends. Here is an example that I have taught for > > years: > > > > def izip(iterable1, iterable2): > > it1 = iter(iterable1) > > it2 = iter(iterable2) > > while True: > > v1 = next(it1) > > v2 = next(it2) > > yield v1, v2 > > Is it obvious to every user that this will consume an element from > it1, then silently terminate if it2 no longer has any content? "Every user"? Of course not. But it should be obvious to those who think carefully about the specification of zip() and what is available to implement it. zip() can't detect that the second argument is empty except by calling next(), which it doesn't do until after it has retrieved a value from the first argument. If it turns out the second argument is empty, what can it do with that first value? It can't shove it back into the iterator. It can't return a single value, or pad it with some sentinel value (that's what izip_longest does). Since zip() is documented as halting on the shorter argument, it can't raise an exception. So what other options are there apart from silently consuming the value? Indeed that is exactly what the built-in zip does: py> a = iter("abcdef") py> b = iter("abc") py> list(zip(a, b)) [('a', 'a'), ('b', 'b'), ('c', 'c')] py> next(a) 'e' -- Steven From donald at stufft.io Fri Nov 21 17:00:53 2014 From: donald at stufft.io (Donald Stufft) Date: Fri, 21 Nov 2014 11:00:53 -0500 Subject: [Python-Dev] Move selected documentation repos to PSF BitBucket account? In-Reply-To: <20141121102647.46e97f04@limelight.wooz.org> References: <20141121102647.46e97f04@limelight.wooz.org> Message-ID: > On Nov 21, 2014, at 10:26 AM, Barry Warsaw wrote: > > On Nov 21, 2014, at 10:36 PM, Nick Coghlan wrote: > >> I'd been taking "must be hosted in PSF infrastructure" as a hard >> requirement, but MAL pointed out earlier this evening that in the age >> of DVCS's, that requirement may not make sense: if you avoid tightly >> coupling your automation to a particular DVCS host's infrastructure, >> then reverting back to self-hosting (if that becomes necessary for >> some reason) is mostly just a matter of "hg push". >> >> If that "must be self-hosted" constraint is removed, then the obvious >> candidate for Mercurial hosting that supports online editing + pull >> requests is the PSF's BitBucket account. > > For the record, I object to moving *official* PSF resources to proprietary, > closed-source infrastructure that we do not control or have access to[*]. > > As nice and friendly as BitBucket or any other code hosting source is today, > there are many reasons why I think this is a bad idea for *official* > branches. We are beholden to their policies and operations, which may not > align with PSF policies or principles today or in the future. We will not be > able to customize the experience for our own needs. We will not have access > to the underlying resources should we need them for any purpose. We cannot > take action ourselves if some problem occurs, e.g. banning an offensive user. > > You're right that in a world of dvcs, branches can be mirrored anywhere. For > that reason, I have no problem allowing developers to use non-PSF controlled > resources *unofficially* if it makes their work easier and doesn't conflict > with their own principles. However, in such cases, I still believe that the > official, master, blessed repositories remain on PSF controlled > infrastructure. Surely that too is possible in the world of dvcs, right? > > Cheers, > -Barry > > [*] Please note that I am not objecting to our use of lower-level resources > donated by our generous sponsors. It's a fine line perhaps, but I have no > problem with a wiki running on a VM hosted on some donated hardware, since we > still have full access to the machine, the OS, and the application software. Personally I care less about proprietary and closed-source and care a lot more about lock-in. Thus my big problem using Bitbucket for these things is that if we ever want to *leave* bitbucket it becomes a lot harder because you have a bunch of links and such pointing at bitbucket instead of a python.org domain. They do offer a redirect feature but that is dependent on them not taking that away in the future. (They also offer a CNAME feature but if you use it you lose the ability to use TLS, which is also a non starter for me). Sadly this also leaves out my favorite host site of Github :/. Something like Github Enterprise or Atlassian stash which are able to be migrated away from are better in this regards. Ironically we do use a propetiary/closed-source/hosted solution for https://status.python.org/ but it?s not terribly difficult to migrate away from that if we ever wanted to. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA From benjamin at python.org Fri Nov 21 17:03:35 2014 From: benjamin at python.org (Benjamin Peterson) Date: Fri, 21 Nov 2014 11:03:35 -0500 Subject: [Python-Dev] Move selected documentation repos to PSF BitBucket account? In-Reply-To: References: Message-ID: <1416585815.1692347.193834457.1CF2483E@webmail.messagingengine.com> On Fri, Nov 21, 2014, at 07:36, Nick Coghlan wrote: > For those that aren't aware, PEP 474 is a PEP I wrote a while back > suggesting we set up a "forge.python.org" service that provides easier > management of Mercurial repos that don't have the complex branching > requirements of the main CPython repo. Think repos like the PEPs repo, > or the developer guide repo. > > The primary objective of the PEP is to enable an online editing + pull > request style workflow for these pure documentation projects that only > have a single active branch. > > I'd been taking "must be hosted in PSF infrastructure" as a hard > requirement, but MAL pointed out earlier this evening that in the age > of DVCS's, that requirement may not make sense: if you avoid tightly > coupling your automation to a particular DVCS host's infrastructure, > then reverting back to self-hosting (if that becomes necessary for > some reason) is mostly just a matter of "hg push". > > If that "must be self-hosted" constraint is removed, then the obvious > candidate for Mercurial hosting that supports online editing + pull > requests is the PSF's BitBucket account. > > There'd still be some work in such a change to make sure we didn't > break automated regeneration of associated site elements, but that's > still a lot simpler than adding an entirely new piece of > infrastructure. > > If folks are amenable to that variant of the idea, I'll undefer PEP > 474 and revise it accordingly, with the developer guide and the PEP's > repo as the initially proposed candidates for transfer. I hate to say this, but if we're going to have "doc" repos hosted in a different place than code, we might as bite the bullet and move them to Git + GitHub. That would surely maximize the community size + ease of use objective function. Otherwise, moving the hosting will probably be more trouble (administratively + infra) than it's worth. I know it's hard to know, but do we have a lot of people comping at the bit to submit PRs to the devguide repo? From benjamin at python.org Fri Nov 21 17:06:52 2014 From: benjamin at python.org (Benjamin Peterson) Date: Fri, 21 Nov 2014 11:06:52 -0500 Subject: [Python-Dev] Move selected documentation repos to PSF BitBucket account? In-Reply-To: References: <20141121102647.46e97f04@limelight.wooz.org> Message-ID: <1416586012.1693116.193837741.16D7417E@webmail.messagingengine.com> On Fri, Nov 21, 2014, at 11:00, Donald Stufft wrote: > > > On Nov 21, 2014, at 10:26 AM, Barry Warsaw wrote: > > > > On Nov 21, 2014, at 10:36 PM, Nick Coghlan wrote: > > > >> I'd been taking "must be hosted in PSF infrastructure" as a hard > >> requirement, but MAL pointed out earlier this evening that in the age > >> of DVCS's, that requirement may not make sense: if you avoid tightly > >> coupling your automation to a particular DVCS host's infrastructure, > >> then reverting back to self-hosting (if that becomes necessary for > >> some reason) is mostly just a matter of "hg push". > >> > >> If that "must be self-hosted" constraint is removed, then the obvious > >> candidate for Mercurial hosting that supports online editing + pull > >> requests is the PSF's BitBucket account. > > > > For the record, I object to moving *official* PSF resources to proprietary, > > closed-source infrastructure that we do not control or have access to[*]. > > > > As nice and friendly as BitBucket or any other code hosting source is today, > > there are many reasons why I think this is a bad idea for *official* > > branches. We are beholden to their policies and operations, which may not > > align with PSF policies or principles today or in the future. We will not be > > able to customize the experience for our own needs. We will not have access > > to the underlying resources should we need them for any purpose. We cannot > > take action ourselves if some problem occurs, e.g. banning an offensive user. > > > > You're right that in a world of dvcs, branches can be mirrored anywhere. For > > that reason, I have no problem allowing developers to use non-PSF controlled > > resources *unofficially* if it makes their work easier and doesn't conflict > > with their own principles. However, in such cases, I still believe that the > > official, master, blessed repositories remain on PSF controlled > > infrastructure. Surely that too is possible in the world of dvcs, right? > > > > Cheers, > > -Barry > > > > [*] Please note that I am not objecting to our use of lower-level resources > > donated by our generous sponsors. It's a fine line perhaps, but I have no > > problem with a wiki running on a VM hosted on some donated hardware, since we > > still have full access to the machine, the OS, and the application software. > > Personally I care less about proprietary and closed-source and care a lot > more > about lock-in. Thus my big problem using Bitbucket for these things is > that if > we ever want to *leave* bitbucket it becomes a lot harder because you > have a > bunch of links and such pointing at bitbucket instead of a python.org > domain. > They do offer a redirect feature but that is dependent on them not taking > that > away in the future. (They also offer a CNAME feature but if you use it > you lose > the ability to use TLS, which is also a non starter for me). Sadly this > also > leaves out my favorite host site of Github :/. Something like Github > Enterprise > or Atlassian stash which are able to be migrated away from are better in > this > regards. > > Ironically we do use a propetiary/closed-source/hosted solution for > https://status.python.org/ but it?s not terribly difficult to migrate > away from > that if we ever wanted to. The more significant one is probably Fastly. From rosuav at gmail.com Fri Nov 21 17:08:38 2014 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 22 Nov 2014 03:08:38 +1100 Subject: [Python-Dev] PEP 479: Change StopIteration handling inside generators In-Reply-To: <20141121155759.GM2748@ando.pearwood.info> References: <20141121155759.GM2748@ando.pearwood.info> Message-ID: On Sat, Nov 22, 2014 at 2:58 AM, Steven D'Aprano wrote: > Since zip() is documented as > halting on the shorter argument, it can't raise an exception. So what > other options are there apart from silently consuming the value? Sure, it's documented as doing that. But imagine something that isn't a well-known function - all you have is someone writing a generator that calls next() in multiple places. Is it obvious that it it'll silently terminate as soon as any one of those iterators is exhausted? In many cases, an exception (probably ValueError?) would be the most obvious response. ChrisA From p.f.moore at gmail.com Fri Nov 21 17:14:58 2014 From: p.f.moore at gmail.com (Paul Moore) Date: Fri, 21 Nov 2014 16:14:58 +0000 Subject: [Python-Dev] PEP 479: Change StopIteration handling inside generators In-Reply-To: <20141121155759.GM2748@ando.pearwood.info> References: <20141121155759.GM2748@ando.pearwood.info> Message-ID: On 21 November 2014 15:58, Steven D'Aprano wrote: >> > def izip(iterable1, iterable2): >> > it1 = iter(iterable1) >> > it2 = iter(iterable2) >> > while True: >> > v1 = next(it1) >> > v2 = next(it2) >> > yield v1, v2 >> >> Is it obvious to every user that this will consume an element from >> it1, then silently terminate if it2 no longer has any content? > > "Every user"? Of course not. But it should be obvious to those who think > carefully about the specification of zip() and what is available to > implement it. > > zip() can't detect that the second argument is empty except by calling > next(), which it doesn't do until after it has retrieved a value from > the first argument. If it turns out the second argument is empty, what > can it do with that first value? It can't shove it back into the > iterator. It can't return a single value, or pad it with some sentinel > value (that's what izip_longest does). Since zip() is documented as > halting on the shorter argument, it can't raise an exception. So what > other options are there apart from silently consuming the value? Interestingly, although I said "yes, it's obvious", I'd missed this subtlety. But I don't consider it "unexpected" or a "gotcha", just subtle. I certainly don't consider it to be the *wrong* behaviour - on the contrary, I'd be more surprised to get a RuntimeError when the second iterator was shorter. What I understand to be the recommended alternative: def izip(iterable1, iterable2): it1 = iter(iterable1) it2 = iter(iterable2) while True: try: # Is it OK to cover both statements with one try...except? # I think it should be, but it's a pattern I try to avoid v1 = next(it1) v2 = next(it2) except StopIteration: return yield v1, v2 looks less obvious to me, and obscures the intent a little. (Note that I understand this is only one example and that others present a much more compelling case in favour of the explicit return form) Paul From ethan at stoneleaf.us Fri Nov 21 17:31:41 2014 From: ethan at stoneleaf.us (Ethan Furman) Date: Fri, 21 Nov 2014 08:31:41 -0800 Subject: [Python-Dev] PEP 479: Change StopIteration handling inside generators In-Reply-To: References: Message-ID: <546F68ED.5050107@stoneleaf.us> On 11/21/2014 05:47 AM, Raymond Hettinger wrote: > > Also, the proposal breaks a reasonably useful pattern of calling next(subiterator) > inside a generator and letting the generator terminate when the data stream ends. > > Here is an example that I have taught for years: > > def [...] > it1 = iter(iterable1) > it2 = iter(iterable2) > while True: > v1 = next(it1) > v2 = next(it2) > yield v1, v2 Stepping back a little and looking at this code, sans header, let's consider the possible desired behaviors: - have an exact match-up between the two iterators, error otherwise - stop when one is exhausted - pad shorter one to longer one Two of those three possible options are going to require dealing with the StopIteration that shouldn't escape -- is the trade of keeping one option short and simple worth the pain caused by the error-at-a-distance bugs caused when a StopIteration does escape that shouldn't have? -- ~Ethan~ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: OpenPGP digital signature URL: From donald at stufft.io Fri Nov 21 17:37:49 2014 From: donald at stufft.io (Donald Stufft) Date: Fri, 21 Nov 2014 11:37:49 -0500 Subject: [Python-Dev] PEP 479: Change StopIteration handling inside generators In-Reply-To: <546F68ED.5050107@stoneleaf.us> References: <546F68ED.5050107@stoneleaf.us> Message-ID: > On Nov 21, 2014, at 11:31 AM, Ethan Furman wrote: > > On 11/21/2014 05:47 AM, Raymond Hettinger wrote: >> >> Also, the proposal breaks a reasonably useful pattern of calling next(subiterator) >> inside a generator and letting the generator terminate when the data stream ends. >> >> Here is an example that I have taught for years: >> >> def [...] >> it1 = iter(iterable1) >> it2 = iter(iterable2) >> while True: >> v1 = next(it1) >> v2 = next(it2) >> yield v1, v2 > > Stepping back a little and looking at this code, sans header, let's consider the possible desired behaviors: > > - have an exact match-up between the two iterators, error otherwise > - stop when one is exhausted > - pad shorter one to longer one > > Two of those three possible options are going to require dealing with the StopIteration that shouldn't escape -- is the > trade of keeping one option short and simple worth the pain caused by the error-at-a-distance bugs caused when a > StopIteration does escape that shouldn't have? > I don?t have an opinion on whether it?s enough of a big deal to actually change it, but I do find wrapping it with a try: except block and returning easier to understand. If you showed me the current code unless I really thought about it I wouldn't think about the fact that the next() calls can cause the generator to terminate. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA From solipsis at pitrou.net Fri Nov 21 17:39:38 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 21 Nov 2014 17:39:38 +0100 Subject: [Python-Dev] Move selected documentation repos to PSF BitBucket account? References: Message-ID: <20141121173938.67eb0d75@fsol> On Fri, 21 Nov 2014 13:29:11 +0000 Brett Cannon wrote: > > If that "must be self-hosted" constraint is removed, then the obvious > > candidate for Mercurial hosting that supports online editing + pull > > requests is the PSF's BitBucket account. > > > > There's also CodePlex and (ironically) SourceForge for open-source hg > hosting. I hope nobody's gonna suggest SourceForge seriously for anything. It's the SharePoint (or Lotus Notes) of software forges. Regards Antoine. From solipsis at pitrou.net Fri Nov 21 17:41:49 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 21 Nov 2014 17:41:49 +0100 Subject: [Python-Dev] Move selected documentation repos to PSF BitBucket account? References: <1416585815.1692347.193834457.1CF2483E@webmail.messagingengine.com> Message-ID: <20141121174149.41ac18cd@fsol> On Fri, 21 Nov 2014 11:03:35 -0500 Benjamin Peterson wrote: > > I hate to say this, but if we're going to have "doc" repos hosted in a > different place than code, we might as bite the bullet and move them to > Git + GitHub. That would surely maximize the community size + ease of > use objective function. > > Otherwise, moving the hosting will probably be more trouble > (administratively + infra) than it's worth. > > I know it's hard to know, but do we have a lot of people comping at the > bit to submit PRs to the devguide repo? People interested in the devguide are probably interested in CPython core in the first place, so they'll have to accept hg anyway (or use a git-hg bridge...). Regards Antoine. From rosuav at gmail.com Fri Nov 21 17:47:02 2014 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 22 Nov 2014 03:47:02 +1100 Subject: [Python-Dev] PEP 479: Change StopIteration handling inside generators In-Reply-To: References: <546F68ED.5050107@stoneleaf.us> Message-ID: On Sat, Nov 22, 2014 at 3:37 AM, Donald Stufft wrote: > I don?t have an opinion on whether it?s enough of a big deal to actually change > it, but I do find wrapping it with a try: except block and returning easier > to understand. If you showed me the current code unless I really thought about > it I wouldn't think about the fact that the next() calls can cause the > generator to terminate. And don't forget, by the way, that you can always request the current behaviour by explicitly wrapping the body of the function in try/except: def izip(iterable1, iterable2): try: it1 = iter(iterable1) it2 = iter(iterable2) while True: v1 = next(it1) v2 = next(it2) yield v1, v2 except StopIteration: pass That's exactly what current behaviour does, and if you think that that try block is too broad and should be narrowed, then you should support this proposal :) ChrisA From solipsis at pitrou.net Fri Nov 21 17:47:31 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 21 Nov 2014 17:47:31 +0100 Subject: [Python-Dev] PEP 479: Change StopIteration handling inside generators References: Message-ID: <20141121174731.2a245ff5@fsol> On Fri, 21 Nov 2014 05:47:58 -0800 Raymond Hettinger wrote: > > Another issue is that it breaks the way I and others have taught for years that generators are a kind of iterator (an object implementing the iterator protocol) and that a primary motivation for generators is to provide a simpler and more direct way of creating iterators. However, Chris explained that, "This proposal causes a separation of generators and iterators, so it's no longer possible to pretend that they're the same thing." That is a major and worrisome conceptual shift. I agree with Raymond on this point. Regards Antoine. From Steve.Dower at microsoft.com Fri Nov 21 17:49:51 2014 From: Steve.Dower at microsoft.com (Steve Dower) Date: Fri, 21 Nov 2014 16:49:51 +0000 Subject: [Python-Dev] Move selected documentation repos to PSF BitBucket account? In-Reply-To: <20141121173938.67eb0d75@fsol> References: <20141121173938.67eb0d75@fsol> Message-ID: > Antoine Pitrou wrote: > On Fri, 21 Nov 2014 13:29:11 +0000 > Brett Cannon wrote: >>> If that "must be self-hosted" constraint is removed, then the >>> obvious candidate for Mercurial hosting that supports online editing >>> + pull requests is the PSF's BitBucket account. >>> >> >> There's also CodePlex and (ironically) SourceForge for open-source hg >> hosting. > > I hope nobody's gonna suggest SourceForge seriously for anything. It's the > SharePoint (or Lotus Notes) of software forges. SourceForge can be Lotus Notes; CodePlex is SharePoint :-) BitBucket is a much better option than either of those. Since I discovered hggit (and got it working), I'm less opposed to github than I have ever been, but I still prefer BitBucket. It just feels more pragmatic and less like a cattle market. I'm not hugely for/against moving the repo, though I do like the general idea of reducing the amount of work committers have to do. Cheers, Steve > Regards > > Antoine. From status at bugs.python.org Fri Nov 21 18:08:14 2014 From: status at bugs.python.org (Python tracker) Date: Fri, 21 Nov 2014 18:08:14 +0100 (CET) Subject: [Python-Dev] Summary of Python tracker Issues Message-ID: <20141121170814.1427056345@psf.upfronthosting.co.za> ACTIVITY SUMMARY (2014-11-14 - 2014-11-21) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 4658 (+11) closed 30014 (+28) total 34672 (+39) Open issues with patches: 2176 Issues opened (30) ================== #22869: Split pylifecycle.c out from pythonrun.c http://bugs.python.org/issue22869 reopened by pitrou #22872: multiprocessing.Queue raises AssertionError http://bugs.python.org/issue22872 opened by Joseph.Siddall #22873: Re: SSLsocket.getpeercert - return ALL the fields of the certi http://bugs.python.org/issue22873 opened by nagle #22875: asyncio: call_soon() documentation unclear on timing http://bugs.python.org/issue22875 opened by Corbin.Simpson #22876: ip_interface can't be broadcast or number net http://bugs.python.org/issue22876 opened by ???????????????? #22881: show median in benchmark results http://bugs.python.org/issue22881 opened by scoder #22882: Document Linux packages you need to compile Python with all de http://bugs.python.org/issue22882 opened by Ludovic.Gasc #22883: Get rid of references to PyInt in Py3 sources http://bugs.python.org/issue22883 opened by serhiy.storchaka #22884: argparse.FileType.__call__ returns unwrapped sys.stdin and std http://bugs.python.org/issue22884 opened by keviv #22885: Arbitrary code execution vulnerability due to unchecked eval() http://bugs.python.org/issue22885 opened by stephen.farris #22886: TestProgram leaves defaultTestLoader.errors dirty http://bugs.python.org/issue22886 opened by rbcollins #22888: ensurepip and distutils' build_scripts fails on Windows when p http://bugs.python.org/issue22888 opened by gmljosea #22890: StringIO.StringIO pickled in 2.7 is not unpickleable on 3.x http://bugs.python.org/issue22890 opened by serhiy.storchaka #22891: code removal from urllib.parse.urlsplit() http://bugs.python.org/issue22891 opened by Alexander.Todorov #22893: Idle: __future__ does not work in startup code. http://bugs.python.org/issue22893 opened by terry.reedy #22894: unittest.TestCase.subTest causes all subsequent tests to be sk http://bugs.python.org/issue22894 opened by Trey.Cucco #22895: test failure introduced by the fix for issue #22462 http://bugs.python.org/issue22895 opened by doko #22896: Don't use PyObject_As*Buffer() functions http://bugs.python.org/issue22896 opened by serhiy.storchaka #22897: IDLE hangs on OS X with Cocoa Tk if encoding dialog is require http://bugs.python.org/issue22897 opened by steipe #22898: segfault during shutdown attempting to log ResourceWarning http://bugs.python.org/issue22898 opened by emptysquare #22899: http.server.BaseHTTPRequestHandler.parse_request (TypeError: d http://bugs.python.org/issue22899 opened by recharti #22901: test.test_uuid.test_find_mac fails because of `ifconfig` depre http://bugs.python.org/issue22901 opened by bru #22902: Use 'ip' for uuid.getnode() http://bugs.python.org/issue22902 opened by bru #22903: unittest creates non-picklable errors http://bugs.python.org/issue22903 opened by pitrou #22904: make profile-opt includes -fprofile* flags in _sysconfigdata C http://bugs.python.org/issue22904 opened by gregory.p.smith #22906: PEP 479: Change StopIteration handling inside generators http://bugs.python.org/issue22906 opened by Rosuav #22907: Misc/python-config.sh.in: ensure sed invocations only match be http://bugs.python.org/issue22907 opened by peko #22908: ZipExtFile in zipfile can be seekable http://bugs.python.org/issue22908 opened by Iridium.Yang #22909: [argparse] Using parse_known_args, unknown arg with space in v http://bugs.python.org/issue22909 opened by TabAtkins #22910: test_pydoc test_synopsis_sourceless is a flaky test http://bugs.python.org/issue22910 opened by gregory.p.smith Most recent 15 issues with no replies (15) ========================================== #22909: [argparse] Using parse_known_args, unknown arg with space in v http://bugs.python.org/issue22909 #22907: Misc/python-config.sh.in: ensure sed invocations only match be http://bugs.python.org/issue22907 #22896: Don't use PyObject_As*Buffer() functions http://bugs.python.org/issue22896 #22893: Idle: __future__ does not work in startup code. http://bugs.python.org/issue22893 #22890: StringIO.StringIO pickled in 2.7 is not unpickleable on 3.x http://bugs.python.org/issue22890 #22886: TestProgram leaves defaultTestLoader.errors dirty http://bugs.python.org/issue22886 #22885: Arbitrary code execution vulnerability due to unchecked eval() http://bugs.python.org/issue22885 #22883: Get rid of references to PyInt in Py3 sources http://bugs.python.org/issue22883 #22872: multiprocessing.Queue raises AssertionError http://bugs.python.org/issue22872 #22865: Allow pty.spawn to ignore data to copy http://bugs.python.org/issue22865 #22859: unittest.TestProgram.usageExit no longer invoked http://bugs.python.org/issue22859 #22853: Multiprocessing.Queue._feed deadlocks on import http://bugs.python.org/issue22853 #22844: test_gdb failure on Debian Wheezy for Z http://bugs.python.org/issue22844 #22832: Tweak parameter names for fcntl module http://bugs.python.org/issue22832 #22831: Use "with" to avoid possible fd leaks http://bugs.python.org/issue22831 Most recent 15 issues waiting for review (15) ============================================= #22908: ZipExtFile in zipfile can be seekable http://bugs.python.org/issue22908 #22907: Misc/python-config.sh.in: ensure sed invocations only match be http://bugs.python.org/issue22907 #22906: PEP 479: Change StopIteration handling inside generators http://bugs.python.org/issue22906 #22904: make profile-opt includes -fprofile* flags in _sysconfigdata C http://bugs.python.org/issue22904 #22902: Use 'ip' for uuid.getnode() http://bugs.python.org/issue22902 #22901: test.test_uuid.test_find_mac fails because of `ifconfig` depre http://bugs.python.org/issue22901 #22898: segfault during shutdown attempting to log ResourceWarning http://bugs.python.org/issue22898 #22894: unittest.TestCase.subTest causes all subsequent tests to be sk http://bugs.python.org/issue22894 #22891: code removal from urllib.parse.urlsplit() http://bugs.python.org/issue22891 #22884: argparse.FileType.__call__ returns unwrapped sys.stdin and std http://bugs.python.org/issue22884 #22882: Document Linux packages you need to compile Python with all de http://bugs.python.org/issue22882 #22881: show median in benchmark results http://bugs.python.org/issue22881 #22869: Split pylifecycle.c out from pythonrun.c http://bugs.python.org/issue22869 #22865: Allow pty.spawn to ignore data to copy http://bugs.python.org/issue22865 #22848: Subparser help does not respect SUPPRESS argument http://bugs.python.org/issue22848 Top 10 most discussed issues (10) ================================= #22869: Split pylifecycle.c out from pythonrun.c http://bugs.python.org/issue22869 12 msgs #11145: '%o' % user-defined instance http://bugs.python.org/issue11145 11 msgs #18813: Speed up slice object processing http://bugs.python.org/issue18813 8 msgs #22536: subprocess should include filename in FileNotFoundError except http://bugs.python.org/issue22536 6 msgs #22823: Use set literals instead of creating a set from a list http://bugs.python.org/issue22823 5 msgs #22882: Document Linux packages you need to compile Python with all de http://bugs.python.org/issue22882 5 msgs #22895: test failure introduced by the fix for issue #22462 http://bugs.python.org/issue22895 5 msgs #22901: test.test_uuid.test_find_mac fails because of `ifconfig` depre http://bugs.python.org/issue22901 5 msgs #14099: ZipFile.open() should not reopen the underlying file http://bugs.python.org/issue14099 4 msgs #17293: uuid.getnode() MAC address on AIX http://bugs.python.org/issue17293 4 msgs Issues closed (26) ================== #18637: mingw: export _PyNode_SizeOf as PyAPI for parser module http://bugs.python.org/issue18637 closed by serhiy.storchaka #20434: Fix error handler of _PyString_Resize() on allocation failure http://bugs.python.org/issue20434 closed by kristjan.jonsson #20450: hg touch fails on System Z Linux buildbot http://bugs.python.org/issue20450 closed by serhiy.storchaka #20604: Missing invalid mode in error message of socket.makefile. http://bugs.python.org/issue20604 closed by serhiy.storchaka #20662: Pydoc doesn't escape parameter defaults in html http://bugs.python.org/issue20662 closed by serhiy.storchaka #20736: test_socket: testSendmsgDontWait needlessly skipped on Linux http://bugs.python.org/issue20736 closed by serhiy.storchaka #21266: test_zipfile fails to run in the installed location http://bugs.python.org/issue21266 closed by serhiy.storchaka #21963: 2.7.8 backport of Issue1856 (avoid daemon thread problems at s http://bugs.python.org/issue21963 closed by pitrou #22370: pathlib OS detection http://bugs.python.org/issue22370 closed by python-dev #22453: PyObject_REPR macro causes refcount leak http://bugs.python.org/issue22453 closed by serhiy.storchaka #22796: Support for httponly/secure cookies reintroduced lax parsing b http://bugs.python.org/issue22796 closed by pitrou #22825: Modernize TextFile http://bugs.python.org/issue22825 closed by serhiy.storchaka #22827: Backport ensurepip to 2.7 (PEP 477) http://bugs.python.org/issue22827 closed by dstufft #22847: Improve method cache efficiency http://bugs.python.org/issue22847 closed by pitrou #22850: Backport ensurepip Windows installer changes to 2.7 http://bugs.python.org/issue22850 closed by steve.dower #22871: datetime documentation incomplete http://bugs.python.org/issue22871 closed by belopolsky #22874: gzip bug in python 2.7.3? http://bugs.python.org/issue22874 closed by jbonerz #22877: Backport ensurepip OS X installer changes to 2.7 http://bugs.python.org/issue22877 closed by ned.deily #22878: PEP 477 (ensurepip backport to 2.7.9): "make install" and "mak http://bugs.python.org/issue22878 closed by ned.deily #22879: Make set's add and discard methods return useful information http://bugs.python.org/issue22879 closed by rhettinger #22880: hmac.new docs show optional args incorrectly http://bugs.python.org/issue22880 closed by berker.peksag #22887: ensurepip and distutils' build_scripts fails on Windows when p http://bugs.python.org/issue22887 closed by ezio.melotti #22889: set a timeout for DNS lookups http://bugs.python.org/issue22889 closed by r.david.murray #22892: Typo in Library's 'threading' module section http://bugs.python.org/issue22892 closed by r.david.murray #22900: decimal.Context Emin, Emax limits restrict functionality witho http://bugs.python.org/issue22900 closed by ncoghlan #22905: spam http://bugs.python.org/issue22905 closed by ned.deily From steve at pearwood.info Fri Nov 21 18:18:57 2014 From: steve at pearwood.info (Steven D'Aprano) Date: Sat, 22 Nov 2014 04:18:57 +1100 Subject: [Python-Dev] PEP 479: Change StopIteration handling inside generators In-Reply-To: References: <1416428127.707439.193050285.1F5F417E@webmail.messagingengine.com> Message-ID: <20141121171857.GC15212@ando.pearwood.info> On Thu, Nov 20, 2014 at 11:36:54AM -0800, Guido van Rossum wrote: [...] > That said, I think for most people the change won't matter, some people > will have to apply one of a few simple fixes, and a rare few will have to > rewrite their code in a non-trivial way (sometimes this will affect > "clever" libraries). > > I wonder if the PEP needs a better transition plan, e.g. > > - right now, start an education campaign > - with Python 3.5, introduce "from __future__ import generator_return", and > silent deprecation warnings > - with Python 3.6, start issuing non-silent deprecation warnings > - with Python 3.7, make the new behavior the default (subject to some kind > of review) I fear that there is one specific corner case that will be impossible to deal with in a backwards-compatible way supporting both Python 2 and 3 in one code base: the use of `return value` in a generator. In Python 2.x through 3.1, `return value` is a syntax error inside generators. Currently, the only way to handle this case in 2+3 code is by using `raise StopIteration(value)` but if that changes in 3.6 or 3.7 then there will be no (obvious?) way to deal with this case. -- Steven From guido at python.org Fri Nov 21 19:18:41 2014 From: guido at python.org (Guido van Rossum) Date: Fri, 21 Nov 2014 10:18:41 -0800 Subject: [Python-Dev] PEP 479: Change StopIteration handling inside generators In-Reply-To: <20141121174731.2a245ff5@fsol> References: <20141121174731.2a245ff5@fsol> Message-ID: On Fri, Nov 21, 2014 at 8:47 AM, Antoine Pitrou wrote: > On Fri, 21 Nov 2014 05:47:58 -0800 > Raymond Hettinger wrote: > > > > Another issue is that it breaks the way I and others have taught for > years that generators are a kind of iterator (an object implementing the > iterator protocol) and that a primary motivation for generators is to > provide a simpler and more direct way of creating iterators. However, > Chris explained that, "This proposal causes a separation of generators and > iterators, so it's no longer possible to pretend that they're the same > thing." That is a major and worrisome conceptual shift. > > I agree with Raymond on this point. > Pretending they're the same thing has always been fraught with subtle errors. From the *calling* side a generator implements the same protocol as any other iterator (though it also has a few others -- send(), throw(), close()). However *inside* they are not at all similar -- generators produce a value is done through "yield", __next__() methods use return. Even if we end up rejecting the PEP we should campaign for better understanding of generators. Raymond may just have to fix some of his examples. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Fri Nov 21 19:44:28 2014 From: guido at python.org (Guido van Rossum) Date: Fri, 21 Nov 2014 10:44:28 -0800 Subject: [Python-Dev] PEP 479: Change StopIteration handling inside generators In-Reply-To: <20141121171857.GC15212@ando.pearwood.info> References: <1416428127.707439.193050285.1F5F417E@webmail.messagingengine.com> <20141121171857.GC15212@ando.pearwood.info> Message-ID: On Fri, Nov 21, 2014 at 9:18 AM, Steven D'Aprano wrote: > I fear that there is one specific corner case that will be impossible to > deal with in a backwards-compatible way supporting both Python 2 and 3 > in one code base: the use of `return value` in a generator. > > In Python 2.x through 3.1, `return value` is a syntax error inside > generators. Currently, the only way to handle this case in 2+3 code is > by using `raise StopIteration(value)` but if that changes in 3.6 or 3.7 > then there will be no (obvious?) way to deal with this case. Note that using StopIteration for this purpose is a recent invention (I believe I invented it for the Google App Engine NDB library). Before Python 3.3 this had to be essentially a private protocol implemented by the framework, and typically the framework defines a custom exception for this purpose -- either an alias for StopIteration, or a subclass of it, or a separate exception altogether. I did a little survey: - ndb uses "return Return(v)" where Return is an alias for StopIteration. - monocle uses "yield Return(v)", so it doesn't even use an exception. - In Twisted you write "returnValue(v)" -- IMO even more primitive since it's not even a control flow statement. - In Tornado you write "raise tornado.gen.Return(v)", where Return does not inherit from StopIteration. In Python 3.3 and later you can also write "return v", and the framework treats Return and StopIteration the same -- but there is no mention of "raise StopIteration(v)" in the docs and given that they have Return there should be no need for it, ever. - In Trollius (the backport of asyncio) you write "raise Return(v)", where Return is currently a subclass of StopIteration -- but it doesn't really have to be, it could be a different exception (like in Tornado). So I haven't found any framework that recommends "raise StopIteration(v)". Sure, some frameworks will have to be changed, but they have until Python 3.6 or 3.6, and the changes can be made to work all the way back to Python 2.7. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Fri Nov 21 19:46:24 2014 From: guido at python.org (Guido van Rossum) Date: Fri, 21 Nov 2014 10:46:24 -0800 Subject: [Python-Dev] Move selected documentation repos to PSF BitBucket account? In-Reply-To: References: <20141121173938.67eb0d75@fsol> Message-ID: Like it or not, github is easily winning this race. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From nad at acm.org Fri Nov 21 21:04:10 2014 From: nad at acm.org (Ned Deily) Date: Fri, 21 Nov 2014 12:04:10 -0800 Subject: [Python-Dev] Move selected documentation repos to PSF BitBucket account? References: <20141121102647.46e97f04@limelight.wooz.org> Message-ID: In article <20141121102647.46e97f04 at limelight.wooz.org>, Barry Warsaw wrote: > On Nov 21, 2014, at 10:36 PM, Nick Coghlan wrote: > > >I'd been taking "must be hosted in PSF infrastructure" as a hard > >requirement, but MAL pointed out earlier this evening that in the age > >of DVCS's, that requirement may not make sense: if you avoid tightly > >coupling your automation to a particular DVCS host's infrastructure, > >then reverting back to self-hosting (if that becomes necessary for > >some reason) is mostly just a matter of "hg push". > > > >If that "must be self-hosted" constraint is removed, then the obvious > >candidate for Mercurial hosting that supports online editing + pull > >requests is the PSF's BitBucket account. > > For the record, I object to moving *official* PSF resources to proprietary, > closed-source infrastructure that we do not control or have access to[*]. > > As nice and friendly as BitBucket or any other code hosting source is today, > there are many reasons why I think this is a bad idea for *official* > branches. We are beholden to their policies and operations, which may not > align with PSF policies or principles today or in the future. We will not be > able to customize the experience for our own needs. We will not have access > to the underlying resources should we need them for any purpose. We cannot > take action ourselves if some problem occurs, e.g. banning an offensive user. > > You're right that in a world of dvcs, branches can be mirrored anywhere. For > that reason, I have no problem allowing developers to use non-PSF controlled > resources *unofficially* if it makes their work easier and doesn't conflict > with their own principles. However, in such cases, I still believe that the > official, master, blessed repositories remain on PSF controlled > infrastructure. Surely that too is possible in the world of dvcs, right? I agree with Barry's arguments. Another issue, so to speak, is what to do about issues? Bitbucket, Github, et al provide integrated issue trackers - like we do with the current hg.python.org / bugs.python.org integration. If we were to move the official repos to another site, then what about the official site for issues? If we don't move the issues as well, some of the power and advantages of using the third-party site are lost (no one-click moving between issues and source, etc.). If we do move the issues, then we have a new source of confusion as to where people should open bugs (adding to the bug janitors workload), the problem of issue history (older and open issues in one tracker, newer ones in another), and a different workflow for tracking the issues. bugs.python.org already supports a kind of pull request with the "Remote hg repo" field. If the goal is to make it easier for newcomers to contribute via pull requests, perhaps that field could be better documented (like adding specific instructions to a readme on the Bitbucket read-only mirror) or improved, if necessary. https://docs.python.org/devguide/triaging.html#mercurial-repository -- Ned Deily, nad at acm.org From donald at stufft.io Fri Nov 21 21:35:53 2014 From: donald at stufft.io (Donald Stufft) Date: Fri, 21 Nov 2014 15:35:53 -0500 Subject: [Python-Dev] Move selected documentation repos to PSF BitBucket account? In-Reply-To: References: <20141121102647.46e97f04@limelight.wooz.org> Message-ID: <19336614-0E4F-42BF-A918-1807BB7F3599@stufft.io> > On Nov 21, 2014, at 3:04 PM, Ned Deily wrote: > > In article <20141121102647.46e97f04 at limelight.wooz.org>, > Barry Warsaw wrote: > >> On Nov 21, 2014, at 10:36 PM, Nick Coghlan wrote: >> >>> I'd been taking "must be hosted in PSF infrastructure" as a hard >>> requirement, but MAL pointed out earlier this evening that in the age >>> of DVCS's, that requirement may not make sense: if you avoid tightly >>> coupling your automation to a particular DVCS host's infrastructure, >>> then reverting back to self-hosting (if that becomes necessary for >>> some reason) is mostly just a matter of "hg push". >>> >>> If that "must be self-hosted" constraint is removed, then the obvious >>> candidate for Mercurial hosting that supports online editing + pull >>> requests is the PSF's BitBucket account. >> >> For the record, I object to moving *official* PSF resources to proprietary, >> closed-source infrastructure that we do not control or have access to[*]. >> >> As nice and friendly as BitBucket or any other code hosting source is today, >> there are many reasons why I think this is a bad idea for *official* >> branches. We are beholden to their policies and operations, which may not >> align with PSF policies or principles today or in the future. We will not be >> able to customize the experience for our own needs. We will not have access >> to the underlying resources should we need them for any purpose. We cannot >> take action ourselves if some problem occurs, e.g. banning an offensive user. >> >> You're right that in a world of dvcs, branches can be mirrored anywhere. For >> that reason, I have no problem allowing developers to use non-PSF controlled >> resources *unofficially* if it makes their work easier and doesn't conflict >> with their own principles. However, in such cases, I still believe that the >> official, master, blessed repositories remain on PSF controlled >> infrastructure. Surely that too is possible in the world of dvcs, right? > > I agree with Barry's arguments. > > Another issue, so to speak, is what to do about issues? Bitbucket, > Github, et al provide integrated issue trackers - like we do with the > current hg.python.org / bugs.python.org integration. If we were to move > the official repos to another site, then what about the official site > for issues? If we don't move the issues as well, some of the power and > advantages of using the third-party site are lost (no one-click moving > between issues and source, etc.). If we do move the issues, then we > have a new source of confusion as to where people should open bugs > (adding to the bug janitors workload), the problem of issue history > (older and open issues in one tracker, newer ones in another), and a > different workflow for tracking the issues. > > bugs.python.org already supports a kind of pull request with the "Remote > hg repo" field. If the goal is to make it easier for newcomers to > contribute via pull requests, perhaps that field could be better > documented (like adding specific instructions to a readme on the > Bitbucket read-only mirror) or improved, if necessary. > > https://docs.python.org/devguide/triaging.html#mercurial-repository Well you can?t document your way out of a bad UX. The thing you?re competing with (on Github at least) is: 1. I notice a docs change I can make 2. I click the ?Edit? button and it automatically creates a fork, and opens up a web based editor where I can make changes to that file. 3. I fill out the commit message and hit Save. 4. An automatic pull request is opened up with my changes. I can submit a fix for a typo in the docs in like ~30 seconds assuming I already have a Github account. That sort of instant win is a great way to get people started contributing to a project and can lead later to bigger more substantial contributions. In contrast as a non core developer right now I have to: 1. Clone the hg repository A. Possibly this requires installing Mercurial and learning how to clone mercurial if I don?t already know how to do that. 2. Re-find the file and open it up in a text editor and make my changes. 3. Generate a patch A. Again this possibly involves figuring out how to create such a patch. 4. Create an issue on bugs.python.org describing the typo and upload my patch. This would probably take me at least 10-15 minutes to do and I already know the tooling involved well enough to do that much. If i wasn?t familiar that could easily take hours in order to do a simple typo fix. Isn?t much better on the core developer side either, assuming in both cases that the patch is perfectly acceptable and you just need to merge it in, the Github flow is: 1. Press big green merge button, rejoice. 2. Possibly manually merge into other branches - This can be done via PRs if you want as well, since you can make a PR that merges one branch into another via the web UI and press the big green button on that too. The current flow is: 1. Download the .patch file. 2. Apply the patch file. - Unlike in the PR style flow, you can?t know ahead of time if this patch is going to cleanly apply or not. This may involve needing to rework the patch which makes it hard to judge the relative level of effort to land any one patch. 3. Commit and push the changes. 4. Possibly manually merge into other branches. Again the PR flow here takes almost no time, like 10 seconds once you?ve reviewed the PR and decided it?s good to land. Maybe another minute or two if you?re going to use the web to merge it into other branches, or a few extra minutes if you?re going to do it manually. The current flow is again probably a good 10-15 minutes worth of effort to land even a tiny one letter change. I?m not going to say that we need to switch to Github/Bitbucket to get this sort of ease of use however those platforms do come with it and it is pretty awesome. There are self hosted options that offer similar things (often times not quite as good in the ease of use though they might be more featureful in other areas). Mostly what I?m trying to say is that documenting a field that essentially requires the end user to not only figure out how to use mercurial, but figure out how to host a public repository somewhere is not even really within the same order of magnitude in ease of use. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA From nad at acm.org Fri Nov 21 21:59:33 2014 From: nad at acm.org (Ned Deily) Date: Fri, 21 Nov 2014 12:59:33 -0800 Subject: [Python-Dev] Move selected documentation repos to PSF BitBucket account? References: <20141121102647.46e97f04@limelight.wooz.org> <19336614-0E4F-42BF-A918-1807BB7F3599@stufft.io> Message-ID: In article <19336614-0E4F-42BF-A918-1807BB7F3599 at stufft.io>, Donald Stufft wrote: [...] > Well you can?t document your way out of a bad UX. The thing you?re > competing with (on Github at least) is: > > 1. I notice a docs change I can make > 2. I click the ?Edit? button and it automatically creates a fork, > and opens up a web based editor where I can make changes to > that file. > 3. I fill out the commit message and hit Save. > 4. An automatic pull request is opened up with my changes. > > I can submit a fix for a typo in the docs in like ~30 seconds assuming > I already have a Github account. That sort of instant win is a great > way to get people started contributing to a project and can lead later > to bigger more substantial contributions. [...] > Mostly what I?m trying to say is that documenting a field that essentially > requires > the end user to not only figure out how to use mercurial, but figure out how > to > host a public repository somewhere is not even really within the same order > of > magnitude in ease of use. Sure, I get that. But we're not even talking here about the main Python docs since they are part of the CPython repos, only ancillary repos like PEPs and the developer's guide. The level of activity on those is quite small. So, thinking about it a bit more, PEPs don't normally have bug tracker issues associated with them so I suppose my concerns about issue tracker aren't a major concern for them. The dev guide does get issues opened about it and I suppose they could be managed. But, without tackling the CPython repo workflow (a *much* bigger deal), is the divergence in workflows worth it? I dunno. -- Ned Deily, nad at acm.org From donald at stufft.io Fri Nov 21 22:36:00 2014 From: donald at stufft.io (Donald Stufft) Date: Fri, 21 Nov 2014 16:36:00 -0500 Subject: [Python-Dev] Move selected documentation repos to PSF BitBucket account? In-Reply-To: References: <20141121102647.46e97f04@limelight.wooz.org> <19336614-0E4F-42BF-A918-1807BB7F3599@stufft.io> Message-ID: <504EB412-57A2-43B6-ABA1-B29D63395CD6@stufft.io> > On Nov 21, 2014, at 3:59 PM, Ned Deily wrote: > > In article <19336614-0E4F-42BF-A918-1807BB7F3599 at stufft.io>, > Donald Stufft wrote: > [...] >> Well you can?t document your way out of a bad UX. The thing you?re >> competing with (on Github at least) is: >> >> 1. I notice a docs change I can make >> 2. I click the ?Edit? button and it automatically creates a fork, >> and opens up a web based editor where I can make changes to >> that file. >> 3. I fill out the commit message and hit Save. >> 4. An automatic pull request is opened up with my changes. >> >> I can submit a fix for a typo in the docs in like ~30 seconds assuming >> I already have a Github account. That sort of instant win is a great >> way to get people started contributing to a project and can lead later >> to bigger more substantial contributions. > [...] >> Mostly what I?m trying to say is that documenting a field that essentially >> requires >> the end user to not only figure out how to use mercurial, but figure out how >> to >> host a public repository somewhere is not even really within the same order >> of >> magnitude in ease of use. > > Sure, I get that. But we're not even talking here about the main Python > docs since they are part of the CPython repos, only ancillary repos like > PEPs and the developer's guide. The level of activity on those is quite > small. So, thinking about it a bit more, PEPs don't normally have bug > tracker issues associated with them so I suppose my concerns about issue > tracker aren't a major concern for them. The dev guide does get issues > opened about it and I suppose they could be managed. But, without > tackling the CPython repo workflow (a *much* bigger deal), is the > divergence in workflows worth it? I dunno. Yea for the smaller repositories I don?t have a whole lot of opinion about if the benefit buys us much, especially since one of the goals is new-person friendliness but the problem is that it doesn?t translate to contributing to CPython itself. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA From jonas.wagner at epfl.ch Sat Nov 22 00:13:00 2014 From: jonas.wagner at epfl.ch (Jonas Wagner) Date: Sat, 22 Nov 2014 00:13:00 +0100 Subject: [Python-Dev] Support for Linux perf In-Reply-To: References: Message-ID: Hi, Anyway, I think we must change CPython to support tools such as perf. Any > thoughts? > Not many thoughts, other than that it would be nice to be able to use a sampling profiler on Python code. I think this would especially benefit applications that use libraries written in C, or applications that call external commands. It would also be useful if you're interested in other metrics than time (e.g., page faults). Python does have support for profiling, though, in the cProfile module. The cleanest solution for this might be to add some sort of plugin support to Perf. Each plugin could teach Perf how to unwind a certain stack. I'm thinking this because the problem is not specific to Python. Any higher-level language would benefit from mapping the low-level instruction pointer and C stack back to higher-level function calls. Databases might use something like this to individual transactions or compiled SQL queries... These plugins would be quite closely tied to a particular interpreter implementation. You only want them in certain circumstances, and it should be possible to turn them off, e.g., when you want to profile the interpreter itself. On the other hand, this strays somewhat far from what Perf was designed for. Maybe a custom stack walker could be more easily implemented in something like SystemTap. Best, Jonas -------------- next part -------------- An HTML attachment was scrubbed... URL: From tshepang at gmail.com Sat Nov 22 06:26:20 2014 From: tshepang at gmail.com (Tshepang Lekhonkhobe) Date: Sat, 22 Nov 2014 07:26:20 +0200 Subject: [Python-Dev] Move selected documentation repos to PSF BitBucket account? In-Reply-To: References: <20141121173938.67eb0d75@fsol> Message-ID: On Fri, Nov 21, 2014 at 8:46 PM, Guido van Rossum wrote: > Like it or not, github is easily winning this race. Are you considering moving CPython development to Github? From guido at python.org Sat Nov 22 06:46:59 2014 From: guido at python.org (Guido van Rossum) Date: Fri, 21 Nov 2014 21:46:59 -0800 Subject: [Python-Dev] Move selected documentation repos to PSF BitBucket account? In-Reply-To: References: <20141121173938.67eb0d75@fsol> Message-ID: On Fri, Nov 21, 2014 at 9:26 PM, Tshepang Lekhonkhobe wrote: > On Fri, Nov 21, 2014 at 8:46 PM, Guido van Rossum > wrote: > > Like it or not, github is easily winning this race. > > Are you considering moving CPython development to Github? > No, but I prefer it for new projects. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From jtaylor.debian at googlemail.com Sat Nov 22 13:44:08 2014 From: jtaylor.debian at googlemail.com (Julian Taylor) Date: Sat, 22 Nov 2014 13:44:08 +0100 Subject: [Python-Dev] Support for Linux perf In-Reply-To: References: Message-ID: <54708518.5030509@googlemail.com> On 17.11.2014 23:09, Francis Giraldeau wrote: > Hi, > ... > The PEP-418 is about performance counters, but there is no mention o > Anyway, I think we must change CPython to support tools such as perf. > Any thoughts? > there are some patches available adding systemtap and dtrace probes, which should at least help getting function level profiles: http://bugs.python.org/issue21590 From ncoghlan at gmail.com Sat Nov 22 15:31:12 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 23 Nov 2014 00:31:12 +1000 Subject: [Python-Dev] PEP 479: Change StopIteration handling inside generators In-Reply-To: <20141121174731.2a245ff5@fsol> References: <20141121174731.2a245ff5@fsol> Message-ID: On 22 Nov 2014 02:51, "Antoine Pitrou" wrote: > > On Fri, 21 Nov 2014 05:47:58 -0800 > Raymond Hettinger wrote: > > > > Another issue is that it breaks the way I and others have taught for years that generators are a kind of iterator (an object implementing the iterator protocol) and that a primary motivation for generators is to provide a simpler and more direct way of creating iterators. However, Chris explained that, "This proposal causes a separation of generators and iterators, so it's no longer possible to pretend that they're the same thing." That is a major and worrisome conceptual shift. > > I agree with Raymond on this point. A particularly relevant variant of the idiom is the approach of writing "__iter__" directly as a generator, rather than creating a separate custom iterator class. In that context, the similarities between the __iter__ implementation and the corresponding explicit __next__ implementation is a beneficial feature. I'm definitely coming around to the point of view that, even if we wouldn't design it the way it currently works given a blank slate, the alternative design doesn't provide sufficient benefit to justify the cost of changing the behaviour & getting people to retrain their brains. Cheers, Nick. > > Regards > > Antoine. > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Sat Nov 22 15:59:42 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 23 Nov 2014 00:59:42 +1000 Subject: [Python-Dev] Move selected documentation repos to PSF BitBucket account? In-Reply-To: <504EB412-57A2-43B6-ABA1-B29D63395CD6@stufft.io> References: <20141121102647.46e97f04@limelight.wooz.org> <19336614-0E4F-42BF-A918-1807BB7F3599@stufft.io> <504EB412-57A2-43B6-ABA1-B29D63395CD6@stufft.io> Message-ID: On 22 Nov 2014 07:37, "Donald Stufft" wrote: > > On Nov 21, 2014, at 3:59 PM, Ned Deily wrote: > > Sure, I get that. But we're not even talking here about the main Python > > docs since they are part of the CPython repos, only ancillary repos like > > PEPs and the developer's guide. The level of activity on those is quite > > small. So, thinking about it a bit more, PEPs don't normally have bug > > tracker issues associated with them so I suppose my concerns about issue > > tracker aren't a major concern for them. The dev guide does get issues > > opened about it and I suppose they could be managed. But, without > > tackling the CPython repo workflow (a *much* bigger deal), is the > > divergence in workflows worth it? I dunno. I also think the tutorial and howto guides should be broken out of the main CPython repo & made version independent (with internal version specific notes). That offers no compelling advantages right now, but becomes far more beneficial if it comes with a switch to enabling online editing. > Yea for the smaller repositories I don?t have a whole lot of opinion > about if the benefit buys us much, especially since one of the goals > is new-person friendliness but the problem is that it doesn?t translate > to contributing to CPython itself. OK, different question. Has anyone here actually even *read* the workflow PEPs I wrote? They were on the agenda for the language summit, but got bumped due to lack of time (which I'm still annoyed about, given the comparatively inconsequential things that chewed up a whole lot of the day). I've only had a couple of folks from outside the core dev community express interest in them. Personally, the lack of online editing support annoys me immensely whenever I need to work on PEPs or the devguide. I also think it's ridiculous that we have dozens (hundreds?) of folks running community workshops, and all creating their own custom documentation, rather than us finding a way to better enable their collaboration on the official tutorial. The BitBucket proposal in this thread came out of a desire to avoid adding yet more work to an understaffed group of primarily volunteers maintaining the infrastructure (the paid admins are more focused on incident response and general infrastructure support, rather than spinning up new workflow services). My preferred answer remains setting up a srlf-hosted forge.python.org, but I've seen little evidence we have the capacity to deploy & maintain such a service effectively, given the relative lack of interest shown in the idea by almost everyone I've spoken to about it. Any progress has only come with a lot of pushing from me, and I just don't have the personal bandwidth to sustain that at this point. That's why the related PEPs were deferred, and the only responses I've received regarding potentially taking them over have come from folks outside the core development community, which really doesn't help very much in removing my availability as a bottleneck in the workflow improvement process. If nobody wants to maintain a self-hosted forge, or even enable the folks that have expressed interest in setting it up & maintaining it, then the right answer is "don't do it" - we should use a commercial service instead. Regards, Nick. > > --- > Donald Stufft > PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From rdmurray at bitdance.com Sat Nov 22 17:29:39 2014 From: rdmurray at bitdance.com (R. David Murray) Date: Sat, 22 Nov 2014 11:29:39 -0500 Subject: [Python-Dev] Move selected documentation repos to PSF BitBucket account? In-Reply-To: References: <20141121102647.46e97f04@limelight.wooz.org> <19336614-0E4F-42BF-A918-1807BB7F3599@stufft.io> <504EB412-57A2-43B6-ABA1-B29D63395CD6@stufft.io> Message-ID: <20141122162940.502A3250EC3@webabinitio.net> On Sun, 23 Nov 2014 00:59:42 +1000, Nick Coghlan wrote: > OK, different question. Has anyone here actually even *read* the workflow > PEPs I wrote? They were on the agenda for the language summit, but got > bumped due to lack of time (which I'm still annoyed about, given the > comparatively inconsequential things that chewed up a whole lot of the day). I have (but I'm core). It's been a while, though, I need to review it. > I've only had a couple of folks from outside the core dev community express > interest in them. Personally, the lack of online editing support annoys me > immensely whenever I need to work on PEPs or the devguide. I also think Eh, I hate online editing ;). But I understand its utility. [...] > My preferred answer remains setting up a srlf-hosted forge.python.org, but > I've seen little evidence we have the capacity to deploy & maintain such a > service effectively, given the relative lack of interest shown in the idea > by almost everyone I've spoken to about it. Any progress has only come with > a lot of pushing from me, and I just don't have the personal bandwidth to > sustain that at this point. That's why the related PEPs were deferred, and > the only responses I've received regarding potentially taking them over > have come from folks outside the core development community, which really > doesn't help very much in removing my availability as a bottleneck in the > workflow improvement process. > > If nobody wants to maintain a self-hosted forge, or even enable the folks > that have expressed interest in setting it up & maintaining it, then the > right answer is "don't do it" - we should use a commercial service instead. I have an interest in this, but like you lack bandwidth. I have no employer that I could petition for 50% open source time, so any time I spend on it is time I'm not billing...so my available time comes and goes depending on my client situation, and I'm also one of the few putting any time in on tracker maintenance (not that I have much lately, Ezio has done most of it) and I've still got the email docs to finish up as well as a list of bugs...yeah, bandwidth is a problem. But maybe I can take over coordination of the volunteers that want to work on the forge. We've already got the mailing list set up. --David From donald at stufft.io Sat Nov 22 19:27:57 2014 From: donald at stufft.io (Donald Stufft) Date: Sat, 22 Nov 2014 13:27:57 -0500 Subject: [Python-Dev] Move selected documentation repos to PSF BitBucket account? In-Reply-To: References: <20141121102647.46e97f04@limelight.wooz.org> <19336614-0E4F-42BF-A918-1807BB7F3599@stufft.io> <504EB412-57A2-43B6-ABA1-B29D63395CD6@stufft.io> Message-ID: <7182BF18-A64C-4AB1-896E-D9EF5A742B6F@stufft.io> > On Nov 22, 2014, at 9:59 AM, Nick Coghlan wrote: > > > On 22 Nov 2014 07:37, "Donald Stufft" > wrote: > > > On Nov 21, 2014, at 3:59 PM, Ned Deily > wrote: > > > Sure, I get that. But we're not even talking here about the main Python > > > docs since they are part of the CPython repos, only ancillary repos like > > > PEPs and the developer's guide. The level of activity on those is quite > > > small. So, thinking about it a bit more, PEPs don't normally have bug > > > tracker issues associated with them so I suppose my concerns about issue > > > tracker aren't a major concern for them. The dev guide does get issues > > > opened about it and I suppose they could be managed. But, without > > > tackling the CPython repo workflow (a *much* bigger deal), is the > > > divergence in workflows worth it? I dunno. > > I also think the tutorial and howto guides should be broken out of the main CPython repo & made version independent (with internal version specific notes). > > That offers no compelling advantages right now, but becomes far more beneficial if it comes with a switch to enabling online editing. > > > Yea for the smaller repositories I don?t have a whole lot of opinion > > about if the benefit buys us much, especially since one of the goals > > is new-person friendliness but the problem is that it doesn?t translate > > to contributing to CPython itself. > > OK, different question. Has anyone here actually even *read* the workflow PEPs I wrote? They were on the agenda for the language summit, but got bumped due to lack of time (which I'm still annoyed about, given the comparatively inconsequential things that chewed up a whole lot of the day). > > I've only had a couple of folks from outside the core dev community express interest in them. Personally, the lack of online editing support annoys me immensely whenever I need to work on PEPs or the devguide. I also think it's ridiculous that we have dozens (hundreds?) of folks running community workshops, and all creating their own custom documentation, rather than us finding a way to better enable their collaboration on the official tutorial. > > The BitBucket proposal in this thread came out of a desire to avoid adding yet more work to an understaffed group of primarily volunteers maintaining the infrastructure (the paid admins are more focused on incident response and general infrastructure support, rather than spinning up new workflow services). > > My preferred answer remains setting up a srlf-hosted forge.python.org , but I've seen little evidence we have the capacity to deploy & maintain such a service effectively, given the relative lack of interest shown in the idea by almost everyone I've spoken to about it. Any progress has only come with a lot of pushing from me, and I just don't have the personal bandwidth to sustain that at this point. That's why the related PEPs were deferred, and the only responses I've received regarding potentially taking them over have come from folks outside the core development community, which really doesn't help very much in removing my availability as a bottleneck in the workflow improvement process. > > If nobody wants to maintain a self-hosted forge, or even enable the folks that have expressed interest in setting it up & maintaining it, then the right answer is "don't do it" - we should use a commercial service instead. > > I think I told you before, but if not I?ll say it now :) From the infrastructure side spinning up a new VM is pretty easy, what we need is someone to write some salt states in the psf-salt repository that will deploy an instance of whatever software. I don't have time to figure out the intracities of the various software and deploy them but I can help with figuring out our infra layout. The other thing the infrastructure side would need is some guidance on what size/flavor of Rackspace VM it needs and what additional services it would need (we have a PostgreSQL database already, does it need extra disk space? A Queue? Object Store?). There is a more or less working vagrant setup in the psf-salt[1] repository (a few of the roles don't work yet without manual configuration, like the hg role) but for what someone would need for setting up a new service it should all be there. I'm not sure what else we can do to enable someone who *does* want to work on it to be able to set it up. However I'm not against using a commerical service for smaller repositories. It's fine with me, I don't have a big opinion one way or the other other than I greatly prefer Github over Bitbucket. [1] https://github.com/python/psf-salt --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: From ron3200 at gmail.com Sat Nov 22 20:49:00 2014 From: ron3200 at gmail.com (Ron Adam) Date: Sat, 22 Nov 2014 13:49:00 -0600 Subject: [Python-Dev] PEP 479: Change StopIteration handling inside generators In-Reply-To: References: <20141121174731.2a245ff5@fsol> Message-ID: On 11/22/2014 08:31 AM, Nick Coghlan wrote: > > On 22 Nov 2014 02:51, "Antoine Pitrou" > wrote: > > > > On Fri, 21 Nov 2014 05:47:58 -0800 > > Raymond Hettinger > wrote: > > > > > > Another issue is that it breaks the way I and others have taught for > years that generators are a kind of iterator (an object implementing the > iterator protocol) and that a primary motivation for generators is to > provide a simpler and more direct way of creating iterators. However, > Chris explained that, "This proposal causes a separation of generators and > iterators, so it's no longer possible to pretend that they're the same > thing." That is a major and worrisome conceptual shift. > > > > I agree with Raymond on this point. > > A particularly relevant variant of the idiom is the approach of writing > "__iter__" directly as a generator, rather than creating a separate custom > iterator class. In that context, the similarities between the __iter__ > implementation and the corresponding explicit __next__ implementation is a > beneficial feature. > > I'm definitely coming around to the point of view that, even if we wouldn't > design it the way it currently works given a blank slate, the alternative > design doesn't provide sufficient benefit to justify the cost of changing > the behaviour & getting people to retrain their brains. This all seems more complex than it should be to me. The way I tend to think about it is simply "for" loops in one form or another, catch StopIteration. So if you iterate an iterator manually rather than using it as a "for" iterator, you need to catch StopIteration. If we write a function to act as an iterator, like __next__, we need to raise StopIteration from somewhere in it, or let one bubble out from a generator if we are manually iterating it on each call... next(g). It's possible we may need to do either or both conditionally. That could mean we need to think about refactoring some part of a program, but it doesn't mean Python needs to be fixed. So the lines that split them isn't quite as clear cut as it may seem they should be. That may just be a misplaced ideal. Any time a StopIteration is raised.. either manually with "raise", or at the end of a generator, it should bubble up until a "for loop" iterating over that bit of code, catches it, or a try-except catches it, or fail loudly. I think it does do this in normal generators, so I don't see an issue with how StopIteration works. Which gets us back to generator expressions and comprehensions. Let me know if I got some part of this wrong... :-) Comprehensions are used as a convenient way to create an object. The expression parts of the comprehension define the *body* of a loop, so a StopIteration raised in it will bubble out. As it would in any other case where it is raised in the body of a loop. Generator exprssions on the other hand define the *iterator* to be used in a for loop. A StopIteration raised in it is caught by the for loop. So they both work as they are designed, but they look so similar, it looks like one is broken. It looks to me that there are three options... OPTION 1: Make comprehensions act more like generator expressions. It would mean a while loop in the object creation point is converted to a for loop. (or something equivalent.) Then both a comprehension and a generator expressions can be viewed as defining iterators, with the same behaviour, rather than comprehensions defining the body of the loop, which has the different but valid behaviour of StopIteration escaping. This would make explaining them a *lot* easier as they become the same thing used in a different context, rather than two different things used in what appears to be similar contexts. I think this fits with what Guido wants, but does so in a narrower scope, only effecting comprehensions. StopIteration is less likely to leak out. But it also allows the use of the stop() hack to raise StopIteration in comprehensions and terminate them early. Currently it doesn't work as it does in generator expressions. If the stop() hack works in both comprehensions and generator expressions the same way, then maybe we can view it as less of a hack. OPTION 2: Make generator expressions more like comprehensions. This would mean StopIteration would bubble out of generator expressions as the person who posted the original topic on python ideas wanted. And the stop hack would no longer work. Both generator expressions and comprehensions could be viewed as supplying the body in a loop. This is inconsistent with defining generators that act as iterators. So I'm definitely -1 on this option. OPTION 3: Document the differences better. Cheers, Ron From ethan at stoneleaf.us Sat Nov 22 21:04:20 2014 From: ethan at stoneleaf.us (Ethan Furman) Date: Sat, 22 Nov 2014 12:04:20 -0800 Subject: [Python-Dev] PEP 479: Change StopIteration handling inside generators In-Reply-To: References: <20141121174731.2a245ff5@fsol> Message-ID: <5470EC44.2060108@stoneleaf.us> On 11/22/2014 06:31 AM, Nick Coghlan wrote: > > A particularly relevant variant of the idiom is the approach of writing > "__iter__" directly as a generator, rather than creating a separate custom > iterator class. In that context, the similarities between the __iter__ > implementation and the corresponding explicit __next__ implementation is a > beneficial feature. https://docs.python.org/3/reference/datamodel.html?highlight=__iter__#object.__iter__ -------------------------------------------------------------------------------------- > This method is called when an iterator is required for a container. > This method should return a new iterator object that can iterate > over all the objects in the container. For mappings, it should > iterate over the keys of the container, and should also be made > available as the method keys(). > Iterator objects also need to implement this method; they are > required to return themselves. For more information on iterator > objects, see Iterator Types. Unless the object is itself at iterator, the __iter__ method is allowed to return any iterator object; whether that iterator is constructed by a separate class entirely, or by using the iter() function, or by writing a generator, should have no bearing on how we write generators themselves. -- ~Ethan~ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: OpenPGP digital signature URL: From rosuav at gmail.com Sat Nov 22 21:16:48 2014 From: rosuav at gmail.com (Chris Angelico) Date: Sun, 23 Nov 2014 07:16:48 +1100 Subject: [Python-Dev] PEP 479: Change StopIteration handling inside generators In-Reply-To: References: <20141121174731.2a245ff5@fsol> Message-ID: On Sun, Nov 23, 2014 at 6:49 AM, Ron Adam wrote: > > OPTION 1: > > Make comprehensions act more like generator expressions. > > It would mean a while loop in the object creation point is converted to a > for loop. (or something equivalent.) > > Then both a comprehension and a generator expressions can be viewed as > defining iterators, with the same behaviour, rather than comprehensions > defining the body of the loop, which has the different but valid behaviour > of StopIteration escaping. > > This would make explaining them a *lot* easier as they become the same thing > used in a different context, rather than two different things used in what > appears to be similar contexts. > > > I think this fits with what Guido wants, but does so in a narrower scope, > only effecting comprehensions. StopIteration is less likely to leak out. A list comp is usually compared to a statement-form loop. def stop(): raise StopIteration lst = [stop() for x in (1,2,3)] lst = [] for x in (1,2,3): lst.append(stop()) At the moment, these are identical (virtually; the pedantic will point out that 'x' doesn't leak out of the comprehension) - in each case, the exception raised by the body will bubble up and be printed to the console. But a generator expression is different: lst = list(stop() for x in (1,2,3)) In this form, lst is an empty list, and nothing is printed to the console. Making comprehensions work more like generator expressions would, IMO, imply making the same change to all for loops: having a StopIteration raised by the body of the loop quietly terminate the loop. This is backwards. Most of Python is about catching exceptions as narrowly as possible: you make your "try" blocks small, you use specific exception types rather than bare "except:" clauses, etc, etc, etc. You do your best to catch ONLY those exceptions that you truly understand, unless you're writing a "catch, log, and reraise" or "catch, log, and go back for more work" generic handler. A 'for' loop currently is written more-or-less like this: for var in expr: body it = iter(expr) while True: try: var = next(it) except StopIteration: break body But this suggestion of yours would make it become: it = iter(expr) while True: try: var = next(it) body except StopIteration: break I believe this to be the wrong direction to make the change. Instead, generator expressions should be the ones to change, because that's a narrowing of exception-catching scope. Currently, every generator function is implicitly guarded by "try: ...... except StopIteration: pass". This is allowing errors to pass silently, to allow a hack involving non-local control flow (letting a chained function call terminate a generator) - violating the Zen of Python. ChrisA From raymond.hettinger at gmail.com Sat Nov 22 21:30:45 2014 From: raymond.hettinger at gmail.com (Raymond Hettinger) Date: Sat, 22 Nov 2014 12:30:45 -0800 Subject: [Python-Dev] PEP 479: Change StopIteration handling inside generators In-Reply-To: References: <20141121174731.2a245ff5@fsol> Message-ID: <289664E5-58CA-4964-8DB6-20E43F3717BC@gmail.com> > On Nov 22, 2014, at 6:31 AM, Nick Coghlan wrote: > > I'm definitely coming around to the point of view that, even if we wouldn't design it the way it currently works given a blank slate, the alternative design doesn't provide sufficient benefit to justify the cost of changing the behaviour & getting people to retrain their brains. Thanks Nick. That was well said. After a couple more days of thinking about PEP 455 and reading many of the mailing list posts, I am also still flatly opposed to the proposal and wanted to summarize my thoughts for everyone's consideration. It looks like the only point in favor of the PEP is makes the internal semantics of genexps appear to more closely match list comprehensions. However, it does so not by fixing anything or adding a capability; rather, it disallows a coding pattern that potentially exposes the true differences between genexps and list comps. Even then, the difference isn't hidden; instead, the proposal just breaks the code loudly by raising a RuntimeError. AFAICT, the problem it solves isn't really a problem in practice. (I do code reviews and teach Python for living, so I get broad exposure to how python is used in practice). As collateral damage, the PEP breaks code that is otherwise well designed, beautiful looking, and functioning correctly. Even worse, the damage will be long lasting. In introduces a new special case in the current clean design of generators. And, it creates an additional learning burden (we would now have to teach the special case and how to work around it with a "try: v=next(it) except StopIteration: return". I realize these are sweeping statements, so I elaborate with more detail and examples below. If you're not interested in the details, feel free to skip the rest of the post; you've already gotten the keys points. New Special Case ---------------- By design, exceptions raised in generators are passed through to the caller. This includes IndexErrors, ValueErrors, StopIteration, and PEP 342's GeneratorExit. Under the proposed PEP, there general rule (exceptions are passed through) is broken and there is a new special case: StopIteration exceptions are caught and reraised as RuntimeErrors. This is a surprising new behavior. Breaks well established, documented, and tested behaviors --------------------------------------------------------- From the first day generators were introduced 13 years ago, we have documented and promised that you can end a terminate a generator by raising StopIteration or by a return-statment (that is in PEP 255, in the whatsnew document for 2.2, in the examples we provided for the myriad of ways to use generators, in standard library code, in the Martelli's Python Cookbook example, in the documention for itertools, etc.) Legitimate Use Cases for Raising StopIteration in a Generator ------------------------------------------------------------ In a producer/consumer generator chain, the input generator signals it is done by raising StopIteration and the output generator signals that it is done by raising StopIteration (this isn't in dispute). That use case is currently implemented something like this: def middleware_generator(source_generator): it = source_generator() input_value = next(it) output_value = do_something_interesting(input_value) yield output_value Termination of the input stream will then terminate middleware stream. You can see several real world examples of this code pattern in Fredrik Lundh's pure python verion of ElementTree (prepare_descendant, prepare_predicate, and iterfind). Under the proposal, the new logic would be to: 1) catch input stream StopIteration 2) return from the generator 3) which in turn raises StopIteration yet again. This doesn't make the code better in any way. The new code is wordy, slow, and unnecessarily convoluted: def middleware_generator(source_generator): it = source_generator() try: input_value = next(it) except StopIteration: return # This causes StopIteration to be reraised output_value = do_something_interesting(input_value) yield output_value I don't look forward to teaching people to have to write code like this and to have to remember yet another special case rule for Python. Is next() Surprising? --------------------- The PEP author uses the word "surprise" many times in describing the motivation for the PEP. In the context of comparing generator expressions to list comprehenions, I can see where someone might be surprised that though similar in appearance, their implementations are quite different and that some of those differences might not expected. However, I believe this is where the "surprise" ends. The behavior of next(it) is to return a value or raise StopIteration. That is fundamental to what is does (what else could it do?). This is as basic as list indexing returning a value or raising an IndexError, or as dict lookups returning a value or raising a KeyError. If someone in this thread says that they were suprised that next() could raise StopIteration, I don't buy it. Being able to consume a value from an iterator stream is a fundamental skill and not hard to learn (when I teach iterators and generators, the operation of next() has never been a stumbling block). The Python iterator protocol isn't rocket science. Like almost every other language, it is based on the Iterator Design Pattern found in the Gang of Four Design patterns book. For the discussion of the PEP to go forward, it is necessary to stop "acting surprised" about everything related to iterators and generators. If there is any surprise, it is confined to uncommon cases that make visible the differences between generator expressions and list comprehensions (IIRC, that is how this thread started). Also, it pays to remember that we have 13 years worth of real world experience with generators and iterators in Python. By and large, it has been one of our biggest success stories. Please keep that in mind when experiencing the temptation to change it. TL;DR Sorry for the long post, but I've been thinking about this deeply for a couple days and wanted to get all the thoughts out to others for consideration, Raymond -------------- next part -------------- An HTML attachment was scrubbed... URL: From tjreedy at udel.edu Sat Nov 22 22:01:41 2014 From: tjreedy at udel.edu (Terry Reedy) Date: Sat, 22 Nov 2014 16:01:41 -0500 Subject: [Python-Dev] PEP 479: Change StopIteration handling inside generators In-Reply-To: References: <20141121174731.2a245ff5@fsol> Message-ID: On 11/22/2014 2:49 PM, Ron Adam wrote: > On 11/22/2014 08:31 AM, Nick Coghlan wrote: >> I'm definitely coming around to the point of view that, even if we >> wouldn't >> design it the way it currently works given a blank slate, the alternative >> design doesn't provide sufficient benefit to justify the cost of changing >> the behaviour & getting people to retrain their brains. Me too. > Which gets us back to generator expressions and comprehensions. > Comprehensions are used as a convenient way to create an object. The > expression parts of the comprehension define the *body* of a loop, so a > StopIteration raised in it will bubble out. As it would in any other > case where it is raised in the body of a loop. > > Generator expressions on the other hand define the *iterator* to be used > in a for loop. A StopIteration raised in it is caught by the for loop. > > So they both work as they are designed, but they look so similar, it > looks like one is broken. > It looks to me that there are three options... > > > OPTION 1: > > Make comprehensions act more like generator expressions. I was thinking about this also. > It would mean a while loop in the object creation point is converted to > a for loop. (or something equivalent.) The code for [e(i) for i in it] already uses a for loop. The issue is when e(i) raise StopIteration. The solution to accomplish the above is to wrap the for loop in try: ... except StopIteration: pass. def _list_comp_temp(): ret = [] try: for i in it: ret.append[e(i)] except StopIteration: pass return ret I believe this would make list(or set)(genexp) == [genexp] (or {genexp}) as desired. In 2.x, there was a second difference, the leaking of 'i'. 3.x eliminated one difference, the leaking of the iteration variable, but not the other, the leaking of StopIteration. The document about execution of comprehensions says "the comprehension is executed in a separate scope, so names assigned to in the target list don?t ?leak?. What we would be adding for comprensions (but not genexps) is "and StopIteration raised in evaluating the expression" before '"leak"'. We could then document the equality above. > Then both a comprehension and a generator expressions can be viewed as > defining iterators, A comprehension is not an iterator. The above would make a list or set comprehension the same as feeding a genexp to list() or set(). > I think this fits with what Guido wants, but does so in a narrower > scope, only effecting comprehensions. StopIteration is less likely to > leak out. StopIteration would *not* leak out ever. This is half of what Guido wants, the other half being the issue of obscure bugs with genrators in general. > But it also allows the use of the stop() hack to raise StopIteration in > comprehensions and terminate them early. Yep. There is no good way to have next(it) work as intended, including in Raymond's example, where it should work, and not let stop() work. I think this falls under our 'consenting adults' principle. > Currently it doesn't work as it does in generator expressions. > If the stop() hack works in both comprehensions and generator > expressions the same way, then maybe we can view it as less of a hack. -- Terry Jan Reedy From ron3200 at gmail.com Sat Nov 22 22:03:25 2014 From: ron3200 at gmail.com (Ron Adam) Date: Sat, 22 Nov 2014 15:03:25 -0600 Subject: [Python-Dev] PEP 479: Change StopIteration handling inside generators In-Reply-To: References: <20141121174731.2a245ff5@fsol> Message-ID: On 11/22/2014 02:16 PM, Chris Angelico wrote: > On Sun, Nov 23, 2014 at 6:49 AM, Ron Adam wrote: >> > >> >OPTION 1: >> > >> >Make comprehensions act more like generator expressions. >> > >> >It would mean a while loop in the object creation point is converted to a >> >for loop. (or something equivalent.) >> > >> >Then both a comprehension and a generator expressions can be viewed as >> >defining iterators, with the same behaviour, rather than comprehensions >> >defining the body of the loop, which has the different but valid behaviour >> >of StopIteration escaping. >> > >> >This would make explaining them a*lot* easier as they become the same thing >> >used in a different context, rather than two different things used in what >> >appears to be similar contexts. >> > >> > >> >I think this fits with what Guido wants, but does so in a narrower scope, >> >only effecting comprehensions. StopIteration is less likely to leak out. > A list comp is usually compared to a statement-form loop. > > def stop(): raise StopIteration > lst = [stop() for x in (1,2,3)] > lst = [] > for x in (1,2,3): > lst.append(stop()) > > At the moment, these are identical (virtually; the pedantic will point > out that 'x' doesn't leak out of the comprehension) - in each case, > the exception raised by the body will bubble up and be printed to the > console. This is what I meant by leaking out. A StopIteration bubbles up. And your examples match my understanding. :-) > But a generator expression is different: Yes, but they work as I expect them to. > lst = list(stop() for x in (1,2,3)) > > In this form, lst is an empty list, and nothing is printed to the > console. I think that is what it should do. I think of it this way... >>> def stop(): raise StopIteration ... >>> def ge(): ... for value in (stop() for x in (1,2,3)): ... yield value ... >>> list(ge()) [] Notice the entire body of the comprehension is in the for loop header, and no parts of it are in the body except the reference to the already assigned value. The StopIteration is caught by the outer for loop. Not the for loop in the generator expression, or iterator part. > Making comprehensions work more like generator expressions > would, IMO, imply making the same change to all for loops: having a > StopIteration raised by the body of the loop quietly terminate the > loop. I'm not suggesting making any changes to generator expressions or for loops at all. They would continue to work like they currently do. Cheers, Ron > This is backwards. Most of Python is about catching exceptions as > narrowly as possible: you make your "try" blocks small, you use > specific exception types rather than bare "except:" clauses, etc, etc, > etc. You do your best to catch ONLY those exceptions that you truly > understand, unless you're writing a "catch, log, and reraise" or > "catch, log, and go back for more work" generic handler. A 'for' loop > currently is written more-or-less like this: > > for var in expr: > body > > it = iter(expr) > while True: > try: var = next(it) > except StopIteration: break > body > > But this suggestion of yours would make it become: > it = iter(expr) > while True: > try: > var = next(it) > body > except StopIteration: break > I believe this to be the wrong direction to make the change. Instead, > generator expressions should be the ones to change, because that's a > narrowing of exception-catching scope. Currently, every generator > function is implicitly guarded by "try: ...... except StopIteration: > pass". This is allowing errors to pass silently, to allow a hack > involving non-local control flow (letting a chained function call > terminate a generator) - violating the Zen of Python. > > ChrisA From ron3200 at gmail.com Sat Nov 22 22:07:53 2014 From: ron3200 at gmail.com (Ron Adam) Date: Sat, 22 Nov 2014 15:07:53 -0600 Subject: [Python-Dev] PEP 479: Change StopIteration handling inside generators In-Reply-To: References: <20141121174731.2a245ff5@fsol> Message-ID: On 11/22/2014 03:01 PM, Terry Reedy wrote: > >> Then both a comprehension and a generator expressions can be viewed as >> defining iterators, > > A comprehension is not an iterator. The above would make a list or set > comprehension the same as feeding a genexp to list() or set(). Correct, but we could think and reason about them in the same way. Cheers, Ron From Steve.Dower at microsoft.com Sat Nov 22 23:10:16 2014 From: Steve.Dower at microsoft.com (Steve Dower) Date: Sat, 22 Nov 2014 22:10:16 +0000 Subject: [Python-Dev] PCBuild project updates Message-ID: <1416694222287.87888@microsoft.com> ?Hi all Just attracting?some attention to http://bugs.python.org/issue22919?for those of us who build Python out of the PCBuild folder. More details/patches there, but the tl;dr is * Still works with VS 2010 (and now VS 2013 and VS 2015 too) * Build time should be reduced * Tools\buildbot\*.bat are also updated, so buildbots shouldn't notice any change If you're concerned/interested, I'll be watching the tracker for comments. Ideally I'd like at least a couple of people to test build with whatever VS version they have, but if nobody can do that then I'll assume that nobody will notice anything break sooner than me :) Cheers, Steve From rosuav at gmail.com Sat Nov 22 23:23:20 2014 From: rosuav at gmail.com (Chris Angelico) Date: Sun, 23 Nov 2014 09:23:20 +1100 Subject: [Python-Dev] PEP 479: Change StopIteration handling inside generators In-Reply-To: References: <20141121174731.2a245ff5@fsol> Message-ID: On Sun, Nov 23, 2014 at 8:03 AM, Ron Adam wrote: >> Making comprehensions work more like generator expressions >> would, IMO, imply making the same change to all for loops: having a >> StopIteration raised by the body of the loop quietly terminate the >> loop. > > > I'm not suggesting making any changes to generator expressions or for loops > at all. They would continue to work like they currently do. But if you're suggesting making list comps react to StopIteration raised by their primary expressions, then to maintain the correspondence between a list comp and its expanded form, for loops would have to change too. Or should that correspondence be broken, in that single-expression loop constructs become semantically different from statement loop constructs? ChrisA From rosuav at gmail.com Sat Nov 22 23:45:14 2014 From: rosuav at gmail.com (Chris Angelico) Date: Sun, 23 Nov 2014 09:45:14 +1100 Subject: [Python-Dev] PEP 479: Change StopIteration handling inside generators In-Reply-To: <289664E5-58CA-4964-8DB6-20E43F3717BC@gmail.com> References: <20141121174731.2a245ff5@fsol> <289664E5-58CA-4964-8DB6-20E43F3717BC@gmail.com> Message-ID: On Sun, Nov 23, 2014 at 7:30 AM, Raymond Hettinger wrote: > Legitimate Use Cases for Raising StopIteration in a Generator > ------------------------------------------------------------ > > In a producer/consumer generator chain, the input generator signals > it is done by raising StopIteration and the output generator signals > that it is done by raising StopIteration (this isn't in dispute). > > That use case is currently implemented something like this: > > def middleware_generator(source_generator): > it = source_generator() > input_value = next(it) > output_value = do_something_interesting(input_value) > yield output_value Does your middleware_generator work with just a single element, yielding either one output value or none? Or is it more likely to be iterating over the source generator: def middleware_generator(source_generator): for input_value in source_generator: yield do_something_interesting(input_value) MUCH tidier code, plus it's safe against unexpected StopIterations. Even if you can't do it this way, all you need is to stick a 'break' at the end of the loop, if the try/except is so abhorrent. Wouldn't be the first time I've seen a loop with a hard break at the end of it. > This doesn't make the code better in any way. The new code > is wordy, slow, and unnecessarily convoluted: > > def middleware_generator(source_generator): > it = source_generator() > try: > input_value = next(it) > except StopIteration: > return # This causes StopIteration to be reraised > output_value = do_something_interesting(input_value) > yield output_value The raising of StopIteration here is an implementation detail; you might just as well have a comment saying "This causes the function to set an exception state and return NULL", which is what happens at the C level. What happens if do_something_interesting happens to raise StopIteration? Will you be surprised that this appears identical to the source generator yielding nothing? That's current behaviour. You don't have the option of narrowing the try/except scope as you've done above. > Is next() Surprising? > --------------------- > > If someone in this thread says that they were suprised that next() could > raise StopIteration, I don't buy it. Agreed, I don't think that's surprising to anyone. > Being able to consume a value from an iterator stream is a fundamental > skill and not hard to learn (when I teach iterators and generators, the > operation of next() has never been a stumbling block). In anything other than a generator, you're expected to cope with two possible results from next(): a returned value or a raised StopIteration. Suppose you want to read a file with a header - you'd need to do something like this: def process_file(f): f = iter(f) try: header=next(f) except StopIteration: cope_somehow_maybe_return for line in f: process_line_with_headers(line, header) Currently, *if and only if* you're writing a generator, you have an implicit "except StopIteration: return" there. Anywhere else, you need to catch that exception. Iterators are an implementation detail of generators. There is no particular reason for a generator author to be aware of iterator protocol, any more than this class needs to be: class X: def __iter__(self): return iter([1,2,3,4]) It's perfectly iterable, just as a generator is, and it knows nothing about StopIteration. ChrisA From brett at python.org Sun Nov 23 00:41:28 2014 From: brett at python.org (Brett Cannon) Date: Sat, 22 Nov 2014 23:41:28 +0000 Subject: [Python-Dev] Move selected documentation repos to PSF BitBucket account? References: <20141121102647.46e97f04@limelight.wooz.org> <19336614-0E4F-42BF-A918-1807BB7F3599@stufft.io> <504EB412-57A2-43B6-ABA1-B29D63395CD6@stufft.io> Message-ID: On Sat Nov 22 2014 at 10:00:03 AM Nick Coghlan wrote: > > On 22 Nov 2014 07:37, "Donald Stufft" wrote: > > > On Nov 21, 2014, at 3:59 PM, Ned Deily wrote: > > > > Sure, I get that. But we're not even talking here about the main > Python > > > docs since they are part of the CPython repos, only ancillary repos > like > > > PEPs and the developer's guide. The level of activity on those is > quite > > > small. So, thinking about it a bit more, PEPs don't normally have bug > > > tracker issues associated with them so I suppose my concerns about > issue > > > tracker aren't a major concern for them. The dev guide does get issues > > > opened about it and I suppose they could be managed. But, without > > > tackling the CPython repo workflow (a *much* bigger deal), is the > > > divergence in workflows worth it? I dunno. > > I also think the tutorial and howto guides should be broken out of the > main CPython repo & made version independent (with internal version > specific notes). > > That offers no compelling advantages right now, but becomes far more > beneficial if it comes with a switch to enabling online editing. > > > Yea for the smaller repositories I don?t have a whole lot of opinion > > about if the benefit buys us much, especially since one of the goals > > is new-person friendliness but the problem is that it doesn?t translate > > to contributing to CPython itself. > > OK, different question. Has anyone here actually even *read* the workflow > PEPs I wrote? They were on the agenda for the language summit, but got > bumped due to lack of time (which I'm still annoyed about, given the > comparatively inconsequential things that chewed up a whole lot of the day). > I did and was looking forward to them coming to fruition. > I've only had a couple of folks from outside the core dev community > express interest in them. Personally, the lack of online editing support > annoys me immensely whenever I need to work on PEPs or the devguide. I also > think it's ridiculous that we have dozens (hundreds?) of folks running > community workshops, and all creating their own custom documentation, > rather than us finding a way to better enable their collaboration on the > official tutorial. > > The BitBucket proposal in this thread came out of a desire to avoid adding > yet more work to an understaffed group of primarily volunteers maintaining > the infrastructure (the paid admins are more focused on incident response > and general infrastructure support, rather than spinning up new workflow > services). > > My preferred answer remains setting up a srlf-hosted forge.python.org, > but I've seen little evidence we have the capacity to deploy & maintain > such a service effectively, given the relative lack of interest shown in > the idea by almost everyone I've spoken to about it. Any progress has only > come with a lot of pushing from me, and I just don't have the personal > bandwidth to sustain that at this point. That's why the related PEPs were > deferred, and the only responses I've received regarding potentially taking > them over have come from folks outside the core development community, > which really doesn't help very much in removing my availability as a > bottleneck in the workflow improvement process. > > If nobody wants to maintain a self-hosted forge, or even enable the folks > that have expressed interest in setting it up & maintaining it, then the > right answer is "don't do it" - we should use a commercial service instead. > There are two goals to any improvement to the development workflow: that which helps the core devs and that which helps everyone else. For helping core devs that's getting some CI set up which will test every patch submitted, single-click patch committal from the issue tracker, etc. For everyone else it's inline editing and whatever it takes to get patches accepted faster (I know Nick is pointing out he wants inline editing for PEPs and docs but I don't view that as critical for core devs who already have the checkouts available and have the workflow memorized). >From my perspective, getting our commit workflow improved is the critical first step before we worry about making it easier to receive patches. If we can't keep up with an influx of patches that might occur from inline editing then there is little point in having it; frustrating people that we can't commit patches as fast as we receive them is not helpful. Now in terms of how the heck we are ever going to improve our workflow, that's tricky. As Nick as pointed out we are low on volunteer time. Take the issue tracker as an example: Ezio Melotti does a large amount of work and R. David Murray also helps, but that's mostly it (Martin von L?wis has helped in the past but has been mostly absent as of late). We are not well covered in the "hit by a bus" scenario. I understand the viewpoint of not wanting to give up control of our process to a third party, and I understand not wanting to use closed-source software. But at some point there is a "practicality beats purity" line we have to either cross to improve our workflow as we simply don't have the software or volunteers to improve our process or we stay as-is and accept our contribution rate will be lower than it could be due to self-imposed barriers to entry for contributors and review overhead for core developers. I see two solutions moving forward. One is to take some bit of documentation, put it up on bitbucket or GitHub and tell people to contribute there as an experiment and some set of core devs can volunteer to accept pull requests over there. Every so often someone can sync up that documentation with the default hg repo and see what that does to contribution rates and patch acceptance turn-around. The tutorial is probably a good one since that's going to be where beginners might want to contribute, it doesn't have a high churn rate, and it's easy to tell people teaching Python to ask for contributions on GitHub or bitbucket to those who may want to contribute. The other solution is to let people fork our GitHub and bitbucket mirrors, let people develop their patch on those services, and then in the issue tracker let them submit the link to their fork. We can then have a Python script for us -- either in Tools/ or built into the issue tracker -- which can take their fork, download the diff, apply it, and then we can follow our normal review workflow (we honestly could probably stand to have a script like this now so that all you need to know is the issue # to download and apply a patch; also having people submit links from other services is we get to use their visual diff tools). This approach has the perk that we still control the infrastructure and workflow, but it also lets people use these other platform's facilities for inline editing, etc. We could even get something like Travis set up so that we can even get free CI for patches (although I think Travis only triggers on PRs so we may need to ask people to send PRs that we actually never accept just to trigger the process). The point is that we could "grease the wheels" between our infrastructure and that of GitHub and bitbucket to make getting contributions from those platforms easier for both contributors through simpler tooling and us through potential free CI and visual diffs on the other platforms. Do either of these approaches sound reasonable to people? -Brett > Regards, > Nick. > > > > > --- > > Donald Stufft > > PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA > > > > _______________________________________________ > > Python-Dev mailing list > > Python-Dev at python.org > > https://mail.python.org/mailman/listinfo/python-dev > > > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ > brett%40python.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ron3200 at gmail.com Sun Nov 23 01:05:14 2014 From: ron3200 at gmail.com (Ron Adam) Date: Sat, 22 Nov 2014 18:05:14 -0600 Subject: [Python-Dev] PEP 479: Change StopIteration handling inside generators In-Reply-To: References: <20141121174731.2a245ff5@fsol> Message-ID: On 11/22/2014 04:23 PM, Chris Angelico wrote: > On Sun, Nov 23, 2014 at 8:03 AM, Ron Adam wrote: >>> >>Making comprehensions work more like generator expressions >>> >>would, IMO, imply making the same change to all for loops: having a >>> >>StopIteration raised by the body of the loop quietly terminate the >>> >>loop. >> > >> > >> >I'm not suggesting making any changes to generator expressions or for loops >> >at all. They would continue to work like they currently do. > But if you're suggesting making list comps react to StopIteration > raised by their primary expressions, then to maintain the > correspondence between a list comp and its expanded form, for loops > would have to change too. > Or should that correspondence be broken, in > that single-expression loop constructs become semantically different > from statement loop constructs? Se we have these... Tuple Comprehension (...) List Comprehension [...] Dict Comprehension {...} Colon make's it different from sets. Set Comprehension {...} I don't think there is any other way to create them. And they don't actually expand to any python code that is visible to a programmer. They are self contained literal expressions for creating these few objects. The expanded form you are referring to is just how we currently explain them. And yes, we will need to alter how we currently explain Comprehensions a bit if this is done. Here is what I think(?) list comps do currently. lst = [expr for items in itr if cond] Is the same as. lst = [] for x in itr: # StopIteration caught here. if cond: # StopIteration bubbles here. lst.append(expr) # StopIteration bubbles here. And it would be changed to this... def comp_expression(): for x in itr: # StopIteration caught here. if cond: list.append(expr) # StopIteration from cond and expr caught here. lst = list(x for x in comp_expression()) So [expr for itr if cond] Becomes a literal form of: list(expr for itr if cond) And I believe that is how they are explained quite often. :-) (Although It's not currently quite true, is it?) Cheers, Ron From rosuav at gmail.com Sun Nov 23 01:20:27 2014 From: rosuav at gmail.com (Chris Angelico) Date: Sun, 23 Nov 2014 11:20:27 +1100 Subject: [Python-Dev] PEP 479: Change StopIteration handling inside generators In-Reply-To: References: <20141121174731.2a245ff5@fsol> Message-ID: On Sun, Nov 23, 2014 at 11:05 AM, Ron Adam wrote: > Se we have these... > > Tuple Comprehension (...) > List Comprehension [...] > Dict Comprehension {...} Colon make's it different from sets. > Set Comprehension {...} > > I don't think there is any other way to create them. And they don't actually > expand to any python code that is visible to a programmer. They are self > contained literal expressions for creating these few objects. Hmmm, there's no such thing as tuple comprehensions. Are you confusing comprehension syntax (which has a 'for' loop in it) with tuple/list/etc display, which doesn't? lst = [1,2,3,4] # not a comprehension even = [n*2 for n in lst] # comprehension > Here is what I think(?) list comps do currently. > > lst = [expr for items in itr if cond] > > Is the same as. > > lst = [] > for x in itr: # StopIteration caught here. > if cond: # StopIteration bubbles here. > lst.append(expr) # StopIteration bubbles here. Pretty much. It's done in a nested function (so x doesn't leak), but otherwise, yes. > And it would be changed to this... > > def comp_expression(): > for x in itr: # StopIteration caught here. > if cond: > list.append(expr) > > # StopIteration from cond and expr caught here. > lst = list(x for x in comp_expression()) That wouldn't quite work, but this would: def (): ret = [] try: for x in itr: if cond: ret.append(x) except StopIteration: pass return ret lst = () (And yes, the name's syntactically illegal, but if you look at a traceback, that's what is used.) ChrisA From ron3200 at gmail.com Sun Nov 23 01:51:22 2014 From: ron3200 at gmail.com (Ron Adam) Date: Sat, 22 Nov 2014 18:51:22 -0600 Subject: [Python-Dev] PEP 479: Change StopIteration handling inside generators In-Reply-To: References: <20141121174731.2a245ff5@fsol> Message-ID: On 11/22/2014 06:20 PM, Chris Angelico wrote: > On Sun, Nov 23, 2014 at 11:05 AM, Ron Adam wrote: >> >Se we have these... >> > >> > Tuple Comprehension (...) >> > List Comprehension [...] >> > Dict Comprehension {...} Colon make's it different from sets. >> > Set Comprehension {...} >> > >> >I don't think there is any other way to create them. And they don't actually >> >expand to any python code that is visible to a programmer. They are self >> >contained literal expressions for creating these few objects. > Hmmm, there's no such thing as tuple comprehensions. Just didn't think it through quite well enough. But you are correct, that would be a generator expression. One less case to worry about. :-) > lst = [1,2,3,4] # not a comprehension > even = [n*2 for n in lst] # comprehension > >> >Here is what I think(?) list comps do currently. >> > >> > lst = [expr for items in itr if cond] >> > >> >Is the same as. >> > >> > lst = [] >> > for x in itr: # StopIteration caught here. >> > if cond: # StopIteration bubbles here. >> > lst.append(expr) # StopIteration bubbles here. > Pretty much. It's done in a nested function (so x doesn't leak), but > otherwise, yes. >> >And it would be changed to this... >> > >> > def comp_expression(): >> > for x in itr: # StopIteration caught here. >> > if cond: >> > list.append(expr) >> > >> > # StopIteration from cond and expr caught here. >> > lst = list(x for x in comp_expression()) > That wouldn't quite work, but this would: Right, the list.append() should be a yield(expr). > def (): > ret = [] > try: > for x in itr: > if cond: > ret.append(x) > except StopIteration: > pass > return ret > lst = () > > (And yes, the name's syntactically illegal, but if you look at a > traceback, that's what is used.) That's fine too. The real question is how much breakage would such a change make? That will probably need a patch in order to test it out. Cheers, Ron From rosuav at gmail.com Sun Nov 23 02:06:18 2014 From: rosuav at gmail.com (Chris Angelico) Date: Sun, 23 Nov 2014 12:06:18 +1100 Subject: [Python-Dev] PEP 479: Change StopIteration handling inside generators In-Reply-To: References: <20141121174731.2a245ff5@fsol> Message-ID: On Sun, Nov 23, 2014 at 11:51 AM, Ron Adam wrote: > > > On 11/22/2014 06:20 PM, Chris Angelico wrote: >> >> Hmmm, there's no such thing as tuple comprehensions. > > Just didn't think it through quite well enough. But you are correct, that > would be a generator expression. > > One less case to worry about. :-) Ah right, no probs. >>> >And it would be changed to this... >>> > >>> > def comp_expression(): >>> > for x in itr: # StopIteration caught here. >>> > if cond: >>> > list.append(expr) >>> > >>> > # StopIteration from cond and expr caught here. >>> > lst = list(x for x in comp_expression()) > > Right, the list.append() should be a yield(expr). In that case, your second generator expression is entirely redundant; all you want is list(comp_expression()). But the example doesn't say *why* this version should terminate on a StopIteration raised by expr, when the statement form would print an exception traceback. > The real question is how much breakage would such a change make? That will > probably need a patch in order to test it out. There's one attached here: http://bugs.python.org/issue22906 It doesn't create a __future__ directive, just applies the change globally. ChrisA From raymond.hettinger at gmail.com Sun Nov 23 02:11:20 2014 From: raymond.hettinger at gmail.com (Raymond Hettinger) Date: Sat, 22 Nov 2014 17:11:20 -0800 Subject: [Python-Dev] PEP 479: Change StopIteration handling inside generators In-Reply-To: References: <20141121174731.2a245ff5@fsol> <289664E5-58CA-4964-8DB6-20E43F3717BC@gmail.com> Message-ID: <23FBBB04-28B5-4CA5-8A1D-3E72CFEB02E0@gmail.com> > On Nov 22, 2014, at 2:45 PM, Chris Angelico wrote: > > Does your middleware_generator work with just a single element, > yielding either one output value or none? I apologize if I didn't make the point clearly. The middleware example was just simple outline of calling next(), doing some processing, and yielding a result while letting the StopIteration float through from the next() call. It was meant to show in summary form a pattern for legitimate uses of next() inside a generator. Some of those uses benefit from letting their StopIteration pass through rather than being caught, returning, and reraising the StopIteration. The worry is that your proposal intentionally breaks that code which is currently bug free, clean, fast, stable, and relying on a part of the API that has been guaranteed and documented from day one. Since the middleware() example was ineffective in communicating the need, here are some real-world examples. Here's one from Fredrick Lundh's ElementTree code in the standard library (there are several other examples besides this one in his code are well): def iterfind(elem, path, namespaces=None): # compile selector pattern cache_key = (path, None if namespaces is None else tuple(sorted(namespaces.items()))) if path[-1:] == "/": path = path + "*" # implicit all (FIXME: keep this?) try: selector = _cache[cache_key] except KeyError: if len(_cache) > 100: _cache.clear() if path[:1] == "/": raise SyntaxError("cannot use absolute path on element") next = iter(xpath_tokenizer(path, namespaces)).__next__ token = next() selector = [] while 1: try: selector.append(ops[token[0]](next, token)) except StopIteration: raise SyntaxError("invalid path") try: token = next() if token[0] == "/": token = next() except StopIteration: break _cache[cache_key] = selector # execute selector pattern result = [elem] context = _SelectorContext(elem) for select in selector: result = select(context, result) return result And here is an example from the pure python version of one of the itertools: def accumulate(iterable, func=operator.add): 'Return running totals' # accumulate([1,2,3,4,5]) --> 1 3 6 10 15 # accumulate([1,2,3,4,5], operator.mul) --> 1 2 6 24 120 it = iter(iterable) total = next(it) yield total for element in it: total = func(total, element) yield total And here is an example from Django: def _generator(): it = iter(text.split(' ')) word = next(it) yield word pos = len(word) - word.rfind('\n') - 1 for word in it: if "\n" in word: lines = word.split('\n') else: lines = (word,) pos += len(lines[0]) + 1 if pos > width: yield '\n' pos = len(lines[-1]) else: yield ' ' if len(lines) > 1: pos = len(lines[-1]) yield word return ''.join(_generator()) I could scan for even more examples, but I think you get the gist. All I'm asking is that you consider that your proposal will do more harm than good. It doesn't add any new capability at all. It just kills some code that currently works. Raymond (the author of the generator expressions pep) -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Sun Nov 23 02:30:57 2014 From: rosuav at gmail.com (Chris Angelico) Date: Sun, 23 Nov 2014 12:30:57 +1100 Subject: [Python-Dev] PEP 479: Change StopIteration handling inside generators In-Reply-To: <23FBBB04-28B5-4CA5-8A1D-3E72CFEB02E0@gmail.com> References: <20141121174731.2a245ff5@fsol> <289664E5-58CA-4964-8DB6-20E43F3717BC@gmail.com> <23FBBB04-28B5-4CA5-8A1D-3E72CFEB02E0@gmail.com> Message-ID: On Sun, Nov 23, 2014 at 12:11 PM, Raymond Hettinger wrote: > The worry is that your proposal intentionally breaks that code which is > currently > bug free, clean, fast, stable, and relying on a part of the API that has > been > guaranteed and documented from day one. (I'd just like to mention that this isn't "my proposal", beyond that I'm doing the summarizing and PEP writing. The proposal itself is primarily derived from one of Guido's posts on -ideas.) > Here's one from Fredrick Lundh's ElementTree code in the standard library > (there are several other examples besides this one in his code are well): > > def iterfind(elem, path, namespaces=None): > # compile selector pattern > cache_key = (path, None if namespaces is None > else tuple(sorted(namespaces.items()))) > if path[-1:] == "/": > path = path + "*" # implicit all (FIXME: keep this?) > try: > selector = _cache[cache_key] > except KeyError: > if len(_cache) > 100: > _cache.clear() > if path[:1] == "/": > raise SyntaxError("cannot use absolute path on element") > next = iter(xpath_tokenizer(path, namespaces)).__next__ > token = next() > selector = [] > while 1: > try: > selector.append(ops[token[0]](next, token)) > except StopIteration: > raise SyntaxError("invalid path") > try: > token = next() > if token[0] == "/": > token = next() > except StopIteration: > break > _cache[cache_key] = selector > # execute selector pattern > result = [elem] > context = _SelectorContext(elem) > for select in selector: > result = select(context, result) > return result Most of the next() calls are already guarded with try/except; from what I can see, only the first one isn't. So this wouldn't be much of a change here. > And here is an example from the pure python version of one of the itertools: > > def accumulate(iterable, func=operator.add): > 'Return running totals' > # accumulate([1,2,3,4,5]) --> 1 3 6 10 15 > # accumulate([1,2,3,4,5], operator.mul) --> 1 2 6 24 120 > it = iter(iterable) > total = next(it) > yield total > for element in it: > total = func(total, element) > yield total Again, only the first one needs protection, and all that happens is that there's clearly a control flow possibility here (that the first "yield total" might not be reached). Currently, *any* function call has the potential to be a silent control flow event. > And here is an example from Django: > > def _generator(): > it = iter(text.split(' ')) > word = next(it) > yield word > pos = len(word) - word.rfind('\n') - 1 > for word in it: > if "\n" in word: > lines = word.split('\n') > else: > lines = (word,) > pos += len(lines[0]) + 1 > if pos > width: > yield '\n' > pos = len(lines[-1]) > else: > yield ' ' > if len(lines) > 1: > pos = len(lines[-1]) > yield word > return ''.join(_generator()) When you split a string, you're guaranteed at least one result, ergo 'it' is guaranteed to yield at least one word. So this one wouldn't need to be changed - it can't possibly raise RuntimeError. > I could scan for even more examples, but I think you get the gist. > All I'm asking is that you consider that your proposal will do more > harm than good. It doesn't add any new capability at all. > It just kills some code that currently works. I have considered it, and I'm not convinced that it will. I see lots of people saying "code will have to be changed", but that's exactly the same concern that people raise about switching from the sloppy Py2 merging of text and bytes to the strict Py3 separation - yes, code has to be changed, but it's definitely much better to have immediate and obvious failures when something's wrong than to have subtle behavioral changes. ChrisA From ethan at stoneleaf.us Sun Nov 23 03:00:18 2014 From: ethan at stoneleaf.us (Ethan Furman) Date: Sat, 22 Nov 2014 18:00:18 -0800 Subject: [Python-Dev] PEP 479: Change StopIteration handling inside generators In-Reply-To: <23FBBB04-28B5-4CA5-8A1D-3E72CFEB02E0@gmail.com> References: <20141121174731.2a245ff5@fsol> <289664E5-58CA-4964-8DB6-20E43F3717BC@gmail.com> <23FBBB04-28B5-4CA5-8A1D-3E72CFEB02E0@gmail.com> Message-ID: <54713FB2.6000207@stoneleaf.us> On 11/22/2014 05:11 PM, Raymond Hettinger wrote: >> On Nov 22, 2014, at 2:45 PM, Chris Angelico wrote: >> >> Does your middleware_generator work with just a single element, >> yielding either one output value or none? > > I apologize if I didn't make the point clearly. The middleware example was > just simple outline of calling next(), doing some processing, and yielding a > result while letting the StopIteration float through from the next() call. [middleware example] def middleware_generator(source_generator): it = source_generator() input_value = next(it) output_value = do_something_interesting(input_value) yield output_value The point that Chris made that you should be refuting is this one: >> What happens if do_something_interesting happens to raise >> StopIteration? Will you be surprised that this appears identical to >> the source generator yielding nothing? -- ~Ethan~ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: OpenPGP digital signature URL: From ron3200 at gmail.com Sun Nov 23 03:48:15 2014 From: ron3200 at gmail.com (Ron Adam) Date: Sat, 22 Nov 2014 20:48:15 -0600 Subject: [Python-Dev] PEP 479: Change StopIteration handling inside generators In-Reply-To: References: <20141121174731.2a245ff5@fsol> Message-ID: On 11/22/2014 07:06 PM, Chris Angelico wrote: > On Sun, Nov 23, 2014 at 11:51 AM, Ron Adam wrote: >> > >> > >> >On 11/22/2014 06:20 PM, Chris Angelico wrote: >>> >> >>> >>Hmmm, there's no such thing as tuple comprehensions. >> > >> >Just didn't think it through quite well enough. But you are correct, that >> >would be a generator expression. >> > >> >One less case to worry about.:-) > Ah right, no probs. >>>>> >>> >And it would be changed to this... >>>>> >>> > >>>>> >>> > def comp_expression(): >>>>> >>> > for x in itr: # StopIteration caught here. >>>>> >>> > if cond: >>>>> >>> > list.append(expr) >>>>> >>> > >>>>> >>> > # StopIteration from cond and expr caught here. >>>>> >>> > lst = list(x for x in comp_expression()) >> > >> >Right, the list.append() should be a yield(expr). > In that case, your second generator expression is entirely redundant; > all you want is list(comp_expression()). Yes, and that is good. Simplies it even more. > But the example doesn't say > *why* this version should terminate on a StopIteration raised by expr, > when the statement form would print an exception traceback. I presume you are asking why do this? And not why the example does that? There has been a desires expressed, more than a few times, to make comprehensions more like generator expressions in the past. It looks to me that that desire is still true. I also think there has been quite a bit of confusion in these discussions that could be reduced substantially by making Comprehensions work a bit more like generator expressions. As to why the example does that.. a list constructor iterates over a generator expression in a way that follows the iterator protocol. If you make comprehsionsons work as if they are a generator expression fed to a constructor... then it too should follow the itorator protocol. Do you agree? >> >The real question is how much breakage would such a change make? That will >> >probably need a patch in order to test it out. > There's one attached here: > > http://bugs.python.org/issue22906 Doesn't that patch effect generators and not comprehensions? If so, it wouldn't do what we are talking about here. Ron From benjamin at python.org Sun Nov 23 05:52:21 2014 From: benjamin at python.org (Benjamin Peterson) Date: Sat, 22 Nov 2014 23:52:21 -0500 Subject: [Python-Dev] 2.7.9 delayed Message-ID: <1416718341.1123748.194286033.0BA37C79@webmail.messagingengine.com> As you may have noticed, 2.7.9rc1 wasn't tagged today. I've been much busier than I expected to be, so I haven't had the time to fix up some 2.7.9 lose ends. Hopefully, the delay won't be more than a few days. From guido at python.org Sun Nov 23 05:53:01 2014 From: guido at python.org (Guido van Rossum) Date: Sat, 22 Nov 2014 20:53:01 -0800 Subject: [Python-Dev] PEP 479: Change StopIteration handling inside generators In-Reply-To: References: <20141121174731.2a245ff5@fsol> Message-ID: In order to save everyone's breath, I am *accepting* the proposal of PEP 479. The transition plan is: - "from __future__ import generator_stop" in 3.5, and a silent deprecation if StopIteration is allowed to bubble out of a generator (i.e. no warning is printed unless you explicitly turn it on) - non-silent deprecation in 3.6 - feature enabled by default in 3.7 The PEP hasn't been updated to include this and it also could use some more editing -- I'll try to get to that Monday. But the specification of the proposal is crystal-clear and I have no doubt that this is the right thing going forward. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Sun Nov 23 06:19:25 2014 From: guido at python.org (Guido van Rossum) Date: Sat, 22 Nov 2014 21:19:25 -0800 Subject: [Python-Dev] Move selected documentation repos to PSF BitBucket account? In-Reply-To: References: <20141121102647.46e97f04@limelight.wooz.org> <19336614-0E4F-42BF-A918-1807BB7F3599@stufft.io> <504EB412-57A2-43B6-ABA1-B29D63395CD6@stufft.io> Message-ID: This thread seems to beg for a decision. I think Donald Stufft has it exactly right: we should move to GitHub, because it is the easiest to use and most contributors already know it (or are eager to learn thee). Honestly, the time for core devs (or some other elite corps of dedicated volunteers) to sysadmin their own machines (virtual or not) is over. We've never been particularly good at this, and I don't see us getting better or more efficient. Moving the CPython code and docs is not a priority, but everything else (PEPs, HOWTOs etc.) can be moved easily and I am in favor of moving to GitHub. For PEPs I've noticed that for most PEPs these days (unless the primary author is a core dev) the author sets up a git repo first anyway, and the friction of moving between such repos and the "official" repo is a pain. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Sun Nov 23 06:30:42 2014 From: donald at stufft.io (Donald Stufft) Date: Sun, 23 Nov 2014 00:30:42 -0500 Subject: [Python-Dev] Move selected documentation repos to PSF BitBucket account? In-Reply-To: References: <20141121102647.46e97f04@limelight.wooz.org> <19336614-0E4F-42BF-A918-1807BB7F3599@stufft.io> <504EB412-57A2-43B6-ABA1-B29D63395CD6@stufft.io> Message-ID: <8CDBE706-B985-4A91-A6F5-7E12A051FE9E@stufft.io> > On Nov 23, 2014, at 12:19 AM, Guido van Rossum wrote: > > This thread seems to beg for a decision. I think Donald Stufft has it exactly right: we should move to GitHub, because it is the easiest to use and most contributors already know it (or are eager to learn thee). Honestly, the time for core devs (or some other elite corps of dedicated volunteers) to sysadmin their own machines (virtual or not) is over. We've never been particularly good at this, and I don't see us getting better or more efficient. > > Moving the CPython code and docs is not a priority, but everything else (PEPs, HOWTOs etc.) can be moved easily and I am in favor of moving to GitHub. For PEPs I've noticed that for most PEPs these days (unless the primary author is a core dev) the author sets up a git repo first anyway, and the friction of moving between such repos and the "official" repo is a pain. > Heck I am a Core dev technically and I setup a git repo first until it?s moved into the hg.python.org . --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: From wes.turner at gmail.com Sun Nov 23 06:45:39 2014 From: wes.turner at gmail.com (Wes Turner) Date: Sat, 22 Nov 2014 23:45:39 -0600 Subject: [Python-Dev] Move selected documentation repos to PSF BitBucket account? In-Reply-To: References: <20141121102647.46e97f04@limelight.wooz.org> <19336614-0E4F-42BF-A918-1807BB7F3599@stufft.io> <504EB412-57A2-43B6-ABA1-B29D63395CD6@stufft.io> Message-ID: re: docs hg repos and static HTTP hosting * I can't remember what the GitHub Pages CDN cache time is * Does BitBucket support more than one pages repo? * Does support Sphinx .. index and :ref: syntax extensions? * https://github.com/yoloseem/awesome-sphinxdoc * https://github.com/rtfd/readthedocs.org ("hostthedocs") * http://conf.writethedocs.org/ * https://docs.python.org/devguide/documenting.html On Sat, Nov 22, 2014 at 11:19 PM, Guido van Rossum wrote: > This thread seems to beg for a decision. I think Donald Stufft has it > exactly right: we should move to GitHub, because it is the easiest to use > and most contributors already know it (or are eager to learn thee). > Honestly, the time for core devs (or some other elite corps of dedicated > volunteers) to sysadmin their own machines (virtual or not) is over. We've > never been particularly good at this, and I don't see us getting better or > more efficient. > > Moving the CPython code and docs is not a priority, but everything else > (PEPs, HOWTOs etc.) can be moved easily and I am in favor of moving to > GitHub. For PEPs I've noticed that for most PEPs these days (unless the > primary author is a core dev) the author sets up a git repo first anyway, > and the friction of moving between such repos and the "official" repo is a > pain. > > -- > --Guido van Rossum (python.org/~guido) > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/wes.turner%40gmail.com > > -- Wes Turner https://westurner.github.io/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Sun Nov 23 06:59:02 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 23 Nov 2014 15:59:02 +1000 Subject: [Python-Dev] Move selected documentation repos to PSF BitBucket account? In-Reply-To: References: <20141121102647.46e97f04@limelight.wooz.org> <19336614-0E4F-42BF-A918-1807BB7F3599@stufft.io> <504EB412-57A2-43B6-ABA1-B29D63395CD6@stufft.io> Message-ID: On 23 November 2014 at 15:19, Guido van Rossum wrote: > This thread seems to beg for a decision. I think Donald Stufft has it > exactly right: we should move to GitHub, because it is the easiest to use > and most contributors already know it (or are eager to learn thee). > Honestly, the time for core devs (or some other elite corps of dedicated > volunteers) to sysadmin their own machines (virtual or not) is over. We've > never been particularly good at this, and I don't see us getting better or > more efficient. The learning curve on git is still awful - it offers no compelling advantages over hg, and GitHub doesn't offer any huge benefits over BitBucket for Sphinx based documentation (ReadTheDocs works just as well with either service). > Moving the CPython code and docs is not a priority, but everything else > (PEPs, HOWTOs etc.) can be moved easily and I am in favor of moving to > GitHub. For PEPs I've noticed that for most PEPs these days (unless the > primary author is a core dev) the author sets up a git repo first anyway, > and the friction of moving between such repos and the "official" repo is a > pain. Note that if folks prefer Git, BitBucket supports both. I would object strongly to unilaterally forcing existing contributors to switch from Mercurial to git. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From donald at stufft.io Sun Nov 23 07:03:07 2014 From: donald at stufft.io (Donald Stufft) Date: Sun, 23 Nov 2014 01:03:07 -0500 Subject: [Python-Dev] Move selected documentation repos to PSF BitBucket account? In-Reply-To: References: <20141121102647.46e97f04@limelight.wooz.org> <19336614-0E4F-42BF-A918-1807BB7F3599@stufft.io> <504EB412-57A2-43B6-ABA1-B29D63395CD6@stufft.io> Message-ID: <3CC02307-6109-4F20-A815-BDA7CFE16AC2@stufft.io> > On Nov 23, 2014, at 12:59 AM, Nick Coghlan wrote: > > On 23 November 2014 at 15:19, Guido van Rossum wrote: >> This thread seems to beg for a decision. I think Donald Stufft has it >> exactly right: we should move to GitHub, because it is the easiest to use >> and most contributors already know it (or are eager to learn thee). >> Honestly, the time for core devs (or some other elite corps of dedicated >> volunteers) to sysadmin their own machines (virtual or not) is over. We've >> never been particularly good at this, and I don't see us getting better or >> more efficient. > > The learning curve on git is still awful - it offers no compelling > advantages over hg, and GitHub doesn't offer any huge benefits over > BitBucket for Sphinx based documentation (ReadTheDocs works just as > well with either service). It does have one very big compelling advantage. It?s way more popular. Besides, the learning curve on hg isn?t any better, it?s just differently hard. > >> Moving the CPython code and docs is not a priority, but everything else >> (PEPs, HOWTOs etc.) can be moved easily and I am in favor of moving to >> GitHub. For PEPs I've noticed that for most PEPs these days (unless the >> primary author is a core dev) the author sets up a git repo first anyway, >> and the friction of moving between such repos and the "official" repo is a >> pain. > > Note that if folks prefer Git, BitBucket supports both. I would object > strongly to unilaterally forcing existing contributors to switch from > Mercurial to git. Going to all the trouble to move to an external repository and choosing the least popular option out of the two main ones seems like a bad idea in general. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA From donald at stufft.io Sun Nov 23 07:06:47 2014 From: donald at stufft.io (Donald Stufft) Date: Sun, 23 Nov 2014 01:06:47 -0500 Subject: [Python-Dev] Move selected documentation repos to PSF BitBucket account? In-Reply-To: <3CC02307-6109-4F20-A815-BDA7CFE16AC2@stufft.io> References: <20141121102647.46e97f04@limelight.wooz.org> <19336614-0E4F-42BF-A918-1807BB7F3599@stufft.io> <504EB412-57A2-43B6-ABA1-B29D63395CD6@stufft.io> <3CC02307-6109-4F20-A815-BDA7CFE16AC2@stufft.io> Message-ID: <413FD9E4-B371-4635-AFDC-3BB57F8811A0@stufft.io> > On Nov 23, 2014, at 1:03 AM, Donald Stufft wrote: > >> >> On Nov 23, 2014, at 12:59 AM, Nick Coghlan wrote: >> >> On 23 November 2014 at 15:19, Guido van Rossum wrote: >>> This thread seems to beg for a decision. I think Donald Stufft has it >>> exactly right: we should move to GitHub, because it is the easiest to use >>> and most contributors already know it (or are eager to learn thee). >>> Honestly, the time for core devs (or some other elite corps of dedicated >>> volunteers) to sysadmin their own machines (virtual or not) is over. We've >>> never been particularly good at this, and I don't see us getting better or >>> more efficient. >> >> The learning curve on git is still awful - it offers no compelling >> advantages over hg, and GitHub doesn't offer any huge benefits over >> BitBucket for Sphinx based documentation (ReadTheDocs works just as >> well with either service). > > It does have one very big compelling advantage. It?s way more popular. > > Besides, the learning curve on hg isn?t any better, it?s just differently > hard. > >> >>> Moving the CPython code and docs is not a priority, but everything else >>> (PEPs, HOWTOs etc.) can be moved easily and I am in favor of moving to >>> GitHub. For PEPs I've noticed that for most PEPs these days (unless the >>> primary author is a core dev) the author sets up a git repo first anyway, >>> and the friction of moving between such repos and the "official" repo is a >>> pain. >> >> Note that if folks prefer Git, BitBucket supports both. I would object >> strongly to unilaterally forcing existing contributors to switch from >> Mercurial to git. > > Going to all the trouble to move to an external repository and choosing the > least popular option out of the two main ones seems like a bad idea in > general. > Also important to note, that if people want to use hg on their own, it?s pretty easy to do that with https://hg-git.github.io/. Last I heard that works pretty well still. I?ve yet to find one that works the other direction that doesn?t choke on some repositories. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA From rosuav at gmail.com Sun Nov 23 07:09:35 2014 From: rosuav at gmail.com (Chris Angelico) Date: Sun, 23 Nov 2014 17:09:35 +1100 Subject: [Python-Dev] Move selected documentation repos to PSF BitBucket account? In-Reply-To: References: <20141121102647.46e97f04@limelight.wooz.org> <19336614-0E4F-42BF-A918-1807BB7F3599@stufft.io> <504EB412-57A2-43B6-ABA1-B29D63395CD6@stufft.io> Message-ID: On Sun, Nov 23, 2014 at 4:59 PM, Nick Coghlan wrote: > The learning curve on git is still awful - it offers no compelling > advantages over hg, and GitHub doesn't offer any huge benefits over > BitBucket for Sphinx based documentation (ReadTheDocs works just as > well with either service). The learning curve for source control as a concept is pretty steep. Once someone's learned one DVCS, learning another is much easier, and I don't know that it makes a lot of difference whether you learn git and then hg, or hg and then git. I learned git first, and mastered hg basics by keeping a Rosetta Stone chart handy; given that the document I was reading was intended for the reverse case, I expect it's not too hard to get the basics of git once you know hg. Just as git offers few advantages over hg, hg offers few advantages over git. If git and GitHub are where most people are, I would support using them for Python. I'm one of those PEP draft authors who starts with his own repo on GitHub and sends drafts in, and Guido had to remind me that I can simply test my edits in the peps repo before posting them (to make sure I haven't mucked up the markup); if the peps repo were itself hosted on GitHub, or at least in a git repo, I could have a workflow that directly incorporates that, instead of being "off to the side" with periodic manual imports. That said, it does make sense for CPython *itself* to use Mercurial, simply because it's written in Python. I don't know how strong that philosophical argument is with people, but I wouldn't argue against it. ChrisA From wes.turner at gmail.com Sun Nov 23 07:03:04 2014 From: wes.turner at gmail.com (Wes Turner) Date: Sun, 23 Nov 2014 00:03:04 -0600 Subject: [Python-Dev] Move selected documentation repos to PSF BitBucket account? In-Reply-To: References: <20141121102647.46e97f04@limelight.wooz.org> <19336614-0E4F-42BF-A918-1807BB7F3599@stufft.io> <504EB412-57A2-43B6-ABA1-B29D63395CD6@stufft.io> Message-ID: > The learning curve on git is still awful - it offers no compelling advantages over hg, and GitHub doesn't offer any huge benefits over BitBucket for Sphinx based documentation (ReadTheDocs works just as well with either service). This was a most helpful resource: https://github.com/sympy/sympy/wiki/Git-hg-rosetta-stone#rosetta-stone Wikipedia: https://en.wikipedia.org/wiki/Git_(software) > Homepage: http://git-scm.com/ > Docs: http://git-scm.com/documentation > Docs: http://git-scm.com/book/en/ > Docs: http://documentup.com/skwp/git-workflows-book > Docs: http://learnxinyminutes.com/docs/git/ > Source: git https://github.com/git/git hg imuutability is certainly a primarily attractive feature; along with the keyring support. On Sat, Nov 22, 2014 at 11:59 PM, Nick Coghlan wrote: > On 23 November 2014 at 15:19, Guido van Rossum wrote: > > This thread seems to beg for a decision. I think Donald Stufft has it > > exactly right: we should move to GitHub, because it is the easiest to use > > and most contributors already know it (or are eager to learn thee). > > Honestly, the time for core devs (or some other elite corps of dedicated > > volunteers) to sysadmin their own machines (virtual or not) is over. > We've > > never been particularly good at this, and I don't see us getting better > or > > more efficient. > > The learning curve on git is still awful - it offers no compelling > advantages over hg, and GitHub doesn't offer any huge benefits over > BitBucket for Sphinx based documentation (ReadTheDocs works just as > well with either service). > > > Moving the CPython code and docs is not a priority, but everything else > > (PEPs, HOWTOs etc.) can be moved easily and I am in favor of moving to > > GitHub. For PEPs I've noticed that for most PEPs these days (unless the > > primary author is a core dev) the author sets up a git repo first anyway, > > and the friction of moving between such repos and the "official" repo is > a > > pain. > > Note that if folks prefer Git, BitBucket supports both. I would object > strongly to unilaterally forcing existing contributors to switch from > Mercurial to git. > > Cheers, > Nick. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/wes.turner%40gmail.com > -- Wes Turner https://westurner.github.io/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Sun Nov 23 07:10:27 2014 From: guido at python.org (Guido van Rossum) Date: Sat, 22 Nov 2014 22:10:27 -0800 Subject: [Python-Dev] Move selected documentation repos to PSF BitBucket account? In-Reply-To: References: <20141121102647.46e97f04@limelight.wooz.org> <19336614-0E4F-42BF-A918-1807BB7F3599@stufft.io> <504EB412-57A2-43B6-ABA1-B29D63395CD6@stufft.io> Message-ID: On Saturday, November 22, 2014, Nick Coghlan wrote: > On 23 November 2014 at 15:19, Guido van Rossum > wrote: > > This thread seems to beg for a decision. I think Donald Stufft has it > > exactly right: we should move to GitHub, because it is the easiest to use > > and most contributors already know it (or are eager to learn thee). > > Honestly, the time for core devs (or some other elite corps of dedicated > > volunteers) to sysadmin their own machines (virtual or not) is over. > We've > > never been particularly good at this, and I don't see us getting better > or > > more efficient. > > The learning curve on git is still awful - it offers no compelling > advantages over hg, and GitHub doesn't offer any huge benefits over > BitBucket for Sphinx based documentation (ReadTheDocs works just as > well with either service). Git may well have a learning curve, but ever since I "got" it I started preferring it over hg. Too bad for BitBucket, but most people who started contributing to open source in the past 5 years already have a GitHub account. > > > Moving the CPython code and docs is not a priority, but everything else > > (PEPs, HOWTOs etc.) can be moved easily and I am in favor of moving to > > GitHub. For PEPs I've noticed that for most PEPs these days (unless the > > primary author is a core dev) the author sets up a git repo first anyway, > > and the friction of moving between such repos and the "official" repo is > a > > pain. > > Note that if folks prefer Git, BitBucket supports both. I would object > strongly to unilaterally forcing existing contributors to switch from > Mercurial to git. > What about potential new contributors? And the hg-git bridges that git fans are always referred to work in the opposite direction too... :-) -- --Guido van Rossum (on iPad) -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Sun Nov 23 07:25:55 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 23 Nov 2014 16:25:55 +1000 Subject: [Python-Dev] Move selected documentation repos to PSF BitBucket account? In-Reply-To: <3CC02307-6109-4F20-A815-BDA7CFE16AC2@stufft.io> References: <20141121102647.46e97f04@limelight.wooz.org> <19336614-0E4F-42BF-A918-1807BB7F3599@stufft.io> <504EB412-57A2-43B6-ABA1-B29D63395CD6@stufft.io> <3CC02307-6109-4F20-A815-BDA7CFE16AC2@stufft.io> Message-ID: On 23 November 2014 at 16:03, Donald Stufft wrote: >> On Nov 23, 2014, at 12:59 AM, Nick Coghlan wrote: >> >> Note that if folks prefer Git, BitBucket supports both. I would object >> strongly to unilaterally forcing existing contributors to switch from >> Mercurial to git. > > Going to all the trouble to move to an external repository and choosing the > least popular option out of the two main ones seems like a bad idea in > general. Moving repos from hg.python.org to bitbucket.org is just a matter of switching some URLs around, and changing some backend systems to cope with the new location. The end result should be to make life better for existing contributors *and* new contributors using the web UI, and be largely transparent to folks using command line tools. By contrast, proposals to switch from Mercurial to Git impose a *massive* burden on contributors that don't already know git. That significant increase in the time investment required will provide *NO* practical benefit for existing contributors (this is coming from someone that has used git and Mercurial in parallel for years - trust me, they're functionally isomorphic), and only make life marginally easier for potential new contributors (you can log in to BitBucket with your GitHub ID, and the functional isomorphism means that many folks already use tools like git-remote-hg to use the git command line to interact with the hg.python.org Mercurial repos). Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From donald at stufft.io Sun Nov 23 07:27:28 2014 From: donald at stufft.io (Donald Stufft) Date: Sun, 23 Nov 2014 01:27:28 -0500 Subject: [Python-Dev] Move selected documentation repos to PSF BitBucket account? In-Reply-To: References: <20141121102647.46e97f04@limelight.wooz.org> <19336614-0E4F-42BF-A918-1807BB7F3599@stufft.io> <504EB412-57A2-43B6-ABA1-B29D63395CD6@stufft.io> <3CC02307-6109-4F20-A815-BDA7CFE16AC2@stufft.io> Message-ID: <013F5AD9-9C9F-4E2E-A750-1C758B66B675@stufft.io> > On Nov 23, 2014, at 1:25 AM, Nick Coghlan wrote: > > On 23 November 2014 at 16:03, Donald Stufft wrote: >>> On Nov 23, 2014, at 12:59 AM, Nick Coghlan wrote: >>> >>> Note that if folks prefer Git, BitBucket supports both. I would object >>> strongly to unilaterally forcing existing contributors to switch from >>> Mercurial to git. >> >> Going to all the trouble to move to an external repository and choosing the >> least popular option out of the two main ones seems like a bad idea in >> general. > > Moving repos from hg.python.org to bitbucket.org is just a matter of > switching some URLs around, and changing some backend systems to cope > with the new location. The end result should be to make life better > for existing contributors *and* new contributors using the web UI, and > be largely transparent to folks using command line tools. > > By contrast, proposals to switch from Mercurial to Git impose a > *massive* burden on contributors that don't already know git. That > significant increase in the time investment required will provide *NO* > practical benefit for existing contributors (this is coming from > someone that has used git and Mercurial in parallel for years - trust > me, they're functionally isomorphic), and only make life marginally > easier for potential new contributors (you can log in to BitBucket > with your GitHub ID, and the functional isomorphism means that many > folks already use tools like git-remote-hg to use the git command > line to interact with the hg.python.org Mercurial repos). Yea, but then you lose out on the entire ecosystem built around Github. Like you won?t be able to run travis tests on the docs to make sure that any Pull Requests don?t silently start breaking the ability to build the docs. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA From rosuav at gmail.com Sun Nov 23 07:40:51 2014 From: rosuav at gmail.com (Chris Angelico) Date: Sun, 23 Nov 2014 17:40:51 +1100 Subject: [Python-Dev] Move selected documentation repos to PSF BitBucket account? In-Reply-To: References: <20141121102647.46e97f04@limelight.wooz.org> <19336614-0E4F-42BF-A918-1807BB7F3599@stufft.io> <504EB412-57A2-43B6-ABA1-B29D63395CD6@stufft.io> Message-ID: On Sun, Nov 23, 2014 at 5:03 PM, Wes Turner wrote: > hg imuutability is certainly a primarily attractive feature; > along with the keyring support. What exactly do you mean by immutability? Are you talking about how git allows a "force push" that can destroy data? That can be rejected in a repo's hook scripts; also, I'm fairly sure I remember reading somewhere about how to do that with hg, too. It's all bits of data inside computers; nothing's immutable. Both DVCSes pledge to make it obvious any time something is altered, not to make it impossible to alter. ChrisA From ncoghlan at gmail.com Sun Nov 23 07:49:42 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 23 Nov 2014 16:49:42 +1000 Subject: [Python-Dev] Move selected documentation repos to PSF BitBucket account? In-Reply-To: References: <20141121102647.46e97f04@limelight.wooz.org> <19336614-0E4F-42BF-A918-1807BB7F3599@stufft.io> <504EB412-57A2-43B6-ABA1-B29D63395CD6@stufft.io> Message-ID: On 23 November 2014 at 16:10, Guido van Rossum wrote: > On Saturday, November 22, 2014, Nick Coghlan wrote: >> The learning curve on git is still awful - it offers no compelling >> advantages over hg, and GitHub doesn't offer any huge benefits over >> BitBucket for Sphinx based documentation (ReadTheDocs works just as >> well with either service). > > > Git may well have a learning curve, but ever since I "got" it I started > preferring it over hg. I took the git knowledge I acquired by necessity at Red Hat and figured out how to apply it to hg. All the same features are there in hg, they're just switched off by default (mainly because the core Mercurial devs are adamant that any potentially history destroying operation in a version control system must be opt-in). In particular, the evolve extension is an impressive tool that allows history to be edited in such a way that the edits can be safely shared amongst repos. > Too bad for BitBucket, but most people who started contributing to open > source in the past 5 years already have a GitHub account. You can log into BitBucket with a GitHub account. More generally, I'm very, very disappointed to see folks so willing to abandon fellow community members for the sake of following the crowd. Perhaps we should all just abandon Python and learn Ruby or JavaScript because they're better at getting press in Silicon Valley? >> Note that if folks prefer Git, BitBucket supports both. I would object >> strongly to unilaterally forcing existing contributors to switch from >> Mercurial to git. > > What about potential new contributors? And the hg-git bridges that git fans > are always referred to work in the opposite direction too... :-) We already have lots of potential contributors (if we didn't, review bandwidth wouldn't be the bottleneck the way it is today), and the functional differences between GitHub and BitBucket from a barrier to entry perspective are so low as to be trivial. For organisations with no vested interest in BitBucket or Mercurial, sure, they may as well just go with the popular choice. Unlike most organisations, we have a vested interest in encouraging the growth of the Python ecosystem and community, and the developers of both Mercurial and BitBucket are members of that community. While we have historically done very little to date in reaching out to the Mercurial developers and seek their assistance in improving the developer guide, remedying that and providing better Mercurial usage guidelines is one of my hoped for outcomes in making it easier for others to contribute to the developer guide. By contrast, GitHub is a Ruby application, and git is a collection of C and shell scripts - they may be tools used *by* the Python community, but they are decidedly not products *of* the Python community. Given the choice between two functionally equivalent solutions, one of which lets us continue using the tools we already use (including all the associated automation), and is created by members of the same development community, I think the burden of proof falls on the folks that want to make the backwards incompatible change that the additional cost will be worth it. Moving from self-hosted Mercurial repos to externally hosted Mercurial repos is a low risk change. It reduces maintenance overhead and lowers barriers to external contribution, both without alienating existing contributors by forcing them to change their workflows. Proposing to *also* switch from Mercurial to git significantly increases the cost of the change, while providing minimal incremental benefit. Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Sun Nov 23 08:01:01 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 23 Nov 2014 17:01:01 +1000 Subject: [Python-Dev] Move selected documentation repos to PSF BitBucket account? In-Reply-To: <013F5AD9-9C9F-4E2E-A750-1C758B66B675@stufft.io> References: <20141121102647.46e97f04@limelight.wooz.org> <19336614-0E4F-42BF-A918-1807BB7F3599@stufft.io> <504EB412-57A2-43B6-ABA1-B29D63395CD6@stufft.io> <3CC02307-6109-4F20-A815-BDA7CFE16AC2@stufft.io> <013F5AD9-9C9F-4E2E-A750-1C758B66B675@stufft.io> Message-ID: On 23 November 2014 at 16:27, Donald Stufft wrote: >> On Nov 23, 2014, at 1:25 AM, Nick Coghlan wrote: >> By contrast, proposals to switch from Mercurial to Git impose a >> *massive* burden on contributors that don't already know git. That >> significant increase in the time investment required will provide *NO* >> practical benefit for existing contributors (this is coming from >> someone that has used git and Mercurial in parallel for years - trust >> me, they're functionally isomorphic), and only make life marginally >> easier for potential new contributors (you can log in to BitBucket >> with your GitHub ID, and the functional isomorphism means that many >> folks already use tools like git-remote-hg to use the git command >> line to interact with the hg.python.org Mercurial repos). > > Yea, but then you lose out on the entire ecosystem built around Github. > > Like you won?t be able to run travis tests on the docs to make sure that > any Pull Requests don?t silently start breaking the ability to build the > docs. Travis isn't the only CI system on the internet, and for pure Sphinx documentation cases, ReadTheDocs runs just as well off BitBucket as it does off GitHub. I personally think changing version control systems would be an incredibly bad idea, and consider it completely out of scope for discussions of Mercurial repo hosting arrangements. There's a lot that could be done, with much lower impact, just by changing the way we manage the existing Mercurial repos. When we can't even work out the practical details of getting those implemented, suggestions of "git/GitHub will fix it!" sound like mere wishful thinking. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From donald at stufft.io Sun Nov 23 08:13:39 2014 From: donald at stufft.io (Donald Stufft) Date: Sun, 23 Nov 2014 02:13:39 -0500 Subject: [Python-Dev] Move selected documentation repos to PSF BitBucket account? In-Reply-To: References: <20141121102647.46e97f04@limelight.wooz.org> <19336614-0E4F-42BF-A918-1807BB7F3599@stufft.io> <504EB412-57A2-43B6-ABA1-B29D63395CD6@stufft.io> Message-ID: > On Nov 23, 2014, at 1:49 AM, Nick Coghlan wrote: > > On 23 November 2014 at 16:10, Guido van Rossum wrote: >> On Saturday, November 22, 2014, Nick Coghlan wrote: >>> The learning curve on git is still awful - it offers no compelling >>> advantages over hg, and GitHub doesn't offer any huge benefits over >>> BitBucket for Sphinx based documentation (ReadTheDocs works just as >>> well with either service). >> >> >> Git may well have a learning curve, but ever since I "got" it I started >> preferring it over hg. > > I took the git knowledge I acquired by necessity at Red Hat and > figured out how to apply it to hg. All the same features are there in > hg, they're just switched off by default (mainly because the core > Mercurial devs are adamant that any potentially history destroying > operation in a version control system must be opt-in). In particular, > the evolve extension is an impressive tool that allows history to be > edited in such a way that the edits can be safely shared amongst > repos. Often times it?s there multiple times in different incompatible ways so figuring out which extension to use can be hard. They also have varying levels of compatibility with other tools. Setuptools uses ?bookmarks? instead of branches sometimes and that confuses bitbucket in some way that makes bitbucket throw errors at me when I attempt to use it. I?ve not had much luck personally with the extensions to Hg, they seem to be second class citizens in terms of workflow and I do think that them being off by default makes them worse too. > >> Too bad for BitBucket, but most people who started contributing to open >> source in the past 5 years already have a GitHub account. > > You can log into BitBucket with a GitHub account. > > More generally, I'm very, very disappointed to see folks so willing to > abandon fellow community members for the sake of following the crowd. > Perhaps we should all just abandon Python and learn Ruby or JavaScript > because they're better at getting press in Silicon Valley? I don?t think this is really fair. It?s not about press, it?s about the best tool for the job and being pragmatic. > >>> Note that if folks prefer Git, BitBucket supports both. I would object >>> strongly to unilaterally forcing existing contributors to switch from >>> Mercurial to git. >> >> What about potential new contributors? And the hg-git bridges that git fans >> are always referred to work in the opposite direction too... :-) > > We already have lots of potential contributors (if we didn't, review > bandwidth wouldn't be the bottleneck the way it is today), and the > functional differences between GitHub and BitBucket from a barrier to > entry perspective are so low as to be trivial. That?s not really true. It?s more than just ?can I log in?, potential contributors are more likely to already know how to use Github too and are more likely to not want to deal with another site. I know personally if I see a project is on bitbucket my chances of contributing to that project drop drastically, even if they are using git on bitbucket, just because I know that I?m going to get frustrated to some degree. > > For organisations with no vested interest in BitBucket or Mercurial, > sure, they may as well just go with the popular choice. > > Unlike most organisations, we have a vested interest in encouraging > the growth of the Python ecosystem and community, and the developers > of both Mercurial and BitBucket are members of that community. > > While we have historically done very little to date in reaching out to > the Mercurial developers and seek their assistance in improving the > developer guide, remedying that and providing better Mercurial usage > guidelines is one of my hoped for outcomes in making it easier for > others to contribute to the developer guide. > > By contrast, GitHub is a Ruby application, and git is a collection of > C and shell scripts - they may be tools used *by* the Python > community, but they are decidedly not products *of* the Python > community. Well CPython is a C application so perhaps we should switch to an interpreter written in Python if the goal is to always pick the thing that?s written in Python? > > Given the choice between two functionally equivalent solutions, one of > which lets us continue using the tools we already use (including all > the associated automation), and is created by members of the same > development community, I think the burden of proof falls on the folks > that want to make the backwards incompatible change that the > additional cost will be worth it. > > Moving from self-hosted Mercurial repos to externally hosted Mercurial > repos is a low risk change. It reduces maintenance overhead and lowers > barriers to external contribution, both without alienating existing > contributors by forcing them to change their workflows. > > Proposing to *also* switch from Mercurial to git significantly > increases the cost of the change, while providing minimal incremental > benefit. > > Regards, > Nick. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/donald%40stufft.io --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA From donald at stufft.io Sun Nov 23 08:14:57 2014 From: donald at stufft.io (Donald Stufft) Date: Sun, 23 Nov 2014 02:14:57 -0500 Subject: [Python-Dev] Move selected documentation repos to PSF BitBucket account? In-Reply-To: References: <20141121102647.46e97f04@limelight.wooz.org> <19336614-0E4F-42BF-A918-1807BB7F3599@stufft.io> <504EB412-57A2-43B6-ABA1-B29D63395CD6@stufft.io> <3CC02307-6109-4F20-A815-BDA7CFE16AC2@stufft.io> <013F5AD9-9C9F-4E2E-A750-1C758B66B675@stufft.io> Message-ID: <5A9CA007-3832-4931-8F05-06BD62BE4A5E@stufft.io> > On Nov 23, 2014, at 2:01 AM, Nick Coghlan wrote: > > On 23 November 2014 at 16:27, Donald Stufft wrote: >>> On Nov 23, 2014, at 1:25 AM, Nick Coghlan wrote: >>> By contrast, proposals to switch from Mercurial to Git impose a >>> *massive* burden on contributors that don't already know git. That >>> significant increase in the time investment required will provide *NO* >>> practical benefit for existing contributors (this is coming from >>> someone that has used git and Mercurial in parallel for years - trust >>> me, they're functionally isomorphic), and only make life marginally >>> easier for potential new contributors (you can log in to BitBucket >>> with your GitHub ID, and the functional isomorphism means that many >>> folks already use tools like git-remote-hg to use the git command >>> line to interact with the hg.python.org Mercurial repos). >> >> Yea, but then you lose out on the entire ecosystem built around Github. >> >> Like you won?t be able to run travis tests on the docs to make sure that >> any Pull Requests don?t silently start breaking the ability to build the >> docs. > > Travis isn't the only CI system on the internet, and for pure Sphinx > documentation cases, ReadTheDocs runs just as well off BitBucket as it > does off GitHub. Sure it?s not the only CI system, but as far as I know bitbucket doesn?t have near the integration possible with CI systems. I make a PR on github I get it tested and the merge button turns green to let me know that the tests run. Travis is popular enough that I?ve seen bitbucket projects hosting a mirror on github *just* for travis. > > I personally think changing version control systems would be an > incredibly bad idea, and consider it completely out of scope for > discussions of Mercurial repo hosting arrangements. There's a lot that > could be done, with much lower impact, just by changing the way we > manage the existing Mercurial repos. When we can't even work out the > practical details of getting those implemented, suggestions of > "git/GitHub will fix it!" sound like mere wishful thinking. > > Cheers, > Nick. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA From ncoghlan at gmail.com Sun Nov 23 08:35:04 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 23 Nov 2014 17:35:04 +1000 Subject: [Python-Dev] Move selected documentation repos to PSF BitBucket account? In-Reply-To: <5A9CA007-3832-4931-8F05-06BD62BE4A5E@stufft.io> References: <20141121102647.46e97f04@limelight.wooz.org> <19336614-0E4F-42BF-A918-1807BB7F3599@stufft.io> <504EB412-57A2-43B6-ABA1-B29D63395CD6@stufft.io> <3CC02307-6109-4F20-A815-BDA7CFE16AC2@stufft.io> <013F5AD9-9C9F-4E2E-A750-1C758B66B675@stufft.io> <5A9CA007-3832-4931-8F05-06BD62BE4A5E@stufft.io> Message-ID: On 23 November 2014 at 17:14, Donald Stufft wrote: >> On Nov 23, 2014, at 2:01 AM, Nick Coghlan wrote: >> Travis isn't the only CI system on the internet, and for pure Sphinx >> documentation cases, ReadTheDocs runs just as well off BitBucket as it >> does off GitHub. > > Sure it?s not the only CI system, but as far as I know bitbucket doesn?t > have near the integration possible with CI systems. I make a PR on github > I get it tested and the merge button turns green to let me know that > the tests run. Travis is popular enough that I?ve seen bitbucket projects > hosting a mirror on github *just* for travis. In the absence of a proposal to change version control systems (again), the lack of Mercurial hosting on GitHub makes it rather a moot point. Given that we can barely muster up any enthusiasm for rehosting *without* changing version control systems (and the number of CI systems that integrate with hg.python.org repos other than the main CPython one is exactly zero), any proposal that involves doing even *more* work seems doubly doomed. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From donald at stufft.io Sun Nov 23 09:11:20 2014 From: donald at stufft.io (Donald Stufft) Date: Sun, 23 Nov 2014 03:11:20 -0500 Subject: [Python-Dev] Move selected documentation repos to PSF BitBucket account? In-Reply-To: References: <20141121102647.46e97f04@limelight.wooz.org> <19336614-0E4F-42BF-A918-1807BB7F3599@stufft.io> <504EB412-57A2-43B6-ABA1-B29D63395CD6@stufft.io> <3CC02307-6109-4F20-A815-BDA7CFE16AC2@stufft.io> <013F5AD9-9C9F-4E2E-A750-1C758B66B675@stufft.io> <5A9CA007-3832-4931-8F05-06BD62BE4A5E@stufft.io> Message-ID: <35957465-A878-4864-9473-9F68B9556387@stufft.io> > On Nov 23, 2014, at 2:35 AM, Nick Coghlan wrote: > > On 23 November 2014 at 17:14, Donald Stufft wrote: >>> On Nov 23, 2014, at 2:01 AM, Nick Coghlan wrote: >>> Travis isn't the only CI system on the internet, and for pure Sphinx >>> documentation cases, ReadTheDocs runs just as well off BitBucket as it >>> does off GitHub. >> >> Sure it?s not the only CI system, but as far as I know bitbucket doesn?t >> have near the integration possible with CI systems. I make a PR on github >> I get it tested and the merge button turns green to let me know that >> the tests run. Travis is popular enough that I?ve seen bitbucket projects >> hosting a mirror on github *just* for travis. > > In the absence of a proposal to change version control systems > (again), the lack of Mercurial hosting on GitHub makes it rather a > moot point. Given that we can barely muster up any enthusiasm for > rehosting *without* changing version control systems (and the number > of CI systems that integrate with hg.python.org repos other than the > main CPython one is exactly zero), any proposal that involves doing > even *more* work seems doubly doomed. > I?d volunteer to do the work to get the PEPs, and possibly other repositories onto Github if we so decided to do so. Don?t let the lack of volunteer stop that because I will find the time to do it if need be. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA From tjreedy at udel.edu Sun Nov 23 11:08:37 2014 From: tjreedy at udel.edu (Terry Reedy) Date: Sun, 23 Nov 2014 05:08:37 -0500 Subject: [Python-Dev] PEP 479: Change StopIteration handling inside generators In-Reply-To: References: <20141121174731.2a245ff5@fsol> Message-ID: On 11/22/2014 5:23 PM, Chris Angelico wrote: > On Sun, Nov 23, 2014 at 8:03 AM, Ron Adam wrote: >>> Making comprehensions work more like generator expressions >>> would, IMO, imply making the same change to all for loops: having a >>> StopIteration raised by the body of the loop quietly terminate the >>> loop. >> >> >> I'm not suggesting making any changes to generator expressions or for loops >> at all. They would continue to work like they currently do. > > But if you're suggesting making list comps react to StopIteration > raised by their primary expressions, then to maintain the > correspondence between a list comp and its expanded form, for loops > would have to change too. Or should that correspondence be broken, in > that single-expression loop constructs become semantically different > from statement loop constructs? The 2.x correspondence between list comp and for loops was *already broken* in 3.0 by moving execution to a new function to prevent name binding in comprehension from leaking the way they did in 2.x. We did not change for loops to match. The following, which is an example of equivalence in 2.7, raises in 3.4 because 'itertools' is not defined. def binder(ns, name, ob): ns[name] = ob for mod in ['sys']: binder(locals(), mod, __import__(mod)) print(sys) [binder(locals(), mod, __import__(mod)) for mod in ['itertools']] print(itertools) Ceasing to leak *any* local bindings in 3.0 broke equivalence and code depending on such bindings. Ceasing to leak StopIteration would break equivalence a bit more, but only a bit, and code dependent on such leakage, which I expect is extremely rare. -- Terry Jan Reedy From ncoghlan at gmail.com Sun Nov 23 12:18:18 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 23 Nov 2014 21:18:18 +1000 Subject: [Python-Dev] Move selected documentation repos to PSF BitBucket account? In-Reply-To: <35957465-A878-4864-9473-9F68B9556387@stufft.io> References: <20141121102647.46e97f04@limelight.wooz.org> <19336614-0E4F-42BF-A918-1807BB7F3599@stufft.io> <504EB412-57A2-43B6-ABA1-B29D63395CD6@stufft.io> <3CC02307-6109-4F20-A815-BDA7CFE16AC2@stufft.io> <013F5AD9-9C9F-4E2E-A750-1C758B66B675@stufft.io> <5A9CA007-3832-4931-8F05-06BD62BE4A5E@stufft.io> <35957465-A878-4864-9473-9F68B9556387@stufft.io> Message-ID: On 23 Nov 2014 18:11, "Donald Stufft" wrote: > > On Nov 23, 2014, at 2:35 AM, Nick Coghlan wrote: > > > > In the absence of a proposal to change version control systems > > (again), the lack of Mercurial hosting on GitHub makes it rather a > > moot point. Given that we can barely muster up any enthusiasm for > > rehosting *without* changing version control systems (and the number > > of CI systems that integrate with hg.python.org repos other than the > > main CPython one is exactly zero), any proposal that involves doing > > even *more* work seems doubly doomed. > > > > I?d volunteer to do the work to get the PEPs, and possibly other repositories > onto Github if we so decided to do so. Don?t let the lack of volunteer stop > that because I will find the time to do it if need be. It's the other way around: someone would have to volunteer to make the case that switching version control systems will actually help in any way whatsoever with the core reviewer bandwidth problem. We do *not* have a shortage of people wanting to contribute. While ongoing outreach efforts are essential to promote increased diversity in the contributor base and to counter natural attrition, there is currently no major problem that needs solving on that front. CPython is high profile enough that folks are willing to do battle with the current complicated contribution process, so we're already one of the most active open source projects in the world, in *spite* of the problems with the existing workflow. This high level of activity also takes place in spite of the fact that direct corporate investment in paid contributions to the CPython runtime currently don't really reflect the key role that CPython holds in the enterprise Linux, OpenStack, data analysis, and education ecosystems (to name a few where I personally have some level of involvement either personally or through work). Where I believe we *do* have a problem is with failing to make the best possible use of core developer contribution time, as typos and other minor fixes to project (rather than product) documentation are managed through the same offline patch & upload process as the reference interpreter itself. (There are other issues as well, but they're out of scope for the current discussion, which is about the support repos, rather than CPython - the same problem exists there, but the solution is unlikely to be as straightforward). Patches getting held up in the review queue for weeks or months is a *huge* barrier to contribution, as it prevents the formation of the positive feedback cycle where having a contribution accepted feels good, so folks are more likely to want to contribute again. In that context, the useful features that a richer repo hosting application can potentially offer are single-click acceptance and merging of documentation changes that aren't directly linked to a specific CPython version, as well as the ability to make trivial fixes to that documentation (like fixing a typo) entirely online. Those features are readily accessible without changing the underlying version control system (whether self-hosted through Kallithea or externally hosted through BitBucket or RhodeCode). Thus the folks that want to change the version control system need to make the case that doing so will provide additional benefits that *can't* be obtained in a less disruptive way. >From my perspective, swapping out Mercurial for git achieves exactly nothing in terms of alleviating the review bottleneck (since the core developers that strongly prefer the git UI will already be using an adapter), and is in fact likely to make it worse by putting the greatest burden in adapting to the change on the folks that are already under the greatest time pressure. It's also worth keeping in mind that changing the underlying VCS means changing *all* the automation scripts, rather than just updating the configuration settings to reflect a new hosting URL. Orchestrating this kind of infrastructure enhancement for Red Hat *is* my day job, and you almost always want to go for the lowest impact, lowest risk approach that will alleviate the bottleneck you're worried about while changing the smallest possible number of elements in the overall workflow management system. That underlying calculation doesn't really change much even when the units shift from budget dollars to volunteers' time and energy. Regards, Nick. > > --- > Donald Stufft > PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA > -------------- next part -------------- An HTML attachment was scrubbed... URL: From stephen at xemacs.org Sun Nov 23 14:58:02 2014 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Sun, 23 Nov 2014 22:58:02 +0900 Subject: [Python-Dev] Move selected documentation repos to PSF BitBucket account? In-Reply-To: References: <20141121102647.46e97f04@limelight.wooz.org> <19336614-0E4F-42BF-A918-1807BB7F3599@stufft.io> <504EB412-57A2-43B6-ABA1-B29D63395CD6@stufft.io> <3CC02307-6109-4F20-A815-BDA7CFE16AC2@stufft.io> Message-ID: <87mw7irpcl.fsf@uwakimon.sk.tsukuba.ac.jp> Nick Coghlan writes: > By contrast, proposals to switch from Mercurial to Git impose a > *massive* burden on contributors that don't already know git. Let's not get carried away here. The *massive* burden is the moaning from git-haters (is there a 12-step program for them?) Agreed, learning any new VCS is a burden, and git seems harder than most -- but even RMS is swallowing git for Emacs, despite the fact that the "g" in "git" doesn't stand for "GNU". > with your GitHub ID, and the functional isomorphism [git <-> hg] I agree that there is an isomorphism, but the philosophical restrictions on hg functionality are quite annoying. I do things like "git reset" and "git commit --amend" a lot. I tend to commit before getting coffee, but I don't want that in the permanent record -- "git rebase --interactive" is a good buddy of mine. And so on. hg *deliberately* gets in the way of such workflows (although perhaps it's not as hard to "opt in" to the necessary features as it used to be). Nevertheless, I tend to agree with you that moving to Github now is a big move. I just think you should avoid the "git dox suck" argument. I'd also like to mention that in my opinion the network externalities argument is being misused. True, "everybody" has a github account, and even if they don't, their little sisters do. So what? There are big network externalities involved, but that doesn't necessarily mean that Bitbucket can't catch up, and most projects I know have branches hosted on both Bitbucket and Github (and often SourceForge or Launchpad as well). People who really prefer one or the other for practical reasons can usually use them without too much difficulty, wherever the "official" repo may be hosted. More likely to have a clear outcome, the main network externality *we* should be concerned with is *within* the Python ecosystem. *If* the "big" projects whose core members tend not to hang out here so much (NumPy, SciPy, Twisted, Django, ...) are vastly more likely to be found on Bitbucket than Github (or vice versa), I think that's potentially much more important than the little sisters with github accounts. I also agree with you that the facts that Mercurial is a Python application, and I guess so is most of Github, are important. But again, let's not get carried away. "Although practicality beats purity" applies here. The Github features are very attractive; we need to look at how useful they will be to contributors before deciding that the warm fuzzy Python community is more important. Finally, Guido is right: Github, and to a somewhat lesser extent Bitbucket and Google Code have gotten code hosting right, compared to the *forges. The people who maintain infrastructure for Python are the kind of contributor who would probably spend more time on reviewing and mentoring and release engineering if they weren't maintaining infrastructure as far as I can see. If the infrastructure maintenance can be delegated (it's not clear to me that it can), that would be a big factor. From solipsis at pitrou.net Sun Nov 23 15:08:59 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sun, 23 Nov 2014 15:08:59 +0100 Subject: [Python-Dev] Move selected documentation repos to PSF BitBucket account? References: <20141121102647.46e97f04@limelight.wooz.org> <19336614-0E4F-42BF-A918-1807BB7F3599@stufft.io> <504EB412-57A2-43B6-ABA1-B29D63395CD6@stufft.io> <3CC02307-6109-4F20-A815-BDA7CFE16AC2@stufft.io> <87mw7irpcl.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <20141123150859.2aaa7e96@fsol> On Sun, 23 Nov 2014 22:58:02 +0900 "Stephen J. Turnbull" wrote: > Nick Coghlan writes: > > > By contrast, proposals to switch from Mercurial to Git impose a > > *massive* burden on contributors that don't already know git. > > Let's not get carried away here. The *massive* burden is the moaning > from git-haters (is there a 12-step program for them?) Is there a 12-step program for people who can't help commenting on the workflow of core developers and contributors while they don't seem to ever contribute anything concrete? Seriously, stick to the topics where you have some semblant of legitimacy. Whether we choose to use git or hg you have absolutely no authority to comment about. Regards Antoine. From benjamin at python.org Sun Nov 23 16:00:37 2014 From: benjamin at python.org (Benjamin Peterson) Date: Sun, 23 Nov 2014 10:00:37 -0500 Subject: [Python-Dev] Move selected documentation repos to PSF BitBucket account? In-Reply-To: References: <20141121102647.46e97f04@limelight.wooz.org> <19336614-0E4F-42BF-A918-1807BB7F3599@stufft.io> <504EB412-57A2-43B6-ABA1-B29D63395CD6@stufft.io> <3CC02307-6109-4F20-A815-BDA7CFE16AC2@stufft.io> Message-ID: <1416754837.1264594.194374273.5EF6A80A@webmail.messagingengine.com> On Sun, Nov 23, 2014, at 01:25, Nick Coghlan wrote: > On 23 November 2014 at 16:03, Donald Stufft wrote: > >> On Nov 23, 2014, at 12:59 AM, Nick Coghlan wrote: > >> > >> Note that if folks prefer Git, BitBucket supports both. I would object > >> strongly to unilaterally forcing existing contributors to switch from > >> Mercurial to git. > > > > Going to all the trouble to move to an external repository and choosing the > > least popular option out of the two main ones seems like a bad idea in > > general. > > Moving repos from hg.python.org to bitbucket.org is just a matter of > switching some URLs around, and changing some backend systems to cope > with the new location. The end result should be to make life better > for existing contributors *and* new contributors using the web UI, and > be largely transparent to folks using command line tools. > > By contrast, proposals to switch from Mercurial to Git impose a > *massive* burden on contributors that don't already know git. But how many people are there who will have this massive burden imposed on them? I imagine there's few among us who haven't had to learn git for our job or other projects. From ethan at stoneleaf.us Sun Nov 23 16:11:50 2014 From: ethan at stoneleaf.us (Ethan Furman) Date: Sun, 23 Nov 2014 07:11:50 -0800 Subject: [Python-Dev] PEP 479: Change StopIteration handling inside generators In-Reply-To: References: <20141121174731.2a245ff5@fsol> Message-ID: <5471F936.2050109@stoneleaf.us> On 11/22/2014 08:53 PM, Guido van Rossum wrote: > > In order to save everyone's breath, I am *accepting* the proposal of PEP > 479. Excellent. Chris, thank you for your time, effort, and thoroughness in championing this PEP. -- ~Ethan~ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: OpenPGP digital signature URL: From rosuav at gmail.com Sun Nov 23 16:25:34 2014 From: rosuav at gmail.com (Chris Angelico) Date: Mon, 24 Nov 2014 02:25:34 +1100 Subject: [Python-Dev] PEP 479: Change StopIteration handling inside generators In-Reply-To: <5471F936.2050109@stoneleaf.us> References: <20141121174731.2a245ff5@fsol> <5471F936.2050109@stoneleaf.us> Message-ID: On Mon, Nov 24, 2014 at 2:11 AM, Ethan Furman wrote: > On 11/22/2014 08:53 PM, Guido van Rossum wrote: >> >> In order to save everyone's breath, I am *accepting* the proposal of PEP >> 479. > > Excellent. > > Chris, thank you for your time, effort, and thoroughness in championing this PEP. > Thank you, it's nice to have a successful one to counterbalance the "failure" of PEP 463. (Which, incidentally, never actually got a resolution. It's still showing up as 'Draft' status.) I'm actually quite impressed with how on-topic the discussion threads went. They didn't ramble nearly as much as I thought they would. Thank you to all who participated, contributed suggestions, complained about the existing text, and generally worked hard to make sure I had more than enough material to draw on! Particular thanks to Guido, pushing changes through and being patient with my mistakes in markup, and adding content directly to the document. ChrisA From p.f.moore at gmail.com Sun Nov 23 16:38:50 2014 From: p.f.moore at gmail.com (Paul Moore) Date: Sun, 23 Nov 2014 15:38:50 +0000 Subject: [Python-Dev] PEP 479: Change StopIteration handling inside generators In-Reply-To: References: <20141121174731.2a245ff5@fsol> <5471F936.2050109@stoneleaf.us> Message-ID: On 23 November 2014 at 15:25, Chris Angelico wrote: > Thank you, it's nice to have a successful one to counterbalance the > "failure" of PEP 463. (Which, incidentally, never actually got a > resolution. It's still showing up as 'Draft' status.) I think it's worth pointing out that both this and PEP 463 were complex an contentious topics, and the discussion in both cases was well-focused and civil. I don't think the fact that you were the author of both PEPs is unrelated to this. Thanks for taking on *both* of these PEPs and handling them so well. Paul From guido at python.org Sun Nov 23 17:55:50 2014 From: guido at python.org (Guido van Rossum) Date: Sun, 23 Nov 2014 08:55:50 -0800 Subject: [Python-Dev] Move selected documentation repos to PSF BitBucket account? In-Reply-To: References: <20141121102647.46e97f04@limelight.wooz.org> <19336614-0E4F-42BF-A918-1807BB7F3599@stufft.io> <504EB412-57A2-43B6-ABA1-B29D63395CD6@stufft.io> Message-ID: On Sat, Nov 22, 2014 at 10:49 PM, Nick Coghlan wrote: > More generally, I'm very, very disappointed to see folks so willing to > abandon fellow community members for the sake of following the crowd. > Perhaps we should all just abandon Python and learn Ruby or JavaScript > because they're better at getting press in Silicon Valley? That's a really low blow, Nick. I think these are the facts: - Hg/Git are equivalent in functionality (at least to the extent that the difference can't be used to force a decision), and ditto for BitBucket/GitHub, with one crucial exception (see below) - We're currently using Hg for most projects under the PSF umbrella (however, there's https://github.com/python/pythondotorg) - Moving from Hg to Git is a fair amount of one-time work (converting repos) and is inconvenient to core devs who aren't already used to Git (learning a new workflow) - Most newer third-party projects are already on GitHub - GitHub is way more popular than BitBucket and slated for long-term success But here's the kicker for me: **A DVCS repo is a social network, so it matters in a functional way what everyone else is using.** So I give you that if you want a quick move into the modern world, while keeping the older generation of core devs happy (not counting myself :-), BitBucket has the lowest cost of entry. But I strongly believe that if we want to do the right thing for the long term, we should switch to GitHub. I promise you that once the pain of the switch is over you will feel much better about it. I am also convinced that we'll get more contributions this way. Note: I am not (yet) proposing we switch CPython itself. Switching it would be a lot of work, and it is specifically out of scope for this discussion. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Sun Nov 23 17:55:04 2014 From: brett at python.org (Brett Cannon) Date: Sun, 23 Nov 2014 16:55:04 +0000 Subject: [Python-Dev] Move selected documentation repos to PSF BitBucket account? References: <20141121102647.46e97f04@limelight.wooz.org> <19336614-0E4F-42BF-A918-1807BB7F3599@stufft.io> <504EB412-57A2-43B6-ABA1-B29D63395CD6@stufft.io> <3CC02307-6109-4F20-A815-BDA7CFE16AC2@stufft.io> <013F5AD9-9C9F-4E2E-A750-1C758B66B675@stufft.io> <5A9CA007-3832-4931-8F05-06BD62BE4A5E@stufft.io> <35957465-A878-4864-9473-9F68B9556387@stufft.io> Message-ID: On Sun Nov 23 2014 at 6:18:46 AM Nick Coghlan wrote: > > On 23 Nov 2014 18:11, "Donald Stufft" wrote: > > > On Nov 23, 2014, at 2:35 AM, Nick Coghlan wrote: > > > > > > > In the absence of a proposal to change version control systems > > > (again), the lack of Mercurial hosting on GitHub makes it rather a > > > moot point. Given that we can barely muster up any enthusiasm for > > > rehosting *without* changing version control systems (and the number > > > of CI systems that integrate with hg.python.org repos other than the > > > main CPython one is exactly zero), any proposal that involves doing > > > even *more* work seems doubly doomed. > > > > > > > I?d volunteer to do the work to get the PEPs, and possibly other > repositories > > onto Github if we so decided to do so. Don?t let the lack of volunteer > stop > > that because I will find the time to do it if need be. > > It's the other way around: someone would have to volunteer to make the > case that switching version control systems will actually help in any way > whatsoever with the core reviewer bandwidth problem. > > We do *not* have a shortage of people wanting to contribute. While ongoing > outreach efforts are essential to promote increased diversity in the > contributor base and to counter natural attrition, there is currently no > major problem that needs solving on that front. CPython is high profile > enough that folks are willing to do battle with the current complicated > contribution process, so we're already one of the most active open source > projects in the world, in *spite* of the problems with the existing > workflow. > The *immediate* problem is making it easier to accept contributions from people. The long-term, never-ending problem is making the whole process of submitting a patch and getting it accepted as easy as possible for everyone involved, contributor and committer alike. If the goal is to make it so we can accept PRs for easier patch acceptances then that can be accomplished on either Bitbucket or GitHub. But if we want to make it easier for people to make contributions then GitHub is arguably better than Bitbucket, whether it's through familiarity of GitHub for most people thanks to other FLOSS projects or from the superior tooling around GitHub (both the platform itself and the ecosystem that has sprung up around it). > This high level of activity also takes place in spite of the fact that > direct corporate investment in paid contributions to the CPython runtime > currently don't really reflect the key role that CPython holds in the > enterprise Linux, OpenStack, data analysis, and education ecosystems (to > name a few where I personally have some level of involvement either > personally or through work). > > Where I believe we *do* have a problem is with failing to make the best > possible use of core developer contribution time, as typos and other minor > fixes to project (rather than product) documentation are managed through > the same offline patch & upload process as the reference interpreter > itself. (There are other issues as well, but they're out of scope for the > current discussion, which is about the support repos, rather than CPython - > the same problem exists there, but the solution is unlikely to be as > straightforward). > > Patches getting held up in the review queue for weeks or months is a > *huge* barrier to contribution, as it prevents the formation of the > positive feedback cycle where having a contribution accepted feels good, so > folks are more likely to want to contribute again. > > In that context, the useful features that a richer repo hosting > application can potentially offer are single-click acceptance and merging > of documentation changes that aren't directly linked to a specific CPython > version, as well as the ability to make trivial fixes to that documentation > (like fixing a typo) entirely online. > > Those features are readily accessible without changing the underlying > version control system (whether self-hosted through Kallithea or externally > hosted through BitBucket or RhodeCode). Thus the folks that want to change > the version control system need to make the case that doing so will provide > additional benefits that *can't* be obtained in a less disruptive way. > I guess my question is who and what is going to be disrupted if we go with Guido's suggestion of switching to GitHub for code hosting? Contributors won't be disrupted at all since most people are more familiar with GitHub vs. Bitbucket (how many times have we all heard the fact someone has even learned Mercurial just to contribute to Python?). Core developers might be based on some learned workflow, but I'm willing to bet we all know git at this point (and for those of us who still don't like it, myself included, there are GUI apps to paper over it or hg-git for those that prefer a CLI). Our infrastructure will need to be updated, but how much of it is that hg-specific short of the command to checkout out the repo? Obviously Bitbucket is much more minor by simply updating just a URL, but changing `hg clone` to `git clone` isn't crazy either. Georg, Antoine, or Benjamin can point out if I'm wrong on this, maybe Donald or someone in the infrastructure committee. Probably the biggest thing I can think of that would need updating is our commit hooks. Once again Georg, Antoine, or Benjamin could say how difficult it would be to update those hooks. > From my perspective, swapping out Mercurial for git achieves exactly > nothing in terms of alleviating the review bottleneck (since the core > developers that strongly prefer the git UI will already be using an > adapter), and is in fact likely to make it worse by putting the greatest > burden in adapting to the change on the folks that are already under the > greatest time pressure. > That's not entirely true. If you are pushing a PR shift in our patch acceptance workflow then Bitbucket vs. GitHub isn't fundamentally any different in terms of benefit, and I would honestly argue that GitHub's PR experience is better. IOW either platform is of equal benefit. > It's also worth keeping in mind that changing the underlying VCS means > changing *all* the automation scripts, rather than just updating the > configuration settings to reflect a new hosting URL. > What are the automation scripts there are that would require updating? I would like to a list and to have the difficulty of moving them mentioned to know what the impact would be. > Orchestrating this kind of infrastructure enhancement for Red Hat *is* my > day job, and you almost always want to go for the lowest impact, lowest > risk approach that will alleviate the bottleneck you're worried about while > changing the smallest possible number of elements in the overall workflow > management system. > Sure, but I would never compare our infrastructure needs to Red Hat. =) You also have to be conservative in order to minimize downtown and impact for cost reasons. As an open source project we don't have those kinds of worry; we just have to worry about keeping everyone happy. > That underlying calculation doesn't really change much even when the units > shift from budget dollars to volunteers' time and energy. > So here is what I want to know to focus this discussion: First, what new workflow are you proposing regardless of repo hosting provider? Are you proposing we maintain just mirrors and update the devguide to tell people to fork on the hosting provider, make their changes, generate a patch (which can be as simple as telling people how find the raw diff on the hosting provider), and then upload the patch the issue tracker just like today? Are you going farther and saying we have people send PRs on the hosting site, have them point to their PR in the issue tracker, and then we accept PRs (I'm going on the assumption we are not dropping our issue tracker)? Second, to properly gauge the impact of switching from git to hg from an infrastructure perspective, what automation scripts do we have and how difficult would it be to update them to use git instead of hg? This is necessary simply to know where we would need to update URLs, let alone change in DVCS. Third, do our release managers care about hg vs. git strongly? They probably use the DVCS the most directly and at a lower level by necessity compared to anyone else. Fourth, do any core developers feel strongly about not using GitHub? Now please notice I said "GitHub" and not "git"; I think the proper way to frame this whole discussion is we are deciding if we want to switch to Bitbucket or GitHub who provide a low-level API for their version control storage service through hg or git, respectively. I personally dislike git, but I really like GitHub and I don't even notice git there since I use GitHub's OS X app; as I said, I view this as choosing a platform and not the underlying DVCS as I have happily chosen to access the GitHub hosting service through an app that is not git (it's like accessing a web app through it's web page or its REST API). At least for me, until we get a clear understanding of what workflow changes we are asking for both contributors and core developers and exactly what work would be necessary to update our infrastructure for either Bitbucket or GitHub we can't really have a reasonable discussion that isn't going to be full of guessing. And I'm still in support no matter what of breaking out the HOWTOs and the tutorial into their own repos for easier updating (having to update the Python porting HOWTO in three branches is a pain when it should be consistent across Python releases). -Brett > Regards, > Nick. > > > > > --- > > Donald Stufft > > PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ > brett%40python.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Sun Nov 23 18:39:08 2014 From: donald at stufft.io (Donald Stufft) Date: Sun, 23 Nov 2014 12:39:08 -0500 Subject: [Python-Dev] Move selected documentation repos to PSF BitBucket account? In-Reply-To: References: <20141121102647.46e97f04@limelight.wooz.org> <19336614-0E4F-42BF-A918-1807BB7F3599@stufft.io> <504EB412-57A2-43B6-ABA1-B29D63395CD6@stufft.io> <3CC02307-6109-4F20-A815-BDA7CFE16AC2@stufft.io> <013F5AD9-9C9F-4E2E-A750-1C758B66B675@stufft.io> <5A9CA007-3832-4931-8F05-06BD62BE4A5E@stufft.io> <35957465-A878-4864-9473-9F68B9556387@stufft.io> Message-ID: > On Nov 23, 2014, at 11:55 AM, Brett Cannon wrote: > > > This high level of activity also takes place in spite of the fact that direct corporate investment in paid contributions to the CPython runtime currently don't really reflect the key role that CPython holds in the enterprise Linux, OpenStack, data analysis, and education ecosystems (to name a few where I personally have some level of involvement either personally or through work). > > Where I believe we *do* have a problem is with failing to make the best possible use of core developer contribution time, as typos and other minor fixes to project (rather than product) documentation are managed through the same offline patch & upload process as the reference interpreter itself. (There are other issues as well, but they're out of scope for the current discussion, which is about the support repos, rather than CPython - the same problem exists there, but the solution is unlikely to be as straightforward). > > Patches getting held up in the review queue for weeks or months is a *huge* barrier to contribution, as it prevents the formation of the positive feedback cycle where having a contribution accepted feels good, so folks are more likely to want to contribute again. > > In that context, the useful features that a richer repo hosting application can potentially offer are single-click acceptance and merging of documentation changes that aren't directly linked to a specific CPython version, as well as the ability to make trivial fixes to that documentation (like fixing a typo) entirely online. > > Those features are readily accessible without changing the underlying version control system (whether self-hosted through Kallithea or externally hosted through BitBucket or RhodeCode). Thus the folks that want to change the version control system need to make the case that doing so will provide additional benefits that *can't* be obtained in a less disruptive way. > > > I guess my question is who and what is going to be disrupted if we go with Guido's suggestion of switching to GitHub for code hosting? Contributors won't be disrupted at all since most people are more familiar with GitHub vs. Bitbucket (how many times have we all heard the fact someone has even learned Mercurial just to contribute to Python?). Core developers might be based on some learned workflow, but I'm willing to bet we all know git at this point (and for those of us who still don't like it, myself included, there are GUI apps to paper over it or hg-git for those that prefer a CLI). Our infrastructure will need to be updated, but how much of it is that hg-specific short of the command to checkout out the repo? Obviously Bitbucket is much more minor by simply updating just a URL, but changing `hg clone` to `git clone` isn't crazy either. Georg, Antoine, or Benjamin can point out if I'm wrong on this, maybe Donald or someone in the infrastructure committee. > > Probably the biggest thing I can think of that would need updating is our commit hooks. Once again Georg, Antoine, or Benjamin could say how difficult it would be to update those hooks. Commit hooks don?t work on Github or bitbucket anyways. So either way they?d need rewritten to support whichever platform?s web hooks instead. > > From my perspective, swapping out Mercurial for git achieves exactly nothing in terms of alleviating the review bottleneck (since the core developers that strongly prefer the git UI will already be using an adapter), and is in fact likely to make it worse by putting the greatest burden in adapting to the change on the folks that are already under the greatest time pressure. > > > That's not entirely true. If you are pushing a PR shift in our patch acceptance workflow then Bitbucket vs. GitHub isn't fundamentally any different in terms of benefit, and I would honestly argue that GitHub's PR experience is better. IOW either platform is of equal benefit. Right, I would argue that Github is at least a little bit better because of the commit status API. It?s *really* nice to see right on a PR if it breaks anything or not before you merge it without ever having to run a thing manually. That isn?t something that only Travis can do so if we want some other system to do it that?s totally possible. > > It's also worth keeping in mind that changing the underlying VCS means changing *all* the automation scripts, rather than just updating the configuration settings to reflect a new hosting URL. > > > What are the automation scripts there are that would require updating? I would like to a list and to have the difficulty of moving them mentioned to know what the impact would be. Off the top of my head for the docs I know of https://github.com/python/docsbuild-scripts/blob/master/build_docs.py which will need to be updated to git clone instead of hg clone. For the record here is the list of commit hooks on the devguide: https://bpaste.net/show/3b151d0ddcb4 but as I said above, those will need to be written on either platform because to my knowledge none of the hosted sites allow you to upload arbitrary commit hooks that get executed on their machines. I?m struggling to think of what other automation we might have for documentation. There?s no buildbot on anything that isn?t CPython. > > Orchestrating this kind of infrastructure enhancement for Red Hat *is* my day job, and you almost always want to go for the lowest impact, lowest risk approach that will alleviate the bottleneck you're worried about while changing the smallest possible number of elements in the overall workflow management system. > > > Sure, but I would never compare our infrastructure needs to Red Hat. =) You also have to be conservative in order to minimize downtown and impact for cost reasons. As an open source project we don't have those kinds of worry; we just have to worry about keeping everyone happy. I think the other consideration here is that when your workforce is volunteers and not people you?re paying to do things you have to be cognizant of what the volunteers are willing to do as well. Nick has said he?s having problem getting interested parties who are motivated to make the move however I am willing to put in the work to move to Github, I?m not however willing to do it for Bitbucket because it?s frankly about as bad to me personally as the current arrangement is. This isn?t to say there isn?t someone who wouldn?t do the work for bitbucket of course, just going off Nick saying he?s having problems finding someone willing to do that work. > > That underlying calculation doesn't really change much even when the units shift from budget dollars to volunteers' time and energy. > > So here is what I want to know to focus this discussion: > > First, what new workflow are you proposing regardless of repo hosting provider? Are you proposing we maintain just mirrors and update the devguide to tell people to fork on the hosting provider, make their changes, generate a patch (which can be as simple as telling people how find the raw diff on the hosting provider), and then upload the patch the issue tracker just like today? Are you going farther and saying we have people send PRs on the hosting site, have them point to their PR in the issue tracker, and then we accept PRs (I'm going on the assumption we are not dropping our issue tracker)? > > Second, to properly gauge the impact of switching from git to hg from an infrastructure perspective, what automation scripts do we have and how difficult would it be to update them to use git instead of hg? This is necessary simply to know where we would need to update URLs, let alone change in DVCS. > > Third, do our release managers care about hg vs. git strongly? They probably use the DVCS the most directly and at a lower level by necessity compared to anyone else. > > Fourth, do any core developers feel strongly about not using GitHub? Now please notice I said "GitHub" and not "git"; I think the proper way to frame this whole discussion is we are deciding if we want to switch to Bitbucket or GitHub who provide a low-level API for their version control storage service through hg or git, respectively. I personally dislike git, but I really like GitHub and I don't even notice git there since I use GitHub's OS X app; as I said, I view this as choosing a platform and not the underlying DVCS as I have happily chosen to access the GitHub hosting service through an app that is not git (it's like accessing a web app through it's web page or its REST API). > > At least for me, until we get a clear understanding of what workflow changes we are asking for both contributors and core developers and exactly what work would be necessary to update our infrastructure for either Bitbucket or GitHub we can't really have a reasonable discussion that isn't going to be full of guessing. > > And I'm still in support no matter what of breaking out the HOWTOs and the tutorial into their own repos for easier updating (having to update the Python porting HOWTO in three branches is a pain when it should be consistent across Python releases). I agree breaking these out is a great idea in general. It?s similar to the packaging guide which is at packaging.python.org and is maintained outside of the CPython repository (also on Github fwiw). > > -Brett > > Regards, > Nick. > > > > > --- > > Donald Stufft > > PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA > > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/brett%40python.org --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: From ethan at stoneleaf.us Sun Nov 23 17:17:00 2014 From: ethan at stoneleaf.us (Ethan Furman) Date: Sun, 23 Nov 2014 08:17:00 -0800 Subject: [Python-Dev] PEP 479: Change StopIteration handling inside generators In-Reply-To: <289664E5-58CA-4964-8DB6-20E43F3717BC@gmail.com> References: <20141121174731.2a245ff5@fsol> <289664E5-58CA-4964-8DB6-20E43F3717BC@gmail.com> Message-ID: <5472087C.4090205@stoneleaf.us> On 11/22/2014 12:30 PM, Raymond Hettinger wrote: Pre-PEP 479: ----------- > def middleware_generator(source_generator): > it = source_generator() > input_value = next(it) > output_value = do_something_interesting(input_value) > yield output_value Post-PEP 479: ------------ > def middleware_generator(source_generator): > it = source_generator() > try: > input_value = next(it) > except StopIteration: > return # This causes StopIteration to be reraised > output_value = do_something_interesting(input_value) > yield output_value While I am in favor of PEP 479, and I have to agree with Raymond that this isn't pretty. Currently, next() accepts an argument of what to return if the iterator is empty. Can we enhance that in some way so that the overall previous behavior could be retained? Something like: def middleware_generator(source_generator): it = source_generator() input_value = next(it, gen_exit=True) # or exc_type=GeneratorExit ? output_value = do_something_interesting(input_value) yield output_value Then, if the iterator is empty, instead of raising StopIteration, or returning some value that would then have to be checked, it could raise some other exception that is understood to be normal generator termination. -- ~Ethan~ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: OpenPGP digital signature URL: From ethan at stoneleaf.us Sun Nov 23 16:35:41 2014 From: ethan at stoneleaf.us (Ethan Furman) Date: Sun, 23 Nov 2014 07:35:41 -0800 Subject: [Python-Dev] Move selected documentation repos to PSF BitBucket account? In-Reply-To: References: <20141121102647.46e97f04@limelight.wooz.org> <19336614-0E4F-42BF-A918-1807BB7F3599@stufft.io> <504EB412-57A2-43B6-ABA1-B29D63395CD6@stufft.io> Message-ID: <5471FECD.3080906@stoneleaf.us> On 11/22/2014 11:13 PM, Donald Stufft wrote: >> On Nov 23, 2014, at 1:49 AM, Nick Coghlan wrote: >> >> I took the git knowledge I acquired by necessity at Red Hat and >> figured out how to apply it to hg. All the same features are there in >> hg, they're just switched off by default (mainly because the core >> Mercurial devs are adamant that any potentially history destroying >> operation in a version control system must be opt-in). If you could find the time to write up something about that I'm sure it would be helpful. :) >> We already have lots of potential contributors (if we didn't, review >> bandwidth wouldn't be the bottleneck the way it is today), and the >> functional differences between GitHub and BitBucket from a barrier to >> entry perspective are so low as to be trivial. > > That?s not really true. It?s more than just ?can I log in?, potential > contributors are more likely to already know how to use Github too and > are more likely to not want to deal with another site. I know personally > if I see a project is on bitbucket my chances of contributing to that > project drop drastically, even if they are using git on bitbucket, > just because I know that I?m going to get frustrated to some degree. I feel the same way, only in reverse. I've learned hg, and to a lesser extent bitbucket, but have not learned git nor github, and would rather not (available bandwidth and all that). >> Moving from self-hosted Mercurial repos to externally hosted Mercurial >> repos is a low risk change. It reduces maintenance overhead and lowers >> barriers to external contribution, both without alienating existing >> contributors by forcing them to change their workflows. >> >> Proposing to *also* switch from Mercurial to git significantly >> increases the cost of the change, while providing minimal incremental >> benefit. Whatever our personal feelings of hg vs git, and bitbucket vs github, that makes sense. -- ~Ethan~ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: OpenPGP digital signature URL: From rymg19 at gmail.com Sun Nov 23 19:03:48 2014 From: rymg19 at gmail.com (Ryan) Date: Sun, 23 Nov 2014 12:03:48 -0600 Subject: [Python-Dev] Move selected documentation repos to PSF BitBucket account? In-Reply-To: References: <20141121102647.46e97f04@limelight.wooz.org> <19336614-0E4F-42BF-A918-1807BB7F3599@stufft.io> <504EB412-57A2-43B6-ABA1-B29D63395CD6@stufft.io> Message-ID: <73f643e5-bd78-4dbc-8a26-335e91735d16@email.android.com> I can agree with most of these points. Some more things to consider: - Git is 20x faster than Hg (that's 99% of the reason I switched and hate using Darcs) - People attached to Hg can use Hg-Git; I've used it several times with nice results. It can also be used to easily convert Hg repos to Git (I was using it to maintain a Pygments mirror, but got bored) - GitHub is much nicer than BitBucket. I usually look for a git mirror of a BitBucket repo since I hate browsing on BB - Even Plan9Port moved from Hg to GitHub a few days ago - Some Google devs even host their projects on GitHub (not Google Code) Thank goodness no one has mentioned CodePlex. BTW, I frown on projects that choose something largely inferior over a better choice just because "one is written in X language." Now, if python.org was written in Ruby, that would just be sad, but certain things just don't matter. Like DVCS's. Guido van Rossum wrote: >On Sat, Nov 22, 2014 at 10:49 PM, Nick Coghlan >wrote: > >> More generally, I'm very, very disappointed to see folks so willing >to >> abandon fellow community members for the sake of following the crowd. >> Perhaps we should all just abandon Python and learn Ruby or >JavaScript >> because they're better at getting press in Silicon Valley? > > >That's a really low blow, Nick. > >I think these are the facts: > >- Hg/Git are equivalent in functionality (at least to the extent that >the >difference can't be used to force a decision), and ditto for >BitBucket/GitHub, with one crucial exception (see below) > >- We're currently using Hg for most projects under the PSF umbrella >(however, there's https://github.com/python/pythondotorg) > >- Moving from Hg to Git is a fair amount of one-time work (converting >repos) and is inconvenient to core devs who aren't already used to Git >(learning a new workflow) > >- Most newer third-party projects are already on GitHub > >- GitHub is way more popular than BitBucket and slated for long-term >success > >But here's the kicker for me: **A DVCS repo is a social network, so it >matters in a functional way what everyone else is using.** > >So I give you that if you want a quick move into the modern world, >while >keeping the older generation of core devs happy (not counting myself >:-), >BitBucket has the lowest cost of entry. But I strongly believe that if >we >want to do the right thing for the long term, we should switch to >GitHub. I >promise you that once the pain of the switch is over you will feel much >better about it. I am also convinced that we'll get more contributions >this >way. > >Note: I am not (yet) proposing we switch CPython itself. Switching it >would >be a lot of work, and it is specifically out of scope for this >discussion. > >-- >--Guido van Rossum (python.org/~guido) > > >------------------------------------------------------------------------ > >_______________________________________________ >Python-Dev mailing list >Python-Dev at python.org >https://mail.python.org/mailman/listinfo/python-dev >Unsubscribe: >https://mail.python.org/mailman/options/python-dev/rymg19%40gmail.com -- Sent from my Android phone with K-9 Mail. Please excuse my brevity. Check out my website: http://kirbyfan64.github.io/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From ethan at stoneleaf.us Sun Nov 23 19:05:15 2014 From: ethan at stoneleaf.us (Ethan Furman) Date: Sun, 23 Nov 2014 10:05:15 -0800 Subject: [Python-Dev] Move selected documentation repos to PSF BitBucket account? In-Reply-To: References: <504EB412-57A2-43B6-ABA1-B29D63395CD6@stufft.io> <3CC02307-6109-4F20-A815-BDA7CFE16AC2@stufft.io> <013F5AD9-9C9F-4E2E-A750-1C758B66B675@stufft.io> <5A9CA007-3832-4931-8F05-06BD62BE4A5E@stufft.io> <35957465-A878-4864-9473-9F68B9556387@stufft.io> Message-ID: <547221DB.6030206@stoneleaf.us> On 11/23/2014 08:55 AM, Brett Cannon wrote: > > Sure, but I would never compare our infrastructure needs to Red Hat. =) You > also have to be conservative in order to minimize downtown and impact for > cost reasons. As an open source project we don't have those kinds of worry; > we just have to worry about keeping everyone happy. Minimizing downtime and impact is important for us, too. The Python job board has now been down for nine months -- that's hardly good PR. -- ~Ethan~ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: OpenPGP digital signature URL: From ethan at stoneleaf.us Sun Nov 23 19:07:50 2014 From: ethan at stoneleaf.us (Ethan Furman) Date: Sun, 23 Nov 2014 10:07:50 -0800 Subject: [Python-Dev] Move selected documentation repos to PSF BitBucket account? In-Reply-To: References: <504EB412-57A2-43B6-ABA1-B29D63395CD6@stufft.io> <3CC02307-6109-4F20-A815-BDA7CFE16AC2@stufft.io> <013F5AD9-9C9F-4E2E-A750-1C758B66B675@stufft.io> <5A9CA007-3832-4931-8F05-06BD62BE4A5E@stufft.io> <35957465-A878-4864-9473-9F68B9556387@stufft.io> Message-ID: <54722276.909@stoneleaf.us> On 11/23/2014 08:55 AM, Brett Cannon wrote: > > Fourth, do any core developers feel strongly about not using GitHub? Dous GitHub support hg? If not, I am strongly opposed. -- ~Ethan~ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: OpenPGP digital signature URL: From brett at python.org Sun Nov 23 19:08:34 2014 From: brett at python.org (Brett Cannon) Date: Sun, 23 Nov 2014 18:08:34 +0000 Subject: [Python-Dev] Move selected documentation repos to PSF BitBucket account? References: <504EB412-57A2-43B6-ABA1-B29D63395CD6@stufft.io> <3CC02307-6109-4F20-A815-BDA7CFE16AC2@stufft.io> <013F5AD9-9C9F-4E2E-A750-1C758B66B675@stufft.io> <5A9CA007-3832-4931-8F05-06BD62BE4A5E@stufft.io> <35957465-A878-4864-9473-9F68B9556387@stufft.io> <547221DB.6030206@stoneleaf.us> Message-ID: On Sun Nov 23 2014 at 1:06:18 PM Ethan Furman wrote: > On 11/23/2014 08:55 AM, Brett Cannon wrote: > > > > Sure, but I would never compare our infrastructure needs to Red Hat. =) > You > > also have to be conservative in order to minimize downtown and impact for > > cost reasons. As an open source project we don't have those kinds of > worry; > > we just have to worry about keeping everyone happy. > > Minimizing downtime and impact is important for us, too. The Python job > board has now been down for nine months -- > that's hardly good PR. > That has nothing to do with downtime and everything to do with volunteer time. My point about "downtime" is that if I can't commit to the cpython repo for a day it isn't going to cause me to freak out or cost anyone thousands of dollars or more in revenue. -Brett > > -- > ~Ethan~ > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ > brett%40python.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Sun Nov 23 19:14:48 2014 From: brett at python.org (Brett Cannon) Date: Sun, 23 Nov 2014 18:14:48 +0000 Subject: [Python-Dev] Move selected documentation repos to PSF BitBucket account? References: <504EB412-57A2-43B6-ABA1-B29D63395CD6@stufft.io> <3CC02307-6109-4F20-A815-BDA7CFE16AC2@stufft.io> <013F5AD9-9C9F-4E2E-A750-1C758B66B675@stufft.io> <5A9CA007-3832-4931-8F05-06BD62BE4A5E@stufft.io> <35957465-A878-4864-9473-9F68B9556387@stufft.io> <54722276.909@stoneleaf.us> Message-ID: On Sun Nov 23 2014 at 1:08:58 PM Ethan Furman wrote: > On 11/23/2014 08:55 AM, Brett Cannon wrote: > > > > Fourth, do any core developers feel strongly about not using GitHub? > > Dous GitHub support hg? If not, I am strongly opposed. > Depends on what you mean by "support". If you mean natively, then no. If you mean "I want more of a hg CLI" then you can get that with http://hg-git.github.io/ . And can I just say this is all bringing back "wonderful" flashbacks of the SourceForge to our own infrastructure move as well as the svn to hg move. =/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From ron3200 at gmail.com Sun Nov 23 19:28:12 2014 From: ron3200 at gmail.com (Ron Adam) Date: Sun, 23 Nov 2014 12:28:12 -0600 Subject: [Python-Dev] PEP 479: Change StopIteration handling inside generators In-Reply-To: References: <20141121174731.2a245ff5@fsol> Message-ID: On 11/23/2014 04:08 AM, Terry Reedy wrote: > On 11/22/2014 5:23 PM, Chris Angelico wrote: >> On Sun, Nov 23, 2014 at 8:03 AM, Ron Adam wrote: >>>> Making comprehensions work more like generator expressions >>>> would, IMO, imply making the same change to all for loops: having a >>>> StopIteration raised by the body of the loop quietly terminate the >>>> loop. >>> >>> >>> I'm not suggesting making any changes to generator expressions or for loops >>> at all. They would continue to work like they currently do. >> >> But if you're suggesting making list comps react to StopIteration >> raised by their primary expressions, then to maintain the >> correspondence between a list comp and its expanded form, for loops >> would have to change too. Or should that correspondence be broken, in >> that single-expression loop constructs become semantically different >> from statement loop constructs? > > The 2.x correspondence between list comp and for loops was *already broken* > in 3.0 by moving execution to a new function to prevent name binding in > comprehension from leaking the way they did in 2.x. We did not change for > loops to match. > > The following, which is an example of equivalence in 2.7, raises in 3.4 > because 'itertools' is not defined. > > def binder(ns, name, ob): > ns[name] = ob > > for mod in ['sys']: > binder(locals(), mod, __import__(mod)) > print(sys) > > [binder(locals(), mod, __import__(mod)) for mod in ['itertools']] > print(itertools) > > Ceasing to leak *any* local bindings in 3.0 broke equivalence and code > depending on such bindings. Ceasing to leak StopIteration would break > equivalence a bit more, but only a bit, and code dependent on such leakage, > which I expect is extremely rare. I think they would be rare too. With the passage of the PEP, it will change what is different about them once it's in full effect. The stop hack won't work in both, and you may get a RuntimeError in generator expressions where you would get StopIteration in list-comps. (Is this correct?) Cheers, Ron From brett at python.org Sun Nov 23 19:31:37 2014 From: brett at python.org (Brett Cannon) Date: Sun, 23 Nov 2014 18:31:37 +0000 Subject: [Python-Dev] Move selected documentation repos to PSF BitBucket account? References: <20141121102647.46e97f04@limelight.wooz.org> <19336614-0E4F-42BF-A918-1807BB7F3599@stufft.io> <504EB412-57A2-43B6-ABA1-B29D63395CD6@stufft.io> Message-ID: On Sun Nov 23 2014 at 11:56:49 AM Guido van Rossum wrote: > On Sat, Nov 22, 2014 at 10:49 PM, Nick Coghlan wrote: > >> More generally, I'm very, very disappointed to see folks so willing to >> abandon fellow community members for the sake of following the crowd. >> Perhaps we should all just abandon Python and learn Ruby or JavaScript >> because they're better at getting press in Silicon Valley? > > > That's a really low blow, Nick. > > I think these are the facts: > > - Hg/Git are equivalent in functionality (at least to the extent that the > difference can't be used to force a decision), and ditto for > BitBucket/GitHub, with one crucial exception (see below) > > - We're currently using Hg for most projects under the PSF umbrella > (however, there's https://github.com/python/pythondotorg) > > - Moving from Hg to Git is a fair amount of one-time work (converting > repos) and is inconvenient to core devs who aren't already used to Git > (learning a new workflow) > > - Most newer third-party projects are already on GitHub > > - GitHub is way more popular than BitBucket and slated for long-term > success > > But here's the kicker for me: **A DVCS repo is a social network, so it > matters in a functional way what everyone else is using.** > > So I give you that if you want a quick move into the modern world, while > keeping the older generation of core devs happy (not counting myself :-), > BitBucket has the lowest cost of entry. But I strongly believe that if we > want to do the right thing for the long term, we should switch to GitHub. I > promise you that once the pain of the switch is over you will feel much > better about it. I am also convinced that we'll get more contributions this > way. > > Note: I am not (yet) proposing we switch CPython itself. Switching it > would be a lot of work, and it is specifically out of scope for this > discussion. > If we want to test the complexity of moving something to GitHub then probably the best repo to use is the peps one: - Very few people directly use that repo (you and me alone could probably manage it if we enforced all changes through a PR as I could then do approvals from work instead of having to wait until I was at home with an hg checkout available) - It's used on the website so it would require updating infrastructure - It isn't a lot of overhead to tell people who email the peps mailing list to "please send a pull request through GitHub" since it isn't tracked in the issue tracker anyway - There is a benefit of setting up some CI integration to know when a PR is actually incorrectly formatted And if people want to test the impact of Bitbucket we could do it for something like the HOWTOs as that too involves infrastructure but is not used by a lot of people. In fact we can make it known we are piloting this approach on Bitbucket and see what kind of contributions it triggers (ditto for the peps since I'm sure some people will want to send in typo PRs and such). IOW I don't see why we can't pilot this between now and April for the language summit and see what difference it all makes so we can have an informed discussion in Montreal with more than 4 full months of experience under our belts. Then we can discuss Bitbucket vs. GitHub, docs vs. everything moving vs. nothing, etc. That was this stops all being conjecture and more about seeing if there is an actual impact. -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Sun Nov 23 19:44:20 2014 From: brett at python.org (Brett Cannon) Date: Sun, 23 Nov 2014 18:44:20 +0000 Subject: [Python-Dev] Move selected documentation repos to PSF BitBucket account? References: <20141121102647.46e97f04@limelight.wooz.org> <19336614-0E4F-42BF-A918-1807BB7F3599@stufft.io> <504EB412-57A2-43B6-ABA1-B29D63395CD6@stufft.io> Message-ID: On Sun Nov 23 2014 at 1:31:36 PM Brett Cannon wrote: > On Sun Nov 23 2014 at 11:56:49 AM Guido van Rossum > wrote: > >> On Sat, Nov 22, 2014 at 10:49 PM, Nick Coghlan >> wrote: >> >>> More generally, I'm very, very disappointed to see folks so willing to >>> abandon fellow community members for the sake of following the crowd. >>> Perhaps we should all just abandon Python and learn Ruby or JavaScript >>> because they're better at getting press in Silicon Valley? >> >> >> That's a really low blow, Nick. >> >> I think these are the facts: >> >> - Hg/Git are equivalent in functionality (at least to the extent that the >> difference can't be used to force a decision), and ditto for >> BitBucket/GitHub, with one crucial exception (see below) >> >> - We're currently using Hg for most projects under the PSF umbrella >> (however, there's https://github.com/python/pythondotorg) >> >> - Moving from Hg to Git is a fair amount of one-time work (converting >> repos) and is inconvenient to core devs who aren't already used to Git >> (learning a new workflow) >> >> - Most newer third-party projects are already on GitHub >> >> - GitHub is way more popular than BitBucket and slated for long-term >> success >> >> But here's the kicker for me: **A DVCS repo is a social network, so it >> matters in a functional way what everyone else is using.** >> >> So I give you that if you want a quick move into the modern world, while >> keeping the older generation of core devs happy (not counting myself :-), >> BitBucket has the lowest cost of entry. But I strongly believe that if we >> want to do the right thing for the long term, we should switch to GitHub. I >> promise you that once the pain of the switch is over you will feel much >> better about it. I am also convinced that we'll get more contributions this >> way. >> >> Note: I am not (yet) proposing we switch CPython itself. Switching it >> would be a lot of work, and it is specifically out of scope for this >> discussion. >> > > If we want to test the complexity of moving something to GitHub then > probably the best repo to use is the peps one: > > - Very few people directly use that repo (you and me alone could > probably manage it if we enforced all changes through a PR as I could then > do approvals from work instead of having to wait until I was at home with > an hg checkout available) > - It's used on the website so it would require updating infrastructure > - It isn't a lot of overhead to tell people who email the peps mailing > list to "please send a pull request through GitHub" since it isn't tracked > in the issue tracker anyway > - There is a benefit of setting up some CI integration to know when a > PR is actually incorrectly formatted > > And if people want to test the impact of Bitbucket we could do it for > something like the HOWTOs as that too involves infrastructure but is not > used by a lot of people. In fact we can make it known we are piloting this > approach on Bitbucket and see what kind of contributions it triggers (ditto > for the peps since I'm sure some people will want to send in typo PRs and > such). > Actually the tutorial might be best to measure ease of contribution for people on Bitbucket since we can also ask people who use the tutorial to test out pointing people to the Bitbucket repo if people want to send a PR. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ethan at stoneleaf.us Sun Nov 23 20:41:07 2014 From: ethan at stoneleaf.us (Ethan Furman) Date: Sun, 23 Nov 2014 11:41:07 -0800 Subject: [Python-Dev] Move selected documentation repos to PSF BitBucket account? In-Reply-To: References: <3CC02307-6109-4F20-A815-BDA7CFE16AC2@stufft.io> <013F5AD9-9C9F-4E2E-A750-1C758B66B675@stufft.io> <5A9CA007-3832-4931-8F05-06BD62BE4A5E@stufft.io> <35957465-A878-4864-9473-9F68B9556387@stufft.io> <54722276.909@stoneleaf.us> Message-ID: <54723853.8060606@stoneleaf.us> On 11/23/2014 10:14 AM, Brett Cannon wrote: > On Sun Nov 23 2014 at 1:08:58 PM Ethan Furman wrote: >> >> Dous GitHub support hg? If not, I am strongly opposed. >> > > Depends on what you mean by "support". If you mean natively, then no. If > you mean "I want more of a hg CLI" then you can get that with > http://hg-git.github.io/ . Well, if somebody documents it, I suppose I can follow along. ;) > And can I just say this is all bringing back "wonderful" flashbacks of the > SourceForge to our own infrastructure move as well as the svn to hg move. =/ My apologies. Change can be hard. My concern is that we will end up with multiple different workflows depending on which part of Python we're working on, which will lead to more time spent learning more about how to do it instead of doing it, and more time spent recovering from errors because of the differences. -- ~Ethan~ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: OpenPGP digital signature URL: From ethan at stoneleaf.us Sun Nov 23 20:43:18 2014 From: ethan at stoneleaf.us (Ethan Furman) Date: Sun, 23 Nov 2014 11:43:18 -0800 Subject: [Python-Dev] Move selected documentation repos to PSF BitBucket account? In-Reply-To: References: <20141121102647.46e97f04@limelight.wooz.org> <19336614-0E4F-42BF-A918-1807BB7F3599@stufft.io> <504EB412-57A2-43B6-ABA1-B29D63395CD6@stufft.io> Message-ID: <547238D6.1040607@stoneleaf.us> On 11/23/2014 10:31 AM, Brett Cannon wrote: > > If we want to test the complexity of moving something to GitHub then > probably the best repo to use is the peps one: > And if people want to test the impact of Bitbucket we could do it for > something like the HOWTOs as that too involves infrastructure but is not > used by a lot of people. +1 -- ~Ethan~ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: OpenPGP digital signature URL: From rymg19 at gmail.com Sun Nov 23 20:51:53 2014 From: rymg19 at gmail.com (Ryan Gonzalez) Date: Sun, 23 Nov 2014 13:51:53 -0600 Subject: [Python-Dev] Move selected documentation repos to PSF BitBucket account? In-Reply-To: <54723853.8060606@stoneleaf.us> References: <3CC02307-6109-4F20-A815-BDA7CFE16AC2@stufft.io> <013F5AD9-9C9F-4E2E-A750-1C758B66B675@stufft.io> <5A9CA007-3832-4931-8F05-06BD62BE4A5E@stufft.io> <35957465-A878-4864-9473-9F68B9556387@stufft.io> <54722276.909@stoneleaf.us> <54723853.8060606@stoneleaf.us> Message-ID: On Sun, Nov 23, 2014 at 1:41 PM, Ethan Furman wrote: > On 11/23/2014 10:14 AM, Brett Cannon wrote: > > On Sun Nov 23 2014 at 1:08:58 PM Ethan Furman > wrote: > >> > >> Dous GitHub support hg? If not, I am strongly opposed. > >> > > > > Depends on what you mean by "support". If you mean natively, then no. If > > you mean "I want more of a hg CLI" then you can get that with > > http://hg-git.github.io/ . > > Well, if somebody documents it, I suppose I can follow along. ;). I haven't used it in a while, but it was something like this (after it's installed): - Put the git repo url in .hgrc: [paths] default=git+https://github.com/my_user/my_repo.git [auth] github.prefix = git+https://github.com/my_user/my_repo.git github.username = my_user github.password = my_password Unless you put the login info there, it won't work (for some reason). Then, whenever you're going to push, first do: hg bookmark -r default master I'd love to test this right now, but I'm on Chrome OS... > > > And can I just say this is all bringing back "wonderful" flashbacks of > the > > SourceForge to our own infrastructure move as well as the svn to hg > move. =/ > > My apologies. Change can be hard. > > My concern is that we will end up with multiple different workflows > depending on which part of Python we're working on, > which will lead to more time spent learning more about how to do it > instead of doing it, and more time spent recovering > from errors because of the differences. > > -- > ~Ethan~ > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/rymg19%40gmail.com > > -- Ryan If anybody ever asks me why I prefer C++ to C, my answer will be simple: "It's becauseslejfp23(@#Q*(E*EIdc-SEGFAULT. Wait, I don't think that was nul-terminated." Personal reality distortion fields are immune to contradictory evidence. - srean Check out my website: http://kirbyfan64.github.io/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From g.brandl at gmx.net Sun Nov 23 21:03:41 2014 From: g.brandl at gmx.net (Georg Brandl) Date: Sun, 23 Nov 2014 21:03:41 +0100 Subject: [Python-Dev] Move selected documentation repos to PSF BitBucket account? In-Reply-To: References: <504EB412-57A2-43B6-ABA1-B29D63395CD6@stufft.io> <3CC02307-6109-4F20-A815-BDA7CFE16AC2@stufft.io> <013F5AD9-9C9F-4E2E-A750-1C758B66B675@stufft.io> <5A9CA007-3832-4931-8F05-06BD62BE4A5E@stufft.io> <35957465-A878-4864-9473-9F68B9556387@stufft.io> Message-ID: On 11/23/2014 05:55 PM, Brett Cannon wrote: > I guess my question is who and what is going to be disrupted if we go with > Guido's suggestion of switching to GitHub for code hosting? Contributors won't > be disrupted at all since most people are more familiar with GitHub vs. > Bitbucket (how many times have we all heard the fact someone has even learned > Mercurial just to contribute to Python?). Core developers might be based on some > learned workflow, but I'm willing to bet we all know git at this point (and for > those of us who still don't like it, myself included, there are GUI apps to > paper over it or hg-git for those that prefer a CLI). Our infrastructure will > need to be updated, but how much of it is that hg-specific short of the command > to checkout out the repo? Obviously Bitbucket is much more minor by simply > updating just a URL, but changing `hg clone` to `git clone` isn't crazy either. > Georg, Antoine, or Benjamin can point out if I'm wrong on this, maybe Donald or > someone in the infrastructure committee. Well, since "most people" already know git this part is probably not a big deal, although we'd have to consider alienating core developers who are not git-savvy. > Probably the biggest thing I can think of that would need updating is our commit > hooks. Once again Georg, Antoine, or Benjamin could say how difficult it would > be to update those hooks. There are two categories of hooks we use: hooks that run before a push is accepted, and hooks that run afterwards. The latter are not a concern, since anything that GH/BB doesn't support can be run as a web service on python.org infrastructure, which then gets POST requests from the hosting platforms. These are * tracker update * IRC notification * sending email to python-checkins * triggering buildbot The more problematic category are pre-push hooks. We use them for checking and rejecting commits with * disallowed branches * non-conformant whitespace * wrong EOL style * multiple heads per named branch As far as I know, neither GH nor BB support such hooks, since they are highly project-specific. However, they are only used in cpython and related repositories, so that doesn't concern migration of doc-only repos. Sure, you can let the CI run the checks, but that doesn't prohibit merging and is circumvented by direct pushes to the repository that don't go through the PR system. (And please don't make me as a coredev open a PR for every change.) > From my perspective, swapping out Mercurial for git achieves exactly nothing > in terms of alleviating the review bottleneck (since the core developers > that strongly prefer the git UI will already be using an adapter), and is in > fact likely to make it worse by putting the greatest burden in adapting to > the change on the folks that are already under the greatest time pressure. > > > That's not entirely true. If you are pushing a PR shift in our patch acceptance > workflow then Bitbucket vs. GitHub isn't fundamentally any different in terms of > benefit, and I would honestly argue that GitHub's PR experience is better. IOW > either platform is of equal benefit. In my opinion, scattering repos over github, bitbucket and hg.python.org is even less friendly to contributors than a centralized place. (We are already approaching this, with pydotorg and infrastructure repos on github.) So I'm going to add some thoughts here about migrating the main CPython to git+hub. We have to consider how well our branch workflow works with the PR workflow. There's no gain in the ability to easily merge a PR to one branch via github when the subsequent merge of 3.x to default/master requires a local pull/push cycle, as well as the 2.x backport. As far as I know, you'd have to open a pull/merge request yourself and instantly merge it, except if there are conflicts between branches, in which case you are again forced to go local. I don't need to mention that this doesn't work well when someone makes a concurrent commit to the source branch in the meantime. And I don't think you'd want to force people to submit 3 pull requests for the individual branches. The next point is that there is no easy way to change the target branch of a pull request (on github or bitbucket). People will usually make patches against the master branch unless told differently explicitly, which means that the pull request will also be against the master branch. Which means, close the PR, ask submitter to resubmit against 3.x branch, or do it yourself. > It's also worth keeping in mind that changing the underlying VCS means > changing *all* the automation scripts, rather than just updating the > configuration settings to reflect a new hosting URL. > > > What are the automation scripts there are that would require updating? I would > like to a list and to have the difficulty of moving them mentioned to know what > the impact would be. Compiling that list is likely the most difficult step of updating them. > So here is what I want to know to focus this discussion: [...] > Third, do our release managers care about hg vs. git strongly? They probably use > the DVCS the most directly and at a lower level by necessity compared to anyone > else. I wouldn't care about git or hg. What I *might* care about is the low-level access to the repository on hg.python.org, mainly because of the hooks. > Fourth, do any core developers feel strongly about not using GitHub? Now please > notice I said "GitHub" and not "git"; I think the proper way to frame this whole > discussion is we are deciding if we want to switch to Bitbucket or GitHub who > provide a low-level API for their version control storage service through hg or > git, respectively. I personally dislike git, but I really like GitHub and I > don't even notice git there since I use GitHub's OS X app; as I said, I view > this as choosing a platform and not the underlying DVCS as I have happily chosen > to access the GitHub hosting service through an app that is not git (it's like > accessing a web app through it's web page or its REST API). Github is a nice platform, but many of its features would not be relevant for the cpython repo (e.g. issue tracker or wiki). The usefulness of pull requests is dubious, see above. The "but it is much easier to contribute simple changes" argument is a bit simplified: for any nontrivial patch, the time spent on working out the code should outweight time spent with "hg diff" or "click on pull request". And while Travis CI is nice, running relevant tests locally is *much* quicker than waiting for a full test suite run on a virtualized testing machine. As for typo fixes, the world does not end when some typos aren't fixed. Anyway, for the docs we have an explicit offer to send anything, patch or just suggestion, to docs at python.org, and people do make use of it. No github account even required. > At least for me, until we get a clear understanding of what workflow changes we > are asking for both contributors and core developers and exactly what work would > be necessary to update our infrastructure for either Bitbucket or GitHub we > can't really have a reasonable discussion that isn't going to be full of guessing. As for svn->hg, a PEP and a champion would be necessary, this time with the possibility of being rejected. > And I'm still in support no matter what of breaking out the HOWTOs and the > tutorial into their own repos for easier updating (having to update the Python > porting HOWTO in three branches is a pain when it should be consistent across > Python releases). I see no problem with that, provided there's a cronjob that syncs the version in Doc/ to the external version reasonably often. cheers, Georg From g.brandl at gmx.net Sun Nov 23 21:14:05 2014 From: g.brandl at gmx.net (Georg Brandl) Date: Sun, 23 Nov 2014 21:14:05 +0100 Subject: [Python-Dev] Move selected documentation repos to PSF BitBucket account? In-Reply-To: <73f643e5-bd78-4dbc-8a26-335e91735d16@email.android.com> References: <20141121102647.46e97f04@limelight.wooz.org> <19336614-0E4F-42BF-A918-1807BB7F3599@stufft.io> <504EB412-57A2-43B6-ABA1-B29D63395CD6@stufft.io> <73f643e5-bd78-4dbc-8a26-335e91735d16@email.android.com> Message-ID: On 11/23/2014 07:03 PM, Ryan wrote: > I can agree with most of these points. Some more things to consider: > > - Git is 20x faster than Hg (that's 99% of the reason I switched and hate using > Darcs) You won't get much traction with this argument around here. As long as there aren't specific complaints about slowness (which probably could be fixed by a discussion on mercurial-devel), Hg is "fast enough", just like Python is "fast enough" compared to C. (Otherwise, why bother.) > - GitHub is much nicer than BitBucket. I usually look for a git mirror of a > BitBucket repo since I hate browsing on BB I agree, Github's UI is nicer than BB. It is also true that most features that are introduced on BB are "inspired" by GH. > - Even Plan9Port moved from Hg to GitHub a few days ago Projects move all the time. Why is this one special? > - Some Google devs even host their projects on GitHub (not Google Code) > > Thank goodness no one has mentioned CodePlex. > > BTW, I frown on projects that choose something largely inferior over a better > choice just because "one is written in X language." Now, if python.org > was written in Ruby, that would just be sad, but certain > things just don't matter. Like DVCS's. You can read up PEP 374 for the reasons for Hg. The main reasons against git were lacking Windows support (which probably is good enough now) and developer preference. Georg From mark at hotpy.org Sun Nov 23 21:18:31 2014 From: mark at hotpy.org (Mark Shannon) Date: Sun, 23 Nov 2014 20:18:31 +0000 Subject: [Python-Dev] Please reconsider PEP 479. Message-ID: <54724117.20408@hotpy.org> Hi, I have serious concerns about this PEP, and would ask you to reconsider it. [ Very short summary: Generators are not the problem. It is the naive use of next() in an iterator that is the problem. (Note that all the examples involve calls to next()). Change next() rather than fiddling with generators. ] I have five main concerns with PEP 479. 1. Is the problem, as stated by the PEP, really the problem at all? 2. The proposed solution does not address the underlying problem. 3. It breaks a fundamental aspect of generators, that they are iterators. 4. This will be a hindrance to porting code from Python 2 to Python 3. 5. The behaviour of next() is not considered, even though it is the real cause of the problem (if there is a problem). 1. The PEP states that "The interaction of generators and StopIteration is currently somewhat surprising, and can conceal obscure bugs." I don't believe that to be the case; if someone knows what StopIteration is and how it is used, then the interaction is entirely as expected. I believe the naive use of next() in an iterator to be the underlying problem. The interaction of generators and next() is just a special case of this. StopIteration is not a normal exception, indicating a problem, rather it exists to signal exhaustion of an iterator. However, next() raises StopIteration for an exhausted iterator, which really is an error. Any iterator code (generator or __next__ method) that calls next() treats the StopIteration as a normal exception and propogates it. The controlling loop then interprets StopIteration as a signal to stop and thus stops. *The problem is the implicit shift from signal to error and back to signal.* 2. The proposed solution does not address this issue at all, but rather legislates against generators raising StopIteration. 3. Generators and the iterator protocol were introduced in Python 2.2, 13 years ago. For all of that time the iterator protocol has been defined by the __iter__(), next()/__next__() methods and the use of StopIteration to terminate iteration. Generators are a way to write iterators without the clunkiness of explicit __iter__() and next()/__next__() methods, but have always obeyed the same protocol as all other iterators. This has allowed code to rewritten from one form to the other whenever desired. Do not forget that despite the addition of the send() and throw() methods and their secondary role as coroutines, generators have primarily always been a clean and elegant way of writing iterators. 4. Porting from Python 2 to Python 3 seems to be hard enough already. 5. I think I've already covered this in the other points, but to reiterate (excuse the pun): Calling next() on an exhausted iterator is, I would suggest, a logical error. However, next() raises StopIteration which is really a signal to the controlling loop. The fault is with next() raising StopIteration. Generators raising StopIteration is not the problem. It also worth noting that calling next() is the only place a StopIteration exception is likely to occur outside of the iterator protocol. An example ---------- Consider a function to return the value from a set with a single member. def value_from_singleton(s): if len(s) < 2: #Intentional error here (should be len(s) == 1) return next(iter(s)) raise ValueError("Not a singleton") Now suppose we pass an empty set to value_from_singleton(s), then we get a StopIteration exception, which is a bit weird, but not too bad. However it is when we use it in a generator (or in the __next__ method of an iterator) that we get a serious problem. Currently the iterator appears to be exhausted early, which is wrong. However, with the proposed change we get RuntimeError("generator raised StopIteration") raised, which is also wrong, just in a different way. Solutions --------- My preferred "solution" is to do nothing except improving the documentation of next(). Explain that it can raise StopIteration which, if allowed to propogate can cause premature exhaustion of an iterator. If something must be done then I would suggest changing the behaviour of next() for an exhausted iterator. Rather than raise StopIteration it should raise ValueError (or IndexError?). Also, it might be worth considering making StopIteration inherit from BaseException, rather than Exception. Cheers, Mark. P.S. 5 days seems a rather short time to respond to a PEP. Could we make it at least a couple of weeks in the future, or better still specify a closing date for comments. From donald at stufft.io Sun Nov 23 21:38:28 2014 From: donald at stufft.io (Donald Stufft) Date: Sun, 23 Nov 2014 15:38:28 -0500 Subject: [Python-Dev] Move selected documentation repos to PSF BitBucket account? In-Reply-To: References: <504EB412-57A2-43B6-ABA1-B29D63395CD6@stufft.io> <3CC02307-6109-4F20-A815-BDA7CFE16AC2@stufft.io> <013F5AD9-9C9F-4E2E-A750-1C758B66B675@stufft.io> <5A9CA007-3832-4931-8F05-06BD62BE4A5E@stufft.io> <35957465-A878-4864-9473-9F68B9556387@stufft.io> Message-ID: <415502D5-3489-4C66-A84D-73C0998A6FFC@stufft.io> > On Nov 23, 2014, at 3:03 PM, Georg Brandl wrote: > > The next point is that there is no easy way to change the target branch of > a pull request (on github or bitbucket). People will usually make patches > against the master branch unless told differently explicitly, which means > that the pull request will also be against the master branch. Which means, > close the PR, ask submitter to resubmit against 3.x branch, or do it > yourself. This in particular is not really true on Github at least. By default PRs are made against whichever branch you have configured as the default branch in the Github UI. This does default to master but it doesn?t have to be, for instance pip has this configured for the develop branch. Although I think this specific point is moot because if things were on Git it?d probably make the most sense to have the default integration branch be ``master`` just like for the Hg repos they use default. Even if someone makes a PR against the wrong branch, it ?degrades? into essentially the same UX as you have now, you can turn a PR into a patch by adding a .patch or .diff to the end of the PR URL and then you have a patch file. In additional github makes it easy to check out PRs directly with only a minor bit of one time configuration in your local clone. I can checkout any PR by doing ``git checkout pr/1`` (replacing 1 with whatever the number is). Honestly the worst part about someone sending a PR to the wrong branch is it degrades into the same terrible UX that *every* patch has to go through on a hg.python.org repository right now. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA From brett at python.org Sun Nov 23 21:42:53 2014 From: brett at python.org (Brett Cannon) Date: Sun, 23 Nov 2014 20:42:53 +0000 Subject: [Python-Dev] Move selected documentation repos to PSF BitBucket account? References: <504EB412-57A2-43B6-ABA1-B29D63395CD6@stufft.io> <3CC02307-6109-4F20-A815-BDA7CFE16AC2@stufft.io> <013F5AD9-9C9F-4E2E-A750-1C758B66B675@stufft.io> <5A9CA007-3832-4931-8F05-06BD62BE4A5E@stufft.io> <35957465-A878-4864-9473-9F68B9556387@stufft.io> Message-ID: On Sun Nov 23 2014 at 3:04:05 PM Georg Brandl wrote: > On 11/23/2014 05:55 PM, Brett Cannon wrote: > > > I guess my question is who and what is going to be disrupted if we go > with > > Guido's suggestion of switching to GitHub for code hosting? Contributors > won't > > be disrupted at all since most people are more familiar with GitHub vs. > > Bitbucket (how many times have we all heard the fact someone has even > learned > > Mercurial just to contribute to Python?). Core developers might be based > on some > > learned workflow, but I'm willing to bet we all know git at this point > (and for > > those of us who still don't like it, myself included, there are GUI apps > to > > paper over it or hg-git for those that prefer a CLI). Our infrastructure > will > > need to be updated, but how much of it is that hg-specific short of the > command > > to checkout out the repo? Obviously Bitbucket is much more minor by > simply > > updating just a URL, but changing `hg clone` to `git clone` isn't crazy > either. > > Georg, Antoine, or Benjamin can point out if I'm wrong on this, maybe > Donald or > > someone in the infrastructure committee. > > Well, since "most people" already know git this part is probably not a big > deal, > although we'd have to consider alienating core developers who are not > git-savvy. > > > Probably the biggest thing I can think of that would need updating is > our commit > > hooks. Once again Georg, Antoine, or Benjamin could say how difficult it > would > > be to update those hooks. > > There are two categories of hooks we use: hooks that run before a push is > accepted, and hooks that run afterwards. The latter are not a concern, > since > anything that GH/BB doesn't support can be run as a web service on > python.org > infrastructure, which then gets POST requests from the hosting platforms. > These > are > > * tracker update > * IRC notification > * sending email to python-checkins > * triggering buildbot > > The more problematic category are pre-push hooks. We use them for checking > and rejecting commits with > > * disallowed branches > * non-conformant whitespace > * wrong EOL style > * multiple heads per named branch > > As far as I know, neither GH nor BB support such hooks, since they are > highly > project-specific. However, they are only used in cpython and related > repositories, so that doesn't concern migration of doc-only repos. > > Sure, you can let the CI run the checks, but that doesn't prohibit merging > and is circumvented by direct pushes to the repository that don't go > through > the PR system. (And please don't make me as a coredev open a PR for every > change.) > I'm not even going to touch the idea of requiring code review for all patches in the middle of this discussion. =) > > From my perspective, swapping out Mercurial for git achieves exactly > nothing > > in terms of alleviating the review bottleneck (since the core > developers > > that strongly prefer the git UI will already be using an adapter), > and is in > > fact likely to make it worse by putting the greatest burden in > adapting to > > the change on the folks that are already under the greatest time > pressure. > > > > > > That's not entirely true. If you are pushing a PR shift in our patch > acceptance > > workflow then Bitbucket vs. GitHub isn't fundamentally any different in > terms of > > benefit, and I would honestly argue that GitHub's PR experience is > better. IOW > > either platform is of equal benefit. > > In my opinion, scattering repos over github, bitbucket and hg.python.org > is > even less friendly to contributors than a centralized place. (We are > already > approaching this, with pydotorg and infrastructure repos on github.) So > I'm > going to add some thoughts here about migrating the main CPython to > git+hub. > I don't think we would ever split ourselves across three separate hosting services. At most I see two -- one for docs, one for CPython -- but I would honestly expect it to be only one long-term. > > We have to consider how well our branch workflow works with the PR > workflow. There's no gain in the ability to easily merge a PR to one > branch > via github when the subsequent merge of 3.x to default/master requires a > local > pull/push cycle, as well as the 2.x backport. > > As far as I know, you'd have to open a pull/merge request yourself and > instantly > merge it, except if there are conflicts between branches, in which case you > are again forced to go local. I don't need to mention that this doesn't > work > well when someone makes a concurrent commit to the source branch in the > meantime. > > And I don't think you'd want to force people to submit 3 pull requests for > the individual branches. > > The next point is that there is no easy way to change the target branch of > a pull request (on github or bitbucket). People will usually make patches > against the master branch unless told differently explicitly, which means > that the pull request will also be against the master branch. Which means, > close the PR, ask submitter to resubmit against 3.x branch, or do it > yourself. > How do other projects tend to manage their bugfix vs. in-dev branches? Is it a lot of cherrypicking? Probably our long development cycles doesn't help with this as it makes cross-branch merging difficult thanks to divergence, else this could be automated somehow or simply not be an issue. Otherwise we would have to do a merge from branch to branch locally as we do now and make sure people committed in the branch we wanted them to. > > > It's also worth keeping in mind that changing the underlying VCS > means > > changing *all* the automation scripts, rather than just updating the > > configuration settings to reflect a new hosting URL. > > > > > > What are the automation scripts there are that would require updating? I > would > > like to a list and to have the difficulty of moving them mentioned to > know what > > the impact would be. > > Compiling that list is likely the most difficult step of updating them. > That's what I figured and why I didn't want them viewed as a blocker. > > > So here is what I want to know to focus this discussion: > > [...] > > > Third, do our release managers care about hg vs. git strongly? They > probably use > > the DVCS the most directly and at a lower level by necessity compared to > anyone > > else. > > I wouldn't care about git or hg. What I *might* care about is the > low-level > access to the repository on hg.python.org, mainly because of the hooks. > > > Fourth, do any core developers feel strongly about not using GitHub? Now > please > > notice I said "GitHub" and not "git"; I think the proper way to frame > this whole > > discussion is we are deciding if we want to switch to Bitbucket or > GitHub who > > provide a low-level API for their version control storage service > through hg or > > git, respectively. I personally dislike git, but I really like GitHub > and I > > don't even notice git there since I use GitHub's OS X app; as I said, I > view > > this as choosing a platform and not the underlying DVCS as I have > happily chosen > > to access the GitHub hosting service through an app that is not git > (it's like > > accessing a web app through it's web page or its REST API). > > Github is a nice platform, but many of its features would not be relevant > for > the cpython repo (e.g. issue tracker or wiki). The usefulness of pull > requests > is dubious, see above. > > The "but it is much easier to contribute simple changes" argument is a bit > simplified: for any nontrivial patch, the time spent on working out the > code > should outweight time spent with "hg diff" or "click on pull request". And > while Travis CI is nice, running relevant tests locally is *much* quicker > than > waiting for a full test suite run on a virtualized testing machine. > Sure, but it's definitely easier to just wait for the Travis green on the PR than have to apply a patch and run the tests myself manually. > > As for typo fixes, the world does not end when some typos aren't fixed. > Anyway, for the docs we have an explicit offer to send anything, patch or > just suggestion, to docs at python.org, and people do make use of it. No > github account even required. > Which is great, but I'm willing to bet clicking "Edit" in GitHub is easier still than creating the patch and emailing it. I have contributed doc fixes to a bunch of projects because of that "Edit" button (which is why I think Nick is so anxious to get it). If I had to do any checkout then I wouldn't have done them. And even having to email a diff might be too much for some changes. > > > At least for me, until we get a clear understanding of what workflow > changes we > > are asking for both contributors and core developers and exactly what > work would > > be necessary to update our infrastructure for either Bitbucket or GitHub > we > > can't really have a reasonable discussion that isn't going to be full of > guessing. > > As for svn->hg, a PEP and a champion would be necessary, this time with the > possibility of being rejected. > > > And I'm still in support no matter what of breaking out the HOWTOs and > the > > tutorial into their own repos for easier updating (having to update the > Python > > porting HOWTO in three branches is a pain when it should be consistent > across > > Python releases). > > I see no problem with that, provided there's a cronjob that syncs the > version > in Doc/ to the external version reasonably often. > Would that really be necessary? At least for the HOWTOs how often are they edited simultaneously as some code in CPython? The Clinic HOWTO is probably the only one that might get updated simultaneously. I'd also be curious to know how often the tutorial is updated simultaneously as well. -------------- next part -------------- An HTML attachment was scrubbed... URL: From g.brandl at gmx.net Sun Nov 23 22:08:29 2014 From: g.brandl at gmx.net (Georg Brandl) Date: Sun, 23 Nov 2014 22:08:29 +0100 Subject: [Python-Dev] Move selected documentation repos to PSF BitBucket account? In-Reply-To: <415502D5-3489-4C66-A84D-73C0998A6FFC@stufft.io> References: <3CC02307-6109-4F20-A815-BDA7CFE16AC2@stufft.io> <013F5AD9-9C9F-4E2E-A750-1C758B66B675@stufft.io> <5A9CA007-3832-4931-8F05-06BD62BE4A5E@stufft.io> <35957465-A878-4864-9473-9F68B9556387@stufft.io> <415502D5-3489-4C66-A84D-73C0998A6FFC@stufft.io> Message-ID: On 11/23/2014 09:38 PM, Donald Stufft wrote: > >> On Nov 23, 2014, at 3:03 PM, Georg Brandl wrote: >> >> The next point is that there is no easy way to change the target branch of >> a pull request (on github or bitbucket). People will usually make patches >> against the master branch unless told differently explicitly, which means >> that the pull request will also be against the master branch. Which means, >> close the PR, ask submitter to resubmit against 3.x branch, or do it >> yourself. > > This in particular is not really true on Github at least. By default PRs are > made against whichever branch you have configured as the default branch in > the Github UI. This does default to master but it doesn?t have to be, for > instance pip has this configured for the develop branch. Although I think > this specific point is moot because if things were on Git it?d probably make > the most sense to have the default integration branch be ``master`` just like > for the Hg repos they use default. Sure, although as is the majority of commits to CPython are bugfixes, which normally go to 2.7, 3.4 and 3.5 (master). > Even if someone makes a PR against the wrong branch, it ?degrades? into > essentially the same UX as you have now, you can turn a PR into a patch by > adding a .patch or .diff to the end of the PR URL and then you have a patch > file. In additional github makes it easy to check out PRs directly with only a > minor bit of one time configuration in your local clone. I can checkout any PR > by doing ``git checkout pr/1`` (replacing 1 with whatever the number is). > > Honestly the worst part about someone sending a PR to the wrong branch is it > degrades into the same terrible UX that *every* patch has to go through on > a hg.python.org repository right now. I'm not saying it's worse, but most of the time it's no better for the committer, especially since the branches have to be juggled in most cases. Although, when it's the same amount of work for the committer, but nicer for the contributor, that's a net win, I can see that. What is absolutely essential though is a way to automatically open an issue on bugs.python.org for each PR, otherwise we have to look for issues in two different places. (Sure, GH treats PRs like GH issues, but we wouldn't use the GH issue tracker.) Georg From g.brandl at gmx.net Sun Nov 23 22:17:49 2014 From: g.brandl at gmx.net (Georg Brandl) Date: Sun, 23 Nov 2014 22:17:49 +0100 Subject: [Python-Dev] Move selected documentation repos to PSF BitBucket account? In-Reply-To: References: <3CC02307-6109-4F20-A815-BDA7CFE16AC2@stufft.io> <013F5AD9-9C9F-4E2E-A750-1C758B66B675@stufft.io> <5A9CA007-3832-4931-8F05-06BD62BE4A5E@stufft.io> <35957465-A878-4864-9473-9F68B9556387@stufft.io> Message-ID: On 11/23/2014 09:42 PM, Brett Cannon wrote: > The more problematic category are pre-push hooks. We use them for checking > and rejecting commits with > > * disallowed branches > * non-conformant whitespace > * wrong EOL style > * multiple heads per named branch > > As far as I know, neither GH nor BB support such hooks, since they are highly > project-specific. However, they are only used in cpython and related > repositories, so that doesn't concern migration of doc-only repos. > > Sure, you can let the CI run the checks, but that doesn't prohibit merging > and is circumvented by direct pushes to the repository that don't go through > the PR system. (And please don't make me as a coredev open a PR for every > change.) > > > I'm not even going to touch the idea of requiring code review for all patches in > the middle of this discussion. =) As far as I can see, our Rietveld is very well used already. *Requiring* code review for *all* patches is detrimental in a volunteer project IMO. (I'm saying this especially because many of my changes are small to trivial doc fixes.) > The next point is that there is no easy way to change the target branch of > a pull request (on github or bitbucket). People will usually make patches > against the master branch unless told differently explicitly, which means > that the pull request will also be against the master branch. Which means, > close the PR, ask submitter to resubmit against 3.x branch, or do it > yourself. > > > How do other projects tend to manage their bugfix vs. in-dev branches? Is it a > lot of cherrypicking? Probably our long development cycles doesn't help with > this as it makes cross-branch merging difficult thanks to divergence, else this > could be automated somehow or simply not be an issue. Otherwise we would have to > do a merge from branch to branch locally as we do now and make sure people > committed in the branch we wanted them to. For smaller projects, it's not as much of an issue. In Sphinx, there are two branches (stable and default), and when there are pull requests to the default branch that should go to stable, I cherry-pick them manually. And as you said, merging or transplanting between branches is much more likely to succeed automatically. > The "but it is much easier to contribute simple changes" argument is a bit > simplified: for any nontrivial patch, the time spent on working out the code > should outweight time spent with "hg diff" or "click on pull request". And > while Travis CI is nice, running relevant tests locally is *much* quicker than > waiting for a full test suite run on a virtualized testing machine. > > > Sure, but it's definitely easier to just wait for the Travis green on the PR > than have to apply a patch and run the tests myself manually. You'll likely have to run them manually anyway while merging to the other branches. > As for typo fixes, the world does not end when some typos aren't fixed. > Anyway, for the docs we have an explicit offer to send anything, patch or > just suggestion, to docs at python.org , and people do > make use of it. No > github account even required. > > > Which is great, but I'm willing to bet clicking "Edit" in GitHub is easier still > than creating the patch and emailing it. I have contributed doc fixes to a bunch > of projects because of that "Edit" button (which is why I think Nick is so > anxious to get it). If I had to do any checkout then I wouldn't have done them. > And even having to email a diff might be too much for some changes. I did say "patch or just suggestion". Most emails to docs at python.org come without a patch, just a plain description of what's wrong, and that is just fine. I would also judge that >50% of the emails come from people who wouldn't have a github account in any case (so that the big shiny Edit button isn't an easy option). > > And I'm still in support no matter what of breaking out the HOWTOs and the > > tutorial into their own repos for easier updating (having to update the Python > > porting HOWTO in three branches is a pain when it should be consistent across > > Python releases). > > I see no problem with that, provided there's a cronjob that syncs the version > in Doc/ to the external version reasonably often. > > > Would that really be necessary? At least for the HOWTOs how often are they > edited simultaneously as some code in CPython? The Clinic HOWTO is probably the > only one that might get updated simultaneously. I'd also be curious to know how > often the tutorial is updated simultaneously as well. I'd like the HOWTOs to stay part of Doc/, so changes in the external repo have to be merged in there somehow, and not only at release time. Georg From donald at stufft.io Sun Nov 23 22:18:40 2014 From: donald at stufft.io (Donald Stufft) Date: Sun, 23 Nov 2014 16:18:40 -0500 Subject: [Python-Dev] Move selected documentation repos to PSF BitBucket account? In-Reply-To: References: <3CC02307-6109-4F20-A815-BDA7CFE16AC2@stufft.io> <013F5AD9-9C9F-4E2E-A750-1C758B66B675@stufft.io> <5A9CA007-3832-4931-8F05-06BD62BE4A5E@stufft.io> <35957465-A878-4864-9473-9F68B9556387@stufft.io> <415502D5-3489-4C66-A84D-73C0998A6FFC@stufft.io> Message-ID: <13392870-0F41-4EB3-BF86-9272869D3DE7@stufft.io> > On Nov 23, 2014, at 4:08 PM, Georg Brandl wrote: > > On 11/23/2014 09:38 PM, Donald Stufft wrote: >> >>> On Nov 23, 2014, at 3:03 PM, Georg Brandl wrote: >>> >>> The next point is that there is no easy way to change the target branch of >>> a pull request (on github or bitbucket). People will usually make patches >>> against the master branch unless told differently explicitly, which means >>> that the pull request will also be against the master branch. Which means, >>> close the PR, ask submitter to resubmit against 3.x branch, or do it >>> yourself. >> >> This in particular is not really true on Github at least. By default PRs are >> made against whichever branch you have configured as the default branch in >> the Github UI. This does default to master but it doesn?t have to be, for >> instance pip has this configured for the develop branch. Although I think >> this specific point is moot because if things were on Git it?d probably make >> the most sense to have the default integration branch be ``master`` just like >> for the Hg repos they use default. > > Sure, although as is the majority of commits to CPython are bugfixes, which > normally go to 2.7, 3.4 and 3.5 (master). > >> Even if someone makes a PR against the wrong branch, it ?degrades? into >> essentially the same UX as you have now, you can turn a PR into a patch by >> adding a .patch or .diff to the end of the PR URL and then you have a patch >> file. In additional github makes it easy to check out PRs directly with only a >> minor bit of one time configuration in your local clone. I can checkout any PR >> by doing ``git checkout pr/1`` (replacing 1 with whatever the number is). >> >> Honestly the worst part about someone sending a PR to the wrong branch is it >> degrades into the same terrible UX that *every* patch has to go through on >> a hg.python.org repository right now. > > I'm not saying it's worse, but most of the time it's no better for the > committer, especially since the branches have to be juggled in most cases. Right, well if you?re just merging one branch into another you can create a PR from one branch to another directly in the Github Web UI, which will run tests and then you can press the merge button on that PR too. It?s something like 3-4 clicks or so. If you want to selectively send a change from one branch to another that will require manually making the change locally yea. > > Although, when it's the same amount of work for the committer, but nicer for > the contributor, that's a net win, I can see that. > > What is absolutely essential though is a way to automatically open an issue > on bugs.python.org for each PR, otherwise we have to look for issues in two > different places. (Sure, GH treats PRs like GH issues, but we wouldn't use > the GH issue tracker.) There?s a web hook that fires when a pull request is created, it could even be smart enough to look for Issue numbers in the PR and commits so that it associates it with an existing issue instead of creating a new issue. You can see all of the events we can subscribe to here: https://developer.github.com/webhooks/#events In a related note, while it?s not possible to *reject* a push if things don?t match up it is entirely possible to detect it after the fact and send an email out to folks or what have you. Similar to how tests themselves are done now. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA From rymg19 at gmail.com Sun Nov 23 22:01:13 2014 From: rymg19 at gmail.com (Ryan) Date: Sun, 23 Nov 2014 15:01:13 -0600 Subject: [Python-Dev] Move selected documentation repos to PSF BitBucket account? In-Reply-To: References: <20141121102647.46e97f04@limelight.wooz.org> <19336614-0E4F-42BF-A918-1807BB7F3599@stufft.io> <504EB412-57A2-43B6-ABA1-B29D63395CD6@stufft.io> <73f643e5-bd78-4dbc-8a26-335e91735d16@email.android.com> Message-ID: <57bfdea3-7b14-435a-8901-ce16bf70ea1c@email.android.com> Georg Brandl wrote: >On 11/23/2014 07:03 PM, Ryan wrote: >> I can agree with most of these points. Some more things to consider: >> >> - Git is 20x faster than Hg (that's 99% of the reason I switched and >hate using >> Darcs) > >You won't get much traction with this argument around here. As long as >there >aren't specific complaints about slowness (which probably could be >fixed by a >discussion on mercurial-devel), Hg is "fast enough", just like Python >is "fast >enough" compared to C. (Otherwise, why bother.) I know, but, when checking out large repos, it stings a little. > >> - GitHub is much nicer than BitBucket. I usually look for a git >mirror of a >> BitBucket repo since I hate browsing on BB > >I agree, Github's UI is nicer than BB. It is also true that most >features >that are introduced on BB are "inspired" by GH. > >> - Even Plan9Port moved from Hg to GitHub a few days ago > >Projects move all the time. Why is this one special? Because, for ages, everything related to Rob Pike used Google Code and Mercurial, largely because he works at Google. > >> - Some Google devs even host their projects on GitHub (not Google >Code) >> >> Thank goodness no one has mentioned CodePlex. >> >> BTW, I frown on projects that choose something largely inferior over >a better >> choice just because "one is written in X language." Now, if >python.org >> was written in Ruby, that would just be sad, but >certain >> things just don't matter. Like DVCS's. > >You can read up PEP 374 for the reasons for Hg. The main reasons >against git >were lacking Windows support (which probably is good enough now) and >developer >preference. > >Georg > > >_______________________________________________ >Python-Dev mailing list >Python-Dev at python.org >https://mail.python.org/mailman/listinfo/python-dev >Unsubscribe: >https://mail.python.org/mailman/options/python-dev/rymg19%40gmail.com -- Sent from my Android phone with K-9 Mail. Please excuse my brevity. Check out my website: http://kirbyfan64.github.io/ From rosuav at gmail.com Sun Nov 23 23:15:10 2014 From: rosuav at gmail.com (Chris Angelico) Date: Mon, 24 Nov 2014 09:15:10 +1100 Subject: [Python-Dev] PEP 479: Change StopIteration handling inside generators In-Reply-To: References: <20141121174731.2a245ff5@fsol> Message-ID: On Mon, Nov 24, 2014 at 5:28 AM, Ron Adam wrote: > With the passage of the PEP, it will change what is different about them > once it's in full effect. The stop hack won't work in both, and you may get > a RuntimeError in generator expressions where you would get StopIteration in > list-comps. (Is this correct?) "them" being list comps and generator expressions? The stop hack won't work in either (currently it does work in genexps), but you'd get a different exception type if you attempt it. This is correct. It's broadly similar to this distinction: >>> {1:2,3:4}[50] Traceback (most recent call last): File "", line 1, in KeyError: 50 >>> [1,2,3,4][50] Traceback (most recent call last): File "", line 1, in IndexError: list index out of range In both lists and dicts, you can't look up something that isn't there. But you get a slightly different exception type (granted, these two do have a common superclass) depending on the exact context. But the behaviour will be effectively the same. ChrisA From rosuav at gmail.com Sun Nov 23 23:54:51 2014 From: rosuav at gmail.com (Chris Angelico) Date: Mon, 24 Nov 2014 09:54:51 +1100 Subject: [Python-Dev] Please reconsider PEP 479. In-Reply-To: <54724117.20408@hotpy.org> References: <54724117.20408@hotpy.org> Message-ID: On Mon, Nov 24, 2014 at 7:18 AM, Mark Shannon wrote: > Hi, > > I have serious concerns about this PEP, and would ask you to reconsider it. Hoping I'm not out of line in responding here, as PEP author. Some of your concerns (eg "5 days is too short") are clearly for Guido, not me, but perhaps I can respond to the rest of it. > [ Very short summary: > Generators are not the problem. It is the naive use of next() in an > iterator that is the problem. (Note that all the examples involve calls to > next()). > Change next() rather than fiddling with generators. > ] > > StopIteration is not a normal exception, indicating a problem, rather it > exists to signal exhaustion of an iterator. > However, next() raises StopIteration for an exhausted iterator, which really > is an error. > Any iterator code (generator or __next__ method) that calls next() treats > the StopIteration as a normal exception and propogates it. > The controlling loop then interprets StopIteration as a signal to stop and > thus stops. > *The problem is the implicit shift from signal to error and back to signal.* The situation is this: Both __next__ and next() need the capability to return literally any object at all. (I raised a hypothetical possibility of some sort of sentinel object, but for such a sentinel to be useful, it will need to have a name, which means that *by definition* that object would have to come up when iterating over the .values() of some namespace.) They both also need to be able to indicate a lack of return value. This means that either they return a (success, value) tuple, or they have some other means of signalling exhaustion. I'm not sure what you mean by your "However" above. In both __next__ and next(), this is a signal; it becomes an error as soon as you call next() and don't cope adequately with the signal, just as KeyError is an error. > 2. The proposed solution does not address this issue at all, but rather > legislates against generators raising StopIteration. Because that's the place where a StopIteration will cause a silent behavioral change, instead of cheerily bubbling up to top-level and printing a traceback. > 3. Generators and the iterator protocol were introduced in Python 2.2, 13 > years ago. > For all of that time the iterator protocol has been defined by the > __iter__(), next()/__next__() methods and the use of StopIteration to > terminate iteration. > > Generators are a way to write iterators without the clunkiness of explicit > __iter__() and next()/__next__() methods, but have always obeyed the same > protocol as all other iterators. This has allowed code to rewritten from one > form to the other whenever desired. > > Do not forget that despite the addition of the send() and throw() methods > and their secondary role as coroutines, generators have primarily always > been a clean and elegant way of writing iterators. This question has been raised several times; there is a distinct difference between __iter__() and __next__(), and it is only the latter which is aware of StopIteration. Compare these three classes: class X: def __init__(self): self.state=0 def __iter__(self): return self def __next__(self): if self.state == 3: raise StopIteration self.state += 1 return self.state class Y: def __iter__(self): return iter([1,2,3]) class Z: def __iter__(self): yield 1 yield 2 yield 3 Note how just one of these classes uses StopIteration, and yet all three are iterable, yielding the same results. Neither Y nor Z is breaking iterator protocol - but neither of them is writing an iterator, either. > 4. Porting from Python 2 to Python 3 seems to be hard enough already. Most of the code broken by this change can be fixed by a mechanical replacement of "raise StopIteration" with "return"; the rest need to be checked to see if they're buggy or unclear. There is an edge case with "return some_value" vs "raise StopIteration(some_value)" (the former's not compatible with 2.7), but apart from that, the recommended form of code for 3.7 will work in all versions of Python since 2.2. > 5. I think I've already covered this in the other points, but to reiterate > (excuse the pun): > Calling next() on an exhausted iterator is, I would suggest, a logical > error. How do you know that it's exhausted, other than by calling next() on it? > It also worth noting that calling next() is the only place a StopIteration > exception is likely to occur outside of the iterator protocol. This I agree with. > An example > ---------- > > Consider a function to return the value from a set with a single member. > def value_from_singleton(s): > if len(s) < 2: #Intentional error here (should be len(s) == 1) > return next(iter(s)) > raise ValueError("Not a singleton") > > Now suppose we pass an empty set to value_from_singleton(s), then we get a > StopIteration exception, which is a bit weird, but not too bad. Only a little weird - and no different from the way you'd get a TypeError if you pass it an integer. > However it is when we use it in a generator (or in the __next__ method of an > iterator) that we get a serious problem. > Currently the iterator appears to be exhausted early, which is wrong. > However, with the proposed change we get RuntimeError("generator raised > StopIteration") raised, which is also wrong, just in a different way. What you have here is two distinct issues. The first is "what happens if an unexpected StopIteration occurs during __next__ processing?", and the second is "ditto ditto a generator's execution?". The first one is extremely hard to deal with, and extremely unlikely. The second is much easier to deal with, and can therefore be solved. > Solutions > --------- > My preferred "solution" is to do nothing except improving the documentation > of next(). Explain that it can raise StopIteration which, if allowed to > propogate can cause premature exhaustion of an iterator. Docs fixing doesn't solve everything. > If something must be done then I would suggest changing the behaviour of > next() for an exhausted iterator. > Rather than raise StopIteration it should raise ValueError (or IndexError?). So, if I've understood you correctly, what you're saying is that __next__ should raise StopIteration, and then next() should absorb that and raise ValueError instead? I'm not sure how this would help anything, but I can see that it would poke the issue with a sharp pointy stick. Can you elaborate on how this would work in practice? > Also, it might be worth considering making StopIteration inherit from > BaseException, rather than Exception. Separate concern altogether, as the bases of StopIteration have nothing to do with a protocol meaning collision. I would probably support this change, on the basis that Exception should be for, well, exceptions, and BaseException can be used for everything that uses the exception-handling mechanism for other purposes. But it wouldn't help or affect this proposal. ChrisA From v+python at g.nevcal.com Mon Nov 24 00:00:24 2014 From: v+python at g.nevcal.com (Glenn Linderman) Date: Sun, 23 Nov 2014 15:00:24 -0800 Subject: [Python-Dev] Move selected documentation repos to PSF BitBucket account? In-Reply-To: References: <19336614-0E4F-42BF-A918-1807BB7F3599@stufft.io> <504EB412-57A2-43B6-ABA1-B29D63395CD6@stufft.io> <3CC02307-6109-4F20-A815-BDA7CFE16AC2@stufft.io> <013F5AD9-9C9F-4E2E-A750-1C758B66B675@stufft.io> <5A9CA007-3832-4931-8F05-06BD62BE4A5E@stufft.io> <35957465-A878-4864-9473-9F68B9556387@stufft.io> Message-ID: <54726708.4020308@g.nevcal.com> On 11/23/2014 3:18 AM, Nick Coghlan wrote: > Patches getting held up in the review queue for weeks or months is a > *huge* barrier to contribution, as it prevents the formation of the > positive feedback cycle where having a contribution accepted feels > good, so folks are more likely to want to contribute again. Exactly. None of the patches I care about have gone anywhere for years. Two of them were done by others, apparently using the right processes, and still went nowhere, so it doesn't encourage me to attempt to even learn the process to make a more appropriate contribution of mine (which, sadly, I fixed too many different bugs in one file or two files, and have not split them into individual patches). Another has recently been making some progress... I guess that means I don't care enough about them, to fight to force them through the system, it is easier just to apply the patches to my installation... but that is a sad state of affairs. -------------- next part -------------- An HTML attachment was scrubbed... URL: From francis.giraldeau at gmail.com Mon Nov 24 00:03:52 2014 From: francis.giraldeau at gmail.com (Francis Giraldeau) Date: Sun, 23 Nov 2014 18:03:52 -0500 Subject: [Python-Dev] Support for Linux perf In-Reply-To: <54708518.5030509@googlemail.com> References: <54708518.5030509@googlemail.com> Message-ID: 2014-11-22 7:44 GMT-05:00 Julian Taylor : > On 17.11.2014 23:09, Francis Giraldeau wrote: > > Hi, > > ... > > The PEP-418 is about performance counters, but there is no mention o > > Anyway, I think we must change CPython to support tools such as perf. > > Any thoughts? > > > > there are some patches available adding systemtap and dtrace probes, > which should at least help getting function level profiles: > > http://bugs.python.org/issue21590 Thanks for these links, the patches looks interesting. As Jonas mentioned, Perf should be able to unwind a Python stack. It does at the interpreter level, and the frame info is scattered in virtual memory. It needs to be access offline. I think it could be possible to use the function entry and exit hooks in the interpreter to save important frame info, such as function name, file and line number, in a memory map known to perf. Then, we can tell Perf to record this compact zone of data in the sample as "extra" field for offline use. Then, at the analysis time, each ELF interpreter frame could be matched with the corresponding Python frame info. I think the perf handler can't sleep, and accesses on each function entry/exit will also ensure the page is present in main memory when the sample is recorded. Thanks again for your inputs, I'll post any further developments. Cheers, Francis -------------- next part -------------- An HTML attachment was scrubbed... URL: From ron3200 at gmail.com Mon Nov 24 00:18:37 2014 From: ron3200 at gmail.com (Ron Adam) Date: Sun, 23 Nov 2014 17:18:37 -0600 Subject: [Python-Dev] PEP 479: Change StopIteration handling inside generators In-Reply-To: References: <20141121174731.2a245ff5@fsol> Message-ID: On 11/23/2014 04:15 PM, Chris Angelico wrote: > On Mon, Nov 24, 2014 at 5:28 AM, Ron Adam wrote: >> >With the passage of the PEP, it will change what is different about them >> >once it's in full effect. The stop hack won't work in both, and you may get >> >a RuntimeError in generator expressions where you would get StopIteration in >> >list-comps. (Is this correct?) > "them" being list comps and generator expressions? Yes > The stop hack won't work in either (currently it does work in > genexps), but you'd get a different exception type if you attempt it. > This is correct. It's broadly similar to this distinction: > >>>> >>>{1:2,3:4}[50] > Traceback (most recent call last): > File "", line 1, in > KeyError: 50 >>>> >>>[1,2,3,4][50] > Traceback (most recent call last): > File "", line 1, in > IndexError: list index out of range Comprehensions insert/append items, so you wouldn't get those unless you are reading from another object and would bubble out of generators too. Which is good because it's most likely an error that needs fixing. > In both lists and dicts, you can't look up something that isn't there. > But you get a slightly different exception type (granted, these two do > have a common superclass) depending on the exact context. But the > behaviour will be effectively the same. I think the difference is very specific to StopIteration. And I think we need to wait and see what happens in real code. Cheers, Ron From rosuav at gmail.com Mon Nov 24 00:22:54 2014 From: rosuav at gmail.com (Chris Angelico) Date: Mon, 24 Nov 2014 10:22:54 +1100 Subject: [Python-Dev] PEP 479: Change StopIteration handling inside generators In-Reply-To: References: <20141121174731.2a245ff5@fsol> Message-ID: On Mon, Nov 24, 2014 at 10:18 AM, Ron Adam wrote: >> The stop hack won't work in either (currently it does work in >> genexps), but you'd get a different exception type if you attempt it. >> This is correct. It's broadly similar to this distinction: >> >>>>> >>>{1:2,3:4}[50] >> >> Traceback (most recent call last): >> File "", line 1, in >> KeyError: 50 >>>>> >>>>> >>>[1,2,3,4][50] >> >> Traceback (most recent call last): >> File "", line 1, in >> IndexError: list index out of range > > > Comprehensions insert/append items, so you wouldn't get those unless you are > reading from another object and would bubble out of generators too. Which is > good because it's most likely an error that needs fixing. My point is that doing the same errant operation on a list or a dict will give different exceptions. In the same way, calling next() on an empty iterator will raise StopIteration normally, but might raise RuntimeError instead. It's still an exception, it still indicates a place where code needs to be changed (unlike, say, a ValueError when doing type conversions, which often means *data* needs to be changed), so it doesn't hugely matter _which_ exception is raised. Also, I'm hoping that someone can improve on my patch by having the full traceback from the original StopException become visible - then, the RuntimeError's __context__ will give all the info you would want. ChrisA From steve at pearwood.info Mon Nov 24 00:24:56 2014 From: steve at pearwood.info (Steven D'Aprano) Date: Mon, 24 Nov 2014 10:24:56 +1100 Subject: [Python-Dev] PEP 479: Change StopIteration handling inside generators In-Reply-To: <5472087C.4090205@stoneleaf.us> References: <20141121174731.2a245ff5@fsol> <289664E5-58CA-4964-8DB6-20E43F3717BC@gmail.com> <5472087C.4090205@stoneleaf.us> Message-ID: <20141123232456.GU2748@ando.pearwood.info> On Sun, Nov 23, 2014 at 08:17:00AM -0800, Ethan Furman wrote: > While I am in favor of PEP 479, and I have to agree with Raymond that > this isn't pretty. > > Currently, next() accepts an argument of what to return if the > iterator is empty. Can we enhance that in some way so that the > overall previous behavior could be retained? [...] > Then, if the iterator is empty, instead of raising StopIteration, or > returning some value that would then have to be checked, it could > raise some other exception that is understood to be normal generator > termination. We *already* have an exception that is understood to be normal generator termination. It is called StopIteration. Removing the long-standing ability to halt generators with StopIteration, but then recreating that ability under a different name is the worst of both worlds: - working code is still broken; - people will complain that the new exception X is silently swallowed by generators, just as they complained about StopIteration; - it is yet another subtle difference between Python 2 and 3; - it involves a code smell ("no constant arguments to functions"); - and does nothing to help generators that don't call next(). The current behaviour is nice and clean and has worked well for over a decade. The new behaviour exchanges consistency in one area (generators behave like all other iterators) for consistency in another (generator expressions will behave like comprehensions in the face of StopIteration). But trying to have both at the same time via a new exception adds even more complexity and would leave everyone unhappy. -- Steven From brett at python.org Mon Nov 24 00:21:15 2014 From: brett at python.org (Brett Cannon) Date: Sun, 23 Nov 2014 23:21:15 +0000 Subject: [Python-Dev] Move selected documentation repos to PSF BitBucket account? References: <3CC02307-6109-4F20-A815-BDA7CFE16AC2@stufft.io> <013F5AD9-9C9F-4E2E-A750-1C758B66B675@stufft.io> <5A9CA007-3832-4931-8F05-06BD62BE4A5E@stufft.io> <35957465-A878-4864-9473-9F68B9556387@stufft.io> Message-ID: On Sun Nov 23 2014 at 4:18:37 PM Georg Brandl wrote: > On 11/23/2014 09:42 PM, Brett Cannon wrote: > [SNIP] > > And I'm still in support no matter what of breaking out the HOWTOs > and the > > > tutorial into their own repos for easier updating (having to > update the Python > > > porting HOWTO in three branches is a pain when it should be > consistent across > > > Python releases). > > > > I see no problem with that, provided there's a cronjob that syncs > the version > > in Doc/ to the external version reasonably often. > > > > > > Would that really be necessary? At least for the HOWTOs how often are > they > > edited simultaneously as some code in CPython? The Clinic HOWTO is > probably the > > only one that might get updated simultaneously. I'd also be curious to > know how > > often the tutorial is updated simultaneously as well. > > I'd like the HOWTOs to stay part of Doc/, so changes in the external repo > have > to be merged in there somehow, and not only at release time. > Right, I'm trying to understand *why* you want the HOWTOs to stay in Doc/. I dread having to update the porting HOWTO because it requires updating 2.7, 3.4, and default. And if the process is automated to pull from an external repo then what is the benefit of syncing the files and duplicating them across 4 repos? -------------- next part -------------- An HTML attachment was scrubbed... URL: From g.brandl at gmx.net Mon Nov 24 00:36:13 2014 From: g.brandl at gmx.net (Georg Brandl) Date: Mon, 24 Nov 2014 00:36:13 +0100 Subject: [Python-Dev] Move selected documentation repos to PSF BitBucket account? In-Reply-To: References: <3CC02307-6109-4F20-A815-BDA7CFE16AC2@stufft.io> <013F5AD9-9C9F-4E2E-A750-1C758B66B675@stufft.io> <5A9CA007-3832-4931-8F05-06BD62BE4A5E@stufft.io> <35957465-A878-4864-9473-9F68B9556387@stufft.io> Message-ID: On 11/24/2014 12:21 AM, Brett Cannon wrote: > > > On Sun Nov 23 2014 at 4:18:37 PM Georg Brandl > wrote: > > On 11/23/2014 09:42 PM, Brett Cannon wrote: > > > [SNIP] > > > > And I'm still in support no matter what of breaking out the HOWTOs > and the > > > tutorial into their own repos for easier updating (having to update > the Python > > > porting HOWTO in three branches is a pain when it should be > consistent across > > > Python releases). > > > > I see no problem with that, provided there's a cronjob that syncs the > version > > in Doc/ to the external version reasonably often. > > > > > > Would that really be necessary? At least for the HOWTOs how often are they > > edited simultaneously as some code in CPython? The Clinic HOWTO is > probably the > > only one that might get updated simultaneously. I'd also be curious to > know how > > often the tutorial is updated simultaneously as well. > > I'd like the HOWTOs to stay part of Doc/, so changes in the external repo have > to be merged in there somehow, and not only at release time. > > > Right, I'm trying to understand *why* you want the HOWTOs to stay in Doc/. I > dread having to update the porting HOWTO because it requires updating 2.7, 3.4, > and default. And if the process is automated to pull from an external repo then > what is the benefit of syncing the files and duplicating them across 4 repos? That they are still part of the docs on docs.python.org and what people download from there. I don't like resources like that scattered about. It makes sense for the devguide, since that is really a different topic. Georg From steve at pearwood.info Mon Nov 24 00:57:27 2014 From: steve at pearwood.info (Steven D'Aprano) Date: Mon, 24 Nov 2014 10:57:27 +1100 Subject: [Python-Dev] Move selected documentation repos to PSF BitBucket account? In-Reply-To: References: <19336614-0E4F-42BF-A918-1807BB7F3599@stufft.io> <504EB412-57A2-43B6-ABA1-B29D63395CD6@stufft.io> Message-ID: <20141123235727.GV2748@ando.pearwood.info> On Sun, Nov 23, 2014 at 08:55:50AM -0800, Guido van Rossum wrote: > But I strongly believe that if we want to do the right thing for the > long term, we should switch to GitHub. Encouraging a software, or social, monopoly is never the right thing for the long term. http://nedbatchelder.com/blog/201405/github_monoculture.html > I promise you that once the pain of the switch is over you will feel > much better about it. I am also convinced that we'll get more > contributions this way. I'm sure that we'll get *more* contributions, but will they be *better* contributions? I know that there are people who think that mailing lists are old and passe, and that we should shift discussion to a social media site like Reddit. If we did, we'd probably get twenty times as many comments, and the average quality would probably plummet. More is not necessarily a good thing. -- Steven From brian at python.org Mon Nov 24 01:08:07 2014 From: brian at python.org (Brian Curtin) Date: Sun, 23 Nov 2014 18:08:07 -0600 Subject: [Python-Dev] Move selected documentation repos to PSF BitBucket account? In-Reply-To: <20141123235727.GV2748@ando.pearwood.info> References: <19336614-0E4F-42BF-A918-1807BB7F3599@stufft.io> <504EB412-57A2-43B6-ABA1-B29D63395CD6@stufft.io> <20141123235727.GV2748@ando.pearwood.info> Message-ID: On Sun, Nov 23, 2014 at 5:57 PM, Steven D'Aprano wrote: > On Sun, Nov 23, 2014 at 08:55:50AM -0800, Guido van Rossum wrote: > >> But I strongly believe that if we want to do the right thing for the >> long term, we should switch to GitHub. > > Encouraging a software, or social, monopoly is never the right thing for > the long term. > > http://nedbatchelder.com/blog/201405/github_monoculture.html > > >> I promise you that once the pain of the switch is over you will feel >> much better about it. I am also convinced that we'll get more >> contributions this way. > > I'm sure that we'll get *more* contributions, but will they be *better* > contributions? > > I know that there are people who think that mailing lists are old and > passe, and that we should shift discussion to a social media site like > Reddit. If we did, we'd probably get twenty times as many comments, and > the average quality would probably plummet. More is not necessarily a > good thing. If we need to ensure that we're getting better contributions than we are now, then we should be interviewing committers, rejecting newcomers (or the opposite, multiplying core-mentors by 100), and running this like a business. I've written some crappy code that got committed, so I should probably be fired. Enabling our community to be active contributors is an important thing. Give them a means to level up and we'll all be better off from it. From mark at hotpy.org Mon Nov 24 01:25:04 2014 From: mark at hotpy.org (Mark Shannon) Date: Mon, 24 Nov 2014 00:25:04 +0000 Subject: [Python-Dev] Please reconsider PEP 479. In-Reply-To: References: <54724117.20408@hotpy.org> Message-ID: <54727AE0.6030301@hotpy.org> On 23/11/14 22:54, Chris Angelico wrote: > On Mon, Nov 24, 2014 at 7:18 AM, Mark Shannon wrote: >> Hi, >> >> I have serious concerns about this PEP, and would ask you to reconsider it. > > Hoping I'm not out of line in responding here, as PEP author. Some of > your concerns (eg "5 days is too short") are clearly for Guido, not > me, but perhaps I can respond to the rest of it. > >> [ Very short summary: >> Generators are not the problem. It is the naive use of next() in an >> iterator that is the problem. (Note that all the examples involve calls to >> next()). >> Change next() rather than fiddling with generators. >> ] >> >> StopIteration is not a normal exception, indicating a problem, rather it >> exists to signal exhaustion of an iterator. >> However, next() raises StopIteration for an exhausted iterator, which really >> is an error. >> Any iterator code (generator or __next__ method) that calls next() treats >> the StopIteration as a normal exception and propogates it. >> The controlling loop then interprets StopIteration as a signal to stop and >> thus stops. >> *The problem is the implicit shift from signal to error and back to signal.* > > The situation is this: Both __next__ and next() need the capability to > return literally any object at all. (I raised a hypothetical > possibility of some sort of sentinel object, but for such a sentinel > to be useful, it will need to have a name, which means that *by > definition* that object would have to come up when iterating over the > .values() of some namespace.) They both also need to be able to > indicate a lack of return value. This means that either they return a > (success, value) tuple, or they have some other means of signalling > exhaustion. You are grouping next() and it.__next__() together, but they are different. I think we agree that the __next__() method is part of the iterator protocol and should raise StopIteration. There is no fundamental reason why next(), the builtin function, should raise StopIteration, just because __next__(), the method, does. Many xxx() functions that wrap __xxx__() methods add additional functionality. Consider max() or min(). Both of these methods take an iterable and if that iterable is empty they raise a ValueError. If next() did likewise then the original example that motivates this PEP would not be a problem. > > I'm not sure what you mean by your "However" above. In both __next__ > and next(), this is a signal; it becomes an error as soon as you call > next() and don't cope adequately with the signal, just as KeyError is > an error. > >> 2. The proposed solution does not address this issue at all, but rather >> legislates against generators raising StopIteration. > > Because that's the place where a StopIteration will cause a silent > behavioral change, instead of cheerily bubbling up to top-level and > printing a traceback. I must disagree. It is the FOR_ITER bytecode (implementing a loop or comprehension) that "silently" converts a StopIteration exception into a branch. I think the generator's __next__() method handling of exceptions is correct; it propogates them, like most other code. > >> 3. Generators and the iterator protocol were introduced in Python 2.2, 13 >> years ago. >> For all of that time the iterator protocol has been defined by the >> __iter__(), next()/__next__() methods and the use of StopIteration to >> terminate iteration. >> >> Generators are a way to write iterators without the clunkiness of explicit >> __iter__() and next()/__next__() methods, but have always obeyed the same >> protocol as all other iterators. This has allowed code to rewritten from one >> form to the other whenever desired. >> >> Do not forget that despite the addition of the send() and throw() methods >> and their secondary role as coroutines, generators have primarily always >> been a clean and elegant way of writing iterators. > > This question has been raised several times; there is a distinct > difference between __iter__() and __next__(), and it is only the I just mentioned __iter__ as it is part of the protocol, I agree that __next__ is relevant method. > latter which is aware of StopIteration. Compare these three classes: > > class X: > def __init__(self): self.state=0 > def __iter__(self): return self > def __next__(self): > if self.state == 3: raise StopIteration > self.state += 1 > return self.state > > class Y: > def __iter__(self): > return iter([1,2,3]) > > class Z: > def __iter__(self): > yield 1 > yield 2 > yield 3 > > Note how just one of these classes uses StopIteration, and yet all > three are iterable, yielding the same results. Neither Y nor Z is > breaking iterator protocol - but neither of them is writing an > iterator, either. All three raise StopIteration, even if it is implicit. This is trivial to demonstrate: def will_it_raise_stop_iteration(it): try: while True: it.__next__() except StopIteration: print("Raises StopIteration") except: print("Raises something else") > >> 4. Porting from Python 2 to Python 3 seems to be hard enough already. > > Most of the code broken by this change can be fixed by a mechanical > replacement of "raise StopIteration" with "return"; the rest need to > be checked to see if they're buggy or unclear. There is an edge case > with "return some_value" vs "raise StopIteration(some_value)" (the > former's not compatible with 2.7), but apart from that, the > recommended form of code for 3.7 will work in all versions of Python > since 2.2. I think that when it comes to porting 2 to 3, the perception is more important than the technical difficultly. Sadly :( > >> 5. I think I've already covered this in the other points, but to reiterate >> (excuse the pun): >> Calling next() on an exhausted iterator is, I would suggest, a logical >> error. > > How do you know that it's exhausted, other than by calling next() on it? Either we add a new method, or you have to handle the exception explicitly. But that is what you are trying to force anyway. I probably should have said "Calling next(), without guarding against the possibility that the iterator is exhausted, is a logical error." > >> It also worth noting that calling next() is the only place a StopIteration >> exception is likely to occur outside of the iterator protocol. > > This I agree with. > >> An example >> ---------- >> >> Consider a function to return the value from a set with a single member. >> def value_from_singleton(s): >> if len(s) < 2: #Intentional error here (should be len(s) == 1) >> return next(iter(s)) >> raise ValueError("Not a singleton") >> >> Now suppose we pass an empty set to value_from_singleton(s), then we get a >> StopIteration exception, which is a bit weird, but not too bad. > > Only a little weird - and no different from the way you'd get a > TypeError if you pass it an integer. Except that TypeError is what is says, an error. StopIteration is a special not-really-an-error thing. > >> However it is when we use it in a generator (or in the __next__ method of an >> iterator) that we get a serious problem. >> Currently the iterator appears to be exhausted early, which is wrong. >> However, with the proposed change we get RuntimeError("generator raised >> StopIteration") raised, which is also wrong, just in a different way. > > What you have here is two distinct issues. The first is "what happens > if an unexpected StopIteration occurs during __next__ processing?", > and the second is "ditto ditto a generator's execution?". The first > one is extremely hard to deal with, and extremely unlikely. The second > is much easier to deal with, and can therefore be solved. I don't think there are two distinct issues. It is only the combination of the two that causes a real problem. There are two places that StopIteration could be convert into a "real" exception. In the next() function or in the generator.__next__() method. Doing so in next() is, IMO, simpler and easier to understand and explain. > >> Solutions >> --------- >> My preferred "solution" is to do nothing except improving the documentation >> of next(). Explain that it can raise StopIteration which, if allowed to >> propogate can cause premature exhaustion of an iterator. > > Docs fixing doesn't solve everything. True, but docs fixing is always backwards compatible :) > >> If something must be done then I would suggest changing the behaviour of >> next() for an exhausted iterator. >> Rather than raise StopIteration it should raise ValueError (or IndexError?). > > So, if I've understood you correctly, what you're saying is that > __next__ should raise StopIteration, and then next() should absorb > that and raise ValueError instead? I'm not sure how this would help > anything, but I can see that it would poke the issue with a sharp > pointy stick. Can you elaborate on how this would work in practice? How would it help? It would prevent propagation of StopIteration causes premature exhaustion of an iterator. That is what the PEP is about, isn't it? > >> Also, it might be worth considering making StopIteration inherit from >> BaseException, rather than Exception. > > Separate concern altogether, as the bases of StopIteration have > nothing to do with a protocol meaning collision. I would probably > support this change, on the basis that Exception should be for, well, > exceptions, and BaseException can be used for everything that uses the > exception-handling mechanism for other purposes. But it wouldn't help > or affect this proposal. Agreed. Cheers, Mark. From donald at stufft.io Mon Nov 24 01:39:30 2014 From: donald at stufft.io (Donald Stufft) Date: Sun, 23 Nov 2014 19:39:30 -0500 Subject: [Python-Dev] Move selected documentation repos to PSF BitBucket account? In-Reply-To: <20141123235727.GV2748@ando.pearwood.info> References: <19336614-0E4F-42BF-A918-1807BB7F3599@stufft.io> <504EB412-57A2-43B6-ABA1-B29D63395CD6@stufft.io> <20141123235727.GV2748@ando.pearwood.info> Message-ID: > On Nov 23, 2014, at 6:57 PM, Steven D'Aprano wrote: > > On Sun, Nov 23, 2014 at 08:55:50AM -0800, Guido van Rossum wrote: > >> But I strongly believe that if we want to do the right thing for the >> long term, we should switch to GitHub. > > Encouraging a software, or social, monopoly is never the right thing for > the long term. > > http://nedbatchelder.com/blog/201405/github_monoculture.html I don?t think this is really all that big of a deal. If we want to move off of Github doing so is easy. There are lots of (not nearly as good as but probably still better than what we have now) OSS software that gives you a github like flow. The only *data* that is really in there is what?s stored in the repository itself (since I don?t think for anything major we?d ever put issues there or use the wiki) which is trivial to move around. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA From wes.turner at gmail.com Mon Nov 24 01:32:18 2014 From: wes.turner at gmail.com (Wes Turner) Date: Sun, 23 Nov 2014 18:32:18 -0600 Subject: [Python-Dev] Move selected documentation repos to PSF BitBucket account? In-Reply-To: <415502D5-3489-4C66-A84D-73C0998A6FFC@stufft.io> References: <504EB412-57A2-43B6-ABA1-B29D63395CD6@stufft.io> <3CC02307-6109-4F20-A815-BDA7CFE16AC2@stufft.io> <013F5AD9-9C9F-4E2E-A750-1C758B66B675@stufft.io> <5A9CA007-3832-4931-8F05-06BD62BE4A5E@stufft.io> <35957465-A878-4864-9473-9F68B9556387@stufft.io> <415502D5-3489-4C66-A84D-73C0998A6FFC@stufft.io> Message-ID: On Sun, Nov 23, 2014 at 2:38 PM, Donald Stufft wrote: > > > On Nov 23, 2014, at 3:03 PM, Georg Brandl wrote: > > > > The next point is that there is no easy way to change the target branch > of > > a pull request (on github or bitbucket). People will usually make > patches > > against the master branch unless told differently explicitly, which means > > that the pull request will also be against the master branch. Which > means, > > close the PR, ask submitter to resubmit against 3.x branch, or do it > > yourself. > > This in particular is not really true on Github at least. By default PRs > are > made against whichever branch you have configured as the default branch in > the Github UI. This does default to master but it doesn?t have to be, for > instance pip has this configured for the develop branch. Although I think > this specific point is moot because if things were on Git it?d probably > make > the most sense to have the default integration branch be ``master`` just > like > for the Hg repos they use default. > I'm liking the hubflow/gitflow git workflow extensions, where "trunk" is also develop. There are feature branches, hotfix branches, release branches, etc. # .git/config : "prefix": "v" git hf release start 0.1.0 echo "0.1.0" >> VERSION.txt git hf release finish 0.1.0 # (merges, release commit message, return to develop) These are some of the most helpful (git) branching diagrams I've ever seen: * https://datasift.github.io/gitflow/IntroducingGitFlow.html * http://www.infoq.com/articles/agile-version-control The support for tagging releases is helpful. (It's possible to host package releases for various platforms as GitHub releases). > > Even if someone makes a PR against the wrong branch, it ?degrades? into > essentially the same UX as you have now, you can turn a PR into a patch by > adding a .patch or .diff to the end of the PR URL and then you have a patch > file. In additional github makes it easy to check out PRs directly with > only a > minor bit of one time configuration in your local clone. I can checkout > any PR > by doing ``git checkout pr/1`` (replacing 1 with whatever the number is). > > Honestly the worst part about someone sending a PR to the wrong branch is > it > degrades into the same terrible UX that *every* patch has to go through on > a hg.python.org repository right now. > In this respect, hg mq is tightly integrated, and written in Python. Patch Queue / build system of VCS integration could be helpful. * http://hgbook.red-bean.com/read/managing-change-with-mercurial-queues.html * http://hgbook.red-bean.com/read/advanced-uses-of-mercurial-queues.html I agree that - [ ] Build status displayed on the PR [is an extremely useful feature for a code forge to offer] - [ ] Markdown/ReStructuredText Checkboxes are helpful * GitHub: Issues * BitBucket: PR Task Lists integrated with review. My mouse cursor tends to be near the scroll bar alot, as well. -- https://en.wikipedia.org/wiki/Forge_(software)#Technology https://en.wikipedia.org/wiki/Comparison_of_open-source_software_hosting_facilities > --- > Donald Stufft > PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/wes.turner%40gmail.com > -- Wes Turner https://westurner.github.io/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From wes.turner at gmail.com Mon Nov 24 01:49:00 2014 From: wes.turner at gmail.com (Wes Turner) Date: Sun, 23 Nov 2014 18:49:00 -0600 Subject: [Python-Dev] Move selected documentation repos to PSF BitBucket account? In-Reply-To: <20141123235727.GV2748@ando.pearwood.info> References: <19336614-0E4F-42BF-A918-1807BB7F3599@stufft.io> <504EB412-57A2-43B6-ABA1-B29D63395CD6@stufft.io> <20141123235727.GV2748@ando.pearwood.info> Message-ID: On Sun, Nov 23, 2014 at 5:57 PM, Steven D'Aprano wrote: > On Sun, Nov 23, 2014 at 08:55:50AM -0800, Guido van Rossum wrote: > > > But I strongly believe that if we want to do the right thing for the > > long term, we should switch to GitHub. > > Encouraging a software, or social, monopoly is never the right thing for > the long term. > > http://nedbatchelder.com/blog/201405/github_monoculture.html > > > > I promise you that once the pain of the switch is over you will feel > > much better about it. I am also convinced that we'll get more > > contributions this way. > > I'm sure that we'll get *more* contributions, but will they be *better* > contributions? > > I know that there are people who think that mailing lists are old and > passe, and that we should shift discussion to a social media site like > Reddit. If we did, we'd probably get twenty times as many comments, and > the average quality would probably plummet. More is not necessarily a > good thing. > > Reddit: Markdown. Edit. Save. Is this a challenge for software configuration management and software requirements traceability? Could this problem be solved by utilizing namespaced URIs to reference commits and issues? In indexing of these repositories, it would be useful to produce JSON-LD with an appropriate schema for expressing primary and other repo URIs, that could be resolved into the appropriate ssh:// URLs for each particular ascii domain regex. - [ ] schema.org/SofwareApplication - [ ] SEON Ontologies - [ ] Warehouse/PyPi RDFa, JSON-LD With secondarily productivity-enhancing URI/URL edges between development artifacts and feedback review cycles being an overall objective for infrastructure and devops comprehension: * Mailing lists, reviews, issues past current and future, documentation pages, releases * Indices (is there a sphinx index that could build the PEPs?) * Reviews (rietveld, issue tracker) * Issues Past * Issues Current * Issues Future * Docs * PEPs * HOWTOs * {this could be a useful set of types with URIs and current URLs} When/if a r'#\d+' is referenced, how do the URLs resolve at render time? When/if a 'r#issue/tracker/uri/{num}' URL is pasted, is there a pingback? * sphinx extlinks * https://github.com/lunaryorn/sphinxcontrib-issuetracker * https://en.wikipedia.org/wiki/Requirements_traceability -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Mon Nov 24 02:15:46 2014 From: steve at pearwood.info (Steven D'Aprano) Date: Mon, 24 Nov 2014 12:15:46 +1100 Subject: [Python-Dev] Move selected documentation repos to PSF BitBucket account? In-Reply-To: References: <504EB412-57A2-43B6-ABA1-B29D63395CD6@stufft.io> <20141123235727.GV2748@ando.pearwood.info> Message-ID: <20141124011546.GY2748@ando.pearwood.info> On Sun, Nov 23, 2014 at 06:08:07PM -0600, Brian Curtin wrote: > On Sun, Nov 23, 2014 at 5:57 PM, Steven D'Aprano wrote: > > I'm sure that we'll get *more* contributions, but will they be *better* > > contributions? > > > > I know that there are people who think that mailing lists are old and > > passe, and that we should shift discussion to a social media site like > > Reddit. If we did, we'd probably get twenty times as many comments, and > > the average quality would probably plummet. More is not necessarily a > > good thing. > > If we need to ensure that we're getting better contributions than we > are now, then we should be interviewing committers, rejecting > newcomers (or the opposite, multiplying core-mentors by 100), and > running this like a business. I've written some crappy code that got > committed, so I should probably be fired. None of those things are guarenteed to lead to better contributions. The quality of code from the average successful business is significantly lower than that from successful FOSS projects like Python. Interviews just weed out people who are poor interviewees, not poor performers. And any organisation that fires contributors for relatively trivial mistakes like "crappy code" would soon run out of developers. My point is that increasing the number of contributions is not, in and of itself, a useful aim to have. More contributions is just a means to an end, the end we want is better Python. > Enabling our community to be active contributors is an important > thing. Give them a means to level up and we'll all be better off from > it. Right. But this isn't a feel-good exercise where anyone who wants a Gold Star for contributing gets commit privileges. (That would "enable our community to be active contributors" too.) Barriers to contribute work two ways: (1) we miss out on good contributions we would want; (2) we also miss out on poor contributions that would just have to be rejected. Enabling more people to contribute increases both. -- Steven From guido at python.org Mon Nov 24 02:18:56 2014 From: guido at python.org (Guido van Rossum) Date: Sun, 23 Nov 2014 17:18:56 -0800 Subject: [Python-Dev] Please reconsider PEP 479. In-Reply-To: <54724117.20408@hotpy.org> References: <54724117.20408@hotpy.org> Message-ID: Chris already responded; I'll top-post to clarify my position. I acted quickly because too many people were expending too much energy debating the issue without coming up with a different resolution, and I am unhappy with the status quo. Many people don't seem to see the difference between the iterator *protocol*, which determines how the caller discovered that the iterator is exhausted, and various iterator *implementations*, which determine how you feed a series of values to the caller and indicate the end of that series. Generators are one possible implementation. The crucial insight, leading to the solution, is that a generator is one step removed from the object representing the protocol. When you call a generator, two objects are created: a generator protocol object (this is what we typically refer to as a generator object, as opposed to a generator function), and a stack frame. The stack frame is initially in a suspended state, and is poised to execute the code of the generator function. The protocol object has a __next__() method. When this is called, the associated stack frame is resumed until it either yields or returns -- if it yields, __next__() returns a value, but if it returns, __next__() raises StopIteration. There is no need for the stack frame to raise StopIteration -- the StopIteration is provided "for free" by the generator protocol object. The confusion stems from the fact that if the stack frame neither yields nor returns but raises an exception, this exception is simply passed through the protocol object, and in most cases (e.g. an AttributeError) this simply bubbles out until it is either caught or terminates the program or thread with a printed stack trace. But in the special case that the frame raises StopIteration, once this is raised by the protocol object's __next__(), this is indistinguishable from the StopIteration that this object throws when the stack frame simply returns. It wouldn't be so bad if we had the occasional generator author writing "raise StopIteration" instead of "return" to exit from a generator. (We could just add a recommendation against this to the style guide.) But the problem is that an unguarded next() call also raises StopIteration. Sometimes this is intentional (as in some itertools examples). But sometimes an unguarded next() call occurs deep in the bowels of some code called by the generator, and this situation is often hard to debug, since there is no stack track. One more thing. You correctly say that the built-in next() function is not the same thing as the __next__() method of the protocol. While this is true, tinkering with next() is not feasible -- if we were to change the interface of next() we would have to change pretty much every call to it. (I did a little research on a large proprietary codebase to which I have access -- I found innumerable next() calls, almost all of them guarded by a "try: ... except StopIteration: ..." block. Even the ones inside generators. On the other hand, I found almost no code depending on the behavior that the PEP changes, apart from a few places where a redundant "raise StopIteration" was found at the end of a generator. So. To reiterate my point. A generator frame uses "yield " to produce a value and "return" to terminate the iteration; the generator protocol object wrapping the frame translates these into "return " and "raise StopIteration", respectively. Raising StopIteration in the frame is redundant and can cause errors to pass silently, hence this is the right thing to change. On Sun, Nov 23, 2014 at 12:18 PM, Mark Shannon wrote: > Hi, > > I have serious concerns about this PEP, and would ask you to reconsider it. > > [ Very short summary: > Generators are not the problem. It is the naive use of next() in an > iterator that is the problem. (Note that all the examples involve calls to > next()). > Change next() rather than fiddling with generators. > ] > > I have five main concerns with PEP 479. > 1. Is the problem, as stated by the PEP, really the problem at all? > 2. The proposed solution does not address the underlying problem. > 3. It breaks a fundamental aspect of generators, that they are iterators. > 4. This will be a hindrance to porting code from Python 2 to Python 3. > 5. The behaviour of next() is not considered, even though it is the real > cause of the problem (if there is a problem). > > 1. The PEP states that "The interaction of generators and StopIteration is > currently somewhat surprising, and can conceal obscure bugs." > I don't believe that to be the case; if someone knows what StopIteration > is and how it is used, then the interaction is entirely as expected. > > I believe the naive use of next() in an iterator to be the underlying > problem. > The interaction of generators and next() is just a special case of this. > > StopIteration is not a normal exception, indicating a problem, rather it > exists to signal exhaustion of an iterator. > However, next() raises StopIteration for an exhausted iterator, which > really is an error. > Any iterator code (generator or __next__ method) that calls next() treats > the StopIteration as a normal exception and propogates it. > The controlling loop then interprets StopIteration as a signal to stop and > thus stops. > *The problem is the implicit shift from signal to error and back to > signal.* > > 2. The proposed solution does not address this issue at all, but rather > legislates against generators raising StopIteration. > > 3. Generators and the iterator protocol were introduced in Python 2.2, 13 > years ago. > For all of that time the iterator protocol has been defined by the > __iter__(), next()/__next__() methods and the use of StopIteration to > terminate iteration. > > Generators are a way to write iterators without the clunkiness of explicit > __iter__() and next()/__next__() methods, but have always obeyed the same > protocol as all other iterators. This has allowed code to rewritten from > one form to the other whenever desired. > > Do not forget that despite the addition of the send() and throw() methods > and their secondary role as coroutines, generators have primarily always > been a clean and elegant way of writing iterators. > > 4. Porting from Python 2 to Python 3 seems to be hard enough already. > > 5. I think I've already covered this in the other points, but to reiterate > (excuse the pun): > Calling next() on an exhausted iterator is, I would suggest, a logical > error. > However, next() raises StopIteration which is really a signal to the > controlling loop. > The fault is with next() raising StopIteration. > Generators raising StopIteration is not the problem. > > It also worth noting that calling next() is the only place a StopIteration > exception is likely to occur outside of the iterator protocol. > > An example > ---------- > > Consider a function to return the value from a set with a single member. > def value_from_singleton(s): > if len(s) < 2: #Intentional error here (should be len(s) == 1) > return next(iter(s)) > raise ValueError("Not a singleton") > > Now suppose we pass an empty set to value_from_singleton(s), then we get a > StopIteration exception, which is a bit weird, but not too bad. > > However it is when we use it in a generator (or in the __next__ method of > an iterator) that we get a serious problem. > Currently the iterator appears to be exhausted early, which is wrong. > However, with the proposed change we get RuntimeError("generator raised > StopIteration") raised, which is also wrong, just in a different way. > > Solutions > --------- > My preferred "solution" is to do nothing except improving the > documentation of next(). Explain that it can raise StopIteration which, if > allowed to propogate can cause premature exhaustion of an iterator. > > If something must be done then I would suggest changing the behaviour of > next() for an exhausted iterator. > Rather than raise StopIteration it should raise ValueError (or > IndexError?). > > Also, it might be worth considering making StopIteration inherit from > BaseException, rather than Exception. > > > Cheers, > Mark. > > P.S. 5 days seems a rather short time to respond to a PEP. > Could we make it at least a couple of weeks in the future, > or better still specify a closing date for comments. > > > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From dw+python-dev at hmmz.org Mon Nov 24 02:03:26 2014 From: dw+python-dev at hmmz.org (David Wilson) Date: Mon, 24 Nov 2014 01:03:26 +0000 Subject: [Python-Dev] Move selected documentation repos to PSF BitBucket account? In-Reply-To: References: <504EB412-57A2-43B6-ABA1-B29D63395CD6@stufft.io> <20141123235727.GV2748@ando.pearwood.info> Message-ID: <20141124010326.GA20756@k3> On Sun, Nov 23, 2014 at 07:39:30PM -0500, Donald Stufft wrote: > I don?t think this is really all that big of a deal. If we want to > move off of Github doing so is easy. There are lots of (not nearly as > good as but probably still better than what we have now) OSS software > that gives you a github like flow. The only *data* that is really in > there is what?s stored in the repository itself (since I don?t think > for anything major we?d ever put issues there or use the wiki) which > is trivial to move around. Assuming PRs are enabled on the new repo, how will that interact with patch review/the issue tracker? Is the goal here to replace the existing process wholesale in one step? It doesn't seem fair to say that moving to GitHub will be easy unless interrupting every contributor's flow is considered "free", or some Rietveld bridge script magically writes itself. The former impacts review bandwidth, the latter infrastructure bandwidth (which already seems quite contended, e.g. given the job board is still MIA). David From olemis at gmail.com Mon Nov 24 02:43:59 2014 From: olemis at gmail.com (Olemis Lang) Date: Sun, 23 Nov 2014 20:43:59 -0500 Subject: [Python-Dev] Please reconsider PEP 479. In-Reply-To: <54727AE0.6030301@hotpy.org> References: <54724117.20408@hotpy.org> <54727AE0.6030301@hotpy.org> Message-ID: On 11/23/14, Mark Shannon wrote: > [...] > > You are grouping next() and it.__next__() together, but they are different. > I think we agree that the __next__() method is part of the iterator > protocol and should raise StopIteration. > There is no fundamental reason why next(), the builtin function, should > raise StopIteration, just because __next__(), the method, does. > Many xxx() functions that wrap __xxx__() methods add additional > functionality. > > Consider max() or min(). Both of these methods take an iterable and if > that iterable is empty they raise a ValueError. > If next() did likewise then the original example that motivates this PEP > would not be a problem. > FWIW , I fully agree with this . My (personal) impression about PEP 479 is that 1. All the drawbacks mentioned by Raymond Hettinger , Nick Coghlan et al. in the other thread are **really** serious 2. The PEP actually over-complicates the so-far-*natural* definition of the iterator protocol for generators ... and proposed solution adds more issues than it really solves . 3. "The fault is with next() raising StopIteration. Generators raising StopIteration is not the problem." since the later is just an instance of sub-typing whereas the former is more about an exceptional branch >> >> I'm not sure what you mean by your "However" above. In both __next__ >> and next(), this is a signal; it becomes an error as soon as you call >> next() and don't cope adequately with the signal, just as KeyError is >> an error. >> >>> 2. The proposed solution does not address this issue at all, but rather >>> legislates against generators raising StopIteration. >> >> Because that's the place where a StopIteration will cause a silent >> behavioral change, instead of cheerily bubbling up to top-level and >> printing a traceback. > I must disagree. It is the FOR_ITER bytecode (implementing a loop or > comprehension) that "silently" converts a StopIteration exception into a > branch. > > I think the generator's __next__() method handling of exceptions is > correct; it propogates them, like most other code. > This is really true and is the basis for composing generator expressions (the discussion's been too long I do not want to add more examples to emphasize this point) . IMHO StopIteration should be propagated up to the caller in the context of iterator composition (i.e. definition) as opposite to the case of client code actually *using* (i.e. consuming) the generator (and the difference between both scenarios is somehow similar to has-a vs is-a in classical OO subtyping scenarios) . In the later case (use) raising ValueError or RuntimeError (I'd prefer the former) would be really helpful , so I really favor doing so in next() method rather than over-complicating generators (and breaking the iterator subtype condition) for no good (IMHO) reason . [...] p.s. I know that the PEP has been accepted by the BDFL , but I really think this is an important concern , that's why I insist for the sake of helping ... in case this represents a serious violation of established rules please send me a private message and I will not do it again (... but I'm hoping , after all, that post-acceptance debate will not be considered as harmful when there's a good reason according to someone ...) -- Regards, Olemis - @olemislc Apache(tm) Bloodhound contributor http://issues.apache.org/bloodhound http://blood-hound.net Blog ES: http://simelo-es.blogspot.com/ Blog EN: http://simelo-en.blogspot.com/ Featured article: From barry at python.org Mon Nov 24 03:55:31 2014 From: barry at python.org (Barry Warsaw) Date: Sun, 23 Nov 2014 21:55:31 -0500 Subject: [Python-Dev] Move selected documentation repos to PSF BitBucket account? In-Reply-To: References: <20141121102647.46e97f04@limelight.wooz.org> <19336614-0E4F-42BF-A918-1807BB7F3599@stufft.io> <504EB412-57A2-43B6-ABA1-B29D63395CD6@stufft.io> Message-ID: <20141123215531.31536cae@limelight.wooz.org> On Nov 23, 2014, at 04:49 PM, Nick Coghlan wrote: >Moving from self-hosted Mercurial repos to externally hosted Mercurial >repos is a low risk change. It reduces maintenance overhead and lowers >barriers to external contribution, both without alienating existing >contributors by forcing them to change their workflows. If those repos are externally maintained, what kind of assurances will people have that they are talking to the *official* repositories of the PSF owned assets? One of the problem IMHO of the democratization of branches that a dvcs provides is knowing when you are interacting with the official code of the project. In general, more democracy is better, but that needs to be balanced with verifiable reputation. Having branches hosted on python.org gives people that immediately. Having branches hosted on other domains means there's more uncertainty. Even aside from the "we should support open source" question (which I feel strongly about but acknowledge others have less allegiance to), it must be absolutely clear that there are repositories which we as the Python development community, bless. Cheers, -Barry From barry at python.org Mon Nov 24 03:59:21 2014 From: barry at python.org (Barry Warsaw) Date: Sun, 23 Nov 2014 21:59:21 -0500 Subject: [Python-Dev] Move selected documentation repos to PSF BitBucket account? In-Reply-To: References: <20141121102647.46e97f04@limelight.wooz.org> <19336614-0E4F-42BF-A918-1807BB7F3599@stufft.io> <504EB412-57A2-43B6-ABA1-B29D63395CD6@stufft.io> Message-ID: <20141123215921.79d2001b@limelight.wooz.org> On Nov 23, 2014, at 08:55 AM, Guido van Rossum wrote: >- Moving from Hg to Git is a fair amount of one-time work For anyone seriously interested in this, even experimentally, I would highly suggest looking at Eric Raymond's reposurgeon code. You can google it or find it covered in the vast discussions of his conversion of the Emacs repository from bazaar to git. I don't know for a fact that it would handle an hg to git conversion, but it's the most likely candidate for something so complex. There's a lot of similarity in the history of the Emacs and Python repositories, having gone through many previous iterations of vcs's - in Python's case, at least cvs, svn, and hg. Cheers, -Barry From barry at python.org Mon Nov 24 04:17:05 2014 From: barry at python.org (Barry Warsaw) Date: Sun, 23 Nov 2014 22:17:05 -0500 Subject: [Python-Dev] Move selected documentation repos to PSF BitBucket account? In-Reply-To: References: <20141121102647.46e97f04@limelight.wooz.org> <19336614-0E4F-42BF-A918-1807BB7F3599@stufft.io> <504EB412-57A2-43B6-ABA1-B29D63395CD6@stufft.io> Message-ID: <20141123221705.087df828@limelight.wooz.org> On Nov 23, 2014, at 03:59 PM, Nick Coghlan wrote: >The learning curve on git is still awful What I find so ironic is that git's model is beautifully simple, but its cli is abysmal, and its manpages are less than helpful. git-push(1) is over 650 lines and it's nearly impossible to dig out the most important bits. Cheers, -Barry From benjamin at python.org Mon Nov 24 04:18:04 2014 From: benjamin at python.org (Benjamin Peterson) Date: Sun, 23 Nov 2014 22:18:04 -0500 Subject: [Python-Dev] ubuntu buildbot Message-ID: <1416799084.1792952.194543661.2FC7FCAA@webmail.messagingengine.com> Hi David, I noticed you run the "Builder x86 Ubuntu Shared" buildbot. It seems it's running a very old version of Ubuntu. Is there any chance of getting that updated? Regards, Benjamin From skip.montanaro at gmail.com Mon Nov 24 04:28:51 2014 From: skip.montanaro at gmail.com (Skip Montanaro) Date: Sun, 23 Nov 2014 21:28:51 -0600 Subject: [Python-Dev] Move selected documentation repos to PSF BitBucket account? In-Reply-To: <20141123221705.087df828@limelight.wooz.org> References: <20141121102647.46e97f04@limelight.wooz.org> <19336614-0E4F-42BF-A918-1807BB7F3599@stufft.io> <504EB412-57A2-43B6-ABA1-B29D63395CD6@stufft.io> <20141123221705.087df828@limelight.wooz.org> Message-ID: > git-push(1) is over 650 lines and it's nearly > impossible to dig out the most important > bits. I use git daily at work. I try to use it in the most simple way possible. My frustration with the man pages got to the point where I basically use Google to ask my questions, then bookmark the solutions I find (which often turn out to be on stackoverflow). Skip -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Mon Nov 24 04:42:42 2014 From: guido at python.org (Guido van Rossum) Date: Sun, 23 Nov 2014 19:42:42 -0800 Subject: [Python-Dev] Move selected documentation repos to PSF BitBucket account? In-Reply-To: References: <20141121102647.46e97f04@limelight.wooz.org> <19336614-0E4F-42BF-A918-1807BB7F3599@stufft.io> <504EB412-57A2-43B6-ABA1-B29D63395CD6@stufft.io> <20141123221705.087df828@limelight.wooz.org> Message-ID: On Sunday, November 23, 2014, Skip Montanaro wrote: > > > git-push(1) is over 650 lines and it's nearly > > impossible to dig out the most important > > bits. > > I use git daily at work. I try to use it in the most simple way possible. > My frustration with the man pages got to the point where I basically use > Google to ask my questions, then bookmark the solutions I find (which often > turn out to be on stackoverflow). > Then there's this. http://git-man-page-generator.lokaltog.net/ -- --Guido van Rossum (on iPad) -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Mon Nov 24 06:03:44 2014 From: rosuav at gmail.com (Chris Angelico) Date: Mon, 24 Nov 2014 16:03:44 +1100 Subject: [Python-Dev] Move selected documentation repos to PSF BitBucket account? In-Reply-To: References: <20141121102647.46e97f04@limelight.wooz.org> <19336614-0E4F-42BF-A918-1807BB7F3599@stufft.io> <504EB412-57A2-43B6-ABA1-B29D63395CD6@stufft.io> <20141123221705.087df828@limelight.wooz.org> Message-ID: On Mon, Nov 24, 2014 at 2:42 PM, Guido van Rossum wrote: > Then there's this. http://git-man-page-generator.lokaltog.net/ Wow.... scarily accurate. http://git-man-page-generator.lokaltog.net/#2d1a13476a5f32c4db27fd7aa89a84f3 Anything to do with git submodules is virtually impossible to distinguish from an elaborate practical joke. I know this because I have tried to use them. But there is hope: the git maintainers *do* accept docs patches. https://www.kernel.org/pub/software/scm/git/docs/git-config.html#_configuration_file Formerly, this said "You will find a description of non-core porcelain configuration variables in the respective porcelain documentation in the appropriate manual page.". Did you know that that means it's acceptable for a third-party git hook to make use of 'git config' to allow end users to configure it? Neither did I, till I asked on the mailing list. However, there are enough git front-ends and tutorials that mean you can generally ignore the man pages until you need something more complicated. Basically, consider the git man pages at the same time you'd consider enabling a Mercurial extension. That's how it seems to be. ChrisA From db3l.net at gmail.com Mon Nov 24 06:33:01 2014 From: db3l.net at gmail.com (David Bolen) Date: Mon, 24 Nov 2014 00:33:01 -0500 Subject: [Python-Dev] ubuntu buildbot In-Reply-To: <1416799084.1792952.194543661.2FC7FCAA@webmail.messagingengine.com> References: <1416799084.1792952.194543661.2FC7FCAA@webmail.messagingengine.com> Message-ID: Yeah, it definitely needs it. Historically it was intentional as my own servers were all on 8.04, but the last of those moved 12.04 last year. I think there's already a 12.04 buildbot, so perhaps 14.04 would be better? I do prefer sticking with an LTS. It'll need to move to 64-bit given the hosting environment, at least for the kernel. I could do 32-bit userspace via multiarch if keeping a 32-bit buildbot in the mix is attractive. -- David On Sun, Nov 23, 2014 at 10:18 PM, Benjamin Peterson wrote: > Hi David, > I noticed you run the "Builder x86 Ubuntu Shared" buildbot. It seems > it's running a very old version of Ubuntu. Is there any chance of > getting that updated? > > Regards, > Benjamin > -------------- next part -------------- An HTML attachment was scrubbed... URL: From stephen at xemacs.org Mon Nov 24 07:19:48 2014 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Mon, 24 Nov 2014 15:19:48 +0900 Subject: [Python-Dev] Move selected documentation repos to PSF BitBucket account? In-Reply-To: References: <504EB412-57A2-43B6-ABA1-B29D63395CD6@stufft.io> <3CC02307-6109-4F20-A815-BDA7CFE16AC2@stufft.io> <013F5AD9-9C9F-4E2E-A750-1C758B66B675@stufft.io> <5A9CA007-3832-4931-8F05-06BD62BE4A5E@stufft.io> <35957465-A878-4864-9473-9F68B9556387@stufft.io> Message-ID: <87egstrugr.fsf@uwakimon.sk.tsukuba.ac.jp> Brett Cannon writes: > How do other projects tend to manage their bugfix vs. in-dev branches? Emacs uses a similar workflow to Python's current one, AIUI: 1. When feasible, developer decides the lowest applicable branch (in Emacs there are only two), commits and pushes there. 2. When needed, merge forward to trunk, supported by a script that helps manage "already cherry-picked from trunk" and "not suitable for trunk" patches. > Is it a lot of cherrypicking? For Emacs, not that much. Most feature development occurs on trunk, and rarely are commits to trunk considered appropriate for cherrypicking to stable. Most bugfixing is in response to reports on stable, or quickly confirmed to apply to stable, is handled in stable, and then merged forward manually. > Probably our long development cycles doesn't help with this as it > makes cross-branch merging difficult thanks to divergence, I suspect that the time from bug injection to bug discovery, which is primarily related to length of support commitment, is more relevant than the length of the development cycle. From ncoghlan at gmail.com Mon Nov 24 08:25:21 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 24 Nov 2014 17:25:21 +1000 Subject: [Python-Dev] Move selected documentation repos to PSF BitBucket account? In-Reply-To: References: <20141121102647.46e97f04@limelight.wooz.org> <19336614-0E4F-42BF-A918-1807BB7F3599@stufft.io> <504EB412-57A2-43B6-ABA1-B29D63395CD6@stufft.io> <3CC02307-6109-4F20-A815-BDA7CFE16AC2@stufft.io> <013F5AD9-9C9F-4E2E-A750-1C758B66B675@stufft.io> <5A9CA007-3832-4931-8F05-06BD62BE4A5E@stufft.io> <35957465-A878-4864-9473-9F68B9556387@stufft.io> Message-ID: On 24 November 2014 at 02:55, Brett Cannon wrote: > On Sun Nov 23 2014 at 6:18:46 AM Nick Coghlan wrote: >> Those features are readily accessible without changing the underlying >> version control system (whether self-hosted through Kallithea or externally >> hosted through BitBucket or RhodeCode). Thus the folks that want to change >> the version control system need to make the case that doing so will provide >> additional benefits that *can't* be obtained in a less disruptive way. > > I guess my question is who and what is going to be disrupted if we go with > Guido's suggestion of switching to GitHub for code hosting? Contributors > won't be disrupted at all since most people are more familiar with GitHub > vs. Bitbucket (how many times have we all heard the fact someone has even > learned Mercurial just to contribute to Python?). Core developers might be > based on some learned workflow, but I'm willing to bet we all know git at > this point (and for those of us who still don't like it, myself included, > there are GUI apps to paper over it or hg-git for those that prefer a CLI). > Our infrastructure will need to be updated, but how much of it is that > hg-specific short of the command to checkout out the repo? Obviously > Bitbucket is much more minor by simply updating just a URL, but changing `hg > clone` to `git clone` isn't crazy either. Georg, Antoine, or Benjamin can > point out if I'm wrong on this, maybe Donald or someone in the > infrastructure committee. Are you volunteering to write a competing PEP for a migration to git and GitHub? I won't be updating PEP 474 to recommend moving to either, as I don't think that would be a good outcome for the Python ecosystem as a whole. It massively undercuts any possible confidence anyone else might have in Mercurial, BitBucket, Rhodecode, Kallithea & Allura (all Python based version control, or version control hosting, systems). If we as the Python core development team don't think any of those are good enough to meet the modest version control needs of our support repos, why on earth would anyone else choose them? In reality, I think most of these services are pretty interchangeable - GitHub's just been the most effective at the venture capital powered mindshare grab business model (note how many of the arguments here stem from the fact folks like *other* things that only interoperate with GitHub, and no other repository hosting providers - that's the core of the A18z funded approach to breaking the "D" in DVCS and ensuring that GitHub's investors are in a position to clip the ticket when GitHub eventually turns around and takes advantage of its dominant market position to increase profit margins). That's why I consider it legitimate to treat supporting fellow Python community members as the determining factor - a number of the available options meet the "good enough" bar from a technical perspective, so it's reasonable to take other commercial and community factors into account when making a final decision. > Probably the biggest thing I can think of that would need updating is our > commit hooks. Once again Georg, Antoine, or Benjamin could say how difficult > it would be to update those hooks. If CPython eventually followed suit in migrating to git (as seems inevitable if all the other repos were to switch), then every buildbot will also need to be updated to have git installed (and Mercurial removed). >> From my perspective, swapping out Mercurial for git achieves exactly >> nothing in terms of alleviating the review bottleneck (since the core >> developers that strongly prefer the git UI will already be using an >> adapter), and is in fact likely to make it worse by putting the greatest >> burden in adapting to the change on the folks that are already under the >> greatest time pressure. > > That's not entirely true. If you are pushing a PR shift in our patch > acceptance workflow then Bitbucket vs. GitHub isn't fundamentally any > different in terms of benefit, and I would honestly argue that GitHub's PR > experience is better. IOW either platform is of equal benefit. Yes, I agree any real benefit comes from the PR workflow, not from git. This is why I consider "written in Python" to be a valid determining factor - multiple services meet the "good enough" bar from a practical perspective, allowing other considerations to come to the fore. (Also note that this proposal does NOT currently cover CPython itself. Neither GitHub nor BitBucket is set up to handle maintenance branches well, and any server side merge based workflow improvements for CPython are gated on fixing the NEWS file maintenance issue. However, once you contemplate moving CPython, then the ripple effects on other systems become much larger) >> It's also worth keeping in mind that changing the underlying VCS means >> changing *all* the automation scripts, rather than just updating the >> configuration settings to reflect a new hosting URL. > > What are the automation scripts there are that would require updating? I > would like to a list and to have the difficulty of moving them mentioned to > know what the impact would be. For the documentation repos, just the devguide and PEP update scripts come to mind. As noted above, the implications get more significant if the main CPython repo eventually follows suit and the buildbot infrastructure all needs to be updated. >> Orchestrating this kind of infrastructure enhancement for Red Hat *is* my >> day job, and you almost always want to go for the lowest impact, lowest risk >> approach that will alleviate the bottleneck you're worried about while >> changing the smallest possible number of elements in the overall workflow >> management system. > > Sure, but I would never compare our infrastructure needs to Red Hat. =) You > also have to be conservative in order to minimize downtown and impact for > cost reasons. As an open source project we don't have those kinds of worry; > we just have to worry about keeping everyone happy. Switching to a proprietary hosting service written in a mixture of Ruby, C & bash wouldn't make me happy. If that's the end result of this thread, I'll be sorry I even suggested the idea of reverting to external hosting at all. That outcome would be the antithesis of the PSF's overall mission, whereas I started this thread at least in part due to a discussion about ways the PSF board might be able to help resolve some of the current CPython workflow issues. Offering the use of the PSF's existing BitBucket org account as hosting location for Mercurial repos was an idea I first brought up in that PSF board thread, and then moved over here since it seemed worthwhile to at least make the suggestion and see what people thought. >> That underlying calculation doesn't really change much even when the units >> shift from budget dollars to volunteers' time and energy. > > So here is what I want to know to focus this discussion: > > First, what new workflow are you proposing regardless of repo hosting > provider? Are you proposing we maintain just mirrors and update the devguide > to tell people to fork on the hosting provider, make their changes, generate > a patch (which can be as simple as telling people how find the raw diff on > the hosting provider), and then upload the patch the issue tracker just like > today? Are you going farther and saying we have people send PRs on the > hosting site, have them point to their PR in the issue tracker, and then we > accept PRs (I'm going on the assumption we are not dropping our issue > tracker)? I am proposing that we switch at least some documentation-only repos to a full PR based workflow, including support for online editing (to make it easy to fix simple typos and the like without even leaving the browser). CPython itself would remain completely unaffected. The proposal in PEP 474 is that we do that by setting up Kallithea as forge.python.org. This thread was about considering BitBucket as an alternative approach. RhodeCode would a third option that still didn't involve switching away from Mercurial. > Second, to properly gauge the impact of switching from git to hg from an > infrastructure perspective, what automation scripts do we have and how > difficult would it be to update them to use git instead of hg? This is > necessary simply to know where we would need to update URLs, let alone > change in DVCS. The problems with changing version control systems don't really become significant until we start talking about switching CPython itself, rather than the support repos. > Third, do our release managers care about hg vs. git strongly? They probably > use the DVCS the most directly and at a lower level by necessity compared to > anyone else. > > Fourth, do any core developers feel strongly about not using GitHub? Now > please notice I said "GitHub" and not "git"; I think the proper way to frame > this whole discussion is we are deciding if we want to switch to Bitbucket > or GitHub who provide a low-level API for their version control storage > service through hg or git, respectively. I personally dislike git, but I > really like GitHub and I don't even notice git there since I use GitHub's OS > X app; as I said, I view this as choosing a platform and not the underlying > DVCS as I have happily chosen to access the GitHub hosting service through > an app that is not git (it's like accessing a web app through it's web page > or its REST API). Yes, I object strongly to the use of GitHub when there are commercially supported services written in Python like BitBucket and RhodeCode available if we want to go the external hosting route, and other options like the RhodeCode derived Kallithea if we want to run a self-hosted forge. RhodeCode are even PSF sponsors - I'm sure they'd be willing to discuss the possibility of hosting core development repos on their service. If I was doing a full risk management breakdown, then RhodeCode would be the obvious winner, as not only are they PSF sponsors, but reverting to self-hosting on Kallithea would remain available as an exit strategy. I only suggested BitBucket in this thread because the PSF already has some repos set up there, so that seemed easier than establishing a new set of repos on a RhodeCode hosted instance. > At least for me, until we get a clear understanding of what workflow changes > we are asking for both contributors and core developers and exactly what > work would be necessary to update our infrastructure for either Bitbucket or > GitHub we can't really have a reasonable discussion that isn't going to be > full of guessing. All repos that migrated away from hg.python.org would move to a PR based workflow, rather than manual patch management on the issue tracker. The migrated repos would likely also use their integrated issue tracker rather than the main CPython one at bugs.python.org. Externally hosted repos would likely retain a regularly updated mirror on hg.python.org to ensure the source remains available even in the event of problems affecting the external hosting provider. > And I'm still in support no matter what of breaking out the HOWTOs and the > tutorial into their own repos for easier updating (having to update the > Python porting HOWTO in three branches is a pain when it should be > consistent across Python releases). Agreed. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Mon Nov 24 08:33:58 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 24 Nov 2014 17:33:58 +1000 Subject: [Python-Dev] Move selected documentation repos to PSF BitBucket account? In-Reply-To: References: <504EB412-57A2-43B6-ABA1-B29D63395CD6@stufft.io> <3CC02307-6109-4F20-A815-BDA7CFE16AC2@stufft.io> <013F5AD9-9C9F-4E2E-A750-1C758B66B675@stufft.io> <5A9CA007-3832-4931-8F05-06BD62BE4A5E@stufft.io> <35957465-A878-4864-9473-9F68B9556387@stufft.io> Message-ID: On 24 November 2014 at 06:42, Brett Cannon wrote: > On Sun Nov 23 2014 at 3:04:05 PM Georg Brandl wrote: >> As for typo fixes, the world does not end when some typos aren't fixed. >> Anyway, for the docs we have an explicit offer to send anything, patch or >> just suggestion, to docs at python.org, and people do make use of it. No >> github account even required. > > > Which is great, but I'm willing to bet clicking "Edit" in GitHub is easier > still than creating the patch and emailing it. I have contributed doc fixes > to a bunch of projects because of that "Edit" button (which is why I think > Nick is so anxious to get it). If I had to do any checkout then I wouldn't > have done them. And even having to email a diff might be too much for some > changes. Yes, it's the ease of noticing a typo or other small, easy to fix error in docs, going to the source page, hitting Edit, and submitting the PR that I like with the online editing support. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Mon Nov 24 10:12:31 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 24 Nov 2014 19:12:31 +1000 Subject: [Python-Dev] Move selected documentation repos to PSF BitBucket account? In-Reply-To: References: <19336614-0E4F-42BF-A918-1807BB7F3599@stufft.io> <504EB412-57A2-43B6-ABA1-B29D63395CD6@stufft.io> <20141123235727.GV2748@ando.pearwood.info> Message-ID: On 24 Nov 2014 10:41, "Donald Stufft" wrote: > > > > On Nov 23, 2014, at 6:57 PM, Steven D'Aprano wrote: > > > > On Sun, Nov 23, 2014 at 08:55:50AM -0800, Guido van Rossum wrote: > > > >> But I strongly believe that if we want to do the right thing for the > >> long term, we should switch to GitHub. > > > > Encouraging a software, or social, monopoly is never the right thing for > > the long term. > > > > http://nedbatchelder.com/blog/201405/github_monoculture.html > > I don?t think this is really all that big of a deal. If we want to move > off of Github doing so is easy. It's only easy to leave if you never adopt any GitHub-only services like Travis CI. It's the same lockin play as the one Google uses: make it easy for people to move their data, so they're less likely to notice you're locking them in with proprietary APIs and ecosystems instead. You unlikely to see significant amounts of VC funding pouring into a company unless the investors see a possible chance to extract monopoly rents at some point in the future. Cheers, Nick. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Mon Nov 24 11:20:35 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 24 Nov 2014 20:20:35 +1000 Subject: [Python-Dev] Move selected documentation repos to PSF BitBucket account? In-Reply-To: <5471FECD.3080906@stoneleaf.us> References: <20141121102647.46e97f04@limelight.wooz.org> <19336614-0E4F-42BF-A918-1807BB7F3599@stufft.io> <504EB412-57A2-43B6-ABA1-B29D63395CD6@stufft.io> <5471FECD.3080906@stoneleaf.us> Message-ID: > On Nov 23, 2014, at 1:49 AM, Nick Coghlan wrote: >>> >>> I took the git knowledge I acquired by necessity at Red Hat and >>> figured out how to apply it to hg. All the same features are there in >>> hg, they're just switched off by default (mainly because the core >>> Mercurial devs are adamant that any potentially history destroying >>> operation in a version control system must be opt-in). > > If you could find the time to write up something about that I'm sure it would be helpful. :) The main thing is to enable some of the extensions mentioned in http://mercurial.selenic.com/wiki/GitConcepts#Command_equivalence_table by adding the appropriate entries to ~/.hgrc. A subset of my own extension set: ============== [extensions] rebase = share = eol = histedit = record = purge = ============== Share (multiple working directories from a single clone) is nice for a multi-branch project like CPython, while record allows selective commits, rebase allows changeset rebasing (as you might expect), histedit allows interactive history editing (akin to git's interactive rebasing), and pure allows deleting commits entirely. eol is the extension for Windows end of line handling that CPython needs. Several of those are off by default due to Mercurial's policy that anything which risks losing data should be off by default - messy history is considered preferable to potentially losing changes you didn't mean to destroy. git, by contrast, is happy to let users destroy history, and considers it their own fault if they mess it up and lose content irrevocably - thus the plethora of "how to use git safely" guides, while there's a relative dearth of "how to enable history modification in Mercurial" guides. The Mercurial team's preferred solution to history modification is changeset evolution, which also version controls any history editing operations so they can be reliably communicated to other users of a shared repo: http://evolution.experimentalworks.net/doc/user-guide.html While that's officially still experimental, it's unlikely to pose significant hazards to anyone coming to Mercurial from git - git's default mode of operation is far more cavalier regarding repository history than evolve will ever be. Another aspect that can be somewhat annoying is the terminology conflict between branches in the git sense and bookmarks vs named branches in the Mercurial sense. In Mercurial, commits made on a branch permanently remember which branch they were made on - you can see in the CPython history, for example, whether a particular commit was made to the 2.7 branch, or the 3.3 branch, etc, just by looking at the commit in isolation. Mercurial bookmarks and git branches, by contrast, are just references to a particular commit that represents the current head of a particular line of development - to find out if a particular commit is on that line of development, you have to trace the commit graph backwards to see if it appears. For a while, Mercurial was also missing support for git-style revision shorthands (you needed an extension to add them), but these days you can do things like "hg diff -r tip^" or "hg diff -r 'p1()^'" to specify relative revset references (while still retaining the full flexibility of Mercurial's programmatic revset syntax if you need it). (Speaking of which, I remembered one key thing which would likely be broken by a switch in the underlying DVCS: the Reitveld/Roundup integration) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From p.f.moore at gmail.com Mon Nov 24 11:44:59 2014 From: p.f.moore at gmail.com (Paul Moore) Date: Mon, 24 Nov 2014 10:44:59 +0000 Subject: [Python-Dev] Move selected documentation repos to PSF BitBucket account? In-Reply-To: References: <20141121102647.46e97f04@limelight.wooz.org> <19336614-0E4F-42BF-A918-1807BB7F3599@stufft.io> <504EB412-57A2-43B6-ABA1-B29D63395CD6@stufft.io> <5471FECD.3080906@stoneleaf.us> Message-ID: On 24 November 2014 at 10:20, Nick Coghlan wrote: > Another aspect that can be somewhat annoying is the terminology > conflict between branches in the git sense and bookmarks vs named > branches in the Mercurial sense. This is probably the thing that hurts me most in git/hg translation. In git, I routinely create branches, play with them, then throw them away (either once they get integrated, or when I decide they are a bad idea). It seems to me to be *much* harder to get that lightweight behaviour in hg - if I create a PR for a hg project that isn't accepted, how do I delete it from my clone? In a way that means it won't reappear when I push/pull from my other PC, where I forgot to *also* delete it? Does the answer to that depend on whether I used a bookmark or a branch (and relatedly, which should I use)? Maybe it's also more complicated than it seems on git, but cleaning up experiments seems much easier to me on git. Not that we're getting offtopic here at all, of course... ;-) Paul From ben+python at benfinney.id.au Mon Nov 24 12:44:35 2014 From: ben+python at benfinney.id.au (Ben Finney) Date: Mon, 24 Nov 2014 22:44:35 +1100 Subject: [Python-Dev] Move selected documentation repos to PSF BitBucket account? References: <3CC02307-6109-4F20-A815-BDA7CFE16AC2@stufft.io> <013F5AD9-9C9F-4E2E-A750-1C758B66B675@stufft.io> <5A9CA007-3832-4931-8F05-06BD62BE4A5E@stufft.io> <35957465-A878-4864-9473-9F68B9556387@stufft.io> Message-ID: <85tx1oyg9o.fsf@benfinney.id.au> Nick Coghlan writes: > Are you volunteering to write a competing PEP for a migration to git > and GitHub? Anyone who does decide to propose either Git or GitHub for hosting Python resources: Please don't conflate the two. Git is a community-supported free-software DVCS system with many viable hosting platforms and providers. GitHub is one proprietary hosting service with features that encourage vendor lock-in, and as Barry Warsaw pointed out it is not answerable to the PSF nor under the PSF's control. There are other hosting solutions without these problems. Everyone here already knows this distinction at some level, but it's far too common to see people assume that an argument in favour of Git will also be a compelling case for GitHub. It isn't, and the case for the latter needs to be made quite separately from the case for the former. No, I'm not offering to write such a PEP either. I'm requesting that we recognise that a promotion of GitHub needs to account for its downsides too, and needs to promote its specific benefits separately from the benefits of Git. -- \ ?There are no significant bugs in our released software that | `\ any significant number of users want fixed.? ?Bill Gates, | _o__) 1995-10-23 | Ben Finney From donald at stufft.io Mon Nov 24 13:01:04 2014 From: donald at stufft.io (Donald Stufft) Date: Mon, 24 Nov 2014 07:01:04 -0500 Subject: [Python-Dev] Move selected documentation repos to PSF BitBucket account? In-Reply-To: <85tx1oyg9o.fsf@benfinney.id.au> References: <3CC02307-6109-4F20-A815-BDA7CFE16AC2@stufft.io> <013F5AD9-9C9F-4E2E-A750-1C758B66B675@stufft.io> <5A9CA007-3832-4931-8F05-06BD62BE4A5E@stufft.io> <35957465-A878-4864-9473-9F68B9556387@stufft.io> <85tx1oyg9o.fsf@benfinney.id.au> Message-ID: <264B330C-F8B9-4775-8B80-019366AF0440@stufft.io> > On Nov 24, 2014, at 6:44 AM, Ben Finney wrote: > > Nick Coghlan writes: > >> Are you volunteering to write a competing PEP for a migration to git >> and GitHub? > > Anyone who does decide to propose either Git or GitHub for hosting > Python resources: Please don't conflate the two. > > Git is a community-supported free-software DVCS system with many viable > hosting platforms and providers. > > GitHub is one proprietary hosting service with features that encourage > vendor lock-in, and as Barry Warsaw pointed out it is not answerable to > the PSF nor under the PSF's control. There are other hosting solutions > without these problems. > > Everyone here already knows this distinction at some level, but it's far > too common to see people assume that an argument in favour of Git will > also be a compelling case for GitHub. It isn't, and the case for the > latter needs to be made quite separately from the case for the former. > > No, I'm not offering to write such a PEP either. I'm requesting that we > recognise that a promotion of GitHub needs to account for its downsides > too, and needs to promote its specific benefits separately from the > benefits of Git. > In many cases Github is git?s killer feature which is why you see a lot of people equate the two. It?s not unusual to see a project switch away from X to git entirely so they can use Github. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA From ncoghlan at gmail.com Mon Nov 24 13:09:10 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 24 Nov 2014 22:09:10 +1000 Subject: [Python-Dev] Move selected documentation repos to PSF BitBucket account? In-Reply-To: <264B330C-F8B9-4775-8B80-019366AF0440@stufft.io> References: <3CC02307-6109-4F20-A815-BDA7CFE16AC2@stufft.io> <013F5AD9-9C9F-4E2E-A750-1C758B66B675@stufft.io> <5A9CA007-3832-4931-8F05-06BD62BE4A5E@stufft.io> <35957465-A878-4864-9473-9F68B9556387@stufft.io> <85tx1oyg9o.fsf@benfinney.id.au> <264B330C-F8B9-4775-8B80-019366AF0440@stufft.io> Message-ID: On 24 November 2014 at 22:01, Donald Stufft wrote: >> On Nov 24, 2014, at 6:44 AM, Ben Finney wrote: >> No, I'm not offering to write such a PEP either. I'm requesting that we >> recognise that a promotion of GitHub needs to account for its downsides >> too, and needs to promote its specific benefits separately from the >> benefits of Git. >> > > In many cases Github is git?s killer feature which is why you see a lot > of people equate the two. It?s not unusual to see a project switch away > from X to git entirely so they can use Github. And this is why the "you can still get your data out" argument doesn't make any sense - if you aren't planning to rely on the proprietary APIs, GitHub is just a fairly mundane git hosting service, not significantly different in capabilities from Gitorious, or RhodeCode, or BitBucket, or GitLab, etc. So you may as well go with one of the open source ones, and be *completely* free from vendor lockin. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From donald at stufft.io Mon Nov 24 13:18:30 2014 From: donald at stufft.io (Donald Stufft) Date: Mon, 24 Nov 2014 07:18:30 -0500 Subject: [Python-Dev] Move selected documentation repos to PSF BitBucket account? In-Reply-To: References: <3CC02307-6109-4F20-A815-BDA7CFE16AC2@stufft.io> <013F5AD9-9C9F-4E2E-A750-1C758B66B675@stufft.io> <5A9CA007-3832-4931-8F05-06BD62BE4A5E@stufft.io> <35957465-A878-4864-9473-9F68B9556387@stufft.io> <85tx1oyg9o.fsf@benfinney.id.au> <264B330C-F8B9-4775-8B80-019366AF0440@stufft.io> Message-ID: <8314B8D5-BFAB-4B68-8067-5B0506FBEDAA@stufft.io> > On Nov 24, 2014, at 7:09 AM, Nick Coghlan wrote: > > On 24 November 2014 at 22:01, Donald Stufft wrote: >>> On Nov 24, 2014, at 6:44 AM, Ben Finney wrote: >>> No, I'm not offering to write such a PEP either. I'm requesting that we >>> recognise that a promotion of GitHub needs to account for its downsides >>> too, and needs to promote its specific benefits separately from the >>> benefits of Git. >>> >> >> In many cases Github is git?s killer feature which is why you see a lot >> of people equate the two. It?s not unusual to see a project switch away >> from X to git entirely so they can use Github. > > And this is why the "you can still get your data out" argument doesn't > make any sense - if you aren't planning to rely on the proprietary > APIs, GitHub is just a fairly mundane git hosting service, not > significantly different in capabilities from Gitorious, or RhodeCode, > or BitBucket, or GitLab, etc. So you may as well go with one of the > open source ones, and be *completely* free from vendor lockin. I don?t really agree unless what you?re saying is that the reason it doesn?t make any sense is because you probably won?t want to get your data out because Github?s features are compelling enough that you don?t want to lose them. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA From donald at stufft.io Mon Nov 24 13:22:51 2014 From: donald at stufft.io (Donald Stufft) Date: Mon, 24 Nov 2014 07:22:51 -0500 Subject: [Python-Dev] Move selected documentation repos to PSF BitBucket account? In-Reply-To: References: <19336614-0E4F-42BF-A918-1807BB7F3599@stufft.io> <504EB412-57A2-43B6-ABA1-B29D63395CD6@stufft.io> <20141123235727.GV2748@ando.pearwood.info> Message-ID: <0706E64B-1319-4FDF-B0EA-EB3C3F0296DF@stufft.io> > On Nov 24, 2014, at 4:12 AM, Nick Coghlan wrote: > > > On 24 Nov 2014 10:41, "Donald Stufft" > wrote: > > > > > > > On Nov 23, 2014, at 6:57 PM, Steven D'Aprano > wrote: > > > > > > On Sun, Nov 23, 2014 at 08:55:50AM -0800, Guido van Rossum wrote: > > > > > >> But I strongly believe that if we want to do the right thing for the > > >> long term, we should switch to GitHub. > > > > > > Encouraging a software, or social, monopoly is never the right thing for > > > the long term. > > > > > > http://nedbatchelder.com/blog/201405/github_monoculture.html > > > > I don?t think this is really all that big of a deal. If we want to move > > off of Github doing so is easy. > > It's only easy to leave if you never adopt any GitHub-only services like Travis CI. It's the same lockin play as the one Google uses: make it easy for people to move their data, so they're less likely to notice you're locking them in with proprietary APIs and ecosystems instead. > > You unlikely to see significant amounts of VC funding pouring into a company unless the investors see a possible chance to extract monopoly rents at some point in the future. > > Cheers, > Nick. > Dropping those services is not harder than running those services on our own anyways. It?s unlikely, for instance, that CPython itself would ever be able to use Travis so if we did move to Github and at some point in the future want to move away from Github we?d simply be reverting to the state we?re in now. We wouldn?t be worse off other than we?d have experienced something better than what we have now. I suppose purposely hobbling ourselves so we never get used to something better is a legitimate way to manage things. It?s not different than saying spam filtering is a ?vendor lockin? on gmail because if you want to switch away from it in the future you?ll have to set up your own spam filtering. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Mon Nov 24 13:54:44 2014 From: donald at stufft.io (Donald Stufft) Date: Mon, 24 Nov 2014 07:54:44 -0500 Subject: [Python-Dev] Move selected documentation repos to PSF BitBucket account? In-Reply-To: References: <20141121102647.46e97f04@limelight.wooz.org> <19336614-0E4F-42BF-A918-1807BB7F3599@stufft.io> <504EB412-57A2-43B6-ABA1-B29D63395CD6@stufft.io> <3CC02307-6109-4F20-A815-BDA7CFE16AC2@stufft.io> <013F5AD9-9C9F-4E2E-A750-1C758B66B675@stufft.io> <5A9CA007-3832-4931-8F05-06BD62BE4A5E@stufft.io> <35957465-A878-4864-9473-9F68B9556387@stufft.io> Message-ID: > On Nov 24, 2014, at 2:25 AM, Nick Coghlan wrote: > > On 24 November 2014 at 02:55, Brett Cannon wrote: >> On Sun Nov 23 2014 at 6:18:46 AM Nick Coghlan wrote: >>> Those features are readily accessible without changing the underlying >>> version control system (whether self-hosted through Kallithea or externally >>> hosted through BitBucket or RhodeCode). Thus the folks that want to change >>> the version control system need to make the case that doing so will provide >>> additional benefits that *can't* be obtained in a less disruptive way. >> >> I guess my question is who and what is going to be disrupted if we go with >> Guido's suggestion of switching to GitHub for code hosting? Contributors >> won't be disrupted at all since most people are more familiar with GitHub >> vs. Bitbucket (how many times have we all heard the fact someone has even >> learned Mercurial just to contribute to Python?). Core developers might be >> based on some learned workflow, but I'm willing to bet we all know git at >> this point (and for those of us who still don't like it, myself included, >> there are GUI apps to paper over it or hg-git for those that prefer a CLI). >> Our infrastructure will need to be updated, but how much of it is that >> hg-specific short of the command to checkout out the repo? Obviously >> Bitbucket is much more minor by simply updating just a URL, but changing `hg >> clone` to `git clone` isn't crazy either. Georg, Antoine, or Benjamin can >> point out if I'm wrong on this, maybe Donald or someone in the >> infrastructure committee. > > Are you volunteering to write a competing PEP for a migration to git and GitHub? > > I won't be updating PEP 474 to recommend moving to either, as I don't > think that would be a good outcome for the Python ecosystem as a > whole. It massively undercuts any possible confidence anyone else > might have in Mercurial, BitBucket, Rhodecode, Kallithea & Allura (all > Python based version control, or version control hosting, systems). If > we as the Python core development team don't think any of those are > good enough to meet the modest version control needs of our support > repos, why on earth would anyone else choose them? If those project?s depend on CPython to pick them then they are doomed because we can only pick one. If we pick Bitbucket than the core development team must not think Rhodecode, Kallithea, & Allura is good enough so why on earth would anyone choose them? Ditto for any of the other options. > > In reality, I think most of these services are pretty interchangeable > - GitHub's just been the most effective at the venture capital powered > mindshare grab business model (note how many of the arguments here > stem from the fact folks like *other* things that only interoperate > with GitHub, and no other repository hosting providers - that's the > core of the A18z funded approach to breaking the "D" in DVCS and > ensuring that GitHub's investors are in a position to clip the ticket > when GitHub eventually turns around and takes advantage of its > dominant market position to increase profit margins). You?ll see those arguments because the other argument is ?softer?, Github just plain works better than Bitbucket does. > > That's why I consider it legitimate to treat supporting fellow Python > community members as the determining factor - a number of the > available options meet the "good enough" bar from a technical > perspective, so it's reasonable to take other commercial and community > factors into account when making a final decision. Sure it?s completely reasonable to take that into account, I don?t think ?Not written in Python? should be a disqualifying statement though. We should pick the best tool for the job. > >> Probably the biggest thing I can think of that would need updating is our >> commit hooks. Once again Georg, Antoine, or Benjamin could say how difficult >> it would be to update those hooks. > > If CPython eventually followed suit in migrating to git (as seems > inevitable if all the other repos were to switch), then every buildbot > will also need to be updated to have git installed (and Mercurial > removed). > >>> From my perspective, swapping out Mercurial for git achieves exactly >>> nothing in terms of alleviating the review bottleneck (since the core >>> developers that strongly prefer the git UI will already be using an >>> adapter), and is in fact likely to make it worse by putting the greatest >>> burden in adapting to the change on the folks that are already under the >>> greatest time pressure. >> >> That's not entirely true. If you are pushing a PR shift in our patch >> acceptance workflow then Bitbucket vs. GitHub isn't fundamentally any >> different in terms of benefit, and I would honestly argue that GitHub's PR >> experience is better. IOW either platform is of equal benefit. > > Yes, I agree any real benefit comes from the PR workflow, not from > git. This is why I consider "written in Python" to be a valid > determining factor - multiple services meet the "good enough" bar from > a practical perspective, allowing other considerations to come to the > fore. > > (Also note that this proposal does NOT currently cover CPython itself. > Neither GitHub nor BitBucket is set up to handle maintenance branches > well, and any server side merge based workflow improvements for > CPython are gated on fixing the NEWS file maintenance issue. However, > once you contemplate moving CPython, then the ripple effects on other > systems become much larger) > >>> It's also worth keeping in mind that changing the underlying VCS means >>> changing *all* the automation scripts, rather than just updating the >>> configuration settings to reflect a new hosting URL. >> >> What are the automation scripts there are that would require updating? I >> would like to a list and to have the difficulty of moving them mentioned to >> know what the impact would be. > > For the documentation repos, just the devguide and PEP update scripts > come to mind. As noted above, the implications get more significant if > the main CPython repo eventually follows suit and the buildbot > infrastructure all needs to be updated. > >>> Orchestrating this kind of infrastructure enhancement for Red Hat *is* my >>> day job, and you almost always want to go for the lowest impact, lowest risk >>> approach that will alleviate the bottleneck you're worried about while >>> changing the smallest possible number of elements in the overall workflow >>> management system. >> >> Sure, but I would never compare our infrastructure needs to Red Hat. =) You >> also have to be conservative in order to minimize downtown and impact for >> cost reasons. As an open source project we don't have those kinds of worry; >> we just have to worry about keeping everyone happy. > > Switching to a proprietary hosting service written in a mixture of > Ruby, C & bash wouldn't make me happy. > > If that's the end result of this thread, I'll be sorry I even > suggested the idea of reverting to external hosting at all. That > outcome would be the antithesis of the PSF's overall mission, whereas > I started this thread at least in part due to a discussion about ways > the PSF board might be able to help resolve some of the current > CPython workflow issues. Offering the use of the PSF's existing > BitBucket org account as hosting location for Mercurial repos was an > idea I first brought up in that PSF board thread, and then moved over > here since it seemed worthwhile to at least make the suggestion and > see what people thought. > >>> That underlying calculation doesn't really change much even when the units >>> shift from budget dollars to volunteers' time and energy. >> >> So here is what I want to know to focus this discussion: >> >> First, what new workflow are you proposing regardless of repo hosting >> provider? Are you proposing we maintain just mirrors and update the devguide >> to tell people to fork on the hosting provider, make their changes, generate >> a patch (which can be as simple as telling people how find the raw diff on >> the hosting provider), and then upload the patch the issue tracker just like >> today? Are you going farther and saying we have people send PRs on the >> hosting site, have them point to their PR in the issue tracker, and then we >> accept PRs (I'm going on the assumption we are not dropping our issue >> tracker)? > > I am proposing that we switch at least some documentation-only repos > to a full PR based workflow, including support for online editing (to > make it easy to fix simple typos and the like without even leaving the > browser). CPython itself would remain completely unaffected. > > The proposal in PEP 474 is that we do that by setting up Kallithea as > forge.python.org. This thread was about considering BitBucket as an > alternative approach. RhodeCode would a third option that still didn't > involve switching away from Mercurial. > >> Second, to properly gauge the impact of switching from git to hg from an >> infrastructure perspective, what automation scripts do we have and how >> difficult would it be to update them to use git instead of hg? This is >> necessary simply to know where we would need to update URLs, let alone >> change in DVCS. > > The problems with changing version control systems don't really become > significant until we start talking about switching CPython itself, > rather than the support repos. > >> Third, do our release managers care about hg vs. git strongly? They probably >> use the DVCS the most directly and at a lower level by necessity compared to >> anyone else. >> >> Fourth, do any core developers feel strongly about not using GitHub? Now >> please notice I said "GitHub" and not "git"; I think the proper way to frame >> this whole discussion is we are deciding if we want to switch to Bitbucket >> or GitHub who provide a low-level API for their version control storage >> service through hg or git, respectively. I personally dislike git, but I >> really like GitHub and I don't even notice git there since I use GitHub's OS >> X app; as I said, I view this as choosing a platform and not the underlying >> DVCS as I have happily chosen to access the GitHub hosting service through >> an app that is not git (it's like accessing a web app through it's web page >> or its REST API). > > Yes, I object strongly to the use of GitHub when there are > commercially supported services written in Python like BitBucket and > RhodeCode available if we want to go the external hosting route, and > other options like the RhodeCode derived Kallithea if we want to run a > self-hosted forge. RhodeCode are even PSF sponsors - I'm sure they'd > be willing to discuss the possibility of hosting core development > repos on their service. > > If I was doing a full risk management breakdown, then RhodeCode would > be the obvious winner, as not only are they PSF sponsors, but > reverting to self-hosting on Kallithea would remain available as an > exit strategy. > > I only suggested BitBucket in this thread because the PSF already has > some repos set up there, so that seemed easier than establishing a new > set of repos on a RhodeCode hosted instance. The PSF account on Bitbucket has 3 repositories. One hasn?t been touched in two years, the other in a year, and the third isn?t in use either. Compare that to Github which has: - The Chef Cookbooks - The Salt states - The Python.org website - The Log Archiver for PyPI/Python.org - The documentation building scripts Which are all actively being used/developed. Beyond that, I don?t have hard numbers (though I could probably attempt to get them) but my gut instinct is that the Python community is primarily there as well. > >> At least for me, until we get a clear understanding of what workflow changes >> we are asking for both contributors and core developers and exactly what >> work would be necessary to update our infrastructure for either Bitbucket or >> GitHub we can't really have a reasonable discussion that isn't going to be >> full of guessing. > > All repos that migrated away from hg.python.org would move to a PR > based workflow, rather than manual patch management on the issue > tracker. The migrated repos would likely also use their integrated > issue tracker rather than the main CPython one at bugs.python.org. > > Externally hosted repos would likely retain a regularly updated mirror > on hg.python.org to ensure the source remains available even in the > event of problems affecting the external hosting provider. > >> And I'm still in support no matter what of breaking out the HOWTOs and the >> tutorial into their own repos for easier updating (having to update the >> Python porting HOWTO in three branches is a pain when it should be >> consistent across Python releases). > > Agreed. > > Cheers, > Nick. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA From wes.turner at gmail.com Mon Nov 24 11:21:03 2014 From: wes.turner at gmail.com (Wes Turner) Date: Mon, 24 Nov 2014 04:21:03 -0600 Subject: [Python-Dev] Move selected documentation repos to PSF BitBucket account? In-Reply-To: References: <19336614-0E4F-42BF-A918-1807BB7F3599@stufft.io> <504EB412-57A2-43B6-ABA1-B29D63395CD6@stufft.io> <20141123235727.GV2748@ando.pearwood.info> Message-ID: On Nov 24, 2014 3:12 AM, "Nick Coghlan" wrote: > > > On 24 Nov 2014 10:41, "Donald Stufft" wrote: > > > > > > > On Nov 23, 2014, at 6:57 PM, Steven D'Aprano wrote: > > > > > > On Sun, Nov 23, 2014 at 08:55:50AM -0800, Guido van Rossum wrote: > > > > > >> But I strongly believe that if we want to do the right thing for the > > >> long term, we should switch to GitHub. > > > > > > Encouraging a software, or social, monopoly is never the right thing for > > > the long term. > > > > > > http://nedbatchelder.com/blog/201405/github_monoculture.html > > > > I don?t think this is really all that big of a deal. If we want to move > > off of Github doing so is easy. > > It's only easy to leave if you never adopt any GitHub-only services like Travis CI. It's the same lockin play as the one Google uses: make it easy for people to move their data, so they're less likely to notice you're locking them in with proprietary APIs and ecosystems instead. - [ ] list of features ranked by stakeholder value - [ ] pingbacks - [ ] in-band build status - [ ] tagging commits with #num - [ ] web standards for integration (HTTP webhooks are the standards here) ... 'Zapier' For this acquisition; is there a feature comparison matrix with alternatives across the top and namespaced features along the side? > > You unlikely to see significant amounts of VC funding pouring into a company unless the investors see a possible chance to extract monopoly rents at some point in the future. > > Cheers, > Nick. > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/wes.turner%40gmail.com > -------------- next part -------------- An HTML attachment was scrubbed... URL: From benjamin at python.org Mon Nov 24 17:07:21 2014 From: benjamin at python.org (Benjamin Peterson) Date: Mon, 24 Nov 2014 11:07:21 -0500 Subject: [Python-Dev] ubuntu buildbot In-Reply-To: References: <1416799084.1792952.194543661.2FC7FCAA@webmail.messagingengine.com> Message-ID: <1416845241.1998368.194775069.7DEEE5BE@webmail.messagingengine.com> On Mon, Nov 24, 2014, at 00:33, David Bolen wrote: > Yeah, it definitely needs it. Historically it was intentional as my own > servers were all on 8.04, but the last of those moved 12.04 last year. > > I think there's already a 12.04 buildbot, so perhaps 14.04 would be > better? I do prefer sticking with an LTS. 14.04 would be good. > > It'll need to move to 64-bit given the hosting environment, at least for > the kernel. I could do 32-bit userspace via multiarch if keeping a > 32-bit > buildbot in the mix is attractive. It'd be nice to keep a 32-bit bot around. From brett at python.org Mon Nov 24 17:28:53 2014 From: brett at python.org (Brett Cannon) Date: Mon, 24 Nov 2014 16:28:53 +0000 Subject: [Python-Dev] Move selected documentation repos to PSF BitBucket account? References: <20141121102647.46e97f04@limelight.wooz.org> <19336614-0E4F-42BF-A918-1807BB7F3599@stufft.io> <504EB412-57A2-43B6-ABA1-B29D63395CD6@stufft.io> <3CC02307-6109-4F20-A815-BDA7CFE16AC2@stufft.io> <013F5AD9-9C9F-4E2E-A750-1C758B66B675@stufft.io> <5A9CA007-3832-4931-8F05-06BD62BE4A5E@stufft.io> <35957465-A878-4864-9473-9F68B9556387@stufft.io> Message-ID: On Mon Nov 24 2014 at 2:25:30 AM Nick Coghlan wrote: > On 24 November 2014 at 02:55, Brett Cannon wrote: > > On Sun Nov 23 2014 at 6:18:46 AM Nick Coghlan > wrote: > >> Those features are readily accessible without changing the underlying > >> version control system (whether self-hosted through Kallithea or > externally > >> hosted through BitBucket or RhodeCode). Thus the folks that want to > change > >> the version control system need to make the case that doing so will > provide > >> additional benefits that *can't* be obtained in a less disruptive way. > > > > I guess my question is who and what is going to be disrupted if we go > with > > Guido's suggestion of switching to GitHub for code hosting? Contributors > > won't be disrupted at all since most people are more familiar with GitHub > > vs. Bitbucket (how many times have we all heard the fact someone has even > > learned Mercurial just to contribute to Python?). Core developers might > be > > based on some learned workflow, but I'm willing to bet we all know git at > > this point (and for those of us who still don't like it, myself included, > > there are GUI apps to paper over it or hg-git for those that prefer a > CLI). > > Our infrastructure will need to be updated, but how much of it is that > > hg-specific short of the command to checkout out the repo? Obviously > > Bitbucket is much more minor by simply updating just a URL, but changing > `hg > > clone` to `git clone` isn't crazy either. Georg, Antoine, or Benjamin can > > point out if I'm wrong on this, maybe Donald or someone in the > > infrastructure committee. > > Are you volunteering to write a competing PEP for a migration to git and > GitHub? > Been there, done that, got the PEP number. I'm just trying to speak from the perspective of the person who drove us off of svn and on to hg (as well as drove us off of SourceForge to our own workflow stack). I personally just want a better workflow. As I said at the beginning, I read your PEPs and talked to you about them at PyCon and I want something like that to happen; push button patch acceptance along with CI of patches would go a long way to making accepting patches easier. But as others have pointed out, we just don't have the volunteer time to make them happen ATM, so I'm willing to entertain moving to GitHub or Bitbucket or whatever to improve our situation. > > I won't be updating PEP 474 to recommend moving to either, as I don't > think that would be a good outcome for the Python ecosystem as a > whole. It massively undercuts any possible confidence anyone else > might have in Mercurial, BitBucket, Rhodecode, Kallithea & Allura (all > Python based version control, or version control hosting, systems). If > we as the Python core development team don't think any of those are > good enough to meet the modest version control needs of our support > repos, why on earth would anyone else choose them? > > In reality, I think most of these services are pretty interchangeable > - GitHub's just been the most effective at the venture capital powered > mindshare grab business model (note how many of the arguments here > stem from the fact folks like *other* things that only interoperate > with GitHub, and no other repository hosting providers - that's the > core of the A18z funded approach to breaking the "D" in DVCS and > ensuring that GitHub's investors are in a position to clip the ticket > when GitHub eventually turns around and takes advantage of its > dominant market position to increase profit margins). > > That's why I consider it legitimate to treat supporting fellow Python > community members as the determining factor - a number of the > available options meet the "good enough" bar from a technical > perspective, so it's reasonable to take other commercial and community > factors into account when making a final decision. > > > Probably the biggest thing I can think of that would need updating is our > > commit hooks. Once again Georg, Antoine, or Benjamin could say how > difficult > > it would be to update those hooks. > > If CPython eventually followed suit in migrating to git (as seems > inevitable if all the other repos were to switch), then every buildbot > will also need to be updated to have git installed (and Mercurial > removed). > > >> From my perspective, swapping out Mercurial for git achieves exactly > >> nothing in terms of alleviating the review bottleneck (since the core > >> developers that strongly prefer the git UI will already be using an > >> adapter), and is in fact likely to make it worse by putting the greatest > >> burden in adapting to the change on the folks that are already under the > >> greatest time pressure. > > > > That's not entirely true. If you are pushing a PR shift in our patch > > acceptance workflow then Bitbucket vs. GitHub isn't fundamentally any > > different in terms of benefit, and I would honestly argue that GitHub's > PR > > experience is better. IOW either platform is of equal benefit. > > Yes, I agree any real benefit comes from the PR workflow, not from > git. This is why I consider "written in Python" to be a valid > determining factor - multiple services meet the "good enough" bar from > a practical perspective, allowing other considerations to come to the > fore. > > (Also note that this proposal does NOT currently cover CPython itself. > Neither GitHub nor BitBucket is set up to handle maintenance branches > well, and any server side merge based workflow improvements for > CPython are gated on fixing the NEWS file maintenance issue. However, > once you contemplate moving CPython, then the ripple effects on other > systems become much larger) > > >> It's also worth keeping in mind that changing the underlying VCS means > >> changing *all* the automation scripts, rather than just updating the > >> configuration settings to reflect a new hosting URL. > > > > What are the automation scripts there are that would require updating? I > > would like to a list and to have the difficulty of moving them mentioned > to > > know what the impact would be. > > For the documentation repos, just the devguide and PEP update scripts > come to mind. As noted above, the implications get more significant if > the main CPython repo eventually follows suit and the buildbot > infrastructure all needs to be updated. > > >> Orchestrating this kind of infrastructure enhancement for Red Hat *is* > my > >> day job, and you almost always want to go for the lowest impact, lowest > risk > >> approach that will alleviate the bottleneck you're worried about while > >> changing the smallest possible number of elements in the overall > workflow > >> management system. > > > > Sure, but I would never compare our infrastructure needs to Red Hat. =) > You > > also have to be conservative in order to minimize downtown and impact for > > cost reasons. As an open source project we don't have those kinds of > worry; > > we just have to worry about keeping everyone happy. > > Switching to a proprietary hosting service written in a mixture of > Ruby, C & bash wouldn't make me happy. > > If that's the end result of this thread, I'll be sorry I even > suggested the idea of reverting to external hosting at all. The irony of that possible outcome was not lost on me this weekend. > That > outcome would be the antithesis of the PSF's overall mission, This might be a little controversial, but the PSF's mission should not influence a decision of python-dev. If we had to break a tie then yes, I would choose the Python-based solution. But if a superior solution existed and it just happened to not be written in Python I'm not going to sacrifice productivity and the overall health of the project just to promote an inferior tool because they happened to choose Python. The only reason we didn't go with Jira for our issue tracker was because of pressure to not go with a closed-source solution *and* I was promised volunteers from the FSF to help manage our issue tracker (which never materialized, BTW). > whereas > I started this thread at least in part due to a discussion about ways > the PSF board might be able to help resolve some of the current > CPython workflow issues. Offering the use of the PSF's existing > BitBucket org account as hosting location for Mercurial repos was an > idea I first brought up in that PSF board thread, and then moved over > here since it seemed worthwhile to at least make the suggestion and > see what people thought. > > >> That underlying calculation doesn't really change much even when the > units > >> shift from budget dollars to volunteers' time and energy. > > > > So here is what I want to know to focus this discussion: > > > > First, what new workflow are you proposing regardless of repo hosting > > provider? Are you proposing we maintain just mirrors and update the > devguide > > to tell people to fork on the hosting provider, make their changes, > generate > > a patch (which can be as simple as telling people how find the raw diff > on > > the hosting provider), and then upload the patch the issue tracker just > like > > today? Are you going farther and saying we have people send PRs on the > > hosting site, have them point to their PR in the issue tracker, and then > we > > accept PRs (I'm going on the assumption we are not dropping our issue > > tracker)? > > I am proposing that we switch at least some documentation-only repos > to a full PR based workflow, including support for online editing (to > make it easy to fix simple typos and the like without even leaving the > browser). CPython itself would remain completely unaffected. > > The proposal in PEP 474 is that we do that by setting up Kallithea as > forge.python.org. This thread was about considering BitBucket as an > alternative approach. RhodeCode would a third option that still didn't > involve switching away from Mercurial. > > > Second, to properly gauge the impact of switching from git to hg from an > > infrastructure perspective, what automation scripts do we have and how > > difficult would it be to update them to use git instead of hg? This is > > necessary simply to know where we would need to update URLs, let alone > > change in DVCS. > > The problems with changing version control systems don't really become > significant until we start talking about switching CPython itself, > rather than the support repos. > > > Third, do our release managers care about hg vs. git strongly? They > probably > > use the DVCS the most directly and at a lower level by necessity > compared to > > anyone else. > > > > Fourth, do any core developers feel strongly about not using GitHub? Now > > please notice I said "GitHub" and not "git"; I think the proper way to > frame > > this whole discussion is we are deciding if we want to switch to > Bitbucket > > or GitHub who provide a low-level API for their version control storage > > service through hg or git, respectively. I personally dislike git, but I > > really like GitHub and I don't even notice git there since I use > GitHub's OS > > X app; as I said, I view this as choosing a platform and not the > underlying > > DVCS as I have happily chosen to access the GitHub hosting service > through > > an app that is not git (it's like accessing a web app through it's web > page > > or its REST API). > > Yes, I object strongly to the use of GitHub when there are > commercially supported services written in Python like BitBucket and > RhodeCode available if we want to go the external hosting route, and > other options like the RhodeCode derived Kallithea if we want to run a > self-hosted forge. RhodeCode are even PSF sponsors - I'm sure they'd > be willing to discuss the possibility of hosting core development > repos on their service. > That's fine if they want to talk about free support, but being a PSF sponsor should not play into it in the slightest, else it's like buying influence. > > If I was doing a full risk management breakdown, then RhodeCode would > be the obvious winner, as not only are they PSF sponsors, but > reverting to self-hosting on Kallithea would remain available as an > exit strategy. > > I only suggested BitBucket in this thread because the PSF already has > some repos set up there, so that seemed easier than establishing a new > set of repos on a RhodeCode hosted instance. > > > At least for me, until we get a clear understanding of what workflow > changes > > we are asking for both contributors and core developers and exactly what > > work would be necessary to update our infrastructure for either > Bitbucket or > > GitHub we can't really have a reasonable discussion that isn't going to > be > > full of guessing. > > All repos that migrated away from hg.python.org would move to a PR > based workflow, rather than manual patch management on the issue > tracker. The migrated repos would likely also use their integrated > issue tracker rather than the main CPython one at bugs.python.org. > > But now we have a fragmented workflow where I have to work with two issue trackers, two ways of dealing with code, etc. That's why I have been talking about a long-term approach of moving everything. I don't need to double my mailing list subscriptions for issues and commits. -Brett > Externally hosted repos would likely retain a regularly updated mirror > on hg.python.org to ensure the source remains available even in the > event of problems affecting the external hosting provider. > > > And I'm still in support no matter what of breaking out the HOWTOs and > the > > tutorial into their own repos for easier updating (having to update the > > Python porting HOWTO in three branches is a pain when it should be > > consistent across Python releases). > > Agreed. > > Cheers, > Nick. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Mon Nov 24 17:36:01 2014 From: donald at stufft.io (Donald Stufft) Date: Mon, 24 Nov 2014 11:36:01 -0500 Subject: [Python-Dev] Move selected documentation repos to PSF BitBucket account? In-Reply-To: References: <20141121102647.46e97f04@limelight.wooz.org> <19336614-0E4F-42BF-A918-1807BB7F3599@stufft.io> <504EB412-57A2-43B6-ABA1-B29D63395CD6@stufft.io> <3CC02307-6109-4F20-A815-BDA7CFE16AC2@stufft.io> <013F5AD9-9C9F-4E2E-A750-1C758B66B675@stufft.io> <5A9CA007-3832-4931-8F05-06BD62BE4A5E@stufft.io> <35957465-A878-4864-9473-9F68B9556387@stufft.io> Message-ID: > On Nov 24, 2014, at 11:28 AM, Brett Cannon wrote: > > On Mon Nov 24 2014 at 2:25:30 AM Nick Coghlan > wrote: > > That > outcome would be the antithesis of the PSF's overall mission, > > This might be a little controversial, but the PSF's mission should not influence a decision of python-dev. If we had to break a tie then yes, I would choose the Python-based solution. But if a superior solution existed and it just happened to not be written in Python I'm not going to sacrifice productivity and the overall health of the project just to promote an inferior tool because they happened to choose Python. > > The only reason we didn't go with Jira for our issue tracker was because of pressure to not go with a closed-source solution and I was promised volunteers from the FSF to help manage our issue tracker (which never materialized, BTW). This is really what I?m trying to do but I?m apparently not getting my point across very well. I want us to pick the best tool for the job regardless of what language it?s written in. I just so happen to think that the best tool for the job in this case is Github. > Yes, I object strongly to the use of GitHub when there are > commercially supported services written in Python like BitBucket and > RhodeCode available if we want to go the external hosting route, and > other options like the RhodeCode derived Kallithea if we want to run a > self-hosted forge. RhodeCode are even PSF sponsors - I'm sure they'd > be willing to discuss the possibility of hosting core development > repos on their service. > > That's fine if they want to talk about free support, but being a PSF sponsor should not play into it in the slightest, else it's like buying influence. Agreed here too (even then, Github has been a PyCon sponsor for awhile and has even ran their own Python conference in the past). I think it?s kind of shitty to reject and demean someone who has supported the Python community through various means just because their primary language isn?t Python. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Mon Nov 24 17:44:43 2014 From: donald at stufft.io (Donald Stufft) Date: Mon, 24 Nov 2014 11:44:43 -0500 Subject: [Python-Dev] Move selected documentation repos to PSF BitBucket account? In-Reply-To: References: <20141121102647.46e97f04@limelight.wooz.org> <19336614-0E4F-42BF-A918-1807BB7F3599@stufft.io> <504EB412-57A2-43B6-ABA1-B29D63395CD6@stufft.io> <3CC02307-6109-4F20-A815-BDA7CFE16AC2@stufft.io> <013F5AD9-9C9F-4E2E-A750-1C758B66B675@stufft.io> <5A9CA007-3832-4931-8F05-06BD62BE4A5E@stufft.io> <35957465-A878-4864-9473-9F68B9556387@stufft.io> Message-ID: <8CD45698-63BC-4F3D-97E2-A3B87F241C89@stufft.io> > On Nov 24, 2014, at 2:25 AM, Nick Coghlan wrote: > > Are you volunteering to write a competing PEP for a migration to git and GitHub? If nobody steps up to do this (and another PEP isn?t accepted before then) I can likely write something up over the upcoming holiday. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA From wes.turner at gmail.com Mon Nov 24 17:57:18 2014 From: wes.turner at gmail.com (Wes Turner) Date: Mon, 24 Nov 2014 10:57:18 -0600 Subject: [Python-Dev] Move selected documentation repos to PSF BitBucket account? In-Reply-To: <8CD45698-63BC-4F3D-97E2-A3B87F241C89@stufft.io> References: <20141121102647.46e97f04@limelight.wooz.org> <19336614-0E4F-42BF-A918-1807BB7F3599@stufft.io> <504EB412-57A2-43B6-ABA1-B29D63395CD6@stufft.io> <3CC02307-6109-4F20-A815-BDA7CFE16AC2@stufft.io> <013F5AD9-9C9F-4E2E-A750-1C758B66B675@stufft.io> <5A9CA007-3832-4931-8F05-06BD62BE4A5E@stufft.io> <35957465-A878-4864-9473-9F68B9556387@stufft.io> <8CD45698-63BC-4F3D-97E2-A3B87F241C89@stufft.io> Message-ID: Is there snapshotting? On Mon, Nov 24, 2014 at 10:44 AM, Donald Stufft wrote: > > > On Nov 24, 2014, at 2:25 AM, Nick Coghlan wrote: > > > > Are you volunteering to write a competing PEP for a migration to git and > GitHub? > > If nobody steps up to do this (and another PEP isn?t accepted before then) > I can > likely write something up over the upcoming holiday. > > --- > Donald Stufft > PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/wes.turner%40gmail.com > -- Wes Turner https://westurner.github.io/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From ischwabacher at wisc.edu Mon Nov 24 17:14:18 2014 From: ischwabacher at wisc.edu (Isaac Schwabacher) Date: Mon, 24 Nov 2014 10:14:18 -0600 Subject: [Python-Dev] Please reconsider PEP 479. In-Reply-To: <7310b9436e4c6.5473594a@wiscmail.wisc.edu> References: <54724117.20408@hotpy.org> <7490f7306e1d6.547352e9@wiscmail.wisc.edu> <760080936a33a.54735326@wiscmail.wisc.edu> <75e0cb746c5ab.54735362@wiscmail.wisc.edu> <7600a1346bf18.5473539f@wiscmail.wisc.edu> <764081036b584.547353db@wiscmail.wisc.edu> <73e0f1a16fa46.54735418@wiscmail.wisc.edu> <71d090cb6d31d.54735454@wiscmail.wisc.edu> <76f0e67b6b934.54735490@wiscmail.wisc.edu> <73e0e1026a9f8.547354cd@wiscmail.wisc.edu> <75e0ee7a6a596.5473550c@wiscmail.wisc.edu> <75f08dbc6e25c.54735548@wiscmail.wisc.edu> <71d0a1946b1a8.54735585@wiscmail.wisc.edu> <7600f74368d06.547355c2@wiscmail.wisc.edu> <7310ce0f6e17a.547355fe@wiscmail.wisc.edu> <76f0b07d68b60.5473563a@wiscmail.wisc.edu> <75f09a2e6eefb.54735677@wiscmail.wisc.edu> <7660ae916d6ba.547356b4@wiscmail.wisc.edu> <7640f76169350.5473572c@wiscmail.wisc.edu> <75e0d70a6a1ed.54735768@wiscmail.wisc.edu> <7660934f6ea3a.547357a5@wiscmail.wisc.edu> <75f0a5716b9aa.54735895@wiscmail.wisc.edu> <7640a38969898.547358d2@wiscmail.wisc.edu> <7730feaa6f94a.5473590e@wiscmail.wisc.edu> <7310b9436e4c6.5473594a@wiscmail.wisc.edu> Message-ID: <75f0876c69053.547304fa@wiscmail.wisc.edu> On 11/23/14, Guido van Rossum wrote: > It wouldn't be so bad if we had the occasional generator author writing "raise StopIteration" instead of "return" to exit from a generator. (We could just add a recommendation against this to the style guide.) But the problem is that an unguarded next() call also raises StopIteration. Sometimes this is intentional (as in some itertools examples). But sometimes an unguarded next() call occurs deep in the bowels of some code called by the generator, and this situation is often hard to debug, since there is no stack track. I'll admit I've only skimmed the massive volume of correspondence this PEP has generated, but it seems to me that this is the main argument for this change. I can only assume that your support for this PEP is informed by your experience building Tulip, but isn't this the kind of thing that can be accomplished with a warning? Then you can get the same behavior without even needing a __future__ import to protect code bases that expect StopIteration to propagate (which seems like the more elegant and natural thing to do, even if it is more error-prone). ijs From berker.peksag at gmail.com Mon Nov 24 19:09:12 2014 From: berker.peksag at gmail.com (=?UTF-8?Q?Berker_Peksa=C4=9F?=) Date: Mon, 24 Nov 2014 20:09:12 +0200 Subject: [Python-Dev] Move selected documentation repos to PSF BitBucket account? In-Reply-To: References: <20141121102647.46e97f04@limelight.wooz.org> <19336614-0E4F-42BF-A918-1807BB7F3599@stufft.io> <504EB412-57A2-43B6-ABA1-B29D63395CD6@stufft.io> <3CC02307-6109-4F20-A815-BDA7CFE16AC2@stufft.io> <013F5AD9-9C9F-4E2E-A750-1C758B66B675@stufft.io> <5A9CA007-3832-4931-8F05-06BD62BE4A5E@stufft.io> <35957465-A878-4864-9473-9F68B9556387@stufft.io> Message-ID: On Mon, Nov 24, 2014 at 6:36 PM, Donald Stufft wrote: > > On Nov 24, 2014, at 11:28 AM, Brett Cannon wrote: > > On Mon Nov 24 2014 at 2:25:30 AM Nick Coghlan wrote: > >> >> That >> outcome would be the antithesis of the PSF's overall mission, > > > This might be a little controversial, but the PSF's mission should not > influence a decision of python-dev. If we had to break a tie then yes, I > would choose the Python-based solution. But if a superior solution existed > and it just happened to not be written in Python I'm not going to sacrifice > productivity and the overall health of the project just to promote an > inferior tool because they happened to choose Python. > > The only reason we didn't go with Jira for our issue tracker was because of > pressure to not go with a closed-source solution and I was promised > volunteers from the FSF to help manage our issue tracker (which never > materialized, BTW). > > > This is really what I?m trying to do but I?m apparently not getting my point > across very well. I want us to pick the best tool for the job regardless of > what language it?s written in. I just so happen to think that the best tool > for the job in this case is Github. For peps and devguide repos GitHub would be the best tool, but not for Doc/ and CPython itself. I also agree that GitHub's big green merge button works well for small or middle sized projects. We can just press the merge button for pep and devguide repos, but for Doc/ I think using GitHub or Bitbucket would be painful. Currently, I do the following steps for a typo fix or a minor markup change: * Import the patch if it's provided: hg imp --no-c * If it's just a typo in the default branch, just verify it and commit: hg commit and hg push * If not: hg update 3.4, hg commit, hg update default, hg merge 3.4, hg push * If it's a markup change, build the documentation locally. It'll take 2-3 minutes. The last two steps would be look tough to follow, it's actually not. I learned Hg when I started to send patches to CPython and learning a couple of commands is not that hard IMO. For GitHub or Bitbucket: * Review the pull request and merge it via the big green merge button. * Ask submitter to send another pull request for the 3.4 branch. If I remember correctly, it's possible to do that from GitHub's UI, but again opening two pull requests to fix a typo or a markup error isn't good. * Or cherrypick it yourself: - Update your local clone - Cherrypick the commit and push * We can use Travis CI to build the documentation on GitHub. I don't think this is a selling point. I tried to fix a broken URL via the "edit" button in the PyPy repo a couple of weeks ago. At my first try, Bitbucket opened a pull request in my fork. It took me 10-15 minutes to open a pull request correctly. This step would be easier on GitHub. However, I think sending an email to docs at python.org is the easiest solution. --Berker From guido at python.org Mon Nov 24 19:19:05 2014 From: guido at python.org (Guido van Rossum) Date: Mon, 24 Nov 2014 10:19:05 -0800 Subject: [Python-Dev] Please reconsider PEP 479. In-Reply-To: <75f0876c69053.547304fa@wiscmail.wisc.edu> References: <54724117.20408@hotpy.org> <7490f7306e1d6.547352e9@wiscmail.wisc.edu> <760080936a33a.54735326@wiscmail.wisc.edu> <75e0cb746c5ab.54735362@wiscmail.wisc.edu> <7600a1346bf18.5473539f@wiscmail.wisc.edu> <764081036b584.547353db@wiscmail.wisc.edu> <73e0f1a16fa46.54735418@wiscmail.wisc.edu> <71d090cb6d31d.54735454@wiscmail.wisc.edu> <76f0e67b6b934.54735490@wiscmail.wisc.edu> <73e0e1026a9f8.547354cd@wiscmail.wisc.edu> <75e0ee7a6a596.5473550c@wiscmail.wisc.edu> <75f08dbc6e25c.54735548@wiscmail.wisc.edu> <71d0a1946b1a8.54735585@wiscmail.wisc.edu> <7600f74368d06.547355c2@wiscmail.wisc.edu> <7310ce0f6e17a.547355fe@wiscmail.wisc.edu> <76f0b07d68b60.5473563a@wiscmail.wisc.edu> <75f09a2e6eefb.54735677@wiscmail.wisc.edu> <7660ae916d6ba.547356b4@wiscmail.wisc.edu> <7640f76169350.5473572c@wiscmail.wisc.edu> <75e0d70a6a1ed.54735768@wiscmail.wisc.edu> <7660934f6ea3a.547357a5@wiscmail.wisc.edu> <75f0a5716b9aa.54735895@wiscmail.wisc.edu> <7640a38969898.547358d2@wiscmail.wisc.edu> <7730feaa6f94a.5473590e@wiscmail.wisc.edu> <7310b9436e4c6.5473594a@wiscmail.wisc.edu> <75f0876c69053.547304fa@wiscmail.wisc.edu> Message-ID: On Mon, Nov 24, 2014 at 8:14 AM, Isaac Schwabacher wrote: > On 11/23/14, Guido van Rossum wrote: > > > It wouldn't be so bad if we had the occasional generator author writing > "raise StopIteration" instead of "return" to exit from a generator. (We > could just add a recommendation against this to the style guide.) But the > problem is that an unguarded next() call also raises StopIteration. > Sometimes this is intentional (as in some itertools examples). But > sometimes an unguarded next() call occurs deep in the bowels of some code > called by the generator, and this situation is often hard to debug, since > there is no stack track. > > I'll admit I've only skimmed the massive volume of correspondence this PEP > has generated, but it seems to me that this is the main argument for this > change. I can only assume that your support for this PEP is informed by > your experience building Tulip, but isn't this the kind of thing that can > be accomplished with a warning? Then you can get the same behavior without > even needing a __future__ import to protect code bases that expect > StopIteration to propagate (which seems like the more elegant and natural > thing to do, even if it is more error-prone). > Yes, this is my main reason for wanting the change -- but not just for tulip/asyncio. The issue can be just as baffling for anyone using unprotected next() calls in the context of a generator. But I'm not sure where to put the warning. Are you proposing to issue a warning under the same conditions the PEP says? But then the itertools examples would issue warnings -- and I bet the advice would typically be "disable warnings" rather than "fix the code, otherwise it will break hard in Python 3.7". -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Mon Nov 24 20:55:54 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 25 Nov 2014 05:55:54 +1000 Subject: [Python-Dev] Move selected documentation repos to PSF BitBucket account? In-Reply-To: References: <20141121102647.46e97f04@limelight.wooz.org> <19336614-0E4F-42BF-A918-1807BB7F3599@stufft.io> <504EB412-57A2-43B6-ABA1-B29D63395CD6@stufft.io> <3CC02307-6109-4F20-A815-BDA7CFE16AC2@stufft.io> <013F5AD9-9C9F-4E2E-A750-1C758B66B675@stufft.io> <5A9CA007-3832-4931-8F05-06BD62BE4A5E@stufft.io> <35957465-A878-4864-9473-9F68B9556387@stufft.io> Message-ID: On 25 Nov 2014 02:28, "Brett Cannon" wrote: > > > > On Mon Nov 24 2014 at 2:25:30 AM Nick Coghlan wrote: >> >> On 24 November 2014 at 02:55, Brett Cannon wrote: >> > On Sun Nov 23 2014 at 6:18:46 AM Nick Coghlan wrote: >> >> Those features are readily accessible without changing the underlying >> >> version control system (whether self-hosted through Kallithea or externally >> >> hosted through BitBucket or RhodeCode). Thus the folks that want to change >> >> the version control system need to make the case that doing so will provide >> >> additional benefits that *can't* be obtained in a less disruptive way. >> > >> > I guess my question is who and what is going to be disrupted if we go with >> > Guido's suggestion of switching to GitHub for code hosting? Contributors >> > won't be disrupted at all since most people are more familiar with GitHub >> > vs. Bitbucket (how many times have we all heard the fact someone has even >> > learned Mercurial just to contribute to Python?). Core developers might be >> > based on some learned workflow, but I'm willing to bet we all know git at >> > this point (and for those of us who still don't like it, myself included, >> > there are GUI apps to paper over it or hg-git for those that prefer a CLI). >> > Our infrastructure will need to be updated, but how much of it is that >> > hg-specific short of the command to checkout out the repo? Obviously >> > Bitbucket is much more minor by simply updating just a URL, but changing `hg >> > clone` to `git clone` isn't crazy either. Georg, Antoine, or Benjamin can >> > point out if I'm wrong on this, maybe Donald or someone in the >> > infrastructure committee. >> >> Are you volunteering to write a competing PEP for a migration to git and GitHub? > > > Been there, done that, got the PEP number. I'm just trying to speak from the perspective of the person who drove us off of svn and on to hg (as well as drove us off of SourceForge to our own workflow stack). I personally just want a better workflow. As I said at the beginning, I read your PEPs and talked to you about them at PyCon and I want something like that to happen; push button patch acceptance along with CI of patches would go a long way to making accepting patches easier. But as others have pointed out, we just don't have the volunteer time to make them happen ATM, so I'm willing to entertain moving to GitHub or Bitbucket or whatever to improve our situation. It may not have been Guido's intention, but his proposal to undercut the entire Python based version control tooling ecosystem by deeming it entirely unfit for our purposes has caused me to dramatically re-evaluate my own priorities. I consider the status quo to be only mildly annoying, which is why I'd been willing to defer proposing improvements. If folks are seriously contemplating proposing a move to GitHub instead, then that changes the equation significantly. Regards, Nick. -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Mon Nov 24 21:25:20 2014 From: donald at stufft.io (Donald Stufft) Date: Mon, 24 Nov 2014 15:25:20 -0500 Subject: [Python-Dev] Move selected documentation repos to PSF BitBucket account? In-Reply-To: References: <20141121102647.46e97f04@limelight.wooz.org> <19336614-0E4F-42BF-A918-1807BB7F3599@stufft.io> <504EB412-57A2-43B6-ABA1-B29D63395CD6@stufft.io> <3CC02307-6109-4F20-A815-BDA7CFE16AC2@stufft.io> <013F5AD9-9C9F-4E2E-A750-1C758B66B675@stufft.io> <5A9CA007-3832-4931-8F05-06BD62BE4A5E@stufft.io> <35957465-A878-4864-9473-9F68B9556387@stufft.io> Message-ID: > On Nov 24, 2014, at 2:55 PM, Nick Coghlan wrote: > > > On 25 Nov 2014 02:28, "Brett Cannon" > wrote: > > > > > > > > On Mon Nov 24 2014 at 2:25:30 AM Nick Coghlan > wrote: > >> > >> On 24 November 2014 at 02:55, Brett Cannon > wrote: > >> > On Sun Nov 23 2014 at 6:18:46 AM Nick Coghlan > wrote: > >> >> Those features are readily accessible without changing the underlying > >> >> version control system (whether self-hosted through Kallithea or externally > >> >> hosted through BitBucket or RhodeCode). Thus the folks that want to change > >> >> the version control system need to make the case that doing so will provide > >> >> additional benefits that *can't* be obtained in a less disruptive way. > >> > > >> > I guess my question is who and what is going to be disrupted if we go with > >> > Guido's suggestion of switching to GitHub for code hosting? Contributors > >> > won't be disrupted at all since most people are more familiar with GitHub > >> > vs. Bitbucket (how many times have we all heard the fact someone has even > >> > learned Mercurial just to contribute to Python?). Core developers might be > >> > based on some learned workflow, but I'm willing to bet we all know git at > >> > this point (and for those of us who still don't like it, myself included, > >> > there are GUI apps to paper over it or hg-git for those that prefer a CLI). > >> > Our infrastructure will need to be updated, but how much of it is that > >> > hg-specific short of the command to checkout out the repo? Obviously > >> > Bitbucket is much more minor by simply updating just a URL, but changing `hg > >> > clone` to `git clone` isn't crazy either. Georg, Antoine, or Benjamin can > >> > point out if I'm wrong on this, maybe Donald or someone in the > >> > infrastructure committee. > >> > >> Are you volunteering to write a competing PEP for a migration to git and GitHub? > > > > > > Been there, done that, got the PEP number. I'm just trying to speak from the perspective of the person who drove us off of svn and on to hg (as well as drove us off of SourceForge to our own workflow stack). I personally just want a better workflow. As I said at the beginning, I read your PEPs and talked to you about them at PyCon and I want something like that to happen; push button patch acceptance along with CI of patches would go a long way to making accepting patches easier. But as others have pointed out, we just don't have the volunteer time to make them happen ATM, so I'm willing to entertain moving to GitHub or Bitbucket or whatever to improve our situation. > > It may not have been Guido's intention, but his proposal to undercut the entire Python based version control tooling ecosystem by deeming it entirely unfit for our purposes has caused me to dramatically re-evaluate my own priorities. > > I think this is a misrepresentation of what Guido said. I?m pretty sure he just said that Github has ?won? over the other sites (which if you define winning in terms of who has the mindshare, I think is indisputable) and that he prefers git over hg now. > I consider the status quo to be only mildly annoying, which is why I'd been willing to defer proposing improvements. If folks are seriously contemplating proposing a move to GitHub instead, then that changes the equation significantly. > > Regards, > Nick. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Mon Nov 24 21:48:23 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 25 Nov 2014 06:48:23 +1000 Subject: [Python-Dev] Move selected documentation repos to PSF BitBucket account? In-Reply-To: References: <20141121102647.46e97f04@limelight.wooz.org> <19336614-0E4F-42BF-A918-1807BB7F3599@stufft.io> <504EB412-57A2-43B6-ABA1-B29D63395CD6@stufft.io> <3CC02307-6109-4F20-A815-BDA7CFE16AC2@stufft.io> <013F5AD9-9C9F-4E2E-A750-1C758B66B675@stufft.io> <5A9CA007-3832-4931-8F05-06BD62BE4A5E@stufft.io> <35957465-A878-4864-9473-9F68B9556387@stufft.io> Message-ID: On 25 Nov 2014 06:25, "Donald Stufft" wrote: > > >> On Nov 24, 2014, at 2:55 PM, Nick Coghlan wrote: >> >> It may not have been Guido's intention, but his proposal to undercut the entire Python based version control tooling ecosystem by deeming it entirely unfit for our purposes has caused me to dramatically re-evaluate my own priorities. >> >> > > I think this is a misrepresentation of what Guido said. I?m pretty sure he just said that Github has ?won? over the other sites (which if you define winning in terms of who has the mindshare, I think is indisputable) and that he prefers git over hg now. That's how a monopoly play works - you invest heavily in rapid growth, aim to lock in an ecosystem through proprietary APIs, then use that powerful market position to deliver monopoly rents to the initial investors. The pattern repeats because "free-as-in-beer" is an extraordinarily effective marketing tactic. Cheers, Nick. > >> I consider the status quo to be only mildly annoying, which is why I'd been willing to defer proposing improvements. If folks are seriously contemplating proposing a move to GitHub instead, then that changes the equation significantly. >> >> Regards, >> Nick. > > > --- > Donald Stufft > PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA > -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris at simplistix.co.uk Mon Nov 24 21:42:01 2014 From: chris at simplistix.co.uk (Chris Withers) Date: Mon, 24 Nov 2014 20:42:01 +0000 Subject: [Python-Dev] Move selected documentation repos to PSF BitBucket account? In-Reply-To: <20141123215921.79d2001b@limelight.wooz.org> References: <20141121102647.46e97f04@limelight.wooz.org> <19336614-0E4F-42BF-A918-1807BB7F3599@stufft.io> <504EB412-57A2-43B6-ABA1-B29D63395CD6@stufft.io> <20141123215921.79d2001b@limelight.wooz.org> Message-ID: <54739819.9040000@simplistix.co.uk> On 24/11/2014 02:59, Barry Warsaw wrote: > On Nov 23, 2014, at 08:55 AM, Guido van Rossum wrote: > >> - Moving from Hg to Git is a fair amount of one-time work > > For anyone seriously interested in this, even experimentally, I would highly > suggest looking at Eric Raymond's reposurgeon code. You can google it or find > it covered in the vast discussions of his conversion of the Emacs repository > from bazaar to git. I don't know for a fact that it would handle an hg to git > conversion, but it's the most likely candidate for something so complex. > There's a lot of similarity in the history of the Emacs and Python > repositories, having gone through many previous iterations of vcs's - in > Python's case, at least cvs, svn, and hg. For what it's worth, Mike Bayer successfully migrated SQLAlchemy from svn to hg and then to git. He now has an amazing system set up whereby commits can be made by contributors through either GitHub or BitBucket, I thought he still had Trac rigged up on sqlalchemy.org, but looks like the issue tracker has migrated to bitbucket... I believe his reasons for moving are similar to those I've read in Guido's posts. I believe he also has notes on what he did and how things are set up... Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk From donald at stufft.io Mon Nov 24 22:00:09 2014 From: donald at stufft.io (Donald Stufft) Date: Mon, 24 Nov 2014 16:00:09 -0500 Subject: [Python-Dev] Move selected documentation repos to PSF BitBucket account? In-Reply-To: References: <20141121102647.46e97f04@limelight.wooz.org> <19336614-0E4F-42BF-A918-1807BB7F3599@stufft.io> <504EB412-57A2-43B6-ABA1-B29D63395CD6@stufft.io> <3CC02307-6109-4F20-A815-BDA7CFE16AC2@stufft.io> <013F5AD9-9C9F-4E2E-A750-1C758B66B675@stufft.io> <5A9CA007-3832-4931-8F05-06BD62BE4A5E@stufft.io> <35957465-A878-4864-9473-9F68B9556387@stufft.io> Message-ID: > On Nov 24, 2014, at 3:48 PM, Nick Coghlan wrote: > > > On 25 Nov 2014 06:25, "Donald Stufft" > wrote: > > > > > >> On Nov 24, 2014, at 2:55 PM, Nick Coghlan > wrote: > >> > >> It may not have been Guido's intention, but his proposal to undercut the entire Python based version control tooling ecosystem by deeming it entirely unfit for our purposes has caused me to dramatically re-evaluate my own priorities. > >> > >> > > > > I think this is a misrepresentation of what Guido said. I?m pretty sure he just said that Github has ?won? over the other sites (which if you define winning in terms of who has the mindshare, I think is indisputable) and that he prefers git over hg now. > > That's how a monopoly play works - you invest heavily in rapid growth, aim to lock in an ecosystem through proprietary APIs, then use that powerful market position to deliver monopoly rents to the initial investors. > > The pattern repeats because "free-as-in-beer" is an extraordinarily effective marketing tactic. > > I don?t really think this is a fair comparison either. ?lock in? here doesn?t make any sense to me unless you define lock-in as any compelling feature what so ever. Wherever a feature Github added was able to be integrated with git itself it did so. PRs for instance could have been done using a proprietary CLI client that uploaded patches which couldn?t be easily exported. Can you please point out in which ways Github has tried to do ?vendor lock in? and in addition can you also point out in which ways they don?t apply to Bitbucket which you were previously fine with? As Guido pointed out, the social aspect to a DVCS is an important part of a DCVS and Git (and Github) has that. Others (including yourself) have pointed out that git vs hg itself is largely isomorphic as far as actual technical ability go when you go through and spend the time to figure out what extensions you need and configure them. It?s largely the *other* tooling around it and the relative sizes of the communities. If you want to pick a toolchain that isn?t as popular nor as polished or as well put together for idealogical reasons than that?s fine, but at least be honest about the fact that you?re choosing to prioritize ideology over selecting the best toolchain. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: From ischwabacher at wisc.edu Mon Nov 24 22:32:03 2014 From: ischwabacher at wisc.edu (Isaac Schwabacher) Date: Mon, 24 Nov 2014 15:32:03 -0600 Subject: [Python-Dev] Please reconsider PEP 479. In-Reply-To: <75e09fb16a85a.5473a350@wiscmail.wisc.edu> References: <54724117.20408@hotpy.org> <7490f7306e1d6.547352e9@wiscmail.wisc.edu> <760080936a33a.54735326@wiscmail.wisc.edu> <75e0cb746c5ab.54735362@wiscmail.wisc.edu> <7600a1346bf18.5473539f@wiscmail.wisc.edu> <764081036b584.547353db@wiscmail.wisc.edu> <73e0f1a16fa46.54735418@wiscmail.wisc.edu> <71d090cb6d31d.54735454@wiscmail.wisc.edu> <76f0e67b6b934.54735490@wiscmail.wisc.edu> <73e0e1026a9f8.547354cd@wiscmail.wisc.edu> <75e0ee7a6a596.5473550c@wiscmail.wisc.edu> <75f08dbc6e25c.54735548@wiscmail.wisc.edu> <71d0a1946b1a8.54735585@wiscmail.wisc.edu> <7600f74368d06.547355c2@wiscmail.wisc.edu> <7310ce0f6e17a.547355fe@wiscmail.wisc.edu> <76f0b07d68b60.5473563a@wiscmail.wisc.edu> <75f09a2e6eefb.54735677@wiscmail.wisc.edu> <7660ae916d6ba.547356b4@wiscmail.wisc.edu> <7640f76169350.5473572c@wiscmail.wisc.edu> <75e0d70a6a1ed.54735768@wiscmail.wisc.edu> <7660934f6ea3a.547357a5@wiscmail.wisc.edu> <75f0a5716b9aa.54735895@wiscmail.wisc.edu> <7640a38969898.547358d2@wiscmail.wisc.edu> <7730feaa6f94a.5473590e@wiscmail.wisc.edu> <7310b9436e4c6.5473594a@wiscmail.wisc.edu> <75f0876c69053.547304fa@wiscmail.wisc.edu> <7490d4706b947.547387da@wiscmail.wisc.edu> <75f0a0a06848a.54738817@wiscmail.wisc.edu> <75f0d8e36ca85.54738853@wiscmail.wisc.edu> <7730cf346a4a5.54738892@wiscmail.wisc.edu> <7640a99a68558.547388d1@wiscmail.wisc.edu> <73e0bf796fc80.5473890d@wiscmail.wisc.edu> <7310e5266f474.5473894a@wiscmail.wisc.edu> <7600f2c86dc25.547389c2@wiscmail.wisc.edu> <73e0bd716a623.547389ff@wiscmail.wisc.edu> <71d0adfa6c157.54738a3b@wiscmail.wisc.edu> <77309add6ffd7.54738ab4@wiscmail.wisc.edu> <73e09c136d72b.54738af0@wiscmail.wisc.edu> <75f0da616db6d.54738b2c@wiscmail.wisc.edu> <7730c8196c8a3.54738b69@wiscmail.wisc.edu> <7730ebff6e532.54738be1@wiscmail.wisc.edu> <75f0c81b6fe80.54738c1e@wiscmail.wisc.edu> <7640f42f6e379.54738c5a@wiscmail.wisc.edu> <7730c8fa6c40c.54738cd3@wiscmail.wisc.edu> <7640f7166b82e.54738d0f@wiscmail.wisc.edu> <75f0bd8d6a318.54738d88@wiscmail.wisc.edu> <75f0815c6e597.54738dc4@wiscmail.wisc.edu> <7310b66f6dfd7.54738e00@wiscmail.wisc.edu> <7490aae46f4d2.54738e3e@wiscmail.wisc.edu> <76f0dd356e74e.54738e7a@wiscmail.wisc.edu> <75f0bc736a09c.54738eb7@wiscmail.wisc.edu> <73e097566d35f.54738ef3@wiscmail.wisc.edu> <7310ea0f6ae5b.54738f30@wiscmail.wisc.edu> <7730ea0a68cb2.54738f6d@wiscmail.wisc.edu> <7490c8556ddb6.54738fa9@wiscmail.wisc.edu> <7640a8896f004.54738fe6@wiscmail.wisc.edu> <75e088366d2a1.54739022@wiscmail.wisc.edu> <7660822e6c8f5.5473905f@wiscmail.wisc.edu> <75f0a59d68c5e.547390d7@wiscmail.wisc.edu> <7600b5566daaa.54739114@wiscmail.wisc.edu> <76f0879969a6f.54739150@wiscmail.wisc.edu> <75e0cc7868051.5473918c@wiscmail.wisc.edu> <7730dc8769c88.547391c9@wiscmail.wisc.edu> <75f0b34a68f51.54739205@wiscmail.wisc.edu> <7600f5956b5ef.54739242@wiscmail.wisc.edu> <76f0eca46f965.5473936e@wiscmail.wisc.edu> <73e0b271684cf.547398d6@wiscmail.wisc.edu> <7600ff196ab43.54739913@wiscmail.wisc.edu> <75e08fc86a335.5473994f@wiscmail.wisc.edu> <75e0a40269708.5473998c@wiscmail.wisc.edu> <75e086196f5dc.547399c8@wiscmail.wisc.edu> <73e0c9926842d.54739a04@wiscmail.wisc.edu> <7660f7206a580.54739a41@wiscmail.wisc.edu> <75e0e77e6a523.54739c35@wiscmail.wisc.edu> <7310f1556d8d5.54739c71@wiscmail.wisc.edu> <7640d2086ceac.54739cae@wiscmail.wisc.edu> <7660f5bc6aa95.54739cea@wiscmail.wisc.edu> <7660a49a6c318.54739d26@wiscmail.wisc.edu> <7600d8be6ad72.54739d63@wiscmail.wisc.edu> <7600f1b46e1e1.54739d9f@wiscmail.wisc.edu> <71d0ddf46d670.54739ddc@wiscmail.wisc.edu> <7310d1a36bdd7.54739e18@wiscmail.wisc.edu> <73e0a9056ae41.54739fbd@wiscmail.wisc.edu> <7730e4e369696.54739ffa@wiscmail.wisc.edu> <73e088b66a755.5473a036@wiscmail.wisc.edu> <76f094e96e0f6.5473a073@wiscmail.wisc.edu> <7660999a6a80e.5473a0af@wiscmail.wisc.edu> <73e0869f6a389.5473a0ed@wiscmail.wisc.edu> <749087326eff5.5473a129@wiscmail.wisc.edu> <75f0f9c66ad98.5473a166@wiscmail.wisc.edu> <75f09ca76968d.5473a29b@wiscmail.wisc.edu> <74909c786f089.5473a2d7@wiscmail.wisc.edu> <7640f6086f726.5473a314@wiscmail.wisc.edu> <75e09fb16a85a.5473a350@wiscmail.wisc.edu> Message-ID: <75f092c168933.54734f73@wiscmail.wisc.edu> On 11/24/14, Guido van Rossum wrote: > On Mon, Nov 24, 2014 at 8:14 AM, Isaac Schwabacher wrote: > > > On 11/23/14, Guido van Rossum wrote: > > > > > It wouldn't be so bad if we had the occasional generator author writing "raise StopIteration" instead of "return" to exit from a generator. (We could just add a recommendation against this to the style guide.) But the problem is that an unguarded next() call also raises StopIteration. Sometimes this is intentional (as in some itertools examples). But sometimes an unguarded next() call occurs deep in the bowels of some code called by the generator, and this situation is often hard to debug, since there is no stack track. > > > > I'll admit I've only skimmed the massive volume of correspondence this PEP has generated, but it seems to me that this is the main argument for this change. I can only assume that your support for this PEP is informed by your experience building Tulip, but isn't this the kind of thing that can be accomplished with a warning? Then you can get the same behavior without even needing a __future__ import to protect code bases that expect StopIteration to propagate (which seems like the more elegant and natural thing to do, even if it is more error-prone). > > Yes, this is my main reason for wanting the change -- but not just for tulip/asyncio. The issue can be just as baffling for anyone using unprotected next() calls in the context of a generator. But I'm not sure where to put the warning. Are you proposing to issue a warning under the same conditions the PEP says? Yes, I'm proposing issuing the warning at the point where the PEP raises, so that the PEP's behavior can be obtained with a warning filter (and such a filter could be installed by default around the asyncio main loop). > But then the itertools examples would issue warnings -- That's definitely problematic. They should either be fixed, or have the warning silenced with a comment about how the bubbling-up case is expected. > and I bet the advice would typically be "disable warnings" rather than "fix the code, otherwise it will break hard in Python 3.7". I don't think it's the language's responsibility to second guess a user who decides to explicitly silence such a warning. And if this *is* accomplished with a warning, then the user can just continue silencing it in 3.7. In my experience, though, python's documentation, StackOverflow presence, blogs, etc. have been absolutely stellar in terms of explaining why things are the way they are and how one should write pythonic code. I don't doubt the community's ability to educate users on this. I think the biggest stumbling block for this proposal is the fact that the current warning machinery doesn't appear to be up to the task of silencing a known-harmless warning in one generator without silencing meaningful warnings in generators it calls. ijs From guido at python.org Mon Nov 24 22:52:53 2014 From: guido at python.org (Guido van Rossum) Date: Mon, 24 Nov 2014 13:52:53 -0800 Subject: [Python-Dev] Please reconsider PEP 479. In-Reply-To: <75f092c168933.54734f73@wiscmail.wisc.edu> References: <54724117.20408@hotpy.org> <7490f7306e1d6.547352e9@wiscmail.wisc.edu> <760080936a33a.54735326@wiscmail.wisc.edu> <75e0cb746c5ab.54735362@wiscmail.wisc.edu> <7600a1346bf18.5473539f@wiscmail.wisc.edu> <764081036b584.547353db@wiscmail.wisc.edu> <73e0f1a16fa46.54735418@wiscmail.wisc.edu> <71d090cb6d31d.54735454@wiscmail.wisc.edu> <76f0e67b6b934.54735490@wiscmail.wisc.edu> <73e0e1026a9f8.547354cd@wiscmail.wisc.edu> <75e0ee7a6a596.5473550c@wiscmail.wisc.edu> <75f08dbc6e25c.54735548@wiscmail.wisc.edu> <71d0a1946b1a8.54735585@wiscmail.wisc.edu> <7600f74368d06.547355c2@wiscmail.wisc.edu> <7310ce0f6e17a.547355fe@wiscmail.wisc.edu> <76f0b07d68b60.5473563a@wiscmail.wisc.edu> <75f09a2e6eefb.54735677@wiscmail.wisc.edu> <7660ae916d6ba.547356b4@wiscmail.wisc.edu> <7640f76169350.5473572c@wiscmail.wisc.edu> <75e0d70a6a1ed.54735768@wiscmail.wisc.edu> <7660934f6ea3a.547357a5@wiscmail.wisc.edu> <75f0a5716b9aa.54735895@wiscmail.wisc.edu> <7640a38969898.547358d2@wiscmail.wisc.edu> <7730feaa6f94a.5473590e@wiscmail.wisc.edu> <7310b9436e4c6.5473594a@wiscmail.wisc.edu> <75f0876c69053.547304fa@wiscmail.wisc.edu> <7490d4706b947.547387da@wiscmail.wisc.edu> <75f0a0a06848a.54738817@wiscmail.wisc.edu> <75f0d8e36ca85.54738853@wiscmail.wisc.edu> <7730cf346a4a5.54738892@wiscmail.wisc.edu> <7640a99a68558.547388d1@wiscmail.wisc.edu> <73e0bf796fc80.5473890d@wiscmail.wisc.edu> <7310e5266f474.5473894a@wiscmail.wisc.edu> <7600f2c86dc25.547389c2@wiscmail.wisc.edu> <73e0bd716a623.547389ff@wiscmail.wisc.edu> <71d0adfa6c157.54738a3b@wiscmail.wisc.edu> <77309add6ffd7.54738ab4@wiscmail.wisc.edu> <73e09c136d72b.54738af0@wiscmail.wisc.edu> <75f0da616db6d.54738b2c@wiscmail.wisc.edu> <7730c8196c8a3.54738b69@wiscmail.wisc.edu> <7730ebff6e532.54738be1@wiscmail.wisc.edu> <75f0c81b6fe80.54738c1e@wiscmail.wisc.edu> <7640f42f6e379.54738c5a@wiscmail.wisc.edu> <7730c8fa6c40c.54738cd3@wiscmail.wisc.edu> <7640f7166b82e.54738d0f@wiscmail.wisc.edu> <75f0bd8d6a318.54738d88@wiscmail.wisc.edu> <75f0815c6e597.54738dc4@wiscmail.wisc.edu> <7310b66f6dfd7.54738e00@wiscmail.wisc.edu> <7490aae46f4d2.54738e3e@wiscmail.wisc.edu> <76f0dd356e74e.54738e7a@wiscmail.wisc.edu> <75f0bc736a09c.54738eb7@wiscmail.wisc.edu> <73e097566d35f.54738ef3@wiscmail.wisc.edu> <7310ea0f6ae5b.54738f30@wiscmail.wisc.edu> <7730ea0a68cb2.54738f6d@wiscmail.wisc.edu> <7490c8556ddb6.54738fa9@wiscmail.wisc.edu> <7640a8896f004.54738fe6@wiscmail.wisc.edu> <75e088366d2a1.54739022@wiscmail.wisc.edu> <7660822e6c8f5.5473905f@wiscmail.wisc.edu> <75f0a59d68c5e.547390d7@wiscmail.wisc.edu> <7600b5566daaa.54739114@wiscmail.wisc.edu> <76f0879969a6f.54739150@wiscmail.wisc.edu> <75e0cc7868051.5473918c@wiscmail.wisc.edu> <7730dc8769c88.547391c9@wiscmail.wisc.edu> <75f0b34a68f51.54739205@wiscmail.wisc.edu> <7600f5956b5ef.54739242@wiscmail.wisc.edu> <76f0eca46f965.5473936e@wiscmail.wisc.edu> <73e0b271684cf.547398d6@wiscmail.wisc.edu> <7600ff196ab43.54739913@wiscmail.wisc.edu> <75e08fc86a335.5473994f@wiscmail.wisc.edu> <75e0a40269708.5473998c@wiscmail.wisc.edu> <75e086196f5dc.547399c8@wiscmail.wisc.edu> <73e0c9926842d.54739a04@wiscmail.wisc.edu> <7660f7206a580.54739a41@wiscmail.wisc.edu> <75e0e77e6a523.54739c35@wiscmail.wisc.edu> <7310f1556d8d5.54739c71@wiscmail.wisc.edu> <7640d2086ceac.54739cae@wiscmail.wisc.edu> <7660f5bc6aa95.54739cea@wiscmail.wisc.edu> <7660a49a6c318.54739d26@wiscmail.wisc.edu> <7600d8be6ad72.54739d63@wiscmail.wisc.edu> <7600f1b46e1e1.54739d9f@wiscmail.wisc.edu> <71d0ddf46d670.54739ddc@wiscmail.wisc.edu> <7310d1a36bdd7.54739e18@wiscmail.wisc.edu> <73e0a9056ae41.54739fbd@wiscmail.wisc.edu> <7730e4e369696.54739ffa@wiscmail.wisc.edu> <73e088b66a755.5473a036@wiscmail.wisc.edu> <76f094e96e0f6.5473a073@wiscmail.wisc.edu> <7660999a6a80e.5473a0af@wiscmail.wisc.edu> <73e0869f6a389.5473a0ed@wiscmail.wisc.edu> <749087326eff5.5473a129@wiscmail.wisc.edu> <75f0f9c66ad98.5473a166@wiscmail.wisc.edu> <75f09ca76968d.5473a29b@wiscmail.wisc.edu> <74909c786f089.5473a2d7@wiscmail.wisc.edu> <7640f6086f726.5473a314@wiscmail.wisc.edu> <75e09fb16a85a.5473a350@wiscmail.wisc.edu> <75f092c168933.54734f73@wiscmail.wisc.edu> Message-ID: On Mon, Nov 24, 2014 at 1:32 PM, Isaac Schwabacher wrote: > On 11/24/14, Guido van Rossum wrote: > > On Mon, Nov 24, 2014 at 8:14 AM, Isaac Schwabacher < > ischwabacher at wisc.edu(javascript:main.compose()> wrote: > > > > > On 11/23/14, Guido van Rossum wrote: > > > > > > > It wouldn't be so bad if we had the occasional generator author > writing "raise StopIteration" instead of "return" to exit from a generator. > (We could just add a recommendation against this to the style guide.) But > the problem is that an unguarded next() call also raises StopIteration. > Sometimes this is intentional (as in some itertools examples). But > sometimes an unguarded next() call occurs deep in the bowels of some code > called by the generator, and this situation is often hard to debug, since > there is no stack track. > > > > > > I'll admit I've only skimmed the massive volume of correspondence this > PEP has generated, but it seems to me that this is the main argument for > this change. I can only assume that your support for this PEP is informed > by your experience building Tulip, but isn't this the kind of thing that > can be accomplished with a warning? Then you can get the same behavior > without even needing a __future__ import to protect code bases that expect > StopIteration to propagate (which seems like the more elegant and natural > thing to do, even if it is more error-prone). > > > > Yes, this is my main reason for wanting the change -- but not just for > tulip/asyncio. The issue can be just as baffling for anyone using > unprotected next() calls in the context of a generator. But I'm not sure > where to put the warning. Are you proposing to issue a warning under the > same conditions the PEP says? > > Yes, I'm proposing issuing the warning at the point where the PEP raises, > so that the PEP's behavior can be obtained with a warning filter (and such > a filter could be installed by default around the asyncio main loop). > > > But then the itertools examples would issue warnings -- > > That's definitely problematic. They should either be fixed, or have the > warning silenced with a comment about how the bubbling-up case is expected. > So you agree with the problem that the PEP is trying to solve, you want people to fix their code in exactly the same way that the PEP is trying to get them to fix it, you want all new code that exhibits the problem to be flagged by a warning, and yet you do not support adding a __future__ statement and a a transition plan that replaces the warnings with hard failures in Python 3.7 (whose release date is going to be at least about four years in the future)? That sounds like the most loyal opposition I can wish for! :-) > > and I bet the advice would typically be "disable warnings" rather than > "fix the code, otherwise it will break hard in Python 3.7". > > I don't think it's the language's responsibility to second guess a user > who decides to explicitly silence such a warning. And if this *is* > accomplished with a warning, then the user can just continue silencing it > in 3.7. In my experience, though, python's documentation, StackOverflow > presence, blogs, etc. have been absolutely stellar in terms of explaining > why things are the way they are and how one should write pythonic code. I > don't doubt the community's ability to educate users on this. > Python's philosophy for (runtime) warnings is pretty clear -- a warning should never be silenced indefinitely. Warnings mean something's wrong with your code that won't get better by ignoring it, and you should fix it at some point. Until then you can silence the warning. Silencing warnings is an important mechanism for users who have no control over the code that issues the warning, and for devs who have more pressing priorities. But they should not be used to permanently enable coding in an "alternate universe" where the language has different features. > I think the biggest stumbling block for this proposal is the fact that the > current warning machinery doesn't appear to be up to the task of silencing > a known-harmless warning in one generator without silencing meaningful > warnings in generators it calls. > You can get pretty darn specific with the warnings silencing machinery: up to the module and line number. It's intentional that you can't specify a class/method -- the latter would just encourage devs to silence a specific warning because they think they know better. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From alexander.belopolsky at gmail.com Tue Nov 25 01:21:32 2014 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Mon, 24 Nov 2014 19:21:32 -0500 Subject: [Python-Dev] PEP 479: Change StopIteration handling inside generators In-Reply-To: References: Message-ID: On Wed, Nov 19, 2014 at 3:10 PM, Guido van Rossum wrote: > There's a new PEP proposing to change how to treat StopIteration bubbling > up out of a generator frame (not caused by a return from the frame). The > proposal is to replace such a StopIteration with a RuntimeError (chained to > the original StopIteration), so that only *returning* from a generator (or > falling off the end) causes the iteration to terminate. I think the PEP should also specify what will happen if the generator's __next__() method is called again after RuntimeError is handled. The two choices are: 1. Raise StopIteration (current behavior for all exceptions). 2. Raise RuntimeError (may be impossible without gi_frame). I think choice 1 is implied by the PEP. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ischwabacher at wisc.edu Tue Nov 25 00:07:19 2014 From: ischwabacher at wisc.edu (Isaac Schwabacher) Date: Mon, 24 Nov 2014 17:07:19 -0600 Subject: [Python-Dev] Please reconsider PEP 479. In-Reply-To: <72c098486e06f.5473ba10@wiscmail.wisc.edu> References: <54724117.20408@hotpy.org> <7490f7306e1d6.547352e9@wiscmail.wisc.edu> <760080936a33a.54735326@wiscmail.wisc.edu> <75e0cb746c5ab.54735362@wiscmail.wisc.edu> <7600a1346bf18.5473539f@wiscmail.wisc.edu> <764081036b584.547353db@wiscmail.wisc.edu> <73e0f1a16fa46.54735418@wiscmail.wisc.edu> <71d090cb6d31d.54735454@wiscmail.wisc.edu> <76f0e67b6b934.54735490@wiscmail.wisc.edu> <73e0e1026a9f8.547354cd@wiscmail.wisc.edu> <75e0ee7a6a596.5473550c@wiscmail.wisc.edu> <75f08dbc6e25c.54735548@wiscmail.wisc.edu> <71d0a1946b1a8.54735585@wiscmail.wisc.edu> <7600f74368d06.547355c2@wiscmail.wisc.edu> <7310ce0f6e17a.547355fe@wiscmail.wisc.edu> <76f0b07d68b60.5473563a@wiscmail.wisc.edu> <75f09a2e6eefb.54735677@wiscmail.wisc.edu> <7660ae916d6ba.547356b4@wiscmail.wisc.edu> <7640f76169350.5473572c@wiscmail.wisc.edu> <75e0d70a6a1ed.54735768@wiscmail.wisc.edu> <7660934f6ea3a.547357a5@wiscmail.wisc.edu> <75f0a5716b9aa.54735895@wiscmail.wisc.edu> <7640a38969898.547358d2@wiscmail.wisc.edu> <7730feaa6f94a.5473590e@wiscmail.wisc.edu> <7310b9436e4c6.5473594a@wiscmail.wisc.edu> <75f0876c69053.547304fa@wiscmail.wisc.edu> <7490d4706b947.547387da@wiscmail.wisc.edu> <75f0a0a06848a.54738817@wiscmail.wisc.edu> <75f0d8e36ca85.54738853@wiscmail.wisc.edu> <7730cf346a4a5.54738892@wiscmail.wisc.edu> <7640a99a68558.547388d1@wiscmail.wisc.edu> <73e0bf796fc80.5473890d@wiscmail.wisc.edu> <7310e5266f474.5473894a@wiscmail.wisc.edu> <7600f2c86dc25.547389c2@wiscmail.wisc.edu> <73e0bd716a623.547389ff@wiscmail.wisc.edu> <71d0adfa6c157.54738a3b@wiscmail.wisc.edu> <77309add6ffd7.54738ab4@wiscmail.wisc.edu> <73e09c136d72b.54738af0@wiscmail.wisc.edu> <75f0da616db6d.54738b2c@wiscmail.wisc.edu> <7730c8196c8a3.54738b69@wiscmail.wisc.edu> <7730ebff6e532.54738be1@wiscmail.wisc.edu> <75f0c81b6fe80.54738c1e@wiscmail.wisc.edu> <7640f42f6e379.54738c5a@wiscmail.wisc.edu> <7730c8fa6c40c.54738cd3@wiscmail.wisc.edu> <7640f7166b82e.54738d0f@wiscmail.wisc.edu> <75f0bd8d6a318.54738d88@wiscmail.wisc.edu> <75f0815c6e597.54738dc4@wiscmail.wisc.edu> <7310b66f6dfd7.54738e00@wiscmail.wisc.edu> <7490aae46f4d2.54738e3e@wiscmail.wisc.edu> <76f0dd356e74e.54738e7a@wiscmail.wisc.edu> <75f0bc736a09c.54738eb7@wiscmail.wisc.edu> <73e097566d35f.54738ef3@wiscmail.wisc.edu> <7310ea0f6ae5b.54738f30@wiscmail.wisc.edu> <7730ea0a68cb2.54738f6d@wiscmail.wisc.edu> <7490c8556ddb6.54738fa9@wiscmail.wisc.edu> <7640a8896f004.54738fe6@wiscmail.wisc.edu> <75e088366d2a1.54739022@wiscmail.wisc.edu> <7660822e6c8f5.5473905f@wiscmail.wisc.edu> <75f0a59d68c5e.547390d7@wiscmail.wisc.edu> <7600b5566daaa.54739114@wiscmail.wisc.edu> <76f0879969a6f.54739150@wiscmail.wisc.edu> <75e0cc7868051.5473918c@wiscmail.wisc.edu> <7730dc8769c88.547391c9@wiscmail.wisc.edu> <75f0b34a68f51.54739205@wiscmail.wisc.edu> <7600f5956b5ef.54739242@wiscmail.wisc.edu> <76f0eca46f965.5473936e@wiscmail.wisc.edu> <73e0b271684cf.547398d6@wiscmail.wisc.edu> <7600ff196ab43.54739913@wiscmail.wisc.edu> <75e08fc86a335.5473994f@wiscmail.wisc.edu> <75e0a40269708.5473998c@wiscmail.wisc.edu> <75e086196f5dc.547399c8@wiscmail.wisc.edu> <73e0c9926842d.54739a04@wiscmail.wisc.edu> <7660f7206a580.54739a41@wiscmail.wisc.edu> <75e0e77e6a523.54739c35@wiscmail.wisc.edu> <7310f1556d8d5.54739c71@wiscmail.wisc.edu> <7640d2086ceac.54739cae@wiscmail.wisc.edu> <7660f5bc6aa95.54739cea@wiscmail.wisc.edu> <7660a49a6c318.54739d26@wiscmail.wisc.edu> <7600d8be6ad72.54739d63@wiscmail.wisc.edu> <7600f1b46e1e1.54739d9f@wiscmail.wisc.edu> <71d0ddf46d670.54739ddc@wiscmail.wisc.edu> <7310d1a36bdd7.54739e18@wiscmail.wisc.edu> <73e0a9056ae41.54739fbd@wiscmail.wisc.edu> <7730e4e369696.54739ffa@wiscmail.wisc.edu> <73e088b66a755.5473a036@wiscmail.wisc.edu> <76f094e96e0f6.5473a073@wiscmail.wisc.edu> <7660999a6a80e.5473a0af@wiscmail.wisc.edu> <73e0869f6a389.5473a0ed@wiscmail.wisc.edu> <749087326eff5.5473a129@wiscmail.wisc.edu> <75f0f9c66ad98.5473a166@wiscmail.wisc.edu> <75f09ca76968d.5473a29b@wiscmail.wisc.edu> <74909c786f089.5473a2d7@wiscmail.wisc.edu> <7640f6086f726.5473a314@wiscmail.wisc.edu> <75e09fb16a85a.5473a350@wiscmail.wisc.edu> <75f092c168933.54734f73@wiscmail.wisc.edu> <73e0995a6dd49.5473b18f@wiscmail.wisc.edu> <76f0b6646cd44.5473b1cc@wiscmail.wisc.edu> <72c0f37b6c4d2.5473b208@wiscmail.wisc.edu> <73009a2d6f797.5473b283@wiscmail.wisc.edu> <75d0c7e168501.5473b2fc@wiscmail.wisc.edu> <73e088946c71c.5473b338@wiscmail.wisc.edu> <74908e966a876.5473b519@wiscmail.wisc.edu> <72c0d36b6aa7f.5473b555@wiscmail.wisc.edu> <7300a7cd6f6f0.5473b592@wiscmail.wisc.edu> <7490f03f6fea8.5473b60d@wiscmail.wisc.edu> <71d0e9d26baaa.5473b649@wiscmail.wisc.edu> <76f0e0f76d935.5473b686@wiscmail.wisc.edu> <75f09fea6a6b0.5473b6c2@wiscmail.wisc.edu> <749081606f44d.5473b6ff@wiscmail.wisc.edu> <749086fd6b319.5473b73b@wiscmail.wisc.edu> <72b0ef3e6a9c3.5473b7b4@wiscmail.wisc.edu> <7610d28b6898c.5473b7f0@wiscmail.wisc.edu> <7730b25069231.5473b82d@wiscmail.wisc.edu> <75f0c52168dbb.5473b869@wiscmail.wisc.edu> <71d0bac569b05.5473b8a6@wiscmail.wisc.edu> <71d0e1f06e79a.5473b95a@wiscmail.wisc.edu> <73e0a8a56f082.5473b997@wiscmail.wisc.edu> <75f09fb16dec1.5473b9d3@wiscmail.wisc.edu> <72c098486e06f.5473ba10@wiscmail.wisc.edu> Message-ID: <71d0b6416c4d4.547365c7@wiscmail.wisc.edu> On 11/24/14, Guido van Rossum wrote: > On Mon, Nov 24, 2014 at 1:32 PM, Isaac Schwabacher ischwabacher at wisc.edu> wrote: > > > On 11/24/14, Guido van Rossum wrote: > > > On Mon, Nov 24, 2014 at 8:14 AM, Isaac Schwabacher (java_script:main.compose()> wrote: > > > > > > > On 11/23/14, Guido van Rossum wrote: > > > > > > > > > It wouldn't be so bad if we had the occasional generator author writing "raise StopIteration" instead of "return" to exit from a generator. (We could just add a recommendation against this to the style guide.) But the problem is that an unguarded next() call also raises StopIteration. Sometimes this is intentional (as in some itertools examples). But sometimes an unguarded next() call occurs deep in the bowels of some code called by the generator, and this situation is often hard to debug, since there is no stack track. > > > > > > > > I'll admit I've only skimmed the massive volume of correspondence this PEP has generated, but it seems to me that this is the main argument for this change. I can only assume that your support for this PEP is informed by your experience building Tulip, but isn't this the kind of thing that can be accomplished with a warning? Then you can get the same behavior without even needing a __future__ import to protect code bases that expect StopIteration to propagate (which seems like the more elegant and natural thing to do, even if it is more error-prone). > > > > > > Yes, this is my main reason for wanting the change -- but not just for tulip/asyncio. The issue can be just as baffling for anyone using unprotected next() calls in the context of a generator. But I'm not sure where to put the warning. Are you proposing to issue a warning under the same conditions the PEP says? > > > > Yes, I'm proposing issuing the warning at the point where the PEP raises, so that the PEP's behavior can be obtained with a warning filter (and such a filter could be installed by default around the asyncio main loop). > > > > > But then the itertools examples would issue warnings -- > > > > That's definitely problematic. They should either be fixed, or have the warning silenced with a comment about how the bubbling-up case is expected. > > So you agree with the problem that the PEP is trying to solve, you want people to fix their code in exactly the same way that the PEP is trying to get them to fix it, you want all new code that exhibits the problem to be flagged by a warning, and yet you do not support adding a __future__ statement and a a transition plan that replaces the warnings with hard failures in Python 3.7 (whose release date is going to be at least about four years in the future)? > > That sounds like the most loyal opposition I can wish for! :-) I agree with you that escaping StopIteration should be easier to notice, but with the opposition that allowing StopIteration to escape on purpose is a useful technique. But when you put it that way... > > > and I bet the advice would typically be "disable warnings" rather than "fix the code, otherwise it will break hard in Python 3.7". > > > > I don't think it's the language's responsibility to second guess a user who decides to explicitly silence such a warning. And if this *is* accomplished with a warning, then the user can just continue silencing it in 3.7. In my experience, though, python's documentation, StackOverflow presence, blogs, etc. have been absolutely stellar in terms of explaining why things are the way they are and how one should write pythonic code. I don't doubt the community's ability to educate users on this. > > Python's philosophy for (runtime) warnings is pretty clear -- a warning should never be silenced indefinitely. Warnings mean something's wrong with your code that won't get better by ignoring it, and you should fix it at some point. Until then you can silence the warning. Silencing warnings is an important mechanism for users who have no control over the code that issues the warning, and for devs who have more pressing priorities. But they should not be used to permanently enable coding in an "alternate universe" where the language has different features. I take it all back. ;D Nothing I've run across in the python ecosystem has made this point to me. (And google searches for "python warning philosophy" and the like don't find it, either.) I was interpreting the warning as indicating that there *might* be a problem in user code, so if the user knows why the system believes this and knows that the problem is a mirage, then it is reasonable to silence the warning indefinitely. In this interpretation it would make sense to simply add the warning and rely on users to determine whether their escaping StopIterations were intentional or not, without ever changing the default handling of the warning to "raise". But if warnings should never be silenced indefinitely, then this is a silly thing to do. > > I think the biggest stumbling block for this proposal is the fact that the current warning machinery doesn't appear to be up to the task of silencing a known-harmless warning in one generator without silencing meaningful warnings in generators it calls. > > You can get pretty darn specific with the warnings silencing machinery: up to the module and line number. It's intentional that you can't specify a class/method -- the latter would just encourage devs to silence a specific warning because they think they know better. And this fact makes so much more sense in context. ijs From guido at python.org Tue Nov 25 01:47:52 2014 From: guido at python.org (Guido van Rossum) Date: Mon, 24 Nov 2014 16:47:52 -0800 Subject: [Python-Dev] Please reconsider PEP 479. In-Reply-To: <71d0b6416c4d4.547365c7@wiscmail.wisc.edu> References: <54724117.20408@hotpy.org> <7730feaa6f94a.5473590e@wiscmail.wisc.edu> <7310b9436e4c6.5473594a@wiscmail.wisc.edu> <75f0876c69053.547304fa@wiscmail.wisc.edu> <7490d4706b947.547387da@wiscmail.wisc.edu> <75f0a0a06848a.54738817@wiscmail.wisc.edu> <75f0d8e36ca85.54738853@wiscmail.wisc.edu> <7730cf346a4a5.54738892@wiscmail.wisc.edu> <7640a99a68558.547388d1@wiscmail.wisc.edu> <73e0bf796fc80.5473890d@wiscmail.wisc.edu> <7310e5266f474.5473894a@wiscmail.wisc.edu> <7600f2c86dc25.547389c2@wiscmail.wisc.edu> <73e0bd716a623.547389ff@wiscmail.wisc.edu> <71d0adfa6c157.54738a3b@wiscmail.wisc.edu> <77309add6ffd7.54738ab4@wiscmail.wisc.edu> <73e09c136d72b.54738af0@wiscmail.wisc.edu> <75f0da616db6d.54738b2c@wiscmail.wisc.edu> <7730c8196c8a3.54738b69@wiscmail.wisc.edu> <7730ebff6e532.54738be1@wiscmail.wisc.edu> <75f0c81b6fe80.54738c1e@wiscmail.wisc.edu> <7640f42f6e379.54738c5a@wiscmail.wisc.edu> <7730c8fa6c40c.54738cd3@wiscmail.wisc.edu> <7640f7166b82e.54738d0f@wiscmail.wisc.edu> <75f0bd8d6a318.54738d88@wiscmail.wisc.edu> <75f0815c6e597.54738dc4@wiscmail.wisc.edu> <7310b66f6dfd7.54738e00@wiscmail.wisc.edu> <7490aae46f4d2.54738e3e@wiscmail.wisc.edu> <76f0dd356e74e.54738e7a@wiscmail.wisc.edu> <75f0bc736a09c.54738eb7@wiscmail.wisc.edu> <73e097566d35f.54738ef3@wiscmail.wisc.edu> <7310ea0f6ae5b.54738f30@wiscmail.wisc.edu> <7730ea0a68cb2.54738f6d@wiscmail.wisc.edu> <7490c8556ddb6.54738fa9@wiscmail.wisc.edu> <7640a8896f004.54738fe6@wiscmail.wisc.edu> <75e088366d2a1.54739022@wiscmail.wisc.edu> <7660822e6c8f5.5473905f@wiscmail.wisc.edu> <75f0a59d68c5e.547390d7@wiscmail.wisc.edu> <7600b5566daaa.54739114@wiscmail.wisc.edu> <76f0879969a6f.54739150@wiscmail.wisc.edu> <75e0cc7868051.5473918c@wiscmail.wisc.edu> <7730dc8769c88.547391c9@wiscmail.wisc.edu> <75f0b34a68f51.54739205@wiscmail.wisc.edu> <7600f5956b5ef.54739242@wiscmail.wisc.edu> <76f0eca46f965.5473936e@wiscmail.wisc.edu> <73e0b271684cf.547398d6@wiscmail.wisc.edu> <7600ff196ab43.54739913@wiscmail.wisc.edu> <75e08fc86a335.5473994f@wiscmail.wisc.edu> <75e0a40269708.5473998c@wiscmail.wisc.edu> <75e086196f5dc.547399c8@wiscmail.wisc.edu> <73e0c9926842d.54739a04@wiscmail.wisc.edu> <7660f7206a580.54739a41@wiscmail.wisc.edu> <75e0e77e6a523.54739c35@wiscmail.wisc.edu> <7310f1556d8d5.54739c71@wiscmail.wisc.edu> <7640d2086ceac.54739cae@wiscmail.wisc.edu> <7660f5bc6aa95.54739cea@wiscmail.wisc.edu> <7660a49a6c318.54739d26@wiscmail.wisc.edu> <7600d8be6ad72.54739d63@wiscmail.wisc.edu> <7600f1b46e1e1.54739d9f@wiscmail.wisc.edu> <71d0ddf46d670.54739ddc@wiscmail.wisc.edu> <7310d1a36bdd7.54739e18@wiscmail.wisc.edu> <73e0a9056ae41.54739fbd@wiscmail.wisc.edu> <7730e4e369696.54739ffa@wiscmail.wisc.edu> <73e088b66a755.5473a036@wiscmail.wisc.edu> <76f094e96e0f6.5473a073@wiscmail.wisc.edu> <7660999a6a80e.5473a0af@wiscmail.wisc.edu> <73e0869f6a389.5473a0ed@wiscmail.wisc.edu> <749087326eff5.5473a129@wiscmail.wisc.edu> <75f0f9c66ad98.5473a166@wiscmail.wisc.edu> <75f09ca76968d.5473a29b@wiscmail.wisc.edu> <74909c786f089.5473a2d7@wiscmail.wisc.edu> <7640f6086f726.5473a314@wiscmail.wisc.edu> <75e09fb16a85a.5473a350@wiscmail.wisc.edu> <75f092c168933.54734f73@wiscmail.wisc.edu> <73e0995a6dd49.5473b18f@wiscmail.wisc.edu> <76f0b6646cd44.5473b1cc@wiscmail.wisc.edu> <72c0f37b6c4d2.5473b208@wiscmail.wisc.edu> <73009a2d6f797.5473b283@wiscmail.wisc.edu> <75d0c7e168501.5473b2fc@wiscmail.wisc.edu> <73e088946c71c.5473b338@wiscmail.wisc.edu> <74908e966a876.5473b519@wiscmail.wisc.edu> <72c0d36b6aa7f.5473b555@wiscmail.wisc.edu> <7300a7cd6f6f0.5473b592@wiscmail.wisc.edu> <7490f03f6fea8.5473b60d@wiscmail.wisc.edu> <71d0e9d26baaa.5473b649@wiscmail.wisc.edu> <76f0e0f76d935.5473b686@wiscmail.wisc.edu> <75f09fea6a6b0.5473b6c2@wiscmail.wisc.edu> <749081606f44d.5473b6ff@wiscmail.wisc.edu> <749086fd6b319.5473b73b@wiscmail.wisc.edu> <72b0ef3e6a9c3.5473b7b4@wiscmail.wisc.edu> <7610d28b6898c.5473b7f0@wiscmail.wisc.edu> <7730b25069231.5473b82d@wiscmail.wisc.edu> <75f0c52168dbb.5473b869@wiscmail.wisc.edu> <71d0bac569b05.5473b8a6@wiscmail.wisc.edu> <71d0e1f06e79a.5473b95a@wiscmail.wisc.edu> <73e0a8a56f082.5473b997@wiscmail.wisc.edu> <75f09fb16dec1.5473b9d3@wiscmail.wisc.edu> <72c098486e06f.5473ba10@wiscmail.wisc.edu> <71d0b6416c4d4.547365c7@wiscmail.wisc.edu> Message-ID: On Mon, Nov 24, 2014 at 3:07 PM, Isaac Schwabacher wrote: > On 11/24/14, Guido van Rossum wrote: > > On Mon, Nov 24, 2014 at 1:32 PM, Isaac Schwabacher < > ischwabacher at wisc.edu > ischwabacher at wisc.edu> wrote: > > > > > On 11/24/14, Guido van Rossum wrote: > > > > On Mon, Nov 24, 2014 at 8:14 AM, Isaac Schwabacher < > python.org/~guido(javascript:main.compose('new', 't=ischwabacher at wisc.edu>(java_script:main.compose()> > wrote: > > > > > > > > > On 11/23/14, Guido van Rossum wrote: > > > > > > > > > > > It wouldn't be so bad if we had the occasional generator author > writing "raise StopIteration" instead of "return" to exit from a generator. > (We could just add a recommendation against this to the style guide.) But > the problem is that an unguarded next() call also raises StopIteration. > Sometimes this is intentional (as in some itertools examples). But > sometimes an unguarded next() call occurs deep in the bowels of some code > called by the generator, and this situation is often hard to debug, since > there is no stack track. > > > > > > > > > > I'll admit I've only skimmed the massive volume of correspondence > this PEP has generated, but it seems to me that this is the main argument > for this change. I can only assume that your support for this PEP is > informed by your experience building Tulip, but isn't this the kind of > thing that can be accomplished with a warning? Then you can get the same > behavior without even needing a __future__ import to protect code bases > that expect StopIteration to propagate (which seems like the more elegant > and natural thing to do, even if it is more error-prone). > > > > > > > > Yes, this is my main reason for wanting the change -- but not just > for tulip/asyncio. The issue can be just as baffling for anyone using > unprotected next() calls in the context of a generator. But I'm not sure > where to put the warning. Are you proposing to issue a warning under the > same conditions the PEP says? > > > > > > Yes, I'm proposing issuing the warning at the point where the PEP > raises, so that the PEP's behavior can be obtained with a warning filter > (and such a filter could be installed by default around the asyncio main > loop). > > > > > > > But then the itertools examples would issue warnings -- > > > > > > That's definitely problematic. They should either be fixed, or have > the warning silenced with a comment about how the bubbling-up case is > expected. > > > > So you agree with the problem that the PEP is trying to solve, you want > people to fix their code in exactly the same way that the PEP is trying to > get them to fix it, you want all new code that exhibits the problem to be > flagged by a warning, and yet you do not support adding a __future__ > statement and a a transition plan that replaces the warnings with hard > failures in Python 3.7 (whose release date is going to be at least about > four years in the future)? > > > > That sounds like the most loyal opposition I can wish for! :-) > > I agree with you that escaping StopIteration should be easier to notice, > but with the opposition that allowing StopIteration to escape on purpose is > a useful technique. But when you put it that way... > > > > > and I bet the advice would typically be "disable warnings" rather > than "fix the code, otherwise it will break hard in Python 3.7". > > > > > > I don't think it's the language's responsibility to second guess a > user who decides to explicitly silence such a warning. And if this *is* > accomplished with a warning, then the user can just continue silencing it > in 3.7. In my experience, though, python's documentation, StackOverflow > presence, blogs, etc. have been absolutely stellar in terms of explaining > why things are the way they are and how one should write pythonic code. I > don't doubt the community's ability to educate users on this. > > > > Python's philosophy for (runtime) warnings is pretty clear -- a warning > should never be silenced indefinitely. Warnings mean something's wrong with > your code that won't get better by ignoring it, and you should fix it at > some point. Until then you can silence the warning. Silencing warnings is > an important mechanism for users who have no control over the code that > issues the warning, and for devs who have more pressing priorities. But > they should not be used to permanently enable coding in an "alternate > universe" where the language has different features. > > I take it all back. ;D Nothing I've run across in the python ecosystem has > made this point to me. (And google searches for "python warning philosophy" > and the like don't find it, either.) I was interpreting the warning as > indicating that there *might* be a problem in user code, so if the user > knows why the system believes this and knows that the problem is a mirage, > then it is reasonable to silence the warning indefinitely. In this > interpretation it would make sense to simply add the warning and rely on > users to determine whether their escaping StopIterations were intentional > or not, without ever changing the default handling of the warning to > "raise". But if warnings should never be silenced indefinitely, then this > is a silly thing to do. > > > > I think the biggest stumbling block for this proposal is the fact that > the current warning machinery doesn't appear to be up to the task of > silencing a known-harmless warning in one generator without silencing > meaningful warnings in generators it calls. > > > > You can get pretty darn specific with the warnings silencing machinery: > up to the module and line number. It's intentional that you can't specify a > class/method -- the latter would just encourage devs to silence a specific > warning because they think they know better. > > And this fact makes so much more sense in context. > In my defense, the docs for the warnings module on docs.python.org at start like this: "Warning messages are typically issued in situations where it is useful to alert the user of some condition in a program, where that condition (normally) doesn?t warrant raising an exception and terminating the program. For example, one might want to issue a warning when a program uses an obsolete module." I admit that's only a single example, but it does refer to something that will break in the future. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Tue Nov 25 01:49:56 2014 From: guido at python.org (Guido van Rossum) Date: Mon, 24 Nov 2014 16:49:56 -0800 Subject: [Python-Dev] PEP 479: Change StopIteration handling inside generators In-Reply-To: References: Message-ID: On Mon, Nov 24, 2014 at 4:21 PM, Alexander Belopolsky < alexander.belopolsky at gmail.com> wrote: > > On Wed, Nov 19, 2014 at 3:10 PM, Guido van Rossum > wrote: > >> There's a new PEP proposing to change how to treat StopIteration bubbling >> up out of a generator frame (not caused by a return from the frame). The >> proposal is to replace such a StopIteration with a RuntimeError (chained to >> the original StopIteration), so that only *returning* from a generator (or >> falling off the end) causes the iteration to terminate. > > > I think the PEP should also specify what will happen if the generator's > __next__() method is called again after RuntimeError is handled. The two > choices are: > > 1. Raise StopIteration (current behavior for all exceptions). > 2. Raise RuntimeError (may be impossible without gi_frame). > > I think choice 1 is implied by the PEP. > Good catch. It has to be #1 because the generator object doesn't retain exception state. I am behind with updating the PEP but I promise I won't mark it as Accepted without adding this, the transition plan, and a discussion of some of the objections that were raised. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From ethan at stoneleaf.us Tue Nov 25 02:59:50 2014 From: ethan at stoneleaf.us (Ethan Furman) Date: Mon, 24 Nov 2014 17:59:50 -0800 Subject: [Python-Dev] Move selected documentation repos to PSF BitBucket account? In-Reply-To: References: <3CC02307-6109-4F20-A815-BDA7CFE16AC2@stufft.io> <013F5AD9-9C9F-4E2E-A750-1C758B66B675@stufft.io> <5A9CA007-3832-4931-8F05-06BD62BE4A5E@stufft.io> <35957465-A878-4864-9473-9F68B9556387@stufft.io> Message-ID: <5473E296.8090306@stoneleaf.us> On 11/24/2014 08:36 AM, Donald Stufft wrote: > On Nov 24, 2014, at 11:28 AM, Brett Cannon wrote: >> >> This might be a little controversial, but the PSF's mission should not influence a decision of python-dev. Yet what we do can reinforce, or undermine, the PSF. >> The only reason we didn't go with Jira for our issue tracker was because of pressure to not go with a closed-source >> solution [...] > This is really what I?m trying to do but I?m apparently not getting my point across very well. I want us to pick the > best tool for the job regardless of what language it?s written in. As an open source project it behooves us to support open source solutions, even if a "better" closed-source solution exists. It is sounding to me like GitHub is not, itself, an open solution, even though they may support open source. -- ~Ethan~ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: OpenPGP digital signature URL: From donald at stufft.io Tue Nov 25 03:27:24 2014 From: donald at stufft.io (Donald Stufft) Date: Mon, 24 Nov 2014 21:27:24 -0500 Subject: [Python-Dev] Move selected documentation repos to PSF BitBucket account? In-Reply-To: <5473E296.8090306@stoneleaf.us> References: <3CC02307-6109-4F20-A815-BDA7CFE16AC2@stufft.io> <013F5AD9-9C9F-4E2E-A750-1C758B66B675@stufft.io> <5A9CA007-3832-4931-8F05-06BD62BE4A5E@stufft.io> <35957465-A878-4864-9473-9F68B9556387@stufft.io> <5473E296.8090306@stoneleaf.us> Message-ID: > On Nov 24, 2014, at 8:59 PM, Ethan Furman wrote: > > On 11/24/2014 08:36 AM, Donald Stufft wrote: >> On Nov 24, 2014, at 11:28 AM, Brett Cannon wrote: >>> >>> This might be a little controversial, but the PSF's mission should not influence a decision of python-dev. > > Yet what we do can reinforce, or undermine, the PSF. > > >>> The only reason we didn't go with Jira for our issue tracker was because of pressure to not go with a closed-source >>> solution [...] > >> This is really what I?m trying to do but I?m apparently not getting my point across very well. I want us to pick the >> best tool for the job regardless of what language it?s written in. > > As an open source project it behooves us to support open source solutions, even if a "better" closed-source solution exists. > > It is sounding to me like GitHub is not, itself, an open solution, even though they may support open source. I?d agree if the tooling was comparable, but at the end of the day the closed source tool is better and more popular. It isn?t Python?s job to fall on the sword in the name of some greater ideology while other languages get to pick the tooling that best enables them to serve the faith that their users have put in them. ?Practicality beats Purity? after all. You might lament the fact that the closed source tool is the better option, but the right response to that is to make an OSS alternative that is more, or at least as, compelling as the closed source solution and then market that and win. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA From ethan at stoneleaf.us Tue Nov 25 03:37:25 2014 From: ethan at stoneleaf.us (Ethan Furman) Date: Mon, 24 Nov 2014 18:37:25 -0800 Subject: [Python-Dev] Move selected documentation repos to PSF BitBucket account? In-Reply-To: References: <3CC02307-6109-4F20-A815-BDA7CFE16AC2@stufft.io> <013F5AD9-9C9F-4E2E-A750-1C758B66B675@stufft.io> <5A9CA007-3832-4931-8F05-06BD62BE4A5E@stufft.io> <35957465-A878-4864-9473-9F68B9556387@stufft.io> <5473E296.8090306@stoneleaf.us> Message-ID: <5473EB65.2070106@stoneleaf.us> On 11/24/2014 06:27 PM, Donald Stufft wrote: > On Nov 24, 2014, at 8:59 PM, Ethan Furman wrote: >> >> It is sounding to me like GitHub is not, itself, an open solution, even though >> they may support open source. > > I?d agree if the tooling was comparable, but at the end of the day the closed > source tool is better and more popular. It isn?t Python?s job to fall on the > sword in the name of some greater ideology while other languages get to pick > the tooling that best enables them to serve the faith that their users have put > in them. ?Practicality beats Purity? after all. > > You might lament the fact that the closed source tool is the better option, but > the right response to that is to make an OSS alternative that is more, or at > least as, compelling as the closed source solution and then market that and win. Or, make a list of the must-haves from the (for whatever reason) controversial choice, and implement them in our own infrastructure. (Yeah, I guess I'm volunteering to help with that effort. ;) -- ~Ethan~ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: OpenPGP digital signature URL: From donald at stufft.io Tue Nov 25 04:18:17 2014 From: donald at stufft.io (Donald Stufft) Date: Mon, 24 Nov 2014 22:18:17 -0500 Subject: [Python-Dev] Move selected documentation repos to PSF BitBucket account? In-Reply-To: <5473EB65.2070106@stoneleaf.us> References: <3CC02307-6109-4F20-A815-BDA7CFE16AC2@stufft.io> <013F5AD9-9C9F-4E2E-A750-1C758B66B675@stufft.io> <5A9CA007-3832-4931-8F05-06BD62BE4A5E@stufft.io> <35957465-A878-4864-9473-9F68B9556387@stufft.io> <5473E296.8090306@stoneleaf.us> <5473EB65.2070106@stoneleaf.us> Message-ID: <1A433321-49C7-461B-ACA0-D0EA7548AA21@stufft.io> > On Nov 24, 2014, at 9:37 PM, Ethan Furman wrote: > > On 11/24/2014 06:27 PM, Donald Stufft wrote: >> On Nov 24, 2014, at 8:59 PM, Ethan Furman wrote: >>> >>> It is sounding to me like GitHub is not, itself, an open solution, even though >>> they may support open source. >> >> I?d agree if the tooling was comparable, but at the end of the day the closed >> source tool is better and more popular. It isn?t Python?s job to fall on the >> sword in the name of some greater ideology while other languages get to pick >> the tooling that best enables them to serve the faith that their users have put >> in them. ?Practicality beats Purity? after all. >> >> You might lament the fact that the closed source tool is the better option, but >> the right response to that is to make an OSS alternative that is more, or at >> least as, compelling as the closed source solution and then market that and win. > > Or, make a list of the must-haves from the (for whatever reason) controversial choice, and implement them in our own > infrastructure. > > (Yeah, I guess I'm volunteering to help with that effort. ;) Isn?t that essentially what I said? ;) Unless you were planning to make the implementations on our own infrastructure closed source. Sadly it?s not just a feature matrix that you need to fill out some check boxes, the closed source software in question just flat out *works* better than any of it?s open source counter parts. Tacking on some features onto an existing solution does not compare. There?s also the social aspects of it as well which is a big concern too IMO. If you want to attract new contributors, not just keep the ones you already have sometimes that means going to where the new contributors are instead of telling them that they need to come to you. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA From ncoghlan at gmail.com Tue Nov 25 06:54:27 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 25 Nov 2014 15:54:27 +1000 Subject: [Python-Dev] Please reconsider PEP 479. In-Reply-To: References: <54724117.20408@hotpy.org> <7310b9436e4c6.5473594a@wiscmail.wisc.edu> <75f0876c69053.547304fa@wiscmail.wisc.edu> <7490d4706b947.547387da@wiscmail.wisc.edu> <75f0a0a06848a.54738817@wiscmail.wisc.edu> <75f0d8e36ca85.54738853@wiscmail.wisc.edu> <7730cf346a4a5.54738892@wiscmail.wisc.edu> <7640a99a68558.547388d1@wiscmail.wisc.edu> <73e0bf796fc80.5473890d@wiscmail.wisc.edu> <7310e5266f474.5473894a@wiscmail.wisc.edu> <7600f2c86dc25.547389c2@wiscmail.wisc.edu> <73e0bd716a623.547389ff@wiscmail.wisc.edu> <71d0adfa6c157.54738a3b@wiscmail.wisc.edu> <77309add6ffd7.54738ab4@wiscmail.wisc.edu> <73e09c136d72b.54738af0@wiscmail.wisc.edu> <75f0da616db6d.54738b2c@wiscmail.wisc.edu> <7730c8196c8a3.54738b69@wiscmail.wisc.edu> <7730ebff6e532.54738be1@wiscmail.wisc.edu> <75f0c81b6fe80.54738c1e@wiscmail.wisc.edu> <7640f42f6e379.54738c5a@wiscmail.wisc.edu> <7730c8fa6c40c.54738cd3@wiscmail.wisc.edu> <7640f7166b82e.54738d0f@wiscmail.wisc.edu> <75f0bd8d6a318.54738d88@wiscmail.wisc.edu> <75f0815c6e597.54738dc4@wiscmail.wisc.edu> <7310b66f6dfd7.54738e00@wiscmail.wisc.edu> <7490aae46f4d2.54738e3e@wiscmail.wisc.edu> <76f0dd356e74e.54738e7a@wiscmail.wisc.edu> <75f0bc736a09c.54738eb7@wiscmail.wisc.edu> <73e097566d35f.54738ef3@wiscmail.wisc.edu> <7310ea0f6ae5b.54738f30@wiscmail.wisc.edu> <7730ea0a68cb2.54738f6d@wiscmail.wisc.edu> <7490c8556ddb6.54738fa9@wiscmail.wisc.edu> <7640a8896f004.54738fe6@wiscmail.wisc.edu> <75e088366d2a1.54739022@wiscmail.wisc.edu> <7660822e6c8f5.5473905f@wiscmail.wisc.edu> <75f0a59d68c5e.547390d7@wiscmail.wisc.edu> <7600b5566daaa.54739114@wiscmail.wisc.edu> <76f0879969a6f.54739150@wiscmail.wisc.edu> <75e0cc7868051.5473918c@wiscmail.wisc.edu> <7730dc8769c88.547391c9@wiscmail.wisc.edu> <75f0b34a68f51.54739205@wiscmail.wisc.edu> <7600f5956b5ef.54739242@wiscmail.wisc.edu> <76f0eca46f965.5473936e@wiscmail.wisc.edu> <73e0b271684cf.547398d6@wiscmail.wisc.edu> <7600ff196ab43.54739913@wiscmail.wisc.edu> <75e08fc86a335.5473994f@wiscmail.wisc.edu> <75e0a40269708.5473998c@wiscmail.wisc.edu> <75e086196f5dc.547399c8@wiscmail.wisc.edu> <73e0c9926842d.54739a04@wiscmail.wisc.edu> <7660f7206a580.54739a41@wiscmail.wisc.edu> <75e0e77e6a523.54739c35@wiscmail.wisc.edu> <7310f1556d8d5.54739c71@wiscmail.wisc.edu> <7640d2086ceac.54739cae@wiscmail.wisc.edu> <7660f5bc6aa95.54739cea@wiscmail.wisc.edu> <7660a49a6c318.54739d26@wiscmail.wisc.edu> <7600d8be6ad72.54739d63@wiscmail.wisc.edu> <7600f1b46e1e1.54739d9f@wiscmail.wisc.edu> <71d0ddf46d670.54739ddc@wiscmail.wisc.edu> <7310d1a36bdd7.54739e18@wiscmail.wisc.edu> <73e0a9056ae41.54739fbd@wiscmail.wisc.edu> <7730e4e369696.54739ffa@wiscmail.wisc.edu> <73e088b66a755.5473a036@wiscmail.wisc.edu> <76f094e96e0f6.5473a073@wiscmail.wisc.edu> <7660999a6a80e.5473a0af@wiscmail.wisc.edu> <73e0869f6a389.5473a0ed@wiscmail.wisc.edu> <749087326eff5.5473a129@wiscmail.wisc.edu> <75f0f9c66ad98.5473a166@wiscmail.wisc.edu> <75f09ca76968d.5473a29b@wiscmail.wisc.edu> <74909c786f089.5473a2d7@wiscmail.wisc.edu> <7640f6086f726.5473a314@wiscmail.wisc.edu> <75e09fb16a85a.5473a350@wiscmail.wisc.edu> <75f092c168933.54734f73@wiscmail.wisc.edu> <73e0995a6dd49.5473b18f@wiscmail.wisc.edu> <76f0b6646cd44.5473b1cc@wiscmail.wisc.edu> <72c0f37b6c4d2.5473b208@wiscmail.wisc.edu> <73009a2d6f797.5473b283@wiscmail.wisc.edu> <75d0c7e168501.5473b2fc@wiscmail.wisc.edu> <73e088946c71c.5473b338@wiscmail.wisc.edu> <74908e966a876.5473b519@wiscmail.wisc.edu> <72c0d36b6aa7f.5473b555@wiscmail.wisc.edu> <7300a7cd6f6f0.5473b592@wiscmail.wisc.edu> <7490f03f6fea8.5473b60d@wiscmail.wisc.edu> <71d0e9d26baaa.5473b649@wiscmail.wisc.edu> <76f0e0f76d935.5473b686@wiscmail.wisc.edu> <75f09fea6a6b0.5473b6c2@wiscmail.wisc.edu> <749081606f44d.5473b6ff@wiscmail.wisc.edu> <749086fd6b319.5473b73b@wiscmail.wisc.edu> <72b0ef3e6a9c3.5473b7b4@wiscmail.wisc.edu> <7610d28b6898c.5473b7f0@wiscmail.wisc.edu> <7730b25069231.5473b82d@wiscmail.wisc.edu> <75f0c52168dbb.5473b869@wiscmail.wisc.edu> <71d0bac569b05.5473b8a6@wiscmail.wisc.edu> <71d0e1f06e79a.5473b95a@wiscmail.wisc.edu> <73e0a8a56f082.5473b997@wiscmail.wisc.edu> <75f09fb16dec1.5473b9d3@wiscmail.wisc.edu> <72c098486e06f.5473ba10@wiscmail.wisc.edu> <71d0b6416c4d4.547365c7@wiscmail.wisc.edu> Message-ID: On 25 November 2014 at 10:47, Guido van Rossum wrote: > In my defense, the docs for the warnings module on docs.python.org at start > like this: > > "Warning messages are typically issued in situations where it is useful to > alert the user of some condition in a program, where that condition > (normally) doesn?t warrant raising an exception and terminating the program. > For example, one might want to issue a warning when a program uses an > obsolete module." > > I admit that's only a single example, but it does refer to something that > will break in the future. The logging tutorial also recommends limiting the use of warning.warn() to cases where the code triggering the warning can and should be changed to avoid the warning: https://docs.python.org/3/howto/logging.html#when-to-use-logging Vinay explicitly noted it there to contrast with the fact that logging.warn() is for cases that are dubious and should potentially be investigated by the application developer, but there's no specific code change to be made to avoid the warning. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From mail at timgolden.me.uk Tue Nov 25 07:09:43 2014 From: mail at timgolden.me.uk (Tim Golden) Date: Tue, 25 Nov 2014 06:09:43 +0000 Subject: [Python-Dev] PCBuild project updates In-Reply-To: <1416694222287.87888@microsoft.com> References: <1416694222287.87888@microsoft.com> Message-ID: <54741D27.407@timgolden.me.uk> On 22/11/2014 22:10, Steve Dower wrote: > Just attracting some attention to http://bugs.python.org/issue22919 > for those of us who build Python out of the PCBuild folder. More > details/patches there, but the tl;dr is > > * Still works with VS 2010 (and now VS 2013 and VS 2015 too) * Build > time should be reduced * Tools\buildbot\*.bat are also updated, so > buildbots shouldn't notice any change > > If you're concerned/interested, I'll be watching the tracker for > comments. Ideally I'd like at least a couple of people to test build > with whatever VS version they have, but if nobody can do that then > I'll assume that nobody will notice anything break sooner than me :) I'm trying to find the time to look, but every time I go there, there's a newer review and a later update :) I view this as a good sign. My VS-fu is minimal but I'll try to pull and build against VS2010 which is all I have installed. As an overview point, though, I think it's great to have a good spring-cleaning of our Win-based setup, so thanks anyway. TJG From ncoghlan at gmail.com Tue Nov 25 07:17:06 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 25 Nov 2014 16:17:06 +1000 Subject: [Python-Dev] Move selected documentation repos to PSF BitBucket account? In-Reply-To: <1A433321-49C7-461B-ACA0-D0EA7548AA21@stufft.io> References: <3CC02307-6109-4F20-A815-BDA7CFE16AC2@stufft.io> <013F5AD9-9C9F-4E2E-A750-1C758B66B675@stufft.io> <5A9CA007-3832-4931-8F05-06BD62BE4A5E@stufft.io> <35957465-A878-4864-9473-9F68B9556387@stufft.io> <5473E296.8090306@stoneleaf.us> <5473EB65.2070106@stoneleaf.us> <1A433321-49C7-461B-ACA0-D0EA7548AA21@stufft.io> Message-ID: On 25 November 2014 at 13:18, Donald Stufft wrote: > > There?s also the social aspects of it as well which is a big concern too IMO. If you want to attract new contributors, not just keep the ones you already have sometimes that means going to where the new contributors are instead of telling them that they need to come to you. Again, the bottleneck at the moment is *reviewing* contributions, not getting more of them. The two aspects are not unrelated, but my key concern at this point is to make the patch review and acceptance process easier, moreso than to increase the rate of incoming patches. My short term proposal to consider BitBucket as an option for support repo hosting purposes was mostly driven by my delays in getting the end-to-end signing PEPs for PyPI updated in a timely fashion - that would have been much easier if the authors had been able to submit pull requests, and I just reviewed and accepted them. The subsequent discussion has made me realise that dissatisfaction with the current state of the infrastructure amongst core developers is higher than I previously realised, so I've re-evaluated my own priorities, and will be spending more time on both PEP 474 (forge.python.org) and PEP 462 (the far more complex proposal to consider introducing OpenStack style merge gating for CPython). At present, it looks like significant workflow improvements for the main CPython repos will require custom tooling - there's very little out there that will adequately support a long term maintenance branch, a short term maintenance branch, additional security fix only branches, and a separate main line of development. Having our own Kallithea installation would provide additional implementation options on that front, so I'll be keeping that in mind as I work to get the proof-of-concept forge instance online. Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From solipsis at pitrou.net Tue Nov 25 10:03:36 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 25 Nov 2014 10:03:36 +0100 Subject: [Python-Dev] Move selected documentation repos to PSF BitBucket account? References: <013F5AD9-9C9F-4E2E-A750-1C758B66B675@stufft.io> <5A9CA007-3832-4931-8F05-06BD62BE4A5E@stufft.io> <35957465-A878-4864-9473-9F68B9556387@stufft.io> <5473E296.8090306@stoneleaf.us> <5473EB65.2070106@stoneleaf.us> <1A433321-49C7-461B-ACA0-D0EA7548AA21@stufft.io> Message-ID: <20141125100336.796401ad@fsol> On Tue, 25 Nov 2014 16:17:06 +1000 Nick Coghlan wrote: > > The subsequent discussion has made me realise that dissatisfaction > with the current state of the infrastructure amongst core developers > is higher than I previously realised, so I've re-evaluated my own > priorities, and will be spending more time on both PEP 474 > (forge.python.org) and PEP 462 (the far more complex proposal to > consider introducing OpenStack style merge gating for CPython). > > At present, it looks like significant workflow improvements for the > main CPython repos will require custom tooling - there's very little > out there that will adequately support a long term maintenance branch, > a short term maintenance branch, additional security fix only > branches, and a separate main line of development. If something constructive comes out of this surrealistic discussion thread then all the better. Thank you Nick. Regards Antoine. From brett at python.org Tue Nov 25 15:52:02 2014 From: brett at python.org (Brett Cannon) Date: Tue, 25 Nov 2014 14:52:02 +0000 Subject: [Python-Dev] Move selected documentation repos to PSF BitBucket account? References: <3CC02307-6109-4F20-A815-BDA7CFE16AC2@stufft.io> <013F5AD9-9C9F-4E2E-A750-1C758B66B675@stufft.io> <5A9CA007-3832-4931-8F05-06BD62BE4A5E@stufft.io> <35957465-A878-4864-9473-9F68B9556387@stufft.io> <5473E296.8090306@stoneleaf.us> <5473EB65.2070106@stoneleaf.us> <1A433321-49C7-461B-ACA0-D0EA7548AA21@stufft.io> Message-ID: On Tue Nov 25 2014 at 1:17:49 AM Nick Coghlan wrote: > On 25 November 2014 at 13:18, Donald Stufft wrote: > > > > There?s also the social aspects of it as well which is a big concern too > IMO. If you want to attract new contributors, not just keep the ones you > already have sometimes that means going to where the new contributors are > instead of telling them that they need to come to you. > > Again, the bottleneck at the moment is *reviewing* contributions, not > getting more of them. The two aspects are not unrelated, but my key > concern at this point is to make the patch review and acceptance > process easier, moreso than to increase the rate of incoming patches. > > My short term proposal to consider BitBucket as an option for support > repo hosting purposes was mostly driven by my delays in getting the > end-to-end signing PEPs for PyPI updated in a timely fashion - that > would have been much easier if the authors had been able to submit > pull requests, and I just reviewed and accepted them. > And then people thought, "ooh, if we are going to open that can of worms we might as well get the better network effect of GitHub" along with Guido going "git >= hg". > > The subsequent discussion has made me realise that dissatisfaction > with the current state of the infrastructure amongst core developers > is higher than I previously realised, so I've re-evaluated my own > priorities, and will be spending more time on both PEP 474 > (forge.python.org) and PEP 462 (the far more complex proposal to > consider introducing OpenStack style merge gating for CPython). > Yay! > > At present, it looks like significant workflow improvements for the > main CPython repos will require custom tooling - there's very little > out there that will adequately support a long term maintenance branch, > a short term maintenance branch, additional security fix only > branches, and a separate main line of development. > Yes, we are unfortunately special. > > Having our own Kallithea installation would provide additional > implementation options on that front, so I'll be keeping that in mind > as I work to get the proof-of-concept forge instance online. > I think this is a reasonable summary of what came up. Short of Donald and maybe Guido really liking the GitHub idea because of their reach, most of us just want better tooling and we all have various compromises we are willing to make to gain that tooling. I suspect if we make sure we add Bitbucket and GitHub login support to the issue tracker then that would help go a fair distance to helping with the GitHub pull of reach (and if we make it so people can simply paste in their fork's URL into the issue tracker and we simply grab a new patch for review that would go even farther). -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Tue Nov 25 16:20:24 2014 From: steve at pearwood.info (Steven D'Aprano) Date: Wed, 26 Nov 2014 02:20:24 +1100 Subject: [Python-Dev] PEP 479: Change StopIteration handling inside generators In-Reply-To: References: Message-ID: <20141125152024.GF2748@ando.pearwood.info> On Mon, Nov 24, 2014 at 10:22:54AM +1100, Chris Angelico wrote: > My point is that doing the same errant operation on a list or a dict > will give different exceptions. In the same way, calling next() on an > empty iterator will raise StopIteration normally, but might raise > RuntimeError instead. It's still an exception, it still indicates a > place where code needs to be changed I wouldn't interpret it like that. Calling next() on an empty iterator raises StopIteration. That's not a bug indicating a failure, it's the protocol working as expected. Your response to that may be to catch the StopIteration and ignore it, or to allow it to bubble up for something else to deal with it. Either way, next() raising StopIteration is not a bug, it is normal behaviour. (Failure to deal with any such StopIteration may be a bug.) However, if next() raises RuntimeError, that's not part of the protocol for iterators, so it is almost certainly a bug to be fixed. (Probably coming from an explicit "raise StopIteration" inside a generator function.) Your fix for the bug may be to refuse to fix it and just catch the exception and ignore it, but that's kind of nasty and hackish and shouldn't be considered good code. Do you agree this is a reasonable way to look at it? -- Steven From rosuav at gmail.com Tue Nov 25 16:31:43 2014 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 26 Nov 2014 02:31:43 +1100 Subject: [Python-Dev] PEP 479: Change StopIteration handling inside generators In-Reply-To: <20141125152024.GF2748@ando.pearwood.info> References: <20141125152024.GF2748@ando.pearwood.info> Message-ID: On Wed, Nov 26, 2014 at 2:20 AM, Steven D'Aprano wrote: > I wouldn't interpret it like that. > > Calling next() on an empty iterator raises StopIteration. That's not a > bug indicating a failure, it's the protocol working as expected. Your > response to that may be to catch the StopIteration and ignore it, or to > allow it to bubble up for something else to deal with it. Either way, > next() raising StopIteration is not a bug, it is normal behaviour. > > (Failure to deal with any such StopIteration may be a bug.) > > However, if next() raises RuntimeError, that's not part of the protocol > for iterators, so it is almost certainly a bug to be fixed. (Probably > coming from an explicit "raise StopIteration" inside a generator > function.) Your fix for the bug may be to refuse to fix it and just > catch the exception and ignore it, but that's kind of nasty and hackish > and shouldn't be considered good code. > > Do you agree this is a reasonable way to look at it? Yes. Specifically, your parenthesis in the middle is the important bit. If you have a csv.DictReader, KeyError might be an important part of your protocol (maybe you have an optional column in the CSV file), but it should be caught before it crosses the boundary of "part of your protocol". At some point, it needs to be converted into ValueError, perhaps, or replaced with a default value, or some other coping mechanism is used. Failure to deal with StopIteration when calling next() is failure to cope with all of that function's protocol, and that is most likely to be a bug. (There are times, and some of them have been mentioned in these discussion threads, where calling next() can never raise StopIteration, so there need be no try/except - eg it=iter(string.split(" ")) - but that just means that a StopIteration from that call is an error somewhere else. I'm definitely happy for that kind of "shouldn't happen" to turn into a RuntimeError rather than being left as an unexpectedly-short generator.) ChrisA From rosuav at gmail.com Tue Nov 25 18:49:59 2014 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 26 Nov 2014 04:49:59 +1100 Subject: [Python-Dev] PEP 479: Change StopIteration handling inside generators In-Reply-To: <75f0c659864dc.54746bd6@wiscmail.wisc.edu> References: <20141125152024.GF2748@ando.pearwood.info> <762099ad85451.5474c008@wiscmail.wisc.edu> <75f0c659864dc.54746bd6@wiscmail.wisc.edu> Message-ID: On Wed, Nov 26, 2014 at 4:45 AM, Isaac Schwabacher wrote: > Yield can also raise StopIteration, if it's thrown in. The current interaction of generator.throw(StopIteration) with yield from can't be emulated under the PEP's behavior, though it's not clear that that's a problem. > Hrm. I have *absolutely* no idea when you would use that, and how you'd go about reworking it to fit this proposal. Do you have any example code (production or synthetic) which throws StopIteration into a generator? ChrisA From guido at python.org Tue Nov 25 19:04:04 2014 From: guido at python.org (Guido van Rossum) Date: Tue, 25 Nov 2014 10:04:04 -0800 Subject: [Python-Dev] PEP 479: Change StopIteration handling inside generators In-Reply-To: References: <20141125152024.GF2748@ando.pearwood.info> <762099ad85451.5474c008@wiscmail.wisc.edu> <75f0c659864dc.54746bd6@wiscmail.wisc.edu> Message-ID: On Tue, Nov 25, 2014 at 9:49 AM, Chris Angelico wrote: > On Wed, Nov 26, 2014 at 4:45 AM, Isaac Schwabacher > wrote: > > Yield can also raise StopIteration, if it's thrown in. The current > interaction of generator.throw(StopIteration) with yield from can't be > emulated under the PEP's behavior, though it's not clear that that's a > problem. > > > > Hrm. I have *absolutely* no idea when you would use that, and how > you'd go about reworking it to fit this proposal. Do you have any > example code (production or synthetic) which throws StopIteration into > a generator? > Sounds like a good one for the obfuscated Python contest. :-) Unless the generator has a try/except surrounding the yield point into which the exception is thrown, it will bubble right out, and PEP 479 will turn this into a RuntimeError. I'll clarify this in the PEP (even though it logically follows from the proposal) -- I don't think there's anything to worry about. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From ischwabacher at wisc.edu Tue Nov 25 19:12:17 2014 From: ischwabacher at wisc.edu (Isaac Schwabacher) Date: Tue, 25 Nov 2014 12:12:17 -0600 Subject: [Python-Dev] PEP 479: Change StopIteration handling inside generators In-Reply-To: <7720eff78085f.5474c65b@wiscmail.wisc.edu> References: <20141125152024.GF2748@ando.pearwood.info> <762099ad85451.5474c008@wiscmail.wisc.edu> <75f0c659864dc.54746bd6@wiscmail.wisc.edu> <7490f4688528b.5474c61e@wiscmail.wisc.edu> <7720eff78085f.5474c65b@wiscmail.wisc.edu> Message-ID: <778099bd8250d.54747221@wiscmail.wisc.edu> On 11/25/14, Guido van Rossum wrote: > On Tue, Nov 25, 2014 at 9:49 AM, Chris Angelico rosuav at gmail.com> wrote: > > > On Wed, Nov 26, 2014 at 4:45 AM, Isaac Schwabacher > > > wrote: > > > Yield can also raise StopIteration, if its thrown in. The current interaction of generator.throw(StopIteration) with yield from cant be emulated under the PEPs behavior, though its not clear that thats a problem. > > > > Hrm. I have *absolutely* no idea when you would use that, To close the innermost generator in a yield-from chain. No, I don't know why you'd want to do that, either. > > and how > > you'd go about reworking it to fit this proposal. Do you have any > > example code (production or synthetic) which throws StopIteration into > > a generator? No. > Sounds like a good one for the obfuscated Python contest. :-) I'm just playing with my food now. :) > Unless the generator has a try/except surrounding the yield point into which the exception is thrown, it will bubble right out, and PEP 479 will turn this into a RuntimeError. I'll clarify this in the PEP (even though it logically follows from the proposal) -- I don't think there's anything to worry about. > > > > -- > --Guido van Rossum ( References: <20141125152024.GF2748@ando.pearwood.info> <762099ad85451.5474c008@wiscmail.wisc.edu> <75f0c659864dc.54746bd6@wiscmail.wisc.edu> <7490f4688528b.5474c61e@wiscmail.wisc.edu> <7720eff78085f.5474c65b@wiscmail.wisc.edu> <778099bd8250d.54747221@wiscmail.wisc.edu> Message-ID: On Tue, Nov 25, 2014 at 10:12 AM, Isaac Schwabacher wrote: > On 11/25/14, Guido van Rossum wrote: > > On Tue, Nov 25, 2014 at 9:49 AM, Chris Angelico rosuav at gmail.com')" target="1">rosuav at gmail.com> wrote: > > > > > On Wed, Nov 26, 2014 at 4:45 AM, Isaac Schwabacher > > > ischwabacher at wisc.edu>> wrote: > > > > Yield can also raise StopIteration, if its thrown in. The current > interaction of generator.throw(StopIteration) with yield from cant be > emulated under the PEPs behavior, though its not clear that thats a problem. > > > > > > Hrm. I have *absolutely* no idea when you would use that, > > To close the innermost generator in a yield-from chain. No, I don't know > why you'd want to do that, either. For that purpose you should call the generator's close() method. This throws a GeneratorExit into the generator to give the generator a chance of cleanup (typically using try/finally). Various reasonable things happen if the generator misbehaves at this point -- if you want to learn what, read the code or experiment a bit on the command line (that's what I usually do). -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From ischwabacher at wisc.edu Tue Nov 25 18:45:26 2014 From: ischwabacher at wisc.edu (Isaac Schwabacher) Date: Tue, 25 Nov 2014 11:45:26 -0600 Subject: [Python-Dev] PEP 479: Change StopIteration handling inside generators In-Reply-To: <762099ad85451.5474c008@wiscmail.wisc.edu> References: <20141125152024.GF2748@ando.pearwood.info> <762099ad85451.5474c008@wiscmail.wisc.edu> Message-ID: <75f0c659864dc.54746bd6@wiscmail.wisc.edu> On 11/25/14, Chris Angelico wrote: > On Wed, Nov 26, 2014 at 2:20 AM, Steven D'Aprano wrote: > > I wouldn't interpret it like that. > > > > Calling next() on an empty iterator raises StopIteration. That's not a > > bug indicating a failure, it's the protocol working as expected. Your > > response to that may be to catch the StopIteration and ignore it, or to > > allow it to bubble up for something else to deal with it. Either way, > > next() raising StopIteration is not a bug, it is normal behaviour. > > > > (Failure to deal with any such StopIteration may be a bug.) > > > > However, if next() raises RuntimeError, that's not part of the protocol > > for iterators, so it is almost certainly a bug to be fixed. (Probably > > coming from an explicit "raise StopIteration" inside a generator > > function.) Your fix for the bug may be to refuse to fix it and just > > catch the exception and ignore it, but that's kind of nasty and hackish > > and shouldn't be considered good code. > > > > Do you agree this is a reasonable way to look at it? > > Yes. Specifically, your parenthesis in the middle is the important > bit. If you have a csv.DictReader, KeyError might be an important part > of your protocol (maybe you have an optional column in the CSV file), > but it should be caught before it crosses the boundary of "part of > your protocol". At some point, it needs to be converted into > ValueError, perhaps, or replaced with a default value, or some other > coping mechanism is used. Failure to deal with StopIteration when > calling next() is failure to cope with all of that function's > protocol, and that is most likely to be a bug. (There are times, and > some of them have been mentioned in these discussion threads, where > calling next() can never raise StopIteration, so there need be no > try/except - eg it=iter(string.split(" ")) - but that just means that > a StopIteration from that call is an error somewhere else. I'm > definitely happy for that kind of "shouldn't happen" to turn into a > RuntimeError rather than being left as an unexpectedly-short > generator.) Yield can also raise StopIteration, if it's thrown in. The current interaction of generator.throw(StopIteration) with yield from can't be emulated under the PEP's behavior, though it's not clear that that's a problem. From solipsis at pitrou.net Wed Nov 26 01:16:33 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 26 Nov 2014 01:16:33 +0100 Subject: [Python-Dev] cpython (3.4): handle errors without a reason attribute References: <20141126000552.55123.63398@psf.io> Message-ID: <20141126011633.483b0258@fsol> On Wed, 26 Nov 2014 00:06:06 +0000 benjamin.peterson wrote: > https://hg.python.org/cpython/rev/e635c3ba75c8 > changeset: 93591:e635c3ba75c8 > branch: 3.4 > user: Benjamin Peterson > date: Tue Nov 25 15:43:58 2014 -0600 > summary: > handle errors without a reason attribute > > files: > Lib/test/support/__init__.py | 2 +- > 1 files changed, 1 insertions(+), 1 deletions(-) > > > diff --git a/Lib/test/support/__init__.py b/Lib/test/support/__init__.py > --- a/Lib/test/support/__init__.py > +++ b/Lib/test/support/__init__.py > @@ -698,7 +698,7 @@ > try: > f(*args, **kwargs) > except IOError as e: > - if e.reason == "CERTIFICATE_VERIFY_FAILED": > + if "CERTIFICATE_VERIFY_FAILED" in str(e): You should be able to keep the e.reason test if you only catch SSLError. Regards Antoine. From benjamin at python.org Wed Nov 26 01:21:08 2014 From: benjamin at python.org (Benjamin Peterson) Date: Tue, 25 Nov 2014 19:21:08 -0500 Subject: [Python-Dev] cpython (3.4): handle errors without a reason attribute In-Reply-To: <20141126011633.483b0258@fsol> References: <20141126000552.55123.63398@psf.io> <20141126011633.483b0258@fsol> Message-ID: <1416961268.2832686.195439937.1CE1FE32@webmail.messagingengine.com> On Tue, Nov 25, 2014, at 19:16, Antoine Pitrou wrote: > On Wed, 26 Nov 2014 00:06:06 +0000 > benjamin.peterson wrote: > > > https://hg.python.org/cpython/rev/e635c3ba75c8 > > changeset: 93591:e635c3ba75c8 > > branch: 3.4 > > user: Benjamin Peterson > > date: Tue Nov 25 15:43:58 2014 -0600 > > summary: > > handle errors without a reason attribute > > > > files: > > Lib/test/support/__init__.py | 2 +- > > 1 files changed, 1 insertions(+), 1 deletions(-) > > > > > > diff --git a/Lib/test/support/__init__.py b/Lib/test/support/__init__.py > > --- a/Lib/test/support/__init__.py > > +++ b/Lib/test/support/__init__.py > > @@ -698,7 +698,7 @@ > > try: > > f(*args, **kwargs) > > except IOError as e: > > - if e.reason == "CERTIFICATE_VERIFY_FAILED": > > + if "CERTIFICATE_VERIFY_FAILED" in str(e): > > You should be able to keep the e.reason test if you only catch SSLError. Unfortunately, test_robotparser seems to manage to raise a cert validation error without a reason. From solipsis at pitrou.net Wed Nov 26 01:29:50 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 26 Nov 2014 01:29:50 +0100 Subject: [Python-Dev] cpython (3.4): handle errors without a reason attribute In-Reply-To: <1416961268.2832686.195439937.1CE1FE32@webmail.messagingengine.com> References: <20141126000552.55123.63398@psf.io> <20141126011633.483b0258@fsol> <1416961268.2832686.195439937.1CE1FE32@webmail.messagingengine.com> Message-ID: <20141126012950.7b62e7fb@fsol> On Tue, 25 Nov 2014 19:21:08 -0500 Benjamin Peterson wrote: > > > > You should be able to keep the e.reason test if you only catch SSLError. > > Unfortunately, test_robotparser seems to manage to raise a cert > validation error without a reason. Ahh... Perhaps it's urllib catching the SSLError and re-raising it as a URLError with the same message with no reason, then. Regards Antoine. From greg.ewing at canterbury.ac.nz Wed Nov 26 01:58:25 2014 From: greg.ewing at canterbury.ac.nz (Greg) Date: Wed, 26 Nov 2014 13:58:25 +1300 Subject: [Python-Dev] Please reconsider PEP 479. In-Reply-To: References: <54724117.20408@hotpy.org> Message-ID: <547525B1.4090303@canterbury.ac.nz> I'm not particularly opposed to PEP 479, but the Abstract and Rationale could do with considerable clarification. They currently appear to promise things that are in disagreement with what the PEP actually delivers. The Abstract claims that the proposal will "unify the behaviour of list comprehensions and generator expressions", but it doesn't do that. What it actually does is provide special protection against escaped StopIteration exceptions in one particular context (the body of a generator). It doesn't prevent StopIteration from escaping anywhere else, including from list comprehensions, so if anything it actually *increases* the difference between generators and comprehensions. There may be merit in preventing rogue StopIterations escaping from generators, but the PEP should sell the idea on that basis, not on what sounds like a false promise that it will make comprehensions and generators behave identically. -- Greg From guido at python.org Wed Nov 26 02:08:46 2014 From: guido at python.org (Guido van Rossum) Date: Tue, 25 Nov 2014 17:08:46 -0800 Subject: [Python-Dev] Please reconsider PEP 479. In-Reply-To: <547525B1.4090303@canterbury.ac.nz> References: <54724117.20408@hotpy.org> <547525B1.4090303@canterbury.ac.nz> Message-ID: On Tue, Nov 25, 2014 at 4:58 PM, Greg wrote: > I'm not particularly opposed to PEP 479, but the Abstract and > Rationale could do with considerable clarification. I know. > They currently > appear to promise things that are in disagreement with what the PEP > actually delivers. > > The Abstract claims that the proposal will "unify the behaviour of > list comprehensions and generator expressions", but it doesn't do > that. What it actually does is provide special protection against > escaped StopIteration exceptions in one particular context (the > body of a generator). It doesn't prevent StopIteration from > escaping anywhere else, including from list comprehensions, so if > anything it actually *increases* the difference between generators > and comprehensions. > Hm, that sounds like you're either being contrarian or Chris and I have explained it even worse than I thought. Currently, there are cases where list(x for x in xs if P(x)) works while [x for x in xs if P(x)] fails (when P(x) raises StopIteration). With the PEP, both cases will raise some exception -- though you (and several others who've pointed this out) are right that the exception raised is different (RuntimeError vs. StopIteration) and if this occurs inside a __next__() method (not a generator) the StopIteration will cause the outer iteration to terminate silently. > There may be merit in preventing rogue StopIterations escaping > from generators, but the PEP should sell the idea on that basis, not > on what sounds like a false promise that it will make comprehensions > and generators behave identically. > I will weaken that language. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Wed Nov 26 02:29:59 2014 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 26 Nov 2014 12:29:59 +1100 Subject: [Python-Dev] Please reconsider PEP 479. In-Reply-To: <547525B1.4090303@canterbury.ac.nz> References: <54724117.20408@hotpy.org> <547525B1.4090303@canterbury.ac.nz> Message-ID: On Wed, Nov 26, 2014 at 11:58 AM, Greg wrote: > The Abstract claims that the proposal will "unify the behaviour of > list comprehensions and generator expressions", but it doesn't do > that. I don't know that it completely unifies the behaviours, but it does merge them on the specific situation of a leaking StopIteration. With the original code examples that sparked this all off (see the first footnote in the PEP), current CPython has the list-comp form terminate cleanly, while the genexp form infinitely loops. With the POC patch on the issue tracker, both forms cause RuntimeError. Is there a better word than "unify" for that? ChrisA From ncoghlan at gmail.com Wed Nov 26 08:06:29 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 26 Nov 2014 17:06:29 +1000 Subject: [Python-Dev] PEP 479: Change StopIteration handling inside generators In-Reply-To: References: <20141125152024.GF2748@ando.pearwood.info> <762099ad85451.5474c008@wiscmail.wisc.edu> <75f0c659864dc.54746bd6@wiscmail.wisc.edu> Message-ID: On 26 November 2014 at 04:04, Guido van Rossum wrote: > On Tue, Nov 25, 2014 at 9:49 AM, Chris Angelico wrote: >> >> On Wed, Nov 26, 2014 at 4:45 AM, Isaac Schwabacher >> wrote: >> > Yield can also raise StopIteration, if it's thrown in. The current >> > interaction of generator.throw(StopIteration) with yield from can't be >> > emulated under the PEP's behavior, though it's not clear that that's a >> > problem. >> > >> >> Hrm. I have *absolutely* no idea when you would use that, and how >> you'd go about reworking it to fit this proposal. Do you have any >> example code (production or synthetic) which throws StopIteration into >> a generator? > > > Sounds like a good one for the obfuscated Python contest. :-) > > Unless the generator has a try/except surrounding the yield point into which > the exception is thrown, it will bubble right out, and PEP 479 will turn > this into a RuntimeError. I'll clarify this in the PEP (even though it > logically follows from the proposal) -- I don't think there's anything to > worry about. This is actually the edge case that needs to be handled in contextlib - a StopIteration raised by the with statement body gets thrown into the generator implementing the context manager. My current porting recommendation is to catch the RuntimeError & look at __cause__ to see if it's the StopIteration instance that was thrown in, but an alternative option would be to just call gen.close() in that case, rather than gen.throw(exc). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From greg.ewing at canterbury.ac.nz Wed Nov 26 09:30:42 2014 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Wed, 26 Nov 2014 21:30:42 +1300 Subject: [Python-Dev] Please reconsider PEP 479. In-Reply-To: References: <54724117.20408@hotpy.org> <547525B1.4090303@canterbury.ac.nz> Message-ID: <54758FB2.6080100@canterbury.ac.nz> Guido van Rossum wrote: > Hm, that sounds like you're either being contrarian or Chris and I have > explained it even worse than I thought. I'm not trying to be contrary, I just think the PEP could explain more clearly what you're trying to achieve. The rationale is too vague and waffly at the moment. > Currently, there are cases where list(x for x in xs if P(x)) works while > [x for x in xs if P(x)] fails (when P(x) raises StopIteration). With the > PEP, both cases will raise some exception That's a better explanation, I think. -- Greg From ncoghlan at gmail.com Wed Nov 26 12:24:10 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 26 Nov 2014 21:24:10 +1000 Subject: [Python-Dev] Please reconsider PEP 479. In-Reply-To: <54758FB2.6080100@canterbury.ac.nz> References: <54724117.20408@hotpy.org> <547525B1.4090303@canterbury.ac.nz> <54758FB2.6080100@canterbury.ac.nz> Message-ID: On 26 November 2014 at 18:30, Greg Ewing wrote: > Guido van Rossum wrote: >> >> Hm, that sounds like you're either being contrarian or Chris and I have >> explained it even worse than I thought. > > I'm not trying to be contrary, I just think the PEP could > explain more clearly what you're trying to achieve. The > rationale is too vague and waffly at the moment. > >> Currently, there are cases where list(x for x in xs if P(x)) works while >> [x for x in xs if P(x)] fails (when P(x) raises StopIteration). With the >> PEP, both cases will raise some exception > > That's a better explanation, I think. The other key aspect is that it changes the answer to the question "How do I gracefully terminate a generator function?". The existing behaviour has an "or" in the answer: "return from the generator frame, OR raise StopIteration from the generator frame". That then leads to the follow on question: "When should I use one over the other?". The "from __future__ import generator_stop" answer drops the "or", so it's just: "return from the generator frame". Raising *any* exception inside the generator, including StopIteration, then counts as non-graceful termination, bringing generators into line with the PEP 343 philosophy that "hiding flow control in macros makes your code inscrutable", where here, the hidden flow control is relying on the fact that a called function raising StopIteration will currently always gracefully terminate generator execution. The key downside is that it means relatively idiomatic code like: def my_generator(): ... yield next(it) ... Now needs to be written out explicitly as: def my_generator(): ... try: yield next(it) except StopIteration return ... That's not especially easy to read, and it's also going to be very slow when working with generator based producer/consumer pipelines. After thinking about that concern for a while, I'd like to suggest the idea of having a new builtin "allow_implicit_stop" decorator that swaps out a GENERATOR code object that has the new "EXPLICIT_STOP" flag set for one with it cleared (attempting to apply "allow_implicit_stop" to a normal function would be an error). Then the updated version of the above example would become: @allow_implicit_stop def my_generator(): ... yield next(it) ... Which would be semantically equivalent to: def my_generator(): try: ... yield next(it) ... except StopIteration return but *much* faster (especially if used in a producer/consumer pipeline) since it would allow a single StopIteration instance to propagate through the entire pipeline, rather than creating and destroying new ones at each stage. Including such a feature in the PEP would also make the fix to contextlib simpler: we'd just update it so that contextlib._GeneratorContextManager automatically calls "allow_implicit_stop" on the passed in generator functions. Single-source Python 2/3 code would also benefit in a 3.7+ world, since libraries like six and python-future could just define their own version of "allow_implicit_stop" that referred to the new builtin in 3.5+, and was implemented as an identity function in other versions. Cheers, Nick. P.S. While I'm less convinced this part is a good idea, if "allow_implicit_stop" accepted both generator functions *and* generator objects, then folks could even still explicitly opt in to the "or stop()" trick - and anyone reading the code would have a name to look up to see what was going on. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Wed Nov 26 12:52:23 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 26 Nov 2014 21:52:23 +1000 Subject: [Python-Dev] Please reconsider PEP 479. In-Reply-To: References: <54724117.20408@hotpy.org> <547525B1.4090303@canterbury.ac.nz> <54758FB2.6080100@canterbury.ac.nz> Message-ID: On 26 November 2014 at 21:44, Petr Viktorin wrote: > On Wed, Nov 26, 2014 at 12:24 PM, Nick Coghlan wrote: >> Now needs to be written out explicitly as: >> >> def my_generator(): >> ... >> try: >> yield next(it) >> except StopIteration >> return >> ... > > It could also be written as: > > def my_generator(): > try: > ... > yield next(it) > ... > except StopIteration > return > > i.e. put the try-block around the whole body, not just the individual > yield. This emulates what's happenning in current Python, and it would > be faster than individual try blocks. Not appreciably though - the potentially slow part that the status quo avoids is the "catch old exception, return from frame, create and raise new exception" step, and that only happens once per generator termination, regardless of how many times the try/except/return pattern appears in the body of a particular generator. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From rosuav at gmail.com Wed Nov 26 12:59:25 2014 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 26 Nov 2014 22:59:25 +1100 Subject: [Python-Dev] Please reconsider PEP 479. In-Reply-To: References: <54724117.20408@hotpy.org> <547525B1.4090303@canterbury.ac.nz> <54758FB2.6080100@canterbury.ac.nz> Message-ID: On Wed, Nov 26, 2014 at 10:24 PM, Nick Coghlan wrote: > The other key aspect is that it changes the answer to the question > "How do I gracefully terminate a generator function?". The existing > behaviour has an "or" in the answer: "return from the generator frame, > OR raise StopIteration from the generator frame". That then leads to > the follow on question: "When should I use one over the other?". > > The "from __future__ import generator_stop" answer drops the "or", so > it's just: "return from the generator frame". If I understand you correctly, you agree that this is a benefit, correct? > The key downside is that it means relatively idiomatic code like: > > def my_generator(): > ... > yield next(it) > ... > > Now needs to be written out explicitly as: > > def my_generator(): > ... > try: > yield next(it) > except StopIteration > return > ... > > That's not especially easy to read, and it's also going to be very > slow when working with generator based producer/consumer pipelines. I'm not sure how often the ease-of-reading concern will come up, but I can at least benchmark the performance of it. I have two virtually-identical builds of CPython 3.5, one with and one without the POC patch. There's no guarantee that this will properly match the performance of the final product, as there'll likely be some additional checks (especially when there's a __future__ directive to concern ourselves with), but it's a start. yield from: https://github.com/Rosuav/GenStopIter/blob/485d1/perftest.py explicit loop: https://github.com/Rosuav/GenStopIter/blob/c071d/perftest.py The numbers are pretty noisy, but I'm seeing about a 5% slowdown in the 'yield from' version, with a recursion depth of 100 generators. (Obviously less percentage slowdown with less depth, as other factors have more impact.) Rewriting the loop to use an explicit try/except and while loop roughly doubles the time taken (so clearly 'yield from' is implemented very efficiently), and also destroys any meaning in the numbers - the two interpreters come out effectively equal. My preliminary conclusion is that there is some impact, but it's unlikely to be significant in the real world. Do you have a more real-world code example to try? > After thinking about that concern for a while, I'd like to suggest the > idea of having a new builtin "allow_implicit_stop" decorator that > swaps out a GENERATOR code object that has the new "EXPLICIT_STOP" > flag set for one with it cleared (attempting to apply > "allow_implicit_stop" to a normal function would be an error). > > Then the updated version of the above example would become: > > @allow_implicit_stop > def my_generator(): > ... > yield next(it) > ... > > Which would be semantically equivalent to: > > def my_generator(): > try: > ... > yield next(it) > ... > except StopIteration > return > > but *much* faster (especially if used in a producer/consumer pipeline) > since it would allow a single StopIteration instance to propagate > through the entire pipeline, rather than creating and destroying new > ones at each stage. If the issue is performance, I would prefer to see something done with a peephole optimizer instead: if there's a "try... except StopIteration: return" construct, it's optimized away down to a magic flag. That way, the code is guaranteed to be correct in all cases, with no hidden behavioral changes, and this is _just_ a performance optimization. I'd still rather see the exception-catching scope narrowed as much as possible, though, which means not having something that's semantically equivalent to wrapping the whole generator in "try... except StopIteration: pass". > P.S. While I'm less convinced this part is a good idea, if > "allow_implicit_stop" accepted both generator functions *and* > generator objects, then folks could even still explicitly opt in to > the "or stop()" trick - and anyone reading the code would have a name > to look up to see what was going on. That would look somewhat thus, then: use_an_iterable(allow_implicit_stop(x+1 for x in itr if x<10 or stop())) I'm definitely not convinced that this would improve matters. However, I don't currently have any recommendation for an "or stop()" replacement, other than "refactor it into an explicitly-written generator function and call it, then you can use return statements". Suggestions welcomed. ChrisA From encukou at gmail.com Wed Nov 26 12:44:08 2014 From: encukou at gmail.com (Petr Viktorin) Date: Wed, 26 Nov 2014 12:44:08 +0100 Subject: [Python-Dev] Please reconsider PEP 479. In-Reply-To: References: <54724117.20408@hotpy.org> <547525B1.4090303@canterbury.ac.nz> <54758FB2.6080100@canterbury.ac.nz> Message-ID: On Wed, Nov 26, 2014 at 12:24 PM, Nick Coghlan wrote: > On 26 November 2014 at 18:30, Greg Ewing wrote: >> Guido van Rossum wrote: >>> >>> Hm, that sounds like you're either being contrarian or Chris and I have >>> explained it even worse than I thought. >> >> I'm not trying to be contrary, I just think the PEP could >> explain more clearly what you're trying to achieve. The >> rationale is too vague and waffly at the moment. >> >>> Currently, there are cases where list(x for x in xs if P(x)) works while >>> [x for x in xs if P(x)] fails (when P(x) raises StopIteration). With the >>> PEP, both cases will raise some exception >> >> That's a better explanation, I think. > > The other key aspect is that it changes the answer to the question > "How do I gracefully terminate a generator function?". The existing > behaviour has an "or" in the answer: "return from the generator frame, > OR raise StopIteration from the generator frame". That then leads to > the follow on question: "When should I use one over the other?". > > The "from __future__ import generator_stop" answer drops the "or", so > it's just: "return from the generator frame". > > Raising *any* exception inside the generator, including StopIteration, > then counts as non-graceful termination, bringing generators into line > with the PEP 343 philosophy that "hiding flow control in macros makes > your code inscrutable", where here, the hidden flow control is relying > on the fact that a called function raising StopIteration will > currently always gracefully terminate generator execution. > > The key downside is that it means relatively idiomatic code like: > > def my_generator(): > ... > yield next(it) > ... > > Now needs to be written out explicitly as: > > def my_generator(): > ... > try: > yield next(it) > except StopIteration > return > ... It could also be written as: def my_generator(): try: ... yield next(it) ... except StopIteration return i.e. put the try-block around the whole body, not just the individual yield. This emulates what's happenning in current Python, and it would be faster than individual try blocks. From ischwabacher at wisc.edu Wed Nov 26 16:34:22 2014 From: ischwabacher at wisc.edu (Isaac Schwabacher) Date: Wed, 26 Nov 2014 09:34:22 -0600 Subject: [Python-Dev] PEP 479: Change StopIteration handling inside generators In-Reply-To: <7640864c78147.5475f2f0@wiscmail.wisc.edu> References: <20141125152024.GF2748@ando.pearwood.info> <762099ad85451.5474c008@wiscmail.wisc.edu> <75f0c659864dc.54746bd6@wiscmail.wisc.edu> <73809b5d7de67.5475f23b@wiscmail.wisc.edu> <75d0db417ff56.5475f277@wiscmail.wisc.edu> <74809efd7adce.5475f2b4@wiscmail.wisc.edu> <7640864c78147.5475f2f0@wiscmail.wisc.edu> Message-ID: <764090a77d7a4.54759e9e@wiscmail.wisc.edu> On 11/26/14, Nick Coghlan wrote: > On 26 November 2014 at 04:04, Guido van Rossum wrote: > > On Tue, Nov 25, 2014 at 9:49 AM, Chris Angelico wrote: > >> > >> On Wed, Nov 26, 2014 at 4:45 AM, Isaac Schwabacher > >> wrote: > >> > Yield can also raise StopIteration, if it's thrown in. The current > >> > interaction of generator.throw(StopIteration) with yield from can't be > >> > emulated under the PEP's behavior, though it's not clear that that's a > >> > problem. > >> > > >> > >> Hrm. I have *absolutely* no idea when you would use that, and how > >> you'd go about reworking it to fit this proposal. Do you have any > >> example code (production or synthetic) which throws StopIteration into > >> a generator? > > > > > > Sounds like a good one for the obfuscated Python contest. :-) > > > > Unless the generator has a try/except surrounding the yield point into which > > the exception is thrown, it will bubble right out, and PEP 479 will turn > > this into a RuntimeError. I'll clarify this in the PEP (even though it > > logically follows from the proposal) -- I don't think there's anything to > > worry about. > > This is actually the edge case that needs to be handled in contextlib > - a StopIteration raised by the with statement body gets thrown into > the generator implementing the context manager. My current porting > recommendation is to catch the RuntimeError & look at __cause__ to see > if it's the StopIteration instance that was thrown in, but an > alternative option would be to just call gen.close() in that case, > rather than gen.throw(exc). If this is the current contextlib implementation, does it break if the yield statement is replaced with yield from another context manager generator? ijs From hrvoje.niksic at avl.com Wed Nov 26 16:55:27 2014 From: hrvoje.niksic at avl.com (Hrvoje Niksic) Date: Wed, 26 Nov 2014 16:55:27 +0100 Subject: [Python-Dev] Please reconsider PEP 479. In-Reply-To: References: <54724117.20408@hotpy.org> <547525B1.4090303@canterbury.ac.nz> <54758FB2.6080100@canterbury.ac.nz> Message-ID: <5475F7EF.7050107@avl.com> On 11/26/2014 12:24 PM, Nick Coghlan wrote: > Now needs to be written out explicitly as: > > def my_generator(): > ... > try: > yield next(it) > except StopIteration > return > ... To retrieve a single value from an iterator, one can use the for/break/else idiom: def my_generator(): ... for val in it: yield val break else: return ... In general, catching and raising StopIteration feels like something that should rarely be done by normal code. From rosuav at gmail.com Wed Nov 26 17:15:21 2014 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 27 Nov 2014 03:15:21 +1100 Subject: [Python-Dev] Please reconsider PEP 479. In-Reply-To: <5475F7EF.7050107@avl.com> References: <54724117.20408@hotpy.org> <547525B1.4090303@canterbury.ac.nz> <54758FB2.6080100@canterbury.ac.nz> <5475F7EF.7050107@avl.com> Message-ID: On Thu, Nov 27, 2014 at 2:55 AM, Hrvoje Niksic wrote: > To retrieve a single value from an iterator, one can use the for/break/else > idiom: > > def my_generator(): > ... > for val in it: > yield val > break > else: > return > ... While that does work, it's not really much more "normal" than a try/except. A for loop implies iteration; having a loop with a hard "break" at the end of it messes with expectations. ChrisA From ischwabacher at wisc.edu Wed Nov 26 17:24:24 2014 From: ischwabacher at wisc.edu (Isaac Schwabacher) Date: Wed, 26 Nov 2014 10:24:24 -0600 Subject: [Python-Dev] PEP 479: Change StopIteration handling inside generators In-Reply-To: <76a091178e62b.5475fe89@wiscmail.wisc.edu> References: <20141125152024.GF2748@ando.pearwood.info> <762099ad85451.5474c008@wiscmail.wisc.edu> <75f0c659864dc.54746bd6@wiscmail.wisc.edu> <7670967d8a424.5475fe11@wiscmail.wisc.edu> <7740f9318c795.5475fe4d@wiscmail.wisc.edu> <76a091178e62b.5475fe89@wiscmail.wisc.edu> Message-ID: <76609b5c8b416.5475aa58@wiscmail.wisc.edu> On 11/26/14, Nick Coghlan wrote: > On 26 November 2014 at 04:04, Guido van Rossum wrote: > > On Tue, Nov 25, 2014 at 9:49 AM, Chris Angelico wrote: > >> > >> On Wed, Nov 26, 2014 at 4:45 AM, Isaac Schwabacher > >> wrote: > >> > Yield can also raise StopIteration, if it's thrown in. The current > >> > interaction of generator.throw(StopIteration) with yield from can't be > >> > emulated under the PEP's behavior, though it's not clear that that's a > >> > problem. > >> > > >> > >> Hrm. I have *absolutely* no idea when you would use that, and how > >> you'd go about reworking it to fit this proposal. Do you have any > >> example code (production or synthetic) which throws StopIteration into > >> a generator? > > > > > > Sounds like a good one for the obfuscated Python contest. :-) > > > > Unless the generator has a try/except surrounding the yield point into which > > the exception is thrown, it will bubble right out, and PEP 479 will turn > > this into a RuntimeError. I'll clarify this in the PEP (even though it > > logically follows from the proposal) -- I don't think there's anything to > > worry about. > > This is actually the edge case that needs to be handled in contextlib > - a StopIteration raised by the with statement body gets thrown into > the generator implementing the context manager. My current porting > recommendation is to catch the RuntimeError & look at __cause__ to see > if it's the StopIteration instance that was thrown in, but an > alternative option would be to just call gen.close() in that case, > rather than gen.throw(exc). This actually leads to a good example of why the PEP is necessary: ``` In [1]: import contextlib In [2]: @contextlib.contextmanager ...: def transact(): ...: print('setup transaction') ...: try: ...: yield from subgenerator() ...: except: ...: print('rollback transaction') ...: raise ...: else: ...: print('commit transaction') ...: finally: ...: print('clean up transaction') ...: In [3]: def subgenerator(): ...: print('setup subgenerator') ...: try: ...: yield ...: except: ...: print('subgenerator failed') ...: raise ...: else: ...: print('subgenerator succeeded') ...: finally: ...: print('clean up subgenerator') ...: In [4]: with transact(): ...: next(iter([])) ...: setup transaction setup subgenerator subgenerator failed clean up subgenerator commit transaction # BAD NOT GOOD BIG FAIL clean up transaction ``` ijs From ethan at stoneleaf.us Wed Nov 26 16:34:40 2014 From: ethan at stoneleaf.us (Ethan Furman) Date: Wed, 26 Nov 2014 07:34:40 -0800 Subject: [Python-Dev] Please reconsider PEP 479. In-Reply-To: References: <54724117.20408@hotpy.org> <547525B1.4090303@canterbury.ac.nz> <54758FB2.6080100@canterbury.ac.nz> Message-ID: <5475F310.3080103@stoneleaf.us> On 11/26/2014 03:24 AM, Nick Coghlan wrote: > > After thinking about that concern for a while, I'd like to suggest the > idea of having a new builtin "allow_implicit_stop" decorator that > swaps out a GENERATOR code object that has the new "EXPLICIT_STOP" > flag set for one with it cleared (attempting to apply > "allow_implicit_stop" to a normal function would be an error). This solution is incomplete -- it's saying that if there is one guarded location, all locations must be guarded, and that is not the case; any other call outside the try/except block has the potential to raise StopIteration. -- ~Ethan~ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: OpenPGP digital signature URL: From p.f.moore at gmail.com Wed Nov 26 17:54:17 2014 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 26 Nov 2014 16:54:17 +0000 Subject: [Python-Dev] PEP 479: Change StopIteration handling inside generators In-Reply-To: <76609b5c8b416.5475aa58@wiscmail.wisc.edu> References: <20141125152024.GF2748@ando.pearwood.info> <762099ad85451.5474c008@wiscmail.wisc.edu> <75f0c659864dc.54746bd6@wiscmail.wisc.edu> <7670967d8a424.5475fe11@wiscmail.wisc.edu> <7740f9318c795.5475fe4d@wiscmail.wisc.edu> <76a091178e62b.5475fe89@wiscmail.wisc.edu> <76609b5c8b416.5475aa58@wiscmail.wisc.edu> Message-ID: On 26 November 2014 at 16:24, Isaac Schwabacher wrote: > This actually leads to a good example of why the PEP is necessary: [...] Oh! If that's the current behaviour, then it probably needs to go into the PEP as a motivating example. It's far more convincing than most of the other arguments I've seen. Just one proviso - is it fixable in contextlib *without* a language change? If so, then it loses a lot of its value. Paul From guido at python.org Wed Nov 26 18:19:57 2014 From: guido at python.org (Guido van Rossum) Date: Wed, 26 Nov 2014 09:19:57 -0800 Subject: [Python-Dev] PEP 479: Change StopIteration handling inside generators In-Reply-To: References: <20141125152024.GF2748@ando.pearwood.info> <762099ad85451.5474c008@wiscmail.wisc.edu> <75f0c659864dc.54746bd6@wiscmail.wisc.edu> <7670967d8a424.5475fe11@wiscmail.wisc.edu> <7740f9318c795.5475fe4d@wiscmail.wisc.edu> <76a091178e62b.5475fe89@wiscmail.wisc.edu> <76609b5c8b416.5475aa58@wiscmail.wisc.edu> Message-ID: On Wed, Nov 26, 2014 at 8:54 AM, Paul Moore wrote: > On 26 November 2014 at 16:24, Isaac Schwabacher > wrote: > > This actually leads to a good example of why the PEP is necessary: > [...] > > Oh! If that's the current behaviour, then it probably needs to go into > the PEP as a motivating example. It's far more convincing than most of > the other arguments I've seen. Just one proviso - is it fixable in > contextlib *without* a language change? If so, then it loses a lot of > its value. > It's hard to use as an example because the behavior of contextlib is an integral part of it -- currently for me the example boils down to "there is a bug in contextlib". Maybe it would have been caught earlier with the change in the PEP, but when using it as a motivating example you have to show the code containing the bug, not just a demonstration. If you want to try though, I'm happy to entertain a pull request for the PEP. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Wed Nov 26 18:58:06 2014 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 26 Nov 2014 17:58:06 +0000 Subject: [Python-Dev] PEP 479: Change StopIteration handling inside generators In-Reply-To: References: <20141125152024.GF2748@ando.pearwood.info> <762099ad85451.5474c008@wiscmail.wisc.edu> <75f0c659864dc.54746bd6@wiscmail.wisc.edu> <7670967d8a424.5475fe11@wiscmail.wisc.edu> <7740f9318c795.5475fe4d@wiscmail.wisc.edu> <76a091178e62b.5475fe89@wiscmail.wisc.edu> <76609b5c8b416.5475aa58@wiscmail.wisc.edu> Message-ID: On 26 November 2014 at 17:19, Guido van Rossum wrote: > It's hard to use as an example because the behavior of contextlib is an > integral part of it -- currently for me the example boils down to "there is > a bug in contextlib" Hmm, fair point. I was assuming that the bug in contextlib can't be fixed with the current language behaviour (and I'd personally be OK with the example simply adding a comment "this can't be fixed without changing Python as proposed in the PEP"). But I'm not sure how true that is, so maybe it's not quite as compelling as it seemed to me at first. Paul From ethan at stoneleaf.us Wed Nov 26 19:07:40 2014 From: ethan at stoneleaf.us (Ethan Furman) Date: Wed, 26 Nov 2014 10:07:40 -0800 Subject: [Python-Dev] PEP 479: Change StopIteration handling inside generators In-Reply-To: References: <20141125152024.GF2748@ando.pearwood.info> <762099ad85451.5474c008@wiscmail.wisc.edu> <75f0c659864dc.54746bd6@wiscmail.wisc.edu> <7670967d8a424.5475fe11@wiscmail.wisc.edu> <7740f9318c795.5475fe4d@wiscmail.wisc.edu> <76a091178e62b.5475fe89@wiscmail.wisc.edu> <76609b5c8b416.5475aa58@wiscmail.wisc.edu> Message-ID: <547616EC.8050603@stoneleaf.us> On 11/26/2014 08:54 AM, Paul Moore wrote: > On 26 November 2014 at 16:24, Isaac Schwabacher wrote: >> This actually leads to a good example of why the PEP is necessary: > [...] > > Oh! If that's the current behaviour, then it probably needs to go into > the PEP as a motivating example. It's far more convincing than most of > the other arguments I've seen. Just one proviso - is it fixable in > contextlib *without* a language change? If so, then it loses a lot of > its value. No value is lost. The PEP exists because this mistake is so easy to make, and can be so hard to track down. -- ~Ethan~ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: OpenPGP digital signature URL: From benjamin at python.org Wed Nov 26 19:40:44 2014 From: benjamin at python.org (Benjamin Peterson) Date: Wed, 26 Nov 2014 13:40:44 -0500 Subject: [Python-Dev] [RELEASE] Python 2.7.9 release candidate 1 Message-ID: <1417027244.3509101.195765385.58D1A695@webmail.messagingengine.com> I'm pleased to announce the first release candidate of Python 2.7.9, which will be the next bugfix release in the Python 2.7 series. Despite technically being a maintenance release, Python 2.7.9 will include several majors changes from 2.7.8: - The "ensurepip" module has been backported to Python 2.7. - Python 3's ssl module has been backported to Python 2.7. - HTTPS certificates are now verified by default using the system's certificate store. - SSLv3 has been disabled by default due to the POODLE attack. Downloads are at https://www.python.org/downloads/release/python-279rc1/ Application and library authors are encouraged test Python 2.7.9 release candidate 1 with their code. This is especially important for 2.7.9 due to significant changes mentioned above. Please report bugs to https://bugs.python.org/ Python 2.7.9 final is currently scheduled for December 10th. Enjoy, Benjamin 2.7 release manager on behalf on python-dev and all of Python's contributors From guido at python.org Wed Nov 26 21:34:48 2014 From: guido at python.org (Guido van Rossum) Date: Wed, 26 Nov 2014 12:34:48 -0800 Subject: [Python-Dev] Please reconsider PEP 479. In-Reply-To: References: <54724117.20408@hotpy.org> <547525B1.4090303@canterbury.ac.nz> <54758FB2.6080100@canterbury.ac.nz> Message-ID: On Wed, Nov 26, 2014 at 3:24 AM, Nick Coghlan wrote: > On 26 November 2014 at 18:30, Greg Ewing > wrote: > > Guido van Rossum wrote: > >> > >> Hm, that sounds like you're either being contrarian or Chris and I have > >> explained it even worse than I thought. > > > > I'm not trying to be contrary, I just think the PEP could > > explain more clearly what you're trying to achieve. The > > rationale is too vague and waffly at the moment. > > > >> Currently, there are cases where list(x for x in xs if P(x)) works while > >> [x for x in xs if P(x)] fails (when P(x) raises StopIteration). With the > >> PEP, both cases will raise some exception > > > > That's a better explanation, I think. > It's now in the PEP. > The other key aspect is that it changes the answer to the question > "How do I gracefully terminate a generator function?". The existing > behaviour has an "or" in the answer: "return from the generator frame, > OR raise StopIteration from the generator frame". That then leads to > the follow on question: "When should I use one over the other?". > > The "from __future__ import generator_stop" answer drops the "or", so > it's just: "return from the generator frame". > That's now also in the PEP. > Raising *any* exception inside the generator, including StopIteration, > then counts as non-graceful termination, bringing generators into line > with the PEP 343 philosophy that "hiding flow control in macros makes > your code inscrutable", where here, the hidden flow control is relying > on the fact that a called function raising StopIteration will > currently always gracefully terminate generator execution. > Right. > The key downside is that it means relatively idiomatic code like: > > def my_generator(): > ... > yield next(it) > ... > I probably considered this an upside of generators when they were introduced. :-( > Now needs to be written out explicitly as: > > def my_generator(): > ... > try: > yield next(it) > except StopIteration > return > ... > > That's not especially easy to read, and it's also going to be very > slow when working with generator based producer/consumer pipelines. > I want to consider this performance argument seriously. Chris did a little benchmark but I don't think he compared the right things -- he showed that "yield from" becomes 5% slower with his patch and that a while loop is twice as slow as "yield from" with or without his patch. I have no idea why his patch would slow down "yield from" but I doubt it's directly related -- his change only adds some extra code when a generator frame is left with an exception, but his "yield from" example code ( https://github.com/Rosuav/GenStopIter/blob/485d1/perftest.py) never raises (unless I really don't understand how the implementation of "yield from" actually works :-). I guess what we *should* benchmark is this: def g(depth): if depth > 0: it = g(depth-1) yield next(it) else: yield 42 vs. the PEP-479-ly corrected version: def g(depth): if depth > 0: it = g(depth-1) try: yield next(it) except StopIteration: pass else: yield 42 This sets up "depth" generators each with a try/except, and then at the very bottom yields a single value (42) which pops up all the way to the top, never raising StopIteration. I wrote the benchmark and here are the code and results: https://gist.github.com/gvanrossum/1adb5bee99400ce615a5 It's clear that the extra try/except setup aren't free, even if the except is never triggered. My summary of the results is that the try/except setup costs 100-200 nsec, while the rest of the code executed in the frame takes about 600-800 nsec. (Hardware: MacBook Pro with 2.8 GHz Intel Core i7.) Incidentally, the try/except cost has come down greatly from Python 2.7, where it's over a microsecond! I also tried a variation where the bottommost generator doesn't yield a value. The conclusion is about the same -- the try/except version is 150 nsec slower. So now we have a number to worry about (150 nsec for a try/except) and I have to think about whether or not that's likely to have a noticeable effect in realistic situations. One recommendation follows: if you have a loop inside your generator, and there's a next() call in the loop, put the try/except around the loop, so you pay the setup cost only once (unless the loop is most likely to have zero iterations, and unlikely to have more than one). > After thinking about that concern for a while, I'd like to suggest the > idea of having a new builtin "allow_implicit_stop" decorator that > swaps out a GENERATOR code object that has the new "EXPLICIT_STOP" > flag set for one with it cleared (attempting to apply > "allow_implicit_stop" to a normal function would be an error). > > Then the updated version of the above example would become: > > @allow_implicit_stop > def my_generator(): > ... > yield next(it) > ... > > Which would be semantically equivalent to: > > def my_generator(): > try: > ... > yield next(it) > ... > except StopIteration > return > > but *much* faster (especially if used in a producer/consumer pipeline) > since it would allow a single StopIteration instance to propagate > through the entire pipeline, rather than creating and destroying new > ones at each stage. > I think we can put a number to "much faster" now -- 150 nsec per try/except. I have serious misgivings about that decorator though -- I'm not sure how viable it is to pass a flag from the function object to the execution (which takes the code object, which is immutable) and how other Python implementations would do that. But I'm sure it can be done through sheer willpower. I'd call it the @hettinger decorator in honor of the PEP's most eloquent detractor. :-) Including such a feature in the PEP would also make the fix to > contextlib simpler: we'd just update it so that > contextlib._GeneratorContextManager automatically calls > "allow_implicit_stop" on the passed in generator functions. > > Single-source Python 2/3 code would also benefit in a 3.7+ world, > since libraries like six and python-future could just define their own > version of "allow_implicit_stop" that referred to the new builtin in > 3.5+, and was implemented as an identity function in other versions. > Well, yes, but you could also trivially write single-source Python, without the use of any adapter library, by just writing the try/except. > Cheers, > Nick. > > P.S. While I'm less convinced this part is a good idea, if > "allow_implicit_stop" accepted both generator functions *and* > generator objects, then folks could even still explicitly opt in to > the "or stop()" trick - and anyone reading the code would have a name > to look up to see what was going on. Well, here's how you could write the @hettinger decorator without any help from the compiler: catch RuntimeError, check for a chained StopIteration, and re-raise that. It wouldn't be any faster, but it would emulate the pre-PEP world perfectly. :-) -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Wed Nov 26 21:36:36 2014 From: guido at python.org (Guido van Rossum) Date: Wed, 26 Nov 2014 12:36:36 -0800 Subject: [Python-Dev] PEP 479: Change StopIteration handling inside generators In-Reply-To: <75f0fa2b78673.5475e0ec@wiscmail.wisc.edu> References: <20141125152024.GF2748@ando.pearwood.info> <762099ad85451.5474c008@wiscmail.wisc.edu> <75f0c659864dc.54746bd6@wiscmail.wisc.edu> <7670967d8a424.5475fe11@wiscmail.wisc.edu> <7740f9318c795.5475fe4d@wiscmail.wisc.edu> <76a091178e62b.5475fe89@wiscmail.wisc.edu> <76609b5c8b416.5475aa58@wiscmail.wisc.edu> <73a0d22c7b73e.54762ae0@wiscmail.wisc.edu> <7740e33d7f1e2.54762c0f@wiscmail.wisc.edu> <7630fc1c7e499.54762c4b@wiscmail.wisc.edu> <7480d5717827c.54762df2@wiscmail.wisc.edu> <7530bd45785c8.54762e2f@wiscmail.wisc.edu> <75f0ff337d57b.54762f5e@wiscmail.wisc.edu> <7530f3ad7f808.54762fd9@wiscmail.wisc.edu> <7480d3c87c41c.54763015@wiscmail.wisc.edu> <763082b77f3cf.54763052@wiscmail.wisc.edu> <73809d277d189.54763091@wiscmail.wisc.edu> <7640c5e878eae.547630cd@wiscmail.wisc.edu> <774091367c0f1.5476310a@wiscmail.wisc.edu> <75d0c4467bf70.54763146@wiscmail.wisc.edu> <764081797ae74.54763182@wiscmail.wisc.edu> <738097f87db8b.547631bf@wiscmail.wisc.edu> <7610ea077f809.547631fb@wiscmail.wisc.edu> <7610d4b57d7f3.54763238@wiscmail.wisc.edu> <73a097287906a.54763274@wiscmail.wisc.edu> <75d0a9817d06a.547632b0@wiscmail.wisc.edu> <73a0ca317a13e.547632ed@wiscmail.wisc.edu> <7480865f7b7b8.54763329@wiscmail.wisc.edu> <7530a1987c5a2.54763366@wiscmail.wisc.edu> <73808f727cc19.547633a2@wiscmail.wisc.edu> <7610ec8f7b703.54763492@wiscmail.wisc.edu> <73a0a83f7fcfd.547634cf@wiscmail.wisc.edu> <73a0c2717c6e3.5476350b@wiscmail.wisc.edu> <7380d6b47eb58.54763548@wiscmail.wisc.edu> <75f0fa2b78673.5475e0ec@wiscmail.wisc.edu> Message-ID: Can you summarize that in a self-contained form for inclusion in the PEP? (That was a rhetorical question. :-) On Wed, Nov 26, 2014 at 12:17 PM, Isaac Schwabacher wrote: > On 14-11-26, Guido van Rossum wrote: > > On Wed, Nov 26, 2014 at 8:54 AM, Paul Moore wrote: > > > > > On 26 November 2014 at 16:24, Isaac Schwabacher wrote: > > > > This actually leads to a good example of why the PEP is necessary: > > > [...] > > > > > > Oh! If that's the current behaviour, then it probably needs to go into > > > the PEP as a motivating example. It's far more convincing than most of > > > the other arguments I've seen. Just one proviso - is it fixable in > > > contextlib *without* a language change? If so, then it loses a lot of > > > its value. > > > > It's hard to use as an example because the behavior of contextlib is an > integral part of it -- currently for me the example boils down to "there is > a bug in contextlib". Maybe it would have been caught earlier with the > change in the PEP, but when using it as a motivating example you have to > show the code containing the bug, not just a demonstration. > > How is this a bug in contextlib? The example behaves the way it does > because gen.throw(StopIteration) behaves differently depending on whether > gen is paused at a yield or a yield from. What *should* > contextlib.contextmanager do in this instance? It has faithfully forwarded > the StopIteration raised in the protected block to the generator, and the > generator has forwarded this to the subgenerator, which has elected to fail > and report success. The bug is in the subgenerator, because it fails to > treat StopIteration as an error. But the subgenerator can't in general be > converted to treat StopIteration as an error, because clearly it's used in > other places than as a nested context manager (otherwise, it would itself > be decorated with @contextlib.contextmanager and accessed as such, instead > of yielded from). And in those places, perhaps it needs to simply allow > StopIteration to bubble up. And can we factor out the error checking so > that we don't have to duplicate subgenerator? Well... yes, but it's tricky > because we'll introduce an extra yield from in the process, so we have to > put the handling in the subgenerator itself and wrap the > *non*-context-manager uses. > > ijs > > > If you want to try though, I'm happy to entertain a pull request for the > PEP. > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From ischwabacher at wisc.edu Wed Nov 26 21:17:16 2014 From: ischwabacher at wisc.edu (Isaac Schwabacher) Date: Wed, 26 Nov 2014 14:17:16 -0600 Subject: [Python-Dev] PEP 479: Change StopIteration handling inside generators In-Reply-To: <7380d6b47eb58.54763548@wiscmail.wisc.edu> References: <20141125152024.GF2748@ando.pearwood.info> <762099ad85451.5474c008@wiscmail.wisc.edu> <75f0c659864dc.54746bd6@wiscmail.wisc.edu> <7670967d8a424.5475fe11@wiscmail.wisc.edu> <7740f9318c795.5475fe4d@wiscmail.wisc.edu> <76a091178e62b.5475fe89@wiscmail.wisc.edu> <76609b5c8b416.5475aa58@wiscmail.wisc.edu> <73a0d22c7b73e.54762ae0@wiscmail.wisc.edu> <7740e33d7f1e2.54762c0f@wiscmail.wisc.edu> <7630fc1c7e499.54762c4b@wiscmail.wisc.edu> <7480d5717827c.54762df2@wiscmail.wisc.edu> <7530bd45785c8.54762e2f@wiscmail.wisc.edu> <75f0ff337d57b.54762f5e@wiscmail.wisc.edu> <7530f3ad7f808.54762fd9@wiscmail.wisc.edu> <7480d3c87c41c.54763015@wiscmail.wisc.edu> <763082b77f3cf.54763052@wiscmail.wisc.edu> <73809d277d189.54763091@wiscmail.wisc.edu> <7640c5e878eae.547630cd@wiscmail.wisc.edu> <774091367c0f1.5476310a@wiscmail.wisc.edu> <75d0c4467bf70.54763146@wiscmail.wisc.edu> <764081797ae74.54763182@wiscmail.wisc.edu> <738097f87db8b.547631bf@wiscmail.wisc.edu> <7610ea077f809.547631fb@wiscmail.wisc.edu> <7610d4b57d7f3.54763238@wiscmail.wisc.edu> <73a097287906a.54763274@wiscmail.wisc.edu> <75d0a9817d06a.547632b0@wiscmail.wisc.edu> <73a0ca317a13e.547632ed@wiscmail.wisc.edu> <7480865f7b7b8.54763329@wiscmail.wisc.edu> <7530a1987c5a2.54763366@wiscmail.wisc.edu> <73808f727cc19.547633a2@wiscmail.wisc.edu> <7610ec8f7b703.54763492@wiscmail.wisc.edu> <73a0a83f7fcfd.547634cf@wiscmail.wisc.edu> <73a0c2717c6e3.5476350b@wiscmail.wisc.edu> <7380d6b47eb58.54763548@wiscmail.wisc.edu> Message-ID: <75f0fa2b78673.5475e0ec@wiscmail.wisc.edu> On 14-11-26, Guido van Rossum wrote: > On Wed, Nov 26, 2014 at 8:54 AM, Paul Moore wrote: > > > On 26 November 2014 at 16:24, Isaac Schwabacher wrote: > > > This actually leads to a good example of why the PEP is necessary: > > [...] > > > > Oh! If that's the current behaviour, then it probably needs to go into > > the PEP as a motivating example. It's far more convincing than most of > > the other arguments I've seen. Just one proviso - is it fixable in > > contextlib *without* a language change? If so, then it loses a lot of > > its value. > > It's hard to use as an example because the behavior of contextlib is an integral part of it -- currently for me the example boils down to "there is a bug in contextlib". Maybe it would have been caught earlier with the change in the PEP, but when using it as a motivating example you have to show the code containing the bug, not just a demonstration. How is this a bug in contextlib? The example behaves the way it does because gen.throw(StopIteration) behaves differently depending on whether gen is paused at a yield or a yield from. What *should* contextlib.contextmanager do in this instance? It has faithfully forwarded the StopIteration raised in the protected block to the generator, and the generator has forwarded this to the subgenerator, which has elected to fail and report success. The bug is in the subgenerator, because it fails to treat StopIteration as an error. But the subgenerator can't in general be converted to treat StopIteration as an error, because clearly it's used in other places than as a nested context manager (otherwise, it would itself be decorated with @contextlib.contextmanager and accessed as such, instead of yielded from). And in those places, perhaps it needs to simply allow StopIteration to bubble up. And can we factor out the error checking so that we don't have to duplicate subgenerator? Well... yes, but it's tricky because we'll introduce an extra yield from in the process, so we have to put the handling in the subgenerator itself and wrap the *non*-context-manager uses. ijs > If you want to try though, I'm happy to entertain a pull request for the PEP. From ncoghlan at gmail.com Wed Nov 26 23:53:10 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 27 Nov 2014 08:53:10 +1000 Subject: [Python-Dev] Please reconsider PEP 479. In-Reply-To: References: <54724117.20408@hotpy.org> <547525B1.4090303@canterbury.ac.nz> <54758FB2.6080100@canterbury.ac.nz> Message-ID: On 27 Nov 2014 06:35, "Guido van Rossum" wrote: > > On Wed, Nov 26, 2014 at 3:24 AM, Nick Coghlan wrote: >> After thinking about that concern for a while, I'd like to suggest the >> idea of having a new builtin "allow_implicit_stop" decorator that >> swaps out a GENERATOR code object that has the new "EXPLICIT_STOP" >> flag set for one with it cleared (attempting to apply >> "allow_implicit_stop" to a normal function would be an error). >> >> Then the updated version of the above example would become: >> >> @allow_implicit_stop >> def my_generator(): >> ... >> yield next(it) >> ... >> >> Which would be semantically equivalent to: >> >> def my_generator(): >> try: >> ... >> yield next(it) >> ... >> except StopIteration >> return >> >> but *much* faster (especially if used in a producer/consumer pipeline) >> since it would allow a single StopIteration instance to propagate >> through the entire pipeline, rather than creating and destroying new >> ones at each stage. > > > I think we can put a number to "much faster" now -- 150 nsec per try/except. > > I have serious misgivings about that decorator though -- I'm not sure how viable it is to pass a flag from the function object to the execution (which takes the code object, which is immutable) and how other Python implementations would do that. But I'm sure it can be done through sheer willpower. I'd call it the @hettinger decorator in honor of the PEP's most eloquent detractor. :-) I agree with everything you wrote in your reply, so I'll just elaborate a bit on my proposed implementation for the decorator idea. What I was thinking is that to implement the __future__ import, we're going to need a code object flag. For the sake of discussion, let's assume that flag is called "EXPLICIT_STOP". When StopIteration escapes from a generator frame, we'll check for that flag and if it's set, StopIteration will be converted to RuntimeError. The details will differ for other implementations, but they're going to need something at least approximately equivalent to handle the transition period. The implicit stop decorator would then check the flags on the code object attached to the passed in function. If GENERATOR wasn't set, that would be an immediate ValueError, while if EXPLICIT_STOP wasn't set, the generator function would be passed through unmodified. However, if EXPLICIT_STOP *was* set, the generator function would be replaced by a *new* generator function with a *new* code object, where the only change was to clear the EXPLICIT_STOP flag. The core part of the idea is actually keeping the runtime check for the EXPLICIT_STOP flag on generator code objects even *after* setting the flag becomes the default compile time behaviour. The builtin (or itertools?) decorator to revert to the current behaviour on a per-generator basis then becomes a convenient way to access (and document!) that functionality without messing about with your own dynamic code object creation, and without coming up with new syntax for it (since new syntax can't be emulated on older versions of Python, or made implicit in other decorators like contextlib.contextmanager). Regards, Nick. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Thu Nov 27 00:15:43 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 27 Nov 2014 09:15:43 +1000 Subject: [Python-Dev] PEP 479: Change StopIteration handling inside generators In-Reply-To: References: <20141125152024.GF2748@ando.pearwood.info> <762099ad85451.5474c008@wiscmail.wisc.edu> <75f0c659864dc.54746bd6@wiscmail.wisc.edu> <7670967d8a424.5475fe11@wiscmail.wisc.edu> <7740f9318c795.5475fe4d@wiscmail.wisc.edu> <76a091178e62b.5475fe89@wiscmail.wisc.edu> <76609b5c8b416.5475aa58@wiscmail.wisc.edu> Message-ID: On 27 Nov 2014 03:58, "Paul Moore" wrote: > > On 26 November 2014 at 17:19, Guido van Rossum wrote: > > It's hard to use as an example because the behavior of contextlib is an > > integral part of it -- currently for me the example boils down to "there is > > a bug in contextlib" > > Hmm, fair point. I was assuming that the bug in contextlib can't be > fixed with the current language behaviour (and I'd personally be OK > with the example simply adding a comment "this can't be fixed without > changing Python as proposed in the PEP"). But I'm not sure how true > that is, so maybe it's not quite as compelling as it seemed to me at > first. The "contextlib only" change would be to map StopIteration in the body of the with statement to gen.close() on the underlying generator rather than gen.throw(StopIteration). (That's backwards incompatible in its own way, since it means you *can't* suppress StopIteration via a generator based context manager any more) This is actually the second iteration of this bug: the original implementation *always* suppressed StopIteration. PJE caught that one before Python 2.5 was released, but we didn't notice that 3.3 had brought it back in a new, more subtle form :( It's worth noting that my "allow_implicit_stop" idea in the other thread wouldn't affect subgenerators - those would still convert StopIteration to RuntimeError unless explicitly silenced. Regards, Nick. -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Thu Nov 27 00:44:57 2014 From: guido at python.org (Guido van Rossum) Date: Wed, 26 Nov 2014 15:44:57 -0800 Subject: [Python-Dev] PEP 479: Change StopIteration handling inside generators In-Reply-To: <7380d3417e532.5475f879@wiscmail.wisc.edu> References: <20141125152024.GF2748@ando.pearwood.info> <762099ad85451.5474c008@wiscmail.wisc.edu> <75f0c659864dc.54746bd6@wiscmail.wisc.edu> <7670967d8a424.5475fe11@wiscmail.wisc.edu> <7740f9318c795.5475fe4d@wiscmail.wisc.edu> <76a091178e62b.5475fe89@wiscmail.wisc.edu> <76609b5c8b416.5475aa58@wiscmail.wisc.edu> <73a0d22c7b73e.54762ae0@wiscmail.wisc.edu> <7740e33d7f1e2.54762c0f@wiscmail.wisc.edu> <7630fc1c7e499.54762c4b@wiscmail.wisc.edu> <7480d5717827c.54762df2@wiscmail.wisc.edu> <7530bd45785c8.54762e2f@wiscmail.wisc.edu> <75f0ff337d57b.54762f5e@wiscmail.wisc.edu> <7530f3ad7f808.54762fd9@wiscmail.wisc.edu> <7480d3c87c41c.54763015@wiscmail.wisc.edu> <763082b77f3cf.54763052@wiscmail.wisc.edu> <73809d277d189.54763091@wiscmail.wisc.edu> <7640c5e878eae.547630cd@wiscmail.wisc.edu> <774091367c0f1.5476310a@wiscmail.wisc.edu> <75d0c4467bf70.54763146@wiscmail.wisc.edu> <764081797ae74.54763182@wiscmail.wisc.edu> <738097f87db8b.547631bf@wiscmail.wisc.edu> <7610ea077f809.547631fb@wiscmail.wisc.edu> <7610d4b57d7f3.54763238@wiscmail.wisc.edu> <73a097287906a.54763274@wiscmail.wisc.edu> <75d0a9817d06a.547632b0@wiscmail.wisc.edu> <73a0ca317a13e.547632ed@wiscmail.wisc.edu> <7480865f7b7b8.54763329@wiscmail.wisc.edu> <7530a1987c5a2.54763366@wiscmail.wisc.edu> <73808f727cc19.547633a2@wiscmail.wisc.edu> <7610ec8f7b703.54763492@wiscmail.wisc.edu> <73a0a83f7fcfd.547634cf@wiscmail.wisc.edu> <73a0c2717c6e3.5476350b@wiscmail.wisc.edu> <7380d6b47eb58.54763548@wiscmail.wisc.edu> <75f0fa2b78673.5475e0ec@wiscmail.wisc.edu> <7480f3917963b.54763b35@wiscmail.wisc.edu> <75f0e0b37a192.54763b71@wiscmail.wisc.edu> <7530b88879d4c.5476451a@wiscmail.wisc.edu> <7380d3417e532.5475f879@wiscmail.wisc.edu> Message-ID: You can use the README here: https://github.com/Rosuav/GenStopIter On Wed, Nov 26, 2014 at 1:57 PM, Isaac Schwabacher wrote: > > Can you summarize that in a self-contained form for inclusion in the PEP? > > > > (That was a rhetorical question. :-) > > Sure. Is it on GitHub? ;D > > ijs > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Thu Nov 27 00:50:16 2014 From: guido at python.org (Guido van Rossum) Date: Wed, 26 Nov 2014 15:50:16 -0800 Subject: [Python-Dev] PEP 479: Change StopIteration handling inside generators In-Reply-To: References: <20141125152024.GF2748@ando.pearwood.info> <762099ad85451.5474c008@wiscmail.wisc.edu> <75f0c659864dc.54746bd6@wiscmail.wisc.edu> <7670967d8a424.5475fe11@wiscmail.wisc.edu> <7740f9318c795.5475fe4d@wiscmail.wisc.edu> <76a091178e62b.5475fe89@wiscmail.wisc.edu> <76609b5c8b416.5475aa58@wiscmail.wisc.edu> Message-ID: On Wed, Nov 26, 2014 at 3:15 PM, Nick Coghlan wrote: > > On 27 Nov 2014 03:58, "Paul Moore" wrote: > > > > On 26 November 2014 at 17:19, Guido van Rossum wrote: > > > It's hard to use as an example because the behavior of contextlib is an > > > integral part of it -- currently for me the example boils down to > "there is > > > a bug in contextlib" > > > > Hmm, fair point. I was assuming that the bug in contextlib can't be > > fixed with the current language behaviour (and I'd personally be OK > > with the example simply adding a comment "this can't be fixed without > > changing Python as proposed in the PEP"). But I'm not sure how true > > that is, so maybe it's not quite as compelling as it seemed to me at > > first. > > The "contextlib only" change would be to map StopIteration in the body of > the with statement to gen.close() on the underlying generator rather than > gen.throw(StopIteration). (That's backwards incompatible in its own way, > since it means you *can't* suppress StopIteration via a generator based > context manager any more) > > This is actually the second iteration of this bug: the original > implementation *always* suppressed StopIteration. PJE caught that one > before Python 2.5 was released, but we didn't notice that 3.3 had brought > it back in a new, more subtle form :( > > It's worth noting that my "allow_implicit_stop" idea in the other thread > wouldn't affect subgenerators - those would still convert StopIteration to > RuntimeError unless explicitly silenced. > You've lost me in this subthread. Am I right to conclude that the PEP change doesn't cause problems for contextlib(*), but that the PEP change also probably wouldn't have helped diagnose any contextlib bugs? (*) Except perhaps that some old 3rd party copy of contextlib may eventually break if it's not updated. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Thu Nov 27 01:03:05 2014 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 27 Nov 2014 11:03:05 +1100 Subject: [Python-Dev] Please reconsider PEP 479. In-Reply-To: References: <54724117.20408@hotpy.org> <547525B1.4090303@canterbury.ac.nz> <54758FB2.6080100@canterbury.ac.nz> Message-ID: On Thu, Nov 27, 2014 at 9:53 AM, Nick Coghlan wrote: > The implicit stop decorator would then check the flags on the code object > attached to the passed in function. If GENERATOR wasn't set, that would be > an immediate ValueError, while if EXPLICIT_STOP wasn't set, the generator > function would be passed through unmodified. However, if EXPLICIT_STOP *was* > set, the generator function would be replaced by a *new* generator function > with a *new* code object, where the only change was to clear the > EXPLICIT_STOP flag. Is it possible to replace the code object without replacing the function? Imagine if you have multiple decorators, one of which retains a reference to the function and then this one which replaces it - the order of decoration would be critical. OTOH, I don't know that anyone would retain references to __code__. ChrisA From ischwabacher at wisc.edu Wed Nov 26 22:57:45 2014 From: ischwabacher at wisc.edu (Isaac Schwabacher) Date: Wed, 26 Nov 2014 15:57:45 -0600 Subject: [Python-Dev] PEP 479: Change StopIteration handling inside generators In-Reply-To: <7530b88879d4c.5476451a@wiscmail.wisc.edu> References: <20141125152024.GF2748@ando.pearwood.info> <762099ad85451.5474c008@wiscmail.wisc.edu> <75f0c659864dc.54746bd6@wiscmail.wisc.edu> <7670967d8a424.5475fe11@wiscmail.wisc.edu> <7740f9318c795.5475fe4d@wiscmail.wisc.edu> <76a091178e62b.5475fe89@wiscmail.wisc.edu> <76609b5c8b416.5475aa58@wiscmail.wisc.edu> <73a0d22c7b73e.54762ae0@wiscmail.wisc.edu> <7740e33d7f1e2.54762c0f@wiscmail.wisc.edu> <7630fc1c7e499.54762c4b@wiscmail.wisc.edu> <7480d5717827c.54762df2@wiscmail.wisc.edu> <7530bd45785c8.54762e2f@wiscmail.wisc.edu> <75f0ff337d57b.54762f5e@wiscmail.wisc.edu> <7530f3ad7f808.54762fd9@wiscmail.wisc.edu> <7480d3c87c41c.54763015@wiscmail.wisc.edu> <763082b77f3cf.54763052@wiscmail.wisc.edu> <73809d277d189.54763091@wiscmail.wisc.edu> <7640c5e878eae.547630cd@wiscmail.wisc.edu> <774091367c0f1.5476310a@wiscmail.wisc.edu> <75d0c4467bf70.54763146@wiscmail.wisc.edu> <764081797ae74.54763182@wiscmail.wisc.edu> <738097f87db8b.547631bf@wiscmail.wisc.edu> <7610ea077f809.547631fb@wiscmail.wisc.edu> <7610d4b57d7f3.54763238@wiscmail.wisc.edu> <73a097287906a.54763274@wiscmail.wisc.edu> <75d0a9817d06a.547632b0@wiscmail.wisc.edu> <73a0ca317a13e.547632ed@wiscmail.wisc.edu> <7480865f7b7b8.54763329@wiscmail.wisc.edu> <7530a1987c5a2.54763366@wiscmail.wisc.edu> <73808f727cc19.547633a2@wiscmail.wisc.edu> <7610ec8f7b703.54763492@wiscmail.wisc.edu> <73a0a83f7fcfd.547634cf@wiscmail.wisc.edu> <73a0c2717c6e3.5476350b@wiscmail.wisc.edu> <7380d6b47eb58.54763548@wiscmail.wisc.edu> <75f0fa2b78673.5475e0ec@wiscmail.wisc.edu> <7480f3917963b.54763b35@wiscmail.wisc.edu> <75f0e0b37a192.54763b71@wiscmail.wisc.edu> <7530b88879d4c.5476451a@wiscmail.wisc.edu> Message-ID: <7380d3417e532.5475f879@wiscmail.wisc.edu> > Can you summarize that in a self-contained form for inclusion in the PEP? > > (That was a rhetorical question. :-) Sure. Is it on GitHub? ;D ijs From guido at python.org Thu Nov 27 01:33:54 2014 From: guido at python.org (Guido van Rossum) Date: Wed, 26 Nov 2014 16:33:54 -0800 Subject: [Python-Dev] Please reconsider PEP 479. In-Reply-To: References: <54724117.20408@hotpy.org> <547525B1.4090303@canterbury.ac.nz> <54758FB2.6080100@canterbury.ac.nz> Message-ID: The design just copies the code object with one flag set differently. Code objects are immutable but they can be copied (though the interface to do that is kind of hidden). On Wed, Nov 26, 2014 at 4:03 PM, Chris Angelico wrote: > On Thu, Nov 27, 2014 at 9:53 AM, Nick Coghlan wrote: > > The implicit stop decorator would then check the flags on the code object > > attached to the passed in function. If GENERATOR wasn't set, that would > be > > an immediate ValueError, while if EXPLICIT_STOP wasn't set, the generator > > function would be passed through unmodified. However, if EXPLICIT_STOP > *was* > > set, the generator function would be replaced by a *new* generator > function > > with a *new* code object, where the only change was to clear the > > EXPLICIT_STOP flag. > > Is it possible to replace the code object without replacing the > function? Imagine if you have multiple decorators, one of which > retains a reference to the function and then this one which replaces > it - the order of decoration would be critical. OTOH, I don't know > that anyone would retain references to __code__. > > ChrisA > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/guido%40python.org > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Thu Nov 27 01:38:40 2014 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 27 Nov 2014 11:38:40 +1100 Subject: [Python-Dev] Please reconsider PEP 479. In-Reply-To: References: <54724117.20408@hotpy.org> <547525B1.4090303@canterbury.ac.nz> <54758FB2.6080100@canterbury.ac.nz> Message-ID: On Thu, Nov 27, 2014 at 11:33 AM, Guido van Rossum wrote: > The design just copies the code object with one flag set differently. Code > objects are immutable but they can be copied (though the interface to do > that is kind of hidden). Yes, but the proposal as written spoke of replacing the generator *function*, which has broader consequences. If it's simply replacing the __code__ attribute of that function, it ought to be safe, I think? ChrisA From guido at python.org Thu Nov 27 01:50:15 2014 From: guido at python.org (Guido van Rossum) Date: Wed, 26 Nov 2014 16:50:15 -0800 Subject: [Python-Dev] Please reconsider PEP 479. In-Reply-To: References: <54724117.20408@hotpy.org> <547525B1.4090303@canterbury.ac.nz> <54758FB2.6080100@canterbury.ac.nz> Message-ID: No, that was a figure of speech. The proposed decorator returns a new function object that references a new code object. The original function and code object are unchanged. On Wed, Nov 26, 2014 at 4:38 PM, Chris Angelico wrote: > On Thu, Nov 27, 2014 at 11:33 AM, Guido van Rossum > wrote: > > The design just copies the code object with one flag set differently. > Code > > objects are immutable but they can be copied (though the interface to do > > that is kind of hidden). > > Yes, but the proposal as written spoke of replacing the generator > *function*, which has broader consequences. If it's simply replacing > the __code__ attribute of that function, it ought to be safe, I think? > > ChrisA > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/guido%40python.org > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Thu Nov 27 01:57:07 2014 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 27 Nov 2014 11:57:07 +1100 Subject: [Python-Dev] Please reconsider PEP 479. In-Reply-To: References: <54724117.20408@hotpy.org> <547525B1.4090303@canterbury.ac.nz> <54758FB2.6080100@canterbury.ac.nz> Message-ID: On Thu, Nov 27, 2014 at 11:50 AM, Guido van Rossum wrote: > No, that was a figure of speech. The proposed decorator returns a new > function object that references a new code object. The original function and > code object are unchanged. Then it has a potentially-confusing interaction with decorators like Flask's app.route(), which return the original function unchanged, but also save a reference to it elsewhere. The order of decoration determines the effect of the @hettinger decorator; there will be two functions around which are almost, but not entirely, identical, and it'd be very easy to not notice that you decorated in the wrong order. ChrisA From guido at python.org Thu Nov 27 02:01:05 2014 From: guido at python.org (Guido van Rossum) Date: Wed, 26 Nov 2014 17:01:05 -0800 Subject: [Python-Dev] Please reconsider PEP 479. In-Reply-To: References: <54724117.20408@hotpy.org> <547525B1.4090303@canterbury.ac.nz> <54758FB2.6080100@canterbury.ac.nz> Message-ID: Well, that's just a general problem with decorator ordering. On Wed, Nov 26, 2014 at 4:57 PM, Chris Angelico wrote: > On Thu, Nov 27, 2014 at 11:50 AM, Guido van Rossum > wrote: > > No, that was a figure of speech. The proposed decorator returns a new > > function object that references a new code object. The original function > and > > code object are unchanged. > > Then it has a potentially-confusing interaction with decorators like > Flask's app.route(), which return the original function unchanged, but > also save a reference to it elsewhere. The order of decoration > determines the effect of the @hettinger decorator; there will be two > functions around which are almost, but not entirely, identical, and > it'd be very easy to not notice that you decorated in the wrong order. > > ChrisA > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/guido%40python.org > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Thu Nov 27 02:07:27 2014 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 27 Nov 2014 12:07:27 +1100 Subject: [Python-Dev] Please reconsider PEP 479. In-Reply-To: References: <54724117.20408@hotpy.org> <547525B1.4090303@canterbury.ac.nz> <54758FB2.6080100@canterbury.ac.nz> Message-ID: On Thu, Nov 27, 2014 at 12:01 PM, Guido van Rossum wrote: > Well, that's just a general problem with decorator ordering. Indeed. I was hoping it could be avoided in this instance by just altering __code__ on an existing function, but if that's not possible, we fall back to what is, after all, a known and documented concern. ChrisA From guido at python.org Thu Nov 27 02:11:14 2014 From: guido at python.org (Guido van Rossum) Date: Wed, 26 Nov 2014 17:11:14 -0800 Subject: [Python-Dev] Please reconsider PEP 479. In-Reply-To: References: <54724117.20408@hotpy.org> <547525B1.4090303@canterbury.ac.nz> <54758FB2.6080100@canterbury.ac.nz> Message-ID: A decorator with a side effect *elsewhere* (like the route registrations) is acceptable; one with a side effect *on the decorated function* is questionable, and instead the decorator should behave "functionally", i.e. return a new object instead. On Wed, Nov 26, 2014 at 5:07 PM, Chris Angelico wrote: > On Thu, Nov 27, 2014 at 12:01 PM, Guido van Rossum > wrote: > > Well, that's just a general problem with decorator ordering. > > Indeed. I was hoping it could be avoided in this instance by just > altering __code__ on an existing function, but if that's not possible, > we fall back to what is, after all, a known and documented concern. > > ChrisA > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/guido%40python.org > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Thu Nov 27 02:12:41 2014 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 27 Nov 2014 12:12:41 +1100 Subject: [Python-Dev] Please reconsider PEP 479. In-Reply-To: References: <54724117.20408@hotpy.org> <547525B1.4090303@canterbury.ac.nz> <54758FB2.6080100@canterbury.ac.nz> Message-ID: On Thu, Nov 27, 2014 at 12:11 PM, Guido van Rossum wrote: > A decorator with a side effect *elsewhere* (like the route registrations) is > acceptable; one with a side effect *on the decorated function* is > questionable, and instead the decorator should behave "functionally", i.e. > return a new object instead. Okay! I learn something new every day. :) Did not know that... so it's a good thing I don't write decorators. I withdraw the suggestion. ChrisA From guido at python.org Thu Nov 27 02:15:58 2014 From: guido at python.org (Guido van Rossum) Date: Wed, 26 Nov 2014 17:15:58 -0800 Subject: [Python-Dev] Please reconsider PEP 479. In-Reply-To: References: <54724117.20408@hotpy.org> <547525B1.4090303@canterbury.ac.nz> <54758FB2.6080100@canterbury.ac.nz> Message-ID: On Wed, Nov 26, 2014 at 2:53 PM, Nick Coghlan wrote: > On 27 Nov 2014 06:35, "Guido van Rossum" wrote: > [...] > > > I think we can put a number to "much faster" now -- 150 nsec per > try/except. > > > > I have serious misgivings about that decorator though -- I'm not sure > how viable it is to pass a flag from the function object to the execution > (which takes the code object, which is immutable) and how other Python > implementations would do that. But I'm sure it can be done through sheer > willpower. I'd call it the @hettinger decorator in honor of the PEP's most > eloquent detractor. :-) > > I agree with everything you wrote in your reply, so I'll just elaborate a > bit on my proposed implementation for the decorator idea. > This remark is ambiguous -- how strongly do you feel that this decorator should be provided? (If so, it should be in the PEP.) (I'm snipping the rest of what you said, since I understand it: the flag on the code object even has a name in the PEP, it's REPLACE_STOPITERATION -- although I could imagine renaming it to GENERATOR_STOP to match the __future__.) -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Thu Nov 27 02:19:00 2014 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 27 Nov 2014 12:19:00 +1100 Subject: [Python-Dev] PEP 479: Change StopIteration handling inside generators In-Reply-To: <7380d3417e532.5475f879@wiscmail.wisc.edu> References: <20141125152024.GF2748@ando.pearwood.info> <762099ad85451.5474c008@wiscmail.wisc.edu> <75f0c659864dc.54746bd6@wiscmail.wisc.edu> <7670967d8a424.5475fe11@wiscmail.wisc.edu> <7740f9318c795.5475fe4d@wiscmail.wisc.edu> <76a091178e62b.5475fe89@wiscmail.wisc.edu> <76609b5c8b416.5475aa58@wiscmail.wisc.edu> <73a0d22c7b73e.54762ae0@wiscmail.wisc.edu> <7740e33d7f1e2.54762c0f@wiscmail.wisc.edu> <7630fc1c7e499.54762c4b@wiscmail.wisc.edu> <7480d5717827c.54762df2@wiscmail.wisc.edu> <7530bd45785c8.54762e2f@wiscmail.wisc.edu> <75f0ff337d57b.54762f5e@wiscmail.wisc.edu> <7530f3ad7f808.54762fd9@wiscmail.wisc.edu> <7480d3c87c41c.54763015@wiscmail.wisc.edu> <763082b77f3cf.54763052@wiscmail.wisc.edu> <73809d277d189.54763091@wiscmail.wisc.edu> <7640c5e878eae.547630cd@wiscmail.wisc.edu> <774091367c0f1.5476310a@wiscmail.wisc.edu> <75d0c4467bf70.54763146@wiscmail.wisc.edu> <764081797ae74.54763182@wiscmail.wisc.edu> <738097f87db8b.547631bf@wiscmail.wisc.edu> <7610ea077f809.547631fb@wiscmail.wisc.edu> <7610d4b57d7f3.54763238@wiscmail.wisc.edu> <73a097287906a.54763274@wiscmail.wisc.edu> <75d0a9817d06a.547632b0@wiscmail.wisc.edu> <73a0ca317a13e.547632ed@wiscmail.wisc.edu> <7480865f7b7b8.54763329@wiscmail.wisc.edu> <7530a1987c5a2.54763366@wiscmail.wisc.edu> <73808f727cc19.547633a2@wiscmail.wisc.edu> <7610ec8f7b703.54763492@wiscmail.wisc.edu> <73a0a83f7fcfd.547634cf@wiscmail.wisc.edu> <73a0c2717c6e3.5476350b@wiscmail.wisc.edu> <7380d6b47eb58.54763548@wiscmail.wisc.edu> <75f0fa2b78673.5475e0ec@wiscmail.wisc.edu> <7480f3917963b.54763b35@wiscmail.wisc.edu> <75f0e0b37a192.54763b71@wiscmail.wisc.edu> <7530b88879d4c.5476451a@wiscmail.wisc.edu> <7380d3417e532.5475f879@wiscmail.wisc.edu> Message-ID: On Thu, Nov 27, 2014 at 8:57 AM, Isaac Schwabacher wrote: >> Can you summarize that in a self-contained form for inclusion in the PEP? >> >> (That was a rhetorical question. :-) > > Sure. Is it on GitHub? ;D Thanks Isaac, I've incorporated your edits. https://raw.githubusercontent.com/Rosuav/GenStopIter/master/pep-0479.txt ChrisA From pmiscml at gmail.com Thu Nov 27 03:42:01 2014 From: pmiscml at gmail.com (Paul Sokolovsky) Date: Thu, 27 Nov 2014 04:42:01 +0200 Subject: [Python-Dev] Python 3.4.2 MSI fails to install and run on Windows For Devices Message-ID: <20141127044201.7f090414@x230> Hello, Following recent discussion of keeping Windows matters to the main dev list, I hope this message will be found useful. Please note that this is not a bug report, mostly a FYI. So, I tried to install Python 3.4.2 on an Intel Galileo board shipped with Windows For Devices. Actually, I don't know if that's correct name for this version of Windows, or rather it's Windows On Devices or Windows IoT. Either way it's essentially a headless Windows, with landing page at http://windowsondevices.com/ . So, I couldn't install python-3.4.2.msi, because that Windows comes without msiexec. Trying to install it via a redistributable (WindowsInstaller-KB893803-v2-x86.exe) fails without any error. At which point I started to think that it may be because running any GUI apps there is not supported. I then installed the MSI with wine (I'm using Linux host), uploaded the installed tree to the board via FTP, and also collected any dependent DLLs from the system folder. python.exe exits immediately in start, just as WindowsInstaller-KB893803-v2-x86.exe. Ironically, that board was gotten to test and better support Windows port of MicroPython. It starts up and executes quick manual command fine, and the reason I needed CPython there is to run MicroPython's testsuite which depends on it. As I mentioned, I don't consider this bug in CPython Windows build, it's rather bug in the discussed Windows version, and I'll try to figure out where to report it properly. But somehow I feel that resolving the underlying issue (running msiexec, [arbitrary] other applications) may take a while, and until then, Python is not usable on this board ;-I. So, until this issue is figured out, MicroPython already supports this embedded Windows version better than CPython ;-). -- Best regards, Paul mailto:pmiscml at gmail.com From ncoghlan at gmail.com Thu Nov 27 12:04:39 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 27 Nov 2014 21:04:39 +1000 Subject: [Python-Dev] Please reconsider PEP 479. In-Reply-To: References: <54724117.20408@hotpy.org> <547525B1.4090303@canterbury.ac.nz> <54758FB2.6080100@canterbury.ac.nz> Message-ID: On 27 November 2014 at 11:15, Guido van Rossum wrote: > On Wed, Nov 26, 2014 at 2:53 PM, Nick Coghlan wrote: >> >> On 27 Nov 2014 06:35, "Guido van Rossum" wrote: >> >> [...] >> >> > I think we can put a number to "much faster" now -- 150 nsec per >> > try/except. >> > >> > I have serious misgivings about that decorator though -- I'm not sure >> > how viable it is to pass a flag from the function object to the execution >> > (which takes the code object, which is immutable) and how other Python >> > implementations would do that. But I'm sure it can be done through sheer >> > willpower. I'd call it the @hettinger decorator in honor of the PEP's most >> > eloquent detractor. :-) >> >> I agree with everything you wrote in your reply, so I'll just elaborate a >> bit on my proposed implementation for the decorator idea. > > This remark is ambiguous -- how strongly do you feel that this decorator > should be provided? (If so, it should be in the PEP.) I think it makes sense to standardise it, but something like "itertools.allow_implicit_stop" would probably be better than having it as a builtin. (The only reason I suggested a builtin initially is because putting it in itertools didn't occur to me until later) Including the decorator provides a straightforward way to immediately start writing forward compatible code that's explicit about the fact it relies on the current StopIteration handling, without being excessively noisy relative to the status quo: # In a module with a generator that relies on the current behaviour from itertools import allow_implicit_stop @allow_implicit_stop def my_generator(): ... yield next(itr) ... In terms of code diffs to ensure forward compatibility, it's 1 import statement per affected module, and 1 decorator line per affected generator, rather than at least 3 lines (for try/except/return) plus indentation changes for each affected generator. That's a useful benefit when it comes to minimising the impact on version control code annotation, etc. If compatibility with older Python versions is needed, then you could put something like the following in a compatibility module: try: from itertools import allow_implicit_stop except ImportError: # Allowing implicit stops is the default in older versions def allow_implicit_stop(g): return g Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Thu Nov 27 12:25:13 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 27 Nov 2014 21:25:13 +1000 Subject: [Python-Dev] PEP 479: Change StopIteration handling inside generators In-Reply-To: References: <20141125152024.GF2748@ando.pearwood.info> <762099ad85451.5474c008@wiscmail.wisc.edu> <75f0c659864dc.54746bd6@wiscmail.wisc.edu> <7670967d8a424.5475fe11@wiscmail.wisc.edu> <7740f9318c795.5475fe4d@wiscmail.wisc.edu> <76a091178e62b.5475fe89@wiscmail.wisc.edu> <76609b5c8b416.5475aa58@wiscmail.wisc.edu> Message-ID: On 27 November 2014 at 09:50, Guido van Rossum wrote: > On Wed, Nov 26, 2014 at 3:15 PM, Nick Coghlan wrote: >> This is actually the second iteration of this bug: the original >> implementation *always* suppressed StopIteration. PJE caught that one before >> Python 2.5 was released, but we didn't notice that 3.3 had brought it back >> in a new, more subtle form :( >> >> It's worth noting that my "allow_implicit_stop" idea in the other thread >> wouldn't affect subgenerators - those would still convert StopIteration to >> RuntimeError unless explicitly silenced. > > You've lost me in this subthread. Am I right to conclude that the PEP change > doesn't cause problems for contextlib(*), but that the PEP change also > probably wouldn't have helped diagnose any contextlib bugs? I think the PEP 479 semantics would have made both bugs (the one PJE found in 2.5, and the newer one Isaac pointed out here) less cryptic, in that they would have caused RuntimeError to be raised, rather than silently consuming the StopIteration and continuing execution after the with statement body. With the new semantics, contextlib just needs to be updated to cope with the StopIteration -> RuntimeError conversion, and Isaac's "spurious success" bug will be fixed*. Without PEP 479, I believe my only recourse to systematically eliminate the risk of generator based context managers silently consuming StopIteration would be to implement the "StopIteration -> gen.close()" workaround, and that would be a backwards incompatible change in its own right. Cheers, Nick. P.S. *(This does mean I was wrong about allow_implicit_stop being useful to contextlib, but I still think the decorator is useful for cases where StopIteration is being used to terminate the generator on purpose) -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From jcea at jcea.es Thu Nov 27 13:28:40 2014 From: jcea at jcea.es (Jesus Cea) Date: Thu, 27 Nov 2014 13:28:40 +0100 Subject: [Python-Dev] Strange "help(int.__lt__)". Probably documentation bug Message-ID: <547718F8.9090102@jcea.es> http://bugs.python.org/issue20530#msg231584 -- Jes?s Cea Avi?n _/_/ _/_/_/ _/_/_/ jcea at jcea.es - http://www.jcea.es/ _/_/ _/_/ _/_/ _/_/ _/_/ Twitter: @jcea _/_/ _/_/ _/_/_/_/_/ jabber / xmpp:jcea at jabber.org _/_/ _/_/ _/_/ _/_/ _/_/ "Things are not so easy" _/_/ _/_/ _/_/ _/_/ _/_/ _/_/ "My name is Dump, Core Dump" _/_/_/ _/_/_/ _/_/ _/_/ "El amor es poner tu felicidad en la felicidad de otro" - Leibniz -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 473 bytes Desc: OpenPGP digital signature URL: From victor.stinner at gmail.com Thu Nov 27 13:41:19 2014 From: victor.stinner at gmail.com (Victor Stinner) Date: Thu, 27 Nov 2014 13:41:19 +0100 Subject: [Python-Dev] Strange "help(int.__lt__)". Probably documentation bug In-Reply-To: <547718F8.9090102@jcea.es> References: <547718F8.9090102@jcea.es> Message-ID: 2014-11-27 13:28 GMT+01:00 Jesus Cea : > http://bugs.python.org/issue20530#msg231584 Copy/paste of the message: Preparing a presentation about Python Magic methods I found something weird: (Python 3.4) """ >>> help(int.__lt__) Help on wrapper_descriptor: __lt__(self, value, /) <- THIS!! Return self References: <547718F8.9090102@jcea.es> Message-ID: 2014-11-27 13:41 GMT+01:00 Victor Stinner : > 2014-11-27 13:28 GMT+01:00 Jesus Cea : >> http://bugs.python.org/issue20530#msg231584 > > Copy/paste of the message: > > Preparing a presentation about Python Magic methods I found something > weird: (Python 3.4) > > """ >>>> help(int.__lt__) > Help on wrapper_descriptor: > > __lt__(self, value, /) <- THIS!! > Return self """ > > I am amused about the "/)" suffix in the signature. It happens to all > magic methods. From victor.stinner at gmail.com Thu Nov 27 13:42:35 2014 From: victor.stinner at gmail.com (Victor Stinner) Date: Thu, 27 Nov 2014 13:42:35 +0100 Subject: [Python-Dev] Strange "help(int.__lt__)". Probably documentation bug In-Reply-To: References: <547718F8.9090102@jcea.es> Message-ID: 2014-11-27 13:41 GMT+01:00 Victor Stinner : > I am amused about the "/)" suffix in the signature. It happens to all > magic methods. If I remember correctly, it means that the function does not accept keywords: >>> (3).__lt__(4) True >>> (3).__lt__(value=4) Traceback (most recent call last): File "", line 1, in TypeError: wrapper __lt__ doesn't take keyword arguments Victor From jcea at jcea.es Thu Nov 27 14:43:05 2014 From: jcea at jcea.es (Jesus Cea) Date: Thu, 27 Nov 2014 14:43:05 +0100 Subject: [Python-Dev] Strange "help(int.__lt__)". Probably documentation bug In-Reply-To: References: <547718F8.9090102@jcea.es> Message-ID: <54772A69.9090407@jcea.es> On 27/11/14 13:42, Victor Stinner wrote: > 2014-11-27 13:41 GMT+01:00 Victor Stinner : >> I am amused about the "/)" suffix in the signature. It happens to all >> magic methods. > > If I remember correctly, it means that the function does not accept keywords: I don't understand. Is that internal annotation for the clinic machinery?. -- Jes?s Cea Avi?n _/_/ _/_/_/ _/_/_/ jcea at jcea.es - http://www.jcea.es/ _/_/ _/_/ _/_/ _/_/ _/_/ Twitter: @jcea _/_/ _/_/ _/_/_/_/_/ jabber / xmpp:jcea at jabber.org _/_/ _/_/ _/_/ _/_/ _/_/ "Things are not so easy" _/_/ _/_/ _/_/ _/_/ _/_/ _/_/ "My name is Dump, Core Dump" _/_/_/ _/_/_/ _/_/ _/_/ "El amor es poner tu felicidad en la felicidad de otro" - Leibniz -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 473 bytes Desc: OpenPGP digital signature URL: From ncoghlan at gmail.com Thu Nov 27 15:11:39 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 28 Nov 2014 00:11:39 +1000 Subject: [Python-Dev] Strange "help(int.__lt__)". Probably documentation bug In-Reply-To: <54772A69.9090407@jcea.es> References: <547718F8.9090102@jcea.es> <54772A69.9090407@jcea.es> Message-ID: On 27 November 2014 at 23:43, Jesus Cea wrote: > On 27/11/14 13:42, Victor Stinner wrote: >> 2014-11-27 13:41 GMT+01:00 Victor Stinner : >>> I am amused about the "/)" suffix in the signature. It happens to all >>> magic methods. >> >> If I remember correctly, it means that the function does not accept keywords: > > I don't understand. Is that internal annotation for the clinic machinery?. See PEP 457 for the broader context: https://www.python.org/dev/peps/pep-0457/ The migration of pydoc (and other introspection APIs) to inspect.signature in Python 3.4 entailed having an unambiguous string representation of positional only parameters - that's the trailing '/' (which mirrors the corresponding syntax in the Argument Clinic DSL). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From guido at python.org Thu Nov 27 17:52:27 2014 From: guido at python.org (Guido van Rossum) Date: Thu, 27 Nov 2014 08:52:27 -0800 Subject: [Python-Dev] Please reconsider PEP 479. In-Reply-To: References: <54724117.20408@hotpy.org> <547525B1.4090303@canterbury.ac.nz> <54758FB2.6080100@canterbury.ac.nz> Message-ID: On Thu, Nov 27, 2014 at 3:04 AM, Nick Coghlan wrote: > On 27 November 2014 at 11:15, Guido van Rossum wrote: > > On Wed, Nov 26, 2014 at 2:53 PM, Nick Coghlan > wrote: > >> > >> On 27 Nov 2014 06:35, "Guido van Rossum" wrote: > >> > >> [...] > >> > >> > I think we can put a number to "much faster" now -- 150 nsec per > >> > try/except. > >> > > >> > I have serious misgivings about that decorator though -- I'm not sure > >> > how viable it is to pass a flag from the function object to the > execution > >> > (which takes the code object, which is immutable) and how other Python > >> > implementations would do that. But I'm sure it can be done through > sheer > >> > willpower. I'd call it the @hettinger decorator in honor of the PEP's > most > >> > eloquent detractor. :-) > >> > >> I agree with everything you wrote in your reply, so I'll just elaborate > a > >> bit on my proposed implementation for the decorator idea. > > > > This remark is ambiguous -- how strongly do you feel that this decorator > > should be provided? (If so, it should be in the PEP.) > > I think it makes sense to standardise it, but something like > "itertools.allow_implicit_stop" would probably be better than having > it as a builtin. (The only reason I suggested a builtin initially is > because putting it in itertools didn't occur to me until later) > > Including the decorator provides a straightforward way to immediately > start writing forward compatible code that's explicit about the fact > it relies on the current StopIteration handling, without being > excessively noisy relative to the status quo: > > # In a module with a generator that relies on the current behaviour > from itertools import allow_implicit_stop > > @allow_implicit_stop > def my_generator(): > ... > yield next(itr) > ... > > In terms of code diffs to ensure forward compatibility, it's 1 import > statement per affected module, and 1 decorator line per affected > generator, rather than at least 3 lines (for try/except/return) plus > indentation changes for each affected generator. That's a useful > benefit when it comes to minimising the impact on version control code > annotation, etc. > > If compatibility with older Python versions is needed, then you could > put something like the following in a compatibility module: > > try: > from itertools import allow_implicit_stop > except ImportError: > # Allowing implicit stops is the default in older versions > def allow_implicit_stop(g): > return g > I understand that @allow_import_stop represents a compromise, an attempt at calming the waves that PEP 479 has caused. But I still want to push back pretty hard on this idea. - It means we're forever stuck with two possible semantics for StopIteration raised in generators. - It complicates the implementation, because (presumably) a generator marked with @allow_stop_import should not cause a warning when a StopIteration bubbles out -- so we actually need another flag to silence the warning. - I don't actually know whether other Python implementations have the ability to copy code objects to change flags. - It actually introduces a new incompatibility, that has to be solved in every module that wants to use it (as you show above), whereas just putting try/except around unguarded next() calls is fully backwards compatible. - Its existence encourage people to use the decorator in favor of fixing their code properly. - The decorator is so subtle that it probably needs to be explained to everyone who encounters it (and wasn't involved in this PEP discussion). Because of this I would strongly advise against using it to "fix" the itertools examples in the docs; it's just too magical. (IIRC only 2 examples actually depend on this.) Let me also present another (minor) argument for PEP 479. Sometimes you want to take a piece of code presented as a generator and turn it into something else. You can usually do this pretty easily by e.g. replacing every "yield" by a call to print() or list.append(). But if there are any bare next() calls in the code you have to beware of those. If the code was originally written without relying on bare next(), the transformation would have been easier. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From victor.stinner at gmail.com Thu Nov 27 19:08:46 2014 From: victor.stinner at gmail.com (Victor Stinner) Date: Thu, 27 Nov 2014 19:08:46 +0100 Subject: [Python-Dev] PEP 479 and asyncio Message-ID: Hi, I'm trying to follow the discussion about the PEP 479 (Change StopIteration handling inside generators), but it's hard to read all messages. I'm concerned by trollius and asyncio which heavily rely on StopIteration. Trollius currently supports running asyncio coroutines: a trollius coroutine can executes an asyncio coroutine, and and asyncio coroutine can execute a trollius coroutine. I modified the Return class of Trollius to not inherit from StopIteration. All trollius tests pass on Python 3.3 except on one (which makes me happy, the test suite is wide enough to detect bugs ;-)): test_trollius_in_asyncio. This specific test executes an asyncio which executes a trollius coroutine. https://bitbucket.org/enovance/trollius/src/873d21ac0badec36835ed24d13e2aeda24f2dc64/tests/test_asyncio.py?at=trollius#cl-60 The problem is that an asyncio coroutine cannot execute a Trollius coroutine anymore: "yield from coro" raises a Return exception instead of simply "stopping" the generator and return the result (value passed to Return). I don't see how an asyncio coroutine calling "yield from trollius_coroutine" can handle the Return exception if it doesn't inherit from StopIteration. It means that I have to drop this feature in Python 3.5 (or later when the PEP 479 becomes effective)? I'm talking about the current behaviour of Python 3.3, I didn't try the PEP 479 (I don't know if an exception exists). Victor -------------- next part -------------- A non-text attachment was scrubbed... Name: asyncio_trollius.py Type: text/x-python Size: 1159 bytes Desc: not available URL: From guido at python.org Thu Nov 27 20:06:46 2014 From: guido at python.org (Guido van Rossum) Date: Thu, 27 Nov 2014 11:06:46 -0800 Subject: [Python-Dev] PEP 479 and asyncio In-Reply-To: References: Message-ID: On Thu, Nov 27, 2014 at 10:08 AM, Victor Stinner wrote: > I'm trying to follow the discussion about the PEP 479 (Change > StopIteration handling inside generators), but it's hard to read all > messages. I'm concerned by trollius and asyncio which heavily rely on > StopIteration. > > Trollius currently supports running asyncio coroutines: a trollius > coroutine can executes an asyncio coroutine, and and asyncio coroutine > can execute a trollius coroutine. > > I modified the Return class of Trollius to not inherit from > StopIteration. All trollius tests pass on Python 3.3 except on one > (which makes me happy, the test suite is wide enough to detect bugs > ;-)): test_trollius_in_asyncio. > > This specific test executes an asyncio which executes a trollius coroutine. > > https://bitbucket.org/enovance/trollius/src/873d21ac0badec36835ed24d13e2aeda24f2dc64/tests/test_asyncio.py?at=trollius#cl-60 > > The problem is that an asyncio coroutine cannot execute a Trollius > coroutine anymore: "yield from coro" raises a Return exception instead > of simply "stopping" the generator and return the result (value passed > to Return). > > I don't see how an asyncio coroutine calling "yield from > trollius_coroutine" can handle the Return exception if it doesn't > inherit from StopIteration. It means that I have to drop this feature > in Python 3.5 (or later when the PEP 479 becomes effective)? > > I'm talking about the current behaviour of Python 3.3, I didn't try > the PEP 479 (I don't know if an exception exists). > The issue here is that asyncio only interprets StopIteration as returning from the generator (with a possible value), while a Trollius coroutine must use "raise Return()" to specify a return value; this works as long as Return is a subclass of StopIteration, but PEP 479 will break this by replacing the StopIteration with RuntimeError. It's an interesting puzzle. The only way out I can think of is to have asyncio special-case the Return exception -- we could do that by defining a new exception (e.g. AlternateReturn) in asyncio that gets treated the same way as StopIteration, so that Trollius can inherit from AlternateReturn (if it exists). What do you think? -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From victor.stinner at gmail.com Thu Nov 27 22:54:32 2014 From: victor.stinner at gmail.com (Victor Stinner) Date: Thu, 27 Nov 2014 22:54:32 +0100 Subject: [Python-Dev] PEP 479 and asyncio In-Reply-To: References: Message-ID: 2014-11-27 20:06 GMT+01:00 Guido van Rossum : > The issue here is that asyncio only interprets StopIteration as returning > from the generator (with a possible value), I'm not sure that the issue is directly related to asyncio. trollius_coro() raises a StopIteration to return the result to caller. To caller is "result = yield from coro", it's not the complex Task._step() method. So it's pure Python, except if I missed something. > The only way out I can think of is to have asyncio special-case the Return > exception -- we could do that by defining a new exception (e.g. > AlternateReturn) in asyncio that gets treated the same way as StopIteration, > so that Trollius can inherit from AlternateReturn (if it exists). I don't see how it would work. Here is a simplified example of my issue. You need to modify all "yield from coro" to write instead "yield from catch_return(coro)", or I missed something important. --- PEP479 = True if not PEP479: # trollius: no need for catch_return() before the PEP 479 class Return(StopIteration): pass else: # PEP 479: need catch_return() class Return(Exception): def __init__(self, value): self.value = value def return_value(value): if 0: yield raise Return(value) def catch_return(gen): try: value = (yield from gen) except Return as exc: return exc.value def add_one(gen): value = (yield from gen) return value + 1 def consume_generator(gen): while True: try: next(gen) except StopIteration as exc: return exc.value gen1 = return_value(3) if PEP479: gen1 = catch_return(gen1) gen2 = add_one(gen1) print(consume_generator(gen2)) --- Victor From victor.stinner at gmail.com Thu Nov 27 23:09:20 2014 From: victor.stinner at gmail.com (Victor Stinner) Date: Thu, 27 Nov 2014 23:09:20 +0100 Subject: [Python-Dev] PEP 479 and asyncio In-Reply-To: References: Message-ID: 2014-11-27 22:54 GMT+01:00 Victor Stinner : > I don't see how it would work. If it cannot be fixed, would it make sense to allow trollius to continue to work as it currently works with something like "from __past__ import generator_dont_stop"? When I talked with a friend about the transition from Python 2 to Python 3, he asked me why there was not "from __past__ import division". He wants to add this to his code to not have to worry that a division may fail "somewhere" in his code. Maybe it would ease upgrades to newer versions of Python if we consider keeping the old behaviour for people who don't have time to port their old code (for no immediate benefit), but need to upgrade because newer OS only provide newer version of Python. (What is the cost of keeping the old behaviour: maintain the code and runtime overhead?) Victor From rosuav at gmail.com Thu Nov 27 23:32:21 2014 From: rosuav at gmail.com (Chris Angelico) Date: Fri, 28 Nov 2014 09:32:21 +1100 Subject: [Python-Dev] PEP 479 and asyncio In-Reply-To: References: Message-ID: On Fri, Nov 28, 2014 at 8:54 AM, Victor Stinner wrote: > def return_value(value): > if 0: > yield > raise Return(value) This is one known significant backward-incompatibility issue with this PEP - it'll be difficult to make this work on Python 2.7, where "return value" would be a problem, and 3.7, where "raise StopIteration" would be a problem. At present, I don't know of a solution to this. In 3.x-only code, you could simply use 'return value' directly; in 2.7 code, StopIteration doesn't seem to even *have* a .value attribute (and .args[0] has to be used instead). But I don't like the idea of a "from __past__" directive. It means backward-compatibility code has to be maintained through eternity (otherwise it just shifts the problem to "why did you remove my __past__ directive, I want a from __paster__ import division"), which means both the Python implementation code (every Python, not just CPython) needs to cope, *and* everyone who reads Python code needs to cope. For python-list, Stack Overflow, and other such coding help places, this means more questions to ask about a piece of code. For real-world usage, it means scanning back up to the top of the file every time you read something that's been affected by a __past__ directive. Plus, which __future__ directives need __past__ equivalents? Personally, I wouldn't bother making "from __past__ import lack_of_with_statement", but your friend is wanting "division", and I'm sure "print_statement" would be wanted... and, this is the one that'd split everyone and put the sides to war: "bytes_literals". Personally, I would want python-dev to say "There will NEVER be a from __past__ import bytes_literals directive", but there are going to be others who say "But my code would be so much cleaner AND faster if you do!", and IMO this is a good reason to avoid having any __past__ directives at all. ChrisA From ncoghlan at gmail.com Fri Nov 28 03:12:22 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 28 Nov 2014 12:12:22 +1000 Subject: [Python-Dev] Please reconsider PEP 479. In-Reply-To: References: <54724117.20408@hotpy.org> <547525B1.4090303@canterbury.ac.nz> <54758FB2.6080100@canterbury.ac.nz> Message-ID: On 28 November 2014 at 02:52, Guido van Rossum wrote: > On Thu, Nov 27, 2014 at 3:04 AM, Nick Coghlan wrote: >> If compatibility with older Python versions is needed, then you could >> put something like the following in a compatibility module: >> >> try: >> from itertools import allow_implicit_stop >> except ImportError: >> # Allowing implicit stops is the default in older versions >> def allow_implicit_stop(g): >> return g > > > I understand that @allow_import_stop represents a compromise, an attempt at > calming the waves that PEP 479 has caused. But I still want to push back > pretty hard on this idea. > > - It means we're forever stuck with two possible semantics for StopIteration > raised in generators. > > - It complicates the implementation, because (presumably) a generator marked > with @allow_stop_import should not cause a warning when a StopIteration > bubbles out -- so we actually need another flag to silence the warning. Ugh, you're right. I'd missed that :( > - I don't actually know whether other Python implementations have the > ability to copy code objects to change flags. I was originally thinking that implicitly catching the RuntimeError and converting it back to StopIteration could be an acceptable "worst case" implementation, but I subsequently realised that interacts differently with yield from than the status quo does. > - It actually introduces a new incompatibility, that has to be solved in > every module that wants to use it (as you show above), whereas just putting > try/except around unguarded next() calls is fully backwards compatible. > > - Its existence encourage people to use the decorator in favor of fixing > their code properly. > > - The decorator is so subtle that it probably needs to be explained to > everyone who encounters it (and wasn't involved in this PEP discussion). > Because of this I would strongly advise against using it to "fix" the > itertools examples in the docs; it's just too magical. (IIRC only 2 examples > actually depend on this.) Yeah, if not for the status quo, there's no way I'd have suggested it at all. As it is, you've persuaded me that preserving this capability indefinitely at the eval loop level isn't worth the extra complexity (in particular, I'd missed the "add yet another flag to suppress the warning" issue). So now I'm wondering if the peephole optimiser could be updated to pick up the "except -> return" idiom... > Let me also present another (minor) argument for PEP 479. Sometimes you want > to take a piece of code presented as a generator and turn it into something > else. You can usually do this pretty easily by e.g. replacing every "yield" > by a call to print() or list.append(). But if there are any bare next() > calls in the code you have to beware of those. If the code was originally > written without relying on bare next(), the transformation would have been > easier. +1 The scenario you describe there strikes me as the statement level equivalent of the behavioural discrepancies between calling next() in a generator expression vs doing it in any other kind of comprehension. In the function definition case, once the "yield" is removed from elsewhere in the function (so its no longer a generator), it changes the semantics of any unguarded next() calls. That's the kind of side effect that's pretty easy for both automated testing and code review to miss. Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Fri Nov 28 03:49:47 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 28 Nov 2014 12:49:47 +1000 Subject: [Python-Dev] PEP 479 and asyncio In-Reply-To: References: Message-ID: On 28 November 2014 at 08:09, Victor Stinner wrote: > 2014-11-27 22:54 GMT+01:00 Victor Stinner : >> I don't see how it would work. > > If it cannot be fixed, would it make sense to allow trollius to > continue to work as it currently works with something like "from > __past__ import generator_dont_stop"? I think between contextlib and Trollius, the case is starting to be made for raising an UnhandledStopIteration subclass of RuntimeError, rather than a generic RuntimeError. We have at least two known cases now where code that works with generators-as-coroutines has a valid reason for wanting to distinguish "arbitrary runtime error" from "unhandled StopIteration exception". While catching RuntimeError and looking for StopIteration in __cause__ *works*, it feels messier and harder to explain than just naming the concept by giving it a dedicated exception type. Trollius would still need an adapter to be called from asyncio, though. Something like: def implicit_stop(g): try: yield from g except UnhandledStopIteration as exc: return exc.__cause__.value Then Victor's example would become: class Return(StopIteration): pass def return_value(value): if 0: yield raise Return(value) def add_one(gen): value = (yield from gen) return value + 1 def consume_generator(gen): while True: try: next(gen) except StopIteration as exc: return exc.value gen1 = return_value(3) if PEP479: gen1 = implicit_stop(gen1) gen2 = add_one(gen1) print(consume_generator(gen2)) > When I talked with a friend about the transition from Python 2 to > Python 3, he asked me why there was not "from __past__ import > division". He wants to add this to his code to not have to worry that > a division may fail "somewhere" in his code. > > Maybe it would ease upgrades to newer versions of Python if we > consider keeping the old behaviour for people who don't have time to > port their old code (for no immediate benefit), but need to upgrade > because newer OS only provide newer version of Python. > > (What is the cost of keeping the old behaviour: maintain the code and > runtime overhead?) The main problem with *never* deprecating anything is an ever-increasing cognitive burden in learning the language, as well as losing the ability to read code in isolation without knowing what flags are in effect. Currently, folks that only work in Python 3 don't need to know how division worked in Python 2, or that print was ever a statement, etc. If those old behaviours could be selectively turned back on, then everyone would still need to learn them, and you couldn't review code in isolation any more: there may be a __past__ import at the top of the module making it do something different. If organisations really want to let their code bitrot (and stay on the treadmill of big expensive high risk updates every decade or so), they can, but they have to do it by running on old versions of Python as well - that gives maintainers a clear understanding that if they want to understand the code, they have to know how Python X.Y worked, rather than being able to assume modern Python. Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From greg.ewing at canterbury.ac.nz Fri Nov 28 10:12:05 2014 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Fri, 28 Nov 2014 22:12:05 +1300 Subject: [Python-Dev] PEP 479 and asyncio In-Reply-To: References: Message-ID: <54783C65.8040301@canterbury.ac.nz> Guido van Rossum wrote: > The issue here is that asyncio only interprets StopIteration as > returning from the generator (with a possible value), while a Trollius > coroutine must use "raise Return()" to specify a return value; > this works as long as Return is a subclass of StopIteration, but PEP 479 > will break this by replacing the StopIteration with RuntimeError. I don't understand. If I'm interpreting PEP 479 correctly, in 'x = yield from foo', a StopIteration raised by foo.__next__() doesn't get turned into a RuntimeError; rather it just stops the sub-iteration as usual and its value attribute gets assigned to x. As long as a Trollius coroutine behaves like something implementing the iterator protocol, it should continue to work fine with Return as a subclass of StopIteration. Or is there something non-obvious about Trollius that I'm missing? -- Greg From victor.stinner at gmail.com Fri Nov 28 10:18:54 2014 From: victor.stinner at gmail.com (Victor Stinner) Date: Fri, 28 Nov 2014 10:18:54 +0100 Subject: [Python-Dev] PEP 479 and asyncio In-Reply-To: <54783C65.8040301@canterbury.ac.nz> References: <54783C65.8040301@canterbury.ac.nz> Message-ID: 2014-11-28 10:12 GMT+01:00 Greg Ewing : > I don't understand. If I'm interpreting PEP 479 correctly, in > 'x = yield from foo', a StopIteration raised by foo.__next__() > doesn't get turned into a RuntimeError The Trollius coroutine uses "raise Return(value)" which is basically a "raise StopIteraton(value)", and this is forbidden by the PEP 479. With the PEP 479, the StopIteration is replaced with a RuntimeError. Victor From rosuav at gmail.com Fri Nov 28 10:27:53 2014 From: rosuav at gmail.com (Chris Angelico) Date: Fri, 28 Nov 2014 20:27:53 +1100 Subject: [Python-Dev] PEP 479 and asyncio In-Reply-To: References: <54783C65.8040301@canterbury.ac.nz> Message-ID: On Fri, Nov 28, 2014 at 8:18 PM, Victor Stinner wrote: > 2014-11-28 10:12 GMT+01:00 Greg Ewing : >> I don't understand. If I'm interpreting PEP 479 correctly, in >> 'x = yield from foo', a StopIteration raised by foo.__next__() >> doesn't get turned into a RuntimeError > > The Trollius coroutine uses "raise Return(value)" which is basically a > "raise StopIteraton(value)", and this is forbidden by the PEP 479. > With the PEP 479, the StopIteration is replaced with a RuntimeError. The question, I guess, is: Why can't it be translated into "return value"? One answer is: Because that's not legal in Python 2.7. And I can't respond to that answer, unfortunately. That's the one major backward compat issue. (Another answer may be "Because it would require changes to many intermediate generators, not all of which are under our control". If that's the issue, then it'll simply be a matter of telling people "When you upgrade to Python 3.6, you will start to see warnings unless you make this change".) ChrisA From victor.stinner at gmail.com Fri Nov 28 11:02:10 2014 From: victor.stinner at gmail.com (Victor Stinner) Date: Fri, 28 Nov 2014 11:02:10 +0100 Subject: [Python-Dev] PEP 479 and asyncio In-Reply-To: References: Message-ID: 2014-11-28 3:49 GMT+01:00 Nick Coghlan : > I think between contextlib and Trollius, the case is starting to be > made for raising an UnhandledStopIteration subclass of RuntimeError, > rather than a generic RuntimeError. I modified Trollius to test such idea: * Return inherits from Exception (not from StopIteration) * on Python 3, @trollius.coroutine wraps the coroutine to catch Runtimerror: if the exc.__context__ is a StopIteration, return exc.__context__.value The test suite pass with such additional coroutine wrapper on Python 3.5 patched with pep479.patch (and unpatched Python 3.3). So yes, it may help to have a new specialized exception, even if "it works" with RuntimeError. The drawback is that a new layer would make trollius even slower. Victor From olemis at gmail.com Fri Nov 28 16:15:16 2014 From: olemis at gmail.com (Olemis Lang) Date: Fri, 28 Nov 2014 10:15:16 -0500 Subject: [Python-Dev] PEP 479 and asyncio In-Reply-To: References: Message-ID: off-topic , not about asyncio but related to the PEP and other things been discussed in this thread On 11/28/14, Victor Stinner wrote: > 2014-11-28 3:49 GMT+01:00 Nick Coghlan : > [...] > > So yes, it may help to have a new specialized exception, even if "it > works" with RuntimeError. > This is somehow the situation I tried to explain in another thread about PEP 479 (though I did not use the right words) and will be a very common situation in practice . > The drawback is that a new layer would make trollius even slower. > e.g. in a (private) library I wrote for a company that's basically about composition of generators there is a situation similar to what Victor explained in this thread . I mostly would have to end-up doing one of a couple of things try: ... except RuntimeError: return which over-complicates function definition and introduces a long chain of (redundant) exception handling code just to end up raising StopIteration once again (i.e. poor performance) or ... # decorate functions in the public API # ... may be improved but you get the idea def myown_stopiter(f) def wrapper(*args, **kwargs): ... try: ... except RuntimeError as exc: if isinstance(exc.args[0], StopIteration): raise StopIteration # exc.args[0] ? else: raise ... return wrapper which is actually a re-implementation of exception matching itself Otherwise ... {{{#!py # in generator definition # rather than natural syntax for defining sequence logic raise MyOwnException(...) # decorate functions in the public API # ... may be improved but you get the idea def myown_stopiter(f) def wrapper(*args, **kwargs): ... try: ... except MyOwnException: raise StopIteration ... return wrapper }}} In the two las cases the library ends up having two functions , the one that allows (MyOwnException | RuntimeError) to bubble up (only used for defining compositions) , and the one that translates the exception (which *should* not be used for compositions, even if it will work, because of performance penalties) ... thus leading to further complications at API level ... Built-in behavior consisting in raising a subclass of RuntimeError is a much better approach similar to the second case mentioned above . This might definitely help to make less painful the process of rewriting things all over to cope with incompatibilities caused by PEP 479 , but afaict performance issues will be there for a while . -- Regards, Olemis - @olemislc Apache(tm) Bloodhound contributor http://issues.apache.org/bloodhound http://blood-hound.net Blog ES: http://simelo-es.blogspot.com/ Blog EN: http://simelo-en.blogspot.com/ Featured article: From olemis at gmail.com Fri Nov 28 16:21:51 2014 From: olemis at gmail.com (Olemis Lang) Date: Fri, 28 Nov 2014 10:21:51 -0500 Subject: [Python-Dev] PEP 479 and asyncio In-Reply-To: References: Message-ID: correction ... On 11/28/14, Olemis Lang wrote: > > try: > ... > except RuntimeError: > return > ... should be {{{#!py # inside generator function body try: ... except StopIteration: return }}} [...] -- Regards, Olemis - @olemislc Apache(tm) Bloodhound contributor http://issues.apache.org/bloodhound http://blood-hound.net Blog ES: http://simelo-es.blogspot.com/ Blog EN: http://simelo-en.blogspot.com/ Featured article: From status at bugs.python.org Fri Nov 28 18:08:16 2014 From: status at bugs.python.org (Python tracker) Date: Fri, 28 Nov 2014 18:08:16 +0100 (CET) Subject: [Python-Dev] Summary of Python tracker Issues Message-ID: <20141128170816.30F9E5613F@psf.upfronthosting.co.za> ACTIVITY SUMMARY (2014-11-21 - 2014-11-28) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 4668 (+10) closed 30056 (+42) total 34724 (+52) Open issues with patches: 2176 Issues opened (35) ================== #22676: _pickle's whichmodule() is slow http://bugs.python.org/issue22676 reopened by zach.ware #22685: memory leak: no transport for pipes by create_subprocess_exec/ http://bugs.python.org/issue22685 reopened by koobs #22912: urlretreive locks up in 2.7.8 http://bugs.python.org/issue22912 opened by TaylorSMarks #22914: Rewrite of Python 2/3 porting HOWTO http://bugs.python.org/issue22914 opened by brett.cannon #22918: Doc for __iter__ makes inexact comment about dict.__iter__ http://bugs.python.org/issue22918 opened by eric.araujo #22919: Update PCBuild for VS 2015 http://bugs.python.org/issue22919 opened by steve.dower #22922: asyncio: call_soon() should raise an exception if the event lo http://bugs.python.org/issue22922 opened by haypo #22923: No prompt for "display all X possibilities" on completion-enab http://bugs.python.org/issue22923 opened by yoha #22924: Use of deprecated cgi.escape http://bugs.python.org/issue22924 opened by serhiy.storchaka #22926: asyncio: raise an exception when called from the wrong thread http://bugs.python.org/issue22926 opened by haypo #22928: HTTP header injection in urrlib2/urllib/httplib/http.client http://bugs.python.org/issue22928 opened by Guido #22931: cookies with square brackets in value http://bugs.python.org/issue22931 opened by Waldemar.Parzonka #22932: email.utils.formatdate uses unreliable time.timezone constant http://bugs.python.org/issue22932 opened by mitya57 #22933: Misleading sentence in doc for shutil.move http://bugs.python.org/issue22933 opened by newbie #22935: Disabling SSLv3 support http://bugs.python.org/issue22935 opened by kroeckx #22936: traceback module has no way to show locals http://bugs.python.org/issue22936 opened by rbcollins #22937: unittest errors can't show locals http://bugs.python.org/issue22937 opened by rbcollins #22939: integer overflow in iterator object http://bugs.python.org/issue22939 opened by hakril #22941: IPv4Interface arithmetic changes subnet mask http://bugs.python.org/issue22941 opened by kwi.dk #22942: Language Reference - optional comma http://bugs.python.org/issue22942 opened by jordan #22943: bsddb: test_queue fails on Windows http://bugs.python.org/issue22943 opened by benjamin.peterson #22945: Ctypes inconsistent between Linux and OS X http://bugs.python.org/issue22945 opened by Daniel.Standage #22946: urllib gives incorrect url after open when using HTTPS http://bugs.python.org/issue22946 opened by John.McKay #22947: Enable 'imageop' - "Multimedia Srvices Feature module" for 64- http://bugs.python.org/issue22947 opened by pankaj.s01 #22949: fnmatch.translate doesn't add ^ at the beginning http://bugs.python.org/issue22949 opened by mstol #22951: unexpected return from float.__repr__() for inf, -inf, nan http://bugs.python.org/issue22951 opened by jaebae17 #22952: multiprocessing doc introduction not in affirmative tone http://bugs.python.org/issue22952 opened by davin #22953: Windows installer configures system PATH also when installing http://bugs.python.org/issue22953 opened by pekka.klarck #22955: Pickling of methodcaller and attrgetter http://bugs.python.org/issue22955 opened by Antony.Lee #22956: Improved support for prepared SQL statements http://bugs.python.org/issue22956 opened by elfring #22958: Constructors of weakref mapping classes don't accept "self" an http://bugs.python.org/issue22958 opened by serhiy.storchaka #22959: http.client.HTTPSConnection checks hostname when SSL context h http://bugs.python.org/issue22959 opened by zodalahtathi #22960: xmlrpc.client.ServerProxy() should accept a custom SSL context http://bugs.python.org/issue22960 opened by zodalahtathi #22961: ctypes.WinError & OSError http://bugs.python.org/issue22961 opened by simonzack #22962: ipaddress: Add optional prefixlen argument to ip_interface and http://bugs.python.org/issue22962 opened by Gary.van.der.Merwe Most recent 15 issues with no replies (15) ========================================== #22962: ipaddress: Add optional prefixlen argument to ip_interface and http://bugs.python.org/issue22962 #22960: xmlrpc.client.ServerProxy() should accept a custom SSL context http://bugs.python.org/issue22960 #22959: http.client.HTTPSConnection checks hostname when SSL context h http://bugs.python.org/issue22959 #22958: Constructors of weakref mapping classes don't accept "self" an http://bugs.python.org/issue22958 #22956: Improved support for prepared SQL statements http://bugs.python.org/issue22956 #22947: Enable 'imageop' - "Multimedia Srvices Feature module" for 64- http://bugs.python.org/issue22947 #22942: Language Reference - optional comma http://bugs.python.org/issue22942 #22931: cookies with square brackets in value http://bugs.python.org/issue22931 #22928: HTTP header injection in urrlib2/urllib/httplib/http.client http://bugs.python.org/issue22928 #22909: [argparse] Using parse_known_args, unknown arg with space in v http://bugs.python.org/issue22909 #22907: Misc/python-config.sh.in: ensure sed invocations only match be http://bugs.python.org/issue22907 #22893: Idle: __future__ does not work in startup code. http://bugs.python.org/issue22893 #22890: StringIO.StringIO pickled in 2.7 is not unpickleable on 3.x http://bugs.python.org/issue22890 #22886: TestProgram leaves defaultTestLoader.errors dirty http://bugs.python.org/issue22886 #22885: Arbitrary code execution vulnerability due to unchecked eval() http://bugs.python.org/issue22885 Most recent 15 issues waiting for review (15) ============================================= #22952: multiprocessing doc introduction not in affirmative tone http://bugs.python.org/issue22952 #22947: Enable 'imageop' - "Multimedia Srvices Feature module" for 64- http://bugs.python.org/issue22947 #22946: urllib gives incorrect url after open when using HTTPS http://bugs.python.org/issue22946 #22943: bsddb: test_queue fails on Windows http://bugs.python.org/issue22943 #22941: IPv4Interface arithmetic changes subnet mask http://bugs.python.org/issue22941 #22935: Disabling SSLv3 support http://bugs.python.org/issue22935 #22932: email.utils.formatdate uses unreliable time.timezone constant http://bugs.python.org/issue22932 #22928: HTTP header injection in urrlib2/urllib/httplib/http.client http://bugs.python.org/issue22928 #22926: asyncio: raise an exception when called from the wrong thread http://bugs.python.org/issue22926 #22924: Use of deprecated cgi.escape http://bugs.python.org/issue22924 #22922: asyncio: call_soon() should raise an exception if the event lo http://bugs.python.org/issue22922 #22919: Update PCBuild for VS 2015 http://bugs.python.org/issue22919 #22914: Rewrite of Python 2/3 porting HOWTO http://bugs.python.org/issue22914 #22908: ZipExtFile in zipfile can be seekable http://bugs.python.org/issue22908 #22907: Misc/python-config.sh.in: ensure sed invocations only match be http://bugs.python.org/issue22907 Top 10 most discussed issues (10) ================================= #21356: Support LibreSSL (instead of OpenSSL): make RAND_egd optional http://bugs.python.org/issue21356 12 msgs #22638: ssl module: the SSLv3 protocol is vulnerable ("POODLE" attack) http://bugs.python.org/issue22638 11 msgs #22685: memory leak: no transport for pipes by create_subprocess_exec/ http://bugs.python.org/issue22685 9 msgs #22906: PEP 479: Change StopIteration handling inside generators http://bugs.python.org/issue22906 9 msgs #22898: segfault during shutdown attempting to log ResourceWarning http://bugs.python.org/issue22898 8 msgs #22919: Update PCBuild for VS 2015 http://bugs.python.org/issue22919 8 msgs #22524: PEP 471 implementation: os.scandir() directory scanning functi http://bugs.python.org/issue22524 6 msgs #22932: email.utils.formatdate uses unreliable time.timezone constant http://bugs.python.org/issue22932 6 msgs #22609: Constructors of some mapping classes don't accept `self` keywo http://bugs.python.org/issue22609 5 msgs #22922: asyncio: call_soon() should raise an exception if the event lo http://bugs.python.org/issue22922 5 msgs Issues closed (39) ================== #1962: ctypes feature request: Automatic type conversion of input arg http://bugs.python.org/issue1962 closed by amaury.forgeotdarc #4887: environment inspection and manipulation API is buggy, inconsis http://bugs.python.org/issue4887 closed by amaury.forgeotdarc #7255: "Default" word boundaries for Unicode data? http://bugs.python.org/issue7255 closed by amaury.forgeotdarc #13212: json library is decoding/encoding when it should not http://bugs.python.org/issue13212 closed by serhiy.storchaka #16056: shadowed test names in std lib regression tests http://bugs.python.org/issue16056 closed by berker.peksag #16561: bdist_wininst installers don't use UAC, then crash http://bugs.python.org/issue16561 closed by python-dev #18905: pydoc -p 0 says the server is available at localhost:0 http://bugs.python.org/issue18905 closed by serhiy.storchaka #19676: Add the "namereplace" error handler http://bugs.python.org/issue19676 closed by serhiy.storchaka #19720: suppress context for some exceptions in importlib? http://bugs.python.org/issue19720 closed by serhiy.storchaka #20002: Cleanup and microoptimize pathlib http://bugs.python.org/issue20002 closed by serhiy.storchaka #20351: Add doc examples for DictReader and DictWriter http://bugs.python.org/issue20351 closed by berker.peksag #21280: shutil.make_archive() with default "root_dir" parameter raises http://bugs.python.org/issue21280 closed by serhiy.storchaka #21514: update json module docs in light of RFC 7159 & ECMA-404 http://bugs.python.org/issue21514 closed by berker.peksag #22314: pydoc.py: TypeError with a $LINES defined to anything http://bugs.python.org/issue22314 closed by serhiy.storchaka #22348: Documentation of asyncio.StreamWriter.drain() http://bugs.python.org/issue22348 closed by python-dev #22417: PEP 476: verify HTTPS certificates by default http://bugs.python.org/issue22417 closed by benjamin.peterson #22788: allow logging.handlers.HTTPHandler to take an SSLContext http://bugs.python.org/issue22788 closed by python-dev #22841: Avoid to use coroutine with add_signal_handler() http://bugs.python.org/issue22841 closed by haypo #22894: unittest.TestCase.subTest causes all subsequent tests to be sk http://bugs.python.org/issue22894 closed by pitrou #22901: test.test_uuid.test_find_mac fails because of `ifconfig` depre http://bugs.python.org/issue22901 closed by bru #22911: Segfault on recursive itertools.chain.from_iterable http://bugs.python.org/issue22911 closed by serhiy.storchaka #22913: Rewrite of Python 2/3 porting HOWTO http://bugs.python.org/issue22913 closed by brett.cannon #22915: sax.parser cannot get xml data from a subprocess pipe http://bugs.python.org/issue22915 closed by serhiy.storchaka #22916: Interpreter segfault on attempted tab complete http://bugs.python.org/issue22916 closed by r.david.murray #22917: Calculating problem http://bugs.python.org/issue22917 closed by SilentGhost #22920: Crash with itertools.chain.from_iterable http://bugs.python.org/issue22920 closed by ethan.furman #22921: SSLContext's check_hostname needlessly intertwined with SNI http://bugs.python.org/issue22921 closed by python-dev #22925: Backporting suppress context manager to 2.7 http://bugs.python.org/issue22925 closed by berker.peksag #22927: Expose an SSLContext parameter on urllib APIs http://bugs.python.org/issue22927 closed by python-dev #22929: cp874 encoding almost empty http://bugs.python.org/issue22929 closed by era #22930: Value of time.timezone is sometimes wrong http://bugs.python.org/issue22930 closed by r.david.murray #22934: Python/importlib.h is now generated by Programs/_freeze_import http://bugs.python.org/issue22934 closed by berker.peksag #22938: assignment operator += does not behave same as a = a + b for L http://bugs.python.org/issue22938 closed by steve.dower #22940: readline does not offer partial saves http://bugs.python.org/issue22940 closed by python-dev #22944: Python argument tuple unpacking http://bugs.python.org/issue22944 closed by benjamin.peterson #22948: Integer type and __add__ http://bugs.python.org/issue22948 closed by berker.peksag #22950: ASLR and DEP protection http://bugs.python.org/issue22950 closed by ned.deily #22954: Logically Dead Code http://bugs.python.org/issue22954 closed by python-dev #22957: Multi-index Containers Library http://bugs.python.org/issue22957 closed by eric.smith From demianbrecht at gmail.com Fri Nov 28 18:34:52 2014 From: demianbrecht at gmail.com (Demian Brecht) Date: Fri, 28 Nov 2014 09:34:52 -0800 Subject: [Python-Dev] Move selected documentation repos to PSF BitBucket account? In-Reply-To: References: <3CC02307-6109-4F20-A815-BDA7CFE16AC2@stufft.io> <013F5AD9-9C9F-4E2E-A750-1C758B66B675@stufft.io> <5A9CA007-3832-4931-8F05-06BD62BE4A5E@stufft.io> <35957465-A878-4864-9473-9F68B9556387@stufft.io> <5473E296.8090306@stoneleaf.us> <5473EB65.2070106@stoneleaf.us> <1A433321-49C7-461B-ACA0-D0EA7548AA21@stufft.io> Message-ID: On Tue, Nov 25, 2014 at 6:52 AM, Brett Cannon wrote: > > I suspect if we make sure we add Bitbucket and GitHub login support to the issue tracker then that would help go a fair distance to helping with the GitHub pull of reach (and if we make it so people can simply paste in their fork's URL into the issue tracker and we simply grab a new patch for review that would go even farther). Chiming in horribly late, so hopefully this hasn't already been mentioned (I've only loosely been following this thread). In addition to the login support (I'm not sold on how much that would help the reach), I think it would be really beneficial to have some documentation on either emulating git-style workflow in hg or detailing a git fork workflow while working on multiple patches concurrently and keeping master in sync with hg default (or perhaps even both). I primarily use git for development. Having little or no effort to context switch to work on CPython in any capacity (PEPs, code, etc) would be hugely beneficial for me. Having a well defined workflow in the docs (perhaps alongside "Lifecycle of a patch"?) would have significantly lowered the initial barrier of entry for me. Given the amount of time I put into that initially, I can only imagine how many people it's entirely turned away from contributing. I'd definitely be interested in contributing documentation around this (I've written up something similar here http://demianbrecht.github.io/vcs/2014/07/31/from-git-to-hg/) if others feel that it would be valuable. IMHO, you don't want to limit submissions due to the tech stack (one of the arguments I've seen for not moving to Github was quality of submissions). This will also limit high quality work from those who simply don't have time to adopt new tech and workflows when they're not being paid to do so. I have no strong opinion of where and how the official repos are stored so long as I can work on them and contribute to them in the way that's most efficient for me. I imagine that statement would also hold true for most. From guido at python.org Fri Nov 28 18:37:43 2014 From: guido at python.org (Guido van Rossum) Date: Fri, 28 Nov 2014 09:37:43 -0800 Subject: [Python-Dev] PEP 479 and asyncio In-Reply-To: References: Message-ID: @Victor: I'm glad you found a work-around. Maybe you can let your users control it with a flag? It is often true that straddling code pays a performance cost. Hopefully the slight performance dip might be an incentive for people to start thinking about porting to asyncio. @Olemis: You never showed examples of how your code would be used, so it's hard to understand what you're trying to do and how PEP 479 affects you. On Fri, Nov 28, 2014 at 7:21 AM, Olemis Lang wrote: > correction ... > > On 11/28/14, Olemis Lang wrote: > > > > try: > > ... > > except RuntimeError: > > return > > > > ... should be > > {{{#!py > > # inside generator function body > > try: > ... > except StopIteration: > return > }}} > > [...] > > -- > Regards, > > Olemis - @olemislc > > Apache(tm) Bloodhound contributor > http://issues.apache.org/bloodhound > http://blood-hound.net > > Blog ES: http://simelo-es.blogspot.com/ > Blog EN: http://simelo-en.blogspot.com/ > > Featured article: > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/guido%40python.org > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From raymond.hettinger at gmail.com Fri Nov 28 21:01:42 2014 From: raymond.hettinger at gmail.com (Raymond Hettinger) Date: Fri, 28 Nov 2014 12:01:42 -0800 Subject: [Python-Dev] Please reconsider PEP 479. In-Reply-To: References: <54724117.20408@hotpy.org> <547525B1.4090303@canterbury.ac.nz> <54758FB2.6080100@canterbury.ac.nz> Message-ID: <492E98A9-0206-44CB-A380-31C2AE12B336@gmail.com> > On Nov 27, 2014, at 8:52 AM, Guido van Rossum wrote: > > I understand that @allow_import_stop represents a compromise, an attempt at calming the waves that PEP 479 has caused. But I still want to push back pretty hard on this idea. > > - It means we're forever stuck with two possible semantics for StopIteration raised in generators. > > - It complicates the implementation, because (presumably) a generator marked with @allow_stop_import should not cause a warning when a StopIteration bubbles out -- so we actually need another flag to silence the warning. > > - I don't actually know whether other Python implementations have the ability to copy code objects to change flags. > > - It actually introduces a new incompatibility, that has to be solved in every module that wants to use it (as you show above), whereas just putting try/except around unguarded next() calls is fully backwards compatible. > > - Its existence encourage people to use the decorator in favor of fixing their code properly. > > - The decorator is so subtle that it probably needs to be explained to everyone who encounters it (and wasn't involved in this PEP discussion). Because of this I would strongly advise against using it to "fix" the itertools examples in the docs; it's just too magical. (IIRC only 2 examples actually depend on this.) I concur. PEP 479 fixes are trivially easy to do without a decorator. After Guido pronounced on the PEP, I fixed-up several parts of the standard library in just a few minutes. It's not hard. https://mail.python.org/pipermail/python-checkins/2014-November/133252.html https://mail.python.org/pipermail/python-checkins/2014-November/133253.html Also, I'm submitting a 479 patch to the Django project so we won't have to worry about this one. I recommend that everyone just accept that the PEP is a done deal and stop adding complexity or work-arounds. We have a lot of things going for us on this one: 1) the affected code isn't common-place (mostly in producer/consumer middleware tools created by tool makers rather than by tool users), 2) the RuntimeError is immediate and clear about both the cause and the repair, 3) the fixes are trivially easy to make (add try/except around next() calls and replace "raise StopIteration" with "return"). Ideally, everyone will let this die and go back to being with family for the holidays (or back to work if you don't have a holiday this week). Raymond -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Sat Nov 29 01:55:26 2014 From: guido at python.org (Guido van Rossum) Date: Fri, 28 Nov 2014 16:55:26 -0800 Subject: [Python-Dev] Please reconsider PEP 479. In-Reply-To: <492E98A9-0206-44CB-A380-31C2AE12B336@gmail.com> References: <54724117.20408@hotpy.org> <547525B1.4090303@canterbury.ac.nz> <54758FB2.6080100@canterbury.ac.nz> <492E98A9-0206-44CB-A380-31C2AE12B336@gmail.com> Message-ID: Thanks for being a good sport, Raymond! I've probably spent too much time fretting about this, so thanks for the reminder. I want to get back to other things too, in particular the type hinting PEP: there's a draft, but there are many things we --the co-authors-- want to change before we bother the community with another review. And that one will certainly take longer than five days! On Fri, Nov 28, 2014 at 12:01 PM, Raymond Hettinger < raymond.hettinger at gmail.com> wrote: > > On Nov 27, 2014, at 8:52 AM, Guido van Rossum wrote: > > I understand that @allow_import_stop represents a compromise, an attempt > at calming the waves that PEP 479 has caused. But I still want to push back > pretty hard on this idea. > > - It means we're forever stuck with two possible semantics for > StopIteration raised in generators. > > - It complicates the implementation, because (presumably) a generator > marked with @allow_stop_import should not cause a warning when a > StopIteration bubbles out -- so we actually need another flag to silence > the warning. > > - I don't actually know whether other Python implementations have the > ability to copy code objects to change flags. > > - It actually introduces a new incompatibility, that has to be solved in > every module that wants to use it (as you show above), whereas just putting > try/except around unguarded next() calls is fully backwards compatible. > > - Its existence encourage people to use the decorator in favor of fixing > their code properly. > > - The decorator is so subtle that it probably needs to be explained to > everyone who encounters it (and wasn't involved in this PEP discussion). > Because of this I would strongly advise against using it to "fix" the > itertools examples in the docs; it's just too magical. (IIRC only 2 > examples actually depend on this.) > > > I concur. PEP 479 fixes are trivially easy to do without a decorator. > > After Guido pronounced on the PEP, I fixed-up several parts of the > standard library in just a few minutes. It's not hard. > https://mail.python.org/pipermail/python-checkins/2014-November/133252.html > https://mail.python.org/pipermail/python-checkins/2014-November/133253.html > > Also, I'm submitting a 479 patch to the Django project so we won't have to > worry about this one. > > I recommend that everyone just accept that the PEP is a done deal and stop > adding complexity or work-arounds. We have a lot of things going for us on > this one: 1) the affected code isn't common-place (mostly in > producer/consumer middleware tools created by tool makers rather than by > tool users), 2) the RuntimeError is immediate and clear about both the > cause and the repair, 3) the fixes are trivially easy to make (add > try/except around next() calls and replace "raise StopIteration" with > "return"). > > Ideally, everyone will let this die and go back to being with family for > the holidays (or back to work if you don't have a holiday this week). > > > Raymond > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Sat Nov 29 02:59:06 2014 From: njs at pobox.com (Nathaniel Smith) Date: Sat, 29 Nov 2014 01:59:06 +0000 Subject: [Python-Dev] advice needed: best approach to enabling "metamodules"? Message-ID: Hi all, There was some discussion on python-ideas last month about how to make it easier/more reliable for a module to override attribute access. This is useful for things like autoloading submodules (accessing 'foo.bar' triggers the import of 'bar'), or for deprecating module attributes that aren't functions. (Accessing 'foo.bar' emits a DeprecationWarning, "the bar attribute will be removed soon".) Python has had some basic support for this for a long time -- if a module overwrites its entry in sys.modules[__name__], then the object that's placed there will be returned by 'import'. This allows one to define custom subclasses of module and use them instead of the default, similar to how metaclasses allow one to use custom subclasses of 'type'. In practice though it's very difficult to make this work safely and correctly for a top-level package. The main problem is that when you create a new object to stick into sys.modules, this necessarily means creating a new namespace dict. And now you have a mess, because now you have two dicts: new_module.__dict__ which is the namespace you export, and old_module.__dict__, which is the globals() for the code that's trying to define the module namespace. Keeping these in sync is extremely error-prone -- consider what happens, e.g., when your package __init__.py wants to import submodules which then recursively import the top-level package -- so it's difficult to justify for the kind of large packages that might be worried about deprecating entries in their top-level namespace. So what we'd really like is a way to somehow end up with an object that (a) has the same __dict__ as the original module, but (b) is of our own custom module subclass. If we can do this then metamodules will become safe and easy to write correctly. (There's a little demo of working metamodules here: https://github.com/njsmith/metamodule/ but it uses ctypes hacks that depend on non-stable parts of the CPython ABI, so it's not a long-term solution.) I've now spent some time trying to hack this capability into CPython and I've made a list of the possible options I can think of to fix this. I'm writing to python-dev because none of them are obviously The Right Way so I'd like to get some opinions/ruling/whatever on which approach to follow up on. Option 1: Make it possible to change the type of a module object in-place, so that we can write something like sys.modules[__name__].__class__ = MyModuleSubclass Option 1 downside: The invariants required to make __class__ assignment safe are complicated, and only implemented for heap-allocated type objects. PyModule_Type is not heap-allocated, so making this work would require lots of delicate surgery to typeobject.c. I'd rather not go down that rabbit-hole. ---- Option 2: Make PyModule_Type into a heap type allocated at interpreter startup, so that the above just works. Option 2 downside: PyModule_Type is exposed as a statically-allocated global symbol, so doing this would involve breaking the stable ABI. ---- Option 3: Make it legal to assign to the __dict__ attribute of a module object, so that we can write something like new_module = MyModuleSubclass(...) new_module.__dict__ = sys.modules[__name__].__dict__ sys.modules[__name__].__dict__ = {} # *** sys.modules[__name__] = new_module The line marked *** is necessary because the way modules are designed, they expect to control the lifecycle of their __dict__. When the module object is initialized, it fills in a bunch of stuff in the __dict__. When the module object (not the dict object!) is deallocated, it deletes everything from the __dict__. This latter feature in particular means that having two module objects sharing the same __dict__ is bad news. Option 3 downside: The paragraph above. Also, there's stuff inside the module struct besides just the __dict__, and more stuff has appeared there over time. ---- Option 4: Add a new function sys.swap_module_internals, which takes two module objects and swaps their __dict__ and other attributes. By making the operation a swap instead of an assignment, we avoid the lifecycle pitfalls from Option 3. By making it a builtin, we can make sure it always handles all the module fields that matter, not just __dict__. Usage: new_module = MyModuleSubclass(...) sys.swap_module_internals(new_module, sys.modules[__name__]) sys.modules[__name__] = new_module Option 4 downside: Obviously a hack. ---- Option 3 or 4 both seem workable, it just depends on which way we prefer to hold our nose. Option 4 is slightly more correct in that it works for *all* modules, but OTOH at the moment the only time Option 3 *really* fails is for compiled modules with PEP 3121 metadata, and compiled modules can already use a module subclass via other means (since they instantiate their own module objects). Thoughts? Suggestions on other options I've missed? Should I go ahead and write a patch for one of these? -n -- Nathaniel J. Smith Postdoctoral researcher - Informatics - University of Edinburgh http://vorpus.org From rosuav at gmail.com Sat Nov 29 04:45:11 2014 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 29 Nov 2014 14:45:11 +1100 Subject: [Python-Dev] advice needed: best approach to enabling "metamodules"? In-Reply-To: References: Message-ID: On Sat, Nov 29, 2014 at 12:59 PM, Nathaniel Smith wrote: > Option 4: Add a new function sys.swap_module_internals, which takes > two module objects and swaps their __dict__ and other attributes. By > making the operation a swap instead of an assignment, we avoid the > lifecycle pitfalls from Option 3. By making it a builtin, we can make > sure it always handles all the module fields that matter, not just > __dict__. Usage: > > new_module = MyModuleSubclass(...) > sys.swap_module_internals(new_module, sys.modules[__name__]) > sys.modules[__name__] = new_module > > Option 4 downside: Obviously a hack. This one corresponds to what I've seen in quite a number of C APIs. It's not ideal, but nothing is; and at least this way, it's clear that you're fiddling with internals. Letting the interpreter do the grunt-work for you is *definitely* preferable to having recipes out there saying "swap in a new __dict__, then don't forget to clear the old module's __dict__", which will have massive versioning issues as soon as a new best-practice comes along; making it a function, like this, means its implementation can smoothly change between versions (even in a bug-fix release). Would it be better to make that function also switch out the entry in sys.modules? That way, it's 100% dedicated to this job of "I want to make a subclass of module and use that for myself", and could then be made atomic against other imports. I've no idea whether there's any other weird shenanigans that could be deployed with this kind of module switch, nor whether cutting them out would be a good or bad thing! ChrisA From guido at python.org Sat Nov 29 05:21:01 2014 From: guido at python.org (Guido van Rossum) Date: Fri, 28 Nov 2014 20:21:01 -0800 Subject: [Python-Dev] advice needed: best approach to enabling "metamodules"? In-Reply-To: References: Message-ID: Are these really all our options? All of them sound like hacks, none of them sound like anything the language (or even the CPython implementation) should sanction. Have I missed the discussion where the use cases and constraints were analyzed and all other approaches were rejected? (I might have some half-baked ideas, but I feel I should read up on the past discussion first, and they are probably more fit for python-ideas than for python-dev. Plus I'm just writing this email because I'm procrastinating on the type hinting PEP. :-) --Guido On Fri, Nov 28, 2014 at 7:45 PM, Chris Angelico wrote: > On Sat, Nov 29, 2014 at 12:59 PM, Nathaniel Smith wrote: > > Option 4: Add a new function sys.swap_module_internals, which takes > > two module objects and swaps their __dict__ and other attributes. By > > making the operation a swap instead of an assignment, we avoid the > > lifecycle pitfalls from Option 3. By making it a builtin, we can make > > sure it always handles all the module fields that matter, not just > > __dict__. Usage: > > > > new_module = MyModuleSubclass(...) > > sys.swap_module_internals(new_module, sys.modules[__name__]) > > sys.modules[__name__] = new_module > > > > Option 4 downside: Obviously a hack. > > This one corresponds to what I've seen in quite a number of C APIs. > It's not ideal, but nothing is; and at least this way, it's clear that > you're fiddling with internals. Letting the interpreter do the > grunt-work for you is *definitely* preferable to having recipes out > there saying "swap in a new __dict__, then don't forget to clear the > old module's __dict__", which will have massive versioning issues as > soon as a new best-practice comes along; making it a function, like > this, means its implementation can smoothly change between versions > (even in a bug-fix release). > > Would it be better to make that function also switch out the entry in > sys.modules? That way, it's 100% dedicated to this job of "I want to > make a subclass of module and use that for myself", and could then be > made atomic against other imports. I've no idea whether there's any > other weird shenanigans that could be deployed with this kind of > module switch, nor whether cutting them out would be a good or bad > thing! > > ChrisA > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/guido%40python.org > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Sat Nov 29 12:32:30 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sat, 29 Nov 2014 12:32:30 +0100 Subject: [Python-Dev] advice needed: best approach to enabling "metamodules"? References: Message-ID: <20141129123230.73c76b31@fsol> On Sat, 29 Nov 2014 01:59:06 +0000 Nathaniel Smith wrote: > > Option 1: Make it possible to change the type of a module object > in-place, so that we can write something like > > sys.modules[__name__].__class__ = MyModuleSubclass > > Option 1 downside: The invariants required to make __class__ > assignment safe are complicated, and only implemented for > heap-allocated type objects. PyModule_Type is not heap-allocated, so > making this work would require lots of delicate surgery to > typeobject.c. I'd rather not go down that rabbit-hole. Option 1b: have __class__ assignment delegate to a tp_classassign slot on the old class, so that typeobject.c doesn't have to be cluttered with many special cases. > Option 3: Make it legal to assign to the __dict__ attribute of a > module object, so that we can write something like > > new_module = MyModuleSubclass(...) > new_module.__dict__ = sys.modules[__name__].__dict__ > sys.modules[__name__].__dict__ = {} # *** > sys.modules[__name__] = new_module > [...] > > Option 4: Add a new function sys.swap_module_internals, which takes > two module objects and swaps their __dict__ and other attributes. By > making the operation a swap instead of an assignment, we avoid the > lifecycle pitfalls from Option 3. By making it a builtin, we can make > sure it always handles all the module fields that matter, not just > __dict__. Usage: How do these two options interact with the fact that module functions store their globals dict, not the module itself? Regards Antoine. From ncoghlan at gmail.com Sat Nov 29 17:08:38 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 30 Nov 2014 02:08:38 +1000 Subject: [Python-Dev] Move selected documentation repos to PSF BitBucket account? In-Reply-To: References: <3CC02307-6109-4F20-A815-BDA7CFE16AC2@stufft.io> <013F5AD9-9C9F-4E2E-A750-1C758B66B675@stufft.io> <5A9CA007-3832-4931-8F05-06BD62BE4A5E@stufft.io> <35957465-A878-4864-9473-9F68B9556387@stufft.io> <5473E296.8090306@stoneleaf.us> <5473EB65.2070106@stoneleaf.us> <1A433321-49C7-461B-ACA0-D0EA7548AA21@stufft.io> Message-ID: On 29 November 2014 at 03:34, Demian Brecht wrote: > On Tue, Nov 25, 2014 at 6:52 AM, Brett Cannon wrote: >> >> I suspect if we make sure we add Bitbucket and GitHub login support to the issue tracker then that would help go a fair distance to helping with the GitHub pull of reach (and if we make it so people can simply paste in their fork's URL into the issue tracker and we simply grab a new patch for review that would go even farther). > > Chiming in horribly late, so hopefully this hasn't already been > mentioned (I've only loosely been following this thread). > > In addition to the login support (I'm not sold on how much that would > help the reach), I think it would be really beneficial to have some > documentation on either emulating git-style workflow in hg or > detailing a git fork workflow while working on multiple patches > concurrently and keeping master in sync with hg default (or perhaps > even both). As far as I'm aware, the easiest way to do that by using git-remote-hg to treat the CPython Mercurial repository as a git remote (although I've never tried it myself). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Sat Nov 29 17:16:25 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 30 Nov 2014 02:16:25 +1000 Subject: [Python-Dev] advice needed: best approach to enabling "metamodules"? In-Reply-To: <20141129123230.73c76b31@fsol> References: <20141129123230.73c76b31@fsol> Message-ID: On 29 November 2014 at 21:32, Antoine Pitrou wrote: > On Sat, 29 Nov 2014 01:59:06 +0000 > Nathaniel Smith wrote: >> >> Option 1: Make it possible to change the type of a module object >> in-place, so that we can write something like >> >> sys.modules[__name__].__class__ = MyModuleSubclass >> >> Option 1 downside: The invariants required to make __class__ >> assignment safe are complicated, and only implemented for >> heap-allocated type objects. PyModule_Type is not heap-allocated, so >> making this work would require lots of delicate surgery to >> typeobject.c. I'd rather not go down that rabbit-hole. > > Option 1b: have __class__ assignment delegate to a tp_classassign slot > on the old class, so that typeobject.c doesn't have to be cluttered with > many special cases. Aye, being able to hook class switching could be potentially useful (including the ability to just disallow it entirely if you really wanted to do that). >> Option 3: Make it legal to assign to the __dict__ attribute of a >> module object, so that we can write something like >> >> new_module = MyModuleSubclass(...) >> new_module.__dict__ = sys.modules[__name__].__dict__ >> sys.modules[__name__].__dict__ = {} # *** >> sys.modules[__name__] = new_module >> > [...] >> >> Option 4: Add a new function sys.swap_module_internals, which takes >> two module objects and swaps their __dict__ and other attributes. By >> making the operation a swap instead of an assignment, we avoid the >> lifecycle pitfalls from Option 3. By making it a builtin, we can make >> sure it always handles all the module fields that matter, not just >> __dict__. Usage: > > How do these two options interact with the fact that module functions > store their globals dict, not the module itself? Right, that's the part I consider the most challenging with metamodules - the fact that there's a longstanding assumption that a module is just a "dictionary with some metadata", so the interpreter is inclined to treat them that way. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From olemis at gmail.com Sat Nov 29 17:45:50 2014 From: olemis at gmail.com (Olemis Lang) Date: Sat, 29 Nov 2014 11:45:50 -0500 Subject: [Python-Dev] PEP 479 and asyncio In-Reply-To: References: Message-ID: On 11/28/14, Guido van Rossum wrote: [...] > > @Olemis: You never showed examples of how your code would be used, so it's > hard to understand what you're trying to do and how PEP 479 affects you. > The intention is not to restart the debate . PEP is approved , it's done ... but ... as a side-effect beware of the consequences that it is a fact that performance will be degraded (under certain circumstances) due to either a chain of (SI = StopIteration) raise SI => except SI: return => raise SI => ... ... or a few other similar cases which I will not describe for the sake of not repeating myself and being brief . -- Regards, Olemis - @olemislc Apache(tm) Bloodhound contributor http://issues.apache.org/bloodhound http://blood-hound.net Blog ES: http://simelo-es.blogspot.com/ Blog EN: http://simelo-en.blogspot.com/ Featured article: From mark at hotpy.org Sat Nov 29 17:46:27 2014 From: mark at hotpy.org (Mark Shannon) Date: Sat, 29 Nov 2014 16:46:27 +0000 Subject: [Python-Dev] advice needed: best approach to enabling "metamodules"? In-Reply-To: References: Message-ID: <5479F863.1000107@hotpy.org> On 29/11/14 01:59, Nathaniel Smith wrote: > Hi all, > [snip] > > Option 3: Make it legal to assign to the __dict__ attribute of a > module object, so that we can write something like > > new_module = MyModuleSubclass(...) > new_module.__dict__ = sys.modules[__name__].__dict__ > sys.modules[__name__].__dict__ = {} # *** > sys.modules[__name__] = new_module > Why does MyModuleClass need to sub-class types.ModuleType? Modules have no special behaviour, apart from the inability to write to their __dict__ attribute, which is the very thing you don't want. If it quacks like a module... Cheers, Mark. From ncoghlan at gmail.com Sat Nov 29 18:07:15 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 30 Nov 2014 03:07:15 +1000 Subject: [Python-Dev] PEP 479 and asyncio In-Reply-To: References: Message-ID: On 30 November 2014 at 02:45, Olemis Lang wrote: > On 11/28/14, Guido van Rossum wrote: > [...] >> >> @Olemis: You never showed examples of how your code would be used, so it's >> hard to understand what you're trying to do and how PEP 479 affects you. >> > > The intention is not to restart the debate . PEP is approved , it's > done ... but ... > > > as a side-effect beware of the consequences that it is a fact that > performance will be degraded (under certain circumstances) due to > either a chain of (SI = StopIteration) > > raise SI => except SI: return => raise SI => ... > > ... or a few other similar cases which I will not describe for the > sake of not repeating myself and being brief . > Guido wrote a specific micro-benchmark for that case in one of the other threads. On his particular system, the overhead was around 150 ns per link in the chain at the point the data processing pipeline was shut down. In most scenarios where a data processing pipeline is worth setting up in the first place, the per-item handling costs (which won't change) are likely to overwhelm the shutdown costs (which will get marginally slower). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From guido at python.org Sat Nov 29 18:47:46 2014 From: guido at python.org (Guido van Rossum) Date: Sat, 29 Nov 2014 09:47:46 -0800 Subject: [Python-Dev] PEP 479 and asyncio In-Reply-To: References: Message-ID: On Sat, Nov 29, 2014 at 9:07 AM, Nick Coghlan wrote: > Guido wrote a specific micro-benchmark for that case in one of the > other threads. On his particular system, the overhead was around 150 > ns per link in the chain at the point the data processing pipeline was > shut down. In most scenarios where a data processing pipeline is worth > setting up in the first place, the per-item handling costs (which > won't change) are likely to overwhelm the shutdown costs (which will > get marginally slower). > If I hadn't written that benchmark I wouldn't recognize what you're talking about here. :-) This is entirely off-topic, but if I didn't know it was about one generator calling next() to iterate over another generator, I wouldn't have understood what pattern you refer to as a data processing pipeline. And I still don't understand how the try/except *setup* cost became *shut down* cost of the pipeline. But that doesn't matter, since the number of setups equals the number of shut downs. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Sat Nov 29 20:37:41 2014 From: njs at pobox.com (Nathaniel Smith) Date: Sat, 29 Nov 2014 19:37:41 +0000 Subject: [Python-Dev] advice needed: best approach to enabling "metamodules"? In-Reply-To: References: Message-ID: On Sat, Nov 29, 2014 at 4:21 AM, Guido van Rossum wrote: > Are these really all our options? All of them sound like hacks, none of them > sound like anything the language (or even the CPython implementation) should > sanction. Have I missed the discussion where the use cases and constraints > were analyzed and all other approaches were rejected? (I might have some > half-baked ideas, but I feel I should read up on the past discussion first, > and they are probably more fit for python-ideas than for python-dev. Plus > I'm just writing this email because I'm procrastinating on the type hinting > PEP. :-) The previous discussions I was referring to are here: http://thread.gmane.org/gmane.comp.python.ideas/29487/focus=29555 http://thread.gmane.org/gmane.comp.python.ideas/29788 There might well be other options; these are just the best ones I could think of :-). The constraints are pretty tight, though: - The "new module" object (whatever it is) should have a __dict__ that aliases the original module globals(). I can elaborate on this if my original email wasn't enough, but hopefully it's obvious that making two copies of the same namespace and then trying to keep them in sync at the very least smells bad :-). - The "new module" object has to be a subtype of ModuleType, b/c there are lots of places that do isinstance(x, ModuleType) checks (notably -- but not only -- reload()). Since a major goal here is to make it possible to do cleaner deprecations, it would be really unfortunate if switching an existing package to use the metamodule support itself broke things :-). - Lookups in the normal case should have no additional performance overhead, because module lookups are extremely extremely common. (So this rules out dict proxies and tricks like that -- we really need 'new_module.__dict__ is globals()' to be true.) AFAICT there are three logically possible strategies for satisfying that first constraint: (a) convert the original module object into the type we want, in-place (b) create a new module object that acts like the original module object (c) somehow arrange for our special type to be used from the start My options 1 and 2 are means of accomplishing (a), and my options 3 and 4 are means of accomplishing (b) while working around the behavioural quirks of module objects (as required by the second constraint). The python-ideas thread did also consider several methods of implementing strategy (c), but they're messy enough that I left them out here. The problem is that somehow we have to execute code to create the new subtype *before* we have an entry in sys.modules for the package that contains the code for the subtype. So one option would be to add a new rule, that if a file pkgname/__new__.py exists, then this is executed first and is required to set up sys.modules["pkgname"] before we exec pkgname/__init__.py. So pkgname/__new__.py might look like: import sys from pkgname._metamodule import MyModuleSubtype sys.modules[__name__] = MyModuleSubtype(__name__, docstring) This runs into a lot of problems though. To start with, the 'from pkgname._metamodule ...' line is an infinite loop, b/c this is the code used to create sys.modules["pkgname"]. It's not clear where the globals dict for executing __new__.py comes from (who defines __name__? Currently that's done by ModuleType.__init__). It only works for packages, not modules. The need to provide the docstring here, before __init__.py is even read, is weird. It adds extra stat() calls to every package lookup. And, the biggest showstopper IMHO: AFAICT it's impossible to write a polyfill to support this code on old python versions, so it's useless to any package which needs to keep compatibility with 2.7 (or even 3.4). Sure, you can backport the whole import system like importlib2, but telling everyone that they need to replace every 'import numpy' with 'import importlib2; import numpy' is a total non-starter. So, yeah, those 4 options are really the only plausible ones I know of. Option 1 and option 3 are pretty nice at the language level! Most Python objects allow assignment to __class__ and __dict__, and both PyPy and Jython at least do support __class__ assignment. Really the only downside with Option 1 is that actually implementing it requires attention from someone with deep knowledge of typeobject.c. -n -- Nathaniel J. Smith Postdoctoral researcher - Informatics - University of Edinburgh http://vorpus.org From njs at pobox.com Sat Nov 29 21:02:50 2014 From: njs at pobox.com (Nathaniel Smith) Date: Sat, 29 Nov 2014 20:02:50 +0000 Subject: [Python-Dev] advice needed: best approach to enabling "metamodules"? In-Reply-To: <20141129123230.73c76b31@fsol> References: <20141129123230.73c76b31@fsol> Message-ID: On Sat, Nov 29, 2014 at 11:32 AM, Antoine Pitrou wrote: > On Sat, 29 Nov 2014 01:59:06 +0000 > Nathaniel Smith wrote: >> >> Option 1: Make it possible to change the type of a module object >> in-place, so that we can write something like >> >> sys.modules[__name__].__class__ = MyModuleSubclass >> >> Option 1 downside: The invariants required to make __class__ >> assignment safe are complicated, and only implemented for >> heap-allocated type objects. PyModule_Type is not heap-allocated, so >> making this work would require lots of delicate surgery to >> typeobject.c. I'd rather not go down that rabbit-hole. > > Option 1b: have __class__ assignment delegate to a tp_classassign slot > on the old class, so that typeobject.c doesn't have to be cluttered with > many special cases. I'm intrigued -- how would this help? I have a vague impression that one could add another branch to object_set_class that went something like if at least one of the types is a subtype of the other type, and the subtype is a heap type with tp_dealloc == subtype_dealloc, and the subtype doesn't add any important slots, and ... then the __class__ assignment is legal. (This is taking advantage of the fact that if you don't have any extra slots added, then subtype_dealloc just basically defers to the base type's tp_dealloc, so it doesn't really matter which one you end up calling.) And my vague impression is that there isn't really anything special about the module type that would allow a tp_classassign function to simplify this logic. But these are just vague impressions :-) >> Option 3: Make it legal to assign to the __dict__ attribute of a >> module object, so that we can write something like >> >> new_module = MyModuleSubclass(...) >> new_module.__dict__ = sys.modules[__name__].__dict__ >> sys.modules[__name__].__dict__ = {} # *** >> sys.modules[__name__] = new_module >> > [...] >> >> Option 4: Add a new function sys.swap_module_internals, which takes >> two module objects and swaps their __dict__ and other attributes. By >> making the operation a swap instead of an assignment, we avoid the >> lifecycle pitfalls from Option 3. By making it a builtin, we can make >> sure it always handles all the module fields that matter, not just >> __dict__. Usage: > > How do these two options interact with the fact that module functions > store their globals dict, not the module itself? I think that's totally fine? The whole point of all these proposals is to make sure that the final module object does in fact have the correct globals dict. ~$ git clone git at github.com:njsmith/metamodule.git ~$ cd metamodule ~/metamodule$ python3.4 >>> import examplepkg >>> examplepkg >>> examplepkg.f.__globals__ is examplepkg.__dict__ True If anything this is another argument for why we NEED something like this :-). -n -- Nathaniel J. Smith Postdoctoral researcher - Informatics - University of Edinburgh http://vorpus.org From mark at hotpy.org Sat Nov 29 21:15:45 2014 From: mark at hotpy.org (Mark Shannon) Date: Sat, 29 Nov 2014 20:15:45 +0000 Subject: [Python-Dev] advice needed: best approach to enabling "metamodules"? In-Reply-To: References: Message-ID: <547A2971.7010202@hotpy.org> On 29/11/14 19:37, Nathaniel Smith wrote: [snip] > - The "new module" object has to be a subtype of ModuleType, b/c there > are lots of places that do isinstance(x, ModuleType) checks (notably It has to be a *subtype* is does not need to be a *subclass* >>> class M: ... __class__ = ModuleType ... >>> isinstance(M(), ModuleType) True Cheers, Mark. From solipsis at pitrou.net Sat Nov 29 21:18:58 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sat, 29 Nov 2014 21:18:58 +0100 Subject: [Python-Dev] advice needed: best approach to enabling "metamodules"? In-Reply-To: References: <20141129123230.73c76b31@fsol> Message-ID: <20141129211858.5da3c1ee@fsol> On Sat, 29 Nov 2014 20:02:50 +0000 Nathaniel Smith wrote: > > > > Option 1b: have __class__ assignment delegate to a tp_classassign slot > > on the old class, so that typeobject.c doesn't have to be cluttered with > > many special cases. > > I'm intrigued -- how would this help? It would allow ModuleType to override tp_classassign to decide whether and how __class__ assignment on a module instance is allowed to work. So typeobject.c needn't know about any specifics of ModuleType or any other type. > > How do these two options interact with the fact that module functions > > store their globals dict, not the module itself? > > I think that's totally fine? The whole point of all these proposals is > to make sure that the final module object does in fact have the > correct globals dict. > > ~$ git clone git at github.com:njsmith/metamodule.git Ok, I see. The code hacks up the new module to take ownership of the old module's __dict__. That doesn't look very clean to me. Regards Antoine. From greg.ewing at canterbury.ac.nz Sat Nov 29 22:24:25 2014 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sun, 30 Nov 2014 10:24:25 +1300 Subject: [Python-Dev] advice needed: best approach to enabling "metamodules"? In-Reply-To: References: Message-ID: <547A3989.5030408@canterbury.ac.nz> Nathaniel Smith wrote: > Option 4: Add a new function sys.swap_module_internals, which takes > two module objects and swaps their __dict__ and other attributes. By > making the operation a swap instead of an assignment, we avoid the > lifecycle pitfalls from Option 3. Didn't I see somewhere that module dicts are not being cleared on shutdown any more? If so, then the lifetime problem mentioned here no longer exists. -- Greg From greg.ewing at canterbury.ac.nz Sat Nov 29 22:36:45 2014 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sun, 30 Nov 2014 10:36:45 +1300 Subject: [Python-Dev] advice needed: best approach to enabling "metamodules"? In-Reply-To: References: Message-ID: <547A3C6D.9050803@canterbury.ac.nz> Guido van Rossum wrote: > Are these really all our options? All of them sound like hacks, none of > them sound like anything the language (or even the CPython > implementation) should sanction. If assignment to the __class__ of a module were permitted (by whatever means) then you could put this in a module: class __class__(types.ModuleType): ... which makes it look almost like a deliberate language feature. :-) Seriously, of the options presented, I think that allowing __class__ assignment is the most elegant, since it solves a lot of problems in one go without introducing any new features -- just removing a restriction that prevents an existing language mechanism from working in this case. -- Greg From contact at ionelmc.ro Sat Nov 29 22:51:32 2014 From: contact at ionelmc.ro (=?UTF-8?Q?Ionel_Cristian_M=C4=83rie=C8=99?=) Date: Sat, 29 Nov 2014 23:51:32 +0200 Subject: [Python-Dev] advice needed: best approach to enabling "metamodules"? In-Reply-To: <547A3C6D.9050803@canterbury.ac.nz> References: <547A3C6D.9050803@canterbury.ac.nz> Message-ID: What if we'd have metaclass semantics on module creation? Eg, suppose the default: __metaclass__ = ModuleType What if Python would support __prepare__ for modules? Thanks, -- Ionel M. On Sat, Nov 29, 2014 at 11:36 PM, Greg Ewing wrote: > Guido van Rossum wrote: > >> Are these really all our options? All of them sound like hacks, none of >> them sound like anything the language (or even the CPython implementation) >> should sanction. >> > > If assignment to the __class__ of a module were permitted > (by whatever means) then you could put this in a module: > > class __class__(types.ModuleType): > ... > > which makes it look almost like a deliberate language > feature. :-) > > Seriously, of the options presented, I think that allowing > __class__ assignment is the most elegant, since it solves > a lot of problems in one go without introducing any new > features -- just removing a restriction that prevents an > existing language mechanism from working in this case. > > -- > Greg > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ > contact%40ionelmc.ro > -------------- next part -------------- An HTML attachment was scrubbed... URL: From greg.ewing at canterbury.ac.nz Sat Nov 29 23:07:57 2014 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sun, 30 Nov 2014 11:07:57 +1300 Subject: [Python-Dev] advice needed: best approach to enabling "metamodules"? In-Reply-To: References: Message-ID: <547A43BD.8080303@canterbury.ac.nz> Nathaniel Smith wrote: > So pkgname/__new__.py might look like: > > import sys > from pkgname._metamodule import MyModuleSubtype > sys.modules[__name__] = MyModuleSubtype(__name__, docstring) > > To start with, the 'from > pkgname._metamodule ...' line is an infinite loop, Why does MyModuleSubtype have to be imported from pkgname? It would make more sense for it to be defined directly in __new__.py, wouldn't it? Isn't the purpose of separating stuff out into __new__.py precisely to avoid circularities like that? -- Greg From ethan at stoneleaf.us Fri Nov 28 20:55:14 2014 From: ethan at stoneleaf.us (Ethan Furman) Date: Fri, 28 Nov 2014 11:55:14 -0800 Subject: [Python-Dev] Move selected documentation repos to PSF BitBucket account? In-Reply-To: References: <5A9CA007-3832-4931-8F05-06BD62BE4A5E@stufft.io> <35957465-A878-4864-9473-9F68B9556387@stufft.io> <5473E296.8090306@stoneleaf.us> <5473EB65.2070106@stoneleaf.us> <1A433321-49C7-461B-ACA0-D0EA7548AA21@stufft.io> Message-ID: <5478D322.8040206@stoneleaf.us> On 11/28/2014 09:34 AM, Demian Brecht wrote: > > I primarily use git for development. Having little or no effort to > context switch to work on CPython in any capacity (PEPs, code, etc) > would be hugely beneficial for me. Having a well defined workflow in > the docs (perhaps alongside "Lifecycle of a patch"?) would have > significantly lowered the initial barrier of entry for me. Given the > amount of time I put into that initially, I can only imagine how many > people it's entirely turned away from contributing. I'd definitely be > interested in contributing documentation around this (I've written up > something similar here > http://demianbrecht.github.io/vcs/2014/07/31/from-git-to-hg/) if > others feel that it would be valuable. That would be very valuable, thank you. -- ~Ethan~ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: OpenPGP digital signature URL: From encukou at gmail.com Sun Nov 30 00:07:02 2014 From: encukou at gmail.com (Petr Viktorin) Date: Sun, 30 Nov 2014 00:07:02 +0100 Subject: [Python-Dev] advice needed: best approach to enabling "metamodules"? In-Reply-To: References: Message-ID: On Sat, Nov 29, 2014 at 8:37 PM, Nathaniel Smith wrote: [...] > The python-ideas thread did also consider several methods of > implementing strategy (c), but they're messy enough that I left them > out here. The problem is that somehow we have to execute code to > create the new subtype *before* we have an entry in sys.modules for > the package that contains the code for the subtype. So one option > would be to add a new rule, that if a file pkgname/__new__.py exists, > then this is executed first and is required to set up > sys.modules["pkgname"] before we exec pkgname/__init__.py. So > pkgname/__new__.py might look like: > > import sys > from pkgname._metamodule import MyModuleSubtype > sys.modules[__name__] = MyModuleSubtype(__name__, docstring) > This runs into a lot of problems though. To start with, the 'from > pkgname._metamodule ...' line is an infinite loop, b/c this is the > code used to create sys.modules["pkgname"]. As Greg Ewing said ? you don't want to import from the package whose metamodule you're defining. You'd want to do as little work as possible in __new__.py. I'd use something like this: import types class __metamodule__(types.ModuleType): def __call__(self): return self.main() where Python would get the attribute __metamodule__ from __new__.py, and use `__metamodule__(name, doc)` as the thing to execute __main__ in. > It's not clear where the > globals dict for executing __new__.py comes from (who defines > __name__? Currently that's done by ModuleType.__init__). Well, it could still be in __metamodule__.__init__(). > It only works for packages, not modules. I don't see a need for this treatment for modules in a package ? if you want `from mypkg import callme`, you can make "callme" a function rather than a callable module. If you *also* want `from mypkg.callme import something_else`, I say you should split "callme" into two differently named things; names are cheap inside a package. If really needed, modules in a package can use an import hook defined in the package, or be converted to subpackages. Single-module projects would be left out, yes ? but those can be simply converted to a package. > The need to provide the docstring here, > before __init__.py is even read, is weird. Does it have to be before __init__.py is read? Can't __init__.py be compiled beforehand, to get __doc__, and only *run* in the new namespace? (Or should __new__.py define import hooks that say how __init__.py should be loaded/compiled? I don't see a case for that.) > It adds extra stat() calls to every package lookup. Fair. > And, the biggest showstopper IMHO: AFAICT > it's impossible to write a polyfill to support this code on old python > versions, so it's useless to any package which needs to keep > compatibility with 2.7 (or even 3.4). Sure, you can backport the whole > import system like importlib2, but telling everyone that they need to > replace every 'import numpy' with 'import importlib2; import numpy' is > a total non-starter. I'm probably missing something obvious, but where would this not work? - As the first thing it does, __init__.py imports the polyfill and calls polyfill(__name__) - The polyfill, if running non-recursively* under old Python: -- compiles __init__.py -- imports __new__.py to get __metamodule__ -- instantiates metamodule with name, and docstring from compiled code -- * remembers the instance, to check for recursion later -- puts it in sys.modules -- execs __init__ in it - afterwards the original __init__.py execution continues, filling up a now-unused module's namespace From steve at pearwood.info Sun Nov 30 00:05:05 2014 From: steve at pearwood.info (Steven D'Aprano) Date: Sun, 30 Nov 2014 10:05:05 +1100 Subject: [Python-Dev] advice needed: best approach to enabling "metamodules"? In-Reply-To: <547A43BD.8080303@canterbury.ac.nz> References: <547A43BD.8080303@canterbury.ac.nz> Message-ID: <20141129230504.GA11424@ando.pearwood.info> On Sun, Nov 30, 2014 at 11:07:57AM +1300, Greg Ewing wrote: > Nathaniel Smith wrote: > >So pkgname/__new__.py might look like: > > > > import sys > > from pkgname._metamodule import MyModuleSubtype > > sys.modules[__name__] = MyModuleSubtype(__name__, docstring) > > > >To start with, the 'from > >pkgname._metamodule ...' line is an infinite loop, > > Why does MyModuleSubtype have to be imported from pkgname? > It would make more sense for it to be defined directly in > __new__.py, wouldn't it? Isn't the purpose of separating > stuff out into __new__.py precisely to avoid circularities > like that? Perhaps I'm missing something, but won't that imply that every module which wants to use a "special" module type has to re-invent the wheel? If this feature is going to be used, I would expect to be able to re-use pre-written module types. E.g. having written "module with properties" (so to speak) once, I can just import it and use it in my next project. -- Steven From donald at stufft.io Sun Nov 30 00:27:05 2014 From: donald at stufft.io (Donald Stufft) Date: Sat, 29 Nov 2014 18:27:05 -0500 Subject: [Python-Dev] PEP 481 - Migrate Some Supporting Repositories to Git and Github Message-ID: <30725AC9-3DCF-416A-BCCD-2D64D489898C@stufft.io> As promised in the "Move selected documentation repos to PSF BitBucket account?" thread I've written up a PEP for moving selected repositories from hg.python.org to Github. You can see this PEP online at: https://www.python.org/dev/peps/pep-0481/ I've also reproduced the PEP below for inline discussion. ----------------------- Abstract ======== This PEP proposes migrating to Git and Github for certain supporting repositories (such as the repository for Python Enhancement Proposals) in a way that is more accessible to new contributors, and easier to manage for core developers. This is offered as an alternative to PEP 474 which aims to achieve the same overall benefits but while continuing to use the Mercurial DVCS and without relying on a commerical entity. In particular this PEP proposes changes to the following repositories: * https://hg.python.org/devguide/ * https://hg.python.org/devinabox/ * https://hg.python.org/peps/ This PEP does not propose any changes to the core development workflow for CPython itself. Rationale ========= As PEP 474 mentions, there are currently a number of repositories hosted on hg.python.org which are not directly used for the development of CPython but instead are supporting or ancillary repositories. These supporting repositories do not typically have complex workflows or often branches at all other than the primary integration branch. This simplicity makes them very good targets for the "Pull Request" workflow that is commonly found on sites like Github. However where PEP 474 wants to continue to use Mercurial and wishes to use an OSS and self-hosted and therefore restricts itself to only those solutions this PEP expands the scope of that to include migrating to Git and using Github. The existing method of contributing to these repositories generally includes generating a patch and either uploading them to bugs.python.org or emailing them to peps at python.org. This process is unfriendly towards non-comitter contributors as well as making the process harder than it needs to be for comitters to accept the patches sent by users. In addition to the benefits in the pull request workflow itself, this style of workflow also enables non techincal contributors, especially those who do not know their way around the DVCS of choice, to contribute using the web based editor. On the committer side the Pull Requests enable them to tell, before merging, whether or not a particular Pull Request will break anything. It also enables them to do a simple "push button" merge which does not require them to check out the changes locally. Another such feature that is useful in particular for docs, is the ability to view a "prose" diff. This Github specific feature enables a committer to view a diff of the rendered output which will hide things like reformatting a paragraph and show you what the actual "meat" of the change actually is. Why Git? -------- Looking at the variety of DVCS which are available today it becomes fairly clear that git has gotten the vast mindshare of people who are currently using it. The Open Hub (Previously Ohloh) statistics [#openhub-stats]_ show that currently 37% of the repositories Open Hub is indexing is using git which is second only to SVN (which has 48%) while Mercurial has just 2% of the indexed repositories (beating only bazaar which has 1%). In additon to the Open Hub statistics a look at the top 100 projects on PyPI (ordered by total download counts) shows us that within the Python space itself there is a majority of projects using git: === ========= ========== ====== === ==== Git Mercurial Subversion Bazaar CVS None === ========= ========== ====== === ==== 62 22 7 4 1 1 === ========= ========== ====== === ==== Chosing a DVCS which has the larger mindshare will make it more likely that any particular person who has experience with DVCS at all will be able to meaningfully use the DVCS that we have chosen without having to learn a new tool. In addition to simply making it more likely that any individual will already know how to use git, the number of projects and people using it means that the resources for learning the tool are likely to be more fully fleshed out and when you run into problems the liklihood that someone else had that problem and posted a question and recieved an answer is also far likelier. Thirdly by using a more popular tool you also increase your options for tooling *around* the DVCS itself. Looking at the various options for hosting repositories it's extremely rare to find a hosting solution (whether OSS or commerical) that supports Mercurial but does not support Git, on the flip side there are a number of tools which support Git but do not support Mercurial. Therefore the popularity of git increases the flexibility of our options going into the future for what toolchain these projects use. Also by moving to the more popular DVCS we increase the likelhood that the knowledge that the person has learned in contributing to these support repositories will transfer to projects outside of the immediate CPython project such as to the larger Python community which is primarily using Git hosted on Github. In previous years there was concern about how well supported git was on Windows in comparison to Mercurial. However git has grown to support Windows as a first class citizen. In addition to that, for Windows users who are not well aquanted with the Windows command line there are GUI options as well. On a techincal level git and Mercurial are fairly similar, however the git branching model is signifcantly better than Mercurial "Named Branches" for non-comitter contributors. Mercurial does have a "Bookmarks" extension however this isn't quite as good as git's branching model. All bookmarks live in the same namespace so it requires individual users to ensure that they namespace the branchnames themselves lest the risk collision. It also is an extension which requires new users to first discover they need an extension at all and then figure out what they need to do in order to enable that extension. Since it is an extension it also means that in general support for them outside of Mercurial core is going to be less than 100% in comparison to git where the feature is built in and core to using git at all. Finally users who are not used to Mercurial are unlikely to discover bookmarks on their own, instead they will likely attempt to use Mercurial's "Named Branches" which, given the fact they live "forever", are not often what a project wants their contributors to use. Why Github? ----------- There are a number of software projects or web services which offer functionality similar to that of Github. These range from commerical web services such as a Bitbucket to self-hosted OSS solutions such as Kallithea or Gitlab. This PEP proposes that we move these repositories to Github. There are two primary reasons for selecting Github: Popularity and Quality/Polish. Github is currently the most popular hosted repository hosting according to Alexa where it currently has a global rank of 121. Much like for Git itself by choosing the most popular tool we gain benefits in increasing the likelhood that a new contributor will have already experienced the toolchain, the quality and availablity of the help, more and better tooling being built around it, and the knowledge transfer to other projects. A look again at the top 100 projects by download counts on PyPI shows the following hosting locations: ====== ========= =========== ========= =========== ========== GitHub BitBucket Google Code Launchpad SourceForge Other/Self ====== ========= =========== ========= =========== ========== 62 18 6 4 3 7 ====== ========= =========== ========= =========== ========== In addition to all of those reasons, Github also has the benefit that while many of the options have similar features when you look at them in a feature matrix the Github version of each of those features tend to work better and be far more polished. This is hard to quantify objectively however it is a fairly common sentiment if you go around and ask people who are using these services often. Finally a reason to choose a web service at all over something that is self-hosted is to be able to more efficiently use volunteer time and donated resources. Every additional service hosted on the PSF infrastruture by the PSF infrastructure team further spreads out the amount of time that the volunteers on that team have to spend and uses some chunk of resources that could potentionally be used for something where there is no free or affordable hosted solution available. One concern that people do have with using a hosted service is that there is a lack of control and that at some point in the future the service may no longer be suitable. It is the opinion of this PEP that Github does not currently and has not in the past engaged in any attempts to lock people into their platform and that if at some point in the future Github is no longer suitable for one reason or another than at that point we can look at migrating away from Github onto a different solution. In other words, we'll cross that bridge if and when we come to it. Example: Scientific Python -------------------------- One of the key ideas behind the move to both git and Github is that a feature of a DVCS, the repository hosting, and the workflow used is the social network and size of the community using said tools. We can see this is true by looking at an example from a sub-community of the Python community: The Scientific Python community. They have already migrated most of the key pieces of the SciPy stack onto Github using the Pull Request based workflow starting with IPython and as more projects moved over it became a natural default for new projects. They claim to have seen a great benefit from this move, where it enables casual contributors to easily move between different projects within their sub-community without having to learn a special, bespoke workflow and a different toolchain for each project. They've found that when people can use their limited time on actually contributing instead of learning the different tools and workflows that not only do they contribute more to one project, that they also expand out and contribute to other projects. This move is also attributed to making it commonplace for members of that community to go so far as publishing their research and educational materials on Github as well. This showcases the real power behind moving to a highly popular toolchain and workflow, as each variance introduces yet another hurdle for new and casual contributors to get past and it makes the time spent learning that workflow less reusable with other projects. Migration ========= Through the use of hg-git [#hg-git]_ we can easily convert a Mercurial repository to a Git repository by simply pushing the Mercurial repository to the Git repository. People who wish to continue to use Mercurual locally can then use hg-git going into the future using the new Github URL, however they will need to re-clone their repositories as using Git as the server seems to trigger a one time change of the changeset ids. As none of the selected repositories have any tags, branches, or bookmarks other than the ``default`` branch the migration will simply map the ``default`` branch in Mercurial to the ``master`` branch in git. In addition since none of the selected projects have any great need of a complex bug tracker, they will also migrate their issue handling to using the GitHub issues. In addition to the migration of the repository hosting itself there are a number of locations for each particular repository which will require updating. The bulk of these will simply be changing commands from the hg equivilant to the git equivilant. In particular this will include: * Updating www.python.org to generate PEPs using a git clone and link to Github. * Updating docs.python.org to pull from Github instead of hg.python.org for the devguide. * Enabling the ability to send an email to python-checkins at python.org for each push. * Enabling the ability to send an IRC message to #python-dev on Freenode for each push. * Migrate any issues for these projects to their respective bug tracker on Github. This will restore these repositories to similar functionality as they currently have. In addition to this the migration will also include enabling testing for each pull request using Travis CI [#travisci]_ where possible to ensure that a new PR does not break the ability to render the documentation or PEPs. User Access =========== Moving to Github would involve adding an additional user account that will need to be managed, however it also offers finer grained control, allowing the ability to grant someone access to only one particular repository instead of the coarser grained ACLs available on hg.python.org. References ========== .. [#openhub-stats] `Open Hub Statistics ` .. [#hg-git] `hg-git ` .. [#travisci] `Travis CI ` --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA From alex.gaynor at gmail.com Sun Nov 30 01:15:55 2014 From: alex.gaynor at gmail.com (Alex Gaynor) Date: Sun, 30 Nov 2014 00:15:55 +0000 (UTC) Subject: [Python-Dev] =?utf-8?q?PEP_481_-_Migrate_Some_Supporting_Reposito?= =?utf-8?q?ries_to_Git=09and_Github?= References: <30725AC9-3DCF-416A-BCCD-2D64D489898C@stufft.io> Message-ID: Donald Stufft stufft.io> writes: > > [words words words] > I strongly support this PEP. I'd like to share two pieces of information. Both of these are personal anecdotes: For the past several years, I've been a contributor on two major projects using mercurial, CPython and PyPy. PyPy has a strong culture of in-repo branching, basically all contributors regularly make branches in the main repo for their work, and we're very free in giving people commit rights, so almost everyone working on PyPy in any way has this level of access. This workflow works ok. I don't love it as much as git, but it's fine, it's not an impediment to my work. By contrast, CPython does not have this type of workflow, there are almost no in-tree branches besides the 2.7, 3.4, etc. ones. Despite being a regular hg user for years, I have no idea how to create a local-only branch, or a branch which is pushed to a remote (to use the git term). I also don't generally commit my own work to CPython, even though I have push privledges, because I prefer to *always* get code review on my work. As a result, I use a git mirror of CPython to do all my work, and generate patches from that. The conclusion I draw from this is that hg's workflow is probably fine if you're a committer on the project, or don't ever need to maintain multiple patches concurrently (and thus can just leave everything uncommitted in the repo). However, the hg workflow seems extremely defficient at non-committer contributors. The seconds experience I have is that of Django's migration to git and github. For a long time we were on SVN, and we were very resistant to moving to DVCS in general, and github in particular. Multiple times I said that I didn't see how exporting a patch and uploading it to trac was more difficult than sending a pull request. That was very wrong on my part. My primary observation is not about new contributors though, it's actually about the behavior of core developers. Before we were on github, it was fairly rare for core developers to ask for reviews for anything besides *gigantic* patches, we'd mostly just commit stuff to trunk. Since the switch to github, I've seen that core developers are *far* more likely to ask for reviews of their work before merging. Big +1 from me, thanks for writing this up Donald, Alex From donald at stufft.io Sun Nov 30 01:25:59 2014 From: donald at stufft.io (Donald Stufft) Date: Sat, 29 Nov 2014 19:25:59 -0500 Subject: [Python-Dev] PEP 481 - Migrate Some Supporting Repositories to Git and Github In-Reply-To: <30725AC9-3DCF-416A-BCCD-2D64D489898C@stufft.io> References: <30725AC9-3DCF-416A-BCCD-2D64D489898C@stufft.io> Message-ID: <5135803B-6729-4ED1-940A-FCBC49CC1736@stufft.io> > On Nov 29, 2014, at 6:27 PM, Donald Stufft wrote: > > [lots of words] Just FYI, I?ve pushed an update to the PEP. Nothing major just some grammatical fixes and such. The revision is here: https://hg.python.org/peps/rev/6c6947dbd13f For whatever it's worth, the person who submitted that patch used Github's online editor to submit them and made a PR to my personal PEP repository where I work on my PEPs (https://github.com/dstufft/peps/pull/3). If someone wanted to see some of the features in action that is a nice PEP to look at. In particular if you hit "Files Changed" then beside the view button is an icon that looks like a piece of paper and when you over over it it'll say "Display the Rich Diff". Clicking on that you'll see the diff of the rendered output which lets you ignore things which were just reflowing the paragraphs and such. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA From encukou at gmail.com Sun Nov 30 00:16:42 2014 From: encukou at gmail.com (Petr Viktorin) Date: Sun, 30 Nov 2014 00:16:42 +0100 Subject: [Python-Dev] advice needed: best approach to enabling "metamodules"? In-Reply-To: <20141129230504.GA11424@ando.pearwood.info> References: <547A43BD.8080303@canterbury.ac.nz> <20141129230504.GA11424@ando.pearwood.info> Message-ID: On Sun, Nov 30, 2014 at 12:05 AM, Steven D'Aprano wrote: > On Sun, Nov 30, 2014 at 11:07:57AM +1300, Greg Ewing wrote: >> Nathaniel Smith wrote: >> >So pkgname/__new__.py might look like: >> > >> > import sys >> > from pkgname._metamodule import MyModuleSubtype >> > sys.modules[__name__] = MyModuleSubtype(__name__, docstring) >> > >> >To start with, the 'from >> >pkgname._metamodule ...' line is an infinite loop, >> >> Why does MyModuleSubtype have to be imported from pkgname? >> It would make more sense for it to be defined directly in >> __new__.py, wouldn't it? Isn't the purpose of separating >> stuff out into __new__.py precisely to avoid circularities >> like that? > > Perhaps I'm missing something, but won't that imply that every module > which wants to use a "special" module type has to re-invent the wheel? > > If this feature is going to be used, I would expect to be able to re-use > pre-written module types. E.g. having written "module with properties" > (so to speak) once, I can just import it and use it in my next project. I expect you'd package the special metamodule class in a stand-alone package, not directly in the ones that use it. You could import other packages freely, just the one that you're currently defining would be unavailable. From donald at stufft.io Sun Nov 30 01:37:17 2014 From: donald at stufft.io (Donald Stufft) Date: Sat, 29 Nov 2014 19:37:17 -0500 Subject: [Python-Dev] PEP 481 - Migrate Some Supporting Repositories to Git and Github In-Reply-To: References: <30725AC9-3DCF-416A-BCCD-2D64D489898C@stufft.io> Message-ID: <4ADD805B-B707-4244-9290-77AB0DAE396A@stufft.io> > On Nov 29, 2014, at 7:15 PM, Alex Gaynor wrote: > > Donald Stufft stufft.io> writes: > >> >> [words words words] >> > > I strongly support this PEP. I'd like to share two pieces of information. Both > of these are personal anecdotes: > > For the past several years, I've been a contributor on two major projects using > mercurial, CPython and PyPy. PyPy has a strong culture of in-repo branching, > basically all contributors regularly make branches in the main repo for their > work, and we're very free in giving people commit rights, so almost everyone > working on PyPy in any way has this level of access. This workflow works ok. I > don't love it as much as git, but it's fine, it's not an impediment to my work. > > By contrast, CPython does not have this type of workflow, there are almost no > in-tree branches besides the 2.7, 3.4, etc. ones. Despite being a regular hg > user for years, I have no idea how to create a local-only branch, or a branch > which is pushed to a remote (to use the git term). I also don't generally > commit my own work to CPython, even though I have push privledges, > because I > prefer to *always* get code review on my work. As a result, I use a git mirror > of CPython to do all my work, and generate patches from that. > > The conclusion I draw from this is that hg's workflow is probably fine if > you're a committer on the project, or don't ever need to maintain multiple > patches concurrently (and thus can just leave everything uncommitted in the > repo). However, the hg workflow seems extremely defficient at non-committer > contributors. I also don?t know how to do this. When I?m doing multiple things for CPython my ?branching? strategy is essentially using hg diff to create a patch file with my ?branch? name (``hg diff > my-branch.patch``), then revert all of my changes (``hg revert ?all ?no-backup``), then either work on a new ?branch? or switch to an old ?branch? by applying the corresponding patch (``patch -p1 < other-branch.patch``). > > The seconds experience I have is that of Django's migration to git and github. > For a long time we were on SVN, and we were very resistant to moving to > DVCS in > general, and github in particular. Multiple times I said that I didn't see how > exporting a patch and uploading it to trac was more difficult than sending a > pull request. That was very wrong on my part. > > My primary observation is not about new contributors though, it's actually > about the behavior of core developers. Before we were on github, it was fairly > rare for core developers to ask for reviews for anything besides *gigantic* > patches, we'd mostly just commit stuff to trunk. Since the switch to github, > I've seen that core developers are *far* more likely to ask for reviews of > their work before merging. I?ve also seen this effect, not just in Django but that I also notice my own behavior. Projects where I have commit access but which aren?t on Github I find myself less likely to look for others to review them and find myself just committing directly to master/default/trunk while I do tend to look for reviews and create PRs to give others the chance to review on Github. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA From solipsis at pitrou.net Sun Nov 30 01:43:31 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sun, 30 Nov 2014 01:43:31 +0100 Subject: [Python-Dev] PEP 481 - Migrate Some Supporting Repositories to Git and Github References: <30725AC9-3DCF-416A-BCCD-2D64D489898C@stufft.io> Message-ID: <20141130014331.31d99bb6@fsol> On Sun, 30 Nov 2014 00:15:55 +0000 (UTC) Alex Gaynor wrote: > > The seconds experience I have is that of Django's migration to git and github. > For a long time we were on SVN, and we were very resistant to moving to > DVCS in > general, and github in particular. Multiple times I said that I didn't see how > exporting a patch and uploading it to trac was more difficult than sending a > pull request. That was very wrong on my part. > > My primary observation is not about new contributors though, it's actually > about the behavior of core developers. Before we were on github, it was fairly > rare for core developers to ask for reviews for anything besides *gigantic* > patches, we'd mostly just commit stuff to trunk. Since the switch to github, > I've seen that core developers are *far* more likely to ask for reviews of > their work before merging. I don't know anything about Django's old SVN setup, but our code review tool (Rietveld) is actually quite good at that, and IMHO slightly better than github, since it will only send an e-mail at the final submission - by contrast, each individual comment you leave on github fires a separate notification, which can feel like spam. Our main problem for reviews these days is the lack of core developer time. For example, Serhiy often asks for reviews (he has many patches pending), which I personally don't have a lot of time to provide. Regards Antoine. From ncoghlan at gmail.com Sun Nov 30 02:12:17 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 30 Nov 2014 11:12:17 +1000 Subject: [Python-Dev] PEP 481 - Migrate Some Supporting Repositories to Git and Github In-Reply-To: <30725AC9-3DCF-416A-BCCD-2D64D489898C@stufft.io> References: <30725AC9-3DCF-416A-BCCD-2D64D489898C@stufft.io> Message-ID: On 30 Nov 2014 09:28, "Donald Stufft" wrote: > > As promised in the "Move selected documentation repos to PSF BitBucket > account?" thread I've written up a PEP for moving selected repositories from > hg.python.org to Github. > > You can see this PEP online at: https://www.python.org/dev/peps/pep-0481/ > > I've also reproduced the PEP below for inline discussion. Given that hg.python.org isn't going anywhere, you could also use hg-git to maintain read-only mirrors at the existing URLs and minimise any breakage (as well as ensuring a full historical copy remains available on PSF infrastructure). Then the only change needed would be to set up appropriate GitHub web hooks to replace anything previously based on a commit hook rather than periodic polling. The PEP should also cover providing clear instructions for setting up git-remote-hg with the remaining Mercurial repos (most notably CPython), as well as documenting a supported workflow for generating patches based on the existing CPython GitHub mirror. Beyond that, GitHub is indeed the most expedient option. My two main reasons for objecting to taking the expedient path are: 1. I strongly believe that the long term sustainability of the overall open source community requires the availability and use of open source infrastructure. While I admire the ingenuity of the "free-as-in-beer" model for proprietary software companies fending off open source competition, I still know a proprietary platform play when I see one (and so do venture capitalists looking to extract monopoly rents from the industry in the future). (So yes, I regret relenting on this principle in previously suggesting the interim use of another proprietary hosted service) 2. I also feel that this proposal is far too cavalier in not even discussing the possibility of helping out the Mercurial team to resolve their documentation and usability issues rather than just yelling at them "your tool isn't popular enough for us, and we find certain aspects of it too hard to use, so we're switching to something else rather than working with you to address our concerns". We consider the Mercurial team a significant enough part of the Python ecosystem that Matt was one of the folks specifically invited to the 2014 language summit to discuss their concerns around the Python 3 transition. Yet we'd prefer to switch to something else entirely rather than organising a sprint with them at PyCon to help ensure that our existing Mercurial based infrastructure is approachable for git & GitHub users? (And yes, I consider some of the core Mercurial devs to be friends, so this isn't an entirely abstract concern for me) Given my proposal to use BitBucket as a near term solution for enabling pull request based workflows, it's clear I consider the second argument the more significant of the two. However, if others consider some short term convenience that may or may not attract additional contributors more important than supporting the broader Python and open source communities (an argument I'm more used to hearing in the ruthlessly commercial environment of Red Hat, rather than in upstream contexts that tend to be less worried about "efficiency at any cost"), I'm not going to expend energy trying to prevent a change I disagree with on principle, but will instead work to eliminate (or significantly reduce) the current expedience argument in GitHub's favour. As a result, I'm -0 on the PEP, rather than -1 (and will try to stay out of further discussions). Given this proposal, I'm planning to refocus PEPs 474 & 462 specifically on resolving the CPython core workflow issues, since that will require infrastructure customisation regardless, and heavy customisation of GitHub based infrastructure requires opting in to the use of the GitHub specific APIs that create platform lockin. (Note that the argument in PEP 481 about saving overall infrastructure work is likely spurious - the vast majority of that work will be in addressing the complex CPython workflow requirements, and moving some support repos to GitHub does little to alleviate that) If folks decide they want to migrate the ancillary repos back from GitHub after that other infrastructure work is done, so be it, but if they don't, that's OK too. We're already running heterogeneous infrastructure across multiple services (especially if you also take PyPA into account), so having additional support repos externally hosted isn't that big a deal from a purely practical perspective. Regards, Nick. > > ----------------------- > > Abstract > ======== > > This PEP proposes migrating to Git and Github for certain supporting > repositories (such as the repository for Python Enhancement Proposals) in a way > that is more accessible to new contributors, and easier to manage for core > developers. This is offered as an alternative to PEP 474 which aims to achieve > the same overall benefits but while continuing to use the Mercurial DVCS and > without relying on a commerical entity. > > In particular this PEP proposes changes to the following repositories: > > * https://hg.python.org/devguide/ > * https://hg.python.org/devinabox/ > * https://hg.python.org/peps/ > > > This PEP does not propose any changes to the core development workflow for > CPython itself. > > > Rationale > ========= > > As PEP 474 mentions, there are currently a number of repositories hosted on > hg.python.org which are not directly used for the development of CPython but > instead are supporting or ancillary repositories. These supporting repositories > do not typically have complex workflows or often branches at all other than the > primary integration branch. This simplicity makes them very good targets for > the "Pull Request" workflow that is commonly found on sites like Github. > > However where PEP 474 wants to continue to use Mercurial and wishes to use an > OSS and self-hosted and therefore restricts itself to only those solutions this > PEP expands the scope of that to include migrating to Git and using Github. > > The existing method of contributing to these repositories generally includes > generating a patch and either uploading them to bugs.python.org or emailing > them to peps at python.org. This process is unfriendly towards non-comitter > contributors as well as making the process harder than it needs to be for > comitters to accept the patches sent by users. In addition to the benefits > in the pull request workflow itself, this style of workflow also enables > non techincal contributors, especially those who do not know their way around > the DVCS of choice, to contribute using the web based editor. On the committer > side the Pull Requests enable them to tell, before merging, whether or not > a particular Pull Request will break anything. It also enables them to do a > simple "push button" merge which does not require them to check out the > changes locally. Another such feature that is useful in particular for docs, > is the ability to view a "prose" diff. This Github specific feature enables > a committer to view a diff of the rendered output which will hide things like > reformatting a paragraph and show you what the actual "meat" of the change > actually is. > > > Why Git? > -------- > > Looking at the variety of DVCS which are available today it becomes fairly > clear that git has gotten the vast mindshare of people who are currently using > it. The Open Hub (Previously Ohloh) statistics [#openhub-stats]_ show that > currently 37% of the repositories Open Hub is indexing is using git which is > second only to SVN (which has 48%) while Mercurial has just 2% of the indexed > repositories (beating only bazaar which has 1%). In additon to the Open Hub > statistics a look at the top 100 projects on PyPI (ordered by total download > counts) shows us that within the Python space itself there is a majority of > projects using git: > > === ========= ========== ====== === ==== > Git Mercurial Subversion Bazaar CVS None > === ========= ========== ====== === ==== > 62 22 7 4 1 1 > === ========= ========== ====== === ==== > > > Chosing a DVCS which has the larger mindshare will make it more likely that any > particular person who has experience with DVCS at all will be able to > meaningfully use the DVCS that we have chosen without having to learn a new > tool. > > In addition to simply making it more likely that any individual will already > know how to use git, the number of projects and people using it means that the > resources for learning the tool are likely to be more fully fleshed out and > when you run into problems the liklihood that someone else had that problem > and posted a question and recieved an answer is also far likelier. > > Thirdly by using a more popular tool you also increase your options for tooling > *around* the DVCS itself. Looking at the various options for hosting > repositories it's extremely rare to find a hosting solution (whether OSS or > commerical) that supports Mercurial but does not support Git, on the flip side > there are a number of tools which support Git but do not support Mercurial. > Therefore the popularity of git increases the flexibility of our options going > into the future for what toolchain these projects use. > > Also by moving to the more popular DVCS we increase the likelhood that the > knowledge that the person has learned in contributing to these support > repositories will transfer to projects outside of the immediate CPython project > such as to the larger Python community which is primarily using Git hosted on > Github. > > In previous years there was concern about how well supported git was on Windows > in comparison to Mercurial. However git has grown to support Windows as a first > class citizen. In addition to that, for Windows users who are not well aquanted > with the Windows command line there are GUI options as well. > > On a techincal level git and Mercurial are fairly similar, however the git > branching model is signifcantly better than Mercurial "Named Branches" for > non-comitter contributors. Mercurial does have a "Bookmarks" extension however > this isn't quite as good as git's branching model. All bookmarks live in the > same namespace so it requires individual users to ensure that they namespace > the branchnames themselves lest the risk collision. It also is an extension > which requires new users to first discover they need an extension at all and > then figure out what they need to do in order to enable that extension. Since > it is an extension it also means that in general support for them outside of > Mercurial core is going to be less than 100% in comparison to git where the > feature is built in and core to using git at all. Finally users who are not > used to Mercurial are unlikely to discover bookmarks on their own, instead they > will likely attempt to use Mercurial's "Named Branches" which, given the fact > they live "forever", are not often what a project wants their contributors to > use. > > > Why Github? > ----------- > > There are a number of software projects or web services which offer > functionality similar to that of Github. These range from commerical web > services such as a Bitbucket to self-hosted OSS solutions such as Kallithea or > Gitlab. This PEP proposes that we move these repositories to Github. > > There are two primary reasons for selecting Github: Popularity and > Quality/Polish. > > Github is currently the most popular hosted repository hosting according to > Alexa where it currently has a global rank of 121. Much like for Git itself by > choosing the most popular tool we gain benefits in increasing the likelhood > that a new contributor will have already experienced the toolchain, the quality > and availablity of the help, more and better tooling being built around it, and > the knowledge transfer to other projects. A look again at the top 100 projects > by download counts on PyPI shows the following hosting locations: > > ====== ========= =========== ========= =========== ========== > GitHub BitBucket Google Code Launchpad SourceForge Other/Self > ====== ========= =========== ========= =========== ========== > 62 18 6 4 3 7 > ====== ========= =========== ========= =========== ========== > > In addition to all of those reasons, Github also has the benefit that while > many of the options have similar features when you look at them in a feature > matrix the Github version of each of those features tend to work better and be > far more polished. This is hard to quantify objectively however it is a fairly > common sentiment if you go around and ask people who are using these services > often. > > Finally a reason to choose a web service at all over something that is > self-hosted is to be able to more efficiently use volunteer time and donated > resources. Every additional service hosted on the PSF infrastruture by the > PSF infrastructure team further spreads out the amount of time that the > volunteers on that team have to spend and uses some chunk of resources that > could potentionally be used for something where there is no free or affordable > hosted solution available. > > One concern that people do have with using a hosted service is that there is a > lack of control and that at some point in the future the service may no longer > be suitable. It is the opinion of this PEP that Github does not currently and > has not in the past engaged in any attempts to lock people into their platform > and that if at some point in the future Github is no longer suitable for one > reason or another than at that point we can look at migrating away from Github > onto a different solution. In other words, we'll cross that bridge if and when > we come to it. > > > Example: Scientific Python > -------------------------- > > One of the key ideas behind the move to both git and Github is that a feature > of a DVCS, the repository hosting, and the workflow used is the social network > and size of the community using said tools. We can see this is true by looking > at an example from a sub-community of the Python community: The Scientific > Python community. They have already migrated most of the key pieces of the > SciPy stack onto Github using the Pull Request based workflow starting with > IPython and as more projects moved over it became a natural default for new > projects. > > They claim to have seen a great benefit from this move, where it enables casual > contributors to easily move between different projects within their > sub-community without having to learn a special, bespoke workflow and a > different toolchain for each project. They've found that when people can use > their limited time on actually contributing instead of learning the different > tools and workflows that not only do they contribute more to one project, that > they also expand out and contribute to other projects. This move is also > attributed to making it commonplace for members of that community to go so far > as publishing their research and educational materials on Github as well. > > This showcases the real power behind moving to a highly popular toolchain and > workflow, as each variance introduces yet another hurdle for new and casual > contributors to get past and it makes the time spent learning that workflow > less reusable with other projects. > > > Migration > ========= > > Through the use of hg-git [#hg-git]_ we can easily convert a Mercurial > repository to a Git repository by simply pushing the Mercurial repository to > the Git repository. People who wish to continue to use Mercurual locally can > then use hg-git going into the future using the new Github URL, however they > will need to re-clone their repositories as using Git as the server seems to > trigger a one time change of the changeset ids. > > As none of the selected repositories have any tags, branches, or bookmarks > other than the ``default`` branch the migration will simply map the ``default`` > branch in Mercurial to the ``master`` branch in git. > > In addition since none of the selected projects have any great need of a > complex bug tracker, they will also migrate their issue handling to using the > GitHub issues. > > In addition to the migration of the repository hosting itself there are a > number of locations for each particular repository which will require updating. > The bulk of these will simply be changing commands from the hg equivilant to > the git equivilant. > > In particular this will include: > > * Updating www.python.org to generate PEPs using a git clone and link to > Github. > * Updating docs.python.org to pull from Github instead of hg.python.org for the > devguide. > * Enabling the ability to send an email to python-checkins at python.org for each > push. > * Enabling the ability to send an IRC message to #python-dev on Freenode for > each push. > * Migrate any issues for these projects to their respective bug tracker on > Github. > > This will restore these repositories to similar functionality as they currently > have. In addition to this the migration will also include enabling testing for > each pull request using Travis CI [#travisci]_ where possible to ensure that > a new PR does not break the ability to render the documentation or PEPs. > > > User Access > =========== > > Moving to Github would involve adding an additional user account that will need > to be managed, however it also offers finer grained control, allowing the > ability to grant someone access to only one particular repository instead of > the coarser grained ACLs available on hg.python.org. > > > References > ========== > > .. [#openhub-stats] `Open Hub Statistics < https://www.openhub.net/repositories/compare>` > .. [#hg-git] `hg-git ` > .. [#travisci] `Travis CI ` > > --- > Donald Stufft > PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Sun Nov 30 02:13:20 2014 From: donald at stufft.io (Donald Stufft) Date: Sat, 29 Nov 2014 20:13:20 -0500 Subject: [Python-Dev] PEP 481 - Migrate Some Supporting Repositories to Git and Github In-Reply-To: <20141130014331.31d99bb6@fsol> References: <30725AC9-3DCF-416A-BCCD-2D64D489898C@stufft.io> <20141130014331.31d99bb6@fsol> Message-ID: > On Nov 29, 2014, at 7:43 PM, Antoine Pitrou wrote: > > On Sun, 30 Nov 2014 00:15:55 +0000 (UTC) > Alex Gaynor wrote: >> >> The seconds experience I have is that of Django's migration to git and github. >> For a long time we were on SVN, and we were very resistant to moving to >> DVCS in >> general, and github in particular. Multiple times I said that I didn't see how >> exporting a patch and uploading it to trac was more difficult than sending a >> pull request. That was very wrong on my part. >> >> My primary observation is not about new contributors though, it's actually >> about the behavior of core developers. Before we were on github, it was fairly >> rare for core developers to ask for reviews for anything besides *gigantic* >> patches, we'd mostly just commit stuff to trunk. Since the switch to github, >> I've seen that core developers are *far* more likely to ask for reviews of >> their work before merging. > > I don't know anything about Django's old SVN setup, but our code review > tool (Rietveld) is actually quite good at that, and IMHO slightly > better than github, since it will only send an e-mail at the final > submission - by contrast, each individual comment you leave on github > fires a separate notification, which can feel like spam. > > Our main problem for reviews these days is the lack of core developer > time. For example, Serhiy often asks for reviews (he has many patches > pending), which I personally don't have a lot of time to provide. > I think one of the issues with Reitveld isn?t related to Reitveld itself at all, it?s all the *other* stuff you have to do to get a patch into Reitveld to allow someone to review it at all. Generating a patch and uploading it to Roundup is a pain and it?s far easier to just commit things directly to default. As far as Reitveld vs Github itself goes for reviews, I don?t personally agree. Sending a single notification vs a notification per comment is going to be a very subjective point of view, I like more notifications for the simple fact that if I'm actively working on a PR while someone is reviewing it I can fix the issues as quickly as they are finding them. However if you move past that one thing, the Github PR review has a number of benefits over Reitveld itself. Since the repositories this PEP currently deals with are largely documentation-esque repositories the "Prose" diff is incredibly useful for viewing a content-based diff instead of a diff designed more for software where you don't normally reflow lines as much. In addition the Github diff viewer also makes it trivial to expand the context of the diff so you can see more of the surrounding file, even allowing you to expand it to the entire file. This is super useful when the automatic amount of context can't really give you enough information for reviewing the change or not. Another difference is in how the review comments are presented. With Reitveld the inline comments go away for each patchset you upload, regardless of if you've addressed the concerns in that comment or not. They do get bubbled up into the overall "Messages" view, however this has the opposite problem where it gives you all of the messages regardless of if they are still a problem with the latest code or not. In contrast the Github PR will hide old comments when they lines they addressed have changed, but still make it easy to see the old comments and the context in which they were made. There's also the UI itself, it's somewhat dated and I think it's not entirely unobvious that it suffers from something a lot of OSS projects suffer from in that it is a "Developer" made UI. I don't think it's a big secret that developers tend to make UIs that are not as nice as proper UX folks make, but OSS has historically had a problem attracting that kind of talent (and is often openly hostile towards it). Finally there's the transferability of knowledge at play too. I likely would not review someone else's patch unless I felt strongly about it on Reitveld largely because I don't know Reitveld very well and I'd rather spend the time that I'd need to learn to use it effectively working on other projects where I already know the entire toolchain well. This is one of the larger points in this PEP that the benefits to using popular tooling is that not only does people's existing knowledge transfer easily into your project, but the time they spend learning your tooling also transfers easily outside of this project. This makes it much more attractive to learn the tooling since the hypothetical person would be able to take that knowledge and apply it elsewhere. It is my experience, and this entirely ancedotal, that it's far easier to get reviews from non-committers and committers alike on projects which are hosted on Github than it is to get reviews on projects which are not hosted there. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA From rosuav at gmail.com Sun Nov 30 02:15:31 2014 From: rosuav at gmail.com (Chris Angelico) Date: Sun, 30 Nov 2014 12:15:31 +1100 Subject: [Python-Dev] PEP 481 - Migrate Some Supporting Repositories to Git and Github In-Reply-To: <4ADD805B-B707-4244-9290-77AB0DAE396A@stufft.io> References: <30725AC9-3DCF-416A-BCCD-2D64D489898C@stufft.io> <4ADD805B-B707-4244-9290-77AB0DAE396A@stufft.io> Message-ID: On Sun, Nov 30, 2014 at 11:37 AM, Donald Stufft wrote: > I also don?t know how to do this. When I?m doing multiple things for CPython > my ?branching? strategy is essentially using hg diff to create a patch file > with my ?branch? name (``hg diff > my-branch.patch``), then revert all of my > changes (``hg revert ?all ?no-backup``), then either work on a new ?branch? > or switch to an old ?branch? by applying the corresponding patch > (``patch -p1 < other-branch.patch``). IMO, this is missing out on part of the benefit of a DVCS. When your patches are always done purely on the basis of files, and have to be managed separately, everything will be manual; and your edits won't (normally) contain commit messages, authorship headers, date/time stamps, and all the other things that a commit will normally have. Using GitHub automatically makes all that available; when someone forks the project and adds a commit, that commit will exist and have its full identity, metadata, etc, and if/when it gets merged into trunk, all that will be carried through automatically. I strongly support this PEP. ChrisA From donald at stufft.io Sun Nov 30 03:01:18 2014 From: donald at stufft.io (Donald Stufft) Date: Sat, 29 Nov 2014 21:01:18 -0500 Subject: [Python-Dev] PEP 481 - Migrate Some Supporting Repositories to Git and Github In-Reply-To: References: <30725AC9-3DCF-416A-BCCD-2D64D489898C@stufft.io> Message-ID: <9BC1BE36-8A24-4AEA-A399-B2FE61A8BBF3@stufft.io> > On Nov 29, 2014, at 8:12 PM, Nick Coghlan wrote: > > > On 30 Nov 2014 09:28, "Donald Stufft" > wrote: > > > > As promised in the "Move selected documentation repos to PSF BitBucket > > account?" thread I've written up a PEP for moving selected repositories from > > hg.python.org to Github. > > > > You can see this PEP online at: https://www.python.org/dev/peps/pep-0481/ > > > > I've also reproduced the PEP below for inline discussion. > > Given that hg.python.org isn't going anywhere, you could also use hg-git to maintain read-only mirrors at the existing URLs and minimise any breakage (as well as ensuring a full historical copy remains available on PSF infrastructure). Then the only change needed would be to set up appropriate GitHub web hooks to replace anything previously based on a commit hook rather than periodic polling. > > Ah yes, I meant to include that and just forgot to do it when I went to test hg-git to see how well it worked and whether I got different commit hashes on different machines. I also thought about adding a git.python.org which just acted as a read-only mirror of what was on Github, but I don?t know if that?s actually generally useful or not. > The PEP should also cover providing clear instructions for setting up git-remote-hg with the remaining Mercurial repos (most notably CPython), as well as documenting a supported workflow for generating patches based on the existing CPython GitHub mirror. > > I can add this. I?ve never actually tried using git-remote-hg with CPython itself because I?ve made it segfault on other Mercurial repositories and I never figured out why so I just generally fight my way through using Mercurial on projects that themselves use Mercurial. I will absolutely test to see if git-remote-hg works with CPython and I can document using that to contribute to CPython. I?m not sure it needs to be part of the PEP or not? Feels like something that would be better inside the devguide itself but I?m not opposed to putting it both locations. > Beyond that, GitHub is indeed the most expedient option. My two main reasons for objecting to taking the expedient path are: > It's not entirely about expedience. I think a lot of the reason why we should look towards outsourcing some of these items is that volunteer time is not a fungible resource. Volunteers are generally only willing to work on things which they personally care about. This is entirely unlike a business where you have employees who will generally work on whatever you tell them to because that's what you're paying them for. To this end I personally don't really have an interest in trying to create a better code hosting platform than Github when Github is doing an amazing job in my opinion and they satisify my needs fine. Given the *current* state of tooling it appears that there are not a lot of people who both care about making that piece of software exist and are capable of competing with Github in terms of quality. > 1. I strongly believe that the long term sustainability of the overall open source community requires the availability and use of open source infrastructure. While I admire the ingenuity of the "free-as-in-beer" model for proprietary software companies fending off open source competition, I still know a proprietary platform play when I see one (and so do venture capitalists looking to extract monopoly rents from the industry in the future). (So yes, I regret relenting on this principle in previously suggesting the interim use of another proprietary hosted service) > > I somewhat agree. However I?m less concerned specifically about where projects are hosted exactly and more about the *ability* to move to a completely OSS infrastructure. In particular if at somepoint we need to move off of Github we can totally do that, it?s not particularly difficult. Currently you lose the higher quality polish of Github if you do that however if at some point in the future Github either turns evil or an OSS software offers a truly compelling alternative to Github then there is really nothing stopping a migration to another platform. As I said in the PEP I view this as a ?let?s cross that bridge if/when we get to it?. The main thing we should look at is things that would be difficult to migrate away from. For code hosting in particular most of the truly valuable data is stored within the DVCS so migrating the bulk of the data is as simple as pushing the repository to a new location. The other data is within the issues, for these repositories I suggest moving the issues to Github entirely because I don?t suspect they?ll get many if any issues themselves so the amount of data stored in issues will be low. However I also think that long term sustainability of any particular project depends on attracting and retaining contributors. To this end going to where the people are already and paving inroads that reduce the barriers to contributing is an important thing to do as well. This proposal is aimed squarely at reducing barriers to contribution while also giving a nod to the first concern by selecting a platform that has done a lot to enable OSS and doing it in a way that the ability to leave that platform is maintained so that in some future we can migrate away if need be. > 2. I also feel that this proposal is far too cavalier in not even discussing the possibility of helping out the Mercurial team to resolve their documentation and usability issues rather than just yelling at them "your tool isn't popular enough for us, and we find certain aspects of it too hard to use, so we're switching to something else rather than working with you to address our concerns". We consider the Mercurial team a significant enough part of the Python ecosystem that Matt was one of the folks specifically invited to the 2014 language summit to discuss their concerns around the Python 3 transition. Yet we'd prefer to switch to something else entirely rather than organising a sprint with them at PyCon to help ensure that our existing Mercurial based infrastructure is approachable for git & GitHub users? (And yes, I consider some of the core Mercurial devs to be friends, so this isn't an entirely abstract concern for me) > I was on the fence about including the bit about branches in the PEP itself and I ended up doing it only because multiple people brought it up when I asked them for a review. I also tried not to include the fact that I haven?t personally figured out how to actually use Mercurial effectively in the PEP itself because honestly I don?t think that is really the core idea behind moving to git and Github. I think if you look at any semi objective measurement between git and Mercurial?s effective popularity git is going to be a clear winner, and if you do the same thing for Github compared to any other code hosting service or software then Github wins that by any even larger margin. The reason the PEP primarily focuses on the popularity of the the tool is because as you mentioned, issues like poor documentation, bad support for a particular platform, a particular workflow not being very good can be solved by working with the tool authors to solve that particular problem. I wouldn?t consider those issues in a vacuum to be a good reason to migrate away from that tool. However there?s very little that CPython can do to get more people using Mercurial, and presumably the authors of Mercurial are already doing what they can to get people to use them. However by choosing, and continuing to choose, a tool that an order of magnitude less people use, we?re choosing to effectively make it harder to contribute. We?re increasing the likelhood that a contributor is going to have to learn our particular DVCS even if they already know one and we?re increasing the likelhood that we?re burdening users to learn a technology that isn?t going to transfer to most other projects that they might want to contribute to, even within the Python ecosystem itself. Even the suggestion of a way to make it approachable for git and Github users still acts as a barrier to contribution. Now it?s true that when you have to select against competing tools the tool you choose is going to be a barrier to *someone*. For example if we do choose to go with git, then that tool will be a barrier to people who already know Mercurial but don?t already know git. However by selecting the most popular tool you ensure that the particular barrier is a barrier to as few as people as possible. To be clear, I don?t consider the technical differences between Mercurial and git to be very large hurdles to overcome, it?s primarily about the mindshare and the fact that Mercurial doesn?t really enable us to do much that git doesn?t also allow us to do (other than support friends, which isn?t an unreasonable desire either) while keeping us from tapping into the collective power of the number of git users. To put it another way, Linux vs the BSDs, I actually prefer FreeBSD over Linux, but I almost never use it because Linux is vastly more popular and by selecting it I?m more likely to be able to transfer that knowledge to different scenarios and find help when I have a problem. > Given my proposal to use BitBucket as a near term solution for enabling pull request based workflows, it's clear I consider the second argument the more significant of the two. > > However, if others consider some short term convenience that may or may not attract additional contributors more important than supporting the broader Python and open source communities (an argument I'm more used to hearing in the ruthlessly commercial environment of Red Hat, rather than in upstream contexts that tend to be less worried about "efficiency at any cost"), I'm not going to expend energy trying to prevent a change I disagree with on principle, but will instead work to eliminate (or significantly reduce) the current expedience argument in GitHub's favour. > > > As a result, I'm -0 on the PEP, rather than -1 (and will try to stay out of further discussions). > > Given this proposal, I'm planning to refocus PEPs 474 & 462 specifically on resolving the CPython core workflow issues, since that will require infrastructure customisation regardless, and heavy customisation of GitHub based infrastructure requires opting in to the use of the GitHub specific APIs that create platform lockin. (Note that the argument in PEP 481 about saving overall infrastructure work is likely spurious - the vast majority of that work will be in addressing the complex CPython workflow requirements, and moving some support repos to GitHub does little to alleviate that > That was specifically about saving infrastructure work to supporting Kallithea (or whatever solution). I don?t suspect that CPython was planning on using Kallithea (maybe I?m wrong?) so forge.python.org was primarily aimed towards the non-CPython repositories. > If folks decide they want to migrate the ancillary repos back from GitHub after that other infrastructure work is done, so be it, but if they don't, that's OK too. We're already running heterogeneous infrastructure across multiple services (especially if you also take PyPA into account), so having additional support repos externally hosted isn't that big a deal from a purely practical perspective. > --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: From jimjjewett at gmail.com Sun Nov 30 03:04:21 2014 From: jimjjewett at gmail.com (Jim J. Jewett) Date: Sat, 29 Nov 2014 21:04:21 -0500 Subject: [Python-Dev] PEP 479 Message-ID: I have a strong suspicion that I'm missing something; I have been persuaded both directions too often to believe I have a grip on the real issue. So I'm putting out some assumptions; please tell me if I'm wrong, and maybe make them more explicit in the PEP. (1) The change will only affect situations where StopIteration is currently raised as an Exception -- i.e., it leaks past the bounds of a loop. (2) This can happen because of an explicit raise StopIteration. This is currently a supported idiom, and that is changing with PEP 479. (2a) Generators in the unwind path will now need to catch and reraise. (3) It can also happen because of an explicit next statement (as opposed the the implicit next of a loop). This is currently supported; after PEP 479, the next statement should be wrapped in a try statement, so that the intent will be explicit. (4) It can happen because of "yield from" yielding from an iterator, rather than a generator? (5) There is no other case where this can happen? (So the generator comprehension case won't matter unless it also includes one of the earlier cases.) -jJ From rosuav at gmail.com Sun Nov 30 03:31:33 2014 From: rosuav at gmail.com (Chris Angelico) Date: Sun, 30 Nov 2014 13:31:33 +1100 Subject: [Python-Dev] PEP 479 In-Reply-To: References: Message-ID: On Sun, Nov 30, 2014 at 1:04 PM, Jim J. Jewett wrote: > I have a strong suspicion that I'm missing something; I have been > persuaded both directions too often to believe I have a grip on the > real issue. > > So I'm putting out some assumptions; please tell me if I'm wrong, and > maybe make them more explicit in the PEP. > > (1) The change will only affect situations where StopIteration is > currently raised as an Exception -- i.e., it leaks past the bounds of > a loop. Where a StopIteration would come up out of the generator. Inside the generator function, it's exactly the same as it is in any other function; you can raise it, you can catch it, everything's normal. > (2) This can happen because of an explicit raise StopIteration. This > is currently a supported idiom, and that is changing with PEP 479. Correct. There is nothing that explicitly-raised StopIteration can do in 3.0-3.4 that a return statement can't do in 3.0-3.7. There is the downside that "raise StopIteration(value)" works on 2.7 where "return value" is a syntax error; the PEP currently has no solution for this. > (2a) Generators in the unwind path will now need to catch and reraise. More likely, catch and return; if your code was allowing "next(iter)" to have the effect of potentially terminating the function, then you now have to spell that "try: next(iter); except StopIteration: return", which makes it clear that there's control flow here. > (3) It can also happen because of an explicit next statement (as > opposed the the implicit next of a loop). > This is currently supported; after PEP 479, the next statement should > be wrapped in a try statement, so that the intent will be explicit. Correct, as per previous point. As you say, the intent will be explicit: take a value, and if there aren't any more, stop processing. > (4) It can happen because of "yield from" yielding from an iterator, > rather than a generator? No; as I understand it (though maybe I'm wrong too), "yield from" will yield every value the other iterator yields, and will then quietly emit a value if the iterator raises StopIteration, or will allow any other exception to propagate. The StopIteration coming from the iterator is absorbed by the "yield from" construct. To completely propagate it out, "return (yield from iter)" should cover all three results (yielded value, returned value, raised exception). > (5) There is no other case where this can happen? (So the generator > comprehension case won't matter unless it also includes one of the > earlier cases.) Correct. In a generator expression (I assume that's what you mean?), the most likely way to leak a StopIteration is the "or stop()" hack, which has always been at least slightly dubious, and is now going to be actively rejected. Control flow in a generator expression is now the same as in a comprehension, with no early-abort option; if you want that, the best way is to break the expression into an out-of-line generator function. This is now very similar to the restrictions on lambda; you can't (eg) raise exceptions in a lambda function, and if anyone comes to python-list asking how to do so, the best response is "use def instead of lambda". ChrisA From guido at python.org Sun Nov 30 03:54:02 2014 From: guido at python.org (Guido van Rossum) Date: Sat, 29 Nov 2014 18:54:02 -0800 Subject: [Python-Dev] advice needed: best approach to enabling "metamodules"? In-Reply-To: References: Message-ID: All the use cases seem to be about adding some kind of getattr hook to modules. They all seem to involve modifying the CPython C code anyway. So why not tackle that problem head-on and modify module_getattro() to look for a global named __getattr__ and if it exists, call that instead of raising AttributeError? On Sat, Nov 29, 2014 at 11:37 AM, Nathaniel Smith wrote: > On Sat, Nov 29, 2014 at 4:21 AM, Guido van Rossum > wrote: > > Are these really all our options? All of them sound like hacks, none of > them > > sound like anything the language (or even the CPython implementation) > should > > sanction. Have I missed the discussion where the use cases and > constraints > > were analyzed and all other approaches were rejected? (I might have some > > half-baked ideas, but I feel I should read up on the past discussion > first, > > and they are probably more fit for python-ideas than for python-dev. Plus > > I'm just writing this email because I'm procrastinating on the type > hinting > > PEP. :-) > > The previous discussions I was referring to are here: > http://thread.gmane.org/gmane.comp.python.ideas/29487/focus=29555 > http://thread.gmane.org/gmane.comp.python.ideas/29788 > > There might well be other options; these are just the best ones I > could think of :-). The constraints are pretty tight, though: > - The "new module" object (whatever it is) should have a __dict__ that > aliases the original module globals(). I can elaborate on this if my > original email wasn't enough, but hopefully it's obvious that making > two copies of the same namespace and then trying to keep them in sync > at the very least smells bad :-). > - The "new module" object has to be a subtype of ModuleType, b/c there > are lots of places that do isinstance(x, ModuleType) checks (notably > -- but not only -- reload()). Since a major goal here is to make it > possible to do cleaner deprecations, it would be really unfortunate if > switching an existing package to use the metamodule support itself > broke things :-). > - Lookups in the normal case should have no additional performance > overhead, because module lookups are extremely extremely common. (So > this rules out dict proxies and tricks like that -- we really need > 'new_module.__dict__ is globals()' to be true.) > > AFAICT there are three logically possible strategies for satisfying > that first constraint: > (a) convert the original module object into the type we want, in-place > (b) create a new module object that acts like the original module object > (c) somehow arrange for our special type to be used from the start > > My options 1 and 2 are means of accomplishing (a), and my options 3 > and 4 are means of accomplishing (b) while working around the > behavioural quirks of module objects (as required by the second > constraint). > > The python-ideas thread did also consider several methods of > implementing strategy (c), but they're messy enough that I left them > out here. The problem is that somehow we have to execute code to > create the new subtype *before* we have an entry in sys.modules for > the package that contains the code for the subtype. So one option > would be to add a new rule, that if a file pkgname/__new__.py exists, > then this is executed first and is required to set up > sys.modules["pkgname"] before we exec pkgname/__init__.py. So > pkgname/__new__.py might look like: > > import sys > from pkgname._metamodule import MyModuleSubtype > sys.modules[__name__] = MyModuleSubtype(__name__, docstring) > > This runs into a lot of problems though. To start with, the 'from > pkgname._metamodule ...' line is an infinite loop, b/c this is the > code used to create sys.modules["pkgname"]. It's not clear where the > globals dict for executing __new__.py comes from (who defines > __name__? Currently that's done by ModuleType.__init__). It only works > for packages, not modules. The need to provide the docstring here, > before __init__.py is even read, is weird. It adds extra stat() calls > to every package lookup. And, the biggest showstopper IMHO: AFAICT > it's impossible to write a polyfill to support this code on old python > versions, so it's useless to any package which needs to keep > compatibility with 2.7 (or even 3.4). Sure, you can backport the whole > import system like importlib2, but telling everyone that they need to > replace every 'import numpy' with 'import importlib2; import numpy' is > a total non-starter. > > So, yeah, those 4 options are really the only plausible ones I know of. > > Option 1 and option 3 are pretty nice at the language level! Most > Python objects allow assignment to __class__ and __dict__, and both > PyPy and Jython at least do support __class__ assignment. Really the > only downside with Option 1 is that actually implementing it requires > attention from someone with deep knowledge of typeobject.c. > > -n > > -- > Nathaniel J. Smith > Postdoctoral researcher - Informatics - University of Edinburgh > http://vorpus.org > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/guido%40python.org > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Sun Nov 30 04:43:39 2014 From: donald at stufft.io (Donald Stufft) Date: Sat, 29 Nov 2014 22:43:39 -0500 Subject: [Python-Dev] PEP 481 - Migrate Some Supporting Repositories to Git and Github In-Reply-To: <9BC1BE36-8A24-4AEA-A399-B2FE61A8BBF3@stufft.io> References: <30725AC9-3DCF-416A-BCCD-2D64D489898C@stufft.io> <9BC1BE36-8A24-4AEA-A399-B2FE61A8BBF3@stufft.io> Message-ID: <61376849-BD78-4642-8F8C-D3AAF03E3772@stufft.io> > On Nov 29, 2014, at 9:01 PM, Donald Stufft wrote: > >> >> The PEP should also cover providing clear instructions for setting up git-remote-hg with the remaining Mercurial repos (most notably CPython), as well as documenting a supported workflow for generating patches based on the existing CPython GitHub mirror. >> >> > > I can add this. I?ve never actually tried using git-remote-hg with CPython > itself because I?ve made it segfault on other Mercurial repositories and I > never figured out why so I just generally fight my way through using Mercurial > on projects that themselves use Mercurial. I will absolutely test to see if > git-remote-hg works with CPython and I can document using that to contribute to > CPython. I?m not sure it needs to be part of the PEP or not? Feels like > something that would be better inside the devguide itself but I?m not opposed > to putting it both locations. Nevermind. I?m not going to add this because git-remote-hg is busted with the latest version of Mercurial and I don?t think it?s really needed for this PEP (though would be good for the devguide at some point). --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: From ben+python at benfinney.id.au Sun Nov 30 06:06:47 2014 From: ben+python at benfinney.id.au (Ben Finney) Date: Sun, 30 Nov 2014 16:06:47 +1100 Subject: [Python-Dev] PEP 481 - Migrate Some Supporting Repositories to Git and Github References: <30725AC9-3DCF-416A-BCCD-2D64D489898C@stufft.io> Message-ID: <85d285wa3c.fsf@benfinney.id.au> Nick Coghlan writes: > 1. I strongly believe that the long term sustainability of the overall > open source community requires the availability and use of open source > infrastructure. I concur. This article makes the arguments well, IMO. > 2. I also feel that this proposal is far too cavalier in not even > discussing the possibility of helping out the Mercurial team [?] we'd > prefer to switch to something else entirely rather than organising a > sprint with them at PyCon to help ensure that our existing Mercurial > based infrastructure is approachable for git & GitHub users? Exactly. For such a core tool, instead of pushing proprietary platforms at the expense of software freedom, the sensible strategy for a project (Python) that hopes to be around in the long term is to use and improve the free software platforms. > As a result, I'm -0 on the PEP, rather than -1 (and will try to stay > out of further discussions). I don't get a vote. So I'm glad there are some within the Python core development team that can see the mistakes inherent in depending on non-free tools for developing free software. -- \ ?The cost of a thing is the amount of what I call life which is | `\ required to be exchanged for it, immediately or in the long | _o__) run.? ?Henry David Thoreau | Ben Finney From rosuav at gmail.com Sun Nov 30 06:23:08 2014 From: rosuav at gmail.com (Chris Angelico) Date: Sun, 30 Nov 2014 16:23:08 +1100 Subject: [Python-Dev] PEP 481 - Migrate Some Supporting Repositories to Git and Github In-Reply-To: <85d285wa3c.fsf@benfinney.id.au> References: <30725AC9-3DCF-416A-BCCD-2D64D489898C@stufft.io> <85d285wa3c.fsf@benfinney.id.au> Message-ID: On Sun, Nov 30, 2014 at 4:06 PM, Ben Finney wrote: > I don't get a vote. So I'm glad there are some within the Python core > development team that can see the mistakes inherent in depending on > non-free tools for developing free software. While this is a laudable view, this kind of extreme stance is contrary to any semblance of practicality. Compare: http://www.gnu.org/distros/free-distros.html http://www.gnu.org/distros/common-distros.html#Debian Debian is not considered sufficiently free because "people can readily learn about these nonfree packages by browsing Debian's online package database", even though you have to be very much explicit about these things (you have to go and enable the non-free repos). Yes, GitHub is proprietary. But all of your actual code is stored in git, which is free, and it's easy to push that to a new host somewhere else, or create your own host. This proposal is for repositories that don't need much in the way of issue trackers etc, so shifting away from GitHub shouldn't demand anything beyond moving the repos themselves. How bad is it, really? Is it worth fighting a philosophical battle for the sake of no real gain, sacrificing real benefits for the intangible "but it's not free" debate? Python is already using quite a bit of non-free software in its ecosystem. The Windows builds of CPython are made with Microsoft's compiler, and the recent discussion about shifting to Cygwin or MinGW basically boiled down to "but it ought to be free software", and that was considered not a sufficiently strong argument. In each case, the decision has impact on other people (using MSVC for the official python.org installers means extension writers need to use MSVC too; and using GitHub means that contributors are strongly encouraged, possibly required, to use GitHub); so why is it acceptable to use a non-free compiler, but not acceptable to use a non-free host? I admire and respect the people who, for their own personal use, absolutely and utterly refuse to use any non-free systems or software. It's great that they do it, because that helps encourage free software to be created. But for myself? I'll use whatever makes the most sense. Proprietary systems have inherent issues (the best-maintained non-free programs seem to have about the same bugginess as a poorly-maintained free program, or at least that's how it feels), but if the available free alternatives have even more issues, I'll not hobble myself for the purity of freedom. Practicality wins. ChrisA From donald at stufft.io Sun Nov 30 06:27:43 2014 From: donald at stufft.io (Donald Stufft) Date: Sun, 30 Nov 2014 00:27:43 -0500 Subject: [Python-Dev] PEP 481 - Migrate Some Supporting Repositories to Git and Github In-Reply-To: <85d285wa3c.fsf@benfinney.id.au> References: <30725AC9-3DCF-416A-BCCD-2D64D489898C@stufft.io> <85d285wa3c.fsf@benfinney.id.au> Message-ID: <7F8618A9-3167-4BF0-8F98-CCF9DA8539BC@stufft.io> > On Nov 30, 2014, at 12:06 AM, Ben Finney wrote: > > Nick Coghlan writes: > >> 1. I strongly believe that the long term sustainability of the overall >> open source community requires the availability and use of open source >> infrastructure. > > I concur. This article > makes the arguments well, IMO. > >> 2. I also feel that this proposal is far too cavalier in not even >> discussing the possibility of helping out the Mercurial team [?] we'd >> prefer to switch to something else entirely rather than organising a >> sprint with them at PyCon to help ensure that our existing Mercurial >> based infrastructure is approachable for git & GitHub users? > > Exactly. For such a core tool, instead of pushing proprietary platforms > at the expense of software freedom, the sensible strategy for a project > (Python) that hopes to be around in the long term is to use and improve > the free software platforms. I think there is a big difference here between using a closed source VCS or compiler and using a closed source code host. Namely in that the protocol is defined by git so switching from one host to another is easy. It?s akin to saying that if we chose to run the PyPI services on a Windows machine that it is somehow makes it less-free even though we could have chosen to run it on a ?free? OS and we weren?t doing much, if anything, to tie us to that particular OS. If it makes people feel better we can continue to support the existing mechanisms of contribution, then people can choose between interacting with a ?non free? host and ?free? tooling. I suspect most people will choose the ?non-free? tooling. From wes.turner at gmail.com Sun Nov 30 06:34:46 2014 From: wes.turner at gmail.com (Wes Turner) Date: Sat, 29 Nov 2014 23:34:46 -0600 Subject: [Python-Dev] PEP 481 - Migrate Some Supporting Repositories to Git and Github In-Reply-To: <7F8618A9-3167-4BF0-8F98-CCF9DA8539BC@stufft.io> References: <30725AC9-3DCF-416A-BCCD-2D64D489898C@stufft.io> <85d285wa3c.fsf@benfinney.id.au> <7F8618A9-3167-4BF0-8F98-CCF9DA8539BC@stufft.io> Message-ID: Specifically, which features are most ideal here? - [ ] Userbase - [ ] TTW editing only over SSL (see: Zope 2) - [ ] Pull Requests (see also: BitBucket, Torvalds rant) - [ ] Simple Issue Tagging - [ ] Pingbacks - [ ] CI Integration On Sat, Nov 29, 2014 at 11:27 PM, Donald Stufft wrote: > > > On Nov 30, 2014, at 12:06 AM, Ben Finney > wrote: > > > > Nick Coghlan writes: > > > >> 1. I strongly believe that the long term sustainability of the overall > >> open source community requires the availability and use of open source > >> infrastructure. > > > > I concur. This article > > makes the arguments well, IMO. > > > >> 2. I also feel that this proposal is far too cavalier in not even > >> discussing the possibility of helping out the Mercurial team [?] we'd > >> prefer to switch to something else entirely rather than organising a > >> sprint with them at PyCon to help ensure that our existing Mercurial > >> based infrastructure is approachable for git & GitHub users? > > > > Exactly. For such a core tool, instead of pushing proprietary platforms > > at the expense of software freedom, the sensible strategy for a project > > (Python) that hopes to be around in the long term is to use and improve > > the free software platforms. > > I think there is a big difference here between using a closed source VCS > or compiler and using a closed source code host. Namely in that the > protocol is defined by git so switching from one host to another is easy. > > It?s akin to saying that if we chose to run the PyPI services on a Windows > machine that it is somehow makes it less-free even though we could > have chosen to run it on a ?free? OS and we weren?t doing much, if > anything, > to tie us to that particular OS. > > If it makes people feel better we can continue to support the existing > mechanisms of contribution, then people can choose between interacting > with a ?non free? host and ?free? tooling. I suspect most people will > choose > the ?non-free? tooling. > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/wes.turner%40gmail.com > -------------- next part -------------- An HTML attachment was scrubbed... URL: From demianbrecht at gmail.com Sun Nov 30 07:14:14 2014 From: demianbrecht at gmail.com (Demian Brecht) Date: Sun, 30 Nov 2014 06:14:14 +0000 Subject: [Python-Dev] PEP 481 - Migrate Some Supporting Repositories to Git and Github References: <30725AC9-3DCF-416A-BCCD-2D64D489898C@stufft.io> Message-ID: On Sat, Nov 29, 2014 at 3:27 PM, Donald Stufft wrote: > As promised in the "Move selected documentation repos to PSF BitBucket > account?" thread I've written up a PEP for moving selected repositories from > hg.python.org to Github. FWIW, I'm a pretty solid -1 to this PEP. Don't get me wrong, I'm much more accustomed to git than I am hg, much prefer git's lightweight branching model and would love to see CPython and all ancillary repos migrated to git & Github. If that was what this PEP was after, I'd be a +1. What I don't like about it is the introduction of multiple tools that directly impact the barrier of entry to contributing. I think that splitting ancillary repos such as PEPs and docs out might be a little short sighted at an overall project level. In my mind, there are three major categories of contributors (and prospective contributors): 1. Those that use git on a daily basis 2. Those that use hg on a daily basis 3. Those who use neither and are more accustomed to Perforce, SVN and the like Let's say this PEP is approved and the suggested repos are moved to Github. For git users, life is suddenly made much easier when contributing to those projects for obvious reasons. However, they still have the same barrier of entry to contributing to CPython (I would imagine that this would be the goal for most users, but maybe I'm wrong about that). I would imagine that contributing to the ancillary repos would be great grounds to ramp up on using hg before hitting CPython with its multiple long lived branches and such. Making the switch as suggested by this PEP removes that. For hg users, you now add a barrier of entry for contributing to the repos now living on Github. In both cases, you've introduced the need to context switch when contributing to CPython and any of the other repos. Two tools that require quite different workflows. Then, for the third class of users, you've now introduced the requirement of learning two different sets of tools (if they want to do anything outside of using the "Edit" button through Github's UI). Now you're looking at conflated contributor documentation or project-specific documentation. IMHO, suboptimal either way you look at it. Personally, I don't think that there are any silver bullets to this problem. In no case is everyone going to be satisfied. In cases like that, I tend to think that no matter what the solution eventually agreed upon is, consistency is of the utmost importance. Moving a number of repos to Github breaks that consistency. What *would* be nice if CPython was to stay on mercurial though is perhaps moving those repos to Bitbucket. In that case, you get both the "Edit" feature that really by all account removes the initial bar of entry, but you still remain consistent with the rest of the project. -------------- next part -------------- An HTML attachment was scrubbed... URL: From larry at hastings.org Sun Nov 30 08:08:08 2014 From: larry at hastings.org (Larry Hastings) Date: Sat, 29 Nov 2014 23:08:08 -0800 Subject: [Python-Dev] PEP 481 - Migrate Some Supporting Repositories to Git and Github In-Reply-To: <4ADD805B-B707-4244-9290-77AB0DAE396A@stufft.io> References: <30725AC9-3DCF-416A-BCCD-2D64D489898C@stufft.io> <4ADD805B-B707-4244-9290-77AB0DAE396A@stufft.io> Message-ID: <547AC258.5090803@hastings.org> On 11/29/2014 04:37 PM, Donald Stufft wrote: > On Nov 29, 2014, at 7:15 PM, Alex Gaynor wrote: >> Despite being a regular hg >> user for years, I have no idea how to create a local-only branch, or a branch >> which is pushed to a remote (to use the git term). > I also don?t know how to do this. Instead of collectively scratching your heads, could one of you guys do the research and figure out whether or not hg supports this workflow? One of the following two things must be true: 1. hg supports this workflow (or a reasonable fascimile), which may lessen the need for this PEP. 2. hg doesn't support this workflow, which may strengthen the need for this PEP. Saying "I've been using hg for years and I don't know whether it supports this IMPORTANT THING" is not a particularly compelling argument. //arry/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Sun Nov 30 10:40:44 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 30 Nov 2014 19:40:44 +1000 Subject: [Python-Dev] PEP 479 In-Reply-To: References: Message-ID: On 30 November 2014 at 12:31, Chris Angelico wrote: > On Sun, Nov 30, 2014 at 1:04 PM, Jim J. Jewett wrote: >> (4) It can happen because of "yield from" yielding from an iterator, >> rather than a generator? > > No; as I understand it (though maybe I'm wrong too), "yield from" will > yield every value the other iterator yields, and will then quietly > emit a value if the iterator raises StopIteration, or will allow any > other exception to propagate. The StopIteration coming from the > iterator is absorbed by the "yield from" construct. To completely > propagate it out, "return (yield from iter)" should cover all three > results (yielded value, returned value, raised exception). Chris is correct here. The key yield from related change due to PEP 479 is to eliminate a subtle difference in the interaction between generators and StopIteration that currently exists when delegating to a subgenerator. If you throw StopIteration directly into a suspended generator, it propagates back out: >>> def gen1(): ... yield 1 ... yield 2 ... >>> g = gen1() >>> next(g) 1 >>> g.throw(StopIteration) Traceback (most recent call last): File "", line 1, in File "", line 2, in gen1 StopIteration However, if the generator is instead suspended inside a *subgenerator*, then the thrown in exception will terminate the subgenerator, and execution of the delegating generator will resume: >>> def gen_inner(): ... yield 1 ... >>> def gen_outer(): ... yield from gen_inner() ... yield 2 ... >>> g2 = gen_outer() >>> next(g2) 1 >>> g2.throw(StopIteration) 2 Under PEP 479, the StopIteration thrown in StopIteration will be turned into RuntimeError regardless of whether or not a subgenerator is involved - the difference in the subgenerator case is that the transformation will happen when the innermost generator is terminated, and it will then appear as RuntimeError in the delegating generator. This change in semantics will make it possible to fix a latent defect in contextlib.contextmanager, where using a subgenerator as part of the context manager implementation may currently lead to inadvertently suppressing StopIteration in the body of a covered with statement. Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From cedric.krier at b2ck.com Sun Nov 30 10:42:27 2014 From: cedric.krier at b2ck.com (=?utf-8?Q?C=C3=A9dric?= Krier) Date: Sun, 30 Nov 2014 10:42:27 +0100 Subject: [Python-Dev] PEP 481 - Migrate Some Supporting Repositories to Git and Github In-Reply-To: References: <30725AC9-3DCF-416A-BCCD-2D64D489898C@stufft.io> <20141130014331.31d99bb6@fsol> Message-ID: <20141130094227.GZ11381@tetsuo.wifi.b2ck.com> On 29 Nov 20:13, Donald Stufft wrote: > I think one of the issues with Reitveld isn?t related to Reitveld itself at > all, it?s all the *other* stuff you have to do to get a patch into Reitveld to > allow someone to review it at all. Generating a patch and uploading it to > Roundup is a pain and it?s far easier to just commit things directly to > default. For the record, there is a mercurial extension to ease upload of patch to Rietveld: https://bitbucket.org/nicoe/hgreview/overview -- C?dric Krier - B2CK SPRL Email/Jabber: cedric.krier at b2ck.com Tel: +32 472 54 46 59 Website: http://www.b2ck.com/ -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 213 bytes Desc: not available URL: From ncoghlan at gmail.com Sun Nov 30 10:54:56 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 30 Nov 2014 19:54:56 +1000 Subject: [Python-Dev] PEP 481 - Migrate Some Supporting Repositories to Git and Github In-Reply-To: References: <30725AC9-3DCF-416A-BCCD-2D64D489898C@stufft.io> <85d285wa3c.fsf@benfinney.id.au> Message-ID: On 30 November 2014 at 15:23, Chris Angelico wrote: > Python is already using quite a bit of non-free software in its > ecosystem. The Windows builds of CPython are made with Microsoft's > compiler, and the recent discussion about shifting to Cygwin or MinGW > basically boiled down to "but it ought to be free software", and that > was considered not a sufficiently strong argument. In each case, the > decision has impact on other people (using MSVC for the official > python.org installers means extension writers need to use MSVC too; > and using GitHub means that contributors are strongly encouraged, > possibly required, to use GitHub); so why is it acceptable to use a > non-free compiler, but not acceptable to use a non-free host? Relying on non-free software to support users of a non-free platform is rather different from *requiring* the use of non-free software to participate in core Python community design processes. Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From greg.ewing at canterbury.ac.nz Sun Nov 30 11:01:33 2014 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sun, 30 Nov 2014 23:01:33 +1300 Subject: [Python-Dev] advice needed: best approach to enabling "metamodules"? In-Reply-To: <20141129230504.GA11424@ando.pearwood.info> References: <547A43BD.8080303@canterbury.ac.nz> <20141129230504.GA11424@ando.pearwood.info> Message-ID: <547AEAFD.4030106@canterbury.ac.nz> Steven D'Aprano wrote: > If this feature is going to be used, I would expect to be able to re-use > pre-written module types. E.g. having written "module with properties" > (so to speak) once, I can just import it and use it in my next project. There would be nothing to stop __new__.py importing it from another module, as long as it's not any of the modules that are going to be using it. -- Greg From rosuav at gmail.com Sun Nov 30 11:17:22 2014 From: rosuav at gmail.com (Chris Angelico) Date: Sun, 30 Nov 2014 21:17:22 +1100 Subject: [Python-Dev] PEP 481 - Migrate Some Supporting Repositories to Git and Github In-Reply-To: References: <30725AC9-3DCF-416A-BCCD-2D64D489898C@stufft.io> <85d285wa3c.fsf@benfinney.id.au> Message-ID: On Sun, Nov 30, 2014 at 8:54 PM, Nick Coghlan wrote: > On 30 November 2014 at 15:23, Chris Angelico wrote: >> Python is already using quite a bit of non-free software in its >> ecosystem. The Windows builds of CPython are made with Microsoft's >> compiler, and the recent discussion about shifting to Cygwin or MinGW >> basically boiled down to "but it ought to be free software", and that >> was considered not a sufficiently strong argument. In each case, the >> decision has impact on other people (using MSVC for the official >> python.org installers means extension writers need to use MSVC too; >> and using GitHub means that contributors are strongly encouraged, >> possibly required, to use GitHub); so why is it acceptable to use a >> non-free compiler, but not acceptable to use a non-free host? > > Relying on non-free software to support users of a non-free platform > is rather different from *requiring* the use of non-free software to > participate in core Python community design processes. But what non-free software is required to use the community design processes? The GitHub client is entirely optional; I don't use it, I just use git itself. Using a free client to access a proprietary server isn't the same as using non-free software. ChrisA From steve at pearwood.info Sun Nov 30 11:42:00 2014 From: steve at pearwood.info (Steven D'Aprano) Date: Sun, 30 Nov 2014 21:42:00 +1100 Subject: [Python-Dev] PEP 481 - Migrate Some Supporting Repositories to Git and Github In-Reply-To: References: <30725AC9-3DCF-416A-BCCD-2D64D489898C@stufft.io> Message-ID: <20141130104153.GD11424@ando.pearwood.info> I have some questions and/or issues with the PEP, but first I'm going to add something to Nick's comments: On Sun, Nov 30, 2014 at 11:12:17AM +1000, Nick Coghlan wrote: > Beyond that, GitHub is indeed the most expedient option. My two main > reasons for objecting to taking the expedient path are: > > 1. I strongly believe that the long term sustainability of the overall open > source community requires the availability and use of open source > infrastructure. While I admire the ingenuity of the "free-as-in-beer" model > for proprietary software companies fending off open source competition, I > still know a proprietary platform play when I see one (and so do venture > capitalists looking to extract monopoly rents from the industry in the > future). (So yes, I regret relenting on this principle in previously > suggesting the interim use of another proprietary hosted service) > > 2. I also feel that this proposal is far too cavalier in not even > discussing the possibility of helping out the Mercurial team to resolve > their documentation and usability issues rather than just yelling at them > "your tool isn't popular enough for us, and we find certain aspects of it > too hard to use, so we're switching to something else rather than working > with you to address our concerns". We consider the Mercurial team a > significant enough part of the Python ecosystem that Matt was one of the > folks specifically invited to the 2014 language summit to discuss their > concerns around the Python 3 transition. Yet we'd prefer to switch to > something else entirely rather than organising a sprint with them at PyCon > to help ensure that our existing Mercurial based infrastructure is > approachable for git & GitHub users? (And yes, I consider some of the core > Mercurial devs to be friends, so this isn't an entirely abstract concern > for me) Thanks Nick, I think these are excellent points, particularly the second. It would be a gross strawman to say that we should "only" use software developed in Python, but we should eat our own dogfood whenever practical and we should support and encourage the Python ecosystem, including Mercurial. Particularly since hg and git are neck and neck feature-wise, we should resist the tendency to jump on bandwagons. If git were clearly the superior product, then maybe there would be an argument for using the best tool for the job, but it isn't. As for the question of using Github hosting, there's another factor which has been conspicuous by its absence. Has GitHub's allegedly toxic and bullying culture changed since Julie Horvath quit in March? And if it has not, do we care? I'm not a saint, but I do try to choose ethical companies and institutions over unethical ones whenever it is possible and practical. I'm not looking for a witch-hunt against GitHub, but if the allegations made by Horvath earlier this year are true, and I don't believe anyone has denied them, then so long as GitHub's internal culture remains sexist and hostile to the degree reported, then I do not believe that we should use GitHub's services even if we shift some repos to git. I have serious doubts about GitHub's compatibility with the ideals expressed by the PSF. Even if our code of conduct does not explicitly forbid it, I think that it goes against the principles that we say we aspire to. Given Horvath's experiences, and the lack of clear evidence that anything has changed in GitHub, I would be deeply disappointed if Python lent even a smidgeon of legitimacy to their company, and I personally will not use their services. I acknowledge that it's hard to prove a negative, and GitHub may have difficulty proving to my satisfaction that they have changed. (My experience is that company culture rarely changes unless there is a change in management, and even then only slowly.) Particularly given GitHub's supposed egalitarian, non-hierarchical, and meritocratic structure, that nobody apparently saw anything wrong with the bullying of staff and workplace sexism until it became public knowledge suggests that it is not just a few bad apples but a problem all through the company. -- Steven From ben+python at benfinney.id.au Sun Nov 30 12:14:14 2014 From: ben+python at benfinney.id.au (Ben Finney) Date: Sun, 30 Nov 2014 22:14:14 +1100 Subject: [Python-Dev] PEP 481 - Migrate Some Supporting Repositories to Git and Github References: <30725AC9-3DCF-416A-BCCD-2D64D489898C@stufft.io> <85d285wa3c.fsf@benfinney.id.au> Message-ID: <858uitvt2x.fsf@benfinney.id.au> Chris Angelico writes: > But what non-free software is required to use the community design > processes? The GitHub client is entirely optional; I don't use it, I > just use git itself. Using a free client to access a proprietary > server isn't the same as using non-free software. For that to remain true, there must be no advantage specifically to GitHub except as a commodity repository host. So all the verbiage about specific GitHub features in this PEP are undermined. On the other hand, if GitHub is being recommended specifically because of features that cannot be migrated elsewhere (such as its proprietary pull requests), then it is not possible to argue that Git alone is sufficient to realise these advantages. -- \ ?DRM doesn't inconvenience [lawbreakers] ? indeed, over time it | `\ trains law-abiding users to become [lawbreakers] out of sheer | _o__) frustration.? ?Charles Stross, 2010-05-09 | Ben Finney From ben+python at benfinney.id.au Sun Nov 30 12:18:32 2014 From: ben+python at benfinney.id.au (Ben Finney) Date: Sun, 30 Nov 2014 22:18:32 +1100 Subject: [Python-Dev] PEP 481 - Migrate Some Supporting Repositories to Git and Github References: <30725AC9-3DCF-416A-BCCD-2D64D489898C@stufft.io> <85d285wa3c.fsf@benfinney.id.au> <7F8618A9-3167-4BF0-8F98-CCF9DA8539BC@stufft.io> Message-ID: <854mthvsvr.fsf@benfinney.id.au> Donald Stufft writes: > I think there is a big difference here between using a closed source > VCS or compiler and using a closed source code host. Namely in that > the protocol is defined by git so switching from one host to another > is easy. GitHub deliberately encourages proprietary features that create valuable data that cannot be exported ? the proprietary GitHub-specific pull requests being a prime example. So it is only true to say one can export the data to a different host if one entirely abandons all those features which create GitHub-proprietary data. If you want to re-write the PEP to be clear you are only talking about the repository hosting of GitHub, and *not* any of its extra features that make it so attractive to use their proprietary service, I'd be interested to see that. On the other hand, if you want to promote those proprietary features as part of the PEP, then it is disingenuous to claim the data can be easily exported to a different host. -- \ ?Do not enter the lift backwards, and only when lit up.? | `\ ?elevator, Leipzig | _o__) | Ben Finney From p.f.moore at gmail.com Sun Nov 30 13:31:45 2014 From: p.f.moore at gmail.com (Paul Moore) Date: Sun, 30 Nov 2014 12:31:45 +0000 Subject: [Python-Dev] PEP 481 - Migrate Some Supporting Repositories to Git and Github In-Reply-To: <30725AC9-3DCF-416A-BCCD-2D64D489898C@stufft.io> References: <30725AC9-3DCF-416A-BCCD-2D64D489898C@stufft.io> Message-ID: On 29 November 2014 at 23:27, Donald Stufft wrote: > In previous years there was concern about how well supported git was on Windows > in comparison to Mercurial. However git has grown to support Windows as a first > class citizen. In addition to that, for Windows users who are not well aquanted > with the Windows command line there are GUI options as well. I have little opinion on the PEP as a whole, but is the above statement true? From the git website, version 2.2.0 is current, and yet the downloadable Windows version is still 1.9.4. That's a fairly significant version lag for a "first class citizen". I like git, and it has a number of windows-specific extensions that are really useful (more than Mercurial, AFAIK), but I wouldn't say that the core product supported Windows on an equal footing to Linux. Paul From solipsis at pitrou.net Sun Nov 30 14:01:37 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sun, 30 Nov 2014 14:01:37 +0100 Subject: [Python-Dev] PEP 481 - Migrate Some Supporting Repositories to Git and Github References: <30725AC9-3DCF-416A-BCCD-2D64D489898C@stufft.io> <85d285wa3c.fsf@benfinney.id.au> Message-ID: <20141130140137.46803df1@fsol> On Sun, 30 Nov 2014 16:23:08 +1100 Chris Angelico wrote: > > Yes, GitHub is proprietary. But all of your actual code is stored in > git, which is free, and it's easy to push that to a new host somewhere > else, or create your own host. This proposal is for repositories that > don't need much in the way of issue trackers etc, so shifting away > from GitHub shouldn't demand anything beyond moving the repos > themselves. I hope we're not proposing to move the issue trackers to github, otherwise I'm -1 on this PEP. Regards Antoine. From brett at python.org Sun Nov 30 15:15:42 2014 From: brett at python.org (Brett Cannon) Date: Sun, 30 Nov 2014 14:15:42 +0000 Subject: [Python-Dev] advice needed: best approach to enabling "metamodules"? References: Message-ID: On Sat, Nov 29, 2014, 21:55 Guido van Rossum wrote: All the use cases seem to be about adding some kind of getattr hook to modules. They all seem to involve modifying the CPython C code anyway. So why not tackle that problem head-on and modify module_getattro() to look for a global named __getattr__ and if it exists, call that instead of raising AttributeError? Not sure if anyone thought of it. :) Seems like a reasonable solution to me. Be curious to know what the benchmark suite said the impact was. -brett On Sat, Nov 29, 2014 at 11:37 AM, Nathaniel Smith wrote: On Sat, Nov 29, 2014 at 4:21 AM, Guido van Rossum wrote: > Are these really all our options? All of them sound like hacks, none of them > sound like anything the language (or even the CPython implementation) should > sanction. Have I missed the discussion where the use cases and constraints > were analyzed and all other approaches were rejected? (I might have some > half-baked ideas, but I feel I should read up on the past discussion first, > and they are probably more fit for python-ideas than for python-dev. Plus > I'm just writing this email because I'm procrastinating on the type hinting > PEP. :-) The previous discussions I was referring to are here: http://thread.gmane.org/gmane.comp.python.ideas/29487/focus=29555 http://thread.gmane.org/gmane.comp.python.ideas/29788 There might well be other options; these are just the best ones I could think of :-). The constraints are pretty tight, though: - The "new module" object (whatever it is) should have a __dict__ that aliases the original module globals(). I can elaborate on this if my original email wasn't enough, but hopefully it's obvious that making two copies of the same namespace and then trying to keep them in sync at the very least smells bad :-). - The "new module" object has to be a subtype of ModuleType, b/c there are lots of places that do isinstance(x, ModuleType) checks (notably -- but not only -- reload()). Since a major goal here is to make it possible to do cleaner deprecations, it would be really unfortunate if switching an existing package to use the metamodule support itself broke things :-). - Lookups in the normal case should have no additional performance overhead, because module lookups are extremely extremely common. (So this rules out dict proxies and tricks like that -- we really need 'new_module.__dict__ is globals()' to be true.) AFAICT there are three logically possible strategies for satisfying that first constraint: (a) convert the original module object into the type we want, in-place (b) create a new module object that acts like the original module object (c) somehow arrange for our special type to be used from the start My options 1 and 2 are means of accomplishing (a), and my options 3 and 4 are means of accomplishing (b) while working around the behavioural quirks of module objects (as required by the second constraint). The python-ideas thread did also consider several methods of implementing strategy (c), but they're messy enough that I left them out here. The problem is that somehow we have to execute code to create the new subtype *before* we have an entry in sys.modules for the package that contains the code for the subtype. So one option would be to add a new rule, that if a file pkgname/__new__.py exists, then this is executed first and is required to set up sys.modules["pkgname"] before we exec pkgname/__init__.py. So pkgname/__new__.py might look like: import sys from pkgname._metamodule import MyModuleSubtype sys.modules[__name__] = MyModuleSubtype(__name__, docstring) This runs into a lot of problems though. To start with, the 'from pkgname._metamodule ...' line is an infinite loop, b/c this is the code used to create sys.modules["pkgname"]. It's not clear where the globals dict for executing __new__.py comes from (who defines __name__? Currently that's done by ModuleType.__init__). It only works for packages, not modules. The need to provide the docstring here, before __init__.py is even read, is weird. It adds extra stat() calls to every package lookup. And, the biggest showstopper IMHO: AFAICT it's impossible to write a polyfill to support this code on old python versions, so it's useless to any package which needs to keep compatibility with 2.7 (or even 3.4). Sure, you can backport the whole import system like importlib2, but telling everyone that they need to replace every 'import numpy' with 'import importlib2; import numpy' is a total non-starter. So, yeah, those 4 options are really the only plausible ones I know of. Option 1 and option 3 are pretty nice at the language level! Most Python objects allow assignment to __class__ and __dict__, and both PyPy and Jython at least do support __class__ assignment. Really the only downside with Option 1 is that actually implementing it requires attention from someone with deep knowledge of typeobject.c. -n -- Nathaniel J. Smith Postdoctoral researcher - Informatics - University of Edinburgh http://vorpus.org _______________________________________________ Python-Dev mailing list Python-Dev at python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/guido%40python.org -- --Guido van Rossum (python.org/~guido) _______________________________________________ Python-Dev mailing list Python-Dev at python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/brett%40python.org -------------- next part -------------- An HTML attachment was scrubbed... URL: From graffatcolmingov at gmail.com Sun Nov 30 16:54:43 2014 From: graffatcolmingov at gmail.com (Ian Cordasco) Date: Sun, 30 Nov 2014 09:54:43 -0600 Subject: [Python-Dev] PEP 481 - Migrate Some Supporting Repositories to Git and Github In-Reply-To: <20141130140137.46803df1@fsol> References: <30725AC9-3DCF-416A-BCCD-2D64D489898C@stufft.io> <85d285wa3c.fsf@benfinney.id.au> <20141130140137.46803df1@fsol> Message-ID: On Sun, Nov 30, 2014 at 7:01 AM, Antoine Pitrou wrote: > On Sun, 30 Nov 2014 16:23:08 +1100 > Chris Angelico wrote: >> >> Yes, GitHub is proprietary. But all of your actual code is stored in >> git, which is free, and it's easy to push that to a new host somewhere >> else, or create your own host. This proposal is for repositories that >> don't need much in the way of issue trackers etc, so shifting away >> from GitHub shouldn't demand anything beyond moving the repos >> themselves. > > I hope we're not proposing to move the issue trackers to github, > otherwise I'm -1 on this PEP. > > Regards > > Antoine. So I usually choose not to weigh in on discussions like these but there seems to be a lot of misdirection in some of these arguments. To start, I'm generally neutral about this proposal or Nick's proposal that spurred this one. I've found the most frustrating part of contributing to anything involving CPython to be the lack of reviewer time. I have had very small (2-line) patches take months (close to a year in reality) to get through in spite of periodically pinging the appropriate people. Moving to git/GitHub will not alleviate this at all. To be clear, the main reasoning behind Nick's was being able to submit changes without ever having to have a local copy of the repository in question on your machine. Having a complete web workflow for editing and contributing makes the barrier to entry far lower than switching VCS or anything else. BitBucket (apparently, although I've never used this) and GitHub both have this capability and *both* are free-as-in-beer systems. No one as I understand it is proposing that we use the per-distro proprietary interface to these websites. All data can be removed from GitHub using it's API and can generally be converted to another platform. The same goes for BitBucket although it's arguably easier to retrieve issue data from BitBucket than GitHub. That said, *the issue tracker is not covered by these proposals* so this is a moot point. Drop it already. If we're seriously considering moving to git as a DVCS, we should consider the real free-as-in-freedom alternatives that come very close to GitHub and can be improved by us (even though they're not written in Python). One of those is GitLab. We can self-host a GitLab instance easily or we can rely on gitlab.com. GitLab aims to provide a very similar user experience to GitHub and it's slowly approaching feature parity and experience parity. GitLab is also what a lot of people chose to switch to after the incident Steven mentioned, which I don't think is something we should dismiss or ignore. We should refocus the discussion with the following in mind: - Migrating "data" from GitHub is easy. There are free-as-in-freedom tools to do it and the only cost is the time it would take to monitor the process - GitHub has a toxic company culture that we should be aware of before moving to it. They have a couple blog posts about attempting to change it but employees became eerily silent after the incident and have remained so from what I've personally seen. - GitHub may be popular but there are popular FOSS solutions that exist that we can also self-host at something like forge.python.org - bugs.python.org is not covered by any of these proposals - The main benefit this proposal (and the proposal to move to BitBucket) are seeking to achieve is an online editing experience allowing for *anyone with a browser and an account* to contribute. This to me is the only reason I would be +1 for either of these proposals (if we can reach consensus). From 4kir4.1i at gmail.com Sun Nov 30 15:08:32 2014 From: 4kir4.1i at gmail.com (Akira Li) Date: Sun, 30 Nov 2014 17:08:32 +0300 Subject: [Python-Dev] PEP 481 - Migrate Some Supporting Repositories to Git and Github References: <30725AC9-3DCF-416A-BCCD-2D64D489898C@stufft.io> <4ADD805B-B707-4244-9290-77AB0DAE396A@stufft.io> <547AC258.5090803@hastings.org> Message-ID: <87oaroaihr.fsf@gmail.com> Larry Hastings writes: > On 11/29/2014 04:37 PM, Donald Stufft wrote: >> On Nov 29, 2014, at 7:15 PM, Alex Gaynor wrote: >>> Despite being a regular hg >>> user for years, I have no idea how to create a local-only branch, or a branch >>> which is pushed to a remote (to use the git term). >> I also don?t know how to do this. > > Instead of collectively scratching your heads, could one of you guys > do the research and figure out whether or not hg supports this > workflow? One of the following two things must be true: > > 1. hg supports this workflow (or a reasonable fascimile), which may > lessen the need for this PEP. > 2. hg doesn't support this workflow, which may strengthen the need for > this PEP. > Assuming git's "all work is done in a local branch" workflow, you could use bookmarks with hg http://lostechies.com/jimmybogard/2010/06/03/translating-my-git-workflow-with-local-branches-to-mercurial/ http://stevelosh.com/blog/2009/08/a-guide-to-branching-in-mercurial/#branching-with-bookmarks http://mercurial.selenic.com/wiki/BookmarksExtension#Usage http://stackoverflow.com/questions/1598759/git-and-mercurial-compare-and-contrast -- Akira From wes.turner at gmail.com Sun Nov 30 07:11:12 2014 From: wes.turner at gmail.com (Wes Turner) Date: Sun, 30 Nov 2014 00:11:12 -0600 Subject: [Python-Dev] PEP 481 - Migrate Some Supporting Repositories to Git and Github In-Reply-To: References: <30725AC9-3DCF-416A-BCCD-2D64D489898C@stufft.io> <85d285wa3c.fsf@benfinney.id.au> <7F8618A9-3167-4BF0-8F98-CCF9DA8539BC@stufft.io> Message-ID: - [ ] Markdown - [ ] ReStructuredText - [ ] Review (why are these out of band?) On Sat, Nov 29, 2014 at 11:34 PM, Wes Turner wrote: > Specifically, which features are most ideal here? > > - [ ] Userbase > - [ ] TTW editing only over SSL (see: Zope 2) > - [ ] Pull Requests (see also: BitBucket, Torvalds rant) > - [ ] Simple Issue Tagging > - [ ] Pingbacks > - [ ] CI Integration > > > On Sat, Nov 29, 2014 at 11:27 PM, Donald Stufft wrote: > >> >> > On Nov 30, 2014, at 12:06 AM, Ben Finney >> wrote: >> > >> > Nick Coghlan writes: >> > >> >> 1. I strongly believe that the long term sustainability of the overall >> >> open source community requires the availability and use of open source >> >> infrastructure. >> > >> > I concur. This article > > >> > makes the arguments well, IMO. >> > >> >> 2. I also feel that this proposal is far too cavalier in not even >> >> discussing the possibility of helping out the Mercurial team [?] we'd >> >> prefer to switch to something else entirely rather than organising a >> >> sprint with them at PyCon to help ensure that our existing Mercurial >> >> based infrastructure is approachable for git & GitHub users? >> > >> > Exactly. For such a core tool, instead of pushing proprietary platforms >> > at the expense of software freedom, the sensible strategy for a project >> > (Python) that hopes to be around in the long term is to use and improve >> > the free software platforms. >> >> I think there is a big difference here between using a closed source VCS >> or compiler and using a closed source code host. Namely in that the >> protocol is defined by git so switching from one host to another is easy. >> >> It?s akin to saying that if we chose to run the PyPI services on a Windows >> machine that it is somehow makes it less-free even though we could >> have chosen to run it on a ?free? OS and we weren?t doing much, if >> anything, >> to tie us to that particular OS. >> >> If it makes people feel better we can continue to support the existing >> mechanisms of contribution, then people can choose between interacting >> with a ?non free? host and ?free? tooling. I suspect most people will >> choose >> the ?non-free? tooling. >> _______________________________________________ >> Python-Dev mailing list >> Python-Dev at python.org >> https://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe: >> https://mail.python.org/mailman/options/python-dev/wes.turner%40gmail.com >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From wes.turner at gmail.com Sun Nov 30 07:31:02 2014 From: wes.turner at gmail.com (Wes Turner) Date: Sun, 30 Nov 2014 00:31:02 -0600 Subject: [Python-Dev] PEP 481 - Migrate Some Supporting Repositories to Git and Github In-Reply-To: References: <30725AC9-3DCF-416A-BCCD-2D64D489898C@stufft.io> <85d285wa3c.fsf@benfinney.id.au> <7F8618A9-3167-4BF0-8F98-CCF9DA8539BC@stufft.io> Message-ID: - [ ] Stable URIs - [ ] Commit hashes On Sun, Nov 30, 2014 at 12:11 AM, Wes Turner wrote: > - [ ] Markdown > - [ ] ReStructuredText > > - [ ] Review (why are these out of band?) > > On Sat, Nov 29, 2014 at 11:34 PM, Wes Turner wrote: > >> Specifically, which features are most ideal here? >> >> - [ ] Userbase >> - [ ] TTW editing only over SSL (see: Zope 2) >> - [ ] Pull Requests (see also: BitBucket, Torvalds rant) >> - [ ] Simple Issue Tagging >> - [ ] Pingbacks >> - [ ] CI Integration >> >> >> On Sat, Nov 29, 2014 at 11:27 PM, Donald Stufft wrote: >> >>> >>> > On Nov 30, 2014, at 12:06 AM, Ben Finney >>> wrote: >>> > >>> > Nick Coghlan writes: >>> > >>> >> 1. I strongly believe that the long term sustainability of the overall >>> >> open source community requires the availability and use of open source >>> >> infrastructure. >>> > >>> > I concur. This article >> http://mako.cc/writing/hill-free_tools.html> >>> > makes the arguments well, IMO. >>> > >>> >> 2. I also feel that this proposal is far too cavalier in not even >>> >> discussing the possibility of helping out the Mercurial team [?] we'd >>> >> prefer to switch to something else entirely rather than organising a >>> >> sprint with them at PyCon to help ensure that our existing Mercurial >>> >> based infrastructure is approachable for git & GitHub users? >>> > >>> > Exactly. For such a core tool, instead of pushing proprietary platforms >>> > at the expense of software freedom, the sensible strategy for a project >>> > (Python) that hopes to be around in the long term is to use and improve >>> > the free software platforms. >>> >>> I think there is a big difference here between using a closed source VCS >>> or compiler and using a closed source code host. Namely in that the >>> protocol is defined by git so switching from one host to another is easy. >>> >>> It?s akin to saying that if we chose to run the PyPI services on a >>> Windows >>> machine that it is somehow makes it less-free even though we could >>> have chosen to run it on a ?free? OS and we weren?t doing much, if >>> anything, >>> to tie us to that particular OS. >>> >>> If it makes people feel better we can continue to support the existing >>> mechanisms of contribution, then people can choose between interacting >>> with a ?non free? host and ?free? tooling. I suspect most people will >>> choose >>> the ?non-free? tooling. >>> _______________________________________________ >>> Python-Dev mailing list >>> Python-Dev at python.org >>> https://mail.python.org/mailman/listinfo/python-dev >>> Unsubscribe: >>> https://mail.python.org/mailman/options/python-dev/wes.turner%40gmail.com >>> >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From wes.turner at gmail.com Sun Nov 30 07:50:27 2014 From: wes.turner at gmail.com (Wes Turner) Date: Sun, 30 Nov 2014 00:50:27 -0600 Subject: [Python-Dev] PEP 481 - Migrate Some Supporting Repositories to Git and Github In-Reply-To: References: <30725AC9-3DCF-416A-BCCD-2D64D489898C@stufft.io> <85d285wa3c.fsf@benfinney.id.au> <7F8618A9-3167-4BF0-8F98-CCF9DA8539BC@stufft.io> Message-ID: - [ ] Paste-able links On Sun, Nov 30, 2014 at 12:31 AM, Wes Turner wrote: > - [ ] Stable URIs > - [ ] Commit hashes > > On Sun, Nov 30, 2014 at 12:11 AM, Wes Turner wrote: > >> - [ ] Markdown >> - [ ] ReStructuredText >> >> - [ ] Review (why are these out of band?) >> >> On Sat, Nov 29, 2014 at 11:34 PM, Wes Turner >> wrote: >> >>> Specifically, which features are most ideal here? >>> >>> - [ ] Userbase >>> - [ ] TTW editing only over SSL (see: Zope 2) >>> - [ ] Pull Requests (see also: BitBucket, Torvalds rant) >>> - [ ] Simple Issue Tagging >>> - [ ] Pingbacks >>> - [ ] CI Integration >>> >>> >>> On Sat, Nov 29, 2014 at 11:27 PM, Donald Stufft >>> wrote: >>> >>>> >>>> > On Nov 30, 2014, at 12:06 AM, Ben Finney >>>> wrote: >>>> > >>>> > Nick Coghlan writes: >>>> > >>>> >> 1. I strongly believe that the long term sustainability of the >>>> overall >>>> >> open source community requires the availability and use of open >>>> source >>>> >> infrastructure. >>>> > >>>> > I concur. This article >>> http://mako.cc/writing/hill-free_tools.html> >>>> > makes the arguments well, IMO. >>>> > >>>> >> 2. I also feel that this proposal is far too cavalier in not even >>>> >> discussing the possibility of helping out the Mercurial team [?] we'd >>>> >> prefer to switch to something else entirely rather than organising a >>>> >> sprint with them at PyCon to help ensure that our existing Mercurial >>>> >> based infrastructure is approachable for git & GitHub users? >>>> > >>>> > Exactly. For such a core tool, instead of pushing proprietary >>>> platforms >>>> > at the expense of software freedom, the sensible strategy for a >>>> project >>>> > (Python) that hopes to be around in the long term is to use and >>>> improve >>>> > the free software platforms. >>>> >>>> I think there is a big difference here between using a closed source VCS >>>> or compiler and using a closed source code host. Namely in that the >>>> protocol is defined by git so switching from one host to another is >>>> easy. >>>> >>>> It?s akin to saying that if we chose to run the PyPI services on a >>>> Windows >>>> machine that it is somehow makes it less-free even though we could >>>> have chosen to run it on a ?free? OS and we weren?t doing much, if >>>> anything, >>>> to tie us to that particular OS. >>>> >>>> If it makes people feel better we can continue to support the existing >>>> mechanisms of contribution, then people can choose between interacting >>>> with a ?non free? host and ?free? tooling. I suspect most people will >>>> choose >>>> the ?non-free? tooling. >>>> _______________________________________________ >>>> Python-Dev mailing list >>>> Python-Dev at python.org >>>> https://mail.python.org/mailman/listinfo/python-dev >>>> Unsubscribe: >>>> https://mail.python.org/mailman/options/python-dev/wes.turner%40gmail.com >>>> >>> >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From wes.turner at gmail.com Sun Nov 30 08:16:18 2014 From: wes.turner at gmail.com (Wes Turner) Date: Sun, 30 Nov 2014 01:16:18 -0600 Subject: [Python-Dev] PEP 481 - Migrate Some Supporting Repositories to Git and Github In-Reply-To: References: <30725AC9-3DCF-416A-BCCD-2D64D489898C@stufft.io> <85d285wa3c.fsf@benfinney.id.au> <7F8618A9-3167-4BF0-8F98-CCF9DA8539BC@stufft.io> Message-ID: Is there a kalithea celery task to mirror / SYNC with github, for pull requests and/or issues? https://pypi.python.org/pypi/Kallithea/ https://kallithea-scm.org/ https://kallithea-scm.org/repos/kallithea https://bitbucket.org/conservancy/kallithea http://pythonhosted.org//Kallithea http://kallithea.readthedocs.org/en/latest/ http://pypi.python.org/pypi/vcs http://pythonhosted.org//vcs https://github.com/codeinn/vcs On Sun, Nov 30, 2014 at 12:50 AM, Wes Turner wrote: > - [ ] Paste-able links > > On Sun, Nov 30, 2014 at 12:31 AM, Wes Turner wrote: > >> - [ ] Stable URIs >> - [ ] Commit hashes >> >> On Sun, Nov 30, 2014 at 12:11 AM, Wes Turner >> wrote: >> >>> - [ ] Markdown >>> - [ ] ReStructuredText >>> >>> - [ ] Review (why are these out of band?) >>> >>> On Sat, Nov 29, 2014 at 11:34 PM, Wes Turner >>> wrote: >>> >>>> Specifically, which features are most ideal here? >>>> >>>> - [ ] Userbase >>>> - [ ] TTW editing only over SSL (see: Zope 2) >>>> - [ ] Pull Requests (see also: BitBucket, Torvalds rant) >>>> - [ ] Simple Issue Tagging >>>> - [ ] Pingbacks >>>> - [ ] CI Integration >>>> >>>> >>>> On Sat, Nov 29, 2014 at 11:27 PM, Donald Stufft >>>> wrote: >>>> >>>>> >>>>> > On Nov 30, 2014, at 12:06 AM, Ben Finney >>>>> wrote: >>>>> > >>>>> > Nick Coghlan writes: >>>>> > >>>>> >> 1. I strongly believe that the long term sustainability of the >>>>> overall >>>>> >> open source community requires the availability and use of open >>>>> source >>>>> >> infrastructure. >>>>> > >>>>> > I concur. This article >>>> http://mako.cc/writing/hill-free_tools.html> >>>>> > makes the arguments well, IMO. >>>>> > >>>>> >> 2. I also feel that this proposal is far too cavalier in not even >>>>> >> discussing the possibility of helping out the Mercurial team [?] >>>>> we'd >>>>> >> prefer to switch to something else entirely rather than organising a >>>>> >> sprint with them at PyCon to help ensure that our existing Mercurial >>>>> >> based infrastructure is approachable for git & GitHub users? >>>>> > >>>>> > Exactly. For such a core tool, instead of pushing proprietary >>>>> platforms >>>>> > at the expense of software freedom, the sensible strategy for a >>>>> project >>>>> > (Python) that hopes to be around in the long term is to use and >>>>> improve >>>>> > the free software platforms. >>>>> >>>>> I think there is a big difference here between using a closed source >>>>> VCS >>>>> or compiler and using a closed source code host. Namely in that the >>>>> protocol is defined by git so switching from one host to another is >>>>> easy. >>>>> >>>>> It?s akin to saying that if we chose to run the PyPI services on a >>>>> Windows >>>>> machine that it is somehow makes it less-free even though we could >>>>> have chosen to run it on a ?free? OS and we weren?t doing much, if >>>>> anything, >>>>> to tie us to that particular OS. >>>>> >>>>> If it makes people feel better we can continue to support the existing >>>>> mechanisms of contribution, then people can choose between interacting >>>>> with a ?non free? host and ?free? tooling. I suspect most people will >>>>> choose >>>>> the ?non-free? tooling. >>>>> _______________________________________________ >>>>> Python-Dev mailing list >>>>> Python-Dev at python.org >>>>> https://mail.python.org/mailman/listinfo/python-dev >>>>> Unsubscribe: >>>>> https://mail.python.org/mailman/options/python-dev/wes.turner%40gmail.com >>>>> >>>> >>>> >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From wes.turner at gmail.com Sun Nov 30 08:18:23 2014 From: wes.turner at gmail.com (Wes Turner) Date: Sun, 30 Nov 2014 01:18:23 -0600 Subject: [Python-Dev] PEP 481 - Migrate Some Supporting Repositories to Git and Github In-Reply-To: References: <30725AC9-3DCF-416A-BCCD-2D64D489898C@stufft.io> <85d285wa3c.fsf@benfinney.id.au> <7F8618A9-3167-4BF0-8F98-CCF9DA8539BC@stufft.io> Message-ID: https://kallithea-scm.org/repos/kallithea/files/tip/setup.py https://github.com/codeinn/vcs/blob/master/setup.py On Sun, Nov 30, 2014 at 1:16 AM, Wes Turner wrote: > Is there a kalithea celery task to mirror / SYNC with github, for pull > requests and/or issues? > > https://pypi.python.org/pypi/Kallithea/ > https://kallithea-scm.org/ > https://kallithea-scm.org/repos/kallithea > https://bitbucket.org/conservancy/kallithea > > http://pythonhosted.org//Kallithea > http://kallithea.readthedocs.org/en/latest/ > > http://pypi.python.org/pypi/vcs > http://pythonhosted.org//vcs > https://github.com/codeinn/vcs > > > On Sun, Nov 30, 2014 at 12:50 AM, Wes Turner wrote: > >> - [ ] Paste-able links >> >> On Sun, Nov 30, 2014 at 12:31 AM, Wes Turner >> wrote: >> >>> - [ ] Stable URIs >>> - [ ] Commit hashes >>> >>> On Sun, Nov 30, 2014 at 12:11 AM, Wes Turner >>> wrote: >>> >>>> - [ ] Markdown >>>> - [ ] ReStructuredText >>>> >>>> - [ ] Review (why are these out of band?) >>>> >>>> On Sat, Nov 29, 2014 at 11:34 PM, Wes Turner >>>> wrote: >>>> >>>>> Specifically, which features are most ideal here? >>>>> >>>>> - [ ] Userbase >>>>> - [ ] TTW editing only over SSL (see: Zope 2) >>>>> - [ ] Pull Requests (see also: BitBucket, Torvalds rant) >>>>> - [ ] Simple Issue Tagging >>>>> - [ ] Pingbacks >>>>> - [ ] CI Integration >>>>> >>>>> >>>>> On Sat, Nov 29, 2014 at 11:27 PM, Donald Stufft >>>>> wrote: >>>>> >>>>>> >>>>>> > On Nov 30, 2014, at 12:06 AM, Ben Finney < >>>>>> ben+python at benfinney.id.au> wrote: >>>>>> > >>>>>> > Nick Coghlan writes: >>>>>> > >>>>>> >> 1. I strongly believe that the long term sustainability of the >>>>>> overall >>>>>> >> open source community requires the availability and use of open >>>>>> source >>>>>> >> infrastructure. >>>>>> > >>>>>> > I concur. This article >>>>> http://mako.cc/writing/hill-free_tools.html> >>>>>> > makes the arguments well, IMO. >>>>>> > >>>>>> >> 2. I also feel that this proposal is far too cavalier in not even >>>>>> >> discussing the possibility of helping out the Mercurial team [?] >>>>>> we'd >>>>>> >> prefer to switch to something else entirely rather than organising >>>>>> a >>>>>> >> sprint with them at PyCon to help ensure that our existing >>>>>> Mercurial >>>>>> >> based infrastructure is approachable for git & GitHub users? >>>>>> > >>>>>> > Exactly. For such a core tool, instead of pushing proprietary >>>>>> platforms >>>>>> > at the expense of software freedom, the sensible strategy for a >>>>>> project >>>>>> > (Python) that hopes to be around in the long term is to use and >>>>>> improve >>>>>> > the free software platforms. >>>>>> >>>>>> I think there is a big difference here between using a closed source >>>>>> VCS >>>>>> or compiler and using a closed source code host. Namely in that the >>>>>> protocol is defined by git so switching from one host to another is >>>>>> easy. >>>>>> >>>>>> It?s akin to saying that if we chose to run the PyPI services on a >>>>>> Windows >>>>>> machine that it is somehow makes it less-free even though we could >>>>>> have chosen to run it on a ?free? OS and we weren?t doing much, if >>>>>> anything, >>>>>> to tie us to that particular OS. >>>>>> >>>>>> If it makes people feel better we can continue to support the existing >>>>>> mechanisms of contribution, then people can choose between interacting >>>>>> with a ?non free? host and ?free? tooling. I suspect most people will >>>>>> choose >>>>>> the ?non-free? tooling. >>>>>> _______________________________________________ >>>>>> Python-Dev mailing list >>>>>> Python-Dev at python.org >>>>>> https://mail.python.org/mailman/listinfo/python-dev >>>>>> Unsubscribe: >>>>>> https://mail.python.org/mailman/options/python-dev/wes.turner%40gmail.com >>>>>> >>>>> >>>>> >>>> >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From balajimarisetti at gmail.com Sun Nov 30 09:07:52 2014 From: balajimarisetti at gmail.com (balaji marisetti) Date: Sun, 30 Nov 2014 13:37:52 +0530 Subject: [Python-Dev] Unicode decode exception Message-ID: Hi, When I try to iterate through the lines of a file("openssl-1.0.1j/crypto/bn/asm/x86_64-gcc.c"), I get a UnicodeDecodeError (in python 3.4.0 on Ubuntu 14.04). But there is no such error with python 2.7.6. What could be the problem? In [39]: with open("openssl-1.0.1j/crypto/bn/asm/x86_64-gcc.c") as f: for line in f: print (line) --------------------------------------------------------------------------- UnicodeDecodeError Traceback (most recent call last) in () 1 with open("../openssl-1.0.1j/crypto/bn/asm/x86_64-gcc.c") as f: ----> 2 for line in f: 3 print (line) 4 /usr/lib/python3.4/codecs.py in decode(self, input, final) 311 # decode input (taking the buffer into account) 312 data = self.buffer + input --> 313 (result, consumed) = self._buffer_decode(data, self.errors, final) 314 # keep undecoded input until the next call 315 self.buffer = data[consumed:] -- :-)balaji From donald at stufft.io Sun Nov 30 17:08:18 2014 From: donald at stufft.io (Donald Stufft) Date: Sun, 30 Nov 2014 11:08:18 -0500 Subject: [Python-Dev] PEP 481 - Migrate Some Supporting Repositories to Git and Github In-Reply-To: References: <30725AC9-3DCF-416A-BCCD-2D64D489898C@stufft.io> Message-ID: <9559CB2D-7237-4D05-98F9-ACAE4354042A@stufft.io> > On Nov 30, 2014, at 7:31 AM, Paul Moore wrote: > > On 29 November 2014 at 23:27, Donald Stufft wrote: >> In previous years there was concern about how well supported git was on Windows >> in comparison to Mercurial. However git has grown to support Windows as a first >> class citizen. In addition to that, for Windows users who are not well aquanted >> with the Windows command line there are GUI options as well. > > I have little opinion on the PEP as a whole, but is the above > statement true? From the git website, version 2.2.0 is current, and > yet the downloadable Windows version is still 1.9.4. That's a fairly > significant version lag for a "first class citizen". > > I like git, and it has a number of windows-specific extensions that > are really useful (more than Mercurial, AFAIK), but I wouldn't say > that the core product supported Windows on an equal footing to Linux. > > Paul I think so yes. I may be wrong, however while 1.9.4 may be the latest downloadable version of git for Windows, there is no downloadable version of the Linux clients at all, they just tell you to go use your package manager which for instance is version 1.7 on Debian. On OS X the latest version is 2.0.1. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA From brett at python.org Sun Nov 30 17:28:49 2014 From: brett at python.org (Brett Cannon) Date: Sun, 30 Nov 2014 16:28:49 +0000 Subject: [Python-Dev] PEP 481 - Migrate Some Supporting Repositories to Git and Github References: <30725AC9-3DCF-416A-BCCD-2D64D489898C@stufft.io> Message-ID: On Sat Nov 29 2014 at 7:16:34 PM Alex Gaynor wrote: > Donald Stufft stufft.io> writes: > > > > > [words words words] > > > > I strongly support this PEP. I'd like to share two pieces of information. > Both > of these are personal anecdotes: > > For the past several years, I've been a contributor on two major projects > using > mercurial, CPython and PyPy. PyPy has a strong culture of in-repo > branching, > basically all contributors regularly make branches in the main repo for > their > work, and we're very free in giving people commit rights, so almost > everyone > working on PyPy in any way has this level of access. This workflow works > ok. I > don't love it as much as git, but it's fine, it's not an impediment to my > work. > > By contrast, CPython does not have this type of workflow, there are almost > no > in-tree branches besides the 2.7, 3.4, etc. ones. Despite being a regular > hg > user for years, I have no idea how to create a local-only branch, or a > branch > which is pushed to a remote (to use the git term). I also don't generally > commit my own work to CPython, even though I have push privledges, > because I > prefer to *always* get code review on my work. As a result, I use a git > mirror > of CPython to do all my work, and generate patches from that. > > The conclusion I draw from this is that hg's workflow is probably fine if > you're a committer on the project, or don't ever need to maintain multiple > patches concurrently (and thus can just leave everything uncommitted in the > repo). However, the hg workflow seems extremely defficient at non-committer > contributors. > One way to come close to that using hg is to have your own clone that you never push to hg.python.org/cpython (e.g. cloning the Bitbucket clone of cpython or hosting on hg.python.org a personal clone). You can then specify the repo and branch on the issue tracker to generate your patch: https://docs.python.org/devguide/triaging.html#mercurial-repository . After that it's just like any other patch workflow for core devs. It's not quite as nice as maybe using named branches where you can just do a final hg merge/commit to get your changes committed, but if you're not going to commit your branches then you might as well get the automatic patch generation perk in the issue tracker rather than using git (unless there is some other git feature you use that you can't get in hg). > > The seconds experience I have is that of Django's migration to git and > github. > For a long time we were on SVN, and we were very resistant to moving to > DVCS in > general, and github in particular. Multiple times I said that I didn't see > how > exporting a patch and uploading it to trac was more difficult than sending > a > pull request. That was very wrong on my part. > > My primary observation is not about new contributors though, it's actually > about the behavior of core developers. Before we were on github, it was > fairly > rare for core developers to ask for reviews for anything besides *gigantic* > patches, we'd mostly just commit stuff to trunk. Since the switch to > github, > I've seen that core developers are *far* more likely to ask for reviews of > their work before merging. > Why specifically? Did you have a web UI for reviewing patches previously? Do you have CI set up for patches now and didn't before? What features did you specifically gain from the switch to GitHub that you didn't have before? IOW was it the "magic" of GitHub or some technical solution that you got as part of the GitHub package and thus could theoretically be replicated on python.org? -Brett -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Sun Nov 30 17:30:29 2014 From: donald at stufft.io (Donald Stufft) Date: Sun, 30 Nov 2014 11:30:29 -0500 Subject: [Python-Dev] PEP 481 - Migrate Some Supporting Repositories to Git and Github In-Reply-To: <547AC258.5090803@hastings.org> References: <30725AC9-3DCF-416A-BCCD-2D64D489898C@stufft.io> <4ADD805B-B707-4244-9290-77AB0DAE396A@stufft.io> <547AC258.5090803@hastings.org> Message-ID: <50A56F56-E636-43BB-BF5D-8DC30920BDB1@stufft.io> > On Nov 30, 2014, at 2:08 AM, Larry Hastings wrote: > > > On 11/29/2014 04:37 PM, Donald Stufft wrote: >> On Nov 29, 2014, at 7:15 PM, Alex Gaynor wrote: >>> Despite being a regular hg >>> user for years, I have no idea how to create a local-only branch, or a branch >>> which is pushed to a remote (to use the git term). >> I also don?t know how to do this. > > Instead of collectively scratching your heads, could one of you guys do the research and figure out whether or not hg supports this workflow? One of the following two things must be true: > hg supports this workflow (or a reasonable fascimile), which may lessen the need for this PEP. > hg doesn't support this workflow, which may strengthen the need for this PEP. > Saying "I've been using hg for years and I don't know whether it supports this IMPORTANT THING" is not a particularly compelling argument. > Comments like this make me feel like I didn?t explain myself very well in the PEP. While I do think that supporting this workflow via an extension is worse than supporting it in core, this isn?t why this PEP exists. The current workflow for contributing is painful, for the repositories this is talking about if I?m a non-comitter I have to email patches to a particular closed mailing list and then sit around and wait. The Pull Request based workflow is *massively* better than uploading/emailing patches around. So the question then becomes, if we're going to move to a PR based workflow how do we do it? PEP 474 says that we should install some software that works with Mercurial and supports Pull Requests. Another thread suggested that we should just use to bitbucket whicih also supports Mercurial and use that. This PEP says that git and Github have the popular vote, which is extremely compelling as a feature because: * Popularity is the one thing you can't replicate featurewise. This feature or that you can find a way to work something like it out. * With popularity comes the fact that people will already know the tooling involved and how to use it. This means that a new contributor whom already knows git will spend less time learning our specific toolset and more time being able to meaningfully contribute. * With popularity also comes transferability, If I don't currently know a VCS tool and I learn git (and github) for the sake of these repositories then that knowledge will directly transfer to greater than 60% of the top 100 projects on PyPI. * With popularity also comes an increased amount of people writing about it, asking questions, etc. This means that if I have a problem while using this tool I am more likely to find someone who already had this problem and they are more likely to have found someone to help them solve it than I am with a less popular tool. I'm not sure why people seem to try and focus on the fact that with this extension or that you can make Mercurial replicate git better when that isn't really the major point of the PEP at all. If Mercurial and git were neck in neck in terms of mindshare and bitbucket and Github were as well then this PEP probably wouldn't exist because on a techincal level I think the two tools are probably close enough. However when most of the world is using one tool and you're using another, that does represent a barrier to entry. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Sun Nov 30 17:33:31 2014 From: donald at stufft.io (Donald Stufft) Date: Sun, 30 Nov 2014 11:33:31 -0500 Subject: [Python-Dev] PEP 481 - Migrate Some Supporting Repositories to Git and Github In-Reply-To: <854mthvsvr.fsf@benfinney.id.au> References: <30725AC9-3DCF-416A-BCCD-2D64D489898C@stufft.io> <85d285wa3c.fsf@benfinney.id.au> <7F8618A9-3167-4BF0-8F98-CCF9DA8539BC@stufft.io> <854mthvsvr.fsf@benfinney.id.au> Message-ID: <225F4EE1-59FE-4D7B-8CF7-BA6A99E01D91@stufft.io> > On Nov 30, 2014, at 6:18 AM, Ben Finney wrote: > > Donald Stufft writes: > >> I think there is a big difference here between using a closed source >> VCS or compiler and using a closed source code host. Namely in that >> the protocol is defined by git so switching from one host to another >> is easy. > > GitHub deliberately encourages proprietary features that create valuable > data that cannot be exported ? the proprietary GitHub-specific pull > requests being a prime example. Once a Pull Request is merged there is little benefit to the PR itself, however if you want the diff you can create a .patch file by appending .diff or .patch to the end of the PR URL. > > So it is only true to say one can export the data to a different host if > one entirely abandons all those features which create GitHub-proprietary > data. That?s not true at all. There is an API that allows you to access *all* of the data that is stored there. Allowing you to replicate it to whatever system you wish. > > If you want to re-write the PEP to be clear you are only talking about > the repository hosting of GitHub, and *not* any of its extra features > that make it so attractive to use their proprietary service, I'd be > interested to see that. > > On the other hand, if you want to promote those proprietary features as > part of the PEP, then it is disingenuous to claim the data can be easily > exported to a different host. > > -- > \ ?Do not enter the lift backwards, and only when lit up.? | > `\ ?elevator, Leipzig | > _o__) | > Ben Finney > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/donald%40stufft.io --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA From alex.gaynor at gmail.com Sun Nov 30 17:37:15 2014 From: alex.gaynor at gmail.com (Alex Gaynor) Date: Sun, 30 Nov 2014 16:37:15 +0000 Subject: [Python-Dev] PEP 481 - Migrate Some Supporting Repositories to Git and Github References: <30725AC9-3DCF-416A-BCCD-2D64D489898C@stufft.io> Message-ID: On Sun Nov 30 2014 at 10:28:50 AM Brett Cannon wrote: > Why specifically? Did you have a web UI for reviewing patches previously? > Do you have CI set up for patches now and didn't before? What features did > you specifically gain from the switch to GitHub that you didn't have > before? IOW was it the "magic" of GitHub or some technical solution that > you got as part of the GitHub package and thus could theoretically be > replicated on python.org? > > -Brett > Previously someone looking for a review (read: any non-committer) would export a diff from their VCS, upload it as a patch to trac, and then reviewers could leave comments as trac comments. CPython's present process is a clear improvement, insofar as Rietveld allows inlining commenting, but it is otherwise basically the same. By contrast, the Github process does not require a patch author to leave their workflow, they simply "git push" to update a patch. We now also have CI for PRs, but that's a recent addition. It's not magic, it's a good UX :-) Alex -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Sun Nov 30 17:44:59 2014 From: brett at python.org (Brett Cannon) Date: Sun, 30 Nov 2014 16:44:59 +0000 Subject: [Python-Dev] PEP 481 - Migrate Some Supporting Repositories to Git and Github References: <30725AC9-3DCF-416A-BCCD-2D64D489898C@stufft.io> <85d285wa3c.fsf@benfinney.id.au> <20141130140137.46803df1@fsol> Message-ID: On Sun Nov 30 2014 at 10:55:26 AM Ian Cordasco wrote: > On Sun, Nov 30, 2014 at 7:01 AM, Antoine Pitrou > wrote: > > On Sun, 30 Nov 2014 16:23:08 +1100 > > Chris Angelico wrote: > >> > >> Yes, GitHub is proprietary. But all of your actual code is stored in > >> git, which is free, and it's easy to push that to a new host somewhere > >> else, or create your own host. This proposal is for repositories that > >> don't need much in the way of issue trackers etc, so shifting away > >> from GitHub shouldn't demand anything beyond moving the repos > >> themselves. > > > > I hope we're not proposing to move the issue trackers to github, > > otherwise I'm -1 on this PEP. > > > > Regards > > > > Antoine. > > So I usually choose not to weigh in on discussions like these but > there seems to be a lot of misdirection in some of these arguments. > > To start, I'm generally neutral about this proposal or Nick's proposal > that spurred this one. I've found the most frustrating part of > contributing to anything involving CPython to be the lack of reviewer > time. I have had very small (2-line) patches take months (close to a > year in reality) to get through in spite of periodically pinging the > appropriate people. Moving to git/GitHub will not alleviate this at > all. > > To be clear, the main reasoning behind Nick's was being able to submit > changes without ever having to have a local copy of the repository in > question on your machine. Having a complete web workflow for editing > and contributing makes the barrier to entry far lower than switching > VCS or anything else. BitBucket (apparently, although I've never used > this) and GitHub both have this capability and *both* are > free-as-in-beer systems. > > No one as I understand it is proposing that we use the per-distro > proprietary interface to these websites. > > All data can be removed from GitHub using it's API and can generally > be converted to another platform. The same goes for BitBucket although > it's arguably easier to retrieve issue data from BitBucket than > GitHub. That said, *the issue tracker is not covered by these > proposals* so this is a moot point. Drop it already. > > If we're seriously considering moving to git as a DVCS, we should > consider the real free-as-in-freedom alternatives that come very close > to GitHub and can be improved by us (even though they're not written > in Python). One of those is GitLab. We can self-host a GitLab instance > easily or we can rely on gitlab.com. GitLab aims to provide a very > similar user experience to GitHub and it's slowly approaching feature > parity and experience parity. GitLab is also what a lot of people > chose to switch to after the incident Steven mentioned, which I don't > think is something we should dismiss or ignore. > > We should refocus the discussion with the following in mind: > > - Migrating "data" from GitHub is easy. There are free-as-in-freedom > tools to do it and the only cost is the time it would take to monitor > the process > > - GitHub has a toxic company culture that we should be aware of before > moving to it. They have a couple blog posts about attempting to change > it but employees became eerily silent after the incident and have > remained so from what I've personally seen. > > - GitHub may be popular but there are popular FOSS solutions that > exist that we can also self-host at something like forge.python.org > > - bugs.python.org is not covered by any of these proposals > > - The main benefit this proposal (and the proposal to move to > BitBucket) are seeking to achieve is an online editing experience > allowing for *anyone with a browser and an account* to contribute. > This to me is the only reason I would be +1 for either of these > proposals (if we can reach consensus). > But that's not just it. As you pointed out, Ian, getting patch submissions committed faster would be a huge improvement over what we have today. GitHub/Bitbucket/whatever could help with this by giving core devs basic CI to know that I patch is sound to some extent as well as push button commits of patches. For me personally, if I knew a simple patch integrated cleanly and passed on at least one buildbot -- when it wasn't a platform-specific fix -- then I could easily push a "Commit" button and be done with it (although this assumes single branch committing; doing this across branches makes all of this difficult unless we finally resolve our Misc/NEWS conflict issues so that in some instances it can be automated). Instead I have to wait until I have a clone I can push from, download a patch, apply it, run the unit tests myself, do the commit, and then repeat a subset of that to whatever branches make sense. It's a lot of work for which some things could be automated. -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Sun Nov 30 17:45:06 2014 From: donald at stufft.io (Donald Stufft) Date: Sun, 30 Nov 2014 11:45:06 -0500 Subject: [Python-Dev] PEP 481 - Migrate Some Supporting Repositories to Git and Github In-Reply-To: References: <30725AC9-3DCF-416A-BCCD-2D64D489898C@stufft.io> Message-ID: > On Nov 30, 2014, at 11:28 AM, Brett Cannon wrote: > > > > On Sat Nov 29 2014 at 7:16:34 PM Alex Gaynor > wrote: > Donald Stufft stufft.io > writes: > > > > > [words words words] > > > > I strongly support this PEP. I'd like to share two pieces of information. Both > of these are personal anecdotes: > > For the past several years, I've been a contributor on two major projects using > mercurial, CPython and PyPy. PyPy has a strong culture of in-repo branching, > basically all contributors regularly make branches in the main repo for their > work, and we're very free in giving people commit rights, so almost everyone > working on PyPy in any way has this level of access. This workflow works ok. I > don't love it as much as git, but it's fine, it's not an impediment to my work. > > By contrast, CPython does not have this type of workflow, there are almost no > in-tree branches besides the 2.7, 3.4, etc. ones. Despite being a regular hg > user for years, I have no idea how to create a local-only branch, or a branch > which is pushed to a remote (to use the git term). I also don't generally > commit my own work to CPython, even though I have push privledges, > because I > prefer to *always* get code review on my work. As a result, I use a git mirror > of CPython to do all my work, and generate patches from that. > > The conclusion I draw from this is that hg's workflow is probably fine if > you're a committer on the project, or don't ever need to maintain multiple > patches concurrently (and thus can just leave everything uncommitted in the > repo). However, the hg workflow seems extremely defficient at non-committer > contributors. > > One way to come close to that using hg is to have your own clone that you never push to hg.python.org/cpython (e.g. cloning the Bitbucket clone of cpython or hosting on hg.python.org a personal clone). You can then specify the repo and branch on the issue tracker to generate your patch: https://docs.python.org/devguide/triaging.html#mercurial-repository . After that it's just like any other patch workflow for core devs. It's not quite as nice as maybe using named branches where you can just do a final hg merge/commit to get your changes committed, but if you're not going to commit your branches then you might as well get the automatic patch generation perk in the issue tracker rather than using git (unless there is some other git feature you use that you can't get in hg). This doesn?t really work in a Pull Request workflow though I think is the point. Uploading patches is it?s own kind of terrible but yes if you?re just uploading patches you can probably do that. I don?t make branches in Mercurial because i?m afraid I?m going to push a permanent branch to hg.python.org and screw something up. > > > The seconds experience I have is that of Django's migration to git and github. > For a long time we were on SVN, and we were very resistant to moving to > DVCS in > general, and github in particular. Multiple times I said that I didn't see how > exporting a patch and uploading it to trac was more difficult than sending a > pull request. That was very wrong on my part. > > My primary observation is not about new contributors though, it's actually > about the behavior of core developers. Before we were on github, it was fairly > rare for core developers to ask for reviews for anything besides *gigantic* > patches, we'd mostly just commit stuff to trunk. Since the switch to github, > I've seen that core developers are *far* more likely to ask for reviews of > their work before merging. > > Why specifically? Did you have a web UI for reviewing patches previously? Do you have CI set up for patches now and didn't before? What features did you specifically gain from the switch to GitHub that you didn't have before? IOW was it the "magic" of GitHub or some technical solution that you got as part of the GitHub package and thus could theoretically be replicated on python.org ? In most of the cases there was some form of a web UI for reviewing patches. I don?t believe there was a CI setup for patches previously and most projects do end up with some sort of CI on Github since Travis makes it particularly easy. Honestly though, I feel like the main reason for it is just that Github?s solution is easy to use, it works with the VCS which is great because that?s what I?m using already, and it?s highly polished. OSS has a long history of having rather poor UX, and poor UX tends to mean that people would rather not use some particular thing unless they *have* to. That is the way I look at Rietveld to be honest. I get frustrated trying to use it so I'd rather just not use it where I feel I can get away with it. It's not really enough to look at a feature matrix and then go "yup we something in all the checkboxes" which it seems that OSS software likes to try and do when comparing themselves to a well done, but non-OSS solution. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: From benjamin at python.org Sun Nov 30 17:47:08 2014 From: benjamin at python.org (Benjamin Peterson) Date: Sun, 30 Nov 2014 11:47:08 -0500 Subject: [Python-Dev] PEP 481 - Migrate Some Supporting Repositories to Git and Github In-Reply-To: References: <30725AC9-3DCF-416A-BCCD-2D64D489898C@stufft.io> Message-ID: <1417366028.1298022.196972041.1FBE18BC@webmail.messagingengine.com> On Sun, Nov 30, 2014, at 11:45, Donald Stufft wrote: > > > On Nov 30, 2014, at 11:28 AM, Brett Cannon wrote: > > > > > > > > On Sat Nov 29 2014 at 7:16:34 PM Alex Gaynor > wrote: > > Donald Stufft stufft.io > writes: > > > > > > > > [words words words] > > > > > > > I strongly support this PEP. I'd like to share two pieces of information. Both > > of these are personal anecdotes: > > > > For the past several years, I've been a contributor on two major projects using > > mercurial, CPython and PyPy. PyPy has a strong culture of in-repo branching, > > basically all contributors regularly make branches in the main repo for their > > work, and we're very free in giving people commit rights, so almost everyone > > working on PyPy in any way has this level of access. This workflow works ok. I > > don't love it as much as git, but it's fine, it's not an impediment to my work. > > > > By contrast, CPython does not have this type of workflow, there are almost no > > in-tree branches besides the 2.7, 3.4, etc. ones. Despite being a regular hg > > user for years, I have no idea how to create a local-only branch, or a branch > > which is pushed to a remote (to use the git term). I also don't generally > > commit my own work to CPython, even though I have push privledges, > > because I > > prefer to *always* get code review on my work. As a result, I use a git mirror > > of CPython to do all my work, and generate patches from that. > > > > The conclusion I draw from this is that hg's workflow is probably fine if > > you're a committer on the project, or don't ever need to maintain multiple > > patches concurrently (and thus can just leave everything uncommitted in the > > repo). However, the hg workflow seems extremely defficient at non-committer > > contributors. > > > > One way to come close to that using hg is to have your own clone that you never push to hg.python.org/cpython (e.g. cloning the Bitbucket clone of cpython or hosting on hg.python.org a personal clone). You can then specify the repo and branch on the issue tracker to generate your patch: https://docs.python.org/devguide/triaging.html#mercurial-repository . After that it's just like any other patch workflow for core devs. It's not quite as nice as maybe using named branches where you can just do a final hg merge/commit to get your changes committed, but if you're not going to commit your branches then you might as well get the automatic patch generation perk in the issue tracker rather than using git (unless there is some other git feature you use that you can't get in hg). > > This doesn?t really work in a Pull Request workflow though I think is the > point. > Uploading patches is it?s own kind of terrible but yes if you?re just > uploading > patches you can probably do that. I don?t make branches in Mercurial > because > i?m afraid I?m going to push a permanent branch to hg.python.org > and screw > something up. Don't worry; we have a hook for that. From barry at python.org Sun Nov 30 17:55:57 2014 From: barry at python.org (Barry Warsaw) Date: Sun, 30 Nov 2014 11:55:57 -0500 Subject: [Python-Dev] PEP 481 - Migrate Some Supporting Repositories to Git and Github In-Reply-To: References: <30725AC9-3DCF-416A-BCCD-2D64D489898C@stufft.io> <85d285wa3c.fsf@benfinney.id.au> <20141130140137.46803df1@fsol> Message-ID: <20141130115557.427918a2@limelight.wooz.org> On Nov 30, 2014, at 09:54 AM, Ian Cordasco wrote: >- Migrating "data" from GitHub is easy. There are free-as-in-freedom >tools to do it and the only cost is the time it would take to monitor >the process *Extracting* data may be easy, but migrating it is a different story. As the Mailman project has seen in trying to migrate from Confluence to Moin, there is a ton of very difficult work involved after extracting the data. Parsing the data, ensuring that you have all the bits you need, fitting it into the new system's schema, working out the edge cases, adapting to semantic differences and gaps, ensuring that all the old links are redirected, and so on, were all exceedingly difficult[*]. Even converting between two FLOSS tools is an amazing amount of work. Look at what Eric Raymond did with reposurgeon to convert from Bazaar to git. It's a good thing that your data isn't locked behind a proprietary door, for now. That's only part of the story. But also, because github is a closed system, there's no guarantee that today's data-freeing APIs will still exist, continue to be functional for practical purposes, remain complete, or stay at parity with new features. Cheers, -Barry [*] And our huge gratitude goes to Paul Boddie for his amazing amount of work on the project. From donald at stufft.io Sun Nov 30 18:00:18 2014 From: donald at stufft.io (Donald Stufft) Date: Sun, 30 Nov 2014 12:00:18 -0500 Subject: [Python-Dev] PEP 481 - Migrate Some Supporting Repositories to Git and Github In-Reply-To: References: <30725AC9-3DCF-416A-BCCD-2D64D489898C@stufft.io> <85d285wa3c.fsf@benfinney.id.au> <20141130140137.46803df1@fsol> Message-ID: <27ABC536-C712-4109-9ED7-3B15C7818143@stufft.io> > On Nov 30, 2014, at 11:44 AM, Brett Cannon wrote: > > > > On Sun Nov 30 2014 at 10:55:26 AM Ian Cordasco > wrote: > On Sun, Nov 30, 2014 at 7:01 AM, Antoine Pitrou > wrote: > > On Sun, 30 Nov 2014 16:23:08 +1100 > > Chris Angelico > wrote: > >> > >> Yes, GitHub is proprietary. But all of your actual code is stored in > >> git, which is free, and it's easy to push that to a new host somewhere > >> else, or create your own host. This proposal is for repositories that > >> don't need much in the way of issue trackers etc, so shifting away > >> from GitHub shouldn't demand anything beyond moving the repos > >> themselves. > > > > I hope we're not proposing to move the issue trackers to github, > > otherwise I'm -1 on this PEP. > > > > Regards > > > > Antoine. > > So I usually choose not to weigh in on discussions like these but > there seems to be a lot of misdirection in some of these arguments. > > To start, I'm generally neutral about this proposal or Nick's proposal > that spurred this one. I've found the most frustrating part of > contributing to anything involving CPython to be the lack of reviewer > time. I have had very small (2-line) patches take months (close to a > year in reality) to get through in spite of periodically pinging the > appropriate people. Moving to git/GitHub will not alleviate this at > all. > > To be clear, the main reasoning behind Nick's was being able to submit > changes without ever having to have a local copy of the repository in > question on your machine. Having a complete web workflow for editing > and contributing makes the barrier to entry far lower than switching > VCS or anything else. BitBucket (apparently, although I've never used > this) and GitHub both have this capability and *both* are > free-as-in-beer systems. > > No one as I understand it is proposing that we use the per-distro > proprietary interface to these websites. > > All data can be removed from GitHub using it's API and can generally > be converted to another platform. The same goes for BitBucket although > it's arguably easier to retrieve issue data from BitBucket than > GitHub. That said, *the issue tracker is not covered by these > proposals* so this is a moot point. Drop it already. > > If we're seriously considering moving to git as a DVCS, we should > consider the real free-as-in-freedom alternatives that come very close > to GitHub and can be improved by us (even though they're not written > in Python). One of those is GitLab. We can self-host a GitLab instance > easily or we can rely on gitlab.com . GitLab aims to provide a very > similar user experience to GitHub and it's slowly approaching feature > parity and experience parity. GitLab is also what a lot of people > chose to switch to after the incident Steven mentioned, which I don't > think is something we should dismiss or ignore. > > We should refocus the discussion with the following in mind: > > - Migrating "data" from GitHub is easy. There are free-as-in-freedom > tools to do it and the only cost is the time it would take to monitor > the process > > - GitHub has a toxic company culture that we should be aware of before > moving to it. They have a couple blog posts about attempting to change > it but employees became eerily silent after the incident and have > remained so from what I've personally seen. > > - GitHub may be popular but there are popular FOSS solutions that > exist that we can also self-host at something like forge.python.org > > - bugs.python.org is not covered by any of these proposals > > - The main benefit this proposal (and the proposal to move to > BitBucket) are seeking to achieve is an online editing experience > allowing for *anyone with a browser and an account* to contribute. > This to me is the only reason I would be +1 for either of these > proposals (if we can reach consensus). > > But that's not just it. As you pointed out, Ian, getting patch submissions committed faster would be a huge improvement over what we have today. GitHub/Bitbucket/whatever could help with this by giving core devs basic CI to know that I patch is sound to some extent as well as push button commits of patches. > > For me personally, if I knew a simple patch integrated cleanly and passed on at least one buildbot -- when it wasn't a platform-specific fix -- then I could easily push a "Commit" button and be done with it (although this assumes single branch committing; doing this across branches makes all of this difficult unless we finally resolve our Misc/NEWS conflict issues so that in some instances it can be automated). Instead I have to wait until I have a clone I can push from, download a patch, apply it, run the unit tests myself, do the commit, and then repeat a subset of that to whatever branches make sense. It's a lot of work for which some things could be automated. Well there?s two sides to the contribution process. There?s making things better/easier for people who *aren?t* committers and there is making things better/easier for people who *are* committers. Tacking extra things on to what we already have to improve the life of committers is easier in many ways. As committers they?ve likely already taken the time to learn the bespoke workflow that the Python project uses and have already gotten through that particular hurdle. Looking to standardize around popular tools is mostly about making it easier for *new* people and making it so that if they learn this set of tools they can go an immediately apply that to most of the other Python projects out there, or that if they are already contributing to those other Python projects they are probably aware of this particular toolchain and workflow and can apply that knowledge directly to the Python project. Moving to some of these tools happens to come with it features like really nice CI integration and a nice "Merge" button that also make it a lot nicer for the committer side of things. I think it's also hard to get a representation of the people for whom the bespoke workflow and less popular tooling are a problem for in a discussion on python-dev. My guess is most of those people would not have signed up for python-dev since, unless they were willing to take the time to learn that, so there is an amount of selection bias at play here as well. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Sun Nov 30 18:09:07 2014 From: donald at stufft.io (Donald Stufft) Date: Sun, 30 Nov 2014 12:09:07 -0500 Subject: [Python-Dev] PEP 481 - Migrate Some Supporting Repositories to Git and Github In-Reply-To: <20141130115557.427918a2@limelight.wooz.org> References: <30725AC9-3DCF-416A-BCCD-2D64D489898C@stufft.io> <85d285wa3c.fsf@benfinney.id.au> <20141130140137.46803df1@fsol> <20141130115557.427918a2@limelight.wooz.org> Message-ID: <31195B20-A4BE-441C-ADC5-31236E2C4E1B@stufft.io> > On Nov 30, 2014, at 11:55 AM, Barry Warsaw wrote: > > On Nov 30, 2014, at 09:54 AM, Ian Cordasco wrote: > >> - Migrating "data" from GitHub is easy. There are free-as-in-freedom >> tools to do it and the only cost is the time it would take to monitor >> the process > > *Extracting* data may be easy, but migrating it is a different story. As the > Mailman project has seen in trying to migrate from Confluence to Moin, there > is a ton of very difficult work involved after extracting the data. Parsing > the data, ensuring that you have all the bits you need, fitting it into the > new system's schema, working out the edge cases, adapting to semantic > differences and gaps, ensuring that all the old links are redirected, and so > on, were all exceedingly difficult[*]. > > Even converting between two FLOSS tools is an amazing amount of work. Look at > what Eric Raymond did with reposurgeon to convert from Bazaar to git. I fail to see how this is a reasonable argument to make at all since, as you mentioned, converting between two FLOSS tools can be an amazing amount of work. Realistically the amount of work is going to be predicated on whether or not there is a tool that already handles the conversion for you. Assuming of course that the data is available to you at all. As a particularly relevant example, switching from Mercurial to Git is as easy as installing hg-git, creating a bookmark for master that tracks default, and then pushing to a git repository. > > It's a good thing that your data isn't locked behind a proprietary door, for > now. That's only part of the story. But also, because github is a closed > system, there's no guarantee that today's data-freeing APIs will still exist, > continue to be functional for practical purposes, remain complete, or stay at > parity with new features. This feels like Chicken Little-ing. If Github closed it?s APIs then you could still get at that data by scraping the web interface. However why would Github do that? That would piss off the vast majority of OSS projects who are currently hosted there and is likely to cause a pretty big migration off of Github for fear that Github is going to attempt to lock people onto Github. The popularity of Github *is* Github?s killer feature and doing something that is going to send the vast bulk of your users running for the hills sounds like something that they would have to be particularly stupid to do. > > Cheers, > -Barry > > [*] And our huge gratitude goes to Paul Boddie for his amazing amount of work > on the project. > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/donald%40stufft.io --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA From donald at stufft.io Sun Nov 30 18:14:20 2014 From: donald at stufft.io (Donald Stufft) Date: Sun, 30 Nov 2014 12:14:20 -0500 Subject: [Python-Dev] PEP 481 - Migrate Some Supporting Repositories to Git and Github In-Reply-To: <31195B20-A4BE-441C-ADC5-31236E2C4E1B@stufft.io> References: <30725AC9-3DCF-416A-BCCD-2D64D489898C@stufft.io> <85d285wa3c.fsf@benfinney.id.au> <20141130140137.46803df1@fsol> <20141130115557.427918a2@limelight.wooz.org> <31195B20-A4BE-441C-ADC5-31236E2C4E1B@stufft.io> Message-ID: > On Nov 30, 2014, at 12:09 PM, Donald Stufft wrote: > >> >> On Nov 30, 2014, at 11:55 AM, Barry Warsaw wrote: >> >> On Nov 30, 2014, at 09:54 AM, Ian Cordasco wrote: >> >>> - Migrating "data" from GitHub is easy. There are free-as-in-freedom >>> tools to do it and the only cost is the time it would take to monitor >>> the process >> >> *Extracting* data may be easy, but migrating it is a different story. As the >> Mailman project has seen in trying to migrate from Confluence to Moin, there >> is a ton of very difficult work involved after extracting the data. Parsing >> the data, ensuring that you have all the bits you need, fitting it into the >> new system's schema, working out the edge cases, adapting to semantic >> differences and gaps, ensuring that all the old links are redirected, and so >> on, were all exceedingly difficult[*]. >> >> Even converting between two FLOSS tools is an amazing amount of work. Look at >> what Eric Raymond did with reposurgeon to convert from Bazaar to git. > > I fail to see how this is a reasonable argument to make at all since, as you > mentioned, converting between two FLOSS tools can be an amazing amount of work. > Realistically the amount of work is going to be predicated on whether or not > there is a tool that already handles the conversion for you. Assuming of course > that the data is available to you at all. > > As a particularly relevant example, switching from Mercurial to Git is as easy > as installing hg-git, creating a bookmark for master that tracks default, and > then pushing to a git repository. When looking for a tool that did this (specifically Github -> Gitlab because the two are most similar) I found https://gitlab.com/sigmavirus24/issues-migration/blob/master/migrate.py which happens to be written by Ian. I would guess that he is likely speaking from experience about migrating off of Github to go to Gitlab. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA From graffatcolmingov at gmail.com Sun Nov 30 18:24:18 2014 From: graffatcolmingov at gmail.com (Ian Cordasco) Date: Sun, 30 Nov 2014 11:24:18 -0600 Subject: [Python-Dev] PEP 481 - Migrate Some Supporting Repositories to Git and Github In-Reply-To: <31195B20-A4BE-441C-ADC5-31236E2C4E1B@stufft.io> References: <30725AC9-3DCF-416A-BCCD-2D64D489898C@stufft.io> <85d285wa3c.fsf@benfinney.id.au> <20141130140137.46803df1@fsol> <20141130115557.427918a2@limelight.wooz.org> <31195B20-A4BE-441C-ADC5-31236E2C4E1B@stufft.io> Message-ID: On Nov 30, 2014 11:09 AM, "Donald Stufft" wrote: > > > > On Nov 30, 2014, at 11:55 AM, Barry Warsaw wrote: > > > > On Nov 30, 2014, at 09:54 AM, Ian Cordasco wrote: > > > >> - Migrating "data" from GitHub is easy. There are free-as-in-freedom > >> tools to do it and the only cost is the time it would take to monitor > >> the process > > > > *Extracting* data may be easy, but migrating it is a different story. As the > > Mailman project has seen in trying to migrate from Confluence to Moin, there > > is a ton of very difficult work involved after extracting the data. Parsing > > the data, ensuring that you have all the bits you need, fitting it into the > > new system's schema, working out the edge cases, adapting to semantic > > differences and gaps, ensuring that all the old links are redirected, and so > > on, were all exceedingly difficult[*]. > > > > Even converting between two FLOSS tools is an amazing amount of work. Look at > > what Eric Raymond did with reposurgeon to convert from Bazaar to git. > > I fail to see how this is a reasonable argument to make at all since, as you > mentioned, converting between two FLOSS tools can be an amazing amount of work. > Realistically the amount of work is going to be predicated on whether or not > there is a tool that already handles the conversion for you. Assuming of course > that the data is available to you at all. > > As a particularly relevant example, switching from Mercurial to Git is as easy > as installing hg-git, creating a bookmark for master that tracks default, and > then pushing to a git repository. > > > > > It's a good thing that your data isn't locked behind a proprietary door, for > > now. That's only part of the story. But also, because github is a closed > > system, there's no guarantee that today's data-freeing APIs will still exist, > > continue to be functional for practical purposes, remain complete, or stay at > > parity with new features. > > This feels like Chicken Little-ing. If Github closed it?s APIs then you could > still get at that data by scraping the web interface. However why would Github > do that? That would piss off the vast majority of OSS projects who are currently > hosted there and is likely to cause a pretty big migration off of Github for > fear that Github is going to attempt to lock people onto Github. The popularity > of Github *is* Github?s killer feature and doing something that is going to > send the vast bulk of your users running for the hills sounds like something that > they would have to be particularly stupid to do. > > > > > Cheers, > > -Barry > > > > [*] And our huge gratitude goes to Paul Boddie for his amazing amount of work > > on the project. > > _______________________________________________ > > Python-Dev mailing list > > Python-Dev at python.org > > https://mail.python.org/mailman/listinfo/python-dev > > Unsubscribe: https://mail.python.org/mailman/options/python-dev/donald%40stufft.io > > --- > Donald Stufft > PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA I tend to agree with Donald that it's highly unlikely for GitHub to close off their APIs but Barry is right that it isn't impossible. That can be mitigated by regularly scheduling a back-up of that data using the tools we have now so that the sky doesn't appear to be falling if the worst case does occur. I also tend to disagree with Barry that it will be extraordinarily difficult because I have migrated issues and pull requests off of GitHub and was able to automate the entirety of it reliably with python. Admittedly, I'm very familiar with GitHub's API as the author of github3.py so for me this is a trivial task. I would also be willing to help set up migrations and back ups if we decide to use GitHub. -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Sun Nov 30 18:31:17 2014 From: donald at stufft.io (Donald Stufft) Date: Sun, 30 Nov 2014 12:31:17 -0500 Subject: [Python-Dev] PEP 481 - Migrate Some Supporting Repositories to Git and Github In-Reply-To: <50A56F56-E636-43BB-BF5D-8DC30920BDB1@stufft.io> References: <30725AC9-3DCF-416A-BCCD-2D64D489898C@stufft.io> <4ADD805B-B707-4244-9290-77AB0DAE396A@stufft.io> <547AC258.5090803@hastings.org> <50A56F56-E636-43BB-BF5D-8DC30920BDB1@stufft.io> Message-ID: <12920E9A-3751-40FA-970D-39B9580CBA6A@stufft.io> > On Nov 30, 2014, at 11:30 AM, Donald Stufft wrote: > > Comments like this make me feel like I didn?t explain myself very well in the > PEP. It?s been pointed out to me that Mercurial bookmarks have been core since 1.8 and since I felt like the technical arguments were really secondary to the social arguments I?ve gone ahead and removed that section from the PEP. I?ve also added the read only Mercurial repository on hg.python.org See changes at https://hg.python.org/peps/rev/d31fe28e2766 --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: From ethan at stoneleaf.us Sun Nov 30 07:39:43 2014 From: ethan at stoneleaf.us (Ethan Furman) Date: Sat, 29 Nov 2014 22:39:43 -0800 Subject: [Python-Dev] PEP 481 - Migrate Some Supporting Repositories to Git and Github In-Reply-To: References: <30725AC9-3DCF-416A-BCCD-2D64D489898C@stufft.io> Message-ID: <547ABBAF.2060000@stoneleaf.us> On 11/29/2014 10:14 PM, Demian Brecht wrote: > > On Sat, Nov 29, 2014 at 3:27 PM, Donald Stufft > wrote: >> As promised in the "Move selected documentation repos to PSF BitBucket >> account?" thread I've written up a PEP for moving selected repositories from >> hg.python.org to Github. > > > FWIW, I'm a pretty solid -1 to this PEP. Agreed. -1 for the same reasons (plus most of Nick's). -- ~Ethan~ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: OpenPGP digital signature URL: From glyph at twistedmatrix.com Sun Nov 30 17:36:52 2014 From: glyph at twistedmatrix.com (Glyph) Date: Sun, 30 Nov 2014 17:36:52 +0100 Subject: [Python-Dev] PEP 481 - Migrate Some Supporting Repositories to Git and Github In-Reply-To: References: <30725AC9-3DCF-416A-BCCD-2D64D489898C@stufft.io> <85d285wa3c.fsf@benfinney.id.au> Message-ID: <63DD4EAE-C63B-4AA7-8549-5E11935476DF@twistedmatrix.com> > On Nov 30, 2014, at 11:17, Chris Angelico wrote: > > On Sun, Nov 30, 2014 at 8:54 PM, Nick Coghlan wrote: >> On 30 November 2014 at 15:23, Chris Angelico wrote: >>> Python is already using quite a bit of non-free software in its >>> ecosystem. The Windows builds of CPython are made with Microsoft's >>> compiler, and the recent discussion about shifting to Cygwin or MinGW >>> basically boiled down to "but it ought to be free software", and that >>> was considered not a sufficiently strong argument. In each case, the >>> decision has impact on other people (using MSVC for the official >>> python.org installers means extension writers need to use MSVC too; >>> and using GitHub means that contributors are strongly encouraged, >>> possibly required, to use GitHub); so why is it acceptable to use a >>> non-free compiler, but not acceptable to use a non-free host? >> >> Relying on non-free software to support users of a non-free platform >> is rather different from *requiring* the use of non-free software to >> participate in core Python community design processes. > > But what non-free software is required to use the community design > processes? The GitHub client is entirely optional; I don't use it, I > just use git itself. Using a free client to access a proprietary > server isn't the same as using non-free software. Also keep in mind that unless you are using a very esoteric hardware setup and dedicated leased lines that you purchased yourself, you are likely to be using copyrighted, patented, proprietary software at a number of levels: the microcode implementation in your CPU the firmware in your GPU the firmware in your network card the boot code (e.g.: BIOS or EFI implementation) of your motherboard the firmware in your router or the firmware in your cable or DSL modem, if you thought to get a free router with OpenWRT or something in it the firmware in your ISP's router the firmware in the backbone's routers the firmware in the PSF's ISP's routers Does this sound like ridiculous nitpicking? Of course it does! If you refused to use all that stuff you just wouldn't be able to access the internet at all, regardless of your personal choices. Most layers of this stack are _less_ constrained to open standards and open data access than Github, and can require layers and layers of additional proprietary software or reverse engineering (ask anyone who has tried to play a video game on a Linux computer what the experience is like without gobs of proprietary blobs from nvidia or ATI). And as the story of BitKeeper shows, if a proprietary platform decides to do something bad, if the cost of migration is within your means, you can just leave. This is far more true of Github than Bitkeeper - Linux had to create a totally new VCS to migrate off of that, we just have to install Trac or Roundup or something again. (Which, as per the presently-proposed PEP, we don't even have to install them again, just change some configuration to put a few repositories back.) The monoculture about Github concerns me. I also have concerns about the long-term consequences of not having an all-free-software stack. But focusing on avoiding services like Github at this point in history is just a gigantic waste of time; it's resolving dependencies in the wrong order. The only reason to avoid Github is ideological purity, and even then it's not even "purity" because you still have to accept this other ideological contamination. Except, what about other ideological concepts that are important to the Python core developers, like equitable representation for women and minorities, and compassionate and polite interactions within the community? Clearly we can't require the use of Linux. If we treat these ideals as an equivalent priority as free software (even if we ignore the "invisible" dependencies like the list I just made above), then there is literally no software left that we can use to develop Python. Linux kernel and GNU low-level user-land development are a perpetual cesspit, and their leaders serve as a continuous embarrassment to our community. And speaking of equitable representation, one proven technique we have learned for dealing with that problem is making it for newcomers of all stripes to access the community. Like it or not, Github's popularity means that it's a place where most newcomers to programming are learning to use version control and bug tracking. This is a network effect that can have a real impact on people's lives. Requiring newcomers to learn our weird technology religion before they can contribute creates a real barrier to entry, which in turn makes our community more insular and homogenous. Some days you get the Git, and some days the Github gets you. The sooner we, as a community and a culture, can accept this and move on, the more time will be available to actually build replacements for these things. -glyph -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Sun Nov 30 19:05:01 2014 From: guido at python.org (Guido van Rossum) Date: Sun, 30 Nov 2014 10:05:01 -0800 Subject: [Python-Dev] PEP 481 - Migrate Some Supporting Repositories to Git and Github In-Reply-To: References: <30725AC9-3DCF-416A-BCCD-2D64D489898C@stufft.io> <85d285wa3c.fsf@benfinney.id.au> <20141130140137.46803df1@fsol> <20141130115557.427918a2@limelight.wooz.org> <31195B20-A4BE-441C-ADC5-31236E2C4E1B@stufft.io> Message-ID: I don't feel it's my job to accept or reject this PEP, but I do have an opinion. The scope of the PSF organization is far beyond just the Python language -- it includes the Python developer community, the Python user community, 3rd party Python packages and their communities (even if some have created their own organizations). But I think that it is "scope creep" to try and be "pure" in our tooling or workflows. Python has a long history (all the way back to my choice of a MIT-style license for the first release) of mixing "free" and "non-free" uses and tools -- for example on Windows we consciously chose to align ourselves with the platform tooling rather than with the (minority) free tools available, Python has been ported to many commercial platforms, and I've always encouraged use of Python in closed-source situations. I bring this up to emphasize that (unlike GNU software and the FSF) Python has no additional hidden agenda of bringing freedom to all software. At least that's how I feel about it -- clearly some of the most vocal contributors to this thread feel differently. Now some entirely practical points. - I am basically the only remaining active PEP editor, so I see most PEP contributions by non-core-committers. Almost all of these uses github. Not bitbucket, not some other git host, but github. I spend a fair amount of time applying patches. It would most definitely be easier if I could get them to send me pull requests. - I am not worried about "lock in". The most important stuff is copied in the local git repos of hundreds of core devs and thousands of others. Pull requests are by nature short-lived -- and if you really want a history of the back-and-forth that led to the eventual set of diffs that was integrated, you could subscribe a mailing list to it to archive it. I'm sure there's a way to back up the issue tracker too. Finally. And this may actually be the most important point. Python people should be doing stuff that makes Python better (both taken in the most inclusive way possible). For stuff that's not unique to Python but can be used by many other open-source projects, such as compilers, DVCS tools, or mailing lists, we should not be wasting our precious time on building and maintaining our own tools or administering the servers on which they run. And historically we've not done a great job on maintenance and administration. Of course it's fun to make tools in Python, and to see them used beyond the Python world. But that's an entirely different argument from what I hear. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Sun Nov 30 19:14:25 2014 From: p.f.moore at gmail.com (Paul Moore) Date: Sun, 30 Nov 2014 18:14:25 +0000 Subject: [Python-Dev] PEP 481 - Migrate Some Supporting Repositories to Git and Github In-Reply-To: <9559CB2D-7237-4D05-98F9-ACAE4354042A@stufft.io> References: <30725AC9-3DCF-416A-BCCD-2D64D489898C@stufft.io> <9559CB2D-7237-4D05-98F9-ACAE4354042A@stufft.io> Message-ID: On 30 November 2014 at 16:08, Donald Stufft wrote: >> On Nov 30, 2014, at 7:31 AM, Paul Moore wrote: >> >> On 29 November 2014 at 23:27, Donald Stufft wrote: >>> In previous years there was concern about how well supported git was on Windows >>> in comparison to Mercurial. However git has grown to support Windows as a first >>> class citizen. In addition to that, for Windows users who are not well aquanted >>> with the Windows command line there are GUI options as well. >> >> I have little opinion on the PEP as a whole, but is the above >> statement true? From the git website, version 2.2.0 is current, and >> yet the downloadable Windows version is still 1.9.4. That's a fairly >> significant version lag for a "first class citizen". >> >> I like git, and it has a number of windows-specific extensions that >> are really useful (more than Mercurial, AFAIK), but I wouldn't say >> that the core product supported Windows on an equal footing to Linux. >> >> Paul > > I think so yes. I may be wrong, however while 1.9.4 may be the latest > downloadable version of git for Windows, there is no downloadable > version of the Linux clients at all, they just tell you to go use > your package manager which for instance is version 1.7 on Debian. On > OS X the latest version is 2.0.1. OTOH, presumably you can build your own copy of git from source on Linux/OSX. I haven't tried this on Windows but it looks pretty difficult (you start by downloading the msysgit development environment and go from there). Also, if it's easy to produce binaries for 2.2.0 on Windows, why haven't the msysgit project (still an external project, to an extent, AFAICT) done so? Paul From graffatcolmingov at gmail.com Sun Nov 30 19:21:15 2014 From: graffatcolmingov at gmail.com (Ian Cordasco) Date: Sun, 30 Nov 2014 12:21:15 -0600 Subject: [Python-Dev] PEP 481 - Migrate Some Supporting Repositories to Git and Github In-Reply-To: References: <30725AC9-3DCF-416A-BCCD-2D64D489898C@stufft.io> <9559CB2D-7237-4D05-98F9-ACAE4354042A@stufft.io> Message-ID: Can this discussion be split off into a separate discussion. It's tangential to the PEP and clearly not actively progressing so it doesn't seem productive. I don't care where it's taken, but I don't think this belongs here. Speculation on the actions of the msysgit project are not fair talk for this PEP. On Sun, Nov 30, 2014 at 12:14 PM, Paul Moore wrote: > On 30 November 2014 at 16:08, Donald Stufft wrote: >>> On Nov 30, 2014, at 7:31 AM, Paul Moore wrote: >>> >>> On 29 November 2014 at 23:27, Donald Stufft wrote: >>>> In previous years there was concern about how well supported git was on Windows >>>> in comparison to Mercurial. However git has grown to support Windows as a first >>>> class citizen. In addition to that, for Windows users who are not well aquanted >>>> with the Windows command line there are GUI options as well. >>> >>> I have little opinion on the PEP as a whole, but is the above >>> statement true? From the git website, version 2.2.0 is current, and >>> yet the downloadable Windows version is still 1.9.4. That's a fairly >>> significant version lag for a "first class citizen". >>> >>> I like git, and it has a number of windows-specific extensions that >>> are really useful (more than Mercurial, AFAIK), but I wouldn't say >>> that the core product supported Windows on an equal footing to Linux. >>> >>> Paul >> >> I think so yes. I may be wrong, however while 1.9.4 may be the latest >> downloadable version of git for Windows, there is no downloadable >> version of the Linux clients at all, they just tell you to go use >> your package manager which for instance is version 1.7 on Debian. On >> OS X the latest version is 2.0.1. > > OTOH, presumably you can build your own copy of git from source on > Linux/OSX. I haven't tried this on Windows but it looks pretty > difficult (you start by downloading the msysgit development > environment and go from there). Also, if it's easy to produce binaries > for 2.2.0 on Windows, why haven't the msysgit project (still an > external project, to an extent, AFAICT) done so? > > Paul > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/graffatcolmingov%40gmail.com From donald at stufft.io Sun Nov 30 19:25:39 2014 From: donald at stufft.io (Donald Stufft) Date: Sun, 30 Nov 2014 13:25:39 -0500 Subject: [Python-Dev] PEP 481 - Migrate Some Supporting Repositories to Git and Github In-Reply-To: References: <30725AC9-3DCF-416A-BCCD-2D64D489898C@stufft.io> <85d285wa3c.fsf@benfinney.id.au> <20141130140137.46803df1@fsol> <20141130115557.427918a2@limelight.wooz.org> <31195B20-A4BE-441C-ADC5-31236E2C4E1B@stufft.io> Message-ID: <17CA895B-DDCA-44B7-A94E-0B74765CA7AD@stufft.io> > On Nov 30, 2014, at 1:05 PM, Guido van Rossum wrote: > > I don't feel it's my job to accept or reject this PEP, but I do have an opinion. So here?s a question. If it?s not your job to accept or reject this PEP, whose is it? This is probably an issue we?re never going to get actual consensus on so unless there is an arbitrator of who gets to decide I feel it?s probably a waste of my time to try and convince absolutely *everyone*. > > The scope of the PSF organization is far beyond just the Python language -- it includes the Python developer community, the Python user community, 3rd party Python packages and their communities (even if some have created their own organizations). But I think that it is "scope creep" to try and be "pure" in our tooling or workflows. > > Python has a long history (all the way back to my choice of a MIT-style license for the first release) of mixing "free" and "non-free" uses and tools -- for example on Windows we consciously chose to align ourselves with the platform tooling rather than with the (minority) free tools available, Python has been ported to many commercial platforms, and I've always encouraged use of Python in closed-source situations. > > I bring this up to emphasize that (unlike GNU software and the FSF) Python has no additional hidden agenda of bringing freedom to all software. At least that's how I feel about it -- clearly some of the most vocal contributors to this thread feel differently. Yes, this is how I feel about Python too, that it?s the pragmatic language/community not the purist language/community. > > Now some entirely practical points. > > - I am basically the only remaining active PEP editor, so I see most PEP contributions by non-core-committers. Almost all of these uses github. Not bitbucket, not some other git host, but github. I spend a fair amount of time applying patches. It would most definitely be easier if I could get them to send me pull requests. > > - I am not worried about "lock in". The most important stuff is copied in the local git repos of hundreds of core devs and thousands of others. Pull requests are by nature short-lived -- and if you really want a history of the back-and-forth that led to the eventual set of diffs that was integrated, you could subscribe a mailing list to it to archive it. I'm sure there's a way to back up the issue tracker too. > > Finally. And this may actually be the most important point. Python people should be doing stuff that makes Python better (both taken in the most inclusive way possible). For stuff that's not unique to Python but can be used by many other open-source projects, such as compilers, DVCS tools, or mailing lists, we should not be wasting our precious time on building and maintaining our own tools or administering the servers on which they run. And historically we've not done a great job on maintenance and administration. > > Of course it's fun to make tools in Python, and to see them used beyond the Python world. But that's an entirely different argument from what I hear. > > -- > --Guido van Rossum (python.org/~guido ) --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Sun Nov 30 20:15:50 2014 From: guido at python.org (Guido van Rossum) Date: Sun, 30 Nov 2014 11:15:50 -0800 Subject: [Python-Dev] advice needed: best approach to enabling "metamodules"? In-Reply-To: References: Message-ID: On Sun, Nov 30, 2014 at 6:15 AM, Brett Cannon wrote: > > On Sat, Nov 29, 2014, 21:55 Guido van Rossum wrote: > > All the use cases seem to be about adding some kind of getattr hook to > modules. They all seem to involve modifying the CPython C code anyway. So > why not tackle that problem head-on and modify module_getattro() to look > for a global named __getattr__ and if it exists, call that instead of > raising AttributeError? > > Not sure if anyone thought of it. :) Seems like a reasonable solution to > me. Be curious to know what the benchmark suite said the impact was. > Why would there be any impact? The __getattr__ hook would be similar to the one on classes -- it's only invoked at the point where otherwise AttributeError would be raised. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Sun Nov 30 20:19:50 2014 From: brett at python.org (Brett Cannon) Date: Sun, 30 Nov 2014 19:19:50 +0000 Subject: [Python-Dev] PEP 481 - Migrate Some Supporting Repositories to Git and Github References: <30725AC9-3DCF-416A-BCCD-2D64D489898C@stufft.io> <85d285wa3c.fsf@benfinney.id.au> <20141130140137.46803df1@fsol> <27ABC536-C712-4109-9ED7-3B15C7818143@stufft.io> Message-ID: On Sun Nov 30 2014 at 12:00:20 PM Donald Stufft wrote: > > On Nov 30, 2014, at 11:44 AM, Brett Cannon wrote: > > > > On Sun Nov 30 2014 at 10:55:26 AM Ian Cordasco > wrote: > >> On Sun, Nov 30, 2014 at 7:01 AM, Antoine Pitrou >> wrote: >> > On Sun, 30 Nov 2014 16:23:08 +1100 >> > Chris Angelico wrote: >> >> >> >> Yes, GitHub is proprietary. But all of your actual code is stored in >> >> git, which is free, and it's easy to push that to a new host somewhere >> >> else, or create your own host. This proposal is for repositories that >> >> don't need much in the way of issue trackers etc, so shifting away >> >> from GitHub shouldn't demand anything beyond moving the repos >> >> themselves. >> > >> > I hope we're not proposing to move the issue trackers to github, >> > otherwise I'm -1 on this PEP. >> > >> > Regards >> > >> > Antoine. >> >> So I usually choose not to weigh in on discussions like these but >> there seems to be a lot of misdirection in some of these arguments. >> >> To start, I'm generally neutral about this proposal or Nick's proposal >> that spurred this one. I've found the most frustrating part of >> contributing to anything involving CPython to be the lack of reviewer >> time. I have had very small (2-line) patches take months (close to a >> year in reality) to get through in spite of periodically pinging the >> appropriate people. Moving to git/GitHub will not alleviate this at >> all. >> >> To be clear, the main reasoning behind Nick's was being able to submit >> changes without ever having to have a local copy of the repository in >> question on your machine. Having a complete web workflow for editing >> and contributing makes the barrier to entry far lower than switching >> VCS or anything else. BitBucket (apparently, although I've never used >> this) and GitHub both have this capability and *both* are >> free-as-in-beer systems. >> >> No one as I understand it is proposing that we use the per-distro >> proprietary interface to these websites. >> >> All data can be removed from GitHub using it's API and can generally >> be converted to another platform. The same goes for BitBucket although >> it's arguably easier to retrieve issue data from BitBucket than >> GitHub. That said, *the issue tracker is not covered by these >> proposals* so this is a moot point. Drop it already. >> >> If we're seriously considering moving to git as a DVCS, we should >> consider the real free-as-in-freedom alternatives that come very close >> to GitHub and can be improved by us (even though they're not written >> in Python). One of those is GitLab. We can self-host a GitLab instance >> easily or we can rely on gitlab.com. GitLab aims to provide a very >> similar user experience to GitHub and it's slowly approaching feature >> parity and experience parity. GitLab is also what a lot of people >> chose to switch to after the incident Steven mentioned, which I don't >> think is something we should dismiss or ignore. >> >> We should refocus the discussion with the following in mind: >> >> - Migrating "data" from GitHub is easy. There are free-as-in-freedom >> tools to do it and the only cost is the time it would take to monitor >> the process >> >> - GitHub has a toxic company culture that we should be aware of before >> moving to it. They have a couple blog posts about attempting to change >> it but employees became eerily silent after the incident and have >> remained so from what I've personally seen. >> >> - GitHub may be popular but there are popular FOSS solutions that >> exist that we can also self-host at something like forge.python.org >> >> - bugs.python.org is not covered by any of these proposals >> >> - The main benefit this proposal (and the proposal to move to >> BitBucket) are seeking to achieve is an online editing experience >> allowing for *anyone with a browser and an account* to contribute. >> This to me is the only reason I would be +1 for either of these >> proposals (if we can reach consensus). >> > > But that's not just it. As you pointed out, Ian, getting patch submissions > committed faster would be a huge improvement over what we have today. > GitHub/Bitbucket/whatever could help with this by giving core devs basic CI > to know that I patch is sound to some extent as well as push button commits > of patches. > > For me personally, if I knew a simple patch integrated cleanly and passed > on at least one buildbot -- when it wasn't a platform-specific fix -- then > I could easily push a "Commit" button and be done with it (although this > assumes single branch committing; doing this across branches makes all of > this difficult unless we finally resolve our Misc/NEWS conflict issues so > that in some instances it can be automated). Instead I have to wait until I > have a clone I can push from, download a patch, apply it, run the unit > tests myself, do the commit, and then repeat a subset of that to whatever > branches make sense. It's a lot of work for which some things could be > automated. > > > Well there?s two sides to the contribution process. > > There?s making things better/easier for people who *aren?t* committers and > there is making things better/easier for people who *are* committers. > Tacking > extra things on to what we already have to improve the life of committers > is > easier in many ways. As committers they?ve likely already taken the time to > learn the bespoke workflow that the Python project uses and have already > gotten > through that particular hurdle. Looking to standardize around popular > tools is > mostly about making it easier for *new* people and making it so that if > they > learn this set of tools they can go an immediately apply that to most of > the > other Python projects out there, or that if they are already contributing > to > those other Python projects they are probably aware of this particular > toolchain and workflow and can apply that knowledge directly to the Python > project. > > Moving to some of these tools happens to come with it features like really > nice > CI integration and a nice "Merge" button that also make it a lot nicer for > the > committer side of things. > All very true, but if we can't improve both sides then we are simply going to end up with even more patches that we take a while to get around to. I want to end up with a solution that advances the situation for *both* committers and non-committers and I feel like that is being lost in the discussion somewhat. As the person who pushed for a migration to DVCS for non-committers I totally support improving the workflow for non-committers, but not at the cost of ignoring the latter half of the contribution workflow of committers which is a chronic problem. As the PEP points out, the devguide, devinabox, and the PEPs have such a shallow development process that hosting them on Bitbucket wouldn't be a big thing. But if we don't view this as a long-term step towards moving cpython development somehow we are bifurcating our code contributors between git and hg which will be annoying. Now it could be argued that it doesn't matter for the peps and devguide since they are purely text and can be done easily through a web UI and a simple CI in Travis can be set up to make sure that the docs compile cleanly. But moving devinabox where there is going to be a code checkout in order to execute code for testing, etc. will be an issue. So I guess my view is +0 for doc-only repos on GitHub as long as we make it clear we are doing it with the expectation that people will do everything through the web UI and never have to know git. But I can't advocate moving code over without moving ALL repos over to git -- hosting location doesn't matter to me -- to prevent having to know both DVCSs in order to do coding work related to Python; the cpython repo shouldn't become this vaunted repo that is special and so it's in hg long-term but everything else is on git. -Brett > > I think it's also hard to get a representation of the people for whom the > bespoke workflow and less popular tooling are a problem for in a discussion > on python-dev. My guess is most of those people would not have signed up > for > python-dev since, unless they were willing to take the time to learn that, > so there is an amount of selection bias at play here as well. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Sun Nov 30 20:27:48 2014 From: brett at python.org (Brett Cannon) Date: Sun, 30 Nov 2014 19:27:48 +0000 Subject: [Python-Dev] advice needed: best approach to enabling "metamodules"? References: Message-ID: On Sun Nov 30 2014 at 2:16:18 PM Guido van Rossum wrote: > On Sun, Nov 30, 2014 at 6:15 AM, Brett Cannon wrote: >> >> On Sat, Nov 29, 2014, 21:55 Guido van Rossum wrote: >> >> All the use cases seem to be about adding some kind of getattr hook to >> modules. They all seem to involve modifying the CPython C code anyway. So >> why not tackle that problem head-on and modify module_getattro() to look >> for a global named __getattr__ and if it exists, call that instead of >> raising AttributeError? >> >> Not sure if anyone thought of it. :) Seems like a reasonable solution >> to me. Be curious to know what the benchmark suite said the impact was. >> > Why would there be any impact? The __getattr__ hook would be similar to > the one on classes -- it's only invoked at the point where otherwise > AttributeError would be raised. > You're right. My brain was thinking __getattribute__ semantics for some reason. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ethan at stoneleaf.us Sun Nov 30 20:27:58 2014 From: ethan at stoneleaf.us (Ethan Furman) Date: Sun, 30 Nov 2014 11:27:58 -0800 Subject: [Python-Dev] advice needed: best approach to enabling "metamodules"? In-Reply-To: References: Message-ID: <547B6FBE.8040205@stoneleaf.us> On 11/30/2014 11:15 AM, Guido van Rossum wrote: > On Sun, Nov 30, 2014 at 6:15 AM, Brett Cannon wrote: >> On Sat, Nov 29, 2014, 21:55 Guido van Rossum wrote: >>> >>> All the use cases seem to be about adding some kind of getattr hook >>> to modules. They all seem to involve modifying the CPython C code >>> anyway. So why not tackle that problem head-on and modify module_getattro() >>> to look for a global named __getattr__ and if it exists, call that instead >>> of raising AttributeError? >> >> Not sure if anyone thought of it. :) Seems like a reasonable solution to me. >> Be curious to know what the benchmark suite said the impact was. > > Why would there be any impact? The __getattr__ hook would be similar to the > one on classes -- it's only invoked at the point where otherwise AttributeError > would be raised. I think the bigger question is how do we support it back on 2.7? -- ~Ethan~ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: OpenPGP digital signature URL: From ethan at stoneleaf.us Sun Nov 30 20:28:20 2014 From: ethan at stoneleaf.us (Ethan Furman) Date: Sun, 30 Nov 2014 11:28:20 -0800 Subject: [Python-Dev] PEP 481 - Migrate Some Supporting Repositories to Git and Github In-Reply-To: References: <30725AC9-3DCF-416A-BCCD-2D64D489898C@stufft.io> <85d285wa3c.fsf@benfinney.id.au> <20141130140137.46803df1@fsol> <20141130115557.427918a2@limelight.wooz.org> <31195B20-A4BE-441C-ADC5-31236E2C4E1B@stufft.io> Message-ID: <547B6FD4.5000408@stoneleaf.us> On 11/30/2014 10:05 AM, Guido van Rossum wrote: > Python has a long history (all the way back to my choice of a MIT-style license for the first release) of mixing "free" > and "non-free" uses and tools -- for example on Windows we consciously chose to align ourselves with the platform > tooling rather than with the (minority) free tools available, Python has been ported to many commercial platforms, and > I've always encouraged use of Python in closed-source situations. For this I am grateful, and agree with. > Finally. And this may actually be the most important point. Python people should be doing stuff that makes Python better > (both taken in the most inclusive way possible). For stuff that's not unique to Python but can be used by many other > open-source projects, such as compilers, DVCS tools, or mailing lists, we should not be wasting our precious time on > building and maintaining our own tools or administering the servers on which they run. And historically we've not done a > great job on maintenance and administration. My issues with GitHub range from selfish to philosophical: - (selfish) I don't want to learn git - (practical) from what I hear git can have issues with losing history -- in a project that has many volunteer and part-time developers, using a tool that can munge your data just doesn't seem very wise - (practical) supporting git and hg means learning two different workflows - (philosophical) in a commercial world we vote with our dollars (don't like how a company behaves? don't buy their product); in an OSS world we vote by whose services/software we use; I don't want to support, or appear to support, a company that is abusive and sexist towards its employees: it is not what the PSF supports, and it's definitely not what I support. Not everyone is suited to demonstrate in the streets, but it shouldn't be that hard to not use a company with acknowledged bad practices. -- ~Ethan~ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: OpenPGP digital signature URL: From njs at pobox.com Sun Nov 30 20:29:10 2014 From: njs at pobox.com (Nathaniel Smith) Date: Sun, 30 Nov 2014 19:29:10 +0000 Subject: [Python-Dev] advice needed: best approach to enabling "metamodules"? In-Reply-To: References: Message-ID: On Sun, Nov 30, 2014 at 2:54 AM, Guido van Rossum wrote: > All the use cases seem to be about adding some kind of getattr hook to > modules. They all seem to involve modifying the CPython C code anyway. So > why not tackle that problem head-on and modify module_getattro() to look for > a global named __getattr__ and if it exists, call that instead of raising > AttributeError? You need to allow overriding __dir__ as well for tab-completion, and some people wanted to use the properties API instead of raw __getattr__, etc. Maybe someone will want __getattribute__ semantics, I dunno. So since we're *so close* to being able to just use the subclassing machinery, it seemed cleaner to try and get that working instead of reimplementing bits of it piecewise. That said, __getattr__ + __dir__ would be enough for my immediate use cases. -n > On Sat, Nov 29, 2014 at 11:37 AM, Nathaniel Smith wrote: >> >> On Sat, Nov 29, 2014 at 4:21 AM, Guido van Rossum >> wrote: >> > Are these really all our options? All of them sound like hacks, none of >> > them >> > sound like anything the language (or even the CPython implementation) >> > should >> > sanction. Have I missed the discussion where the use cases and >> > constraints >> > were analyzed and all other approaches were rejected? (I might have some >> > half-baked ideas, but I feel I should read up on the past discussion >> > first, >> > and they are probably more fit for python-ideas than for python-dev. >> > Plus >> > I'm just writing this email because I'm procrastinating on the type >> > hinting >> > PEP. :-) >> >> The previous discussions I was referring to are here: >> http://thread.gmane.org/gmane.comp.python.ideas/29487/focus=29555 >> http://thread.gmane.org/gmane.comp.python.ideas/29788 >> >> There might well be other options; these are just the best ones I >> could think of :-). The constraints are pretty tight, though: >> - The "new module" object (whatever it is) should have a __dict__ that >> aliases the original module globals(). I can elaborate on this if my >> original email wasn't enough, but hopefully it's obvious that making >> two copies of the same namespace and then trying to keep them in sync >> at the very least smells bad :-). >> - The "new module" object has to be a subtype of ModuleType, b/c there >> are lots of places that do isinstance(x, ModuleType) checks (notably >> -- but not only -- reload()). Since a major goal here is to make it >> possible to do cleaner deprecations, it would be really unfortunate if >> switching an existing package to use the metamodule support itself >> broke things :-). >> - Lookups in the normal case should have no additional performance >> overhead, because module lookups are extremely extremely common. (So >> this rules out dict proxies and tricks like that -- we really need >> 'new_module.__dict__ is globals()' to be true.) >> >> AFAICT there are three logically possible strategies for satisfying >> that first constraint: >> (a) convert the original module object into the type we want, in-place >> (b) create a new module object that acts like the original module object >> (c) somehow arrange for our special type to be used from the start >> >> My options 1 and 2 are means of accomplishing (a), and my options 3 >> and 4 are means of accomplishing (b) while working around the >> behavioural quirks of module objects (as required by the second >> constraint). >> >> The python-ideas thread did also consider several methods of >> implementing strategy (c), but they're messy enough that I left them >> out here. The problem is that somehow we have to execute code to >> create the new subtype *before* we have an entry in sys.modules for >> the package that contains the code for the subtype. So one option >> would be to add a new rule, that if a file pkgname/__new__.py exists, >> then this is executed first and is required to set up >> sys.modules["pkgname"] before we exec pkgname/__init__.py. So >> pkgname/__new__.py might look like: >> >> import sys >> from pkgname._metamodule import MyModuleSubtype >> sys.modules[__name__] = MyModuleSubtype(__name__, docstring) >> >> This runs into a lot of problems though. To start with, the 'from >> pkgname._metamodule ...' line is an infinite loop, b/c this is the >> code used to create sys.modules["pkgname"]. It's not clear where the >> globals dict for executing __new__.py comes from (who defines >> __name__? Currently that's done by ModuleType.__init__). It only works >> for packages, not modules. The need to provide the docstring here, >> before __init__.py is even read, is weird. It adds extra stat() calls >> to every package lookup. And, the biggest showstopper IMHO: AFAICT >> it's impossible to write a polyfill to support this code on old python >> versions, so it's useless to any package which needs to keep >> compatibility with 2.7 (or even 3.4). Sure, you can backport the whole >> import system like importlib2, but telling everyone that they need to >> replace every 'import numpy' with 'import importlib2; import numpy' is >> a total non-starter. >> >> So, yeah, those 4 options are really the only plausible ones I know of. >> >> Option 1 and option 3 are pretty nice at the language level! Most >> Python objects allow assignment to __class__ and __dict__, and both >> PyPy and Jython at least do support __class__ assignment. Really the >> only downside with Option 1 is that actually implementing it requires >> attention from someone with deep knowledge of typeobject.c. >> >> -n >> >> -- >> Nathaniel J. Smith >> Postdoctoral researcher - Informatics - University of Edinburgh >> http://vorpus.org >> _______________________________________________ >> Python-Dev mailing list >> Python-Dev at python.org >> https://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe: >> https://mail.python.org/mailman/options/python-dev/guido%40python.org > > > > > -- > --Guido van Rossum (python.org/~guido) -- Nathaniel J. Smith Postdoctoral researcher - Informatics - University of Edinburgh http://vorpus.org From donald at stufft.io Sun Nov 30 20:33:32 2014 From: donald at stufft.io (Donald Stufft) Date: Sun, 30 Nov 2014 14:33:32 -0500 Subject: [Python-Dev] PEP 481 - Migrate Some Supporting Repositories to Git and Github In-Reply-To: References: <30725AC9-3DCF-416A-BCCD-2D64D489898C@stufft.io> <85d285wa3c.fsf@benfinney.id.au> <20141130140137.46803df1@fsol> <27ABC536-C712-4109-9ED7-3B15C7818143@stufft.io> Message-ID: <3AC4141B-354B-47BA-8CFD-3CAAAC852ECC@stufft.io> > On Nov 30, 2014, at 2:19 PM, Brett Cannon wrote: > > All very true, but if we can't improve both sides then we are simply going to end up with even more patches that we take a while to get around to. I want to end up with a solution that advances the situation for *both* committers and non-committers and I feel like that is being lost in the discussion somewhat. As the person who pushed for a migration to DVCS for non-committers I totally support improving the workflow for non-committers, but not at the cost of ignoring the latter half of the contribution workflow of committers which is a chronic problem. > > As the PEP points out, the devguide, devinabox, and the PEPs have such a shallow development process that hosting them on Bitbucket wouldn't be a big thing. But if we don't view this as a long-term step towards moving cpython development somehow we are bifurcating our code contributors between git and hg which will be annoying. Now it could be argued that it doesn't matter for the peps and devguide since they are purely text and can be done easily through a web UI and a simple CI in Travis can be set up to make sure that the docs compile cleanly. But moving devinabox where there is going to be a code checkout in order to execute code for testing, etc. will be an issue. > > So I guess my view is +0 for doc-only repos on GitHub as long as we make it clear we are doing it with the expectation that people will do everything through the web UI and never have to know git. But I can't advocate moving code over without moving ALL repos over to git -- hosting location doesn't matter to me -- to prevent having to know both DVCSs in order to do coding work related to Python; the cpython repo shouldn't become this vaunted repo that is special and so it's in hg long-term but everything else is on git. So a goal of mine here is to sort of use these as a bit of a test bed. Moving CPython itself is a big and drastic change with a lot of implications, but moving the ?support? repositories is not nearly as much, especially with a read only mirror on hg.python.org which would allow things like the PEP rendering on www.python.org to stay the same if we wanted to. My hope was that we?d try this out, see how it works out, and if it seems to be a good thing, then at a later time we can either look at moving CPython itself or decide if it makes sense to do something different. Maybe this should be spelled out in the PEP? I?ve seen a few people say they were -1 because they didn?t want to split between hg on the CPython side and git on the supporting repos side. I?m not sure you can really get away from that because we?re *already* in that situation, things like the docs building script is a Git repo on Github, the python infrastructure itself is a git repo on Github, the new python.org website is a git repo on Github, the future PyPI is a git repo on GitHub. IOW I?m not sure what the best way forward is. I think moving to these tools for *all* repos is likely to be in the best interests of making things better for both sides of that coin however I didn?t want to go wholesale and try and make everything switch at all at once. If you think it makes sense to drop devinabox and make the dividing line between Code and not code (although I?d argue that line is already crossed with other code things already being on github) that?s fine with me. Or I can expand the scope if people think that makes more sense in the PEP too. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Sun Nov 30 20:41:03 2014 From: brett at python.org (Brett Cannon) Date: Sun, 30 Nov 2014 19:41:03 +0000 Subject: [Python-Dev] advice needed: best approach to enabling "metamodules"? References: <547B6FBE.8040205@stoneleaf.us> Message-ID: On Sun Nov 30 2014 at 2:28:31 PM Ethan Furman wrote: > On 11/30/2014 11:15 AM, Guido van Rossum wrote: > > On Sun, Nov 30, 2014 at 6:15 AM, Brett Cannon wrote: > >> On Sat, Nov 29, 2014, 21:55 Guido van Rossum wrote: > >>> > >>> All the use cases seem to be about adding some kind of getattr hook > >>> to modules. They all seem to involve modifying the CPython C code > >>> anyway. So why not tackle that problem head-on and modify > module_getattro() > >>> to look for a global named __getattr__ and if it exists, call that > instead > >>> of raising AttributeError? > >> > >> Not sure if anyone thought of it. :) Seems like a reasonable solution > to me. > >> Be curious to know what the benchmark suite said the impact was. > > > > Why would there be any impact? The __getattr__ hook would be similar to > the > > one on classes -- it's only invoked at the point where otherwise > AttributeError > > would be raised. > > I think the bigger question is how do we support it back on 2.7? > You don't; you just can't shoehorn everything back to 2.7. And just to make sure everyone participating in this discussion is up on the latest import stuff, Python 3.4 does have Loader.create_module() which lets you control what object is used for a module in the import machinery (this is prior to loading, though, so you can't specify it in a module but at the loader level only). This is how I was able to implement lazy loading for 3.5 . -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Sun Nov 30 20:49:36 2014 From: njs at pobox.com (Nathaniel Smith) Date: Sun, 30 Nov 2014 19:49:36 +0000 Subject: [Python-Dev] advice needed: best approach to enabling "metamodules"? In-Reply-To: <547B6FBE.8040205@stoneleaf.us> References: <547B6FBE.8040205@stoneleaf.us> Message-ID: On Sun, Nov 30, 2014 at 7:27 PM, Ethan Furman wrote: > On 11/30/2014 11:15 AM, Guido van Rossum wrote: >> On Sun, Nov 30, 2014 at 6:15 AM, Brett Cannon wrote: >>> On Sat, Nov 29, 2014, 21:55 Guido van Rossum wrote: >>>> >>>> All the use cases seem to be about adding some kind of getattr hook >>>> to modules. They all seem to involve modifying the CPython C code >>>> anyway. So why not tackle that problem head-on and modify module_getattro() >>>> to look for a global named __getattr__ and if it exists, call that instead >>>> of raising AttributeError? >>> >>> Not sure if anyone thought of it. :) Seems like a reasonable solution to me. >>> Be curious to know what the benchmark suite said the impact was. >> >> Why would there be any impact? The __getattr__ hook would be similar to the >> one on classes -- it's only invoked at the point where otherwise AttributeError >> would be raised. > > I think the bigger question is how do we support it back on 2.7? I think that's doable -- assuming I'm remembering correctly the slightly weird class vs. instance lookup rules for special methods, you can write a module subclass like class GetAttrModule(types.ModuleType): def __getattr__(self, name): return self.__dict__["__getattr__"](name) and then use ctypes hacks to get it into sys.modules[__name__]. -n -- Nathaniel J. Smith Postdoctoral researcher - Informatics - University of Edinburgh http://vorpus.org From donald at stufft.io Sun Nov 30 20:56:22 2014 From: donald at stufft.io (Donald Stufft) Date: Sun, 30 Nov 2014 14:56:22 -0500 Subject: [Python-Dev] PEP 481 - Migrate Some Supporting Repositories to Git and Github In-Reply-To: <547B6FD4.5000408@stoneleaf.us> References: <30725AC9-3DCF-416A-BCCD-2D64D489898C@stufft.io> <85d285wa3c.fsf@benfinney.id.au> <20141130140137.46803df1@fsol> <20141130115557.427918a2@limelight.wooz.org> <31195B20-A4BE-441C-ADC5-31236E2C4E1B@stufft.io> <547B6FD4.5000408@stoneleaf.us> Message-ID: <609E6EF0-2A6C-4F1E-B25F-DC43588B04C4@stufft.io> > On Nov 30, 2014, at 2:28 PM, Ethan Furman wrote: > > On 11/30/2014 10:05 AM, Guido van Rossum wrote: > >> Python has a long history (all the way back to my choice of a MIT-style license for the first release) of mixing "free" >> and "non-free" uses and tools -- for example on Windows we consciously chose to align ourselves with the platform >> tooling rather than with the (minority) free tools available, Python has been ported to many commercial platforms, and >> I've always encouraged use of Python in closed-source situations. > > For this I am grateful, and agree with. > >> Finally. And this may actually be the most important point. Python people should be doing stuff that makes Python better >> (both taken in the most inclusive way possible). For stuff that's not unique to Python but can be used by many other >> open-source projects, such as compilers, DVCS tools, or mailing lists, we should not be wasting our precious time on >> building and maintaining our own tools or administering the servers on which they run. And historically we've not done a >> great job on maintenance and administration. > > My issues with GitHub range from selfish to philosophical: > > - (selfish) I don't want to learn git Note: That you don?t actually have to learn git, you can clone a git repository with Mercurial using hg-git and continue to use Mercurial locally. The same of course can be said for the *other* way, but I?d argue that putting the burden of using things like hg-git or git-remote-hg on the less popular tool is a better overall decision. > > - (practical) from what I hear git can have issues with losing history -- in a > project that has many volunteer and part-time developers, using a tool that > can munge your data just doesn't seem very wise I have never heard of git losing history. Git goes out of it?s way not to lose things. Even unreferences commits don?t go away for months unless you purposely prune them. I?d personally call this FUD unless there?s some citation here. > > - (practical) supporting git and hg means learning two different workflows As I mentioned in my other email, we?re already supporting two different tools, and it?s a hope of mine to use this as a sort of testbed to moving the other repositories as well. > > - (philosophical) in a commercial world we vote with our dollars (don't like how > a company behaves? don't buy their product); in an OSS world we vote by whose > services/software we use; I don't want to support, or appear to support, a > company that is abusive and sexist towards its employees: it is not what the > PSF supports, and it's definitely not what I support. I?m assuming this is about Github. I?ll say that Github has, at least publicly, made steps towards doing better than they had previously there. I?m not a Github employee so I can?t speak towards that. It almost feels like there is some amount of co-opting this incident as a shield to hide behind. Most people who make this statement are more than happy to continue to use Linux for example, even though Linus is well documented being extremely abusive towards people and has more or less said that he?s never going to change that. I also think it?s hard to look at a company like bitbucket, for example, and say they are *better* than Github just because they didn?t have a public and inflammatory event. Attempting to reduce the cognitive burden for contributing and aligning ourselves with the most popular tools allows us to take advantage of the network effects of these tools popularity. This can be the difference between someone with limited amount of time being able to contribute or not, which can make real inroads towards making it easier for under privileged people to contribute much more than refusing to use a product of one group of people over another just because the other group hasn?t had a public and inflammatory event. > > Not everyone is suited to demonstrate in the streets, but it shouldn't be that hard to not use a company with > acknowledged bad practices. > > -- > ~Ethan~ > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/donald%40stufft.io --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA From rosuav at gmail.com Sun Nov 30 21:05:47 2014 From: rosuav at gmail.com (Chris Angelico) Date: Mon, 1 Dec 2014 07:05:47 +1100 Subject: [Python-Dev] PEP 481 - Migrate Some Supporting Repositories to Git and Github In-Reply-To: <547B6FD4.5000408@stoneleaf.us> References: <30725AC9-3DCF-416A-BCCD-2D64D489898C@stufft.io> <85d285wa3c.fsf@benfinney.id.au> <20141130140137.46803df1@fsol> <20141130115557.427918a2@limelight.wooz.org> <31195B20-A4BE-441C-ADC5-31236E2C4E1B@stufft.io> <547B6FD4.5000408@stoneleaf.us> Message-ID: On Mon, Dec 1, 2014 at 6:28 AM, Ethan Furman wrote: > My issues with GitHub range from selfish to philosophical: > > - (selfish) I don't want to learn git This ties in directly with the popularity argument. How many people are there who know hg and don't know git? How many who know git and don't know hg? So while this is a git issue for you, it's going to be a Mercurial issue for a lot more people. (And even people like me who kinda know both are going to spend a lot more time with git than with hg, on average. My knowledge of git is fairly advanced - I use git hooks, have scripts that manage things for me, can repair a repository that's seen some mess in it, etc - but my Mercurial knowledge is basic - I can clone and pull, not even sure I can push as I have literally never done so, and I basically turn to a dev guide to figure out how to make a patch in the appropriate way.) > - (practical) from what I hear git can have issues with losing history -- in a > project that has many volunteer and part-time developers, using a tool that > can munge your data just doesn't seem very wise It is possible to rewrite history in git, but - assuming you're not going to go to the ridiculous extent of SHA-1 cracking - it'll always be done by creating a new stream of commits, and git is very careful about confirming that you want to do this sort of thing. When you have a central server, you can simply configure it to reject any non-fast-forward pushes. (I'm not sure how to set receive.denyNonFastForwards on GitHub, but it's likely to be possible. In any case, you can always have an authoritative clone on python.org somewhere which mandates this kind of thing.) The git philosophy is: Your repository is yours. What you do with it is up to you. If that means rewriting history, git will confirm with you that you really want to do that, and then go right ahead and do what you ask. But then if your repo is a clone of someone else's, well, that someone else controls the other repo, so you might well not be allowed to merge changes in. > - (practical) supporting git and hg means learning two different workflows Already a problem for a lot of people. Unless CPython is the only project you ever contribute to, chances are you're going to meet git somewhere. If CPython used some mindblowingly brilliant but utterly unheard-of DVCS, sure it might be possible for core devs to avoid using any of the popular systems, but nobody else could. Personally, and somewhat selfishly, I would love to see the PEPs repo move to GitHub. For the two PEPs that I wrote, I had to juggle my own personal PEP draft repo and the central Mercurial peps repo; making sure changes got deployed from one to the other was basically a manual job, without any tool support. If I could send PRs against a clone of the peps repo, I would work that way instead of creating a dedicated repo at all (at least until such time as I need other files; the PEP 463 repo grew a few other bits and bobs, but they could just as easily have been in an ancillary repo without the PEP text in it). There'll be easily as many people who selfishly want git as selfishly want hg :) ChrisA From rosuav at gmail.com Sun Nov 30 21:19:11 2014 From: rosuav at gmail.com (Chris Angelico) Date: Mon, 1 Dec 2014 07:19:11 +1100 Subject: [Python-Dev] Unicode decode exception In-Reply-To: References: Message-ID: On Sun, Nov 30, 2014 at 7:07 PM, balaji marisetti wrote: > Hi, Hi. This list is for the development *of* Python, not development *with* Python, so I'm sending this reply also to python-list at python.org where it can be better handled. You'll probably want to subscribe here: https://mail.python.org/mailman/listinfo/python-list or alternatively, point a news reader at comp.lang.python. Let's continue this conversation on python-list rather than python-dev. > When I try to iterate through the lines of a > file("openssl-1.0.1j/crypto/bn/asm/x86_64-gcc.c"), I get a > UnicodeDecodeError (in python 3.4.0 on Ubuntu 14.04). But there is no > such error with python 2.7.6. What could be the problem? The difference between the two Python versions is that 2.7 lets you be a bit sloppy about Unicode vs bytes, but 3.4 requires that you keep them properly separate. > In [39]: with open("openssl-1.0.1j/crypto/bn/asm/x86_64-gcc.c") as f: > for line in f: > print (line) > > --------------------------------------------------------------------------- > UnicodeDecodeError Traceback (most recent call last) > in () > 1 with open("../openssl-1.0.1j/crypto/bn/asm/x86_64-gcc.c") as f: > ----> 2 for line in f: > 3 print (line) > 4 > > /usr/lib/python3.4/codecs.py in decode(self, input, final) > 311 # decode input (taking the buffer into account) > 312 data = self.buffer + input > --> 313 (result, consumed) = self._buffer_decode(data, > self.errors, final) > 314 # keep undecoded input until the next call > 315 self.buffer = data[consumed:] > > > -- > :-)balaji Most likely, the line of input that you just reached has a non-ASCII character, and the default encoding is ASCII. (Though without the actual exception message, I can't be sure of that.) The best fix would be to know what the file's encoding is, and simply add that as a parameter to your open() call - perhaps this: with open("filename", encoding="utf-8") as f: If you use the right encoding, and the file is correctly encoded, you should have no errors. If you still have errors... welcome to data problems, life can be hard. :| ChrisA From brett at python.org Sun Nov 30 21:26:19 2014 From: brett at python.org (Brett Cannon) Date: Sun, 30 Nov 2014 20:26:19 +0000 Subject: [Python-Dev] PEP 481 - Migrate Some Supporting Repositories to Git and Github References: <30725AC9-3DCF-416A-BCCD-2D64D489898C@stufft.io> <85d285wa3c.fsf@benfinney.id.au> <20141130140137.46803df1@fsol> <27ABC536-C712-4109-9ED7-3B15C7818143@stufft.io> <3AC4141B-354B-47BA-8CFD-3CAAAC852ECC@stufft.io> Message-ID: On Sun Nov 30 2014 at 2:33:35 PM Donald Stufft wrote: > > On Nov 30, 2014, at 2:19 PM, Brett Cannon wrote: > > All very true, but if we can't improve both sides then we are simply going > to end up with even more patches that we take a while to get around to. I > want to end up with a solution that advances the situation for *both* > committers and non-committers and I feel like that is being lost in the > discussion somewhat. As the person who pushed for a migration to DVCS for > non-committers I totally support improving the workflow for non-committers, > but not at the cost of ignoring the latter half of the contribution > workflow of committers which is a chronic problem. > > As the PEP points out, the devguide, devinabox, and the PEPs have such a > shallow development process that hosting them on Bitbucket wouldn't be a > big thing. But if we don't view this as a long-term step towards moving > cpython development somehow we are bifurcating our code contributors > between git and hg which will be annoying. Now it could be argued that it > doesn't matter for the peps and devguide since they are purely text and can > be done easily through a web UI and a simple CI in Travis can be set up to > make sure that the docs compile cleanly. But moving devinabox where there > is going to be a code checkout in order to execute code for testing, etc. > will be an issue. > > So I guess my view is +0 for doc-only repos on GitHub as long as we make > it clear we are doing it with the expectation that people will do > everything through the web UI and never have to know git. But I can't > advocate moving code over without moving ALL repos over to git -- hosting > location doesn't matter to me -- to prevent having to know both DVCSs in > order to do coding work related to Python; the cpython repo shouldn't > become this vaunted repo that is special and so it's in hg long-term but > everything else is on git. > > > So a goal of mine here is to sort of use these as a bit of a test bed. > Moving CPython itself is a big and drastic change with a lot of > implications, but moving the ?support? repositories is not nearly as much, > especially with a read only mirror on hg.python.org which would allow > things like the PEP rendering on www.python.org to stay the same if we > wanted to. My hope was that we?d try this out, see how it works out, and if > it seems to be a good thing, then at a later time we can either look at > moving CPython itself or decide if it makes sense to do something > different. Maybe this should be spelled out in the PEP? > > I?ve seen a few people say they were -1 because they didn?t want to split > between hg on the CPython side and git on the supporting repos side. I?m > not sure you can really get away from that because we?re *already* in that > situation, things like the docs building script is a Git repo on Github, > the python infrastructure itself is a git repo on Github, the new > python.org website is a git repo on Github, the future PyPI is a git repo > on GitHub. > That doesn't bother as that is support infrastructure around CPython but in no way directly tied to CPython releases. But devinabox, for instance, is specifically for helping people contribute to CPython, so asking people to use devinabox in git but then work in hg for some repos and git in others that devinabox checks out is just asking for trouble (e.g. devinabox checks out the peps, devguide, and cpython repos). > > > IOW I?m not sure what the best way forward is. I think moving to these > tools for *all* repos is likely to be in the best interests of making > things better for both sides of that coin however I didn?t want to go > wholesale and try and make everything switch at all at once. If you think > it makes sense to drop devinabox and make the dividing line between Code > and not code (although I?d argue that line is already crossed with other > code things already being on github) that?s fine with me. Or I can expand > the scope if people think that makes more sense in the PEP too. > Depends what other people think, but for me it's "we are going to move to git long-term and we are starting an experiment with docs on GitHub to see if that impacts contributions and committer maintenance at least for docs, maybe for code eventually". -------------- next part -------------- An HTML attachment was scrubbed... URL: From ethan at stoneleaf.us Sun Nov 30 21:28:09 2014 From: ethan at stoneleaf.us (Ethan Furman) Date: Sun, 30 Nov 2014 12:28:09 -0800 Subject: [Python-Dev] PEP 481 - Migrate Some Supporting Repositories to Git and Github In-Reply-To: <609E6EF0-2A6C-4F1E-B25F-DC43588B04C4@stufft.io> References: <30725AC9-3DCF-416A-BCCD-2D64D489898C@stufft.io> <85d285wa3c.fsf@benfinney.id.au> <20141130140137.46803df1@fsol> <20141130115557.427918a2@limelight.wooz.org> <31195B20-A4BE-441C-ADC5-31236E2C4E1B@stufft.io> <547B6FD4.5000408@stoneleaf.us> <609E6EF0-2A6C-4F1E-B25F-DC43588B04C4@stufft.io> Message-ID: <547B7DD9.6030000@stoneleaf.us> On 11/30/2014 11:56 AM, Donald Stufft wrote: >> On Nov 30, 2014, at 2:28 PM, Ethan Furman wrote: >> >> My issues with GitHub range from selfish to philosophical: >> >> - (selfish) I don't want to learn git > > Note: That you don?t actually have to learn git, you can clone a git repository > with Mercurial using hg-git and continue to use Mercurial locally. The same of > course can be said for the *other* way, but I?d argue that putting the burden of > using things like hg-git or git-remote-hg on the less popular tool is a better > overall decision. Fair enough. >> - (practical) from what I hear git can have issues with losing history -- in a >> project that has many volunteer and part-time developers, using a tool that >> can munge your data just doesn't seem very wise > > I have never heard of git losing history. Git goes out of it?s way not to lose > things. Even unreferences commits don?t go away for months unless you purposely > prune them. I?d personally call this FUD unless there?s some citation here. Okay. >> - (practical) supporting git and hg means learning two different workflows > > As I mentioned in my other email, we?re already supporting two different tools, > and it?s a hope of mine to use this as a sort of testbed to moving the other > repositories as well. That should be in the PEP then. >> - (philosophical) in a commercial world we vote with our dollars (don't like how >> a company behaves? don't buy their product); in an OSS world we vote by whose >> services/software we use; I don't want to support, or appear to support, a >> company that is abusive and sexist towards its employees: it is not what the >> PSF supports, and it's definitely not what I support. > > I?m assuming this is about Github. Yes. > I?ll say that Github has, at least publicly, made steps towards doing better than > they had previously there. I?m not a Github employee so I can?t speak towards that. > > It almost feels like there is some amount of co-opting this incident as a shield > to hide behind. Most people who make this statement are more than happy to continue > to use Linux for example, even though Linus is well documented being extremely > abusive towards people and has more or less said that he?s never going to change > that. Linux is not my choice. ;) Linus is also one person, not an entire company. > I also think it?s hard to look at a company like bitbucket, for example, and > say they are *better* than Github just because they didn?t have a public and > inflammatory event. In cases like this it's not "better" but "willing to work with". BitBucket might be as hostile as GitHub, only someone who has worked for both could really know. > [...] than refusing to use a product of one group of people over another just because > the other group hasn?t had a public and inflammatory event. We can only make decisions on information we have; pretending we don't have it, or that some other company /could/ be the same, is hiding our heads in the sand. If git is the wave of the future, there are other git hosts besides GitHub. -- ~Ethan~ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: OpenPGP digital signature URL: From guido at python.org Sun Nov 30 21:54:58 2014 From: guido at python.org (Guido van Rossum) Date: Sun, 30 Nov 2014 12:54:58 -0800 Subject: [Python-Dev] advice needed: best approach to enabling "metamodules"? In-Reply-To: References: Message-ID: On Sun, Nov 30, 2014 at 11:29 AM, Nathaniel Smith wrote: > On Sun, Nov 30, 2014 at 2:54 AM, Guido van Rossum > wrote: > > All the use cases seem to be about adding some kind of getattr hook to > > modules. They all seem to involve modifying the CPython C code anyway. So > > why not tackle that problem head-on and modify module_getattro() to look > for > > a global named __getattr__ and if it exists, call that instead of raising > > AttributeError? > > You need to allow overriding __dir__ as well for tab-completion, and > some people wanted to use the properties API instead of raw > __getattr__, etc. Maybe someone will want __getattribute__ semantics, > I dunno. Hm... I agree about __dir__ but the other things feel too speculative. > So since we're *so close* to being able to just use the > subclassing machinery, it seemed cleaner to try and get that working > instead of reimplementing bits of it piecewise. > That would really be option 1, right? It's the one that looks cleanest from the user's POV (or at least from the POV of a developer who wants to build a framework using this feature -- for a simple one-off use case, __getattr__ sounds pretty attractive). I think that if we really want option 1, the issue of PyModuleType not being a heap type can be dealt with. > That said, __getattr__ + __dir__ would be enough for my immediate use > cases. Perhaps it would be a good exercise to try and write the "lazy submodule import"(*) use case three ways: (a) using only CPython 3.4; (b) using __class__ assignment; (c) using customizable __getattr__ and __dir__. I think we can learn a lot about the alternatives from this exercise. I presume there's already a version of (a) floating around, but if it's been used in practice at all, it's probably too gnarly to serve as a useful comparison (though its essence may be extracted to serve as such). FWIW I believe all proposals here have a big limitation: the module *itself* cannot benefit much from all these shenanigans, because references to globals from within the module's own code are just dictionary accesses, and we don't want to change that. (*) I originally wrote "lazy import", but I realized that messing with the module class object probably isn't the best way to implement that -- it requires a proxy for the module that's managed by an import hook. But if you think it's possible, feel free to use this example, as "lazy import" seems a pretty useful thing to have in many situations. (At least that's how I would do it. And I would probably add some atrocious hack to patch up the importing module's globals once the module is actually loaded, to reduce the cost of using the proxy over the lifetime of the process. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Sun Nov 30 22:00:35 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sun, 30 Nov 2014 22:00:35 +0100 Subject: [Python-Dev] PEP 481 - Migrate Some Supporting Repositories to Git and Github In-Reply-To: References: <30725AC9-3DCF-416A-BCCD-2D64D489898C@stufft.io> <85d285wa3c.fsf@benfinney.id.au> <20141130140137.46803df1@fsol> <27ABC536-C712-4109-9ED7-3B15C7818143@stufft.io> Message-ID: <20141130220035.655f07a2@fsol> On Sun, 30 Nov 2014 19:19:50 +0000 Brett Cannon wrote: > > As the PEP points out, the devguide, devinabox, and the PEPs have such a > shallow development process that hosting them on Bitbucket wouldn't be a > big thing. But if we don't view this as a long-term step towards moving > cpython development somehow we are bifurcating our code contributors > between git and hg which will be annoying. Now it could be argued that it > doesn't matter for the peps and devguide since they are purely text and can > be done easily through a web UI and a simple CI in Travis can be set up to > make sure that the docs compile cleanly. Web-based text editors are a poor UI so, while it *can* be done over the Web, it's not really a good thing to promote (unless we're talking mini-edits such as fixing typos). Regards Antoine. From solipsis at pitrou.net Sun Nov 30 22:05:53 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sun, 30 Nov 2014 22:05:53 +0100 Subject: [Python-Dev] PEP 481 - Migrate Some Supporting Repositories to Git and Github References: <30725AC9-3DCF-416A-BCCD-2D64D489898C@stufft.io> <85d285wa3c.fsf@benfinney.id.au> <20141130140137.46803df1@fsol> <20141130115557.427918a2@limelight.wooz.org> <31195B20-A4BE-441C-ADC5-31236E2C4E1B@stufft.io> Message-ID: <20141130220553.2aa256f6@fsol> On Sun, 30 Nov 2014 10:05:01 -0800 Guido van Rossum wrote: > > I bring this up to emphasize that (unlike GNU software and the FSF) Python > has no additional hidden agenda of bringing freedom to all software. As far as GNU and the FSF are concerned, I don't think the agenda is "hidden" at all ;-) > Now some entirely practical points. > > - I am basically the only remaining active PEP editor, so I see most PEP > contributions by non-core-committers. Almost all of these uses github. Not > bitbucket, not some other git host, but github. I spend a fair amount of > time applying patches. It would most definitely be easier if I could get > them to send me pull requests. Are you sure that those contributors wouldn't want to use Bitbucket - or another hg-based hosting service? A PEP contributor is someone who is likely to contribute CPython code as well - so they would have to know hg. Regards Antoine. From brett at python.org Sun Nov 30 22:10:42 2014 From: brett at python.org (Brett Cannon) Date: Sun, 30 Nov 2014 21:10:42 +0000 Subject: [Python-Dev] advice needed: best approach to enabling "metamodules"? References: Message-ID: On Sun Nov 30 2014 at 3:55:39 PM Guido van Rossum wrote: > On Sun, Nov 30, 2014 at 11:29 AM, Nathaniel Smith wrote: > >> On Sun, Nov 30, 2014 at 2:54 AM, Guido van Rossum >> wrote: >> > All the use cases seem to be about adding some kind of getattr hook to >> > modules. They all seem to involve modifying the CPython C code anyway. >> So >> > why not tackle that problem head-on and modify module_getattro() to >> look for >> > a global named __getattr__ and if it exists, call that instead of >> raising >> > AttributeError? >> >> You need to allow overriding __dir__ as well for tab-completion, and >> some people wanted to use the properties API instead of raw >> __getattr__, etc. Maybe someone will want __getattribute__ semantics, >> I dunno. > > > Hm... I agree about __dir__ but the other things feel too speculative. > > >> So since we're *so close* to being able to just use the >> subclassing machinery, it seemed cleaner to try and get that working >> instead of reimplementing bits of it piecewise. >> > > That would really be option 1, right? It's the one that looks cleanest > from the user's POV (or at least from the POV of a developer who wants to > build a framework using this feature -- for a simple one-off use case, > __getattr__ sounds pretty attractive). I think that if we really want > option 1, the issue of PyModuleType not being a heap type can be dealt with. > > >> That said, __getattr__ + __dir__ would be enough for my immediate use >> cases. > > > Perhaps it would be a good exercise to try and write the "lazy submodule > import"(*) use case three ways: (a) using only CPython 3.4; (b) using > __class__ assignment; (c) using customizable __getattr__ and __dir__. I > think we can learn a lot about the alternatives from this exercise. I > presume there's already a version of (a) floating around, but if it's been > used in practice at all, it's probably too gnarly to serve as a useful > comparison (though its essence may be extracted to serve as such). > > FWIW I believe all proposals here have a big limitation: the module > *itself* cannot benefit much from all these shenanigans, because references > to globals from within the module's own code are just dictionary accesses, > and we don't want to change that. > > (*) I originally wrote "lazy import", but I realized that messing with the > module class object probably isn't the best way to implement that -- it > requires a proxy for the module that's managed by an import hook. But if > you think it's possible, feel free to use this example, as "lazy import" > seems a pretty useful thing to have in many situations. (At least that's > how I would do it. And I would probably add some atrocious hack to patch up > the importing module's globals once the module is actually loaded, to > reduce the cost of using the proxy over the lifetime of the process. > Start at https://hg.python.org/cpython/file/64bb01bce12c/Lib/importlib/util.py#l207 and read down the rest of the file. It really only requires changing __class__ to drop the proxy and that's done immediately after the lazy import. The approach also occurs *after* the finder so you don't get ImportError for at least missing a file. -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Sun Nov 30 22:12:51 2014 From: donald at stufft.io (Donald Stufft) Date: Sun, 30 Nov 2014 16:12:51 -0500 Subject: [Python-Dev] PEP 481 - Migrate Some Supporting Repositories to Git and Github In-Reply-To: <20141130220553.2aa256f6@fsol> References: <30725AC9-3DCF-416A-BCCD-2D64D489898C@stufft.io> <85d285wa3c.fsf@benfinney.id.au> <20141130140137.46803df1@fsol> <20141130115557.427918a2@limelight.wooz.org> <31195B20-A4BE-441C-ADC5-31236E2C4E1B@stufft.io> <20141130220553.2aa256f6@fsol> Message-ID: <63D2C7FE-7628-41E7-BE05-538073AB8602@stufft.io> > On Nov 30, 2014, at 4:05 PM, Antoine Pitrou wrote: > > On Sun, 30 Nov 2014 10:05:01 -0800 > Guido van Rossum wrote: >> >> I bring this up to emphasize that (unlike GNU software and the FSF) Python >> has no additional hidden agenda of bringing freedom to all software. > > As far as GNU and the FSF are concerned, I don't think the agenda is > "hidden" at all ;-) > >> Now some entirely practical points. >> >> - I am basically the only remaining active PEP editor, so I see most PEP >> contributions by non-core-committers. Almost all of these uses github. Not >> bitbucket, not some other git host, but github. I spend a fair amount of >> time applying patches. It would most definitely be easier if I could get >> them to send me pull requests. > > Are you sure that those contributors wouldn't want to use Bitbucket - > or another hg-based hosting service? > > A PEP contributor is someone who is likely to contribute CPython code as > well - so they would have to know hg. Speaking as someone who has written a number of PEPS, plans to write a number more, and also has commit bit on hg.python.org and is maintaining code in CPython. I would greatly prefer using Github for PEPs than I would any hg hosting service I?ve seen to date. I can?t obviously speak for every single committer, however Guido also stated that through his work in committing patches of incoming PEPs he?s seen that a lot of the PEPs are being written on Github using git. I think it?s fair to say that those people would prefer PRs on Github over using Bitbucket as well since they were choosing Github over Bitbucket when there was no incentive to do so. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA From solipsis at pitrou.net Sun Nov 30 22:12:43 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sun, 30 Nov 2014 22:12:43 +0100 Subject: [Python-Dev] advice needed: best approach to enabling "metamodules"? References: Message-ID: <20141130221243.79e4feae@fsol> On Sun, 30 Nov 2014 11:15:50 -0800 Guido van Rossum wrote: > On Sun, Nov 30, 2014 at 6:15 AM, Brett Cannon wrote: > > > > On Sat, Nov 29, 2014, 21:55 Guido van Rossum wrote: > > > > All the use cases seem to be about adding some kind of getattr hook to > > modules. They all seem to involve modifying the CPython C code anyway. So > > why not tackle that problem head-on and modify module_getattro() to look > > for a global named __getattr__ and if it exists, call that instead of > > raising AttributeError? > > > > Not sure if anyone thought of it. :) Seems like a reasonable solution to > > me. Be curious to know what the benchmark suite said the impact was. > > > Why would there be any impact? The __getattr__ hook would be similar to the > one on classes -- it's only invoked at the point where otherwise > AttributeError would be raised. builtins are typically found by first looking up in the current globals (module) scope, failing, and then falling back on __builtins__. Depending on how much overhead is added to the "failing" step, there /might/ be a performance difference. Of course, that would only occur wherever a __getattr__ hook is defined. Regards Antoine. From guido at python.org Sun Nov 30 22:34:04 2014 From: guido at python.org (Guido van Rossum) Date: Sun, 30 Nov 2014 13:34:04 -0800 Subject: [Python-Dev] advice needed: best approach to enabling "metamodules"? In-Reply-To: <20141130221243.79e4feae@fsol> References: <20141130221243.79e4feae@fsol> Message-ID: On Sun, Nov 30, 2014 at 1:12 PM, Antoine Pitrou wrote: > On Sun, 30 Nov 2014 11:15:50 -0800 > Guido van Rossum wrote: > > On Sun, Nov 30, 2014 at 6:15 AM, Brett Cannon wrote: > > > > > > On Sat, Nov 29, 2014, 21:55 Guido van Rossum > wrote: > > > > > > All the use cases seem to be about adding some kind of getattr hook to > > > modules. They all seem to involve modifying the CPython C code anyway. > So > > > why not tackle that problem head-on and modify module_getattro() to > look > > > for a global named __getattr__ and if it exists, call that instead of > > > raising AttributeError? > > > > > > Not sure if anyone thought of it. :) Seems like a reasonable solution > to > > > me. Be curious to know what the benchmark suite said the impact was. > > > > > Why would there be any impact? The __getattr__ hook would be similar to > the > > one on classes -- it's only invoked at the point where otherwise > > AttributeError would be raised. > > builtins are typically found by first looking up in the current globals > (module) scope, failing, and then falling back on __builtins__. > > Depending on how much overhead is added to the "failing" step, there > /might/ be a performance difference. Of course, that would only occur > wherever a __getattr__ hook is defined. > The builtins lookup process never does a module attribute lookup -- it only does dict lookups. So it would not be affected by a module __getattr__ hook (unless we were to use dict proxies, which Nathaniel already rejected). @Nathaniel: perhaps you could get what you want without any C code changes using the approach of Brett's LazyLoader? -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Sun Nov 30 22:45:54 2014 From: donald at stufft.io (Donald Stufft) Date: Sun, 30 Nov 2014 16:45:54 -0500 Subject: [Python-Dev] PEP 481 - Migrate Some Supporting Repositories to Git and Github In-Reply-To: References: <30725AC9-3DCF-416A-BCCD-2D64D489898C@stufft.io> <85d285wa3c.fsf@benfinney.id.au> <20141130140137.46803df1@fsol> <27ABC536-C712-4109-9ED7-3B15C7818143@stufft.io> <3AC4141B-354B-47BA-8CFD-3CAAAC852ECC@stufft.io> Message-ID: > On Nov 30, 2014, at 3:26 PM, Brett Cannon wrote: > > > > On Sun Nov 30 2014 at 2:33:35 PM Donald Stufft > wrote: > >> On Nov 30, 2014, at 2:19 PM, Brett Cannon > wrote: >> >> All very true, but if we can't improve both sides then we are simply going to end up with even more patches that we take a while to get around to. I want to end up with a solution that advances the situation for *both* committers and non-committers and I feel like that is being lost in the discussion somewhat. As the person who pushed for a migration to DVCS for non-committers I totally support improving the workflow for non-committers, but not at the cost of ignoring the latter half of the contribution workflow of committers which is a chronic problem. >> >> As the PEP points out, the devguide, devinabox, and the PEPs have such a shallow development process that hosting them on Bitbucket wouldn't be a big thing. But if we don't view this as a long-term step towards moving cpython development somehow we are bifurcating our code contributors between git and hg which will be annoying. Now it could be argued that it doesn't matter for the peps and devguide since they are purely text and can be done easily through a web UI and a simple CI in Travis can be set up to make sure that the docs compile cleanly. But moving devinabox where there is going to be a code checkout in order to execute code for testing, etc. will be an issue. >> >> So I guess my view is +0 for doc-only repos on GitHub as long as we make it clear we are doing it with the expectation that people will do everything through the web UI and never have to know git. But I can't advocate moving code over without moving ALL repos over to git -- hosting location doesn't matter to me -- to prevent having to know both DVCSs in order to do coding work related to Python; the cpython repo shouldn't become this vaunted repo that is special and so it's in hg long-term but everything else is on git. > > > So a goal of mine here is to sort of use these as a bit of a test bed. Moving CPython itself is a big and drastic change with a lot of implications, but moving the ?support? repositories is not nearly as much, especially with a read only mirror on hg.python.org which would allow things like the PEP rendering on www.python.org to stay the same if we wanted to. My hope was that we?d try this out, see how it works out, and if it seems to be a good thing, then at a later time we can either look at moving CPython itself or decide if it makes sense to do something different. Maybe this should be spelled out in the PEP? > > I?ve seen a few people say they were -1 because they didn?t want to split between hg on the CPython side and git on the supporting repos side. I?m not sure you can really get away from that because we?re *already* in that situation, things like the docs building script is a Git repo on Github, the python infrastructure itself is a git repo on Github, the new python.org website is a git repo on Github, the future PyPI is a git repo on GitHub. > > That doesn't bother as that is support infrastructure around CPython but in no way directly tied to CPython releases. But devinabox, for instance, is specifically for helping people contribute to CPython, so asking people to use devinabox in git but then work in hg for some repos and git in others that devinabox checks out is just asking for trouble (e.g. devinabox checks out the peps, devguide, and cpython repos). I?m not sure what you?re proposing here. If devinabox checks out peps, devguide, and cpython aren?t they going to have to use git and hg both unless we move all of those repositories together? Beyond just the tools the status quo of those is that if you do make a change you have different ways to submit a contribution depending on which repository it is. > > > > IOW I?m not sure what the best way forward is. I think moving to these tools for *all* repos is likely to be in the best interests of making things better for both sides of that coin however I didn?t want to go wholesale and try and make everything switch at all at once. If you think it makes sense to drop devinabox and make the dividing line between Code and not code (although I?d argue that line is already crossed with other code things already being on github) that?s fine with me. Or I can expand the scope if people think that makes more sense in the PEP too. > > Depends what other people think, but for me it's "we are going to move to git long-term and we are starting an experiment with docs on GitHub to see if that impacts contributions and committer maintenance at least for docs, maybe for code eventually?. What do you mean by ?docs?, is that the devguide and PEPs repository? Here?s another idea for an experiment that might be more generally useful. As we've said there are two sides to the coin here, non-comitters and comitters, a lot of the benefit of moving to Github is focused at non-comitters although there are benefits for comitters themselves. Django hosts it's git repository on Github and it's issue tracker is Trac. Although it doesn't require an issue to be created for small issues, you can just send a PR. What if we focused an experiment on the benefits to non-comitters? It's possible to maintain a git mirror of a Mercurial repository, in fact we already have that at github.com/python/cpython. What if we permit people to make PRs against that repository, and then take those PRs and paste them into roundup? Sort of like the "Remote hg repo" field. Then we can create some integration that would post a comment to the ticket whenever that PR is updated (sort of like the notification that happens when a new patch is uploaded). The cannonical repository would still be hg.python.org and in order to actually commit the PR commiters would need to turn the PR into a patch (trivially easy, just add .diff or .patch to the PR URL). If we ever move CPython to git (and Github) this sort of integration is going to be required anyways unless we plan on moving the issue tracker as well which I don't think we would since the Github issues are not fully featured enough for us. This would give us some real information on how people would prefer to contribute I think. If people start sending a lot of patches using Github PRs then we can look at moving the possibility of moving everything whole hog over to the git / Github. If people end up not really using it, or it doesn't seem to be actually useful we can look at disabling or (or keeping it around for people who like to use it). I still think moving at least the PEPs and devguide repository over to git / Github is a generally good idea that should be part of all of this as well. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: From tjreedy at udel.edu Sun Nov 30 23:14:38 2014 From: tjreedy at udel.edu (Terry Reedy) Date: Sun, 30 Nov 2014 17:14:38 -0500 Subject: [Python-Dev] PEP 481 - Migrate Some Supporting Repositories to Git and Github In-Reply-To: <3AC4141B-354B-47BA-8CFD-3CAAAC852ECC@stufft.io> References: <30725AC9-3DCF-416A-BCCD-2D64D489898C@stufft.io> <85d285wa3c.fsf@benfinney.id.au> <20141130140137.46803df1@fsol> <27ABC536-C712-4109-9ED7-3B15C7818143@stufft.io> <3AC4141B-354B-47BA-8CFD-3CAAAC852ECC@stufft.io> Message-ID: On 11/30/2014 2:33 PM, Donald Stufft wrote: > So a goal of mine here is to sort of use these as a bit of a test bed. > Moving CPython itself is a big and drastic change with a lot of > implications, but moving the ?support? repositories is not nearly as > much, especially with a read only mirror on hg.python.org > which would allow things like the PEP rendering > on www.python.org to stay the same if we wanted > to. My hope was that we?d try this out, see how it works out, and if it > seems to be a good thing, then at a later time we can either look at > moving CPython itself or decide if it makes sense to do something > different. Maybe this should be spelled out in the PEP? > > I?ve seen a few people say they were -1 because they didn?t want to > split between hg on the CPython side and git on the supporting repos > side. Being only recently somewhat comfortable with hg, I do not really want to have to learn git at this time. > I?m not sure you can really get away from that because we?re > *already* in that situation, things like the docs building script is a > Git repo on Github, As long as I can do test doc builds with the scripts in /Docs, this does not matter to me. If the people who work with the site doc build script want it in git on Github, fine. > the python infrastructure itself is a git repo on Github, Since I am not sure what you mean, its location does not matter to me. > the new python.org website is a git repo on > Github, the future PyPI is a git repo on GitHub. Since I do not commit to either, ditto. As far as I am concerned, the people involved with specialized repositories may as well have their preference. This includes devinabox. I have made a few commits to the PEP repository, but only to the one I co-authored, 434. Using the web form might make it easier to suggest changes to other peoples' PEPs. > IOW I?m not sure what the best way forward is. I think moving to these > tools for *all* repos is likely to be in the best interests of making > things better for both sides of that coin however I didn?t want to go > wholesale and try and make everything switch at all at once. I agree with others that the current bottleneck is disposing of patches, especially those from non-core developers, not getting more patches to considier. I am quite sure that if we significantly reduce the current backlog of 4600 issues and more expeditiously handles the contributions we do get, will will get more. More than one person has said that they are disinclined to submit patches when they have multiple patches on the tracker already. So I think the focus should be on making better use of developer time and having more of it. -- Terry Jan Reedy From mark at hotpy.org Sun Nov 30 23:14:48 2014 From: mark at hotpy.org (Mark Shannon) Date: Sun, 30 Nov 2014 22:14:48 +0000 Subject: [Python-Dev] advice needed: best approach to enabling "metamodules"? In-Reply-To: References: Message-ID: <547B96D8.7050700@hotpy.org> Hi, This discussion has been going on for a while, but no one has questioned the basic premise. Does this needs any change to the language or interpreter? I believe it does not. I'm modified your original metamodule.py to not use ctypes and support reloading: https://gist.github.com/markshannon/1868e7e6115d70ce6e76 Cheers, Mark. On 29/11/14 01:59, Nathaniel Smith wrote: > Hi all, > > There was some discussion on python-ideas last month about how to make > it easier/more reliable for a module to override attribute access. > This is useful for things like autoloading submodules (accessing > 'foo.bar' triggers the import of 'bar'), or for deprecating module > attributes that aren't functions. (Accessing 'foo.bar' emits a > DeprecationWarning, "the bar attribute will be removed soon".) Python > has had some basic support for this for a long time -- if a module > overwrites its entry in sys.modules[__name__], then the object that's > placed there will be returned by 'import'. This allows one to define > custom subclasses of module and use them instead of the default, > similar to how metaclasses allow one to use custom subclasses of > 'type'. > > In practice though it's very difficult to make this work safely and > correctly for a top-level package. The main problem is that when you > create a new object to stick into sys.modules, this necessarily means > creating a new namespace dict. And now you have a mess, because now > you have two dicts: new_module.__dict__ which is the namespace you > export, and old_module.__dict__, which is the globals() for the code > that's trying to define the module namespace. Keeping these in sync is > extremely error-prone -- consider what happens, e.g., when your > package __init__.py wants to import submodules which then recursively > import the top-level package -- so it's difficult to justify for the > kind of large packages that might be worried about deprecating entries > in their top-level namespace. So what we'd really like is a way to > somehow end up with an object that (a) has the same __dict__ as the > original module, but (b) is of our own custom module subclass. If we > can do this then metamodules will become safe and easy to write > correctly. > > (There's a little demo of working metamodules here: > https://github.com/njsmith/metamodule/ > but it uses ctypes hacks that depend on non-stable parts of the > CPython ABI, so it's not a long-term solution.) > > I've now spent some time trying to hack this capability into CPython > and I've made a list of the possible options I can think of to fix > this. I'm writing to python-dev because none of them are obviously The > Right Way so I'd like to get some opinions/ruling/whatever on which > approach to follow up on. > > Option 1: Make it possible to change the type of a module object > in-place, so that we can write something like > > sys.modules[__name__].__class__ = MyModuleSubclass > > Option 1 downside: The invariants required to make __class__ > assignment safe are complicated, and only implemented for > heap-allocated type objects. PyModule_Type is not heap-allocated, so > making this work would require lots of delicate surgery to > typeobject.c. I'd rather not go down that rabbit-hole. > > ---- > > Option 2: Make PyModule_Type into a heap type allocated at interpreter > startup, so that the above just works. > > Option 2 downside: PyModule_Type is exposed as a statically-allocated > global symbol, so doing this would involve breaking the stable ABI. > > ---- > > Option 3: Make it legal to assign to the __dict__ attribute of a > module object, so that we can write something like > > new_module = MyModuleSubclass(...) > new_module.__dict__ = sys.modules[__name__].__dict__ > sys.modules[__name__].__dict__ = {} # *** > sys.modules[__name__] = new_module > > The line marked *** is necessary because the way modules are designed, > they expect to control the lifecycle of their __dict__. When the > module object is initialized, it fills in a bunch of stuff in the > __dict__. When the module object (not the dict object!) is > deallocated, it deletes everything from the __dict__. This latter > feature in particular means that having two module objects sharing the > same __dict__ is bad news. > > Option 3 downside: The paragraph above. Also, there's stuff inside the > module struct besides just the __dict__, and more stuff has appeared > there over time. > > ---- > > Option 4: Add a new function sys.swap_module_internals, which takes > two module objects and swaps their __dict__ and other attributes. By > making the operation a swap instead of an assignment, we avoid the > lifecycle pitfalls from Option 3. By making it a builtin, we can make > sure it always handles all the module fields that matter, not just > __dict__. Usage: > > new_module = MyModuleSubclass(...) > sys.swap_module_internals(new_module, sys.modules[__name__]) > sys.modules[__name__] = new_module > > Option 4 downside: Obviously a hack. > > ---- > > Option 3 or 4 both seem workable, it just depends on which way we > prefer to hold our nose. Option 4 is slightly more correct in that it > works for *all* modules, but OTOH at the moment the only time Option 3 > *really* fails is for compiled modules with PEP 3121 metadata, and > compiled modules can already use a module subclass via other means > (since they instantiate their own module objects). > > Thoughts? Suggestions on other options I've missed? Should I go ahead > and write a patch for one of these? > > -n > From brunocauet at gmail.com Sun Nov 30 17:18:46 2014 From: brunocauet at gmail.com (Bruno Cauet) Date: Sun, 30 Nov 2014 08:18:46 -0800 (PST) Subject: [Python-Dev] Unicode decode exception In-Reply-To: References: Message-ID: <1417364326393.ac7250c2@Nodemailer> Did you try opening it as a binary file? open(filename, 'rb'): ? Tagada tsouin tsouin On Sun, Nov 30, 2014 at 5:06 PM, balaji marisetti wrote: > Hi, > When I try to iterate through the lines of a > file("openssl-1.0.1j/crypto/bn/asm/x86_64-gcc.c"), I get a > UnicodeDecodeError (in python 3.4.0 on Ubuntu 14.04). But there is no > such error with python 2.7.6. What could be the problem? > In [39]: with open("openssl-1.0.1j/crypto/bn/asm/x86_64-gcc.c") as f: > for line in f: > print (line) > --------------------------------------------------------------------------- > UnicodeDecodeError Traceback (most recent call last) > in () > 1 with open("../openssl-1.0.1j/crypto/bn/asm/x86_64-gcc.c") as f: > ----> 2 for line in f: > 3 print (line) > 4 > /usr/lib/python3.4/codecs.py in decode(self, input, final) > 311 # decode input (taking the buffer into account) > 312 data = self.buffer + input > --> 313 (result, consumed) = self._buffer_decode(data, > self.errors, final) > 314 # keep undecoded input until the next call > 315 self.buffer = data[consumed:] > -- > :-)balaji > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/brunocauet%40gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: