From ericsnowcurrently at gmail.com Sun Mar 1 02:30:55 2015 From: ericsnowcurrently at gmail.com (Eric Snow) Date: Sat, 28 Feb 2015 18:30:55 -0700 Subject: [Python-ideas] An even simpler customization of class creation In-Reply-To: References: Message-ID: On Sat, Feb 28, 2015 at 10:10 AM, Nick Coghlan wrote: > That leaves my one concern being the "namespace" keyword parameter. > I'd started reconsidering that for PEP 422, and I think the argument > holds at least as strongly for your improved design: > https://www.python.org/dev/peps/pep-0422/#is-the-namespace-concept-worth-the-extra-complexity > > Since speed usually isn't a critical concern for class definitions, > it's likely better to just make them ordered by default, which can be > its own PEP, rather than being part of this one. FWIW, Guido already told me a while back that OrderedDict by default for class definition namespace was okay. It's just been waiting on me to get back to the C OrderedDict implementation (which I've started to recently). -eric From ncoghlan at gmail.com Sun Mar 1 07:48:10 2015 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 1 Mar 2015 16:48:10 +1000 Subject: [Python-ideas] An even simpler customization of class creation In-Reply-To: References: Message-ID: On 1 March 2015 at 11:30, Eric Snow wrote: > On Sat, Feb 28, 2015 at 10:10 AM, Nick Coghlan wrote: >> That leaves my one concern being the "namespace" keyword parameter. >> I'd started reconsidering that for PEP 422, and I think the argument >> holds at least as strongly for your improved design: >> https://www.python.org/dev/peps/pep-0422/#is-the-namespace-concept-worth-the-extra-complexity >> >> Since speed usually isn't a critical concern for class definitions, >> it's likely better to just make them ordered by default, which can be >> its own PEP, rather than being part of this one. > > FWIW, Guido already told me a while back that OrderedDict by default > for class definition namespace was okay. It's just been waiting on me > to get back to the C OrderedDict implementation (which I've started to > recently). Yeah, that was the conversation that prompted me to question the utility of the namespace keyword idea. The prototype class and class extension examples are cute, but fall into the "don't do this in production" category for me, making them prime candidates for custom metaclasses. The one really useful one is using an ordered dictionary for namespace execution, and your solution addresses that better by just making it the normal behaviour. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From lkb.teichmann at gmail.com Sun Mar 1 10:27:43 2015 From: lkb.teichmann at gmail.com (Martin Teichmann) Date: Sun, 1 Mar 2015 10:27:43 +0100 Subject: [Python-ideas] An even simpler customization of class creation In-Reply-To: References: Message-ID: > Aye, I like this design, it just wasn't until I saw it written out > fully that it finally clicked why restricting it to subclasses let > both zero-argument super() and explicit namespace access to the > defining class both work. Getting the latter to also work is a > definite strength of your approach over mine. That said, I would love if it was just possible to use super() in classmethods called from the __init__ of a metaclass. Until now, the __class__ of a method is only initialized after the metaclasses __init__ has run. Fortunately, this is easily changeable: it is possible to move the initialization of __class__ into type.__new__. As the class is (typically) created there, that is about as early as possible. I put the necessary changes to github: https://github.com/tecki/cpython/commits/cell > That leaves my one concern being the "namespace" keyword parameter. > I'd started reconsidering that for PEP 422, and I think the argument > holds at least as strongly for your improved design: > https://www.python.org/dev/peps/pep-0422/#is-the-namespace-concept-worth-the-extra-complexity > > Since speed usually isn't a critical concern for class definitions, > it's likely better to just make them ordered by default, which can be > its own PEP, rather than being part of this one. I feel the same way as you, so I am happy to drop the namespace idea from PEP 487, but I agree that then ordered namespaces would simply be great, but reading Eric's comment, that seems just around the corner. Greetings Martin From ncoghlan at gmail.com Mon Mar 2 10:45:38 2015 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 2 Mar 2015 19:45:38 +1000 Subject: [Python-ideas] An even simpler customization of class creation In-Reply-To: References: Message-ID: On 1 Mar 2015 19:28, "Martin Teichmann" wrote: > > > Aye, I like this design, it just wasn't until I saw it written out > > fully that it finally clicked why restricting it to subclasses let > > both zero-argument super() and explicit namespace access to the > > defining class both work. Getting the latter to also work is a > > definite strength of your approach over mine. > > That said, I would love if it was just possible to use super() in > classmethods called from the __init__ of a metaclass. Until > now, the __class__ of a method is only initialized after > the metaclasses __init__ has run. > > Fortunately, this is easily changeable: it is possible to move the > initialization of __class__ into type.__new__. As the class is > (typically) created there, that is about as early as possible. Unfortunately, it's too early - we can't control whether or not subclasses actually return the result of type.__new__ from their own __new__ implementation, so that object could have been replaced by something else by the time __init__ gets called. However, it may be feasible to move it into type.__init__. That way class methods would start working properly as soon as the metaclass __init__ implementation had called up into type.__init__. Regards, Nick. -------------- next part -------------- An HTML attachment was scrubbed... URL: From lkb.teichmann at gmail.com Mon Mar 2 11:57:09 2015 From: lkb.teichmann at gmail.com (Martin Teichmann) Date: Mon, 2 Mar 2015 11:57:09 +0100 Subject: [Python-ideas] An even simpler customization of class creation In-Reply-To: References: Message-ID: >> Fortunately, this is easily changeable: it is possible to move the >> initialization of __class__ into type.__new__. As the class is >> (typically) created there, that is about as early as possible. > > Unfortunately, it's too early - we can't control whether or not subclasses > actually return the result of type.__new__ from their own __new__ > implementation, so that object could have been replaced by something else by > the time __init__ gets called. Well, this is actually just a question of definition and documentation. Sure, a metaclasses __new__ may return whatever it pleases, but this is not in contradiction of setting __class__ of the methods in type.__new__. There are two possible scenarios: 1. metaclass.__new__ does not call type.__new__. Once we document that it is type.__new__'s job to set __class__, nobody can complain if __class__ is not set if type.__new__ is not called, as this would be documented behavior. In my changes I actually still left in the old code, so __build_class__ would still try to set __class__ even if type.__new__ did not get called. 2. somehow type.__new__ does get called (most likely by metaclass.__new__). It will do its job and set the __class__ of the methods of the class being created. Whatever metaclass.__new__ returns then doesn't matter much, because what sense would it make to set __class__ of that object it creates? To give an example: Imagine a soliton-generating metaclass: class Soliton(type): def __new__(cls, name, bases, ns): self = super().__new__(name, bases, ns) return self() And generate such a soliton: class A(metaclass=Soliton): def f(self): print(__class__) As of now, writing "A.f()" interestingly prints "<__main__.A object>", so __class__ is indeed set to what Soliton.__new__ returns, the object, not the class. This is currently correct behavior, but I think it actually is not what one expects, nor what one desires. (Does anyone out there make use of such a construct? Please speak up!) super() certainly won't work. So I think it would actually be a change for the better to let type.__new__ set the __class__ of the generated class. I know that it technically breaks backward compatibility. But I ran all the tests and none failed, so apparently until now nobody got weird ideas like me... I even tried to remove the old code from __build_class__ and still all tests run. (just a note, if anyone else here is trying to do so, I tampered with the compiler, so you better delete your .pyc files before running the tests) Greetings Martin From steve at pearwood.info Mon Mar 2 12:46:59 2015 From: steve at pearwood.info (Steven D'Aprano) Date: Mon, 2 Mar 2015 22:46:59 +1100 Subject: [Python-ideas] An even simpler customization of class creation In-Reply-To: References: Message-ID: <20150302114658.GF7655@ando.pearwood.info> On Mon, Mar 02, 2015 at 11:57:09AM +0100, Martin Teichmann wrote: > Imagine a soliton-generating metaclass: > > class Soliton(type): > def __new__(cls, name, bases, ns): > self = super().__new__(name, bases, ns) > return self() You have to pass cls as an explicit argument to __new__, otherwise you get a TypeError: TypeError: type.__new__(X): X is not a type object (str) (that's in Python 3.3). The line should be: self = super().__new__(cls, name, bases, ns) > And generate such a soliton: > > class A(metaclass=Soliton): > def f(self): > print(__class__) > > As of now, writing "A.f()" interestingly prints "<__main__.A object>", so > __class__ is indeed set to what Soliton.__new__ returns, the object, > not the class. Is "soliton" the usual terminology for this? Do you perhaps mean singleton? I've googled for "soliton" but nothing relevant is coming up. > This is currently correct behavior, but I think it actually is not what one > expects, nor what one desires. (Does anyone out there make use of > such a construct? Please speak up!) I do now! Seriously, I think you have just solved a major problem for me. I'll need to do some further experimentation, but that is almost exactly what I have been looking for. I've been looking for a way to have a class statement return a custom instance and I think this might just do it. -- Steve From ncoghlan at gmail.com Mon Mar 2 13:21:44 2015 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 2 Mar 2015 22:21:44 +1000 Subject: [Python-ideas] An even simpler customization of class creation In-Reply-To: References: Message-ID: On 2 March 2015 at 20:57, Martin Teichmann wrote: >>> Fortunately, this is easily changeable: it is possible to move the >>> initialization of __class__ into type.__new__. As the class is >>> (typically) created there, that is about as early as possible. >> >> Unfortunately, it's too early - we can't control whether or not subclasses >> actually return the result of type.__new__ from their own __new__ >> implementation, so that object could have been replaced by something else by >> the time __init__ gets called. > > Well, this is actually just a question of definition and documentation. > Sure, a metaclasses __new__ may return whatever it pleases, but this > is not in contradiction of setting __class__ of the methods in type.__new__. > > There are two possible scenarios: > > 1. metaclass.__new__ does not call type.__new__. Once we document that > it is type.__new__'s job to set __class__, nobody can complain if __class__ > is not set if type.__new__ is not called, as this would be documented behavior. > In my changes I actually still left in the old code, so __build_class__ would > still try to set __class__ even if type.__new__ did not get called. > > 2. somehow type.__new__ does get called (most likely by metaclass.__new__). > It will do its job and set the __class__ of the methods of the class being > created. Whatever metaclass.__new__ returns then doesn't matter much, > because what sense would it make to set __class__ of that object it creates? > > To give an example: > Imagine a soliton-generating metaclass: > > class Soliton(type): > def __new__(cls, name, bases, ns): > self = super().__new__(name, bases, ns) > return self() > > And generate such a soliton: > > class A(metaclass=Soliton): > def f(self): > print(__class__) > > As of now, writing "A.f()" interestingly prints "<__main__.A object>", so > __class__ is indeed set to what Soliton.__new__ returns, the object, > not the class. > > This is currently correct behavior, but I think it actually is not what one > expects, nor what one desires. (Does anyone out there make use of > such a construct? Please speak up!) super() certainly won't work. > So I think it would actually be a change for the better to let > type.__new__ set the __class__ of the generated class. Thinking about this a bit more, I think a good way to look at it is to consider the impact of: 1. Using explicit super() rather than implicit super() 2. Using explicit decorators Currently, in your example, "__class__" and "A" will both refer to an *instance* of the class. If, instead, you used a "singleton" explicit decorator, then "__class__ " would refer to the actual class object, while "A" would refer to the instance. With your proposed change, then "__class__" will always refer to the actual class object, even if the metaclass __new__ plays implicit decoration games. It also means that class methods that rely on __class__ (whether explicitly or implicitly through super()) will "just work" in metaclass __init__ and __new__ methods, although only after the call up to the base class implementation in the latter case. > I know that it technically breaks backward compatibility. But I ran > all the tests and none failed, so apparently until now nobody got > weird ideas like me... I even tried to remove the old code from > __build_class__ and still all tests run. (just a note, if anyone else > here is trying to do so, I tampered with the compiler, so you better > delete your .pyc files before running the tests) Aye, and the discrepancy I was concerned about can already happen when using explicit decorators. That suggest to me that even if someone *was* relying on __class__ pointing to the same object as the bound name when a metaclass does something odd, they can likely switch to referring to it by name instead. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Mon Mar 2 13:25:31 2015 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 2 Mar 2015 22:25:31 +1000 Subject: [Python-ideas] An even simpler customization of class creation In-Reply-To: <20150302114658.GF7655@ando.pearwood.info> References: <20150302114658.GF7655@ando.pearwood.info> Message-ID: On 2 March 2015 at 21:46, Steven D'Aprano wrote: > On Mon, Mar 02, 2015 at 11:57:09AM +0100, Martin Teichmann wrote: >> This is currently correct behavior, but I think it actually is not what one >> expects, nor what one desires. (Does anyone out there make use of >> such a construct? Please speak up!) > > I do now! Seriously, I think you have just solved a major problem for > me. I'll need to do some further experimentation, but that is almost > exactly what I have been looking for. I've been looking for a way to > have a class statement return a custom instance and I think this might > just do it. Martin's question wasn't about using __new__ on the metaclass to do factory function magic (that's "normal usage" as far as metaclasses go). The "construct" in question here is: 1. Returning something from a metaclass __new__ implementation other than the result of type.__new__; AND 2. Relying on __class__ to refer to the returned object, rather than the result of type.__new__ That seems likely to be a vanishingly small subset of Python usage, and even if it does happen, the subset that couldn't switch to using explicit super() via the bound name is likely to be even smaller. That seems like a risk well worth taking when the pay-off is having class methods that use implicit super() just work when called from the metaclass. Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Mon Mar 2 13:45:22 2015 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 2 Mar 2015 22:45:22 +1000 Subject: [Python-ideas] An even simpler customization of class creation In-Reply-To: References: Message-ID: On 28 February 2015 at 02:56, Martin Teichmann wrote: > Connection to other PEP > ======================= > > This is a competing proposal to PEP 422 by Nick Coughlan and Daniel Urban. > It shares both most of the PEP text and proposed code, but has major > differences in how to achieve its goals. I've now withdrawn PEP 422 in favour of this proposal: https://hg.python.org/peps/rev/4db0c1a3e363 (main site hadn't updated at time of writing, but it should propagate before too long) > Proposal > ======== > > This PEP proposes that a new mechanism to customise class creation be > added to Python 3.5 that meets the following criteria: > > 1. Integrates nicely with class inheritance structures (including mixins and > multiple inheritance), > 2. Integrates nicely with the implicit ``__class__`` reference and > zero-argument ``super()`` syntax introduced by PEP 3135, > 3. Can be added to an existing base class without a significant risk of > introducing backwards compatibility problems, and > 4. Restores the ability for class namespaces to have some influence on the > class creation process (above and beyond populating the namespace itself), > but potentially without the full flexibility of the Python 2 style > ``__metaclass__`` hook. > > Those goals can be achieved by adding two functionalities: > > 1. A ``__init_subclass__`` hook that initializes all subclasses of a > given class, and In writing the withdrawal notice for PEP 422, I realised there's a case that needs to be more clearly specified here: the handling of cases where the base class already defines the hook. Since object has a default implementation, we should be able formally define this as the type creation machinery calling: super(cls, cls).__init_subclass__(ns, **kwargs) Rather than: cls.__init_subclass__(ns, **kwargs) We may also want to consider defining object.__init_subclass__ as accepting "cls, ns, *, metaclass=type" so we don't need to implicitly pop metaclass from the kwargs dict when it gets specified. Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From g.rodola at gmail.com Tue Mar 3 01:46:51 2015 From: g.rodola at gmail.com (Giampaolo Rodola') Date: Tue, 3 Mar 2015 01:46:51 +0100 Subject: [Python-ideas] Show deprecation warnings in the interactive interpreter In-Reply-To: References: Message-ID: On Wed, Feb 25, 2015 at 8:51 AM, Serhiy Storchaka wrote: > What you are think about turning deprecation warnings on by default in the > interactive interpreter? > > Deprecation warnings are silent by default because they just annoys end > user that uses old Python program with new Python. End user usually is not > a programmer and can't do something with warnings. But in the interactive > interpreter the user is the author of executed code and can avoid to use > deprecated functions if warned. > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > Am I the only one thinking that disabling deprecation warnings by default (2.6?) was a bad idea? It's particularly counterproductive for library vendors and their users because the latters have no visibility of what APIs are going to be removed: they will just wake up one day, upgrade to the next major version of their favorite lib and have their code broken. -- Giampaolo - http://grodola.blogspot.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From guettliml at thomas-guettler.de Tue Mar 3 09:33:18 2015 From: guettliml at thomas-guettler.de (=?UTF-8?B?VGhvbWFzIEfDvHR0bGVy?=) Date: Tue, 03 Mar 2015 09:33:18 +0100 Subject: [Python-ideas] unittest.failureExeption(msg, data=None): add optional data kwarg to failureException() of unittest Message-ID: <54F571CE.3040901@thomas-guettler.de> Hi, what do you think about the idea to add an optional data kwarg to unittest.failureExeption()? Up to now a the exception class "failureException" does only get a message which is a string. I would like to log the failed Exceptions to a structured logging back end. Parsing the string message to extract the real value is possible, but somehow ugly and error prone. My high level use case: Have a system (maybe ELK-Stack) to investigate if a test failure was seen before. Doing this on strings could work, but I would like to keep the original (json serializable) values. What do you think? Thomas G?ttler From ncoghlan at gmail.com Tue Mar 3 13:58:18 2015 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 3 Mar 2015 22:58:18 +1000 Subject: [Python-ideas] Show deprecation warnings in the interactive interpreter In-Reply-To: References: Message-ID: On 3 March 2015 at 10:46, Giampaolo Rodola' wrote: > > > On Wed, Feb 25, 2015 at 8:51 AM, Serhiy Storchaka > wrote: >> >> What you are think about turning deprecation warnings on by default in the >> interactive interpreter? >> >> Deprecation warnings are silent by default because they just annoys end >> user that uses old Python program with new Python. End user usually is not a >> programmer and can't do something with warnings. But in the interactive >> interpreter the user is the author of executed code and can avoid to use >> deprecated functions if warned. >> >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ > > > Am I the only one thinking that disabling deprecation warnings by default > (2.6?) was a bad idea? > It's particularly counterproductive for library vendors and their users > because the latters have no visibility of what APIs are going to be removed: > they will just wake up one day, upgrade to the next major version of their > favorite lib and have their code broken. Not really, because if you don't have automated regression tests, not seeing the deprecation warnings from your dependencies is the least of your worries - you can break your own stuff completely and have no idea until after you ship the broken version. If I ever find time to implement the PEP 432 changes to the startup sequence (or a volunteer interested in trying to unknot some of the most tangled C code we have left in the interpreter), then it might be feasible to introduce a system wide config file that redistributors and end users could use to decide whether to show deprecation warnings by default or not. In the meantime though, while the status quo has its flaws, I think its less flawed than the previous situation where end users were getting cryptic warnings they couldn't do anything about just because they were running a tool that happened to be written in Python (e.g. consider Mercurial users getting deprecation warnings on Python 2.6 when Python 2.7 was still over a year away). The previous approach was leading to things like tools explicitly silencing the warnings (so end users didn't see them), but then their tests weren't seeing them either. The flip in moving to the status quo (including adjusting test frameworks to turn the deprecation warnings back on) improved the situation for folks doing things right (i.e. using automated regression testing), while making it slightly worse for folks that are trusting to luck (i.e. they may not even get the deprecation warnings before things outright break on them). Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From storchaka at gmail.com Tue Mar 3 16:52:59 2015 From: storchaka at gmail.com (Serhiy Storchaka) Date: Tue, 03 Mar 2015 17:52:59 +0200 Subject: [Python-ideas] IntFlags Message-ID: Enum and IntEnum classes allow constants to have nice str() and repr() representations. >>> socket.AF_INET >>> socket.socket() But when integer constants are flags that should be ORed, IntEnum doesn't help, because the result of bitwise OR of two IntEnum instances is int, and this value can't be represented as IntEnum. We need new type IntFlags. It is like IntEnum, but has differences: 1. The value of an instance should be not limited to the set of predefined constants. It can be a combination of predefined constants or even arbitrary integer. 2. The result of "|", "&" and "~" operators for IntFlags arguments should be an instance of the same IntFlags subclass. 3. It should have nice str() and repr(). >>> print(stat.S_IROTH | stat.S_IWOTH) stat.S_IROTH|stat.S_IWOTH >>> stat.S_IROTH | stat.S_IWOTH Any thoughts? From victor.stinner at gmail.com Tue Mar 3 17:11:39 2015 From: victor.stinner at gmail.com (Victor Stinner) Date: Tue, 3 Mar 2015 17:11:39 +0100 Subject: [Python-ideas] IntFlags In-Reply-To: References: Message-ID: 2015-03-03 16:52 GMT+01:00 Serhiy Storchaka : > Enum and IntEnum classes allow constants to have nice str() and repr() > representations. > >>>> socket.AF_INET > >>>> socket.socket() > type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)> By the way, since the PEP 446, sock.type may contain the flag SOCK_CLOEXEC ;-) Extract of test_socket.py: sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM) self.assertEqual(sock.family, socket.AF_INET) if hasattr(socket, 'SOCK_CLOEXEC'): self.assertIn(sock.type, (socket.SOCK_STREAM | socket.SOCK_CLOEXEC, socket.SOCK_STREAM)) Victor From ethan at stoneleaf.us Tue Mar 3 17:40:11 2015 From: ethan at stoneleaf.us (Ethan Furman) Date: Tue, 03 Mar 2015 08:40:11 -0800 Subject: [Python-ideas] IntFlags In-Reply-To: References: Message-ID: <54F5E3EB.6080506@stoneleaf.us> On 03/03/2015 07:52 AM, Serhiy Storchaka wrote: > > Enum and IntEnum classes allow constants to have nice str() and repr() representations. > > But when integer constants are flags that should be ORed, IntEnum doesn't help, because the result of bitwise OR of two > IntEnum instances is int, and this value can't be represented as IntEnum. > > We need new type IntFlags. It is like IntEnum, but has differences: > > 1. The value of an instance should be not limited to the set of predefined constants. It can be a combination of > predefined constants or even arbitrary integer. > > 2. The result of "|", "&" and "~" operators for IntFlags arguments should be an instance of the same IntFlags subclass. > > 3. It should have nice str() and repr(). This was in my original Enum code, but stripped out as not being needed at the time. If there is sufficient interest (and use-cases) I can add it back in. -- ~Ethan~ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: OpenPGP digital signature URL: From jsbueno at python.org.br Tue Mar 3 17:44:15 2015 From: jsbueno at python.org.br (Joao S. O. Bueno) Date: Tue, 3 Mar 2015 13:44:15 -0300 Subject: [Python-ideas] IntFlags In-Reply-To: <54F5E3EB.6080506@stoneleaf.us> References: <54F5E3EB.6080506@stoneleaf.us> Message-ID: On 3 March 2015 at 13:40, Ethan Furman wrote: > On 03/03/2015 07:52 AM, Serhiy Storchaka wrote: >> >> Enum and IntEnum classes allow constants to have nice str() and repr() representations. >> >> But when integer constants are flags that should be ORed, IntEnum doesn't help, because the result of bitwise OR of two >> IntEnum instances is int, and this value can't be represented as IntEnum. >> >> We need new type IntFlags. It is like IntEnum, but has differences: >> >> 1. The value of an instance should be not limited to the set of predefined constants. It can be a combination of >> predefined constants or even arbitrary integer. >> >> 2. The result of "|", "&" and "~" operators for IntFlags arguments should be an instance of the same IntFlags subclass. >> >> 3. It should have nice str() and repr(). > > This was in my original Enum code, but stripped out as not being needed at the time. > > If there is sufficient interest (and use-cases) I can add it back in. I think that would be nice. The problem is that it would be too late now for changing the behavior of "IntEnum" itself, right? > > -- > ~Ethan~ > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ From me at the-compiler.org Tue Mar 3 17:55:35 2015 From: me at the-compiler.org (Florian Bruhin) Date: Tue, 3 Mar 2015 17:55:35 +0100 Subject: [Python-ideas] IntFlags In-Reply-To: <54F5E3EB.6080506@stoneleaf.us> References: <54F5E3EB.6080506@stoneleaf.us> Message-ID: <20150303165535.GJ11094@tonks> * Ethan Furman [2015-03-03 08:40:11 -0800]: > On 03/03/2015 07:52 AM, Serhiy Storchaka wrote: > > > > Enum and IntEnum classes allow constants to have nice str() and repr() representations. > > > > But when integer constants are flags that should be ORed, IntEnum doesn't help, because the result of bitwise OR of two > > IntEnum instances is int, and this value can't be represented as IntEnum. > > > > We need new type IntFlags. It is like IntEnum, but has differences: > > > > 1. The value of an instance should be not limited to the set of predefined constants. It can be a combination of > > predefined constants or even arbitrary integer. > > > > 2. The result of "|", "&" and "~" operators for IntFlags arguments should be an instance of the same IntFlags subclass. > > > > 3. It should have nice str() and repr(). > > This was in my original Enum code, but stripped out as not being needed at the time. > > If there is sufficient interest (and use-cases) I can add it back in. I'd gladly use it - for example when communicating with embedded electronic devices. Florian -- http://www.the-compiler.org | me at the-compiler.org (Mail/XMPP) GPG: 916E B0C8 FD55 A072 | http://the-compiler.org/pubkey.asc I love long mails! | http://email.is-not-s.ms/ From storchaka at gmail.com Tue Mar 3 18:27:22 2015 From: storchaka at gmail.com (Serhiy Storchaka) Date: Tue, 03 Mar 2015 19:27:22 +0200 Subject: [Python-ideas] IntFlags In-Reply-To: <54F5E3EB.6080506@stoneleaf.us> References: <54F5E3EB.6080506@stoneleaf.us> Message-ID: On 03.03.15 18:40, Ethan Furman wrote: > This was in my original Enum code, but stripped out as not being needed at the time. > > If there is sufficient interest (and use-cases) I can add it back in. This would be good. I need IntFlags even more than IntEnum. I made an implementation based on early implementations of IntEnum, but then IntEnum was changed too much and became incompatible with my extension. IntFlags should be a separate class, IntEnum is good with its restrictions. From ethan at stoneleaf.us Tue Mar 3 18:28:05 2015 From: ethan at stoneleaf.us (Ethan Furman) Date: Tue, 03 Mar 2015 09:28:05 -0800 Subject: [Python-ideas] An even simpler customization of class creation In-Reply-To: References: Message-ID: <54F5EF25.4030002@stoneleaf.us> On 03/02/2015 02:57 AM, Martin Teichmann wrote: > class Soliton(type): > def __new__(cls, name, bases, ns): > self = super().__new__(name, bases, ns) > return self() > > class A(metaclass=Soliton): > def f(self): > print(__class__) This should be `self.__class__`, yes? -- ~Ethan~ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: OpenPGP digital signature URL: From rosuav at gmail.com Tue Mar 3 19:48:51 2015 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 4 Mar 2015 05:48:51 +1100 Subject: [Python-ideas] IntFlags In-Reply-To: References: <54F5E3EB.6080506@stoneleaf.us> Message-ID: On Wed, Mar 4, 2015 at 4:27 AM, Serhiy Storchaka wrote: > IntFlags should be a separate class, IntEnum is good with its restrictions. Also, some operations that make sense for an IntEnum don't make sense for IntFlags (or FlagEnum or whatever it gets called), such as greater-than/less-than comparisons. Definitely a separate class IMO. ChrisA From cs at zip.com.au Tue Mar 3 23:04:01 2015 From: cs at zip.com.au (Cameron Simpson) Date: Wed, 4 Mar 2015 09:04:01 +1100 Subject: [Python-ideas] IntFlags In-Reply-To: References: Message-ID: <20150303220401.GA13067@cskk.homeip.net> On 03Mar2015 19:27, Serhiy Storchaka wrote: >On 03.03.15 18:40, Ethan Furman wrote: >>This was in my original Enum code, but stripped out as not being needed at the time. >> >>If there is sufficient interest (and use-cases) I can add it back in. > >This would be good. I need IntFlags even more than IntEnum. I am also +1. I also probably use IntFlags (euqivalents) more often than IntEnums. It would have been nice for my PEP-418 demo implementation: https://bitbucket.org/cameron_simpson/css/src/tip/lib/python/cs/clockutils.py I rolled my own _Clock_Flags there for this very reason. >I made an implementation based on early implementations of IntEnum, >but then IntEnum was changed too much and became incompatible with my >extension. > >IntFlags should be a separate class, IntEnum is good with its restrictions. I agree. I presume IntEnum|IntEnum might return an IntFlags? Or would that make for a TypeError or ValueError and one be expected to start with IntFlags? Cheers, Cameron Simpson Trust the computer... the computer is your friend. - Richard Dominelli From rosuav at gmail.com Tue Mar 3 23:16:47 2015 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 4 Mar 2015 09:16:47 +1100 Subject: [Python-ideas] IntFlags In-Reply-To: <20150303220401.GA13067@cskk.homeip.net> References: <20150303220401.GA13067@cskk.homeip.net> Message-ID: On Wed, Mar 4, 2015 at 9:04 AM, Cameron Simpson wrote: > I agree. I presume IntEnum|IntEnum might return an IntFlags? Or would that > make for a TypeError or ValueError and one be expected to start with > IntFlags? That should be a sanity error, but since an IntEnum devolves to an int, it just produces a nonsensical integer. It definitely shouldn't become an IntFlags. There are two distinctly different use-cases here: class Color(IntEnum): black = 0 red = 1 green = 2 orange = 3 blue = 4 magenta = 5 cyan = 6 white = 7 class FileMode(IntFlags): owner_read = 0o400 owner_write= 0o200 owner_exec = 0o100 group_read = 0o040 group_write= 0o020 group_exec = 0o010 other_read = 0o004 other_write= 0o002 other_exec = 0o001 With colors, it makes no sense to combine them in any way (addition, bitwise or, etc). You can't put blue and cyan together and expect to get something usable. File modes are meant to be combined, and their values are deliberately chosen to make this possible. So IntEnum|IntEnum is most likely going to be combining values that were assigned sequentially or arbitrarily, which isn't likely to be very useful. ChrisA From ethan at stoneleaf.us Tue Mar 3 23:29:34 2015 From: ethan at stoneleaf.us (Ethan Furman) Date: Tue, 03 Mar 2015 14:29:34 -0800 Subject: [Python-ideas] IntFlags In-Reply-To: References: <20150303220401.GA13067@cskk.homeip.net> Message-ID: <54F635CE.6070508@stoneleaf.us> On 03/03/2015 02:16 PM, Chris Angelico wrote: > On Wed, Mar 4, 2015 at 9:04 AM, Cameron Simpson wrote: >> I agree. I presume IntEnum|IntEnum might return an IntFlags? Or would that >> make for a TypeError or ValueError and one be expected to start with >> IntFlags? > > That should be a sanity error, but since an IntEnum devolves to an > int, it just produces a nonsensical integer. It definitely shouldn't > become an IntFlags. Agreed. IntEnums are just fancy ints, and will continue to behave as ints with the exception of how they are displayed. IntFlags, should such a thing come into being, can be more proprietary with its behavior. -- ~Ethan~ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: OpenPGP digital signature URL: From mistersheik at gmail.com Wed Mar 4 02:45:38 2015 From: mistersheik at gmail.com (Neil Girdhar) Date: Tue, 3 Mar 2015 17:45:38 -0800 (PST) Subject: [Python-ideas] unittest.failureExeption(msg, data=None): add optional data kwarg to failureException() of unittest In-Reply-To: <54F571CE.3040901@thomas-guettler.de> References: <54F571CE.3040901@thomas-guettler.de> Message-ID: <3e24116d-16af-40fc-b1b9-ab4ebefc8d2a@googlegroups.com> Says it right in the documentation: failureException This class attribute gives the exception raised by the test method. If a test framework needs to use a specialized exception, possibly to carry additional information, it must subclass this exception in order to ?play fair? with the framework. The initial value of this attribute is AssertionError . On Tuesday, March 3, 2015 at 3:34:02 AM UTC-5, Thomas G?ttler wrote: > > Hi, > > what do you think about the idea to add an optional data kwarg to > unittest.failureExeption()? > > Up to now a the exception class "failureException" does only get a message > which is a string. > > I would like to log the failed Exceptions to a structured logging back end. > > Parsing the string message to extract the real value is possible, but > somehow ugly and error prone. > > My high level use case: Have a system (maybe ELK-Stack) to investigate if > a test failure was seen before. > > Doing this on strings could work, but I would like to keep the original > (json serializable) values. > > What do you think? > > Thomas G?ttler > _______________________________________________ > Python-ideas mailing list > Python... at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From cs at zip.com.au Wed Mar 4 04:42:47 2015 From: cs at zip.com.au (Cameron Simpson) Date: Wed, 4 Mar 2015 14:42:47 +1100 Subject: [Python-ideas] IntFlags In-Reply-To: References: Message-ID: <20150304034247.GA77820@cskk.homeip.net> On 04Mar2015 09:16, Chris Angelico wrote: >On Wed, Mar 4, 2015 at 9:04 AM, Cameron Simpson wrote: >> I agree. I presume IntEnum|IntEnum might return an IntFlags? Or would that >> make for a TypeError or ValueError and one be expected to start with >> IntFlags? > >That should be a sanity error, but since an IntEnum devolves to an >int, it just produces a nonsensical integer. It definitely shouldn't >become an IntFlags. Yes, agreed. Brain fade. Sorry, Cameron Simpson From mistersheik at gmail.com Wed Mar 4 05:55:23 2015 From: mistersheik at gmail.com (Neil Girdhar) Date: Tue, 3 Mar 2015 20:55:23 -0800 (PST) Subject: [Python-ideas] IntFlags In-Reply-To: <20150303220401.GA13067@cskk.homeip.net> References: <20150303220401.GA13067@cskk.homeip.net> Message-ID: <77e8ae02-ed09-4771-9da7-c881f0321f6b@googlegroups.com> Also +1. I think IntFlags should also have a named constructor that constructs an IntFlags instance from a set: reduce(lambda x, y: x | y, (c[key] for key in some_set)) Best, Neil On Tuesday, March 3, 2015 at 5:04:50 PM UTC-5, Cameron Simpson wrote: > > On 03Mar2015 19:27, Serhiy Storchaka > > wrote: > >On 03.03.15 18:40, Ethan Furman wrote: > >>This was in my original Enum code, but stripped out as not being needed > at the time. > >> > >>If there is sufficient interest (and use-cases) I can add it back in. > > > >This would be good. I need IntFlags even more than IntEnum. > > I am also +1. > > I also probably use IntFlags (euqivalents) more often than IntEnums. It > would > have been nice for my PEP-418 demo implementation: > > > https://bitbucket.org/cameron_simpson/css/src/tip/lib/python/cs/clockutils.py > > I rolled my own _Clock_Flags there for this very reason. > > >I made an implementation based on early implementations of IntEnum, > >but then IntEnum was changed too much and became incompatible with my > >extension. > > > >IntFlags should be a separate class, IntEnum is good with its > restrictions. > > I agree. I presume IntEnum|IntEnum might return an IntFlags? Or would that > make > for a TypeError or ValueError and one be expected to start with IntFlags? > > Cheers, > Cameron Simpson > > > Trust the computer... the computer is your friend. > - Richard Dominelli > > _______________________________________________ > Python-ideas mailing list > Python... at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From guettliml at thomas-guettler.de Wed Mar 4 06:55:38 2015 From: guettliml at thomas-guettler.de (=?UTF-8?B?VGhvbWFzIEfDvHR0bGVy?=) Date: Wed, 04 Mar 2015 06:55:38 +0100 Subject: [Python-ideas] unittest.failureExeption(msg, data=None): add optional data kwarg to failureException() of unittest In-Reply-To: <3e24116d-16af-40fc-b1b9-ab4ebefc8d2a@googlegroups.com> References: <54F571CE.3040901@thomas-guettler.de> <3e24116d-16af-40fc-b1b9-ab4ebefc8d2a@googlegroups.com> Message-ID: <54F69E5A.1020907@thomas-guettler.de> Am 04.03.2015 um 02:45 schrieb Neil Girdhar: > Says it right in the documentation: > > failureException > > > This class attribute gives the exception raised by the test method. If a > test framework needs to use a specialized exception, possibly to carry > additional information, it must subclass this exception in order to ?play > fair? with the framework. The initial value of this attribute is > AssertionError > You are right. And if you want to assertEqual(a, b) to capture a and b, you need to replace assertEqual to forward the values to your own failureException. I guess we will go this way: replace failureException and assertEqual(), assertIn(), .... No change in Python's unittest lib is needed. Thank you for your reply. Thomas G?ttler -- http://www.thomas-guettler.de/ From mistersheik at gmail.com Wed Mar 4 06:59:32 2015 From: mistersheik at gmail.com (Neil Girdhar) Date: Wed, 4 Mar 2015 00:59:32 -0500 Subject: [Python-ideas] unittest.failureExeption(msg, data=None): add optional data kwarg to failureException() of unittest In-Reply-To: <54F69E5A.1020907@thomas-guettler.de> References: <54F571CE.3040901@thomas-guettler.de> <3e24116d-16af-40fc-b1b9-ab4ebefc8d2a@googlegroups.com> <54F69E5A.1020907@thomas-guettler.de> Message-ID: You got it. On Wed, Mar 4, 2015 at 12:55 AM, Thomas G?ttler < guettliml at thomas-guettler.de> wrote: > Am 04.03.2015 um 02:45 schrieb Neil Girdhar: > > Says it right in the documentation: > > > > failureException > > < > https://docs.python.org/2/library/unittest.html#unittest.TestCase.failureException > > > > > > This class attribute gives the exception raised by the test method. If a > > test framework needs to use a specialized exception, possibly to carry > > additional information, it must subclass this exception in order to ?play > > fair? with the framework. The initial value of this attribute is > > AssertionError > > < > https://docs.python.org/2/library/exceptions.html#exceptions.AssertionError > > > > You are right. > > And if you want to assertEqual(a, b) to capture a and b, > you need to replace assertEqual to forward the values to > your own failureException. > > I guess we will go this way: replace failureException and > assertEqual(), assertIn(), .... > > No change in Python's unittest lib is needed. > > Thank you for your reply. > > Thomas G?ttler > > -- > http://www.thomas-guettler.de/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From abarnert at yahoo.com Wed Mar 4 16:17:49 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Wed, 4 Mar 2015 07:17:49 -0800 Subject: [Python-ideas] IntFlags In-Reply-To: References: Message-ID: <9509685D-34AA-4D65-9AB0-2599E6E10D88@yahoo.com> One of the big questions that (IIRC) derailed this last time and got it dropped from the enum stdlib design was: what does ~ do? Does it give you the 2's complement negative integer? What does that display as in the str and repr? And, if you add in conversion from an IntFlags to/from a set of separate values, as has been suggested again in this thread, how does that work? All of this is trivial when you're dealing with C fixed-size unsigned ints: ~READ means 15 of the 16 bits (all except the READ bit) are set. Another issue that came up was that C flags often have "combined" names that are ambiguous: RDWR = RDONLY | WRONLY), which is fine until you want a repr (in C, it's just going to print 3); does it have to be smart enough to show RDWR? (Or, worse, RDWR | CLOEXEC.) Sent from a random iPhone On Mar 3, 2015, at 7:52, Serhiy Storchaka wrote: > Enum and IntEnum classes allow constants to have nice str() and repr() representations. > > >>> socket.AF_INET > > >>> socket.socket() > > > But when integer constants are flags that should be ORed, IntEnum doesn't help, because the result of bitwise OR of two IntEnum instances is int, and this value can't be represented as IntEnum. > > We need new type IntFlags. It is like IntEnum, but has differences: > > 1. The value of an instance should be not limited to the set of predefined constants. It can be a combination of predefined constants or even arbitrary integer. > > 2. The result of "|", "&" and "~" operators for IntFlags arguments should be an instance of the same IntFlags subclass. > > 3. It should have nice str() and repr(). > > >>> print(stat.S_IROTH | stat.S_IWOTH) > stat.S_IROTH|stat.S_IWOTH > >>> stat.S_IROTH | stat.S_IWOTH > > > Any thoughts? > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ From rosuav at gmail.com Wed Mar 4 16:31:25 2015 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 5 Mar 2015 02:31:25 +1100 Subject: [Python-ideas] IntFlags In-Reply-To: <9509685D-34AA-4D65-9AB0-2599E6E10D88@yahoo.com> References: <9509685D-34AA-4D65-9AB0-2599E6E10D88@yahoo.com> Message-ID: On Thu, Mar 5, 2015 at 2:17 AM, Andrew Barnert wrote: > Another issue that came up was that C flags often have "combined" names that are ambiguous: RDWR = RDONLY | WRONLY), which is fine until you want a repr (in C, it's just going to print 3); does it have to be smart enough to show RDWR? (Or, worse, RDWR | CLOEXEC.) > That could probably be handled by going through the flags in iteration order. If the flag is present, emit it and move on. Something like this: from enum import IntEnum class Flags(IntEnum): RDWR = 3 RDONLY = 1 WRONLY = 2 CLOEXEC = 4 def flag_str(flg): names = [] for flag in Flags: if (flg&flag) == flag: flg -= flag names.append(str(flag)) return "|".join(names) print(flag_str(Flags.RDWR|Flags.CLOEXEC)) print(flag_str(Flags.RDONLY|Flags.CLOEXEC)) As long as the combined versions come up ahead of the others, they'll be used. Alternatively, if you prefer them _not_ to be used, just put them after the individual forms, and then the str() will expand them out. ChrisA From mistersheik at gmail.com Wed Mar 4 16:44:33 2015 From: mistersheik at gmail.com (Neil Girdhar) Date: Wed, 4 Mar 2015 10:44:33 -0500 Subject: [Python-ideas] IntFlags In-Reply-To: References: <9509685D-34AA-4D65-9AB0-2599E6E10D88@yahoo.com> Message-ID: Why do you need these composite flags? On Wed, Mar 4, 2015 at 10:31 AM, Chris Angelico wrote: > On Thu, Mar 5, 2015 at 2:17 AM, Andrew Barnert > wrote: > > Another issue that came up was that C flags often have "combined" names > that are ambiguous: RDWR = RDONLY | WRONLY), which is fine until you want a > repr (in C, it's just going to print 3); does it have to be smart enough to > show RDWR? (Or, worse, RDWR | CLOEXEC.) > > > > That could probably be handled by going through the flags in iteration > order. If the flag is present, emit it and move on. Something like > this: > > from enum import IntEnum > > class Flags(IntEnum): > RDWR = 3 > RDONLY = 1 > WRONLY = 2 > CLOEXEC = 4 > > def flag_str(flg): > names = [] > for flag in Flags: > if (flg&flag) == flag: > flg -= flag > names.append(str(flag)) > return "|".join(names) > > print(flag_str(Flags.RDWR|Flags.CLOEXEC)) > print(flag_str(Flags.RDONLY|Flags.CLOEXEC)) > > > As long as the combined versions come up ahead of the others, they'll > be used. Alternatively, if you prefer them _not_ to be used, just put > them after the individual forms, and then the str() will expand them > out. > > ChrisA > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > > -- > > --- > You received this message because you are subscribed to a topic in the > Google Groups "python-ideas" group. > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/python-ideas/L5KfCEXFaII/unsubscribe. > To unsubscribe from this group and all its topics, send an email to > python-ideas+unsubscribe at googlegroups.com. > For more options, visit https://groups.google.com/d/optout. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mistersheik at gmail.com Wed Mar 4 16:51:21 2015 From: mistersheik at gmail.com (Neil Girdhar) Date: Wed, 4 Mar 2015 07:51:21 -0800 (PST) Subject: [Python-ideas] IntFlags In-Reply-To: References: Message-ID: <47349895-ea31-4f85-aa57-4a637c93e0a4@googlegroups.com> For ~, I suggest either having an extra bit on the object that remembers negation or replacing the patterns a &= ~b with a.clear_flags(b) and a & ~b with a.without_flags(b) or something like that. On Tuesday, March 3, 2015 at 10:54:15 AM UTC-5, Serhiy Storchaka wrote: > > Enum and IntEnum classes allow constants to have nice str() and repr() > representations. > > >>> socket.AF_INET > > >>> socket.socket() > type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)> > > But when integer constants are flags that should be ORed, IntEnum > doesn't help, because the result of bitwise OR of two IntEnum instances > is int, and this value can't be represented as IntEnum. > > We need new type IntFlags. It is like IntEnum, but has differences: > > 1. The value of an instance should be not limited to the set of > predefined constants. It can be a combination of predefined constants or > even arbitrary integer. > > 2. The result of "|", "&" and "~" operators for IntFlags arguments > should be an instance of the same IntFlags subclass. > > 3. It should have nice str() and repr(). > > >>> print(stat.S_IROTH | stat.S_IWOTH) > stat.S_IROTH|stat.S_IWOTH > >>> stat.S_IROTH | stat.S_IWOTH > > > Any thoughts? > > _______________________________________________ > Python-ideas mailing list > Python... at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From storchaka at gmail.com Wed Mar 4 16:54:29 2015 From: storchaka at gmail.com (Serhiy Storchaka) Date: Wed, 04 Mar 2015 17:54:29 +0200 Subject: [Python-ideas] IntFlags In-Reply-To: <9509685D-34AA-4D65-9AB0-2599E6E10D88@yahoo.com> References: <9509685D-34AA-4D65-9AB0-2599E6E10D88@yahoo.com> Message-ID: On 04.03.15 17:17, Andrew Barnert wrote: > One of the big questions that (IIRC) derailed this last time and got it dropped from the enum stdlib design was: what does ~ do? Does it give you the 2's complement negative integer? What does that display as in the str and repr? And, if you add in conversion from an IntFlags to/from a set of separate values, as has been suggested again in this thread, how does that work? All of this is trivial when you're dealing with C fixed-size unsigned ints: ~READ means 15 of the 16 bits (all except the READ bit) are set. IntFlags is just fancy int. int(~flags) == int(~int(flags)). Python supports ~ for arbitrary integers. ~x == -x-1 There are no problems with conversions from a set to IntFlags, but the conversion from IntFlags to set is not always possible. > Another issue that came up was that C flags often have "combined" names that are ambiguous: RDWR = RDONLY | WRONLY), which is fine until you want a repr (in C, it's just going to print 3); does it have to be smart enough to show RDWR? (Or, worse, RDWR | CLOEXEC.) This problem is nor so hard. My implementation was smart enough. >>> print(OpenMode.RDONLY | OpenMode.WRONLY | OpenMode.CLOEXEC) OpenMode.RDWR|OpenMode.CLOEXEC >>> print(~(OpenMode.RDONLY | OpenMode.WRONLY | OpenMode.CLOEXEC)) ~(OpenMode.RDWR|OpenMode.CLOEXEC) Of course the repr can be senseless if the value is senseless (such as RDONLY | ~WRONLY). From storchaka at gmail.com Wed Mar 4 16:58:37 2015 From: storchaka at gmail.com (Serhiy Storchaka) Date: Wed, 04 Mar 2015 17:58:37 +0200 Subject: [Python-ideas] IntFlags In-Reply-To: References: <9509685D-34AA-4D65-9AB0-2599E6E10D88@yahoo.com> Message-ID: On 04.03.15 17:31, Chris Angelico wrote: > On Thu, Mar 5, 2015 at 2:17 AM, Andrew Barnert > wrote: >> Another issue that came up was that C flags often have "combined" names that are ambiguous: RDWR = RDONLY | WRONLY), which is fine until you want a repr (in C, it's just going to print 3); does it have to be smart enough to show RDWR? (Or, worse, RDWR | CLOEXEC.) > That could probably be handled by going through the flags in iteration > order. If the flag is present, emit it and move on. Something like > this: Yes, something like this, but with iterating flags in descended sorted order, and with special case for negative value. From rosuav at gmail.com Wed Mar 4 17:04:09 2015 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 5 Mar 2015 03:04:09 +1100 Subject: [Python-ideas] IntFlags In-Reply-To: References: <9509685D-34AA-4D65-9AB0-2599E6E10D88@yahoo.com> Message-ID: On Thu, Mar 5, 2015 at 2:58 AM, Serhiy Storchaka wrote: > On 04.03.15 17:31, Chris Angelico wrote: >> >> On Thu, Mar 5, 2015 at 2:17 AM, Andrew Barnert >> wrote: >>> >>> Another issue that came up was that C flags often have "combined" names >>> that are ambiguous: RDWR = RDONLY | WRONLY), which is fine until you want a >>> repr (in C, it's just going to print 3); does it have to be smart enough to >>> show RDWR? (Or, worse, RDWR | CLOEXEC.) >> >> That could probably be handled by going through the flags in iteration >> order. If the flag is present, emit it and move on. Something like >> this: > > > Yes, something like this, but with iterating flags in descended sorted > order, and with special case for negative value. Sure. Going automatically like that is a way of guaranteeing that the combined flags will be used, which is probably what you want most of the time anyway. And yes, I hadn't coped with negatives in that. ChrisA From python at 2sn.net Wed Mar 4 19:37:38 2015 From: python at 2sn.net (Alexander Heger) Date: Thu, 5 Mar 2015 05:37:38 +1100 Subject: [Python-ideas] IntFlags In-Reply-To: References: <20150303220401.GA13067@cskk.homeip.net> Message-ID: > That should be a sanity error, but since an IntEnum devolves to an > int, it just produces a nonsensical integer. It definitely shouldn't > become an IntFlags. There are two distinctly different use-cases here: > > class Color(IntEnum): > black = 0 > red = 1 > green = 2 > orange = 3 > blue = 4 > magenta = 5 > cyan = 6 > white = 7 > > class FileMode(IntFlags): > owner_read = 0o400 > owner_write= 0o200 > owner_exec = 0o100 > group_read = 0o040 > group_write= 0o020 > group_exec = 0o010 > other_read = 0o004 > other_write= 0o002 > other_exec = 0o001 > > With colors, it makes no sense to combine them in any way (addition, > bitwise or, etc). You can't put blue and cyan together and expect to > get something usable. I don't see your point. With colours it would be exactly the same: red | blue == magenta and for your example: cyan | blue = cyan As with the later discussion on file flags, cyan is just a combination flag for green and blue. -Alexander From python at 2sn.net Wed Mar 4 19:49:25 2015 From: python at 2sn.net (Alexander Heger) Date: Thu, 5 Mar 2015 05:49:25 +1100 Subject: [Python-ideas] IntFlags In-Reply-To: References: <9509685D-34AA-4D65-9AB0-2599E6E10D88@yahoo.com> Message-ID: >> One of the big questions that (IIRC) derailed this last time and got it >> dropped from the enum stdlib design was: what does ~ do? Does it give you >> the 2's complement negative integer? What does that display as in the str >> and repr? And, if you add in conversion from an IntFlags to/from a set of >> separate values, as has been suggested again in this thread, how does that >> work? All of this is trivial when you're dealing with C fixed-size unsigned >> ints: ~READ means 15 of the 16 bits (all except the READ bit) are set. > > > IntFlags is just fancy int. int(~flags) == int(~int(flags)). Python supports > ~ for arbitrary integers. ~x == -x-1 > > There are no problems with conversions from a set to IntFlags, but the > conversion from IntFlags to set is not always possible. > >> Another issue that came up was that C flags often have "combined" names >> that are ambiguous: RDWR = RDONLY | WRONLY), which is fine until you want a >> repr (in C, it's just going to print 3); does it have to be smart enough to >> show RDWR? (Or, worse, RDWR | CLOEXEC.) > > > This problem is nor so hard. My implementation was smart enough. > >>>> print(OpenMode.RDONLY | OpenMode.WRONLY | OpenMode.CLOEXEC) > OpenMode.RDWR|OpenMode.CLOEXEC >>>> print(~(OpenMode.RDONLY | OpenMode.WRONLY | OpenMode.CLOEXEC)) > ~(OpenMode.RDWR|OpenMode.CLOEXEC) > > Of course the repr can be senseless if the value is senseless (such as > RDONLY | ~WRONLY). Wouldn't it be possible to for the class to determine what bits are all used from the constants defined, and then for the ~ operator to just invert those? It may require some sanity check for users only defining "combination constants" in the class such that the result could not be represented. class Stupid(IntFlags): CAT = 3 DOG = 6 in which case ~CAT would not make sense, but neither could CAT | DOG be represented. I suppose in such cases an error should be raised by the metaclass on class definition. -Alexander From random832 at fastmail.us Wed Mar 4 20:45:07 2015 From: random832 at fastmail.us (random832 at fastmail.us) Date: Wed, 04 Mar 2015 14:45:07 -0500 Subject: [Python-ideas] IntFlags In-Reply-To: References: Message-ID: <1425498307.1916175.235567981.156D6415@webmail.messagingengine.com> On Tue, Mar 3, 2015, at 10:52, Serhiy Storchaka wrote: > 3. It should have nice str() and repr(). > > >>> print(stat.S_IROTH | stat.S_IWOTH) > stat.S_IROTH|stat.S_IWOTH > >>> stat.S_IROTH | stat.S_IWOTH > > > Any thoughts? Any reason StatFlags shouldn't have a representation like "rwxr-x---" for 0750? From skip.montanaro at gmail.com Wed Mar 4 20:47:17 2015 From: skip.montanaro at gmail.com (Skip Montanaro) Date: Wed, 4 Mar 2015 13:47:17 -0600 Subject: [Python-ideas] IntFlags In-Reply-To: <1425498307.1916175.235567981.156D6415@webmail.messagingengine.com> References: <1425498307.1916175.235567981.156D6415@webmail.messagingengine.com> Message-ID: On Wed, Mar 4, 2015 at 1:45 PM, wrote: > Any reason StatFlags shouldn't have a representation like "rwxr-x---" for 0750? I've never considered this before, but is this sort of thing portable to/meaningful on Windows? Skip From random832 at fastmail.us Wed Mar 4 20:48:26 2015 From: random832 at fastmail.us (random832 at fastmail.us) Date: Wed, 04 Mar 2015 14:48:26 -0500 Subject: [Python-ideas] IntFlags In-Reply-To: References: <20150303220401.GA13067@cskk.homeip.net> Message-ID: <1425498506.1917011.235568685.4E5E7144@webmail.messagingengine.com> On Tue, Mar 3, 2015, at 17:16, Chris Angelico wrote: > class Color(IntEnum): > black = 0 > red = 1 > green = 2 > orange = 3 > blue = 4 > magenta = 5 > cyan = 6 > white = 7 > > With colors, it makes no sense to combine them in any way (addition, > bitwise or, etc). You can't put blue and cyan together and expect to > get something usable. Except for the part where green|blue==cyan... and if only you'd said yellow instead of orange it'd hold across the board. I assume you took these values from ANSI colors. From random832 at fastmail.us Wed Mar 4 20:52:50 2015 From: random832 at fastmail.us (random832 at fastmail.us) Date: Wed, 04 Mar 2015 14:52:50 -0500 Subject: [Python-ideas] IntFlags In-Reply-To: <9509685D-34AA-4D65-9AB0-2599E6E10D88@yahoo.com> References: <9509685D-34AA-4D65-9AB0-2599E6E10D88@yahoo.com> Message-ID: <1425498770.1920134.235569725.07B56128@webmail.messagingengine.com> On Wed, Mar 4, 2015, at 10:17, Andrew Barnert wrote: > One of the big questions that (IIRC) derailed this last time and got it > dropped from the enum stdlib design was: what does ~ do? Does it give you > the 2's complement negative integer? What does that display as in the str > and repr? And, if you add in conversion from an IntFlags to/from a set of > separate values, as has been suggested again in this thread, how does > that work? All of this is trivial when you're dealing with C fixed-size > unsigned ints: ~READ means 15 of the 16 bits (all except the READ bit) > are set. str should be ~(READ) obviously. And more generally ~(all|bits|that|are|not|set). > Another issue that came up was that C flags often have "combined" names > that are ambiguous: RDWR = RDONLY | WRONLY), Nope. You've actually got a different, much worse, issue: traditionally, RDONLY is 0 (and should be printed if and only if 1 and 2 are not set), WRONLY is 1, RDWR is 2, and 3 is traditionally invalid and may have a platform-dependent meaning. From random832 at fastmail.us Wed Mar 4 20:54:43 2015 From: random832 at fastmail.us (random832 at fastmail.us) Date: Wed, 04 Mar 2015 14:54:43 -0500 Subject: [Python-ideas] IntFlags In-Reply-To: References: <20150303220401.GA13067@cskk.homeip.net> Message-ID: <1425498883.1920337.235571721.245D6167@webmail.messagingengine.com> On Wed, Mar 4, 2015, at 13:37, Alexander Heger wrote: > I don't see your point. With colours it would be exactly the same: > > red | blue == magenta > > and for your example: > > cyan | blue = cyan > > As with the later discussion on file flags, cyan is just a combination > flag for green and blue. I think it was a badly chosen example - someone easily _could_ have a color enum that doesn't have these properties. Definitely there are enums that it doesn't make sense to combine together with bitwise ops. From p.f.moore at gmail.com Wed Mar 4 21:17:27 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 4 Mar 2015 20:17:27 +0000 Subject: [Python-ideas] IntFlags In-Reply-To: References: <1425498307.1916175.235567981.156D6415@webmail.messagingengine.com> Message-ID: On 4 March 2015 at 19:47, Skip Montanaro wrote: > On Wed, Mar 4, 2015 at 1:45 PM, wrote: > >> Any reason StatFlags shouldn't have a representation like "rwxr-x---" for 0750? > > I've never considered this before, but is this sort of thing portable > to/meaningful on Windows? Not really. Windows' actual ACL system is far too complex to fit into 9 bits. But the readable and writable bits are use to reflect the readonly flag (which is a separate value from the ACLs) of a file (so S_IREAD is always true, and S_IWRITE is true unless the file is readonly). So you only ever get "'-rw-rw-rw-'" or "'-r--r--r--'" Paul From rosuav at gmail.com Wed Mar 4 21:58:38 2015 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 5 Mar 2015 07:58:38 +1100 Subject: [Python-ideas] IntFlags In-Reply-To: <1425498506.1917011.235568685.4E5E7144@webmail.messagingengine.com> References: <20150303220401.GA13067@cskk.homeip.net> <1425498506.1917011.235568685.4E5E7144@webmail.messagingengine.com> Message-ID: On Thu, Mar 5, 2015 at 6:48 AM, wrote: > On Tue, Mar 3, 2015, at 17:16, Chris Angelico wrote: >> class Color(IntEnum): >> black = 0 >> red = 1 >> green = 2 >> orange = 3 >> blue = 4 >> magenta = 5 >> cyan = 6 >> white = 7 >> >> With colors, it makes no sense to combine them in any way (addition, >> bitwise or, etc). You can't put blue and cyan together and expect to >> get something usable. > > Except for the part where green|blue==cyan... and if only you'd said > yellow instead of orange it'd hold across the board. I assume you took > these values from ANSI colors. Those are indeed the ANSI colors, but even though you might think that you can combine them, they don't really combine usefully in all cases. But sure. If you're bothered by the fact that blue+green == cyan, use a different example. Turn 'em into animals, cars, CPU models, whatever you like; the point of an IntEnum is usually just that the values are unique, _not_ that they can be combined in any meaningful way. ChrisA From random832 at fastmail.us Wed Mar 4 23:31:55 2015 From: random832 at fastmail.us (random832 at fastmail.us) Date: Wed, 04 Mar 2015 17:31:55 -0500 Subject: [Python-ideas] IntFlags In-Reply-To: References: <1425498307.1916175.235567981.156D6415@webmail.messagingengine.com> Message-ID: <1425508315.1963424.235639341.7EF9350A@webmail.messagingengine.com> On Wed, Mar 4, 2015, at 14:47, Skip Montanaro wrote: > On Wed, Mar 4, 2015 at 1:45 PM, wrote: > > > Any reason StatFlags shouldn't have a representation like "rwxr-x---" for 0750? > > I've never considered this before, but is this sort of thing portable > to/meaningful on Windows? No, but neither are 90% of the flags themselves, especially as actually implemented by the C runtime library (they're basically all hardcoded, except the write flags are turned off if a file is readonly, the exec flags are set based on a hardcoded list of filename extensions, and a handful of the file type bit states are implemented) If someone wants to do something meaningful on windows, they'll need a richer API than the one currently implemented in the os module. From abarnert at yahoo.com Wed Mar 4 23:30:07 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Wed, 4 Mar 2015 14:30:07 -0800 Subject: [Python-ideas] IntFlags In-Reply-To: References: <9509685D-34AA-4D65-9AB0-2599E6E10D88@yahoo.com> Message-ID: On Mar 4, 2015, at 7:44, Neil Girdhar wrote: > Why do you need these composite flags? Because the whole point of this proposal is to deal with C types (otherwise, who cares about the int value?). And most such C types define combined values--the motivating example, stat, has S_IRWXU, etc. If the Python code is less readable than the equivalent C code... > On Wed, Mar 4, 2015 at 10:31 AM, Chris Angelico wrote: >> On Thu, Mar 5, 2015 at 2:17 AM, Andrew Barnert >> wrote: >> > Another issue that came up was that C flags often have "combined" names that are ambiguous: RDWR = RDONLY | WRONLY), which is fine until you want a repr (in C, it's just going to print 3); does it have to be smart enough to show RDWR? (Or, worse, RDWR | CLOEXEC.) >> > >> >> That could probably be handled by going through the flags in iteration >> order. If the flag is present, emit it and move on. Something like >> this: >> >> from enum import IntEnum >> >> class Flags(IntEnum): >> RDWR = 3 >> RDONLY = 1 >> WRONLY = 2 >> CLOEXEC = 4 >> >> def flag_str(flg): >> names = [] >> for flag in Flags: >> if (flg&flag) == flag: >> flg -= flag >> names.append(str(flag)) >> return "|".join(names) >> >> print(flag_str(Flags.RDWR|Flags.CLOEXEC)) >> print(flag_str(Flags.RDONLY|Flags.CLOEXEC)) >> >> >> As long as the combined versions come up ahead of the others, they'll >> be used. Alternatively, if you prefer them _not_ to be used, just put >> them after the individual forms, and then the str() will expand them >> out. >> >> ChrisA >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ >> >> -- >> >> --- >> You received this message because you are subscribed to a topic in the Google Groups "python-ideas" group. >> To unsubscribe from this topic, visit https://groups.google.com/d/topic/python-ideas/L5KfCEXFaII/unsubscribe. >> To unsubscribe from this group and all its topics, send an email to python-ideas+unsubscribe at googlegroups.com. >> For more options, visit https://groups.google.com/d/optout. > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From abarnert at yahoo.com Wed Mar 4 23:59:48 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Wed, 4 Mar 2015 14:59:48 -0800 Subject: [Python-ideas] IntFlags In-Reply-To: <47349895-ea31-4f85-aa57-4a637c93e0a4@googlegroups.com> References: <47349895-ea31-4f85-aa57-4a637c93e0a4@googlegroups.com> Message-ID: <8A695AF5-3B25-4273-A7FF-D77A62CD490A@yahoo.com> On Mar 4, 2015, at 7:51, Neil Girdhar wrote: > For ~, I suggest either having an extra bit on the object that remembers negation Then what exactly happens when you & or | two values, one of which has the negation bit set? For example, using stat values, what is S_IRUSR & ~S_IXUSR? Or, worse, with |. You can end up with some bits set, some negatively set, and some set neither way. How are you going to represent that? In C, because the values are stored as unsigned fixed-size (say, 16-bit) ints and the ~ is just the usual 1's complement operator on unsigned int types, ~S_IXUSR is all the bits except 0o400 (that is 0o1377). There are multiple ways to solve this: * Use signed 1's complement (as I suggested in my original message, which Serhily ignored and suggested the same thing), but then the value of ~S_IXUSR is -0o401, which isn't likely to match == to a value you got from C (or to be obvious to someone using/debugging the code). * Scan the values and assume the highest bit seen is the max bit (as two other people suggested) and manually unsigned-1's-comp, but this is not only more complicated, it only provides the same values as C if you've used all 8/16/32/64 bits or you explicitly mask off & ALL_BITS at the end. * Require specifying the max bit somewhere in the definition, maybe defaulting to 32, or maybe using C++ enum rules (round the highest value up to 8, 16, 32, or 64 bits). * Store two separate ints, one for negated and one for non-negated. * Don't allow ~ at all (as you suggest at the end of the paragraph), using methods instead (although you only gave mutating methods, which is going to make a lot of C 1-liners turn into verbose 3-liners in Python). And if you're wondering why I keep harping on having the same value as C: if you don't care about that, you don't need to use an int as a bitset in the first place; the only reason this proposal is useful in the first place is that functions like stat return ints that have well-known meanings from C, and everyone knows how to manipulate them in C, and we want to be able to do the same thing in Python but with readable reprs and all the other benefits of a real enum type. > or replacing the patterns a &= ~b with a.clear_flags(b) and a & ~b with a.without_flags(b) or something like that. > > On Tuesday, March 3, 2015 at 10:54:15 AM UTC-5, Serhiy Storchaka wrote: >> >> Enum and IntEnum classes allow constants to have nice str() and repr() >> representations. >> >> >>> socket.AF_INET >> >> >>> socket.socket() >> > type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)> >> >> But when integer constants are flags that should be ORed, IntEnum >> doesn't help, because the result of bitwise OR of two IntEnum instances >> is int, and this value can't be represented as IntEnum. >> >> We need new type IntFlags. It is like IntEnum, but has differences: >> >> 1. The value of an instance should be not limited to the set of >> predefined constants. It can be a combination of predefined constants or >> even arbitrary integer. >> >> 2. The result of "|", "&" and "~" operators for IntFlags arguments >> should be an instance of the same IntFlags subclass. >> >> 3. It should have nice str() and repr(). >> >> >>> print(stat.S_IROTH | stat.S_IWOTH) >> stat.S_IROTH|stat.S_IWOTH >> >>> stat.S_IROTH | stat.S_IWOTH >> >> >> Any thoughts? >> >> _______________________________________________ >> Python-ideas mailing list >> Python... at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From abarnert at yahoo.com Thu Mar 5 00:19:00 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Wed, 4 Mar 2015 15:19:00 -0800 Subject: [Python-ideas] IntFlags In-Reply-To: <1425498770.1920134.235569725.07B56128@webmail.messagingengine.com> References: <9509685D-34AA-4D65-9AB0-2599E6E10D88@yahoo.com> <1425498770.1920134.235569725.07B56128@webmail.messagingengine.com> Message-ID: On Mar 4, 2015, at 11:52, random832 at fastmail.us wrote: > On Wed, Mar 4, 2015, at 10:17, Andrew Barnert wrote: >> One of the big questions that (IIRC) derailed this last time and got it >> dropped from the enum stdlib design was: what does ~ do? Does it give you >> the 2's complement negative integer? What does that display as in the str >> and repr? And, if you add in conversion from an IntFlags to/from a set of >> separate values, as has been suggested again in this thread, how does >> that work? All of this is trivial when you're dealing with C fixed-size >> unsigned ints: ~READ means 15 of the 16 bits (all except the READ bit) >> are set. > > str should be ~(READ) obviously. And more generally > ~(all|bits|that|are|not|set). Think about how that extends to the result of |. Of course the answer depends on how you store ~ in the first place, but for most choices, str is not obvious. For example, using fixed-size unsigned with automatic highest-bit detection, for an enum with READ, WRITE, EXEC, STICKY, ~(READ) is the same value as (WRITE|EXEC|STICKY), so how does str know which to print? The one with the fewest flags? Some other rule? (The signed 1's comp choice actually has a reasonable answer here, it just means that you get silly results for silly values, which is fine...) >> Another issue that came up was that C flags often have "combined" names >> that are ambiguous: RDWR = RDONLY | WRONLY), > > Nope. You've actually got a different, much worse, issue: traditionally, > RDONLY is 0 (and should be printed if and only if 1 and 2 are not set), > WRONLY is 1, RDWR is 2, and 3 is traditionally invalid and may have a > platform-dependent meaning. Yeah, open flags are especially screwy, where flags & 3 has a special non-bitmapped meaning but the rest of the bits are flags. There are other cases where multiple ints, only some of which are bitmaps, are packed as separate bitfields into the same int (e.g., TCP/IP headers), but I don't know of any others that pretend to be a single bitmap even though they aren't, so that's really a unique problem, which can be ignored. Just think about stat results or mmap prot flags or anything else where read and write are separate bits. (PS, IIRC, Linux treats 3 as "open for fstat only", and you have that access on most files even if you can't do anything else to them, which was an accident left in place because lilo used it, which made lilo a pain to port to FreeBSD as part of a Linux repair kit...) > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ From abarnert at yahoo.com Thu Mar 5 00:42:03 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Wed, 4 Mar 2015 15:42:03 -0800 Subject: [Python-ideas] IntFlags In-Reply-To: References: <9509685D-34AA-4D65-9AB0-2599E6E10D88@yahoo.com> Message-ID: On Mar 4, 2015, at 7:58, Serhiy Storchaka wrote: > On 04.03.15 17:31, Chris Angelico wrote: >> On Thu, Mar 5, 2015 at 2:17 AM, Andrew Barnert >> wrote: >>> Another issue that came up was that C flags often have "combined" names that are ambiguous: RDWR = RDONLY | WRONLY), which is fine until you want a repr (in C, it's just going to print 3); does it have to be smart enough to show RDWR? (Or, worse, RDWR | CLOEXEC.) >> That could probably be handled by going through the flags in iteration >> order. If the flag is present, emit it and move on. Something like >> this: > > Yes, something like this, but with iterating flags in descended sorted order, and with special case for negative value. I think the very fact that the two of you immediately knew which order was obvious, but you chose the opposite one as the obvious one, proves that it's not obvious. For the record, I think your version is better, because usually the C definitions define the combined values after the individual ones, and it would be nice to be able to mirror the C definitions (or, even better, auto generate the Python from the C header*) and get the desired results. I think you're also right that using signed 1's complement is the best way to handle negated flags, despite the tradeoffs (not being able to compare to negated C values, having a confusing numerical value in the repr, having silly str for silly cases), especially since that's what ~ already does with IntEnum (except that the result is just plain int, of course). But regardless, the point is that these questions don't have a single obvious answer; you have to think about them, decide what makes sense, and explain the tradeoffs and why one should win (and implement it). That's why this "simple" proposal didn't make it into 3.4's enum: because it's not actually simple, and everyone who insisted that it was disagreed, and someone (Guido or Eli?) finally told everyone to shut up, flags weren't going into 3.4, and put your competing implementations on PyPI and see which one people use. Do we really want to rehash all those arguments from scratch? --- * The parenthetical brings up another issue: if you look in your platform's sys/stat.h (or whichever header actually defines these things), S_IRWXG is probably not defined as 0o700, but as S_IRGRP | S_IWGRP | S_IXGRP. Can we do that in the enum definition? If not, it may be less readable than the C. (In fact, on many platforms, S_IRGRP is itself defined as something like _S_IREAD << _S_GRP, and S_IRWXG may be defined as (_S_IREAD | _S_IWRITE | _S_IEXEC) << _S_GRP. I think Linux took this even further and defined it as S_IRWXU >> (_S_USR - _S_GRP) or something silly. But at that point, not being able to clone the C no longer looks like a loss of readability...) From rosuav at gmail.com Thu Mar 5 00:56:57 2015 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 5 Mar 2015 10:56:57 +1100 Subject: [Python-ideas] IntFlags In-Reply-To: References: <9509685D-34AA-4D65-9AB0-2599E6E10D88@yahoo.com> Message-ID: On Thu, Mar 5, 2015 at 10:42 AM, Andrew Barnert wrote: > On Mar 4, 2015, at 7:58, Serhiy Storchaka wrote: > >> On 04.03.15 17:31, Chris Angelico wrote: >>> On Thu, Mar 5, 2015 at 2:17 AM, Andrew Barnert >>> wrote: >>>> Another issue that came up was that C flags often have "combined" names that are ambiguous: RDWR = RDONLY | WRONLY), which is fine until you want a repr (in C, it's just going to print 3); does it have to be smart enough to show RDWR? (Or, worse, RDWR | CLOEXEC.) >>> That could probably be handled by going through the flags in iteration >>> order. If the flag is present, emit it and move on. Something like >>> this: >> >> Yes, something like this, but with iterating flags in descended sorted order, and with special case for negative value. > > I think the very fact that the two of you immediately knew which order was obvious, but you chose the opposite one as the obvious one, proves that it's not obvious. Actually, we chose the same thing, only in slightly different ways. Serhiy suggested (in effect) sorting the flags by value and stepping through from highest to lowest, which enforces that the combined flags will be the ones picked. I suggested putting the responsibility onto the class author - if you want the combined ones to be used, place them first - which is like how aliasing works (the first one with a given value is used in str/repr, any others are aliases). That's a relatively minor point, and it depends on whether there'd ever be a time when you want to provide a combined flag that _isn't_ used in str/repr; if there is, you need my plan, but if not, go with the simpler route. ChrisA From rob.cliffe at btinternet.com Thu Mar 5 01:18:27 2015 From: rob.cliffe at btinternet.com (Rob Cliffe) Date: Thu, 05 Mar 2015 00:18:27 +0000 Subject: [Python-ideas] IntFlags In-Reply-To: References: <9509685D-34AA-4D65-9AB0-2599E6E10D88@yahoo.com> Message-ID: <54F7A0D3.1090001@btinternet.com> > It may require some sanity check for users only > defining "combination constants" in the class such that the result > could not be represented. > > class Stupid(IntFlags): > CAT = 3 > DOG = 6 > > in which case ~CAT would not make sense, but neither could CAT | DOG > be represented. I suppose in such cases an error should be raised by > the metaclass on class definition. > > -Alexander Personally, I find that CAT & DOG is almost invariably an error. :-) From abarnert at yahoo.com Thu Mar 5 10:25:53 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Thu, 5 Mar 2015 01:25:53 -0800 Subject: [Python-ideas] IntFlags In-Reply-To: References: <9509685D-34AA-4D65-9AB0-2599E6E10D88@yahoo.com> Message-ID: On Mar 4, 2015, at 15:56, Chris Angelico wrote: > On Thu, Mar 5, 2015 at 10:42 AM, Andrew Barnert > wrote: >> On Mar 4, 2015, at 7:58, Serhiy Storchaka wrote: >> >>> On 04.03.15 17:31, Chris Angelico wrote: >>>> On Thu, Mar 5, 2015 at 2:17 AM, Andrew Barnert >>>> wrote: >>>>> Another issue that came up was that C flags often have "combined" names that are ambiguous: RDWR = RDONLY | WRONLY), which is fine until you want a repr (in C, it's just going to print 3); does it have to be smart enough to show RDWR? (Or, worse, RDWR | CLOEXEC.) >>>> That could probably be handled by going through the flags in iteration >>>> order. If the flag is present, emit it and move on. Something like >>>> this: >>> >>> Yes, something like this, but with iterating flags in descended sorted order, and with special case for negative value. >> >> I think the very fact that the two of you immediately knew which order was obvious, but you chose the opposite one as the obvious one, proves that it's not obvious. > > Actually, we chose the same thing, only in slightly different ways. > Serhiy suggested (in effect) sorting the flags by value and stepping > through from highest to lowest, which enforces that the combined flags > will be the ones picked. I suggested putting the responsibility onto > the class author - if you want the combined ones to be used, place > them first - which is like how aliasing works (the first one with a > given value is used in str/repr, any others are aliases). That's a > relatively minor point, and it depends on whether there'd ever be a > time when you want to provide a combined flag that _isn't_ used in > str/repr; if there is, you need my plan, but if not, go with the > simpler route. Except with yours, any time you copy the definition (whether by hand, or with an automated tool) from C, where the combined ones almost always come last, they won't be used; with his, they will. So, in the most common use case, you'll get the opposite result. (Of course if you always pick the _last_ instead of the first or the highest, which gives you the choice for the rare case, but the nice answer for the ubiquitous case. That breaks your analogy with aliases, but it does work the same way as normal class attributes, where the last value wins...) From mistersheik at gmail.com Thu Mar 5 10:58:53 2015 From: mistersheik at gmail.com (Neil Girdhar) Date: Thu, 5 Mar 2015 04:58:53 -0500 Subject: [Python-ideas] IntFlags In-Reply-To: References: <9509685D-34AA-4D65-9AB0-2599E6E10D88@yahoo.com> Message-ID: Can I propose that instead of IntFlags, we have IntFields? class Permissions(IntFields): # Some Boolean flags (the bit index) owner_read = 0 owner_write = 1 owner_exec = 2 group_read = 3 group_write = 4 group_exec = 5 user_read = 6 user_write = 7 user_exec = 8 # Some named fields (can overlap) owner_flags = range(3) group_flags = range(3, 6) user_flags = range(6, 9) # Some field values (can overlap with flags and each other) regular_file = range(9), 0o755 character_file = range(9), 0o664 directory = range(9), 0o600 And then you can do the following operations: p = Permissions(Permissions.character_file) p.set(Permissions.regular_file) p.owner_write = True print(p.user_read) p.owner_flags = 6 Another example: class IEEE754(IntFields): fraction = range(23) exponent = range(23, 23+8) sign = 31 denormalized = 'exponent', 0 infinity = 'exponent', 255 In case it's not clear, there are two kinds of members: named fields and field values. Named fields are specified with a range instance or integer represent the bit range or index. Field values are specified with a pair, the first member of which is either a range, an integer, or a string (which would refer to a declared range); and a second member, which is the value. IntFields.set accepts a field value name, and sets the bits according to the value. IntFields.__setattr__ accepts a named field name; it sets the corresponding bits according to the value. IntFields.__getattr__ accepts a named field name or field value name There is no ~ or & operator. Something should probably be done for __or__. Best, Neil On Thu, Mar 5, 2015 at 4:25 AM, 'Andrew Barnert' via python-ideas < python-ideas at googlegroups.com> wrote: > On Mar 4, 2015, at 15:56, Chris Angelico wrote: > > > On Thu, Mar 5, 2015 at 10:42 AM, Andrew Barnert > > wrote: > >> On Mar 4, 2015, at 7:58, Serhiy Storchaka wrote: > >> > >>> On 04.03.15 17:31, Chris Angelico wrote: > >>>> On Thu, Mar 5, 2015 at 2:17 AM, Andrew Barnert > >>>> wrote: > >>>>> Another issue that came up was that C flags often have "combined" > names that are ambiguous: RDWR = RDONLY | WRONLY), which is fine until you > want a repr (in C, it's just going to print 3); does it have to be smart > enough to show RDWR? (Or, worse, RDWR | CLOEXEC.) > >>>> That could probably be handled by going through the flags in iteration > >>>> order. If the flag is present, emit it and move on. Something like > >>>> this: > >>> > >>> Yes, something like this, but with iterating flags in descended sorted > order, and with special case for negative value. > >> > >> I think the very fact that the two of you immediately knew which order > was obvious, but you chose the opposite one as the obvious one, proves that > it's not obvious. > > > > Actually, we chose the same thing, only in slightly different ways. > > Serhiy suggested (in effect) sorting the flags by value and stepping > > through from highest to lowest, which enforces that the combined flags > > will be the ones picked. I suggested putting the responsibility onto > > the class author - if you want the combined ones to be used, place > > them first - which is like how aliasing works (the first one with a > > given value is used in str/repr, any others are aliases). That's a > > relatively minor point, and it depends on whether there'd ever be a > > time when you want to provide a combined flag that _isn't_ used in > > str/repr; if there is, you need my plan, but if not, go with the > > simpler route. > > Except with yours, any time you copy the definition (whether by hand, or > with an automated tool) from C, where the combined ones almost always come > last, they won't be used; with his, they will. So, in the most common use > case, you'll get the opposite result. > > (Of course if you always pick the _last_ instead of the first or the > highest, which gives you the choice for the rare case, but the nice answer > for the ubiquitous case. That breaks your analogy with aliases, but it does > work the same way as normal class attributes, where the last value wins...) > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > > -- > > --- > You received this message because you are subscribed to a topic in the > Google Groups "python-ideas" group. > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/python-ideas/L5KfCEXFaII/unsubscribe. > To unsubscribe from this group and all its topics, send an email to > python-ideas+unsubscribe at googlegroups.com. > For more options, visit https://groups.google.com/d/optout. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mistersheik at gmail.com Thu Mar 5 11:04:14 2015 From: mistersheik at gmail.com (Neil Girdhar) Date: Thu, 5 Mar 2015 05:04:14 -0500 Subject: [Python-ideas] IntFlags In-Reply-To: <8A695AF5-3B25-4273-A7FF-D77A62CD490A@yahoo.com> References: <47349895-ea31-4f85-aa57-4a637c93e0a4@googlegroups.com> <8A695AF5-3B25-4273-A7FF-D77A62CD490A@yahoo.com> Message-ID: This is exactly why we should not have & and ~. It creates the possibility to write meaningless code. Have methods that clear fields and methods that set them. I proposed an interface to IntFields in another message. Best, Neil On Wed, Mar 4, 2015 at 5:59 PM, Andrew Barnert wrote: > On Mar 4, 2015, at 7:51, Neil Girdhar wrote: > > For ~, I suggest either having an extra bit on the object that remembers > negation > > > Then what exactly happens when you & or | two values, one of which has the > negation bit set? For example, using stat values, what is S_IRUSR & > ~S_IXUSR? Or, worse, with |. You can end up with some bits set, some > negatively set, and some set neither way. How are you going to represent > that? > > In C, because the values are stored as unsigned fixed-size (say, 16-bit) > ints and the ~ is just the usual 1's complement operator on unsigned int > types, ~S_IXUSR is all the bits except 0o400 (that is 0o1377). > > There are multiple ways to solve this: > > * Use signed 1's complement (as I suggested in my original message, which > Serhily ignored and suggested the same thing), but then the value of > ~S_IXUSR is -0o401, which isn't likely to match == to a value you got from > C (or to be obvious to someone using/debugging the code). > > * Scan the values and assume the highest bit seen is the max bit (as two > other people suggested) and manually unsigned-1's-comp, but this is not > only more complicated, it only provides the same values as C if you've used > all 8/16/32/64 bits or you explicitly mask off & ALL_BITS at the end. > > * Require specifying the max bit somewhere in the definition, maybe > defaulting to 32, or maybe using C++ enum rules (round the highest value up > to 8, 16, 32, or 64 bits). > > * Store two separate ints, one for negated and one for non-negated. > > * Don't allow ~ at all (as you suggest at the end of the paragraph), > using methods instead (although you only gave mutating methods, which is > going to make a lot of C 1-liners turn into verbose 3-liners in Python). > (I did give a pair of methods one of which was not mutating) > And if you're wondering why I keep harping on having the same value as C: > if you don't care about that, you don't need to use an int as a bitset in > the first place; the only reason this proposal is useful in the first place > is that functions like stat return ints that have well-known meanings from > C, and everyone knows how to manipulate them in C, and we want to be able > to do the same thing in Python but with readable reprs and all the other > benefits of a real enum type. > > or replacing the patterns a &= ~b with a.clear_flags(b) and a & ~b with > a.without_flags(b) or something like that. > > On Tuesday, March 3, 2015 at 10:54:15 AM UTC-5, Serhiy Storchaka wrote: >> >> Enum and IntEnum classes allow constants to have nice str() and repr() >> representations. >> >> >>> socket.AF_INET >> >> >>> socket.socket() >> > type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)> >> >> But when integer constants are flags that should be ORed, IntEnum >> doesn't help, because the result of bitwise OR of two IntEnum instances >> is int, and this value can't be represented as IntEnum. >> >> We need new type IntFlags. It is like IntEnum, but has differences: >> >> 1. The value of an instance should be not limited to the set of >> predefined constants. It can be a combination of predefined constants or >> even arbitrary integer. >> >> 2. The result of "|", "&" and "~" operators for IntFlags arguments >> should be an instance of the same IntFlags subclass. >> >> 3. It should have nice str() and repr(). >> >> >>> print(stat.S_IROTH | stat.S_IWOTH) >> stat.S_IROTH|stat.S_IWOTH >> >>> stat.S_IROTH | stat.S_IWOTH >> >> >> Any thoughts? >> >> _______________________________________________ >> Python-ideas mailing list >> Python... at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ >> > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mistersheik at gmail.com Thu Mar 5 11:31:05 2015 From: mistersheik at gmail.com (Neil Girdhar) Date: Thu, 5 Mar 2015 02:31:05 -0800 (PST) Subject: [Python-ideas] IntFlags In-Reply-To: References: <9509685D-34AA-4D65-9AB0-2599E6E10D88@yahoo.com> Message-ID: <6761e23a-d640-4b86-af74-0acb82d96c25@googlegroups.com> Actually, it would be cool to support nesting too, e.g., class SubPermissions(IntFields): read = 0 write = 1 exec_ = 2 class Permissions(IntFields): owner_flags = range(3), SubPermissions group_flags = range(3, 6), SubPermissions user_flags = range(6, 9), SubPermissions regular_file = range(9), 0o755 character_file = range(9), 0o664 directory = range(9), 0o600 -------------- next part -------------- An HTML attachment was scrubbed... URL: From storchaka at gmail.com Thu Mar 5 16:32:23 2015 From: storchaka at gmail.com (Serhiy Storchaka) Date: Thu, 05 Mar 2015 17:32:23 +0200 Subject: [Python-ideas] IntFlags In-Reply-To: References: Message-ID: Opened an issue: http://bugs.python.org/issue23591. From random832 at fastmail.us Thu Mar 5 16:58:44 2015 From: random832 at fastmail.us (random832 at fastmail.us) Date: Thu, 05 Mar 2015 10:58:44 -0500 Subject: [Python-ideas] IntFlags In-Reply-To: References: <9509685D-34AA-4D65-9AB0-2599E6E10D88@yahoo.com> <1425498770.1920134.235569725.07B56128@webmail.messagingengine.com> Message-ID: <1425571124.2853338.236000061.7B2455B9@webmail.messagingengine.com> On Wed, Mar 4, 2015, at 18:19, Andrew Barnert wrote: > Think about how that extends to the result of |. The value is either negative (is the complement of a finite set of bits) or it isn't. Remember, the underlying representation is an int. Of course the answer > depends on how you store ~ in the first place, but for most choices, str > is not obvious. For example, using fixed-size unsigned with automatic > highest-bit detection, for an enum with READ, WRITE, EXEC, STICKY, > ~(READ) is the same value as (WRITE|EXEC|STICKY), Why would that be the case? > Yeah, open flags are especially screwy, where flags & 3 has a special > non-bitmapped meaning but the rest of the bits are flags. There are other > cases where multiple ints, only some of which are bitmaps, are packed as > separate bitfields into the same int (e.g., TCP/IP headers), but I don't > know of any others that pretend to be a single bitmap even though they > aren't, so that's really a unique problem, which can be ignored. Technically file modes have the file type field (flags & 0xF000) which has some states that don't mean anything on common platforms. From luciano at ramalho.org Thu Mar 5 17:15:58 2015 From: luciano at ramalho.org (Luciano Ramalho) Date: Thu, 5 Mar 2015 13:15:58 -0300 Subject: [Python-ideas] IntFlags In-Reply-To: References: Message-ID: I like the IntFlags concept, as long as it's clearly separated from IntEnum -- it seems that's the consensus now. I don't like the name IntFlags: BitFlags makes more sense to me, since the key feature is supporting bitwise operators. Calling it BitFlags has the additional advantage of making it very clear that it's not closely related to IntEnum. Best, Luciano On Tue, Mar 3, 2015 at 12:52 PM, Serhiy Storchaka wrote: > Enum and IntEnum classes allow constants to have nice str() and repr() > representations. > >>>> socket.AF_INET > >>>> socket.socket() > type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)> > > But when integer constants are flags that should be ORed, IntEnum doesn't > help, because the result of bitwise OR of two IntEnum instances is int, and > this value can't be represented as IntEnum. > > We need new type IntFlags. It is like IntEnum, but has differences: > > 1. The value of an instance should be not limited to the set of predefined > constants. It can be a combination of predefined constants or even arbitrary > integer. > > 2. The result of "|", "&" and "~" operators for IntFlags arguments should be > an instance of the same IntFlags subclass. > > 3. It should have nice str() and repr(). > >>>> print(stat.S_IROTH | stat.S_IWOTH) > stat.S_IROTH|stat.S_IWOTH >>>> stat.S_IROTH | stat.S_IWOTH > > > Any thoughts? > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -- Luciano Ramalho Twitter: @ramalhoorg Professor em: http://python.pro.br Twitter: @pythonprobr From eltoder at gmail.com Thu Mar 5 17:58:06 2015 From: eltoder at gmail.com (Eugene Toder) Date: Thu, 5 Mar 2015 11:58:06 -0500 Subject: [Python-ideas] IntFlags In-Reply-To: References: Message-ID: Maybe even call it BitSet, and model the interface based on frozenset, except that every element is also a set of one. The operations will be: X | Y -- union X & Y -- intersection X ^ Y -- symmetric difference X - Y -- difference X in Y == (X & Y) == X len(X) -- number of set bits bool(X) -- any bits set isdisjoint, issubset, issuperset == not X & Y, not X - Y, not Y - X (Note no negation.) Eugene On Thu, Mar 5, 2015 at 11:15 AM, Luciano Ramalho wrote: > I like the IntFlags concept, as long as it's clearly separated from > IntEnum -- it seems that's the consensus now. > > I don't like the name IntFlags: BitFlags makes more sense to me, since > the key feature is supporting bitwise operators. > > Calling it BitFlags has the additional advantage of making it very > clear that it's not closely related to IntEnum. > > Best, > > Luciano > > On Tue, Mar 3, 2015 at 12:52 PM, Serhiy Storchaka > wrote: > > Enum and IntEnum classes allow constants to have nice str() and repr() > > representations. > > > >>>> socket.AF_INET > > > >>>> socket.socket() > > > type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)> > > > > But when integer constants are flags that should be ORed, IntEnum doesn't > > help, because the result of bitwise OR of two IntEnum instances is int, > and > > this value can't be represented as IntEnum. > > > > We need new type IntFlags. It is like IntEnum, but has differences: > > > > 1. The value of an instance should be not limited to the set of > predefined > > constants. It can be a combination of predefined constants or even > arbitrary > > integer. > > > > 2. The result of "|", "&" and "~" operators for IntFlags arguments > should be > > an instance of the same IntFlags subclass. > > > > 3. It should have nice str() and repr(). > > > >>>> print(stat.S_IROTH | stat.S_IWOTH) > > stat.S_IROTH|stat.S_IWOTH > >>>> stat.S_IROTH | stat.S_IWOTH > > > > > > Any thoughts? > > > > _______________________________________________ > > Python-ideas mailing list > > Python-ideas at python.org > > https://mail.python.org/mailman/listinfo/python-ideas > > Code of Conduct: http://python.org/psf/codeofconduct/ > > > > -- > Luciano Ramalho > Twitter: @ramalhoorg > > Professor em: http://python.pro.br > Twitter: @pythonprobr > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From luciano at ramalho.org Thu Mar 5 18:03:35 2015 From: luciano at ramalho.org (Luciano Ramalho) Date: Thu, 5 Mar 2015 14:03:35 -0300 Subject: [Python-ideas] IntFlags In-Reply-To: References: Message-ID: The problem is that a set is by definition unordered, but the position of the bits in BitFlags is crucial, so the name BitSet may not give the right idea. On Thu, Mar 5, 2015 at 1:58 PM, Eugene Toder wrote: > Maybe even call it BitSet, and model the interface based on frozenset, > except that every element is also a set of one. The operations will be: > X | Y -- union > X & Y -- intersection > X ^ Y -- symmetric difference > X - Y -- difference > X in Y == (X & Y) == X > len(X) -- number of set bits > bool(X) -- any bits set > isdisjoint, issubset, issuperset == not X & Y, not X - Y, not Y - X > (Note no negation.) > > Eugene > > On Thu, Mar 5, 2015 at 11:15 AM, Luciano Ramalho > wrote: >> >> I like the IntFlags concept, as long as it's clearly separated from >> IntEnum -- it seems that's the consensus now. >> >> I don't like the name IntFlags: BitFlags makes more sense to me, since >> the key feature is supporting bitwise operators. >> >> Calling it BitFlags has the additional advantage of making it very >> clear that it's not closely related to IntEnum. >> >> Best, >> >> Luciano >> >> On Tue, Mar 3, 2015 at 12:52 PM, Serhiy Storchaka >> wrote: >> > Enum and IntEnum classes allow constants to have nice str() and repr() >> > representations. >> > >> >>>> socket.AF_INET >> > >> >>>> socket.socket() >> > > > type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)> >> > >> > But when integer constants are flags that should be ORed, IntEnum >> > doesn't >> > help, because the result of bitwise OR of two IntEnum instances is int, >> > and >> > this value can't be represented as IntEnum. >> > >> > We need new type IntFlags. It is like IntEnum, but has differences: >> > >> > 1. The value of an instance should be not limited to the set of >> > predefined >> > constants. It can be a combination of predefined constants or even >> > arbitrary >> > integer. >> > >> > 2. The result of "|", "&" and "~" operators for IntFlags arguments >> > should be >> > an instance of the same IntFlags subclass. >> > >> > 3. It should have nice str() and repr(). >> > >> >>>> print(stat.S_IROTH | stat.S_IWOTH) >> > stat.S_IROTH|stat.S_IWOTH >> >>>> stat.S_IROTH | stat.S_IWOTH >> > >> > >> > Any thoughts? >> > >> > _______________________________________________ >> > Python-ideas mailing list >> > Python-ideas at python.org >> > https://mail.python.org/mailman/listinfo/python-ideas >> > Code of Conduct: http://python.org/psf/codeofconduct/ >> >> >> >> -- >> Luciano Ramalho >> Twitter: @ramalhoorg >> >> Professor em: http://python.pro.br >> Twitter: @pythonprobr >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ > > -- Luciano Ramalho Twitter: @ramalhoorg Professor em: http://python.pro.br Twitter: @pythonprobr From storchaka at gmail.com Thu Mar 5 18:11:48 2015 From: storchaka at gmail.com (Serhiy Storchaka) Date: Thu, 05 Mar 2015 19:11:48 +0200 Subject: [Python-ideas] IntFlags In-Reply-To: References: <9509685D-34AA-4D65-9AB0-2599E6E10D88@yahoo.com> Message-ID: On 05.03.15 01:42, Andrew Barnert wrote: > I think you're also right that using signed 1's complement is the best way to handle negated flags, despite the tradeoffs (not being able to compare to negated C values, having a confusing numerical value in the repr, having silly str for silly cases), especially since that's what ~ already does with IntEnum (except that the result is just plain int, of course). Sorry, I don't understand your argument. Why you can't compare complemented IntFlags with complemented int? ~(os.OpenMode.O_CLOEXEC) == int(~os.OpenMode.O_CLOEXEC) == ~int(os.OpenMode.O_CLOEXEC) == posix.O_CLOEXEC IntFlags is an int subclass and behaves as plain int, except that it has special repr and results of bitwise operations preserve a type. > * The parenthetical brings up another issue: if you look in your platform's sys/stat.h (or whichever header actually defines these things), S_IRWXG is probably not defined as 0o700, but as S_IRGRP | S_IWGRP | S_IXGRP. Can we do that in the enum definition? If not, it may be less readable than the C. (In fact, on many platforms, S_IRGRP is itself defined as something like _S_IREAD << _S_GRP, and S_IRWXG may be defined as (_S_IREAD | _S_IWRITE | _S_IEXEC) << _S_GRP. I think Linux took this even further and defined it as S_IRWXU >> (_S_USR - _S_GRP) or something silly. But at that point, not being able to clone the C no longer looks like a loss of readability...) Yes, of course you can define: class Permissions(enum.IntFlags): S_IRGRP = 0o0040 # read by group S_IWGRP = 0o0020 # write by group S_IXGRP = 0o0010 # execute by group S_IRWXG = S_IRGRP | S_IWGRP | S_IXGRP ... or what your like. I tried and this works. From eltoder at gmail.com Thu Mar 5 18:14:09 2015 From: eltoder at gmail.com (Eugene Toder) Date: Thu, 5 Mar 2015 12:14:09 -0500 Subject: [Python-ideas] IntFlags In-Reply-To: References: Message-ID: On Thu, Mar 5, 2015 at 12:03 PM, Luciano Ramalho wrote: > The problem is that a set is by definition unordered, but the position > of the bits in BitFlags is crucial, so the name BitSet may not give > the right idea. The flags discussed so far are unordered as well. E.g. OpenMode.RDONLY | OpenMode.WRONLY | OpenMode.CLOEXEC == OpenMode.RDONLY | OpenMode.CLOEXEC | OpenMode.WRONLY or any other permutation. In other words, flags (aka bitmasks) are just an optimized representation of an (unordered) set of small integer values. Eugene -------------- next part -------------- An HTML attachment was scrubbed... URL: From luciano at ramalho.org Thu Mar 5 18:20:27 2015 From: luciano at ramalho.org (Luciano Ramalho) Date: Thu, 5 Mar 2015 14:20:27 -0300 Subject: [Python-ideas] IntFlags In-Reply-To: References: Message-ID: On Thu, Mar 5, 2015 at 2:14 PM, Eugene Toder wrote: > The flags discussed so far are unordered as well. E.g. OpenMode.RDONLY | OpenMode.WRONLY | OpenMode.CLOEXEC == OpenMode.RDONLY | OpenMode.CLOEXEC | OpenMode.WRONLY or any other permutation. In other words, flags (aka bitmasks) are just an optimized representation of an (unordered) set of small integer values. I agree that they are unordered in the common use cases. But the kind of low-level code where such flags are used often involves building a byte or a word out of those flags to pass to a low-level C API. How do you make sure that the byte you will build makes sense if you don't care about the position, or the log2 of each flag value? So I definitely think ordering and positioning are to be preserved in a data structure intended to manage and combine bits. Best, Luciano -- Luciano Ramalho Twitter: @ramalhoorg Professor em: http://python.pro.br Twitter: @pythonprobr From storchaka at gmail.com Thu Mar 5 18:26:40 2015 From: storchaka at gmail.com (Serhiy Storchaka) Date: Thu, 05 Mar 2015 19:26:40 +0200 Subject: [Python-ideas] IntFlags In-Reply-To: References: Message-ID: On 05.03.15 18:15, Luciano Ramalho wrote: > I don't like the name IntFlags: BitFlags makes more sense to me, since > the key feature is supporting bitwise operators. I chose this name because the concept IntFlags is very similar to the concept of Flags enums in C#. The Flags decorator in C# is as close to IntFlags as enums in C# close to IntEnum. The Int prefix is here because IntFlags is just an funny int (as IntEnum) and both IntFlags and IntEnum are purposed to replace int constants. > Calling it BitFlags has the additional advantage of making it very > clear that it's not closely related to IntEnum. But it is closely related to IntEnum. Both are int subclass and fully compatible with ints, both provides named constants, both have funny str and repr, both inherit common useful interface from Enum, both are purposed to replace integer constants. From mistersheik at gmail.com Thu Mar 5 18:29:22 2015 From: mistersheik at gmail.com (Neil Girdhar) Date: Thu, 5 Mar 2015 12:29:22 -0500 Subject: [Python-ideas] IntFlags In-Reply-To: References: Message-ID: Have you looked at my IntFields generalization of IntFlags? It seems that many of your examples (permissions, e.g.) are better expressed with fields than with flags. On Thu, Mar 5, 2015 at 12:26 PM, Serhiy Storchaka wrote: > On 05.03.15 18:15, Luciano Ramalho wrote: > >> I don't like the name IntFlags: BitFlags makes more sense to me, since >> the key feature is supporting bitwise operators. >> > > I chose this name because the concept IntFlags is very similar to the > concept of Flags enums in C#. The Flags decorator in C# is as close to > IntFlags as enums in C# close to IntEnum. The Int prefix is here because > IntFlags is just an funny int (as IntEnum) and both IntFlags and IntEnum > are purposed to replace int constants. > > Calling it BitFlags has the additional advantage of making it very >> clear that it's not closely related to IntEnum. >> > > But it is closely related to IntEnum. Both are int subclass and fully > compatible with ints, both provides named constants, both have funny str > and repr, both inherit common useful interface from Enum, both are purposed > to replace integer constants. > > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > > -- > > --- You received this message because you are subscribed to a topic in the > Google Groups "python-ideas" group. > To unsubscribe from this topic, visit https://groups.google.com/d/ > topic/python-ideas/L5KfCEXFaII/unsubscribe. > To unsubscribe from this group and all its topics, send an email to > python-ideas+unsubscribe at googlegroups.com. > For more options, visit https://groups.google.com/d/optout. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From eltoder at gmail.com Thu Mar 5 18:42:08 2015 From: eltoder at gmail.com (Eugene Toder) Date: Thu, 5 Mar 2015 12:42:08 -0500 Subject: [Python-ideas] IntFlags In-Reply-To: References: Message-ID: On Thu, Mar 5, 2015 at 12:20 PM, Luciano Ramalho wrote: > On Thu, Mar 5, 2015 at 2:14 PM, Eugene Toder wrote: > > The flags discussed so far are unordered as well. E.g. OpenMode.RDONLY | > OpenMode.WRONLY | OpenMode.CLOEXEC == OpenMode.RDONLY | OpenMode.CLOEXEC | > OpenMode.WRONLY or any other permutation. In other words, flags (aka > bitmasks) are just an optimized representation of an (unordered) set of > small integer values. > > I agree that they are unordered in the common use cases. But the kind > of low-level code where such flags are used often involves building a > byte or a word out of those flags to pass to a low-level C API. How do > you make sure that the byte you will build makes sense if you don't > care about the position, or the log2 of each flag value? So I > definitely think ordering and positioning are to be preserved in a > data structure intended to manage and combine bits. > I think you conflate the two separate issues: ordering of the elements in the set, and the bit representation of the set. Flags don't preserve the former, but guarantee the later. Normal sets don't preserve the former too, and the latter doesn't make sense for them. BitSet would not preserve the former, but add the later in the way compatible with flags. Specifically, when you define a BitSet, you assign every possible element a specific bit position (i.e. integer value). When you perform bit operations the natural thing happens, and you get the expected bit representation. Eugene -------------- next part -------------- An HTML attachment was scrubbed... URL: From storchaka at gmail.com Thu Mar 5 18:57:58 2015 From: storchaka at gmail.com (Serhiy Storchaka) Date: Thu, 05 Mar 2015 19:57:58 +0200 Subject: [Python-ideas] IntFlags In-Reply-To: References: Message-ID: On 05.03.15 19:29, Neil Girdhar wrote: > Have you looked at my IntFields generalization of IntFlags? It seems > that many of your examples (permissions, e.g.) are better expressed with > fields than with flags. It looks too complicated for such simple case. And it has an interface incompatible with plain int. From chris.barker at noaa.gov Thu Mar 5 18:11:29 2015 From: chris.barker at noaa.gov (Chris Barker) Date: Thu, 5 Mar 2015 09:11:29 -0800 Subject: [Python-ideas] IntFlags In-Reply-To: References: Message-ID: On Thu, Mar 5, 2015 at 9:03 AM, Luciano Ramalho wrote: > The problem is that a set is by definition unordered, but the position > of the bits in BitFlags is crucial, Is it? or is that an implementation detail? What I'm getting at is that if you use an integer to store bits, then the nth bit is, well, the nth bit in the value. But you could conceptually think of it as the bit with the name, 'n', in which case order no longer matters. -Chris > so the name BitSet may not give > the right idea. > > On Thu, Mar 5, 2015 at 1:58 PM, Eugene Toder wrote: > > Maybe even call it BitSet, and model the interface based on frozenset, > > except that every element is also a set of one. The operations will be: > > X | Y -- union > > X & Y -- intersection > > X ^ Y -- symmetric difference > > X - Y -- difference > > X in Y == (X & Y) == X > > len(X) -- number of set bits > > bool(X) -- any bits set > > isdisjoint, issubset, issuperset == not X & Y, not X - Y, not Y - X > > (Note no negation.) > > > > Eugene > > > > On Thu, Mar 5, 2015 at 11:15 AM, Luciano Ramalho > > wrote: > >> > >> I like the IntFlags concept, as long as it's clearly separated from > >> IntEnum -- it seems that's the consensus now. > >> > >> I don't like the name IntFlags: BitFlags makes more sense to me, since > >> the key feature is supporting bitwise operators. > >> > >> Calling it BitFlags has the additional advantage of making it very > >> clear that it's not closely related to IntEnum. > >> > >> Best, > >> > >> Luciano > >> > >> On Tue, Mar 3, 2015 at 12:52 PM, Serhiy Storchaka > >> wrote: > >> > Enum and IntEnum classes allow constants to have nice str() and repr() > >> > representations. > >> > > >> >>>> socket.AF_INET > >> > > >> >>>> socket.socket() > >> > >> > type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)> > >> > > >> > But when integer constants are flags that should be ORed, IntEnum > >> > doesn't > >> > help, because the result of bitwise OR of two IntEnum instances is > int, > >> > and > >> > this value can't be represented as IntEnum. > >> > > >> > We need new type IntFlags. It is like IntEnum, but has differences: > >> > > >> > 1. The value of an instance should be not limited to the set of > >> > predefined > >> > constants. It can be a combination of predefined constants or even > >> > arbitrary > >> > integer. > >> > > >> > 2. The result of "|", "&" and "~" operators for IntFlags arguments > >> > should be > >> > an instance of the same IntFlags subclass. > >> > > >> > 3. It should have nice str() and repr(). > >> > > >> >>>> print(stat.S_IROTH | stat.S_IWOTH) > >> > stat.S_IROTH|stat.S_IWOTH > >> >>>> stat.S_IROTH | stat.S_IWOTH > >> > > >> > > >> > Any thoughts? > >> > > >> > _______________________________________________ > >> > Python-ideas mailing list > >> > Python-ideas at python.org > >> > https://mail.python.org/mailman/listinfo/python-ideas > >> > Code of Conduct: http://python.org/psf/codeofconduct/ > >> > >> > >> > >> -- > >> Luciano Ramalho > >> Twitter: @ramalhoorg > >> > >> Professor em: http://python.pro.br > >> Twitter: @pythonprobr > >> _______________________________________________ > >> Python-ideas mailing list > >> Python-ideas at python.org > >> https://mail.python.org/mailman/listinfo/python-ideas > >> Code of Conduct: http://python.org/psf/codeofconduct/ > > > > > > > > -- > Luciano Ramalho > Twitter: @ramalhoorg > > Professor em: http://python.pro.br > Twitter: @pythonprobr > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From greg.ewing at canterbury.ac.nz Thu Mar 5 21:30:39 2015 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Fri, 06 Mar 2015 09:30:39 +1300 Subject: [Python-ideas] IntFlags In-Reply-To: <1425571124.2853338.236000061.7B2455B9@webmail.messagingengine.com> References: <9509685D-34AA-4D65-9AB0-2599E6E10D88@yahoo.com> <1425498770.1920134.235569725.07B56128@webmail.messagingengine.com> <1425571124.2853338.236000061.7B2455B9@webmail.messagingengine.com> Message-ID: <54F8BCEF.1070806@canterbury.ac.nz> Concerning ~, if you think of it as a set, then ~x should be the result of exlusive-oring x with an int containing all the valid bits, i.e. the value obtained by oring all the defined values together. This is different from what you would get in C, but it's self-consistent and the repr() would make sense. -- Greg From rosuav at gmail.com Thu Mar 5 22:22:50 2015 From: rosuav at gmail.com (Chris Angelico) Date: Fri, 6 Mar 2015 08:22:50 +1100 Subject: [Python-ideas] IntFlags In-Reply-To: References: Message-ID: On Fri, Mar 6, 2015 at 4:11 AM, Chris Barker wrote: > On Thu, Mar 5, 2015 at 9:03 AM, Luciano Ramalho wrote: >> >> The problem is that a set is by definition unordered, but the position >> of the bits in BitFlags is crucial, > > > Is it? or is that an implementation detail? > > What I'm getting at is that if you use an integer to store bits, then the > nth bit is, well, the nth bit in the value. But you could conceptually think > of it as the bit with the name, 'n', in which case order no longer matters. The position of bits is the values of things in the set. These two are more-or-less expressing the same concept: flags1 = {"ReadOnly", "AllowRead", "CloseOnExec"} flags2 = FLG_READONLY | FLG_ALLOWREAD | FLG_CLOEXEC The order of elements in the braced set is meaningless, just as the order of bitflags in the piped list is meaningless. The positions of bits in the piped set corresponds to the string names in the braced set. As the other Chris says, it's basically the bit with name "n", and then we give it a much more convenient alias. ChrisA From eltoder at gmail.com Thu Mar 5 23:07:04 2015 From: eltoder at gmail.com (Eugene Toder) Date: Thu, 5 Mar 2015 17:07:04 -0500 Subject: [Python-ideas] IntFlags In-Reply-To: <54F8BCEF.1070806@canterbury.ac.nz> References: <9509685D-34AA-4D65-9AB0-2599E6E10D88@yahoo.com> <1425498770.1920134.235569725.07B56128@webmail.messagingengine.com> <1425571124.2853338.236000061.7B2455B9@webmail.messagingengine.com> <54F8BCEF.1070806@canterbury.ac.nz> Message-ID: On Thu, Mar 5, 2015 at 3:30 PM, Greg Ewing wrote: > Concerning ~, if you think of it as a set, then ~x should > be the result of exlusive-oring x with an int containing > all the valid bits, i.e. the value obtained by oring all > the defined values together. > Yes, that will work. Then we might want predefined constants for all set bits and zero. (Zero can be just calling MyBitSet() with no arguments). Then ~X is type(X).ALL_ONES - X. The main question is whether this operation makes much sense. I think in practice ~ is only used with flags to clear bits (X & ~Y), which can be done in one explicit operation. It may still be worth supporting ~ to allow painless transition from plain ints. Eugene -------------- next part -------------- An HTML attachment was scrubbed... URL: From cs at zip.com.au Thu Mar 5 23:26:18 2015 From: cs at zip.com.au (Cameron Simpson) Date: Fri, 6 Mar 2015 09:26:18 +1100 Subject: [Python-ideas] IntFlags In-Reply-To: References: Message-ID: <20150305222618.GA92958@cskk.homeip.net> On 05Mar2015 05:04, Neil Girdhar wrote: >This is exactly why we should not have & and ~. It creates the possibility >to write meaningless code. Have methods that clear fields and methods that >set them. I proposed an interface to IntFields in another message. I can see the argument against ~, unless you preagree a 1's complelment bit range (or equivalent, whatever): feasible, given that you might refuse to accept "unknown" flags. With a preagreed range, "~" is perfectly meaningful. I do not see _any_ argument again "&". I would be very unhappy with an IntFlags that didn't accept "&" and "|", and immediately subclass it and never use the raw one again. I'm not sure I am a fan of "remembering negation". Keep it simple. Forbid (or document as undefined) ~ in the absense of a range agreement at setup time. Cheers, Cameron Simpson Favourite proverb: Ein mann der motorrad fahrt ist fuer immer jung. From cs at zip.com.au Thu Mar 5 23:34:03 2015 From: cs at zip.com.au (Cameron Simpson) Date: Fri, 6 Mar 2015 09:34:03 +1100 Subject: [Python-ideas] IntFlags In-Reply-To: References: Message-ID: <20150305223403.GA7148@cskk.homeip.net> On 03Mar2015 17:52, Serhiy Storchaka wrote: >Enum and IntEnum classes allow constants to have nice str() and repr() >representations. [...] >3. It should have nice str() and repr(). >>>> print(stat.S_IROTH | stat.S_IWOTH) >stat.S_IROTH|stat.S_IWOTH >>>> stat.S_IROTH | stat.S_IWOTH > Regarding this one, and regarding some of the later discussion about "meaningless" values from some operations, when I wrote my _Clock_Flags I made the repr() pull off known flags and include the remainer if not zero. So your example above would come out: and: if there was an extra unknown "32" bit in the mix. On reflection, I'd like these to come out: (with space or "." or whatever). Cheers, Cameron Simpson Isaac Asimov once remarked that friends had chided him for not patenting the electronic pocket calculator, since he wrote of similar devices back in the 1940's. His reply, "Have you ever noticed I only described what it looked like on the *outside*?" - ijl at mediaone.net From ckaynor at zindagigames.com Thu Mar 5 23:34:54 2015 From: ckaynor at zindagigames.com (Chris Kaynor) Date: Thu, 5 Mar 2015 14:34:54 -0800 Subject: [Python-ideas] IntFlags In-Reply-To: References: <9509685D-34AA-4D65-9AB0-2599E6E10D88@yahoo.com> <1425498770.1920134.235569725.07B56128@webmail.messagingengine.com> <1425571124.2853338.236000061.7B2455B9@webmail.messagingengine.com> <54F8BCEF.1070806@canterbury.ac.nz> Message-ID: On Thu, Mar 5, 2015 at 2:07 PM, Eugene Toder wrote: > On Thu, Mar 5, 2015 at 3:30 PM, Greg Ewing > wrote: >> >> Concerning ~, if you think of it as a set, then ~x should >> be the result of exlusive-oring x with an int containing >> all the valid bits, i.e. the value obtained by oring all >> the defined values together. > > Yes, that will work. Then we might want predefined constants for all set > bits and zero. (Zero can be just calling MyBitSet() with no arguments). Then > ~X is type(X).ALL_ONES - X. > The main question is whether this operation makes much sense. I think in > practice ~ is only used with flags to clear bits (X & ~Y), which can be done > in one explicit operation. It may still be worth supporting ~ to allow > painless transition from plain ints. I wonder if it would be worth it to have an ALL value supported, which defaults to the other values ored together, but could be overridden for specific cases. This could be useful for compatibility with C code where there are not publicly defined values for some bits, but they have some meaning (possibly only for compatibility). If it were purely computed automatically, handling of cases where there are unused bits could get messy in some cases. Even if methods are implemented for clearing flags (the primary case for bit-wise not on a bit field), supporting a bit-wise not would still be extremely useful for compatibility with other languages. In terms of printing, I would expect that, for ease of reading, a negated flags would print with the negation (storing it as one-complement), however every other operation would collapse the negation into the full int value, losing the printing value. Something like: >>> READ = 1; WRITE = 2; EXECUTE = 4 # Obviously, different syntax. This would default ALL = 4 | 2 | 1, however it could be overridden in the special cases where other values have meaning. I would imagine the most common override would be to 0xFFFFFFFF, though I could also see others. >>> a = READ | EXECUTE >>> a (READ | EXECUTE) >>> ~a # Stores it as a negation of the values. ~(READ | EXECUTE) >>> b = EXECUTE | WRITE >>> b # Reordered based on values. (WRITE | EXECUTE) >>> b & ~a # Collapses the negation into the full bit field, resulting in a non-negated bit field. WRITE >>> ~b | ~a (READ | WRITE) On Thu, Mar 5, 2015 at 1:22 PM, Chris Angelico wrote: > As the other Chris says, it's basically the bit with name "n", > and then we give it a much more convenient alias. Now that I've joined the conversation, there are TWO other Chris's :). From cs at zip.com.au Thu Mar 5 23:42:11 2015 From: cs at zip.com.au (Cameron Simpson) Date: Fri, 6 Mar 2015 09:42:11 +1100 Subject: [Python-ideas] IntFlags In-Reply-To: References: Message-ID: <20150305224211.GA35388@cskk.homeip.net> On 05Mar2015 19:26, Serhiy Storchaka wrote: >On 05.03.15 18:15, Luciano Ramalho wrote: >>I don't like the name IntFlags: BitFlags makes more sense to me, since >>the key feature is supporting bitwise operators. > >I chose this name because the concept IntFlags is very similar to the >concept of Flags enums in C#. The Flags decorator in C# is as close to >IntFlags as enums in C# close to IntEnum. The Int prefix is here >because IntFlags is just an funny int (as IntEnum) and both IntFlags >and IntEnum are purposed to replace int constants. > >>Calling it BitFlags has the additional advantage of making it very >>clear that it's not closely related to IntEnum. > >But it is closely related to IntEnum. Both are int subclass and fully >compatible with ints, both provides named constants, both have funny >str and repr, both inherit common useful interface from Enum, both are >purposed to replace integer constants. I also prefer IntFlags, like IntEnum. I certainly have use cases where I do not care that there is an underlying numeric value, and am treating things like a set. But with fixed names (versus sets, which are open ended in their native form) and using an int underneath is both efficient an natural. But conversely, I have use cases where I do care. I am -1 on BitFlags. Cheers, Cameron Simpson You can be psychotic and still be competent. - John L. Young, American Academy of Psychiatry and the Law on Ted Kaczynski, and probably most internet users From greg.ewing at canterbury.ac.nz Thu Mar 5 21:27:28 2015 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Fri, 06 Mar 2015 09:27:28 +1300 Subject: [Python-ideas] IntFlags In-Reply-To: References: Message-ID: <54F8BC30.6080307@canterbury.ac.nz> Luciano Ramalho wrote: > I don't like the name IntFlags: BitFlags makes more sense to me, since > the key feature is supporting bitwise operators. Or perhaps BitSet, emphasising set-like behaviour. -- Greg From greg.ewing at canterbury.ac.nz Thu Mar 5 21:39:11 2015 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Fri, 06 Mar 2015 09:39:11 +1300 Subject: [Python-ideas] IntFlags In-Reply-To: References: Message-ID: <54F8BEEF.3070607@canterbury.ac.nz> Luciano Ramalho wrote: > The problem is that a set is by definition unordered, but the position > of the bits in BitFlags is crucial, so the name BitSet may not give > the right idea. I think the "Bit" part of BitSet covers that. It's a set that's represented using bits, and bits have a position. Also, the elements of the set are conceptually ints, and ints are ordered. -- Greg From storchaka at gmail.com Fri Mar 6 01:41:18 2015 From: storchaka at gmail.com (Serhiy Storchaka) Date: Fri, 06 Mar 2015 02:41:18 +0200 Subject: [Python-ideas] IntFlags In-Reply-To: <54F8BC30.6080307@canterbury.ac.nz> References: <54F8BC30.6080307@canterbury.ac.nz> Message-ID: On 05.03.15 22:27, Greg Ewing wrote: > Or perhaps BitSet, emphasising set-like behaviour. bitset in C++ and BitSet in Java are fixed-size arrays of booleans. They support such operations as changing a range of bits, but are not compatible with ints and sets. From greg.ewing at canterbury.ac.nz Fri Mar 6 02:36:14 2015 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Fri, 06 Mar 2015 14:36:14 +1300 Subject: [Python-ideas] IntFlags In-Reply-To: References: <54F8BC30.6080307@canterbury.ac.nz> Message-ID: <54F9048E.7050402@canterbury.ac.nz> On 03/06/2015 01:41 PM, Serhiy Storchaka wrote: > > bitset in C++ and BitSet in Java are fixed-size arrays of booleans. They > support such operations as changing a range of bits, but are not compatible > with ints and sets. Java has an EnumSet, which is closer to what we're talking about. It supports set operations and uses a bit vector internally. -- Greg From abarnert at yahoo.com Fri Mar 6 03:13:56 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Thu, 5 Mar 2015 18:13:56 -0800 Subject: [Python-ideas] IntFlags In-Reply-To: References: <9509685D-34AA-4D65-9AB0-2599E6E10D88@yahoo.com> Message-ID: On Mar 5, 2015, at 9:11, Serhiy Storchaka wrote: > On 05.03.15 01:42, Andrew Barnert wrote: >> I think you're also right that using signed 1's complement is the best way to handle negated flags, despite the tradeoffs (not being able to compare to negated C values, having a confusing numerical value in the repr, having silly str for silly cases), especially since that's what ~ already does with IntEnum (except that the result is just plain int, of course). > > Sorry, I don't understand your argument. Why you can't compare complemented IntFlags with complemented int? You can, but what you can't do is compare it with the values that you'll likely get from C, because -0o200001 != 0o37775777777 in Python, whereas they are equal in C. > ~(os.OpenMode.O_CLOEXEC) == int(~os.OpenMode.O_CLOEXEC) == ~int(os.OpenMode.O_CLOEXEC) == posix.O_CLOEXEC I certainly hope that's not true, or ~ is a no-op... But even if you add the missing ~, none of these is going to be == the value of an unsigned int holding ~O_CLOEXEC in C. Again, I already said I think you chose the best tradeoff, but there is a downside compared to treating the values as unsigned or to the other options people have proposed here; that's why it's a tradeoff rather than a one obvious choice. From abarnert at yahoo.com Fri Mar 6 03:25:15 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Thu, 5 Mar 2015 18:25:15 -0800 Subject: [Python-ideas] IntFlags In-Reply-To: <54F9048E.7050402@canterbury.ac.nz> References: <54F8BC30.6080307@canterbury.ac.nz> <54F9048E.7050402@canterbury.ac.nz> Message-ID: On Mar 5, 2015, at 17:36, Greg Ewing wrote: > On 03/06/2015 01:41 PM, Serhiy Storchaka wrote: >> >> bitset in C++ and BitSet in Java are fixed-size arrays of booleans. They >> support such operations as changing a range of bits, but are not compatible >> with ints and sets. > > Java has an EnumSet, which is closer to what we're talking about. It > supports set operations and uses a bit vector internally. If you're looking for similar names in use, PyPI has (at least) modules named bitsets, bitset, bitmap, bitstring, bitarray, and flags, which are all related to what's being discussed here, but crucially without the enum-like naming of the bits. (I've used bitstring and bitarray for things like examining the bits of a TCP header; the others I just saw when searching for bitarray when I couldn't remember its name...) From mistersheik at gmail.com Fri Mar 6 05:26:42 2015 From: mistersheik at gmail.com (Neil Girdhar) Date: Thu, 5 Mar 2015 23:26:42 -0500 Subject: [Python-ideas] IntFlags In-Reply-To: References: Message-ID: Even if you constrain yourself to the BitFlags rather than the more general BitFields, I strongly disagree with the interface that people are proposing involving & and ~ operators. In general, good interface design reflects the way we think about objects ? not their underlying representation. The fact is that a BitSet's main operations are set and clear individual bits. It is as if the BitFlags are a namedtuple with Boolean elements whose underlying storage happens to be an integer. Therefore, the interface that makes the most sense is member access: my_bit_flags.some_bit = True my_bit_flags.some_bit = False I don't see the justification for writing these as my_bit_flags |= TheBitFlagsClass.some_bit my_bit_flags &= ~TheBitFlagsClass.some_bit The second line is particularly terrible because it exposes you to making mistakes like: my_bit_flags &= TheBitFlagsClass.some_bit my_bit_flags |= ~TheBitFlagsClass.some_bit ? both of which are meaningless. It also makes it hard to convert code between the alternate implementation of using a namedtuple. It should be easy to do that in my opinion. Best, Neil On Thu, Mar 5, 2015 at 12:57 PM, Serhiy Storchaka wrote: > On 05.03.15 19:29, Neil Girdhar wrote: > >> Have you looked at my IntFields generalization of IntFlags? It seems >> that many of your examples (permissions, e.g.) are better expressed with >> fields than with flags. >> > > It looks too complicated for such simple case. And it has an interface > incompatible with plain int. > > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > > -- > > --- You received this message because you are subscribed to a topic in the > Google Groups "python-ideas" group. > To unsubscribe from this topic, visit https://groups.google.com/d/ > topic/python-ideas/L5KfCEXFaII/unsubscribe. > To unsubscribe from this group and all its topics, send an email to > python-ideas+unsubscribe at googlegroups.com. > For more options, visit https://groups.google.com/d/optout. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mistersheik at gmail.com Fri Mar 6 05:35:46 2015 From: mistersheik at gmail.com (Neil Girdhar) Date: Thu, 5 Mar 2015 23:35:46 -0500 Subject: [Python-ideas] IntFlags In-Reply-To: References: Message-ID: FYI: ctypes already has a BitFlags/BitFields mechanism. class Flags_bits(ctypes.LittleEndianStructure): _fields_ = [ ("logout", c_uint8, 1), ("userswitch", c_uint8, 1), ("suspend", c_uint8, 1), ("idle", c_uint8, 1), ] On Thu, Mar 5, 2015 at 11:26 PM, Neil Girdhar wrote: > Even if you constrain yourself to the BitFlags rather than the more > general BitFields, I strongly disagree with the interface that people are > proposing involving & and ~ operators. In general, good interface design > reflects the way we think about objects ? not their underlying > representation. The fact is that a BitSet's main operations are set and > clear individual bits. It is as if the BitFlags are a namedtuple with > Boolean elements whose underlying storage happens to be an integer. > Therefore, the interface that makes the most sense is member access: > > my_bit_flags.some_bit = True > my_bit_flags.some_bit = False > > I don't see the justification for writing these as > > my_bit_flags |= TheBitFlagsClass.some_bit > my_bit_flags &= ~TheBitFlagsClass.some_bit > > The second line is particularly terrible because it exposes you to making > mistakes like: > > my_bit_flags &= TheBitFlagsClass.some_bit > my_bit_flags |= ~TheBitFlagsClass.some_bit > > ? both of which are meaningless. > > It also makes it hard to convert code between the alternate implementation > of using a namedtuple. It should be easy to do that in my opinion. > > Best, > > Neil > > On Thu, Mar 5, 2015 at 12:57 PM, Serhiy Storchaka > wrote: > >> On 05.03.15 19:29, Neil Girdhar wrote: >> >>> Have you looked at my IntFields generalization of IntFlags? It seems >>> that many of your examples (permissions, e.g.) are better expressed with >>> fields than with flags. >>> >> >> It looks too complicated for such simple case. And it has an interface >> incompatible with plain int. >> >> >> >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ >> >> -- >> >> --- You received this message because you are subscribed to a topic in >> the Google Groups "python-ideas" group. >> To unsubscribe from this topic, visit https://groups.google.com/d/ >> topic/python-ideas/L5KfCEXFaII/unsubscribe. >> To unsubscribe from this group and all its topics, send an email to >> python-ideas+unsubscribe at googlegroups.com. >> For more options, visit https://groups.google.com/d/optout. >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From abarnert at yahoo.com Fri Mar 6 10:28:29 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Fri, 6 Mar 2015 01:28:29 -0800 Subject: [Python-ideas] IntFlags In-Reply-To: References: Message-ID: <5D40A256-DF55-4B4B-A681-A43AEF2ABB85@yahoo.com> On Mar 5, 2015, at 20:26, Neil Girdhar wrote: > Even if you constrain yourself to the BitFlags rather than the more general BitFields, I strongly disagree with the interface that people are proposing involving & and ~ operators. In general, good interface design reflects the way we think about objects ? not their underlying representation. But sometimes the object really is "an integer used as a set of bits in some C structure/protocol field/well-known API". For example, if we were designing os.open or mmap or whatever as a Pythonic interface, it wouldn't have a "flags" value that or's together multiple integers. We'd probably have separate keyword-only arguments for the less common flags, etc. But they weren't designed from scratch; they were designed to closely mirror the POSIX APIs. And that doesn't just mean simpler implementation, it means people who are familiar with those APIs know how to use them. It means the vast volumes of tutorials and sample code for opening file handles or mapping memory written for C applies to Python. And so on. So, the interface makes sense. So, one very good use for something like IntFlags is to allow people to keep using that C sample code (with trivial, easy-to-understand changes), but get better debugging, etc. when they do so--e.g., when you introspect an mmap object, it would be great if it could tell you that it was opened with PROT_READ | PROT_EXEC, instead of telling you "3", which you have to manually convert to bits and reverse-lookup in the docs or the module dict. Not allowing people to use C-style operations if they use named bits means that someone who wants the advantages of named bits has to rewrite their familiar C-style code. Sure, maybe the result will be more readable (although that's arguable; the suggested alternatives are pretty verbose--especially since people keep suggesting mutating-only APIs...), but it means many people will stick with plain ints rather than rewrite, and those who do rewrite will end up with code that doesn't look like the familiar code that everyone knows how to read from C. > The fact is that a BitSet's main operations are set and clear individual bits. It is as if the BitFlags are a namedtuple with Boolean elements whose underlying storage happens to be an integer. In the case where you don't really care that the underlying storage is an integer, why use an integer in the first place? Why not use a namedtuple, or a set, or whatever else is appropriate? In the very rare case where you need to store a million of these things (and can't store them even more compactly with array or NumPy or similar), you can go get a third-party lib; the vast majority of the time, there's no advantage to using an integer. Except, of course, when the underlying representation is the whole point, because you're dealing with an API that's written in terms of integers. > Therefore, the interface that makes the most sense is member access: > > my_bit_flags.some_bit = True > my_bit_flags.some_bit = False > > I don't see the justification for writing these as > > my_bit_flags |= TheBitFlagsClass.some_bit > my_bit_flags &= ~TheBitFlagsClass.some_bit > > The second line is particularly terrible because it exposes you to making mistakes like: > > my_bit_flags &= TheBitFlagsClass.some_bit > my_bit_flags |= ~TheBitFlagsClass.some_bit > > ? both of which are meaningless. No they're not. Put some real names instead of toy names there: readable = m.prot readable &= ProtFlags.Readable Now it's true iff m.prot includes the Readable flag. Of course usually you'd write this in a single line without mutation: readable = m.prot & ProtFlags.Readable But that just goes to show that the primary interface of bit flags is an immutable one; trying to force people to use mutating methods like set_bit and clear_bit is just getting in people's way. (And try to come up with a good name for the non-mutating operation that's obvious and reads like English and isn't approaching the ridiculous Apple level of verbosity you get in Cocoa methods like "bitSetWithBitClear:".) > It also makes it hard to convert code between the alternate implementation of using a namedtuple. It should be easy to do that in my opinion. > > Best, > > Neil > > On Thu, Mar 5, 2015 at 12:57 PM, Serhiy Storchaka wrote: >> On 05.03.15 19:29, Neil Girdhar wrote: >>> Have you looked at my IntFields generalization of IntFlags? It seems >>> that many of your examples (permissions, e.g.) are better expressed with >>> fields than with flags. >> >> It looks too complicated for such simple case. And it has an interface incompatible with plain int. >> >> >> >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ >> >> -- >> >> --- You received this message because you are subscribed to a topic in the Google Groups "python-ideas" group. >> To unsubscribe from this topic, visit https://groups.google.com/d/topic/python-ideas/L5KfCEXFaII/unsubscribe. >> To unsubscribe from this group and all its topics, send an email to python-ideas+unsubscribe at googlegroups.com. >> For more options, visit https://groups.google.com/d/optout. > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From mistersheik at gmail.com Fri Mar 6 10:42:51 2015 From: mistersheik at gmail.com (Neil Girdhar) Date: Fri, 6 Mar 2015 04:42:51 -0500 Subject: [Python-ideas] IntFlags In-Reply-To: <5D40A256-DF55-4B4B-A681-A43AEF2ABB85@yahoo.com> References: <5D40A256-DF55-4B4B-A681-A43AEF2ABB85@yahoo.com> Message-ID: On Fri, Mar 6, 2015 at 4:28 AM, Andrew Barnert wrote: > On Mar 5, 2015, at 20:26, Neil Girdhar wrote: > > Even if you constrain yourself to the BitFlags rather than the more > general BitFields, I strongly disagree with the interface that people are > proposing involving & and ~ operators. In general, good interface design > reflects the way we think about objects ? not their underlying > representation. > > > But sometimes the object really is "an integer used as a set of bits in > some C structure/protocol field/well-known API". > You can always get that integer by casting to integer. > > For example, if we were designing os.open or mmap or whatever as a > Pythonic interface, it wouldn't have a "flags" value that or's together > multiple integers. We'd probably have separate keyword-only arguments for > the less common flags, etc. But they weren't designed from scratch; they > were designed to closely mirror the POSIX APIs. And that doesn't just mean > simpler implementation, it means people who are familiar with those APIs > know how to use them. It means the vast volumes of tutorials and sample > code for opening file handles or mapping memory written for C applies to > Python. And so on. So, the interface makes sense. > I disagree that there is any need to follow the style of the "vast volumes of tutorials and sample code in C" when designing Python libraries. The goal is for the Python code to be as natural as possible. Member access, and building constants using | are natural. Using &~ to clear a bit is not natural; It is a coincidence of implementation that distracts from what is happening. > > So, one very good use for something like IntFlags is to allow people to > keep using that C sample code (with trivial, easy-to-understand changes), > but get better debugging, etc. when they do so--e.g., when you introspect > an mmap object, it would be great if it could tell you that it was opened > with PROT_READ | PROT_EXEC, instead of telling you "3", which you have to > manually convert to bits and reverse-lookup in the docs or the module dict. > Yes, totally agree. > > Not allowing people to use C-style operations if they use named bits means > that someone who wants the advantages of named bits has to rewrite their > familiar C-style code. Sure, maybe the result will be more readable > (although that's arguable; the suggested alternatives are pretty > verbose--especially since people keep suggesting mutating-only APIs...), > but it means many people will stick with plain ints rather than rewrite, > and those who do rewrite will end up with code that doesn't look like the > familiar code that everyone knows how to read from C. > I totally agree with you that there should not only be mutating-only functions. I agree that | should be used for comining bit fields or flags. However, the people who are "familiar with C" (including me) are frankly dying :) Pandering to the past really gets you nowhere. Try to be a bit idealistic so that new Python code is natural, succinct, and human-readable ? rather than the C values of reflecting the underlying representation in spite of the human being. > The fact is that a BitSet's main operations are set and clear individual > bits. It is as if the BitFlags are a namedtuple with Boolean elements > whose underlying storage happens to be an integer. > > > In the case where you don't really care that the underlying storage is an > integer, why use an integer in the first place? Why not use a namedtuple, > or a set, or whatever else is appropriate? In the very rare case where you > need to store a million of these things (and can't store them even more > compactly with array or NumPy or similar), you can go get a third-party > lib; the vast majority of the time, there's no advantage to using an > integer. > The main reason is so that you can cast it to "int" and produce something that some API requires. > > Except, of course, when the underlying representation is the whole point, > because you're dealing with an API that's written in terms of integers. > > right. > Therefore, the interface that makes the most sense is member access: > > my_bit_flags.some_bit = True > my_bit_flags.some_bit = False > > I don't see the justification for writing these as > > my_bit_flags |= TheBitFlagsClass.some_bit > my_bit_flags &= ~TheBitFlagsClass.some_bit > > The second line is particularly terrible because it exposes you to making > mistakes like: > > my_bit_flags &= TheBitFlagsClass.some_bit > my_bit_flags |= ~TheBitFlagsClass.some_bit > > ? both of which are meaningless. > > > No they're not. Put some real names instead of toy names there: > > readable = m.prot > readable &= ProtFlags.Readable > > Now it's true iff m.prot includes the Readable flag. > > Of course usually you'd write this in a single line without mutation: > > readable = m.prot & ProtFlags.Readable > We both know that the most readable version is just member access, like you would on any object: readable = m.prot.readable This usage of & to filter is unnecessarily complicated. The fact that the machine does so is no reason for the programmer to write it so. > > But that just goes to show that the primary interface of bit flags is an > immutable one; trying to force people to use mutating methods like set_bit > and clear_bit is just getting in people's way. (And try to come up with a > good name for the non-mutating operation that's obvious and reads like > English and isn't approaching the ridiculous Apple level of verbosity you > get in Cocoa methods like "bitSetWithBitClear:".) > I agree with you here. I think you should also have | so that you can build constants the way you're used to, although I'm not sure about & since I don't see when you would use it in preference to member access. > It also makes it hard to convert code between the alternate implementation > of using a namedtuple. It should be easy to do that in my opinion. > > > > Best, > > Neil > > On Thu, Mar 5, 2015 at 12:57 PM, Serhiy Storchaka > wrote: > >> On 05.03.15 19:29, Neil Girdhar wrote: >> >>> Have you looked at my IntFields generalization of IntFlags? It seems >>> that many of your examples (permissions, e.g.) are better expressed with >>> fields than with flags. >>> >> >> It looks too complicated for such simple case. And it has an interface >> incompatible with plain int. >> >> >> >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ >> >> -- >> >> --- You received this message because you are subscribed to a topic in >> the Google Groups "python-ideas" group. >> To unsubscribe from this topic, visit https://groups.google.com/d/ >> topic/python-ideas/L5KfCEXFaII/unsubscribe. >> To unsubscribe from this group and all its topics, send an email to >> python-ideas+unsubscribe at googlegroups.com. >> For more options, visit https://groups.google.com/d/optout. >> > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Fri Mar 6 14:21:36 2015 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 6 Mar 2015 23:21:36 +1000 Subject: [Python-ideas] An even simpler customization of class creation In-Reply-To: <54F5EF25.4030002@stoneleaf.us> References: <54F5EF25.4030002@stoneleaf.us> Message-ID: On 4 March 2015 at 03:28, Ethan Furman wrote: > On 03/02/2015 02:57 AM, Martin Teichmann wrote: >> class Soliton(type): >> def __new__(cls, name, bases, ns): >> self = super().__new__(name, bases, ns) >> return self() >> >> class A(metaclass=Soliton): >> def f(self): >> print(__class__) > > This should be `self.__class__`, yes? No, it's printing out the type referenced from the definition time cell*, rather than the type of the object actually passed in to the method. Cheers, Nick. P.S. *For anyone that isn't already aware, the general gist of how that works: * The compiler actually special cases "super()" and "__class__" in method bodies * If it sees either of them, it implicitly adds a "__class__" cell reference to the function * In the "super()" case, the call is implicitly rewritten as "super(, __class__)" * __class__ is later filled in with a reference to the class being defined, once that actually exists In my view, that's never going to win any awards for "elegance in language design", but it makes super() so much easier to use I'm happy to tell my sense of aesthetics to be quiet :) -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From random832 at fastmail.us Fri Mar 6 14:51:07 2015 From: random832 at fastmail.us (random832 at fastmail.us) Date: Fri, 06 Mar 2015 08:51:07 -0500 Subject: [Python-ideas] IntFlags In-Reply-To: <5D40A256-DF55-4B4B-A681-A43AEF2ABB85@yahoo.com> References: <5D40A256-DF55-4B4B-A681-A43AEF2ABB85@yahoo.com> Message-ID: <1425649867.3365145.236454285.358944E3@webmail.messagingengine.com> On Fri, Mar 6, 2015, at 04:28, Andrew Barnert wrote: > For example, if we were designing os.open or mmap or whatever as a > Pythonic interface, it wouldn't have a "flags" value that or's together > multiple integers. We'd probably have separate keyword-only arguments for > the less common flags, etc. But they weren't designed from scratch; they > were designed to closely mirror the POSIX APIs. It makes me wonder what the os module would look like if Windows (or VMS, OS/2, classic Mac, or whatever other OSes have been supported by python in the present or past) *didn't* provide close mirrors of the POSIX APIs as part of their C runtime library. I mean, it's not like we have opendir and readdir. And while we do have fork and exec, there's a reason beyond convenience for spawn. Because those are the functions that _don't_ exist, or don't work right, on non-Unix platforms. From marky1991 at gmail.com Fri Mar 6 15:11:50 2015 From: marky1991 at gmail.com (Mark Young) Date: Fri, 6 Mar 2015 09:11:50 -0500 Subject: [Python-ideas] An even simpler customization of class creation In-Reply-To: References: <54F5EF25.4030002@stoneleaf.us> Message-ID: Is that documented somewhere? I have never heard of that functionality. Thanks -------------- next part -------------- An HTML attachment was scrubbed... URL: From abarnert at yahoo.com Fri Mar 6 17:09:07 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Fri, 6 Mar 2015 08:09:07 -0800 Subject: [Python-ideas] IntFlags In-Reply-To: <1425649867.3365145.236454285.358944E3@webmail.messagingengine.com> References: <5D40A256-DF55-4B4B-A681-A43AEF2ABB85@yahoo.com> <1425649867.3365145.236454285.358944E3@webmail.messagingengine.com> Message-ID: <21134B82-9842-4959-8294-F3AC7CBCB6EA@yahoo.com> On Mar 6, 2015, at 5:51, random832 at fastmail.us wrote: > On Fri, Mar 6, 2015, at 04:28, Andrew Barnert wrote: >> For example, if we were designing os.open or mmap or whatever as a >> Pythonic interface, it wouldn't have a "flags" value that or's together >> multiple integers. We'd probably have separate keyword-only arguments for >> the less common flags, etc. But they weren't designed from scratch; they >> were designed to closely mirror the POSIX APIs. > > It makes me wonder what the os module would look like if Windows (or > VMS, OS/2, classic Mac, or whatever other OSes have been supported by > python in the present or past) *didn't* provide close mirrors of the > POSIX APIs as part of their C runtime library. Classic Mac didn't provide anything remotely close to the POSIX APIs. IIRC, you called FSSpecMake to create an FSSpec structure from a volume ID, directory ID, and bare filename, then called FSSpecOpenDF with it, after which you'd call various other APIs to map blocks of data to memory handles. For that matter, Win16 and early Win9x didn't have anything quite like the POSIX APIs, but mapping CreateFile and HANDLE values to open and file descriptors wouldn't be quite as ridiculous as mapping FSSpecOpenDF and refnum values. At any rate, I think os.open has always been only available on Unix and NT. The whole point of it is to interface with libraries that want to use file descriptors or to access platform-specific features; if you want cross-platform files, you just call open. That's why the API is so close to the POSIX API; if it tried to wrap things up at a higher level, it wouldn't be able to provide access to flags that only Solaris offers, etc. > I mean, it's not like we have opendir and readdir. And while we do have > fork and exec, there's a reason beyond convenience for spawn. Because > those are the functions that _don't_ exist, or don't work right, on > non-Unix platforms. > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ From abarnert at yahoo.com Fri Mar 6 17:41:00 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Fri, 6 Mar 2015 08:41:00 -0800 Subject: [Python-ideas] IntFlags In-Reply-To: References: <5D40A256-DF55-4B4B-A681-A43AEF2ABB85@yahoo.com> Message-ID: <938519D8-D41C-411D-8CE8-C49BCB1666BA@yahoo.com> On Mar 6, 2015, at 1:42, Neil Girdhar wrote: > On Fri, Mar 6, 2015 at 4:28 AM, Andrew Barnert wrote: >> On Mar 5, 2015, at 20:26, Neil Girdhar wrote: >> >>> Even if you constrain yourself to the BitFlags rather than the more general BitFields, I strongly disagree with the interface that people are proposing involving & and ~ operators. In general, good interface design reflects the way we think about objects ? not their underlying representation. >> >> But sometimes the object really is "an integer used as a set of bits in some C structure/protocol field/well-known API". > > You can always get that integer by casting to integer. > >> >> For example, if we were designing os.open or mmap or whatever as a Pythonic interface, it wouldn't have a "flags" value that or's together multiple integers. We'd probably have separate keyword-only arguments for the less common flags, etc. But they weren't designed from scratch; they were designed to closely mirror the POSIX APIs. And that doesn't just mean simpler implementation, it means people who are familiar with those APIs know how to use them. It means the vast volumes of tutorials and sample code for opening file handles or mapping memory written for C applies to Python. And so on. So, the interface makes sense. > > I disagree that there is any need to follow the style of the "vast volumes of tutorials and sample code in C" when designing Python libraries. The goal is for the Python code to be as natural as possible. Member access, and building constants using | are natural. Using &~ to clear a bit is not natural; It is a coincidence of implementation that distracts from what is happening. For the vast majority of libraries, I agree. An XML parser or audio decoder has no need to follow cryptic C API standards. But libraries that are designed for close-to-the-metal access can be an exception--again, consider os.open, which automatically gives you access to every *nix plafform's platform-specific features. And wrapping C libraries that don't have much of a Python userbase can be another example. If enough people start using it, someone will write and document a higher-level Pythonic interface, but until that happens, having an interface which closely matches what people can find documentation, StackOverflow help, sample code, etc. for is a huge help. Consider PyGame. Much of it is still sparsely documented, but because it wraps the SDL APIs, you can almost always figure out what you need to do, which is part of the reason it's so popular while higher-level wrappers are not. (The other part of the reason is that it wraps almost all of the functionality of SDL, and nothing else can claim that, and again that's probably because it's a thin wrapper.) Or consider PyWin32: it has almost no documentation, and it's not at all Pythonic, but because you can look up a function on MSDN and directly use the C documentation, it's useful for all those areas of the Win32 API (and third-party COM libraries, etc.) that don't have higher-level wrappers. And there are plenty of protocols, file formats, etc. for which the documentation is written for C (or is just a C implementation, as with the predecessor to RTSP that I forget the name of) as well. In an ideal world, everything you wanted would have a high-level, Pythonic API--in fact, everything would be designed for Python in the first place. In the real world, you're better off with a C API than with no API at all. >> So, one very good use for something like IntFlags is to allow people to keep using that C sample code (with trivial, easy-to-understand changes), but get better debugging, etc. when they do so--e.g., when you introspect an mmap object, it would be great if it could tell you that it was opened with PROT_READ | PROT_EXEC, instead of telling you "3", which you have to manually convert to bits and reverse-lookup in the docs or the module dict. > > Yes, totally agree. >> >> Not allowing people to use C-style operations if they use named bits means that someone who wants the advantages of named bits has to rewrite their familiar C-style code. Sure, maybe the result will be more readable (although that's arguable; the suggested alternatives are pretty verbose--especially since people keep suggesting mutating-only APIs...), but it means many people will stick with plain ints rather than rewrite, and those who do rewrite will end up with code that doesn't look like the familiar code that everyone knows how to read from C. > > I totally agree with you that there should not only be mutating-only functions. I agree that | should be used for comining bit fields or flags. However, the people who are "familiar with C" (including me) are frankly dying :) People have been saying that for a couple decades now, but there's still tons of functionality--not just system-level stuff, but APIs for high-level things like audio fingerprinting or animating sprites or streaming video or extending a Python interpreter--that only exists in C (or sometimes C++ or ObjC), or with very thin wrappers for higher-level languages. And that's still going to be true for a long time to come. More importantly, if C really were dead and irrelevant, there would be no need for this proposal; again, the only reason you ever care about packing flags into an int in the first place is for compatibility with C or C-style code. When you don't need that, just use a namedtuple or a set or keyword arguments or whatever in the first place. > Pandering to the past really gets you nowhere. Try to be a bit idealistic so that new Python code is natural, succinct, and human-readable ? rather than the C values of reflecting the underlying representation in spite of the human being. > >>> The fact is that a BitSet's main operations are set and clear individual bits. It is as if the BitFlags are a namedtuple with Boolean elements whose underlying storage happens to be an integer. >> >> In the case where you don't really care that the underlying storage is an integer, why use an integer in the first place? Why not use a namedtuple, or a set, or whatever else is appropriate? In the very rare case where you need to store a million of these things (and can't store them even more compactly with array or NumPy or similar), you can go get a third-party lib; the vast majority of the time, there's no advantage to using an integer. > > The main reason is so that you can cast it to "int" and produce something that some API requires. >> >> Except, of course, when the underlying representation is the whole point, because you're dealing with an API that's written in terms of integers. > right. >>> Therefore, the interface that makes the most sense is member access: >>> >>> my_bit_flags.some_bit = True >>> my_bit_flags.some_bit = False >>> >>> I don't see the justification for writing these as >>> >>> my_bit_flags |= TheBitFlagsClass.some_bit >>> my_bit_flags &= ~TheBitFlagsClass.some_bit >>> >>> The second line is particularly terrible because it exposes you to making mistakes like: >>> >>> my_bit_flags &= TheBitFlagsClass.some_bit >>> my_bit_flags |= ~TheBitFlagsClass.some_bit >>> >>> ? both of which are meaningless. >> >> No they're not. Put some real names instead of toy names there: >> >> readable = m.prot >> readable &= ProtFlags.Readable >> >> Now it's true iff m.prot includes the Readable flag. >> >> Of course usually you'd write this in a single line without mutation: >> >> readable = m.prot & ProtFlags.Readable > > We both know that the most readable version is just member access, like you would on any object: > > readable = m.prot.readable > > This usage of & to filter is unnecessarily complicated. The fact that the machine does so is no reason for the programmer to write it so. Right, so someone should write a higher-level library that wraps up mmap so you don't have to use it. But no one has done so yet, and if you want to use it without waiting another couple decades until someone gets around to it, you're using the C-style API. >> But that just goes to show that the primary interface of bit flags is an immutable one; trying to force people to use mutating methods like set_bit and clear_bit is just getting in people's way. (And try to come up with a good name for the non-mutating operation that's obvious and reads like English and isn't approaching the ridiculous Apple level of verbosity you get in Cocoa methods like "bitSetWithBitClear:".) > > I agree with you here. I think you should also have | so that you can build constants the way you're used to, although I'm not sure about & since I don't see when you would use it in preference to member access. OK, if you have | and &, you automatically have |= and &=. There's no way to implement the former without automatically getting the latter. So if that's your suggestion, it's not possible in the first place, so you have to choose whether we get both or neither. >>> It also makes it hard to convert code between the alternate implementation of using a namedtuple. It should be easy to do that in my opinion. > >>> >>> Best, >>> >>> Neil >>> >>> On Thu, Mar 5, 2015 at 12:57 PM, Serhiy Storchaka wrote: >>>> On 05.03.15 19:29, Neil Girdhar wrote: >>>>> Have you looked at my IntFields generalization of IntFlags? It seems >>>>> that many of your examples (permissions, e.g.) are better expressed with >>>>> fields than with flags. >>>> >>>> It looks too complicated for such simple case. And it has an interface incompatible with plain int. >>>> >>>> >>>> >>>> _______________________________________________ >>>> Python-ideas mailing list >>>> Python-ideas at python.org >>>> https://mail.python.org/mailman/listinfo/python-ideas >>>> Code of Conduct: http://python.org/psf/codeofconduct/ >>>> >>>> -- >>>> >>>> --- You received this message because you are subscribed to a topic in the Google Groups "python-ideas" group. >>>> To unsubscribe from this topic, visit https://groups.google.com/d/topic/python-ideas/L5KfCEXFaII/unsubscribe. >>>> To unsubscribe from this group and all its topics, send an email to python-ideas+unsubscribe at googlegroups.com. >>>> For more options, visit https://groups.google.com/d/optout. >>> >>> _______________________________________________ >>> Python-ideas mailing list >>> Python-ideas at python.org >>> https://mail.python.org/mailman/listinfo/python-ideas >>> Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From abarnert at yahoo.com Fri Mar 6 17:47:09 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Fri, 6 Mar 2015 08:47:09 -0800 Subject: [Python-ideas] An even simpler customization of class creation In-Reply-To: References: <54F5EF25.4030002@stoneleaf.us> Message-ID: <4972E8D7-965E-4D3E-8650-9663CD88B240@yahoo.com> On Mar 6, 2015, at 5:21, Nick Coghlan wrote: > On 4 March 2015 at 03:28, Ethan Furman wrote: >> On 03/02/2015 02:57 AM, Martin Teichmann wrote: >>> class Soliton(type): >>> def __new__(cls, name, bases, ns): >>> self = super().__new__(name, bases, ns) >>> return self() >>> >>> class A(metaclass=Soliton): >>> def f(self): >>> print(__class__) >> >> This should be `self.__class__`, yes? > > No, it's printing out the type referenced from the definition time > cell*, rather than the type of the object actually passed in to the > method. > > Cheers, > Nick. > > P.S. *For anyone that isn't already aware, the general gist of how that works: > > * The compiler actually special cases "super()" and "__class__" in method bodies > * If it sees either of them, it implicitly adds a "__class__" cell > reference to the function > * In the "super()" case, the call is implicitly rewritten as > "super(, __class__)" > * __class__ is later filled in with a reference to the class being > defined, once that actually exists I think it's clearer to explain it as: Every method has a nonlocal variable named __class__ that refers to the class it's defined in. (As an optimization, CPython doesn't create a closure cell for __class__ if the method body doesn't need it.) The only problem with this definition is that someone implementing Python and writing type.__new__ has to know at exactly what point the __class__ cell becomes available. But I don't think an implementation that made it available too early would be considered broken, would it? > In my view, that's never going to win any awards for "elegance in > language design", but it makes super() so much easier to use I'm happy > to tell my sense of aesthetics to be quiet :) > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ From mistersheik at gmail.com Fri Mar 6 18:03:58 2015 From: mistersheik at gmail.com (Neil Girdhar) Date: Fri, 6 Mar 2015 12:03:58 -0500 Subject: [Python-ideas] IntFlags In-Reply-To: <938519D8-D41C-411D-8CE8-C49BCB1666BA@yahoo.com> References: <5D40A256-DF55-4B4B-A681-A43AEF2ABB85@yahoo.com> <938519D8-D41C-411D-8CE8-C49BCB1666BA@yahoo.com> Message-ID: It seems to me that we probably would agree on an interface even if we have a philosophical disagreement. I think it's possible to have clean, Pythonic interface that produces whatever integers you want. In short, my preferred interface is: __or__ (and __ior__) __setattr__ and __getattr__ __int__ and that's it. Is there really a use case for __and__, or __invert__? Given that you want to follow C so closely, I'm surprised that you don't prefer IntFields to IntFlags. I also gave a couple motivating examples for fields (here's a third: http://www.tagwith.com/question_332767_rgb-color-converting-into-565-format ). Best, Neil On Fri, Mar 6, 2015 at 11:41 AM, Andrew Barnert wrote: > On Mar 6, 2015, at 1:42, Neil Girdhar wrote: > > On Fri, Mar 6, 2015 at 4:28 AM, Andrew Barnert wrote: > >> On Mar 5, 2015, at 20:26, Neil Girdhar wrote: >> >> Even if you constrain yourself to the BitFlags rather than the more >> general BitFields, I strongly disagree with the interface that people are >> proposing involving & and ~ operators. In general, good interface design >> reflects the way we think about objects ? not their underlying >> representation. >> >> >> But sometimes the object really is "an integer used as a set of bits in >> some C structure/protocol field/well-known API". >> > > You can always get that integer by casting to integer. > > >> >> For example, if we were designing os.open or mmap or whatever as a >> Pythonic interface, it wouldn't have a "flags" value that or's together >> multiple integers. We'd probably have separate keyword-only arguments for >> the less common flags, etc. But they weren't designed from scratch; they >> were designed to closely mirror the POSIX APIs. And that doesn't just mean >> simpler implementation, it means people who are familiar with those APIs >> know how to use them. It means the vast volumes of tutorials and sample >> code for opening file handles or mapping memory written for C applies to >> Python. And so on. So, the interface makes sense. >> > > I disagree that there is any need to follow the style of the "vast volumes > of tutorials and sample code in C" when designing Python libraries. The > goal is for the Python code to be as natural as possible. Member access, > and building constants using | are natural. Using &~ to clear a bit is not > natural; It is a coincidence of implementation that distracts from what is > happening. > > > For the vast majority of libraries, I agree. An XML parser or audio > decoder has no need to follow cryptic C API standards. > > But libraries that are designed for close-to-the-metal access can be an > exception--again, consider os.open, which automatically gives you access to > every *nix plafform's platform-specific features. > It's just as "close to the metal" to write things with member access. It's not as if it's going to be much slower! It's just a question of how you express setting and clearing bits. > > And wrapping C libraries that don't have much of a Python userbase can be > another example. If enough people start using it, someone will write and > document a higher-level Pythonic interface, but until that happens, having > an interface which closely matches what people can find documentation, > StackOverflow help, sample code, etc. for is a huge help. > Most people use StackOverflow and SO always adapts. > Consider PyGame. Much of it is still sparsely documented, but because it > wraps the SDL APIs, you can almost always figure out what you need to do, > which is part of the reason it's so popular while higher-level wrappers are > not. (The other part of the reason is that it wraps almost all of the > functionality of SDL, and nothing else can claim that, and again that's > probably because it's a thin wrapper.) Or consider PyWin32: it has almost > no documentation, and it's not at all Pythonic, but because you can look up > a function on MSDN and directly use the C documentation, it's useful for > all those areas of the Win32 API (and third-party COM libraries, etc.) that > don't have higher-level wrappers. > > And there are plenty of protocols, file formats, etc. for which the > documentation is written for C (or is just a C implementation, as with the > predecessor to RTSP that I forget the name of) as well. > > In an ideal world, everything you wanted would have a high-level, Pythonic > API--in fact, everything would be designed for Python in the first place. > In the real world, you're better off with a C API than with no API at all. > I don't think the above API is so "high level". I think "x.b = False" is just better design than "x &= ~Class.b". > > So, one very good use for something like IntFlags is to allow people to >> keep using that C sample code (with trivial, easy-to-understand changes), >> but get better debugging, etc. when they do so--e.g., when you introspect >> an mmap object, it would be great if it could tell you that it was opened >> with PROT_READ | PROT_EXEC, instead of telling you "3", which you have to >> manually convert to bits and reverse-lookup in the docs or the module dict. >> > > Yes, totally agree. > >> >> Not allowing people to use C-style operations if they use named bits >> means that someone who wants the advantages of named bits has to rewrite >> their familiar C-style code. Sure, maybe the result will be more readable >> (although that's arguable; the suggested alternatives are pretty >> verbose--especially since people keep suggesting mutating-only APIs...), >> but it means many people will stick with plain ints rather than rewrite, >> and those who do rewrite will end up with code that doesn't look like the >> familiar code that everyone knows how to read from C. >> > > I totally agree with you that there should not only be mutating-only > functions. I agree that | should be used for comining bit fields or > flags. However, the people who are "familiar with C" (including me) are > frankly dying :) > > > People have been saying that for a couple decades now, but there's still > tons of functionality--not just system-level stuff, but APIs for high-level > things like audio fingerprinting or animating sprites or streaming video > or extending a Python interpreter--that only exists in C (or sometimes C++ > or ObjC), or with very thin wrappers for higher-level languages. And that's > still going to be true for a long time to come. > > More importantly, if C really were dead and irrelevant, there would be no > need for this proposal; again, the only reason you ever care about packing > flags into an int in the first place is for compatibility with C or C-style > code. When you don't need that, just use a namedtuple or a set or keyword > arguments or whatever in the first place. > I'm not saying it's dead. I'm saying that pandering to an audience who knows C is a waste. I have a feeling that the real inertia has nothing to do with other people who might know C, and more to do with people like me and you who want to keep writing things the same way we've been writing things. Sometimes, we've been doing things the long way, and the next generation can write things the short way. It's not much more "high level". It's just simpler. > Pandering to the past really gets you nowhere. Try to be a bit idealistic > so that new Python code is natural, succinct, and human-readable ? rather > than the C values of reflecting the underlying representation in spite of > the human being. > > >> The fact is that a BitSet's main operations are set and clear >> individual bits. It is as if the BitFlags are a namedtuple with Boolean >> elements whose underlying storage happens to be an integer. >> >> >> In the case where you don't really care that the underlying storage is an >> integer, why use an integer in the first place? Why not use a namedtuple, >> or a set, or whatever else is appropriate? In the very rare case where you >> need to store a million of these things (and can't store them even more >> compactly with array or NumPy or similar), you can go get a third-party >> lib; the vast majority of the time, there's no advantage to using an >> integer. >> > > The main reason is so that you can cast it to "int" and produce something > that some API requires. > >> >> Except, of course, when the underlying representation is the whole point, >> because you're dealing with an API that's written in terms of integers. >> >> right. > >> Therefore, the interface that makes the most sense is member access: >> >> my_bit_flags.some_bit = True >> my_bit_flags.some_bit = False >> >> I don't see the justification for writing these as >> >> my_bit_flags |= TheBitFlagsClass.some_bit >> my_bit_flags &= ~TheBitFlagsClass.some_bit >> >> The second line is particularly terrible because it exposes you to making >> mistakes like: >> >> my_bit_flags &= TheBitFlagsClass.some_bit >> my_bit_flags |= ~TheBitFlagsClass.some_bit >> >> ? both of which are meaningless. >> >> >> No they're not. Put some real names instead of toy names there: >> >> readable = m.prot >> readable &= ProtFlags.Readable >> >> Now it's true iff m.prot includes the Readable flag. >> >> Of course usually you'd write this in a single line without mutation: >> >> readable = m.prot & ProtFlags.Readable >> > > We both know that the most readable version is just member access, like > you would on any object: > > readable = m.prot.readable > > This usage of & to filter is unnecessarily complicated. The fact that the > machine does so is no reason for the programmer to write it so. > > > Right, so someone should write a higher-level library that wraps up mmap > so you don't have to use it. But no one has done so yet, and if you want to > use it without waiting another couple decades until someone gets around to > it, you're using the C-style API. > > But that just goes to show that the primary interface of bit flags is an >> immutable one; trying to force people to use mutating methods like set_bit >> and clear_bit is just getting in people's way. (And try to come up with a >> good name for the non-mutating operation that's obvious and reads like >> English and isn't approaching the ridiculous Apple level of verbosity you >> get in Cocoa methods like "bitSetWithBitClear:".) >> > > I agree with you here. I think you should also have | so that you can > build constants the way you're used to, although I'm not sure about & since > I don't see when you would use it in preference to member access. > > > OK, if you have | and &, you automatically have |= and &=. There's no way > to implement the former without automatically getting the latter. So if > that's your suggestion, it's not possible in the first place, so you have > to choose whether we get both or neither. > > It also makes it hard to convert code between the alternate implementation >> of using a namedtuple. It should be easy to do that in my opinion. >> >> > >> >> Best, >> >> Neil >> >> On Thu, Mar 5, 2015 at 12:57 PM, Serhiy Storchaka >> wrote: >> >>> On 05.03.15 19:29, Neil Girdhar wrote: >>> >>>> Have you looked at my IntFields generalization of IntFlags? It seems >>>> that many of your examples (permissions, e.g.) are better expressed with >>>> fields than with flags. >>>> >>> >>> It looks too complicated for such simple case. And it has an interface >>> incompatible with plain int. >>> >>> >>> >>> _______________________________________________ >>> Python-ideas mailing list >>> Python-ideas at python.org >>> https://mail.python.org/mailman/listinfo/python-ideas >>> Code of Conduct: http://python.org/psf/codeofconduct/ >>> >>> -- >>> >>> --- You received this message because you are subscribed to a topic in >>> the Google Groups "python-ideas" group. >>> To unsubscribe from this topic, visit https://groups.google.com/d/ >>> topic/python-ideas/L5KfCEXFaII/unsubscribe. >>> To unsubscribe from this group and all its topics, send an email to >>> python-ideas+unsubscribe at googlegroups.com. >>> For more options, visit https://groups.google.com/d/optout. >>> >> >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From random832 at fastmail.us Fri Mar 6 18:19:12 2015 From: random832 at fastmail.us (random832 at fastmail.us) Date: Fri, 06 Mar 2015 12:19:12 -0500 Subject: [Python-ideas] IntFlags In-Reply-To: <21134B82-9842-4959-8294-F3AC7CBCB6EA@yahoo.com> References: <5D40A256-DF55-4B4B-A681-A43AEF2ABB85@yahoo.com> <1425649867.3365145.236454285.358944E3@webmail.messagingengine.com> <21134B82-9842-4959-8294-F3AC7CBCB6EA@yahoo.com> Message-ID: <1425662352.4091428.236549341.705B2877@webmail.messagingengine.com> On Fri, Mar 6, 2015, at 11:09, Andrew Barnert wrote: > Classic Mac didn't provide anything remotely close to the POSIX APIs. > IIRC, you called FSSpecMake to create an FSSpec structure from a volume > ID, directory ID, and bare filename, then called FSSpecOpenDF with it, > after which you'd call various other APIs to map blocks of data to memory > handles. > > For that matter, Win16 and early Win9x didn't have anything quite like > the POSIX APIs, but mapping CreateFile and HANDLE values to open and file > descriptors wouldn't be quite as ridiculous as mapping FSSpecOpenDF and > refnum values. In both cases, open/read/write are in the libraries provided with the commonly available C compilers on the platform (MPW or Codewarrior in the Mac case, MSVC or Borland etc in the Windows/DOS case), and it doesn't really matter that they're not part of the platform proper. I don't think there's ever been a compiler for Windows or DOS that _didn't_ provide these functions. It depends on how you define "quite like" the POSIX APIs, I guess. Anyway, the source code of the os module for classic mac clearly shows that it is implemented in terms of pre-existing POSIX-like functions provided either by the supported build platforms or by something called "GUSI", rather than directly in terms of the native API. https://hg.python.org/cpython/file/364638d6434d/Mac/Modules/macmodule.c - you have to go further back to find a version of python that actually had dosmodule.c, but it looks basically the same. > At any rate, I think os.open has always been only available on Unix and > NT. Nope. List of functions on the os module on Mac in python 2.0: https://docs.python.org/release/2.0/mac/module-mac.html > The whole point of it is to interface with libraries that want to use > file descriptors or to access platform-specific features; if you want > cross-platform files, you just call open. The os module provides the same interface on all platforms _so that_ the higher-level functions like builtin open can be implemented in terms of it. It's the implementation (and list of available functions) that's different between platforms, not the interface. > That's why the API is so close > to the POSIX API; if it tried to wrap things up at a higher level, it > wouldn't be able to provide access to flags that only Solaris offers, > etc. Your theory doesn't explain listdir and walk. The reason the API is so close to the POSIX API is because the work of emulating that API was already done on these platforms in order to support ports of Unix C programs, and because no-one saw any value in supporting platform-specific features not exposed through that layer. From techtonik at gmail.com Thu Mar 5 15:23:43 2015 From: techtonik at gmail.com (anatoly techtonik) Date: Thu, 5 Mar 2015 17:23:43 +0300 Subject: [Python-ideas] Accessible IDE In-Reply-To: <35CE8F26-4970-438C-A8B9-E1429252A341@gmail.com> References: <35CE8F26-4970-438C-A8B9-E1429252A341@gmail.com> Message-ID: On Tue, Feb 17, 2015 at 12:55 AM, Bryan Duarte wrote: > > I was wondering if anyone had a suggestion on an accessible IDE for Python using Voiceover on the Mac. I use a Mac with Voiceover to interact with my computer, but can also use Linux or Windows if necessary. I would really like to find a tool that will allow me to use Python with some of the features of an IDE. I have tried: > ? PyCharm > ? iPy > ? iPy Notebook > ? Idel > ? and Eclipse > > So far Eclipse worked the best but still caused problems. Any suggestions and/or help would be great. I think it is highly dependent on the reader, and reader is highly dependent on platform. What reader do you use on Linux and Windows? Also, I am certain that some web plumbing is very possible to add for IPython Notebook, because it is just a web page. Do you know which standards are actual to look into? Is it WAI-ARIA https://en.wikipedia.org/wiki/WAI-ARIA ? From techtonik at gmail.com Thu Mar 5 14:40:05 2015 From: techtonik at gmail.com (anatoly techtonik) Date: Thu, 5 Mar 2015 16:40:05 +0300 Subject: [Python-ideas] .from and .to instead of .encode and .decode Message-ID: Hi, While looking at the code like: 'os': sysinfo['os'].decode('utf-8'), 'hostname': sysinfo['hostname'].decode('utf-8'), I can't really read if the result will be unicode or binary string in utf-8. It would be more convenient for readability to have these instead: bytes.from(encoding) -> unicode unicode.to(encoding) -> bytes .encode/.decode are confusing, because it Python 2 it was: str.encode(encoding) -> str str.decode(encoding) -> str with no encoding info attached. -- anatoly t. From ethan at stoneleaf.us Sat Mar 7 09:07:38 2015 From: ethan at stoneleaf.us (Ethan Furman) Date: Sat, 07 Mar 2015 00:07:38 -0800 Subject: [Python-ideas] IntFlags In-Reply-To: References: Message-ID: <54FAB1CA.3000105@stoneleaf.us> On 03/03/2015 07:52 AM, Serhiy Storchaka wrote: > We need new type IntFlags. It is like IntEnum, but has differences: > > 1. The value of an instance should be not limited to the set of predefined constants. It can be a combination of > predefined constants or even arbitrary integer. > > 2. The result of "|", "&" and "~" operators for IntFlags arguments should be an instance of the same IntFlags subclass. > > 3. It should have nice str() and repr(). As long as we are dreaming :) class Stat(IntFlag): RDONLY = 1 NOSUID = 2 NODEV = 4 NOEXEC = 8 SYNCHRONOUS = 16 MANDLOCK = 64 WRITE = 128 APPEND = 256 NOATIME = 1024 NODIRATIME = 2048 RELATIME = 4096 a = Stat.RDONLY # creates a new instance of Stat, not a singleton b = Stat.RDONLY a is b # False a == b # True c = a a |= Stat.MANDLOCK c.MANDLOCK # 64 b.MANDLOCK # 0 c is a # True repr(a) # repr(b) # d = b | 32 # undefined value repr(d) # d.MANDLOCK = True repr(d) # repr(~d) # I'm not at all sure I have that last one correct. -- ~Ethan~ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: OpenPGP digital signature URL: From storchaka at gmail.com Sat Mar 7 11:50:46 2015 From: storchaka at gmail.com (Serhiy Storchaka) Date: Sat, 07 Mar 2015 12:50:46 +0200 Subject: [Python-ideas] IntFlags In-Reply-To: <54FAB1CA.3000105@stoneleaf.us> References: <54FAB1CA.3000105@stoneleaf.us> Message-ID: On 07.03.15 10:07, Ethan Furman wrote: > On 03/03/2015 07:52 AM, Serhiy Storchaka wrote: > >> We need new type IntFlags. It is like IntEnum, but has differences: >> >> 1. The value of an instance should be not limited to the set of predefined constants. It can be a combination of >> predefined constants or even arbitrary integer. >> >> 2. The result of "|", "&" and "~" operators for IntFlags arguments should be an instance of the same IntFlags subclass. >> >> 3. It should have nice str() and repr(). > > As long as we are dreaming :) > > class Stat(IntFlag): > RDONLY = 1 > NOSUID = 2 > NODEV = 4 > NOEXEC = 8 > SYNCHRONOUS = 16 > MANDLOCK = 64 > WRITE = 128 > APPEND = 256 > NOATIME = 1024 > NODIRATIME = 2048 > RELATIME = 4096 > > a = Stat.RDONLY # creates a new instance of Stat, not a singleton IntFlags is purposed to replace existing integer constants (as IntEnum). globals().update(Stat.__members__) From rosuav at gmail.com Sat Mar 7 12:41:35 2015 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 7 Mar 2015 22:41:35 +1100 Subject: [Python-ideas] .from and .to instead of .encode and .decode In-Reply-To: References: Message-ID: On Fri, Mar 6, 2015 at 12:40 AM, anatoly techtonik wrote: > Hi, > > While looking at the code like: > > 'os': sysinfo['os'].decode('utf-8'), > 'hostname': sysinfo['hostname'].decode('utf-8'), > > I can't really read if the result will be unicode or binary string in > utf-8. If it says "decode", the result is a Unicode string. If it says "encode", the result is bytes. I'm not sure what is difficult here. > .encode/.decode are confusing, because it Python 2 it was: > > str.encode(encoding) -> str > str.decode(encoding) -> str > > with no encoding info attached. $ python Python 2.7.3 (default, Mar 13 2014, 11:03:55) [GCC 4.7.2] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> "test".decode("utf-8") u'test' >>> u"test".encode("utf-8") 'test' Looks to me like str.decode() -> unicode, and unicode.encode() -> str, at least for UTF-8 and other encodings that apply to Unicode. Yes, there are some oddities in Py2: >>> "74657374".decode("hex") 'test' in which str.decode returns a str, but AFAIK all of those are buried away in Python 3: >>> codecs.decode(b"74657374","hex") b'test' So where's the confusion? ChrisA From storchaka at gmail.com Sat Mar 7 14:27:57 2015 From: storchaka at gmail.com (Serhiy Storchaka) Date: Sat, 07 Mar 2015 15:27:57 +0200 Subject: [Python-ideas] .from and .to instead of .encode and .decode In-Reply-To: References: Message-ID: On 05.03.15 15:40, anatoly techtonik wrote: > While looking at the code like: > > 'os': sysinfo['os'].decode('utf-8'), > 'hostname': sysinfo['hostname'].decode('utf-8'), > > I can't really read if the result will be unicode or binary string in > utf-8. It would be more convenient for readability to have these > instead: > > bytes.from(encoding) -> unicode > unicode.to(encoding) -> bytes Don't write sysinfo['os'].decode('utf-8') if it confuses you. Write str(sysinfo['os'], 'utf-8'). From luciano at ramalho.org Sat Mar 7 14:42:59 2015 From: luciano at ramalho.org (Luciano Ramalho) Date: Sat, 7 Mar 2015 10:42:59 -0300 Subject: [Python-ideas] .from and .to instead of .encode and .decode In-Reply-To: References: Message-ID: On Sat, Mar 7, 2015 at 8:41 AM, Chris Angelico wrote: > If it says "decode", the result is a Unicode string. If it says > "encode", the result is bytes. I'm not sure what is difficult here. Yep. When I teach, I use this mnemonic, which I can now quote from my book [1] ;-) [TIP] ==== If you need a memory aid to distinguish `.decode()` from `.encode()`, convince yourself that a Unicode `str` contains "human" text, while byte sequences can be cryptic machine core dumps. Therefore, it makes sense that we *decode* `bytes` to `str` to get human readable text, and we *encode* text to `bytes` for storage or transmission. ==== [1] http://shop.oreilly.com/product/0636920032519.do Cheers, Luciano > >> .encode/.decode are confusing, because it Python 2 it was: >> >> str.encode(encoding) -> str >> str.decode(encoding) -> str >> >> with no encoding info attached. > > $ python > Python 2.7.3 (default, Mar 13 2014, 11:03:55) > [GCC 4.7.2] on linux2 > Type "help", "copyright", "credits" or "license" for more information. >>>> "test".decode("utf-8") > u'test' >>>> u"test".encode("utf-8") > 'test' > > Looks to me like str.decode() -> unicode, and unicode.encode() -> str, > at least for UTF-8 and other encodings that apply to Unicode. Yes, > there are some oddities in Py2: > >>>> "74657374".decode("hex") > 'test' > > in which str.decode returns a str, but AFAIK all of those are buried > away in Python 3: > >>>> codecs.decode(b"74657374","hex") > b'test' > > So where's the confusion? > > ChrisA > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -- Luciano Ramalho Twitter: @ramalhoorg Professor em: http://python.pro.br Twitter: @pythonprobr From abarnert at yahoo.com Sat Mar 7 14:41:30 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Sat, 7 Mar 2015 05:41:30 -0800 Subject: [Python-ideas] .from and .to instead of .encode and .decode In-Reply-To: References: Message-ID: <54D70D5D-23E6-476F-86BD-23B9475FCCB9@yahoo.com> On Mar 5, 2015, at 5:40 AM, anatoly techtonik wrote: > > Hi, > > While looking at the code like: > > 'os': sysinfo['os'].decode('utf-8'), > 'hostname': sysinfo['hostname'].decode('utf-8'), > > I can't really read if the result will be unicode or binary string in > utf-8. It would be more convenient for readability to have these > instead: > > bytes.from(encoding) -> unicode > unicode.to(encoding) -> bytes > > .encode/.decode are confusing, because it Python 2 it was: > > str.encode(encoding) -> str > str.decode(encoding) -> str Except for a handful of confusing cases like hex and rot13, str.decode always returned unicode, and unicode.encode always returned str. However, str.encode _did_ return str--which it did by first decoding the str with sys.getdefaultencoding() so it had something to encode, which often failed, giving those wonderful error messages about being unable to decode ASCII when you asked it to encode UTF-8. It's little wonder that so many novices resorted to throwing in extra random calls to encode and decode and the str constructor until it seemed to work for one example, and then threw their hands up in despair when it didn't work for the next example input. Python 3 fixed that, simply by getting rid of the Unicode decode and bytes encode methods (which were occasionally useful for codecs like hex and rot13, but far more often just confusing). So now you can only encode Unicode to bytes, and decode bytes to Unicode, and you're the first person I've seen who found the Python 3 API at all confusing. I suspect that's because you didn't realize you were confused by Python 2, and carried that confusion over to 3, and somehow didn't run into any problems for the first few years? > with no encoding info attached. > -- > anatoly t. > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ From abarnert at yahoo.com Sat Mar 7 14:53:09 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Sat, 7 Mar 2015 05:53:09 -0800 Subject: [Python-ideas] IntFlags In-Reply-To: <54FAB1CA.3000105@stoneleaf.us> References: <54FAB1CA.3000105@stoneleaf.us> Message-ID: <5C81EBF0-6229-452E-9B73-CBEB2B84FF6F@yahoo.com> On Mar 7, 2015, at 12:07 AM, Ethan Furman wrote: > >> On 03/03/2015 07:52 AM, Serhiy Storchaka wrote: >> >> We need new type IntFlags. It is like IntEnum, but has differences: >> >> 1. The value of an instance should be not limited to the set of predefined constants. It can be a combination of >> predefined constants or even arbitrary integer. >> >> 2. The result of "|", "&" and "~" operators for IntFlags arguments should be an instance of the same IntFlags subclass. >> >> 3. It should have nice str() and repr(). > > As long as we are dreaming :) > > class Stat(IntFlag): > RDONLY = 1 > NOSUID = 2 > NODEV = 4 > NOEXEC = 8 > SYNCHRONOUS = 16 > MANDLOCK = 64 > WRITE = 128 > APPEND = 256 > NOATIME = 1024 > NODIRATIME = 2048 > RELATIME = 4096 > > a = Stat.RDONLY # creates a new instance of Stat, not a singleton Why? Assuming the values are immutable (like int), the only difference is your is test, and I have no idea why your code would care about that. (If you're worried about the "a |=" below changing b--or, worse, changing the constant--there's no reason to worry. Just as "a = 1; a |= 2" doesn't affect any other variable holding the int 1, the same would be true here.) > b = Stat.RDONLY > a is b # False > a == b # True > > c = a > a |= Stat.MANDLOCK > c.MANDLOCK # 64 > b.MANDLOCK # 0 > c is a # True > > repr(a) # > repr(b) # > > d = b | 32 # undefined value > repr(d) # > d.MANDLOCK = True > repr(d) # > > repr(~d) # > > I'm not at all sure I have that last one correct. That last one is the big question. In C, ~d is going to have the 8192, 16384, ... 2b bits set, not just the bits you defined and the gaps in between. Are you sure you want a different result in Python? (Especially if you're on a platform that does something with those extra bits, so you can get values back that actually have them set. Although I don't think that's an issue with stat, it is with lots of other flags from POSIX-land.) Look at all of the alternatives that have been suggested (round up to power of 256, guess what C would do on your platform with the most likely equivalent definitions, use signed instead of unsigned so the issue moves somewhere else, handle complementing with a special bool, maybe others); why is this one better? Why is it OK to set the undefined 512 bit but not the equally-undefined 8192 bit when someone does ~ on a value of 32? From ethan at stoneleaf.us Sat Mar 7 15:50:38 2015 From: ethan at stoneleaf.us (Ethan Furman) Date: Sat, 07 Mar 2015 06:50:38 -0800 Subject: [Python-ideas] IntFlags In-Reply-To: References: <54FAB1CA.3000105@stoneleaf.us> Message-ID: <54FB103E.4060107@stoneleaf.us> On 03/07/2015 02:50 AM, Serhiy Storchaka wrote: > On 07.03.15 10:07, Ethan Furman wrote: >> On 03/03/2015 07:52 AM, Serhiy Storchaka wrote: >> >>> We need new type IntFlags. It is like IntEnum, but has differences: >>> >>> 1. The value of an instance should be not limited to the set of predefined constants. It can be a combination of >>> predefined constants or even arbitrary integer. >>> >>> 2. The result of "|", "&" and "~" operators for IntFlags arguments should be an instance of the same IntFlags subclass. >>> >>> 3. It should have nice str() and repr(). >> >> As long as we are dreaming :) >> >> class Stat(IntFlag): >> RDONLY = 1 >> NOSUID = 2 >> NODEV = 4 >> NOEXEC = 8 >> SYNCHRONOUS = 16 >> MANDLOCK = 64 >> WRITE = 128 >> APPEND = 256 >> NOATIME = 1024 >> NODIRATIME = 2048 >> RELATIME = 4096 >> >> a = Stat.RDONLY # creates a new instance of Stat, not a singleton > > IntFlags is purposed to replace existing integer constants (as IntEnum). > > globals().update(Stat.__members__) And that can still work -- if the flag is accessed from the class (Stat.RDONLY) it will always be the value assigned (1); if it is accessed from a member, it will be the value assigned /if set/, otherwise 0. This also has the advantage of supporting both C style operations (x = Stat.WRITE | Stat.APPEND), or the more customary Python operations (x = Stat.WRITE; x.APPEND = True). -- ~Ethan~ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: OpenPGP digital signature URL: From ethan at stoneleaf.us Sat Mar 7 16:05:05 2015 From: ethan at stoneleaf.us (Ethan Furman) Date: Sat, 07 Mar 2015 07:05:05 -0800 Subject: [Python-ideas] IntFlags In-Reply-To: <5C81EBF0-6229-452E-9B73-CBEB2B84FF6F@yahoo.com> References: <54FAB1CA.3000105@stoneleaf.us> <5C81EBF0-6229-452E-9B73-CBEB2B84FF6F@yahoo.com> Message-ID: <54FB13A1.5020008@stoneleaf.us> On 03/07/2015 05:53 AM, Andrew Barnert wrote: > On Mar 7, 2015, at 12:07 AM, Ethan Furman wrote: >> >>> On 03/03/2015 07:52 AM, Serhiy Storchaka wrote: >>> >>> We need new type IntFlags. It is like IntEnum, but has differences: >>> >>> 1. The value of an instance should be not limited to the set of predefined constants. It can be a combination of >>> predefined constants or even arbitrary integer. >>> >>> 2. The result of "|", "&" and "~" operators for IntFlags arguments should be an instance of the same IntFlags subclass. >>> >>> 3. It should have nice str() and repr(). >> >> As long as we are dreaming :) >> >> class Stat(IntFlag): >> RDONLY = 1 >> NOSUID = 2 >> NODEV = 4 >> NOEXEC = 8 >> SYNCHRONOUS = 16 >> MANDLOCK = 64 >> WRITE = 128 >> APPEND = 256 >> NOATIME = 1024 >> NODIRATIME = 2048 >> RELATIME = 4096 >> >> a = Stat.RDONLY # creates a new instance of Stat, not a singleton > > Why? Because by having mutable instances of Stat we can have more Python operations: instead of: x = Stat(some_Stat_value_from_somewhere) x = x | Stat.NOEXEC # to set the bit we can say: x = Stat(some_Stat_value_from_somewhere) x.NOEXEC = True or, even more illustratively, instead of: x = Stat(some_Stat_value_from_somewhere) x &= ~Stat.NOEXEC # to clear the bit we can have: x = Stat(some_Stat_value_from_somewhere) x.NOEXEC = False # much more readable > Assuming the values are immutable (like int), [...] Drat. Instances of Stat would not be immutuable, but that means they can't be thin wrappers around `int`, doesn't it? Which also means more work around C call sites. Drat and double-drat. >> repr(~d) # >> >> I'm not at all sure I have that last one correct. > > That last one is the big question. In C, ~d is going to have the 8192, 16384, ... 2b > bits set, not just the bits you defined and the gaps in between. Are you sure you want > a different result in Python? (Especially if you're on a platform that does something > with those extra bits, so you can get values back that actually have them set. > Although I don't think that's an issue with stat, it is with lots of other flags from > POSIX-land.) A `bytes` (or `byte_size`) would need to be set to control how many bits got flipped. If not set, then only defined bits get flipped. -- ~Ethan~ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: OpenPGP digital signature URL: From mistersheik at gmail.com Sat Mar 7 19:21:44 2015 From: mistersheik at gmail.com (Neil Girdhar) Date: Sat, 7 Mar 2015 13:21:44 -0500 Subject: [Python-ideas] IntFlags In-Reply-To: <54FB13A1.5020008@stoneleaf.us> References: <54FAB1CA.3000105@stoneleaf.us> <5C81EBF0-6229-452E-9B73-CBEB2B84FF6F@yahoo.com> <54FB13A1.5020008@stoneleaf.us> Message-ID: On Sat, Mar 7, 2015 at 10:05 AM, Ethan Furman wrote: > On 03/07/2015 05:53 AM, Andrew Barnert wrote: > > On Mar 7, 2015, at 12:07 AM, Ethan Furman wrote: > >> > >>> On 03/03/2015 07:52 AM, Serhiy Storchaka wrote: > >>> > >>> We need new type IntFlags. It is like IntEnum, but has differences: > >>> > >>> 1. The value of an instance should be not limited to the set of > predefined constants. It can be a combination of > >>> predefined constants or even arbitrary integer. > >>> > >>> 2. The result of "|", "&" and "~" operators for IntFlags arguments > should be an instance of the same IntFlags subclass. > >>> > >>> 3. It should have nice str() and repr(). > >> > >> As long as we are dreaming :) > >> > >> class Stat(IntFlag): > >> RDONLY = 1 > >> NOSUID = 2 > >> NODEV = 4 > >> NOEXEC = 8 > >> SYNCHRONOUS = 16 > >> MANDLOCK = 64 > >> WRITE = 128 > >> APPEND = 256 > >> NOATIME = 1024 > >> NODIRATIME = 2048 > >> RELATIME = 4096 > >> > >> a = Stat.RDONLY # creates a new instance of Stat, not a singleton > > > > Why? > > Because by having mutable instances of Stat we can have more Python > operations: > > instead of: > > x = Stat(some_Stat_value_from_somewhere) > x = x | Stat.NOEXEC # to set the bit > > we can say: > > x = Stat(some_Stat_value_from_somewhere) > x.NOEXEC = True > > or, even more illustratively, instead of: > > x = Stat(some_Stat_value_from_somewhere) > x &= ~Stat.NOEXEC # to clear the bit > > we can have: > > x = Stat(some_Stat_value_from_somewhere) > x.NOEXEC = False # much more readable > Yes, +1 > > > > Assuming the values are immutable (like int), [...] > > Drat. Instances of Stat would not be immutuable, but that means they > can't be thin wrappers around `int`, doesn't it? > Which also means more work around C call sites. Drat and double-drat. > Can't your C call sites cast the object to int? > > > >> repr(~d) # > 8094> > >> > >> I'm not at all sure I have that last one correct. > > > > That last one is the big question. In C, ~d is going to have the 8192, > 16384, ... 2b > > bits set, not just the bits you defined and the gaps in between. Are you > sure you want > > a different result in Python? (Especially if you're on a platform that > does something > > with those extra bits, so you can get values back that actually have > them set. > > Although I don't think that's an issue with stat, it is with lots of > other flags from > > POSIX-land.) > > A `bytes` (or `byte_size`) would need to be set to control how many bits > got flipped. If not set, then only defined > bits get flipped. > Why do you need ~ at all? Do any API calls that you want to make want the inverted flags? Isn't the only point of inverting the bits in order to clear a field? > > -- > ~Ethan~ > > -- > > --- > You received this message because you are subscribed to a topic in the > Google Groups "python-ideas" group. > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/python-ideas/L5KfCEXFaII/unsubscribe. > To unsubscribe from this group and all its topics, send an email to > python-ideas+unsubscribe at googlegroups.com. > For more options, visit https://groups.google.com/d/optout. > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > > -- > > --- > You received this message because you are subscribed to a topic in the > Google Groups "python-ideas" group. > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/python-ideas/L5KfCEXFaII/unsubscribe. > To unsubscribe from this group and all its topics, send an email to > python-ideas+unsubscribe at googlegroups.com. > For more options, visit https://groups.google.com/d/optout. > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ethan at stoneleaf.us Sat Mar 7 19:51:18 2015 From: ethan at stoneleaf.us (Ethan Furman) Date: Sat, 07 Mar 2015 10:51:18 -0800 Subject: [Python-ideas] IntFlags In-Reply-To: References: <54FAB1CA.3000105@stoneleaf.us> <5C81EBF0-6229-452E-9B73-CBEB2B84FF6F@yahoo.com> <54FB13A1.5020008@stoneleaf.us> Message-ID: <54FB48A6.5080405@stoneleaf.us> On 03/07/2015 10:21 AM, Neil Girdhar wrote: > On Sat, Mar 7, 2015 at 10:05 AM, Ethan Furman wrote: >> or, even more illustratively, instead of: >> >> x = Stat(some_Stat_value_from_somewhere) >> x &= ~Stat.NOEXEC # to clear the bit >> >> we can have: >> >> x = Stat(some_Stat_value_from_somewhere) >> x.NOEXEC = False # much more readable > > Yes, +1 >> Drat. Instances of Stat would not be immutuable, but that means they can't be thin wrappers around `int`, doesn't it? >> Which also means more work around C call sites. Drat and double-drat. > > Can't your C call sites cast the object to int? They could, but then you no longer have a drop-in replacement, which is what IntEnum is. Plus it's a hassle. So IntFlag (or whatever it's called) would to be immutable, which means no neat tricks like `obj.bit_name = False`. > Why do you need ~ at all? Do any API calls that you want to make want the inverted > flags? Isn't the only point of inverting the bits in order to clear a field? Hopefully somebody else can address this point. -- ~Ethan~ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: OpenPGP digital signature URL: From storchaka at gmail.com Sat Mar 7 19:58:33 2015 From: storchaka at gmail.com (Serhiy Storchaka) Date: Sat, 07 Mar 2015 20:58:33 +0200 Subject: [Python-ideas] IntFlags In-Reply-To: References: <54FAB1CA.3000105@stoneleaf.us> <5C81EBF0-6229-452E-9B73-CBEB2B84FF6F@yahoo.com> <54FB13A1.5020008@stoneleaf.us> Message-ID: On 07.03.15 20:21, Neil Girdhar wrote: > Why do you need ~ at all? Do any API calls that you want to make want > the inverted flags? Isn't the only point of inverting the bits in order > to clear a field? Because existing code uses it. mode = os.stat(path).st_mode os.chmod(path, mode & ~(stat.S_IRUSR|stat.S_IRGRP|stat.S_IROTH)) From mistersheik at gmail.com Sat Mar 7 20:02:13 2015 From: mistersheik at gmail.com (Neil Girdhar) Date: Sat, 7 Mar 2015 14:02:13 -0500 Subject: [Python-ideas] IntFlags In-Reply-To: <54FB48A6.5080405@stoneleaf.us> References: <54FAB1CA.3000105@stoneleaf.us> <5C81EBF0-6229-452E-9B73-CBEB2B84FF6F@yahoo.com> <54FB13A1.5020008@stoneleaf.us> <54FB48A6.5080405@stoneleaf.us> Message-ID: On Sat, Mar 7, 2015 at 1:51 PM, Ethan Furman wrote: > On 03/07/2015 10:21 AM, Neil Girdhar wrote: > > On Sat, Mar 7, 2015 at 10:05 AM, Ethan Furman wrote: > > >> or, even more illustratively, instead of: > >> > >> x = Stat(some_Stat_value_from_somewhere) > >> x &= ~Stat.NOEXEC # to clear the bit > >> > >> we can have: > >> > >> x = Stat(some_Stat_value_from_somewhere) > >> x.NOEXEC = False # much more readable > > > > Yes, +1 > > >> Drat. Instances of Stat would not be immutuable, but that means they > can't be thin wrappers around `int`, doesn't it? > >> Which also means more work around C call sites. Drat and double-drat. > > > > Can't your C call sites cast the object to int? > > They could, but then you no longer have a drop-in replacement, which is > what IntEnum is. Plus it's a hassle. So > IntFlag (or whatever it's called) would to be immutable, which means no > neat tricks like `obj.bit_name = False`. > A drop-in replacement for what? Is there a lot of Python code that manipulates flags that you expect people to convert to IntFlags from int? Isn't the point of IntFlags for writing new code? > > Why do you need ~ at all? Do any API calls that you want to make want > the inverted > > flags? Isn't the only point of inverting the bits in order to clear a > field? > > Hopefully somebody else can address this point. > > -- > ~Ethan~ > > -- > > --- > You received this message because you are subscribed to a topic in the > Google Groups "python-ideas" group. > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/python-ideas/L5KfCEXFaII/unsubscribe. > To unsubscribe from this group and all its topics, send an email to > python-ideas+unsubscribe at googlegroups.com. > For more options, visit https://groups.google.com/d/optout. > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > > -- > > --- > You received this message because you are subscribed to a topic in the > Google Groups "python-ideas" group. > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/python-ideas/L5KfCEXFaII/unsubscribe. > To unsubscribe from this group and all its topics, send an email to > python-ideas+unsubscribe at googlegroups.com. > For more options, visit https://groups.google.com/d/optout. > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From storchaka at gmail.com Sat Mar 7 20:39:40 2015 From: storchaka at gmail.com (Serhiy Storchaka) Date: Sat, 07 Mar 2015 21:39:40 +0200 Subject: [Python-ideas] IntFlags In-Reply-To: References: <54FAB1CA.3000105@stoneleaf.us> <5C81EBF0-6229-452E-9B73-CBEB2B84FF6F@yahoo.com> <54FB13A1.5020008@stoneleaf.us> <54FB48A6.5080405@stoneleaf.us> Message-ID: On 07.03.15 21:02, Neil Girdhar wrote: > A drop-in replacement for what? Is there a lot of Python code that > manipulates flags that you expect people to convert to IntFlags from > int? Isn't the point of IntFlags for writing new code? No, the point of IntFlags to be drop-in replacement for integer constants used as flags. See my patch in issue23591. From g.brandl at gmx.net Sat Mar 7 21:28:14 2015 From: g.brandl at gmx.net (Georg Brandl) Date: Sat, 07 Mar 2015 21:28:14 +0100 Subject: [Python-ideas] IntFlags In-Reply-To: <54FB13A1.5020008@stoneleaf.us> References: <54FAB1CA.3000105@stoneleaf.us> <5C81EBF0-6229-452E-9B73-CBEB2B84FF6F@yahoo.com> <54FB13A1.5020008@stoneleaf.us> Message-ID: On 03/07/2015 04:05 PM, Ethan Furman wrote: >>> As long as we are dreaming :) >>> >>> class Stat(IntFlag): >>> RDONLY = 1 >>> NOSUID = 2 >>> NODEV = 4 >>> NOEXEC = 8 >>> SYNCHRONOUS = 16 >>> MANDLOCK = 64 >>> WRITE = 128 >>> APPEND = 256 >>> NOATIME = 1024 >>> NODIRATIME = 2048 >>> RELATIME = 4096 >>> >>> a = Stat.RDONLY # creates a new instance of Stat, not a singleton >> >> Why? > > Because by having mutable instances of Stat we can have more Python operations: > > instead of: > > x = Stat(some_Stat_value_from_somewhere) > x = x | Stat.NOEXEC # to set the bit > > we can say: > > x = Stat(some_Stat_value_from_somewhere) > x.NOEXEC = True Please no. You're making a mutable type out of something that is conceptually (and in people's minds) an integer. Remember how long it can take to understand that a = 1 a = 2 does not change the integer "1" to now be "2". Georg From ethan at stoneleaf.us Sat Mar 7 21:38:20 2015 From: ethan at stoneleaf.us (Ethan Furman) Date: Sat, 07 Mar 2015 12:38:20 -0800 Subject: [Python-ideas] IntFlags In-Reply-To: References: <54FAB1CA.3000105@stoneleaf.us> <5C81EBF0-6229-452E-9B73-CBEB2B84FF6F@yahoo.com> <54FB13A1.5020008@stoneleaf.us> Message-ID: <54FB61BC.8090107@stoneleaf.us> On 03/07/2015 12:28 PM, Georg Brandl wrote: > On 03/07/2015 04:05 PM, Ethan Furman wrote: > >>>> As long as we are dreaming :) >>>> >>>> class Stat(IntFlag): >>>> RDONLY = 1 >>>> NOSUID = 2 >>>> NODEV = 4 >>>> NOEXEC = 8 >>>> SYNCHRONOUS = 16 >>>> MANDLOCK = 64 >>>> WRITE = 128 >>>> APPEND = 256 >>>> NOATIME = 1024 >>>> NODIRATIME = 2048 >>>> RELATIME = 4096 >>>> >>>> a = Stat.RDONLY # creates a new instance of Stat, not a singleton >>> >>> Why? >> >> Because by having mutable instances of Stat we can have more Python operations: >> >> instead of: >> >> x = Stat(some_Stat_value_from_somewhere) >> x = x | Stat.NOEXEC # to set the bit >> >> we can say: >> >> x = Stat(some_Stat_value_from_somewhere) >> x.NOEXEC = True > > Please no. You're making a mutable type out of something that is conceptually > (and in people's minds) an integer. Remember how long it can take to understand > that > > a = 1 > a = 2 > > does not change the integer "1" to now be "2". Good point. To do something like that the name would have be BitFlags or something no so solidly tied to "immutable". At any rate, for this to work would require `int(x)` around every call to a lower-level API, and that's a non-starter. -- ~Ethan~ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: OpenPGP digital signature URL: From mistersheik at gmail.com Sun Mar 8 10:36:10 2015 From: mistersheik at gmail.com (Neil Girdhar) Date: Sun, 8 Mar 2015 05:36:10 -0400 Subject: [Python-ideas] IntFlags In-Reply-To: <54FB48A6.5080405@stoneleaf.us> References: <54FAB1CA.3000105@stoneleaf.us> <5C81EBF0-6229-452E-9B73-CBEB2B84FF6F@yahoo.com> <54FB13A1.5020008@stoneleaf.us> <54FB48A6.5080405@stoneleaf.us> Message-ID: On Sat, Mar 7, 2015 at 1:51 PM, Ethan Furman wrote: > On 03/07/2015 10:21 AM, Neil Girdhar wrote: > > On Sat, Mar 7, 2015 at 10:05 AM, Ethan Furman wrote: > > >> or, even more illustratively, instead of: > >> > >> x = Stat(some_Stat_value_from_somewhere) > >> x &= ~Stat.NOEXEC # to clear the bit > >> > >> we can have: > >> > >> x = Stat(some_Stat_value_from_somewhere) > >> x.NOEXEC = False # much more readable > > > > Yes, +1 > > >> Drat. Instances of Stat would not be immutuable, but that means they > can't be thin wrappers around `int`, doesn't it? > >> Which also means more work around C call sites. Drat and double-drat. > > > > Can't your C call sites cast the object to int? > > They could, but then you no longer have a drop-in replacement, which is > what IntEnum is. Plus it's a hassle. So > IntFlag (or whatever it's called) would to be immutable, which means no > neat tricks like `obj.bit_name = False`. > Looks like they already do cast to int: In [3]: class X: ...: def __int__(self): return 0 ...: In [4]: os.chmod('a', X()) I really don't see the problem. > > > Why do you need ~ at all? Do any API calls that you want to make want > the inverted > > flags? Isn't the only point of inverting the bits in order to clear a > field? > > Hopefully somebody else can address this point. > > -- > ~Ethan~ > > -- > > --- > You received this message because you are subscribed to a topic in the > Google Groups "python-ideas" group. > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/python-ideas/L5KfCEXFaII/unsubscribe. > To unsubscribe from this group and all its topics, send an email to > python-ideas+unsubscribe at googlegroups.com. > For more options, visit https://groups.google.com/d/optout. > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > > -- > > --- > You received this message because you are subscribed to a topic in the > Google Groups "python-ideas" group. > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/python-ideas/L5KfCEXFaII/unsubscribe. > To unsubscribe from this group and all its topics, send an email to > python-ideas+unsubscribe at googlegroups.com. > For more options, visit https://groups.google.com/d/optout. > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mistersheik at gmail.com Sun Mar 8 10:45:30 2015 From: mistersheik at gmail.com (Neil Girdhar) Date: Sun, 8 Mar 2015 05:45:30 -0400 Subject: [Python-ideas] IntFlags In-Reply-To: <54FB61BC.8090107@stoneleaf.us> References: <54FAB1CA.3000105@stoneleaf.us> <5C81EBF0-6229-452E-9B73-CBEB2B84FF6F@yahoo.com> <54FB13A1.5020008@stoneleaf.us> <54FB61BC.8090107@stoneleaf.us> Message-ID: On Sat, Mar 7, 2015 at 3:38 PM, Ethan Furman wrote: > On 03/07/2015 12:28 PM, Georg Brandl wrote: > > On 03/07/2015 04:05 PM, Ethan Furman wrote: > > > >>>> As long as we are dreaming :) > >>>> > >>>> class Stat(IntFlag): > >>>> RDONLY = 1 > >>>> NOSUID = 2 > >>>> NODEV = 4 > >>>> NOEXEC = 8 > >>>> SYNCHRONOUS = 16 > >>>> MANDLOCK = 64 > >>>> WRITE = 128 > >>>> APPEND = 256 > >>>> NOATIME = 1024 > >>>> NODIRATIME = 2048 > >>>> RELATIME = 4096 > >>>> > >>>> a = Stat.RDONLY # creates a new instance of Stat, not a singleton > >>> > >>> Why? > >> > >> Because by having mutable instances of Stat we can have more Python > operations: > >> > >> instead of: > >> > >> x = Stat(some_Stat_value_from_somewhere) > >> x = x | Stat.NOEXEC # to set the bit > >> > >> we can say: > >> > >> x = Stat(some_Stat_value_from_somewhere) > >> x.NOEXEC = True > > > > Please no. You're making a mutable type out of something that is > conceptually > > (and in people's minds) an integer. Remember how long it can take to > understand > > that > > > > a = 1 > > a = 2 > > > > does not change the integer "1" to now be "2". > The fact that the flags are stored in an int ? that is, the implementation has nothing at all to do with the conceptual nature of flags. Flags are properties. Your knowledge of the underlying implementation is misleading you. It would be just as easy and intuitive to implement those flags as keys and values in a dict. If flags were conceptually subtypes of int, then you should be able to do things like: flags ** 7 or flags // 91 Do you agree that this is totally meaningless? There is no "is a" relationship between Flags and int. There is a conversion between the conceptual mapping that is a Flags object to int for the sole purpose of calling into APIs. > Good point. To do something like that the name would have be BitFlags or > something no so solidly tied to "immutable". > At any rate, for this to work would require `int(x)` around every call to > a lower-level API, and that's a non-starter. > Correct me if I'm wrong, but doesn't Boost.Python when generating Python methods that accept ints, automatically call __int__ on the arguments. Doesn't SWIG do the same? Can you give me some examples of methods that accept ints but don't call __int__? It seems to me to be a bug in those methods than in the caller. A method that wants an int should call __int__ on its argument ? not expect that isinstance(X, int) etc. > > -- > ~Ethan~ > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Sun Mar 8 13:28:57 2015 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 8 Mar 2015 22:28:57 +1000 Subject: [Python-ideas] An even simpler customization of class creation In-Reply-To: References: <54F5EF25.4030002@stoneleaf.us> Message-ID: On 7 March 2015 at 00:11, Mark Young wrote: > Is that documented somewhere? I have never heard of that functionality. The implicit __class__ cell? PEP 3135 introduced the ability to call super() without any arguments in Python 3 to ask the interpreter to just "do the right thing": https://www.python.org/dev/peps/pep-3135/ The actual language reference docs are here: https://docs.python.org/3/reference/datamodel.html#creating-the-class-object Cheers, Nick. P.S. Those language reference class creation docs weren't fully updated for the original Python 3.0 release. They were updated in 3.3 when the new dynamic type creation APIs were added to the types module to simplify PEP 3115 compliant dynamic type creation: https://docs.python.org/3/library/types.html#dynamic-type-creation -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From g.rodola at gmail.com Sun Mar 8 14:07:10 2015 From: g.rodola at gmail.com (Giampaolo Rodola') Date: Sun, 8 Mar 2015 14:07:10 +0100 Subject: [Python-ideas] .from and .to instead of .encode and .decode In-Reply-To: References: Message-ID: On Sat, Mar 7, 2015 at 2:42 PM, Luciano Ramalho wrote: > On Sat, Mar 7, 2015 at 8:41 AM, Chris Angelico wrote: > > If it says "decode", the result is a Unicode string. If it says > > "encode", the result is bytes. I'm not sure what is difficult here. > > Yep. When I teach, I use this mnemonic, which I can now quote from my > book [1] ;-) > > [TIP] > ==== > If you need a memory aid to distinguish `.decode()` from `.encode()`, > convince yourself that a Unicode `str` contains "human" text, while > byte sequences can be cryptic machine core dumps. Therefore, it makes > sense that we *decode* `bytes` to `str` to get human readable text, > and we *encode* text to `bytes` for storage or transmission. > ==== This is a great advice (and yes, I also often get confused by the two). -- Giampaolo - http://grodola.blogspot.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From mistersheik at gmail.com Sun Mar 8 16:50:12 2015 From: mistersheik at gmail.com (Neil Girdhar) Date: Sun, 8 Mar 2015 08:50:12 -0700 (PDT) Subject: [Python-ideas] IntFlags In-Reply-To: References: <54FAB1CA.3000105@stoneleaf.us> <5C81EBF0-6229-452E-9B73-CBEB2B84FF6F@yahoo.com> <54FB13A1.5020008@stoneleaf.us> <54FB48A6.5080405@stoneleaf.us> Message-ID: <8ed6ef09-8203-4389-9378-e0dce0190382@googlegroups.com> For what it's worth, I really like your patch. My suggestion is to change a couple dozen lines so that IntFlags uses composition rather than inheritance, and provides setattr and getattr. As far as I can tell, this would still be a "drop-in replacement", although I'm still not sure in which contexts this has to work. Best, Neil On Saturday, March 7, 2015 at 2:40:29 PM UTC-5, Serhiy Storchaka wrote: > > On 07.03.15 21:02, Neil Girdhar wrote: > > A drop-in replacement for what? Is there a lot of Python code that > > manipulates flags that you expect people to convert to IntFlags from > > int? Isn't the point of IntFlags for writing new code? > > No, the point of IntFlags to be drop-in replacement for integer > constants used as flags. See my patch in issue23591. > > _______________________________________________ > Python-ideas mailing list > Python... at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From marky1991 at gmail.com Sun Mar 8 18:17:41 2015 From: marky1991 at gmail.com (Mark Young) Date: Sun, 8 Mar 2015 13:17:41 -0400 Subject: [Python-ideas] An even simpler customization of class creation In-Reply-To: References: <54F5EF25.4030002@stoneleaf.us> Message-ID: Thanks Nick! I swear I checked on the 3.x "Data Model" page, but I guess not. -------------- next part -------------- An HTML attachment was scrubbed... URL: From tjreedy at udel.edu Sun Mar 8 19:31:49 2015 From: tjreedy at udel.edu (Terry Reedy) Date: Sun, 08 Mar 2015 14:31:49 -0400 Subject: [Python-ideas] .from and .to instead of .encode and .decode In-Reply-To: References: Message-ID: On 3/8/2015 9:07 AM, Giampaolo Rodola' wrote: > On Sat, Mar 7, 2015 at 2:42 PM, Luciano Ramalho > [TIP] > ==== > If you need a memory aid to distinguish `.decode()` from `.encode()`, > convince yourself that a Unicode `str` contains "human" text, while > byte sequences can be cryptic machine core dumps. Therefore, it makes > sense that we *decode* `bytes` to `str` to get human readable text, > and we *encode* text to `bytes` for storage or transmission. > ==== > > > This is a great advice (and yes, I also often get confused by the two). Unless a bytes object simply represents a sequence of counts in range(256), which in unusual, then the bytes are an encoding of some other information, which in Python is embodied in an object. Struct.pack (en)codes python ints to bytes. Here the coding issue is big versus little endian. Struct.unpack decodes bytes back to int objects. PIL can decode (unpack and deserialize) bytes in various image encodings to an image object and can likewise encode an image object to bytes in various formats. The common theme is that non-bytes objects are encoded to bytes for transmission and storage and decoded back to non-bytes objects. -- Terry Jan Reedy From ncoghlan at gmail.com Mon Mar 9 01:12:38 2015 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 9 Mar 2015 10:12:38 +1000 Subject: [Python-ideas] An even simpler customization of class creation In-Reply-To: References: <54F5EF25.4030002@stoneleaf.us> Message-ID: On 9 Mar 2015 03:18, "Mark Young" wrote: > > Thanks Nick! I swear I checked on the 3.x "Data Model" page, but I guess not. Every other reference on that page is to the class attribute rather than the implicit method nonlocal, so it took a moment for me to relocate it myself. I should have searched for "super" instead :) Cheers, Nick. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Mon Mar 9 01:22:25 2015 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 9 Mar 2015 10:22:25 +1000 Subject: [Python-ideas] An even simpler customization of class creation In-Reply-To: References: <54F5EF25.4030002@stoneleaf.us> Message-ID: On 9 Mar 2015 10:12, "Nick Coghlan" wrote: > > > On 9 Mar 2015 03:18, "Mark Young" wrote: > > > > Thanks Nick! I swear I checked on the 3.x "Data Model" page, but I guess not. > > Every other reference on that page is to the class attribute rather than the implicit method nonlocal, so it took a moment for me to relocate it myself. I should have searched for "super" instead :) With the benefit of hindsight, if I had the chance to do this over, I'd suggest we call the implicit method nonlocal "__classdef__" instead, specifically to avoid the name collision with the runtime type information on object instances. My general impression is that the current name triggers too many incorrect assumptions for most readers, and the name collision makes it harder than it needs to be to build a more correct mental model. It would actually be possible to still make that transition, with a view to deprecating the current "__class__" spelling at some point in the distant future. I'm too close to the problem to judge whether or not it would actually help reduce confusion, though. Regards, Nick. > > Cheers, > Nick. -------------- next part -------------- An HTML attachment was scrubbed... URL: From abarnert at yahoo.com Mon Mar 9 13:29:21 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Mon, 9 Mar 2015 05:29:21 -0700 Subject: [Python-ideas] IntFlags In-Reply-To: References: <54FAB1CA.3000105@stoneleaf.us> <5C81EBF0-6229-452E-9B73-CBEB2B84FF6F@yahoo.com> <54FB13A1.5020008@stoneleaf.us> <54FB61BC.8090107@stoneleaf.us> Message-ID: On Mar 8, 2015, at 1:45 AM, Neil Girdhar wrote: > > > >> On Sat, Mar 7, 2015 at 3:38 PM, Ethan Furman wrote: >> On 03/07/2015 12:28 PM, Georg Brandl wrote: >> > On 03/07/2015 04:05 PM, Ethan Furman wrote: >> > >> >>>> As long as we are dreaming :) >> >>>> >> >>>> class Stat(IntFlag): >> >>>> RDONLY = 1 >> >>>> NOSUID = 2 >> >>>> NODEV = 4 >> >>>> NOEXEC = 8 >> >>>> SYNCHRONOUS = 16 >> >>>> MANDLOCK = 64 >> >>>> WRITE = 128 >> >>>> APPEND = 256 >> >>>> NOATIME = 1024 >> >>>> NODIRATIME = 2048 >> >>>> RELATIME = 4096 >> >>>> >> >>>> a = Stat.RDONLY # creates a new instance of Stat, not a singleton >> >>> >> >>> Why? >> >> >> >> Because by having mutable instances of Stat we can have more Python operations: >> >> >> >> instead of: >> >> >> >> x = Stat(some_Stat_value_from_somewhere) >> >> x = x | Stat.NOEXEC # to set the bit >> >> >> >> we can say: >> >> >> >> x = Stat(some_Stat_value_from_somewhere) >> >> x.NOEXEC = True >> > >> > Please no. You're making a mutable type out of something that is conceptually >> > (and in people's minds) an integer. Remember how long it can take to understand >> > that >> > >> > a = 1 >> > a = 2 >> > >> > does not change the integer "1" to now be "2". > > The fact that the flags are stored in an int ? that is, the implementation has nothing at all to do with the conceptual nature of flags. Flags are properties. Your knowledge of the underlying implementation is misleading you. It would be just as easy and intuitive to implement those flags as keys and values in a dict. Usually that's true. And when it's true, you use a dict (or a set or a namedtuple or whatever's appropriate), so you have no need for IntFlags in the first place. The only reason you'd ever want this class is when the fact that the flags are stored in an int is important. Sure, an IntFlags that isn't a subclass on int can define an __int__ method. But so can some class that doesn't use an int for storage in the first place. If you want to convert back and forth, that's perfectly fine, but why are you using an int for storage? > If flags were conceptually subtypes of int, then you should be able to do things like: > > flags ** 7 > > or > > flags // 91 > > Do you agree that this is totally meaningless? There is no "is a" relationship between Flags and int. There is a conversion between the conceptual mapping that is a Flags object to int for the sole purpose of calling into APIs. > >> >> Good point. To do something like that the name would have be BitFlags or something no so solidly tied to "immutable". >> At any rate, for this to work would require `int(x)` around every call to a lower-level API, and that's a non-starter. > > Correct me if I'm wrong, but doesn't Boost.Python when generating Python methods that accept ints, automatically call __int__ on the arguments. Doesn't SWIG do the same? Can you give me some examples of methods that accept ints but don't call __int__? It seems to me to be a bug in those methods than in the caller. A method that wants an int should call __int__ on its argument ? not expect that isinstance(X, int) etc. > >> >> -- >> ~Ethan~ > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From abarnert at yahoo.com Mon Mar 9 13:40:03 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Mon, 9 Mar 2015 05:40:03 -0700 Subject: [Python-ideas] An even simpler customization of class creation In-Reply-To: References: <54F5EF25.4030002@stoneleaf.us> Message-ID: <1719F758-5F28-457A-A609-F9E7CADA2A1A@yahoo.com> On Mar 8, 2015, at 5:22 PM, Nick Coghlan wrote: > > On 9 Mar 2015 10:12, "Nick Coghlan" wrote: > > > > > > On 9 Mar 2015 03:18, "Mark Young" wrote: > > > > > > Thanks Nick! I swear I checked on the 3.x "Data Model" page, but I guess not. > > > > Every other reference on that page is to the class attribute rather than the implicit method nonlocal, so it took a moment for me to relocate it myself. I should have searched for "super" instead :) > > With the benefit of hindsight, if I had the chance to do this over, I'd suggest we call the implicit method nonlocal "__classdef__" instead, specifically to avoid the name collision with the runtime type information on object instances. My general impression is that the current name triggers too many incorrect assumptions for most readers, and the name collision makes it harder than it needs to be to build a more correct mental model. > I can tell you that every single time I've mentioned the __class__ nonlocal in a StackOverflow answer, someone mistakenly "corrected" me to tell me either "you need to write self.__class__" or "that won't work because it's the runtime type of self, which could be a subclass". I think a better solution, if you were doing it from scratch, might be to get rid of the __class__ member and force people to use type(self) to get it, so TOOTDI. (Of course that implies that code that needs to change __class__ would have to call some function to do so, but I'm not sure that's a bad thing. Dynamically changing your type is a rare thing to do in Python, and unusual or impossible in most other OO languages, except maybe during initialization in the Smalltalk family, so why shouldn't it be signaled by something scary?) On the other hand, is __class__ the only thing that's spelled with double underscored that isn't an attribute? Maybe that's the part that's confusing people... > It would actually be possible to still make that transition, with a view to deprecating the current "__class__" spelling at some point in the distant future. I'm too close to the problem to judge whether or not it would actually help reduce confusion, though. > > Regards, > Nick. > > > > > Cheers, > > Nick. > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Mon Mar 9 14:36:59 2015 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 9 Mar 2015 23:36:59 +1000 Subject: [Python-ideas] An even simpler customization of class creation In-Reply-To: <1719F758-5F28-457A-A609-F9E7CADA2A1A@yahoo.com> References: <54F5EF25.4030002@stoneleaf.us> <1719F758-5F28-457A-A609-F9E7CADA2A1A@yahoo.com> Message-ID: On 9 March 2015 at 22:40, Andrew Barnert wrote: > On Mar 8, 2015, at 5:22 PM, Nick Coghlan wrote: > > > On 9 Mar 2015 10:12, "Nick Coghlan" wrote: >> >> >> On 9 Mar 2015 03:18, "Mark Young" wrote: >> > >> > Thanks Nick! I swear I checked on the 3.x "Data Model" page, but I guess >> > not. >> >> Every other reference on that page is to the class attribute rather than >> the implicit method nonlocal, so it took a moment for me to relocate it >> myself. I should have searched for "super" instead :) > > With the benefit of hindsight, if I had the chance to do this over, I'd > suggest we call the implicit method nonlocal "__classdef__" instead, > specifically to avoid the name collision with the runtime type information > on object instances. My general impression is that the current name triggers > too many incorrect assumptions for most readers, and the name collision > makes it harder than it needs to be to build a more correct mental model. > > I can tell you that every single time I've mentioned the __class__ nonlocal > in a StackOverflow answer, someone mistakenly "corrected" me to tell me > either "you need to write self.__class__" or "that won't work because it's > the runtime type of self, which could be a subclass". > > I think a better solution, if you were doing it from scratch, might be to > get rid of the __class__ member and force people to use type(self) to get > it, so TOOTDI. They're not substitutes for each other, as __class__ lets you lie about your type to the consumer, while type() doesn't, and this is by design: >>> import weakref >>> class Example(): ... pass ... >>> a = Example() >>> b = weakref.proxy(a) >>> b.__class__ >>> type(b) For many use cases, you actually want obj.__class__ in order to handle proxy objects correctly. > (Of course that implies that code that needs to change > __class__ would have to call some function to do so, but I'm not sure that's > a bad thing. Dynamically changing your type is a rare thing to do in Python, > and unusual or impossible in most other OO languages, except maybe during > initialization in the Smalltalk family, so why shouldn't it be signaled by > something scary?) It's not (just) about being able to dynamically change your type, it's about the fact that __class__ goes through the descriptor and class attribute lookup machinery, while type() doesn't. > > On the other hand, is __class__ the only thing that's spelled with double > underscored that isn't an attribute? Maybe that's the part that's confusing > people... It's not the only one - implicit module attributes like __name__ work that way, as do the __debug__ and __import__ builtins. As far as I can tell, it's specifically the fact that "__class__" looks like a plausible typo for "self.__class__" that causes problems, as this also creates the problem that if you search for "__class__" not only on the data model page, but anywhere, you're far more likely to get a hit on the attribute (which has been around for as long as Python has had classes) than you will on the implicit method nonlocal introduced in Python 3. Hence why I suspect changing/aliasing the preferred name for referencing the nonlocal to be __classdef__ may help improve things, as it better hints at the lexical scoping, where you're referring to the class where the method is being defined at compile time, rather than the one where it was retrieved at run time. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From mistersheik at gmail.com Mon Mar 9 14:41:57 2015 From: mistersheik at gmail.com (Neil Girdhar) Date: Mon, 9 Mar 2015 09:41:57 -0400 Subject: [Python-ideas] IntFlags In-Reply-To: References: <54FAB1CA.3000105@stoneleaf.us> <5C81EBF0-6229-452E-9B73-CBEB2B84FF6F@yahoo.com> <54FB13A1.5020008@stoneleaf.us> <54FB61BC.8090107@stoneleaf.us> Message-ID: On Mon, Mar 9, 2015 at 8:29 AM, Andrew Barnert wrote: > On Mar 8, 2015, at 1:45 AM, Neil Girdhar wrote: > > > > On Sat, Mar 7, 2015 at 3:38 PM, Ethan Furman wrote: > >> On 03/07/2015 12:28 PM, Georg Brandl wrote: >> > On 03/07/2015 04:05 PM, Ethan Furman wrote: >> > >> >>>> As long as we are dreaming :) >> >>>> >> >>>> class Stat(IntFlag): >> >>>> RDONLY = 1 >> >>>> NOSUID = 2 >> >>>> NODEV = 4 >> >>>> NOEXEC = 8 >> >>>> SYNCHRONOUS = 16 >> >>>> MANDLOCK = 64 >> >>>> WRITE = 128 >> >>>> APPEND = 256 >> >>>> NOATIME = 1024 >> >>>> NODIRATIME = 2048 >> >>>> RELATIME = 4096 >> >>>> >> >>>> a = Stat.RDONLY # creates a new instance of Stat, not a singleton >> >>> >> >>> Why? >> >> >> >> Because by having mutable instances of Stat we can have more Python >> operations: >> >> >> >> instead of: >> >> >> >> x = Stat(some_Stat_value_from_somewhere) >> >> x = x | Stat.NOEXEC # to set the bit >> >> >> >> we can say: >> >> >> >> x = Stat(some_Stat_value_from_somewhere) >> >> x.NOEXEC = True >> > >> > Please no. You're making a mutable type out of something that is >> conceptually >> > (and in people's minds) an integer. Remember how long it can take to >> understand >> > that >> > >> > a = 1 >> > a = 2 >> > >> > does not change the integer "1" to now be "2". >> > > The fact that the flags are stored in an int ? that is, the implementation > has nothing at all to do with the conceptual nature of flags. Flags are > properties. Your knowledge of the underlying implementation is misleading > you. It would be just as easy and intuitive to implement those flags as > keys and values in a dict. > > > Usually that's true. And when it's true, you use a dict (or a set or a > namedtuple or whatever's appropriate), so you have no need for IntFlags in > the first place. The only reason you'd ever want this class is when the > fact that the flags are stored in an int is important. > > Sure, an IntFlags that isn't a subclass on int can define an __int__ > method. But so can some class that doesn't use an int for storage in the > first place. If you want to convert back and forth, that's perfectly fine, > but why are you using an int for storage? > I would word it this way: the only time you use IntFlags is when you need access to an integer representation. The implementation is free to do what it likes provided it meets the interface guarantees. You're not "using an int for storage" and interfaces should never make implementation promises. They should only make interface guarantees. > > If flags were conceptually subtypes of int, then you should be able to do > things like: > > flags ** 7 > > or > > flags // 91 > > Do you agree that this is totally meaningless? There is no "is a" > relationship between Flags and int. There is a conversion between the > conceptual mapping that is a Flags object to int for the sole purpose of > calling into APIs. > > In my opinion, this is the most important point when deciding on inheritance. > > >> Good point. To do something like that the name would have be BitFlags or >> something no so solidly tied to "immutable". >> At any rate, for this to work would require `int(x)` around every call to >> a lower-level API, and that's a non-starter. >> > > Correct me if I'm wrong, but doesn't Boost.Python when generating Python > methods that accept ints, automatically call __int__ on the arguments. > Doesn't SWIG do the same? Can you give me some examples of methods that > accept ints but don't call __int__? It seems to me to be a bug in those > methods than in the caller. A method that wants an int should call __int__ > on its argument ? not expect that isinstance(X, int) etc. > > >> >> -- >> ~Ethan~ >> >> _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From random832 at fastmail.us Mon Mar 9 14:49:35 2015 From: random832 at fastmail.us (random832 at fastmail.us) Date: Mon, 09 Mar 2015 09:49:35 -0400 Subject: [Python-ideas] IntFlags In-Reply-To: References: <54FAB1CA.3000105@stoneleaf.us> <5C81EBF0-6229-452E-9B73-CBEB2B84FF6F@yahoo.com> <54FB13A1.5020008@stoneleaf.us> <54FB61BC.8090107@stoneleaf.us> Message-ID: <1425908975.1315177.237847797.212D6AA4@webmail.messagingengine.com> On Sun, Mar 8, 2015, at 05:45, Neil Girdhar wrote: > If flags were conceptually subtypes of int, then you should be able to do > things like: > > flags ** 7 > > or > > flags // 91 *cough* bool *cough* > > Do you agree that this is totally meaningless? There is no "is a" > relationship between Flags and int. There is a conversion between the > conceptual mapping that is a Flags object to int for the sole purpose of > calling into APIs. From mistersheik at gmail.com Mon Mar 9 15:00:20 2015 From: mistersheik at gmail.com (Neil Girdhar) Date: Mon, 9 Mar 2015 10:00:20 -0400 Subject: [Python-ideas] IntFlags In-Reply-To: <1425908975.1315177.237847797.212D6AA4@webmail.messagingengine.com> References: <54FAB1CA.3000105@stoneleaf.us> <5C81EBF0-6229-452E-9B73-CBEB2B84FF6F@yahoo.com> <54FB13A1.5020008@stoneleaf.us> <54FB61BC.8090107@stoneleaf.us> <1425908975.1315177.237847797.212D6AA4@webmail.messagingengine.com> Message-ID: People have been discussing bool since it was introduced. Many people have proposed that bool should not subclass int exactly for this reason (the Liskov substitution principle), but unfortunately bool evolved from int in Python 2.2.1 (I think?) when there was already a lot of code using int to implement Boolean variables with constants like False, True = 0, 1. In order not to break too much code, it was decided to allow bool to inherit from int. The inheritance of bool from int is not a good precedent when making inheritance decisions in the future. On Mon, Mar 9, 2015 at 9:49 AM, wrote: > On Sun, Mar 8, 2015, at 05:45, Neil Girdhar wrote: > > If flags were conceptually subtypes of int, then you should be able to do > > things like: > > > > flags ** 7 > > > > or > > > > flags // 91 > > *cough* bool *cough* > > > > > Do you agree that this is totally meaningless? There is no "is a" > > relationship between Flags and int. There is a conversion between the > > conceptual mapping that is a Flags object to int for the sole purpose of > > calling into APIs. > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > > -- > > --- > You received this message because you are subscribed to a topic in the > Google Groups "python-ideas" group. > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/python-ideas/L5KfCEXFaII/unsubscribe. > To unsubscribe from this group and all its topics, send an email to > python-ideas+unsubscribe at googlegroups.com. > For more options, visit https://groups.google.com/d/optout. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From random832 at fastmail.us Mon Mar 9 16:01:12 2015 From: random832 at fastmail.us (random832 at fastmail.us) Date: Mon, 09 Mar 2015 11:01:12 -0400 Subject: [Python-ideas] IntFlags In-Reply-To: References: <54FAB1CA.3000105@stoneleaf.us> <5C81EBF0-6229-452E-9B73-CBEB2B84FF6F@yahoo.com> <54FB13A1.5020008@stoneleaf.us> <54FB61BC.8090107@stoneleaf.us> <1425908975.1315177.237847797.212D6AA4@webmail.messagingengine.com> Message-ID: <1425913272.1336142.237883657.1A24DE1B@webmail.messagingengine.com> On Mon, Mar 9, 2015, at 10:00, Neil Girdhar wrote: > People have been discussing bool since it was introduced. Many people > have > proposed that bool should not subclass int exactly for this reason (the > Liskov substitution principle), but unfortunately bool evolved from int > in > Python 2.2.1 (I think?) when there was already a lot of code using int to > implement Boolean variables with constants like False, True = 0, 1. In > order not to break too much code, it was decided to allow bool to inherit > from int. The inheritance of bool from int is not a good precedent when > making inheritance decisions in the future. And we're already using int in all these places people are talking about dropping this new IntFlags class into. The situations seem exactly analogous to me. From mistersheik at gmail.com Mon Mar 9 16:54:07 2015 From: mistersheik at gmail.com (Neil Girdhar) Date: Mon, 9 Mar 2015 11:54:07 -0400 Subject: [Python-ideas] IntFlags In-Reply-To: <1425913272.1336142.237883657.1A24DE1B@webmail.messagingengine.com> References: <54FAB1CA.3000105@stoneleaf.us> <5C81EBF0-6229-452E-9B73-CBEB2B84FF6F@yahoo.com> <54FB13A1.5020008@stoneleaf.us> <54FB61BC.8090107@stoneleaf.us> <1425908975.1315177.237847797.212D6AA4@webmail.messagingengine.com> <1425913272.1336142.237883657.1A24DE1B@webmail.messagingengine.com> Message-ID: The differences are that the flags are used in a tiny portion of Python code, and of that tiny amount of code, 99% of it probably uses nothing more than the minimum "flags interface" that the composition solution has. bool was not like that. Changing bool to be its own type would led to many more problems. On Mon, Mar 9, 2015 at 11:01 AM, wrote: > On Mon, Mar 9, 2015, at 10:00, Neil Girdhar wrote: > > People have been discussing bool since it was introduced. Many people > > have > > proposed that bool should not subclass int exactly for this reason (the > > Liskov substitution principle), but unfortunately bool evolved from int > > in > > Python 2.2.1 (I think?) when there was already a lot of code using int to > > implement Boolean variables with constants like False, True = 0, 1. In > > order not to break too much code, it was decided to allow bool to inherit > > from int. The inheritance of bool from int is not a good precedent when > > making inheritance decisions in the future. > > And we're already using int in all these places people are talking about > dropping this new IntFlags class into. The situations seem exactly > analogous to me. > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > > -- > > --- > You received this message because you are subscribed to a topic in the > Google Groups "python-ideas" group. > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/python-ideas/L5KfCEXFaII/unsubscribe. > To unsubscribe from this group and all its topics, send an email to > python-ideas+unsubscribe at googlegroups.com. > For more options, visit https://groups.google.com/d/optout. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rymg19 at gmail.com Mon Mar 9 18:40:11 2015 From: rymg19 at gmail.com (Ryan Gonzalez) Date: Mon, 9 Mar 2015 12:40:11 -0500 Subject: [Python-ideas] CPython needs to revamp/replace it's build system && Cross-compiling Python sucks Message-ID: I've been trying to work on Issue 23496. And want to tear my hair out. I've noticed a few problems with Python's current build system: - There is no separation of the host C compiler and the target C compiler. When you give configure a C compiler for cross-compiling, make will happily try to build Parser/pgen with the cross-compiler. And try to run the generated executable. And fail. I know you can use `make touch` to get around this, but it still sucks. - For some reason, autoconf decided to re-configure everything about 2 minutes into make. But it didn't pass any of the flags I needed, resulting in everything failing. - I do not like having to use this for configuring: PYTHON_FOR_BUILD=`realpath ../cpython/python` CC=arm-linux-androideabi-clang CXX=arm-linux-androideabi-clang++ ./configure --host=arm-linux-androideabi --build=x86_64 --disable-ipv6 ac_cv_file__dev_ptmx=no ac_cv_file__dev_ptc=no ac_cv_have_long_long_format=yes At the same time, I'm not quite sure how many people would want to sift through the 5000 lines in configure.ac or the 1650 lines in Makefile.pre.in. I just get the feeling that, whenever I build Python, there are too many holes/hacks to get around make/autoconf things. -- Ryan If I were in a 10-story building glass-sided building and forced to write either Go or autconf scripts, I?d jump out a window. http://kirbyfan64.github.io/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From russell at keith-magee.com Tue Mar 10 01:57:54 2015 From: russell at keith-magee.com (Russell Keith-Magee) Date: Tue, 10 Mar 2015 08:57:54 +0800 Subject: [Python-ideas] CPython needs to revamp/replace it's build system && Cross-compiling Python sucks In-Reply-To: References: Message-ID: Hi Ryan, First off I'm no fan of autoconf, but... On Tue, Mar 10, 2015 at 1:40 AM, Ryan Gonzalez wrote: > I've been trying to work on Issue 23496. And want to tear my hair out. > I've noticed a few problems with Python's current build system: > > - There is no separation of the host C compiler and the target C compiler. > When you give configure a C compiler for cross-compiling, make will happily > try to build Parser/pgen with the cross-compiler. And try to run the > generated executable. And fail. I know you can use `make touch` to get > around this, but it still sucks. > True there is no separation of host/target compilers, but are you sure this matters? In my cross-compiling for iOS, the pgen steps are skipped -- Parser/pgen is compiled, but not ever used. Are you sure that this failed execution isn't an artefact of identifying the Android build a "linux"? I've had to make a bunch of small modifications to configure.ac to make the iOS build process not build and run things that won't work on the build platform. > - For some reason, autoconf decided to re-configure everything about 2 > minutes into make. But it didn't pass any of the flags I needed, resulting > in everything failing. > That's the step compiling libffi/ctypes. libffi is vendored into the Python source, and the mid-build configure step is the configuration of that vendored source code. You'll see the mid-build configure execute regardless of whether you're cross-compiling or not -- unless you're for OS/X, because the build instructions for OS/X libffi are hard-coded into setup.py. > - I do not like having to use this for configuring: > > PYTHON_FOR_BUILD=`realpath ../cpython/python` > CC=arm-linux-androideabi-clang CXX=arm-linux-androideabi-clang++ > ./configure --host=arm-linux-androideabi --build=x86_64 --disable-ipv6 > ac_cv_file__dev_ptmx=no ac_cv_file__dev_ptc=no > ac_cv_have_long_long_format=yes > > The PYTHON_FOR_BUILD specification *shouldn't* be necessary. If you're compiling Python 3.X, as long as the build platform has a python3.X executable on the path, PYTHON_FOR_BUILD should auto detect. It's only necessary to manually specify PYTHON_FOR_BUILD if you *haven't* got an appropriate python in your path. ac_cv_file__dev_ptmx=no and ac_cv_file__dev_ptc=no are manually specified flags because they can't be auto detected on a cross-compile (at least, not in any way I can think of). They're checking whether the *target* device has /dev/ptc and /dev/ptmx. In the case of Android/iOS, you could probably hard-code the responses on the basis that the target platform is a known quantity, but in the general case (which is the current case), you can't auto detect the properties of the target platform. I can't comment on ac_cv_have_long_long_format, but I'm guessing the cause is the same - you can't compile and run the configure test program. --disable-ipv6 is a feature flag - I don't know why it's necessary on Android. At the same time, I'm not quite sure how many people would want to sift > through the 5000 lines in configure.ac or the 1650 lines in > Makefile.pre.in. > > I just get the feeling that, whenever I build Python, there are too many > holes/hacks to get around make/autoconf things. > Sure - configure sucks. What's the alternative? Sometimes things are complex because... they're complex. This is a build system that will support building everything from AiX to Android, and in *most* cases, with little more than a ./configure && make && make install. Android (and other cross compiling cases) are a little more complex because you have to *tell* configure what the target looks like, because it can't detect anything about the platform. Yours, Russ Magee %-) -------------- next part -------------- An HTML attachment was scrubbed... URL: From rymg19 at gmail.com Tue Mar 10 03:43:46 2015 From: rymg19 at gmail.com (Ryan) Date: Mon, 09 Mar 2015 21:43:46 -0500 Subject: [Python-ideas] CPython needs to revamp/replace it's build system && Cross-compiling Python sucks In-Reply-To: References: Message-ID: <49c23aa9-51cf-47b7-843b-8c4b58d01cbd@email.android.com> Russell Keith-Magee wrote: >Hi Ryan, > >First off I'm no fan of autoconf, but... > Who is other than the one who designed it? :) >On Tue, Mar 10, 2015 at 1:40 AM, Ryan Gonzalez >wrote: > >> I've been trying to work on Issue 23496. And want to tear my hair >out. >> I've noticed a few problems with Python's current build system: >> >> - There is no separation of the host C compiler and the target C >compiler. >> When you give configure a C compiler for cross-compiling, make will >happily >> try to build Parser/pgen with the cross-compiler. And try to run the >> generated executable. And fail. I know you can use `make touch` to >get >> around this, but it still sucks. >> > >True there is no separation of host/target compilers, but are you sure >this >matters? In my cross-compiling for iOS, the pgen steps are skipped -- >Parser/pgen is compiled, but not ever used. Are you sure that this >failed >execution isn't an artefact of identifying the Android build a "linux"? >I've had to make a bunch of small modifications to configure.ac to make >the >iOS build process not build and run things that won't work on the build >platform. > Hmmm... But you gave an example. You shouldn't need to edit configure.ac to not try to run what can't be run because the host != the builder. > >> - For some reason, autoconf decided to re-configure everything about >2 >> minutes into make. But it didn't pass any of the flags I needed, >resulting >> in everything failing. >> > >That's the step compiling libffi/ctypes. libffi is vendored into the >Python >source, and the mid-build configure step is the configuration of that >vendored source code. You'll see the mid-build configure execute >regardless >of whether you're cross-compiling or not -- unless you're for OS/X, >because >the build instructions for OS/X libffi are hard-coded into setup.py. > Still sucks. > >> - I do not like having to use this for configuring: >> >> PYTHON_FOR_BUILD=`realpath ../cpython/python` >> CC=arm-linux-androideabi-clang CXX=arm-linux-androideabi-clang++ >> ./configure --host=arm-linux-androideabi --build=x86_64 >--disable-ipv6 >> ac_cv_file__dev_ptmx=no ac_cv_file__dev_ptc=no >> ac_cv_have_long_long_format=yes >> >> >The PYTHON_FOR_BUILD specification *shouldn't* be necessary. If you're >compiling Python 3.X, as long as the build platform has a python3.X >executable on the path, PYTHON_FOR_BUILD should auto detect. It's only >necessary to manually specify PYTHON_FOR_BUILD if you *haven't* got an >appropriate python in your path. The build was giving errors because apparently 3.4's _sre module doesn't have some constant that the build script needs (MAXGROUPS). > >ac_cv_file__dev_ptmx=no and ac_cv_file__dev_ptc=no are manually >specified >flags because they can't be auto detected on a cross-compile (at least, >not >in any way I can think of). They're checking whether the *target* >device >has /dev/ptc and /dev/ptmx. In the case of Android/iOS, you could >probably >hard-code the responses on the basis that the target platform is a >known >quantity, but in the general case (which is the current case), you >can't >auto detect the properties of the target platform. It took me a while to figure out that that was the option I needed to pass, though. > >I can't comment on ac_cv_have_long_long_format, but I'm guessing the >cause >is the same - you can't compile and run the configure test program. > >--disable-ipv6 is a feature flag - I don't know why it's necessary on >Android. I don't really know why, either. Something to do with a getaddrline bug. > >At the same time, I'm not quite sure how many people would want to sift >> through the 5000 lines in configure.ac or the 1650 lines in >> Makefile.pre.in. >> >> I just get the feeling that, whenever I build Python, there are too >many >> holes/hacks to get around make/autoconf things. >> > >Sure - configure sucks. What's the alternative? All 1000 other build systems out there? > >Sometimes things are complex because... they're complex. This is a >build >system that will support building everything from AiX to Android, and >in >*most* cases, with little more than a ./configure && make && make >install. >Android (and other cross compiling cases) are a little more complex >because >you have to *tell* configure what the target looks like, because it >can't >detect anything about the platform. But autotools tends to make things more complex than needed. When Qt (or was it KDE?) moved to CMake from autotools, the build script size dropped a large amount. Just take a look at the code in configure.ac that manually detects the compiler on OSX systems because autoconf SOMEHOW manages to get it wrong. > >Yours, >Russ Magee %-) -- Sent from my Android phone with K-9 Mail. Please excuse my brevity. Check out my website: http://kirbyfan64.github.io/ From russell at keith-magee.com Tue Mar 10 12:20:42 2015 From: russell at keith-magee.com (Russell Keith-Magee) Date: Tue, 10 Mar 2015 19:20:42 +0800 Subject: [Python-ideas] CPython needs to revamp/replace it's build system && Cross-compiling Python sucks In-Reply-To: <49c23aa9-51cf-47b7-843b-8c4b58d01cbd@email.android.com> References: <49c23aa9-51cf-47b7-843b-8c4b58d01cbd@email.android.com> Message-ID: On Tue, Mar 10, 2015 at 10:43 AM, Ryan wrote: > > > Russell Keith-Magee wrote: > >Hi Ryan, > > > >First off I'm no fan of autoconf, but... > > > > Who is other than the one who designed it? :) > Let me be clear - there's a difference between "not a fan" and "calling for it's head". Autoconf is crufty, annoying, and sometimes aggravating - but it *works*, and has had some pretty seriously battle testing - both in the abstract "autoconf as a product", and in the specific "autoconf as used in Python". This isn't something that can be said of most of the alternatives in the abstract case, or *any* of the alternatives in the Python case. >On Tue, Mar 10, 2015 at 1:40 AM, Ryan Gonzalez > >wrote: > > > >> I've been trying to work on Issue 23496. And want to tear my hair > >out. > >> I've noticed a few problems with Python's current build system: > >> > >> - There is no separation of the host C compiler and the target C > >compiler. > >> When you give configure a C compiler for cross-compiling, make will > >happily > >> try to build Parser/pgen with the cross-compiler. And try to run the > >> generated executable. And fail. I know you can use `make touch` to > >get > >> around this, but it still sucks. > >> > > > >True there is no separation of host/target compilers, but are you sure > >this > >matters? In my cross-compiling for iOS, the pgen steps are skipped -- > >Parser/pgen is compiled, but not ever used. Are you sure that this > >failed > >execution isn't an artefact of identifying the Android build a "linux"? > >I've had to make a bunch of small modifications to configure.ac to make > >the > >iOS build process not build and run things that won't work on the build > >platform. > > > > Hmmm... > > But you gave an example. You shouldn't need to edit configure.ac to not > try to run what can't be run because the host != the builder. > Agreed. My claim is that the behaviour you're seeing is a bug in the existing configure script, probably born of the fact that using the "linux" system type for Android confuses the build system into thinking that the host executable will run on the build machine. I suspect that the fix will be quite simple, and will reveal itself almost immediately if you build with --host=armv7-google-android instead of --host=armv7-linux-gnu or --host=armv7-linux-android The existence of bugs isn't a reason to replace the entire build system, unless you've got good reason to believe that a new build system won't be subject to the same class of bugs. >> - For some reason, autoconf decided to re-configure everything about > >2 > >> minutes into make. But it didn't pass any of the flags I needed, > >resulting > >> in everything failing. > >> > > > >That's the step compiling libffi/ctypes. libffi is vendored into the > >Python > >source, and the mid-build configure step is the configuration of that > >vendored source code. You'll see the mid-build configure execute > >regardless > >of whether you're cross-compiling or not -- unless you're for OS/X, > >because > >the build instructions for OS/X libffi are hard-coded into setup.py. > > > > Still sucks. > I don't see why. libffi is a complex piece of machinery. Every platform/CPU architecture needs a different set of files to be compiled and included. libffi's sources includes those definitions. The alternative is hard-coding every possible target platform. Using configure (the same configure script that libffi ships, I might add) means this isn't required. I'd also add that the other alternative is to configure using --with-system-ffi, and ensure there is an appropriate libffi.so in your path, etc. No internal configure step is required in this case. >> - I do not like having to use this for configuring: > >> > >> PYTHON_FOR_BUILD=`realpath ../cpython/python` > >> CC=arm-linux-androideabi-clang CXX=arm-linux-androideabi-clang++ > >> ./configure --host=arm-linux-androideabi --build=x86_64 > >--disable-ipv6 > >> ac_cv_file__dev_ptmx=no ac_cv_file__dev_ptc=no > >> ac_cv_have_long_long_format=yes > >> > >> > >The PYTHON_FOR_BUILD specification *shouldn't* be necessary. If you're > >compiling Python 3.X, as long as the build platform has a python3.X > >executable on the path, PYTHON_FOR_BUILD should auto detect. It's only > >necessary to manually specify PYTHON_FOR_BUILD if you *haven't* got an > >appropriate python in your path. > > The build was giving errors because apparently 3.4's _sre module doesn't > have some constant that the build script needs (MAXGROUPS). > > > > >ac_cv_file__dev_ptmx=no and ac_cv_file__dev_ptc=no are manually > >specified > >flags because they can't be auto detected on a cross-compile (at least, > >not > >in any way I can think of). They're checking whether the *target* > >device > >has /dev/ptc and /dev/ptmx. In the case of Android/iOS, you could > >probably > >hard-code the responses on the basis that the target platform is a > >known > >quantity, but in the general case (which is the current case), you > >can't > >auto detect the properties of the target platform. > > It took me a while to figure out that that was the option I needed to > pass, though. > Granted it isn't obvious. However, I'd suggest the logical extension of that argument is "Lets document the existing build system", not "Lets replace the build system with something new". >I can't comment on ac_cv_have_long_long_format, but I'm guessing the > >cause > >is the same - you can't compile and run the configure test program. > > > >--disable-ipv6 is a feature flag - I don't know why it's necessary on > >Android. > > I don't really know why, either. Something to do with a getaddrline bug. > Sure. And if it turns out that there's a good systemic reason why ipv6 won't ever work on Android, it's a couple of line patch to configure.ac (around line 2666) to fix the problem permanently, removing the need for the manual flag to be specified. >At the same time, I'm not quite sure how many people would want to sift > >> through the 5000 lines in configure.ac or the 1650 lines in > >> Makefile.pre.in. > >> > >> I just get the feeling that, whenever I build Python, there are too > >many > >> holes/hacks to get around make/autoconf things. > >> > > > >Sure - configure sucks. What's the alternative? > > All 1000 other build systems out there? > Care to state a specific preference, rather than engage in hyperbole? >Sometimes things are complex because... they're complex. This is a > >build > >system that will support building everything from AiX to Android, and > >in > >*most* cases, with little more than a ./configure && make && make > >install. > >Android (and other cross compiling cases) are a little more complex > >because > >you have to *tell* configure what the target looks like, because it > >can't > >detect anything about the platform. > > But autotools tends to make things more complex than needed. When Qt (or > was it KDE?) moved to CMake from autotools, the build script size dropped a > large amount. > > Just take a look at the code in configure.ac that manually detects the > compiler on OSX systems because autoconf SOMEHOW manages to get it wrong. > Wow - who would have thought that a complex set of build requirements that have to work on a wide range of platforms and has been developed over 20+ years would occasionally yield edge cases? Seriously, if your claim is that moving to a new build system will remove the existence of edge cases, I want some of what you are smoking. To address the specific example you gave: OSX is a particularly complex platform to support because it spans 2 completely different CPU architectures, has the concept of Framework builds that other platforms don't have, and has, at various time, used clang, llvm-gcc and gcc as the default compiler. Net result: AC_PROG_CC autoconf tool doesn't handle this complexity well. But then, most build systems don't. However, Autoconf is rich enough that it is possible to work around this problem. I can't speak for the core team, but from my perspective, this sounds like a proposal where the only argument that is going to work is a working (or mostly working) port of the existing build system that proves it is possible to support all the existing edge cases in a much simpler fashion. For my money - of all the problems I've hit trying to port Python to iOS, autoconf is the least of them. No, it isn't my favourite tool - but for all it's faults, it works. If you want to convince me that there is a better alternative, you're going to need to do more than just repeating that "autoconf sucks". Yours, Russ Magee %-) -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Tue Mar 10 13:51:30 2015 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 10 Mar 2015 22:51:30 +1000 Subject: [Python-ideas] IntFlags In-Reply-To: References: <54FAB1CA.3000105@stoneleaf.us> <5C81EBF0-6229-452E-9B73-CBEB2B84FF6F@yahoo.com> <54FB13A1.5020008@stoneleaf.us> <54FB61BC.8090107@stoneleaf.us> <1425908975.1315177.237847797.212D6AA4@webmail.messagingengine.com> <1425913272.1336142.237883657.1A24DE1B@webmail.messagingengine.com> Message-ID: On 10 March 2015 at 01:54, Neil Girdhar wrote: > The differences are that the flags are used in a tiny portion of Python > code, and of that tiny amount of code, 99% of it probably uses nothing more > than the minimum "flags interface" that the composition solution has. bool > was not like that. Changing bool to be its own type would led to many more > problems. The flag values in modules like "stat" and "socket" are not niche use cases where we can be cavalier with backwards compatibility concerns, as they get passed to operating system level APIs both inside and outside the standard library. Most application level code will never need to touch those modules, but system boundary code does it all the time. It's the exact same rationale as was used for adopting IntEnum in the socket and errno modules rather than the base Enum class. There's likely no need to get especially creative with the overall design here, as the 3.4 enum module should provide a good general architecture to follow, it's just the runtime behaviour that will be changing to be "ordered collection of combinatorial flags" rather than "ordered collection of named values". The main question in my mind is whether you might want named bitmask support as well, but that's probably not worth the extra complexity, since you don't need it for debugging purposes. Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Tue Mar 10 14:43:14 2015 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 10 Mar 2015 23:43:14 +1000 Subject: [Python-ideas] CPython needs to revamp/replace it's build system && Cross-compiling Python sucks In-Reply-To: References: <49c23aa9-51cf-47b7-843b-8c4b58d01cbd@email.android.com> Message-ID: On 10 March 2015 at 21:20, Russell Keith-Magee wrote: > > On Tue, Mar 10, 2015 at 10:43 AM, Ryan wrote: >> >> Russell Keith-Magee wrote: >> >Hi Ryan, >> > >> >First off I'm no fan of autoconf, but... >> > >> >> Who is other than the one who designed it? :) > > Let me be clear - there's a difference between "not a fan" and "calling for > it's head". Autoconf is crufty, annoying, and sometimes aggravating - but it > *works*, and has had some pretty seriously battle testing - both in the > abstract "autoconf as a product", and in the specific "autoconf as used in > Python". This isn't something that can be said of most of the alternatives > in the abstract case, or *any* of the alternatives in the Python case. Building architecture dependent software is a genuinely hard problem, which is why it's full of "80%" solutions like autotools and the setuptools/distutils pairing. Yes, they're quirky and annoying, but they also basically do their job, so folks tend to develop a high tolerance level for their quirks. In our particular case, we also have the fun problem of "How do you build CPython locally when you don't have a working Python interpreter yet?". A number of the more interesting alternative C/C++ build systems (like Meson, Scons, Waf and the Google "gyp" project that powers the node-gyp native extension build system for Node.js) require an already working Python interpreter to get things started (when starting from a completely blank slate, even our own build sequence has to build a deliberately broken partial interpreter just to freeze importlib._bootstrap before going on to build a properly configured one that can actually import modules). CMake seems like a plausible alternative that would avoid the bootstrapping problem (as even building it from source only requires a working C++ compiler for the platform, either native or cross-compiled): http://www.cmake.org/ KDE switched over to CMake from autotools ~9 years ago (http://lwn.net/Articles/188693/) and plenty of other projects have either followed suit or simply never used autotools in the first place: http://www.cmake.org/Wiki/CMake_Projects It's far too late in the 3.5 dev cycle to contemplate a change like that, but if anyone was looking for a potentially interesting side project, seeing exactly what was involved in getting CPython to build with CMake instead of autotools could be a worthwhile investigation: https://stackoverflow.com/questions/7132862/tutorial-for-converting-autotools-to-cmake Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From rymg19 at gmail.com Tue Mar 10 16:39:50 2015 From: rymg19 at gmail.com (Ryan Gonzalez) Date: Tue, 10 Mar 2015 10:39:50 -0500 Subject: [Python-ideas] CPython needs to revamp/replace it's build system && Cross-compiling Python sucks In-Reply-To: References: <49c23aa9-51cf-47b7-843b-8c4b58d01cbd@email.android.com> Message-ID: On Tue, Mar 10, 2015 at 8:43 AM, Nick Coghlan wrote: > On 10 March 2015 at 21:20, Russell Keith-Magee > wrote: > > > > On Tue, Mar 10, 2015 at 10:43 AM, Ryan wrote: > >> > >> Russell Keith-Magee wrote: > >> >Hi Ryan, > >> > > >> >First off I'm no fan of autoconf, but... > >> > > >> > >> Who is other than the one who designed it? :) > > > > Let me be clear - there's a difference between "not a fan" and "calling > for > > it's head". Autoconf is crufty, annoying, and sometimes aggravating - > but it > > *works*, and has had some pretty seriously battle testing - both in the > > abstract "autoconf as a product", and in the specific "autoconf as used > in > > Python". This isn't something that can be said of most of the > alternatives > > in the abstract case, or *any* of the alternatives in the Python case. > > Building architecture dependent software is a genuinely hard problem, > which is why it's full of "80%" solutions like autotools and the > setuptools/distutils pairing. Yes, they're quirky and annoying, but > they also basically do their job, so folks tend to develop a high > tolerance level for their quirks. > > In our particular case, we also have the fun problem of "How do you > build CPython locally when you don't have a working Python interpreter > yet?". A number of the more interesting alternative C/C++ build > systems (like Meson, Scons, Waf and the Google "gyp" project that > powers the node-gyp native extension build system for Node.js) require > an already working Python interpreter to get things started (when > starting from a completely blank slate, even our own build sequence > has to build a deliberately broken partial interpreter just to freeze > importlib._bootstrap before going on to build a properly configured > one that can actually import modules). > What about something like Google's new GN? It's designed to replace Gyp in building Chromium and is written in C++. I'm pretty sure using a build system written in Lua to build Python is kind of weird. With all that in mind...what exactly would be requirements that Python would need in an autotools replacement? > > CMake seems like a plausible alternative that would avoid the > bootstrapping problem (as even building it from source only requires a > working C++ compiler for the platform, either native or > cross-compiled): http://www.cmake.org/ > > KDE switched over to CMake from autotools ~9 years ago > (http://lwn.net/Articles/188693/) and plenty of other projects have > either followed suit or simply never used autotools in the first > place: http://www.cmake.org/Wiki/CMake_Projects > > That was what I was referring to before. > It's far too late in the 3.5 dev cycle to contemplate a change like > that, but if anyone was looking for a potentially interesting side > project, seeing exactly what was involved in getting CPython to build > with CMake instead of autotools could be a worthwhile investigation: > > https://stackoverflow.com/questions/7132862/tutorial-for-converting-autotools-to-cmake > > Someone is maintaining a set of CMake build files for Python 2.7 somewhere. > Regards, > Nick. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > -- Ryan If I were in a 10-story building glass-sided building and forced to write either Go or autotools scripts, I?d jump out a window. http://kirbyfan64.github.io/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From dmalinovsky at thumbtack.net Tue Mar 10 16:43:17 2015 From: dmalinovsky at thumbtack.net (Dmitry Malinovsky) Date: Tue, 10 Mar 2015 21:43:17 +0600 Subject: [Python-ideas] "Regular" namespace packages Message-ID: Hello, I'm implementing an extensible library. Along with the library itself I want to implement various extensions (and some of these extensions will be also pluggable). I found that creating a namespace package will be user-friendly (to distinguish default implementations from third-party ones). However, current namespace packages implementation does not allow me to have a dedicated regular package with the same name as the namespace. E.g. if I have to namespaced packages: top.packageA and top.packageB, I can't have dedicated top package. (Actually, I can do that: py module is the example, but this way is not easy to implement for the library developers and to understand by external contributors). So my idea is to have a standard (and easy) way to declare a package which can act both as a namespace package and a regular package, like pep-420 (implicit namespace packages) standardized way to create namespace packages. Here is an example: first/first/__init__.py: X = 1 second/first/second/__init__.py: Y = 2 import sys sys.path += ['first', 'second'] import first first.X # 1 import first.second # I expect to get 2, first.second.Y # but currently ImportError is raised If I understand this correctly, this change will require to modify the default submodule lookup algorithm, so when the import machinery finds the cached first module, and there is no second submodule defined, the machinery launches the standard namespace packages finder algorithm (as described in [1]), excluding first package directory from the search path. >From a user perspective, assuming we have a mature library with lots of extensions (for example, flask), and a pluggable extension (like flask-flatpages), the following imports from flask import Flask from flask_flatpages import FlatPages from flask_flatpages_knitr import FlatPagesKnitr can be rewritten as follows: from flask import Flask from flask.flatpages import FlatPages from flask.flatpages.knitr import FlatPagesKnitr which I find more readable than the first snippet. What do you think about it? Is flat better than nested, or namespaces are really great? Thank you for any feedback! [1] https://www.python.org/dev/peps/pep-0420/#dynamic-path-computation -- Dmitry Malinovsky Senior DevOps Thumbtack Technology Email: dmalinovsky at thumbtack.net Skype: d_malinoff -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Tue Mar 10 17:14:59 2015 From: guido at python.org (Guido van Rossum) Date: Tue, 10 Mar 2015 09:14:59 -0700 Subject: [Python-ideas] "Regular" namespace packages In-Reply-To: References: Message-ID: Your example code can be corrected by updating first.__path__: first.__path__.append('second/first') Now "import first.second" will succeed. On Tue, Mar 10, 2015 at 8:43 AM, Dmitry Malinovsky < dmalinovsky at thumbtack.net> wrote: > Hello, > > I'm implementing an extensible library. Along with the library itself I > want to implement various extensions (and some of these extensions will be > also pluggable). > > I found that creating a namespace package will be user-friendly (to > distinguish default implementations from third-party ones). > However, current namespace packages implementation does not allow me to > have a dedicated regular package with the same name as the namespace. > E.g. if I have to namespaced packages: top.packageA and top.packageB, I > can't have dedicated top package. > (Actually, I can do that: py module is the example, but this way is not > easy to implement for the library developers and to understand > by external contributors). > > So my idea is to have a standard (and easy) way to declare a package which > can act both as a namespace package and a regular package, > like pep-420 (implicit namespace packages) standardized way to create > namespace packages. > > Here is an example: > > first/first/__init__.py: > X = 1 > > second/first/second/__init__.py: > Y = 2 > > import sys > sys.path += ['first', 'second'] > > import first > first.X # 1 > import first.second # I expect to get 2, > first.second.Y # but currently ImportError is raised > > If I understand this correctly, this change will require to modify the > default submodule lookup algorithm, > so when the import machinery finds the cached first module, and there is > no second submodule defined, > the machinery launches the standard namespace packages finder algorithm > (as described in [1]), > excluding first package directory from the search path. > > From a user perspective, assuming we have a mature library with lots of > extensions (for example, flask), > and a pluggable extension (like flask-flatpages), the following imports > > from flask import Flask > from flask_flatpages import FlatPages > from flask_flatpages_knitr import FlatPagesKnitr > > can be rewritten as follows: > > from flask import Flask > from flask.flatpages import FlatPages > from flask.flatpages.knitr import FlatPagesKnitr > > which I find more readable than the first snippet. > > What do you think about it? Is flat better than nested, or namespaces are > really great? > > Thank you for any feedback! > > [1] https://www.python.org/dev/peps/pep-0420/#dynamic-path-computation > > -- > Dmitry Malinovsky > Senior DevOps > Thumbtack Technology > Email: dmalinovsky at thumbtack.net > Skype: d_malinoff > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From dmalinovsky at thumbtack.net Tue Mar 10 18:20:38 2015 From: dmalinovsky at thumbtack.net (Dmitry Malinovsky) Date: Tue, 10 Mar 2015 23:20:38 +0600 Subject: [Python-ideas] "Regular" namespace packages In-Reply-To: References: Message-ID: Thank you, Guido, however the whole idea is to avoid modifying "first" package, so the "first" package author can freely leave his "first" namespace for extensions developers without even knowing that such extensions exist. On Tue, Mar 10, 2015 at 10:14 PM, Guido van Rossum wrote: > Your example code can be corrected by updating first.__path__: > > first.__path__.append('second/first') > > Now "import first.second" will succeed. > > On Tue, Mar 10, 2015 at 8:43 AM, Dmitry Malinovsky < > dmalinovsky at thumbtack.net> wrote: > >> Hello, >> >> I'm implementing an extensible library. Along with the library itself I >> want to implement various extensions (and some of these extensions will be >> also pluggable). >> >> I found that creating a namespace package will be user-friendly (to >> distinguish default implementations from third-party ones). >> However, current namespace packages implementation does not allow me to >> have a dedicated regular package with the same name as the namespace. >> E.g. if I have to namespaced packages: top.packageA and top.packageB, I >> can't have dedicated top package. >> (Actually, I can do that: py module is the example, but this way is not >> easy to implement for the library developers and to understand >> by external contributors). >> >> So my idea is to have a standard (and easy) way to declare a package >> which can act both as a namespace package and a regular package, >> like pep-420 (implicit namespace packages) standardized way to create >> namespace packages. >> >> Here is an example: >> >> first/first/__init__.py: >> X = 1 >> >> second/first/second/__init__.py: >> Y = 2 >> >> import sys >> sys.path += ['first', 'second'] >> >> import first >> first.X # 1 >> import first.second # I expect to get 2, >> first.second.Y # but currently ImportError is raised >> >> If I understand this correctly, this change will require to modify the >> default submodule lookup algorithm, >> so when the import machinery finds the cached first module, and there is >> no second submodule defined, >> the machinery launches the standard namespace packages finder algorithm >> (as described in [1]), >> excluding first package directory from the search path. >> >> From a user perspective, assuming we have a mature library with lots of >> extensions (for example, flask), >> and a pluggable extension (like flask-flatpages), the following imports >> >> from flask import Flask >> from flask_flatpages import FlatPages >> from flask_flatpages_knitr import FlatPagesKnitr >> >> can be rewritten as follows: >> >> from flask import Flask >> from flask.flatpages import FlatPages >> from flask.flatpages.knitr import FlatPagesKnitr >> >> which I find more readable than the first snippet. >> >> What do you think about it? Is flat better than nested, or namespaces are >> really great? >> >> Thank you for any feedback! >> >> [1] https://www.python.org/dev/peps/pep-0420/#dynamic-path-computation >> >> -- >> Dmitry Malinovsky >> Senior DevOps >> Thumbtack Technology >> Email: dmalinovsky at thumbtack.net >> Skype: d_malinoff >> >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ >> > > > > -- > --Guido van Rossum (python.org/~guido) > -- Dmitry Malinovsky Support Engineer Thumbtack Technology Email: dmalinovsky at thumbtack.net Skype: d_malinoff -------------- next part -------------- An HTML attachment was scrubbed... URL: From eltoder at gmail.com Tue Mar 10 19:00:23 2015 From: eltoder at gmail.com (Eugene Toder) Date: Tue, 10 Mar 2015 14:00:23 -0400 Subject: [Python-ideas] "Regular" namespace packages In-Reply-To: References: Message-ID: On Tue, Mar 10, 2015 at 1:20 PM, Dmitry Malinovsky < dmalinovsky at thumbtack.net> wrote: > Thank you, Guido, however the whole idea is to avoid modifying "first" > package, so the "first" package author can freely leave his "first" > namespace for extensions developers without even knowing that such > extensions exist. > The simplest solution is probably to make the installation process of package "second" to install itself in the directory of "first", or make a symlink, e.g.: first/first/__init__.py first/first/second --> second/first/second # added by package second second/first/second/__init__.py Alternatively, you can add "second" to the package __path__ as Guido pointed above via .pth file[1], to avoid changing anything in the "first" package. [1] https://docs.python.org/3.4/library/site.html, especially the part about lines starting with "import" Eugene -------------- next part -------------- An HTML attachment was scrubbed... URL: From julien at tayon.net Wed Mar 11 13:28:49 2015 From: julien at tayon.net (julien tayon) Date: Wed, 11 Mar 2015 08:28:49 -0400 Subject: [Python-ideas] Implementation of adding dicts Message-ID: Hello, Just in case I have to implementation of dict/sub/mult/div for dict. I have chosen only one rule: + - / * should behave with the least surprise principle which is so far linear algebrae when dealing with stuff that looks like mathematics Implementation 2 (trying trait and mixin) http://archery.readthedocs.org/en/latest/bow.html Implementation 1 (experimenting weired stuff with dict (like addition and transforming dict with dict in a ?matrix like? fashion) http://vectordict.readthedocs.org/en/latest/algebrae.html And the linear algebrae test: http://vectordict.readthedocs.org/en/latest/consistent.html And the results https://github.com/jul/ADictAdd_iction/blob/master/docs/source/result.txt I just noticed the dicussion, I have been using the + - / * for aggregations for 2 years now and it is really helpful. I can give any feedback. I also had to overload + - for Sets ^^ and I have the cosine similarities that works nicely out of the box with my definition -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Wed Mar 11 16:07:19 2015 From: brett at python.org (Brett Cannon) Date: Wed, 11 Mar 2015 15:07:19 +0000 Subject: [Python-ideas] CPython needs to revamp/replace it's build system && Cross-compiling Python sucks In-Reply-To: References: <49c23aa9-51cf-47b7-843b-8c4b58d01cbd@email.android.com> Message-ID: On Tue, Mar 10, 2015 at 11:40 AM Ryan Gonzalez wrote: > On Tue, Mar 10, 2015 at 8:43 AM, Nick Coghlan wrote: > >> On 10 March 2015 at 21:20, Russell Keith-Magee >> wrote: >> > >> > On Tue, Mar 10, 2015 at 10:43 AM, Ryan wrote: >> >> >> >> Russell Keith-Magee wrote: >> >> >Hi Ryan, >> >> > >> >> >First off I'm no fan of autoconf, but... >> >> > >> >> >> >> Who is other than the one who designed it? :) >> > >> > Let me be clear - there's a difference between "not a fan" and "calling >> for >> > it's head". Autoconf is crufty, annoying, and sometimes aggravating - >> but it >> > *works*, and has had some pretty seriously battle testing - both in the >> > abstract "autoconf as a product", and in the specific "autoconf as used >> in >> > Python". This isn't something that can be said of most of the >> alternatives >> > in the abstract case, or *any* of the alternatives in the Python case. >> >> Building architecture dependent software is a genuinely hard problem, >> which is why it's full of "80%" solutions like autotools and the >> setuptools/distutils pairing. Yes, they're quirky and annoying, but >> they also basically do their job, so folks tend to develop a high >> tolerance level for their quirks. >> >> In our particular case, we also have the fun problem of "How do you >> build CPython locally when you don't have a working Python interpreter >> yet?". A number of the more interesting alternative C/C++ build >> systems (like Meson, Scons, Waf and the Google "gyp" project that >> powers the node-gyp native extension build system for Node.js) require >> an already working Python interpreter to get things started (when >> starting from a completely blank slate, even our own build sequence >> has to build a deliberately broken partial interpreter just to freeze >> importlib._bootstrap before going on to build a properly configured >> one that can actually import modules). >> > > What about something like Google's new GN? It's designed to replace Gyp in > building Chromium and is written in C++. > > I'm pretty sure using a build system written in Lua to build Python is > kind of weird. > > > With all that in mind...what exactly would be requirements that Python > would need in an autotools replacement? > Can't require anything more than an ANSI C compiler basically. People run Python in places where they don't have much more than gcc. > > >> >> CMake seems like a plausible alternative that would avoid the >> bootstrapping problem (as even building it from source only requires a >> working C++ compiler for the platform, either native or >> cross-compiled): http://www.cmake.org/ >> >> KDE switched over to CMake from autotools ~9 years ago >> (http://lwn.net/Articles/188693/) and plenty of other projects have >> either followed suit or simply never used autotools in the first >> place: http://www.cmake.org/Wiki/CMake_Projects >> >> > That was what I was referring to before. > I casually discussed the idea back at PyCon 2008 about using CMake and was told "no" because of potential build tool dependencies for CMake itself. So you have to make sure that you can prove CMake is highly portable. > > >> It's far too late in the 3.5 dev cycle to contemplate a change like >> that, but if anyone was looking for a potentially interesting side >> project, seeing exactly what was involved in getting CPython to build >> with CMake instead of autotools could be a worthwhile investigation: >> >> https://stackoverflow.com/questions/7132862/tutorial-for-converting-autotools-to-cmake >> >> > Someone is maintaining a set of CMake build files for Python 2.7 somewhere. > That's fine, but obviously things have been tweaked since then and this could only go into 3.x. -Brett > > >> Regards, >> Nick. >> >> -- >> Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia >> > > > > -- > Ryan > If I were in a 10-story building glass-sided building and forced to write > either Go or autotools scripts, I?d jump out a window. > http://kirbyfan64.github.io/ > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From rymg19 at gmail.com Wed Mar 11 19:30:12 2015 From: rymg19 at gmail.com (Ryan Gonzalez) Date: Wed, 11 Mar 2015 13:30:12 -0500 Subject: [Python-ideas] CPython needs to revamp/replace it's build system && Cross-compiling Python sucks In-Reply-To: References: <49c23aa9-51cf-47b7-843b-8c4b58d01cbd@email.android.com> Message-ID: On Wed, Mar 11, 2015 at 10:07 AM, Brett Cannon wrote: > > > On Tue, Mar 10, 2015 at 11:40 AM Ryan Gonzalez wrote: > >> On Tue, Mar 10, 2015 at 8:43 AM, Nick Coghlan wrote: >> >>> On 10 March 2015 at 21:20, Russell Keith-Magee >>> wrote: >>> > >>> > On Tue, Mar 10, 2015 at 10:43 AM, Ryan wrote: >>> >> >>> >> Russell Keith-Magee wrote: >>> >> >Hi Ryan, >>> >> > >>> >> >First off I'm no fan of autoconf, but... >>> >> > >>> >> >>> >> Who is other than the one who designed it? :) >>> > >>> > Let me be clear - there's a difference between "not a fan" and >>> "calling for >>> > it's head". Autoconf is crufty, annoying, and sometimes aggravating - >>> but it >>> > *works*, and has had some pretty seriously battle testing - both in the >>> > abstract "autoconf as a product", and in the specific "autoconf as >>> used in >>> > Python". This isn't something that can be said of most of the >>> alternatives >>> > in the abstract case, or *any* of the alternatives in the Python case. >>> >>> Building architecture dependent software is a genuinely hard problem, >>> which is why it's full of "80%" solutions like autotools and the >>> setuptools/distutils pairing. Yes, they're quirky and annoying, but >>> they also basically do their job, so folks tend to develop a high >>> tolerance level for their quirks. >>> >>> In our particular case, we also have the fun problem of "How do you >>> build CPython locally when you don't have a working Python interpreter >>> yet?". A number of the more interesting alternative C/C++ build >>> systems (like Meson, Scons, Waf and the Google "gyp" project that >>> powers the node-gyp native extension build system for Node.js) require >>> an already working Python interpreter to get things started (when >>> starting from a completely blank slate, even our own build sequence >>> has to build a deliberately broken partial interpreter just to freeze >>> importlib._bootstrap before going on to build a properly configured >>> one that can actually import modules). >>> >> >> What about something like Google's new GN? It's designed to replace Gyp >> in building Chromium and is written in C++. >> >> I'm pretty sure using a build system written in Lua to build Python is >> kind of weird. >> > > >> >> With all that in mind...what exactly would be requirements that Python >> would need in an autotools replacement? >> > > Can't require anything more than an ANSI C compiler basically. People run > Python in places where they don't have much more than gcc. > Not even C99? Darn it... > > >> >> >>> >>> CMake seems like a plausible alternative that would avoid the >>> bootstrapping problem (as even building it from source only requires a >>> working C++ compiler for the platform, either native or >>> cross-compiled): http://www.cmake.org/ >>> >>> KDE switched over to CMake from autotools ~9 years ago >>> (http://lwn.net/Articles/188693/) and plenty of other projects have >>> either followed suit or simply never used autotools in the first >>> place: http://www.cmake.org/Wiki/CMake_Projects >>> >>> >> That was what I was referring to before. >> > > I casually discussed the idea back at PyCon 2008 about using CMake and was > told "no" because of potential build tool dependencies for CMake itself. So > you have to make sure that you can prove CMake is highly portable. > I'm guessing that's because CMake is written in C++, which is a large dependency. I *really* wish someone on this earth still had a C++-to-C99 compiler like cfront was. > > >> >> >>> It's far too late in the 3.5 dev cycle to contemplate a change like >>> that, but if anyone was looking for a potentially interesting side >>> project, seeing exactly what was involved in getting CPython to build >>> with CMake instead of autotools could be a worthwhile investigation: >>> >>> https://stackoverflow.com/questions/7132862/tutorial-for-converting-autotools-to-cmake >>> >>> >> Someone is maintaining a set of CMake build files for Python 2.7 >> somewhere. >> > > That's fine, but obviously things have been tweaked since then and this > could only go into 3.x. > > I know. Just an example to show it's possible. So, here's what I've gathered thus far: - Needs to be written in ANSI C89 and *only* ANSI C89 - Needs to support autotools-style try-to-build-this-and-do-something-if-it-fails - Needs to be nicer than autotools, otherwise this entire thread is 100% pointless. > -Brett > > >> >> >>> Regards, >>> Nick. >>> >>> -- >>> Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia >>> >> >> >> >> -- >> Ryan >> If I were in a 10-story building glass-sided building and forced to write >> either Go or autotools scripts, I?d jump out a window. >> http://kirbyfan64.github.io/ >> >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ > > -- Ryan [ERROR]: Your autotools build scripts are 200 lines longer than your program. Something?s wrong. http://kirbyfan64.github.io/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From nas-python at arctrix.com Wed Mar 11 20:43:25 2015 From: nas-python at arctrix.com (Neil Schemenauer) Date: Wed, 11 Mar 2015 13:43:25 -0600 Subject: [Python-ideas] Migration of /usr/bin/python to python3 Message-ID: <20150311194325.GA15390@python.ca> This has been brought up elsewhere, I think this is a better forum to discuss it. Changing /usr/bin/python directly to python3.x is bad idea, in my option. It's going to cause users trouble and the benefit to Python 3.x users is not worth it. Instead, let's try to find a smooth migration path. One idea is to install Python 3 as /usr/bin/python but have it always execute scripts with Python 2 unless they are explicitly marked as Python 3 compatible. The interpreter could parse the first line of the script to look for options. Traditionally the OS only passes the first option to the interpreter so anything after that could be used to pass 3.x specific options. For example, a script starting with #!/usr/bin/python -x -3 could signify that the script is compatible with a Python 3 interpreter. An alternative idea is to allow something like the 'encoding' declaration. It wouldn't have to be in the first line and would allow more flexibility. Many years in the future when all scripts are marked as Python 3 compatible, Python 2 can be removed from the system and /usr/bin/python will start Python 3 by default. I suppose running '/usr/bin/python' interactively could start Python 3 by default. However, I strongly feel that existing Python 2 scripts should not get broken in a flag day like way. Neil From mertz at gnosis.cx Wed Mar 11 21:03:36 2015 From: mertz at gnosis.cx (David Mertz) Date: Wed, 11 Mar 2015 13:03:36 -0700 Subject: [Python-ideas] Migration of /usr/bin/python to python3 In-Reply-To: <20150311194325.GA15390@python.ca> References: <20150311194325.GA15390@python.ca> Message-ID: https://www.python.org/dev/peps/pep-0394/ On Mar 11, 2015 12:44 PM, "Neil Schemenauer" wrote: > This has been brought up elsewhere, I think this is a better forum > to discuss it. Changing /usr/bin/python directly to python3.x is > bad idea, in my option. It's going to cause users trouble and the > benefit to Python 3.x users is not worth it. Instead, let's try to > find a smooth migration path. > > One idea is to install Python 3 as /usr/bin/python but have it > always execute scripts with Python 2 unless they are explicitly > marked as Python 3 compatible. > > The interpreter could parse the first line of the script to look for > options. Traditionally the OS only passes the first option to the > interpreter so anything after that could be used to pass 3.x > specific options. > > For example, a script starting with > > #!/usr/bin/python -x -3 > > could signify that the script is compatible with a Python 3 > interpreter. > > An alternative idea is to allow something like the 'encoding' > declaration. It wouldn't have to be in the first line and would > allow more flexibility. Many years in the future when all scripts > are marked as Python 3 compatible, Python 2 can be removed from the > system and /usr/bin/python will start Python 3 by default. > > I suppose running '/usr/bin/python' interactively could start Python > 3 by default. However, I strongly feel that existing Python 2 > scripts should not get broken in a flag day like way. > > > Neil > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rymg19 at gmail.com Wed Mar 11 21:10:24 2015 From: rymg19 at gmail.com (Ryan Gonzalez) Date: Wed, 11 Mar 2015 15:10:24 -0500 Subject: [Python-ideas] Migration of /usr/bin/python to python3 In-Reply-To: <20150311194325.GA15390@python.ca> References: <20150311194325.GA15390@python.ca> Message-ID: On Wed, Mar 11, 2015 at 2:43 PM, Neil Schemenauer wrote: > This has been brought up elsewhere, I think this is a better forum > to discuss it. Changing /usr/bin/python directly to python3.x is > bad idea, in my option. It's going to cause users trouble and the > benefit to Python 3.x users is not worth it. Instead, let's try to > find a smooth migration path. > > One idea is to install Python 3 as /usr/bin/python but have it > always execute scripts with Python 2 unless they are explicitly > marked as Python 3 compatible. > > So Python 2 needs to be there anyway? > The interpreter could parse the first line of the script to look for > options. Traditionally the OS only passes the first option to the > interpreter so anything after that could be used to pass 3.x > specific options. > > For example, a script starting with > > #!/usr/bin/python -x -3 > > could signify that the script is compatible with a Python 3 > interpreter. > What about using */usr/bin/env*? That wouldn't work; on my Linux system, */usr/bin/env* doesn't like any command like arguments. Whenever I download a Python program, I usually check the hashbang to see if it works with Python 2 or 3. This is bound to cause confusion, since a quick glance would say "hey, this looks like it works under Python 2", followed by the error-fest. > > An alternative idea is to allow something like the 'encoding' > declaration. It wouldn't have to be in the first line and would > allow more flexibility. Many years in the future when all scripts > are marked as Python 3 compatible, Python 2 can be removed from the > system and /usr/bin/python will start Python 3 by default. > > I guess that's better. > I suppose running '/usr/bin/python' interactively could start Python > 3 by default. However, I strongly feel that existing Python 2 > scripts should not get broken in a flag day like way. > > But parsing hashbangs really isn't the answer. It's just really Perl-magic-ish confusing. > > Neil > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- Ryan [ERROR]: Your autotools build scripts are 200 lines longer than your program. Something?s wrong. http://kirbyfan64.github.io/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From skip.montanaro at gmail.com Wed Mar 11 21:13:18 2015 From: skip.montanaro at gmail.com (Skip Montanaro) Date: Wed, 11 Mar 2015 15:13:18 -0500 Subject: [Python-ideas] Migration of /usr/bin/python to python3 In-Reply-To: References: <20150311194325.GA15390@python.ca> Message-ID: On Wed, Mar 11, 2015 at 3:10 PM, Ryan Gonzalez wrote: > What about using */usr/bin/env*? That wouldn't work; on my Linux system, > */usr/bin/env* doesn't like any command like arguments. Not sure what planet your Linux system is from, but I use #!/usr/bin/env python in all my scripts. It likes the argument ("python" in this case) just fine... Skip -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Wed Mar 11 21:14:14 2015 From: donald at stufft.io (Donald Stufft) Date: Wed, 11 Mar 2015 16:14:14 -0400 Subject: [Python-ideas] Migration of /usr/bin/python to python3 In-Reply-To: <20150311194325.GA15390@python.ca> References: <20150311194325.GA15390@python.ca> Message-ID: <041B5A0E-4C77-4683-9B0A-98432C139F48@stufft.io> What's worse. Breaking everything that uses an unversioned shebang because /usr/bin/python doesn't exist or breaking some things that doesn't work with Python 3 because /usr/bin/python is now 3? Generally discussions along this line talk about Python 2 not being installed by default at all (whereas Python 3 would be). So really it's two different broken by default. > On Mar 11, 2015, at 3:43 PM, Neil Schemenauer wrote: > > This has been brought up elsewhere, I think this is a better forum > to discuss it. Changing /usr/bin/python directly to python3.x is > bad idea, in my option. It's going to cause users trouble and the > benefit to Python 3.x users is not worth it. Instead, let's try to > find a smooth migration path. > > One idea is to install Python 3 as /usr/bin/python but have it > always execute scripts with Python 2 unless they are explicitly > marked as Python 3 compatible. > > The interpreter could parse the first line of the script to look for > options. Traditionally the OS only passes the first option to the > interpreter so anything after that could be used to pass 3.x > specific options. > > For example, a script starting with > > #!/usr/bin/python -x -3 > > could signify that the script is compatible with a Python 3 > interpreter. > > An alternative idea is to allow something like the 'encoding' > declaration. It wouldn't have to be in the first line and would > allow more flexibility. Many years in the future when all scripts > are marked as Python 3 compatible, Python 2 can be removed from the > system and /usr/bin/python will start Python 3 by default. > > I suppose running '/usr/bin/python' interactively could start Python > 3 by default. However, I strongly feel that existing Python 2 > scripts should not get broken in a flag day like way. > > > Neil > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ From phd at phdru.name Wed Mar 11 21:22:35 2015 From: phd at phdru.name (Oleg Broytman) Date: Wed, 11 Mar 2015 21:22:35 +0100 Subject: [Python-ideas] Migration of /usr/bin/python to python3 In-Reply-To: <20150311194325.GA15390@python.ca> References: <20150311194325.GA15390@python.ca> Message-ID: <20150311202235.GA30739@phdru.name> Hi! On Wed, Mar 11, 2015 at 01:43:25PM -0600, Neil Schemenauer wrote: [skip] > a script starting with > > #!/usr/bin/python -x -3 [skip] That wouldn't work. Shebang line accepts one and only one parameter. There are operating systems that allow more but they are exceptions. $ cat test1 #! /usr/local/bin/python -x3 print "Ok" $ ./test1 Ok $ cat test2 #! /usr/local/bin/python -x -3 print "Ok" $ ./test2 Unknown option: - usage: /usr/local/bin/python [option] ... [-c cmd | -m mod | file | -] [arg] ... Try `python -h' for more information. $ cat test3 #! /usr/bin/env python -x3 print "Ok" $ ./test3 env: python -x3: No such file or directory $ cat test4 #! /usr/bin/env python -x -3 print "Ok" $ ./test4 env: python -x -3: No such file or directory $ uname -rs FreeBSD 9.3-RELEASE > Neil Oleg. -- Oleg Broytman http://phdru.name/ phd at phdru.name Programmers don't die, they just GOSUB without RETURN. From skip.montanaro at gmail.com Wed Mar 11 21:26:07 2015 From: skip.montanaro at gmail.com (Skip Montanaro) Date: Wed, 11 Mar 2015 15:26:07 -0500 Subject: [Python-ideas] Migration of /usr/bin/python to python3 In-Reply-To: <20150311202235.GA30739@phdru.name> References: <20150311194325.GA15390@python.ca> <20150311202235.GA30739@phdru.name> Message-ID: On Wed, Mar 11, 2015 at 3:22 PM, Oleg Broytman wrote: > That wouldn't work. Shebang line accepts one and only one parameter. > There are operating systems that allow more but they are exceptions. > I think Neal was suggesting that you rely on that property (in which case the exceptions might be the problem). If I understood correctly, /usr/bin/python would be started by the kernel, and it would be passed the -x flag. It would then inspect the #! line again, paying attention to the second (and following) arguments. Seeing the -3, it would continue executing. If it saw nothing (or saw -2?) it would exec /usr/bin/python2 (or similar). I hope I'm not putting words in Neal's mouth. Skip -------------- next part -------------- An HTML attachment was scrubbed... URL: From phd at phdru.name Wed Mar 11 21:32:45 2015 From: phd at phdru.name (Oleg Broytman) Date: Wed, 11 Mar 2015 21:32:45 +0100 Subject: [Python-ideas] Migration of /usr/bin/python to python3 In-Reply-To: References: <20150311194325.GA15390@python.ca> <20150311202235.GA30739@phdru.name> Message-ID: <20150311203245.GB30739@phdru.name> On Wed, Mar 11, 2015 at 03:26:07PM -0500, Skip Montanaro wrote: > On Wed, Mar 11, 2015 at 3:22 PM, Oleg Broytman wrote: > > > That wouldn't work. Shebang line accepts one and only one parameter. > > There are operating systems that allow more but they are exceptions. > > If I understood correctly, > /usr/bin/python would be started by the kernel, and it would be passed the > -x flag. But it (the kernel) doesn't: $ cat test2 #! /usr/local/bin/python -x -3 print "Ok" $ ./test2 Unknown option: - usage: /usr/local/bin/python [option] ... [-c cmd | -m mod | file | -] [arg] ... Try `python -h' for more information. Oleg. -- Oleg Broytman http://phdru.name/ phd at phdru.name Programmers don't die, they just GOSUB without RETURN. From mal at egenix.com Wed Mar 11 21:38:06 2015 From: mal at egenix.com (M.-A. Lemburg) Date: Wed, 11 Mar 2015 21:38:06 +0100 Subject: [Python-ideas] Migration of /usr/bin/python to python3 In-Reply-To: References: <20150311194325.GA15390@python.ca> Message-ID: <5500A7AE.9020400@egenix.com> On 11.03.2015 21:03, David Mertz wrote: > https://www.python.org/dev/peps/pep-0394/ I think the migration should start with modifying scripts to use #!/usr/bin/env python2 when they are Python 2 only (*) and #!/usr/bin/env python3 when they are Python 3 only and #!/usr/bin/env python only when they support both Python 2 and 3. "Explicit is better than implicit" and all that Zen :-) Once that's done, switching the symlink is really a no-brainer. The recipe for this is easy too: 1. replace all "#!/usr/bin/env python" with "#!/usr/bin/env python2" 2. migrate your scripts one by one to either Python 3.x or to Python 2.7 + 3.4+ 3. after migration replace "#!/usr/bin/env python2" with "#!/usr/bin/env python3" or "#!/usr/bin/env python" resp. (*) Some OSes may require to use python2.7, if they don't come with a symlink from python2 -> python2.7. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Mar 11 2015) >>> Python Projects, Coaching and Consulting ... http://www.egenix.com/ >>> mxODBC Plone/Zope Database Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From rymg19 at gmail.com Wed Mar 11 21:50:31 2015 From: rymg19 at gmail.com (Ryan Gonzalez) Date: Wed, 11 Mar 2015 15:50:31 -0500 Subject: [Python-ideas] Migration of /usr/bin/python to python3 In-Reply-To: References: <20150311194325.GA15390@python.ca> Message-ID: No; this won't work: #!/usr/bin/env python -x -3 As Oleg said, anything in the hashbang can only be passed one argument. It's not the kernel that invokes that; it's the shell. http://mail-index.netbsd.org/netbsd-users/2008/11/09/msg002388.html On Wed, Mar 11, 2015 at 3:13 PM, Skip Montanaro wrote: > > On Wed, Mar 11, 2015 at 3:10 PM, Ryan Gonzalez wrote: > >> What about using */usr/bin/env*? That wouldn't work; on my Linux system, >> */usr/bin/env* doesn't like any command like arguments. > > > Not sure what planet your Linux system is from, but I use > > #!/usr/bin/env python > > in all my scripts. It likes the argument ("python" in this case) just > fine... > > Skip > > -- Ryan [ERROR]: Your autotools build scripts are 200 lines longer than your program. Something?s wrong. http://kirbyfan64.github.io/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From dwblas at gmail.com Wed Mar 11 22:08:20 2015 From: dwblas at gmail.com (David Blaschke) Date: Wed, 11 Mar 2015 14:08:20 -0700 Subject: [Python-ideas] Migration of /usr/bin/python to python3 In-Reply-To: References: <20150311194325.GA15390@python.ca> Message-ID: Sorry if this is a double post-didn't reply to all Checking for default version is trivial unless I am missing something .import sys .if sys.version_info[0] < 3: . ## /usr/bin/python2 via subprocess.call or whatever .else: . ##/usr/bin/python3 -- Only when you actually get to the state where there is neither delusion nor enlightenment are you finally comfortable...Foyan From ethan at stoneleaf.us Wed Mar 11 22:19:06 2015 From: ethan at stoneleaf.us (Ethan Furman) Date: Wed, 11 Mar 2015 14:19:06 -0700 Subject: [Python-ideas] Migration of /usr/bin/python to python3 In-Reply-To: <5500A7AE.9020400@egenix.com> References: <20150311194325.GA15390@python.ca> <5500A7AE.9020400@egenix.com> Message-ID: <5500B14A.6090705@stoneleaf.us> On 03/11/2015 01:38 PM, M.-A. Lemburg wrote: > On 11.03.2015 21:03, David Mertz wrote: > I think the migration should start with modifying scripts to use > > #!/usr/bin/env python2 > > when they are Python 2 only (*) and > > #!/usr/bin/env python3 > > when they are Python 3 only and > > #!/usr/bin/env python > > only when they support both Python 2 and 3. +1 -- ~Ethan~ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: OpenPGP digital signature URL: From ianlee1521 at gmail.com Wed Mar 11 22:52:06 2015 From: ianlee1521 at gmail.com (Ian Lee) Date: Wed, 11 Mar 2015 14:52:06 -0700 Subject: [Python-ideas] Migration of /usr/bin/python to python3 In-Reply-To: <5500A7AE.9020400@egenix.com> References: <20150311194325.GA15390@python.ca> <5500A7AE.9020400@egenix.com> Message-ID: +1 An issue that I've noticed recently (I haven't had the time to fully track this one down yet, but I suspect it's a setuptools issue) takes a "last one wins" approach to scripts that get written into the same bin directory (e.g. --user installs). $ python2.7 -m pip install --user ipython > > $ head -n1 ~/.local/bin/ipython* > ==> /home/ianlee1521/.local/bin/ipython <== > #!/usr/bin/python2.7 > > ==> /home/ianlee1521/.local/bin/ipython2 <== > #!/usr/bin/python2.7 > > $ python3.4 -m pip install --user ipython > > $ head -n1 ~/.local/bin/ipython* > ==> /home/ianlee1521/.local/bin/ipython <== > #!/usr/bin/python3.4 > > ==> /home/ianlee1521/.local/bin/ipython2 <== > #!/usr/bin/python2.7 > > ==> /home/ianlee1521/.local/bin/ipython3 <== > #!/usr/bin/python3.4 So the packages end up in the separate, versioned directories (~/.local/lib/pythonX.Y/site-packages), but with the scripts ending up in the same bin directory, the implicit script "~/.local/bin/ipython" ends up changing to use the hashbang of the Python version which last installed the package (see highlighted above). I would expect that the implicit script ("~/.local/bin/ipython") should use the implicit python version, e.g. "#!/usr/bin/python". Alternatively, maybe I just shouldn't ever be installing user versions of both Python 2 and Python 3 versions of a package. ~ Ian Lee On Wed, Mar 11, 2015 at 1:38 PM, M.-A. Lemburg wrote: > On 11.03.2015 21:03, David Mertz wrote: > > https://www.python.org/dev/peps/pep-0394/ > > I think the migration should start with modifying scripts to use > > #!/usr/bin/env python2 > > when they are Python 2 only (*) and > > #!/usr/bin/env python3 > > when they are Python 3 only and > > #!/usr/bin/env python > > only when they support both Python 2 and 3. > > "Explicit is better than implicit" and all that Zen :-) > > Once that's done, switching the symlink is really a no-brainer. > > The recipe for this is easy too: > > 1. replace all "#!/usr/bin/env python" with "#!/usr/bin/env python2" > 2. migrate your scripts one by one to either Python 3.x or > to Python 2.7 + 3.4+ > 3. after migration replace "#!/usr/bin/env python2" with > "#!/usr/bin/env python3" or "#!/usr/bin/env python" resp. > > (*) Some OSes may require to use python2.7, if they don't come > with a symlink from python2 -> python2.7. > > -- > Marc-Andre Lemburg > eGenix.com > > Professional Python Services directly from the Source (#1, Mar 11 2015) > >>> Python Projects, Coaching and Consulting ... http://www.egenix.com/ > >>> mxODBC Plone/Zope Database Adapter ... http://zope.egenix.com/ > >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ > ________________________________________________________________________ > > ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: > > eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 > D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg > Registered at Amtsgericht Duesseldorf: HRB 46611 > http://www.egenix.com/company/contact/ > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From abarnert at yahoo.com Wed Mar 11 23:29:20 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Wed, 11 Mar 2015 15:29:20 -0700 Subject: [Python-ideas] Migration of /usr/bin/python to python3 In-Reply-To: References: <20150311194325.GA15390@python.ca> <5500A7AE.9020400@egenix.com> Message-ID: <9F862F85-77CE-4F68-8223-EDCEF9818836@yahoo.com> On Mar 11, 2015, at 2:52 PM, Ian Lee wrote: > > +1 > > An issue that I've noticed recently (I haven't had the time to fully track this one down yet, but I suspect it's a setuptools issue) takes a "last one wins" approach to scripts that get written into the same bin directory (e.g. --user installs). Many packages will install spamX and spamX.Y scripts as well as plain spam, so you can deal with this the same way you deal with Python instead--run ipython3 or ipython3.4 instead of just ipython. (Not to mention ipython_pypy3!) Also, sometimes python2 and python3 will install to different directories (/usr for the one that's part of the system, /usr/local for others, or /Library/Framework/Python/Versions/X.Y/bin...), so "last one wins" isn't even predictable cross-platform; you have to be able to deal with either "last one wins" or "controlled by PATH" when you want to just run "ipython". But really, since these scripts don't need to be run by a shbang or other automated mechanism, and the user who intentionally installs IPython for both 2.7 and 3.4 is going to need some way to manually select the one they want to run each time anyway. So I think this solution isn't too bad: 90% of the time, a script is only installed for one Python version so "last one wins" works trivially; most of the rest of the time, the user has two or more and needs to use "ipython2" vs. "ipython3" rather than just "ipython" because how else could be use both as desired, so "last one wins" is irrelevant.The thing is, I don't think you get this script versioning automatically from setuptools. If you don't, maybe you should? > >> $ python2.7 -m pip install --user ipython >> >> $ head -n1 ~/.local/bin/ipython* >> ==> /home/ianlee1521/.local/bin/ipython <== >> #!/usr/bin/python2.7 >> >> ==> /home/ianlee1521/.local/bin/ipython2 <== >> #!/usr/bin/python2.7 >> >> $ python3.4 -m pip install --user ipython >> >> $ head -n1 ~/.local/bin/ipython* >> ==> /home/ianlee1521/.local/bin/ipython <== >> #!/usr/bin/python3.4 >> >> ==> /home/ianlee1521/.local/bin/ipython2 <== >> #!/usr/bin/python2.7 >> >> ==> /home/ianlee1521/.local/bin/ipython3 <== >> #!/usr/bin/python3.4 > > So the packages end up in the separate, versioned directories (~/.local/lib/pythonX.Y/site-packages), but with the scripts ending up in the same bin directory, the implicit script "~/.local/bin/ipython" ends up changing to use the hashbang of the Python version which last installed the package (see highlighted above). > > I would expect that the implicit script ("~/.local/bin/ipython") should use the implicit python version, e.g. "#!/usr/bin/python". > > Alternatively, maybe I just shouldn't ever be installing user versions of both Python 2 and Python 3 versions of a package. > > > ~ Ian Lee > >> On Wed, Mar 11, 2015 at 1:38 PM, M.-A. Lemburg wrote: >> On 11.03.2015 21:03, David Mertz wrote: >> > https://www.python.org/dev/peps/pep-0394/ >> >> I think the migration should start with modifying scripts to use >> >> #!/usr/bin/env python2 >> >> when they are Python 2 only (*) and >> >> #!/usr/bin/env python3 >> >> when they are Python 3 only and >> >> #!/usr/bin/env python >> >> only when they support both Python 2 and 3. >> >> "Explicit is better than implicit" and all that Zen :-) >> >> Once that's done, switching the symlink is really a no-brainer. >> >> The recipe for this is easy too: >> >> 1. replace all "#!/usr/bin/env python" with "#!/usr/bin/env python2" >> 2. migrate your scripts one by one to either Python 3.x or >> to Python 2.7 + 3.4+ >> 3. after migration replace "#!/usr/bin/env python2" with >> "#!/usr/bin/env python3" or "#!/usr/bin/env python" resp. >> >> (*) Some OSes may require to use python2.7, if they don't come >> with a symlink from python2 -> python2.7. >> >> -- >> Marc-Andre Lemburg >> eGenix.com >> >> Professional Python Services directly from the Source (#1, Mar 11 2015) >> >>> Python Projects, Coaching and Consulting ... http://www.egenix.com/ >> >>> mxODBC Plone/Zope Database Adapter ... http://zope.egenix.com/ >> >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ >> ________________________________________________________________________ >> >> ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: >> >> eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 >> D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg >> Registered at Amtsgericht Duesseldorf: HRB 46611 >> http://www.egenix.com/company/contact/ >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From greg at krypto.org Wed Mar 11 23:41:47 2015 From: greg at krypto.org (Gregory P. Smith) Date: Wed, 11 Mar 2015 22:41:47 +0000 Subject: [Python-ideas] Migration of /usr/bin/python to python3 In-Reply-To: <5500A7AE.9020400@egenix.com> References: <20150311194325.GA15390@python.ca> <5500A7AE.9020400@egenix.com> Message-ID: I think /usr/bin/env is a bad idea to promote. It changes from an explicit designation of what you need in order to execute to something being derived from the users $PATH. It is *less specific* than /usr/bin/python. For some people that is the goal, but I discourage it. I do agree with Donald (and Nick from the other conversation elsewhere) that once this is framed as "the OS won't have Python 2.7 installed at all" it is a much better idea: Yes, we will eventually need to update PEP 394 to recommend under what circumstances /usr/bin/python -> python3.6 is acceptable. (it should never change meaning on a given OS release based on the packages installed and 2.7 won't be required for the default install images so 3.x makes sense) The harsh reality is that anyone who does not explicitly specify a specific X.Y version of Python in their #! line via has effectively declared that they do not care, consequences be damned. If your code supports multiple versions, fixing up the #! line to use the locally appropriate one is an install time task. -gps On Wed, Mar 11, 2015 at 1:38 PM M.-A. Lemburg wrote: > On 11.03.2015 21:03, David Mertz wrote: > > https://www.python.org/dev/peps/pep-0394/ > > I think the migration should start with modifying scripts to use > > #!/usr/bin/env python2 > > when they are Python 2 only (*) and > > #!/usr/bin/env python3 > > when they are Python 3 only and > > #!/usr/bin/env python > > only when they support both Python 2 and 3. > > "Explicit is better than implicit" and all that Zen :-) > > Once that's done, switching the symlink is really a no-brainer. > > The recipe for this is easy too: > > 1. replace all "#!/usr/bin/env python" with "#!/usr/bin/env python2" > 2. migrate your scripts one by one to either Python 3.x or > to Python 2.7 + 3.4+ > 3. after migration replace "#!/usr/bin/env python2" with > "#!/usr/bin/env python3" or "#!/usr/bin/env python" resp. > > (*) Some OSes may require to use python2.7, if they don't come > with a symlink from python2 -> python2.7. > > -- > Marc-Andre Lemburg > eGenix.com > > Professional Python Services directly from the Source (#1, Mar 11 2015) > >>> Python Projects, Coaching and Consulting ... http://www.egenix.com/ > >>> mxODBC Plone/Zope Database Adapter ... http://zope.egenix.com/ > >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ > ________________________________________________________________________ > > ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: > > eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 > D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg > Registered at Amtsgericht Duesseldorf: HRB 46611 > http://www.egenix.com/company/contact/ > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ianlee1521 at gmail.com Wed Mar 11 23:45:15 2015 From: ianlee1521 at gmail.com (Ian Lee) Date: Wed, 11 Mar 2015 15:45:15 -0700 Subject: [Python-ideas] Migration of /usr/bin/python to python3 In-Reply-To: <9F862F85-77CE-4F68-8223-EDCEF9818836@yahoo.com> References: <20150311194325.GA15390@python.ca> <5500A7AE.9020400@egenix.com> <9F862F85-77CE-4F68-8223-EDCEF9818836@yahoo.com> Message-ID: Sure, and I agree that called "ipython2" / "ipython3" explicitly is better. I just expected it would default to use whatever "python" is implicitly on my path. E.g. that "python" and "ipython" would use the same interpreter. ~ Ian Lee On Mar 11, 2015 3:29 PM, "Andrew Barnert" wrote: > On Mar 11, 2015, at 2:52 PM, Ian Lee wrote: > > +1 > > An issue that I've noticed recently (I haven't had the time to fully track > this one down yet, but I suspect it's a setuptools issue) takes a "last one > wins" approach to scripts that get written into the same bin directory > (e.g. --user installs). > > > Many packages will install spamX and spamX.Y scripts as well as plain > spam, so you can deal with this the same way you deal with Python > instead--run ipython3 or ipython3.4 instead of just ipython. (Not to > mention ipython_pypy3!) Also, sometimes python2 and python3 will install to > different directories (/usr for the one that's part of the system, > /usr/local for others, or /Library/Framework/Python/Versions/X.Y/bin...), > so "last one wins" isn't even predictable cross-platform; you have to be > able to deal with either "last one wins" or "controlled by PATH" when you > want to just run "ipython". But really, since these scripts don't need to > be run by a shbang or other automated mechanism, and the user who > intentionally installs IPython for both 2.7 and 3.4 is going to need some > way to manually select the one they want to run each time anyway. So I > think this solution isn't too bad: 90% of the time, a script is only > installed for one Python version so "last one wins" works trivially; most > of the rest of the time, the user has two or more and needs to use > "ipython2" vs. "ipython3" rather than just "ipython" because how else could > be use both as desired, so "last one wins" is irrelevant.The thing is, I > don't think you get this script versioning automatically from setuptools. > If you don't, maybe you should? > > > $ python2.7 -m pip install --user ipython >> >> $ head -n1 ~/.local/bin/ipython* >> ==> /home/ianlee1521/.local/bin/ipython <== >> #!/usr/bin/python2.7 >> >> ==> /home/ianlee1521/.local/bin/ipython2 <== >> #!/usr/bin/python2.7 >> >> $ python3.4 -m pip install --user ipython >> >> $ head -n1 ~/.local/bin/ipython* >> ==> /home/ianlee1521/.local/bin/ipython <== >> #!/usr/bin/python3.4 >> >> ==> /home/ianlee1521/.local/bin/ipython2 <== >> #!/usr/bin/python2.7 >> >> ==> /home/ianlee1521/.local/bin/ipython3 <== >> #!/usr/bin/python3.4 > > > So the packages end up in the separate, versioned directories > (~/.local/lib/pythonX.Y/site-packages), but with the scripts ending up in > the same bin directory, the implicit script "~/.local/bin/ipython" ends up > changing to use the hashbang of the Python version which last installed the > package (see highlighted above). > > I would expect that the implicit script ("~/.local/bin/ipython") should > use the implicit python version, e.g. "#!/usr/bin/python". > > Alternatively, maybe I just shouldn't ever be installing user versions of > both Python 2 and Python 3 versions of a package. > > > > ~ Ian Lee > > On Wed, Mar 11, 2015 at 1:38 PM, M.-A. Lemburg wrote: > >> On 11.03.2015 21:03, David Mertz wrote: >> > https://www.python.org/dev/peps/pep-0394/ >> >> I think the migration should start with modifying scripts to use >> >> #!/usr/bin/env python2 >> >> when they are Python 2 only (*) and >> >> #!/usr/bin/env python3 >> >> when they are Python 3 only and >> >> #!/usr/bin/env python >> >> only when they support both Python 2 and 3. >> >> "Explicit is better than implicit" and all that Zen :-) >> >> Once that's done, switching the symlink is really a no-brainer. >> >> The recipe for this is easy too: >> >> 1. replace all "#!/usr/bin/env python" with "#!/usr/bin/env python2" >> 2. migrate your scripts one by one to either Python 3.x or >> to Python 2.7 + 3.4+ >> 3. after migration replace "#!/usr/bin/env python2" with >> "#!/usr/bin/env python3" or "#!/usr/bin/env python" resp. >> >> (*) Some OSes may require to use python2.7, if they don't come >> with a symlink from python2 -> python2.7. >> >> -- >> Marc-Andre Lemburg >> eGenix.com >> >> Professional Python Services directly from the Source (#1, Mar 11 2015) >> >>> Python Projects, Coaching and Consulting ... http://www.egenix.com/ >> >>> mxODBC Plone/Zope Database Adapter ... http://zope.egenix.com/ >> >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ >> ________________________________________________________________________ >> >> ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: >> >> eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 >> D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg >> Registered at Amtsgericht Duesseldorf: HRB 46611 >> http://www.egenix.com/company/contact/ >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ >> > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From abarnert at yahoo.com Wed Mar 11 23:45:19 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Wed, 11 Mar 2015 15:45:19 -0700 Subject: [Python-ideas] Migration of /usr/bin/python to python3 In-Reply-To: <5500A7AE.9020400@egenix.com> References: <20150311194325.GA15390@python.ca> <5500A7AE.9020400@egenix.com> Message-ID: <3485D6D4-E09B-4617-AC5B-2D07BB79435F@yahoo.com> On Mar 11, 2015, at 1:38 PM, M.-A. Lemburg wrote: > >> On 11.03.2015 21:03, David Mertz wrote: >> https://www.python.org/dev/peps/pep-0394/ > > I think the migration should start with modifying scripts to use > > #!/usr/bin/env python2 > > when they are Python 2 only (*) and > > #!/usr/bin/env python3 > > when they are Python 3 only and > > #!/usr/bin/env python > > only when they support both Python 2 and 3. Except that you need a step 0: Not every system out there is PEP 394-compliant. Most notably, the built-in Python on OS X has nothing named python2, and python is a wrapper executable that selects between python2.5, python2.6, and python2.7. Some of the popular third-party Python-with-extra-batteries installs still give you only python and maybe python2.7, but not python2. And most linux distros' built-in and standard repo Pythons are PEP 394-compliant, but "most Linux distros" is not the same thing as "all non-OS X Unix-like systems". So if you want to move this forward, I think someone needs to create a central place with all the info on how far we are from 100% PEP 394 compliance (including but reports to each vendor, etc.) and start pressuring the vendors who don't comply. In the case of Apple, unless you can get them to add a python2 executable in bug fix releases of 10.9 and 10.10, rather than just in 10.11, it'll be a long time before python2 becomes a portable command. (Even if you could, would dropping OS X 10.6-10.8 be acceptable today for most scripts?) > "Explicit is better than implicit" and all that Zen :-) > > Once that's done, switching the symlink is really a no-brainer. > > The recipe for this is easy too: > > 1. replace all "#!/usr/bin/env python" with "#!/usr/bin/env python2" > 2. migrate your scripts one by one to either Python 3.x or > to Python 2.7 + 3.4+ > 3. after migration replace "#!/usr/bin/env python2" with > "#!/usr/bin/env python3" or "#!/usr/bin/env python" resp. > > (*) Some OSes may require to use python2.7, if they don't come > with a symlink from python2 -> python2.7. > > -- > Marc-Andre Lemburg > eGenix.com > > Professional Python Services directly from the Source (#1, Mar 11 2015) >>>> Python Projects, Coaching and Consulting ... http://www.egenix.com/ >>>> mxODBC Plone/Zope Database Adapter ... http://zope.egenix.com/ >>>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ > ________________________________________________________________________ > > ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: > > eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 > D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg > Registered at Amtsgericht Duesseldorf: HRB 46611 > http://www.egenix.com/company/contact/ > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ From chris.barker at noaa.gov Wed Mar 11 23:54:04 2015 From: chris.barker at noaa.gov (Chris Barker) Date: Wed, 11 Mar 2015 15:54:04 -0700 Subject: [Python-ideas] Migration of /usr/bin/python to python3 In-Reply-To: References: <20150311194325.GA15390@python.ca> <5500A7AE.9020400@egenix.com> Message-ID: On Wed, Mar 11, 2015 at 3:41 PM, Gregory P. Smith wrote: > I think /usr/bin/env is a bad idea to promote. It changes from an explicit > designation of what you need in order to execute to something being derived > from the users $PATH. It is *less specific* than /usr/bin/python. For some > people that is the goal, but I discourage it. > I don't think the goal is specificity, or lack there of, but a shift in control: Who controls where python is installed? If you put an explicit full path on the #! line, then the script author controls which python is used, AND where it must be installed. If you put /usr/bin env on the #! line, then the script author is saying 'use python", and the script user can have python installed anywhere they want, and can use PATH to tell everything what the default it. I think that _should_ be encouraged -- but ideally with some versioning: #!/usr/bin/env python2 or even: #!/usr/bin/env python3.4 If any of you remember what a pain it was when RedHat installed system tools with (I think): !#/usr/bin/env python You'll know what a nightmare that was -- you could not upgrade the 'default" python without breaking stuff. So system tools should probably use the full specific path -- the script is tied closely to the environment. But if they had at least put a version in there, it would have worked fine: #!/usr/bin/env python1.5 These days, I try to use distutils or setuptools to install scripts, and they can do the right thing to use the correct python to start up anyway... -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From abarnert at yahoo.com Wed Mar 11 23:52:12 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Wed, 11 Mar 2015 15:52:12 -0700 Subject: [Python-ideas] Migration of /usr/bin/python to python3 In-Reply-To: References: <20150311194325.GA15390@python.ca> <5500A7AE.9020400@egenix.com> <9F862F85-77CE-4F68-8223-EDCEF9818836@yahoo.com> Message-ID: <0E24CF04-2B74-4E37-B69D-BF2B53A6E416@yahoo.com> On Mar 11, 2015, at 3:45 PM, Ian Lee wrote: > Sure, and I agree that called "ipython2" / "ipython3" explicitly is better. I just expected it would default to use whatever "python" is implicitly on my path. E.g. that "python" and "ipython" would use the same interpreter. > That's actually not a bad idea; I didn't even think about it because Unix stuff doesn't "naturally" work that way, but it's probably not too hard to do it "unnaturally". Maybe setuptools could check "there's something on the path called python that doesn't run the same version of Python that I'm running right now, so only install ipython3.4 and ipython3, not ipython". > ~ Ian Lee > >> On Mar 11, 2015 3:29 PM, "Andrew Barnert" wrote: >>> On Mar 11, 2015, at 2:52 PM, Ian Lee wrote: >>> >>> +1 >>> >>> An issue that I've noticed recently (I haven't had the time to fully track this one down yet, but I suspect it's a setuptools issue) takes a "last one wins" approach to scripts that get written into the same bin directory (e.g. --user installs). >> >> Many packages will install spamX and spamX.Y scripts as well as plain spam, so you can deal with this the same way you deal with Python instead--run ipython3 or ipython3.4 instead of just ipython. (Not to mention ipython_pypy3!) Also, sometimes python2 and python3 will install to different directories (/usr for the one that's part of the system, /usr/local for others, or /Library/Framework/Python/Versions/X.Y/bin...), so "last one wins" isn't even predictable cross-platform; you have to be able to deal with either "last one wins" or "controlled by PATH" when you want to just run "ipython". But really, since these scripts don't need to be run by a shbang or other automated mechanism, and the user who intentionally installs IPython for both 2.7 and 3.4 is going to need some way to manually select the one they want to run each time anyway. So I think this solution isn't too bad: 90% of the time, a script is only installed for one Python version so "last one wins" works trivially; most of the rest of the time, the user has two or more and needs to use "ipython2" vs. "ipython3" rather than just "ipython" because how else could be use both as desired, so "last one wins" is irrelevant.The thing is, I don't think you get this script versioning automatically from setuptools. If you don't, maybe you should? >> >>> >>>> $ python2.7 -m pip install --user ipython >>>> >>>> $ head -n1 ~/.local/bin/ipython* >>>> ==> /home/ianlee1521/.local/bin/ipython <== >>>> #!/usr/bin/python2.7 >>>> >>>> ==> /home/ianlee1521/.local/bin/ipython2 <== >>>> #!/usr/bin/python2.7 >>>> >>>> $ python3.4 -m pip install --user ipython >>>> >>>> $ head -n1 ~/.local/bin/ipython* >>>> ==> /home/ianlee1521/.local/bin/ipython <== >>>> #!/usr/bin/python3.4 >>>> >>>> ==> /home/ianlee1521/.local/bin/ipython2 <== >>>> #!/usr/bin/python2.7 >>>> >>>> ==> /home/ianlee1521/.local/bin/ipython3 <== >>>> #!/usr/bin/python3.4 >>> >>> So the packages end up in the separate, versioned directories (~/.local/lib/pythonX.Y/site-packages), but with the scripts ending up in the same bin directory, the implicit script "~/.local/bin/ipython" ends up changing to use the hashbang of the Python version which last installed the package (see highlighted above). >>> >>> I would expect that the implicit script ("~/.local/bin/ipython") should use the implicit python version, e.g. "#!/usr/bin/python". >>> >>> Alternatively, maybe I just shouldn't ever be installing user versions of both Python 2 and Python 3 versions of a package. >>> >>> >>> ~ Ian Lee >>> >>>> On Wed, Mar 11, 2015 at 1:38 PM, M.-A. Lemburg wrote: >>>> On 11.03.2015 21:03, David Mertz wrote: >>>> > https://www.python.org/dev/peps/pep-0394/ >>>> >>>> I think the migration should start with modifying scripts to use >>>> >>>> #!/usr/bin/env python2 >>>> >>>> when they are Python 2 only (*) and >>>> >>>> #!/usr/bin/env python3 >>>> >>>> when they are Python 3 only and >>>> >>>> #!/usr/bin/env python >>>> >>>> only when they support both Python 2 and 3. >>>> >>>> "Explicit is better than implicit" and all that Zen :-) >>>> >>>> Once that's done, switching the symlink is really a no-brainer. >>>> >>>> The recipe for this is easy too: >>>> >>>> 1. replace all "#!/usr/bin/env python" with "#!/usr/bin/env python2" >>>> 2. migrate your scripts one by one to either Python 3.x or >>>> to Python 2.7 + 3.4+ >>>> 3. after migration replace "#!/usr/bin/env python2" with >>>> "#!/usr/bin/env python3" or "#!/usr/bin/env python" resp. >>>> >>>> (*) Some OSes may require to use python2.7, if they don't come >>>> with a symlink from python2 -> python2.7. >>>> >>>> -- >>>> Marc-Andre Lemburg >>>> eGenix.com >>>> >>>> Professional Python Services directly from the Source (#1, Mar 11 2015) >>>> >>> Python Projects, Coaching and Consulting ... http://www.egenix.com/ >>>> >>> mxODBC Plone/Zope Database Adapter ... http://zope.egenix.com/ >>>> >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ >>>> ________________________________________________________________________ >>>> >>>> ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: >>>> >>>> eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 >>>> D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg >>>> Registered at Amtsgericht Duesseldorf: HRB 46611 >>>> http://www.egenix.com/company/contact/ >>>> _______________________________________________ >>>> Python-ideas mailing list >>>> Python-ideas at python.org >>>> https://mail.python.org/mailman/listinfo/python-ideas >>>> Code of Conduct: http://python.org/psf/codeofconduct/ >>> >>> _______________________________________________ >>> Python-ideas mailing list >>> Python-ideas at python.org >>> https://mail.python.org/mailman/listinfo/python-ideas >>> Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From rymg19 at gmail.com Thu Mar 12 00:02:49 2015 From: rymg19 at gmail.com (Ryan Gonzalez) Date: Wed, 11 Mar 2015 18:02:49 -0500 Subject: [Python-ideas] Migration of /usr/bin/python to python3 In-Reply-To: References: <20150311194325.GA15390@python.ca> <5500A7AE.9020400@egenix.com> Message-ID: Bad idea? What about when you're running simple scripts you've downloaded and Python 3 is on an external hard drive, not /usr/bin? That's my usage case, anyway. On Wed, Mar 11, 2015 at 5:41 PM, Gregory P. Smith wrote: > I think /usr/bin/env is a bad idea to promote. It changes from an explicit > designation of what you need in order to execute to something being derived > from the users $PATH. It is *less specific* than /usr/bin/python. For some > people that is the goal, but I discourage it. > > I do agree with Donald (and Nick from the other conversation elsewhere) > that once this is framed as "the OS won't have Python 2.7 installed at all" > it is a much better idea: Yes, we will eventually need to update PEP 394 to > recommend under what circumstances /usr/bin/python -> python3.6 is > acceptable. (it should never change meaning on a given OS release based on > the packages installed and 2.7 won't be required for the default install > images so 3.x makes sense) > > The harsh reality is that anyone who does not explicitly specify a > specific X.Y version of Python in their #! line via has effectively > declared that they do not care, consequences be damned. If your code > supports multiple versions, fixing up the #! line to use the locally > appropriate one is an install time task. > > -gps > > On Wed, Mar 11, 2015 at 1:38 PM M.-A. Lemburg wrote: > >> On 11.03.2015 21:03, David Mertz wrote: >> > https://www.python.org/dev/peps/pep-0394/ >> >> I think the migration should start with modifying scripts to use >> >> #!/usr/bin/env python2 >> >> when they are Python 2 only (*) and >> >> #!/usr/bin/env python3 >> >> when they are Python 3 only and >> >> #!/usr/bin/env python >> >> only when they support both Python 2 and 3. >> >> "Explicit is better than implicit" and all that Zen :-) >> >> Once that's done, switching the symlink is really a no-brainer. >> >> The recipe for this is easy too: >> >> 1. replace all "#!/usr/bin/env python" with "#!/usr/bin/env python2" >> 2. migrate your scripts one by one to either Python 3.x or >> to Python 2.7 + 3.4+ >> 3. after migration replace "#!/usr/bin/env python2" with >> "#!/usr/bin/env python3" or "#!/usr/bin/env python" resp. >> >> (*) Some OSes may require to use python2.7, if they don't come >> with a symlink from python2 -> python2.7. >> >> -- >> Marc-Andre Lemburg >> eGenix.com >> >> Professional Python Services directly from the Source (#1, Mar 11 2015) >> >>> Python Projects, Coaching and Consulting ... http://www.egenix.com/ >> >>> mxODBC Plone/Zope Database Adapter ... http://zope.egenix.com/ >> >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ >> ________________________________________________________________________ >> >> ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: >> >> eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 >> D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg >> Registered at Amtsgericht Duesseldorf: HRB 46611 >> http://www.egenix.com/company/contact/ >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ >> > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- Ryan [ERROR]: Your autotools build scripts are 200 lines longer than your program. Something?s wrong. http://kirbyfan64.github.io/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Thu Mar 12 00:03:33 2015 From: donald at stufft.io (Donald Stufft) Date: Wed, 11 Mar 2015 19:03:33 -0400 Subject: [Python-ideas] Migration of /usr/bin/python to python3 In-Reply-To: <0E24CF04-2B74-4E37-B69D-BF2B53A6E416@yahoo.com> References: <20150311194325.GA15390@python.ca> <5500A7AE.9020400@egenix.com> <9F862F85-77CE-4F68-8223-EDCEF9818836@yahoo.com> <0E24CF04-2B74-4E37-B69D-BF2B53A6E416@yahoo.com> Message-ID: <8E9600FA-2CE4-4A01-BBC2-B1F950418288@stufft.io> > On Mar 11, 2015, at 6:52 PM, Andrew Barnert wrote: > > On Mar 11, 2015, at 3:45 PM, Ian Lee > wrote: >> Sure, and I agree that called "ipython2" / "ipython3" explicitly is better. I just expected it would default to use whatever "python" is implicitly on my path. E.g. that "python" and "ipython" would use the same interpreter. >> > That's actually not a bad idea; I didn't even think about it because Unix stuff doesn't "naturally" work that way, but it's probably not too hard to do it "unnaturally". > > Maybe setuptools could check "there's something on the path called python that doesn't run the same version of Python that I'm running right now, so only install ipython3.4 and ipython3, not ipython?. This is a problem that we?ve yet to tackle really (besides special cases for pip itself). You can?t really do it reasonably without not making Wheels, or making wheels that are python version specific. This is because the build process runs on the author?s machine and ?bakes? in the names of the commands at that time. It?s likely going to require figuring out which trade off makes the most sense, adding explicit fooX.Y is one option, but that leaves into question what we do with alternate interpreters like PyPy or Jython (including versioning them, because you might have multiple versions of them installed). It also can cause issues when you have multiple things called ?python2? or ?python2.7? on your path, which version of ?python2? does ?foo2? call? Obviously it?s possible to just define something and say ?this is what it is? but we need to figure out what that thing should be, and if the trade offs make sense. On the other side, executing versioned commands with -m is not quite as nice (python2.7 -m pip is more letters than pip2.7 and doesn?t autocomplete) however it is really explicit under which interpreter it?s going to run and support for things like alternate interpreters and multiple versions is built in. >> ~ Ian Lee >> >> On Mar 11, 2015 3:29 PM, "Andrew Barnert" > wrote: >> On Mar 11, 2015, at 2:52 PM, Ian Lee > wrote: >> >>> +1 >>> >>> An issue that I've noticed recently (I haven't had the time to fully track this one down yet, but I suspect it's a setuptools issue) takes a "last one wins" approach to scripts that get written into the same bin directory (e.g. --user installs). >> >> Many packages will install spamX and spamX.Y scripts as well as plain spam, so you can deal with this the same way you deal with Python instead--run ipython3 or ipython3.4 instead of just ipython. (Not to mention ipython_pypy3!) Also, sometimes python2 and python3 will install to different directories (/usr for the one that's part of the system, /usr/local for others, or /Library/Framework/Python/Versions/X.Y/bin...), so "last one wins" isn't even predictable cross-platform; you have to be able to deal with either "last one wins" or "controlled by PATH" when you want to just run "ipython". But really, since these scripts don't need to be run by a shbang or other automated mechanism, and the user who intentionally installs IPython for both 2.7 and 3.4 is going to need some way to manually select the one they want to run each time anyway. So I think this solution isn't too bad: 90% of the time, a script is only installed for one Python version so "last one wins" works trivially; most of the rest of the time, the user has two or more and needs to use "ipython2" vs. "ipython3" rather than just "ipython" because how else could be use both as desired, so "last one wins" is irrelevant.The thing is, I don't think you get this script versioning automatically from setuptools. If you don't, maybe you should? >> >>> >>> $ python2.7 -m pip install --user ipython >>> >>> $ head -n1 ~/.local/bin/ipython* >>> ==> /home/ianlee1521/.local/bin/ipython <== >>> #!/usr/bin/python2.7 >>> >>> ==> /home/ianlee1521/.local/bin/ipython2 <== >>> #!/usr/bin/python2.7 >>> >>> $ python3.4 -m pip install --user ipython >>> >>> $ head -n1 ~/.local/bin/ipython* >>> ==> /home/ianlee1521/.local/bin/ipython <== >>> #!/usr/bin/python3.4 >>> >>> ==> /home/ianlee1521/.local/bin/ipython2 <== >>> #!/usr/bin/python2.7 >>> >>> ==> /home/ianlee1521/.local/bin/ipython3 <== >>> #!/usr/bin/python3.4 >>> >>> So the packages end up in the separate, versioned directories (~/.local/lib/pythonX.Y/site-packages), but with the scripts ending up in the same bin directory, the implicit script "~/.local/bin/ipython" ends up changing to use the hashbang of the Python version which last installed the package (see highlighted above). >>> >>> I would expect that the implicit script ("~/.local/bin/ipython") should use the implicit python version, e.g. "#!/usr/bin/python". >>> >>> Alternatively, maybe I just shouldn't ever be installing user versions of both Python 2 and Python 3 versions of a package. >>> >>> >>> ~ Ian Lee >>> >>> On Wed, Mar 11, 2015 at 1:38 PM, M.-A. Lemburg > wrote: >>> On 11.03.2015 21:03, David Mertz wrote: >>> > https://www.python.org/dev/peps/pep-0394/ >>> >>> I think the migration should start with modifying scripts to use >>> >>> #!/usr/bin/env python2 >>> >>> when they are Python 2 only (*) and >>> >>> #!/usr/bin/env python3 >>> >>> when they are Python 3 only and >>> >>> #!/usr/bin/env python >>> >>> only when they support both Python 2 and 3. >>> >>> "Explicit is better than implicit" and all that Zen :-) >>> >>> Once that's done, switching the symlink is really a no-brainer. >>> >>> The recipe for this is easy too: >>> >>> 1. replace all "#!/usr/bin/env python" with "#!/usr/bin/env python2" >>> 2. migrate your scripts one by one to either Python 3.x or >>> to Python 2.7 + 3.4+ >>> 3. after migration replace "#!/usr/bin/env python2" with >>> "#!/usr/bin/env python3" or "#!/usr/bin/env python" resp. >>> >>> (*) Some OSes may require to use python2.7, if they don't come >>> with a symlink from python2 -> python2.7. >>> >>> -- >>> Marc-Andre Lemburg >>> eGenix.com >>> >>> Professional Python Services directly from the Source (#1, Mar 11 2015) >>> >>> Python Projects, Coaching and Consulting ... http://www.egenix.com/ >>> >>> mxODBC Plone/Zope Database Adapter ... http://zope.egenix.com/ >>> >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ >>> ________________________________________________________________________ >>> >>> ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: >>> >>> eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 >>> D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg >>> Registered at Amtsgericht Duesseldorf: HRB 46611 >>> http://www.egenix.com/company/contact/ >>> _______________________________________________ >>> Python-ideas mailing list >>> Python-ideas at python.org >>> https://mail.python.org/mailman/listinfo/python-ideas >>> Code of Conduct: http://python.org/psf/codeofconduct/ >>> >>> _______________________________________________ >>> Python-ideas mailing list >>> Python-ideas at python.org >>> https://mail.python.org/mailman/listinfo/python-ideas >>> Code of Conduct: http://python.org/psf/codeofconduct/ _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From ianlee1521 at gmail.com Thu Mar 12 00:03:35 2015 From: ianlee1521 at gmail.com (Ian Lee) Date: Wed, 11 Mar 2015 16:03:35 -0700 Subject: [Python-ideas] Migration of /usr/bin/python to python3 In-Reply-To: <0E24CF04-2B74-4E37-B69D-BF2B53A6E416@yahoo.com> References: <20150311194325.GA15390@python.ca> <5500A7AE.9020400@egenix.com> <9F862F85-77CE-4F68-8223-EDCEF9818836@yahoo.com> <0E24CF04-2B74-4E37-B69D-BF2B53A6E416@yahoo.com> Message-ID: ~ Ian Lee On Wed, Mar 11, 2015 at 3:52 PM, Andrew Barnert wrote: > On Mar 11, 2015, at 3:45 PM, Ian Lee wrote: > > Sure, and I agree that called "ipython2" / "ipython3" explicitly is > better. I just expected it would default to use whatever "python" is > implicitly on my path. E.g. that "python" and "ipython" would use the same > interpreter. > > That's actually not a bad idea; I didn't even think about it because Unix > stuff doesn't "naturally" work that way, but it's probably not too hard to > do it "unnaturally". > > Maybe setuptools could check "there's something on the path called python > that doesn't run the same version of Python that I'm running right now, so > only install ipython3.4 and ipython3, not ipython". > I'm not sure this should be a decision at install time to set the hashbang. Instead I'd propose the hashbangs should end up: ipython ==> "#! /usr/bin/env python" ipython2 ==> "#! /usr/bin/env python2" ipython3 ==> "#! /usr/bin/env python3" > ~ Ian Lee > On Mar 11, 2015 3:29 PM, "Andrew Barnert" wrote: > >> On Mar 11, 2015, at 2:52 PM, Ian Lee wrote: >> >> +1 >> >> An issue that I've noticed recently (I haven't had the time to fully >> track this one down yet, but I suspect it's a setuptools issue) takes a >> "last one wins" approach to scripts that get written into the same bin >> directory (e.g. --user installs). >> >> >> Many packages will install spamX and spamX.Y scripts as well as plain >> spam, so you can deal with this the same way you deal with Python >> instead--run ipython3 or ipython3.4 instead of just ipython. (Not to >> mention ipython_pypy3!) Also, sometimes python2 and python3 will install to >> different directories (/usr for the one that's part of the system, >> /usr/local for others, or /Library/Framework/Python/Versions/X.Y/bin...), >> so "last one wins" isn't even predictable cross-platform; you have to be >> able to deal with either "last one wins" or "controlled by PATH" when you >> want to just run "ipython". But really, since these scripts don't need to >> be run by a shbang or other automated mechanism, and the user who >> intentionally installs IPython for both 2.7 and 3.4 is going to need some >> way to manually select the one they want to run each time anyway. So I >> think this solution isn't too bad: 90% of the time, a script is only >> installed for one Python version so "last one wins" works trivially; most >> of the rest of the time, the user has two or more and needs to use >> "ipython2" vs. "ipython3" rather than just "ipython" because how else could >> be use both as desired, so "last one wins" is irrelevant.The thing is, I >> don't think you get this script versioning automatically from setuptools. >> If you don't, maybe you should? >> >> >> $ python2.7 -m pip install --user ipython >>> >>> $ head -n1 ~/.local/bin/ipython* >>> ==> /home/ianlee1521/.local/bin/ipython <== >>> #!/usr/bin/python2.7 >>> >>> ==> /home/ianlee1521/.local/bin/ipython2 <== >>> #!/usr/bin/python2.7 >>> >>> $ python3.4 -m pip install --user ipython >>> >>> $ head -n1 ~/.local/bin/ipython* >>> ==> /home/ianlee1521/.local/bin/ipython <== >>> #!/usr/bin/python3.4 >>> >>> ==> /home/ianlee1521/.local/bin/ipython2 <== >>> #!/usr/bin/python2.7 >>> >>> ==> /home/ianlee1521/.local/bin/ipython3 <== >>> #!/usr/bin/python3.4 >> >> >> So the packages end up in the separate, versioned directories >> (~/.local/lib/pythonX.Y/site-packages), but with the scripts ending up in >> the same bin directory, the implicit script "~/.local/bin/ipython" ends up >> changing to use the hashbang of the Python version which last installed the >> package (see highlighted above). >> >> I would expect that the implicit script ("~/.local/bin/ipython") should >> use the implicit python version, e.g. "#!/usr/bin/python". >> >> Alternatively, maybe I just shouldn't ever be installing user versions of >> both Python 2 and Python 3 versions of a package. >> >> >> >> ~ Ian Lee >> >> On Wed, Mar 11, 2015 at 1:38 PM, M.-A. Lemburg wrote: >> >>> On 11.03.2015 21:03, David Mertz wrote: >>> > https://www.python.org/dev/peps/pep-0394/ >>> >>> I think the migration should start with modifying scripts to use >>> >>> #!/usr/bin/env python2 >>> >>> when they are Python 2 only (*) and >>> >>> #!/usr/bin/env python3 >>> >>> when they are Python 3 only and >>> >>> #!/usr/bin/env python >>> >>> only when they support both Python 2 and 3. >>> >>> "Explicit is better than implicit" and all that Zen :-) >>> >>> Once that's done, switching the symlink is really a no-brainer. >>> >>> The recipe for this is easy too: >>> >>> 1. replace all "#!/usr/bin/env python" with "#!/usr/bin/env python2" >>> 2. migrate your scripts one by one to either Python 3.x or >>> to Python 2.7 + 3.4+ >>> 3. after migration replace "#!/usr/bin/env python2" with >>> "#!/usr/bin/env python3" or "#!/usr/bin/env python" resp. >>> >>> (*) Some OSes may require to use python2.7, if they don't come >>> with a symlink from python2 -> python2.7. >>> >>> -- >>> Marc-Andre Lemburg >>> eGenix.com >>> >>> Professional Python Services directly from the Source (#1, Mar 11 2015) >>> >>> Python Projects, Coaching and Consulting ... http://www.egenix.com/ >>> >>> mxODBC Plone/Zope Database Adapter ... http://zope.egenix.com/ >>> >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ >>> ________________________________________________________________________ >>> >>> ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: >>> >>> eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 >>> D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg >>> Registered at Amtsgericht Duesseldorf: HRB 46611 >>> http://www.egenix.com/company/contact/ >>> _______________________________________________ >>> Python-ideas mailing list >>> Python-ideas at python.org >>> https://mail.python.org/mailman/listinfo/python-ideas >>> Code of Conduct: http://python.org/psf/codeofconduct/ >>> >> >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ >> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From ianlee1521 at gmail.com Thu Mar 12 00:03:42 2015 From: ianlee1521 at gmail.com (Ian Lee) Date: Wed, 11 Mar 2015 16:03:42 -0700 Subject: [Python-ideas] Migration of /usr/bin/python to python3 In-Reply-To: References: <20150311194325.GA15390@python.ca> <5500A7AE.9020400@egenix.com> Message-ID: ~ Ian Lee On Wed, Mar 11, 2015 at 3:54 PM, Chris Barker wrote: > On Wed, Mar 11, 2015 at 3:41 PM, Gregory P. Smith wrote: > >> I think /usr/bin/env is a bad idea to promote. It changes from an >> explicit designation of what you need in order to execute to something >> being derived from the users $PATH. It is *less specific* than >> /usr/bin/python. For some people that is the goal, but I discourage it. >> > > I don't think the goal is specificity, or lack there of, but a shift in > control: > > Who controls where python is installed? > > If you put an explicit full path on the #! line, then the script author > controls which python is used, AND where it must be installed. > > If you put /usr/bin env on the #! line, then the script author is saying > 'use python", and the script user can have python installed anywhere they > want, and can use PATH to tell everything what the default it. > > I think that _should_ be encouraged -- but ideally with some versioning: > > #!/usr/bin/env python2 > > or even: > > #!/usr/bin/env python3.4 > +1 to encouraging this. > > If any of you remember what a pain it was when RedHat installed system > tools with (I think): > > !#/usr/bin/env python > > You'll know what a nightmare that was -- you could not upgrade the > 'default" python without breaking stuff. > > So system tools should probably use the full specific path -- the script > is tied closely to the environment. But if they had at least put a version > in there, it would have worked fine: > > #!/usr/bin/env python1.5 > > These days, I try to use distutils or setuptools to install scripts, and > they can do the right thing to use the correct python to start up anyway... > > -CHB > > > > -- > > Christopher Barker, Ph.D. > Oceanographer > > Emergency Response Division > NOAA/NOS/OR&R (206) 526-6959 voice > 7600 Sand Point Way NE (206) 526-6329 fax > Seattle, WA 98115 (206) 526-6317 main reception > > Chris.Barker at noaa.gov > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rymg19 at gmail.com Thu Mar 12 00:12:00 2015 From: rymg19 at gmail.com (Ryan Gonzalez) Date: Wed, 11 Mar 2015 18:12:00 -0500 Subject: [Python-ideas] Migration of /usr/bin/python to python3 In-Reply-To: References: <20150311194325.GA15390@python.ca> <5500A7AE.9020400@egenix.com> Message-ID: On Wed, Mar 11, 2015 at 5:54 PM, Chris Barker wrote: > On Wed, Mar 11, 2015 at 3:41 PM, Gregory P. Smith wrote: > >> I think /usr/bin/env is a bad idea to promote. It changes from an >> explicit designation of what you need in order to execute to something >> being derived from the users $PATH. It is *less specific* than >> /usr/bin/python. For some people that is the goal, but I discourage it. >> > > I don't think the goal is specificity, or lack there of, but a shift in > control: > > Who controls where python is installed? > > If you put an explicit full path on the #! line, then the script author > controls which python is used, AND where it must be installed. > > If you put /usr/bin env on the #! line, then the script author is saying > 'use python", and the script user can have python installed anywhere they > want, and can use PATH to tell everything what the default it. > > I think that _should_ be encouraged -- but ideally with some versioning: > > #!/usr/bin/env python2 > > or even: > > #!/usr/bin/env python3.4 > > Please, just explicit major versions, not minor ones. python3, not python3.4. Just earlier this year I had to fix about 20 scripts that ran perfectly on 3.4 but whose hashbang explicitly used 3.1. It sucks. > If any of you remember what a pain it was when RedHat installed system > tools with (I think): > > !#/usr/bin/env python > > You'll know what a nightmare that was -- you could not upgrade the > 'default" python without breaking stuff. > > So system tools should probably use the full specific path -- the script > is tied closely to the environment. But if they had at least put a version > in there, it would have worked fine: > > #!/usr/bin/env python1.5 > > These days, I try to use distutils or setuptools to install scripts, and > they can do the right thing to use the correct python to start up anyway... > > -CHB > > > > -- > > Christopher Barker, Ph.D. > Oceanographer > > Emergency Response Division > NOAA/NOS/OR&R (206) 526-6959 voice > 7600 Sand Point Way NE (206) 526-6329 fax > Seattle, WA 98115 (206) 526-6317 main reception > > Chris.Barker at noaa.gov > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- Ryan [ERROR]: Your autotools build scripts are 200 lines longer than your program. Something?s wrong. http://kirbyfan64.github.io/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.barker at noaa.gov Wed Mar 11 22:27:55 2015 From: chris.barker at noaa.gov (Chris Barker) Date: Wed, 11 Mar 2015 14:27:55 -0700 Subject: [Python-ideas] Migration of /usr/bin/python to python3 In-Reply-To: <5500B14A.6090705@stoneleaf.us> References: <20150311194325.GA15390@python.ca> <5500A7AE.9020400@egenix.com> <5500B14A.6090705@stoneleaf.us> Message-ID: On Wed, Mar 11, 2015 at 2:19 PM, Ethan Furman wrote: > On 03/11/2015 01:38 PM, M.-A. Lemburg wrote: > > On 11.03.2015 21:03, David Mertz wrote: > > > I think the migration should start with modifying scripts to use > > > > #!/usr/bin/env python2 > > > > when they are Python 2 only (*) and > > > > #!/usr/bin/env python3 > > > > when they are Python 3 only and > > > > #!/usr/bin/env python > > > > only when they support both Python 2 and 3. > and for that matter: #!/usr/bin/env python3.4 If the minor version matters. We all should have been doing this for years! -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.barker at noaa.gov Thu Mar 12 00:06:22 2015 From: chris.barker at noaa.gov (Chris Barker) Date: Wed, 11 Mar 2015 16:06:22 -0700 Subject: [Python-ideas] Migration of /usr/bin/python to python3 In-Reply-To: <0E24CF04-2B74-4E37-B69D-BF2B53A6E416@yahoo.com> References: <20150311194325.GA15390@python.ca> <5500A7AE.9020400@egenix.com> <9F862F85-77CE-4F68-8223-EDCEF9818836@yahoo.com> <0E24CF04-2B74-4E37-B69D-BF2B53A6E416@yahoo.com> Message-ID: On Wed, Mar 11, 2015 at 3:52 PM, Andrew Barnert wrote: > Maybe setuptools could check "there's something on the path called python > that doesn't run the same version of Python that I'm running right now, so > only install ipython3.4 and ipython3, not ipython". > hmm -- that seems odd -- if you re-arranged the path after it was installed, it would be different??? I think you can count on folks use PATH to determine what they want as default -- so in python2, setuptools would put ipython and ipython2 and in the python3 install, it would put: ipython and ipython3 so if you asked for ipython3 or ipython2, you'd get what you asked for -- and if you asked for ipython the PATH would determine what you'd get. -CHB > ~ Ian Lee > On Mar 11, 2015 3:29 PM, "Andrew Barnert" wrote: > >> On Mar 11, 2015, at 2:52 PM, Ian Lee wrote: >> >> +1 >> >> An issue that I've noticed recently (I haven't had the time to fully >> track this one down yet, but I suspect it's a setuptools issue) takes a >> "last one wins" approach to scripts that get written into the same bin >> directory (e.g. --user installs). >> >> >> Many packages will install spamX and spamX.Y scripts as well as plain >> spam, so you can deal with this the same way you deal with Python >> instead--run ipython3 or ipython3.4 instead of just ipython. (Not to >> mention ipython_pypy3!) Also, sometimes python2 and python3 will install to >> different directories (/usr for the one that's part of the system, >> /usr/local for others, or /Library/Framework/Python/Versions/X.Y/bin...), >> so "last one wins" isn't even predictable cross-platform; you have to be >> able to deal with either "last one wins" or "controlled by PATH" when you >> want to just run "ipython". But really, since these scripts don't need to >> be run by a shbang or other automated mechanism, and the user who >> intentionally installs IPython for both 2.7 and 3.4 is going to need some >> way to manually select the one they want to run each time anyway. So I >> think this solution isn't too bad: 90% of the time, a script is only >> installed for one Python version so "last one wins" works trivially; most >> of the rest of the time, the user has two or more and needs to use >> "ipython2" vs. "ipython3" rather than just "ipython" because how else could >> be use both as desired, so "last one wins" is irrelevant.The thing is, I >> don't think you get this script versioning automatically from setuptools. >> If you don't, maybe you should? >> >> >> $ python2.7 -m pip install --user ipython >>> >>> $ head -n1 ~/.local/bin/ipython* >>> ==> /home/ianlee1521/.local/bin/ipython <== >>> #!/usr/bin/python2.7 >>> >>> ==> /home/ianlee1521/.local/bin/ipython2 <== >>> #!/usr/bin/python2.7 >>> >>> $ python3.4 -m pip install --user ipython >>> >>> $ head -n1 ~/.local/bin/ipython* >>> ==> /home/ianlee1521/.local/bin/ipython <== >>> #!/usr/bin/python3.4 >>> >>> ==> /home/ianlee1521/.local/bin/ipython2 <== >>> #!/usr/bin/python2.7 >>> >>> ==> /home/ianlee1521/.local/bin/ipython3 <== >>> #!/usr/bin/python3.4 >> >> >> So the packages end up in the separate, versioned directories >> (~/.local/lib/pythonX.Y/site-packages), but with the scripts ending up in >> the same bin directory, the implicit script "~/.local/bin/ipython" ends up >> changing to use the hashbang of the Python version which last installed the >> package (see highlighted above). >> >> I would expect that the implicit script ("~/.local/bin/ipython") should >> use the implicit python version, e.g. "#!/usr/bin/python". >> >> Alternatively, maybe I just shouldn't ever be installing user versions of >> both Python 2 and Python 3 versions of a package. >> >> >> >> ~ Ian Lee >> >> On Wed, Mar 11, 2015 at 1:38 PM, M.-A. Lemburg wrote: >> >>> On 11.03.2015 21:03, David Mertz wrote: >>> > https://www.python.org/dev/peps/pep-0394/ >>> >>> I think the migration should start with modifying scripts to use >>> >>> #!/usr/bin/env python2 >>> >>> when they are Python 2 only (*) and >>> >>> #!/usr/bin/env python3 >>> >>> when they are Python 3 only and >>> >>> #!/usr/bin/env python >>> >>> only when they support both Python 2 and 3. >>> >>> "Explicit is better than implicit" and all that Zen :-) >>> >>> Once that's done, switching the symlink is really a no-brainer. >>> >>> The recipe for this is easy too: >>> >>> 1. replace all "#!/usr/bin/env python" with "#!/usr/bin/env python2" >>> 2. migrate your scripts one by one to either Python 3.x or >>> to Python 2.7 + 3.4+ >>> 3. after migration replace "#!/usr/bin/env python2" with >>> "#!/usr/bin/env python3" or "#!/usr/bin/env python" resp. >>> >>> (*) Some OSes may require to use python2.7, if they don't come >>> with a symlink from python2 -> python2.7. >>> >>> -- >>> Marc-Andre Lemburg >>> eGenix.com >>> >>> Professional Python Services directly from the Source (#1, Mar 11 2015) >>> >>> Python Projects, Coaching and Consulting ... http://www.egenix.com/ >>> >>> mxODBC Plone/Zope Database Adapter ... http://zope.egenix.com/ >>> >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ >>> ________________________________________________________________________ >>> >>> ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: >>> >>> eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 >>> D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg >>> Registered at Amtsgericht Duesseldorf: HRB 46611 >>> http://www.egenix.com/company/contact/ >>> _______________________________________________ >>> Python-ideas mailing list >>> Python-ideas at python.org >>> https://mail.python.org/mailman/listinfo/python-ideas >>> Code of Conduct: http://python.org/psf/codeofconduct/ >>> >> >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ >> >> > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From random832 at fastmail.us Thu Mar 12 00:42:38 2015 From: random832 at fastmail.us (random832 at fastmail.us) Date: Wed, 11 Mar 2015 19:42:38 -0400 Subject: [Python-ideas] Migration of /usr/bin/python to python3 In-Reply-To: <20150311194325.GA15390@python.ca> References: <20150311194325.GA15390@python.ca> Message-ID: <1426117358.3338188.239148661.1F774D54@webmail.messagingengine.com> On Wed, Mar 11, 2015, at 15:43, Neil Schemenauer wrote: > Traditionally the OS only passes the first option to the > interpreter so anything after that could be used to pass 3.x > specific options. I think some OSes pass everything after the interpreter as one long string rather than ignoring extra arguments. Is there any standard for this? From ethan at stoneleaf.us Thu Mar 12 00:43:27 2015 From: ethan at stoneleaf.us (Ethan Furman) Date: Wed, 11 Mar 2015 16:43:27 -0700 Subject: [Python-ideas] Migration of /usr/bin/python to python3 In-Reply-To: References: <20150311194325.GA15390@python.ca> <5500A7AE.9020400@egenix.com> <9F862F85-77CE-4F68-8223-EDCEF9818836@yahoo.com> <0E24CF04-2B74-4E37-B69D-BF2B53A6E416@yahoo.com> Message-ID: <5500D31F.9080700@stoneleaf.us> On 03/11/2015 04:06 PM, Chris Barker wrote: > I think you can count on folks use PATH to determine what they want as default -- so in python2, setuptools would put > ipython and ipython2 > > and in the python3 install, it would put: > > ipython and ipython3 > > so if you asked for ipython3 or ipython2, you'd get what you asked for -- and if you asked for ipython the PATH would > determine what you'd get. +1 Use the Path, Luke! *ducks and runs* -- ~Ethan~ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: OpenPGP digital signature URL: From mal at egenix.com Thu Mar 12 01:19:32 2015 From: mal at egenix.com (M.-A. Lemburg) Date: Thu, 12 Mar 2015 01:19:32 +0100 Subject: [Python-ideas] Migration of /usr/bin/python to python3 In-Reply-To: <3485D6D4-E09B-4617-AC5B-2D07BB79435F@yahoo.com> References: <20150311194325.GA15390@python.ca> <5500A7AE.9020400@egenix.com> <3485D6D4-E09B-4617-AC5B-2D07BB79435F@yahoo.com> Message-ID: <5500DB94.3020606@egenix.com> On 11.03.2015 23:45, Andrew Barnert wrote: > On Mar 11, 2015, at 1:38 PM, M.-A. Lemburg wrote: >> >>> On 11.03.2015 21:03, David Mertz wrote: >>> https://www.python.org/dev/peps/pep-0394/ >> >> I think the migration should start with modifying scripts to use >> >> #!/usr/bin/env python2 >> >> when they are Python 2 only (*) and >> >> #!/usr/bin/env python3 >> >> when they are Python 3 only and >> >> #!/usr/bin/env python >> >> only when they support both Python 2 and 3. > > Except that you need a step 0: Not every system out there is PEP 394-compliant. Most notably, the built-in Python on OS X has nothing named python2, and python is a wrapper executable that selects between python2.5, python2.6, and python2.7. Some of the popular third-party Python-with-extra-batteries installs still give you only python and maybe python2.7, but not python2. And most linux distros' built-in and standard repo Pythons are PEP 394-compliant, but "most Linux distros" is not the same thing as "all non-OS X Unix-like systems". > > So if you want to move this forward, I think someone needs to create a central place with all the info on how far we are from 100% PEP 394 compliance (including but reports to each vendor, etc.) and start pressuring the vendors who don't comply. > > In the case of Apple, unless you can get them to add a python2 executable in bug fix releases of 10.9 and 10.10, rather than just in 10.11, it'll be a long time before python2 becomes a portable command. (Even if you could, would dropping OS X 10.6-10.8 be acceptable today for most scripts?) I won't comment on the Mac Python installations. There's a reason why we've been using our own installations since the early days of Max OS X :-) In any case, if those binaries don't exist on the target system, the user will get a nice error message saying e.g.: python2: Command not found. and even better: modern OSes will then point the user to the missing packages on their system to make the script run again. I think that's a lot more informative than SyntaxError: Missing parentheses in call to 'print' :-) Anyway, the above is just what I recommend. I'm not a big fan of doing setuptools like magic all over the place when there's a clean solution right in front of you (ok, one may need to take off those cool Python monkey-patching and meta-class sunglasses occasionally to see it ;-)). >> "Explicit is better than implicit" and all that Zen :-) >> >> Once that's done, switching the symlink is really a no-brainer. >> >> The recipe for this is easy too: >> >> 1. replace all "#!/usr/bin/env python" with "#!/usr/bin/env python2" >> 2. migrate your scripts one by one to either Python 3.x or >> to Python 2.7 + 3.4+ >> 3. after migration replace "#!/usr/bin/env python2" with >> "#!/usr/bin/env python3" or "#!/usr/bin/env python" resp. >> >> (*) Some OSes may require to use python2.7, if they don't come >> with a symlink from python2 -> python2.7. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Mar 12 2015) >>> Python Projects, Coaching and Consulting ... http://www.egenix.com/ >>> mxODBC Plone/Zope Database Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From steve at pearwood.info Thu Mar 12 01:20:49 2015 From: steve at pearwood.info (Steven D'Aprano) Date: Thu, 12 Mar 2015 11:20:49 +1100 Subject: [Python-ideas] Migration of /usr/bin/python to python3 In-Reply-To: References: <20150311194325.GA15390@python.ca> <5500A7AE.9020400@egenix.com> Message-ID: <20150312002049.GC7655@ando.pearwood.info> On Wed, Mar 11, 2015 at 06:12:00PM -0500, Ryan Gonzalez wrote: > Please, just explicit major versions, not minor ones. python3, not > python3.4. Just earlier this year I had to fix about 20 scripts that ran > perfectly on 3.4 but whose hashbang explicitly used 3.1. It sucks. Funny about that, I had exactly the opposite experience: a bunch of scripts which worked fine with Python2.3 but failed with 2.6. (`raise "string"` was removed in 2.6.) Fortunately, the author of the scripts was an old-school Unix grey-beard who used the full path to the versioned interpreter, so I was able to install 2.3 and have them just work. Fixing hashbang lines is much simpler than trying to fix the scripts themselves. -- Steve From abarnert at yahoo.com Thu Mar 12 01:45:39 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Wed, 11 Mar 2015 17:45:39 -0700 Subject: [Python-ideas] Migration of /usr/bin/python to python3 In-Reply-To: References: <20150311194325.GA15390@python.ca> <5500A7AE.9020400@egenix.com> <9F862F85-77CE-4F68-8223-EDCEF9818836@yahoo.com> <0E24CF04-2B74-4E37-B69D-BF2B53A6E416@yahoo.com> Message-ID: <84D7FEF9-E1DA-4878-99B3-CF26399CD218@yahoo.com> On Mar 11, 2015, at 4:06 PM, Chris Barker wrote: > >> On Wed, Mar 11, 2015 at 3:52 PM, Andrew Barnert wrote: >> Maybe setuptools could check "there's something on the path called python that doesn't run the same version of Python that I'm running right now, so only install ipython3.4 and ipython3, not ipython". > > hmm -- that seems odd -- if you re-arranged the path after it was installed, it would be different??? Yes. That's essentially why I said Unix doesn't naturally work that way. But it may be appropriate for this specific use case, even if it's a hack. > I think you can count on folks use PATH to determine what they want as default -- so in python2, setuptools would put > ipython and ipython2 > > and in the python3 install, it would put: > > ipython and ipython3 > > so if you asked for ipython3 or ipython2, you'd get what you asked for -- and if you asked for ipython the PATH would determine what you'd get. That's exactly what we already have; the problem is when python2 and python3 have the same scripts directory--say, /usr/local/bin--they both install a script called /usr/local/bin/ipython, with whichever was installed last overwriting the one that was installed first. You might argue that this should never come up, but the fact is that it does. When you install python2 and python3 packages from most package managers, they both go into /usr/local, and both install scripts to /usr/local/bin. And on many platforms, even if the system Python is in /usr, it installs scripts to /usr/local/bin (and that's the right thing to do if /usr/bin is only for system binaries or binaries owned by the system package manager). So, Ian (I think it was him; it's hard to see from this pruned reply...) suggested that just "ipython" should be able to run whatever version of Python is just "python" rather than whichever was installed last. How could we make that work? One option is to have the script #!/usr/bin/env python. But, while that works great for the case where you've installed ipython2 and ipython3 and want ipython to follow your python, the case Ian is trying to solve, it doesn't work an all for the case where you've installed only ipython3 and want to be able to access it as just plain ipython, which is an even more common case. (I do have 4 versions of ipython installed for different CPython and PyPy versions, but I only have one chardetect, etc., because I don't care which version of Python runs the script, I just care what the script does. So, the installer hack would solve both problems, on every kind of platform, for every kind of script. But it's a pretty bad hack. Is it worth doing? I'm not sure. But it's worth thinking about why it would get what Ian wants, at least to see why other solutions won't. > > -CHB > > > > >>> ~ Ian Lee >>> >>>> On Mar 11, 2015 3:29 PM, "Andrew Barnert" wrote: >>>>> On Mar 11, 2015, at 2:52 PM, Ian Lee wrote: >>>>> >>>>> +1 >>>>> >>>>> An issue that I've noticed recently (I haven't had the time to fully track this one down yet, but I suspect it's a setuptools issue) takes a "last one wins" approach to scripts that get written into the same bin directory (e.g. --user installs). >>>> >>>> Many packages will install spamX and spamX.Y scripts as well as plain spam, so you can deal with this the same way you deal with Python instead--run ipython3 or ipython3.4 instead of just ipython. (Not to mention ipython_pypy3!) Also, sometimes python2 and python3 will install to different directories (/usr for the one that's part of the system, /usr/local for others, or /Library/Framework/Python/Versions/X.Y/bin...), so "last one wins" isn't even predictable cross-platform; you have to be able to deal with either "last one wins" or "controlled by PATH" when you want to just run "ipython". But really, since these scripts don't need to be run by a shbang or other automated mechanism, and the user who intentionally installs IPython for both 2.7 and 3.4 is going to need some way to manually select the one they want to run each time anyway. So I think this solution isn't too bad: 90% of the time, a script is only installed for one Python version so "last one wins" works trivially; most of the rest of the time, the user has two or more and needs to use "ipython2" vs. "ipython3" rather than just "ipython" because how else could be use both as desired, so "last one wins" is irrelevant.The thing is, I don't think you get this script versioning automatically from setuptools. If you don't, maybe you should? >>>> >>>>> >>>>>> $ python2.7 -m pip install --user ipython >>>>>> >>>>>> $ head -n1 ~/.local/bin/ipython* >>>>>> ==> /home/ianlee1521/.local/bin/ipython <== >>>>>> #!/usr/bin/python2.7 >>>>>> >>>>>> ==> /home/ianlee1521/.local/bin/ipython2 <== >>>>>> #!/usr/bin/python2.7 >>>>>> >>>>>> $ python3.4 -m pip install --user ipython >>>>>> >>>>>> $ head -n1 ~/.local/bin/ipython* >>>>>> ==> /home/ianlee1521/.local/bin/ipython <== >>>>>> #!/usr/bin/python3.4 >>>>>> >>>>>> ==> /home/ianlee1521/.local/bin/ipython2 <== >>>>>> #!/usr/bin/python2.7 >>>>>> >>>>>> ==> /home/ianlee1521/.local/bin/ipython3 <== >>>>>> #!/usr/bin/python3.4 >>>>> >>>>> So the packages end up in the separate, versioned directories (~/.local/lib/pythonX.Y/site-packages), but with the scripts ending up in the same bin directory, the implicit script "~/.local/bin/ipython" ends up changing to use the hashbang of the Python version which last installed the package (see highlighted above). >>>>> >>>>> I would expect that the implicit script ("~/.local/bin/ipython") should use the implicit python version, e.g. "#!/usr/bin/python". >>>>> >>>>> Alternatively, maybe I just shouldn't ever be installing user versions of both Python 2 and Python 3 versions of a package. >>>>> >>>>> >>>>> ~ Ian Lee >>>>> >>>>>> On Wed, Mar 11, 2015 at 1:38 PM, M.-A. Lemburg wrote: >>>>>> On 11.03.2015 21:03, David Mertz wrote: >>>>>> > https://www.python.org/dev/peps/pep-0394/ >>>>>> >>>>>> I think the migration should start with modifying scripts to use >>>>>> >>>>>> #!/usr/bin/env python2 >>>>>> >>>>>> when they are Python 2 only (*) and >>>>>> >>>>>> #!/usr/bin/env python3 >>>>>> >>>>>> when they are Python 3 only and >>>>>> >>>>>> #!/usr/bin/env python >>>>>> >>>>>> only when they support both Python 2 and 3. >>>>>> >>>>>> "Explicit is better than implicit" and all that Zen :-) >>>>>> >>>>>> Once that's done, switching the symlink is really a no-brainer. >>>>>> >>>>>> The recipe for this is easy too: >>>>>> >>>>>> 1. replace all "#!/usr/bin/env python" with "#!/usr/bin/env python2" >>>>>> 2. migrate your scripts one by one to either Python 3.x or >>>>>> to Python 2.7 + 3.4+ >>>>>> 3. after migration replace "#!/usr/bin/env python2" with >>>>>> "#!/usr/bin/env python3" or "#!/usr/bin/env python" resp. >>>>>> >>>>>> (*) Some OSes may require to use python2.7, if they don't come >>>>>> with a symlink from python2 -> python2.7. >>>>>> >>>>>> -- >>>>>> Marc-Andre Lemburg >>>>>> eGenix.com >>>>>> >>>>>> Professional Python Services directly from the Source (#1, Mar 11 2015) >>>>>> >>> Python Projects, Coaching and Consulting ... http://www.egenix.com/ >>>>>> >>> mxODBC Plone/Zope Database Adapter ... http://zope.egenix.com/ >>>>>> >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ >>>>>> ________________________________________________________________________ >>>>>> >>>>>> ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: >>>>>> >>>>>> eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 >>>>>> D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg >>>>>> Registered at Amtsgericht Duesseldorf: HRB 46611 >>>>>> http://www.egenix.com/company/contact/ >>>>>> _______________________________________________ >>>>>> Python-ideas mailing list >>>>>> Python-ideas at python.org >>>>>> https://mail.python.org/mailman/listinfo/python-ideas >>>>>> Code of Conduct: http://python.org/psf/codeofconduct/ >>>>> >>>>> _______________________________________________ >>>>> Python-ideas mailing list >>>>> Python-ideas at python.org >>>>> https://mail.python.org/mailman/listinfo/python-ideas >>>>> Code of Conduct: http://python.org/psf/codeofconduct/ >> >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ > > > > -- > > Christopher Barker, Ph.D. > Oceanographer > > Emergency Response Division > NOAA/NOS/OR&R (206) 526-6959 voice > 7600 Sand Point Way NE (206) 526-6329 fax > Seattle, WA 98115 (206) 526-6317 main reception > > Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From skip.montanaro at gmail.com Thu Mar 12 01:58:55 2015 From: skip.montanaro at gmail.com (Skip Montanaro) Date: Wed, 11 Mar 2015 19:58:55 -0500 Subject: [Python-ideas] Migration of /usr/bin/python to python3 In-Reply-To: References: <20150311194325.GA15390@python.ca> Message-ID: On Mar 11, 2015 3:50 PM, "Ryan Gonzalez" wrote: > > No; this won't work: > > #!/usr/bin/env python -x -3 > > As Oleg said, anything in the hashbang can only be passed one argument. It's not the kernel that invokes that; it's the shell. What is to prevent Python itself from going back and reading that line? Will the shell barf completely if there are two args? Or will it exec .../env python? Skip -------------- next part -------------- An HTML attachment was scrubbed... URL: From abarnert at yahoo.com Thu Mar 12 01:59:32 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Wed, 11 Mar 2015 17:59:32 -0700 Subject: [Python-ideas] Migration of /usr/bin/python to python3 In-Reply-To: <5500DB94.3020606@egenix.com> References: <20150311194325.GA15390@python.ca> <5500A7AE.9020400@egenix.com> <3485D6D4-E09B-4617-AC5B-2D07BB79435F@yahoo.com> <5500DB94.3020606@egenix.com> Message-ID: <9D539754-054B-498A-B095-B94744852071@yahoo.com> On Mar 11, 2015, at 5:19 PM, M.-A. Lemburg wrote: > >> On 11.03.2015 23:45, Andrew Barnert wrote: >>> On Mar 11, 2015, at 1:38 PM, M.-A. Lemburg wrote: >>> >>>> On 11.03.2015 21:03, David Mertz wrote: >>>> https://www.python.org/dev/peps/pep-0394/ >>> >>> I think the migration should start with modifying scripts to use >>> >>> #!/usr/bin/env python2 >>> >>> when they are Python 2 only (*) and >>> >>> #!/usr/bin/env python3 >>> >>> when they are Python 3 only and >>> >>> #!/usr/bin/env python >>> >>> only when they support both Python 2 and 3. >> >> Except that you need a step 0: Not every system out there is PEP 394-compliant. Most notably, the built-in Python on OS X has nothing named python2, and python is a wrapper executable that selects between python2.5, python2.6, and python2.7. Some of the popular third-party Python-with-extra-batteries installs still give you only python and maybe python2.7, but not python2. And most linux distros' built-in and standard repo Pythons are PEP 394-compliant, but "most Linux distros" is not the same thing as "all non-OS X Unix-like systems". >> >> So if you want to move this forward, I think someone needs to create a central place with all the info on how far we are from 100% PEP 394 compliance (including but reports to each vendor, etc.) and start pressuring the vendors who don't comply. >> >> In the case of Apple, unless you can get them to add a python2 executable in bug fix releases of 10.9 and 10.10, rather than just in 10.11, it'll be a long time before python2 becomes a portable command. (Even if you could, would dropping OS X 10.6-10.8 be acceptable today for most scripts?) > > I won't comment on the Mac Python installations. There's a reason > why we've been using our own installations since the early days of > Max OS X :-) In the early days of Mac OS X, Apple shipped an incomplete and broken version of a 4-year-old Python, so that made perfect sense. For half a decade, Apple has been shipping a complete version of the latest 2.x, and with things like PyObjC available out of the box. On top of that, when you tell people to use a separate Python 2.7 installation, they invariably run into problems where they install things for Apple's Python and then try to use them in the other Python or vice-versa, which is compounded by the fact that many people want to use /usr/local/bin as their script directory, but that's the same directory Apple uses, which is one of the ways you get to Ian's problem--whichever ipython was installed last wins--but of course here version numbers don't even help because they're both 2.7. There are hundreds of people on StackOverflow with this problem every year, and surely orders of magnitude more who don't go there for help. The root problem is that PEP 394 only solves the problem of having two different versions of Python, not two different Pythons of the same version, but you're encouraging people to do the latter. And if PATH doesn't solve it for them (because they're using /usr/local), what does? Also, if your program only works on Macs with an extra Python installation, then you don't work on the majority of Unix desktops out there, and it's hard to call that "portable". If you're not interested in working out of the box on most *nix systems, just require Python 3 and all these problems go away. :) > In any case, if those binaries don't exist on the target system, > the user will get a nice error message saying e.g.: > > python2: Command not found. > > and even better: modern OSes will then point the user to the > missing packages on their system to make the script run again. So nothing other than some of the linux distros counts as a modern OS? > I think that's a lot more informative than > > SyntaxError: Missing parentheses in call to 'print' > > :-) > > Anyway, the above is just what I recommend. > > I'm not a big fan of doing setuptools like magic all over the > place when there's a clean solution right in front of you (ok, > one may need to take off those cool Python monkey-patching and > meta-class sunglasses occasionally to see it ;-)). > >>> "Explicit is better than implicit" and all that Zen :-) >>> >>> Once that's done, switching the symlink is really a no-brainer. >>> >>> The recipe for this is easy too: >>> >>> 1. replace all "#!/usr/bin/env python" with "#!/usr/bin/env python2" >>> 2. migrate your scripts one by one to either Python 3.x or >>> to Python 2.7 + 3.4+ >>> 3. after migration replace "#!/usr/bin/env python2" with >>> "#!/usr/bin/env python3" or "#!/usr/bin/env python" resp. >>> >>> (*) Some OSes may require to use python2.7, if they don't come >>> with a symlink from python2 -> python2.7. > > -- > Marc-Andre Lemburg > eGenix.com > > Professional Python Services directly from the Source (#1, Mar 12 2015) >>>> Python Projects, Coaching and Consulting ... http://www.egenix.com/ >>>> mxODBC Plone/Zope Database Adapter ... http://zope.egenix.com/ >>>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ > ________________________________________________________________________ > > ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: > > eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 > D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg > Registered at Amtsgericht Duesseldorf: HRB 46611 > http://www.egenix.com/company/contact/ From mal at egenix.com Thu Mar 12 02:30:29 2015 From: mal at egenix.com (M.-A. Lemburg) Date: Thu, 12 Mar 2015 02:30:29 +0100 Subject: [Python-ideas] Migration of /usr/bin/python to python3 In-Reply-To: <9D539754-054B-498A-B095-B94744852071@yahoo.com> References: <20150311194325.GA15390@python.ca> <5500A7AE.9020400@egenix.com> <3485D6D4-E09B-4617-AC5B-2D07BB79435F@yahoo.com> <5500DB94.3020606@egenix.com> <9D539754-054B-498A-B095-B94744852071@yahoo.com> Message-ID: <5500EC35.40105@egenix.com> On 12.03.2015 01:59, Andrew Barnert wrote: > On Mar 11, 2015, at 5:19 PM, M.-A. Lemburg wrote: >> >>> On 11.03.2015 23:45, Andrew Barnert wrote: >>>> On Mar 11, 2015, at 1:38 PM, M.-A. Lemburg wrote: >>>> >>>>> On 11.03.2015 21:03, David Mertz wrote: >>>>> https://www.python.org/dev/peps/pep-0394/ >>>> >>>> I think the migration should start with modifying scripts to use >>>> >>>> #!/usr/bin/env python2 >>>> >>>> when they are Python 2 only (*) and >>>> >>>> #!/usr/bin/env python3 >>>> >>>> when they are Python 3 only and >>>> >>>> #!/usr/bin/env python >>>> >>>> only when they support both Python 2 and 3. >>> >>> Except that you need a step 0: Not every system out there is PEP 394-compliant. Most notably, the built-in Python on OS X has nothing named python2, and python is a wrapper executable that selects between python2.5, python2.6, and python2.7. Some of the popular third-party Python-with-extra-batteries installs still give you only python and maybe python2.7, but not python2. And most linux distros' built-in and standard repo Pythons are PEP 394-compliant, but "most Linux distros" is not the same thing as "all non-OS X Unix-like systems". >>> >>> So if you want to move this forward, I think someone needs to create a central place with all the info on how far we are from 100% PEP 394 compliance (including but reports to each vendor, etc.) and start pressuring the vendors who don't comply. >>> >>> In the case of Apple, unless you can get them to add a python2 executable in bug fix releases of 10.9 and 10.10, rather than just in 10.11, it'll be a long time before python2 becomes a portable command. (Even if you could, would dropping OS X 10.6-10.8 be acceptable today for most scripts?) >> >> I won't comment on the Mac Python installations. There's a reason >> why we've been using our own installations since the early days of >> Max OS X :-) > > In the early days of Mac OS X, Apple shipped an incomplete and broken version of a 4-year-old Python, so that made perfect sense. For half a decade, Apple has been shipping a complete version of the latest 2.x, and with things like PyObjC available out of the box. On top of that, when you tell people to use a separate Python 2.7 installation, they invariably run into problems where they install things for Apple's Python and then try to use them in the other Python or vice-versa, which is compounded by the fact that many people want to use /usr/local/bin as their script directory, but that's the same directory Apple uses, which is one of the ways you get to Ian's problem--whichever ipython was installed last wins--but of course here version numbers don't even help because they're both 2.7. There are hundreds of people on StackOverflow with this problem every year, and surely orders of magnitude more who don't go there for help. As I said, we've been using our own installations. That doesn't mean the resulting packages won't run with the stock Apple versions, it just means that we're not relying on those Apple versions when compiling our software. > The root problem is that PEP 394 only solves the problem of having two different versions of Python, not two different Pythons of the same version, but you're encouraging people to do the latter. And if PATH doesn't solve it for them (because they're using /usr/local), what does? This thread is not trying to solve the problem of having multiple Python installations of the same version. The subject problem is migration to Python 3, not running multiple installations of Python 2.7. Let's not get side tracked (I know it's python-ideas and side tracking is fun, but still :-)). > Also, if your program only works on Macs with an extra Python installation, then you don't work on the majority of Unix desktops out there, and it's hard to call that "portable". If you're not interested in working out of the box on most *nix systems, just require Python 3 and all these problems go away. :) Actually, we don't work much on Macs, but mostly on Linux, FreeBSD and also on Windows, where you have a different set of problems than what's being discussed here, but which can be solved in a similar way by the way of the py launcher: https://www.python.org/dev/peps/pep-0397/ With the right py.ini file, the above approach will work on Windows as well. >> In any case, if those binaries don't exist on the target system, >> the user will get a nice error message saying e.g.: >> >> python2: Command not found. >> >> and even better: modern OSes will then point the user to the >> missing packages on their system to make the script run again. > > So nothing other than some of the linux distros counts as a modern OS? Of course they do. This was just a tangent I couldn't resist, since I think that more modern OSes should provide the same usability features. It's one of the nice Linux distro features I'd love to see in other OSes as well. But this was not the main argument. I think that even the simple python2: Command not found. is better than the standard Python SyntaxError (or other similar Python 3 vs. 2 related error) you get when running a script with the wrong Python version. >> I think that's a lot more informative than >> >> SyntaxError: Missing parentheses in call to 'print' >> >> :-) >> >> Anyway, the above is just what I recommend. >> >> I'm not a big fan of doing setuptools like magic all over the >> place when there's a clean solution right in front of you (ok, >> one may need to take off those cool Python monkey-patching and >> meta-class sunglasses occasionally to see it ;-)). >> >>>> "Explicit is better than implicit" and all that Zen :-) >>>> >>>> Once that's done, switching the symlink is really a no-brainer. >>>> >>>> The recipe for this is easy too: >>>> >>>> 1. replace all "#!/usr/bin/env python" with "#!/usr/bin/env python2" >>>> 2. migrate your scripts one by one to either Python 3.x or >>>> to Python 2.7 + 3.4+ >>>> 3. after migration replace "#!/usr/bin/env python2" with >>>> "#!/usr/bin/env python3" or "#!/usr/bin/env python" resp. >>>> >>>> (*) Some OSes may require to use python2.7, if they don't come >>>> with a symlink from python2 -> python2.7. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Mar 12 2015) >>> Python Projects, Coaching and Consulting ... http://www.egenix.com/ >>> mxODBC Plone/Zope Database Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From abarnert at yahoo.com Thu Mar 12 02:39:11 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Wed, 11 Mar 2015 18:39:11 -0700 Subject: [Python-ideas] Migration of /usr/bin/python to python3 In-Reply-To: <1426117358.3338188.239148661.1F774D54@webmail.messagingengine.com> References: <20150311194325.GA15390@python.ca> <1426117358.3338188.239148661.1F774D54@webmail.messagingengine.com> Message-ID: <4402E693-1F8A-4865-BD6B-F5F3BE60417B@yahoo.com> On Mar 11, 2015, at 4:42 PM, random832 at fastmail.us wrote: > >> On Wed, Mar 11, 2015, at 15:43, Neil Schemenauer wrote: >> Traditionally the OS only passes the first option to the >> interpreter so anything after that could be used to pass 3.x >> specific options. > > I think some OSes pass everything after the interpreter as one long > string rather than ignoring extra arguments. And some pass all the arguments separately. Try "#! /usr/bin/env python -O -i" on different platforms; you'll find all three behaviors. > Is there any standard for > this? Nope. POSIX allows for compliant sh implementations to do shbang processing by saying that 'If the first line of a file of shell commands starts with the characters "#!", the results are unspecified". That means there's actually no way to write portable #! scripts according to POSIX. Given that POSIX doesn't require any particular path to env, that's not too surprising; you're already non-portable using /usr/bin/env At least (as of 2008) POSIX does ban doing shbang processing inside the exec functions instead of in sh. (Although even there, Linux's pluggable executable formats provide a way around that...) Somewhere in the POSIX standard, there's a non-normative recommendation that if you want an application to depend on #!, you should include an installer that uses getconf PATH, searches for a suitable interpreter, and used sed to change the shbang to point at the absolute path to that interpreter. So, forget theory; in practice, what's portable? Using exactly one argument works as expected on any system you're likely to find. Which means if you want to use /usr/bin/env instead of using an installer, you don't get any arguments. From phd at phdru.name Thu Mar 12 09:24:01 2015 From: phd at phdru.name (Oleg Broytman) Date: Thu, 12 Mar 2015 09:24:01 +0100 Subject: [Python-ideas] Migration of /usr/bin/python to python3 In-Reply-To: <1426117358.3338188.239148661.1F774D54@webmail.messagingengine.com> References: <20150311194325.GA15390@python.ca> <1426117358.3338188.239148661.1F774D54@webmail.messagingengine.com> Message-ID: <20150312082401.GA12046@phdru.name> Hi! On Wed, Mar 11, 2015 at 07:42:38PM -0400, random832 at fastmail.us wrote: > On Wed, Mar 11, 2015, at 15:43, Neil Schemenauer wrote: > > Traditionally the OS only passes the first option to the > > interpreter so anything after that could be used to pass 3.x > > specific options. > > I think some OSes pass everything after the interpreter as one long > string rather than ignoring extra arguments. Is there any standard for > this? There *are* standards ;-) See http://www.in-ulm.de/~mascheck/various/shebang/#splitting and the table below. Oleg. -- Oleg Broytman http://phdru.name/ phd at phdru.name Programmers don't die, they just GOSUB without RETURN. From phd at phdru.name Thu Mar 12 09:25:43 2015 From: phd at phdru.name (Oleg Broytman) Date: Thu, 12 Mar 2015 09:25:43 +0100 Subject: [Python-ideas] Migration of /usr/bin/python to python3 In-Reply-To: References: <20150311194325.GA15390@python.ca> Message-ID: <20150312082543.GB12046@phdru.name> Hi! On Wed, Mar 11, 2015 at 07:58:55PM -0500, Skip Montanaro wrote: > On Mar 11, 2015 3:50 PM, "Ryan Gonzalez" wrote: > > > > No; this won't work: > > > > #!/usr/bin/env python -x -3 > > > > As Oleg said, anything in the hashbang can only be passed one argument. > It's not the kernel that invokes that; it's the shell. Why do you think it's the shell? It's exec() system call. > Skip Oleg. -- Oleg Broytman http://phdru.name/ phd at phdru.name Programmers don't die, they just GOSUB without RETURN. From solipsis at pitrou.net Thu Mar 12 10:25:03 2015 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 12 Mar 2015 10:25:03 +0100 Subject: [Python-ideas] Migration of /usr/bin/python to python3 References: <20150311194325.GA15390@python.ca> Message-ID: <20150312102503.02b22f00@fsol> On Wed, 11 Mar 2015 13:43:25 -0600 Neil Schemenauer wrote: > This has been brought up elsewhere, I think this is a better forum > to discuss it. Changing /usr/bin/python directly to python3.x is > bad idea, in my option. It's going to cause users trouble and the > benefit to Python 3.x users is not worth it. Instead, let's try to > find a smooth migration path. As a data point, when you install Python 3 using Conda, the executable is already called "python": $ python Python 3.4.2 |Continuum Analytics, Inc.| (default, Oct 21 2014, 17:16:37) [GCC 4.4.7 20120313 (Red Hat 4.4.7-1)] on linux Type "help", "copyright", "credits" or "license" for more information. >>> Regards Antoine. From abarnert at yahoo.com Thu Mar 12 11:17:47 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Thu, 12 Mar 2015 03:17:47 -0700 Subject: [Python-ideas] Migration of /usr/bin/python to python3 In-Reply-To: <20150312082401.GA12046@phdru.name> References: <20150311194325.GA15390@python.ca> <1426117358.3338188.239148661.1F774D54@webmail.messagingengine.com> <20150312082401.GA12046@phdru.name> Message-ID: <74313125-C895-49EA-ACC0-31AE69522A86@yahoo.com> On Mar 12, 2015, at 1:24 AM, Oleg Broytman wrote: > > Hi! > >> On Wed, Mar 11, 2015 at 07:42:38PM -0400, random832 at fastmail.us wrote: >>> On Wed, Mar 11, 2015, at 15:43, Neil Schemenauer wrote: >>> Traditionally the OS only passes the first option to the >>> interpreter so anything after that could be used to pass 3.x >>> specific options. >> >> I think some OSes pass everything after the interpreter as one long >> string rather than ignoring extra arguments. Is there any standard for >> this? > > There *are* standards ;-) See > http://www.in-ulm.de/~mascheck/various/shebang/#splitting > and the table below. A table showing that a variety of systems do things in all three of the conceivable ways is pretty much the exact opposite of a standard. > > Oleg. > -- > Oleg Broytman http://phdru.name/ phd at phdru.name > Programmers don't die, they just GOSUB without RETURN. > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ From phd at phdru.name Thu Mar 12 11:23:54 2015 From: phd at phdru.name (Oleg Broytman) Date: Thu, 12 Mar 2015 11:23:54 +0100 Subject: [Python-ideas] Migration of /usr/bin/python to python3 In-Reply-To: <74313125-C895-49EA-ACC0-31AE69522A86@yahoo.com> References: <20150311194325.GA15390@python.ca> <1426117358.3338188.239148661.1F774D54@webmail.messagingengine.com> <20150312082401.GA12046@phdru.name> <74313125-C895-49EA-ACC0-31AE69522A86@yahoo.com> Message-ID: <20150312102354.GA17548@phdru.name> On Thu, Mar 12, 2015 at 03:17:47AM -0700, Andrew Barnert wrote: > On Mar 12, 2015, at 1:24 AM, Oleg Broytman wrote: > >> On Wed, Mar 11, 2015 at 07:42:38PM -0400, random832 at fastmail.us wrote: > >>> On Wed, Mar 11, 2015, at 15:43, Neil Schemenauer wrote: > >>> Traditionally the OS only passes the first option to the > >>> interpreter so anything after that could be used to pass 3.x > >>> specific options. > >> > >> I think some OSes pass everything after the interpreter as one long > >> string rather than ignoring extra arguments. Is there any standard for > >> this? > > > > There *are* standards ;-) See > > http://www.in-ulm.de/~mascheck/various/shebang/#splitting > > and the table below. > > A table showing that a variety of systems do things in all three of the conceivable ways is pretty much the exact opposite of a standard. Isn't that how all standards are implemented in real life?! ;-) Oleg. -- Oleg Broytman http://phdru.name/ phd at phdru.name Programmers don't die, they just GOSUB without RETURN. From abarnert at yahoo.com Thu Mar 12 11:29:59 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Thu, 12 Mar 2015 03:29:59 -0700 Subject: [Python-ideas] Migration of /usr/bin/python to python3 In-Reply-To: <20150312082543.GB12046@phdru.name> References: <20150311194325.GA15390@python.ca> <20150312082543.GB12046@phdru.name> Message-ID: <58EFCAFB-A5A8-4FF5-89C8-145D6D01412A@yahoo.com> On Mar 12, 2015, at 1:25 AM, Oleg Broytman wrote: > > Hi! > >> On Wed, Mar 11, 2015 at 07:58:55PM -0500, Skip Montanaro wrote: >>> On Mar 11, 2015 3:50 PM, "Ryan Gonzalez" wrote: >>> >>> No; this won't work: >>> >>> #!/usr/bin/env python -x -3 >>> >>> As Oleg said, anything in the hashbang can only be passed one argument. >> It's not the kernel that invokes that; it's the shell. > > Why do you think it's the shell? It's exec() system call. Maybe he thinks that because he read the POSIX standards, which forbid doing it in exec, or the man pages for his system's execl or sh? The earliest versions of Unix shbang processing were done in exec, but that turned out to lead to a variety of problems, so most stopped, and after a few decades, POSIX finally (in 2008) forbade it: exec must use the standard shell processor for anything that's not a recognized, loadable executable format. The standard shell processor (usually /bin/sh) is then allowed to do anything it wants for files starting with "#!" (the behavior is left undefined by the standard); on pretty much all systems what it does is (one of the three variations of) the usual shbang process. (As I briefly mentioned earlier, there is a way to weasel around the wording, and Linux does give you a way to use it: the linux kernel has pluggable executable formats, and there's nothing stopping you from, say, defining an executable format that's recognized by treating "#!/usr/bin/env python" as a magic number and "loaded" by exec'ing the Python interpreter, which would still be within the letter of the standard.) > >> Skip > > Oleg. > -- > Oleg Broytman http://phdru.name/ phd at phdru.name > Programmers don't die, they just GOSUB without RETURN. > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ From abarnert at yahoo.com Thu Mar 12 11:42:51 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Thu, 12 Mar 2015 03:42:51 -0700 Subject: [Python-ideas] Migration of /usr/bin/python to python3 In-Reply-To: <20150312102354.GA17548@phdru.name> References: <20150311194325.GA15390@python.ca> <1426117358.3338188.239148661.1F774D54@webmail.messagingengine.com> <20150312082401.GA12046@phdru.name> <74313125-C895-49EA-ACC0-31AE69522A86@yahoo.com> <20150312102354.GA17548@phdru.name> Message-ID: On Mar 12, 2015, at 3:23 AM, Oleg Broytman wrote: > >> On Thu, Mar 12, 2015 at 03:17:47AM -0700, Andrew Barnert wrote: >> On Mar 12, 2015, at 1:24 AM, Oleg Broytman wrote: >>>>> On Wed, Mar 11, 2015 at 07:42:38PM -0400, random832 at fastmail.us wrote: >>>>> On Wed, Mar 11, 2015, at 15:43, Neil Schemenauer wrote: >>>>> Traditionally the OS only passes the first option to the >>>>> interpreter so anything after that could be used to pass 3.x >>>>> specific options. >>>> >>>> I think some OSes pass everything after the interpreter as one long >>>> string rather than ignoring extra arguments. Is there any standard for >>>> this? >>> >>> There *are* standards ;-) See >>> http://www.in-ulm.de/~mascheck/various/shebang/#splitting >>> and the table below. >> >> A table showing that a variety of systems do things in all three of the conceivable ways is pretty much the exact opposite of a standard. > > Isn't that how all standards are implemented in real life?! ;-) Sure, there are cases where there is a standard but nobody follows it, but that's not the case here: there is no standard to follow in the first place. The relevant standard document (POSIX 2008 or 2013, for most *nixes) explicitly says the behavior is not defined by the standard. (And practically, that makes even more of a difference than it should, as various C libs have at times _intentionally_ done things differently when the standard leaves it open, for good and bad reasons ranging from "so people don't rely on 'it works on Linux and FreeBSD, so it must be standard and portable, so I won't look it up'" to "screw those BSD people for implying something is portable without checking with us".) > -- > Oleg Broytman http://phdru.name/ phd at phdru.name > Programmers don't die, they just GOSUB without RETURN. > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ From guettliml at thomas-guettler.de Thu Mar 12 12:56:58 2015 From: guettliml at thomas-guettler.de (=?UTF-8?B?VGhvbWFzIEfDvHR0bGVy?=) Date: Thu, 12 Mar 2015 12:56:58 +0100 Subject: [Python-ideas] datetime: Support infinity Message-ID: <55017F0A.4040109@thomas-guettler.de> Hi, at the end of january there was a mail thread about "datetime: Support infinity". Some people liked it :-) What can I do to get this done? Regards, Thomas From skip.montanaro at gmail.com Thu Mar 12 14:21:33 2015 From: skip.montanaro at gmail.com (Skip Montanaro) Date: Thu, 12 Mar 2015 08:21:33 -0500 Subject: [Python-ideas] Migration of /usr/bin/python to python3 In-Reply-To: <20150312082543.GB12046@phdru.name> References: <20150311194325.GA15390@python.ca> <20150312082543.GB12046@phdru.name> Message-ID: I think the quoting got screwed up somewhere. I wrote: skip> If I understood correctly, /usr/bin/python would be started by the kernel, and it would be passed the -x flag. to which Ryan replied: ryan> It's not the kernel that invokes that; it's the shell. then Oleg responded (but it looked like he was quoting me, not Ryan): oleg> Why do you think it's the shell? It's exec() system call. I ignored Ryan's original reply, since it's been so long that I considered things at this level, I thought I might have been mistaken or that the field had changed in the meantime. So, are we all on the same page now? The shell calls exec(...whatever...), and within that call (in the kernel), the #! line is interpreted. There is still some question about whether it consumes the first two "words" after the #! (command and one argument), or in some cases consumes all words, treating them as separate arguments (seems feasible, though problematic from a security POV) or as one big argument (seems unlikely, as that would foster an enormous amount of breakage). Skip From skip.montanaro at gmail.com Thu Mar 12 14:23:11 2015 From: skip.montanaro at gmail.com (Skip Montanaro) Date: Thu, 12 Mar 2015 08:23:11 -0500 Subject: [Python-ideas] Migration of /usr/bin/python to python3 In-Reply-To: References: <20150311194325.GA15390@python.ca> <20150312082543.GB12046@phdru.name> Message-ID: On Thu, Mar 12, 2015 at 8:21 AM, Skip Montanaro wrote: > ... I thought I might have been mistaken > or that the field had changed in the meantime. Which, I see by reading the rest of the thread, it has... S From wes.turner at gmail.com Thu Mar 12 14:55:15 2015 From: wes.turner at gmail.com (Wes Turner) Date: Thu, 12 Mar 2015 08:55:15 -0500 Subject: [Python-ideas] Migration of /usr/bin/python to python3 In-Reply-To: <20150312102503.02b22f00@fsol> References: <20150311194325.GA15390@python.ca> <20150312102503.02b22f00@fsol> Message-ID: On Mar 12, 2015 4:25 AM, "Antoine Pitrou" wrote: > > On Wed, 11 Mar 2015 13:43:25 -0600 > Neil Schemenauer > wrote: > > This has been brought up elsewhere, I think this is a better forum > > to discuss it. Changing /usr/bin/python directly to python3.x is > > bad idea, in my option. It's going to cause users trouble and the > > benefit to Python 3.x users is not worth it. Instead, let's try to > > find a smooth migration path. > > As a data point, when you install Python 3 using Conda, the executable > is already called "python": > > $ python Brew and pyenv also rely upon $PATH to determine which executable named 'python[maj.min]' to run a script marked as executable with. +1 for the /usr/bin/env python[maj[.min]] approach is likely most portable (because it uses $PATH for what it is for) * OSX 10.9 is still on 2.7.5 (i think w/o SSL security updates(!)) * OSX 10.10 has ?.? ([1] lists 2.7.1 as most recent) * I think it would be reasonable to request an update that adds a '/usr/bin/python2' symlink, so that paths can be '/usr/bin/env python[maj[.min]'. * They seem more focused on (!OpenStack) Swift lately. [1] https://www.apple.com/opensource/ Again, +1 for the /usr/bin/env python[maj[.min]] approach (because it uses $PATH for what it is for) -------------- next part -------------- An HTML attachment was scrubbed... URL: From random832 at fastmail.us Thu Mar 12 16:43:36 2015 From: random832 at fastmail.us (random832 at fastmail.us) Date: Thu, 12 Mar 2015 11:43:36 -0400 Subject: [Python-ideas] Migration of /usr/bin/python to python3 In-Reply-To: <20150312082401.GA12046@phdru.name> References: <20150311194325.GA15390@python.ca> <1426117358.3338188.239148661.1F774D54@webmail.messagingengine.com> <20150312082401.GA12046@phdru.name> Message-ID: <1426175016.2903413.239449765.7FD189FA@webmail.messagingengine.com> On Thu, Mar 12, 2015, at 04:24, Oleg Broytman wrote: > > There *are* standards ;-) See > http://www.in-ulm.de/~mascheck/various/shebang/#splitting > and the table below. > > Oleg. One additional data point of purely historical interest is that DEMOS (a soviet unix clone derived from 3BSD but the functionality here is apparently its unique innovation) supports "#!CMD A1 $* A2 A3" which does more or less what it looks like it does. It also supports "/*#!" as the magic prefix. From random832 at fastmail.us Thu Mar 12 16:51:06 2015 From: random832 at fastmail.us (random832 at fastmail.us) Date: Thu, 12 Mar 2015 11:51:06 -0400 Subject: [Python-ideas] Migration of /usr/bin/python to python3 In-Reply-To: <58EFCAFB-A5A8-4FF5-89C8-145D6D01412A@yahoo.com> References: <20150311194325.GA15390@python.ca> <20150312082543.GB12046@phdru.name> <58EFCAFB-A5A8-4FF5-89C8-145D6D01412A@yahoo.com> Message-ID: <1426175466.2906539.239453817.319E6C2C@webmail.messagingengine.com> On Thu, Mar 12, 2015, at 06:29, Andrew Barnert wrote: > The earliest versions of Unix shbang processing were done in exec, but > that turned out to lead to a variety of problems, so most stopped, and > after a few decades, POSIX finally (in 2008) forbade it: exec must use > the standard shell processor for anything that's not a recognized, > loadable executable format. Show me where POSIX defines "recognized, loadable executable format" to exclude #! as a magic number with its usual behavior. What the standard requires is that execvp and execlp treat what would be a failure of the other members of the exec family of functions (i.e. execve, execle, execv, execl), with [ENOEXEC] by falling back to sh. However, shebang does not cause these other functions to fail with [ENOEXEC], and therefore does not fall under this clause. It does not forbid any processing from being done "in exec" (i.e. in execve). From random832 at fastmail.us Thu Mar 12 16:58:22 2015 From: random832 at fastmail.us (random832 at fastmail.us) Date: Thu, 12 Mar 2015 11:58:22 -0400 Subject: [Python-ideas] Migration of /usr/bin/python to python3 In-Reply-To: <1426175466.2906539.239453817.319E6C2C@webmail.messagingengine.com> References: <20150311194325.GA15390@python.ca> <20150312082543.GB12046@phdru.name> <58EFCAFB-A5A8-4FF5-89C8-145D6D01412A@yahoo.com> <1426175466.2906539.239453817.319E6C2C@webmail.messagingengine.com> Message-ID: <1426175902.2908116.239457549.634FE590@webmail.messagingengine.com> On Thu, Mar 12, 2015, at 11:51, random832 at fastmail.us wrote: > It does not forbid any processing from being done "in exec" (i.e. in > execve). Just for completeness, I will note that it _probably_ does forbid exec from returning [ENOENT], [EACCES], etc, on the basis of problems executing the interpreter specified in the shbang line, and implementations should simply return [EINVAL] in that case. However, that's quite different from the processing being forbidden entirely. From rymg19 at gmail.com Thu Mar 12 18:50:53 2015 From: rymg19 at gmail.com (Ryan Gonzalez) Date: Thu, 12 Mar 2015 12:50:53 -0500 Subject: [Python-ideas] Migration of /usr/bin/python to python3 In-Reply-To: <20150312082543.GB12046@phdru.name> References: <20150311194325.GA15390@python.ca> <20150312082543.GB12046@phdru.name> Message-ID: On Thu, Mar 12, 2015 at 3:25 AM, Oleg Broytman wrote: > Hi! > > On Wed, Mar 11, 2015 at 07:58:55PM -0500, Skip Montanaro < > skip.montanaro at gmail.com> wrote: > > On Mar 11, 2015 3:50 PM, "Ryan Gonzalez" wrote: > > > > > > No; this won't work: > > > > > > #!/usr/bin/env python -x -3 > > > > > > As Oleg said, anything in the hashbang can only be passed one argument. > > It's not the kernel that invokes that; it's the shell. > > Why do you think it's the shell? It's exec() system call. > > Right. My bad. I was thinking about the parsing part, not the execution part. > > Skip > > Oleg. > -- > Oleg Broytman http://phdru.name/ phd at phdru.name > Programmers don't die, they just GOSUB without RETURN. > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- Ryan [ERROR]: Your autotools build scripts are 200 lines longer than your program. Something?s wrong. http://kirbyfan64.github.io/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From abarnert at yahoo.com Thu Mar 12 21:38:01 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Thu, 12 Mar 2015 13:38:01 -0700 Subject: [Python-ideas] Migration of /usr/bin/python to python3 In-Reply-To: <1426175466.2906539.239453817.319E6C2C@webmail.messagingengine.com> References: <20150311194325.GA15390@python.ca> <20150312082543.GB12046@phdru.name> <58EFCAFB-A5A8-4FF5-89C8-145D6D01412A@yahoo.com> <1426175466.2906539.239453817.319E6C2C@webmail.messagingengine.com> Message-ID: > On Mar 12, 2015, at 8:51 AM, random832 at fastmail.us wrote: > > Show me where POSIX defines "recognized, loadable executable format" to > exclude #! as a magic number with its usual behavior. I already quoted the standard earlier in the thread, and I already explained how something like linux's pluggable binfmt provides a way around it that's still at least according to the letter of the standard. I don't see much point in repeating what I already wrote. But notice that in the RATIONALE section of the 2001, 2004, 2008, and 2013 versions of the standard (e.g., http://pubs.opengroup.org/onlinepubs/009695399/functions/exec.html) effectively describes that as legacy behavior in contrast to the behavior that's "now required by IEEE Std 1003.1-2001": > Another way that some historical implementations handle shell scripts is by recognizing the first two bytes of the file as the character string "#!" and using the remainder of the first line of the file as the name of the command interpreter to execute. At any rate, this is getting way off track. The point here is that if you're looking for a standard on what #! does everywhere, you're not going to find one. The working group on executable scripts for POSIX-2001 eventually decided to add nothing more than the statement that scripts that start with #! have behavior undefined by the standard, and the recommendation apps that want to use #! should include an installer that sed's the script based on what's on the PATH at install time. Which makes the standard useless if you want to write portable shbang scripts. But practicality beats purity. If all of the *nix platforms that Python currently supports do shbang processing, all have env in /usr/bin (also specifically not required by POSIX, which explicitly says that path locations are not determined by the standard), and none of them swallow all args (as early BSD did), but some drop all args after the first, some pass all, and some pass a single arg with spaces, then that's what you have to work with. You can use "#!/usr/bin/env python", but you can't use "#!/usr/bin/env python -m ipython". Of course if you're building a Mac installer or packages for a particular linux distro or the like, you don't have to worry about portability at all, and can do what makes sense for your platform. (This also means not having to worry about other distros that provide a Python 3.x as /usr/bin/python, etc.) From random832 at fastmail.us Thu Mar 12 21:49:36 2015 From: random832 at fastmail.us (random832 at fastmail.us) Date: Thu, 12 Mar 2015 16:49:36 -0400 Subject: [Python-ideas] Migration of /usr/bin/python to python3 In-Reply-To: References: <20150311194325.GA15390@python.ca> <20150312082543.GB12046@phdru.name> <58EFCAFB-A5A8-4FF5-89C8-145D6D01412A@yahoo.com> <1426175466.2906539.239453817.319E6C2C@webmail.messagingengine.com> Message-ID: <1426193376.2984303.239645173.35C9AF3F@webmail.messagingengine.com> On Thu, Mar 12, 2015, at 16:38, Andrew Barnert wrote: > effectively describes that as legacy behavior in contrast to the behavior > that's "now required by IEEE Std 1003.1-2001": The legacy behavior being discussed is that of _requiring_ it and _failing_ to execute the shell for files that didn't have it. There's nothing forbidding an implementation from having a magic number of "#!" as a standard non-"pluggable" executable format. > > Another way that some historical implementations handle shell scripts is by recognizing the first two bytes of the file as the character string "#!" and using the remainder of the first line of the file as the name of the command interpreter to execute. > > At any rate, this is getting way off track. You have a point - sorry about that. Though, speaking of the posix shell script exec issue, and to wrench things back on topic... what about allowing python scripts to start with the line "exec python [options...] $0", a line which will be ignored by the python interpreter? From abarnert at yahoo.com Thu Mar 12 22:05:16 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Thu, 12 Mar 2015 14:05:16 -0700 Subject: [Python-ideas] Migration of /usr/bin/python to python3 In-Reply-To: <1426193376.2984303.239645173.35C9AF3F@webmail.messagingengine.com> References: <20150311194325.GA15390@python.ca> <20150312082543.GB12046@phdru.name> <58EFCAFB-A5A8-4FF5-89C8-145D6D01412A@yahoo.com> <1426175466.2906539.239453817.319E6C2C@webmail.messagingengine.com> <1426193376.2984303.239645173.35C9AF3F@webmail.messagingengine.com> Message-ID: <3C2D8DFB-3F09-483D-B8E2-5F1D1291215D@yahoo.com> On Mar 12, 2015, at 1:49 PM, random832 at fastmail.us wrote: > Though, speaking of the posix shell script exec issue, and to wrench > things back on topic... what about allowing python scripts to start with > the line "exec python [options...] $0", a line which will be ignored by > the python interpreter? I vaguely remember either perl or Tcl considering something equivalent long ago as a way of dealing with BSD4.x versions that didn't pass any arguments from the shbang line (before that became irrelevant but env became a de facto practical standard and ate up the first arg, putting us right back in the equivalent place), so it might be worth searching their archives to see what came of that. (Although given the languages we're talking about, I wouldn't be surprised if the answer was "you can already do it in one line in these six different hacky ways, so we don't need to add a special way to do it", which won't apply to Python...) However, I'm not sure this solves the problem the thread is trying to solve. Yes, it gives you a portable way to let 3.5+ scripts pass extra arguments to the interpreter, which might be useful for other reasons (and if so, I think a new thread laying out the rationale would be handy). But the problem people were trying to solve here is that `#!/usr/bin/env python2 -m spam` isn't portable (and neither is `#!/usr/local/bin/spam` where spam is a Python script), and you can't retroactively change 2.x. From p.f.moore at gmail.com Thu Mar 12 22:33:35 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 12 Mar 2015 21:33:35 +0000 Subject: [Python-ideas] Migration of /usr/bin/python to python3 In-Reply-To: <1426193376.2984303.239645173.35C9AF3F@webmail.messagingengine.com> References: <20150311194325.GA15390@python.ca> <20150312082543.GB12046@phdru.name> <58EFCAFB-A5A8-4FF5-89C8-145D6D01412A@yahoo.com> <1426175466.2906539.239453817.319E6C2C@webmail.messagingengine.com> <1426193376.2984303.239645173.35C9AF3F@webmail.messagingengine.com> Message-ID: On 12 March 2015 at 20:49, wrote: > Though, speaking of the posix shell script exec issue, and to wrench > things back on topic... what about allowing python scripts to start with > the line "exec python [options...] $0", a line which will be ignored by > the python interpreter? Can't you use the -x option (skip first line of source) for that? Paul From abarnert at yahoo.com Thu Mar 12 22:34:49 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Thu, 12 Mar 2015 14:34:49 -0700 Subject: [Python-ideas] Migration of /usr/bin/python to python3 In-Reply-To: References: <20150311194325.GA15390@python.ca> <20150312102503.02b22f00@fsol> Message-ID: <2D3F4998-7AF3-4FAF-96CF-9A6E733C7067@yahoo.com> On Mar 12, 2015, at 6:55 AM, Wes Turner wrote: > > > On Mar 12, 2015 4:25 AM, "Antoine Pitrou" wrote: > > > > On Wed, 11 Mar 2015 13:43:25 -0600 > > Neil Schemenauer > > wrote: > > > This has been brought up elsewhere, I think this is a better forum > > > to discuss it. Changing /usr/bin/python directly to python3.x is > > > bad idea, in my option. It's going to cause users trouble and the > > > benefit to Python 3.x users is not worth it. Instead, let's try to > > > find a smooth migration path. > > > > As a data point, when you install Python 3 using Conda, the executable > > is already called "python": > > > > $ python > > Brew and pyenv also rely upon $PATH to determine which executable named 'python[maj.min]' to run a script marked as executable with. > > +1 for the /usr/bin/env python[maj[.min]] approach is likely most portable (because it uses $PATH for what it is for) > > * OSX 10.9 is still on 2.7.5 (i think w/o SSL security updates(!)) > * OSX 10.10 has ?.? ([1] lists 2.7.1 as most recent) > 2.7.6 on both my base 10.10 box and my 10.10.3 prerelease. (I think it's no longer violating NDA to answer questions about prereleases now that they're public betas rather than DRs... If I'm wrong, pretend I didn't say anything and just consider that they've only changed the Python version in a dot release once in the history of OS X, and that was before they got semi-serious about Python...) > * I think it would be reasonable to request an update that adds a '/usr/bin/python2' symlink, so that paths can be '/usr/bin/env python[maj[.min]'. > Is there someone who can make that request who Apple will listen to more carefully than just a random user's radar? Or should we all just go file radars and link them on OpenRadar for tracking? Also, is there any change of getting it added to 10.9 and 10.10 bug fix releases? Anyway, this would get us a step closer, but as long as there are millions more OS X 10.8, RHEL5, etc. boxes out there (that have python but not python2) than Arch boxes (where python means 3.x); 2.x-only app's still can't portably follow what PEP 394 recommends. > * They seem more focused on (!OpenStack) Swift lately. > It's not as if Python was ever their top priority in the first place. But they've clearly got at least someone working on Python integration given that they included a newer Python 2.7 (and things like newer PyObjC, newer and better-optimized NumPy, PyOpenSSL linked to the same dylib as Python itself, etc.) in each new major OS release, so it's not like there's nobody who could add the symlink... -------------- next part -------------- An HTML attachment was scrubbed... URL: From random832 at fastmail.us Thu Mar 12 22:49:42 2015 From: random832 at fastmail.us (random832 at fastmail.us) Date: Thu, 12 Mar 2015 17:49:42 -0400 Subject: [Python-ideas] Migration of /usr/bin/python to python3 In-Reply-To: References: <20150311194325.GA15390@python.ca> <20150312082543.GB12046@phdru.name> <58EFCAFB-A5A8-4FF5-89C8-145D6D01412A@yahoo.com> <1426175466.2906539.239453817.319E6C2C@webmail.messagingengine.com> <1426193376.2984303.239645173.35C9AF3F@webmail.messagingengine.com> Message-ID: <1426196982.2999114.239678065.6219273A@webmail.messagingengine.com> On Thu, Mar 12, 2015, at 17:33, Paul Moore wrote: > On 12 March 2015 at 20:49, wrote: > > Though, speaking of the posix shell script exec issue, and to wrench > > things back on topic... what about allowing python scripts to start with > > the line "exec python [options...] $0", a line which will be ignored by > > the python interpreter? > > Can't you use the -x option (skip first line of source) for that? > Paul I'd guess the "DOS specific hack" it is intended for is, in fact, precisely "run the python interpreter using the native scripting language in the first line". So... I guess you could. Might be worth cleaning up the line number issue and replacing the env hack with it going forward, given, you know, POSIX compliance. From barry at python.org Thu Mar 12 22:52:20 2015 From: barry at python.org (Barry Warsaw) Date: Thu, 12 Mar 2015 17:52:20 -0400 Subject: [Python-ideas] Migration of /usr/bin/python to python3 References: <20150311194325.GA15390@python.ca> <5500A7AE.9020400@egenix.com> Message-ID: <20150312175220.6f125d03@anarchist.wooz.org> On Mar 11, 2015, at 09:38 PM, M.-A. Lemburg wrote: >I think the migration should start with modifying scripts to use > >#!/usr/bin/env python2 > >when they are Python 2 only (*) and > >#!/usr/bin/env python3 > >when they are Python 3 only and > >#!/usr/bin/env python > >only when they support both Python 2 and 3. Mostly, I agree, although I would not recommend /usr/bin/env for system installed scripts. Using /usr/bin/env is fine for development trees, but a really bad idea for anything installed in system locations. Modify your suggestion to /usr/bin/python2 and /usr/bin/python3 and I'm on board. :) I would very much like to see explicit use of /usr/bin/python2 replace /usr/bin/python for scripts. Then I think the problem is, what do users expect when they type $ python foo.py at their shell prompt? Cheers, -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 819 bytes Desc: OpenPGP digital signature URL: From abarnert at yahoo.com Thu Mar 12 23:00:47 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Thu, 12 Mar 2015 15:00:47 -0700 Subject: [Python-ideas] Migration of /usr/bin/python to python3 In-Reply-To: References: <20150311194325.GA15390@python.ca> <20150312102503.02b22f00@fsol> Message-ID: <1E007D06-742A-4933-BDF9-E2910ED3DF62@yahoo.com> On Mar 12, 2015, at 6:55 AM, Wes Turner wrote: > On Mar 12, 2015 4:25 AM, "Antoine Pitrou" wrote: > > > > On Wed, 11 Mar 2015 13:43:25 -0600 > > Neil Schemenauer > > wrote: > > > This has been brought up elsewhere, I think this is a better forum > > > to discuss it. Changing /usr/bin/python directly to python3.x is > > > bad idea, in my option. It's going to cause users trouble and the > > > benefit to Python 3.x users is not worth it. Instead, let's try to > > > find a smooth migration path. > > > > As a data point, when you install Python 3 using Conda, the executable > > is already called "python": > > > > $ python > > Brew and pyenv also rely upon $PATH to determine which executable named 'python[maj.min]' to run a script marked as executable with. > > +1 for the /usr/bin/env python[maj[.min]] approach is likely most portable (because it uses $PATH for what it is for) > > * OSX 10.9 is still on 2.7.5 (i think w/o SSL security updates(!)) > * OSX 10.10 has ?.? ([1] lists 2.7.1 as most recent) > * I think it would be reasonable to request an update that adds a '/usr/bin/python2' symlink, so that paths can be '/usr/bin/env python[maj[.min]'. > http://openradar.appspot.com/radar?id=5833516721897472 If you want to file your own bug report with Apple, you can mention that it's a dup of radar 20143311. > * They seem more focused on (!OpenStack) Swift lately. > > [1] https://www.apple.com/opensource/ > > Again, +1 for the /usr/bin/env python[maj[.min]] approach (because it uses $PATH for what it is for) > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From wes.turner at gmail.com Thu Mar 12 23:01:13 2015 From: wes.turner at gmail.com (Wes Turner) Date: Thu, 12 Mar 2015 17:01:13 -0500 Subject: [Python-ideas] Migration of /usr/bin/python to python3 In-Reply-To: <20150312175220.6f125d03@anarchist.wooz.org> References: <20150311194325.GA15390@python.ca> <5500A7AE.9020400@egenix.com> <20150312175220.6f125d03@anarchist.wooz.org> Message-ID: On Mar 12, 2015 4:53 PM, "Barry Warsaw" wrote: > > On Mar 11, 2015, at 09:38 PM, M.-A. Lemburg wrote: > > >I think the migration should start with modifying scripts to use > > > >#!/usr/bin/env python2 > > > >when they are Python 2 only (*) and > > > >#!/usr/bin/env python3 > > > >when they are Python 3 only and > > > >#!/usr/bin/env python > > > >only when they support both Python 2 and 3. > > Mostly, I agree, although I would not recommend /usr/bin/env for system > installed scripts. Using /usr/bin/env is fine for development trees, but a > really bad idea for anything installed in system locations. Modify your > suggestion to /usr/bin/python2 and /usr/bin/python3 and I'm on board. :) For (re-)packaged scripts, it seems reasonable to expect package maintainers to use explicit platform-specific paths. Otherwise, for checksums and diff noise, does /usr/bin/env python2 make sense? > > I would very much like to see explicit use of /usr/bin/python2 replace > /usr/bin/python for scripts. +1. This will require documented guidelines and many emails to package maintainers. Where should a note about this go in /docs? > > Then I think the problem is, what do users expect when they type > > $ python foo.py > > at their shell prompt? For the first python binary on $PATH to supersede whatever might be defined in the shebang line of foo.py (thus skipping any shebang path and args). -------------- next part -------------- An HTML attachment was scrubbed... URL: From wes.turner at gmail.com Thu Mar 12 23:04:30 2015 From: wes.turner at gmail.com (Wes Turner) Date: Thu, 12 Mar 2015 17:04:30 -0500 Subject: [Python-ideas] Migration of /usr/bin/python to python3 In-Reply-To: <1E007D06-742A-4933-BDF9-E2910ED3DF62@yahoo.com> References: <20150311194325.GA15390@python.ca> <20150312102503.02b22f00@fsol> <1E007D06-742A-4933-BDF9-E2910ED3DF62@yahoo.com> Message-ID: On Mar 12, 2015 5:00 PM, "Andrew Barnert" wrote: > > On Mar 12, 2015, at 6:55 AM, Wes Turner wrote: >> >> On Mar 12, 2015 4:25 AM, "Antoine Pitrou" wrote: >> > >> > On Wed, 11 Mar 2015 13:43:25 -0600 >> > Neil Schemenauer >> > wrote: >> > > This has been brought up elsewhere, I think this is a better forum >> > > to discuss it. Changing /usr/bin/python directly to python3.x is >> > > bad idea, in my option. It's going to cause users trouble and the >> > > benefit to Python 3.x users is not worth it. Instead, let's try to >> > > find a smooth migration path. >> > >> > As a data point, when you install Python 3 using Conda, the executable >> > is already called "python": >> > >> > $ python >> >> Brew and pyenv also rely upon $PATH to determine which executable named 'python[maj.min]' to run a script marked as executable with. >> >> +1 for the /usr/bin/env python[maj[.min]] approach is likely most portable (because it uses $PATH for what it is for) >> >> * OSX 10.9 is still on 2.7.5 (i think w/o SSL security updates(!)) >> * OSX 10.10 has ?.? ([1] lists 2.7.1 as most recent) >> * I think it would be reasonable to request an update that adds a '/usr/bin/python2' symlink, so that paths can be '/usr/bin/env python[maj[.min]'. > > http://openradar.appspot.com/radar?id=5833516721897472 > > If you want to file your own bug report with Apple, you can mention that it's a dup of radar 20143311. I sent an email to product-security AT apple.com ( https://www.apple.com/support/security/) with a link to this thread; because TLS. -------------- next part -------------- An HTML attachment was scrubbed... URL: From barry at python.org Fri Mar 13 02:40:42 2015 From: barry at python.org (Barry Warsaw) Date: Thu, 12 Mar 2015 21:40:42 -0400 Subject: [Python-ideas] Migration of /usr/bin/python to python3 References: <20150311194325.GA15390@python.ca> <5500A7AE.9020400@egenix.com> Message-ID: <20150312214042.31055748@anarchist.wooz.org> On Mar 11, 2015, at 06:02 PM, Ryan Gonzalez wrote: >Bad idea? What about when you're running simple scripts you've downloaded >and Python 3 is on an external hard drive, not /usr/bin? I don't get it. Regardless of what a script's shebang says, you can always be explicit about which version you want to use to run the script. You don't *have* to use the shebang's interpreter. Okay, so it's a little more typing - is that really a problem for this use case? Cheers, -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 819 bytes Desc: OpenPGP digital signature URL: From barry at python.org Fri Mar 13 02:45:36 2015 From: barry at python.org (Barry Warsaw) Date: Thu, 12 Mar 2015 21:45:36 -0400 Subject: [Python-ideas] Migration of /usr/bin/python to python3 References: <20150311194325.GA15390@python.ca> <5500A7AE.9020400@egenix.com> <20150312175220.6f125d03@anarchist.wooz.org> Message-ID: <20150312214536.65583928@anarchist.wooz.org> On Mar 12, 2015, at 05:01 PM, Wes Turner wrote: >For (re-)packaged scripts, it seems reasonable to expect package >maintainers to use explicit platform-specific paths. > >Otherwise, for checksums and diff noise, does /usr/bin/env python2 make >sense? Sorry, I don't understand the question. ;) Cheers, -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 819 bytes Desc: OpenPGP digital signature URL: From bkabrda at redhat.com Fri Mar 13 10:00:14 2015 From: bkabrda at redhat.com (Bohuslav Kabrda) Date: Fri, 13 Mar 2015 05:00:14 -0400 (EDT) Subject: [Python-ideas] Migration of /usr/bin/python to python3 In-Reply-To: <20150312175220.6f125d03@anarchist.wooz.org> References: <20150311194325.GA15390@python.ca> <5500A7AE.9020400@egenix.com> <20150312175220.6f125d03@anarchist.wooz.org> Message-ID: <1848995041.4098752.1426237214459.JavaMail.zimbra@redhat.com> ----- Original Message ----- > On Mar 11, 2015, at 09:38 PM, M.-A. Lemburg wrote: > > >I think the migration should start with modifying scripts to use > > > >#!/usr/bin/env python2 > > > >when they are Python 2 only (*) and > > > >#!/usr/bin/env python3 > > > >when they are Python 3 only and > > > >#!/usr/bin/env python > > > >only when they support both Python 2 and 3. > > Mostly, I agree, although I would not recommend /usr/bin/env for system > installed scripts. Using /usr/bin/env is fine for development trees, but a > really bad idea for anything installed in system locations. Modify your > suggestion to /usr/bin/python2 and /usr/bin/python3 and I'm on board. :) +1, the system scripts should use system Python explicitly to keep working when user's environment changes. I'm not sure about Debian/Ubuntu, but we do this in Fedora as much as we can. And while I'm primarily a packager, I also use /usr/bin/env python[2|3] while coding, because it's just more handy during development. > I would very much like to see explicit use of /usr/bin/python2 replace > /usr/bin/python for scripts. +1 (or, better yet, /usr/bin/python3 ;)) > Then I think the problem is, what do users expect when they type > > $ python foo.py > > at their shell prompt? For more than a year now, I've been trying to push python 3 as default in Fedora. "Default" means that it should be the only Python on live media/cloud images/... and various system applications and system scripts will run on it. I want Fedora to follow PEP 394, so this doesn't mean switching /usr/bin/python to Python 3, but that's one of the biggest sources of confusion. Most users expect /usr/bin/python to point to "the" Python. We're not switching for switching /usr/bin/python (for now) for several reasons: 1) PEP 394 2) System packages still using /usr/bin/python 3) Custom user/sysadmin scripts pointing to /usr/bin/python Point 1 is a recommendation that can change in time, since it's in hands of Python upstream (but then again, it's mostly there because of 3, I think); we'll make point 2 go away in time, because that's something that we can change. Point 3 is the real issue, because (from my experience) there are tons of users/sysadmins that use Python as a shell with nice features and don't really care about what's going on in Python upstream. These scripts are outside of Python upstream and distro control and we can't do anything about them. My thinking is that these won't change - not now, not in two years, not in ten years. The only way to change them is to break them - and then the real question is: do we want to break them or do we want to keep them working forever? (well, at least until we retire Python 2, which would make /usr/bin/python disappear completely) So what can be done: a) PEP 394 changes in few years to recommend pointing /usr/bin/python to Python 3, thus breaking (3). Hopefully this would happen at point where most distros are ready (and willing) to do the change. b) PEP 394 never changes while Python 2 is supported, suggesting switching /usr/bin/python after Python 2 EOL (2020?). c) PEP 394 never changes, not even after Python 2 EOL. d) We remove /usr/bin/python. I've already heard several people suggesting this, but I'm not sure what that should mean for us on distro level. For example, should the system-packaged pip have /usr/bin/pip then? Or should that be removed too? Either way, this would nicely solve (3), since people would just get "python not found" and would have to investigate and learn what's going on, doing an explicit decision about which Python they want to use. On the other hand, I can pretty much guarantee that lots of people would hate this, since they just want to type "python" :) e) Something else? I have to say I prefer a), since I think that we should just move forward at one point and make Python 3 "the Python". My second preference would be d), since that at least doesn't make it look that Python 2 is still "the Python". > Cheers, > -Barry -- Regards, Slavek Kabrda From wes.turner at gmail.com Fri Mar 13 11:38:26 2015 From: wes.turner at gmail.com (Wes Turner) Date: Fri, 13 Mar 2015 05:38:26 -0500 Subject: [Python-ideas] Migration of /usr/bin/python to python3 In-Reply-To: <1848995041.4098752.1426237214459.JavaMail.zimbra@redhat.com> References: <20150311194325.GA15390@python.ca> <5500A7AE.9020400@egenix.com> <20150312175220.6f125d03@anarchist.wooz.org> <1848995041.4098752.1426237214459.JavaMail.zimbra@redhat.com> Message-ID: On Mar 13, 2015 4:02 AM, "Bohuslav Kabrda" wrote: > > ----- Original Message ----- > > On Mar 11, 2015, at 09:38 PM, M.-A. Lemburg wrote: > > > > >I think the migration should start with modifying scripts to use > > > > > >#!/usr/bin/env python2 > > > > > >when they are Python 2 only (*) and > > > > > >#!/usr/bin/env python3 > > > > > >when they are Python 3 only and > > > > > >#!/usr/bin/env python > > > > > >only when they support both Python 2 and 3. > > > > Mostly, I agree, although I would not recommend /usr/bin/env for system > > installed scripts. Using /usr/bin/env is fine for development trees, but a > > really bad idea for anything installed in system locations. Modify your > > suggestion to /usr/bin/python2 and /usr/bin/python3 and I'm on board. :) > > +1, the system scripts should use system Python explicitly to keep working when user's environment changes. I'm not sure about Debian/Ubuntu, but we do this in Fedora as much as we can. > And while I'm primarily a packager, I also use /usr/bin/env python[2|3] while coding, because it's just more handy during development. > > > I would very much like to see explicit use of /usr/bin/python2 replace > > /usr/bin/python for scripts. > > +1 (or, better yet, /usr/bin/python3 ;)) > > > Then I think the problem is, what do users expect when they type > > > > $ python foo.py > > > > at their shell prompt? > > For more than a year now, I've been trying to push python 3 as default in Fedora. "Default" means that it should be the only Python on live media/cloud images/... and various system applications and system scripts will run on it. I want Fedora to follow PEP 394, so this doesn't mean switching /usr/bin/python to Python 3, but that's one of the biggest sources of confusion. Most users expect /usr/bin/python to point to "the" Python. > > We're not switching for switching /usr/bin/python (for now) for several reasons: > 1) PEP 394 > 2) System packages still using /usr/bin/python > 3) Custom user/sysadmin scripts pointing to /usr/bin/python > > Point 1 is a recommendation that can change in time, since it's in hands of Python upstream (but then again, it's mostly there because of 3, I think); we'll make point 2 go away in time, because that's something that we can change. Point 3 is the real issue, because (from my experience) there are tons of users/sysadmins that use Python as a shell with nice features and don't really care about what's going on in Python upstream. These scripts are outside of Python upstream and distro control and we can't do anything about them. My thinking is that these won't change - not now, not in two years, not in ten years. The only way to change them is to break them - and then the real question is: do we want to break them or do we want to keep them working forever? (well, at least until we retire Python 2, which would make /usr/bin/python disappear completely) > > So what can be done: > a) PEP 394 changes in few years to recommend pointing /usr/bin/python to Python 3, thus breaking (3). Hopefully this would happen at point where most distros are ready (and willing) to do the change. > b) PEP 394 never changes while Python 2 is supported, suggesting switching /usr/bin/python after Python 2 EOL (2020?). > c) PEP 394 never changes, not even after Python 2 EOL. > d) We remove /usr/bin/python. I've already heard several people suggesting this, but I'm not sure what that should mean for us on distro level. For example, should the system-packaged pip have /usr/bin/pip then? Or should that be removed too? Either way, this would nicely solve (3), since people would just get "python not found" and would have to investigate and learn what's going on, doing an explicit decision about which Python they want to use. On the other hand, I can pretty much guarantee that lots of people would hate this, since they just want to type "python" :) > e) Something else? > > I have to say I prefer a), since I think that we should just move forward at one point and make Python 3 "the Python". My second preference would be d), since that at least doesn't make it look that Python 2 is still "the Python". Values / Criteria (for dealing with change): * backward compatibility (non-breaking) * cross-platform support * consistent file checksums * interpreter arguments * PYTHONHASHSEED * -O2 * simplest solution * convenience Things that require/generate shebang lines: * OS Packages in ./bin (can be expected to sed /usr/bin/env \(python.*\)) * ./scripts (should not need to be sed-ed to work w/ the current virtualenv (?)) * console_scripts in ./bin (stubs generated with an explicit path; that can later be 'made relocatable') PEP 394: The "python" Command on Unix-Like Systems * https://www.python.org/dev/peps/pep-0394/#recommendation * http://legacy.python.org/dev/peps/pep-0394/#recommendation -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Fri Mar 13 12:57:36 2015 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 13 Mar 2015 21:57:36 +1000 Subject: [Python-ideas] Migration of /usr/bin/python to python3 In-Reply-To: <041B5A0E-4C77-4683-9B0A-98432C139F48@stufft.io> References: <20150311194325.GA15390@python.ca> <041B5A0E-4C77-4683-9B0A-98432C139F48@stufft.io> Message-ID: On 12 March 2015 at 06:14, Donald Stufft wrote: > What's worse. Breaking everything that uses an unversioned shebang because /usr/bin/python doesn't exist or breaking some things that doesn't work with Python 3 because /usr/bin/python is now 3? > > Generally discussions along this line talk about Python 2 not being installed by default at all (whereas Python 3 would be). So really it's two different broken by default. Right, this most recently came up in the context of Fedora's plans to transition to only having Python 3 in the base install (https://fedoraproject.org/wiki/Changes/Python_3_as_Default). At the moment the expected consequence is that there will come a Fedora version (tentatively Fedora 23) where, out of the box, "/usr/bin/python" and "/usr/bin/env python" will just stop working. Packages that are part of the distro aren't a problem - those are all being dealt with as the transition progresses, and any which specifically need Python 2 will depend on it accordingly. The problem arises with custom user packages and scripts, and third party packages and scripts. For those, we either just let them break, or we automatically try running them in Python 3 by change the package that provides the "/usr/bin/python" symlink. This is actually going to happen well in advance of Python 3.6, so my original suggestion of deferring a decision until then (the other discussion Neal was referring to when starting this thread) won't actually help (as Fedora 23 is due out later this year, while even a slip to Fedora 24 would still put this change in early 2016). In favour of moving the symlink to the Python 3 package when Python 3 is the only version a distro installs by default: * Python 2/3 compatible scripts will "just work" * Python 3 incompatible scripts will hopefully get a useful error that points them in the right direction (such as https://stackoverflow.com/questions/25445439/what-does-syntaxerror-missing-parentheses-in-call-to-print-mean-in-python) * if the script author is only worried about PEP 394 compliant environments, they can update their shebang line to "python2", or else they can set it to "python2.7" if they aren't worried about older Linux distros that only provide Python 2.6 or earlier. If we went down this path, Fedora would likely consider enabling deprecation warnings and Py3k warnings in Rawhide and the Alphas of the previous release (where Python 2 was still the default) to help gauge the likely impact. The case against moving the symlink: * while it's convenient, running arbitrary user code in the system Python is also going to be a compatibility/maintainability issue for both distro developers and our users * for the Fedora/CentOS/RHEL ecosystem, Software Collections use environment modules to switch the active Python: https://www.softwarecollections.org/en/ * if the python symlink is taken away entirely, then it's an opportunity to give users the choice of either installing the "system Python 2 stack" (which will be effectively unchanged, just not installed by default) *or* of installing the Python 2.7 Software Collection, and getting used to the idea of leaving the system Python alone * the advantage of Software Collections is that they support parallel installation by design, so user's can upgrade their distro without upgrading the collections they're using, and vice-versa * other utilities like pyenv and conda similarly support parallel installs by design Going down this path would basically mean saying that "python" at the system level would no longer be defined at all once Python 2 goes EOL in 2020. A "default" Python would only exist inside an activated environment module, whether defined by scl, pyenv, conda, or another tool. Adopting the latter approach would mean saying "No change to PEP 394" upstream, and thus passing the situation back to Fedora to decide how to handle. Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Fri Mar 13 13:15:22 2015 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 13 Mar 2015 22:15:22 +1000 Subject: [Python-ideas] CPython needs to revamp/replace it's build system && Cross-compiling Python sucks In-Reply-To: References: <49c23aa9-51cf-47b7-843b-8c4b58d01cbd@email.android.com> Message-ID: On 12 March 2015 at 04:30, Ryan Gonzalez wrote: > On Wed, Mar 11, 2015 at 10:07 AM, Brett Cannon wrote: >> >> >> >> On Tue, Mar 10, 2015 at 11:40 AM Ryan Gonzalez wrote: >>> >>> On Tue, Mar 10, 2015 at 8:43 AM, Nick Coghlan wrote: >>>> >>>> On 10 March 2015 at 21:20, Russell Keith-Magee >>>> wrote: >>>> > >>>> > On Tue, Mar 10, 2015 at 10:43 AM, Ryan wrote: >>>> >> >>>> >> Russell Keith-Magee wrote: >>>> >> >Hi Ryan, >>>> >> > >>>> >> >First off I'm no fan of autoconf, but... >>>> >> > >>>> >> >>>> >> Who is other than the one who designed it? :) >>>> > >>>> > Let me be clear - there's a difference between "not a fan" and >>>> > "calling for >>>> > it's head". Autoconf is crufty, annoying, and sometimes aggravating - >>>> > but it >>>> > *works*, and has had some pretty seriously battle testing - both in >>>> > the >>>> > abstract "autoconf as a product", and in the specific "autoconf as >>>> > used in >>>> > Python". This isn't something that can be said of most of the >>>> > alternatives >>>> > in the abstract case, or *any* of the alternatives in the Python case. >>>> >>>> Building architecture dependent software is a genuinely hard problem, >>>> which is why it's full of "80%" solutions like autotools and the >>>> setuptools/distutils pairing. Yes, they're quirky and annoying, but >>>> they also basically do their job, so folks tend to develop a high >>>> tolerance level for their quirks. >>>> >>>> In our particular case, we also have the fun problem of "How do you >>>> build CPython locally when you don't have a working Python interpreter >>>> yet?". A number of the more interesting alternative C/C++ build >>>> systems (like Meson, Scons, Waf and the Google "gyp" project that >>>> powers the node-gyp native extension build system for Node.js) require >>>> an already working Python interpreter to get things started (when >>>> starting from a completely blank slate, even our own build sequence >>>> has to build a deliberately broken partial interpreter just to freeze >>>> importlib._bootstrap before going on to build a properly configured >>>> one that can actually import modules). >>> >>> >>> What about something like Google's new GN? It's designed to replace Gyp >>> in building Chromium and is written in C++. >>> >>> I'm pretty sure using a build system written in Lua to build Python is >>> kind of weird. >> >> >>> >>> >>> With all that in mind...what exactly would be requirements that Python >>> would need in an autotools replacement? >> >> >> Can't require anything more than an ANSI C compiler basically. People run >> Python in places where they don't have much more than gcc. > > > Not even C99? Darn it... C99 would probably be OK these days. If there's a good enough cross-compiling story, I'm also not sure we should entirely rule out C++, as it's potentially OK if *developing* CPython requires a mainstream platform with a C++ compiler. But yeah, you can see why folks look at this problem and end up thinking "well, you know, maybe autotools isn't *that* bad..." Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From bcannon at gmail.com Fri Mar 13 13:27:15 2015 From: bcannon at gmail.com (Brett Cannon) Date: Fri, 13 Mar 2015 12:27:15 +0000 Subject: [Python-ideas] About people's accounts being deactivated due to bounces Message-ID: We're trying to figure out what's going on. Just re-enable your account if you got a notice that it was temporarily disabled. -------------- next part -------------- An HTML attachment was scrubbed... URL: From barry at python.org Fri Mar 13 18:32:04 2015 From: barry at python.org (Barry Warsaw) Date: Fri, 13 Mar 2015 13:32:04 -0400 Subject: [Python-ideas] Migration of /usr/bin/python to python3 In-Reply-To: <1848995041.4098752.1426237214459.JavaMail.zimbra@redhat.com> References: <20150311194325.GA15390@python.ca> <5500A7AE.9020400@egenix.com> <20150312175220.6f125d03@anarchist.wooz.org> <1848995041.4098752.1426237214459.JavaMail.zimbra@redhat.com> Message-ID: <20150313133204.6753253b@marathon> On Mar 13, 2015, at 05:00 AM, Bohuslav Kabrda wrote: >+1, the system scripts should use system Python explicitly to keep working >when user's environment changes. I'm not sure about Debian/Ubuntu, but we do >this in Fedora as much as we can. We do this in Debian/Ubuntu too, although it looks like there are one or two outliers so they probably aren't built with the common Python build tools. >And while I'm primarily a packager, I also use /usr/bin/env python[2|3] while >coding, because it's just more handy during development. Oh, definitely. >For more than a year now, I've been trying to push python 3 as default in >Fedora. "Default" means that it should be the only Python on live media/cloud >images/... and various system applications and system scripts will run on >it. This is a great goal, and something that we've been working toward in Debian/Ubuntu also. I hope all the Linux distros can do the same thing. >I want Fedora to follow PEP 394, so this doesn't mean switching >/usr/bin/python to Python 3, but that's one of the biggest sources of >confusion. Most users expect /usr/bin/python to point to "the" Python. This is what we need to tease out I think. For the most part, users don't care what the shebang line of their scripts is, or even if they have one. They *do* care about what interpreter fires up when they type `python` or any of the variants at their shell, as well as... >3) Custom user/sysadmin scripts pointing to /usr/bin/python [...] >Point 3 is the real issue, because (from my experience) there are tons of >users/sysadmins that use Python as a shell with nice features and don't >really care about what's going on in Python upstream. These scripts are >outside of Python upstream and distro control and we can't do anything about >them. My thinking is that these won't change - not now, not in two years, not >in ten years. The only way to change them is to break them - and then the >real question is: do we want to break them or do we want to keep them working >forever? (well, at least until we retire Python 2, which would make >/usr/bin/python disappear completely) That is indeed the question. I think if we break them, we're just pissing people off for a not compelling enough reason (notice I didn't say "good" reason :). It think the first step is to strongly encourage people to start using /usr/bin/python2 for their Python 2 scripts, and to strongly encourage all distributions of Python to be including a 'python2' symlink. Of course, for anyone with even a rudimentary understanding of *nix, it's easy enough to create that symlink if it's missing, but we might even go a step further and include a 'fix_py2_usr_bin' script that people can run which wouldn't do anything more than ensure /usr/bin/python2 exists. We've committed to bug fix releases for 2.7 until 2020. To me, that's the time frame for any change to /usr/bin/python, whether that's complete removal or repointing the symlink. By then, we'll have 5 more years of educating people to be using 'python2' and 'python3' explicitly so the impact of resolving the ambiguity around 'python' should be lessened. Cheers, -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 819 bytes Desc: OpenPGP digital signature URL: From rymg19 at gmail.com Fri Mar 13 19:05:21 2015 From: rymg19 at gmail.com (Ryan Gonzalez) Date: Fri, 13 Mar 2015 13:05:21 -0500 Subject: [Python-ideas] CPython needs to revamp/replace it's build system && Cross-compiling Python sucks In-Reply-To: References: <49c23aa9-51cf-47b7-843b-8c4b58d01cbd@email.android.com> Message-ID: If C99's ok, there's hope. :) C++'s main issue is that there are platforms that support C99 but have out-of-date or non-complaint C++ compilers. I love the language (yes, I'm weird), but it's tricky to have truly portable C++ programs because if all the holes in various compilers. Just look at all the #ifdef's in Boost. On Fri, Mar 13, 2015 at 7:15 AM, Nick Coghlan wrote: > On 12 March 2015 at 04:30, Ryan Gonzalez wrote: > > On Wed, Mar 11, 2015 at 10:07 AM, Brett Cannon wrote: > >> > >> > >> > >> On Tue, Mar 10, 2015 at 11:40 AM Ryan Gonzalez > wrote: > >>> > >>> On Tue, Mar 10, 2015 at 8:43 AM, Nick Coghlan > wrote: > >>>> > >>>> On 10 March 2015 at 21:20, Russell Keith-Magee < > russell at keith-magee.com> > >>>> wrote: > >>>> > > >>>> > On Tue, Mar 10, 2015 at 10:43 AM, Ryan wrote: > >>>> >> > >>>> >> Russell Keith-Magee wrote: > >>>> >> >Hi Ryan, > >>>> >> > > >>>> >> >First off I'm no fan of autoconf, but... > >>>> >> > > >>>> >> > >>>> >> Who is other than the one who designed it? :) > >>>> > > >>>> > Let me be clear - there's a difference between "not a fan" and > >>>> > "calling for > >>>> > it's head". Autoconf is crufty, annoying, and sometimes aggravating > - > >>>> > but it > >>>> > *works*, and has had some pretty seriously battle testing - both in > >>>> > the > >>>> > abstract "autoconf as a product", and in the specific "autoconf as > >>>> > used in > >>>> > Python". This isn't something that can be said of most of the > >>>> > alternatives > >>>> > in the abstract case, or *any* of the alternatives in the Python > case. > >>>> > >>>> Building architecture dependent software is a genuinely hard problem, > >>>> which is why it's full of "80%" solutions like autotools and the > >>>> setuptools/distutils pairing. Yes, they're quirky and annoying, but > >>>> they also basically do their job, so folks tend to develop a high > >>>> tolerance level for their quirks. > >>>> > >>>> In our particular case, we also have the fun problem of "How do you > >>>> build CPython locally when you don't have a working Python interpreter > >>>> yet?". A number of the more interesting alternative C/C++ build > >>>> systems (like Meson, Scons, Waf and the Google "gyp" project that > >>>> powers the node-gyp native extension build system for Node.js) require > >>>> an already working Python interpreter to get things started (when > >>>> starting from a completely blank slate, even our own build sequence > >>>> has to build a deliberately broken partial interpreter just to freeze > >>>> importlib._bootstrap before going on to build a properly configured > >>>> one that can actually import modules). > >>> > >>> > >>> What about something like Google's new GN? It's designed to replace Gyp > >>> in building Chromium and is written in C++. > >>> > >>> I'm pretty sure using a build system written in Lua to build Python is > >>> kind of weird. > >> > >> > >>> > >>> > >>> With all that in mind...what exactly would be requirements that Python > >>> would need in an autotools replacement? > >> > >> > >> Can't require anything more than an ANSI C compiler basically. People > run > >> Python in places where they don't have much more than gcc. > > > > > > Not even C99? Darn it... > > C99 would probably be OK these days. If there's a good enough > cross-compiling story, I'm also not sure we should entirely rule out > C++, as it's potentially OK if *developing* CPython requires a > mainstream platform with a C++ compiler. > > But yeah, you can see why folks look at this problem and end up > thinking "well, you know, maybe autotools isn't *that* bad..." > > Cheers, > Nick. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > -- Ryan [ERROR]: Your autotools build scripts are 200 lines longer than your program. Something?s wrong. http://kirbyfan64.github.io/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From rymg19 at gmail.com Fri Mar 13 19:43:46 2015 From: rymg19 at gmail.com (Ryan Gonzalez) Date: Fri, 13 Mar 2015 13:43:46 -0500 Subject: [Python-ideas] Rewriting the build system (part 2) Message-ID: So...I've been looking stuff up. Don't have too much free time right now, but it was mostly Googleing. I have deemed three ideas to be best for Python: - Write a custom makefile generator. I have some ideas, but they likely won't come to light any time soon. - Use Boost.Build. It seems ugly and complicated, but it actually has a full configuring system that can detect libraries and try to compile source files. It's written in C89, not C++, and its dependency is the Boehm GC, which probably supports everywhere that Python runs on. Only issue? It's undergoing a rewrite in Python. However, at the rate it's going, Python will likely be at version 7 by the time they finish. - Stick with autotools. Trying to avoid this. Thoughts? Again? -- Ryan [ERROR]: Your autotools build scripts are 200 lines longer than your program. Something?s wrong. http://kirbyfan64.github.io/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From wes.turner at gmail.com Fri Mar 13 20:27:29 2015 From: wes.turner at gmail.com (Wes Turner) Date: Fri, 13 Mar 2015 14:27:29 -0500 Subject: [Python-ideas] Rewriting the build system (part 2) In-Reply-To: References: Message-ID: Scons? http://www.scons.org/doc/production/HTML/scons-user.html On Mar 13, 2015 1:44 PM, "Ryan Gonzalez" wrote: > So...I've been looking stuff up. Don't have too much free time right now, > but it was mostly Googleing. > > I have deemed three ideas to be best for Python: > > - Write a custom makefile generator. I have some ideas, but they likely > won't come to light any time soon. > > - Use Boost.Build. It seems ugly and complicated, but it actually has a > full configuring system that can detect libraries and try to compile source > files. It's written in C89, not C++, and its dependency is the Boehm GC, > which probably supports everywhere that Python runs on. > > Only issue? It's undergoing a rewrite in Python. However, at the rate it's > going, Python will likely be at version 7 by the time they finish. > > - Stick with autotools. Trying to avoid this. > > Thoughts? Again? > > -- > Ryan > [ERROR]: Your autotools build scripts are 200 lines longer than your > program. Something?s wrong. > http://kirbyfan64.github.io/ > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rymg19 at gmail.com Fri Mar 13 20:40:50 2015 From: rymg19 at gmail.com (Ryan Gonzalez) Date: Fri, 13 Mar 2015 14:40:50 -0500 Subject: [Python-ideas] Rewriting the build system (part 2) In-Reply-To: References: Message-ID: That written in Python, so Python would require Python to build which would require Python to build which would require... Besides, SCons sucks. If I were using a build system written in Python, I'd use fbuild or Waf. On Fri, Mar 13, 2015 at 2:27 PM, Wes Turner wrote: > Scons? http://www.scons.org/doc/production/HTML/scons-user.html > On Mar 13, 2015 1:44 PM, "Ryan Gonzalez" wrote: > >> So...I've been looking stuff up. Don't have too much free time right now, >> but it was mostly Googleing. >> >> I have deemed three ideas to be best for Python: >> >> - Write a custom makefile generator. I have some ideas, but they likely >> won't come to light any time soon. >> >> - Use Boost.Build. It seems ugly and complicated, but it actually has a >> full configuring system that can detect libraries and try to compile source >> files. It's written in C89, not C++, and its dependency is the Boehm GC, >> which probably supports everywhere that Python runs on. >> >> Only issue? It's undergoing a rewrite in Python. However, at the rate >> it's going, Python will likely be at version 7 by the time they finish. >> >> - Stick with autotools. Trying to avoid this. >> >> Thoughts? Again? >> >> -- >> Ryan >> [ERROR]: Your autotools build scripts are 200 lines longer than your >> program. Something?s wrong. >> http://kirbyfan64.github.io/ >> >> >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ >> > -- Ryan [ERROR]: Your autotools build scripts are 200 lines longer than your program. Something?s wrong. http://kirbyfan64.github.io/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From wes.turner at gmail.com Fri Mar 13 20:46:17 2015 From: wes.turner at gmail.com (Wes Turner) Date: Fri, 13 Mar 2015 14:46:17 -0500 Subject: [Python-ideas] Rewriting the build system (part 2) In-Reply-To: References: Message-ID: The subject line mentions "Part 2". Is there a link to "Part 1"? What is the rationale for changing? Make, autoconf, and automake are already widely ported/portable. What are the criteria for evaluating alternatives? On Mar 13, 2015 2:41 PM, "Ryan Gonzalez" wrote: > That written in Python, so Python would require Python to build which > would require Python to build which would require... > > Besides, SCons sucks. If I were using a build system written in Python, > I'd use fbuild or Waf. > > > On Fri, Mar 13, 2015 at 2:27 PM, Wes Turner wrote: > >> Scons? http://www.scons.org/doc/production/HTML/scons-user.html >> On Mar 13, 2015 1:44 PM, "Ryan Gonzalez" wrote: >> >>> So...I've been looking stuff up. Don't have too much free time right >>> now, but it was mostly Googleing. >>> >>> I have deemed three ideas to be best for Python: >>> >>> - Write a custom makefile generator. I have some ideas, but they likely >>> won't come to light any time soon. >>> >>> - Use Boost.Build. It seems ugly and complicated, but it actually has a >>> full configuring system that can detect libraries and try to compile source >>> files. It's written in C89, not C++, and its dependency is the Boehm GC, >>> which probably supports everywhere that Python runs on. >>> >>> Only issue? It's undergoing a rewrite in Python. However, at the rate >>> it's going, Python will likely be at version 7 by the time they finish. >>> >>> - Stick with autotools. Trying to avoid this. >>> >>> Thoughts? Again? >>> >>> -- >>> Ryan >>> [ERROR]: Your autotools build scripts are 200 lines longer than your >>> program. Something?s wrong. >>> http://kirbyfan64.github.io/ >>> >>> >>> _______________________________________________ >>> Python-ideas mailing list >>> Python-ideas at python.org >>> https://mail.python.org/mailman/listinfo/python-ideas >>> Code of Conduct: http://python.org/psf/codeofconduct/ >>> >> > > > -- > Ryan > [ERROR]: Your autotools build scripts are 200 lines longer than your > program. Something?s wrong. > http://kirbyfan64.github.io/ > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jeanpierreda at gmail.com Fri Mar 13 20:47:45 2015 From: jeanpierreda at gmail.com (Devin Jeanpierre) Date: Fri, 13 Mar 2015 12:47:45 -0700 Subject: [Python-ideas] CPython needs to revamp/replace it's build system && Cross-compiling Python sucks In-Reply-To: References: Message-ID: On Mon, Mar 9, 2015 at 10:40 AM, Ryan Gonzalez wrote: > I've been trying to work on Issue 23496. And want to tear my hair out. I've > noticed a few problems with Python's current build system: > > - There is no separation of the host C compiler and the target C compiler. > When you give configure a C compiler for cross-compiling, make will happily > try to build Parser/pgen with the cross-compiler. And try to run the > generated executable. And fail. I know you can use `make touch` to get > around this, but it still sucks. I just want to echo this. I had really horrible problems trying to get Python working with emscripten a year ago, because of this and other problems. (Gave up.) It turned out emscripten was a bigger cause for complaint, but, still. -- Devin > - For some reason, autoconf decided to re-configure everything about 2 > minutes into make. But it didn't pass any of the flags I needed, resulting > in everything failing. > > - I do not like having to use this for configuring: > > PYTHON_FOR_BUILD=`realpath ../cpython/python` > CC=arm-linux-androideabi-clang CXX=arm-linux-androideabi-clang++ ./configure > --host=arm-linux-androideabi --build=x86_64 --disable-ipv6 > ac_cv_file__dev_ptmx=no ac_cv_file__dev_ptc=no > ac_cv_have_long_long_format=yes > > At the same time, I'm not quite sure how many people would want to sift > through the 5000 lines in configure.ac or the 1650 lines in Makefile.pre.in. > > I just get the feeling that, whenever I build Python, there are too many > holes/hacks to get around make/autoconf things. > > -- > Ryan > If I were in a 10-story building glass-sided building and forced to write > either Go or autconf scripts, I?d jump out a window. > http://kirbyfan64.github.io/ > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ From rymg19 at gmail.com Fri Mar 13 20:58:06 2015 From: rymg19 at gmail.com (Ryan Gonzalez) Date: Fri, 13 Mar 2015 14:58:06 -0500 Subject: [Python-ideas] Rewriting the build system (part 2) In-Reply-To: References: Message-ID: https://mail.python.org/pipermail/python-ideas/2015-March/032381.html On Fri, Mar 13, 2015 at 2:46 PM, Wes Turner wrote: > The subject line mentions "Part 2". Is there a link to "Part 1"? What is > the rationale for changing? > > Make, autoconf, and automake are already widely ported/portable. > > What are the criteria for evaluating alternatives? > On Mar 13, 2015 2:41 PM, "Ryan Gonzalez" wrote: > >> That written in Python, so Python would require Python to build which >> would require Python to build which would require... >> >> Besides, SCons sucks. If I were using a build system written in Python, >> I'd use fbuild or Waf. >> >> >> On Fri, Mar 13, 2015 at 2:27 PM, Wes Turner wrote: >> >>> Scons? http://www.scons.org/doc/production/HTML/scons-user.html >>> On Mar 13, 2015 1:44 PM, "Ryan Gonzalez" wrote: >>> >>>> So...I've been looking stuff up. Don't have too much free time right >>>> now, but it was mostly Googleing. >>>> >>>> I have deemed three ideas to be best for Python: >>>> >>>> - Write a custom makefile generator. I have some ideas, but they likely >>>> won't come to light any time soon. >>>> >>>> - Use Boost.Build. It seems ugly and complicated, but it actually has a >>>> full configuring system that can detect libraries and try to compile source >>>> files. It's written in C89, not C++, and its dependency is the Boehm GC, >>>> which probably supports everywhere that Python runs on. >>>> >>>> Only issue? It's undergoing a rewrite in Python. However, at the rate >>>> it's going, Python will likely be at version 7 by the time they finish. >>>> >>>> - Stick with autotools. Trying to avoid this. >>>> >>>> Thoughts? Again? >>>> >>>> -- >>>> Ryan >>>> [ERROR]: Your autotools build scripts are 200 lines longer than your >>>> program. Something?s wrong. >>>> http://kirbyfan64.github.io/ >>>> >>>> >>>> _______________________________________________ >>>> Python-ideas mailing list >>>> Python-ideas at python.org >>>> https://mail.python.org/mailman/listinfo/python-ideas >>>> Code of Conduct: http://python.org/psf/codeofconduct/ >>>> >>> >> >> >> -- >> Ryan >> [ERROR]: Your autotools build scripts are 200 lines longer than your >> program. Something?s wrong. >> http://kirbyfan64.github.io/ >> >> > -- Ryan [ERROR]: Your autotools build scripts are 200 lines longer than your program. Something?s wrong. http://kirbyfan64.github.io/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From rymg19 at gmail.com Fri Mar 13 21:00:03 2015 From: rymg19 at gmail.com (Ryan Gonzalez) Date: Fri, 13 Mar 2015 15:00:03 -0500 Subject: [Python-ideas] Rewriting the build system (part 2) In-Reply-To: References: Message-ID: On Fri, Mar 13, 2015 at 2:47 PM, Chris Barker wrote: > On Fri, Mar 13, 2015 at 12:40 PM, Ryan Gonzalez wrote: > >> That written in Python, so Python would require Python to build which >> would require Python to build which would require... >> > > Well, you can't buid a C compiler without a C compiler either (or SOME > compiler). And given that you should be able to do cross-compilation for > platforms that don't currently have a python build, I'm not sure that's a > killer. > > It it so wrong that python eats its own dog food in this manner? > > It's not as much "Python eat its own dog food" as "cross-compiling natively would be even harder than it is right now." C compilers are available on virtually every UNIX that's ever been created since the 70's, including AIX, Plan 9, Solaris, etc. Python? Not as much. > -CHB > >> >> -- > > Christopher Barker, Ph.D. > Oceanographer > > Emergency Response Division > NOAA/NOS/OR&R (206) 526-6959 voice > 7600 Sand Point Way NE (206) 526-6329 fax > Seattle, WA 98115 (206) 526-6317 main reception > > Chris.Barker at noaa.gov > -- Ryan [ERROR]: Your autotools build scripts are 200 lines longer than your program. Something?s wrong. http://kirbyfan64.github.io/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From wes.turner at gmail.com Fri Mar 13 21:01:30 2015 From: wes.turner at gmail.com (Wes Turner) Date: Fri, 13 Mar 2015 15:01:30 -0500 Subject: [Python-ideas] Rewriting the build system (part 2) In-Reply-To: References: Message-ID: On Mar 13, 2015 2:41 PM, "Ryan Gonzalez" wrote: > > That written in Python, so Python would require Python to build which would require Python to build which would require... > > Besides, SCons sucks. If I were using a build system written in Python, I'd use fbuild or Waf. Apparently, Waf started as a fork of SCons. [1][2] [1] http://stackoverflow.com/questions/12569463/choosing-between-scons-and-waf-in-large-projects [2] http://www.scons.org/wiki/SconsVsOtherBuildTools#Waf I don't know anything about fbuild; others might. A feature/alternative matrix could be helpful (in addition to a preliminary implementation) for making a cogent argument (in PEP form [3][4]). This is possible in RST with table or list-table syntax. [3] https://www.python.org/dev/peps/pep-0001/#pep-formats-and-templates [4] https://www.python.org/dev/peps/pep-0009/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From rymg19 at gmail.com Fri Mar 13 21:11:41 2015 From: rymg19 at gmail.com (Ryan Gonzalez) Date: Fri, 13 Mar 2015 15:11:41 -0500 Subject: [Python-ideas] Rewriting the build system (part 2) In-Reply-To: References: Message-ID: On Fri, Mar 13, 2015 at 3:01 PM, Wes Turner wrote: > > On Mar 13, 2015 2:41 PM, "Ryan Gonzalez" wrote: > > > > That written in Python, so Python would require Python to build which > would require Python to build which would require... > > > > Besides, SCons sucks. If I were using a build system written in Python, > I'd use fbuild or Waf. > > Apparently, Waf started as a fork of SCons. [1][2] > > [1] > http://stackoverflow.com/questions/12569463/choosing-between-scons-and-waf-in-large-projects > [2] http://www.scons.org/wiki/SconsVsOtherBuildTools#Waf > But it's now better. > I don't know anything about fbuild; others might. > https://github.com/felix-lang/fbuild > A feature/alternative matrix could be helpful (in addition to a > preliminary implementation) for making a cogent argument (in PEP form > [3][4]). This is possible in RST with table or list-table syntax. > > [3] https://www.python.org/dev/peps/pep-0001/#pep-formats-and-templates > [4] https://www.python.org/dev/peps/pep-0009/ > -- Ryan [ERROR]: Your autotools build scripts are 200 lines longer than your program. Something?s wrong. http://kirbyfan64.github.io/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.barker at noaa.gov Fri Mar 13 20:47:05 2015 From: chris.barker at noaa.gov (Chris Barker) Date: Fri, 13 Mar 2015 12:47:05 -0700 Subject: [Python-ideas] Rewriting the build system (part 2) In-Reply-To: References: Message-ID: On Fri, Mar 13, 2015 at 12:40 PM, Ryan Gonzalez wrote: > That written in Python, so Python would require Python to build which > would require Python to build which would require... > Well, you can't buid a C compiler without a C compiler either (or SOME compiler). And given that you should be able to do cross-compilation for platforms that don't currently have a python build, I'm not sure that's a killer. It it so wrong that python eats its own dog food in this manner? -CHB > > -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From abarnert at yahoo.com Fri Mar 13 22:46:20 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Fri, 13 Mar 2015 14:46:20 -0700 Subject: [Python-ideas] About people's accounts being deactivated due to bounces In-Reply-To: References: Message-ID: <5260F722-8D2C-420D-BCBD-6014579E993F@yahoo.com> I think this may be related to a problem with changes in DKIM policy at Yahoo, or an expiration of DKIM certs at Python.org, or something else that may have changed somewhere in that area. I've noticed that since around 15 January, mailing lists (at least those that munge headers and preserve the original sender's From, as many do) have had two problems: (1) messages from Yahoo users (or other users whose provider has a DKIM policy but isn't gmail) through the list to gmail get marked as spam, and (2) a flood of bounces. I don't know why this didn't just affect every user on every mailing list all at once instead of randomly hitting some but not all users at some random time in the last two months. Sent from my iPhone > On Mar 13, 2015, at 5:27 AM, Brett Cannon wrote: > > We're trying to figure out what's going on. Just re-enable your account if you got a notice that it was temporarily disabled. > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ From robertc at robertcollins.net Sat Mar 14 01:07:57 2015 From: robertc at robertcollins.net (Robert Collins) Date: Sat, 14 Mar 2015 13:07:57 +1300 Subject: [Python-ideas] Rewriting the build system (part 2) In-Reply-To: References: Message-ID: On 14 Mar 2015 09:44, "Chris Barker" wrote: > > On Fri, Mar 13, 2015 at 12:40 PM, Ryan Gonzalez wrote: >> >> That written in Python, so Python would require Python to build which would require Python to build which would require... > > > Well, you can't buid a C compiler without a C compiler either (or SOME compiler). And given that you should be able to do cross-compilation for platforms that don't currently have a python build, I'm not sure that's a killer. > > It it so wrong that python eats its own dog food in this manner? Wrong, no. harder yes. C compilers have mature cross compiler build chains including stage aware test suites and the like. When a tool is it's own build dependency bootstrapping it on a new platform gains a hard requirement on cross compiling rather than it being a thing for special cases like platforms that cannot compile the tool. This makes the life of distribution maintainers harder too, since cross arch platform bootstrapping becomes even more manual. Rob -------------- next part -------------- An HTML attachment was scrubbed... URL: From random832 at fastmail.us Sat Mar 14 06:10:43 2015 From: random832 at fastmail.us (random832 at fastmail.us) Date: Sat, 14 Mar 2015 01:10:43 -0400 Subject: [Python-ideas] Rewriting the build system (part 2) In-Reply-To: References: Message-ID: <1426309843.2639153.240240573.03996CBC@webmail.messagingengine.com> On Fri, Mar 13, 2015, at 20:07, Robert Collins wrote: > Wrong, no. harder yes. C compilers have mature cross compiler build > chains > including stage aware test suites and the like. When a tool is it's own > build dependency bootstrapping it on a new platform gains a hard > requirement on cross compiling rather than it being a thing for special > cases like platforms that cannot compile the tool. That's what bootstrapping is. If you're not doing this, you're not bootstrapping. From stefan at bytereef.org Sat Mar 14 12:54:52 2015 From: stefan at bytereef.org (Stefan Krah) Date: Sat, 14 Mar 2015 11:54:52 +0000 (UTC) Subject: [Python-ideas] CPython needs to revamp/replace it's build system && Cross-compiling Python sucks References: Message-ID: Devin Jeanpierre writes: > On Mon, Mar 9, 2015 at 10:40 AM, Ryan Gonzalez gmail.com> wrote: > > I've been trying to work on Issue 23496. And want to tear my hair out. I've > > noticed a few problems with Python's current build system: > > > > - There is no separation of the host C compiler and the target C compiler. > > When you give configure a C compiler for cross-compiling, make will happily > > try to build Parser/pgen with the cross-compiler. And try to run the > > generated executable. And fail. I know you can use `make touch` to get > > around this, but it still sucks. > > I just want to echo this. I had really horrible problems trying to get > Python working with emscripten a year ago, because of this and other > problems. (Gave up.) It turned out emscripten was a bigger cause for > complaint, but, still. Someone just needs to open an issue that we need 'make bootstrap' or something similar. Replacing autotools does not seem very realistic or desirable. A couple of years ago I did LinuxFromScratch and autotools was *by far* the most reliable and problem free build system. I can't say the same about Cmake. Stefan Krah From solipsis at pitrou.net Sat Mar 14 14:03:23 2015 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sat, 14 Mar 2015 14:03:23 +0100 Subject: [Python-ideas] Rewriting the build system (part 2) References: Message-ID: <20150314140323.1964d347@fsol> On Sat, 14 Mar 2015 13:07:57 +1300 Robert Collins wrote: > On 14 Mar 2015 09:44, "Chris Barker" wrote: > > > > On Fri, Mar 13, 2015 at 12:40 PM, Ryan Gonzalez wrote: > >> > >> That written in Python, so Python would require Python to build which > would require Python to build which would require... > > > > > > Well, you can't buid a C compiler without a C compiler either (or SOME > compiler). And given that you should be able to do cross-compilation for > platforms that don't currently have a python build, I'm not sure that's a > killer. > > > > It it so wrong that python eats its own dog food in this manner? > > Wrong, no. harder yes. C compilers have mature cross compiler build chains > including stage aware test suites and the like. Not to mention that any decent platform manufacturer or system vendor will do the job of writing or porting a C compiler for the platform. Few of them will port Python for us, though. Regards Antoine. From flying-sheep at web.de Sat Mar 14 14:32:42 2015 From: flying-sheep at web.de (Philipp A.) Date: Sat, 14 Mar 2015 13:32:42 +0000 Subject: [Python-ideas] Migration of /usr/bin/python to python3 In-Reply-To: References: <20150311194325.GA15390@python.ca> <5500A7AE.9020400@egenix.com> Message-ID: the PEP is pretty perfect. #!/usr/bin/env python_binary_this_script_runs_with if it runs with python 2: #!/usr/bin/env python2 if it runs with python 3: #!/usr/bin/env python3 if it runs with both: #!/usr/bin/env python the only thing worth discussing IMHO is if python 3 should start shipping with *all* symlinks from past 3.x minor versions to the newest minor version. then we could switch to #!/usr/bin/env python3.4 if it uses e.g. pathlib and would be sure that the script will work with future versions. but then again, users can install pathlib for 3.3? ? Ryan Gonzalez schrieb am Do., 12. M?rz 2015 um 00:13 Uhr: > On Wed, Mar 11, 2015 at 5:54 PM, Chris Barker > wrote: > >> On Wed, Mar 11, 2015 at 3:41 PM, Gregory P. Smith >> wrote: >> >>> I think /usr/bin/env is a bad idea to promote. It changes from an >>> explicit designation of what you need in order to execute to something >>> being derived from the users $PATH. It is *less specific* than >>> /usr/bin/python. For some people that is the goal, but I discourage it. >>> >> >> I don't think the goal is specificity, or lack there of, but a shift in >> control: >> >> Who controls where python is installed? >> >> If you put an explicit full path on the #! line, then the script author >> controls which python is used, AND where it must be installed. >> >> If you put /usr/bin env on the #! line, then the script author is saying >> 'use python", and the script user can have python installed anywhere they >> want, and can use PATH to tell everything what the default it. >> >> I think that _should_ be encouraged -- but ideally with some versioning: >> >> #!/usr/bin/env python2 >> >> or even: >> >> #!/usr/bin/env python3.4 >> >> > Please, just explicit major versions, not minor ones. python3, not > python3.4. Just earlier this year I had to fix about 20 scripts that ran > perfectly on 3.4 but whose hashbang explicitly used 3.1. It sucks. > > >> If any of you remember what a pain it was when RedHat installed system >> tools with (I think): >> >> !#/usr/bin/env python >> >> You'll know what a nightmare that was -- you could not upgrade the >> 'default" python without breaking stuff. >> >> So system tools should probably use the full specific path -- the script >> is tied closely to the environment. But if they had at least put a version >> in there, it would have worked fine: >> >> #!/usr/bin/env python1.5 >> >> These days, I try to use distutils or setuptools to install scripts, and >> they can do the right thing to use the correct python to start up anyway... >> >> -CHB >> >> >> >> -- >> >> Christopher Barker, Ph.D. >> Oceanographer >> >> Emergency Response Division >> NOAA/NOS/OR&R (206) 526-6959 voice >> 7600 Sand Point Way NE (206) 526-6329 fax >> Seattle, WA 98115 (206) 526-6317 main reception >> >> Chris.Barker at noaa.gov >> >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ >> > > -- > Ryan > [ERROR]: Your autotools build scripts are 200 lines longer than your > program. Something?s wrong. > http://kirbyfan64.github.io/ > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From abarnert at yahoo.com Sat Mar 14 16:03:41 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Sat, 14 Mar 2015 08:03:41 -0700 Subject: [Python-ideas] Migration of /usr/bin/python to python3 In-Reply-To: References: <20150311194325.GA15390@python.ca> <5500A7AE.9020400@egenix.com> Message-ID: On Mar 14, 2015, at 6:32 AM, Philipp A. wrote: > > the PEP is pretty perfect. > > #!/usr/bin/env python_binary_this_script_runs_with > if it runs with python 2: #!/usr/bin/env python2 > if it runs with python 3: #!/usr/bin/env python3 > if it runs with both: #!/usr/bin/env python > I don't see how this is perfect in the real world, or even a good recommendation. Continuing to use python rather than python2 will work out of the box on nearly every *nix system today, with no exceptions larger than Arch Linux. Switching to python2 means you no longer work on a number of widely-deployed and still-supported versions (including even the latest betas of Mac OS X). What script developer in his right mind would take that tradeoff? (Especially what developer who isn't willing to move to 3.x, or to distribute or deploy their scripts in a more reasonable way, either of which eliminates the problem...) And the only way that's likely to change over the next decade is that some platforms just won't include Python 2.7 out of the box at all--in which case python2 won't work any better than plain python. > the only thing worth discussing IMHO is if python 3 should start shipping with all symlinks from past 3.x minor versions to the newest minor version. then we could switch to #!/usr/bin/env python3.4 if it uses e.g. pathlib and would be sure that the script will work with future versions. but then again, users can install pathlib for 3.3? > > > Ryan Gonzalez schrieb am Do., 12. M?rz 2015 um 00:13 Uhr: >>> On Wed, Mar 11, 2015 at 5:54 PM, Chris Barker wrote: >> >>>> On Wed, Mar 11, 2015 at 3:41 PM, Gregory P. Smith wrote: >>>> I think /usr/bin/env is a bad idea to promote. It changes from an explicit designation of what you need in order to execute to something being derived from the users $PATH. It is *less specific* than /usr/bin/python. For some people that is the goal, but I discourage it. >>> >>> I don't think the goal is specificity, or lack there of, but a shift in control: >>> >>> Who controls where python is installed? >>> >>> If you put an explicit full path on the #! line, then the script author controls which python is used, AND where it must be installed. >>> >>> If you put /usr/bin env on the #! line, then the script author is saying 'use python", and the script user can have python installed anywhere they want, and can use PATH to tell everything what the default it. >>> >>> I think that _should_ be encouraged -- but ideally with some versioning: >>> >>> #!/usr/bin/env python2 >>> >>> or even: >>> >>> #!/usr/bin/env python3.4 >> >> Please, just explicit major versions, not minor ones. python3, not python3.4. Just earlier this year I had to fix about 20 scripts that ran perfectly on 3.4 but whose hashbang explicitly used 3.1. It sucks. >> >>> If any of you remember what a pain it was when RedHat installed system tools with (I think): >>> >>> !#/usr/bin/env python >>> >>> You'll know what a nightmare that was -- you could not upgrade the 'default" python without breaking stuff. >>> >>> So system tools should probably use the full specific path -- the script is tied closely to the environment. But if they had at least put a version in there, it would have worked fine: >>> >>> #!/usr/bin/env python1.5 >>> >>> These days, I try to use distutils or setuptools to install scripts, and they can do the right thing to use the correct python to start up anyway... >>> >>> -CHB >>> >>> >>> >>> -- >>> >>> Christopher Barker, Ph.D. >>> Oceanographer >>> >>> Emergency Response Division >>> NOAA/NOS/OR&R (206) 526-6959 voice >>> 7600 Sand Point Way NE (206) 526-6329 fax >>> Seattle, WA 98115 (206) 526-6317 main reception >>> >>> Chris.Barker at noaa.gov >>> >>> _______________________________________________ >>> Python-ideas mailing list >>> Python-ideas at python.org >>> https://mail.python.org/mailman/listinfo/python-ideas >>> Code of Conduct: http://python.org/psf/codeofconduct/ >> >> >> -- >> Ryan >> [ERROR]: Your autotools build scripts are 200 lines longer than your program. Something?s wrong. >> http://kirbyfan64.github.io/ >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Sat Mar 14 16:27:02 2015 From: rosuav at gmail.com (Chris Angelico) Date: Sun, 15 Mar 2015 02:27:02 +1100 Subject: [Python-ideas] Migration of /usr/bin/python to python3 In-Reply-To: References: <20150311194325.GA15390@python.ca> <5500A7AE.9020400@egenix.com> Message-ID: On Sun, Mar 15, 2015 at 12:32 AM, Philipp A. wrote: > the only thing worth discussing IMHO is if python 3 should start shipping > with all symlinks from past 3.x minor versions to the newest minor version. > then we could switch to #!/usr/bin/env python3.4 if it uses e.g. pathlib and > would be sure that the script will work with future versions. but then > again, users can install pathlib for 3.3? I would strongly advise NOT doing this. If you say "python3.4", then it should run on Python 3.4, not on Python 3.4-or-anything-later. ChrisA From barry at python.org Sat Mar 14 16:47:59 2015 From: barry at python.org (Barry Warsaw) Date: Sat, 14 Mar 2015 11:47:59 -0400 Subject: [Python-ideas] Migration of /usr/bin/python to python3 References: <20150311194325.GA15390@python.ca> <5500A7AE.9020400@egenix.com> Message-ID: <20150314114759.5fb7f8e3@anarchist.wooz.org> On Mar 14, 2015, at 08:03 AM, Andrew Barnert wrote: >Continuing to use python rather than python2 will work out of the box on >nearly every *nix system today, with no exceptions larger than Arch >Linux. The part of the PEP aimed at Python distribution providers should recommend that `python2` be used in preference of just `python` and that the symlink be present. Cheers, -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 819 bytes Desc: OpenPGP digital signature URL: From barry at python.org Sat Mar 14 16:57:19 2015 From: barry at python.org (Barry Warsaw) Date: Sat, 14 Mar 2015 11:57:19 -0400 Subject: [Python-ideas] Migration of /usr/bin/python to python3 References: <20150311194325.GA15390@python.ca> <041B5A0E-4C77-4683-9B0A-98432C139F48@stufft.io> Message-ID: <20150314115719.23ea9d75@anarchist.wooz.org> On Mar 13, 2015, at 09:57 PM, Nick Coghlan wrote: >Right, this most recently came up in the context of Fedora's plans to >transition to only having Python 3 in the base install >(https://fedoraproject.org/wiki/Changes/Python_3_as_Default). At the >moment the expected consequence is that there will come a Fedora >version (tentatively Fedora 23) where, out of the box, >"/usr/bin/python" and "/usr/bin/env python" will just stop working. > >Packages that are part of the distro aren't a problem - those are all >being dealt with as the transition progresses, and any which >specifically need Python 2 will depend on it accordingly. > >The problem arises with custom user packages and scripts, and third >party packages and scripts. Sure, agreed so far. But you're not going to be removing Python 2 from the Fedora archive right? So any user with third party packages that require Python 2 could just install it and their stuff would continue to work. At least that's the plan on Debian/Ubuntu. I'd like to be at the point[*] where the default installs would not contain Python 2, but a simple `apt-get` would install it from the archives for anybody who needs it. That seems like a safer option than changing the symlink. Cheers, -Barry [*] I'm aiming for Debian Stretch and Ubuntu 16.04 Xylophonic X-ray fish LTS. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 819 bytes Desc: OpenPGP digital signature URL: From barry at python.org Sat Mar 14 16:59:23 2015 From: barry at python.org (Barry Warsaw) Date: Sat, 14 Mar 2015 11:59:23 -0400 Subject: [Python-ideas] About people's accounts being deactivated due to bounces References: <5260F722-8D2C-420D-BCBD-6014579E993F@yahoo.com> Message-ID: <20150314115923.27aa5ead@anarchist.wooz.org> On Mar 13, 2015, at 02:46 PM, Andrew Barnert wrote: >I think this may be related to a problem with changes in DKIM policy at >Yahoo, or an expiration of DKIM certs at Python.org, or something else that >may have changed somewhere in that area. Please contact postmaster at python.org. Mark recently upgraded Mailman on mail.python.org and the new version has some DKIM mitigation stuff. Cheers, -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 819 bytes Desc: OpenPGP digital signature URL: From rosuav at gmail.com Sat Mar 14 17:44:38 2015 From: rosuav at gmail.com (Chris Angelico) Date: Sun, 15 Mar 2015 03:44:38 +1100 Subject: [Python-ideas] Rewriting the build system (part 2) In-Reply-To: References: Message-ID: On Sat, Mar 14, 2015 at 6:47 AM, Chris Barker wrote: > On Fri, Mar 13, 2015 at 12:40 PM, Ryan Gonzalez wrote: >> >> That written in Python, so Python would require Python to build which >> would require Python to build which would require... > > > Well, you can't buid a C compiler without a C compiler either (or SOME > compiler). And given that you should be able to do cross-compilation for > platforms that don't currently have a python build, I'm not sure that's a > killer. > > It it so wrong that python eats its own dog food in this manner? > > -CHB I've been just skimming this thread rather than reading in detail, so please forgive me if this question is completely unrelated. Will this affect the commands required to build Python from source, or is all this just for actual development? Consider two completely different workflows here: Firstly, a core Python dev. S/he wants to be able to efficiently build, edit, etc, and wants to be able to edit any part of the code. and secondly, someone who has little knowledge of Python's internals, and just wants to build the latest version (either from hg or from a source tarball). In the first case, it's fine to require that some version of Python be already installed. If editing file XYZ requires that file ABC be rebuilt from it, and the rebuild process requires Python 3.3 or newer, that's not going to hurt people. The modified ABC gets checked in along with the change to XYZ, and all's well. But in the second case, it'd be much safer to minimize the requirements. If you can untar the source and just "./configure; make; sudo make install" your way to having the latest Python, that's a LOT easier than I've just been doing trying to get my Dad onto a newer Mozilla Thunderbird. (I don't understand what it's doing all along there, but it pulls a whole bunch of separate repositories when I say update, and there's a separate bootstrap step, and ... I dunno.) ChrisA From brett at python.org Sat Mar 14 17:52:22 2015 From: brett at python.org (Brett Cannon) Date: Sat, 14 Mar 2015 16:52:22 +0000 Subject: [Python-ideas] About people's accounts being deactivated due to bounces In-Reply-To: <20150314115923.27aa5ead@anarchist.wooz.org> References: <5260F722-8D2C-420D-BCBD-6014579E993F@yahoo.com> <20150314115923.27aa5ead@anarchist.wooz.org> Message-ID: Already did. On Sat, Mar 14, 2015, 12:02 Barry Warsaw wrote: > On Mar 13, 2015, at 02:46 PM, Andrew Barnert wrote: > > >I think this may be related to a problem with changes in DKIM policy at > >Yahoo, or an expiration of DKIM certs at Python.org, or something else > that > >may have changed somewhere in that area. > > Please contact postmaster at python.org. Mark recently upgraded Mailman on > mail.python.org and the new version has some DKIM mitigation stuff. > > Cheers, > -Barry > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From abarnert at yahoo.com Sun Mar 15 04:50:37 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Sat, 14 Mar 2015 20:50:37 -0700 Subject: [Python-ideas] Migration of /usr/bin/python to python3 In-Reply-To: <20150314114759.5fb7f8e3@anarchist.wooz.org> References: <20150311194325.GA15390@python.ca> <5500A7AE.9020400@egenix.com> <20150314114759.5fb7f8e3@anarchist.wooz.org> Message-ID: On Mar 14, 2015, at 8:47 AM, Barry Warsaw wrote: > >> On Mar 14, 2015, at 08:03 AM, Andrew Barnert wrote: >> >> Continuing to use python rather than python2 will work out of the box on >> nearly every *nix system today, with no exceptions larger than Arch >> Linux. > > The part of the PEP aimed at Python distribution providers should recommend > that `python2` be used in preference of just `python` and that the symlink be > present. Sure, that's exactly what it recommends. But you can't retroactively make the symlink present on, say, OS X 10.10 or CentOS 5, so what good does that recommendation actually do? Maybe in 5-10 years we'll be at a point where you can count on the symlink being there on most *nix systems that have Python 2 pre-installed. Except, of course, that most *nix systems won't have Python 2 pre-installed anymore by then (and those that do will probably overlap quite a bit with the ones that don't have the symlink). From barry at python.org Mon Mar 16 00:06:34 2015 From: barry at python.org (Barry Warsaw) Date: Sun, 15 Mar 2015 19:06:34 -0400 Subject: [Python-ideas] Migration of /usr/bin/python to python3 In-Reply-To: References: <20150311194325.GA15390@python.ca> <5500A7AE.9020400@egenix.com> <20150314114759.5fb7f8e3@anarchist.wooz.org> Message-ID: <20150315190634.4a41c33c@marathon> On Mar 14, 2015, at 08:50 PM, Andrew Barnert wrote: >Maybe in 5-10 years we'll be at a point where you can count on the symlink >being there on most *nix systems that have Python 2 pre-installed. Except, of >course, that most *nix systems won't have Python 2 pre-installed anymore by >then (and those that do will probably overlap quite a bit with the ones that >don't have the symlink). That's the point. By the time Python 2.7 EOLs and (IMHO) we make any change to /usr/bin/python, there will have been plenty of time to make sure anybody running a modern OS version will have the python2 symlink. Given how easy it is to add yourself, I'm not worried about legacy systems or rebels who wouldn't add the link. Cheers, -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 819 bytes Desc: OpenPGP digital signature URL: From tjreedy at udel.edu Mon Mar 16 00:09:39 2015 From: tjreedy at udel.edu (Terry Reedy) Date: Sun, 15 Mar 2015 19:09:39 -0400 Subject: [Python-ideas] Add str.bmp() to only expand non-BMP chars, for tkinter use Message-ID: 3.x comes with builtin ascii(obj) to return a string representation of obj that only has ascii characters. >>> s = 'a\xaa\ua000\U0001a000' >>> len(s) 4 >>> sa = ascii(s) >>> sa, len(sa) ("'a\\xaa\\ua000\\U0001a000'", 23) This allows any string to be printed on even a minimal ascii terminal. Python also comes with the tkinter interface to tk. Tk widgets are not limited to ascii but support the full BMP subset of unicode. (This is better than Windows consoles limited by codepages.) Thus, for use with tkinter, ascii() has two faults: it adds a quote at beginning and end of the string (like repr); it expands too much. I looked at repr, which expands less, but it seems to be buggy in that it is not consistent in its handling of non-BMP chars. >>> s1 = 'a\xaa\ua000\U00011000' >>> sa = 'a\xaa\ua000\U0001a000' >>> s1r = repr(s1); len(s1r) 6 >>> sar = repr(sa); len(sar) 15 '\U0001a000' gets expanded, and can be printed. '\U00011000' does not, and cannot be consistently printed. >>> s1r # only works at >>> prompt "'a\xaa\ua000\U00011000'" >>> print(s1r) # required in programs Traceback (most recent call last): File "", line 1, in print(s1r) File "C:\Programs\Python34\lib\idlelib\PyShell.py", line 1347, in write return self.shell.write(s, self.tags) UnicodeEncodeError: 'UCS-2' codec can't encode characters in position 4-4: Non-BMP character not supported in Tk Printing s1 or sa directly, by either means, gives the same error. (Since '>>> expr' is supposed to be the same as 'print(expr)' the above difference puzzles me.) Even if repr always worked as it does for '\U0001a000', there would still be the problem of the added quotes. I therefore proposed the addition of a new str method, such as 's.bmp()', that returns s with all non-BMP chars, and only such chars, expanded. Since strings (in CPython) are internally marked by 'kind', the method would just return s when no expansion is needed. I presume it could otherwise re-use the expansion code already in repr. Aside from tkinter programmers in general, this issue bites Idle in at least two ways. Internally, filenames can contain non-BMP chars and Idle displays them in 3 places. See http://bugs.python.org/issue23672. Externally, Idle users sometimes want to print strings with non-BMP chars. I believe the automatic use of .bmp() with console prints could be user selectable. There have been issues about this on both our tracker and StackOverflow. I believe that the use of non-BMP chars is becoming more common and can no longer be simply dismissed as too rare to worry about. Telling Windows users that they are better off than if they use python directly, with the windows console, does not solve the inability to print any Python string. This proposal would. -- Terry Jan Reedy From cfkaran2 at gmail.com Mon Mar 16 01:02:31 2015 From: cfkaran2 at gmail.com (Cem Karan) Date: Sun, 15 Mar 2015 20:02:31 -0400 Subject: [Python-ideas] Modify inspect.getmembers() to return object members in source code order Message-ID: I'd like to suggest that inspect.getmembers() be modified to return elements in the order they are defined in the source code rather than in alphabetical order. For people that want the output in sorted order, the call would become 'sorted(inspect.getmembers())', which is relatively easy to do. For me, and anyone else that is generating code from source code, the advantage is that the generated code is 'in the right order'. As an example, consider generating unit tests for code that really should have had unit tests as the code was written, but didn't[1]. If inspect.getmembers() returned the elements in the same order as they appear in the source code, then we can generate code similar to the following[2]: """ members = inspect.getmembers(SomeoneNeedsToCleanThisUp) names = [x[0] for x in members] def test_gen(name): return (" def test_{0!s}(self):\n".format(name) + " raise NotImplementedError('He Who Shall Not Be Named, But Is On My Special List had better fill this in NOW!')\n" + " # End of test_{0!s}()\n".format(name)) tests = "\n".join([test_gen(x) for x in names]) # Writing out names to the right location, etc. """ Thoughts? Thanks, Cem Karan [1] This may possibly be the reason why I want to propose this change... [2] Banged out in Mail.app very quickly, may have bugs, and definitely is not production code. From storchaka at gmail.com Mon Mar 16 01:03:03 2015 From: storchaka at gmail.com (Serhiy Storchaka) Date: Mon, 16 Mar 2015 02:03:03 +0200 Subject: [Python-ideas] Add str.bmp() to only expand non-BMP chars, for tkinter use In-Reply-To: References: Message-ID: On 16.03.15 01:09, Terry Reedy wrote: > 3.x comes with builtin ascii(obj) to return a string representation of > obj that only has ascii characters. > > >>> s = 'a\xaa\ua000\U0001a000' > >>> len(s) > 4 > >>> sa = ascii(s) > >>> sa, len(sa) > ("'a\\xaa\\ua000\\U0001a000'", 23) > > This allows any string to be printed on even a minimal ascii terminal. > > Python also comes with the tkinter interface to tk. Tk widgets are not > limited to ascii but support the full BMP subset of unicode. (This is > better than Windows consoles limited by codepages.) Thus, for use with > tkinter, ascii() has two faults: it adds a quote at beginning and end of > the string (like repr); it expands too much. > > I looked at repr, which expands less, but it seems to be buggy in that > it is not consistent in its handling of non-BMP chars. > > >>> s1 = 'a\xaa\ua000\U00011000' > >>> sa = 'a\xaa\ua000\U0001a000' > >>> s1r = repr(s1); len(s1r) > 6 > >>> sar = repr(sa); len(sar) > 15 > > '\U0001a000' gets expanded, and can be printed. > '\U00011000' does not, and cannot be consistently printed. > > >>> s1r # only works at >>> prompt > "'a\xaa\ua000\U00011000'" > >>> print(s1r) # required in programs > Traceback (most recent call last): > File "", line 1, in > print(s1r) > File "C:\Programs\Python34\lib\idlelib\PyShell.py", line 1347, in write > return self.shell.write(s, self.tags) > UnicodeEncodeError: 'UCS-2' codec can't encode characters in position > 4-4: Non-BMP character not supported in Tk > > Printing s1 or sa directly, by either means, gives the same error. > (Since '>>> expr' is supposed to be the same as 'print(expr)' the above > difference puzzles me.) > > Even if repr always worked as it does for '\U0001a000', there would > still be the problem of the added quotes. I therefore proposed the > addition of a new str method, such as 's.bmp()', that returns s with all > non-BMP chars, and only such chars, expanded. Since strings (in > CPython) are internally marked by 'kind', the method would just return s > when no expansion is needed. I presume it could otherwise re-use the > expansion code already in repr. > > Aside from tkinter programmers in general, this issue bites Idle in at > least two ways. Internally, filenames can contain non-BMP chars and > Idle displays them in 3 places. > See http://bugs.python.org/issue23672. > Externally, Idle users sometimes want to print strings with non-BMP > chars. I believe the automatic use of .bmp() with console prints could > be user selectable. There have been issues about this on both our > tracker and StackOverflow. > > I believe that the use of non-BMP chars is becoming more common and can > no longer be simply dismissed as too rare to worry about. Telling > Windows users that they are better off than if they use python directly, > with the windows console, does not solve the inability to print any > Python string. This proposal would. > Right now I'm writing a patch that implements similar idea for issue18814. >>> codecs.convert_astral('a\u20ac\U000e007f', 'strict') Traceback (most recent call last): File "", line 1, in File "/home/serhiy/py/cpython/Lib/codecs.py", line 1159, in convert_astral 'astral characters')) UnicodeTranslateError: can't translate character '\U000e007f' in position 2: astral characters >>> codecs.convert_astral('a\u20ac\U000e007f', 'ignore') 'a?' >>> codecs.convert_astral('a\u20ac\U000e007f', 'replace') 'a??' >>> codecs.convert_astral('a\u20ac\U000e007f', 'backslashreplace') 'a?\\U000e007f' >>> codecs.convert_astral('a\u20ac\U000e007f', 'namereplace') 'a?\\N{CANCEL TAG}' >>> codecs.convert_astral('a\u20ac\U000e007f', 'xmlcharrefreplace') 'a?󠁿' From rosuav at gmail.com Mon Mar 16 01:08:55 2015 From: rosuav at gmail.com (Chris Angelico) Date: Mon, 16 Mar 2015 11:08:55 +1100 Subject: [Python-ideas] Modify inspect.getmembers() to return object members in source code order In-Reply-To: References: Message-ID: On Mon, Mar 16, 2015 at 11:02 AM, Cem Karan wrote: > I'd like to suggest that inspect.getmembers() be modified to return elements in the order they are defined in the source code rather than in alphabetical order. > I'm not sure that that's really possible, unless you also are proposing that OrderedDict be used internally instead of a regular dict for all namespaces. That information doesn't exist. > As an example, consider generating unit tests for code that really > should have had unit tests as the code was written, but didn't If I had to solve this problem, I'd parse the source code. Chances are your functions/methods follow a straight-forward layout, so you should be able to read them and parse them fairly easily. ChrisA From njs at pobox.com Mon Mar 16 01:13:05 2015 From: njs at pobox.com (Nathaniel Smith) Date: Sun, 15 Mar 2015 17:13:05 -0700 Subject: [Python-ideas] Modify inspect.getmembers() to return object members in source code order In-Reply-To: References: Message-ID: Line number information is retained, so you could sort by that. In fact you could do this without modifying inspect.getmembers... -n On Mar 15, 2015 5:09 PM, "Chris Angelico" wrote: > On Mon, Mar 16, 2015 at 11:02 AM, Cem Karan wrote: > > I'd like to suggest that inspect.getmembers() be modified to return > elements in the order they are defined in the source code rather than in > alphabetical order. > > > > I'm not sure that that's really possible, unless you also are > proposing that OrderedDict be used internally instead of a regular > dict for all namespaces. That information doesn't exist. > > > As an example, consider generating unit tests for code that really > > should have had unit tests as the code was written, but didn't > > If I had to solve this problem, I'd parse the source code. Chances are > your functions/methods follow a straight-forward layout, so you should > be able to read them and parse them fairly easily. > > ChrisA > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From random832 at fastmail.us Mon Mar 16 01:23:53 2015 From: random832 at fastmail.us (random832 at fastmail.us) Date: Sun, 15 Mar 2015 20:23:53 -0400 Subject: [Python-ideas] Modify inspect.getmembers() to return object members in source code order In-Reply-To: References: Message-ID: <1426465433.1977968.240744977.79DC0319@webmail.messagingengine.com> On Sun, Mar 15, 2015, at 20:08, Chris Angelico wrote: > I'm not sure that that's really possible, unless you also are > proposing that OrderedDict be used internally instead of a regular > dict for all namespaces. That information doesn't exist. There are dict implementations where all items that were added with no deletions starting from when the dict was empty will be returned by iteration in the order they were originally added. Adding such an implementation (or changing dict itself to have this guarantee) and using it for various things like keyword args is perennially suggested. From cfkaran2 at gmail.com Mon Mar 16 01:29:25 2015 From: cfkaran2 at gmail.com (Cem Karan) Date: Sun, 15 Mar 2015 20:29:25 -0400 Subject: [Python-ideas] Modify inspect.getmembers() to return object members in source code order In-Reply-To: References: Message-ID: <1BF7A457-39AB-4D98-9CE9-CAC7100079C9@gmail.com> On Mar 15, 2015, at 8:08 PM, Chris Angelico wrote: > On Mon, Mar 16, 2015 at 11:02 AM, Cem Karan wrote: >> I'd like to suggest that inspect.getmembers() be modified to return elements in the order they are defined in the source code rather than in alphabetical order. >> > > I'm not sure that that's really possible, unless you also are > proposing that OrderedDict be used internally instead of a regular > dict for all namespaces. That information doesn't exist. That was what I was afraid of. >> As an example, consider generating unit tests for code that really >> should have had unit tests as the code was written, but didn't > > If I had to solve this problem, I'd parse the source code. Chances are > your functions/methods follow a straight-forward layout, so you should > be able to read them and parse them fairly easily. They are, and I can do so, but I got my hopes up that inspect.getmembers() could be modified easily. Oh well, such is life. Thanks, Cem Karan From rosuav at gmail.com Mon Mar 16 01:37:48 2015 From: rosuav at gmail.com (Chris Angelico) Date: Mon, 16 Mar 2015 11:37:48 +1100 Subject: [Python-ideas] Modify inspect.getmembers() to return object members in source code order In-Reply-To: <1BF7A457-39AB-4D98-9CE9-CAC7100079C9@gmail.com> References: <1BF7A457-39AB-4D98-9CE9-CAC7100079C9@gmail.com> Message-ID: On Mon, Mar 16, 2015 at 11:29 AM, Cem Karan wrote: > On Mar 15, 2015, at 8:08 PM, Chris Angelico wrote: > >> On Mon, Mar 16, 2015 at 11:02 AM, Cem Karan wrote: >>> I'd like to suggest that inspect.getmembers() be modified to return elements in the order they are defined in the source code rather than in alphabetical order. >>> >> >> I'm not sure that that's really possible, unless you also are >> proposing that OrderedDict be used internally instead of a regular >> dict for all namespaces. That information doesn't exist. > > That was what I was afraid of. Nathaniel offered information which I hadn't known, which probably solves your problem. You get line number info, and can sort by that - just on the result of getmembers. ChrisA From cfkaran2 at gmail.com Mon Mar 16 01:39:59 2015 From: cfkaran2 at gmail.com (Cem Karan) Date: Sun, 15 Mar 2015 20:39:59 -0400 Subject: [Python-ideas] Modify inspect.getmembers() to return object members in source code order In-Reply-To: References: Message-ID: On Mar 15, 2015, at 8:13 PM, Nathaniel Smith wrote: > Line number information is retained, so you could sort by that. In fact you could do this without modifying inspect.getmembers... You mean via a combinations of inspect.getmembers() and inspect.getsourcelines()? Yeah, I think that could work. I'll have to play around a bit to be sure I've got the hang of it, but I'm going to provisionally withdraw my suggestion about changing getmembers(). Thank you Nathaniel! > -n > > On Mar 15, 2015 5:09 PM, "Chris Angelico" wrote: > On Mon, Mar 16, 2015 at 11:02 AM, Cem Karan wrote: > > I'd like to suggest that inspect.getmembers() be modified to return elements in the order they are defined in the source code rather than in alphabetical order. > > > > I'm not sure that that's really possible, unless you also are > proposing that OrderedDict be used internally instead of a regular > dict for all namespaces. That information doesn't exist. > > > As an example, consider generating unit tests for code that really > > should have had unit tests as the code was written, but didn't > > If I had to solve this problem, I'd parse the source code. Chances are > your functions/methods follow a straight-forward layout, so you should > be able to read them and parse them fairly easily. > > ChrisA Thanks, Cem Karan From abarnert at yahoo.com Mon Mar 16 06:43:46 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Sun, 15 Mar 2015 22:43:46 -0700 Subject: [Python-ideas] Migration of /usr/bin/python to python3 In-Reply-To: <20150315190634.4a41c33c@marathon> References: <20150311194325.GA15390@python.ca> <5500A7AE.9020400@egenix.com> <20150314114759.5fb7f8e3@anarchist.wooz.org> <20150315190634.4a41c33c@marathon> Message-ID: <7512BEB1-7654-43C3-B35F-21F9638CD7EB@yahoo.com> On Mar 15, 2015, at 4:06 PM, Barry Warsaw wrote: > >> On Mar 14, 2015, at 08:50 PM, Andrew Barnert wrote: >> >> Maybe in 5-10 years we'll be at a point where you can count on the symlink >> being there on most *nix systems that have Python 2 pre-installed. Except, of >> course, that most *nix systems won't have Python 2 pre-installed anymore by >> then (and those that do will probably overlap quite a bit with the ones that >> don't have the symlink). > > That's the point. By the time Python 2.7 EOLs and (IMHO) we make any change > to /usr/bin/python, there will have been plenty of time to make sure anybody > running a modern OS version will have the python2 symlink. If you're suggesting that people distributing Python 2.7 scripts in 5-10 years should use python2, then sure, I agree. But you're suggesting they should do that today (or, if not you, then the PEP, and others in this thread). And there hasn't been plenty of time today. And people shouldn't be distributing new Python 2.7 scripts in 5-10 years. If this is still a major issue when 2.7 EOLs, then 2.7 shouldn't EOL, and 3.x has been a failure. But I personally think that's unlikely. Most people will switch to 3 by then, and those who don't will mostly have special circumstances (e.g., they're adding to a server that has other components that require legacy software). > Given how easy it is to add yourself, I'm not worried about legacy systems or > rebels who wouldn't add the link. People using their brand-new MacBook or a hosted CentOS server aren't rebels, and they're not using legacy systems. (They're also less likely to know how to deal with a problem than someone running a linux desktop or maintaining their own server.) From abarnert at yahoo.com Mon Mar 16 06:45:19 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Sun, 15 Mar 2015 22:45:19 -0700 Subject: [Python-ideas] Add str.bmp() to only expand non-BMP chars, for tkinter use In-Reply-To: References: Message-ID: It seems like what you really want here is a new codec, like unicode-escape but only escaping non-BMP characters, not a new repr-like function. On Mar 15, 2015, at 4:09 PM, Terry Reedy wrote: > > 3.x comes with builtin ascii(obj) to return a string representation of obj that only has ascii characters. > > >>> s = 'a\xaa\ua000\U0001a000' > >>> len(s) > 4 > >>> sa = ascii(s) > >>> sa, len(sa) > ("'a\\xaa\\ua000\\U0001a000'", 23) > > This allows any string to be printed on even a minimal ascii terminal. > > Python also comes with the tkinter interface to tk. Tk widgets are not limited to ascii but support the full BMP subset of unicode. (This is better than Windows consoles limited by codepages.) Thus, for use with tkinter, ascii() has two faults: it adds a quote at beginning and end of the string (like repr); it expands too much. > > I looked at repr, which expands less, but it seems to be buggy in that it is not consistent in its handling of non-BMP chars. > > >>> s1 = 'a\xaa\ua000\U00011000' > >>> sa = 'a\xaa\ua000\U0001a000' > >>> s1r = repr(s1); len(s1r) > 6 > >>> sar = repr(sa); len(sar) > 15 > > '\U0001a000' gets expanded, and can be printed. > '\U00011000' does not, and cannot be consistently printed. > > >>> s1r # only works at >>> prompt > "'a\xaa\ua000\U00011000'" > >>> print(s1r) # required in programs > Traceback (most recent call last): > File "", line 1, in > print(s1r) > File "C:\Programs\Python34\lib\idlelib\PyShell.py", line 1347, in write > return self.shell.write(s, self.tags) > UnicodeEncodeError: 'UCS-2' codec can't encode characters in position 4-4: Non-BMP character not supported in Tk > > Printing s1 or sa directly, by either means, gives the same error. (Since '>>> expr' is supposed to be the same as 'print(expr)' the above difference puzzles me.) > > Even if repr always worked as it does for '\U0001a000', there would still be the problem of the added quotes. I therefore proposed the addition of a new str method, such as 's.bmp()', that returns s with all non-BMP chars, and only such chars, expanded. Since strings (in CPython) are internally marked by 'kind', the method would just return s when no expansion is needed. I presume it could otherwise re-use the expansion code already in repr. > > Aside from tkinter programmers in general, this issue bites Idle in at least two ways. Internally, filenames can contain non-BMP chars and Idle displays them in 3 places. > See http://bugs.python.org/issue23672. > Externally, Idle users sometimes want to print strings with non-BMP chars. I believe the automatic use of .bmp() with console prints could be user selectable. There have been issues about this on both our tracker and StackOverflow. > > I believe that the use of non-BMP chars is becoming more common and can no longer be simply dismissed as too rare to worry about. Telling Windows users that they are better off than if they use python directly, with the windows console, does not solve the inability to print any Python string. This proposal would. > > -- > Terry Jan Reedy > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ From ncoghlan at gmail.com Mon Mar 16 09:27:29 2015 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 16 Mar 2015 18:27:29 +1000 Subject: [Python-ideas] CPython needs to revamp/replace it's build system && Cross-compiling Python sucks In-Reply-To: References: <49c23aa9-51cf-47b7-843b-8c4b58d01cbd@email.android.com> Message-ID: On 14 Mar 2015 04:05, "Ryan Gonzalez" wrote: > > If C99's ok, there's hope. :) As far as I can recall, the main culprit there in relation to CPython itself has long been MSVC's lack of support for C99, but the very fact we use the native build chain on Windows means it isn't relevant to the choice of build system on other platforms. I may have forgotten another reason for the C89 requirement, but if I'm right then a build tool for non-Windows systems could potentially be allowed to require C99 support. Regards, Nick. > > C++'s main issue is that there are platforms that support C99 but have out-of-date or non-complaint C++ compilers. I love the language (yes, I'm weird), but it's tricky to have truly portable C++ programs because if all the holes in various compilers. Just look at all the #ifdef's in Boost. > > On Fri, Mar 13, 2015 at 7:15 AM, Nick Coghlan wrote: >> >> On 12 March 2015 at 04:30, Ryan Gonzalez wrote: >> > On Wed, Mar 11, 2015 at 10:07 AM, Brett Cannon wrote: >> >> >> >> >> >> >> >> On Tue, Mar 10, 2015 at 11:40 AM Ryan Gonzalez wrote: >> >>> >> >>> On Tue, Mar 10, 2015 at 8:43 AM, Nick Coghlan wrote: >> >>>> >> >>>> On 10 March 2015 at 21:20, Russell Keith-Magee < russell at keith-magee.com> >> >>>> wrote: >> >>>> > >> >>>> > On Tue, Mar 10, 2015 at 10:43 AM, Ryan wrote: >> >>>> >> >> >>>> >> Russell Keith-Magee wrote: >> >>>> >> >Hi Ryan, >> >>>> >> > >> >>>> >> >First off I'm no fan of autoconf, but... >> >>>> >> > >> >>>> >> >> >>>> >> Who is other than the one who designed it? :) >> >>>> > >> >>>> > Let me be clear - there's a difference between "not a fan" and >> >>>> > "calling for >> >>>> > it's head". Autoconf is crufty, annoying, and sometimes aggravating - >> >>>> > but it >> >>>> > *works*, and has had some pretty seriously battle testing - both in >> >>>> > the >> >>>> > abstract "autoconf as a product", and in the specific "autoconf as >> >>>> > used in >> >>>> > Python". This isn't something that can be said of most of the >> >>>> > alternatives >> >>>> > in the abstract case, or *any* of the alternatives in the Python case. >> >>>> >> >>>> Building architecture dependent software is a genuinely hard problem, >> >>>> which is why it's full of "80%" solutions like autotools and the >> >>>> setuptools/distutils pairing. Yes, they're quirky and annoying, but >> >>>> they also basically do their job, so folks tend to develop a high >> >>>> tolerance level for their quirks. >> >>>> >> >>>> In our particular case, we also have the fun problem of "How do you >> >>>> build CPython locally when you don't have a working Python interpreter >> >>>> yet?". A number of the more interesting alternative C/C++ build >> >>>> systems (like Meson, Scons, Waf and the Google "gyp" project that >> >>>> powers the node-gyp native extension build system for Node.js) require >> >>>> an already working Python interpreter to get things started (when >> >>>> starting from a completely blank slate, even our own build sequence >> >>>> has to build a deliberately broken partial interpreter just to freeze >> >>>> importlib._bootstrap before going on to build a properly configured >> >>>> one that can actually import modules). >> >>> >> >>> >> >>> What about something like Google's new GN? It's designed to replace Gyp >> >>> in building Chromium and is written in C++. >> >>> >> >>> I'm pretty sure using a build system written in Lua to build Python is >> >>> kind of weird. >> >> >> >> >> >>> >> >>> >> >>> With all that in mind...what exactly would be requirements that Python >> >>> would need in an autotools replacement? >> >> >> >> >> >> Can't require anything more than an ANSI C compiler basically. People run >> >> Python in places where they don't have much more than gcc. >> > >> > >> > Not even C99? Darn it... >> >> C99 would probably be OK these days. If there's a good enough >> cross-compiling story, I'm also not sure we should entirely rule out >> C++, as it's potentially OK if *developing* CPython requires a >> mainstream platform with a C++ compiler. >> >> But yeah, you can see why folks look at this problem and end up >> thinking "well, you know, maybe autotools isn't *that* bad..." >> >> Cheers, >> Nick. >> >> -- >> Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > > > > > -- > Ryan > [ERROR]: Your autotools build scripts are 200 lines longer than your program. Something?s wrong. > http://kirbyfan64.github.io/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Mon Mar 16 09:55:30 2015 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 16 Mar 2015 09:55:30 +0100 Subject: [Python-ideas] CPython needs to revamp/replace it's build system && Cross-compiling Python sucks References: <49c23aa9-51cf-47b7-843b-8c4b58d01cbd@email.android.com> Message-ID: <20150316095530.548e590b@fsol> On Mon, 16 Mar 2015 18:27:29 +1000 Nick Coghlan wrote: > On 14 Mar 2015 04:05, "Ryan Gonzalez" wrote: > > > > If C99's ok, there's hope. :) > > As far as I can recall, the main culprit there in relation to CPython > itself has long been MSVC's lack of support for C99, but the very fact we > use the native build chain on Windows means it isn't relevant to the choice > of build system on other platforms. VS 2015 (which Python 3.5 is gonna use) should support at least a subset of C99. > I may have forgotten another reason for the C89 requirement, but if I'm > right then a build tool for non-Windows systems could potentially be > allowed to require C99 support. Other (legacy? enterprise?) C compilers may not support C99, but at this point we can probably declare them unsupported. Regards Antoine. From victor.stinner at gmail.com Mon Mar 16 11:39:29 2015 From: victor.stinner at gmail.com (Victor Stinner) Date: Mon, 16 Mar 2015 11:39:29 +0100 Subject: [Python-ideas] Add str.bmp() to only expand non-BMP chars, for tkinter use In-Reply-To: References: Message-ID: I agree that a codec would be more appropriate. See: http://bugs.python.org/issue18814 Victor 2015-03-16 6:45 GMT+01:00 Andrew Barnert : > It seems like what you really want here is a new codec, like unicode-escape but only escaping non-BMP characters, not a new repr-like function. > > On Mar 15, 2015, at 4:09 PM, Terry Reedy wrote: >> >> 3.x comes with builtin ascii(obj) to return a string representation of obj that only has ascii characters. >> >> >>> s = 'a\xaa\ua000\U0001a000' >> >>> len(s) >> 4 >> >>> sa = ascii(s) >> >>> sa, len(sa) >> ("'a\\xaa\\ua000\\U0001a000'", 23) >> >> This allows any string to be printed on even a minimal ascii terminal. >> >> Python also comes with the tkinter interface to tk. Tk widgets are not limited to ascii but support the full BMP subset of unicode. (This is better than Windows consoles limited by codepages.) Thus, for use with tkinter, ascii() has two faults: it adds a quote at beginning and end of the string (like repr); it expands too much. >> >> I looked at repr, which expands less, but it seems to be buggy in that it is not consistent in its handling of non-BMP chars. >> >> >>> s1 = 'a\xaa\ua000\U00011000' >> >>> sa = 'a\xaa\ua000\U0001a000' >> >>> s1r = repr(s1); len(s1r) >> 6 >> >>> sar = repr(sa); len(sar) >> 15 >> >> '\U0001a000' gets expanded, and can be printed. >> '\U00011000' does not, and cannot be consistently printed. >> >> >>> s1r # only works at >>> prompt >> "'a\xaa\ua000\U00011000'" >> >>> print(s1r) # required in programs >> Traceback (most recent call last): >> File "", line 1, in >> print(s1r) >> File "C:\Programs\Python34\lib\idlelib\PyShell.py", line 1347, in write >> return self.shell.write(s, self.tags) >> UnicodeEncodeError: 'UCS-2' codec can't encode characters in position 4-4: Non-BMP character not supported in Tk >> >> Printing s1 or sa directly, by either means, gives the same error. (Since '>>> expr' is supposed to be the same as 'print(expr)' the above difference puzzles me.) >> >> Even if repr always worked as it does for '\U0001a000', there would still be the problem of the added quotes. I therefore proposed the addition of a new str method, such as 's.bmp()', that returns s with all non-BMP chars, and only such chars, expanded. Since strings (in CPython) are internally marked by 'kind', the method would just return s when no expansion is needed. I presume it could otherwise re-use the expansion code already in repr. >> >> Aside from tkinter programmers in general, this issue bites Idle in at least two ways. Internally, filenames can contain non-BMP chars and Idle displays them in 3 places. >> See http://bugs.python.org/issue23672. >> Externally, Idle users sometimes want to print strings with non-BMP chars. I believe the automatic use of .bmp() with console prints could be user selectable. There have been issues about this on both our tracker and StackOverflow. >> >> I believe that the use of non-BMP chars is becoming more common and can no longer be simply dismissed as too rare to worry about. Telling Windows users that they are better off than if they use python directly, with the windows console, does not solve the inability to print any Python string. This proposal would. >> >> -- >> Terry Jan Reedy >> >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ From victor.stinner at gmail.com Mon Mar 16 11:55:03 2015 From: victor.stinner at gmail.com (Victor Stinner) Date: Mon, 16 Mar 2015 11:55:03 +0100 Subject: [Python-ideas] About people's accounts being deactivated due to bounces In-Reply-To: References: Message-ID: Hi, Be careful: recently, many emails sent to python-ideas and core-mentorship were sorted as spam in my Gmail account. It may be related to the recent upgrade of Mailman of these mailing lists. Victor 2015-03-13 13:27 GMT+01:00 Brett Cannon : > We're trying to figure out what's going on. Just re-enable your account if > you got a notice that it was temporarily disabled. > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ From random832 at fastmail.us Mon Mar 16 14:55:59 2015 From: random832 at fastmail.us (random832 at fastmail.us) Date: Mon, 16 Mar 2015 09:55:59 -0400 Subject: [Python-ideas] Add str.bmp() to only expand non-BMP chars, for tkinter use In-Reply-To: References: Message-ID: <1426514159.1203554.240988925.5344514A@webmail.messagingengine.com> On Sun, Mar 15, 2015, at 19:09, Terry Reedy wrote: > Tk widgets ... support the full BMP subset of unicode. Is it actually true that Tk widgets cannot support non-BMP characters? On my Python 2.7 on Windows (which is a narrow build, and therefore is passing in a UTF-16 string containing surrogates), it appears to work just fine. The character itself appears as a box, because there is no font support, but it is one box, not two. It seems like on the versions where people are getting the error message, Tkinter is arbitrarily (i.e. not based on a real limitation in Tk) refusing to convert extended characters to surrogates. From techtonik at gmail.com Mon Mar 16 13:12:44 2015 From: techtonik at gmail.com (anatoly techtonik) Date: Mon, 16 Mar 2015 15:12:44 +0300 Subject: [Python-ideas] math.tau Message-ID: >>> import math >>> math.tau Traceback (most recent call last): File "", line 1, in AttributeError: 'module' object has no attribute 'tau' >>> math.pi * 2 6.283185307179586 https://www.youtube.com/watch?v=2hhjsSN-AiU =) From techtonik at gmail.com Mon Mar 16 14:09:36 2015 From: techtonik at gmail.com (anatoly techtonik) Date: Mon, 16 Mar 2015 16:09:36 +0300 Subject: [Python-ideas] .from and .to instead of .encode and .decode In-Reply-To: References: Message-ID: On Sun, Mar 8, 2015 at 4:07 PM, Giampaolo Rodola' wrote: > On Sat, Mar 7, 2015 at 2:42 PM, Luciano Ramalho wrote: >> >> On Sat, Mar 7, 2015 at 8:41 AM, Chris Angelico wrote: >> > If it says "decode", the result is a Unicode string. If it says >> > "encode", the result is bytes. I'm not sure what is difficult here. >> >> Yep. When I teach, I use this mnemonic, which I can now quote from my >> book [1] ;-) >> >> [TIP] >> ==== >> If you need a memory aid to distinguish `.decode()` from `.encode()`, >> convince yourself that a Unicode `str` contains "human" text, while >> byte sequences can be cryptic machine core dumps. Therefore, it makes >> sense that we *decode* `bytes` to `str` to get human readable text, >> and we *encode* text to `bytes` for storage or transmission. >> ==== > > > This is a great advice (and yes, I also often get confused by the two). I use the analogy of "encoding abstract numbers into specific bytes" myself. It is still hard to review the source, which may use it wrong. db.get(query).from('utf-8') gives me a clear indication that a person expects that db contents is in 'utf-8'. db.get(query).decode('utf-8') gives me a feeling that query result should be decoded from whatever format it is in to 'utf-8'. So if Python supported .from and .to, the part about encoding wouldn't feel like math. From techtonik at gmail.com Mon Mar 16 14:24:05 2015 From: techtonik at gmail.com (anatoly techtonik) Date: Mon, 16 Mar 2015 16:24:05 +0300 Subject: [Python-ideas] Rewriting the build system (part 2) In-Reply-To: References: Message-ID: On Fri, Mar 13, 2015 at 10:46 PM, Wes Turner wrote: > The subject line mentions "Part 2". Is there a link to "Part 1"? What is the > rationale for changing? > > Make, autoconf, and automake are already widely ported/portable. "Autotools has reached the Chandrasekhar mass limit of software and become a black hole of suck that consumes endless hours of bright peoples? time without any good result. It is well past time for it to die." (c) "Autotools must die", ESR. http://esr.ibiblio.org/?p=1877 I couldn't sum up this better. It looks even worse from Windows machine. > What are the criteria for evaluating alternatives? Python 3 support and beautiful output to console? =) From techtonik at gmail.com Mon Mar 16 14:43:40 2015 From: techtonik at gmail.com (anatoly techtonik) Date: Mon, 16 Mar 2015 16:43:40 +0300 Subject: [Python-ideas] CPython needs to revamp/replace it's build system && Cross-compiling Python sucks In-Reply-To: References: Message-ID: On Fri, Mar 13, 2015 at 10:47 PM, Devin Jeanpierre wrote: > On Mon, Mar 9, 2015 at 10:40 AM, Ryan Gonzalez wrote: >> I've been trying to work on Issue 23496. And want to tear my hair out. I've >> noticed a few problems with Python's current build system: >> >> - There is no separation of the host C compiler and the target C compiler. >> When you give configure a C compiler for cross-compiling, make will happily >> try to build Parser/pgen with the cross-compiler. And try to run the >> generated executable. And fail. I know you can use `make touch` to get >> around this, but it still sucks. > > I just want to echo this. I had really horrible problems trying to get > Python working with emscripten a year ago, because of this and other > problems. (Gave up.) It turned out emscripten was a bigger cause for > complaint, but, still. There is an awesome SCons extension called Parts, but is not integrated back due to a lack of time and/or funding. Here is the lightning talk: https://www.youtube.com/watch?v=Ft6-r_vksDI#t=1188 http://parts.stage.tigris.org/ -- anatoly t. From bcannon at gmail.com Mon Mar 16 16:11:09 2015 From: bcannon at gmail.com (Brett Cannon) Date: Mon, 16 Mar 2015 15:11:09 +0000 Subject: [Python-ideas] Warn about comparing bytes to int for `python3 -b` Message-ID: One of the rather subtle issues with writing Python 2/3 code is that indexing on bytes in Python 2 returns a length-1 bytes object while in Python 3 it returns an int. Because ==/!= always returns True/False it can be a very subtle failure and tough to track down. What do people think of extending -b/-bb in Python 3 to warn when performing equality between an int and a bytes object of any length? I don't want to restrict to length-1 bytes objects because people may be doing comparisons where the result can be length-1 or any other length and thus would still have a subtle bug to pick up. Do people think this would raise a ton of false-positives? Would people find it useful? -------------- next part -------------- An HTML attachment was scrubbed... URL: From victor.stinner at gmail.com Mon Mar 16 16:15:01 2015 From: victor.stinner at gmail.com (Victor Stinner) Date: Mon, 16 Mar 2015 16:15:01 +0100 Subject: [Python-ideas] math.tau In-Reply-To: References: Message-ID: Hi Anatoly, It looks like you forgot to write the body of your email, there is no text, only code? 2015-03-16 13:12 GMT+01:00 anatoly techtonik : >>>> import math >>>> math.tau > Traceback (most recent call last): > File "", line 1, in > AttributeError: 'module' object has no attribute 'tau' >>>> math.pi * 2 > 6.283185307179586 Once a year, someone asks for math.tau and then disappears. I don't think that there is a real need to add math.tau. It's really trivial to compute such constant in your own application or in the Python interpreter. They are much more math constants, it's not useful to add more constants to the math modules. We added recently math.inf and math.nan which are float("int") and float("nan")! Even if I'm not convinced that it was useful to add them :-p http://bugs.python.org/issue23185 (see also http://bugs.python.org/issue23229 for cmath) The previous attempt to add math.tau was rejected: http://bugs.python.org/issue12345 Please read the issue for the rationale. Victor From guido at python.org Mon Mar 16 16:16:22 2015 From: guido at python.org (Guido van Rossum) Date: Mon, 16 Mar 2015 08:16:22 -0700 Subject: [Python-ideas] Warn about comparing bytes to int for `python3 -b` In-Reply-To: References: Message-ID: As long as it's part of -b/-bb this sounds like a useful (though small) bit of help for people in the last throes of porting a big package to PY3. As for how many false positives it will trigger, who knows? The most likely case would be if people use dicts whose keys can be bytestrings or ints -- I know that's a popular hobby when it comes to str/int, but I don't know if it's also common with bytes/int. I guess the only way to find out is to build and release it. On Mon, Mar 16, 2015 at 8:11 AM, Brett Cannon wrote: > One of the rather subtle issues with writing Python 2/3 code is that > indexing on bytes in Python 2 returns a length-1 bytes object while in > Python 3 it returns an int. Because ==/!= always returns True/False it can > be a very subtle failure and tough to track down. > > What do people think of extending -b/-bb in Python 3 to warn when > performing equality between an int and a bytes object of any length? I > don't want to restrict to length-1 bytes objects because people may be > doing comparisons where the result can be length-1 or any other length and > thus would still have a subtle bug to pick up. Do people think this would > raise a ton of false-positives? Would people find it useful? > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From victor.stinner at gmail.com Mon Mar 16 16:21:57 2015 From: victor.stinner at gmail.com (Victor Stinner) Date: Mon, 16 Mar 2015 16:21:57 +0100 Subject: [Python-ideas] Warn about comparing bytes to int for `python3 -b` In-Reply-To: References: Message-ID: Hi, 2015-03-16 16:11 GMT+01:00 Brett Cannon : > One of the rather subtle issues with writing Python 2/3 code is that > indexing on bytes in Python 2 returns a length-1 bytes object while in > Python 3 it returns an int. Because ==/!= always returns True/False it can > be a very subtle failure and tough to track down. I worked on such patch in the past, but I lost it :-) I can help to rewrite it if needed. So yes, it *is* very useful to port a large Python 2 project to Python 3. For example, you may not be able to run your application with Python 3 because a third party library cannot be imported on Python 3, so it blocks the whole work on porting an application to Python 3. Until the module is ported, you may want to prepare the port. Checking bytes==str just by reading the source code is difficult. Other issues which can only be "seen" at runtime when running an application on Python 3 : - "x > 0" with x=None => TypeError is raised in Python 3 - x / 8 where x is an int => becomes a float in Python 3, it's hard to detect this issue in Python 2 just by reading the source code :-/ > What do people think of extending -b/-bb in Python 3 to warn when performing > equality between an int and a bytes object of any length? I don't want to > restrict to length-1 bytes objects because people may be doing comparisons > where the result can be length-1 or any other length and thus would still > have a subtle bug to pick up. Do people think this would raise a ton of > false-positives? Would people find it useful? First ensure that the stdlib doesn't raise any BytesWarning exception. For example, os.get_exec_path() has to modify warnings filters temporary in Python 3 :-/ # {b'PATH': ...}.get('PATH') and {'PATH': ...}.get(b'PATH') emit a # BytesWarning when using python -b or python -bb: ignore the warning Victor From chris.barker at noaa.gov Mon Mar 16 17:27:14 2015 From: chris.barker at noaa.gov (Chris Barker) Date: Mon, 16 Mar 2015 09:27:14 -0700 Subject: [Python-ideas] Rewriting the build system (part 2) In-Reply-To: References: Message-ID: > > >> That written in Python, so Python would require Python to build which > >> would require Python to build which would require... > A bit more here: It seems there are two cases: 1) Development on and for a common, mature platform: In this case finding a functional python with which to run the build tool sis trivial. We're all set, no need to exclude python-based tools. 2) Development for / porting to a new and/or obscure platform: In this case, then there is no existing python with which to run the build tools. And there are two ways you might go about this (in general): a) cross-compiling -- run the compiler on a existing mature platform, and generate a compiled python for the new platform. IN this case, the build tools only need to run on the development platform, so we are back to (1), and all set. Note that the hard part of cross-compiling is still done by the C compiler, not really the build/configuration tools, The build tools need to be able to set up the cross-compile -- but presumable a new tool is chosen be cause it's easier to do things like that with it... b) Native compiling -- compiling python on the new platform. This is the tricky one. In this case, the entire goal is to get python building/running on the new platform. Clearly you can't just use the nifty python-based build system to do that. But as that's the goal, you'll still need to do the tricky parts anyway. So you could cross-compile the first version -- with maybe as few modules as possible to get started. Another option is a build system that needs python for the configuration stage, but not the compilation stage -- kind of like autotools: autotools is used in a three-step process: i) run autoconf, etc, to generate a configure script -- this is often distributed with the source tarball, so that end users don't need the autoconf tools ii) run the configure script to generate Makefiles, etc. configured for the platform at hand. iii) run make to do the actual build. Usually the last two steps are run natively -- but it's not too much of a stretch to run i and ii on a known build machine, and then only ned to run make on the final target. That would require the build tools to have a cross-configure ability. I have no idea if any of the known tools can do this, but if so, it would be pretty usable. And also worth keeping in mind that truly new platforms are pretty rare -- this kind of bootstrapping would not have to be done often --not a killer if it takes some hand-work. But the grain of salt : I have neer and am unlikely ever, to try to port a significant project to a new platform. Folks that actually need to do this should make these decisions. Will this affect the commands required to build Python from source, or > is all this just for actual development? > well, it may or may not be both: If the system supported a three-step process I outlined above, then makefiles (or the like) could be shipped for the common platforms, though probably easier to run the configure step on the final target. But the vast majority of final targets do have a python ready to run the configure/build machinery. someone > who has little knowledge of Python's internals, and just wants to > build the latest version (either from hg or from a source tarball). > That's the assumption I'm following -- the user wants the latest version -- so probably already has easy access to an older version with which to run the build tools... But in the second case, it'd be much safer to minimize the > requirements. If you can untar the source and just "./configure; make; > sudo make install" your way to having the latest Python, that's a LOT > easier than I've just been doing trying to get my Dad onto a newer > Mozilla Thunderbird. (I don't understand what it's doing all along > there, but it pulls a whole bunch of separate repositories when I say > update, and there's a separate bootstrap step, and ... I dunno.) > requiring _some_, not very recent, python is not exactly """a whole bunch of separate repositories when I say update, and there's a separate bootstrap step""" If there is an already build python (like OS-X, any general purpose linux distro, Windows if you don't mind point and clicking the python.org installer, etc, then it's not really a "bootstrap". In short -- using a python-based build system would make porting to a brand-new platform trickier -- but not really affect most other building. -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.barker at noaa.gov Mon Mar 16 17:41:05 2015 From: chris.barker at noaa.gov (Chris Barker) Date: Mon, 16 Mar 2015 09:41:05 -0700 Subject: [Python-ideas] .from and .to instead of .encode and .decode In-Reply-To: References: Message-ID: On Mon, Mar 16, 2015 at 6:09 AM, anatoly techtonik wrote: > db.get(query).from('utf-8') gives me a clear indication that a person > expects > that db contents is in 'utf-8'. > > db.get(query).decode('utf-8') gives me a feeling that query result should > be > decoded from whatever format it is in to 'utf-8'. > In teaching this, yes, newbies do get confused about what "decode" vs. "encode" mean. The fact that py2 strings have BOTH methods does confuse the heck out of it! -- but that problem is solved in py3. And in this case "from" might be a bit more obvious to a newbie. (though only might -- hard to tell without usability testing...) But "decode" and "encode" are widely used terms that clearly specify what's going on -- I think that even with teh "from", it's clear to you because you already understand a bit about the concept of unicode and the distiction between a string object and encoded bytes. And once you get that, then you kind know what "encode" and "decode" mean. Hopefully, folks will get that "decoded to utf-8" is meaningless... I suppose we could have used "decode_from" and "encode_to" in the first place to be really clear, but I don't think it's worth making that change now. -CHB > So if Python supported .from and .to, the part about encoding wouldn't feel > like math. > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Mon Mar 16 17:50:06 2015 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 17 Mar 2015 03:50:06 +1100 Subject: [Python-ideas] Rewriting the build system (part 2) In-Reply-To: References: Message-ID: On Tue, Mar 17, 2015 at 3:27 AM, Chris Barker wrote: >> someone >> who has little knowledge of Python's internals, and just wants to >> build the latest version (either from hg or from a source tarball). > > > That's the assumption I'm following -- the user wants the latest version -- > so probably already has easy access to an older version with which to run > the build tools... The tricky part here is defining the oldest supported version. "To build Python X.Y, you must already have Python ??? or later installed" - how do you pick the minimum? The older that minimum, the more warped the build code has to be (especially if you want to support 2.x as well as 3.x - imagine trying to make it possible to use the system Python on RHEL, as well as any 3.[1-4]), but go too far the other way and you end up with a multi-step build process ("first build Python 3.4, then use that to build Python 3.7") which is just going to be a pain. >> But in the second case, it'd be much safer to minimize the >> requirements. If you can untar the source and just "./configure; make; >> sudo make install" your way to having the latest Python, that's a LOT >> easier than I've just been doing trying to get my Dad onto a newer >> Mozilla Thunderbird. (I don't understand what it's doing all along >> there, but it pulls a whole bunch of separate repositories when I say >> update, and there's a separate bootstrap step, and ... I dunno.) > > > requiring _some_, not very recent, python is not exactly > > """a whole bunch of separate repositories when I say update, and there's a > separate bootstrap step""" > > If there is an already build python (like OS-X, any general purpose linux > distro, Windows if you don't mind point and clicking the python.org > installer, etc, then it's not really a "bootstrap". No, that's not what I'm talking about. I'm talking about trying to build Thunderbird and having to do a bunch of steps where I run this program, then run that, then let it chug its way through stuff, and then eventually it'll start building. Without detailed knowledge of the build internals I have no way of knowing what steps I can skip, so every time I want to test the build again, I have to go through them all. It took three parts of forever to figure out what was going on. I much prefer projects that ship with the standard "./configure; make" pattern. I don't care if the configure script is built by some arcane incantation, as long as it's checked into source control (or at very least is in the downloadable source tarball for any particular version). If I want to build your program, I don't want to have to hunt down your notes file, read it through in detail, and jump through a dozen hoops; I just want to point my compiler at your source, and then go make myself a hot chocolate while it does its work. ChrisA From rosuav at gmail.com Mon Mar 16 17:55:00 2015 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 17 Mar 2015 03:55:00 +1100 Subject: [Python-ideas] .from and .to instead of .encode and .decode In-Reply-To: References: Message-ID: On Tue, Mar 17, 2015 at 3:41 AM, Chris Barker wrote: > I suppose we could have used "decode_from" and "encode_to" in the first > place to be really clear, but I don't think it's worth making that change > now. I don't think it'd help much. It'd just be noise words, at best. What you're doing is encoding or decoding using a particular scheme. What scheme? That's the parameter. You're encoding to bytes and decoding to text; and in order to perform that transformation, you use a standard technique by the name of UTF-8. The result of "spam".encode("UTF-8") is not a UTF-8, it's a stream of bytes. So there'd be just as much confusion the other way around. *shrug* ChrisA From bcannon at gmail.com Mon Mar 16 18:06:44 2015 From: bcannon at gmail.com (Brett Cannon) Date: Mon, 16 Mar 2015 17:06:44 +0000 Subject: [Python-ideas] Warn about comparing bytes to int for `python3 -b` In-Reply-To: References: Message-ID: http://bugs.python.org/issue23681 to track this for Python 3.5. On Mon, Mar 16, 2015 at 11:16 AM Guido van Rossum wrote: > As long as it's part of -b/-bb this sounds like a useful (though small) > bit of help for people in the last throes of porting a big package to PY3. > As for how many false positives it will trigger, who knows? The most likely > case would be if people use dicts whose keys can be bytestrings or ints -- > I know that's a popular hobby when it comes to str/int, but I don't know if > it's also common with bytes/int. I guess the only way to find out is to > build and release it. > > On Mon, Mar 16, 2015 at 8:11 AM, Brett Cannon wrote: > >> One of the rather subtle issues with writing Python 2/3 code is that >> indexing on bytes in Python 2 returns a length-1 bytes object while in >> Python 3 it returns an int. Because ==/!= always returns True/False it can >> be a very subtle failure and tough to track down. >> >> What do people think of extending -b/-bb in Python 3 to warn when >> performing equality between an int and a bytes object of any length? I >> don't want to restrict to length-1 bytes objects because people may be >> doing comparisons where the result can be length-1 or any other length and >> thus would still have a subtle bug to pick up. Do people think this would >> raise a ton of false-positives? Would people find it useful? >> >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ >> > > > > -- > --Guido van Rossum (python.org/~guido) > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rymg19 at gmail.com Mon Mar 16 18:18:22 2015 From: rymg19 at gmail.com (Ryan Gonzalez) Date: Mon, 16 Mar 2015 12:18:22 -0500 Subject: [Python-ideas] Rewriting the build system (part 2) In-Reply-To: References: Message-ID: On Mon, Mar 16, 2015 at 11:50 AM, Chris Angelico wrote: > On Tue, Mar 17, 2015 at 3:27 AM, Chris Barker > wrote: > >> someone > >> who has little knowledge of Python's internals, and just wants to > >> build the latest version (either from hg or from a source tarball). > > > > > > That's the assumption I'm following -- the user wants the latest version > -- > > so probably already has easy access to an older version with which to run > > the build tools... > > The tricky part here is defining the oldest supported version. "To > build Python X.Y, you must already have Python ??? or later installed" > - how do you pick the minimum? The older that minimum, the more warped > the build code has to be (especially if you want to support 2.x as > well as 3.x - imagine trying to make it possible to use the system > Python on RHEL, as well as any 3.[1-4]), but go too far the other way > and you end up with a multi-step build process ("first build Python > 3.4, then use that to build Python 3.7") which is just going to be a > pain. > > Sounds like Go is now. It's a pain. > >> But in the second case, it'd be much safer to minimize the > >> requirements. If you can untar the source and just "./configure; make; > >> sudo make install" your way to having the latest Python, that's a LOT > >> easier than I've just been doing trying to get my Dad onto a newer > >> Mozilla Thunderbird. (I don't understand what it's doing all along > >> there, but it pulls a whole bunch of separate repositories when I say > >> update, and there's a separate bootstrap step, and ... I dunno.) > > > > > > requiring _some_, not very recent, python is not exactly > > > > """a whole bunch of separate repositories when I say update, and > there's a > > separate bootstrap step""" > > > > If there is an already build python (like OS-X, any general purpose linux > > distro, Windows if you don't mind point and clicking the python.org > > installer, etc, then it's not really a "bootstrap". > > No, that's not what I'm talking about. I'm talking about trying to > build Thunderbird and having to do a bunch of steps where I run this > program, then run that, then let it chug its way through stuff, and > then eventually it'll start building. Without detailed knowledge of > the build internals I have no way of knowing what steps I can skip, so > every time I want to test the build again, I have to go through them > all. It took three parts of forever to figure out what was going on. > > All of Mozilla's build systems need help. They have a build system genius who wrote Tup working for them, and they somehow still have 20,000 stupid mozbuild files with some weird, intensely slow make-based system that everyone complains about. > I much prefer projects that ship with the standard "./configure; make" > pattern. I don't care if the configure script is built by some arcane > incantation, as long as it's checked into source control (or at very > least is in the downloadable source tarball for any particular > version). If I want to build your program, I don't want to have to > hunt down your notes file, read it through in detail, and jump through > a dozen hoops; I just want to point my compiler at your source, and > then go make myself a hot chocolate while it does its work. > > Most of that is bad documentation. If there's no configure script but a file labeled INSTALLING, then that should contain simple instructions to build. I hate most tools that ship with configure scripts because 70% of the time some stupid test does some stupid thing wrong and gives the worst errors, 29% of the time the configure system doesn't pick up my options correctly, and 1% of the time it actually *works*. SCons also constantly gives me trouble. Of course, you could always do it the plan9port way: ryan at DevPC-LX:~/stuff/plan9port$ ./configure read the README file. ryan at DevPC-LX:~/stuff/plan9port$ > ChrisA > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- Ryan [ERROR]: Your autotools build scripts are 200 lines longer than your program. Something?s wrong. http://kirbyfan64.github.io/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From bcannon at gmail.com Mon Mar 16 18:25:58 2015 From: bcannon at gmail.com (Brett Cannon) Date: Mon, 16 Mar 2015 17:25:58 +0000 Subject: [Python-ideas] Warn about comparing bytes to int for `python3 -b` In-Reply-To: References: Message-ID: On Mon, Mar 16, 2015 at 11:22 AM Victor Stinner wrote: > Hi, > > 2015-03-16 16:11 GMT+01:00 Brett Cannon : > > One of the rather subtle issues with writing Python 2/3 code is that > > indexing on bytes in Python 2 returns a length-1 bytes object while in > > Python 3 it returns an int. Because ==/!= always returns True/False it > can > > be a very subtle failure and tough to track down. > > I worked on such patch in the past, but I lost it :-) I can help to > rewrite it if needed. > I filed http://bugs.python.org/issue23681 if you want to help (although I don't expect the patch to be complicated; I might ask you for a code review though =). > > So yes, it *is* very useful to port a large Python 2 project to Python 3. > > For example, you may not be able to run your application with Python 3 > because a third party library cannot be imported on Python 3, so it > blocks the whole work on porting an application to Python 3. Until the > module is ported, you may want to prepare the port. Checking > bytes==str just by reading the source code is difficult. > > Other issues which can only be "seen" at runtime when running an > application on Python 3 : > > - "x > 0" with x=None => TypeError is raised in Python 3 > Testing can easily catch that *and* have a traceback to trace down where the problem originated from, so I'm not worried. > - x / 8 where x is an int => becomes a float in Python 3, it's hard to > detect this issue in Python 2 just by reading the source code :-/ > -Q in Python 2 and/or adding the division __future__ statement solves this one. Basically I'm just trying to plug the last few holes that can't be automated by tools like Modernize or Futurize *and* won't surface where the problem is easily during testing under Python 2 or 3 (e.g., a traceback during testing makes it easy to find unless you swallow the exception, in which case a warning won't help you either if you use -Werror). > > > What do people think of extending -b/-bb in Python 3 to warn when > performing > > equality between an int and a bytes object of any length? I don't want to > > restrict to length-1 bytes objects because people may be doing > comparisons > > where the result can be length-1 or any other length and thus would still > > have a subtle bug to pick up. Do people think this would raise a ton of > > false-positives? Would people find it useful? > > First ensure that the stdlib doesn't raise any BytesWarning exception. > Yep, I always test with -Werror whenever I muck with something that can raise a warning (side-effect from helping to write _warnings.c). -Brett > > For example, os.get_exec_path() has to modify warnings filters > temporary in Python 3 :-/ > > # {b'PATH': ...}.get('PATH') and {'PATH': ...}.get(b'PATH') emit a > # BytesWarning when using python -b or python -bb: ignore the warning > > Victor > -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.barker at noaa.gov Mon Mar 16 18:39:22 2015 From: chris.barker at noaa.gov (Chris Barker) Date: Mon, 16 Mar 2015 10:39:22 -0700 Subject: [Python-ideas] Rewriting the build system (part 2) In-Reply-To: References: Message-ID: On Mon, Mar 16, 2015 at 9:50 AM, Chris Angelico wrote: > The tricky part here is defining the oldest supported version. "To > build Python X.Y, you must already have Python ??? or later installed" > - how do you pick the minimum? The older that minimum, the more warped > the build code has to be sure -- but it seems what's on the table now is: must be buildable by an old version of C -- not even C99 which is nominally 16 years old... So yeah, the build system itself would need to support a "old" python -- though probably python2.7 would be OK at this point -- sorry, not py3 yet, but what can you do? > If there is an already build python (like OS-X, any general purpose linux > > distro, Windows if you don't mind point and clicking the python.org > > installer, etc, then it's not really a "bootstrap". > > No, that's not what I'm talking about. I'm talking about trying to > build Thunderbird and having to do a bunch of steps where I run this > program, then run that, then let it chug its way through stuff, and > then eventually it'll start building. It sounds like that build setup suck, yes -- but not sure why that's relevant. Clearly we don't want something that ugly. We want something that is both easy and robust to configure for new platfroms, etc, and easy on the end user to "jsut build" the source. All I"m proposing that is requireing a not-very-recent python as a first step is only a big hurdle for folks poritn got new platforms -- your use case would be, in INSTALL.txt: """ 1) You need at least python 2.7 to build this software. try: python --version if it starts up and reports a version greater than 2.7, then you are all set. If not, then It can probably be installed with your system package manager: apt_get install python or similar. On Windows, you can download an installer for Python2.7 from python.org 2) Once a python is working, you can build the this latest version from source with: ./buildme """ Of course, there may be options, etc, but that's the basic deal. And heck, if people REALLY want the ./configure && make && make install dance, then we could ship a little configure script that does nothing but call the python-based one... Of course, all this is predicated on there BEING some miraculous as-powerful-and-flexible-and-robust-as-autotools-but-much-easier-to-use python-based build system..... One more note: so adding a python dependency is a pain. But the current system requires separate stems for Windows and *nix-y systems, and who knows what for hypothetical systems that don't support autotools. So maybe a python-based build system that worked on all the common platforms, with an additional makefile that you have to hand-mangle to bootstrap a new platform isn't any more to maintain. -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From rymg19 at gmail.com Mon Mar 16 18:49:21 2015 From: rymg19 at gmail.com (Ryan Gonzalez) Date: Mon, 16 Mar 2015 12:49:21 -0500 Subject: [Python-ideas] Rewriting the build system (part 2) In-Reply-To: References: Message-ID: On Mon, Mar 16, 2015 at 12:39 PM, Chris Barker wrote: > On Mon, Mar 16, 2015 at 9:50 AM, Chris Angelico wrote: > >> The tricky part here is defining the oldest supported version. "To >> build Python X.Y, you must already have Python ??? or later installed" >> - how do you pick the minimum? The older that minimum, the more warped >> the build code has to be > > > sure -- but it seems what's on the table now is: must be buildable by an > old version of C -- not even C99 which is nominally 16 years old... > > So yeah, the build system itself would need to support a "old" python -- > though probably python2.7 would be OK at this point -- sorry, not py3 yet, > but what can you do? > > > If there is an already build python (like OS-X, any general purpose linux >> > distro, Windows if you don't mind point and clicking the python.org >> > installer, etc, then it's not really a "bootstrap". >> >> No, that's not what I'm talking about. I'm talking about trying to >> build Thunderbird and having to do a bunch of steps where I run this >> program, then run that, then let it chug its way through stuff, and >> then eventually it'll start building. > > > It sounds like that build setup suck, yes -- but not sure why that's > relevant. Clearly we don't want something that ugly. We want something that > is both easy and robust to configure for new platfroms, etc, and easy on > the end user to "jsut build" the source. All I"m proposing that is > requireing a not-very-recent python as a first step is only a big hurdle > for folks poritn got new platforms -- your use case would be, in > INSTALL.txt: > > """ > 1) You need at least python 2.7 to build this software. > > try: > > python --version > > if it starts up and reports a version greater than 2.7, then you are all > set. If not, then It can probably be installed with your system package > manager: > > apt_get install python > > or similar. On Windows, you can download an installer for Python2.7 from > python.org > > 2) Once a python is working, you can build the this latest version from > source with: > > ./buildme > """ > > Of course, there may be options, etc, but that's the basic deal. > > And heck, if people REALLY want the ./configure && make && make install dance, > then we could ship a little configure script that does nothing but call the > python-based one... > > Of course, all this is predicated on there BEING some miraculous > as-powerful-and-flexible-and-robust-as-autotools-but-much-easier-to-use > python-based build system..... > > One more note: so adding a python dependency is a pain. But the current > system requires separate stems for Windows and *nix-y systems, and who > knows what for hypothetical systems that don't support autotools. So maybe > a python-based build system that worked on all the common platforms, with > an additional makefile that you have to hand-mangle to bootstrap a new > platform isn't any more to maintain. > > The best build systems I know in Python are fbuild, Waf, and Meson. Meson generates Ninja scripts, and Ninja's in C++, so that's out of the question. I have used fbuild a LOT and can say that, after you get through the small amount of command-line boilerplate, it works very well. It's also very easy to extend with host/target compiler differences. Not sure about Waf, but I do know it does about anything you'd guess. I guess this narrows the result down to Boost.Build or a Python-based build system. > -CHB > > -- > > Christopher Barker, Ph.D. > Oceanographer > > Emergency Response Division > NOAA/NOS/OR&R (206) 526-6959 voice > 7600 Sand Point Way NE (206) 526-6329 fax > Seattle, WA 98115 (206) 526-6317 main reception > > Chris.Barker at noaa.gov > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- Ryan [ERROR]: Your autotools build scripts are 200 lines longer than your program. Something?s wrong. http://kirbyfan64.github.io/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From abarnert at yahoo.com Mon Mar 16 19:05:31 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Mon, 16 Mar 2015 11:05:31 -0700 Subject: [Python-ideas] Rewriting the build system (part 2) In-Reply-To: References: Message-ID: Something nobody's mentioned yet in this thread: The autotools stuff doesn't just handle different platforms, it also handles local differences (not every FooLinux x.y system has Tcl/Tk; if yours doesn't, you can still build Python, you just won't get tkinter) and user options (disabling features, using a specific OpenSSL instead of the older system one that would get autodetected, etc.). Autoconf makes this relatively easy (not that anything in autoconf is easy, but if you can write the OpenSSL detection, it's not significantly harder to write OpenSSL detection with a flag to override the path). Is the same thing true for other build configuration systems? Sent from my iPhone On Mar 16, 2015, at 9:27 AM, Chris Barker wrote: >> >> That written in Python, so Python would require Python to build which >> >> would require Python to build which would require... > > A bit more here: > > It seems there are two cases: > > 1) Development on and for a common, mature platform: > > In this case finding a functional python with which to run the build tool sis trivial. We're all set, no need to exclude python-based tools. > > 2) Development for / porting to a new and/or obscure platform: > > In this case, then there is no existing python with which to run the build tools. And there are two ways you might go about this (in general): > > a) cross-compiling -- run the compiler on a existing mature platform, and generate a compiled python for the new platform. IN this case, the build tools only need to run on the development platform, so we are back to (1), and all set. Note that the hard part of cross-compiling is still done by the C compiler, not really the build/configuration tools, The build tools need to be able to set up the cross-compile -- but presumable a new tool is chosen be cause it's easier to do things like that with it... > > b) Native compiling -- compiling python on the new platform. This is the tricky one. In this case, the entire goal is to get python building/running on the new platform. Clearly you can't just use the nifty python-based build system to do that. But as that's the goal, you'll still need to do the tricky parts anyway. So you could cross-compile the first version -- with maybe as few modules as possible to get started. Another option is a build system that needs python for the configuration stage, but not the compilation stage -- kind of like autotools: autotools is used in a three-step process: > > i) run autoconf, etc, to generate a configure script -- this is often distributed with the source tarball, so that end users don't need the autoconf tools > > ii) run the configure script to generate Makefiles, etc. configured for the platform at hand. > > iii) run make to do the actual build. > > Usually the last two steps are run natively -- but it's not too much of a stretch to run i and ii on a known build machine, and then only ned to run make on the final target. That would require the build tools to have a cross-configure ability. > > I have no idea if any of the known tools can do this, but if so, it would be pretty usable. > > And also worth keeping in mind that truly new platforms are pretty rare -- this kind of bootstrapping would not have to be done often --not a killer if it takes some hand-work. > > But the grain of salt : I have neer and am unlikely ever, to try to port a significant project to a new platform. Folks that actually need to do this should make these decisions. > >> Will this affect the commands required to build Python from source, or >> is all this just for actual development? > > well, it may or may not be both: > > If the system supported a three-step process I outlined above, then makefiles (or the like) could be shipped for the common platforms, though probably easier to run the configure step on the final target. But the vast majority of final targets do have a python ready to run the configure/build machinery. > >> someone >> who has little knowledge of Python's internals, and just wants to >> build the latest version (either from hg or from a source tarball). > > That's the assumption I'm following -- the user wants the latest version -- so probably already has easy access to an older version with which to run the build tools... > >> But in the second case, it'd be much safer to minimize the >> requirements. If you can untar the source and just "./configure; make; >> sudo make install" your way to having the latest Python, that's a LOT >> easier than I've just been doing trying to get my Dad onto a newer >> Mozilla Thunderbird. (I don't understand what it's doing all along >> there, but it pulls a whole bunch of separate repositories when I say >> update, and there's a separate bootstrap step, and ... I dunno.) > > requiring _some_, not very recent, python is not exactly > > """a whole bunch of separate repositories when I say update, and there's a separate bootstrap step""" > > If there is an already build python (like OS-X, any general purpose linux distro, Windows if you don't mind point and clicking the python.org installer, etc, then it's not really a "bootstrap". > > In short -- using a python-based build system would make porting to a brand-new platform trickier -- but not really affect most other building. > > -CHB > > > -- > > Christopher Barker, Ph.D. > Oceanographer > > Emergency Response Division > NOAA/NOS/OR&R (206) 526-6959 voice > 7600 Sand Point Way NE (206) 526-6329 fax > Seattle, WA 98115 (206) 526-6317 main reception > > Chris.Barker at noaa.gov > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefan at bytereef.org Mon Mar 16 19:33:23 2015 From: stefan at bytereef.org (Stefan Krah) Date: Mon, 16 Mar 2015 18:33:23 +0000 (UTC) Subject: [Python-ideas] Rewriting the build system (part 2) References: Message-ID: Ryan Gonzalez writes: > I hate most tools that ship with configure scripts because 70% of the time some stupid test does some stupid thing wrong and gives the worst errors, 29% of the time the configure system doesn't pick up my options correctly, and 1% of the time it actually *works*. That's just not true. Also, the small build issue that triggered this thread is already tracked here: http://bugs.python.org/issue22625 It seems more productive to me to fix that rather than rewrite the build system. Users don't have an unlimited tolerance for pointless churn. Stefan Krah From rymg19 at gmail.com Mon Mar 16 20:19:36 2015 From: rymg19 at gmail.com (Ryan Gonzalez) Date: Mon, 16 Mar 2015 14:19:36 -0500 Subject: [Python-ideas] Rewriting the build system (part 2) In-Reply-To: References: Message-ID: Most due let you do that easily. For instance, fbuild lets you just trap the raised error and move on. On Mon, Mar 16, 2015 at 1:05 PM, Andrew Barnert < abarnert at yahoo.com.dmarc.invalid> wrote: > Something nobody's mentioned yet in this thread: > > The autotools stuff doesn't just handle different platforms, it also > handles local differences (not every FooLinux x.y system has Tcl/Tk; if > yours doesn't, you can still build Python, you just won't get tkinter) and > user options (disabling features, using a specific OpenSSL instead of the > older system one that would get autodetected, etc.). > > Autoconf makes this relatively easy (not that anything in autoconf is > easy, but if you can write the OpenSSL detection, it's not significantly > harder to write OpenSSL detection with a flag to override the path). Is the > same thing true for other build configuration systems? > > > Sent from my iPhone > > On Mar 16, 2015, at 9:27 AM, Chris Barker wrote: > > >> That written in Python, so Python would require Python to build which >> >> would require Python to build which would require... >> > > A bit more here: > > It seems there are two cases: > > 1) Development on and for a common, mature platform: > > In this case finding a functional python with which to run the build tool > sis trivial. We're all set, no need to exclude python-based tools. > > 2) Development for / porting to a new and/or obscure platform: > > In this case, then there is no existing python with which to run the build > tools. And there are two ways you might go about this (in general): > > a) cross-compiling -- run the compiler on a existing mature platform, > and generate a compiled python for the new platform. IN this case, the > build tools only need to run on the development platform, so we are back to > (1), and all set. Note that the hard part of cross-compiling is still done > by the C compiler, not really the build/configuration tools, The build > tools need to be able to set up the cross-compile -- but presumable a new > tool is chosen be cause it's easier to do things like that with it... > > b) Native compiling -- compiling python on the new platform. This is the > tricky one. In this case, the entire goal is to get python building/running > on the new platform. Clearly you can't just use the nifty python-based > build system to do that. But as that's the goal, you'll still need to do > the tricky parts anyway. So you could cross-compile the first version -- > with maybe as few modules as possible to get started. Another option is a > build system that needs python for the configuration stage, but not the > compilation stage -- kind of like autotools: autotools is used in a > three-step process: > > i) run autoconf, etc, to generate a configure script -- this is often > distributed with the source tarball, so that end users don't need the > autoconf tools > > ii) run the configure script to generate Makefiles, etc. configured for > the platform at hand. > > iii) run make to do the actual build. > > Usually the last two steps are run natively -- but it's not too much of a > stretch to run i and ii on a known build machine, and then only ned to run > make on the final target. That would require the build tools to have a > cross-configure ability. > > I have no idea if any of the known tools can do this, but if so, it would > be pretty usable. > > And also worth keeping in mind that truly new platforms are pretty rare -- > this kind of bootstrapping would not have to be done often --not a killer > if it takes some hand-work. > > But the grain of salt : I have neer and am unlikely ever, to try to port a > significant project to a new platform. Folks that actually need to do this > should make these decisions. > > Will this affect the commands required to build Python from source, or >> is all this just for actual development? >> > > well, it may or may not be both: > > If the system supported a three-step process I outlined above, then > makefiles (or the like) could be shipped for the common platforms, though > probably easier to run the configure step on the final target. But the vast > majority of final targets do have a python ready to run the configure/build > machinery. > > someone >> who has little knowledge of Python's internals, and just wants to >> build the latest version (either from hg or from a source tarball). >> > > That's the assumption I'm following -- the user wants the latest version > -- so probably already has easy access to an older version with which to > run the build tools... > > But in the second case, it'd be much safer to minimize the >> > requirements. If you can untar the source and just "./configure; make; >> sudo make install" your way to having the latest Python, that's a LOT >> easier than I've just been doing trying to get my Dad onto a newer >> Mozilla Thunderbird. (I don't understand what it's doing all along >> there, but it pulls a whole bunch of separate repositories when I say >> update, and there's a separate bootstrap step, and ... I dunno.) >> > > requiring _some_, not very recent, python is not exactly > > """a whole bunch of separate repositories when I say update, and there's > a separate bootstrap step""" > > If there is an already build python (like OS-X, any general purpose linux > distro, Windows if you don't mind point and clicking the python.org > installer, etc, then it's not really a "bootstrap". > > In short -- using a python-based build system would make porting to a > brand-new platform trickier -- but not really affect most other building. > > -CHB > > > -- > > Christopher Barker, Ph.D. > Oceanographer > > Emergency Response Division > NOAA/NOS/OR&R (206) 526-6959 voice > 7600 Sand Point Way NE (206) 526-6329 fax > Seattle, WA 98115 (206) 526-6317 main reception > > Chris.Barker at noaa.gov > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- Ryan [ERROR]: Your autotools build scripts are 200 lines longer than your program. Something?s wrong. http://kirbyfan64.github.io/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From rymg19 at gmail.com Mon Mar 16 20:24:42 2015 From: rymg19 at gmail.com (Ryan Gonzalez) Date: Mon, 16 Mar 2015 14:24:42 -0500 Subject: [Python-ideas] Rewriting the build system (part 2) In-Reply-To: References: Message-ID: On Mon, Mar 16, 2015 at 1:33 PM, Stefan Krah wrote: > Ryan Gonzalez writes: > > I hate most tools that ship with configure scripts because 70% of the > time > some stupid test does some stupid thing wrong and gives the worst errors, > 29% of the time the configure system doesn't pick up my options correctly, > and 1% of the time it actually *works*. > > That's just not true. Also, the small build issue that triggered > this thread is already tracked here: > > http://bugs.python.org/issue22625 > > You've evidently had better luck than I usually have. > > It seems more productive to me to fix that rather than rewrite the > build system. Users don't have an unlimited tolerance for pointless > churn. > > Lots and lots of projects have moved from autotools to about everything else. It's slow, annoying, and generates huge files (the 5k-line configure.ac generates a 17k-line configure script). And largely pointless, since nowadays few like it anymore. > > > Stefan Krah > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- Ryan [ERROR]: Your autotools build scripts are 200 lines longer than your program. Something?s wrong. http://kirbyfan64.github.io/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From breamoreboy at yahoo.co.uk Mon Mar 16 21:04:59 2015 From: breamoreboy at yahoo.co.uk (Mark Lawrence) Date: Mon, 16 Mar 2015 20:04:59 +0000 Subject: [Python-ideas] Rewriting the build system (part 2) In-Reply-To: References: Message-ID: On 16/03/2015 19:24, Ryan Gonzalez wrote: > On Mon, Mar 16, 2015 at 1:33 PM, Stefan Krah > > wrote: > > Ryan Gonzalez writes: > > I hate most tools that ship with configure scripts because 70% of the time > some stupid test does some stupid thing wrong and gives the worst > errors, > 29% of the time the configure system doesn't pick up my options > correctly, > and 1% of the time it actually *works*. > > That's just not true. Also, the small build issue that triggered > this thread is already tracked here: > > http://bugs.python.org/issue22625 > > > You've evidently had better luck than I usually have. Luck doesn't enter in to it. More likely a bad workan always blames his tools? > > > It seems more productive to me to fix that rather than rewrite the > build system. Users don't have an unlimited tolerance for pointless > churn. > > > Lots and lots of projects have moved from autotools to about everything > else. > > It's slow, annoying, and generates huge files (the 5k-line configure.ac > generates a 17k-line configure script). And > largely pointless, since nowadays few like it anymore. > Perhaps everybody should move to MSVC, that way at least everything would be consistent :) -- My fellow Pythonistas, ask not what our language can do for you, ask what you can do for our language. Mark Lawrence From chris.barker at noaa.gov Mon Mar 16 17:31:32 2015 From: chris.barker at noaa.gov (Chris Barker) Date: Mon, 16 Mar 2015 09:31:32 -0700 Subject: [Python-ideas] Rewriting the build system (part 2) In-Reply-To: References: Message-ID: On Mon, Mar 16, 2015 at 6:24 AM, anatoly techtonik wrote: > "Autotools has reached the Chandrasekhar mass limit of software and > become a black hole of suck that consumes endless hours of bright > peoples? time without any good result. It is well past time for it to > die." (c) "Autotools must die", ESR. http://esr.ibiblio.org/?p=1877 > > I couldn't sum up this better. It looks even worse from Windows machine. > Is it even used for building Python on Windows (for the "normal" python.org builds)? But to miss-paraphrase Winston Churchill [1]: "Autotools is the worst build system except for all the others." Is there really something better out there? -CHB [1] http://www.goodreads.com/quotes/267224-democracy-is-the-worst-form-of-government-except-for-all -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From rymg19 at gmail.com Mon Mar 16 21:30:47 2015 From: rymg19 at gmail.com (Ryan Gonzalez) Date: Mon, 16 Mar 2015 15:30:47 -0500 Subject: [Python-ideas] Rewriting the build system (part 2) In-Reply-To: References: Message-ID: On Mon, Mar 16, 2015 at 3:04 PM, Mark Lawrence wrote: > On 16/03/2015 19:24, Ryan Gonzalez wrote: > >> On Mon, Mar 16, 2015 at 1:33 PM, Stefan Krah >> > > wrote: >> >> Ryan Gonzalez writes: >> > I hate most tools that ship with configure scripts because 70% of >> the time >> some stupid test does some stupid thing wrong and gives the worst >> errors, >> 29% of the time the configure system doesn't pick up my options >> correctly, >> and 1% of the time it actually *works*. >> >> That's just not true. Also, the small build issue that triggered >> this thread is already tracked here: >> >> http://bugs.python.org/issue22625 >> >> >> You've evidently had better luck than I usually have. >> > > Luck doesn't enter in to it. More likely a bad workan always blames his > tools? > > :P I never have that much trouble with other build tools, or even hand-written makefiles/configure scripts. Just autotools. > >> >> It seems more productive to me to fix that rather than rewrite the >> build system. Users don't have an unlimited tolerance for pointless >> churn. >> >> >> Lots and lots of projects have moved from autotools to about everything >> else. >> >> It's slow, annoying, and generates huge files (the 5k-line configure.ac >> generates a 17k-line configure script). And >> largely pointless, since nowadays few like it anymore. >> >> > Perhaps everybody should move to MSVC, that way at least everything would > be consistent :) > > And non-standards-compliant. :) > -- > My fellow Pythonistas, ask not what our language can do for you, ask > what you can do for our language. > > Mark Lawrence > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- Ryan [ERROR]: Your autotools build scripts are 200 lines longer than your program. Something?s wrong. http://kirbyfan64.github.io/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From rymg19 at gmail.com Mon Mar 16 21:33:11 2015 From: rymg19 at gmail.com (Ryan Gonzalez) Date: Mon, 16 Mar 2015 15:33:11 -0500 Subject: [Python-ideas] Rewriting the build system (part 2) In-Reply-To: References: Message-ID: On Mon, Mar 16, 2015 at 11:31 AM, Chris Barker wrote: > On Mon, Mar 16, 2015 at 6:24 AM, anatoly techtonik > wrote: > >> "Autotools has reached the Chandrasekhar mass limit of software and >> become a black hole of suck that consumes endless hours of bright >> peoples? time without any good result. It is well past time for it to >> die." (c) "Autotools must die", ESR. http://esr.ibiblio.org/?p=1877 >> >> I couldn't sum up this better. It looks even worse from Windows machine. >> > > Is it even used for building Python on Windows (for the "normal" > python.org builds)? > > No. Some random batch file is used. That's all I know. However, when building without MSVC/VS2008, you need MinGW, which means you'll end up using autotools. > But to miss-paraphrase Winston Churchill [1]: > > "Autotools is the worst build system except for all the others." > > Is there really something better out there? > > Fbuild, Waf, Meson, GN, Boost.Build, Tundra, Bam, ... > -CHB > > [1] > http://www.goodreads.com/quotes/267224-democracy-is-the-worst-form-of-government-except-for-all > > -- > > Christopher Barker, Ph.D. > Oceanographer > > Emergency Response Division > NOAA/NOS/OR&R (206) 526-6959 voice > 7600 Sand Point Way NE (206) 526-6329 fax > Seattle, WA 98115 (206) 526-6317 main reception > > Chris.Barker at noaa.gov > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- Ryan [ERROR]: Your autotools build scripts are 200 lines longer than your program. Something?s wrong. http://kirbyfan64.github.io/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.barker at noaa.gov Mon Mar 16 16:51:37 2015 From: chris.barker at noaa.gov (Chris Barker) Date: Mon, 16 Mar 2015 08:51:37 -0700 Subject: [Python-ideas] math.tau In-Reply-To: References: Message-ID: On Mon, Mar 16, 2015 at 8:15 AM, Victor Stinner wrote: > Once a year, someone asks for math.tau and then disappears. well, it's s small enough deal that overcoming the inertia is probably way too much work for someone to carry it through -- which doesn't mean it's not a good idea. > The previous attempt to add math.tau was rejected: > http://bugs.python.org/issue12345 well, toward teh end of that issue thread, we have fro Guido: """ Uh, not until I've seen a lot more evidence that people are habitually writing "TAU = 2 * math.pi" in their programs... """ then a bit of evidence about how common the use is -- but no commentary on that. I'd say "Habitually" is not well defined. The trick with "evidence" is that many people compute a 2*pi constant, but not many people call it "tau". I'm for one have some version of : twopi = 2*math.pi in a huge fraction of the code I use pi in -- though I'm can't say I'm consistent as to what I call it, even in my own code. So I'm going to equivocate: +1 because while trivial, it's also useful and really commonly used -1 because the name "tau" is unknown enough that even if it's there, a LOT of people will compute themselves anyway :-( -- which, as pointed out is not a big deal. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From nad at acm.org Mon Mar 16 23:45:10 2015 From: nad at acm.org (Ned Deily) Date: Mon, 16 Mar 2015 15:45:10 -0700 Subject: [Python-ideas] Rewriting the build system (part 2) References: Message-ID: In article , Stefan Krah wrote: > That's just not true. Also, the small build issue that triggered > this thread is already tracked here: > > http://bugs.python.org/issue22625 > > It seems more productive to me to fix that rather than rewrite the > build system. Users don't have an unlimited tolerance for pointless > churn. +10 How many different platforms and configurations on each platform do we explicitly or implicitly support today for current CPython 2 and CPython 3 releases? I don't know (and I should since I help release them) but it's clearly at least in the dozens. We do not currently have formal tests or test platforms (e.g. buildbots) for many of them like we should and it would be a monumental undertaking to try to migrate the current build system to something substantially different. It's fine to let off steam about frustrations with build systems but talking about it here is not gonna cause it to change. And it won't change unless someone (or, more likely, some big company) is willing to invest an enormous effort in people time and machine resources to do so. Stefan's suggestion is much more practical. Along with it, if someone is motivated, better documenting the current processes for cross-compilation and which pairs of build/target systems are supported would be a welcome and extremely useful improvement. Much of what is there today for cross-compilation has slipped in over the years with little discussion or documentation. It's easy to break because it's not always clear how it is supposed to work and because it isn't regularly tested. -- Ned Deily, nad at acm.org From chris.barker at noaa.gov Tue Mar 17 00:45:17 2015 From: chris.barker at noaa.gov (Chris Barker) Date: Mon, 16 Mar 2015 16:45:17 -0700 Subject: [Python-ideas] Rewriting the build system (part 2) In-Reply-To: References: Message-ID: On Mon, Mar 16, 2015 at 3:45 PM, Ned Deily wrote: > it would be a monumental undertaking to try to migrate the current > build system to something substantially different. It's fine to let off > steam about frustrations with build systems but talking about it here is > not gonna cause it to change. And it won't change unless someone (or, > more likely, some big company) is willing to invest an enormous effort > in people time and machine resources to do so. Stefan's suggestion is > much more practical. This doesn't seem to me to be a winner takes all proposition -- there is no reason someone couldn't set up a new build system for Python, get it all working nicely on their platform(s) of choice, persuade others how wonderful it is, get contributions, etc. If it turns out to really be better/easier etc. than autoconf, then , and only then, would it make any sense to talk about replacing the existing official build system. And even then, the two could live in parallel for quite some time... -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From nad at acm.org Tue Mar 17 01:54:34 2015 From: nad at acm.org (Ned Deily) Date: Mon, 16 Mar 2015 17:54:34 -0700 Subject: [Python-ideas] Rewriting the build system (part 2) References: Message-ID: In article , Chris Barker wrote: > This doesn't seem to me to be a winner takes all proposition -- there is no > reason someone couldn't set up a new build system for Python, get it all > working nicely on their platform(s) of choice, persuade others how > wonderful it is, get contributions, etc. > > If it turns out to really be better/easier etc. than autoconf, then , and > only then, would it make any sense to talk about replacing the existing > official build system. And even then, the two could live in parallel for > quite some time... Sorry, that's just not realistic. We already have two different build systems, one for Windows and one for all other platforms, and it's difficult enough to keep both of them running. Adding yet another system in parallel is not in the cards without a major infusion of resources. There are frequent minor changes to the current build systems as part of new Python features or bug fixes and those run the risk of breaking things today and too often do. Major changes to build systems impact most parts of the Python ecosphere, for example, development, automated testing, manual testing, release management, downstream distributors of CPython (many of them managing their own sets of local modifications to the current processes), etc etc. To be even able to evaluate the merits of a major build system migration, we would first need to understand what we support today, what are the actual shortcomings of what we have today, and what we want to support in the future. Then, given that, we'd need to decide what are the options and what are their costs. One prerequisite to that is having a much more comprehensive test build environment so we could make even substantial incremental changes with a reasonable degree of confidence that we're not breaking support of a defined set of platforms and configurations. We don't have that today. Working towards that would be a necessary and important step. Also, note that, in all this talk about autoconf, be aware the CPython build processes don't even make full use of current automake/autoconf features. Much of the current Python build process predates the addition of features to those tools. Just bringing the current configure scripts and Makefiles up-to-date to use all the current features would be a big task that would imply a big testing effort on resources we don't currently have. Also keep in mind that builds and installations are areas that few core CPython developers are all that interested in and with good reason. It's usually tedious and generally unrewarding work especially when operating in a vacuum. Let's first work towards properly identifying and testing current important environments (for example, from typical installed Python configurations, not build directories and, as noted before, the almost totally undocumented and tested world of cross-building) and then we can start incrementally improving things with some degree of confidence that we're not going to break things for us and our users. -- Ned Deily, nad at acm.org From stephen at xemacs.org Tue Mar 17 03:27:21 2015 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Tue, 17 Mar 2015 11:27:21 +0900 Subject: [Python-ideas] Rewriting the build system (part 2) In-Reply-To: References: Message-ID: <87wq2g73zq.fsf@uwakimon.sk.tsukuba.ac.jp> Ryan Gonzalez writes: > I have used fbuild a LOT Actually, I suspect that in the relevant sense, you haven't used it a lot. Have you used it for a single program on 1. more than 3 CPUs, AND 2. on each CPU, more than one (and preferably more than two) OSes, AND 3. for each CPU-OS combination, more than two different configurations of the program being built, AND 4. for each CPU-OS-configuration combination, at least 3 versions of the program source (preferably spanning a major version bump)? The number of programs/versions you have built on a single (or small selection of) CPU-OS combinations is irrelevant to the discussion. The problem autotools helps to solve better than any other system I know of is bootstrapping on a new system. It's also quite good at handling foibles (== regressions) across version bumps of the source program. I agree it's suboptimal (at best :-) to have a PyPI module that depends on the autotools. I've found it to be a real PITA for Windows users who want to develop certain Emacs Lisp applications, for example, which is a similar kind of development environment to Python. But for bootstrapping Python itself, what else is there, really? Every time I've seen a project move to something like SCons, I've also seen their port rate to new platforms decrease dramatically. (Hard to interpret that trend, though -- if a project has the free cycles to change build systems, maybe they have ported to almost all of the plausible platforms already.) Autotools alternatives just have way fewer modules for dealing with new and/or old ;-) platforms. Bright! Shiny! New! system use crufty old system :-( + => + missing library :-( library Caveat: Autotools is the only build system (well, Imake -- now you know how old I am!) I have substantial experience with. But I've heard stories from colleagues about the limitations of Scons and cmake in this respect, and I find it hard to believe that there are systems out there with the accumulated knowledge of platforms that has been assembled in Autotools over the decades. From breamoreboy at yahoo.co.uk Tue Mar 17 03:28:29 2015 From: breamoreboy at yahoo.co.uk (Mark Lawrence) Date: Tue, 17 Mar 2015 02:28:29 +0000 Subject: [Python-ideas] Rewriting the build system (part 2) In-Reply-To: References: Message-ID: On 16/03/2015 23:45, Chris Barker wrote: > On Mon, Mar 16, 2015 at 3:45 PM, Ned Deily > > wrote: > > it would be a monumental undertaking to try to migrate the current > build system to something substantially different. It's fine to let off > steam about frustrations with build systems but talking about it here is > not gonna cause it to change. And it won't change unless someone (or, > more likely, some big company) is willing to invest an enormous effort > in people time and machine resources to do so. Stefan's suggestion is > much more practical. > > > This doesn't seem to me to be a winner takes all proposition -- there is > no reason someone couldn't set up a new build system for Python, get it > all working nicely on their platform(s) of choice, persuade others how > wonderful it is, get contributions, etc. > This sounds really simple to me so when do you start work on it? -- My fellow Pythonistas, ask not what our language can do for you, ask what you can do for our language. Mark Lawrence From abarnert at yahoo.com Tue Mar 17 04:38:03 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Mon, 16 Mar 2015 20:38:03 -0700 Subject: [Python-ideas] Rewriting the build system (part 2) In-Reply-To: References: Message-ID: <9173781D-E734-4BDD-88C5-EC8755A923E8@yahoo.com> On Mar 16, 2015, at 12:19 PM, Ryan Gonzalez wrote: > > Most due let you do that easily. For instance, fbuild lets you just trap the raised error and move on. How does trapping an error and moving on give you a user-configurable option to select which OpenSSL to use, whether he wants a fat i386/x86_64 build or just x86_64, whether he wants a framework bundle or a traditional layout...? (These are the first three flags that Homebrew passes to Python's configure aside from the install prefix, which I'd hope everything can handle.) >> On Mon, Mar 16, 2015 at 1:05 PM, Andrew Barnert wrote: >> Something nobody's mentioned yet in this thread: >> >> The autotools stuff doesn't just handle different platforms, it also handles local differences (not every FooLinux x.y system has Tcl/Tk; if yours doesn't, you can still build Python, you just won't get tkinter) and user options (disabling features, using a specific OpenSSL instead of the older system one that would get autodetected, etc.). >> >> Autoconf makes this relatively easy (not that anything in autoconf is easy, but if you can write the OpenSSL detection, it's not significantly harder to write OpenSSL detection with a flag to override the path). Is the same thing true for other build configuration systems? >> >> >> Sent from my iPhone >> >> On Mar 16, 2015, at 9:27 AM, Chris Barker wrote: >> >>>> >> That written in Python, so Python would require Python to build which >>>> >> would require Python to build which would require... >>> >>> A bit more here: >>> >>> It seems there are two cases: >>> >>> 1) Development on and for a common, mature platform: >>> >>> In this case finding a functional python with which to run the build tool sis trivial. We're all set, no need to exclude python-based tools. >>> >>> 2) Development for / porting to a new and/or obscure platform: >>> >>> In this case, then there is no existing python with which to run the build tools. And there are two ways you might go about this (in general): >>> >>> a) cross-compiling -- run the compiler on a existing mature platform, and generate a compiled python for the new platform. IN this case, the build tools only need to run on the development platform, so we are back to (1), and all set. Note that the hard part of cross-compiling is still done by the C compiler, not really the build/configuration tools, The build tools need to be able to set up the cross-compile -- but presumable a new tool is chosen be cause it's easier to do things like that with it... >>> >>> b) Native compiling -- compiling python on the new platform. This is the tricky one. In this case, the entire goal is to get python building/running on the new platform. Clearly you can't just use the nifty python-based build system to do that. But as that's the goal, you'll still need to do the tricky parts anyway. So you could cross-compile the first version -- with maybe as few modules as possible to get started. Another option is a build system that needs python for the configuration stage, but not the compilation stage -- kind of like autotools: autotools is used in a three-step process: >>> >>> i) run autoconf, etc, to generate a configure script -- this is often distributed with the source tarball, so that end users don't need the autoconf tools >>> >>> ii) run the configure script to generate Makefiles, etc. configured for the platform at hand. >>> >>> iii) run make to do the actual build. >>> >>> Usually the last two steps are run natively -- but it's not too much of a stretch to run i and ii on a known build machine, and then only ned to run make on the final target. That would require the build tools to have a cross-configure ability. >>> >>> I have no idea if any of the known tools can do this, but if so, it would be pretty usable. >>> >>> And also worth keeping in mind that truly new platforms are pretty rare -- this kind of bootstrapping would not have to be done often --not a killer if it takes some hand-work. >>> >>> But the grain of salt : I have neer and am unlikely ever, to try to port a significant project to a new platform. Folks that actually need to do this should make these decisions. >>> >>>> Will this affect the commands required to build Python from source, or >>>> is all this just for actual development? >>> >>> well, it may or may not be both: >>> >>> If the system supported a three-step process I outlined above, then makefiles (or the like) could be shipped for the common platforms, though probably easier to run the configure step on the final target. But the vast majority of final targets do have a python ready to run the configure/build machinery. >>> >>>> someone >>>> who has little knowledge of Python's internals, and just wants to >>>> build the latest version (either from hg or from a source tarball). >>> >>> That's the assumption I'm following -- the user wants the latest version -- so probably already has easy access to an older version with which to run the build tools... >>> >>>> But in the second case, it'd be much safer to minimize the >>>> requirements. If you can untar the source and just "./configure; make; >>>> sudo make install" your way to having the latest Python, that's a LOT >>>> easier than I've just been doing trying to get my Dad onto a newer >>>> Mozilla Thunderbird. (I don't understand what it's doing all along >>>> there, but it pulls a whole bunch of separate repositories when I say >>>> update, and there's a separate bootstrap step, and ... I dunno.) >>> >>> requiring _some_, not very recent, python is not exactly >>> >>> """a whole bunch of separate repositories when I say update, and there's a separate bootstrap step""" >>> >>> If there is an already build python (like OS-X, any general purpose linux distro, Windows if you don't mind point and clicking the python.org installer, etc, then it's not really a "bootstrap". >>> >>> In short -- using a python-based build system would make porting to a brand-new platform trickier -- but not really affect most other building. >>> >>> -CHB >>> >>> >>> -- >>> >>> Christopher Barker, Ph.D. >>> Oceanographer >>> >>> Emergency Response Division >>> NOAA/NOS/OR&R (206) 526-6959 voice >>> 7600 Sand Point Way NE (206) 526-6329 fax >>> Seattle, WA 98115 (206) 526-6317 main reception >>> >>> Chris.Barker at noaa.gov >>> _______________________________________________ >>> Python-ideas mailing list >>> Python-ideas at python.org >>> https://mail.python.org/mailman/listinfo/python-ideas >>> Code of Conduct: http://python.org/psf/codeofconduct/ >> >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ > > > > -- > Ryan > [ERROR]: Your autotools build scripts are 200 lines longer than your program. Something?s wrong. > http://kirbyfan64.github.io/ > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.barker at noaa.gov Tue Mar 17 05:13:05 2015 From: chris.barker at noaa.gov (Chris Barker) Date: Mon, 16 Mar 2015 21:13:05 -0700 Subject: [Python-ideas] Rewriting the build system (part 2) In-Reply-To: References: Message-ID: On Mon, Mar 16, 2015 at 7:28 PM, Mark Lawrence wrote: > On Mon, Mar 16, 2015 at 3:45 PM, Ned Deily > >> This doesn't seem to me to be a winner takes all proposition -- there is >> no reason someone couldn't set up a new build system for Python, get it >> all working nicely on their platform(s) of choice, persuade others how >> wonderful it is, get contributions, etc. >> >> > This sounds really simple to me so when do you start work on it? I think I was misunderstood -- my point was that there is no reason to start all this with trying to persuade the python dev team to accept a new build system up front. The only possible way it could happen were if someone(s) did the work and got a new system in place that worked well and proved to be better than autoconf in various ways, while being just as capable. And that would be LOT of work to get that far. But it could be done without having to persuade anyone up front that it 's a good idea. Frankly, I doubt anyone's going to put that work in -- I know I won't -- remember that I'm one of the folks that's skeptical that this mythical build system exists at all -- never mind the work to transition to it. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From lkb.teichmann at gmail.com Tue Mar 17 16:55:52 2015 From: lkb.teichmann at gmail.com (Martin Teichmann) Date: Tue, 17 Mar 2015 16:55:52 +0100 Subject: [Python-ideas] Simpler Customization of Class Creation, next round Message-ID: Hello everybody, recently I posted PEP 487, a simpler customization of class creation. For newcomers: I propose the introduction of a __init_subclass__ classmethod which initializes subclasses of a class, simplifying what metaclasses can already do. It took me a while to digest all the ideas from the list here, but well, we're not in a hurry. So, I updated PEP 487, and pushed the changes to github at https://github.com/tecki/peps/commits/master I applied the following changes: PEP 487 contained the possibility to set a namespace for subclasses. The most important usecase for this feature would be to have an OrderedDict as the class definition namespace. As Eric Snow pointed out, that will be soon standard anyways, so I took out this feature from the PEP. The implementation on PyPI now just uses an OrderedDict as a namespace, anticipating the expected changes to CPython. I also did some reading on possible usecases for PEP 487, so that it actually may be used by someone. Some Traits-like thing is a standard usecase, so I looked especially at IPython's traitlets, which are a simple example of that usecase. Currently traitlets use both __new__ and __init__ of a metaclass. So I tried to also introduce a __new_subclass__ the same way I introduced __init_subclass__. This turned out much harder than I thought, actually impossible, because it is type.__new__ that sets the method resolution order, so making super() work in a __new_subclass__ hook is a chicken-egg problem: we need the MRO to find the next base class to call, but the most basic base class is the one creating the MRO. Nick, how did you solve that problem in PEP 422? Anyhow, I think that traitlets can also be written just using __init_subclass__. There is just this weird hint in the docs that you should use __new__ for metaclasses, not __init__, a hint I never understood as the reasons when to use __new__ or __init__ are precisely the same for normal classes and metaclasses. So I think we don't miss out much when not having __new_subclass__. I also updated the implementation of PEP 487, it's still at https://pypi.python.org/pypi/metaclass Greetings Martin From luciano at ramalho.org Tue Mar 17 17:10:32 2015 From: luciano at ramalho.org (Luciano Ramalho) Date: Tue, 17 Mar 2015 13:10:32 -0300 Subject: [Python-ideas] .from and .to instead of .encode and .decode In-Reply-To: References: Message-ID: On Mon, Mar 16, 2015 at 10:09 AM, anatoly techtonik wrote: > I use the analogy of "encoding abstract numbers into specific bytes" myself. Your version of the analogy doesn't sound as helpful to me; "abastract numbers" sounds even more cryptic than bytes. My mnemonics are "decode from cryptic bytes to human text" and "encode from human text to cryptic bytes". A more objective problem is: "from" is a keyword, so it can't be a method name. Best, Luciano -- Luciano Ramalho | Author of Fluent Python (O'Reilly, 2015) | http://shop.oreilly.com/product/0636920032519.do | Professor em: http://python.pro.br | Twitter: @ramalhoorg From luciano at ramalho.org Tue Mar 17 17:52:04 2015 From: luciano at ramalho.org (Luciano Ramalho) Date: Tue, 17 Mar 2015 13:52:04 -0300 Subject: [Python-ideas] A mutable alternative to namedtuple Message-ID: Sometimes we need a simple class to hold some mutable attributes, provide a nice repr, support == for testing, and support iterable unpacking, so you can write: >>> p = Point(3, 4) >>> x, y = p That's very much like the classes built by namedtuple, but mutable. I propose we add to the collections module another class factory. I am calling it plainclass, but perhaps we can think of a better name. Here is how it would be used: >>> import collections >>> Point = collections.plainclass('Point', 'x y') The signature of the plainclass function would be exactly the same as namedtuple, supporting the same alternative ways of naming the attributes. The semantics of the generated Point class would be like this code: https://gist.github.com/ramalho/fd3d367e9d3b2a659faf What do you think? Cheers, Luciano PS. I am aware that there are "Namespace" classes in the standard library (e.g. [2]). They solve a different problem. [2] https://docs.python.org/3/library/argparse.html#argparse.Namespace -- Luciano Ramalho | Author of Fluent Python (O'Reilly, 2015) | http://shop.oreilly.com/product/0636920032519.do | Professor em: http://python.pro.br | Twitter: @ramalhoorg From rymg19 at gmail.com Tue Mar 17 18:01:24 2015 From: rymg19 at gmail.com (Ryan Gonzalez) Date: Tue, 17 Mar 2015 12:01:24 -0500 Subject: [Python-ideas] Rewriting the build system (part 2) In-Reply-To: <87wq2g73zq.fsf@uwakimon.sk.tsukuba.ac.jp> References: <87wq2g73zq.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: On Mon, Mar 16, 2015 at 9:27 PM, Stephen J. Turnbull wrote: > Ryan Gonzalez writes: > > > I have used fbuild a LOT > > Actually, I suspect that in the relevant sense, you haven't used it a > lot. Have you used it for a single program on > > 1. more than 3 CPUs, AND > 2. on each CPU, more than one (and preferably more than two) OSes, AND > 3. for each CPU-OS combination, more than two different configurations > of the program being built, AND > 4. for each CPU-OS-configuration combination, at least 3 versions of > the program source (preferably spanning a major version bump)? > > No...but I know that Felix uses it for iPhone, OSX, Linux, ... with separation of host and target and multiple configurations. > The number of programs/versions you have built on a single (or small > selection of) CPU-OS combinations is irrelevant to the discussion. > The problem autotools helps to solve better than any other system I > know of is bootstrapping on a new system. It's also quite good at > handling foibles (== regressions) across version bumps of the source > program. > > I agree it's suboptimal (at best :-) to have a PyPI module that > depends on the autotools. I've found it to be a real PITA for Windows > users who want to develop certain Emacs Lisp applications, for > example, which is a similar kind of development environment to Python. > > But for bootstrapping Python itself, what else is there, really? > Every time I've seen a project move to something like SCons, I've also > seen their port rate to new platforms decrease dramatically. (Hard to > interpret that trend, though -- if a project has the free cycles to > change build systems, maybe they have ported to almost all of the > plausible platforms already.) Autotools alternatives just have way > fewer modules for dealing with new and/or old ;-) platforms. > > This is also SCons. Everybody loved it until they had to maintain it. :) > Bright! Shiny! New! system use crufty old system :-( > + => + > missing library :-( library > > Caveat: Autotools is the only build system (well, Imake -- now you > know how old I am!) I have substantial experience with. But I've > heard stories from colleagues about the limitations of Scons and > cmake in this respect, and I find it hard to believe that there are > systems out there with the accumulated knowledge of platforms that has > been assembled in Autotools over the decades. > > SCons is very annoying. CMake is weird and only really useful for those huge, 20,000 file C++ projects. -- Ryan [ERROR]: Your autotools build scripts are 200 lines longer than your program. Something?s wrong. http://kirbyfan64.github.io/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From rymg19 at gmail.com Tue Mar 17 18:06:52 2015 From: rymg19 at gmail.com (Ryan Gonzalez) Date: Tue, 17 Mar 2015 12:06:52 -0500 Subject: [Python-ideas] Rewriting the build system (part 2) In-Reply-To: <9173781D-E734-4BDD-88C5-EC8755A923E8@yahoo.com> References: <9173781D-E734-4BDD-88C5-EC8755A923E8@yahoo.com> Message-ID: On Mon, Mar 16, 2015 at 10:38 PM, Andrew Barnert wrote: > On Mar 16, 2015, at 12:19 PM, Ryan Gonzalez wrote: > > Most due let you do that easily. For instance, fbuild lets you just trap > the raised error and move on. > > > How does trapping an error and moving on give you a user-configurable > option to select which OpenSSL to use, whether he wants a fat i386/x86_64 > build or just x86_64, whether he wants a framework bundle or a traditional > layout...? (These are the first three flags that Homebrew passes to > Python's configure aside from the install prefix, which I'd hope everything > can handle.) > > Just mix a command-line argument with a try-catch block. It's actually much easier/less-hackish than it sounds. The nice thing is that fbuild allows multiple builders, so separating target/host stuff and command-line args is super-easy. > On Mon, Mar 16, 2015 at 1:05 PM, Andrew Barnert < > abarnert at yahoo.com.dmarc.invalid> wrote: > >> Something nobody's mentioned yet in this thread: >> >> The autotools stuff doesn't just handle different platforms, it also >> handles local differences (not every FooLinux x.y system has Tcl/Tk; if >> yours doesn't, you can still build Python, you just won't get tkinter) and >> user options (disabling features, using a specific OpenSSL instead of the >> older system one that would get autodetected, etc.). >> >> Autoconf makes this relatively easy (not that anything in autoconf is >> easy, but if you can write the OpenSSL detection, it's not significantly >> harder to write OpenSSL detection with a flag to override the path). Is the >> same thing true for other build configuration systems? >> >> >> Sent from my iPhone >> >> On Mar 16, 2015, at 9:27 AM, Chris Barker wrote: >> >> >> That written in Python, so Python would require Python to build which >>> >> would require Python to build which would require... >>> >> >> A bit more here: >> >> It seems there are two cases: >> >> 1) Development on and for a common, mature platform: >> >> In this case finding a functional python with which to run the build tool >> sis trivial. We're all set, no need to exclude python-based tools. >> >> 2) Development for / porting to a new and/or obscure platform: >> >> In this case, then there is no existing python with which to run the >> build tools. And there are two ways you might go about this (in general): >> >> a) cross-compiling -- run the compiler on a existing mature platform, >> and generate a compiled python for the new platform. IN this case, the >> build tools only need to run on the development platform, so we are back to >> (1), and all set. Note that the hard part of cross-compiling is still done >> by the C compiler, not really the build/configuration tools, The build >> tools need to be able to set up the cross-compile -- but presumable a new >> tool is chosen be cause it's easier to do things like that with it... >> >> b) Native compiling -- compiling python on the new platform. This is >> the tricky one. In this case, the entire goal is to get python >> building/running on the new platform. Clearly you can't just use the nifty >> python-based build system to do that. But as that's the goal, you'll still >> need to do the tricky parts anyway. So you could cross-compile the first >> version -- with maybe as few modules as possible to get started. Another >> option is a build system that needs python for the configuration stage, but >> not the compilation stage -- kind of like autotools: autotools is used in a >> three-step process: >> >> i) run autoconf, etc, to generate a configure script -- this is often >> distributed with the source tarball, so that end users don't need the >> autoconf tools >> >> ii) run the configure script to generate Makefiles, etc. configured for >> the platform at hand. >> >> iii) run make to do the actual build. >> >> Usually the last two steps are run natively -- but it's not too much of a >> stretch to run i and ii on a known build machine, and then only ned to run >> make on the final target. That would require the build tools to have a >> cross-configure ability. >> >> I have no idea if any of the known tools can do this, but if so, it would >> be pretty usable. >> >> And also worth keeping in mind that truly new platforms are pretty rare >> -- this kind of bootstrapping would not have to be done often --not a >> killer if it takes some hand-work. >> >> But the grain of salt : I have neer and am unlikely ever, to try to port >> a significant project to a new platform. Folks that actually need to do >> this should make these decisions. >> >> Will this affect the commands required to build Python from source, or >>> is all this just for actual development? >>> >> >> well, it may or may not be both: >> >> If the system supported a three-step process I outlined above, then >> makefiles (or the like) could be shipped for the common platforms, though >> probably easier to run the configure step on the final target. But the vast >> majority of final targets do have a python ready to run the configure/build >> machinery. >> >> someone >>> who has little knowledge of Python's internals, and just wants to >>> build the latest version (either from hg or from a source tarball). >>> >> >> That's the assumption I'm following -- the user wants the latest version >> -- so probably already has easy access to an older version with which to >> run the build tools... >> >> But in the second case, it'd be much safer to minimize the >>> >> requirements. If you can untar the source and just "./configure; make; >>> sudo make install" your way to having the latest Python, that's a LOT >>> easier than I've just been doing trying to get my Dad onto a newer >>> Mozilla Thunderbird. (I don't understand what it's doing all along >>> there, but it pulls a whole bunch of separate repositories when I say >>> update, and there's a separate bootstrap step, and ... I dunno.) >>> >> >> requiring _some_, not very recent, python is not exactly >> >> """a whole bunch of separate repositories when I say update, and there's >> a separate bootstrap step""" >> >> If there is an already build python (like OS-X, any general purpose linux >> distro, Windows if you don't mind point and clicking the python.org >> installer, etc, then it's not really a "bootstrap". >> >> In short -- using a python-based build system would make porting to a >> brand-new platform trickier -- but not really affect most other building. >> >> -CHB >> >> >> -- >> >> Christopher Barker, Ph.D. >> Oceanographer >> >> Emergency Response Division >> NOAA/NOS/OR&R (206) 526-6959 voice >> 7600 Sand Point Way NE (206) 526-6329 fax >> Seattle, WA 98115 (206) 526-6317 main reception >> >> Chris.Barker at noaa.gov >> >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ >> >> >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ >> > > > > -- > Ryan > [ERROR]: Your autotools build scripts are 200 lines longer than your > program. Something?s wrong. > http://kirbyfan64.github.io/ > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > > -- Ryan [ERROR]: Your autotools build scripts are 200 lines longer than your program. Something?s wrong. http://kirbyfan64.github.io/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From ericsnowcurrently at gmail.com Tue Mar 17 18:14:33 2015 From: ericsnowcurrently at gmail.com (Eric Snow) Date: Tue, 17 Mar 2015 11:14:33 -0600 Subject: [Python-ideas] A mutable alternative to namedtuple In-Reply-To: References: Message-ID: On Tue, Mar 17, 2015 at 10:52 AM, Luciano Ramalho wrote: > Sometimes we need a simple class to hold some mutable attributes, > provide a nice repr, support == for testing, and support iterable > unpacking, so you can write: > >>>> p = Point(3, 4) >>>> x, y = p > > That's very much like the classes built by namedtuple, but mutable. > > I propose we add to the collections module another class factory. I am > calling it plainclass, but perhaps we can think of a better name. Here > is how it would be used: > >>>> import collections >>>> Point = collections.plainclass('Point', 'x y') > > The signature of the plainclass function would be exactly the same as > namedtuple, supporting the same alternative ways of naming the > attributes. > > The semantics of the generated Point class would be like this code: > > https://gist.github.com/ramalho/fd3d367e9d3b2a659faf Should it also have all the same methods as a namedtuple class (e.g. the tuple methods, _make, _replace)? What about the other namedtuple attrs (_fields, etc.)? What's the motivation for parity with namedtuple (particularly iteration)? I suppose I see the desire for a fixed set of mutable attributes specific to the generated type. However, there have been numerous discussions on this list about alternative approaches to namedtuple which apply here. Simply adapting namedtuple may not be the right thing. Regardless, this is the sort of thing that should bake outside the stdlib for a while to prove it's approach and its worth, much as namedtuple did. It would also help if there were a concrete use case in the stdlib that this new class (factory) would satisfy. > PS. I am aware that there are "Namespace" classes in the standard > library (e.g. [2]). They solve a different problem. > > [2] https://docs.python.org/3/library/argparse.html#argparse.Namespace Don't forget types.SimpleNamespace. :) -eric From eric at trueblade.com Tue Mar 17 18:20:08 2015 From: eric at trueblade.com (Eric V. Smith) Date: Tue, 17 Mar 2015 13:20:08 -0400 Subject: [Python-ideas] A mutable alternative to namedtuple In-Reply-To: References: Message-ID: <55086248.2010009@trueblade.com> On 03/17/2015 12:52 PM, Luciano Ramalho wrote: > Sometimes we need a simple class to hold some mutable attributes, > provide a nice repr, support == for testing, and support iterable > unpacking, so you can write: > >>>> p = Point(3, 4) >>>> x, y = p > > That's very much like the classes built by namedtuple, but mutable. https://pypi.python.org/pypi/namedlist It also adds default values to the generated constructor, which may or may not be desirable. But if used exactly like collections.namedtuple, it ignores the default values. Eric. > > I propose we add to the collections module another class factory. I am > calling it plainclass, but perhaps we can think of a better name. Here > is how it would be used: > >>>> import collections >>>> Point = collections.plainclass('Point', 'x y') > > The signature of the plainclass function would be exactly the same as > namedtuple, supporting the same alternative ways of naming the > attributes. > > The semantics of the generated Point class would be like this code: > > https://gist.github.com/ramalho/fd3d367e9d3b2a659faf > > What do you think? > > Cheers, > > Luciano > > PS. I am aware that there are "Namespace" classes in the standard > library (e.g. [2]). They solve a different problem. > > [2] https://docs.python.org/3/library/argparse.html#argparse.Namespace > > From nicholas.chammas at gmail.com Tue Mar 17 19:39:59 2015 From: nicholas.chammas at gmail.com (Nicholas Chammas) Date: Tue, 17 Mar 2015 11:39:59 -0700 (PDT) Subject: [Python-ideas] gofmt for Python: standardized styling as a language feature In-Reply-To: References: Message-ID: <92ac0f1b-617e-4da5-a4d0-47491ea446ad@googlegroups.com> I posted this via Google Groups and got a response back by email saying my post was rejected because I wasn't a list member. So I signed up via Mailman and resubmitted this post by mail. Not sure if I should expect a duplicate post to show up here on not... Apologies. On Tuesday, March 17, 2015 at 2:13:31 PM UTC-4, Nicholas Chammas wrote: > > The Zen of Python says: > > There should be one? and preferably only one ?obvious way to do it. > > Python already has a style guide in PEP 8 > . More importantly, from a > practical standpoint, Python has automated third-party tools for checking > and enforcing PEP 8 guidelines: > > - pep8 : Python style guide checker > - autopep8 : A tool that > automatically formats Python code to conform to the PEP 8 style guide > > I say ?more importantly? because these tools are what make the style guide > alive. If a Python project of any non-trivial size is PEP 8-compliant, you > can bet it?s because the project contributors automated the process of > enforcing compliance using these or similar tools. (Certainly, some parts > of PEP 8 are hard to automate. But those parts that *can* be automated > have been, to the benefit of Python programmers everywhere.) > > Having a style guide provides benefits that I think are well-understood. > Additionally having a standardized style that can be *applied > automatically* makes whatever benefits of the style guide that much more > prevalent in the greater community and actually effective. > > As many of you may already know, the Go-lang folks have such a feature for > their language (though they call it formatting as opposed to styling). They > note: > > go fmt your code : > > Gofmt is a tool that automatically formats > Go source code. > > Gofmt?d code is: > > - easier to *write*: never worry about minor formatting concerns while > hacking away, > - easier to *read*: when all code looks the same you need not mentally > convert others? formatting style into something you can understand. > - easier to *maintain*: mechanical changes to the source don?t cause > unrelated changes to the file?s formatting; diffs show only the real > changes. > - *uncontroversial*: never have a debate about spacing or brace > position ever again! > > Of course, we don?t worry about brace positions, but these benefits of a > standardized style carry over into any language. :) > > So, in the spirit of The Zen of Python, and taking a cue from Go-lang?s > gofmt , I propose we promote > standardized auto-styling to a first-class language feature. > > UI-wise, I?m not sure how exactly we?d present this to the user. Perhaps > something like: > > python -m style script.py > > Don?t read too much into this UI suggestion, as I?m really not sure how it > would be implemented. The point is to expose some auto-styler as a standard > language tool?i.e. something like autopep8, but as a built-in utility. > > On running a command like that, script.py would automatically be updated > to conform to the standard style we want to promote. This is what gofmt > does for Go code. > > Promoting an auto-styler to be a first-class language feature builds on > the foundation laid in PEP 8 and in tools like pep8 and autopep8. It > helps reduce code style to more of a non-issue for Python projects than it > already is. > ------------------------------ > > *Side Note 1:* There was a previous python-ideas discussion about > introducing standardized formatting here > , > but it was totally derailed by the OP?s suggestion that the formatter > introduce block delimiters. I am proposing no such thing here. > > *Side Note 2:* I found these additional comments > about gofmt interesting: > > Gofmt also enables gofix , which can make > arbitrarily complex source transformations. Gofix was an invaluable tool > during the early days when we regularly made breaking changes to the > language and libraries. > > Introducing Gofix : > > Gofix gives us the ability to fix mistakes or completely rethink package > APIs without worrying about the cost of converting existing code. > > Python has its own analog to gofix in 2to3 > , though that was built for > a specific migration rather than as a regular language feature. I wonder if > having a gofmt analog in Python would similarly facilitate future > migrations, though I doubt we?ll ever have a big as migration as the 2-to-3 > one. > ? > -------------- next part -------------- An HTML attachment was scrubbed... URL: From nicholas.chammas at gmail.com Tue Mar 17 19:31:10 2015 From: nicholas.chammas at gmail.com (Nicholas Chammas) Date: Tue, 17 Mar 2015 18:31:10 +0000 Subject: [Python-ideas] gofmt for Python: standardized styling as a language feature Message-ID: The Zen of Python says: There should be one? and preferably only one ?obvious way to do it. Python already has a style guide in PEP 8 . More importantly, from a practical standpoint, Python has automated third-party tools for checking and enforcing PEP 8 guidelines: - pep8 : Python style guide checker - autopep8 : A tool that automatically formats Python code to conform to the PEP 8 style guide I say ?more importantly? because these tools are what make the style guide alive. If a Python project of any non-trivial size is PEP 8-compliant, you can bet it?s because the project contributors automated the process of enforcing compliance using these or similar tools. (Certainly, some parts of PEP 8 are hard to automate. But those parts that *can* be automated have been, to the benefit of Python programmers everywhere.) Having a style guide provides benefits that I think are well-understood. Additionally having a standardized style that can be *applied automatically* makes whatever benefits of the style guide that much more prevalent in the greater community and actually effective. As many of you may already know, the Go-lang folks have such a feature for their language (though they call it formatting as opposed to styling). They note: go fmt your code : Gofmt is a tool that automatically formats Go source code. Gofmt?d code is: - easier to *write*: never worry about minor formatting concerns while hacking away, - easier to *read*: when all code looks the same you need not mentally convert others? formatting style into something you can understand. - easier to *maintain*: mechanical changes to the source don?t cause unrelated changes to the file?s formatting; diffs show only the real changes. - *uncontroversial*: never have a debate about spacing or brace position ever again! Of course, we don?t worry about brace positions, but these benefits of a standardized style carry over into any language. :) So, in the spirit of The Zen of Python, and taking a cue from Go-lang?s gofmt , I propose we promote standardized auto-styling to a first-class language feature. UI-wise, I?m not sure how exactly we?d present this to the user. Perhaps something like: python -m style script.py Don?t read too much into this UI suggestion, as I?m really not sure how it would be implemented. The point is to expose some auto-styler as a standard language tool?i.e. something like autopep8, but as a built-in utility. On running a command like that, script.py would automatically be updated to conform to the standard style we want to promote. This is what gofmt does for Go code. Promoting an auto-styler to be a first-class language feature builds on the foundation laid in PEP 8 and in tools like pep8 and autopep8. It helps reduce code style to more of a non-issue for Python projects than it already is. ------------------------------ *Side Note 1:* There was a previous python-ideas discussion about introducing standardized formatting here , but it was totally derailed by the OP?s suggestion that the formatter introduce block delimiters. I am proposing no such thing here. *Side Note 2:* I found these additional comments about gofmt interesting: Gofmt also enables gofix , which can make arbitrarily complex source transformations. Gofix was an invaluable tool during the early days when we regularly made breaking changes to the language and libraries. Introducing Gofix : Gofix gives us the ability to fix mistakes or completely rethink package APIs without worrying about the cost of converting existing code. Python has its own analog to gofix in 2to3 , though that was built for a specific migration rather than as a regular language feature. I wonder if having a gofmt analog in Python would similarly facilitate future migrations, though I doubt we?ll ever have a big as migration as the 2-to-3 one. -------------- next part -------------- An HTML attachment was scrubbed... URL: From nicholas.chammas at gmail.com Tue Mar 17 21:51:30 2015 From: nicholas.chammas at gmail.com (Nicholas Chammas) Date: Tue, 17 Mar 2015 13:51:30 -0700 (PDT) Subject: [Python-ideas] gofmt for Python: standardized styling as a language feature In-Reply-To: References: Message-ID: <0592d0ad-ddea-4bcc-8139-930370cad143@googlegroups.com> Please disregard this thread in favor of the one that went successfully through via Mailman here: https://groups.google.com/forum/#!topic/python-ideas/xute5NmXRps Apologies again about the duplicate posting. -------------- next part -------------- An HTML attachment was scrubbed... URL: From abarnert at yahoo.com Tue Mar 17 23:16:33 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Tue, 17 Mar 2015 15:16:33 -0700 Subject: [Python-ideas] gofmt for Python: standardized styling as a language feature In-Reply-To: References: Message-ID: <63A4AF9C-5CD1-407B-8F20-3697EC62466D@yahoo.com> On Mar 17, 2015, at 11:31 AM, Nicholas Chammas wrote: > The Zen of Python says: > > There should be one? and preferably only one ?obvious way to do it. > As someone who uses autopep8 regularly but has at least once had resistance to getting into a team's workflow, +1 on the idea. The analogy with gofmt isn't perfect because gofmt only deals with some simple issues where there's an unambiguous 100%-clear standard, while PEP 8 includes a wide range of guidelines that are usually but not always consistent and appropriate, but that (obviously) doesn't mean autopep8 is an impossible or pointless non-starter. So, is your proposal that autopep8 (and therefore pep8, on which it depends) be added to the stdlib with only minor changes? If so: * You'd probably want to ask whether Hideo Hattori and Johann C. Rocholl are willing to relicense and contribute their code, and maintain it for years to come, before asking whether Python will accept the result. * Will it be easy to transform a new version of the PyPI libraries into a new version of the stdlib module in some mostly-automated way for each new Python release? * Given that Python updates less often than pep8 and autopep8, will it generally be "good enough" to use the autopep8 1.1.3 that came with your Python 3.5 instead of installing the 1.1.7 that's on PyPI? What if you're still using Python 3.5 in another two years, and Python 3.6 came with 1.2.1, and 1.2.5 is on PyPI; is using 1.1.3 still a good idea? (Compare with pysqlite; I've used sqlite3 many times, and only wanted to get the newer version twice--and once just because it was the easiest way to build with a local variant of the C library on Windows.) * Are all of the current features and bugs (see issues on GitHub) acceptable as-is for the stdlib today? If the answers to those questions is "yes", this seems like the way to go. Otherwise, you're probably suggesting someone either fork the existing projects or write a brand new one (and as the one with the itch and the ideas, who better than you?). > Python already has a style guide in PEP 8. More importantly, from a practical standpoint, Python has automated third-party tools for checking and enforcing PEP 8 guidelines: > > pep8: Python style guide checker > autopep8: A tool that automatically formats Python code to conform to the PEP 8 style guide > I say ?more importantly? because these tools are what make the style guide alive. If a Python project of any non-trivial size is PEP 8-compliant, you can bet it?s because the project contributors automated the process of enforcing compliance using these or similar tools. (Certainly, some parts of PEP 8 are hard to automate. But those parts that can be automated have been, to the benefit of Python programmers everywhere.) > > Having a style guide provides benefits that I think are well-understood. Additionally having a standardized style that can be applied automatically makes whatever benefits of the style guide that much more prevalent in the greater community and actually effective. > > As many of you may already know, the Go-lang folks have such a feature for their language (though they call it formatting as opposed to styling). They note: > > go fmt your code: > > Gofmt is a tool that automatically formats Go source code. > > Gofmt?d code is: > > easier to write: never worry about minor formatting concerns while hacking away, > easier to read: when all code looks the same you need not mentally convert others? formatting style into something you can understand. > easier to maintain: mechanical changes to the source don?t cause unrelated changes to the file?s formatting; diffs show only the real changes. > uncontroversial: never have a debate about spacing or brace position ever again! > Of course, we don?t worry about brace positions, but these benefits of a standardized style carry over into any language. :) > > So, in the spirit of The Zen of Python, and taking a cue from Go-lang?s gofmt, I propose we promote standardized auto-styling to a first-class language feature. > > UI-wise, I?m not sure how exactly we?d present this to the user. Perhaps something like: > > python -m style script.py > Don?t read too much into this UI suggestion, as I?m really not sure how it would be implemented. The point is to expose some auto-styler as a standard language tool?i.e. something like autopep8, but as a built-in utility. > > On running a command like that, script.py would automatically be updated to conform to the standard style we want to promote. This is what gofmt does for Go code. > > Promoting an auto-styler to be a first-class language feature builds on the foundation laid in PEP 8 and in tools like pep8 and autopep8. It helps reduce code style to more of a non-issue for Python projects than it already is. > > Side Note 1: There was a previous python-ideas discussion about introducing standardized formatting here, but it was totally derailed by the OP?s suggestion that the formatter introduce block delimiters. I am proposing no such thing here. > > Side Note 2: I found these additional comments about gofmt interesting: > > Gofmt also enables gofix, which can make arbitrarily complex source transformations. Gofix was an invaluable tool during the early days when we regularly made breaking changes to the language and libraries. > > Introducing Gofix: > > Gofix gives us the ability to fix mistakes or completely rethink package APIs without worrying about the cost of converting existing code. > > Python has its own analog to gofix in 2to3, though that was built for a specific migration rather than as a regular language feature. I wonder if having a gofmt analog in Python would similarly facilitate future migrations, though I doubt we?ll ever have a big as migration as the 2-to-3 one. > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Tue Mar 17 23:28:52 2015 From: steve at pearwood.info (Steven D'Aprano) Date: Wed, 18 Mar 2015 09:28:52 +1100 Subject: [Python-ideas] gofmt for Python: standardized styling as a language feature In-Reply-To: References: Message-ID: <20150317222851.GS7655@ando.pearwood.info> On Tue, Mar 17, 2015 at 06:31:10PM +0000, Nicholas Chammas wrote: > Additionally having a standardized style that can be *applied > automatically* makes whatever benefits of the style guide that much > more prevalent in the greater community and actually effective. If you're talking about Gofmt and Gofix and enforcing PEP-8 rules for variable names, you're basically talking about refactoring tools, not merely source code reformatters. Do you have such tools you would like to propose for the standard library? If you do have such tools, or would like to volunteer to write them, then they should spend some time as a third party project to knock the sharp edges off them. Once their usefulness is well-established and proven, and their API and featureset stable, then they can be proposed for the standard library. Unlike Go-lang, which has the resources of Google behind it and can just pay some person or persons to work on this with as high a priority as Google wishes to give it, Python is reliant on volunteers. For major proposals like this, you're likely to get much more traction if you are volunteering yourself, rather than just hoping to convince somebody else to volunteer. I think good refactoring tools for Python would be useful. I'm not volunteering to write them. -- Steve From tjreedy at udel.edu Wed Mar 18 01:04:53 2015 From: tjreedy at udel.edu (Terry Reedy) Date: Tue, 17 Mar 2015 20:04:53 -0400 Subject: [Python-ideas] gofmt for Python: standardized styling as a language feature In-Reply-To: References: Message-ID: On 3/17/2015 2:31 PM, Nicholas Chammas wrote: > The Zen of Python says: > > There should be one? and preferably only one ?obvious way to do it. > > Python already has a style guide in PEP 8 > . More importantly, from a > practical standpoint, Python has automated third-party tools for > checking and enforcing PEP 8 guidelines: > > * |pep8| : Python style guide checker > * |autopep8| : A tool that > automatically formats Python code to conform to the PEP 8 style guide Except for idlelib and more recently turtledemo, the stdlib consists of *library* modules that define constants, mostly functions and classes, for import into and use by other libraries and applications. Notice that the stdlib contains lib2to3, while the 2to3 application that uses lib2to3 is in /Tools/Scripts, along with other application scripts, mostly short, written by core developers. pep8, autopep8, and similar applications do not belong in the stdlib. They are easily installed with, for instance, 'pip install autopep8'. The install process puts a startup file in /Scripts, which (for Windows at least) is on PATH (unlike /Tools/Scripts). After installing, I entered 'autopep8 file.py' and it runs. There is an issue on the tracker to add to the Idle IDE a gui frontend to pip, so beginners who use Idle will be able to manage (install, update, delete) 3rd-party apps like these without knowing about 'command lines' or having to remember pip options. (There is also an issue to apply such apps to the contents of an Editor window.) -- Terry Jan Reedy From nicholas.chammas at gmail.com Wed Mar 18 01:55:17 2015 From: nicholas.chammas at gmail.com (Nicholas Chammas) Date: Wed, 18 Mar 2015 00:55:17 +0000 Subject: [Python-ideas] gofmt for Python: standardized styling as a language feature In-Reply-To: <63A4AF9C-5CD1-407B-8F20-3697EC62466D@yahoo.com> References: <63A4AF9C-5CD1-407B-8F20-3697EC62466D@yahoo.com> Message-ID: On Tue, Mar 17, 2015 at 6:16 PM, Andrew Barnert wrote: > So, is your proposal that autopep8 (and therefore pep8, on which it > depends) be added to the stdlib with only minor changes? Not necessarily, though that is certainly a possible approach. (In fact, it's probably the approach most likely to succeed, since it builds on existing and tested tools.) I was more interested in first testing the _idea_ of an included auto-styler before proposing a specific path to implementing it. On Tue, Mar 17, 2015 at 6:28 PM, Steven D'Aprano wrote: Unlike Go-lang, which has the resources of Google behind it and can just > pay some person or persons to work on this with as high a priority as > Google wishes to give it, Python is reliant on volunteers. For major > proposals like this, you're likely to get much more traction if you are > volunteering yourself, rather than just hoping to convince somebody else > to volunteer. > Totally understood and agreed. To be upfront, I am not volunteering myself for this effort just yet. This is my first time posting to python-ideas. (I subscribed today.) I gather from Andrew and Steven's responses that ideas discussed on this list are supposed to be more "fully baked" than my proposal here. My apologies. For what it's worth, I certainly wouldn't expect myself or anyone else to invest any effort in an initiative like this if there is a chance that it will be rejected for reasons that can be explained upfront. On the other hand, if there is some consensus that this would be good thing to have at least in theory, then perhaps that would better set the stage for it to happen in practice. Hence the motivation for my original post. Nick -------------- next part -------------- An HTML attachment was scrubbed... URL: From nicholas.chammas at gmail.com Wed Mar 18 02:54:11 2015 From: nicholas.chammas at gmail.com (Nicholas Chammas) Date: Tue, 17 Mar 2015 21:54:11 -0400 Subject: [Python-ideas] gofmt for Python: standardized styling as a language feature In-Reply-To: References: Message-ID: On Tue, Mar 17, 2015 at 8:04 PM, Terry Reedy wrote: pep8, autopep8, and similar applications do not belong in the stdlib. They > are easily installed with, for instance, 'pip install autopep8'. I am not proposing putting autopep8 in the standard library. I am proposing that an auto-styler be included with Python as a first-class utility. The emphasis here is on *included* and *first-class*. That auto-styler may well be based somehow on autopep8, but that is a separate discussion about *implementation*. I am first checking that the *idea* is sound. I can?t tell if you had any objections to that idea. Yes, I know that today I can install and use autopep8 as a third-party utility using pip, but autopep8 today is neither included with Python nor a first-class utility. (FWIW, couldn?t autopep8 somehow be adapted to be an included module that can be invoked via python -m ? e.g. python -m compileall, python -m SimpleHTTPServer ? just as far as the mechanics are concerned? As I noted in my original post, I?m not in a position yet to specifically propose how an included auto-styler would work, but let?s say for argument?s sake that 1) we decided we liked the idea and 2) autopep8 fit the bill for promotion to an included utility, and 3) we addressed Andrew?s points.) Nick ? -------------- next part -------------- An HTML attachment was scrubbed... URL: From ethan at stoneleaf.us Wed Mar 18 02:58:08 2015 From: ethan at stoneleaf.us (Ethan Furman) Date: Tue, 17 Mar 2015 18:58:08 -0700 Subject: [Python-ideas] gofmt for Python: standardized styling as a language feature In-Reply-To: References: Message-ID: <5508DBB0.2010209@stoneleaf.us> On 03/17/2015 06:54 PM, Nicholas Chammas wrote: > I am not proposing putting |autopep8| in the standard library. I am proposing that an auto-styler be included with > Python as a first-class utility. The emphasis here is on /included/ and /first-class/. Perhaps you could explain what you mean by a /first-class/ utility? -- ~Ethan~ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: OpenPGP digital signature URL: From nicholas.chammas at gmail.com Wed Mar 18 03:17:28 2015 From: nicholas.chammas at gmail.com (Nicholas Chammas) Date: Wed, 18 Mar 2015 02:17:28 +0000 Subject: [Python-ideas] gofmt for Python: standardized styling as a language feature In-Reply-To: <5508DBB0.2010209@stoneleaf.us> References: <5508DBB0.2010209@stoneleaf.us> Message-ID: First class means it's like a language feature. Actually, in my case I am using "first-class" and "included with the language" synonymously. Not sure if that's appropriate terminology, but that's what I mean. To explain by rough analogy, Twisted is to asyncio what autopep8 would be to a first-class auto-styler. (I say "rough" because I understand that Twisted is not supposed to be replaced entirely by asyncio, but I hope my point is clear.) Nick On Tue, Mar 17, 2015 at 9:58 PM Ethan Furman wrote: > On 03/17/2015 06:54 PM, Nicholas Chammas wrote: > > > I am not proposing putting |autopep8| in the standard library. I am > proposing that an auto-styler be included with > > Python as a first-class utility. The emphasis here is on /included/ and > /first-class/. > > Perhaps you could explain what you mean by a /first-class/ utility? > > -- > ~Ethan~ > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From eliben at gmail.com Wed Mar 18 04:29:43 2015 From: eliben at gmail.com (Eli Bendersky) Date: Tue, 17 Mar 2015 20:29:43 -0700 Subject: [Python-ideas] gofmt for Python: standardized styling as a language feature In-Reply-To: References: <5508DBB0.2010209@stoneleaf.us> Message-ID: Interesting coincidence :-) We (at Google) are just on the verge of releasing an open-source tool for auto-formatting of Python code. It should be posted to Github within the next few days or weeks, under a permissive license (Apache 2.0 or something similar). The tool works like gofmt and clang-format (the C++/JS version provided as part of the Clang toolchain) - it only acts on whitespace and performs re-formatting of code (PEP-8 oriented for Python but it's configurable). It does not rename variables/classes/functions to conform to any particular style. The algorithm is based on clang-format, and lib2to3 is used for the parsing side - so it's very solid. We use it on Python 2 code internally, but Python 3 support is also planned and should be easy. We intend to keep actively developing the tool in the open and will be happy to accept test cases, fixes and patches in general. So stay tuned! P.S. I'm also against such a tool being a part of the Python distribution, at least in the near term. A tool/library has to first prove itself outside of core for a while, and only when/if it proves to be widely used and stable, an incorporation into core should be considered. The stdlib is big as it is, and any additional code is a burden on all core developers. Incorporation into the stdlib also imposes a very slow pace of development/release on said library/tool, and hence should only be considered when it's been extremely stable for a while. Eli On Tue, Mar 17, 2015 at 7:17 PM, Nicholas Chammas < nicholas.chammas at gmail.com> wrote: > First class means it's like a language feature. Actually, in my case I am > using "first-class" and "included with the language" synonymously. Not sure > if that's appropriate terminology, but that's what I mean. > > To explain by rough analogy, Twisted is to asyncio what autopep8 would be > to a first-class auto-styler. (I say "rough" because I understand that > Twisted is not supposed to be replaced entirely by asyncio, but I hope my > point is clear.) > > Nick > > On Tue, Mar 17, 2015 at 9:58 PM Ethan Furman wrote: > >> On 03/17/2015 06:54 PM, Nicholas Chammas wrote: >> >> > I am not proposing putting |autopep8| in the standard library. I am >> proposing that an auto-styler be included with >> > Python as a first-class utility. The emphasis here is on /included/ and >> /first-class/. >> >> Perhaps you could explain what you mean by a /first-class/ utility? >> >> -- >> ~Ethan~ >> >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From tritium-list at sdamon.com Wed Mar 18 04:42:39 2015 From: tritium-list at sdamon.com (Alexander Walters) Date: Tue, 17 Mar 2015 23:42:39 -0400 Subject: [Python-ideas] gofmt for Python: standardized styling as a language feature In-Reply-To: References: <5508DBB0.2010209@stoneleaf.us> Message-ID: <5508F42F.8020604@sdamon.com> What coding style would be used for such a tool? PEP-8? What about a tool for those of us who believe that PEP-8 got a few things wrong? What about for the tribal knowledge that pythonistas develop over time that are not part of the standard? Would such a tool include this? And what of third party tools that do the same thing? I have no evidence to support this, but I have the feeling that when the standard library includes something in a new space, third party work in that space slows (json and sqlite come to mind). Good coding style really should not be something we ship in the library, it should be something we teach. 2? On Tue, Mar 17, 2015 at 7:17 PM, Nicholas Chammas > wrote: > > First class means it's like a language feature. Actually, in my > case I am using "first-class" and "included with the language" > synonymously. Not sure if that's appropriate terminology, but > that's what I mean. > > To explain by rough analogy, Twisted is to asyncio what autopep8 > would be to a first-class auto-styler. (I say "rough" because I > understand that Twisted is not supposed to be replaced entirely by > asyncio, but I hope my point is clear.) > > Nick > > On Tue, Mar 17, 2015 at 9:58 PM Ethan Furman > wrote: > > On 03/17/2015 06:54 PM, Nicholas Chammas wrote: > > > I am not proposing putting |autopep8| in the standard > library. I am proposing that an auto-styler be included with > > Python as a first-class utility. The emphasis here is on > /included/ and /first-class/. > > Perhaps you could explain what you mean by a /first-class/ > utility? > > -- > ~Ethan~ > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > > > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From ianlee1521 at gmail.com Wed Mar 18 07:33:32 2015 From: ianlee1521 at gmail.com (Ian Lee) Date: Tue, 17 Mar 2015 23:33:32 -0700 Subject: [Python-ideas] Evolution of Style Guides Message-ID: tl;dr -- How should style guides evolve? So this is a post inspired largely by a thread that popped up today [1], and specifically Alex's message [2]. I didn't want to derail that conversation (specifically dealing with style checking / correcting as a part of the standard library), but I did want to start a separate thread for an idea that's been rattling around in my brain for a while now. (Disclaimer, I am the current lead maintainer of the pep8 package, so part of this comes from my work there.) Namely, how should style guides, and here I'm particularly looking at PEP-8, evolve? I was influenced early on in my Python learning by a number of youtube Python conference talks. In particular "A Python ?sthetic: Beauty and Why I Python" by Brandon Rhodes [3] and "Transforming Code into Beautiful, Idiomatic Python" by Raymond Hettinger [4] had quite an impact on my thinking while I was moving from C / C++ to Python a couple of years back, and I highly recommend them. Somewhat of an open question, I'm not sure I have a particularly hard and fast opinion on this issue, and I'd love to get feedback from the community. In particular, a couple of points that are worth mentioning: * There are a number of checks that are disabled in pep8 by default, due to their being somewhat controversial for many users [5] which you can read about what each of them are for in the docs [6] if you're interested. * Even within the checker community, there is some discussion about whether the tool should be rigidly set by the PEP-8 document (I tend to think it should be) and what that means for folks that don't want to follow a rule (I just disable it in my code) (see discussion about the use of ``isinstance()`` at [7]). * There are several large code bases, the Python stdlib and Twisted for starters, that don't necessarily follow PEP-8 to the letter, due to pre-dating PEP-8 or being highly specialized, or etc, and proposals to bring them more in line have received pushback over their potential for more harm than good (see [8] for an example) (which is completely valid). [1] https://mail.python.org/pipermail/python-ideas/2015-March/032545.html [2] https://mail.python.org/pipermail/python-ideas/2015-March/032556.html [3] https://www.youtube.com/watch?v=x-kB2o8sd5c [4] https://www.youtube.com/watch?v=OSGv2VnC0go [5] https://github.com/jcrocholl/pep8/blob/master/pep8.py#L68 [6] http://pep8.readthedocs.org/en/latest/intro.html#error-codes [7] https://github.com/jcrocholl/pep8/pull/313 [8] http://bugs.python.org/issue23061#msg233065 ~ Ian Lee -------------- next part -------------- An HTML attachment was scrubbed... URL: From ianlee1521 at gmail.com Wed Mar 18 07:38:24 2015 From: ianlee1521 at gmail.com (Ian Lee) Date: Tue, 17 Mar 2015 23:38:24 -0700 Subject: [Python-ideas] gofmt for Python: standardized styling as a language feature In-Reply-To: <5508F42F.8020604@sdamon.com> References: <5508DBB0.2010209@stoneleaf.us> <5508F42F.8020604@sdamon.com> Message-ID: ~ Ian Lee On Tue, Mar 17, 2015 at 8:42 PM, Alexander Walters wrote: > What coding style would be used for such a tool? PEP-8? What about a > tool for those of us who believe that PEP-8 got a few things wrong? > > What about for the tribal knowledge that pythonistas develop over time > that are not part of the standard? Would such a tool include this? > And perhaps more to the point, should PEP-8 (and if so how), evolve over time to incorporate newer ideas that the community has now moved towards accepting as the "more Pythonic way to do things" now with the benefit of additional hindsight. I didn't want to derail this conversation, so I spun off a separate thread to discuss: https://mail.python.org/pipermail/python-ideas/2015-March/032557.html > > And what of third party tools that do the same thing? I have no evidence > to support this, but I have the feeling that when the standard library > includes something in a new space, third party work in that space slows > (json and sqlite come to mind). > > Good coding style really should not be something we ship in the library, > it should be something we teach. > > 2? > > > On Tue, Mar 17, 2015 at 7:17 PM, Nicholas Chammas < > nicholas.chammas at gmail.com> wrote: > > First class means it's like a language feature. Actually, in my case >> I am using "first-class" and "included with the language" synonymously. Not >> sure if that's appropriate terminology, but that's what I mean. >> >> To explain by rough analogy, Twisted is to asyncio what autopep8 would >> be to a first-class auto-styler. (I say "rough" because I understand that >> Twisted is not supposed to be replaced entirely by asyncio, but I hope my >> point is clear.) >> >> Nick >> >> On Tue, Mar 17, 2015 at 9:58 PM Ethan Furman wrote: >> >>> On 03/17/2015 06:54 PM, Nicholas Chammas wrote: >>> >>> > I am not proposing putting |autopep8| in the standard library. I am >>> proposing that an auto-styler be included with >>> > Python as a first-class utility. The emphasis here is on /included/ >>> and /first-class/. >>> >>> Perhaps you could explain what you mean by a /first-class/ utility? >>> >>> -- >>> ~Ethan~ >>> >>> _______________________________________________ >>> Python-ideas mailing list >>> Python-ideas at python.org >>> https://mail.python.org/mailman/listinfo/python-ideas >>> Code of Conduct: http://python.org/psf/codeofconduct/ >> >> >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ >> > > > > _______________________________________________ > Python-ideas mailing listPython-ideas at python.orghttps://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From abarnert at yahoo.com Wed Mar 18 08:18:02 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Wed, 18 Mar 2015 00:18:02 -0700 Subject: [Python-ideas] Evolution of Style Guides In-Reply-To: References: Message-ID: On Mar 17, 2015, at 11:33 PM, Ian Lee wrote: > > > * There are a number of checks that are disabled in pep8 by default, due to their being somewhat controversial for many users [5] which you can read about what each of them are for in the docs [6] if you're interested. I don't know that the rules themselves are necessarily controversial, so much as the way the checks are implemented. I enabled E226 to see what it does (just because it sounded like the easiest one to test without reading further). It flags the "Yes" examples from the PEP ("c = (a+b) * (a-b)") as an error, and passes the corresponding "No" examples ("c = (a + b) * (a - b)"). No wonder it's controversial. What's going on? The rule it enforces is "always put whitespace around arithmetic operators", but the rule in the PEP is "If operators with different priorities are used, consider adding whitespace around the operators with the lowest priority(ies)." But how exactly can an automated checker "consider" doing something? Clearly always doing it is not what's intended. Even besides the examples, in a document that starts off by telling you it's only a guideline and should not be followed slavishly or rigidly, any part that goes out of its way to say "consider" and follow with "use your own judgment" must have a reason to do so. So, your judgment can tell you that the No example has too much whitespace. But the pep8 code can't.* And this is, presumably, why the E226 code has no problem with it, which in turn is presumably why users have a problem with the E226 code. So, I don't think the fact that some of the checks are disabled by default necessarily means there's any dissonance between PEP8 and the Python community, it just means there are things in PEP8 that, though worth following, are hard to codify as rigidly checkable rules. * Maybe "can't" is a bit strong. But consider: Would your judgment be the same if the variables had longer names? If this were one line of a complicated multi-line expression? Even if you knew the rules that provided the answers to those questions, would you want to code them and turn pep8 into a massive 80s-style expert system? From me at the-compiler.org Wed Mar 18 08:30:04 2015 From: me at the-compiler.org (Florian Bruhin) Date: Wed, 18 Mar 2015 08:30:04 +0100 Subject: [Python-ideas] Evolution of Style Guides In-Reply-To: References: Message-ID: <20150318073004.GJ10871@tonks> * Andrew Barnert [2015-03-18 00:18:02 -0700]: > On Mar 17, 2015, at 11:33 PM, Ian Lee wrote: > > > > > > * There are a number of checks that are disabled in pep8 by default, due to their being somewhat controversial for many users [5] which you can read about what each of them are for in the docs [6] if you're interested. > > I don't know that the rules themselves are necessarily controversial, so much as the way the checks are implemented. > > I enabled E226 to see what it does (just because it sounded like the easiest one to test without reading further). > > It flags the "Yes" examples from the PEP ("c = (a+b) * (a-b)") > as an error, and passes the corresponding "No" examples ("c = (a + b) * (a - b)"). No wonder it's controversial. > > What's going on? The rule it enforces is "always put whitespace around arithmetic operators", but the rule in the PEP is "If operators with different priorities are used, consider adding whitespace around the operators with the lowest priority(ies)." That was an addition made "recently" to pep8: https://hg.python.org/peps/diff/37af28ad2972/pep-0008.txt I guess pep8 adopted by disabling that check by default, rather than completely removing it. Florian -- http://www.the-compiler.org | me at the-compiler.org (Mail/XMPP) GPG: 916E B0C8 FD55 A072 | http://the-compiler.org/pubkey.asc I love long mails! | http://email.is-not-s.ms/ -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 819 bytes Desc: not available URL: From abarnert at yahoo.com Wed Mar 18 09:48:29 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Wed, 18 Mar 2015 01:48:29 -0700 Subject: [Python-ideas] Evolution of Style Guides In-Reply-To: <20150318073004.GJ10871@tonks> References: <20150318073004.GJ10871@tonks> Message-ID: <08381887-47D4-4B1D-A326-9F7EF5D7F703@yahoo.com> On Mar 18, 2015, at 12:30 AM, Florian Bruhin wrote: > > * Andrew Barnert [2015-03-18 00:18:02 -0700]: >>> On Mar 17, 2015, at 11:33 PM, Ian Lee wrote: >>> >>> >>> * There are a number of checks that are disabled in pep8 by default, due to their being somewhat controversial for many users [5] which you can read about what each of them are for in the docs [6] if you're interested. >> >> I don't know that the rules themselves are necessarily controversial, so much as the way the checks are implemented. >> >> I enabled E226 to see what it does (just because it sounded like the easiest one to test without reading further). >> >> It flags the "Yes" examples from the PEP ("c = (a+b) * (a-b)") >> as an error, and passes the corresponding "No" examples ("c = (a + b) * (a - b)"). No wonder it's controversial. >> >> What's going on? The rule it enforces is "always put whitespace around arithmetic operators", but the rule in the PEP is "If operators with different priorities are used, consider adding whitespace around the operators with the lowest priority(ies)." > > That was an addition made "recently" to pep8: > https://hg.python.org/peps/diff/37af28ad2972/pep-0008.txt > > I guess pep8 adopted by disabling that check by default, rather than > completely removing it. So PEP 8 evolved to match changing community standards, maybe even in part because of community experience with the pep8 tool, and the tool adapted to those changes... So what needs to change here? (If Ian was arguing that pep8, autopep8, etc. should remain outside the stdlib so they can continue to adapt at a reasonable pace, then I think I agree, but I think he was suggesting something different; I'll go re-read the post, and apologies in advance if I'm wrong.) From steve at pearwood.info Wed Mar 18 09:52:46 2015 From: steve at pearwood.info (Steven D'Aprano) Date: Wed, 18 Mar 2015 19:52:46 +1100 Subject: [Python-ideas] gofmt for Python: standardized styling as a language feature In-Reply-To: References: <63A4AF9C-5CD1-407B-8F20-3697EC62466D@yahoo.com> Message-ID: <20150318085246.GB23538@ando.pearwood.info> On Wed, Mar 18, 2015 at 12:55:17AM +0000, Nicholas Chammas wrote: > This is my first time posting to python-ideas. (I subscribed today.) I > gather from Andrew and Steven's responses that ideas discussed on this list > are supposed to be more "fully baked" than my proposal here. My apologies. Not necessarily. This list frequently includes half-baked ideas, but the "bigger" the proposal, the more baked it should be to be taken seriously. I can proposal that Python should include a fully-functioning self-aware computer AI, but unless I actually have a fully-functioning self-aware AI written in Python to include, I might as well just wish for a flying pony. On the other hand, "Add math.zero through math.ten constants" doesn't need anything but a proposal to be discussed. Still, occassionally its useful to make a really big proposal. Even if it doesn't end up being accepted, it might lead somewhere interesting. -- Steve From mal at egenix.com Wed Mar 18 09:58:10 2015 From: mal at egenix.com (M.-A. Lemburg) Date: Wed, 18 Mar 2015 09:58:10 +0100 Subject: [Python-ideas] Evolution of Style Guides In-Reply-To: References: Message-ID: <55093E22.7000200@egenix.com> On 18.03.2015 07:33, Ian Lee wrote: > tl;dr -- How should style guides evolve? > > So this is a post inspired largely by a thread that popped up today [1], > and specifically Alex's message [2]. I didn't want to derail that > conversation (specifically dealing with style checking / correcting as a > part of the standard library), but I did want to start a separate thread > for an idea that's been rattling around in my brain for a while now. > (Disclaimer, I am the current lead maintainer of the pep8 package, so part > of this comes from my work there.) > > Namely, how should style guides, and here I'm particularly looking at > PEP-8, evolve? By revisiting them every now and them :-) I think that people sometimes miss the "guide" aspect of PEP 8. PEP 8 is a guideline for people to follow which results in readable code. It's not a the only such guideline. Many companies have their own which usually build on PEP 8 and then add some extra rules or modify PEP 8 rules. Whether or not to use a tool enforcing a guideline like PEP 8 is really up to the user or the programming context. I'm -1 on having a go like feature in Python which mandates one particular style, but absolutely nothing against tools applying a formatting specification to some Python code on demand :-) -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Mar 18 2015) >>> Python Projects, Coaching and Consulting ... http://www.egenix.com/ >>> mxODBC Plone/Zope Database Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ 2015-03-12: Released mxODBC 3.3.2 ... http://egenix.com/go71 ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From encukou at gmail.com Wed Mar 18 11:53:52 2015 From: encukou at gmail.com (Petr Viktorin) Date: Wed, 18 Mar 2015 11:53:52 +0100 Subject: [Python-ideas] A macro for easier rich comparisons Message-ID: <55095940.4010707@gmail.com> Hello, Rich comparison functions of many builtin types include a block of boilerplate code with a switch on the "op" argument, like this example from tupleobject: int cmp; PyObject *res; switch (op) { case Py_LT: cmp = vlen < wlen; break; case Py_LE: cmp = vlen <= wlen; break; case Py_EQ: cmp = vlen == wlen; break; case Py_NE: cmp = vlen != wlen; break; case Py_GT: cmp = vlen > wlen; break; case Py_GE: cmp = vlen >= wlen; break; default: return NULL; /* cannot happen */ } if (cmp) res = Py_True; else res = Py_False; Py_INCREF(res); return res; Each time it's implemented slightly differently: sometimes there are two values to compare, sometimes there's a cmp-style 0/1/-1 result to process; invalid "op" values are handled in various ways. Such code is also required in third-party extensions. I propose adding a public macro to ease this. My version takes two C-orderable values and the operation, similar to richcmpfunc: #define Py_RICHCOMPARE(val1, val2, op) ( \ ((op) == Py_EQ) ? PyBool_FromLong((val1) == (val2)) : \ ((op) == Py_NE) ? PyBool_FromLong((val1) != (val2)) : \ ((op) == Py_LT) ? PyBool_FromLong((val1) < (val2)) : \ ((op) == Py_GT) ? PyBool_FromLong((val1) > (val2)) : \ ((op) == Py_LE) ? PyBool_FromLong((val1) <= (val2)) : \ ((op) == Py_GE) ? PyBool_FromLong((val1) >= (val2)) : \ (Py_INCREF(Py_NotImplemented), Py_NotImplemented)) (As for the behavior for unknown op: for most cases the best thing to do is setting an error and returning NULL, but that doesn't fit into a macro. A surprising number of richcmpfunc's in CPython either return NULL without error (e.g. tupleobject), or fall through and return True/False arbitrarily (as in bytearrayobject). Datetime does an assert. I think Py_NotImplemented is a good value to return) I've made a patch with the macro and its use in CPython itself. Read it as examples of usage: I'm not saying that all these modules should start using it immediately (or ever, e.g. for speed reasons), and if they do, many could use some refactoring in other code paths. The patch is here: https://github.com/encukou/cpython/compare/master...richcmp Why I'm interested in this: When porting old code to Python 3, rich comparisons a major cause of added boilerplate ? especially when wrapping a C library that exposes either trivially comparable types or "cmp" functions. With this macro, a typical trivial rich comparison function reduces to: static PyObject* mytype_richcmp(PyObject *obj1, PyObject *obj2, int op) { if (mytype_Check(obj1) && mytype_Check(obj2)) { return PY_RICHCOMPARE(mytype_get_data(obj1), mytype_get_data(obj2), op); } Py_RETURN_NOTIMPLEMENTED; } ... or with `PY_RICHCOMPARE(mytype_cmp(obj1, obj2), 0, op)`. (Note that even adding this to 3.5/3.6, would help in porting efforts: the macro itself is be easy to backport, but there are big benefits in having a standard name, known semantics, and official documentation.) Is this a PEP-worthy idea? From mal at egenix.com Wed Mar 18 12:01:07 2015 From: mal at egenix.com (M.-A. Lemburg) Date: Wed, 18 Mar 2015 12:01:07 +0100 Subject: [Python-ideas] A macro for easier rich comparisons In-Reply-To: <55095940.4010707@gmail.com> References: <55095940.4010707@gmail.com> Message-ID: <55095AF3.6080201@egenix.com> On 18.03.2015 11:53, Petr Viktorin wrote: > ... > I propose adding a public macro to ease this. My version takes two C-orderable values and the > operation, similar to richcmpfunc: > > #define Py_RICHCOMPARE(val1, val2, op) ( \ > ((op) == Py_EQ) ? PyBool_FromLong((val1) == (val2)) : \ > ((op) == Py_NE) ? PyBool_FromLong((val1) != (val2)) : \ > ((op) == Py_LT) ? PyBool_FromLong((val1) < (val2)) : \ > ((op) == Py_GT) ? PyBool_FromLong((val1) > (val2)) : \ > ((op) == Py_LE) ? PyBool_FromLong((val1) <= (val2)) : \ > ((op) == Py_GE) ? PyBool_FromLong((val1) >= (val2)) : \ > (Py_INCREF(Py_NotImplemented), Py_NotImplemented)) > > (As for the behavior for unknown op: for most cases the best thing to do is setting an error and > returning NULL, but that doesn't fit into a macro. A surprising number of richcmpfunc's in CPython > either return NULL without error (e.g. tupleobject), or fall through and return True/False > arbitrarily (as in bytearrayobject). Datetime does an assert. > I think Py_NotImplemented is a good value to return) > ... > Is this a PEP-worthy idea? This is a great idea, no need for a PEP :-) Please submit the patch on the bug tracker and also include a patch for the C API documentation. Thanks, -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Mar 18 2015) >>> Python Projects, Coaching and Consulting ... http://www.egenix.com/ >>> mxODBC Plone/Zope Database Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ 2015-03-12: Released mxODBC 3.3.2 ... http://egenix.com/go71 ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From luciano at ramalho.org Wed Mar 18 12:24:24 2015 From: luciano at ramalho.org (Luciano Ramalho) Date: Wed, 18 Mar 2015 08:24:24 -0300 Subject: [Python-ideas] .from and .to instead of .encode and .decode In-Reply-To: References: Message-ID: On Wed, Mar 18, 2015 at 2:54 AM, anatoly techtonik wrote: >> My mnemonics are "decode from cryptic bytes to human text" and "encode >> from human text to cryptic bytes". > > Maybe I am thinking too low level, but there is no such thing as "human text" in > computer memory. But I agree that using the words like "abstraction" is not a > good idea, because Python is used by children of age 7, and things should be > simpler. It's a didactic mnemonic ;-). Although str objects hold codepoints, they *are* used to represent human text. On the other hand, we convert str to bytes when we need to talk to the machine: writing to disk, sending over the network... > >> A more objective problem is: "from" is a keyword, so it can't be a method name. > > I don't see why "from" can not be used as a method name in Python. It can not > be used as function name, because it will override the keyword, but method > names are always namespaced with self. or class prefixes. You can't write it's definition, "def from..." and you can't even write "x.from" without a SyntaxError (instead of NameError or AttributeError). Try it in the Python console. The parser would need to be changed. Best, Luciano -- Luciano Ramalho | Author of Fluent Python (O'Reilly, 2015) | http://shop.oreilly.com/product/0636920032519.do | Professor em: http://python.pro.br | Twitter: @ramalhoorg From luciano at ramalho.org Wed Mar 18 12:36:23 2015 From: luciano at ramalho.org (Luciano Ramalho) Date: Wed, 18 Mar 2015 08:36:23 -0300 Subject: [Python-ideas] A mutable alternative to namedtuple In-Reply-To: References: Message-ID: On Wed, Mar 18, 2015 at 3:05 AM, anatoly techtonik wrote: > On Tue, Mar 17, 2015 at 7:52 PM, Luciano Ramalho wrote: >> Sometimes we need a simple class to hold some mutable attributes, >> provide a nice repr, support == for testing, and support iterable >> unpacking > > +1, but I think that the core problem with such proposals is that they > lack the use cases. The only reason for me to have such class is to > work with tabular data. > > For example, for querying the capability of the system, I need to build > an inmemory table of features, and then set parameters for each > feature one by one. sqlite/SQL is an overkill for that, and dicts are > just not enough to do readable lookups and updates to specific > cells, so I'd more appreciate a full table class than just its "named > row" model. Thanks for your input and example, Anatoly. What would the full table class offer that could not be easily done with a simple list of items produced with a plainclass class? > Practical example that I came up with: > https://bitbucket.org/techtonik/discovery/src/c4f3d306bb43772dcf3c03be8db941ff67d077b4/graphics/canvas2d/canvas2d/__init__.py?at=default#cl-22 Oh, I see you have an update method, so that's the key value add of the Table class, right? I see the Table and plainclass as complementary ideas. You could use plainclass to implement Table more easily. The Table.__iter__ could return plainclass class instances instead of OrderedDict. Cheers, Luciano -- Luciano Ramalho | Author of Fluent Python (O'Reilly, 2015) | http://shop.oreilly.com/product/0636920032519.do | Professor em: http://python.pro.br | Twitter: @ramalhoorg From encukou at gmail.com Wed Mar 18 14:48:46 2015 From: encukou at gmail.com (Petr Viktorin) Date: Wed, 18 Mar 2015 14:48:46 +0100 Subject: [Python-ideas] A macro for easier rich comparisons In-Reply-To: <55095AF3.6080201@egenix.com> References: <55095940.4010707@gmail.com> <55095AF3.6080201@egenix.com> Message-ID: On Wed, Mar 18, 2015 at 12:01 PM, M.-A. Lemburg wrote: > On 18.03.2015 11:53, Petr Viktorin wrote: >> ... >> I propose adding a public macro to ease this. My version takes two C-orderable values and the >> operation, similar to richcmpfunc: >> >> #define Py_RICHCOMPARE(val1, val2, op) ( \ >> ((op) == Py_EQ) ? PyBool_FromLong((val1) == (val2)) : \ >> ((op) == Py_NE) ? PyBool_FromLong((val1) != (val2)) : \ >> ((op) == Py_LT) ? PyBool_FromLong((val1) < (val2)) : \ >> ((op) == Py_GT) ? PyBool_FromLong((val1) > (val2)) : \ >> ((op) == Py_LE) ? PyBool_FromLong((val1) <= (val2)) : \ >> ((op) == Py_GE) ? PyBool_FromLong((val1) >= (val2)) : \ >> (Py_INCREF(Py_NotImplemented), Py_NotImplemented)) >> >> (As for the behavior for unknown op: for most cases the best thing to do is setting an error and >> returning NULL, but that doesn't fit into a macro. A surprising number of richcmpfunc's in CPython >> either return NULL without error (e.g. tupleobject), or fall through and return True/False >> arbitrarily (as in bytearrayobject). Datetime does an assert. >> I think Py_NotImplemented is a good value to return) >> ... >> Is this a PEP-worthy idea? > > This is a great idea, no need for a PEP :-) Please submit the patch on > the bug tracker and also include a patch for the C API documentation. Issue with patches is here: http://bugs.python.org/issue23699 From techtonik at gmail.com Wed Mar 18 06:11:46 2015 From: techtonik at gmail.com (anatoly techtonik) Date: Wed, 18 Mar 2015 08:11:46 +0300 Subject: [Python-ideas] gofmt for Python: standardized styling as a language feature In-Reply-To: <5508F42F.8020604@sdamon.com> References: <5508DBB0.2010209@stoneleaf.us> <5508F42F.8020604@sdamon.com> Message-ID: On Wed, Mar 18, 2015 at 6:42 AM, Alexander Walters wrote: > What coding style would be used for such a tool? PEP-8? What about a tool > for those of us who believe that PEP-8 got a few things wrong? +1 to make it optional. I use 2 space indent for most of my code. I'd also add that policy to enforce rigid rules is harmful. The recent normalization of Python package versions made some of my apps that use semantic version invalid for PyPI. From techtonik at gmail.com Wed Mar 18 06:33:52 2015 From: techtonik at gmail.com (anatoly techtonik) Date: Wed, 18 Mar 2015 08:33:52 +0300 Subject: [Python-ideas] math.tau In-Reply-To: References: Message-ID: On Mon, Mar 16, 2015 at 6:15 PM, Victor Stinner wrote: > It looks like you forgot to write the body of your email, there is no > text, only code? There is video, which is a much better media for me. For me the reason to add tau is that Python feels too boring now that Monty Python is passed away and stdlib doesn't shine in comparison with awesome-python too. -- anatoly t. From techtonik at gmail.com Wed Mar 18 06:54:51 2015 From: techtonik at gmail.com (anatoly techtonik) Date: Wed, 18 Mar 2015 08:54:51 +0300 Subject: [Python-ideas] .from and .to instead of .encode and .decode In-Reply-To: References: Message-ID: On Tue, Mar 17, 2015 at 7:10 PM, Luciano Ramalho wrote: > On Mon, Mar 16, 2015 at 10:09 AM, anatoly techtonik wrote: >> I use the analogy of "encoding abstract numbers into specific bytes" myself. > > Your version of the analogy doesn't sound as helpful to me; "abastract > numbers" sounds even more cryptic than bytes. > > My mnemonics are "decode from cryptic bytes to human text" and "encode > from human text to cryptic bytes". Maybe I am thinking too low level, but there is no such thing as "human text" in computer memory. But I agree that using the words like "abstraction" is not a good idea, because Python is used by children of age 7, and things should be simpler. > A more objective problem is: "from" is a keyword, so it can't be a method name. I don't see why "from" can not be used as a method name in Python. It can not be used as function name, because it will override the keyword, but method names are always namespaced with self. or class prefixes. -- anatoly t. From techtonik at gmail.com Wed Mar 18 07:05:12 2015 From: techtonik at gmail.com (anatoly techtonik) Date: Wed, 18 Mar 2015 09:05:12 +0300 Subject: [Python-ideas] A mutable alternative to namedtuple In-Reply-To: References: Message-ID: On Tue, Mar 17, 2015 at 7:52 PM, Luciano Ramalho wrote: > Sometimes we need a simple class to hold some mutable attributes, > provide a nice repr, support == for testing, and support iterable > unpacking +1, but I think that the core problem with such proposals is that they lack the use cases. The only reason for me to have such class is to work with tabular data. For example, for querying the capability of the system, I need to build an inmemory table of features, and then set parameters for each feature one by one. sqlite/SQL is an overkill for that, and dicts are just not enough to do readable lookups and updates to specific cells, so I'd more appreciate a full table class than just its "named row" model. Practical example that I came up with: https://bitbucket.org/techtonik/discovery/src/c4f3d306bb43772dcf3c03be8db941ff67d077b4/graphics/canvas2d/canvas2d/__init__.py?at=default#cl-22 -- anatoly t. From techtonik at gmail.com Wed Mar 18 07:19:27 2015 From: techtonik at gmail.com (anatoly techtonik) Date: Wed, 18 Mar 2015 09:19:27 +0300 Subject: [Python-ideas] Rewriting the build system (part 2) In-Reply-To: References: Message-ID: On Mon, Mar 16, 2015 at 8:18 PM, Ryan Gonzalez wrote: > They have a build system genius who wrote Tup working for them I like the conceptual brevity of the Tup page http://gittup.org/tup/ It is the core of that I am try to bring from SCons internals to the surface. It is very often that you need to process some DAG and touch nodes based on their changed dependencies. The problem is so common that it is even worth to spend time on visualization for it. So far no build system has this stuff, and I run of time to complete the SCons plumbing. -- anatoly t. From techtonik at gmail.com Wed Mar 18 07:21:23 2015 From: techtonik at gmail.com (anatoly techtonik) Date: Wed, 18 Mar 2015 09:21:23 +0300 Subject: [Python-ideas] Rewriting the build system (part 2) In-Reply-To: References: Message-ID: On Mon, Mar 16, 2015 at 8:39 PM, Chris Barker wrote: > One more note: so adding a python dependency is a pain. But the current > system requires separate stems for Windows and *nix-y systems, and who knows > what for hypothetical systems that don't support autotools. So maybe a > python-based build system that worked on all the common platforms, with an > additional makefile that you have to hand-mangle to bootstrap a new platform > isn't any more to maintain. Manual instruction with lists of files that can be parsed out of it is better than Makefile IMO. Pumped up Markdown or something.. -- anatoly t. From techtonik at gmail.com Wed Mar 18 08:18:26 2015 From: techtonik at gmail.com (anatoly techtonik) Date: Wed, 18 Mar 2015 10:18:26 +0300 Subject: [Python-ideas] Rewriting the build system (part 2) In-Reply-To: References: Message-ID: On Mon, Mar 16, 2015 at 7:31 PM, Chris Barker wrote: > On Mon, Mar 16, 2015 at 6:24 AM, anatoly techtonik > wrote: >> >> "Autotools has reached the Chandrasekhar mass limit of software and >> become a black hole of suck that consumes endless hours of bright >> peoples? time without any good result. It is well past time for it to >> die." (c) "Autotools must die", ESR. http://esr.ibiblio.org/?p=1877 >> >> I couldn't sum up this better. It looks even worse from Windows machine. > > Is it even used for building Python on Windows (for the "normal" python.org > builds)? There was some success with MinGW toolchain, but I am not tracking if it was integrated into official source. The FUD was that compiled extensions are not compatible between MSVC builds and MinGW builds, but I could not confirm that. If I understand correctly, the incompatibility is only for C++ programs, because of strange SEH strategy and proprietary name mangling. > But to miss-paraphrase Winston Churchill [1]: > > "Autotools is the worst build system except for all the others." > > Is there really something better out there? A lot of attempts are made, but reviewing them requires a lot of time and effort. From techtonik at gmail.com Wed Mar 18 08:21:46 2015 From: techtonik at gmail.com (anatoly techtonik) Date: Wed, 18 Mar 2015 10:21:46 +0300 Subject: [Python-ideas] Rewriting the build system (part 2) In-Reply-To: References: Message-ID: On Mon, Mar 16, 2015 at 11:33 PM, Ryan Gonzalez wrote: >> >> But to miss-paraphrase Winston Churchill [1]: >> >> "Autotools is the worst build system except for all the others." >> >> Is there really something better out there? >> > > Fbuild, Waf, Meson, GN, Boost.Build, Tundra, Bam, ... Is there anything Javascriptey? Python needs to move to web. I heard something about Grunt, which was replaced by Gulp, but no idea if they really mature enough to include "system features detection" and "user configured builds" (or work with C/C++ compilers at all). From techtonik at gmail.com Wed Mar 18 08:31:37 2015 From: techtonik at gmail.com (anatoly techtonik) Date: Wed, 18 Mar 2015 10:31:37 +0300 Subject: [Python-ideas] Rewriting the build system (part 2) In-Reply-To: References: Message-ID: On Tue, Mar 17, 2015 at 2:45 AM, Chris Barker wrote: > > This doesn't seem to me to be a winner takes all proposition -- there is no > reason someone couldn't set up a new build system for Python, get it all > working nicely on their platform(s) of choice, persuade others how wonderful > it is, get contributions, etc. > > If it turns out to really be better/easier etc. than autoconf, then , and > only then, would it make any sense to talk about replacing the existing > official build system. And even then, the two could live in parallel for > quite some time... Both Blender and Wesnoth have SCons and CMake running in parallel for a long time. Both camps are just maintaining their own chains and it works well. From techtonik at gmail.com Wed Mar 18 08:36:31 2015 From: techtonik at gmail.com (anatoly techtonik) Date: Wed, 18 Mar 2015 10:36:31 +0300 Subject: [Python-ideas] Rewriting the build system (part 2) In-Reply-To: <87wq2g73zq.fsf@uwakimon.sk.tsukuba.ac.jp> References: <87wq2g73zq.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: On Tue, Mar 17, 2015 at 5:27 AM, Stephen J. Turnbull wrote: > systems out there with the accumulated knowledge of platforms that has > been assembled in Autotools over the decades. Given that Autotools still doesn't have native Windows support, the value of that knowledge is overstated. From techtonik at gmail.com Wed Mar 18 08:44:04 2015 From: techtonik at gmail.com (anatoly techtonik) Date: Wed, 18 Mar 2015 10:44:04 +0300 Subject: [Python-ideas] Rewriting the build system (part 2) In-Reply-To: References: <87wq2g73zq.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: On Tue, Mar 17, 2015 at 8:01 PM, Ryan Gonzalez wrote: > On Mon, Mar 16, 2015 at 9:27 PM, Stephen J. Turnbull > wrote: >> >> Ryan Gonzalez writes: >> >> > I have used fbuild a LOT >> >> Actually, I suspect that in the relevant sense, you haven't used it a >> lot. Have you used it for a single program on >> >> 1. more than 3 CPUs, AND >> 2. on each CPU, more than one (and preferably more than two) OSes, AND >> 3. for each CPU-OS combination, more than two different configurations >> of the program being built, AND >> 4. for each CPU-OS-configuration combination, at least 3 versions of >> the program source (preferably spanning a major version bump)? >> > > No...but I know that Felix uses it for iPhone, OSX, Linux, ... with > separation of host and target and multiple configurations. What or who is Felix? > This is also SCons. Everybody loved it until they had to maintain it. :) I know a blind girl, who made a voice synthesizer, which works on Windows, Linux and Android, and she used SCons to build it. That gave me another reason to stick with this build system and try to enhance it. > SCons is very annoying. CMake is weird and only really useful for those > huge, 20,000 file C++ projects. For me the advantage of SCons is that I already know its codebase. There a lot of room for improvement if you can tag yourself as a Python hacker it addition to inherent potential user property. In any, I'd appreciate the feedback on most annoying SCons feature, because it will give a criteria for comparing it with others and for future development. -- anatoly t. From me at the-compiler.org Wed Mar 18 16:21:50 2015 From: me at the-compiler.org (Florian Bruhin) Date: Wed, 18 Mar 2015 16:21:50 +0100 Subject: [Python-ideas] Rewriting the build system (part 2) In-Reply-To: References: Message-ID: <20150318152150.GN10871@tonks> * anatoly techtonik [2015-03-18 10:21:46 +0300]: > On Mon, Mar 16, 2015 at 11:33 PM, Ryan Gonzalez wrote: > >> > >> But to miss-paraphrase Winston Churchill [1]: > >> > >> "Autotools is the worst build system except for all the others." > >> > >> Is there really something better out there? > >> > > > > Fbuild, Waf, Meson, GN, Boost.Build, Tundra, Bam, ... > > Is there anything Javascriptey? Python needs to move to web. I heard > something about Grunt, which was replaced by Gulp, but no idea if they > really mature enough to include "system features detection" and "user > configured builds" (or work with C/C++ compilers at all). What does "Python needs to move to web" to do with it's build system being "Javascriptey"? Florian -- http://www.the-compiler.org | me at the-compiler.org (Mail/XMPP) GPG: 916E B0C8 FD55 A072 | http://the-compiler.org/pubkey.asc I love long mails! | http://email.is-not-s.ms/ -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 819 bytes Desc: not available URL: From rymg19 at gmail.com Wed Mar 18 17:11:33 2015 From: rymg19 at gmail.com (Ryan Gonzalez) Date: Wed, 18 Mar 2015 11:11:33 -0500 Subject: [Python-ideas] Rewriting the build system (part 2) In-Reply-To: References: Message-ID: Then you need Node.js installed, which is kind of big. On Wed, Mar 18, 2015 at 2:21 AM, anatoly techtonik wrote: > On Mon, Mar 16, 2015 at 11:33 PM, Ryan Gonzalez wrote: > >> > >> But to miss-paraphrase Winston Churchill [1]: > >> > >> "Autotools is the worst build system except for all the others." > >> > >> Is there really something better out there? > >> > > > > Fbuild, Waf, Meson, GN, Boost.Build, Tundra, Bam, ... > > Is there anything Javascriptey? Python needs to move to web. I heard > something about Grunt, which was replaced by Gulp, but no idea if they > really mature enough to include "system features detection" and "user > configured builds" (or work with C/C++ compilers at all). > -- Ryan [ERROR]: Your autotools build scripts are 200 lines longer than your program. Something?s wrong. http://kirbyfan64.github.io/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From rymg19 at gmail.com Wed Mar 18 17:18:39 2015 From: rymg19 at gmail.com (Ryan Gonzalez) Date: Wed, 18 Mar 2015 11:18:39 -0500 Subject: [Python-ideas] Rewriting the build system (part 2) In-Reply-To: References: <87wq2g73zq.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: On Wed, Mar 18, 2015 at 2:44 AM, anatoly techtonik wrote: > On Tue, Mar 17, 2015 at 8:01 PM, Ryan Gonzalez wrote: > > On Mon, Mar 16, 2015 at 9:27 PM, Stephen J. Turnbull > > > wrote: > >> > >> Ryan Gonzalez writes: > >> > >> > I have used fbuild a LOT > >> > >> Actually, I suspect that in the relevant sense, you haven't used it a > >> lot. Have you used it for a single program on > >> > >> 1. more than 3 CPUs, AND > >> 2. on each CPU, more than one (and preferably more than two) OSes, AND > >> 3. for each CPU-OS combination, more than two different configurations > >> of the program being built, AND > >> 4. for each CPU-OS-configuration combination, at least 3 versions of > >> the program source (preferably spanning a major version bump)? > >> > > > > No...but I know that Felix uses it for iPhone, OSX, Linux, ... with > > separation of host and target and multiple configurations. > > What or who is Felix? > > A programming language. http://felix-lang.org/. The code base is pretty huge; since it comes with most of its dependencies (re2, Judy, etc.), it consists of about 1600 source files. Fbuild works pretty well, other than the fact that the language's creator is 100% positive that dynamic languages are cruel, evil, and unmaintainable. > > This is also SCons. Everybody loved it until they had to maintain it. :) > > I know a blind girl, who made a voice synthesizer, which works on > Windows, Linux and Android, and she used SCons to build it. That > gave me another reason to stick with this build system and try to > enhance it. > > > SCons is very annoying. CMake is weird and only really useful for those > > huge, 20,000 file C++ projects. > > For me the advantage of SCons is that I already know its codebase. > There a lot of room for improvement if you can tag yourself as a Python > hacker it addition to inherent potential user property. > > In any, I'd appreciate the feedback on most annoying SCons feature, > because it will give a criteria for comparing it with others and for future > development. > Ah, right...you work on SCons...please don't take anything I said personally. :) It's just that 99% of the time I try to build a project with SCons, some horribly stupid thing goes wrong because of some weird API call. Also, variant directories are WAY to weird for me. It's also kind of slow. I'm one of those people that gets impatient if I have more than 1.01s for my build to complete, which is why I don't really like make too much. > -- > anatoly t. > -- Ryan [ERROR]: Your autotools build scripts are 200 lines longer than your program. Something?s wrong. http://kirbyfan64.github.io/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From nicholas.chammas at gmail.com Wed Mar 18 17:52:07 2015 From: nicholas.chammas at gmail.com (Nicholas Chammas) Date: Wed, 18 Mar 2015 12:52:07 -0400 Subject: [Python-ideas] gofmt for Python: standardized styling as a language feature In-Reply-To: References: <5508DBB0.2010209@stoneleaf.us> <5508F42F.8020604@sdamon.com> Message-ID: On Tue, Mar 17, 2015 at 11:29 PM, Eli Bendersky wrote: > We (at Google) are just on the verge of releasing an open-source tool for > auto-formatting of Python code. Interesting. Are y'all familiar with `autopep8`? What are the differences between the tool y'all are releasing and `autopep8`? On Tue, Mar 17, 2015 at 11:29 PM, Eli Bendersky wrote: > P.S. I'm also against such a tool being a part of the Python distribution, > at least in the near term. A tool/library has to first prove itself outside > of core for a while, and only when/if it proves to be widely used and > stable, an incorporation into core should be considered. The stdlib is big > as it is, and any additional code is a burden on all core developers. > Incorporation into the stdlib also imposes a very slow pace of > development/release on said library/tool, and hence should only be > considered when it's been extremely stable for a while. This makes sense. In fact, I recently watched a keynote that Guido gave and he said something similar . On Tue, Mar 17, 2015 at 11:42 PM, Alexander Walters wrote: > What coding style would be used for such a tool? PEP-8? What about a > tool for those of us who believe that PEP-8 got a few things wrong? > > What about for the tribal knowledge that pythonistas develop over time > that are not part of the standard? Would such a tool include this? > I might be wrong, but I see PEP 8 as a living document. If the community believes parts of it are out of date, I would think it's better to update PEP 8 (or issue a replacement PEP) to reflect current thinking. Any auto-styling tool included in the standard library would work best if it was based on a written "spec" like PEP 8 (granted, I don't think PEP 8 was meant to be a spec for programmatic implementation, but you get the idea). If stuff is not part of PEP 8 that should be, or if parts of PEP 8 are agreed to be "wrong", then PEP 8 should be updated accordingly. Any auto-styling tool, I would hope, would just follow from PEP 8 (or a similar document). The tool should not be the specification. On Tue, Mar 17, 2015 at 11:42 PM, Alexander Walters wrote: > > And what of third party tools that do the same thing? I have no evidence > to support this, but I have the feeling that when the standard library > includes something in a new space, third party work in that space slows > (json and sqlite come to mind). > I see what you mean. Several people have made comments to this effect, and this is a good point. I guess this is a separate discussion, but is there some way to offer users the advantages of having a library be included with Python without imposing the 2-year release cycle and additional burden on the library's developers? This seems to be the root of the concern about including, for example, something like autopep8 with Python. On Tue, Mar 17, 2015 at 11:42 PM, Alexander Walters wrote: > Good coding style really should not be something we ship in the library, > it should be something we teach. > This point is orthogonal to the discussion of whether or not to package an auto-styler, but there are many style points that are so mechanical I see little benefit in expending energy on teaching them. Best would be to automate as much of it away so that people can focus on bigger problems like, what is the right abstraction, did I design this API well, etc. Put differently, consider the following straw man: "Good memory management should not be something we ship in the language, it should be something we teach." There is always something to teach. But when we can automate part of the problem away, I think we generally should. On Wed, Mar 18, 2015 at 2:38 AM, Ian Lee wrote: > And perhaps more to the point, should PEP-8 (and if so how), evolve over > time to incorporate newer ideas that the community has now moved towards > accepting as the "more Pythonic way to do things" now with the benefit of > additional hindsight. I consider PEP 8 to be a living document. That is, it is already changing with time. For example, I see that though PEP 8 was created in 2001, it was updated in 2013 . I assume it will continue to be updated as accepted standards change. On Wed, Mar 18, 2015 at 4:52 AM, Steven D'Aprano wrote: > Not necessarily. This list frequently includes half-baked ideas, but the > "bigger" the proposal, the more baked it should be to be taken > seriously. > Duly noted! Nick -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Wed Mar 18 18:31:13 2015 From: njs at pobox.com (Nathaniel Smith) Date: Wed, 18 Mar 2015 10:31:13 -0700 Subject: [Python-ideas] Rewriting the build system (part 2) In-Reply-To: References: Message-ID: On Mar 18, 2015 8:14 AM, "anatoly techtonik" wrote: > > On Mon, Mar 16, 2015 at 7:31 PM, Chris Barker wrote: > > On Mon, Mar 16, 2015 at 6:24 AM, anatoly techtonik > > wrote: > >> > >> "Autotools has reached the Chandrasekhar mass limit of software and > >> become a black hole of suck that consumes endless hours of bright > >> peoples? time without any good result. It is well past time for it to > >> die." (c) "Autotools must die", ESR. http://esr.ibiblio.org/?p=1877 > >> > >> I couldn't sum up this better. It looks even worse from Windows machine. > > > > Is it even used for building Python on Windows (for the "normal" python.org > > builds)? > > There was some success with MinGW toolchain, but I am not tracking if it was > integrated into official source. The FUD was that compiled extensions are not > compatible between MSVC builds and MinGW builds, but I could not confirm > that. If I understand correctly, the incompatibility is only for C++ programs, > because of strange SEH strategy and proprietary name mangling. This isn't FUD, alas. mingw in almost all of its variants uses effectively a different "libc" than MSVC, so code compiled with two compilers can't even agree on the meaning of file numbers. It is mostly possible to set up a mingw-w64 build that produces MSVC-compatible extensions, but it requires using a configuration that's undocumented and untested upstream (I.e. you will run into mingw bugs). NumPy *may* switch to this at some point -- we've been working on it for like a year now -- but it's extremely non-trivial. -n -------------- next part -------------- An HTML attachment was scrubbed... URL: From barry at python.org Wed Mar 18 19:22:02 2015 From: barry at python.org (Barry Warsaw) Date: Wed, 18 Mar 2015 14:22:02 -0400 Subject: [Python-ideas] Evolution of Style Guides References: <55093E22.7000200@egenix.com> Message-ID: <20150318142202.0e2c302d@anarchist.wooz.org> On Mar 18, 2015, at 09:58 AM, M.-A. Lemburg wrote: >PEP 8 is a guideline for people to follow which results >in readable code. It's not a the only such guideline. Many >companies have their own which usually build on PEP 8 and then >add some extra rules or modify PEP 8 rules. Exactly. Also PEP 8 is deliberately noncommittal about certain style points. It shouldn't dot every t and cross every i . Besides, PEP 8 was originally - and as its introduction still states - intended for the coding conventions of the standard library, where consistency is as important as it is lacking in practice. ;) You're supposed to define the coding standards for your own projects, and it's fine to include PEP 8 by reference. That's what we do for example with GNU Mailman (although it needs an update). Cheers, -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 819 bytes Desc: OpenPGP digital signature URL: From abarnert at yahoo.com Wed Mar 18 21:38:58 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Wed, 18 Mar 2015 13:38:58 -0700 Subject: [Python-ideas] gofmt for Python: standardized styling as a language feature In-Reply-To: References: <5508DBB0.2010209@stoneleaf.us> <5508F42F.8020604@sdamon.com> Message-ID: <5F528E38-D44A-4A7A-9FC2-25232F7D26AA@yahoo.com> On Mar 18, 2015, at 9:52 AM, Nicholas Chammas wrote: > > I might be wrong, but I see PEP 8 as a living document. If the community believes parts of it are out of date, I would think it's better to update PEP 8 (or issue a replacement PEP) to reflect current thinking. > > Any auto-styling tool included in the standard library would work best if it was based on a written "spec" like PEP 8 (granted, I don't think PEP 8 was meant to be a spec for programmatic implementation, but you get the idea). > > If stuff is not part of PEP 8 that should be, or if parts of PEP 8 are agreed to be "wrong", then PEP 8 should be updated accordingly. Any auto-styling tool, I would hope, would just follow from PEP 8 (or a similar document). The tool should not be the specification. I agree with all of this. But that implies that it's worth someone making a spec out of PEP 8. That implies going through PEP 8, getting a sense of what parts are (unambiguously and uncontroversially) mechanizable, and reorganizing and minimally rewriting it to get that sense across. Having numbered rules, and named anchors for each one in the document (or at least to narrower sections, so you can link to something more specific than "Programming Recommendations") would also be helpful. I'm not sure such a thing would be accepted if you did it. But it seems like if it were rejected, the standard styler would be no more "standard" than what already exists--and likely to be rejected for the same reasons. > On Tue, Mar 17, 2015 at 11:42 PM, Alexander Walters wrote: >> >> And what of third party tools that do the same thing? I have no evidence to support this, but I have the feeling that when the standard library includes something in a new space, third party work in that space slows (json and sqlite come to mind). > > I see what you mean. Several people have made comments to this effect, and this is a good point. > > I guess this is a separate discussion, but is there some way to offer users the advantages of having a library be included with Python without imposing the 2-year release cycle and additional burden on the library's developers? It depends what you mean. First, if pip comes with Python, and the library doesn't require C compilation, users already pretty much have that advantage. (And the really nice thing is that 2.7, 3.4, etc. users also have that advantage, which isn't true for something added to the stdlib in 3.5.) It might even be possible to have language in the official docs and/or the Getting Started or other parts of the site that suggest (as strongly as seems appropriate) that users "pip install autopep8" and use it. (I'm saying "autopep8" as a shorthand for "autopep8, or whatever alternative you'd end up suggesting if you did".) If the suggestion that IDLE add a graphical package manager wrapping pip goes forward, it could presumably add some kind of "recommended packages" feature, maybe even suggesting it at first run, but it seems like you'd have to jump-start the discussion among IDLE developers and users and then bring it back here, or wait and see if they add such a thing so you can suggest adding autopep8 to the list. Beyond that, there's really no way to be "included without being included". If it's in the standard source and binary distributions, it's part of the release cycle. I suppose you could have something that came with Python but could be updated with pip, with users strongly suggested to update it--but that doesn't seem like a good idea; if you want to get everyone to pip install -U it, the best way to do that is not to give them an obsolete version, but to just not give it to them. And bootstrapping it the same way as pip doesn't seem reasonable (it's reasonable for pip because "just pip install pip" obviously doesn't work--sometimes, special cases _are_ special enough...). One thing that might be reasonable is to include an autopep8 script that runs autopep8 if present, and otherwise just tells you it's not present and suggests pip installing it (similar to what Apple does with Java and X11 binaries). Of course nothing else in Python works that way today, and I suspect a lot of people would resist it, but technically it's certainly doable, if you want to make a case for it. Finally, remember that not everyone gets Python through an official Python.org installer. Almost every non-Windows, non-mobile platform comes with a "system python", and also has a package manager (possibly third-party, like Homebrew on Mac) that many people who want a different Python turn to. And some people build from source. And others (including on Windows) use third-party Python distributions like Enthought or PythonWin. To these users, what "comes with" Python is not the same thing that Python.org crams into the Windows and Mac installers--and can be much more up-to-date (but isn't automatically, of course). Apple's system Python comes with PyObjC. Some linux distros split off parts of the stdlib into packages named "python-spam" and then have non-stdlib packages in the same repo named "python-eggs". Enthought comes with the SciPy stack, and PythonWin with PyWin32. Anaconda has its own package management system, and a set of recommended extra packages. And so on. > This seems to be the root of the concern about including, for example, something like autopep8 with Python. > > >> On Tue, Mar 17, 2015 at 11:42 PM, Alexander Walters wrote: >> Good coding style really should not be something we ship in the library, it should be something we teach. > > This point is orthogonal to the discussion of whether or not to package an auto-styler, but there are many style points that are so mechanical I see little benefit in expending energy on teaching them. Best would be to automate as much of it away so that people can focus on bigger problems like, what is the right abstraction, did I design this API well, etc. > > Put differently, consider the following straw man: "Good memory management should not be something we ship in the language, it should be something we teach." That's a great example, because it's a little of both. Python automates memory management as far as is reasonable, and makes it easy for people who need to do more (clear recommendations on when to del/reassign variables, tools like weakrefdict, etc.), so some users don't need to be taught anything, and the more advanced users can be taught more easily than in, say, Objective C. > There is always something to teach. But when we can automate part of the problem away, I think we generally should. > >> On Wed, Mar 18, 2015 at 4:52 AM, Steven D'Aprano wrote: >> Not necessarily. This list frequently includes half-baked ideas, but the >> "bigger" the proposal, the more baked it should be to be taken >> seriously. > > Duly noted! Also keep in mind that, to whatever extent it isn't fully-baked, it should be clear that it is at least bakeable, and that you look willing and able to do the baking. That's a bit harder for a brand new list member than for, say, Guido (his initial proposal for asyncio, for example, wouldn't have gotten the same interest of it was your initial proposal), but it's doable, and necessary. -------------- next part -------------- An HTML attachment was scrubbed... URL: From tjreedy at udel.edu Wed Mar 18 23:17:12 2015 From: tjreedy at udel.edu (Terry Reedy) Date: Wed, 18 Mar 2015 18:17:12 -0400 Subject: [Python-ideas] gofmt for Python: standardized styling as a language feature In-Reply-To: References: <5508DBB0.2010209@stoneleaf.us> <5508F42F.8020604@sdamon.com> Message-ID: On 3/18/2015 12:52 PM, Nicholas Chammas wrote: > I might be wrong, but I see PEP 8 as a living document. PEP 8 is Guido's view on how new CPython stdlib code should be written, with input from other core developers. "This document gives coding conventions for the Python code comprising the standard library in the main Python distribution." It changes as he and we change our views. It is against Guido's policy to blindly change old stdlib code to follow PEP 8. (Note that autoPEP8 only changes whitespace, which is a small part of PEP 8 recommendations, and the safest aspect of code to change.) > If the community > believes parts of it are out of date, I would think it's better to > update PEP 8 (or issue a replacement PEP) to reflect current thinking. The 'community' is free to follow it or not, but not to change it. Other people and organizations have written their own style guides. Good for them. > If stuff is not part of PEP 8 that should be, or if parts of PEP 8 are > agreed to be "wrong", then PEP 8 should be updated accordingly. PEP 8 is by definition 'right' until Guido decides otherwise, with input from other core devs. > I guess this is a separate discussion, but is there some way to offer > users the advantages of having a library be included with Python without > imposing the 2-year release cycle and additional burden on the library's > developers? Use pip. It was added to make pypi (and other repositories) an easy-to-use extension of the stdlib, without imposing stdlib requirements and burdens. We are working on making pip even easier to use. -- Terry Jan Reedy From tjreedy at udel.edu Thu Mar 19 00:21:23 2015 From: tjreedy at udel.edu (Terry Reedy) Date: Wed, 18 Mar 2015 19:21:23 -0400 Subject: [Python-ideas] gofmt for Python: standardized styling as a language feature In-Reply-To: References: Message-ID: On 3/17/2015 9:54 PM, Nicholas Chammas wrote: > I am not proposing putting |autopep8| in the standard library. You gave that as an example and I went with it the absence of there being anything else concrete, even a name. Substitute a mythical 'autostyle' if you wish. > I am > proposing that an auto-styler be included with Python as a first-class > utility. The emphasis here is on /included/ and /first-class/. We are wary about including anything more with the CPython distribution. It is already too large to be managed by the core developer group as well as we would like. I disagree than not being included in 'cpython' necessarily makes something less than first-class, as you seemed to say in another response. > That auto-styler may well be based somehow on |autopep8|, but that is a > separate discussion about /implementation/. I am first checking that the > /idea/ is sound. I can?t tell if you had any objections to that idea. I do. We should be careful about adding anything. When we do, we should add libraries, not applications. We should especially not officially endorse one of many opinion-based competitors. Pip is the intended way to include 3rd party modules, as one chooses, as virtual members of the stdlib or distribution. FWIW, CPython already comes with a minimal autostyle script, Tools/Scripts/reindent.py, which is used to enforce minimal requirements for committing new or altered stdlib .py files to the cpython hg repository. > couldn?t |autopep8| somehow be adapted to be an included module > that can be invoked via |python -m| Today it is invoked with 'autopep8'. Why would you want to have to add 'python -m'? For Idle, I would like to move in the other direction and allow Idle to be started on a command line with 'idle' rather than 'python -m idlelib'. (This is possible today, but only if one adds '.../python/lib/idlelib' to PATH, or, on windows, moves an edited version of idle.bat to /Scripts.) -- Terry Jan Reedy From eliben at gmail.com Thu Mar 19 00:30:10 2015 From: eliben at gmail.com (Eli Bendersky) Date: Wed, 18 Mar 2015 16:30:10 -0700 Subject: [Python-ideas] gofmt for Python: standardized styling as a language feature In-Reply-To: References: <5508DBB0.2010209@stoneleaf.us> <5508F42F.8020604@sdamon.com> Message-ID: On Wed, Mar 18, 2015 at 9:52 AM, Nicholas Chammas < nicholas.chammas at gmail.com> wrote: > On Tue, Mar 17, 2015 at 11:29 PM, Eli Bendersky wrote: > >> We (at Google) are just on the verge of releasing an open-source tool for >> auto-formatting of Python code. > > > Interesting. Are y'all familiar with `autopep8`? > Yes, very familiar. > What are the differences between the tool y'all are releasing and > `autopep8`? > > autopep8's scope is both smaller and larger. It only detects PEP8 violations and attempts to fix them, not restricting itself to whitespace. Our tool works more like the gofmt tool mentioned in the beginning of the thread - it pretty much only cares about formatting, and it auto-reformats the code to its own liking, covering more whitespace-related stuff than PEP8. Basically the idea is "forget about formatting your code, just run it through the tool", which is a similar philosophy as gofmt (and clang-format) and has been hugely successful for those tools/languages. Eli > > On Tue, Mar 17, 2015 at 11:29 PM, Eli Bendersky wrote: > >> P.S. I'm also against such a tool being a part of the Python >> distribution, at least in the near term. A tool/library has to first prove >> itself outside of core for a while, and only when/if it proves to be widely >> used and stable, an incorporation into core should be considered. The >> stdlib is big as it is, and any additional code is a burden on all core >> developers. Incorporation into the stdlib also imposes a very slow pace of >> development/release on said library/tool, and hence should only be >> considered when it's been extremely stable for a while. > > > This makes sense. In fact, I recently watched a keynote that Guido gave > and he said something similar . > > > On Tue, Mar 17, 2015 at 11:42 PM, Alexander Walters < > tritium-list at sdamon.com> wrote: > >> What coding style would be used for such a tool? PEP-8? What about a >> tool for those of us who believe that PEP-8 got a few things wrong? >> >> What about for the tribal knowledge that pythonistas develop over time >> that are not part of the standard? Would such a tool include this? >> > > I might be wrong, but I see PEP 8 as a living document. If the community > believes parts of it are out of date, I would think it's better to update > PEP 8 (or issue a replacement PEP) to reflect current thinking. > > Any auto-styling tool included in the standard library would work best if > it was based on a written "spec" like PEP 8 (granted, I don't think PEP 8 > was meant to be a spec for programmatic implementation, but you get the > idea). > > If stuff is not part of PEP 8 that should be, or if parts of PEP 8 are > agreed to be "wrong", then PEP 8 should be updated accordingly. Any > auto-styling tool, I would hope, would just follow from PEP 8 (or a similar > document). The tool should not be the specification. > > > On Tue, Mar 17, 2015 at 11:42 PM, Alexander Walters < > tritium-list at sdamon.com> wrote: >> >> And what of third party tools that do the same thing? I have no evidence >> to support this, but I have the feeling that when the standard library >> includes something in a new space, third party work in that space slows >> (json and sqlite come to mind). >> > > I see what you mean. Several people have made comments to this effect, and > this is a good point. > > I guess this is a separate discussion, but is there some way to offer > users the advantages of having a library be included with Python without > imposing the 2-year release cycle and additional burden on the library's > developers? > > This seems to be the root of the concern about including, for example, > something like autopep8 with Python. > > > On Tue, Mar 17, 2015 at 11:42 PM, Alexander Walters < > tritium-list at sdamon.com> wrote: > >> Good coding style really should not be something we ship in the library, >> it should be something we teach. >> > > This point is orthogonal to the discussion of whether or not to package an > auto-styler, but there are many style points that are so mechanical I see > little benefit in expending energy on teaching them. Best would be to > automate as much of it away so that people can focus on bigger problems > like, what is the right abstraction, did I design this API well, etc. > > Put differently, consider the following straw man: "Good memory management > should not be something we ship in the language, it should be something we > teach." > > There is always something to teach. But when we can automate part of the > problem away, I think we generally should. > > > On Wed, Mar 18, 2015 at 2:38 AM, Ian Lee wrote: > >> And perhaps more to the point, should PEP-8 (and if so how), evolve over >> time to incorporate newer ideas that the community has now moved towards >> accepting as the "more Pythonic way to do things" now with the benefit of >> additional hindsight. > > > I consider PEP 8 to be a living document. That is, it is already changing > with time. For example, I see that though PEP 8 was created in 2001, it was updated > in 2013 . I assume it will > continue to be updated as accepted standards change. > > > On Wed, Mar 18, 2015 at 4:52 AM, Steven D'Aprano > wrote: > >> Not necessarily. This list frequently includes half-baked ideas, but the >> "bigger" the proposal, the more baked it should be to be taken >> seriously. >> > > Duly noted! > > Nick > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From szport at gmail.com Thu Mar 19 08:04:38 2015 From: szport at gmail.com (Zaur Shibzukhov) Date: Thu, 19 Mar 2015 00:04:38 -0700 (PDT) Subject: [Python-ideas] A mutable alternative to namedtuple In-Reply-To: <55086248.2010009@trueblade.com> References: <55086248.2010009@trueblade.com> Message-ID: ???????, 17 ????? 2015 ?., 20:21:01 UTC+3 ???????????? Eric V. Smith ???????: > > On 03/17/2015 12:52 PM, Luciano Ramalho wrote: > > Sometimes we need a simple class to hold some mutable attributes, > > provide a nice repr, support == for testing, and support iterable > > unpacking, so you can write: > > > >>>> p = Point(3, 4) > >>>> x, y = p > > > > That's very much like the classes built by namedtuple, but mutable. > > https://pypi.python.org/pypi/namedlist > > It also adds default values to the generated constructor, which may or > may not be desirable. But if used exactly like collections.namedtuple, > it ignores the default values. > > Eric. > > Since named tuple is considered as an object that is a tuple with attribute access. The mutable alternative could be considered as an array with attribute access. Array in this context is tuple-like object that support assign operation. Since python have not such object there are different approaches tomutable named tuple alternatives. One should note that particular property of named tuple is memory saving. So one can expect similar property of mutable named tuple too. -------------- next part -------------- An HTML attachment was scrubbed... URL: From abarnert at yahoo.com Thu Mar 19 09:37:26 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Thu, 19 Mar 2015 01:37:26 -0700 Subject: [Python-ideas] A mutable alternative to namedtuple In-Reply-To: References: <55086248.2010009@trueblade.com> Message-ID: <5F372C2E-00E0-4432-9832-91C69AB69092@yahoo.com> On Mar 19, 2015, at 12:04 AM, Zaur Shibzukhov wrote: > > > > ???????, 17 ????? 2015 ?., 20:21:01 UTC+3 ???????????? Eric V. Smith ???????: >> >> On 03/17/2015 12:52 PM, Luciano Ramalho wrote: >> > Sometimes we need a simple class to hold some mutable attributes, >> > provide a nice repr, support == for testing, and support iterable >> > unpacking, so you can write: >> > >> >>>> p = Point(3, 4) >> >>>> x, y = p >> > >> > That's very much like the classes built by namedtuple, but mutable. >> >> https://pypi.python.org/pypi/namedlist >> >> It also adds default values to the generated constructor, which may or >> may not be desirable. But if used exactly like collections.namedtuple, >> it ignores the default values. >> >> Eric. > Since named tuple is considered as an object that is a tuple with attribute access. > The mutable alternative could be considered as an array with attribute access. > Array in this context is tuple-like object that support assign operation. > Since python have not such object there are different approaches tomutable named tuple alternatives. Python definitely does have such an object: list. A list is effectively the same as a tuple but mutable; it's the paradigm MutableSequence while tuple is the paradigm Sequence. Under the covers they have very similar headers that both use the same storage (a C array of pointers to Python objects, in CPython), and C API functions like PySequence_Fast_GET_ITEM don't distinguish between the two. However, list is resizable, and presumably a "namedlist" would not be. That makes things more complicated for both the interface (there's no is-a relationship; a type without append is not a list--and, worse, a type that has __setitem__ but can't handle slice replacement is not a list but that's very hard to detect...) and the implementation (e.g., a list reserves extra space at the end to avoid having to reallocate on every append). (Python _also_ has an array type, which is for homogenous simple types (like 32-bit int) which can store the values directly, as opposed to tuple and list, which store (pointers to) heterogenous normal Python objects.) > One should note that particular property of named tuple is memory saving. > So one can expect similar property of mutable named tuple too. If you don't need to access the items by index for whatever reason, you don't need a namedtuple, and using one as a misguided misoptimization is a bad idea. Besides the fact that a normal class with __slots__ is also small, and even a normal class with a dict (in newer CPython versions and PyPy) not that much bigger, besides the fact that you can eliminate the row overhead rather than just slightly reducing it by using, e.g., a 2D array, you're optimizing the wrong thing in the first place--if your rows have 9 elements, reducing the row overhead is focusing on fixing 10% of your overhead, while reducing or eliminating the element overhead by using, e.g., a 2D numpy array of low-level values fixes the 90% (along with the 10%). -------------- next part -------------- An HTML attachment was scrubbed... URL: From szport at gmail.com Thu Mar 19 10:11:56 2015 From: szport at gmail.com (Zaur Shibzukhov) Date: Thu, 19 Mar 2015 12:11:56 +0300 Subject: [Python-ideas] A mutable alternative to namedtuple In-Reply-To: <5F372C2E-00E0-4432-9832-91C69AB69092@yahoo.com> References: <55086248.2010009@trueblade.com> <5F372C2E-00E0-4432-9832-91C69AB69092@yahoo.com> Message-ID: That all right. But I want to note that `collections.namedtuple` has several properties that make them exclusive: 1. Fast creation; 2. Minimal memory capacity; 3. Fast sequence interface; 4. Attribute access to elements via properties. Different namedtuple alternatives has different sets of properties that make them more ore less suitable depending on use cases. So if someone search alternative of collections.namedtuple that support assignment too then it could be constructed on top of array (actually "tuple" but with assignment support, but python seems have not such array type). --- *Zaur Shibzukhov* 2015-03-19 11:37 GMT+03:00 Andrew Barnert : > On Mar 19, 2015, at 12:04 AM, Zaur Shibzukhov wrote: > > > > ???????, 17 ????? 2015 ?., 20:21:01 UTC+3 ???????????? Eric V. Smith > ???????: >> >> On 03/17/2015 12:52 PM, Luciano Ramalho wrote: >> > Sometimes we need a simple class to hold some mutable attributes, >> > provide a nice repr, support == for testing, and support iterable >> > unpacking, so you can write: >> > >> >>>> p = Point(3, 4) >> >>>> x, y = p >> > >> > That's very much like the classes built by namedtuple, but mutable. >> >> https://pypi.python.org/pypi/namedlist >> >> It also adds default values to the generated constructor, which may or >> may not be desirable. But if used exactly like collections.namedtuple, >> it ignores the default values. >> >> Eric. >> >> Since named tuple is considered as an object that is a tuple with > attribute access. > The mutable alternative could be considered as an array with attribute > access. > Array in this context is tuple-like object that support assign operation. > Since python have not such object there are different approaches > tomutable named tuple alternatives. > > > Python definitely does have such an object: list. A list is effectively > the same as a tuple but mutable; it's the paradigm MutableSequence while > tuple is the paradigm Sequence. Under the covers they have very similar > headers that both use the same storage (a C array of pointers to Python > objects, in CPython), and C API functions like PySequence_Fast_GET_ITEM > don't distinguish between the two. > > However, list is resizable, and presumably a "namedlist" would not be. > That makes things more complicated for both the interface (there's no is-a > relationship; a type without append is not a list--and, worse, a type that > has __setitem__ but can't handle slice replacement is not a list but that's > very hard to detect...) and the implementation (e.g., a list reserves extra > space at the end to avoid having to reallocate on every append). > > (Python _also_ has an array type, which is for homogenous simple types > (like 32-bit int) which can store the values directly, as opposed to tuple > and list, which store (pointers to) heterogenous normal Python objects.) > > One should note that particular property of named tuple is memory saving. > So one can expect similar property of mutable named tuple too. > > > If you don't need to access the items by index for whatever reason, you > don't need a namedtuple, and using one as a misguided misoptimization is a > bad idea. > > Besides the fact that a normal class with __slots__ is also small, and > even a normal class with a dict (in newer CPython versions and PyPy) not > that much bigger, besides the fact that you can eliminate the row overhead > rather than just slightly reducing it by using, e.g., a 2D array, you're > optimizing the wrong thing in the first place--if your rows have 9 > elements, reducing the row overhead is focusing on fixing 10% of your > overhead, while reducing or eliminating the element overhead by using, > e.g., a 2D numpy array of low-level values fixes the 90% (along with the > 10%). > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From abarnert at yahoo.com Thu Mar 19 11:07:20 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Thu, 19 Mar 2015 03:07:20 -0700 Subject: [Python-ideas] A mutable alternative to namedtuple In-Reply-To: References: <55086248.2010009@trueblade.com> <5F372C2E-00E0-4432-9832-91C69AB69092@yahoo.com> Message-ID: <4B123971-940F-4ADD-943C-6C82B0FA282D@yahoo.com> On Mar 19, 2015, at 2:11 AM, Zaur Shibzukhov wrote: > > That all right. But I want to note that `collections.namedtuple` has several properties that make them exclusive: > > 1. Fast creation; > 2. Minimal memory capacity; > 3. Fast sequence interface; > 4. Attribute access to elements via properties. > > Different namedtuple alternatives has different sets of properties that make them more ore less suitable depending on use cases. > > So if someone search alternative of collections.namedtuple that support assignment too then it could be constructed on top of array (actually "tuple" but with assignment support, but python seems have not such array type). Again, do you not know about list? > > --- > Zaur Shibzukhov > > > 2015-03-19 11:37 GMT+03:00 Andrew Barnert : >>> On Mar 19, 2015, at 12:04 AM, Zaur Shibzukhov wrote: >>> >>> >>> >>> ???????, 17 ????? 2015 ?., 20:21:01 UTC+3 ???????????? Eric V. Smith ???????: >>>> >>>> On 03/17/2015 12:52 PM, Luciano Ramalho wrote: >>>> > Sometimes we need a simple class to hold some mutable attributes, >>>> > provide a nice repr, support == for testing, and support iterable >>>> > unpacking, so you can write: >>>> > >>>> >>>> p = Point(3, 4) >>>> >>>> x, y = p >>>> > >>>> > That's very much like the classes built by namedtuple, but mutable. >>>> >>>> https://pypi.python.org/pypi/namedlist >>>> >>>> It also adds default values to the generated constructor, which may or >>>> may not be desirable. But if used exactly like collections.namedtuple, >>>> it ignores the default values. >>>> >>>> Eric. >>> Since named tuple is considered as an object that is a tuple with attribute access. >>> The mutable alternative could be considered as an array with attribute access. >>> Array in this context is tuple-like object that support assign operation. >>> Since python have not such object there are different approaches tomutable named tuple alternatives. >> >> Python definitely does have such an object: list. A list is effectively the same as a tuple but mutable; it's the paradigm MutableSequence while tuple is the paradigm Sequence. Under the covers they have very similar headers that both use the same storage (a C array of pointers to Python objects, in CPython), and C API functions like PySequence_Fast_GET_ITEM don't distinguish between the two. >> >> However, list is resizable, and presumably a "namedlist" would not be. That makes things more complicated for both the interface (there's no is-a relationship; a type without append is not a list--and, worse, a type that has __setitem__ but can't handle slice replacement is not a list but that's very hard to detect...) and the implementation (e.g., a list reserves extra space at the end to avoid having to reallocate on every append). >> >> (Python _also_ has an array type, which is for homogenous simple types (like 32-bit int) which can store the values directly, as opposed to tuple and list, which store (pointers to) heterogenous normal Python objects.) >> >>> One should note that particular property of named tuple is memory saving. >>> So one can expect similar property of mutable named tuple too. >> >> If you don't need to access the items by index for whatever reason, you don't need a namedtuple, and using one as a misguided misoptimization is a bad idea. >> >> Besides the fact that a normal class with __slots__ is also small, and even a normal class with a dict (in newer CPython versions and PyPy) not that much bigger, besides the fact that you can eliminate the row overhead rather than just slightly reducing it by using, e.g., a 2D array, you're optimizing the wrong thing in the first place--if your rows have 9 elements, reducing the row overhead is focusing on fixing 10% of your overhead, while reducing or eliminating the element overhead by using, e.g., a 2D numpy array of low-level values fixes the 90% (along with the 10%). > -------------- next part -------------- An HTML attachment was scrubbed... URL: From szport at gmail.com Thu Mar 19 11:43:32 2015 From: szport at gmail.com (Zaur Shibzukhov) Date: Thu, 19 Mar 2015 13:43:32 +0300 Subject: [Python-ideas] A mutable alternative to namedtuple In-Reply-To: <4B123971-940F-4ADD-943C-6C82B0FA282D@yahoo.com> References: <55086248.2010009@trueblade.com> <5F372C2E-00E0-4432-9832-91C69AB69092@yahoo.com> <4B123971-940F-4ADD-943C-6C82B0FA282D@yahoo.com> Message-ID: Yes. But: First. >>>print(sys.getsizeof(list([])), sys.getsizeof(tuple([]))) 64 48 print(sys.getsizeof(list([1,2])), sys.getsizeof(tuple([1,2]))) 104 64 Second. Tuple object allocates it's memory 1 time, list object allocates it's memory 2 time. That is why >>>%time for i in range(100000): tuple(i for i in range(15)) CPU times: user 241 ms, sys: 1.75 ms, total: 243 ms Wall time: 243 ms >>>%time for i in range(100000): list(i for i in range(15)) CPU times: user 318 ms, sys: 1.94 ms, total: 320 ms Wall time: 321 ms Certainly this can have or not have a value depending on use case. --- *Zaur Shibzukhov* 2015-03-19 13:07 GMT+03:00 Andrew Barnert : > On Mar 19, 2015, at 2:11 AM, Zaur Shibzukhov wrote: > > That all right. But I want to note that `collections.namedtuple` has > several properties that make them exclusive: > > 1. Fast creation; > 2. Minimal memory capacity; > 3. Fast sequence interface; > 4. Attribute access to elements via properties. > > Different namedtuple alternatives has different sets of properties that > make them more ore less suitable depending on use cases. > > So if someone search alternative of collections.namedtuple that support > assignment too then it could be constructed on top of array (actually > "tuple" but with assignment support, but python seems have not such array > type). > > > Again, do you not know about list? > > > --- > *Zaur Shibzukhov* > > > 2015-03-19 11:37 GMT+03:00 Andrew Barnert : > >> On Mar 19, 2015, at 12:04 AM, Zaur Shibzukhov wrote: >> >> >> >> ???????, 17 ????? 2015 ?., 20:21:01 UTC+3 ???????????? Eric V. Smith >> ???????: >>> >>> On 03/17/2015 12:52 PM, Luciano Ramalho wrote: >>> > Sometimes we need a simple class to hold some mutable attributes, >>> > provide a nice repr, support == for testing, and support iterable >>> > unpacking, so you can write: >>> > >>> >>>> p = Point(3, 4) >>> >>>> x, y = p >>> > >>> > That's very much like the classes built by namedtuple, but mutable. >>> >>> https://pypi.python.org/pypi/namedlist >>> >>> It also adds default values to the generated constructor, which may or >>> may not be desirable. But if used exactly like collections.namedtuple, >>> it ignores the default values. >>> >>> Eric. >>> >>> Since named tuple is considered as an object that is a tuple with >> attribute access. >> The mutable alternative could be considered as an array with attribute >> access. >> Array in this context is tuple-like object that support assign operation. >> Since python have not such object there are different approaches >> tomutable named tuple alternatives. >> >> >> Python definitely does have such an object: list. A list is effectively >> the same as a tuple but mutable; it's the paradigm MutableSequence while >> tuple is the paradigm Sequence. Under the covers they have very similar >> headers that both use the same storage (a C array of pointers to Python >> objects, in CPython), and C API functions like PySequence_Fast_GET_ITEM >> don't distinguish between the two. >> >> However, list is resizable, and presumably a "namedlist" would not be. >> That makes things more complicated for both the interface (there's no is-a >> relationship; a type without append is not a list--and, worse, a type that >> has __setitem__ but can't handle slice replacement is not a list but that's >> very hard to detect...) and the implementation (e.g., a list reserves extra >> space at the end to avoid having to reallocate on every append). >> >> (Python _also_ has an array type, which is for homogenous simple types >> (like 32-bit int) which can store the values directly, as opposed to tuple >> and list, which store (pointers to) heterogenous normal Python objects.) >> >> One should note that particular property of named tuple is memory saving. >> So one can expect similar property of mutable named tuple too. >> >> >> If you don't need to access the items by index for whatever reason, you >> don't need a namedtuple, and using one as a misguided misoptimization is a >> bad idea. >> >> Besides the fact that a normal class with __slots__ is also small, and >> even a normal class with a dict (in newer CPython versions and PyPy) not >> that much bigger, besides the fact that you can eliminate the row overhead >> rather than just slightly reducing it by using, e.g., a 2D array, you're >> optimizing the wrong thing in the first place--if your rows have 9 >> elements, reducing the row overhead is focusing on fixing 10% of your >> overhead, while reducing or eliminating the element overhead by using, >> e.g., a 2D numpy array of low-level values fixes the 90% (along with the >> 10%). >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From abarnert at yahoo.com Thu Mar 19 12:22:14 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Thu, 19 Mar 2015 04:22:14 -0700 Subject: [Python-ideas] A mutable alternative to namedtuple In-Reply-To: References: <55086248.2010009@trueblade.com> <5F372C2E-00E0-4432-9832-91C69AB69092@yahoo.com> <4B123971-940F-4ADD-943C-6C82B0FA282D@yahoo.com> Message-ID: On Mar 19, 2015, at 3:43 AM, Zaur Shibzukhov wrote: > > Yes. But: > > First. > > >>>print(sys.getsizeof(list([])), sys.getsizeof(tuple([]))) > 64 48 > print(sys.getsizeof(list([1,2])), sys.getsizeof(tuple([1,2]))) > 104 64 > > > Second. > > Tuple object allocates it's memory 1 time, list object allocates it's memory 2 time. As I explained, list leaves room for expansion, so it doesn't have to keep reallocating with every append. A 2-element list actually has room for... off the top of my head, I think 8 elements; then it multiplies the capacity every time it runs out on an append. In a C extension, you can create a list with whatever specific capacity you want, but that isn't exposed to Python (presumably because such a micro-optimization is rarely useful, and would complicate the API, as well as being an attractive nuisance to people who insist on optimizing the wrong things). So, if you really needed this optimization, you could implement your namedlist in C. But what use case are you imagining where the extra 40 bytes per row matter, but the 64 bytes per row don't, and neither do the 56 or so bytes per element per row? I'm having a hard time imagining any case where using tuples would be necessary where it wouldn't be woefully insufficient. > That is why > > >>>%time for i in range(100000): tuple(i for i in range(15)) > CPU times: user 241 ms, sys: 1.75 ms, total: 243 ms > Wall time: 243 ms > > >>>%time for i in range(100000): list(i for i in range(15)) > CPU times: user 318 ms, sys: 1.94 ms, total: 320 ms > Wall time: 321 ms > > Certainly this can have or not have a value depending on use case. If you're going to show benchmarks, you really should use %timeit rather than %time, and also understand that "i for i in range(15)" is just going to give you a slower but equivalent iterable to just using "range(15)" in the first place. But more importantly, what use case are you considering where this extra 0.8us for construction of each row object will matter, but the difference between a namedtuple (or, presumably, namedlist) vs. a tuple won't? And, even if you did, you're focusing on something that accounts for at worst 25% of the construction time, and 0% of the time for all the actual work you do with the object after construction. Again, unless you have many millions of these, neither the memory not the construction time is going to matter--and if you do, a new type that's more like tuple isn't going to be anywhere near sufficient to make a noticeable difference, because you're optimizing the wrong part of both the memory and the time. > > --- > Zaur Shibzukhov > > > 2015-03-19 13:07 GMT+03:00 Andrew Barnert : >>> On Mar 19, 2015, at 2:11 AM, Zaur Shibzukhov wrote: >>> >>> That all right. But I want to note that `collections.namedtuple` has several properties that make them exclusive: >>> >>> 1. Fast creation; >>> 2. Minimal memory capacity; >>> 3. Fast sequence interface; >>> 4. Attribute access to elements via properties. >>> >>> Different namedtuple alternatives has different sets of properties that make them more ore less suitable depending on use cases. >>> >>> So if someone search alternative of collections.namedtuple that support assignment too then it could be constructed on top of array (actually "tuple" but with assignment support, but python seems have not such array type). >> >> Again, do you not know about list? >> >>> >>> --- >>> Zaur Shibzukhov >>> >>> >>> 2015-03-19 11:37 GMT+03:00 Andrew Barnert : >>>>> On Mar 19, 2015, at 12:04 AM, Zaur Shibzukhov wrote: >>>>> >>>>> >>>>> >>>>> ???????, 17 ????? 2015 ?., 20:21:01 UTC+3 ???????????? Eric V. Smith ???????: >>>>>> >>>>>> On 03/17/2015 12:52 PM, Luciano Ramalho wrote: >>>>>> > Sometimes we need a simple class to hold some mutable attributes, >>>>>> > provide a nice repr, support == for testing, and support iterable >>>>>> > unpacking, so you can write: >>>>>> > >>>>>> >>>> p = Point(3, 4) >>>>>> >>>> x, y = p >>>>>> > >>>>>> > That's very much like the classes built by namedtuple, but mutable. >>>>>> >>>>>> https://pypi.python.org/pypi/namedlist >>>>>> >>>>>> It also adds default values to the generated constructor, which may or >>>>>> may not be desirable. But if used exactly like collections.namedtuple, >>>>>> it ignores the default values. >>>>>> >>>>>> Eric. >>>>> Since named tuple is considered as an object that is a tuple with attribute access. >>>>> The mutable alternative could be considered as an array with attribute access. >>>>> Array in this context is tuple-like object that support assign operation. >>>>> Since python have not such object there are different approaches tomutable named tuple alternatives. >>>> >>>> Python definitely does have such an object: list. A list is effectively the same as a tuple but mutable; it's the paradigm MutableSequence while tuple is the paradigm Sequence. Under the covers they have very similar headers that both use the same storage (a C array of pointers to Python objects, in CPython), and C API functions like PySequence_Fast_GET_ITEM don't distinguish between the two. >>>> >>>> However, list is resizable, and presumably a "namedlist" would not be. That makes things more complicated for both the interface (there's no is-a relationship; a type without append is not a list--and, worse, a type that has __setitem__ but can't handle slice replacement is not a list but that's very hard to detect...) and the implementation (e.g., a list reserves extra space at the end to avoid having to reallocate on every append). >>>> >>>> (Python _also_ has an array type, which is for homogenous simple types (like 32-bit int) which can store the values directly, as opposed to tuple and list, which store (pointers to) heterogenous normal Python objects.) >>>> >>>>> One should note that particular property of named tuple is memory saving. >>>>> So one can expect similar property of mutable named tuple too. >>>> >>>> If you don't need to access the items by index for whatever reason, you don't need a namedtuple, and using one as a misguided misoptimization is a bad idea. >>>> >>>> Besides the fact that a normal class with __slots__ is also small, and even a normal class with a dict (in newer CPython versions and PyPy) not that much bigger, besides the fact that you can eliminate the row overhead rather than just slightly reducing it by using, e.g., a 2D array, you're optimizing the wrong thing in the first place--if your rows have 9 elements, reducing the row overhead is focusing on fixing 10% of your overhead, while reducing or eliminating the element overhead by using, e.g., a 2D numpy array of low-level values fixes the 90% (along with the 10%). > -------------- next part -------------- An HTML attachment was scrubbed... URL: From nicholas.chammas at gmail.com Thu Mar 19 15:32:02 2015 From: nicholas.chammas at gmail.com (Nicholas Chammas) Date: Thu, 19 Mar 2015 10:32:02 -0400 Subject: [Python-ideas] gofmt for Python: standardized styling as a language feature In-Reply-To: References: <5508DBB0.2010209@stoneleaf.us> <5508F42F.8020604@sdamon.com> Message-ID: OK, I hereby abandon this proposal. :) To summarize the conclusion I believe this thread has arrived at: - An auto-styler needs to have wide community adoption before it can be considered for inclusion into the standard library. - Even then, adding stuff to the standard library at this point in Python?s life is generally a bad thing. Putting an auto-styler in the standard library would be a big burden on its maintainers and slow down improvements that could be made to it. - Ignoring that difficulty, there is also the problem of what ?spec? to use for any such auto-styler. PEP 8 is an obvious choice, but it was written as a guide for humans, not a spec for programmatic implementation. Furthermore, there are doubts about PEP 8?s utility as the basis for a universal auto-styler, as it was intended for code specifically in the Python standard library. - Let a thousand auto-stylers bloom outside the standard library. Some final responses to various people?s comments: On Wed, Mar 18, 2015 at 4:38 PM, Andrew Barnert wrote: But that implies that it's worth someone making a spec out of PEP 8. That > implies going through PEP 8, getting a sense of what parts are > (unambiguously and uncontroversially) mechanizable, and reorganizing and > minimally rewriting it to get that sense across. Having numbered rules, and > named anchors for each one in the document (or at least to narrower > sections, so you can link to something more specific than "Programming > Recommendations") would also be helpful. This is an excellent point and one I did not consider. On Wed, Mar 18, 2015 at 6:17 PM, Terry Reedy wrote: PEP 8 is Guido's view on how new CPython stdlib code should be written, > with input from other core developers. "This document gives coding > conventions for the Python code comprising the standard library in the main > Python distribution." Good point. For my all my talk about PEP 8, I missed this key point. Of course, it should be noted that PEP 8 is being used by all kinds of people for stuff outside of stdlib, but then again as you pointed out people are also putting together their own style guides. On Wed, Mar 18, 2015 at 6:17 PM, Terry Reedy wrote: (Note that autoPEP8 only changes whitespace, which is a small part of PEP 8 > recommendations, and the safest aspect of code to change.) > Minor point: autopep8 can do more than just whitespace changes with the --aggressive flag . On Wed, Mar 18, 2015 at 7:21 PM, Terry Reedy wrote: We should be careful about adding anything. When we do, we should add > libraries, not applications. We should especially not officially endorse > one of many opinion-based competitors. I have come around to this view. On Wed, Mar 18, 2015 at 7:30 PM, Eli Bendersky wrote: Basically the idea is "forget about formatting your code, just run it > through the tool", which is a similar philosophy as gofmt (and > clang-format) and has been hugely successful for those tools/languages. > Sounds good. I?m looking forward to seeing it released. As I said in an earlier email, I think any part of the development process we automate away is a good thing (stdlib inclusion or not notwithstanding). Nick ? -------------- next part -------------- An HTML attachment was scrubbed... URL: From slavek at redhat.com Thu Mar 19 15:47:44 2015 From: slavek at redhat.com (Bohuslav Kabrda) Date: Thu, 19 Mar 2015 10:47:44 -0400 (EDT) Subject: [Python-ideas] Migration of /usr/bin/python to python3 In-Reply-To: <20150314115719.23ea9d75@anarchist.wooz.org> References: <20150311194325.GA15390@python.ca> <041B5A0E-4C77-4683-9B0A-98432C139F48@stufft.io> <20150314115719.23ea9d75@anarchist.wooz.org> Message-ID: <1073335302.7431680.1426776464310.JavaMail.zimbra@redhat.com> ----- Original Message ----- > On Mar 13, 2015, at 09:57 PM, Nick Coghlan wrote: > > >Right, this most recently came up in the context of Fedora's plans to > >transition to only having Python 3 in the base install > >(https://fedoraproject.org/wiki/Changes/Python_3_as_Default). At the > >moment the expected consequence is that there will come a Fedora > >version (tentatively Fedora 23) where, out of the box, > >"/usr/bin/python" and "/usr/bin/env python" will just stop working. > > > >Packages that are part of the distro aren't a problem - those are all > >being dealt with as the transition progresses, and any which > >specifically need Python 2 will depend on it accordingly. > > > >The problem arises with custom user packages and scripts, and third > >party packages and scripts. > > Sure, agreed so far. But you're not going to be removing Python 2 from the > Fedora archive right? So any user with third party packages that require > Python 2 could just install it and their stuff would continue to work. Correct. > At least that's the plan on Debian/Ubuntu. I'd like to be at the point[*] > where the default installs would not contain Python 2, but a simple `apt-get` > would install it from the archives for anybody who needs it. That seems like > a safer option than changing the symlink. Yes, this is precisely where we're heading with this; we won't change the symlink unless PEP 394 changes. We will also prefer Python 3 for *all* applications (assuming upstream is compatible), not only those that are on live media. I think we'll have Python 2 in Fedora repositories as long as upstream will support it (and maybe even a bit longer). > Cheers, > -Barry > > [*] I'm aiming for Debian Stretch and Ubuntu 16.04 Xylophonic X-ray fish LTS. -- Regards, Slavek Kabrda From Steve.Dower at microsoft.com Thu Mar 19 16:52:41 2015 From: Steve.Dower at microsoft.com (Steve Dower) Date: Thu, 19 Mar 2015 15:52:41 +0000 Subject: [Python-ideas] Rewriting the build system (part 2) In-Reply-To: References: , Message-ID: And unfortunately, a lot of that work may need to be redone for 3.5 and VC14... Sorry about that, but once the other compilers have caught up with the new CRT interface then it will (should) all be better in the long run. Cheers, Steve Top-posted from my Windows Phone ________________________________ From: Nathaniel Smith Sent: ?3/?18/?2015 10:31 To: anatoly techtonik Cc: python-ideas at python.org Subject: Re: [Python-ideas] Rewriting the build system (part 2) On Mar 18, 2015 8:14 AM, "anatoly techtonik" > wrote: > > On Mon, Mar 16, 2015 at 7:31 PM, Chris Barker > wrote: > > On Mon, Mar 16, 2015 at 6:24 AM, anatoly techtonik > > > wrote: > >> > >> "Autotools has reached the Chandrasekhar mass limit of software and > >> become a black hole of suck that consumes endless hours of bright > >> peoples? time without any good result. It is well past time for it to > >> die." (c) "Autotools must die", ESR. http://esr.ibiblio.org/?p=1877 > >> > >> I couldn't sum up this better. It looks even worse from Windows machine. > > > > Is it even used for building Python on Windows (for the "normal" python.org > > builds)? > > There was some success with MinGW toolchain, but I am not tracking if it was > integrated into official source. The FUD was that compiled extensions are not > compatible between MSVC builds and MinGW builds, but I could not confirm > that. If I understand correctly, the incompatibility is only for C++ programs, > because of strange SEH strategy and proprietary name mangling. This isn't FUD, alas. mingw in almost all of its variants uses effectively a different "libc" than MSVC, so code compiled with two compilers can't even agree on the meaning of file numbers. It is mostly possible to set up a mingw-w64 build that produces MSVC-compatible extensions, but it requires using a configuration that's undocumented and untested upstream (I.e. you will run into mingw bugs). NumPy *may* switch to this at some point -- we've been working on it for like a year now -- but it's extremely non-trivial. -n -------------- next part -------------- An HTML attachment was scrubbed... URL: From techtonik at gmail.com Thu Mar 19 11:23:15 2015 From: techtonik at gmail.com (anatoly techtonik) Date: Thu, 19 Mar 2015 13:23:15 +0300 Subject: [Python-ideas] A mutable alternative to namedtuple In-Reply-To: References: Message-ID: On Wed, Mar 18, 2015 at 2:36 PM, Luciano Ramalho wrote: > On Wed, Mar 18, 2015 at 3:05 AM, anatoly techtonik wrote: >> On Tue, Mar 17, 2015 at 7:52 PM, Luciano Ramalho wrote: >>> Sometimes we need a simple class to hold some mutable attributes, >>> provide a nice repr, support == for testing, and support iterable >>> unpacking >> >> +1, but I think that the core problem with such proposals is that they >> lack the use cases. The only reason for me to have such class is to >> work with tabular data. >> >> For example, for querying the capability of the system, I need to build >> an inmemory table of features, and then set parameters for each >> feature one by one. sqlite/SQL is an overkill for that, and dicts are >> just not enough to do readable lookups and updates to specific >> cells, so I'd more appreciate a full table class than just its "named >> row" model. > > Thanks for your input and example, Anatoly. What would the full table > class offer that could not be easily done with a simple list of items > produced with a plainclass class? Lookup operations. Attaching new columns to existing table or providing a supplementary table with additional columns for existing ones. If Python had first class tables, things right now could be very different. We could have a better structured data handling. Take logging for example. It is not extensible. You have message and level. Then it got component. Then timestamp. But you can't add anything yourself to that message - error code, binary dump, selfie or some read/ unread flags. If logging used a table, there could be an ability to add your own data to its events. And this would be interoperable between programs. >> Practical example that I came up with: >> https://bitbucket.org/techtonik/discovery/src/c4f3d306bb43772dcf3c03be8db941ff67d077b4/graphics/canvas2d/canvas2d/__init__.py?at=default#cl-22 > > Oh, I see you have an update method, so that's the key value add of > the Table class, right? Right. The update method says "change the value of cell with column name==name, in row where column name idname==idvalue". It is basically lookup method with cell modification. So it adds a second dimension to 1D "mutable namedtuple". > I see the Table and plainclass as complementary ideas. You could use > plainclass to implement Table more easily. The Table.__iter__ could > return plainclass class instances instead of OrderedDict. That's one of the uses. But I am concerned that it is the only example where this "mutable namedtuple" is useful. I don't like the dynamic class construction as with namedtuple. I believe you can not serialize it reliably, and there are problems with static analysis tools to deal with it (like locate the definition). -- anatoly t. From techtonik at gmail.com Thu Mar 19 11:41:01 2015 From: techtonik at gmail.com (anatoly techtonik) Date: Thu, 19 Mar 2015 13:41:01 +0300 Subject: [Python-ideas] .from and .to instead of .encode and .decode In-Reply-To: References: Message-ID: On Wed, Mar 18, 2015 at 2:24 PM, Luciano Ramalho wrote: >> >>> A more objective problem is: "from" is a keyword, so it can't be a method name. >> >> I don't see why "from" can not be used as a method name in Python. It can not >> be used as function name, because it will override the keyword, but method >> names are always namespaced with self. or class prefixes. > > You can't write it's definition, "def from..." and you can't even > write "x.from" without a SyntaxError (instead of NameError or > AttributeError). Try it in the Python console. > > The parser would need to be changed. And that won't break anything. -- anatoly t. From techtonik at gmail.com Thu Mar 19 12:12:36 2015 From: techtonik at gmail.com (anatoly techtonik) Date: Thu, 19 Mar 2015 14:12:36 +0300 Subject: [Python-ideas] Rewriting the build system (part 2) In-Reply-To: <20150318152150.GN10871@tonks> References: <20150318152150.GN10871@tonks> Message-ID: On Wed, Mar 18, 2015 at 6:21 PM, Florian Bruhin wrote: > * anatoly techtonik [2015-03-18 10:21:46 +0300]: >> On Mon, Mar 16, 2015 at 11:33 PM, Ryan Gonzalez wrote: >> >> >> >> But to miss-paraphrase Winston Churchill [1]: >> >> >> >> "Autotools is the worst build system except for all the others." >> >> >> >> Is there really something better out there? >> >> >> > >> > Fbuild, Waf, Meson, GN, Boost.Build, Tundra, Bam, ... >> >> Is there anything Javascriptey? Python needs to move to web. I heard >> something about Grunt, which was replaced by Gulp, but no idea if they >> really mature enough to include "system features detection" and "user >> configured builds" (or work with C/C++ compilers at all). > > What does "Python needs to move to web" to do with it's build system > being "Javascriptey"? Because things like Twisted are de-facto standard in JavaScript world (Node.js), and Python will benefit more if people look for how things are implemented in non-C worlds. -- anatoly t. From szport at gmail.com Thu Mar 19 18:57:38 2015 From: szport at gmail.com (Zaur Shibzukhov) Date: Thu, 19 Mar 2015 20:57:38 +0300 Subject: [Python-ideas] A mutable alternative to namedtuple In-Reply-To: References: <55086248.2010009@trueblade.com> <5F372C2E-00E0-4432-9832-91C69AB69092@yahoo.com> <4B123971-940F-4ADD-943C-6C82B0FA282D@yahoo.com> Message-ID: 2015-03-19 14:22 GMT+03:00 Andrew Barnert : > On Mar 19, 2015, at 3:43 AM, Zaur Shibzukhov wrote: > > Yes. But: > > First. > > >>>print(sys.getsizeof(list([])), sys.getsizeof(tuple([]))) > 64 48 > print(sys.getsizeof(list([1,2])), sys.getsizeof(tuple([1,2]))) > 104 64 > > > Second. > > Tuple object allocates it's memory 1 time, list object allocates it's > memory 2 time. > > > As I explained, list leaves room for expansion, so it doesn't have to keep > reallocating with every append. A 2-element list actually has room for... > off the top of my head, I think 8 elements; then it multiplies the capacity > every time it runs out on an append. > >>>for i in [4, 12, 28, 68]: print(sys.getsizeof(list(range(i))), end=' ') 120 216 360 720 > In a C extension, you can create a list with whatever specific capacity you want, but that isn't exposed to Python > (presumably because such a micro-optimization is rarely useful, and would complicate the API, as well as being an > attractive nuisance to people who insist on optimizing the wrong things). So, if you really needed this optimization, you could > implement your namedlist in C. Yes you right. Actual optimization one could archive with the help of Cython if necessary. But what use case are you imagining where the extra 40 bytes per row matter, but the 64 bytes per row don't, and neither do the 56 or so bytes per element per row? I'm having a hard time imagining any case where using tuples would be necessary where it wouldn't be woefully insufficient. That is why >>>%time for i in range(100000): tuple(i for i in range(15)) CPU times: user 241 ms, sys: 1.75 ms, total: 243 ms Wall time: 243 ms >>>%time for i in range(100000): list(i for i in range(15)) CPU times: user 318 ms, sys: 1.94 ms, total: 320 ms Wall time: 321 ms Certainly this can have or not have a value depending on use case. >>>import namedlist > If you're going to show benchmarks, you really should use %timeit rather than %time, and also understand that "i for i in > range(15)" is just going to give you a slower but equivalent iterable to just using "range(15)" in the first place. > But more importantly, what use case are you considering where this extra 0.8us for construction of each row object will > matter, but the difference between a namedtuple (or, presumably, namedlist) vs. a tuple won't? >>>class Point: __slots__ = 'x', 'y', 'z' def __init__(self, x, y, z): self.x, self.y, self.z = x, y, z >>>asizeof.asizeof(Point(1,2,3)) 232 >>>import collections >>>Point2 = collections.namedtuple('Point', 'x y z') >>>Point3 = namedlist.namedtuple('Point', 'x y z') >>>Point4 = namedlist.namedlist('Point', 'x y z') >>>asizeof.asizeof(Point2(1,2,3)) 120 >>>asizeof.asizeof(Point3(1,2,3)) 120 >>>asizeof.asizeof(Point4(1,2,3)) 1704 > And, even if you did, you're focusing on something that accounts for at worst 25% of the construction time, and 0% of the > time for all the actual work you do with the object after construction. > Again, unless you have many millions of these, neither the memory not the construction time is going to matter--and if you do, > a new type that's more like tuple isn't going to be anywhere near sufficient to make a noticeable difference, because you're > optimizing the wrong part of both the memory and the time. Possibly basic gain in the savings of memory, especially in the context of long running processes, limitations on the memory on hostings, long loop + memory fragmentation in python heap space, ... --- *Zaur Shibzukhov* 2015-03-19 13:07 GMT+03:00 Andrew Barnert : > On Mar 19, 2015, at 2:11 AM, Zaur Shibzukhov wrote: > > That all right. But I want to note that `collections.namedtuple` has > several properties that make them exclusive: > > 1. Fast creation; > 2. Minimal memory capacity; > 3. Fast sequence interface; > 4. Attribute access to elements via properties. > > Different namedtuple alternatives has different sets of properties that > make them more ore less suitable depending on use cases. > > So if someone search alternative of collections.namedtuple that support > assignment too then it could be constructed on top of array (actually > "tuple" but with assignment support, but python seems have not such array > type). > > > Again, do you not know about list? > > > --- > *Zaur Shibzukhov* > > > 2015-03-19 11:37 GMT+03:00 Andrew Barnert : > >> On Mar 19, 2015, at 12:04 AM, Zaur Shibzukhov wrote: >> >> >> >> ???????, 17 ????? 2015 ?., 20:21:01 UTC+3 ???????????? Eric V. Smith >> ???????: >>> >>> On 03/17/2015 12:52 PM, Luciano Ramalho wrote: >>> > Sometimes we need a simple class to hold some mutable attributes, >>> > provide a nice repr, support == for testing, and support iterable >>> > unpacking, so you can write: >>> > >>> >>>> p = Point(3, 4) >>> >>>> x, y = p >>> > >>> > That's very much like the classes built by namedtuple, but mutable. >>> >>> https://pypi.python.org/pypi/namedlist >>> >>> It also adds default values to the generated constructor, which may or >>> may not be desirable. But if used exactly like collections.namedtuple, >>> it ignores the default values. >>> >>> Eric. >>> >>> Since named tuple is considered as an object that is a tuple with >> attribute access. >> The mutable alternative could be considered as an array with attribute >> access. >> Array in this context is tuple-like object that support assign operation. >> Since python have not such object there are different approaches >> tomutable named tuple alternatives. >> >> >> Python definitely does have such an object: list. A list is effectively >> the same as a tuple but mutable; it's the paradigm MutableSequence while >> tuple is the paradigm Sequence. Under the covers they have very similar >> headers that both use the same storage (a C array of pointers to Python >> objects, in CPython), and C API functions like PySequence_Fast_GET_ITEM >> don't distinguish between the two. >> >> However, list is resizable, and presumably a "namedlist" would not be. >> That makes things more complicated for both the interface (there's no is-a >> relationship; a type without append is not a list--and, worse, a type that >> has __setitem__ but can't handle slice replacement is not a list but that's >> very hard to detect...) and the implementation (e.g., a list reserves extra >> space at the end to avoid having to reallocate on every append). >> >> (Python _also_ has an array type, which is for homogenous simple types >> (like 32-bit int) which can store the values directly, as opposed to tuple >> and list, which store (pointers to) heterogenous normal Python objects.) >> >> One should note that particular property of named tuple is memory saving. >> So one can expect similar property of mutable named tuple too. >> >> >> If you don't need to access the items by index for whatever reason, you >> don't need a namedtuple, and using one as a misguided misoptimization is a >> bad idea. >> >> Besides the fact that a normal class with __slots__ is also small, and >> even a normal class with a dict (in newer CPython versions and PyPy) not >> that much bigger, besides the fact that you can eliminate the row overhead >> rather than just slightly reducing it by using, e.g., a 2D array, you're >> optimizing the wrong thing in the first place--if your rows have 9 >> elements, reducing the row overhead is focusing on fixing 10% of your >> overhead, while reducing or eliminating the element overhead by using, >> e.g., a 2D numpy array of low-level values fixes the 90% (along with the >> 10%). >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From abarnert at yahoo.com Thu Mar 19 19:42:55 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Thu, 19 Mar 2015 11:42:55 -0700 Subject: [Python-ideas] A mutable alternative to namedtuple In-Reply-To: References: <55086248.2010009@trueblade.com> <5F372C2E-00E0-4432-9832-91C69AB69092@yahoo.com> <4B123971-940F-4ADD-943C-6C82B0FA282D@yahoo.com> Message-ID: > On Mar 19, 2015, at 10:57 AM, Zaur Shibzukhov wrote: > > > > 2015-03-19 14:22 GMT+03:00 Andrew Barnert : >>> On Mar 19, 2015, at 3:43 AM, Zaur Shibzukhov wrote: >>> >>> Yes. But: >>> >>> First. >>> >>> >>>print(sys.getsizeof(list([])), sys.getsizeof(tuple([]))) >>> 64 48 >>> print(sys.getsizeof(list([1,2])), sys.getsizeof(tuple([1,2]))) >>> 104 64 >>> >>> >>> Second. >>> >>> Tuple object allocates it's memory 1 time, list object allocates it's memory 2 time. >> >> As I explained, list leaves room for expansion, so it doesn't have to keep reallocating with every append. A 2-element list actually has room for... off the top of my head, I think 8 elements; then it multiplies the capacity every time it runs out on an append. > > >>>for i in [4, 12, 28, 68]: print(sys.getsizeof(list(range(i))), end=' ') > 120 216 360 720 And what's your point? That's exactly what you should expect if lists keep extra slack space at the end to avoid reallocation on expansion. > > In a C extension, you can create a list with whatever specific capacity you want, but that isn't exposed to Python > > (presumably because such a micro-optimization is rarely useful, and would complicate the API, as well as being an > > attractive nuisance to people who insist on optimizing the wrong things). So, if you really needed this optimization, you could > > implement your namedlist in C. > > Yes you right. Actual optimization one could archive with the help of Cython if necessary. Not if you're trying to put this in the stdlib; you have to write the C extension. But if the person who thinks this is necessary (you) prefers to use Cython instead of C, that seems like another argument against putting it in the stdlib... > But what use case are you imagining where the extra 40 bytes per row matter, but the 64 bytes per row don't, and neither do the 56 or so bytes per element per row? I'm having a hard time imagining any case where using tuples would be necessary where it wouldn't be woefully insufficient. I've now asked this twice, and you've avoided answering it both times to present further irrelevancies. This is the key question. If you don't have any use case for which this matters, then it doesn't matter. >> That is why >> >> >>>%time for i in range(100000): tuple(i for i in range(15)) >> CPU times: user 241 ms, sys: 1.75 ms, total: 243 ms >> Wall time: 243 ms >> >> >>>%time for i in range(100000): list(i for i in range(15)) >> CPU times: user 318 ms, sys: 1.94 ms, total: 320 ms >> Wall time: 321 ms >> >> Certainly this can have or not have a value depending on use case. >>>import namedlist > > > If you're going to show benchmarks, you really should use %timeit rather than %time, and also understand that "i for i in > > range(15)" is just going to give you a slower but equivalent iterable to just using "range(15)" in the first place. > > > But more importantly, what use case are you considering where this extra 0.8us for construction of each row object will > > matter, but the difference between a namedtuple (or, presumably, namedlist) vs. a tuple won't? > > >>>class Point: > __slots__ = 'x', 'y', 'z' > def __init__(self, x, y, z): > self.x, self.y, self.z = x, y, z > > >>>asizeof.asizeof(Point(1,2,3)) > 232 > >>>import collections > >>>Point2 = collections.namedtuple('Point', 'x y z') > >>>Point3 = namedlist.namedtuple('Point', 'x y z') > >>>Point4 = namedlist.namedlist('Point', 'x y z') > >>>asizeof.asizeof(Point2(1,2,3)) > 120 > >>>asizeof.asizeof(Point3(1,2,3)) > 120 > >>>asizeof.asizeof(Point4(1,2,3)) > 1704 Despite the name, the third-party module "namedlist" does not appear to make classes that inherit from list, or use a list for storage. You just get a normal class (using an OrderedDict __dict__ by default, or optionally __slots__) that provides a __getitem__ method implemented in terms of __getattr__. So I don't see how this is even vaguely relevant here. And, even if that module were relevant, how would it have anything to do with answering "Where is the use case where this construction time matters?" You don't even _mention_ construction time here. > > And, even if you did, you're focusing on something that accounts for at worst 25% of the construction time, and 0% of the > > time for all the actual work you do with the object after construction. > > > Again, unless you have many millions of these, neither the memory not the construction time is going to matter--and if you do, > > a new type that's more like tuple isn't going to be anywhere near sufficient to make a noticeable difference, because you're > > optimizing the wrong part of both the memory and the time. > > Possibly basic gain in the savings of memory, especially in the context of long running processes, limitations on the memory on hostings, long loop + memory fragmentation in python heap space, ... No. This is basic computer science. Optimizing parts of your code that aren't actually relevant because they contribute only a tiny percentage to the waste doesn't actually improve anything. >> >> --- >> Zaur Shibzukhov >> >> >> 2015-03-19 13:07 GMT+03:00 Andrew Barnert : >>>> On Mar 19, 2015, at 2:11 AM, Zaur Shibzukhov wrote: >>>> >>>> That all right. But I want to note that `collections.namedtuple` has several properties that make them exclusive: >>>> >>>> 1. Fast creation; >>>> 2. Minimal memory capacity; >>>> 3. Fast sequence interface; >>>> 4. Attribute access to elements via properties. >>>> >>>> Different namedtuple alternatives has different sets of properties that make them more ore less suitable depending on use cases. >>>> >>>> So if someone search alternative of collections.namedtuple that support assignment too then it could be constructed on top of array (actually "tuple" but with assignment support, but python seems have not such array type). >>> >>> Again, do you not know about list? >>> >>>> >>>> --- >>>> Zaur Shibzukhov >>>> >>>> >>>> 2015-03-19 11:37 GMT+03:00 Andrew Barnert : >>>>>> On Mar 19, 2015, at 12:04 AM, Zaur Shibzukhov wrote: >>>>>> >>>>>> >>>>>> >>>>>> ???????, 17 ????? 2015 ?., 20:21:01 UTC+3 ???????????? Eric V. Smith ???????: >>>>>>> >>>>>>> On 03/17/2015 12:52 PM, Luciano Ramalho wrote: >>>>>>> > Sometimes we need a simple class to hold some mutable attributes, >>>>>>> > provide a nice repr, support == for testing, and support iterable >>>>>>> > unpacking, so you can write: >>>>>>> > >>>>>>> >>>> p = Point(3, 4) >>>>>>> >>>> x, y = p >>>>>>> > >>>>>>> > That's very much like the classes built by namedtuple, but mutable. >>>>>>> >>>>>>> https://pypi.python.org/pypi/namedlist >>>>>>> >>>>>>> It also adds default values to the generated constructor, which may or >>>>>>> may not be desirable. But if used exactly like collections.namedtuple, >>>>>>> it ignores the default values. >>>>>>> >>>>>>> Eric. >>>>>> Since named tuple is considered as an object that is a tuple with attribute access. >>>>>> The mutable alternative could be considered as an array with attribute access. >>>>>> Array in this context is tuple-like object that support assign operation. >>>>>> Since python have not such object there are different approaches tomutable named tuple alternatives. >>>>> >>>>> Python definitely does have such an object: list. A list is effectively the same as a tuple but mutable; it's the paradigm MutableSequence while tuple is the paradigm Sequence. Under the covers they have very similar headers that both use the same storage (a C array of pointers to Python objects, in CPython), and C API functions like PySequence_Fast_GET_ITEM don't distinguish between the two. >>>>> >>>>> However, list is resizable, and presumably a "namedlist" would not be. That makes things more complicated for both the interface (there's no is-a relationship; a type without append is not a list--and, worse, a type that has __setitem__ but can't handle slice replacement is not a list but that's very hard to detect...) and the implementation (e.g., a list reserves extra space at the end to avoid having to reallocate on every append). >>>>> >>>>> (Python _also_ has an array type, which is for homogenous simple types (like 32-bit int) which can store the values directly, as opposed to tuple and list, which store (pointers to) heterogenous normal Python objects.) >>>>> >>>>>> One should note that particular property of named tuple is memory saving. >>>>>> So one can expect similar property of mutable named tuple too. >>>>> >>>>> If you don't need to access the items by index for whatever reason, you don't need a namedtuple, and using one as a misguided misoptimization is a bad idea. >>>>> >>>>> Besides the fact that a normal class with __slots__ is also small, and even a normal class with a dict (in newer CPython versions and PyPy) not that much bigger, besides the fact that you can eliminate the row overhead rather than just slightly reducing it by using, e.g., a 2D array, you're optimizing the wrong thing in the first place--if your rows have 9 elements, reducing the row overhead is focusing on fixing 10% of your overhead, while reducing or eliminating the element overhead by using, e.g., a 2D numpy array of low-level values fixes the 90% (along with the 10%). > -------------- next part -------------- An HTML attachment was scrubbed... URL: From abarnert at yahoo.com Thu Mar 19 19:48:02 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Thu, 19 Mar 2015 11:48:02 -0700 Subject: [Python-ideas] Rewriting the build system (part 2) In-Reply-To: References: <20150318152150.GN10871@tonks> Message-ID: On Mar 19, 2015, at 4:12 AM, anatoly techtonik wrote: > >> On Wed, Mar 18, 2015 at 6:21 PM, Florian Bruhin wrote: >> * anatoly techtonik [2015-03-18 10:21:46 +0300]: >>> On Mon, Mar 16, 2015 at 11:33 PM, Ryan Gonzalez wrote: >>>>> >>>>> But to miss-paraphrase Winston Churchill [1]: >>>>> >>>>> "Autotools is the worst build system except for all the others." >>>>> >>>>> Is there really something better out there? >>>> >>>> Fbuild, Waf, Meson, GN, Boost.Build, Tundra, Bam, ... >>> >>> Is there anything Javascriptey? Python needs to move to web. I heard >>> something about Grunt, which was replaced by Gulp, but no idea if they >>> really mature enough to include "system features detection" and "user >>> configured builds" (or work with C/C++ compilers at all). >> >> What does "Python needs to move to web" to do with it's build system >> being "Javascriptey"? > > Because things like Twisted are de-facto standard in JavaScript world > (Node.js), and Python will benefit more if people look for how things are > implemented in non-C worlds. Is this a sarcastic comment? Or are you seriously suggesting that because people use things like the stdlib's asyncio, gevent, Tornado, etc. in Python, which have different paradigms than Twisted, the one Node copied, we need to force some other part of Python to be more like Node in some other way, because "move to the web" means "be more like the latest (but nowhere near most popular) framework for building web services"? Also, correct me if I'm wrong, but isn't Node implemented in C, with a build system in C and Python? So if we really want to be more like Node for some reason, we should make the build system Pythony instead of Javascripty? From rymg19 at gmail.com Thu Mar 19 19:55:05 2015 From: rymg19 at gmail.com (Ryan Gonzalez) Date: Thu, 19 Mar 2015 13:55:05 -0500 Subject: [Python-ideas] Rewriting the build system (part 2) In-Reply-To: References: <20150318152150.GN10871@tonks> Message-ID: On Thu, Mar 19, 2015 at 1:48 PM, Andrew Barnert < abarnert at yahoo.com.dmarc.invalid> wrote: > On Mar 19, 2015, at 4:12 AM, anatoly techtonik > wrote: > > > >> On Wed, Mar 18, 2015 at 6:21 PM, Florian Bruhin > wrote: > >> * anatoly techtonik [2015-03-18 10:21:46 +0300]: > >>> On Mon, Mar 16, 2015 at 11:33 PM, Ryan Gonzalez > wrote: > >>>>> > >>>>> But to miss-paraphrase Winston Churchill [1]: > >>>>> > >>>>> "Autotools is the worst build system except for all the others." > >>>>> > >>>>> Is there really something better out there? > >>>> > >>>> Fbuild, Waf, Meson, GN, Boost.Build, Tundra, Bam, ... > >>> > >>> Is there anything Javascriptey? Python needs to move to web. I heard > >>> something about Grunt, which was replaced by Gulp, but no idea if they > >>> really mature enough to include "system features detection" and "user > >>> configured builds" (or work with C/C++ compilers at all). > >> > >> What does "Python needs to move to web" to do with it's build system > >> being "Javascriptey"? > > > > Because things like Twisted are de-facto standard in JavaScript world > > (Node.js), and Python will benefit more if people look for how things are > > implemented in non-C worlds. > > Is this a sarcastic comment? > > Or are you seriously suggesting that because people use things like the > stdlib's asyncio, gevent, Tornado, etc. in Python, which have different > paradigms than Twisted, the one Node copied, we need to force some other > part of Python to be more like Node in some other way, because "move to the > web" means "be more like the latest (but nowhere near most popular) > framework for building web services"? > > Also, correct me if I'm wrong, but isn't Node implemented in C, with a > build system in C and Python? So if we really want to be more like Node for > some reason, we should make the build system Pythony instead of Javascripty? > You're not wrong. Node uses Gyp, although, if Chromium switches to GN (written in C++), Node might switch, too. > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- Ryan [ERROR]: Your autotools build scripts are 200 lines longer than your program. Something?s wrong. http://kirbyfan64.github.io/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From tjreedy at udel.edu Thu Mar 19 21:40:52 2015 From: tjreedy at udel.edu (Terry Reedy) Date: Thu, 19 Mar 2015 16:40:52 -0400 Subject: [Python-ideas] gofmt for Python: standardized styling as a language feature In-Reply-To: References: <5508DBB0.2010209@stoneleaf.us> <5508F42F.8020604@sdamon.com> Message-ID: On 3/19/2015 10:32 AM, Nicholas Chammas wrote: > OK, I hereby abandon this proposal. :) Thank you for being so sensible ;-) Let me first quote (out of order) your conclusion: > I think any part of the development process we automate > away is a good thing (stdlib inclusion or not notwithstanding).> I agree, which is why I expect to do some work making it easier to download 3rd party modules and apply them to python code, both in a gui IDE framework (Idle). I think automation should respect and augment, not replace. the human programmer. > To summarize the conclusion I believe this thread has arrived at: ... > Ignoring that difficulty, there is also the problem of what ?spec? > to use for any such auto-styler. PEP 8 is an obvious choice, but it > was written as a guide for humans, not a spec for programmatic > implementation. A program that treats PEP 8 guidelines as rigid rules, that ignores the first guideline "A Foolish Consistency is the Hobgoblin of Little Minds", that removes the human choices intentionally included in PEP 8, is not PEP 8 compliant. I believe one of the motivations for some recent PEP 8 amendments was some unhappiness with pep8.py or other checkers. > Furthermore, there are doubts about PEP 8?s utility > as the basis for a universal auto-styler, as it was intended for > code specifically in the Python standard library. Pydev is a relatively small group of volunteers who otherwise work either for themselves or for various organizations scattered around the world. Having the freedom to put actual readability ahead of rigid consistency makes volunteering more attractive. I regard code as a form of poetry, but prefer a flexible but defined structure to free-form mishmash. Google, as an example, is a very large corporation that pays people well to adhere to its guidelines and rules. This is a different situation and its best choice for Python styling may well be different from pydev. > On Wed, Mar 18, 2015 at 4:38 PM, Andrew Barnert > > wrote: > > But that implies that it's worth someone making a spec out of PEP 8. > That implies going through PEP 8, getting a sense of what parts are > (unambiguously and uncontroversially) mechanizable, and reorganizing > and minimally rewriting it to get that sense across. Having numbered > rules, and named anchors for each one in the document (or at least > to narrower sections, so you can link to something more specific > than "Programming Recommendations") would also be helpful. > > This is an excellent point and one I did not consider. Having the guidelines numbered (id'ed), even though still regarded as guidelines, could help communication. Checkers could then easily refer to specific guidelines. > On Wed, Mar 18, 2015 at 6:17 PM, Terry Reedy > wrote: > > (Note that autoPEP8 only changes whitespace, which is a small part > of PEP 8 recommendations, and the safest aspect of code to change.) > > Minor point: autopep8 can do more than just whitespace changes with the > |--aggressive| flag > . Thanks for the link. I was specifically thinking of global renamings to satisfy PEP 8's Naming Conventions. Idlelib has a mishmash of module and function/method name styles. (I might look and see what can be done with the undocumented (except as 'unstable') libe2to3.) > On Wed, Mar 18, 2015 at 7:21 PM, Terry Reedy > wrote: > > We should be careful about adding anything. When we do, we should > add libraries, not applications. We should especially not > officially endorse one of many opinion-based competitors. > > I have come around to this view. Accepting this view, we can still ask if the stdlib should have more (policy-neutral) string or list of strings reforming components than it does now. Textwrap has a few functions for strings in general. Idle has about 10 formatting functions exposed on its editor Format menu, but they are not exposed for general use. Tools/Scripts/reindent.py contain some of the same functions as does autopep8. Should there be a code format module that all three, and other programs, could import? Idle and Tools/Scrips/highlight.py both have code to analyze Python code tokens into categories and colorize by category. (Idle uses tkinter tags, highlight.py uses ANSI or HTML tags). Can and should both use a common code analysis function that could be imported? This might actually ease maintainance. Should a generalized 2to3 package be renamed 'codefix' and documented? As 2to3, it is already used by both Tools/Scripts/2to2 and optionally by autopep8, and perhaps by others. I believe it could also be used for some non-2to3 pep8 fixes. -- Terry Jan Reedy From guido at python.org Thu Mar 19 22:01:39 2015 From: guido at python.org (Guido van Rossum) Date: Thu, 19 Mar 2015 14:01:39 -0700 Subject: [Python-ideas] gofmt for Python: standardized styling as a language feature In-Reply-To: References: <5508DBB0.2010209@stoneleaf.us> <5508F42F.8020604@sdamon.com> Message-ID: On Thu, Mar 19, 2015 at 1:40 PM, Terry Reedy wrote: > > A program that treats PEP 8 guidelines as rigid rules, that ignores the > first guideline "A Foolish Consistency is the Hobgoblin of Little Minds", > that removes the human choices intentionally included in PEP 8, is not PEP > 8 compliant. I believe one of the motivations for some recent PEP 8 > amendments was some unhappiness with pep8.py or other checkers. > I have always been skeptical of automatic style checkers. They encourage a nit-pickish attitude that makes the development process less effective, by encouraging critique of superficial style elements rather than of the code's structure and meaning. In contrast, the intention of PEP 8 was to make code more readable so that its structure and meaning are more easily understood. > On Wed, Mar 18, 2015 at 4:38 PM, Andrew Barnert >> > > wrote: >> >> But that implies that it's worth someone making a spec out of PEP 8. >> That implies going through PEP 8, getting a sense of what parts are >> (unambiguously and uncontroversially) mechanizable, and reorganizing >> and minimally rewriting it to get that sense across. Having numbered >> rules, and named anchors for each one in the document (or at least >> to narrower sections, so you can link to something more specific >> than "Programming Recommendations") would also be helpful. >> >> This is an excellent point and one I did not consider. >> > > Having the guidelines numbered (id'ed), even though still regarded as > guidelines, could help communication. Checkers could then easily refer to > specific guidelines. > I worry this will just encourage the nit-picky attitude more. > > On Wed, Mar 18, 2015 at 6:17 PM, Terry Reedy > > wrote: >> >> (Note that autoPEP8 only changes whitespace, which is a small part >> of PEP 8 recommendations, and the safest aspect of code to change.) >> >> Minor point: autopep8 can do more than just whitespace changes with the >> |--aggressive| flag >> . >> > > Thanks for the link. I was specifically thinking of global renamings to > satisfy PEP 8's Naming Conventions. Idlelib has a mishmash of module and > function/method name styles. (I might look and see what can be done with > the undocumented (except as 'unstable') libe2to3.) > This feels hard to automate, because you don't know what names are part of an external specification. > On Wed, Mar 18, 2015 at 7:21 PM, Terry Reedy > > wrote: >> >> We should be careful about adding anything. When we do, we should >> add libraries, not applications. We should especially not >> officially endorse one of many opinion-based competitors. >> >> I have come around to this view. >> > > Accepting this view, we can still ask if the stdlib should have more > (policy-neutral) string or list of strings reforming components than it > does now. Textwrap has a few functions for strings in general. Idle has > about 10 formatting functions exposed on its editor Format menu, but they > are not exposed for general use. Tools/Scripts/reindent.py contain some of > the same functions as does autopep8. Should there be a code format module > that all three, and other programs, could import? > > Idle and Tools/Scrips/highlight.py both have code to analyze Python code > tokens into categories and colorize by category. (Idle uses tkinter tags, > highlight.py uses ANSI or HTML tags). Can and should both use a common > code analysis function that could be imported? This might actually ease > maintainance. > > Should a generalized 2to3 package be renamed 'codefix' and documented? As > 2to3, it is already used by both Tools/Scripts/2to2 and optionally by > autopep8, and perhaps by others. I believe it could also be used for some > non-2to3 pep8 fixes. I think that a quick "clean up whitespace" feature would be a nice addition to IDLE, provided it's a user-selectable menu items (similar to the existing indent/dedent operations). Refactoring, however, is fiendishly hard for Python -- give it a try in e.g. PyCharm and you will agree. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Thu Mar 19 22:41:50 2015 From: donald at stufft.io (Donald Stufft) Date: Thu, 19 Mar 2015 17:41:50 -0400 Subject: [Python-ideas] gofmt for Python: standardized styling as a language feature In-Reply-To: References: <5508DBB0.2010209@stoneleaf.us> <5508F42F.8020604@sdamon.com> Message-ID: > On Mar 19, 2015, at 5:01 PM, Guido van Rossum wrote: > > On Thu, Mar 19, 2015 at 1:40 PM, Terry Reedy > wrote: > A program that treats PEP 8 guidelines as rigid rules, that ignores the first guideline "A Foolish Consistency is the Hobgoblin of Little Minds", that removes the human choices intentionally included in PEP 8, is not PEP 8 compliant. I believe one of the motivations for some recent PEP 8 amendments was some unhappiness with pep8.py or other checkers. > > I have always been skeptical of automatic style checkers. They encourage a nit-pickish attitude that makes the development process less effective, by encouraging critique of superficial style elements rather than of the code's structure and meaning. In contrast, the intention of PEP 8 was to make code more readable so that its structure and meaning are more easily understood. My experience is actually the exact opposite. When a machine is responsible for style enforcement people generally don?t feel as bad about being asked to adjust the formatting because it?s just a machine, it can?t make value judgements. When a person asks someone to adjust something to make it easier to read it often times ends up causing a distraction where people tend to feel like since it?s a person asking them to do it they are passing judgement on their code, or they don?t agree with it and since it?s a human saying it it feels more like there is room to budge so you end up with people more likely to be willing to ask for an exemption. In my experience projects without mechnical enforcement ends up with massively inconsistent style all throughout the code base. The Python stdlib is a good example of this. Contrast this to projects like github.com/pyca/cryptography where there is an automated mechanical enforcement of style and it?s got some bare minimum of consistency throughout the entire code base. > > On Wed, Mar 18, 2015 at 4:38 PM, Andrew Barnert > > >> wrote: > > But that implies that it's worth someone making a spec out of PEP 8. > That implies going through PEP 8, getting a sense of what parts are > (unambiguously and uncontroversially) mechanizable, and reorganizing > and minimally rewriting it to get that sense across. Having numbered > rules, and named anchors for each one in the document (or at least > to narrower sections, so you can link to something more specific > than "Programming Recommendations") would also be helpful. > > This is an excellent point and one I did not consider. > > Having the guidelines numbered (id'ed), even though still regarded as guidelines, could help communication. Checkers could then easily refer to specific guidelines. > > I worry this will just encourage the nit-picky attitude more. > > On Wed, Mar 18, 2015 at 6:17 PM, Terry Reedy > >> wrote: > > (Note that autoPEP8 only changes whitespace, which is a small part > of PEP 8 recommendations, and the safest aspect of code to change.) > > Minor point: autopep8 can do more than just whitespace changes with the > |--aggressive| flag > >. > > Thanks for the link. I was specifically thinking of global renamings to satisfy PEP 8's Naming Conventions. Idlelib has a mishmash of module and function/method name styles. (I might look and see what can be done with the undocumented (except as 'unstable') libe2to3.) > > This feels hard to automate, because you don't know what names are part of an external specification. > > On Wed, Mar 18, 2015 at 7:21 PM, Terry Reedy > >> wrote: > > We should be careful about adding anything. When we do, we should > add libraries, not applications. We should especially not > officially endorse one of many opinion-based competitors. > > I have come around to this view. > > Accepting this view, we can still ask if the stdlib should have more (policy-neutral) string or list of strings reforming components than it does now. Textwrap has a few functions for strings in general. Idle has about 10 formatting functions exposed on its editor Format menu, but they are not exposed for general use. Tools/Scripts/reindent.py contain some of the same functions as does autopep8. Should there be a code format module that all three, and other programs, could import? > > Idle and Tools/Scrips/highlight.py both have code to analyze Python code tokens into categories and colorize by category. (Idle uses tkinter tags, highlight.py uses ANSI or HTML tags). Can and should both use a common code analysis function that could be imported? This might actually ease maintainance. > > Should a generalized 2to3 package be renamed 'codefix' and documented? As 2to3, it is already used by both Tools/Scripts/2to2 and optionally by autopep8, and perhaps by others. I believe it could also be used for some non-2to3 pep8 fixes. > > I think that a quick "clean up whitespace" feature would be a nice addition to IDLE, provided it's a user-selectable menu items (similar to the existing indent/dedent operations). Refactoring, however, is fiendishly hard for Python -- give it a try in e.g. PyCharm and you will agree. > > -- > --Guido van Rossum (python.org/~guido ) > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From nicholas.chammas at gmail.com Thu Mar 19 23:13:53 2015 From: nicholas.chammas at gmail.com (Nicholas Chammas) Date: Thu, 19 Mar 2015 22:13:53 +0000 Subject: [Python-ideas] gofmt for Python: standardized styling as a language feature In-Reply-To: References: <5508DBB0.2010209@stoneleaf.us> <5508F42F.8020604@sdamon.com> Message-ID: I second Donald?s remarks regarding the impact that automatic style checkers have on the dev process. The Apache Spark project, for example, makes a number of automated checks on all proposed patches. One of those checks is a style check. When a proposed patch fails a style check, an informative message explaining why is posted in the CI build log for the author to review, and the author typically just updates their patch accordingly. Here?s an example from an Apache Spark PR that touched some Python code. The author?s patch had some blank lines that contained whitespace, so the patch failed the automated style check. The author just updated their patch to pass the style check and the discussion quickly returned to the meat of what the patch was about. >From what I?ve seen so far, the process of a patch failing style checks plays out almost exactly like it would if a contributor had submitted a patch that failed unit tests. The contributor just updates their patch to pass those checks and everyone moves on. There is no arguing or friction. And the result in the case of the Spark project is that all the Python code in the repo passes the pep8 checks we?ve enabled. Granted, this doesn?t mean the code is readable or beautiful or correct, but it simply means that there is a small but non-trivial set of style problems and inconsistencies that we just don?t have to worry about. For a large, open source project with hundreds of contributors from many different organizations and backgrounds, that?s a solid win. Nick ? On Thu, Mar 19, 2015 at 5:42 PM Donald Stufft wrote: > On Mar 19, 2015, at 5:01 PM, Guido van Rossum wrote: > > On Thu, Mar 19, 2015 at 1:40 PM, Terry Reedy wrote: >> >> A program that treats PEP 8 guidelines as rigid rules, that ignores the >> first guideline "A Foolish Consistency is the Hobgoblin of Little Minds", >> that removes the human choices intentionally included in PEP 8, is not PEP >> 8 compliant. I believe one of the motivations for some recent PEP 8 >> amendments was some unhappiness with pep8.py or other checkers. >> > > I have always been skeptical of automatic style checkers. They encourage a > nit-pickish attitude that makes the development process less effective, by > encouraging critique of superficial style elements rather than of the > code's structure and meaning. In contrast, the intention of PEP 8 was to > make code more readable so that its structure and meaning are more easily > understood. > > > My experience is actually the exact opposite. When a machine is > responsible for style enforcement people generally don?t feel as bad about > being asked to adjust the formatting because it?s just a machine, it can?t > make value judgements. When a person asks someone to adjust something to > make it easier to read it often times ends up causing a distraction where > people tend to feel like since it?s a person asking them to do it they are > passing judgement on their code, or they don?t agree with it and since it?s > a human saying it it feels more like there is room to budge so you end up > with people more likely to be willing to ask for an exemption. > > In my experience projects without mechnical enforcement ends up with > massively inconsistent style all throughout the code base. The Python > stdlib is a good example of this. Contrast this to projects like > github.com/pyca/cryptography where there is an automated mechanical > enforcement of style and it?s got some bare minimum of consistency > throughout the entire code base. > > > >> On Wed, Mar 18, 2015 at 4:38 PM, Andrew Barnert >>> >> > wrote: >>> >>> But that implies that it's worth someone making a spec out of PEP 8. >>> That implies going through PEP 8, getting a sense of what parts are >>> (unambiguously and uncontroversially) mechanizable, and reorganizing >>> and minimally rewriting it to get that sense across. Having numbered >>> rules, and named anchors for each one in the document (or at least >>> to narrower sections, so you can link to something more specific >>> than "Programming Recommendations") would also be helpful. >>> >>> This is an excellent point and one I did not consider. >>> >> >> Having the guidelines numbered (id'ed), even though still regarded as >> guidelines, could help communication. Checkers could then easily refer to >> specific guidelines. >> > > I worry this will just encourage the nit-picky attitude more. > >> >> On Wed, Mar 18, 2015 at 6:17 PM, Terry Reedy >> > wrote: >>> >>> (Note that autoPEP8 only changes whitespace, which is a small part >>> of PEP 8 recommendations, and the safest aspect of code to change.) >>> >>> Minor point: autopep8 can do more than just whitespace changes with the >>> |--aggressive| flag >>> . >>> >> >> Thanks for the link. I was specifically thinking of global renamings to >> satisfy PEP 8's Naming Conventions. Idlelib has a mishmash of module and >> function/method name styles. (I might look and see what can be done with >> the undocumented (except as 'unstable') libe2to3.) >> > > This feels hard to automate, because you don't know what names are part of > an external specification. > > >> On Wed, Mar 18, 2015 at 7:21 PM, Terry Reedy >> > wrote: >>> >>> We should be careful about adding anything. When we do, we should >>> add libraries, not applications. We should especially not >>> officially endorse one of many opinion-based competitors. >>> >>> I have come around to this view. >>> >> >> Accepting this view, we can still ask if the stdlib should have more >> (policy-neutral) string or list of strings reforming components than it >> does now. Textwrap has a few functions for strings in general. Idle has >> about 10 formatting functions exposed on its editor Format menu, but they >> are not exposed for general use. Tools/Scripts/reindent.py contain some of >> the same functions as does autopep8. Should there be a code format module >> that all three, and other programs, could import? >> >> Idle and Tools/Scrips/highlight.py both have code to analyze Python code >> tokens into categories and colorize by category. (Idle uses tkinter tags, >> highlight.py uses ANSI or HTML tags). Can and should both use a common >> code analysis function that could be imported? This might actually ease >> maintainance. >> >> Should a generalized 2to3 package be renamed 'codefix' and documented? As >> 2to3, it is already used by both Tools/Scripts/2to2 and optionally by >> autopep8, and perhaps by others. I believe it could also be used for some >> non-2to3 pep8 fixes. > > > I think that a quick "clean up whitespace" feature would be a nice > addition to IDLE, provided it's a user-selectable menu items (similar to > the existing indent/dedent operations). Refactoring, however, is fiendishly > hard for Python -- give it a try in e.g. PyCharm and you will agree. > > -- > --Guido van Rossum (python.org/~guido) > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > > > --- > Donald Stufft > PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From cs at zip.com.au Thu Mar 19 23:29:01 2015 From: cs at zip.com.au (Cameron Simpson) Date: Fri, 20 Mar 2015 09:29:01 +1100 Subject: [Python-ideas] gofmt for Python: standardized styling as a language feature In-Reply-To: <5508F42F.8020604@sdamon.com> References: <5508F42F.8020604@sdamon.com> Message-ID: <20150319222901.GA72481@cskk.homeip.net> On 17Mar2015 23:42, Alexander Walters wrote: >What coding style would be used for such a tool? PEP-8? What about a >tool for those of us who believe that PEP-8 got a few things wrong? autopep8 at least lets you pick and choose specific issues to apply or ignore. As someone who now aims for a "mostly pep8" style, that is very handy. Aside: I always wish "fix it" tools to have this kind of granularity. In a former life I got to use SGI's C compiler. Like GCC, it had a "lots of warnings" mode, somewhat like a builtin lint. _Unlike_ GCC, the SGI compiler let one suppress specific warnings, which was very very nice. >What about for the tribal knowledge that pythonistas develop over time >that are not part of the standard? Would such a tool include this? Such things get progressively harder to formalise into stuff a tool can implement. I imagine one could implement quite a lot of knobs, but I also imagine that some knobs would require huge code support, and perhaps be intrusive enough to make the default knobs (very?) hard to debug/maintain. Using PEP8 as a guide at least gives a concrete baseline for the feature set. I'm not arguing that that is all one should ever want, just that it is well defined and very broadly appplicable. >And what of third party tools that do the same thing? I have no >evidence to support this, but I have the feeling that when the >standard library includes something in a new space, third party work >in that space slows (json and sqlite come to mind). Sure. But that isn't always a bad thing; life is too short. In an ideal world the included battery is amenable to extension and then the supplied battery enables faster third party work, at least in the form of adding to the battery. >Good coding style really should not be something we ship in the >library, it should be something we teach. Yes and no. I wholely agree that good style comes from within, and that it should be taught "what and why"; blindly applied mechanical style does not itself improve code quality. Though having consistent style does at least usually improve code readability and maintainability. Not to mention diff noise. That said, having a tool that will _apply_ a desirable style is very useful. Taking myself as an example: my personal library has gads of old code with bad style (and bad design choices!) I tend to mostly-PEP8 the old code as I come across it during development (committing separately from semantic changes unless I've rewritten something outright). That work is slow and incremental. Having just downloaded autopep8 and given it a tiny trial run against a single module I found that a bare run produced a diff of more or less the entire file, but adding "--indent-size 2" (my personal major divergence from PEP8) got the diff down to a reviewable bunch of small changes, all of which I approve. Performing that task by hand was an ongoing indefinite inconsistent minor improvement activity. Now I can branch and restyle vigorously, because I have a tool to do the picky editing. Yay! Aside: why the bad style from the outset? When coming to a new language or system I find it useful to write batteries myself, sometimes because the supplied battery does not match my needs but usually because the battery is tractable and I learn quite a lot about the problem domain during this process. And the same applies to style; my old code has poor style, some leaked across from other languages. Having written plenty of code according to my whim, I can then look at a suggested style like PEP8 and evaluate it with context. Now I mostly aim for PEP8 because I've come to informed decisions about its choices, and mostly agree with them. Metaaside: the above is also something of an excuse for the state of some of the code I've put out there:-) Cheers, Cameron Simpson Mac OS X. Because making Unix user-friendly is easier than debugging Windows. - Mike Dawson, Macintosh Systems Administrator and Consultation. mdawson at mac.com http://herowars.onestop.net From guido at python.org Fri Mar 20 00:55:08 2015 From: guido at python.org (Guido van Rossum) Date: Thu, 19 Mar 2015 16:55:08 -0700 Subject: [Python-ideas] gofmt for Python: standardized styling as a language feature In-Reply-To: <550B3FED.9020200@udel.edu> References: <5508DBB0.2010209@stoneleaf.us> <5508F42F.8020604@sdamon.com> <550B3FED.9020200@udel.edu> Message-ID: On Thu, Mar 19, 2015 at 2:30 PM, Terry Reedy wrote: > On 3/19/2015 5:01 PM, Guido van Rossum wrote: > >> On Thu, Mar 19, 2015 at 1:40 PM, Terry Reedy > > > Having the guidelines numbered (id'ed), even though still regarded >> as guidelines, could help communication. Checkers could then easily >> refer to specific guidelines. >> >> I worry this will just encourage the nit-picky attitude more. >> > > I agree that this is a possible negative. > I'll leave the rest of the discussion to other participants. It seems all possible POVs are well-represented. > Thanks for the link. I was specifically thinking of global >> renamings to satisfy PEP 8's Naming Conventions. Idlelib has a >> mishmash of module and function/method name styles. (I might look >> and see what can be done with the undocumented (except as >> 'unstable') libe2to3.) >> >> This feels hard to automate, because you don't know what names are part >> of an external specification. >> > > Modernizing idlelib names, in relation to PEP 434, is a separate > discussion. > > I think that a quick "clean up whitespace" feature would be a nice >> addition to IDLE, provided it's a user-selectable menu items (similar to >> the existing indent/dedent operations). >> > > The Format menu now has 'Strip trailing whitespace', which may be new > since you last looked. It needs to strip trailing blank lines, as > reindent.py does, in addition to trailing whitespace on each line, to make > files ready to commit to the cpython repository. Did you have in mind also > doing within-line cleanups (as a separate menu entry)? > I probably spoke out of turn (I haven't really used IDLE in years). But I could imagine another command that did something like autopep8 on a selection, making sure the whitespace around various operators is consistent with PEP 8, fixing indentation levels, and even inserting/removing blank lines in certain cases. I'm not sure I would want it to automatically break long lines, as I often have very specific ideas about the best place to break a long line, which require understanding the meaning of the code. Have you used Emacs? The python-mode written by Tim Peters and maintained by Barry Warsaw has some pretty amazing reformatting capabilities (though I rarely use them, because my fingers automatically type correctly formatted code :-). -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From ben+python at benfinney.id.au Fri Mar 20 04:02:29 2015 From: ben+python at benfinney.id.au (Ben Finney) Date: Fri, 20 Mar 2015 14:02:29 +1100 Subject: [Python-ideas] gofmt for Python: standardized styling as a language feature References: <5508F42F.8020604@sdamon.com> <20150319222901.GA72481@cskk.homeip.net> Message-ID: <857fucqsl6.fsf@benfinney.id.au> Cameron Simpson writes: > In a former life I got to use SGI's C compiler. Like GCC, it had a > "lots of warnings" mode, somewhat like a builtin lint. _Unlike_ GCC, > the SGI compiler let one suppress specific warnings, which was very > very nice. GCC now has ?diagnostic pragmas? to turn on/off specific diagnostics , is that what you mean? -- \ ?I cannot be angry at God, in whom I do not believe.? ?Simone | `\ De Beauvoir | _o__) | Ben Finney From stephen at xemacs.org Fri Mar 20 06:02:47 2015 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Fri, 20 Mar 2015 14:02:47 +0900 Subject: [Python-ideas] gofmt for Python: standardized styling as a language feature In-Reply-To: References: <5508DBB0.2010209@stoneleaf.us> <5508F42F.8020604@sdamon.com> Message-ID: <87iodwqn0n.fsf@uwakimon.sk.tsukuba.ac.jp> Donald Stufft writes: >>>>> Guido van Rossum writes: > > I have always been skeptical of automatic style checkers. They > > encourage a nit-pickish attitude that makes the development > > process less effective, by encouraging critique of superficial > > style elements rather than of the code's structure and > > meaning. In contrast, the intention of PEP 8 was to make code > > more readable so that its structure and meaning are more easily > > understood. > > My experience is actually the exact opposite. This is one of those things where people differ. I'm happy to follow Guido on this, although personally I've been able to adapt to mechanical rules where required. > In my experience projects without mechnical enforcement ends up > with massively inconsistent style all throughout the code base. The > Python stdlib is a good example of this. Contrast this to projects > like github.com/pyca/cryptography > where there is an automated > mechanical enforcement of style and it?s got some bare minimum of > consistency throughout the entire code base. Which proves what? That enforcing minimal style rules (and I think we all agree that there's plenty of good and bad style that machines can't yet reliably distinguish) results in a code base that follows minimal style rules? From tjreedy at udel.edu Fri Mar 20 06:11:27 2015 From: tjreedy at udel.edu (Terry Reedy) Date: Fri, 20 Mar 2015 01:11:27 -0400 Subject: [Python-ideas] gofmt for Python: standardized styling as a language feature In-Reply-To: References: <5508DBB0.2010209@stoneleaf.us> <5508F42F.8020604@sdamon.com> Message-ID: On 3/19/2015 5:01 PM, Guido van Rossum wrote: > On Thu, Mar 19, 2015 at 1:40 PM, Terry Reedy Having the guidelines numbered (id'ed), even though still regarded > as guidelines, could help communication. Checkers could then easily > refer to specific guidelines. > > I worry this will just encourage the nit-picky attitude more. I agree that this is a possible negative. > Thanks for the link. I was specifically thinking of global > renamings to satisfy PEP 8's Naming Conventions. Idlelib has a > mishmash of module and function/method name styles. (I might look > and see what can be done with the undocumented (except as > 'unstable') libe2to3.) > > This feels hard to automate, because you don't know what names are part > of an external specification. Modernizing idlelib names, in relation to PEP 434, is a separate discussion. > I think that a quick "clean up whitespace" feature would be a nice > addition to IDLE, provided it's a user-selectable menu items (similar to > the existing indent/dedent operations). The Format menu now has 'Strip trailing whitespace', which may be new since you last looked. It needs to strip trailing blank lines, as reindent.py does, in addition to trailing whitespace on each line, to make files ready to commit to the cpython repository. Did you have in mind also doing within-line cleanups (as a separate menu entry)? -- Terry Jan Reedy From donald at stufft.io Fri Mar 20 06:12:11 2015 From: donald at stufft.io (Donald Stufft) Date: Fri, 20 Mar 2015 01:12:11 -0400 Subject: [Python-ideas] gofmt for Python: standardized styling as a language feature In-Reply-To: <87iodwqn0n.fsf@uwakimon.sk.tsukuba.ac.jp> References: <5508DBB0.2010209@stoneleaf.us> <5508F42F.8020604@sdamon.com> <87iodwqn0n.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <07429E95-B816-4E99-9B59-5A4FD29D5F75@stufft.io> > On Mar 20, 2015, at 1:02 AM, Stephen J. Turnbull wrote: > > Donald Stufft writes: >>>>>> Guido van Rossum writes: > >>> I have always been skeptical of automatic style checkers. They >>> encourage a nit-pickish attitude that makes the development >>> process less effective, by encouraging critique of superficial >>> style elements rather than of the code's structure and >>> meaning. In contrast, the intention of PEP 8 was to make code >>> more readable so that its structure and meaning are more easily >>> understood. >> >> My experience is actually the exact opposite. > > This is one of those things where people differ. I'm happy to follow > Guido on this, although personally I've been able to adapt to > mechanical rules where required. > >> In my experience projects without mechnical enforcement ends up >> with massively inconsistent style all throughout the code base. The >> Python stdlib is a good example of this. Contrast this to projects >> like github.com/pyca/cryptography >> where there is an automated >> mechanical enforcement of style and it?s got some bare minimum of >> consistency throughout the entire code base. > > Which proves what? That enforcing minimal style rules (and I think we > all agree that there's plenty of good and bad style that machines > can't yet reliably distinguish) results in a code base that follows > minimal style rules? > Sure, just like tests can?t guarantee that a code base works in every situation but that it works in some minimal set of tests (where minimal is whatever tests were written). Humans are inconsistent and emotional, people are more likely to nitpick a patch that they don?t particularly like in an unconscious attempt to discourage the contribution. They are also far more likely to let things slip when it?s a contribution from someone they like or even from themselves. Working on the Python stdlib is somewhat frustrating to me in this regard because the code in the stdlib is often times wildly inconsistent even within the same module. Maybe having inconsistent style which is inconsistently applied doesn?t bother you, that?s fine. I don?t particularly mind if the stdlib gains an automated style checker or formatter because I?m happy using third party modules. I was merely offering my experience that *anything* which relies on a human to verify it is, without exception, going to be verified unevenly and that using a human to verify it invites people to attempt to argue against it more often than when a machine does it. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From abarnert at yahoo.com Fri Mar 20 06:32:20 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Thu, 19 Mar 2015 22:32:20 -0700 Subject: [Python-ideas] gofmt for Python: standardized styling as a language feature In-Reply-To: <07429E95-B816-4E99-9B59-5A4FD29D5F75@stufft.io> References: <5508DBB0.2010209@stoneleaf.us> <5508F42F.8020604@sdamon.com> <87iodwqn0n.fsf@uwakimon.sk.tsukuba.ac.jp> <07429E95-B816-4E99-9B59-5A4FD29D5F75@stufft.io> Message-ID: <94D070AA-D5F6-4FAA-91E4-250A1011A823@yahoo.com> On Mar 19, 2015, at 10:12 PM, Donald Stufft wrote: > > Working on the Python stdlib is somewhat frustrating to me in this regard > because the code in the stdlib is often times wildly inconsistent even within > the same module. The stdlib is full of milkmen^H^H^H^H^H^H^Hmodules. Some of them are very old. Often older than PEP 8--and presumably the inspiration for PEP 8 to be written in the first place. At the time PEP 8 was written, obviously nobody went through and rewrote the whole stdlib to be compliant. Why not? Maybe no one volunteered--or maybe someone did, but it was rejected because the costs (risk of introducing new bugs, headaches every time you need to backport a fix across the rewrite, or even just look at the diffs to see what changed when, etc.) outweighed the benefits. Over time, some parts that touched on the visible interface, that needed to be rewritten anyway, or that are intended to serve as sample code (as in, the source is linked from the docs) have been restyled, but most of the code hasn't. Since most of those updates were in 3.0, and it seems an obvious place for a flag-day restyling if ever there were to be one, I'll bet if you search the py3k archives you'll find some relevant discussions. (Unless Guido or someone else knows off the top of his head.) From stephen at xemacs.org Fri Mar 20 07:06:37 2015 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Fri, 20 Mar 2015 15:06:37 +0900 Subject: [Python-ideas] gofmt for Python: standardized styling as a language feature In-Reply-To: <07429E95-B816-4E99-9B59-5A4FD29D5F75@stufft.io> References: <5508DBB0.2010209@stoneleaf.us> <5508F42F.8020604@sdamon.com> <87iodwqn0n.fsf@uwakimon.sk.tsukuba.ac.jp> <07429E95-B816-4E99-9B59-5A4FD29D5F75@stufft.io> Message-ID: <877fucqk2a.fsf@uwakimon.sk.tsukuba.ac.jp> Donald Stufft writes: > Working on the Python stdlib is somewhat frustrating to me in this > regard because the code in the stdlib is often times wildly > inconsistent even within the same module. Maybe you haven't paid attention to past discussions, but the Python stdlib is a terrible example because fixing it up is a FAQ on this list, on core-mentoriship, and (in the past) on python-dev. It has been repeatedly vetoed on the grounds that changes, even in whitespace, are likely to introduce more bugs than consistent style is worth. If you're actually working *on* the stdlib, then improve the style as you fix bugs or add features. If you are working on something that requires studying the stdlib, then you're out of luck. None of the above means that improving the stdlib's coding style is off the table as far as I'm concerned. I personally wouldn't mind seeing it happen, but I don't care that much. What it means is simply that the stdlib is the worst possible example to make your case -- everybody agrees with you about the inconsistent style, but Those Who Make Decisions Around Here think there are more important considerations than style. > I was merely offering my experience that *anything* which relies on > a human to verify it is, without exception, going to be verified > unevenly and that using a human to verify it invites people to > attempt to argue against it more often than when a machine does it. But the stdlib is a terrible example for that purpose for the same reason. But I take your point that using a machine can smooth correction of "mechanical" errors. Mine is simply that I don't see that it makes such a big difference in my experience; most people who have horrible coding styles respond to human advice quite well, and the "minor" differences are -- minor. YMMV. From encukou at gmail.com Fri Mar 20 15:24:04 2015 From: encukou at gmail.com (Petr Viktorin) Date: Fri, 20 Mar 2015 15:24:04 +0100 Subject: [Python-ideas] A macro for easier rich comparisons In-Reply-To: References: <55095940.4010707@gmail.com> <55095AF3.6080201@egenix.com> Message-ID: On Wed, Mar 18, 2015 at 2:48 PM, Petr Viktorin wrote: > On Wed, Mar 18, 2015 at 12:01 PM, M.-A. Lemburg wrote: >> On 18.03.2015 11:53, Petr Viktorin wrote: >>> ... >>> I propose adding a public macro to ease this. My version takes two C-orderable values and the >>> operation, similar to richcmpfunc: >>> >>> #define Py_RICHCOMPARE(val1, val2, op) ( \ >>> ((op) == Py_EQ) ? PyBool_FromLong((val1) == (val2)) : \ >>> ((op) == Py_NE) ? PyBool_FromLong((val1) != (val2)) : \ >>> ((op) == Py_LT) ? PyBool_FromLong((val1) < (val2)) : \ >>> ((op) == Py_GT) ? PyBool_FromLong((val1) > (val2)) : \ >>> ((op) == Py_LE) ? PyBool_FromLong((val1) <= (val2)) : \ >>> ((op) == Py_GE) ? PyBool_FromLong((val1) >= (val2)) : \ >>> (Py_INCREF(Py_NotImplemented), Py_NotImplemented)) >>> >>> (As for the behavior for unknown op: for most cases the best thing to do is setting an error and >>> returning NULL, but that doesn't fit into a macro. A surprising number of richcmpfunc's in CPython >>> either return NULL without error (e.g. tupleobject), or fall through and return True/False >>> arbitrarily (as in bytearrayobject). Datetime does an assert. >>> I think Py_NotImplemented is a good value to return) >>> ... >>> Is this a PEP-worthy idea? >> >> This is a great idea, no need for a PEP :-) Please submit the patch on >> the bug tracker and also include a patch for the C API documentation. > > Issue with patches is here: http://bugs.python.org/issue23699 Notes from the patch review: it is not so clear-cut after all. A sequence of comparisons is less efficient than a switch statement. Same with PyBool_FromLong vs. Py_RETURN_* Also, it might be too complex for a macro. These points could be solved by making it a function instead. Another issue raised is that the use cases are too limited limited for this to be a part of the Python API. I don't agree with that; I think boilerplate-reducing features are even if their usage is limited (see Py_RETURN_NOTIMPLEMENTED or PyModule_AddIntConstant). From donald at stufft.io Fri Mar 20 16:50:44 2015 From: donald at stufft.io (Donald Stufft) Date: Fri, 20 Mar 2015 11:50:44 -0400 Subject: [Python-ideas] gofmt for Python: standardized styling as a language feature In-Reply-To: <877fucqk2a.fsf@uwakimon.sk.tsukuba.ac.jp> References: <5508DBB0.2010209@stoneleaf.us> <5508F42F.8020604@sdamon.com> <87iodwqn0n.fsf@uwakimon.sk.tsukuba.ac.jp> <07429E95-B816-4E99-9B59-5A4FD29D5F75@stufft.io> <877fucqk2a.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <04C22794-C552-4DBF-A53F-E430D420F7AC@stufft.io> > On Mar 20, 2015, at 2:06 AM, Stephen J. Turnbull wrote: > > Donald Stufft writes: > >> Working on the Python stdlib is somewhat frustrating to me in this >> regard because the code in the stdlib is often times wildly >> inconsistent even within the same module. > > Maybe you haven't paid attention to past discussions, but the Python > stdlib is a terrible example because fixing it up is a FAQ on this > list, on core-mentoriship, and (in the past) on python-dev. It has > been repeatedly vetoed on the grounds that changes, even in > whitespace, are likely to introduce more bugs than consistent style is > worth. > > If you're actually working *on* the stdlib, then improve the style as > you fix bugs or add features. If you are working on something that > requires studying the stdlib, then you're out of luck. > > None of the above means that improving the stdlib's coding style is > off the table as far as I'm concerned. I personally wouldn't mind > seeing it happen, but I don't care that much. > > What it means is simply that the stdlib is the worst possible example > to make your case -- everybody agrees with you about the inconsistent > style, but Those Who Make Decisions Around Here think there are more > important considerations than style. > >> I was merely offering my experience that *anything* which relies on >> a human to verify it is, without exception, going to be verified >> unevenly and that using a human to verify it invites people to >> attempt to argue against it more often than when a machine does it. > > But the stdlib is a terrible example for that purpose for the same > reason. > > But I take your point that using a machine can smooth correction of > "mechanical" errors. Mine is simply that I don't see that it makes > such a big difference in my experience; most people who have horrible > coding styles respond to human advice quite well, and the "minor" > differences are -- minor. > > YMMV. > I used the stdlib primarily because it was an agreed upon fact that the style there was inconsistent. However it?s not just in old modules, it?s in any module that has had more than a handful of contributors to it. I was picking on the stdlib to avoid calling out other people?s software where it wasn?t an agreed upon thing that it was bad. If you want non stdlib examples, pip itself and virtualenv are both great (terrible?) examples of wildly inconsistent style. Pip is better now because we went through and fixed it all and are now running automated style checks as part of our CI. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From luoyonggang at gmail.com Fri Mar 20 19:24:54 2015 From: luoyonggang at gmail.com (=?UTF-8?B?572X5YuH5YiaKFlvbmdnYW5nIEx1bykg?=) Date: Sat, 21 Mar 2015 02:24:54 +0800 Subject: [Python-ideas] class os.DirEntry is confusing, Message-ID: Should be replaced with os.FsEntry or os.ScanEntry -- ?? ? ??? Yours sincerely, Yonggang Luo From cs at zip.com.au Fri Mar 20 23:00:25 2015 From: cs at zip.com.au (Cameron Simpson) Date: Sat, 21 Mar 2015 09:00:25 +1100 Subject: [Python-ideas] gofmt for Python: standardized styling as a language feature In-Reply-To: <857fucqsl6.fsf@benfinney.id.au> References: <857fucqsl6.fsf@benfinney.id.au> Message-ID: <20150320220025.GA91756@cskk.homeip.net> On 20Mar2015 14:02, Ben Finney wrote: >Cameron Simpson writes: >> In a former life I got to use SGI's C compiler. Like GCC, it had a >> "lots of warnings" mode, somewhat like a builtin lint. _Unlike_ GCC, >> the SGI compiler let one suppress specific warnings, which was very >> very nice. > >GCC now has ?diagnostic pragmas? to turn on/off specific diagnostics >, >is that what you mean? In terms of function, yes, though it says "only warnings [...] can be controlled, and not all of them". In terms of usability, no. I'd rather put compiler options to control noise in Makefiles (or whatever built system) rather than litter code with compiler specific junk which has no effect (one hopes) elsewhere (other platforms, other compilers). Also, I don't think these #pragmas were around at the time:-) Cheers, Cameron Simpson Tim's Top 5: #5. Know when to give cages the bird. #4. I'm not hanging off, so you don't need to hang off either. (especially on the "up" side) #3. Always let me know when you're gonna get on and off. And by God, don't let your boot scratch my polished exhaust canister. #2. Remain seated until the aircraft has safely stopped at the passenger terminal and the seat belt sign has been extinguished. #1. Shut the hell up, it'll all be over in a minute. - Timothy R. Redden From barry at python.org Fri Mar 20 23:07:47 2015 From: barry at python.org (Barry Warsaw) Date: Fri, 20 Mar 2015 18:07:47 -0400 Subject: [Python-ideas] gofmt for Python: standardized styling as a language feature References: <5508DBB0.2010209@stoneleaf.us> <5508F42F.8020604@sdamon.com> <550B3FED.9020200@udel.edu> Message-ID: <20150320180747.2507167e@anarchist.wooz.org> On Mar 19, 2015, at 04:55 PM, Guido van Rossum wrote: >Have you used Emacs? The python-mode written by Tim Peters and maintained >by Barry Warsaw has some pretty amazing reformatting capabilities (though I >rarely use them, because my fingers automatically type correctly formatted >code :-). I must admit to a long-standing lack of enthusiasm for Elisp hacking, so much of the recent credit for python-mode's development goes to Andreas R?hler. :) Simple additions of pyflakes (some folks like flake8) and whitespace-mode make python-mode really fantastic for quickly spotting questionable code. My fingers too are well-trained but sometimes my brain gets in the way, and then Emacs will clearly show me lines that need clean up. I'd like that better than something that automatically reformats my code. Cheers, -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 819 bytes Desc: OpenPGP digital signature URL: From rymg19 at gmail.com Fri Mar 20 23:06:53 2015 From: rymg19 at gmail.com (Ryan Gonzalez) Date: Fri, 20 Mar 2015 17:06:53 -0500 Subject: [Python-ideas] gofmt for Python: standardized styling as a language feature In-Reply-To: <20150320220025.GA91756@cskk.homeip.net> References: <857fucqsl6.fsf@benfinney.id.au> <20150320220025.GA91756@cskk.homeip.net> Message-ID: Can't you do that witout pragmas? gcc tst.c -Wall -Wno-parenthesis -Wno-unused-variable ... On Fri, Mar 20, 2015 at 5:00 PM, Cameron Simpson wrote: > On 20Mar2015 14:02, Ben Finney wrote: > >> Cameron Simpson writes: >> >>> In a former life I got to use SGI's C compiler. Like GCC, it had a >>> "lots of warnings" mode, somewhat like a builtin lint. _Unlike_ GCC, >>> the SGI compiler let one suppress specific warnings, which was very >>> very nice. >>> >> >> GCC now has ?diagnostic pragmas? to turn on/off specific diagnostics >> > Diagnostic-Pragmas.html#Diagnostic-Pragmas>, >> is that what you mean? >> > > In terms of function, yes, though it says "only warnings [...] can be > controlled, and not all of them". In terms of usability, no. > > I'd rather put compiler options to control noise in Makefiles (or whatever > built system) rather than litter code with compiler specific junk which has > no effect (one hopes) elsewhere (other platforms, other compilers). > > Also, I don't think these #pragmas were around at the time:-) > > Cheers, > Cameron Simpson > > Tim's Top 5: > #5. Know when to give cages the bird. > #4. I'm not hanging off, so you don't need to hang off either. > (especially on the "up" side) > #3. Always let me know when you're gonna get on and off. And by God, > don't let your boot scratch my polished exhaust canister. > #2. Remain seated until the aircraft has safely stopped at the > passenger terminal and the seat belt sign has been extinguished. > #1. Shut the hell up, it'll all be over in a minute. > - Timothy R. Redden > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- Ryan [ERROR]: Your autotools build scripts are 200 lines longer than your program. Something?s wrong. http://kirbyfan64.github.io/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From cs at zip.com.au Fri Mar 20 23:18:36 2015 From: cs at zip.com.au (Cameron Simpson) Date: Sat, 21 Mar 2015 09:18:36 +1100 Subject: [Python-ideas] gofmt for Python: standardized styling as a language feature In-Reply-To: References: Message-ID: <20150320221836.GA52250@cskk.homeip.net> On 20Mar2015 17:06, Ryan Gonzalez wrote: >Can't you do that witout pragmas? > >gcc tst.c -Wall -Wno-parenthesis -Wno-unused-variable ... Now, presumably yes. Then, definitely no. So I presume the situation is improved a lot. Cheers, Cameron Simpson If you 'aint falling off, you ar'nt going hard enough. - Fred Gassit From abarnert at yahoo.com Fri Mar 20 23:19:09 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Fri, 20 Mar 2015 15:19:09 -0700 Subject: [Python-ideas] A macro for easier rich comparisons In-Reply-To: References: <55095940.4010707@gmail.com> <55095AF3.6080201@egenix.com> Message-ID: <6D6DC4C3-BB82-49CE-8615-E07B1F048FA1@yahoo.com> On Mar 20, 2015, at 7:24 AM, Petr Viktorin wrote: > > A sequence of comparisons is less efficient than a switch statement. > Same with PyBool_FromLong vs. Py_RETURN_* > Also, it might be too complex for a macro. > These points could be solved by making it a function instead. Or by just making the macro a statement instead of a function; then the code can be exactly the same code you're abstracting out, and instead of using it like this: return PY_RICHCOMPARE(mytype_get_data(obj1), mytype_get_data(obj2), op); ... you use it like this: PY_RETURNRICHCOMPARE(mytype_get_data(obj1), mytype_get_data(obj2), op); From paddy3118 at gmail.com Sat Mar 21 05:41:03 2015 From: paddy3118 at gmail.com (Paddy3118) Date: Fri, 20 Mar 2015 21:41:03 -0700 (PDT) Subject: [Python-ideas] os.path.commonprefix: Yes that old chestnut. Message-ID: <22b04160-2ff7-4c98-9d2c-97ac13b3bc25@googlegroups.com> I had to add a comment to an RC entry that was using os.path.commonprefix to compute the longest common prefix which it does regardless of path separators, (which is what it should do). It has been discussed before - it goes back to the nineties it seems, but we are still left with a misplaced function, (it should be str.commonprefix), with an awful hack of saying "yes we know its ballsed up" in the documentation . I guess we missed a great opportunity to fix this when we moved to Python 3 too!? The fix seems clear: deprecate os.path.commonprefix whilst creating a true os.path.commonpath and str.commonprefix. The deprecated function should hang around and the standard libs modified to switch to the new function(s) I've heard that some religious people put obvious faults in their work as only their god should get so close to perfection - maybe that's why we still have this wart :-) -------------- next part -------------- An HTML attachment was scrubbed... URL: From victor.stinner at gmail.com Sat Mar 21 11:23:26 2015 From: victor.stinner at gmail.com (Victor Stinner) Date: Sat, 21 Mar 2015 11:23:26 +0100 Subject: [Python-ideas] class os.DirEntry is confusing, In-Reply-To: References: Message-ID: Hi, The "DirEntry" name comes from the POSIX structure "direntry". You should not instanciate the DirEntry class nor store it, so not check its type neither (isinstance(something, os.DirEntry)). I don't think that the exact name of the class matters. I had prefer DirectoryEntry, but "dir" shortcut is common in the Python os module (os.listdir) and in POSIX strandard (opendir, struct direntry). You are coming late. The PEP was discussed during 3 months (may-july 2014), and Ben started to work on it since 2012! See the PEP for details: https://www.python.org/dev/peps/pep-0471/#naming I accepted the PEP and I don't plan to modify it even if you have a serious rationale for the change. Bikeshedding is not :-) By the way, the PEP is implemnted. os.scandir() is already in the latest python 3.5 alpha. Optimized os.walk() will be in the next alpha. Victor Le vendredi 20 mars 2015, ???(Yonggang Luo) a ?crit : > Should be replaced with os.FsEntry or os.ScanEntry > > -- > ?? > ? > ??? > Yours > sincerely, > Yonggang Luo > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Sat Mar 21 12:47:32 2015 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sat, 21 Mar 2015 12:47:32 +0100 Subject: [Python-ideas] os.path.commonprefix: Yes that old chestnut. References: <22b04160-2ff7-4c98-9d2c-97ac13b3bc25@googlegroups.com> Message-ID: <20150321124732.0b47730e@fsol> On Fri, 20 Mar 2015 21:41:03 -0700 (PDT) Paddy3118 wrote: > > The fix seems clear: deprecate os.path.commonprefix whilst creating a true > os.path.commonpath and str.commonprefix. The deprecated function should > hang around and the standard libs modified to switch to the new function(s) +1 from me. > I've heard that some religious people put obvious faults in their work as > only their god should get so close to perfection - maybe that's why we > still have this wart :-) If by chance their god is Guido, then at least /he/ can intervene to fix his disciples' mess ;-) Regards Antoine. From guido at python.org Fri Mar 20 17:59:32 2015 From: guido at python.org (Guido van Rossum) Date: Fri, 20 Mar 2015 09:59:32 -0700 Subject: [Python-ideas] PEP 484 (Type Hints) -- second draft Message-ID: Here's an updated draft for PEP 484. There's more to do (especially generic types need much more thought and writing) but I've done a bunch of editing and thinking so I think this is ready for another round of review. Remember, for technical questions it's often best to use the GitHub tracker for this PEP at https://github.com/ambv/typehinting/issues . Significant changes in this draft: - Define stubs. - Define `@overload`. - Describe `cast()`. - Fix description of `Any`. - Describe `Callable[..., t]`. - Explain why `List[t]` instead of `List`. - Add section on rejected alternatives. - Various other edits for clarity. Incomplete list of TODOs: - Define and explain generics. - Covariance vs. contravariance ( https://github.com/ambv/typehinting/issues/2) - Other edits as indicated by FIXME comments. - Other edits mentioned in https://github.com/ambv/typehinting/blob/master/README.rst . Here's the full text of the PEP (hopefully it will appear soon at https://www.python.org/dev/peps/pep-0484/): PEP: 484 Title: Type Hints Version: $Revision$ Last-Modified: $Date$ Author: Guido van Rossum , Jukka Lehtosalo < jukka.lehtosalo at iki.fi>, ?ukasz Langa Discussions-To: Python-Dev Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 29-Sep-2014 Post-History: 16-Jan-2015,20-Mar-2015 Resolution: Abstract ======== This PEP introduces a standard syntax for type hints using annotations (PEP 3107) on function definitions. For example, here is a simple function whose argument and return type are declared in the annotations:: def greeting(name: str) -> str: return 'Hello ' + name While these annotations are available at runtime through the usual ``__annotations__`` attribute, *no type checking happens at runtime*. Instead, the proposal assumes the existence of a separate off-line type checker which users can run over their source code voluntarily. Essentially, such a type checker acts as a very powerful linter. The proposal is strongly inspired by mypy [mypy]_. For example, the type "sequence of integers" can be written as ``Sequence[int]``. The square brackets mean that no new syntax needs to be added to the language. The example here uses a custom class ``Sequence``, imported from a pure-Python module ``typing.py``. The ``Sequence[int]`` notation works by implementing ``__getitem__()`` in the metaclass. The type system supports unions, generic types, and a special type named ``Any`` which is consistent with (i.e. assignable to and from) all types. This latter feature is taken from the idea of gradual typing. Gradual typing and the full type system are explained in PEP 483. Other approaches from which we have borrowed or to which ours can be compared and contrasted are described in PEP 482. Rationale and Goals =================== PEP 3107 added support for arbitrary annotations on parts of a function definition. Although no meaning was assigned to annotations then, there has always been an implicit goal to use them for type hinting, which is listed as the first possible use case in said PEP. This PEP aims to provide a standard syntax for type annotations, opening up Python code to easier static analysis and refactoring, potential runtime type checking, and performance optimizations utilizing type information. Of these goals, static analysis is the most important. This includes support for off-line type checkers such as mypy, as well as providing a standard notation that can be used by IDEs for code completion and refactoring. Non-goals --------- While the proposed typing module will contain some building blocks for runtime type checking -- in particular a useful ``isinstance()`` implementation -- third party packages would have to be developed to implement specific runtime type checking functionality, for example using decorators or metaclasses. Using type hints for performance optimizations is left as an exercise for the reader. It should also be emphasized that Python will remain a dynamically typed language, and the authors have no desire to ever make type hints mandatory, even by convention. Type Definition Syntax ====================== The syntax leverages PEP 3107-style annotations with a number of extensions described in sections below. In its basic form, type hinting is used by filling function annotations with classes:: def greeting(name: str) -> str: return 'Hello ' + name This denotes that the expected type of the ``name`` argument is ``str``. Analogically, the expected return type is ``str``. Subclasses of a specified argument type are also accepted as valid types for that argument. Abstract base classes, types available in the ``types`` module, and user-defined classes may be used as type hints as well. Annotations must be valid expressions that evaluate without raising exceptions at the time the function is defined. In addition, the needs of static analysis require that annotations must be simple enough to be interpreted by static analysis tools. (This is an intentionally somewhat vague requirement.) .. FIXME: Define rigorously what is/isn't supported. When used as an annotation, the expression ``None`` is considered equivalent to ``NoneType`` (i.e., ``type(None)`` for type hinting purposes. Type aliases are also valid type hints:: integer = int def retry(url: str, retry_count: integer) -> None: ... New names that are added to support features described in following sections are available in the ``typing`` package. Callbacks --------- Frameworks expecting callback functions of specific signatures might be type hinted using ``Callable[[Arg1Type, Arg2Type], ReturnType]``. Examples:: from typing import Callable def feeder(get_next_item: Callable[[], str]) -> None: # Body def async_query(on_success: Callable[[int], None], on_error: Callable[[int, Exception], None]) -> None: # Body It is possible to declare the return type of a callable without specifying the call signature by substituting a literal ellipsis (three dots) for the list of arguments:: def partial(func: Callable[..., str], *args) -> Callable[..., str]: # Body Note that there are no square brackets around the ellipsis. The arguments of the callback are completely unconstrained in this case (and keyword arguments are acceptable). Since using callbacks with keyword arguments is not perceived as a common use case, there is currently no support for specifying keyword arguments with ``Callable``. Similarly, there is no support for specifying callback signatures with a variable number of argument of a specific type. Generics -------- Since type information about objects kept in containers cannot be statically inferred in a generic way, abstract base classes have been extended to support subscription to denote expected types for container elements. Example:: from typing import Mapping, Set def notify_by_email(employees: Set[Employee], overrides: Mapping[str, str]) -> None: ... Generics can be parametrized by using a new factory available in ``typing`` called ``TypeVar``. Example:: from typing import Sequence, TypeVar T = TypeVar('T') # Declare type variable def first(l: Sequence[T]) -> T: # Generic function return l[0] In this case the contract is that the returning value is consistent with the elements held by the collection. ``TypeVar`` supports constraining parametric types to classes with any of the specified bases. Example:: from typing import Iterable X = TypeVar('X') Y = TypeVar('Y', Iterable[X]) def filter(rule: Callable[[X], bool], input: Y) -> Y: ... .. FIXME: Add an example with multiple bases defined. In the example above we specify that ``Y`` can be any subclass of Iterable with elements of type ``X``, as long as the return type of ``filter()`` will be the same as the type of the ``input`` argument. .. FIXME: Explain more about how this works. Forward references ------------------ When a type hint contains names that have not been defined yet, that definition may be expressed as a string, to be resolved later. For example, instead of writing:: def notify_by_email(employees: Set[Employee]) -> None: ... one might write:: def notify_by_email(employees: 'Set[Employee]') -> None: ... .. FIXME: Rigorously define this, and give a motivational example. Union types ----------- Since accepting a small, limited set of expected types for a single argument is common, there is a new special factory called ``Union``. Example:: from typing import Union def handle_employees(e: Union[Employee, Sequence[Employee]]) -> None: if isinstance(e, Employee): e = [e] ... A type factored by ``Union[T1, T2, ...]`` responds ``True`` to ``issubclass`` checks for ``T1`` and any of its subclasses, ``T2`` and any of its subclasses, and so on. One common case of union types are *optional* types. By default, ``None`` is an invalid value for any type, unless a default value of ``None`` has been provided in the function definition. Examples:: def handle_employee(e: Union[Employee, None]) -> None: ... As a shorthand for ``Union[T1, None]`` you can write ``Optional[T1]``; for example, the above is equivalent to:: from typing import Optional def handle_employee(e: Optional[Employee]) -> None: ... An optional type is also automatically assumed when the default value is ``None``, for example:: def handle_employee(e: Employee = None): ... This is equivalent to:: def handle_employee(e: Optional[Employee] = None) -> None: ... The ``Any`` type ---------------- A special kind of type is ``Any``. Every class is a subclass of ``Any``. This is also true for the builtin class ``object``. However, to the static type checker these are completely different. When the type of a value is ``object``, the type checker will reject almost all operations on it, and assigning it to a variable (or using it as a return value) of a more specialized type is a type error. On the other hand, when a value has type ``Any``, the type checker will allow all operations on it, and a value of type `Any`` can be assigned to a variable (or used as a return value) of a more constrained type. Platform-specific type checking ------------------------------- In some cases the typing information will depend on the platform that the program is being executed on. To enable specifying those differences, simple conditionals can be used:: from typing import PY2, WINDOWS if PY2: text = unicode else: text = str def f() -> text: ... if WINDOWS: loop = ProactorEventLoop else: loop = UnixSelectorEventLoop .. FIXME: Also define PY3 and POSIX? Arbitrary literals defined in the form of ``NAME = True`` will also be accepted by the type checker to differentiate type resolution:: DEBUG = False ... if DEBUG: class Tracer: else: class Tracer: For the purposes of type hinting, the type checker assumes ``__debug__`` is set to ``True``, in other words the ``-O`` command-line option is not used while type checking. Compatibility with other uses of function annotations ----------------------------------------------------- A number of existing or potential use cases for function annotations exist, which are incompatible with type hinting. These may confuse a static type checker. However, since type hinting annotations have no runtime behavior (other than evaluation of the annotation expression and storing annotations in the ``__annotations__`` attribute of the function object), this does not make the program incorrect -- it just makes it issue warnings when a static analyzer is used. To mark portions of the program that should not be covered by type hinting, use the following: * a ``@no_type_check`` decorator on classes and functions * a ``# type: ignore`` comment on arbitrary lines .. FIXME: should we have a module-wide comment as well? .. FIXME: suggest that other uses of annotations be replaced with decorators .. FIXME: add reference to "rejected alternatives" Type Hints on Local and Global Variables ======================================== No first-class syntax support for explicitly marking variables as being of a specific type is added by this PEP. To help with type inference in complex cases, a comment of the following format may be used:: x = [] # type: List[Employee] In the case where type information for a local variable is needed before it is declared, an ``Undefined`` placeholder might be used:: from typing import Undefined x = Undefined # type: List[Employee] y = Undefined(int) If type hinting proves useful in general, a syntax for typing variables may be provided in a future Python version. Casts ===== Occasionally the type checker may need a different kind of hint: the programmer may know that an expression is of a more constrained type than the type checker infers. For example:: from typing import List def find_first_str(a: List[object]) -> str: index = next(i for i, x in enumerate(a) if isinstance(x, str)) # We only get here if there's at least one string in a return cast(str, a[index]) The type checker infers the type ``object`` for ``a[index]``, but we know that (if the code gets to that point) it must be a string. The ``cast(t, x)`` call tells the type checker that we are confident that the type of ``x`` is ``t``. At runtime a cast always returns the expression unchanged -- it does not check the type, and it does not convert or coerce the value. Casts differ from type comments (see the previous section). When using a type comment, the type checker should still verify that the inferred type is consistent with the stated type. When using a cast, the type checker trusts the programmer. Also, casts can be used in expressions, while type comments only apply to assignments. Stub Files ========== Stub files are files containing type hints that are only for use by the type checker, not at runtime. There are several use cases for stub files: * Extension modules * 3rd party modules whose authors have not yet added type hints * Standard library modules for which type hints have not yet been written * Modules that must be compatible with Python 2 and 3 * Modules that use annotations for other purposes Stub files have the same syntax as regular Python modules. There is one feature of the ``typing`` module that may only be used in stub files: the ``@overload`` decorator described below. The type checker should only check function signatures in stub files; function bodies in stub files should just be a single ``pass`` statement. The type checker should have a configurable search path for stub files. If a stub file is found the type checker should not read the corresponding "real" module. Stub files may use the ``.py`` extension or alternatively may use the ``.pyi`` extension. The latter makes it possible to maintain stub files in the same directory as the corresponding real module. Function overloading -------------------- The ``@overload`` decorator allows describing functions that support multiple different combinations of argument types. This pattern is used frequently in builtin modules and types. For example, the ``__getitem__()`` method of the ``bytes`` type can be described as follows:: from typing import overload class bytes: ... @overload def __getitem__(self, i: int) -> int: pass @overload def __getitem__(self, s: slice) -> bytes: pass This description is more precise than would be possible using unions (which cannot express the relationship between the argument and return types):: from typing import Union class bytes: ... def __getitem__(self, a: Union[int, slice]) -> Union[int, bytes]: pass Another example where ``@overload`` comes in handy is the type of the builtin ``map()`` function, which takes a different number of arguments depending on the type of the callable:: from typing import Callable, Iterable, Iterator, Tuple, TypeVar, overload T1 = TypeVar('T1') T2 = TypeVar('T2) S = TypeVar('S') @overload def map(func: Callable[[T1], S], iter1: Iterable[T1]) -> Iterator[S]: pass @overload def map(func: Callable[[T1, T2], S], iter1: Iterable[T1], iter2: Iterable[T2]) -> Iterator[S]: pass # ... and we could add more items to support more than two iterables Note that we could also easily add items to support ``map(None, ...)``:: @overload def map(func: None, iter1: Iterable[T1]) -> Iterable[T1]: pass @overload def map(func: None, iter1: Iterable[T1], iter2: Iterable[T2]) -> Iterable[Tuple[T1, T2]]: pass The ``@overload`` decorator may only be used in stub files. While it would be possible to provide a multiple dispatch implementation using this syntax, its implementation would require using ``sys._getframe()``, which is frowned upon. Also, designing and implementing an efficient multiple dispatch mechanism is hard, which is why previous attempts were abandoned in favor of ``functools.singledispatch()``. (See PEP 443, especially its section "Alternative approaches".) In the future we may come up with a satisfactory multiple dispatch design, but we don't want such a design to be constrained by the overloading syntax defined for type hints in stub files. Exceptions ========== No syntax for listing explicitly raised exceptions is proposed. Currently the only known use case for this feature is documentational, in which case the recommendation is to put this information in a docstring. The ``typing`` Package ====================== To open the usage of static type checking to Python 3.5 as well as older versions, a uniform namespace is required. For this purpose, a new package in the standard library is introduced called ``typing``. It holds a set of classes representing builtin types with generics, namely: * Dict, used as ``Dict[key_type, value_type]`` * List, used as ``List[element_type]`` * Set, used as ``Set[element_type]``. See remark for ``AbstractSet`` below. * FrozenSet, used as ``FrozenSet[element_type]`` * Tuple, used by listing the element types, for example ``Tuple[int, int, str]``. Arbitrary-length homogeneous tuples can be expressed using one type and ellipsis, for example ``Tuple[int, ...]``. (The ``...`` here are part of the syntax.) The generic versions of concrete collection types (``Dict``, ``List``, ``Set``, ``FrozenSet``, and homogeneous arbitrary-length ``Tuple``) are mainly useful for annotating return values. For arguments, prefer the abstract collection types defined below, e.g. ``Mapping``, ``Sequence`` or ``AbstractSet``. It also introduces factories and helper members needed to express generics and union types: * Any, used as ``def get(key: str) -> Any: ...`` * Union, used as ``Union[Type1, Type2, Type3]`` * TypeVar, used as ``X = TypeVar('X', Type1, Type2, Type3)`` or simply ``Y = TypeVar('Y')`` * Undefined, used as ``local_variable = Undefined # type: List[int]`` or ``local_variable = Undefined(List[int])`` (the latter being slower during runtime) * Callable, used as ``Callable[[Arg1Type, Arg2Type], ReturnType]`` * AnyStr, equivalent to ``TypeVar('AnyStr', str, bytes)`` All abstract base classes available in ``collections.abc`` are importable from the ``typing`` package, with added generics support: * ByteString * Callable * Container * Hashable * ItemsView * Iterable * Iterator * KeysView * Mapping * MappingView * MutableMapping * MutableSequence * MutableSet * Sequence * Set as ``AbstractSet``. This name change was required because ``Set`` in the ``typing`` module means ``set()`` with generics. * Sized * ValuesView * Mapping The library includes literals for platform-specific type hinting: * PY2 * PY3, equivalent to ``not PY2`` * WINDOWS * POSIX, equivalent to ``not WINDOWS`` The following types are available in the ``typing.io`` module: * IO * BinaryIO * TextIO The following types are provided by the ``typing.re`` module: * Match and Pattern, types of ``re.match()`` and ``re.compile()`` results As a convenience measure, types from ``typing.io`` and ``typing.re`` are also available in ``typing`` (quoting Guido, "There's a reason those modules have two-letter names."). The place of the ``typing`` module in the standard library ---------------------------------------------------------- .. FIXME: complete this section (or discard?) Usage Patterns ============== The main use case of type hinting is static analysis using an external tool without executing the analyzed program. Existing tools used for that purpose like ``pyflakes`` [pyflakes]_ or ``pylint`` [pylint]_ might be extended to support type checking. New tools, like mypy [mypy]_, can be adopted specifically for this purpose. Type checking based on type hints is understood as a best-effort mechanism. In other words, whenever types are not annotated and cannot be inferred, the type checker considers such code valid. Type errors are only reported in case of explicit or inferred conflict. Moreover, as a mechanism that is not tied to execution of the code, it does not affect runtime behaviour. In other words, even in the case of a typing error, the program will continue running. The implementation of a type checker, whether linting source files or enforcing type information during runtime, is out of scope for this PEP. .. FIXME: This is somewhat redundant with the updated initial sections. .. FIXME: Describe run-time behavior of generic types. Rejected Alternatives ===================== During discussion of earlier drafts of this PEP, various objections were raised and alternatives were proposed. We discuss some of these here and explain why we reject them. Several main objections were raised. Which brackets for generic type parameters? ------------------------------------------- Most people are familiar with the use of angular brackets (e.g. ``List``) in languages like C++, Java, C# and Swift to express the parametrization of generic types. The problem with these is that they are really hard to parse, especially for a simple-minded parser like Python. In most languages the ambiguities are usually dealy with by only allowing angular brackets in specific syntactic positions, where general expressions aren't allowed. (And also by using very powerful parsing techniques that can backtrack over an arbitrary section of code.) But in Python, we'd like type expressions to be (syntactically) the same as other expressions, so that we can use e.g. variable assignment to create type aliases. Consider this simple type expression:: List >From the Python parser's perspective, the expression begins with the same four tokens (NAME, LESS, NAME, GREATER) as a chained comparison:: a < b > c # I.e., (a < b) and (b > c) We can even make up an example that could be parsed both ways:: a < b > [ c ] Assuming we had angular brackets in the language, this could be interpreted as either of the following two:: (a)[c] # I.e., (a).__getitem__(c) a < b > ([c]) # I.e., (a < b) and (b > [c]) It would surely be possible to come up with a rule to disambiguate such cases, but to most users the rules would feel arbitrary and complex. It would also require us to dramatically change the CPython parser (and every other parser for Python). It should be noted that Python's current parser is intentionally "dumb" -- a simple grammar is easier for users to reason about. For all these reasons, square brackets (e.g. ``List[int]``) are (and have long been) the preferred syntax for generic type parameters. They can be implemented by defining the ``__getitem__()`` method on the metaclass, and no new syntax is required at all. This option works in all recent versions of Python (starting with Python 2.2). Python is not alone in this syntactic choice -- generic classes in Scala also use square brackets. What about existing uses of annotations? ---------------------------------------- One line of argument points out that PEP 3107 explicitly supports the use of arbitrary expressions in function annotations. The new proposal is then considered incompatible with the specification of PEP 3107. Our response to this is that, first of all, the current proposal does not introduce any direct incompatibilities, so programs using annotations in Python 3.4 will still work correctly and without prejudice in Python 3.5. We do hope that type hints will eventually become the sole use for annotations, but this will require additional discussion and a deprecation period after the initial roll-out of the typing module with Python 3.5. The current PEP will have provisional status (see PEP 411) until Python 3.6 is released. The fastest conceivable scheme would introduce silent deprecation of non-type-hint annotations in 3.6, full deprecation in 3.7, and declare type hints as the only allowed use of annotations in Python 3.8. This should give authors of packages that use annotations plenty of time to devise another approach, even if type hints become an overnight success. Another possible outcome would be that type hints will eventually become the default meaning for annotations, but that there will always remain an option to disable them. For this purpose the current proposal defines a decorator ``@no_type_check`` which disables the default interpretation of annotations as type hints in a given class or function. It also defines a meta-decorator ``@no_type_check_decorator`` which can be used to decorate a decorator (!), causing annotations in any function or class decorated with the latter to be ignored by the type checker. There are also ``# type: ignore`` comments, and static checkers should support configuration options to disable type checking in selected packages. Despite all these options, proposals have been circulated to allow type hints and other forms of annotations to coexist for individual arguments. One proposal suggests that if an annotation for a given argument is a dictionary literal, each key represents a different form of annotation, and the key ``'type'`` would be use for type hints. The problem with this idea and its variants is that the notation becomes very "noisy" and hard to read. Also, in most cases where existing libraries use annotations, there would be little need to combine them with type hints. So the simpler approach of selectively disabling type hints appears sufficient. The problem of forward declarations ----------------------------------- The current proposal is admittedly sub-optimal when type hints must contain forward references. Python requires all names to be defined by the time they are used. Apart from circular imports this is rarely a problem: "use" here means "look up at runtime", and with most "forward" references there is no problem in ensuring that a name is defined before the function using it is called. The problem with type hints is that annotations (per PEP 3107, and similar to default values) are evaluated at the time a function is defined, and thus any names used in an annotation must be already defined when the function is being defined. A common scenario is a class definition whose methods need to reference the class itself in their annotations. (More general, it can also occur with mutually recursive classes.) This is natural for container types, for example:: class Node: """Binary tree node.""" def __init__(self, left: Node, right: None): self.left = left self.right = right As written this will not work, because of the peculiarity in Python that class names become defined once the entire body of the class has been executed. Our solution, which isn't particularly elegant, but gets the job done, is to allow using string literals in annotations. Most of the time you won't have to use this though -- most _uses_ of type hints are expected to reference builtin types or types defined in other modules. A counterproposal would change the semantics of type hints so they aren't evaluated at runtime at all (after all, type checking happens off-line, so why would type hints need to be evaluated at runtime at all). This of course would run afoul of backwards compatibility, since the Python interpreter doesn't actually know whether a particular annotation is meant to be a type hint or something else. The double colon ---------------- A few creative souls have tried to invent solutions for this problem. For example, it was proposed to use a double colon (``::``) for type hints, solving two problems at once: disambiguating between type hints and other annotations, and changing the semantics to preclude runtime evaluation. There are several things wrong with this idea, however. * It's ugly. The single colon in Python has many uses, and all of them look familiar because they resemble the use of the colon in English text. This is a general rule of thumb by which Python abides for most forms of punctuation; the exceptions are typically well known from other programming languages. But this use of ``::`` is unheard of in English, and in other languages (e.g. C++) it is used as a scoping operator, which is a very different beast. In contrast, the single colon for type hints reads natural -- and no wonder, since it was carefully designed for this purpose (the idea long predates PEP 3107 [gvr-artima]_). It is also used in the same fashion in other languages from Pascal to Swift. * What would you do for return type annotations? * It's actually a feature that type hints are evaluated at runtime. * Making type hints available at runtime allows runtime type checkers to be built on top of type hints. * It catches mistakes even when the type checker is not run. Since it is a separate program, users may choose not to run it (or even install it), but might still want to use type hints as a concise form of documentation. Broken type hints are no use even for documentation. * Because it's new syntax, using the double colon for type hints would limit them to code that works with Python 3.5 only. By using existing syntax, the current proposal can easily work for older versions of Python 3. (And in fact mypy supports Python 3.2 and newer.) * If type hints become successful we may well decide to add new syntax in the future to declare the type for variables, for example ``var age: int = 42``. If we were to use a double colon for argument type hints, for consistency we'd have to use the same convention for future syntax, perpetuating the ugliness. Other forms of new syntax ------------------------- A few other forms of alternative syntax have been proposed, e.g. the introduction of a ``where`` keyword [roberge]_, and Cobra-inspired ``requires`` clauses. But these all share a problem with the double colon: they won't work for earlier versions of Python 3. The same would apply to a new ``__future__`` import. Other backwards compatible conventions -------------------------------------- The ideas put forward include: * A decorator, e.g. ``@typehints(name=str, returns=str)``. This could work, but it's pretty verbose (an extra line, and the argument names must be repeated), and a far cry in elegance from the PEP 3107 notation. * Stub files. We do want stub files, but they are primarily useful for adding type hints to existing code that doesn't lend itself to adding type hints, e.g. 3rd party packages, code that needs to support both Python 2 and Python 3, and especially extension modules. For most situations, having the annotations in line with the function definitions makes them much more useful. * Docstrings. There is an existing convention for docstrings, based on the Sphinx notation (``:type arg1: description``). This is pretty verbose (an extra line per parameter), and not very elegant. We could also make up something new, but the annotation syntax is hard to beat (because it was designed for this very purpose). It's also been proposed to simply wait another release. But what problem would that solve? It would just be procrastination. Is Type Hinting Pythonic? ========================= .. FIXME: Do we really need this section? Type annotations provide important documentation for how a unit of code should be used. Programmers should therefore provide type hints on public APIs, namely argument and return types on functions and methods considered public. However, because types of local and global variables can be often inferred, they are rarely necessary. The kind of information that type hints hold has always been possible to achieve by means of docstrings. In fact, a number of formalized mini-languages for describing accepted arguments have evolved. Moving this information to the function declaration makes it more visible and easier to access both at runtime and by static analysis. Adding to that the notion that ?explicit is better than implicit?, type hints are indeed *Pythonic*. Acknowledgements ================ This document could not be completed without valuable input, encouragement and advice from Jim Baker, Jeremy Siek, Michael Matson Vitousek, Andrey Vlasovskikh, and Radomir Dopieralski. Influences include existing languages, libraries and frameworks mentioned in PEP 482. Many thanks to their creators, in alphabetical order: Stefan Behnel, William Edwards, Greg Ewing, Larry Hastings, Anders Hejlsberg, Alok Menghrajani, Travis E. Oliphant, Joe Pamer, Raoul-Gabriel Urma, and Julien Verlaguet. References ========== .. [mypy] http://mypy-lang.org .. [pyflakes] https://github.com/pyflakes/pyflakes/ .. [pylint] http://www.pylint.org .. [gvr-artima] http://www.artima.com/weblogs/viewpost.jsp?thread=85551 .. [roberge] http://aroberge.blogspot.com/2015/01/type-hinting-in-python-focus-on.html Copyright ========= This document has been placed in the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End: -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From bcannon at gmail.com Sat Mar 21 14:57:56 2015 From: bcannon at gmail.com (Brett Cannon) Date: Sat, 21 Mar 2015 13:57:56 +0000 Subject: [Python-ideas] Warn about comparing bytes to int for `python3 -b` In-Reply-To: References: Message-ID: Just to update this thread, Serhiy beat me to a patch and committed it. Through the change he found a few latent bugs in the stdlib itself so this idea has already proven/paid for itself. On Mon, Mar 16, 2015 at 1:25 PM Brett Cannon wrote: > On Mon, Mar 16, 2015 at 11:22 AM Victor Stinner > wrote: > >> Hi, >> >> 2015-03-16 16:11 GMT+01:00 Brett Cannon : >> > One of the rather subtle issues with writing Python 2/3 code is that >> > indexing on bytes in Python 2 returns a length-1 bytes object while in >> > Python 3 it returns an int. Because ==/!= always returns True/False it >> can >> > be a very subtle failure and tough to track down. >> >> I worked on such patch in the past, but I lost it :-) I can help to >> rewrite it if needed. >> > > I filed http://bugs.python.org/issue23681 if you want to help (although I > don't expect the patch to be complicated; I might ask you for a code review > though =). > > >> >> So yes, it *is* very useful to port a large Python 2 project to Python 3. >> >> For example, you may not be able to run your application with Python 3 >> because a third party library cannot be imported on Python 3, so it >> blocks the whole work on porting an application to Python 3. Until the >> module is ported, you may want to prepare the port. Checking >> bytes==str just by reading the source code is difficult. >> >> Other issues which can only be "seen" at runtime when running an >> application on Python 3 : >> >> - "x > 0" with x=None => TypeError is raised in Python 3 >> > > Testing can easily catch that *and* have a traceback to trace down where > the problem originated from, so I'm not worried. > > >> - x / 8 where x is an int => becomes a float in Python 3, it's hard to >> detect this issue in Python 2 just by reading the source code :-/ >> > > -Q in Python 2 and/or adding the division __future__ statement solves this > one. > > Basically I'm just trying to plug the last few holes that can't be > automated by tools like Modernize or Futurize *and* won't surface where > the problem is easily during testing under Python 2 or 3 (e.g., a traceback > during testing makes it easy to find unless you swallow the exception, in > which case a warning won't help you either if you use -Werror). > > > >> >> > What do people think of extending -b/-bb in Python 3 to warn when >> performing >> > equality between an int and a bytes object of any length? I don't want >> to >> > restrict to length-1 bytes objects because people may be doing >> comparisons >> > where the result can be length-1 or any other length and thus would >> still >> > have a subtle bug to pick up. Do people think this would raise a ton of >> > false-positives? Would people find it useful? >> >> First ensure that the stdlib doesn't raise any BytesWarning exception. >> > > Yep, I always test with -Werror whenever I muck with something that can > raise a warning (side-effect from helping to write _warnings.c). > > -Brett > > >> >> For example, os.get_exec_path() has to modify warnings filters >> temporary in Python 3 :-/ >> >> # {b'PATH': ...}.get('PATH') and {'PATH': ...}.get(b'PATH') emit a >> # BytesWarning when using python -b or python -bb: ignore the warning >> >> Victor >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Sat Mar 21 16:37:32 2015 From: steve at pearwood.info (Steven D'Aprano) Date: Sun, 22 Mar 2015 02:37:32 +1100 Subject: [Python-ideas] PEP 484 (Type Hints) -- second draft In-Reply-To: References: Message-ID: <20150321153731.GN31862@ando.pearwood.info> On Fri, Mar 20, 2015 at 09:59:32AM -0700, Guido van Rossum wrote: > Union types > ----------- [...] > As a shorthand for ``Union[T1, None]`` you can write ``Optional[T1]``; That only saves three characters. Is it worth it? > An optional type is also automatically assumed when the default value is > ``None``, for example:: > > def handle_employee(e: Employee = None): ... Should that apply to all default values or just None? E.g. if I have def spam(s: str = 23): ... should that be inferred as Union[str, int] or be flagged as a type error? I think that we want a type error here, and it's only None that actually should be treated as special. Perhaps that should be made explicit in the PEP. [...] > For the purposes of type hinting, the type checker assumes ``__debug__`` > is set to ``True``, in other words the ``-O`` command-line option is not > used while type checking. I'm afraid I don't understand what you are trying to say here. I would have expected that __debug__ and -O and the type checker would be independent of each other. [...] > To mark portions of the program that should not be covered by type > hinting, use the following: > > * a ``@no_type_check`` decorator on classes and functions > > * a ``# type: ignore`` comment on arbitrary lines > > .. FIXME: should we have a module-wide comment as well? I think so, if for no other reason than it will reduce the fear of some people that type checks will be mandatory. > Type Hints on Local and Global Variables > ======================================== > > No first-class syntax support for explicitly marking variables as being > of a specific type is added by this PEP. To help with type inference in > complex cases, a comment of the following format may be used:: > > x = [] # type: List[Employee] > > In the case where type information for a local variable is needed before > it is declared, an ``Undefined`` placeholder might be used:: > > from typing import Undefined > > x = Undefined # type: List[Employee] > y = Undefined(int) How is that better than just bringing forward the variable declaration? x = [] # type: List[Employee] y = 0 > Casts > ===== > > Occasionally the type checker may need a different kind of hint: the > programmer may know that an expression is of a more constrained type > than the type checker infers. For example:: > > from typing import List > > def find_first_str(a: List[object]) -> str: > index = next(i for i, x in enumerate(a) if isinstance(x, str)) > # We only get here if there's at least one string in a > return cast(str, a[index]) > > The type checker infers the type ``object`` for ``a[index]``, but we > know that (if the code gets to that point) it must be a string. The > ``cast(t, x)`` call tells the type checker that we are confident that > the type of ``x`` is ``t``. Is the type checker supposed to unconditionally believe the cast, or only if the cast is more constrained than the infered type (like str and object, or bool and int)? E.g. if the type checker infers int, and the cast says list, I'm not entirely sure I would trust the programmer more than the type checker. My feeling here is that some type checkers will unconditionally trust the cast, and some will flag the mismatch, or offer a config option to swap between the two, and that will be a feature for type checkers to compete on. I'm also going to bike-shed the order of arguments. It seems to me that we want to say: cast(x, T) # pronounced "cast x to T" rather than Yoda-speak "cast T x to we shall" *wink*. That also matches the order of isinstance(obj, type) calls and makes it easier to remember. > At runtime a cast always returns the > expression unchanged -- it does not check the type, and it does not > convert or coerce the value. I'm a little concerned about cast() being a function. I know that it's a do-nothing function, but there's still the overhead of the name lookup and function call. It saddens me that giving a hint to the type checker has a runtime cost, small as it is. (I know that *technically* annotations have a runtime cost too, but they're once-only, at function definition time, not every time you call the function.) Your point below that cast() can be used inside expressions is a valid point, so there has to be a cast() function to support those cases, but for the example given here where the cast occurs in a return statement, wouldn't a type comment do? return some_expression # type: T hints that some_expression is to be treated as type T, regardless of what was infered. > Casts differ from type comments (see the previous section). When > using a type comment, the type checker should still verify that the > inferred type is consistent with the stated type. When using a cast, > the type checker trusts the programmer. Also, casts can be used in > expressions, while type comments only apply to assignments. > Stub Files > ========== [...] > Stub files may use the ``.py`` extension or alternatively may use the > ``.pyi`` extension. The latter makes it possible to maintain stub > files in the same directory as the corresponding real module. I don't like anything that could cause confusion between stub files and actual Python files. If we allow .py extension on stub files, I'm sure there will be confusing errors where people somehow manage to get the stub file imported instead of the actual module they want. Is there any advantage to allowing stub files use a .py extension? If not, then don't allow it. -- Steve From luciano at ramalho.org Sat Mar 21 19:30:27 2015 From: luciano at ramalho.org (Luciano Ramalho) Date: Sat, 21 Mar 2015 15:30:27 -0300 Subject: [Python-ideas] PEP 484 (Type Hints) -- second draft In-Reply-To: <20150321153731.GN31862@ando.pearwood.info> References: <20150321153731.GN31862@ando.pearwood.info> Message-ID: On Saturday, March 21, 2015, Steven D'Aprano wrote: > On Fri, Mar 20, 2015 at 09:59:32AM -0700, Guido van Rossum wrote: > > > Union types > > ----------- > [...] > > As a shorthand for ``Union[T1, None]`` you can write ``Optional[T1]``; > > That only saves three characters. Is it worth it? I think it is worth it because will be a very common case, and the shortcut is more readable. cheers, Luciano > > > > An optional type is also automatically assumed when the default value is > > ``None``, for example:: > > > > def handle_employee(e: Employee = None): ... > > Should that apply to all default values or just None? E.g. if I have > > def spam(s: str = 23): ... > > should that be inferred as Union[str, int] or be flagged as a type > error? I think that we want a type error here, and it's only None that > actually should be treated as special. Perhaps that should be made > explicit in the PEP. > > > [...] > > For the purposes of type hinting, the type checker assumes ``__debug__`` > > is set to ``True``, in other words the ``-O`` command-line option is not > > used while type checking. > > I'm afraid I don't understand what you are trying to say here. I would > have expected that __debug__ and -O and the type checker would be > independent of each other. > > > [...] > > To mark portions of the program that should not be covered by type > > hinting, use the following: > > > > * a ``@no_type_check`` decorator on classes and functions > > > > * a ``# type: ignore`` comment on arbitrary lines > > > > .. FIXME: should we have a module-wide comment as well? > > I think so, if for no other reason than it will reduce the fear of some > people that type checks will be mandatory. > > > > > Type Hints on Local and Global Variables > > ======================================== > > > > No first-class syntax support for explicitly marking variables as being > > of a specific type is added by this PEP. To help with type inference in > > complex cases, a comment of the following format may be used:: > > > > x = [] # type: List[Employee] > > > > In the case where type information for a local variable is needed before > > it is declared, an ``Undefined`` placeholder might be used:: > > > > from typing import Undefined > > > > x = Undefined # type: List[Employee] > > y = Undefined(int) > > How is that better than just bringing forward the variable declaration? > > x = [] # type: List[Employee] > y = 0 > > > > > Casts > > ===== > > > > Occasionally the type checker may need a different kind of hint: the > > programmer may know that an expression is of a more constrained type > > than the type checker infers. For example:: > > > > from typing import List > > > > def find_first_str(a: List[object]) -> str: > > index = next(i for i, x in enumerate(a) if isinstance(x, str)) > > # We only get here if there's at least one string in a > > return cast(str, a[index]) > > > > The type checker infers the type ``object`` for ``a[index]``, but we > > know that (if the code gets to that point) it must be a string. The > > ``cast(t, x)`` call tells the type checker that we are confident that > > the type of ``x`` is ``t``. > > Is the type checker supposed to unconditionally believe the cast, or > only if the cast is more constrained than the infered type (like str and > object, or bool and int)? > > E.g. if the type checker infers int, and the cast says list, I'm not > entirely sure I would trust the programmer more than the type checker. > > My feeling here is that some type checkers will unconditionally trust > the cast, and some will flag the mismatch, or offer a config option to > swap between the two, and that will be a feature for type checkers to > compete on. > > > I'm also going to bike-shed the order of arguments. It seems to me that > we want to say: > > cast(x, T) # pronounced "cast x to T" > > rather than Yoda-speak "cast T x to we shall" *wink*. That also matches > the order of isinstance(obj, type) calls and makes it easier to > remember. > > > > At runtime a cast always returns the > > expression unchanged -- it does not check the type, and it does not > > convert or coerce the value. > > I'm a little concerned about cast() being a function. I know that it's a > do-nothing function, but there's still the overhead of the name lookup > and function call. It saddens me that giving a hint to the type checker > has a runtime cost, small as it is. > > (I know that *technically* annotations have a runtime cost too, but > they're once-only, at function definition time, not every time you call > the function.) > > Your point below that cast() can be used inside expressions is a valid > point, so there has to be a cast() function to support those cases, but > for the example given here where the cast occurs in a return statement, > wouldn't a type comment do? > > return some_expression # type: T > > hints that some_expression is to be treated as type T, regardless of > what was infered. > > > > Casts differ from type comments (see the previous section). When > > using a type comment, the type checker should still verify that the > > inferred type is consistent with the stated type. When using a cast, > > the type checker trusts the programmer. Also, casts can be used in > > expressions, while type comments only apply to assignments. > > > > Stub Files > > ========== > [...] > > Stub files may use the ``.py`` extension or alternatively may use the > > ``.pyi`` extension. The latter makes it possible to maintain stub > > files in the same directory as the corresponding real module. > > I don't like anything that could cause confusion between stub files and > actual Python files. If we allow .py extension on stub files, I'm sure > there will be confusing errors where people somehow manage to get the > stub file imported instead of the actual module they want. > > Is there any advantage to allowing stub files use a .py extension? If > not, then don't allow it. > > > -- > Steve > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- Luciano Ramalho | Author of Fluent Python (O'Reilly, 2015) | http://shop.oreilly.com/product/0636920032519.do | Professor em: http://python.pro.br | Twitter: @ramalhoorg -------------- next part -------------- An HTML attachment was scrubbed... URL: From probiner at gmail.com Sun Mar 22 07:38:07 2015 From: probiner at gmail.com (pedro santos) Date: Sun, 22 Mar 2015 06:38:07 +0000 Subject: [Python-ideas] 0-base and 1-base indexed iterables? Custom slicing rules? Message-ID: Hi I'm an Art and CG student learning Python and today's exercise was about positions in a tiled room. The fact that I had to check if a position was inside the room and given that in a 1x1 room, 0.0 was considered in and 1.0 was considered out, it kept me thinking about 0-base indexing iterables and slicing. Read some articles and discussions, some pros and cons to each 0-base and 1-base, concerns about slicing, etc. But ultimately the question that got stuck in me and didn't found an answer was: Why can't both 0-base and 1-base indexing exist in the same language, and why can't slicing be customized? If I'm indexing the ruler marks, intervals, boundaries, dots, makes sense to start of at 0; rul=[0,1,2,3,4,5,6] would index every mark on my ruler so that accordingly rul[0]=0, rul[5]=5. If I'm indexing the blue circles, natural number quantities, objects, spans, makes sense to start at 1; cir= [1,2,3,4,5] so that cir[1]=1 and cir[5]=5. Now, a lot of the discussion was to do with slicing coupled with the indexing and I don't totally understand why. a ? x < b is not so intuitive when dealing with objects ("I want balls 1 up to the the one before 3"), so on one side, you put the finger on what you want and on the other, on what you don't want. But this method does have the neat property of producing neighbor selections that border perfectly, as in [:a][a:b][b:c]. Although in inverse order(-1), the results can be unexpected as it returns values off-by-one from its counterpart like; L=[0,1,2,3,4,5] so that L[1:3]=[1,2] and L[3:1:-1]=[3:2]. So it's consistent with the rule a ? x < b, grabbing the lower limit item, but it can feel strange by not producing the same selection with inverse order. a ? x ? b is a natural way to select objets ("I want the balls 1 up to 3"), so you're putting the finger on the things you want. If you inverse the order(-1) it's still very easy to grasp what are you picking because whatever you select it's included like: L=[0,1,2,3,4,5] so that L[1:3]=[1,2,3] and L[3:1:-1]=[3,2,1]. Problems seem to arrive though, when trying to do neighbor selections, where one would have to do [:a][a+1:b][b+1:c] to have the border perfectly. That terrible? Even though one could see a ? x < b to be more adept to 0-base, and a ? x ? b to be more adept to 1-base, the way I see it, index base and slicing rules could be somehow independent. And one would index and slice the way it would fit the rationale of the problem or the data, because even slicing a 1-base indexed array with a ? x < b, would still produce an expected outcome, as in cir=[1,2,3,4,5] so that cir[1:3]=[1,2] or cir[:3]=[1,2]. Same thing applying a ? x ? b to a 0-base indexed array, as in rul[0,1,2,3,4,5] so that rul[:2]=[0,1,2] or rul[0:2]=[0,1,2]. Given that python is an example of human friendly code language, emphasizing readability, wouldn't having 0 and 1 base indexing and custom slicing methods, improve the thought process when writing and reading the code, by fitting them better to specific contexts or data? Is there some language that provides both or each language picks only one? Cheers -- *------------------------------[image: http://i153.photobucket.com/albums/s202/animatics/probiner-sig.gif]Pedro Alpiar?a dos Santos Animator 3DModeler Illustrator >> http://probiner.x10.mx/ * -------------- next part -------------- An HTML attachment was scrubbed... URL: From techtonik at gmail.com Sat Mar 21 10:46:48 2015 From: techtonik at gmail.com (anatoly techtonik) Date: Sat, 21 Mar 2015 12:46:48 +0300 Subject: [Python-ideas] Geo coordinates conversion in stdlib Message-ID: I think it should be in stdlib. 11? 18? 0? S, 2? 42? 0? E to -11.3, 2.7 and back. This is from Wikipedia. -- anatoly t. From luciano at ramalho.org Sun Mar 22 08:24:12 2015 From: luciano at ramalho.org (Luciano Ramalho) Date: Sun, 22 Mar 2015 04:24:12 -0300 Subject: [Python-ideas] 0-base and 1-base indexed iterables? Custom slicing rules? In-Reply-To: References: Message-ID: On Sun, Mar 22, 2015 at 3:38 AM, pedro santos wrote: > Why can't both 0-base and 1-base indexing exist in the same language, and why can't slicing be customized? Pedro, The best answer ever is by one of the most important computer scientists of the 20th century, Edsger W. Dijkstra -- who was also Dutch, like Guido van Rossum: https://www.cs.utexas.edu/users/EWD/ewd08xx/EWD831.PDF The answer is also visually beautiful, as are all Dijkstra's manuscripts. Some font designer should create a font based on Dijkstra's handwriting. Cheers, Luciano > > > > If I'm indexing the ruler marks, intervals, boundaries, dots, makes sense to start of at 0; rul=[0,1,2,3,4,5,6] would index every mark on my ruler so that accordingly rul[0]=0, rul[5]=5. > If I'm indexing the blue circles, natural number quantities, objects, spans, makes sense to start at 1; cir= [1,2,3,4,5] so that cir[1]=1 and cir[5]=5. > > Now, a lot of the discussion was to do with slicing coupled with the indexing and I don't totally understand why. > > a ? x < b is not so intuitive when dealing with objects ("I want balls 1 up to the the one before 3"), so on one side, you put the finger on what you want and on the other, on what you don't want. But this method does have the neat property of producing neighbor selections that border perfectly, as in [:a][a:b][b:c]. Although in inverse order(-1), the results can be unexpected as it returns values off-by-one from its counterpart like; L=[0,1,2,3,4,5] so that L[1:3]=[1,2] and L[3:1:-1]=[3:2]. So it's consistent with the rule a ? x < b, grabbing the lower limit item, but it can feel strange by not producing the same selection with inverse order. > > a ? x ? b is a natural way to select objets ("I want the balls 1 up to 3"), so you're putting the finger on the things you want. If you inverse the order(-1) it's still very easy to grasp what are you picking because whatever you select it's included like: L=[0,1,2,3,4,5] so that L[1:3]=[1,2,3] and L[3:1:-1]=[3,2,1]. Problems seem to arrive though, when trying to do neighbor selections, where one would have to do [:a][a+1:b][b+1:c] to have the border perfectly. That terrible? > > Even though one could see a ? x < b to be more adept to 0-base, and a ? x ? b to be more adept to 1-base, the way I see it, index base and slicing rules could be somehow independent. And one would index and slice the way it would fit the rationale of the problem or the data, because even slicing a 1-base indexed array with a ? x < b, would still produce an expected outcome, as in cir=[1,2,3,4,5] so that cir[1:3]=[1,2] or cir[:3]=[1,2]. > Same thing applying a ? x ? b to a 0-base indexed array, as in rul[0,1,2,3,4,5] so that rul[:2]=[0,1,2] or rul[0:2]=[0,1,2]. > > Given that python is an example of human friendly code language, emphasizing readability, wouldn't having 0 and 1 base indexing and custom slicing methods, improve the thought process when writing and reading the code, by fitting them better to specific contexts or data? > Is there some language that provides both or each language picks only one? > > Cheers > > > -- > ________________________________ > > Pedro Alpiar?a dos Santos > Animator 3DModeler Illustrator > >> http://probiner.x10.mx/ > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -- Luciano Ramalho | Author of Fluent Python (O'Reilly, 2015) | http://shop.oreilly.com/product/0636920032519.do | Professor em: http://python.pro.br | Twitter: @ramalhoorg From ben+python at benfinney.id.au Sun Mar 22 08:54:28 2015 From: ben+python at benfinney.id.au (Ben Finney) Date: Sun, 22 Mar 2015 18:54:28 +1100 Subject: [Python-ideas] =?utf-8?b?TyDigJNBUE9P4oCTIE9XVERJICh3YXM6IDAtYmFz?= =?utf-8?q?e_and_1-base_indexed_iterables=3F_Custom_slicing_rules=3F=29?= References: Message-ID: <85d24179hn.fsf_-_@benfinney.id.au> Luciano Ramalho writes: > The best answer ever is by one of the most important computer > scientists of the 20th century, Edsger W. Dijkstra -- who was also > Dutch, like Guido van Rossum: > > https://www.cs.utexas.edu/users/EWD/ewd08xx/EWD831.PDF Fascinatingly ? to me, at least ? Dijkstra's orthography for parenthetic en dash is the opposite of that Tim Peters used in the thirteenth aphorism of the Zen of Python. Dijkstra, EWD831: Exclusion of the lower bound ?as in a) and b)? forces [?] Peters, Zen of Python: There should be one? and preferably only one ?obvious way to do it. How wonderful to see another dimension of combinations for that style issue, in a Dijkstra paper talking about exploring the dimensions of another style issue :-) -- \ ?When I get new information, I change my position. What, sir, | `\ do you do with new information?? ?John Maynard Keynes | _o__) | Ben Finney From rosuav at gmail.com Sun Mar 22 11:11:38 2015 From: rosuav at gmail.com (Chris Angelico) Date: Sun, 22 Mar 2015 21:11:38 +1100 Subject: [Python-ideas] Geo coordinates conversion in stdlib In-Reply-To: References: Message-ID: On Sat, Mar 21, 2015 at 8:46 PM, anatoly techtonik wrote: > I think it should be in stdlib. > > 11? 18? 0? S, 2? 42? 0? E to -11.3, 2.7 and back. > > This is from Wikipedia. Put some code together, stick it on PyPI... oh, and while you're there, see if someone else has already written it. You never know, it might well be there! ChrisA From liik.joonas at gmail.com Sun Mar 22 15:15:39 2015 From: liik.joonas at gmail.com (Joonas Liik) Date: Sun, 22 Mar 2015 16:15:39 +0200 Subject: [Python-ideas] 0-base and 1-base indexed iterables? Custom slicing rules? In-Reply-To: References: Message-ID: Hey In one of my projects I'm stuck in an environment where there are multiple co-existing coding environments, one of which has a convention of using 0 based indexing and the other using 1 based indexing. All i can say is that is is a VERY (i cant possibly emphasize this enough) bad idea. it is an unending source of confusion and off by 1 bugs. It is like the line endings issue all over again ... CR or LF and then you end up with CR, LF and CR LF, choosing 1 is definitely better than choosing both. Python got this right, please do not inflict this upon the python community! I got real life experience with this and i'm telling you this is a disaster waiting to happen, please don't.. On 22 March 2015 at 09:24, Luciano Ramalho wrote: > On Sun, Mar 22, 2015 at 3:38 AM, pedro santos wrote: > > Why can't both 0-base and 1-base indexing exist in the same language, > and why can't slicing be customized? > > Pedro, > > The best answer ever is by one of the most important computer > scientists of the 20th century, Edsger W. Dijkstra -- who was also > Dutch, like Guido van Rossum: > > https://www.cs.utexas.edu/users/EWD/ewd08xx/EWD831.PDF > > The answer is also visually beautiful, as are all Dijkstra's > manuscripts. Some font designer should create a font based on > Dijkstra's handwriting. > > Cheers, > > Luciano > > > > > > > > > > If I'm indexing the ruler marks, intervals, boundaries, dots, makes > sense to start of at 0; rul=[0,1,2,3,4,5,6] would index every mark on my > ruler so that accordingly rul[0]=0, rul[5]=5. > > If I'm indexing the blue circles, natural number quantities, objects, > spans, makes sense to start at 1; cir= [1,2,3,4,5] so that cir[1]=1 and > cir[5]=5. > > > > Now, a lot of the discussion was to do with slicing coupled with the > indexing and I don't totally understand why. > > > > a ? x < b is not so intuitive when dealing with objects ("I want balls 1 > up to the the one before 3"), so on one side, you put the finger on what > you want and on the other, on what you don't want. But this method does > have the neat property of producing neighbor selections that border > perfectly, as in [:a][a:b][b:c]. Although in inverse order(-1), the results > can be unexpected as it returns values off-by-one from its counterpart > like; L=[0,1,2,3,4,5] so that L[1:3]=[1,2] and L[3:1:-1]=[3:2]. So it's > consistent with the rule a ? x < b, grabbing the lower limit item, but it > can feel strange by not producing the same selection with inverse order. > > > > a ? x ? b is a natural way to select objets ("I want the balls 1 up to > 3"), so you're putting the finger on the things you want. If you inverse > the order(-1) it's still very easy to grasp what are you picking because > whatever you select it's included like: L=[0,1,2,3,4,5] so that > L[1:3]=[1,2,3] and L[3:1:-1]=[3,2,1]. Problems seem to arrive though, when > trying to do neighbor selections, where one would have to do > [:a][a+1:b][b+1:c] to have the border perfectly. That terrible? > > > > Even though one could see a ? x < b to be more adept to 0-base, and a ? > x ? b to be more adept to 1-base, the way I see it, index base and slicing > rules could be somehow independent. And one would index and slice the way > it would fit the rationale of the problem or the data, because even slicing > a 1-base indexed array with a ? x < b, would still produce an expected > outcome, as in cir=[1,2,3,4,5] so that cir[1:3]=[1,2] or cir[:3]=[1,2]. > > Same thing applying a ? x ? b to a 0-base indexed array, as in > rul[0,1,2,3,4,5] so that rul[:2]=[0,1,2] or rul[0:2]=[0,1,2]. > > > > Given that python is an example of human friendly code language, > emphasizing readability, wouldn't having 0 and 1 base indexing and custom > slicing methods, improve the thought process when writing and reading the > code, by fitting them better to specific contexts or data? > > Is there some language that provides both or each language picks only > one? > > > > Cheers > > > > > > -- > > ________________________________ > > > > Pedro Alpiar?a dos Santos > > Animator 3DModeler Illustrator > > >> http://probiner.x10.mx/ > > > > > > _______________________________________________ > > Python-ideas mailing list > > Python-ideas at python.org > > https://mail.python.org/mailman/listinfo/python-ideas > > Code of Conduct: http://python.org/psf/codeofconduct/ > > > > > -- > Luciano Ramalho > | Author of Fluent Python (O'Reilly, 2015) > | http://shop.oreilly.com/product/0636920032519.do > | Professor em: http://python.pro.br > | Twitter: @ramalhoorg > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Sun Mar 22 15:30:12 2015 From: rosuav at gmail.com (Chris Angelico) Date: Mon, 23 Mar 2015 01:30:12 +1100 Subject: [Python-ideas] 0-base and 1-base indexed iterables? Custom slicing rules? In-Reply-To: References: Message-ID: On Sun, Mar 22, 2015 at 5:38 PM, pedro santos wrote: > > a ? x < b is not so intuitive when dealing with objects ("I want balls 1 up to the the one before 3"), so on one side, you put the finger on what you want and on the other, on what you don't want. But this method does have the neat property of producing neighbor selections that border perfectly, as in [:a][a:b][b:c]. Although in inverse order(-1), the results can be unexpected as it returns values off-by-one from its counterpart like; L=[0,1,2,3,4,5] so that L[1:3]=[1,2] and L[3:1:-1]=[3:2]. So it's consistent with the rule a ? x < b, grabbing the lower limit item, but it can feel strange by not producing the same selection with inverse order. > > a ? x ? b is a natural way to select objets ("I want the balls 1 up to 3"), so you're putting the finger on the things you want. If you inverse the order(-1) it's still very easy to grasp what are you picking because whatever you select it's included like: L=[0,1,2,3,4,5] so that L[1:3]=[1,2,3] and L[3:1:-1]=[3,2,1]. Problems seem to arrive though, when trying to do neighbor selections, where one would have to do [:a][a+1:b][b+1:c] to have the border perfectly. That terrible? > To add to what other people have said in support of Python's indexing style, I'd like to point out one situation where fully-closed ranges are used: Scripture references. Have a look at this page: https://www.biblegateway.com/passage/?search=Mark+3&version=NIV This is one chapter from the Bible. Passages within that chapter can be identified by starting and ending verse, inclusive at both ends. So, for instance, the paragraph under the heading "Jesus Appoints the Twelve" would be Mark 3:13-19. Note how the end point is actually one less than the number that starts the next passage. Python identifies the boundaries between elements, which corresponds perfectly to a numbering system such as this; noting the passage as "Mark 3:13-20" would mean you start reading at the indicator for verse 13, and stop reading as soon as you reach the indicator for verse 20, which makes a lot more sense. Obviously it's way WAY too hard to change something as extensively used as Bible references - the backward incompatibility concern is just too strong - but this may give some indication of why it's more sensible to work with half-open ranges. A half-open range also gives a convenient notation for "to the end of the chapter": instead of "Mark 3:20-3:35", it would simply be "Mark 3:20-4:1". (Chapters and verses are numbered from one, not zero, but in an ideal world, this would also change, so it would be "3:19-4:0".) ChrisA From abarnert at yahoo.com Sun Mar 22 15:52:07 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Sun, 22 Mar 2015 07:52:07 -0700 Subject: [Python-ideas] 0-base and 1-base indexed iterables? Custom slicing rules? In-Reply-To: References: Message-ID: Your first example is 0-based even though it's closed (L[1:3]=[1,2,3]), which weakens your whole argument for 1-based indexing. and makes me think such a possibility might actually confuse you--which is exactly why we don't have it. As you say, it may still make sense to have closed ranges and slices alongside half-open, even with 0-based indices. And many newer languages do that, often by using ".." vs. "...", which seems to be pretty readable. But there's no obvious way to add something similar to Python at this point. (Also, I think the fact that ranges and slices are different things with different syntax is a benefit of Python, not a deficit, and it's even harder to design a good solution without losing that benefit...) Sent from my iPhone > On Mar 21, 2015, at 11:38 PM, pedro santos wrote: > > Hi > > I'm an Art and CG student learning Python and today's exercise was about positions in a tiled room. The fact that I had to check if a position was inside the room and given that in a 1x1 room, 0.0 was considered in and 1.0 was considered out, it kept me thinking about 0-base indexing iterables and slicing. > > Read some articles and discussions, some pros and cons to each 0-base and 1-base, concerns about slicing, etc. But ultimately the question that got stuck in me and didn't found an answer was: > Why can't both 0-base and 1-base indexing exist in the same language, and why can't slicing be customized? > > > > > > > If I'm indexing the ruler marks, intervals, boundaries, dots, makes sense to start of at 0; rul=[0,1,2,3,4,5,6] would index every mark on my ruler so that accordingly rul[0]=0, rul[5]=5. > If I'm indexing the blue circles, natural number quantities, objects, spans, makes sense to start at 1; cir= [1,2,3,4,5] so that cir[1]=1 and cir[5]=5. > > Now, a lot of the discussion was to do with slicing coupled with the indexing and I don't totally understand why. > > a ? x < b is not so intuitive when dealing with objects ("I want balls 1 up to the the one before 3"), so on one side, you put the finger on what you want and on the other, on what you don't want. But this method does have the neat property of producing neighbor selections that border perfectly, as in [:a][a:b][b:c]. Although in inverse order(-1), the results can be unexpected as it returns values off-by-one from its counterpart like; L=[0,1,2,3,4,5] so that L[1:3]=[1,2] and L[3:1:-1]=[3:2]. So it's consistent with the rule a ? x < b, grabbing the lower limit item, but it can feel strange by not producing the same selection with inverse order. > > a ? x ? b is a natural way to select objets ("I want the balls 1 up to 3"), so you're putting the finger on the things you want. If you inverse the order(-1) it's still very easy to grasp what are you picking because whatever you select it's included like: L=[0,1,2,3,4,5] so that L[1:3]=[1,2,3] and L[3:1:-1]=[3,2,1]. Problems seem to arrive though, when trying to do neighbor selections, where one would have to do [:a][a+1:b][b+1:c] to have the border perfectly. That terrible? > > Even though one could see a ? x < b to be more adept to 0-base, and a ? x ? b to be more adept to 1-base, the way I see it, index base and slicing rules could be somehow independent. And one would index and slice the way it would fit the rationale of the problem or the data, because even slicing a 1-base indexed array with a ? x < b, would still produce an expected outcome, as in cir=[1,2,3,4,5] so that cir[1:3]=[1,2] or cir[:3]=[1,2]. > Same thing applying a ? x ? b to a 0-base indexed array, as in rul[0,1,2,3,4,5] so that rul[:2]=[0,1,2] or rul[0:2]=[0,1,2]. > > Given that python is an example of human friendly code language, emphasizing readability, wouldn't having 0 and 1 base indexing and custom slicing methods, improve the thought process when writing and reading the code, by fitting them better to specific contexts or data? > Is there some language that provides both or each language picks only one? > > Cheers > > > -- > > Pedro Alpiar?a dos Santos > Animator 3DModeler Illustrator > >> http://probiner.x10.mx/ > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From ron3200 at gmail.com Sun Mar 22 19:50:16 2015 From: ron3200 at gmail.com (Ron Adam) Date: Sun, 22 Mar 2015 14:50:16 -0400 Subject: [Python-ideas] 0-base and 1-base indexed iterables? Custom slicing rules? In-Reply-To: References: Message-ID: On 03/22/2015 10:30 AM, Chris Angelico wrote: > To add to what other people have said in support of Python's indexing > style, I'd like to point out one situation where fully-closed ranges > are used: Scripture references. While indexing and slicing use natural numbers, the advantage of the half open interval becomes more obvious if you consider examples with real numbers. When using a closed range interval with real numbers ... n1 <= i <= n2 ? <= i <= n3 Picking the start value for the next adjacent sequence is not simply a matter of adding 1. The half open sequence makes that case both trivial and elegant. n1 <= i < n2 n2 <= i < n3 And it works just as well with natural numbers. And works best if the end value is a multiple of the step value. As to weather to use 0 or 1 indexing, 0 indexing is very useful for interfacing with the hardware. 0 is a valid memory address, and the 0th bit is the first bit of a byte. Indexing at that level directly correspond to how the hardware is built. For other higher level things, ones indexing can work equally well, (in my opinion), we just don't want to mix them. 0 <= i < n # n equals number of items 1 <= i < n + 1 # Dijkstra prefers the first case. But consider in programming, the indexes are frequently variables rather than literals. start <= i < start + count This works in both cases. In the above case of zero indexing it's really. 0 <= i < 0 + n It just happens that it can be simplified a bit in the case where there start value is zero. And then the end index becomes the count, but only when the start index is zero. When ones indexing is used, the zero index can be used to specify the end value when stepping towards zero from both directions. When zero indexing is used. it allows for wrap around behaviour when the index crosses zero. Sometimes that can be very nice. A reason a language should choose one or the other, and not both, is that the choice defines how many other things interact with that choice. Trying to do both can mean having two versions of many functions and statements. Cheers, Ron From mertz at gnosis.cx Sun Mar 22 20:52:40 2015 From: mertz at gnosis.cx (David Mertz) Date: Sun, 22 Mar 2015 12:52:40 -0700 Subject: [Python-ideas] 0-base and 1-base indexed iterables? Custom slicing rules? In-Reply-To: References: Message-ID: The strongest proof that mixed index bases are a bad idea is that this is a valid statement in VB6: Dim array(1 to 10) as Integer For all X, if VB6 allows X, then X is a bad idea. :-) -- Keeping medicines from the bloodstreams of the sick; food from the bellies of the hungry; books from the hands of the uneducated; technology from the underdeveloped; and putting advocates of freedom in prisons. Intellectual property is to the 21st century what the slave trade was to the 16th. -------------- next part -------------- An HTML attachment was scrubbed... URL: From tritium-list at sdamon.com Sun Mar 22 22:15:39 2015 From: tritium-list at sdamon.com (Alexander Walters) Date: Sun, 22 Mar 2015 17:15:39 -0400 Subject: [Python-ideas] Geo coordinates conversion in stdlib In-Reply-To: References: Message-ID: <550F30FB.5010000@sdamon.com> On 3/21/2015 05:46, anatoly techtonik wrote: > I think it should be in stdlib. > > 11? 18? 0? S, 2? 42? 0? E to -11.3, 2.7 and back. > > This is from Wikipedia. Why does this need to be in the standard library? From solipsis at pitrou.net Sun Mar 22 22:38:15 2015 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sun, 22 Mar 2015 22:38:15 +0100 Subject: [Python-ideas] Geo coordinates conversion in stdlib References: <550F30FB.5010000@sdamon.com> Message-ID: <20150322223815.06082280@fsol> On Sun, 22 Mar 2015 17:15:39 -0400 Alexander Walters wrote: > On 3/21/2015 05:46, anatoly techtonik wrote: > > I think it should be in stdlib. > > > > 11? 18? 0? S, 2? 42? 0? E to -11.3, 2.7 and back. > > > > This is from Wikipedia. > Why does this need to be in the standard library? Because Anatoly needs it, of course :) Regards Antoine. From abarnert at yahoo.com Sun Mar 22 23:37:51 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Sun, 22 Mar 2015 15:37:51 -0700 Subject: [Python-ideas] Geo coordinates conversion in stdlib In-Reply-To: <20150322223815.06082280@fsol> References: <550F30FB.5010000@sdamon.com> <20150322223815.06082280@fsol> Message-ID: <21A308F0-5F2E-48BB-9F3F-4A3CFAB1D6DD@yahoo.com> On Mar 22, 2015, at 2:38 PM, Antoine Pitrou wrote: > > On Sun, 22 Mar 2015 17:15:39 -0400 > Alexander Walters > wrote: >>> On 3/21/2015 05:46, anatoly techtonik wrote: >>> I think it should be in stdlib. >>> >>> 11? 18? 0? S, 2? 42? 0? E to -11.3, 2.7 and back. >>> >>> This is from Wikipedia. >> Why does this need to be in the standard library? > > Because Anatoly needs it, of course :) Well, it would certainly be nice to have all of Wikipedia in the stdlib, but I wonder how that would affect Python's release schedule, and its use in resource-constrained systems. More seriously, if you're just looking to convert arcminutes and arcseconds to and from fractional degrees, it's exactly the same algorithm as converting minutes and seconds to fractional hours, which we already have in the datetime.timedelta class. If you can't figure out the complexity of multiplying and divmodding by 60 twice, you can borrow its code, or just wrap it and delegate. That seems trivial enough to do in any project that needs it. And that means projects that want degrees-arcminutes-arcseconds-milliarcseconds (e.g., for astronomy) or just degrees and fractional arcminutes with no arcseconds (e.g., for marine navigation) or degrees, arcminutes, and eighth-arcminutes (e.g., for US military scopes) can just as easily get what they need, instead of being forced to use the variation surveyors prefer. From chris.barker at noaa.gov Mon Mar 23 00:13:57 2015 From: chris.barker at noaa.gov (Chris Barker) Date: Sun, 22 Mar 2015 16:13:57 -0700 Subject: [Python-ideas] Geo coordinates conversion in stdlib In-Reply-To: <21A308F0-5F2E-48BB-9F3F-4A3CFAB1D6DD@yahoo.com> References: <550F30FB.5010000@sdamon.com> <20150322223815.06082280@fsol> <21A308F0-5F2E-48BB-9F3F-4A3CFAB1D6DD@yahoo.com> Message-ID: On Sun, Mar 22, 2015 at 3:37 PM, Andrew Barnert < abarnert at yahoo.com.dmarc.invalid> wrote: > If you can't figure out the complexity of multiplying and divmodding by > 60 twice, you can borrow its code, or just wrap it and delegate. That seems > trivial enough to do in any project that needs it. I"m not suggesting this is an stdlib candiate, but it'snot quite that simple. With FP precision issues you can end up with results like: 1? 60? 13? which you don't want. I haven't checked to see if datetime deal siwththis corectly but I'd guess it does :-) -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From davidfstr at gmail.com Mon Mar 23 07:02:40 2015 From: davidfstr at gmail.com (David Foster) Date: Sun, 22 Mar 2015 23:02:40 -0700 Subject: [Python-ideas] PEP 484 (Type Hints) -- second draft In-Reply-To: References: Message-ID: <550FAC80.8040405@gmail.com> * It's a bit vague how local variables should be typed in the presence of multiple assignments. Consider: ``` if x > 0: root = sqrt(x) # type: int else: root = None ``` It seems that the type of `root` should be inferred as `Union[int, None]` (AKA `Optional[int]`), but there's no discussion about this type of inference in the PEP. Cases like this are the only way that the proposed blank `Undefined` value makes sense to me, using an example like: ``` root = Undefined # type: Optional[int] if x > 0: root = sqrt(x) # type: int else: root = None ``` * +1 for only allowing .pyi as the extension for stub files (and not also .py). Makes it very clear that they should only be interpreted as stubs. * -1 that `Optional[T1]` is unnecessary. It's conceptually more clean and directly corresponds conceptually to types like `int?` and `?int` from other languages. * +1 for the argument order cast(x, T). This is consistent with (x: T) elsewhere. -- David Foster http://dafoster.net/ P.S. Apologies if this doesn't add to the message chain correctly. There's no clearly correct way to reply to a digest email from Mailman. From techtonik at gmail.com Mon Mar 23 07:08:23 2015 From: techtonik at gmail.com (anatoly techtonik) Date: Mon, 23 Mar 2015 09:08:23 +0300 Subject: [Python-ideas] Geo coordinates conversion in stdlib In-Reply-To: <21A308F0-5F2E-48BB-9F3F-4A3CFAB1D6DD@yahoo.com> References: <550F30FB.5010000@sdamon.com> <20150322223815.06082280@fsol> <21A308F0-5F2E-48BB-9F3F-4A3CFAB1D6DD@yahoo.com> Message-ID: On Mon, Mar 23, 2015 at 1:37 AM, Andrew Barnert wrote: > On Mar 22, 2015, at 2:38 PM, Antoine Pitrou wrote: >> >> On Sun, 22 Mar 2015 17:15:39 -0400 >> Alexander Walters >> wrote: >>>> On 3/21/2015 05:46, anatoly techtonik wrote: >>>> I think it should be in stdlib. >>>> >>>> 11? 18? 0? S, 2? 42? 0? E to -11.3, 2.7 and back. >>>> >>>> This is from Wikipedia. >>> Why does this need to be in the standard library? >> >> Because Anatoly needs it, of course :) > > Well, it would certainly be nice to have all of Wikipedia in the stdlib, but I wonder how that would affect Python's release schedule, and its use in resource-constrained systems. > > More seriously, if you're just looking to convert arcminutes and arcseconds to and from fractional degrees, it's exactly the same algorithm as converting minutes and seconds to fractional hours, which we already have in the datetime.timedelta class. If you can't figure out the complexity of multiplying and divmodding by 60 twice, you can borrow its code, or just wrap it and delegate. That seems trivial enough to do in any project that needs it. And that means projects that want degrees-arcminutes-arcseconds-milliarcseconds (e.g., for astronomy) or just degrees and fractional arcminutes with no arcseconds (e.g., for marine navigation) or degrees, arcminutes, and eighth-arcminutes (e.g., for US military scopes) can just as easily get what they need, instead of being forced to use the variation surveyors prefer. That's nice to know, but IIRC datetime is from the top 10 Python modules that need a redesign. Things contained therein doesn't pass human usability check, and are not used as a result. My story for that day is that I needed to process geodata from Wikipedia to see the distribution over the Earth surface, so I thought that if I know the Python it is not a problem. It appeared that the problem is there, and the problem is recurring (last time I had GPS points). Python docs explain a lot about core OS concepts, but Python is not only a systems language. I'd say that it is now being replaced with Go for all system stuff, and more attempts are made to use is as a data processing language. But that is not possible if Python doesn't have a basic concepts of data representation from the Real Life (RL) (tm) in its stdlib. Handling geo coordinates in most used formats is as important as handling .zip archives in stdlib. -- anatoly t. From rosuav at gmail.com Mon Mar 23 08:21:32 2015 From: rosuav at gmail.com (Chris Angelico) Date: Mon, 23 Mar 2015 18:21:32 +1100 Subject: [Python-ideas] Geo coordinates conversion in stdlib In-Reply-To: References: <550F30FB.5010000@sdamon.com> <20150322223815.06082280@fsol> <21A308F0-5F2E-48BB-9F3F-4A3CFAB1D6DD@yahoo.com> Message-ID: On Mon, Mar 23, 2015 at 5:08 PM, anatoly techtonik wrote: > Handling geo coordinates in most used formats is as important as > handling .zip archives in stdlib. That may or may not be the case, but if it is, the best way to argue the point is: 1) Find or create a third-party package on PyPI 2) Demonstrate that this package has a stable API 3) Demonstrate or pledge support for the package 4) Appeal for the package's inclusion in the stdlib You'll need to show that there's benefit to having it in the stdlib as opposed to on PyPI - demonstrate the problem with "pip install packagename" that means that you want it in the standard library. Just *stating* that there is a need isn't sufficient - you need to prove it. ChrisA From abarnert at yahoo.com Mon Mar 23 09:29:16 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Mon, 23 Mar 2015 01:29:16 -0700 Subject: [Python-ideas] Geo coordinates conversion in stdlib In-Reply-To: References: <550F30FB.5010000@sdamon.com> <20150322223815.06082280@fsol> <21A308F0-5F2E-48BB-9F3F-4A3CFAB1D6DD@yahoo.com> Message-ID: <019975C9-137D-4834-80E6-09F471A789B3@yahoo.com> On Mar 22, 2015, at 11:08 PM, anatoly techtonik wrote: > > On Mon, Mar 23, 2015 at 1:37 AM, Andrew Barnert > wrote: >> On Mar 22, 2015, at 2:38 PM, Antoine Pitrou wrote: >>> >>> On Sun, 22 Mar 2015 17:15:39 -0400 >>> Alexander Walters >>> wrote: >>>>> On 3/21/2015 05:46, anatoly techtonik wrote: >>>>> I think it should be in stdlib. >>>>> >>>>> 11? 18? 0? S, 2? 42? 0? E to -11.3, 2.7 and back. >>>>> >>>>> This is from Wikipedia. >>>> Why does this need to be in the standard library? >>> >>> Because Anatoly needs it, of course :) >> >> Well, it would certainly be nice to have all of Wikipedia in the stdlib, but I wonder how that would affect Python's release schedule, and its use in resource-constrained systems. >> >> More seriously, if you're just looking to convert arcminutes and arcseconds to and from fractional degrees, it's exactly the same algorithm as converting minutes and seconds to fractional hours, which we already have in the datetime.timedelta class. If you can't figure out the complexity of multiplying and divmodding by 60 twice, you can borrow its code, or just wrap it and delegate. That seems trivial enough to do in any project that needs it. And that means projects that want degrees-arcminutes-arcseconds-milliarcseconds (e.g., for astronomy) or just degrees and fractional arcminutes with no arcseconds (e.g., for marine navigation) or degrees, arcminutes, and eighth-arcminutes (e.g., for US military scopes) can just as easily get what they need, instead of being forced to use the variation surveyors prefer. > > That's nice to know, but IIRC datetime is from the top 10 Python > modules that need a redesign. Things contained therein doesn't pass > human usability check, and are not used as a result. There are thousands of projects that actually do use it; thousands of people asking questions on StackOverflow, Python-tutor, etc. and getting actual answers; etc. So your theoretical argument that it's not usable doesn't really matter given that practically it is actually usable, and used. If you just opened up an interactive interpreter and player with the datetime.timedelta class for a few seconds instead of trying to guess based on hearsay and abstract principles whether it might be useful, you could have figured this out for yourself. > My story for that day is that I needed to process geodata from > Wikipedia to see the distribution over the Earth surface, so I thought > that if I know the Python it is not a problem. It appeared that the > problem is there, and the problem is recurring (last time I had GPS > points). I suspect that people who do a lot of geodata processing in Python already have third-party libs that they use, and would have no interest in a very small limited subset being duplicated in the stdlib. > Python docs explain a lot about core OS concepts, but Python > is not only a systems language. I'd say that it is now being replaced > with Go for all system stuff, I don't know anyone outside Google who's using Go for any system (or sysadmin, which seems to be more likely what you're talking about) stuff. I'm sure there are exceptions, but the idea that Python is being universally replaced by Go in that area is so ridiculous that it's hard to believe you actually meant it seriously. It's like saying that Ruby is being replaced by Scala for all web server stuff just because Scala is the fastest-growing language in that area this week. > and more attempts are made to use is as > a data processing language. But that is not possible if Python doesn't > have a basic concepts of data representation from the Real Life (RL) > (tm) in its stdlib. Most people I know who use Python as a serious data processing language don't care much what's in the stdlib because they're using numpy and/or pandas. > Handling geo coordinates in most used formats is as important as > handling .zip archives in stdlib. Not even close. Zip archives are used in a ridiculously wide range of application areas; geoccordinates are used only in processing geocoordinate data (and there only in small enough scales that you don't need numpy or some specialized geodata library that I don't know about but that I'll bet exists). From breamoreboy at yahoo.co.uk Mon Mar 23 09:51:09 2015 From: breamoreboy at yahoo.co.uk (Mark Lawrence) Date: Mon, 23 Mar 2015 08:51:09 +0000 Subject: [Python-ideas] Geo coordinates conversion in stdlib In-Reply-To: References: <550F30FB.5010000@sdamon.com> <20150322223815.06082280@fsol> <21A308F0-5F2E-48BB-9F3F-4A3CFAB1D6DD@yahoo.com> Message-ID: On 23/03/2015 06:08, anatoly techtonik wrote: > On Mon, Mar 23, 2015 at 1:37 AM, Andrew Barnert > wrote: >> On Mar 22, 2015, at 2:38 PM, Antoine Pitrou wrote: >>> >>> On Sun, 22 Mar 2015 17:15:39 -0400 >>> Alexander Walters >>> wrote: >>>>> On 3/21/2015 05:46, anatoly techtonik wrote: >>>>> I think it should be in stdlib. >>>>> >>>>> 11? 18? 0? S, 2? 42? 0? E to -11.3, 2.7 and back. >>>>> >>>>> This is from Wikipedia. >>>> Why does this need to be in the standard library? >>> >>> Because Anatoly needs it, of course :) >> >> Well, it would certainly be nice to have all of Wikipedia in the stdlib, but I wonder how that would affect Python's release schedule, and its use in resource-constrained systems. >> >> More seriously, if you're just looking to convert arcminutes and arcseconds to and from fractional degrees, it's exactly the same algorithm as converting minutes and seconds to fractional hours, which we already have in the datetime.timedelta class. If you can't figure out the complexity of multiplying and divmodding by 60 twice, you can borrow its code, or just wrap it and delegate. That seems trivial enough to do in any project that needs it. And that means projects that want degrees-arcminutes-arcseconds-milliarcseconds (e.g., for astronomy) or just degrees and fractional arcminutes with no arcseconds (e.g., for marine navigation) or degrees, arcminutes, and eighth-arcminutes (e.g., for US military scopes) can just as easily get what they need, instead of being forced to use the variation surveyors prefer. > > That's nice to know, but IIRC datetime is from the top 10 Python > modules that need a redesign. I'm looking forward to seeing your redesign for all 10 modules on the bug tracker. Can you get this done in time for 3.5 beta? -- My fellow Pythonistas, ask not what our language can do for you, ask what you can do for our language. Mark Lawrence From tjreedy at udel.edu Mon Mar 23 09:55:55 2015 From: tjreedy at udel.edu (Terry Reedy) Date: Mon, 23 Mar 2015 04:55:55 -0400 Subject: [Python-ideas] PEP 484 (Type Hints) -- second draft In-Reply-To: <550FAC80.8040405@gmail.com> References: <550FAC80.8040405@gmail.com> Message-ID: On 3/23/2015 2:02 AM, David Foster wrote: > * It's a bit vague how local variables should be typed in the presence of > multiple assignments. Consider: > > ``` > if x > 0: > root = sqrt(x) # type: int > else: > root = None > ``` root = sqrt(x) if x > 0 or None # type: Union(int, None) > * +1 for only allowing .pyi as the extension for stub files (and not > also .py). > Makes it very clear that they should only be interpreted as stubs. Agreed > * +1 for the argument order cast(x, T). This is consistent with (x: T) > elsewhere. cast x to T and is x instance of T, agreed. -- Terry Jan Reedy From mal at egenix.com Mon Mar 23 10:16:55 2015 From: mal at egenix.com (M.-A. Lemburg) Date: Mon, 23 Mar 2015 10:16:55 +0100 Subject: [Python-ideas] Geo coordinates conversion in stdlib In-Reply-To: References: <550F30FB.5010000@sdamon.com> <20150322223815.06082280@fsol> <21A308F0-5F2E-48BB-9F3F-4A3CFAB1D6DD@yahoo.com> Message-ID: <550FDA07.5020704@egenix.com> On 23.03.2015 07:08, anatoly techtonik wrote: > My story for that day is that I needed to process geodata from > Wikipedia to see the distribution over the Earth surface, so I thought > that if I know the Python it is not a problem. It appeared that the > problem is there, and the problem is recurring (last time I had GPS > points). Python docs explain a lot about core OS concepts, but Python > is not only a systems language. I'd say that it is now being replaced > with Go for all system stuff, and more attempts are made to use is as > a data processing language. But that is not possible if Python doesn't > have a basic concepts of data representation from the Real Life (RL) > (tm) in its stdlib. Well, sorry Anatoly that we haven't already solved all your problems for you ;-) If you are really interested in a solution to your problem, have a look at this large collection of tools for doing GIS in Python: https://pythongisresources.wordpress.com/categories/ More specifically, you're probably looking for one of these packages: https://pythongisresources.wordpress.com/projections-2/ It's all just a pip install away. BTW: If you expect the Python stdlib to make coffee, clean your house and drive your kids to school, you're probably looking at the wrong language - we only do batteries in the stdlib :-) -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Mar 23 2015) >>> Python Projects, Coaching and Consulting ... http://www.egenix.com/ >>> mxODBC Plone/Zope Database Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ 2015-03-12: Released mxODBC 3.3.2 ... http://egenix.com/go71 ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From random832 at fastmail.us Mon Mar 23 12:36:11 2015 From: random832 at fastmail.us (random832 at fastmail.us) Date: Mon, 23 Mar 2015 07:36:11 -0400 Subject: [Python-ideas] Geo coordinates conversion in stdlib In-Reply-To: References: <550F30FB.5010000@sdamon.com> <20150322223815.06082280@fsol> <21A308F0-5F2E-48BB-9F3F-4A3CFAB1D6DD@yahoo.com> Message-ID: <1427110571.3985053.243972942.3E7E5E29@webmail.messagingengine.com> On Sun, Mar 22, 2015, at 19:13, Chris Barker wrote: > On Sun, Mar 22, 2015 at 3:37 PM, Andrew Barnert < > abarnert at yahoo.com.dmarc.invalid> wrote: > > > If you can't figure out the complexity of multiplying and divmodding by > > 60 twice, you can borrow its code, or just wrap it and delegate. That seems > > trivial enough to do in any project that needs it. > > > I"m not suggesting this is an stdlib candiate, but it'snot quite that > simple. With FP precision issues you can end up with results like: > > 1? 60? 13? I'm not sure I believe this is true. Do you have a concrete example of an actual floating point value that will do this? From greg.ewing at canterbury.ac.nz Mon Mar 23 09:15:26 2015 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Mon, 23 Mar 2015 21:15:26 +1300 Subject: [Python-ideas] Geo coordinates conversion in stdlib In-Reply-To: References: <550F30FB.5010000@sdamon.com> <20150322223815.06082280@fsol> <21A308F0-5F2E-48BB-9F3F-4A3CFAB1D6DD@yahoo.com> Message-ID: <550FCB9E.3000709@canterbury.ac.nz> Chris Barker wrote: > With FP precision issues you can end up with results like: > > 1? 60? 13? Do you have an example that gives this result? -- Greg From taleinat at gmail.com Mon Mar 23 15:52:06 2015 From: taleinat at gmail.com (Tal Einat) Date: Mon, 23 Mar 2015 16:52:06 +0200 Subject: [Python-ideas] os.path.commonprefix: Yes that old chestnut. In-Reply-To: <20150321124732.0b47730e@fsol> References: <22b04160-2ff7-4c98-9d2c-97ac13b3bc25@googlegroups.com> <20150321124732.0b47730e@fsol> Message-ID: On Sat, Mar 21, 2015 at 1:47 PM, Antoine Pitrou wrote: > On Fri, 20 Mar 2015 21:41:03 -0700 (PDT) > Paddy3118 wrote: >> >> The fix seems clear: deprecate os.path.commonprefix whilst creating a true >> os.path.commonpath and str.commonprefix. The deprecated function should >> hang around and the standard libs modified to switch to the new function(s) > > +1 from me. +1 From p.f.moore at gmail.com Mon Mar 23 16:24:22 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Mon, 23 Mar 2015 15:24:22 +0000 Subject: [Python-ideas] os.path.commonprefix: Yes that old chestnut. In-Reply-To: References: <22b04160-2ff7-4c98-9d2c-97ac13b3bc25@googlegroups.com> <20150321124732.0b47730e@fsol> Message-ID: On 23 March 2015 at 14:52, Tal Einat wrote: > On Sat, Mar 21, 2015 at 1:47 PM, Antoine Pitrou wrote: >> On Fri, 20 Mar 2015 21:41:03 -0700 (PDT) >> Paddy3118 wrote: >>> >>> The fix seems clear: deprecate os.path.commonprefix whilst creating a true >>> os.path.commonpath and str.commonprefix. The deprecated function should >>> hang around and the standard libs modified to switch to the new function(s) >> >> +1 from me. > > +1 +1. Maybe adding a commonprefix operation to pathlib would be a good idea as well - using that would avoid the confusion between the deprecated os.path.commonprefix and os.path.commonpath... Paul From encukou at gmail.com Mon Mar 23 16:39:01 2015 From: encukou at gmail.com (Petr Viktorin) Date: Mon, 23 Mar 2015 16:39:01 +0100 Subject: [Python-ideas] A macro for easier rich comparisons In-Reply-To: <6D6DC4C3-BB82-49CE-8615-E07B1F048FA1@yahoo.com> References: <55095940.4010707@gmail.com> <55095AF3.6080201@egenix.com> <6D6DC4C3-BB82-49CE-8615-E07B1F048FA1@yahoo.com> Message-ID: On Fri, Mar 20, 2015 at 11:19 PM, Andrew Barnert wrote: > On Mar 20, 2015, at 7:24 AM, Petr Viktorin wrote: >> >> A sequence of comparisons is less efficient than a switch statement. >> Same with PyBool_FromLong vs. Py_RETURN_* >> Also, it might be too complex for a macro. >> These points could be solved by making it a function instead. > > Or by just making the macro a statement instead of a function; then the code can be exactly the same code you're abstracting out, and instead of using it like this: > > return PY_RICHCOMPARE(mytype_get_data(obj1), mytype_get_data(obj2), op); > > ... you use it like this: > > PY_RETURNRICHCOMPARE(mytype_get_data(obj1), mytype_get_data(obj2), op); Thanks Andrew. This makes it harder to do cleanup after the comparison (e.g. jump to a common block of Py_DECREFs, or do PyFPE_END_PROTECT in float code). But those should be rare enough, or can be easily worked around, so a "return" macro is still useful. From tritium-list at sdamon.com Mon Mar 23 17:44:43 2015 From: tritium-list at sdamon.com (Alexander Walters) Date: Mon, 23 Mar 2015 12:44:43 -0400 Subject: [Python-ideas] Geo coordinates conversion in stdlib In-Reply-To: References: <550F30FB.5010000@sdamon.com> <20150322223815.06082280@fsol> <21A308F0-5F2E-48BB-9F3F-4A3CFAB1D6DD@yahoo.com> Message-ID: <551042FB.4000203@sdamon.com> I could go on an on about my experience and what problems I solve, but... Nope. that is just flat wrong. Zipfiles are part of the import system, and everyone uses them all the time in python. Coordinate conversion is not a problem I have encountered, and a far as i can tell, not many people will. On 3/23/2015 02:08, anatoly techtonik wrote: > Handling geo coordinates in most used formats is as important as > handling .zip archives in stdlib. From alexander.belopolsky at gmail.com Mon Mar 23 18:21:36 2015 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Mon, 23 Mar 2015 13:21:36 -0400 Subject: [Python-ideas] Top 10 Python modules that need a redesign Was: Geo coordinates conversion in stdlib Message-ID: On Mon, Mar 23, 2015 at 2:08 AM, anatoly techtonik wrote: > > That's nice to know, but IIRC datetime is from the top 10 Python > modules that need a redesign. Things contained therein doesn't pass > human usability check, and are not used as a result. Where have you been when PEP 3108 was discussed? I have not seen any other list of Python modules that needed a redesign, so I cannot tell what's on your top ten list. Speaking of the datetime module, in what sense does it not "pass human usability check"? It does have a few quirks, for example I would rather see date accept a single argument in the constructor which may be a string, another date or a tuple, but I am not even sure this desire is shared by many other humans. It would be nice if datetime classes were named in CamelCase according to PEP 8 conventions, but again this is a very minor quirk. In my view, if anyone is to blame for the "human usability" of the datetime module, it would be Pope Gregory XIII, Benjamin Franklin and scores of unnamed astronomers who made modern timekeeping such a mess. -------------- next part -------------- An HTML attachment was scrubbed... URL: From tritium-list at sdamon.com Mon Mar 23 18:32:49 2015 From: tritium-list at sdamon.com (Alexander Walters) Date: Mon, 23 Mar 2015 13:32:49 -0400 Subject: [Python-ideas] Top 10 Python modules that need a redesign Was: Geo coordinates conversion in stdlib In-Reply-To: References: Message-ID: <55104E41.10609@sdamon.com> It works fine for me. It does not do EVERYTHING I need, but with dateutil and pytz, I am satisfied with the datetime infrastructure in python. Sure, I would like things changed in it, but its not like I expect the every module in the standard library to work exactly how I want. On 3/23/2015 13:21, Alexander Belopolsky wrote: > > Speaking of the datetime module, in what sense does it not "pass human > usability check"? It does have a few quirks, for example I would > rather see date accept a single argument in the constructor which may > be a string, another date or a tuple, but I am not even sure this > desire is shared by many other humans. It would be nice if datetime > classes were named in CamelCase according to PEP 8 conventions, but > again this is a very minor quirk. From tjreedy at udel.edu Mon Mar 23 19:59:55 2015 From: tjreedy at udel.edu (Terry Reedy) Date: Mon, 23 Mar 2015 14:59:55 -0400 Subject: [Python-ideas] Top 10 Python modules that need a redesign Was: Geo coordinates conversion in stdlib In-Reply-To: References: Message-ID: On 3/23/2015 1:21 PM, Alexander Belopolsky wrote: > > On Mon, Mar 23, 2015 at 2:08 AM, anatoly techtonik > > wrote: > > > > That's nice to know, but IIRC datetime is from the top 10 Python > > modules that need a redesign. Things contained therein doesn't pass > > human usability check, and are not used as a result. ... > in what sense does it not "pass human usability check"? ... Please don't feed the troll. He is known for opinionated digs like the above and historically, nothing good comes from trying to discuss them. -- Terry Jan Reedy From guido at python.org Mon Mar 23 20:55:37 2015 From: guido at python.org (Guido van Rossum) Date: Mon, 23 Mar 2015 12:55:37 -0700 Subject: [Python-ideas] PEP 484 (Type Hints) -- second draft In-Reply-To: <550FAC80.8040405@gmail.com> References: <550FAC80.8040405@gmail.com> Message-ID: On Sun, Mar 22, 2015 at 11:02 PM, David Foster wrote: > * It's a bit vague how local variables should be typed in the presence of > multiple assignments. Consider: > > ``` > if x > 0: > root = sqrt(x) # type: int > else: > root = None > ``` > > It seems that the type of `root` should be inferred as `Union[int, None]` > (AKA `Optional[int]`), but there's no discussion about this type of > inference in the PEP. > > Cases like this are the only way that the proposed blank `Undefined` > value makes sense to me, using an example like: > > ``` > root = Undefined # type: Optional[int] > if x > 0: > root = sqrt(x) # type: int > else: > root = None > ``` > Right. Another use case for Undefined is class variables (serving as instance variable initialization/declaration). If you feel strongly that this needs to be mentioned in the PEP, can you send a PR to github.com/ambv/typehinting ? > * +1 for only allowing .pyi as the extension for stub files (and not also > .py). > Makes it very clear that they should only be interpreted as stubs. > Perhaps. Though there are also downsides. > * -1 that `Optional[T1]` is unnecessary. It's conceptually more clean and > directly corresponds conceptually to types like `int?` and `?int` from > other languages. > Indeed, Optional[t] is much clearer to the human reader than Union[t, None]. > * +1 for the argument order cast(x, T). This is consistent with (x: T) > elsewhere. > I disagree on this. It goes against the argument order of casts in other languages, e.g. C, C++, Java, and even in Python -- you write int(x), not x(int). -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From rymg19 at gmail.com Mon Mar 23 21:13:46 2015 From: rymg19 at gmail.com (Ryan Gonzalez) Date: Mon, 23 Mar 2015 15:13:46 -0500 Subject: [Python-ideas] PEP 484 (Type Hints) -- second draft In-Reply-To: References: <550FAC80.8040405@gmail.com> Message-ID: On Mon, Mar 23, 2015 at 2:55 PM, Guido van Rossum wrote: > On Sun, Mar 22, 2015 at 11:02 PM, David Foster > wrote: > >> * It's a bit vague how local variables should be typed in the presence of >> multiple assignments. Consider: >> >> ``` >> if x > 0: >> root = sqrt(x) # type: int >> else: >> root = None >> ``` >> >> It seems that the type of `root` should be inferred as `Union[int, >> None]` >> (AKA `Optional[int]`), but there's no discussion about this type of >> inference in the PEP. >> >> Cases like this are the only way that the proposed blank `Undefined` >> value makes sense to me, using an example like: >> >> ``` >> root = Undefined # type: Optional[int] >> if x > 0: >> root = sqrt(x) # type: int >> else: >> root = None >> ``` >> > > Right. Another use case for Undefined is class variables (serving as > instance variable initialization/declaration). > > If you feel strongly that this needs to be mentioned in the PEP, can you > send a PR to github.com/ambv/typehinting ? > > >> * +1 for only allowing .pyi as the extension for stub files (and not also >> .py). >> Makes it very clear that they should only be interpreted as stubs. >> > > Perhaps. Though there are also downsides. > > >> * -1 that `Optional[T1]` is unnecessary. It's conceptually more clean and >> directly corresponds conceptually to types like `int?` and `?int` from >> other languages. >> > > Indeed, Optional[t] is much clearer to the human reader than Union[t, > None]. > > >> * +1 for the argument order cast(x, T). This is consistent with (x: T) >> elsewhere. >> > > I disagree on this. It goes against the argument order of casts in other > languages, e.g. C, C++, Java, and even in Python -- you write int(x), not > x(int). > I brought this up somewhere before, but I still think this would be great for casts: cast[T](x) Like C++'s named casts (static_cast(x), dynamic_cast(x), etc.). It gives the notion that T is a type, not some random variable, and that cast is NOT a normal function call. > > -- > --Guido van Rossum (python.org/~guido) > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- Ryan [ERROR]: Your autotools build scripts are 200 lines longer than your program. Something?s wrong. http://kirbyfan64.github.io/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Mon Mar 23 21:17:32 2015 From: guido at python.org (Guido van Rossum) Date: Mon, 23 Mar 2015 13:17:32 -0700 Subject: [Python-ideas] PEP 484 (Type Hints) -- second draft In-Reply-To: References: <550FAC80.8040405@gmail.com> Message-ID: Yeah, I've considered that, but it feels too punctuation-heavy (as does the C++ cast(x)). Also, we need cast() to be a fast no-op; cast[T](x) necessarily means calling two functions rather than one. On Mon, Mar 23, 2015 at 1:13 PM, Ryan Gonzalez wrote: > On Mon, Mar 23, 2015 at 2:55 PM, Guido van Rossum > wrote: > >> On Sun, Mar 22, 2015 at 11:02 PM, David Foster >> wrote: >> >>> * It's a bit vague how local variables should be typed in the presence of >>> multiple assignments. Consider: >>> >>> ``` >>> if x > 0: >>> root = sqrt(x) # type: int >>> else: >>> root = None >>> ``` >>> >>> It seems that the type of `root` should be inferred as `Union[int, >>> None]` >>> (AKA `Optional[int]`), but there's no discussion about this type of >>> inference in the PEP. >>> >>> Cases like this are the only way that the proposed blank `Undefined` >>> value makes sense to me, using an example like: >>> >>> ``` >>> root = Undefined # type: Optional[int] >>> if x > 0: >>> root = sqrt(x) # type: int >>> else: >>> root = None >>> ``` >>> >> >> Right. Another use case for Undefined is class variables (serving as >> instance variable initialization/declaration). >> >> If you feel strongly that this needs to be mentioned in the PEP, can you >> send a PR to github.com/ambv/typehinting ? >> >> >>> * +1 for only allowing .pyi as the extension for stub files (and not >>> also .py). >>> Makes it very clear that they should only be interpreted as stubs. >>> >> >> Perhaps. Though there are also downsides. >> >> >>> * -1 that `Optional[T1]` is unnecessary. It's conceptually more clean and >>> directly corresponds conceptually to types like `int?` and `?int` from >>> other languages. >>> >> >> Indeed, Optional[t] is much clearer to the human reader than Union[t, >> None]. >> >> >>> * +1 for the argument order cast(x, T). This is consistent with (x: T) >>> elsewhere. >>> >> >> I disagree on this. It goes against the argument order of casts in other >> languages, e.g. C, C++, Java, and even in Python -- you write int(x), not >> x(int). >> > > I brought this up somewhere before, but I still think this would be great > for casts: > > cast[T](x) > > Like C++'s named casts (static_cast(x), dynamic_cast(x), etc.). It > gives the notion that T is a type, not some random variable, and that cast > is NOT a normal function call. > > >> >> -- >> --Guido van Rossum (python.org/~guido) >> >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ >> > > > > -- > Ryan > [ERROR]: Your autotools build scripts are 200 lines longer than your > program. Something?s wrong. > http://kirbyfan64.github.io/ > > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Mon Mar 23 22:05:04 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Mon, 23 Mar 2015 21:05:04 +0000 Subject: [Python-ideas] PEP 484 (Type Hints) -- second draft In-Reply-To: References: <550FAC80.8040405@gmail.com> Message-ID: On 23 March 2015 at 19:55, Guido van Rossum wrote: >> * +1 for the argument order cast(x, T). This is consistent with (x: T) >> elsewhere. > > I disagree on this. It goes against the argument order of casts in other > languages, e.g. C, C++, Java, and even in Python -- you write int(x), not > x(int). However, in SQL, which is the only language I know of with an with an explicitly named cast function (unless you count C++'s static_cast(foo) which isn't normal function syntax), the syntax is CAST(foo AS type). So there's definitely a precedent (arguably stronger, depending on your background) for cast(x, T). Paul. From njs at pobox.com Mon Mar 23 22:17:20 2015 From: njs at pobox.com (Nathaniel Smith) Date: Mon, 23 Mar 2015 14:17:20 -0700 Subject: [Python-ideas] PEP 484 (Type Hints) -- second draft In-Reply-To: References: <550FAC80.8040405@gmail.com> Message-ID: On Mar 23, 2015 1:03 PM, "Guido van Rossum" wrote: > > On Sun, Mar 22, 2015 at 11:02 PM, David Foster wrote: >> >> * +1 for the argument order cast(x, T). This is consistent with (x: T) elsewhere. > > > I disagree on this. It goes against the argument order of casts in other languages, e.g. C, C++, Java, and even in Python -- you write int(x), not x(int). I don't have any strong opinion here, but I don't find the consistency argument convincing. In int(x), 'int' is the verb, and in English verbs come before undergoers ("I am inting the x"). In cast(...), though, cast is the verb, x remains the undergoer, and int is conceptualized as the destination (or something like that), and destinations go in prepositional clauses after the object. You'd say "I'm casting the x to int"; (cf "I'm throwing the ball to Sarah"). "I'm casting to int the x" is extremely weird. -n -------------- next part -------------- An HTML attachment was scrubbed... URL: From greg at krypto.org Mon Mar 23 22:33:48 2015 From: greg at krypto.org (Gregory P. Smith) Date: Mon, 23 Mar 2015 21:33:48 +0000 Subject: [Python-ideas] os.path.commonprefix: Yes that old chestnut. In-Reply-To: References: <22b04160-2ff7-4c98-9d2c-97ac13b3bc25@googlegroups.com> <20150321124732.0b47730e@fsol> Message-ID: +1 pathlib would be the appropriate place for the correctly behaving function to appear. os.path.commonprefix()'s behavior would be better off as a str and bytes method rather than a crazy function in the os.path library but I doubt anyone *really* wants to add more methods to those. On Mon, Mar 23, 2015 at 8:25 AM Paul Moore wrote: > On 23 March 2015 at 14:52, Tal Einat wrote: > > On Sat, Mar 21, 2015 at 1:47 PM, Antoine Pitrou > wrote: > >> On Fri, 20 Mar 2015 21:41:03 -0700 (PDT) > >> Paddy3118 wrote: > >>> > >>> The fix seems clear: deprecate os.path.commonprefix whilst creating a > true > >>> os.path.commonpath and str.commonprefix. The deprecated function should > >>> hang around and the standard libs modified to switch to the new > function(s) > >> > >> +1 from me. > > > > +1 > > +1. Maybe adding a commonprefix operation to pathlib would be a good > idea as well - using that would avoid the confusion between the > deprecated os.path.commonprefix and os.path.commonpath... > > Paul > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From cs at zip.com.au Mon Mar 23 22:41:45 2015 From: cs at zip.com.au (Cameron Simpson) Date: Tue, 24 Mar 2015 08:41:45 +1100 Subject: [Python-ideas] PEP 484 (Type Hints) -- second draft In-Reply-To: References: Message-ID: <20150323214145.GA94958@cskk.homeip.net> On 23Mar2015 04:55, Terry Reedy wrote: >On 3/23/2015 2:02 AM, David Foster wrote: >>* It's a bit vague how local variables should be typed in the presence of >> multiple assignments. Consider: >> >> ``` >> if x > 0: >> root = sqrt(x) # type: int >> else: >> root = None >> ``` > >root = sqrt(x) if x > 0 or None # type: Union(int, None) Should people need to reshape their code for this? In the example above it is small, though less readable already. But it scales very badly; what of a 4 way if etc? It moves from a bit less readable to a nightmare. Cheers, Cameron Simpson Do not cross the oncoming lanes of death that are California's highways. - SJ Mercury News From abarnert at yahoo.com Mon Mar 23 22:59:21 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Mon, 23 Mar 2015 14:59:21 -0700 Subject: [Python-ideas] PEP 484 (Type Hints) -- second draft In-Reply-To: <20150323214145.GA94958@cskk.homeip.net> References: <20150323214145.GA94958@cskk.homeip.net> Message-ID: <76D24940-328A-46A0-86D0-A073F8CB51D7@yahoo.com> On Mar 23, 2015, at 2:41 PM, Cameron Simpson wrote: > >> On 23Mar2015 04:55, Terry Reedy wrote: >>> On 3/23/2015 2:02 AM, David Foster wrote: >>> * It's a bit vague how local variables should be typed in the presence of >>> multiple assignments. Consider: >>> >>> ``` >>> if x > 0: >>> root = sqrt(x) # type: int >>> else: >>> root = None >>> ``` >> >> root = sqrt(x) if x > 0 or None # type: Union(int, None) > > Should people need to reshape their code for this? > In the example above it is small, though less readable already. If this is going to be the example we keep focusing on... Why would you have a custom int sqrt function whose return type is neither declared nor inferrable? I'd suspect that most variables that are initialized with a function call don't need a type hint, and if there are special cases where it comes up, they may be special enough that a real-life-plausible example is worth having rather than an example that's almost certainly unnecessary (or, in this case, an explicit type error, or code that type checks but only because you accidentally fooled the checker, because the last time I actually wrote, used, or wanted an int sqrt function it was in 6502 assembly...) > But it scales very badly; what of a 4 way if etc? It moves from a bit less readable to a nightmare. Agreed. In fact, even when what you have is an expression in the first place, Python makes it easy (and relatively cheap) to break it up into a sequence of statements (e.g., assignments to temporary variables), and that's more often a good idea than the other way around, so typing had better work nicely for the statement version of things (and Jukka and Guido seem to have worked hard to make that true in general). > > Cheers, > Cameron Simpson > > Do not cross the oncoming lanes of death that are California's highways. > - SJ Mercury News > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ From solipsis at pitrou.net Mon Mar 23 23:08:06 2015 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 23 Mar 2015 23:08:06 +0100 Subject: [Python-ideas] PEP 484 (Type Hints) -- second draft References: <550FAC80.8040405@gmail.com> Message-ID: <20150323230806.7a0653b9@fsol> On Mon, 23 Mar 2015 14:17:20 -0700 Nathaniel Smith wrote: > On Mar 23, 2015 1:03 PM, "Guido van Rossum" wrote: > > > > On Sun, Mar 22, 2015 at 11:02 PM, David Foster > wrote: > >> > >> * +1 for the argument order cast(x, T). This is consistent with (x: T) > elsewhere. > > > > > > I disagree on this. It goes against the argument order of casts in other > languages, e.g. C, C++, Java, and even in Python -- you write int(x), not > x(int). > > I don't have any strong opinion here, but I don't find the consistency > argument convincing. In int(x), 'int' is the verb, and in English verbs > come before undergoers ("I am inting the x"). In cast(...), though, cast is > the verb, x remains the undergoer, and int is conceptualized as the > destination (or something like that), and destinations go in prepositional > clauses after the object. You'd say "I'm casting the x to int"; (cf "I'm > throwing the ball to Sarah"). "I'm casting to int the x" is extremely weird. I agree with cast(x, T). It also nicely mirrors isinstance(x, T). Regards Antoine. From solipsis at pitrou.net Mon Mar 23 23:09:37 2015 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 23 Mar 2015 23:09:37 +0100 Subject: [Python-ideas] os.path.commonprefix: Yes that old chestnut. References: <22b04160-2ff7-4c98-9d2c-97ac13b3bc25@googlegroups.com> <20150321124732.0b47730e@fsol> Message-ID: <20150323230937.0af5ee32@fsol> On Mon, 23 Mar 2015 21:33:48 +0000 "Gregory P. Smith" wrote: > +1 pathlib would be the appropriate place for the correctly behaving > function to appear. Patches welcome :-) Regards Antoine. > > os.path.commonprefix()'s behavior would be better off as a str and bytes > method rather than a crazy function in the os.path library but I doubt > anyone *really* wants to add more methods to those. > > On Mon, Mar 23, 2015 at 8:25 AM Paul Moore wrote: > > > On 23 March 2015 at 14:52, Tal Einat wrote: > > > On Sat, Mar 21, 2015 at 1:47 PM, Antoine Pitrou > > wrote: > > >> On Fri, 20 Mar 2015 21:41:03 -0700 (PDT) > > >> Paddy3118 wrote: > > >>> > > >>> The fix seems clear: deprecate os.path.commonprefix whilst creating a > > true > > >>> os.path.commonpath and str.commonprefix. The deprecated function should > > >>> hang around and the standard libs modified to switch to the new > > function(s) > > >> > > >> +1 from me. > > > > > > +1 > > > > +1. Maybe adding a commonprefix operation to pathlib would be a good > > idea as well - using that would avoid the confusion between the > > deprecated os.path.commonprefix and os.path.commonpath... > > > > Paul > > _______________________________________________ > > Python-ideas mailing list > > Python-ideas at python.org > > https://mail.python.org/mailman/listinfo/python-ideas > > Code of Conduct: http://python.org/psf/codeofconduct/ > > > From chris.barker at noaa.gov Mon Mar 23 23:19:29 2015 From: chris.barker at noaa.gov (Chris Barker) Date: Mon, 23 Mar 2015 15:19:29 -0700 Subject: [Python-ideas] PEP 484 (Type Hints) -- second draft In-Reply-To: <20150323214145.GA94958@cskk.homeip.net> References: <20150323214145.GA94958@cskk.homeip.net> Message-ID: On Mon, Mar 23, 2015 at 2:41 PM, Cameron Simpson wrote: > But it scales very badly; what of a 4 way if etc? It moves from a bit less > readable to a nightmare. > in python one particular type or None is a very common idiom -- well worth special casing. Somethign that could take or return 4 different types is a far less usual case, we can be more verbose there. Note that Cython and Pyrex default to "or NOne", and you have to do func( int x not None) if you don't want to allow the None. That's because None as a not-set-yet value is so common. In this case, it's a mistake to make it the default, as Cython does not generate checks later for None, so it can get ugly if you don't specifically test for it in your code. None the less, it's very useful. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Mon Mar 23 23:35:23 2015 From: guido at python.org (Guido van Rossum) Date: Mon, 23 Mar 2015 15:35:23 -0700 Subject: [Python-ideas] PEP 484 (Type Hints) -- second draft In-Reply-To: References: <20150323214145.GA94958@cskk.homeip.net> Message-ID: On Mon, Mar 23, 2015 at 3:19 PM, Chris Barker wrote: > > On Mon, Mar 23, 2015 at 2:41 PM, Cameron Simpson wrote: > >> But it scales very badly; what of a 4 way if etc? It moves from a bit >> less readable to a nightmare. >> > > in python one particular type or None is a very common idiom -- well worth > special casing. Somethign that could take or return 4 different types is a > far less usual case, we can be more verbose there. > > Note that Cython and Pyrex default to "or NOne", and you have to do > > func( int x not None) > > if you don't want to allow the None. That's because None as a not-set-yet > value is so common. In this case, it's a mistake to make it the default, as > Cython does not generate checks later for None, so it can get ugly if you > don't specifically test for it in your code. None the less, it's very > useful. > There's a lot of discussion in the typehinting tracker and also in the mypy tracker about what to do with "or None" types. mypy currently doesn't complain about this because it's deemed too noisy (Jukka regularly tries to convert various real-world code bases to type hints, and he learns a lot from these). That said, about the specific example used here, the best way to deal with this currently is to put an annotation comment on the first line, like this: if a < 0: r = None # type: Optional[float] else: r = sqrt(a) There's currently one place in the PEP where Optional[t] is automatically inferred: an argument that's annotated and has None for a default: def foo(x: int = None): ... The actual type of x here is Optional[int]. Note that the PEP doesn't prescribe that the type of a variable becomes the union of the type assigned to in in separate branches of a conditional. This is intentional -- except for a few special cases (like discussed here) giving a variable a different type in different branches (if it is used after the branches join) is more likely a bug than desired. Experimenting with this in mypy is difficult because it explicitly silences complaints about None (see above); if you want to understand what it does in general it's better to experiment with two distinct non-None types, e.g. float and str. Using this approach I found that when you use an if-statement, you should put a #type: comment on the assignment in the first branch to silence it; when you use an if-expression, there's no way to shut it up! (This is exactly the opposite of the assumption that was made earlier in this thread.) PS. About the cast() argument order, I've opened a tracker issue for this: https://github.com/ambv/typehinting/issues/63 -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Mon Mar 23 23:48:47 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Mon, 23 Mar 2015 22:48:47 +0000 Subject: [Python-ideas] os.path.commonprefix: Yes that old chestnut. In-Reply-To: <20150323230937.0af5ee32@fsol> References: <22b04160-2ff7-4c98-9d2c-97ac13b3bc25@googlegroups.com> <20150321124732.0b47730e@fsol> <20150323230937.0af5ee32@fsol> Message-ID: On 23 March 2015 at 22:09, Antoine Pitrou wrote: > On Mon, 23 Mar 2015 21:33:48 +0000 > "Gregory P. Smith" wrote: >> +1 pathlib would be the appropriate place for the correctly behaving >> function to appear. > > Patches welcome :-) I'll see what I can do. Basically it's just def commonprefix(p1, p2): cp = [] for (pp1, pp2) in zip(p1.parts, p2.parts): if pp1 != pp2: break cp.append(pp1) return pathlib.Path(*cp) (extended to an arbitrary number of args) but getting the corner cases right is a little more tricky. If there is no common prefix, I guess returning None is correct - it's the nearest equivalent to os.path.commonprefix returning ''. The type of the return value should be a concrete type - probably type(p1)? Maybe this should be a method on Path objects - p.commonprefix(p1, p2, ...) although it seems a bit odd that the first argument (self) is treated differently in the signature. Anyone got any preferences? Paul From p.f.moore at gmail.com Mon Mar 23 23:56:17 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Mon, 23 Mar 2015 22:56:17 +0000 Subject: [Python-ideas] os.path.commonprefix: Yes that old chestnut. In-Reply-To: References: <22b04160-2ff7-4c98-9d2c-97ac13b3bc25@googlegroups.com> <20150321124732.0b47730e@fsol> <20150323230937.0af5ee32@fsol> Message-ID: On 23 March 2015 at 22:48, Paul Moore wrote: > I'll see what I can do. Basically it's just More specifically, as a pathlib.Path method: def commonprefix(self, *rest): cp = [] for p0, *ps in zip(self.parts, *[pp.parts for pp in rest]): if any(p0 != pi for pi in ps): break cp.append(p0) if not cp: return None return type(self)(*cp) Paul From guido at python.org Tue Mar 24 00:13:57 2015 From: guido at python.org (Guido van Rossum) Date: Mon, 23 Mar 2015 16:13:57 -0700 Subject: [Python-ideas] PEP 484 (Type Hints) -- second draft In-Reply-To: <20150321153731.GN31862@ando.pearwood.info> References: <20150321153731.GN31862@ando.pearwood.info> Message-ID: (This one took longer to respond to because there are so many nits. :-) On Sat, Mar 21, 2015 at 8:37 AM, Steven D'Aprano wrote: > On Fri, Mar 20, 2015 at 09:59:32AM -0700, Guido van Rossum wrote: > > > Union types > > ----------- > [...] > > As a shorthand for ``Union[T1, None]`` you can write ``Optional[T1]``; > > That only saves three characters. Is it worth it? > Yes (see other messages in the thread). > > An optional type is also automatically assumed when the default value is > > ``None``, for example:: > > > > def handle_employee(e: Employee = None): ... > > Should that apply to all default values or just None? E.g. if I have > > def spam(s: str = 23): ... > > should that be inferred as Union[str, int] or be flagged as a type > error? I think that we want a type error here, and it's only None that > actually should be treated as special. Perhaps that should be made > explicit in the PEP. > I only want this special treatment for None. The PEP seems to be pretty clear about this. > [...] > > For the purposes of type hinting, the type checker assumes ``__debug__`` > > is set to ``True``, in other words the ``-O`` command-line option is not > > used while type checking. > > I'm afraid I don't understand what you are trying to say here. I would > have expected that __debug__ and -O and the type checker would be > independent of each other. > Well, __debug__ tracks -O (__debug__ is false iff -O is given). But a type checker usually doesn't have the luxury to know whether the code is meant to run with -O or without it. So it assumes -O is *not* given, i.e. __debug__ is True. (You could try to type-check twice, once with -O and once without, but this idea doesn't really scale when you consider all the other flags that might be treated this way. And it's not really worth the trouble.) > [...] > > To mark portions of the program that should not be covered by type > > hinting, use the following: > > > > * a ``@no_type_check`` decorator on classes and functions > > > > * a ``# type: ignore`` comment on arbitrary lines > > > > .. FIXME: should we have a module-wide comment as well? > > I think so, if for no other reason than it will reduce the fear of some > people that type checks will be mandatory. > That fear ought to be reduced to zero by words in the PEP; have you got any suggestions? The mere presence of a module-wide comment to disable type checks might actually *increase* the fear that (absent such a comment) type checks might be mandatory. I could imagine other reasons for wanting a file-scoped directive, but I'm not sure -- there's a lot of discussion in https://github.com/ambv/typehinting/issues/35, maybe you can make sense of it. > > Type Hints on Local and Global Variables > > ======================================== > > > > No first-class syntax support for explicitly marking variables as being > > of a specific type is added by this PEP. To help with type inference in > > complex cases, a comment of the following format may be used:: > > > > x = [] # type: List[Employee] > > > > In the case where type information for a local variable is needed before > > it is declared, an ``Undefined`` placeholder might be used:: > > > > from typing import Undefined > > > > x = Undefined # type: List[Employee] > > y = Undefined(int) > > How is that better than just bringing forward the variable declaration? > > x = [] # type: List[Employee] > y = 0 > The actual initialization might have to happen later, separately in different branches of an if-statement; or this might be a class variable. Jukka gave some more reasons for having Undefined in https://github.com/ambv/typehinting/issues/20 > > Casts > > ===== > > > > Occasionally the type checker may need a different kind of hint: the > > programmer may know that an expression is of a more constrained type > > than the type checker infers. For example:: > > > > from typing import List > > > > def find_first_str(a: List[object]) -> str: > > index = next(i for i, x in enumerate(a) if isinstance(x, str)) > > # We only get here if there's at least one string in a > > return cast(str, a[index]) > > > > The type checker infers the type ``object`` for ``a[index]``, but we > > know that (if the code gets to that point) it must be a string. The > > ``cast(t, x)`` call tells the type checker that we are confident that > > the type of ``x`` is ``t``. > > Is the type checker supposed to unconditionally believe the cast, or > only if the cast is more constrained than the infered type (like str and > object, or bool and int)? > > E.g. if the type checker infers int, and the cast says list, I'm not > entirely sure I would trust the programmer more than the type checker. > > My feeling here is that some type checkers will unconditionally trust > the cast, and some will flag the mismatch, or offer a config option to > swap between the two, and that will be a feature for type checkers to > compete on. > It should unconditionally believe the cast. (Reference: https://github.com/ambv/typehinting/issues/15#issuecomment-69136820) > I'm also going to bike-shed the order of arguments. It seems to me that > we want to say: > > cast(x, T) # pronounced "cast x to T" > > rather than Yoda-speak "cast T x to we shall" *wink*. That also matches > the order of isinstance(obj, type) calls and makes it easier to > remember. > Seems you're not alone here. :-) I've opened https://github.com/ambv/typehinting/issues/63 > > At runtime a cast always returns the > > expression unchanged -- it does not check the type, and it does not > > convert or coerce the value. > > I'm a little concerned about cast() being a function. I know that it's a > do-nothing function, but there's still the overhead of the name lookup > and function call. It saddens me that giving a hint to the type checker > has a runtime cost, small as it is. > > (I know that *technically* annotations have a runtime cost too, but > they're once-only, at function definition time, not every time you call > the function.) > > Your point below that cast() can be used inside expressions is a valid > point, so there has to be a cast() function to support those cases, but > for the example given here where the cast occurs in a return statement, > wouldn't a type comment do? > > return some_expression # type: T > > hints that some_expression is to be treated as type T, regardless of > what was infered. > That has slightly different semantics -- while the type checker should unconditionally believe cast(), #type: comments are required to be consistent. In the example from the PEP (find_first_str()) the cas() is required, since the derived type is object. > > Casts differ from type comments (see the previous section). When > > using a type comment, the type checker should still verify that the > > inferred type is consistent with the stated type. When using a cast, > > the type checker trusts the programmer. Also, casts can be used in > > expressions, while type comments only apply to assignments. > > > > Stub Files > > ========== > [...] > > Stub files may use the ``.py`` extension or alternatively may use the > > ``.pyi`` extension. The latter makes it possible to maintain stub > > files in the same directory as the corresponding real module. > > I don't like anything that could cause confusion between stub files and > actual Python files. If we allow .py extension on stub files, I'm sure > there will be confusing errors where people somehow manage to get the > stub file imported instead of the actual module they want. > > Is there any advantage to allowing stub files use a .py extension? If > not, then don't allow it. > I'm tracking this at https://github.com/ambv/typehinting/issues/64 now. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From cs at zip.com.au Tue Mar 24 00:50:53 2015 From: cs at zip.com.au (Cameron Simpson) Date: Tue, 24 Mar 2015 10:50:53 +1100 Subject: [Python-ideas] PEP 484 (Type Hints) -- second draft In-Reply-To: References: Message-ID: <20150323235053.GA36178@cskk.homeip.net> On 23Mar2015 15:19, Chris Barker - NOAA Federal wrote: >On Mon, Mar 23, 2015 at 2:41 PM, Cameron Simpson wrote: >> But it scales very badly; what of a 4 way if etc? It moves from a bit less >> readable to a nightmare. > >in python one particular type or None is a very common idiom -- well worth >special casing. Somethign that could take or return 4 different types is a >far less usual case, we can be more verbose there. I did not mean 4 different types, I meant a four (or more) way decision which gave a (probably) single type or None. If "a if b else c" construction rapidly becomes ery hard to read/verify. Cheers, Cameron Simpson A crash reduces your expensive computer to a simple stone. - Haiku Error Messages http://www.salonmagazine.com/21st/chal/1998/02/10chal2.html From rosuav at gmail.com Tue Mar 24 00:56:46 2015 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 24 Mar 2015 10:56:46 +1100 Subject: [Python-ideas] PEP 484 (Type Hints) -- second draft In-Reply-To: References: <550FAC80.8040405@gmail.com> Message-ID: On Tue, Mar 24, 2015 at 8:05 AM, Paul Moore wrote: > On 23 March 2015 at 19:55, Guido van Rossum wrote: >>> * +1 for the argument order cast(x, T). This is consistent with (x: T) >>> elsewhere. >> >> I disagree on this. It goes against the argument order of casts in other >> languages, e.g. C, C++, Java, and even in Python -- you write int(x), not >> x(int). > > However, in SQL, which is the only language I know of with an with an > explicitly named cast function (unless you count C++'s > static_cast(foo) which isn't normal function syntax), the syntax is > CAST(foo AS type). So there's definitely a precedent (arguably > stronger, depending on your background) for cast(x, T). I would *hope* that some stronger argument is used than "SQL puts it this way". SQL has so many bizarre oddities of both syntax and semantics that it's seldom a good guideline. In this case, the isinstance() parallel is a better argument than the SQL cast, but it's a small point and I could easily be tipped back to preferring cast(T, x). ChrisA From mertz at gnosis.cx Tue Mar 24 01:03:47 2015 From: mertz at gnosis.cx (David Mertz) Date: Mon, 23 Mar 2015 17:03:47 -0700 Subject: [Python-ideas] PEP 484 (Type Hints) -- second draft In-Reply-To: <20150323230806.7a0653b9@fsol> References: <550FAC80.8040405@gmail.com> <20150323230806.7a0653b9@fsol> Message-ID: For what it's worth, 'cast(x, T)' also reads far more naturally to me. On Mon, Mar 23, 2015 at 3:08 PM, Antoine Pitrou wrote: > On Mon, 23 Mar 2015 14:17:20 -0700 > Nathaniel Smith wrote: > > > On Mar 23, 2015 1:03 PM, "Guido van Rossum" wrote: > > > > > > On Sun, Mar 22, 2015 at 11:02 PM, David Foster > > wrote: > > >> > > >> * +1 for the argument order cast(x, T). This is consistent with (x: T) > > elsewhere. > > > > > > > > > I disagree on this. It goes against the argument order of casts in > other > > languages, e.g. C, C++, Java, and even in Python -- you write int(x), not > > x(int). > > > > I don't have any strong opinion here, but I don't find the consistency > > argument convincing. In int(x), 'int' is the verb, and in English verbs > > come before undergoers ("I am inting the x"). In cast(...), though, cast > is > > the verb, x remains the undergoer, and int is conceptualized as the > > destination (or something like that), and destinations go in > prepositional > > clauses after the object. You'd say "I'm casting the x to int"; (cf "I'm > > throwing the ball to Sarah"). "I'm casting to int the x" is extremely > weird. > > I agree with cast(x, T). It also nicely mirrors isinstance(x, T). > > Regards > > Antoine. > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- Keeping medicines from the bloodstreams of the sick; food from the bellies of the hungry; books from the hands of the uneducated; technology from the underdeveloped; and putting advocates of freedom in prisons. Intellectual property is to the 21st century what the slave trade was to the 16th. -------------- next part -------------- An HTML attachment was scrubbed... URL: From cs at zip.com.au Tue Mar 24 01:11:02 2015 From: cs at zip.com.au (Cameron Simpson) Date: Tue, 24 Mar 2015 11:11:02 +1100 Subject: [Python-ideas] PEP 484 (Type Hints) -- second draft In-Reply-To: References: Message-ID: <20150324001102.GA77518@cskk.homeip.net> On 24Mar2015 10:56, Chris Angelico wrote: >On Tue, Mar 24, 2015 at 8:05 AM, Paul Moore wrote: >> On 23 March 2015 at 19:55, Guido van Rossum wrote: >>>> * +1 for the argument order cast(x, T). This is consistent with (x: T) >>>> elsewhere. >>> >>> I disagree on this. It goes against the argument order of casts in other >>> languages, e.g. C, C++, Java, and even in Python -- you write int(x), not >>> x(int). Well, in C you write (T)x, but I would expect that to be a syntactic requirement. Had it been framed as a function call I would not find cast(x, T) surprising at all. >> However, in SQL, which is the only language I know of with an with an >> explicitly named cast function [...] the syntax is >> CAST(foo AS type). [...] >> stronger, depending on your background) for cast(x, T). > >I would *hope* that some stronger argument is used than "SQL puts it >this way". Shrug. It is a consistency argument. And language like SQL and (gasp) COBOL strive in a sense for English prose friendliness. >In this case, the isinstance() parallel is a better argument than the >SQL cast, but it's a small point and I could easily be tipped back to >preferring cast(T, x). I would not. Leaving aside object method calls (fp.write(blah)), usually the thing acted upon or through is the first function agrument: write(fd, data, size) find(string, substring) heapsort(mylist) frob(widget, frobbificationness) I also like the instance() argument. For the record, I am +1 for cast(x, T). Cheers, Cameron Simpson Artificial intelligence won't make a micrometer out of a monkeywrench. - Rick Gordon From random832 at fastmail.us Tue Mar 24 02:54:29 2015 From: random832 at fastmail.us (random832 at fastmail.us) Date: Mon, 23 Mar 2015 21:54:29 -0400 Subject: [Python-ideas] os.path.commonprefix: Yes that old chestnut. In-Reply-To: References: <22b04160-2ff7-4c98-9d2c-97ac13b3bc25@googlegroups.com> <20150321124732.0b47730e@fsol> <20150323230937.0af5ee32@fsol> Message-ID: <1427162069.275798.244337286.403E06BC@webmail.messagingengine.com> On Mon, Mar 23, 2015, at 18:48, Paul Moore wrote: > The type of the return value should > be a concrete type - probably type(p1)? I'd argue it should be the common supertype, so a PurePosixPath if both are posix and one is pure, a Path or PurePath if one is windows and the other is posix. From greg.ewing at canterbury.ac.nz Tue Mar 24 03:58:46 2015 From: greg.ewing at canterbury.ac.nz (Greg) Date: Tue, 24 Mar 2015 15:58:46 +1300 Subject: [Python-ideas] PEP 484 (Type Hints) -- second draft In-Reply-To: <20150323230806.7a0653b9@fsol> References: <550FAC80.8040405@gmail.com> <20150323230806.7a0653b9@fsol> Message-ID: <5510D2E6.2030802@canterbury.ac.nz> On 24/03/2015 11:08 a.m., Antoine Pitrou wrote: > I agree with cast(x, T). It also nicely mirrors isinstance(x, T). I think that cast(T, x) is more readable when x is a long expression. It puts the type next to the thing using or being assigned the result, making it easier to visually verify that the types match up. -- Greg From greg.ewing at canterbury.ac.nz Tue Mar 24 03:59:41 2015 From: greg.ewing at canterbury.ac.nz (Greg) Date: Tue, 24 Mar 2015 15:59:41 +1300 Subject: [Python-ideas] PEP 484 (Type Hints) -- second draft In-Reply-To: References: <20150323214145.GA94958@cskk.homeip.net> Message-ID: <5510D31D.3080307@canterbury.ac.nz> On 24/03/2015 11:35 a.m., Guido van Rossum wrote: > when you use an > if-expression, there's no way to shut it up! Maybe an if-expression should assume the union type, then? -- Greg From p.f.moore at gmail.com Tue Mar 24 11:05:17 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 24 Mar 2015 10:05:17 +0000 Subject: [Python-ideas] os.path.commonprefix: Yes that old chestnut. In-Reply-To: <1427162069.275798.244337286.403E06BC@webmail.messagingengine.com> References: <22b04160-2ff7-4c98-9d2c-97ac13b3bc25@googlegroups.com> <20150321124732.0b47730e@fsol> <20150323230937.0af5ee32@fsol> <1427162069.275798.244337286.403E06BC@webmail.messagingengine.com> Message-ID: On 24 March 2015 at 01:54, wrote: > On Mon, Mar 23, 2015, at 18:48, Paul Moore wrote: >> The type of the return value should >> be a concrete type - probably type(p1)? > > I'd argue it should be the common supertype, so a PurePosixPath if both > are posix and one is pure, a Path or PurePath if one is windows and the > other is posix. That's not really possible in the face of the possibility that an argument could be a user-defined class, possibly not even a pathlib.Path subclass (given duck typing). That's a clear benefit of a Path method, actually - the type of the return value is easy to specify - it's the type of self. Actually, in many ways, this is really a list (sequence) method - common_prefix - applied to the "parts" property of a Path. It's a shame there isn't a sequence utils module in the stdlib... One thing my implementation doesn't (yet) handle is case sensitivity. The common prefix of WindowsPath('c:\\FOO\\bar') and WindowsPath('C:\\Foo\\BAR') should be WindowsPath('C:\\Foo'). But not for PosixPath. (And again, when they are mixed, which is silly but possible, what behaviour should apply? "Work like self" is the obvious answer if we have a method). Paul From probiner at gmail.com Tue Mar 24 11:31:12 2015 From: probiner at gmail.com (pedro santos) Date: Tue, 24 Mar 2015 10:31:12 +0000 Subject: [Python-ideas] 0-base and 1-base indexed iterables? Custom slicing rules? In-Reply-To: References: Message-ID: *Newbie slowly steps down from the table* I guess then 1-indexing in most cases the it's something relegated to fleshing, so that the index offset is done on the user level and not at the code level. Yes because no one will pick a gambling dice with face 0 :P I guess it's a matter of brain education... I did fell I needed to run a bunch of tests just to re-assure myself about the 1-indexed problem being solved with a 0-base representation. But I totally understand the arguments presented here as code consistency also makes it more malleable and somewhat more black-boxy. As someone that deals a lot with vectors with linear color workflows I know that what could be a clean work environment can be become a minefield of conversion routines that add an overhead to the brain when trying to operate Frankenstein operations into data flow setups. Thanks for the extra discussion, especially when indexing was addressed separately from slicing routines. Cheers On Sun, Mar 22, 2015 at 7:52 PM, David Mertz wrote: > The strongest proof that mixed index bases are a bad idea is that this is > a valid statement in VB6: > > Dim array(1 to 10) as Integer > > For all X, if VB6 allows X, then X is a bad idea. :-) > > -- > Keeping medicines from the bloodstreams of the sick; food > from the bellies of the hungry; books from the hands of the > uneducated; technology from the underdeveloped; and putting > advocates of freedom in prisons. Intellectual property is > to the 21st century what the slave trade was to the 16th. > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- *------------------------------[image: http://i153.photobucket.com/albums/s202/animatics/probiner-sig.gif]Pedro Alpiar?a dos Santos Animator 3DModeler Illustrator >> http://probiner.x10.mx/ * -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Tue Mar 24 12:56:25 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 24 Mar 2015 11:56:25 +0000 Subject: [Python-ideas] os.path.commonprefix: Yes that old chestnut. In-Reply-To: References: <22b04160-2ff7-4c98-9d2c-97ac13b3bc25@googlegroups.com> <20150321124732.0b47730e@fsol> Message-ID: On 23 March 2015 at 21:33, Gregory P. Smith wrote: > +1 pathlib would be the appropriate place for the correctly behaving > function to appear. OK, so here's a question. What actual use cases exist for a common_prefix function? The reason I ask is that I'm looking at some of the edge cases, and the obvious behaviour isn't particularly clear to me. For example, common_prefix('a/b/file.c', 'a/b/file.c'). The common prefix is obviously 'a/b/file.c' - but I can imagine people *actually* wanting the common *directory* containing both files. But taken literally, that's only possible if you check the filesystem, so it would no longer be a PurePath operation. And what about common_prefix('foo/bar', '../here/foo')? Or common_prefix('bar/baz', 'foo/../bar/baz')? Pathlib avoids collapsing .. because the meaning could change in the face of symlinks. I believe the same applies here. Maybe you need to call resolve() before doing the common prefix operation (but that gives an absolute path). With the above limitations, would a common_prefix function actually help typical use cases? In my experience, doing list operations on pathobj.parts is often simple enough that I don't need specialised functions like common_prefix... Getting the edge cases right is fiddly enough for common_prefix that a specialised function is a reasonable idea, but only if the "obvious" behaviour is clear. If there's a lot of conflicting possibilities, maybe a recipe in the docs would be a better option. Paul From p.f.moore at gmail.com Tue Mar 24 13:08:25 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 24 Mar 2015 12:08:25 +0000 Subject: [Python-ideas] pathlib.Path.parts - tuple of path objects rather than strings? Message-ID: One occasional problem I have with pathlib.Path.parts is that it returns a tuple of strings. For WindowsPath objects, paths are case insensitive but strings aren't. So we get the situation that >>> p1 = pathlib.PureWindowsPath('foo') >>> p2 = pathlib.PureWindowsPath('FOO') >>> p1 == p2 True >>> p1.parts == p2.parts False Making p1.parts return a tuple of objects of type p1.__class__ would address this: >>> [p1.__class__(p) for p in p1.parts] == [p2.__class__(p) for p in p2.parts] True Is this something that's worth changing (pathlib is still provisional, so the backward compatibility break would technically be allowed, even if it's not ideal)? Or would an extra property, say p.path_parts that returned a tuple of path objects be worth considering? As it is, use of path.parts is a potential source of portability bugs, where Unix users don't consider the case sensitivity issue. (Heck, as a Windows user, I almost missed the problem in my implementation of common_prefix, so it's not just Unix users...) Paul From markus at unterwaditzer.net Tue Mar 24 13:43:57 2015 From: markus at unterwaditzer.net (Markus Unterwaditzer) Date: Tue, 24 Mar 2015 13:43:57 +0100 Subject: [Python-ideas] pathlib.Path.parts - tuple of path objects rather than strings? In-Reply-To: References: Message-ID: <85174692-228C-4725-8972-3B81350FF136@unterwaditzer.net> No, I always forget. I've forwarded the message. I understand that the current behavior creates bugs too, however, I'd say the drawbacks I described are much bigger. The next option would be to create a separate type for path segments which has the appropriate comparison behavior: PathSegment, or more generically, PathString. On 24 March 2015 13:37:34 CET, Paul Moore wrote: >On 24 March 2015 at 12:25, Markus Unterwaditzer > wrote: >> I'd say treating each part as full-blown Path doesn't make sense from >a semantical POV. >> >> Simply mapping Path over the results is also a potential source of >bugs, since each part is now a relative path to the CWD, instead of the >previous segment. > >Fair point. Although I tend to view them as just "relative" - to what >depends on context (in this case, "to the preceding bits in the list" >is implied). But I agree, people may not be expecting that and so >could make mistakes. > >I still think using strings is equally a bug magnet, though. There's >actually no clean way of getting the correct case sensitivity >behaviour *apart* from making the parts into path objects. > >I don't think your suggestion of each part being a prefix of the next >would actually help in any of the places I've ever used .parts. > >Paul > >PS BTW, did you mean to send to just me rather than the list? I'd like >to know what other people think of your point. From markus at unterwaditzer.net Tue Mar 24 13:39:55 2015 From: markus at unterwaditzer.net (Markus Unterwaditzer) Date: Tue, 24 Mar 2015 13:39:55 +0100 Subject: [Python-ideas] pathlib.Path.parts - tuple of path objects rather than strings? In-Reply-To: References: Message-ID: <690A535E-4A78-4B0D-9BB5-153BFCF70757@unterwaditzer.net> Forgot to reply to list! -------- Original Message -------- From: Markus Unterwaditzer Sent: 24 March 2015 13:25:27 CET To: Paul Moore Subject: Re: [Python-ideas] pathlib.Path.parts - tuple of path objects rather than strings? I'd say treating each part as full-blown Path doesn't make sense from a semantical POV. Simply mapping Path over the results is also a potential source of bugs, since each part is now a relative path to the CWD, instead of the previous segment. I'd suggest that an additional property is added where: Path("/usr/bin/true").newproperty == (Path("/"), Path("/usr"), Path("/usr/bin"), Path("/usr/bin/true")) I.e. each part is a prefix of the next one. But I'm not sure where you'd use that. -- Markus On 24 March 2015 13:08:25 CET, Paul Moore wrote: >One occasional problem I have with pathlib.Path.parts is that it >returns a tuple of strings. For WindowsPath objects, paths are case >insensitive but strings aren't. So we get the situation that > >>>> p1 = pathlib.PureWindowsPath('foo') >>>> p2 = pathlib.PureWindowsPath('FOO') >>>> p1 == p2 >True >>>> p1.parts == p2.parts >False > >Making p1.parts return a tuple of objects of type p1.__class__ would >address this: > >>>> [p1.__class__(p) for p in p1.parts] == [p2.__class__(p) for p in >p2.parts] >True > >Is this something that's worth changing (pathlib is still provisional, >so the backward compatibility break would technically be allowed, even >if it's not ideal)? Or would an extra property, say p.path_parts that >returned a tuple of path objects be worth considering? > >As it is, use of path.parts is a potential source of portability bugs, >where Unix users don't consider the case sensitivity issue. (Heck, as >a Windows user, I almost missed the problem in my implementation of >common_prefix, so it's not just Unix users...) > >Paul >_______________________________________________ >Python-ideas mailing list >Python-ideas at python.org >https://mail.python.org/mailman/listinfo/python-ideas >Code of Conduct: http://python.org/psf/codeofconduct/ From random832 at fastmail.us Tue Mar 24 14:36:23 2015 From: random832 at fastmail.us (random832 at fastmail.us) Date: Tue, 24 Mar 2015 09:36:23 -0400 Subject: [Python-ideas] os.path.commonprefix: Yes that old chestnut. In-Reply-To: References: <22b04160-2ff7-4c98-9d2c-97ac13b3bc25@googlegroups.com> <20150321124732.0b47730e@fsol> <20150323230937.0af5ee32@fsol> <1427162069.275798.244337286.403E06BC@webmail.messagingengine.com> Message-ID: <1427204183.1820818.244539086.02E8E5DB@webmail.messagingengine.com> On Tue, Mar 24, 2015, at 06:05, Paul Moore wrote: > One thing my implementation doesn't (yet) handle is case sensitivity. > The common prefix of WindowsPath('c:\\FOO\\bar') and > WindowsPath('C:\\Foo\\BAR') should be WindowsPath('C:\\Foo'). But not > for PosixPath. (And again, when they are mixed, which is silly but > possible, what behaviour should apply? "Work like self" is the obvious > answer if we have a method). So, speaking of windows path oddities... what should be done for paths where one has a drive and the other does not? No common prefix? Interpret as the current drive? Do the former for pure paths and the latter for concrete paths? Pass in an option? Keeping in mind that people may write "portable" code but put in no effort to specifically support windows. > And again, when they are mixed, which is silly but possible, what behaviour should apply For concrete paths, using the actual names of the files if they exist would be another option. From p.f.moore at gmail.com Tue Mar 24 14:37:13 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 24 Mar 2015 13:37:13 +0000 Subject: [Python-ideas] Fwd: pathlib.Path.parts - tuple of path objects rather than strings? In-Reply-To: References: Message-ID: For completeness, here's my reply that got sent off-list... On 24 March 2015 at 12:25, Markus Unterwaditzer wrote: > I'd say treating each part as full-blown Path doesn't make sense from a semantical POV. > > Simply mapping Path over the results is also a potential source of bugs, since each part is now a relative path to the CWD, instead of the previous segment. Fair point. Although I tend to view them as just "relative" - to what depends on context (in this case, "to the preceding bits in the list" is implied). But I agree, people may not be expecting that and so could make mistakes. I still think using strings is equally a bug magnet, though. There's actually no clean way of getting the correct case sensitivity behaviour *apart* from making the parts into path objects. I don't think your suggestion of each part being a prefix of the next would actually help in any of the places I've ever used .parts. Paul PS BTW, did you mean to send to just me rather than the list? I'd like to know what other people think of your point. From random832 at fastmail.us Tue Mar 24 14:41:51 2015 From: random832 at fastmail.us (random832 at fastmail.us) Date: Tue, 24 Mar 2015 09:41:51 -0400 Subject: [Python-ideas] os.path.commonprefix: Yes that old chestnut. In-Reply-To: References: <22b04160-2ff7-4c98-9d2c-97ac13b3bc25@googlegroups.com> <20150321124732.0b47730e@fsol> Message-ID: <1427204511.1822328.244542110.44A2A94D@webmail.messagingengine.com> On Tue, Mar 24, 2015, at 07:56, Paul Moore wrote: > For example, common_prefix('a/b/file.c', 'a/b/file.c'). The common > prefix is obviously 'a/b/file.c' - but I can imagine people *actually* > wanting the common *directory* containing both files. But taken > literally, that's only possible if you check the filesystem, so it > would no longer be a PurePath operation. Or you could _always_ reject the last component. That is, even if "file.c" is a directory, return the directory it is in rather than itself. Does parts differentiate between "a/b/c" and "a/b/c/"? There are certainly contexts where real filesystems differentiate between them (symlinks, for example). From p.f.moore at gmail.com Tue Mar 24 14:43:44 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 24 Mar 2015 13:43:44 +0000 Subject: [Python-ideas] os.path.commonprefix: Yes that old chestnut. In-Reply-To: <1427204183.1820818.244539086.02E8E5DB@webmail.messagingengine.com> References: <22b04160-2ff7-4c98-9d2c-97ac13b3bc25@googlegroups.com> <20150321124732.0b47730e@fsol> <20150323230937.0af5ee32@fsol> <1427162069.275798.244337286.403E06BC@webmail.messagingengine.com> <1427204183.1820818.244539086.02E8E5DB@webmail.messagingengine.com> Message-ID: On 24 March 2015 at 13:36, wrote: > On Tue, Mar 24, 2015, at 06:05, Paul Moore wrote: >> One thing my implementation doesn't (yet) handle is case sensitivity. >> The common prefix of WindowsPath('c:\\FOO\\bar') and >> WindowsPath('C:\\Foo\\BAR') should be WindowsPath('C:\\Foo'). But not >> for PosixPath. (And again, when they are mixed, which is silly but >> possible, what behaviour should apply? "Work like self" is the obvious >> answer if we have a method). > > So, speaking of windows path oddities... what should be done for paths > where one has a drive and the other does not? No common prefix? > Interpret as the current drive? Do the former for pure paths and the > latter for concrete paths? Pass in an option? It's worth pointing out that all of these edge cases only occur for relative paths. It's quite possible that the only actual use cases would be completely fine if the operation was only defined on absolute paths. My instinct says you do this *purely* based on the common prefix of path.parts. The behaviour is then easy to define, and if it's not precisely what someone wants they can convert the paths to absolute and do the operations with that. I'd have to see real use cases to justify anything else. > Keeping in mind that people may write "portable" code but put in no > effort to specifically support windows. Precisely :-) >> And again, when they are mixed, which is silly but possible, what behaviour should apply > > For concrete paths, using the actual names of the files if they exist > would be another option. You can't have mixed concrete paths, you can only ever instantiate one of PosixPath or WindowsPath on a given system, AIUI. Paul From taleinat at gmail.com Tue Mar 24 14:44:39 2015 From: taleinat at gmail.com (Tal Einat) Date: Tue, 24 Mar 2015 15:44:39 +0200 Subject: [Python-ideas] pathlib.Path.parts - tuple of path objects rather than strings? In-Reply-To: <690A535E-4A78-4B0D-9BB5-153BFCF70757@unterwaditzer.net> References: <690A535E-4A78-4B0D-9BB5-153BFCF70757@unterwaditzer.net> Message-ID: On Tue, Mar 24, 2015 at 2:39 PM, Markus Unterwaditzer wrote: > > I'd suggest that an additional property is added where: > > Path("/usr/bin/true").newproperty == (Path("/"), > Path("/usr"), > Path("/usr/bin"), > Path("/usr/bin/true")) > > I.e. each part is a prefix of the next one. Path.parents already exists: Python 3.4.2 (default, Feb 23 2015, 21:16:28) [GCC 4.2.1 Compatible Apple LLVM 5.1 (clang-503.0.40)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> import pathlib >>> list(pathlib.Path("/usr/bin/true").parents) [PosixPath('/usr/bin'), PosixPath('/usr'), PosixPath('/')] - Tal Einat From p.f.moore at gmail.com Tue Mar 24 14:45:37 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 24 Mar 2015 13:45:37 +0000 Subject: [Python-ideas] os.path.commonprefix: Yes that old chestnut. In-Reply-To: <1427204511.1822328.244542110.44A2A94D@webmail.messagingengine.com> References: <22b04160-2ff7-4c98-9d2c-97ac13b3bc25@googlegroups.com> <20150321124732.0b47730e@fsol> <1427204511.1822328.244542110.44A2A94D@webmail.messagingengine.com> Message-ID: On 24 March 2015 at 13:41, wrote: > On Tue, Mar 24, 2015, at 07:56, Paul Moore wrote: >> For example, common_prefix('a/b/file.c', 'a/b/file.c'). The common >> prefix is obviously 'a/b/file.c' - but I can imagine people *actually* >> wanting the common *directory* containing both files. But taken >> literally, that's only possible if you check the filesystem, so it >> would no longer be a PurePath operation. > > Or you could _always_ reject the last component. That is, even if > "file.c" is a directory, return the directory it is in rather than > itself. I'm pretty sure there would be cases where that's not right. For example, commonprefix('a/b', 'a/b/file.c') would be 'a'. That seems wrong. > Does parts differentiate between "a/b/c" and "a/b/c/"? There are > certainly contexts where real filesystems differentiate between them > (symlinks, for example). No, it doesn't: >>> pathlib.Path('a/b/').parts ('a', 'b') >>> pathlib.Path('a/b').parts ('a', 'b') Paul From abarnert at yahoo.com Tue Mar 24 14:44:16 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Tue, 24 Mar 2015 06:44:16 -0700 Subject: [Python-ideas] os.path.commonprefix: Yes that old chestnut. In-Reply-To: References: <22b04160-2ff7-4c98-9d2c-97ac13b3bc25@googlegroups.com> <20150321124732.0b47730e@fsol> <20150323230937.0af5ee32@fsol> <1427162069.275798.244337286.403E06BC@webmail.messagingengine.com> Message-ID: <4458B94D-D294-42BC-BF7A-BCD90E8A8A0B@yahoo.com> On Mar 24, 2015, at 3:05 AM, Paul Moore wrote: > >> On 24 March 2015 at 01:54, wrote: >>> On Mon, Mar 23, 2015, at 18:48, Paul Moore wrote: >>> The type of the return value should >>> be a concrete type - probably type(p1)? >> >> I'd argue it should be the common supertype, so a PurePosixPath if both >> are posix and one is pure, a Path or PurePath if one is windows and the >> other is posix. > > That's not really possible in the face of the possibility that an > argument could be a user-defined class, possibly not even a > pathlib.Path subclass (given duck typing). That's a clear benefit of a > Path method, actually - the type of the return value is easy to > specify - it's the type of self. > > Actually, in many ways, this is really a list (sequence) method - > common_prefix - applied to the "parts" property of a Path. It's a > shame there isn't a sequence utils module in the stdlib... That's a good point. But do you really care that the result is a list (actually, isn't parts a tuple, not a list?), or just that it's some kind of iterable--or, even more generally, something you can make a Path object out of? Because there _is_ an iterable utils module in the stdlib, and I think the implementation is simpler if you think of it that way too: def common_prefix(x: Iterable[X], y: Iterable[X]) -> Iterator[X]: for a, b in zip(x, y): if a != b: return yield a (Or, if you prefer, implement it as a chain of zip, takewhile, and map(itemgetter) then yield from the result.) If you as a user want to turn that back into a tuple, you can, but normally you're just going to want to join them back up into a Path (or a type(p1)) without bothering with that. > One thing my implementation doesn't (yet) handle is case sensitivity. > The common prefix of WindowsPath('c:\\FOO\\bar') and > WindowsPath('C:\\Foo\\BAR') should be WindowsPath('C:\\Foo'). Not 'c:\\FOO'? I'd expect the left one to win--especially if it's a method, so the left one is self. > But not > for PosixPath. (And again, when they are mixed, which is silly but > possible, what behaviour should apply? "Work like self" is the obvious > answer if we have a method). Needless to say, an itertools (or "sequencetools") function that you call on parts does nothing to either help or hinder this problem. But it does seem to lend itself better to approaches where parts holds some new FooPathComponent type, or maybe a str on POSIX but a new CaseInsensitiveStr on Windows. From p.f.moore at gmail.com Tue Mar 24 14:51:30 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 24 Mar 2015 13:51:30 +0000 Subject: [Python-ideas] os.path.commonprefix: Yes that old chestnut. In-Reply-To: <4458B94D-D294-42BC-BF7A-BCD90E8A8A0B@yahoo.com> References: <22b04160-2ff7-4c98-9d2c-97ac13b3bc25@googlegroups.com> <20150321124732.0b47730e@fsol> <20150323230937.0af5ee32@fsol> <1427162069.275798.244337286.403E06BC@webmail.messagingengine.com> <4458B94D-D294-42BC-BF7A-BCD90E8A8A0B@yahoo.com> Message-ID: On 24 March 2015 at 13:44, Andrew Barnert wrote: >> Actually, in many ways, this is really a list (sequence) method - >> common_prefix - applied to the "parts" property of a Path. It's a >> shame there isn't a sequence utils module in the stdlib... > > That's a good point. But do you really care that the result is a list (actually, isn't parts a tuple, not a list?), or just that it's some kind of iterable--or, even more generally, something you can make a Path object out of? Because there _is_ an iterable utils module in the stdlib, and I think the implementation is simpler if you think of it that way too: > > def common_prefix(x: Iterable[X], y: Iterable[X]) -> Iterator[X]: > for a, b in zip(x, y): > if a != b: return > yield a > > (Or, if you prefer, implement it as a chain of zip, takewhile, and map(itemgetter) then yield from the result.) > > If you as a user want to turn that back into a tuple, you can, but normally you're just going to want to join them back up into a Path (or a type(p1)) without bothering with that. I was thinking that there might be a reason it wouldn't work for arbitrary iterators, so you'd need at least a Sequence. But you're right, an itertool is sufficient. Although given the itertools focus on building blocks, it may end up being simply a recipe. >> One thing my implementation doesn't (yet) handle is case sensitivity. >> The common prefix of WindowsPath('c:\\FOO\\bar') and >> WindowsPath('C:\\Foo\\BAR') should be WindowsPath('C:\\Foo'). > > Not 'c:\\FOO'? I'd expect the left one to win--especially if it's a method, so the left one is self. Technically, WindowsPath('C:\\FOO') and WindowsPath('C:\\Foo') are the same, so I stand by what I said :-) But yeah, the natural implementation would give you the relevant part of self. >> But not >> for PosixPath. (And again, when they are mixed, which is silly but >> possible, what behaviour should apply? "Work like self" is the obvious >> answer if we have a method). > > Needless to say, an itertools (or "sequencetools") function that you call on parts does nothing to either help or hinder this problem. But it does seem to lend itself better to approaches where parts holds some new FooPathComponent type, or maybe a str on POSIX but a new CaseInsensitiveStr on Windows. (p.__class__(pp) for pp in p.parts) (See the thread about parts containing strings - technically using Path objects is dodgy, but as long as you don't leak the working values out of your function it's perfectly adequate). Paul From abarnert at yahoo.com Tue Mar 24 14:55:18 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Tue, 24 Mar 2015 06:55:18 -0700 Subject: [Python-ideas] os.path.commonprefix: Yes that old chestnut. In-Reply-To: References: <22b04160-2ff7-4c98-9d2c-97ac13b3bc25@googlegroups.com> <20150321124732.0b47730e@fsol> Message-ID: <239E08B3-8FD0-4744-95E7-691900325A1D@yahoo.com> On Mar 24, 2015, at 4:56 AM, Paul Moore wrote: > >> On 23 March 2015 at 21:33, Gregory P. Smith wrote: >> +1 pathlib would be the appropriate place for the correctly behaving >> function to appear. > > OK, so here's a question. What actual use cases exist for a > common_prefix function? The reason I ask is that I'm looking at some > of the edge cases, and the obvious behaviour isn't particularly clear > to me. > > For example, common_prefix('a/b/file.c', 'a/b/file.c'). The common > prefix is obviously 'a/b/file.c' - but I can imagine people *actually* > wanting the common *directory* containing both files. But taken > literally, that's only possible if you check the filesystem, so it > would no longer be a PurePath operation. The traditional way to handle this is that the basename (the part after the last '/') is assumed to be a file (if you don't want that, include the trailing slash). POSIX even defines the technical term "path prefix" to mean everything up to the last slash, so something called a "common path prefix" sounds like it should be the common prefix of the path prefixes, right? Except that not command and function in POSIX works this way, requiring you to memorize or look up the man page to see what someone chose as "obvious" back in the 1970s.... At any rate, we probably don't need to figure this out from first principles; I'm pretty sure some subset of {Java, Boost, Cocoa, .NET, JUCE, one overwhelming popular CPAN library, etc.} have already come up with an answer, and if most of them agree, we probably want to follow suit (even if it seems silly). > And what about common_prefix('foo/bar', '../here/foo')? Or > common_prefix('bar/baz', 'foo/../bar/baz')? Pathlib avoids collapsing > .. because the meaning could change in the face of symlinks. I believe > the same applies here. Maybe you need to call resolve() before doing > the common prefix operation (but that gives an absolute path). > > With the above limitations, would a common_prefix function actually > help typical use cases? In my experience, doing list operations on > pathobj.parts is often simple enough that I don't need specialised > functions like common_prefix... > > Getting the edge cases right is fiddly enough for common_prefix that a > specialised function is a reasonable idea, but only if the "obvious" > behaviour is clear. If there's a lot of conflicting possibilities, > maybe a recipe in the docs would be a better option. > > Paul > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ From abarnert at yahoo.com Tue Mar 24 15:07:30 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Tue, 24 Mar 2015 07:07:30 -0700 Subject: [Python-ideas] os.path.commonprefix: Yes that old chestnut. In-Reply-To: References: <22b04160-2ff7-4c98-9d2c-97ac13b3bc25@googlegroups.com> <20150321124732.0b47730e@fsol> <20150323230937.0af5ee32@fsol> <1427162069.275798.244337286.403E06BC@webmail.messagingengine.com> <4458B94D-D294-42BC-BF7A-BCD90E8A8A0B@yahoo.com> Message-ID: On Mar 24, 2015, at 6:51 AM, Paul Moore wrote: > > On 24 March 2015 at 13:44, Andrew Barnert wrote: >>> Actually, in many ways, this is really a list (sequence) method - >>> common_prefix - applied to the "parts" property of a Path. It's a >>> shame there isn't a sequence utils module in the stdlib... >> >> That's a good point. But do you really care that the result is a list (actually, isn't parts a tuple, not a list?), or just that it's some kind of iterable--or, even more generally, something you can make a Path object out of? Because there _is_ an iterable utils module in the stdlib, and I think the implementation is simpler if you think of it that way too: >> >> def common_prefix(x: Iterable[X], y: Iterable[X]) -> Iterator[X]: >> for a, b in zip(x, y): >> if a != b: return >> yield a >> >> (Or, if you prefer, implement it as a chain of zip, takewhile, and map(itemgetter) then yield from the result.) >> >> If you as a user want to turn that back into a tuple, you can, but normally you're just going to want to join them back up into a Path (or a type(p1)) without bothering with that. > > I was thinking that there might be a reason it wouldn't work for > arbitrary iterators, so you'd need at least a Sequence. But you're > right, an itertool is sufficient. Although given the itertools focus > on building blocks, it may end up being simply a recipe. > >>> One thing my implementation doesn't (yet) handle is case sensitivity. >>> The common prefix of WindowsPath('c:\\FOO\\bar') and >>> WindowsPath('C:\\Foo\\BAR') should be WindowsPath('C:\\Foo'). >> >> Not 'c:\\FOO'? I'd expect the left one to win--especially if it's a method, so the left one is self. > > Technically, WindowsPath('C:\\FOO') and WindowsPath('C:\\Foo') are the > same, so I stand by what I said :-) But yeah, the natural > implementation would give you the relevant part of self. > >>> But not >>> for PosixPath. (And again, when they are mixed, which is silly but >>> possible, what behaviour should apply? "Work like self" is the obvious >>> answer if we have a method). >> >> Needless to say, an itertools (or "sequencetools") function that you call on parts does nothing to either help or hinder this problem. But it does seem to lend itself better to approaches where parts holds some new FooPathComponent type, or maybe a str on POSIX but a new CaseInsensitiveStr on Windows. > > (p.__class__(pp) for pp in p.parts) Sure, but then your whole expression looks something like: p1.__class__(*more_itertools.common_prefix( (p1.__class__(pp) for pp in p1.parts), (p2.__class__(pp) for pp in p2.parts))) Which doesn't read quite as nicely as "just call an itertools function on the parts and construct a Path from them" sounds like it should. Which implies that you'd probably want at least a recipe in the pathlib docs that referenced the recipe in the itertools docs or something. And that many people who aren't on Windows just wouldn't bother and would write something non-portable until they got a complaint from a Windows user and found it worth investigating... (While we're at it: most POSIX OS's can handle both case-sensitive and case-insensitive filesystems, and at least some OS X functions take that into account, although that may not be true at the BSD level, only at the POSIX level. For that matter, doesn't the HFS+ filesystem also consider two paths equal if they have the same NFKD, even if they have different code points? But I guess if I'm remembering right, this would be no more or less broken than any other use of PosixPath on Mac, so it's not worth worrying about here, right?) > (See the thread about parts containing strings - technically using > Path objects is dodgy, but as long as you don't leak the working > values out of your function it's perfectly adequate). Yeah, I agree that it's safe here even though it isn't safe in general. From p.f.moore at gmail.com Tue Mar 24 15:16:58 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 24 Mar 2015 14:16:58 +0000 Subject: [Python-ideas] os.path.commonprefix: Yes that old chestnut. In-Reply-To: References: <22b04160-2ff7-4c98-9d2c-97ac13b3bc25@googlegroups.com> <20150321124732.0b47730e@fsol> <20150323230937.0af5ee32@fsol> <1427162069.275798.244337286.403E06BC@webmail.messagingengine.com> <4458B94D-D294-42BC-BF7A-BCD90E8A8A0B@yahoo.com> Message-ID: On 24 March 2015 at 14:07, Andrew Barnert wrote: >>> Needless to say, an itertools (or "sequencetools") function that you call on parts does nothing to either help or hinder this problem. But it does seem to lend itself better to approaches where parts holds some new FooPathComponent type, or maybe a str on POSIX but a new CaseInsensitiveStr on Windows. >> >> (p.__class__(pp) for pp in p.parts) > > Sure, but then your whole expression looks something like: > > p1.__class__(*more_itertools.common_prefix( > (p1.__class__(pp) for pp in p1.parts), > (p2.__class__(pp) for pp in p2.parts))) > > Which doesn't read quite as nicely as "just call an itertools function on the parts and construct a Path from them" sounds like it should. Agreed, absolutely :-) > Which implies that you'd probably want at least a recipe in the pathlib docs that referenced the recipe in the itertools docs or something. > > And that many people who aren't on Windows just wouldn't bother and would write something non-portable until they got a complaint from a Windows user and found it worth investigating... I still think it's worth considering as a path object method, for convenience. I just think the *semantics* should be that of the equivalent list operation, because that's easily understandable. But to an extent, that's the trap that os.path.commonprefix fell into (although to a much worse level...) Hence my concern that we find some real use cases for the operation, so that we can ensure that the list semantics match what people actually want the operation to do. > (While we're at it: most POSIX OS's can handle both case-sensitive and case-insensitive filesystems, and at least some OS X functions take that into account, although that may not be true at the BSD level, only at the POSIX level. For that matter, doesn't the HFS+ filesystem also consider two paths equal if they have the same NFKD, even if they have different code points? But I guess if I'm remembering right, this would be no more or less broken than any other use of PosixPath on Mac, so it's not worth worrying about here, right?) This (and the decision to treat a/b/ and a/b as the same) are decisions that have already been made, for better or worse, by pathlib, and I have no intention of getting sucked into them here. IMO, a common_prefix method on pathlib objects should follow pathlib semantics, and can easily do that by being built on top of existing pathlib operations and uncontroversial list operations. That's the only approach that I think makes sense - and the only remaining question is whether it meets a real-world requirement, or whether it will end up being an oddity that no-one ever uses because it doesn't *quite* do what they want. Paul From abarnert at yahoo.com Tue Mar 24 15:44:40 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Tue, 24 Mar 2015 07:44:40 -0700 Subject: [Python-ideas] os.path.commonprefix: Yes that old chestnut. In-Reply-To: <239E08B3-8FD0-4744-95E7-691900325A1D@yahoo.com> References: <22b04160-2ff7-4c98-9d2c-97ac13b3bc25@googlegroups.com> <20150321124732.0b47730e@fsol> <239E08B3-8FD0-4744-95E7-691900325A1D@yahoo.com> Message-ID: <850BA2F2-AF08-4DE8-8661-ABF9A66A2D3A@yahoo.com> On Mar 24, 2015, at 6:55 AM, Andrew Barnert wrote: > >> On Mar 24, 2015, at 4:56 AM, Paul Moore wrote: >> >>> On 23 March 2015 at 21:33, Gregory P. Smith wrote: >>> +1 pathlib would be the appropriate place for the correctly behaving >>> function to appear. >> >> OK, so here's a question. What actual use cases exist for a >> common_prefix function? The reason I ask is that I'm looking at some >> of the edge cases, and the obvious behaviour isn't particularly clear >> to me. >> >> For example, common_prefix('a/b/file.c', 'a/b/file.c'). The common >> prefix is obviously 'a/b/file.c' - but I can imagine people *actually* >> wanting the common *directory* containing both files. But taken >> literally, that's only possible if you check the filesystem, so it >> would no longer be a PurePath operation. > > The traditional way to handle this is that the basename (the part after the last '/') is assumed to be a file (if you don't want that, include the trailing slash). POSIX even defines the technical term "path prefix" to mean everything up to the last slash, so something called a "common path prefix" sounds like it should be the common prefix of the path prefixes, right? Except that not command and function in POSIX works this way, requiring you to memorize or look up the man page to see what someone chose as "obvious" back in the 1970s.... Sorry, forgot to fill in the cite. See 3.2.69 at http://pubs.opengroup.org/stage7tc1/basedefs/V1_chap03.html for the 2008 definition of "path prefix". > At any rate, we probably don't need to figure this out from first principles; I'm pretty sure some subset of {Java, Boost, Cocoa, .NET, JUCE, one overwhelming popular CPAN library, etc.} have already come up with an answer, and if most of them agree, we probably want to follow suit (even if it seems silly). From a quick search, it looks like many other languages don't define a common prefix path method/function, but many do define a generic iterable common-prefix function (or a first-mismatch function and random-access iterables so you can easily build one trivially, as in C++). This implies that the obvious solution in most languages will include the basename, not skip it. And it looks like http://rosettacode.org/wiki/Find_common_directory_path agrees with that. From my other reply, looking over the functions used in some of the rosettacode examples, it looks like the generic iterable function, when it exists, and the language makes it feasible, often handles an arbitrary number of arguments, not just two. Which makes sense, now that I think about it. So: def common_prefix(*iterables): for first, *rest in zip(*iterables): if any(first != part for part in rest): return yield first And of course to wrap it up: def common_path_prefix(*paths): return paths[0].__class__( *(common_prefix(*((path.__class__(p) for p in path.parts) for path in paths))) Except I'll bet I got the parens wrong somewhere; it's probably clearer as: def common_path_prefix(*paths): parts = (map(path.__class__, path.parts) for path in paths) prefix = common_prefix(*parts) return paths[0].__class__(*prefix) >> And what about common_prefix('foo/bar', '../here/foo')? Or >> common_prefix('bar/baz', 'foo/../bar/baz')? Pathlib avoids collapsing >> .. because the meaning could change in the face of symlinks. I believe >> the same applies here. Maybe you need to call resolve() before doing >> the common prefix operation (but that gives an absolute path). >> >> With the above limitations, would a common_prefix function actually >> help typical use cases? In my experience, doing list operations on >> pathobj.parts is often simple enough that I don't need specialised >> functions like common_prefix... >> >> Getting the edge cases right is fiddly enough for common_prefix that a >> specialised function is a reasonable idea, but only if the "obvious" >> behaviour is clear. If there's a lot of conflicting possibilities, >> maybe a recipe in the docs would be a better option. >> >> Paul >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ From abarnert at yahoo.com Tue Mar 24 15:56:06 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Tue, 24 Mar 2015 07:56:06 -0700 Subject: [Python-ideas] pathlib.Path.parts - tuple of path objects rather than strings? In-Reply-To: References: <690A535E-4A78-4B0D-9BB5-153BFCF70757@unterwaditzer.net> Message-ID: On Mar 24, 2015, at 6:44 AM, Tal Einat wrote: > > On Tue, Mar 24, 2015 at 2:39 PM, Markus Unterwaditzer > wrote: >> >> I'd suggest that an additional property is added where: >> >> Path("/usr/bin/true").newproperty == (Path("/"), >> Path("/usr"), >> Path("/usr/bin"), >> Path("/usr/bin/true")) >> >> I.e. each part is a prefix of the next one. > > Path.parents already exists: > > Python 3.4.2 (default, Feb 23 2015, 21:16:28) > [GCC 4.2.1 Compatible Apple LLVM 5.1 (clang-503.0.40)] on darwin > Type "help", "copyright", "credits" or "license" for more information. >>>> import pathlib >>>> list(pathlib.Path("/usr/bin/true").parents) > [PosixPath('/usr/bin'), PosixPath('/usr'), PosixPath('/')] Ooh, forgot about that! That implies a conceptually simpler implementation: it's just first(last(takewhile(all_equal(reversed(map(parents)))))), but I'm too tired to finish translating that from Haskelly terms to itertoolsy terms in my head. :) I'm not sure if that would be simpler (or more efficient) in practice, however. Also, it would give the opposite answer to is the 'true' bit a potential part of the result if you common-prefix /usr/bin/true with itself (or only pass one argument), which goes nicely with my intuition but not so nicely with the rosettacode page. But maybe that's just a matter of naming--with "common_parent" or "common_ancestor" instead of "common_prefix" it seems pretty obvious and unambiguous? From p.f.moore at gmail.com Tue Mar 24 16:15:45 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 24 Mar 2015 15:15:45 +0000 Subject: [Python-ideas] os.path.commonprefix: Yes that old chestnut. In-Reply-To: <850BA2F2-AF08-4DE8-8661-ABF9A66A2D3A@yahoo.com> References: <22b04160-2ff7-4c98-9d2c-97ac13b3bc25@googlegroups.com> <20150321124732.0b47730e@fsol> <239E08B3-8FD0-4744-95E7-691900325A1D@yahoo.com> <850BA2F2-AF08-4DE8-8661-ABF9A66A2D3A@yahoo.com> Message-ID: On 24 March 2015 at 14:44, Andrew Barnert wrote: > From my other reply, looking over the functions used in some of the rosettacode examples, it looks like the generic iterable function, when it exists, and the language makes it feasible, often handles an arbitrary number of arguments, not just two. Which makes sense, now that I think about it. So: > > def common_prefix(*iterables): > for first, *rest in zip(*iterables): > if any(first != part for part in rest): > return > yield first Thanks for the research. This is probably something that could be included as a recipe in the itertools documentation. (I doubt it would be viewed as a common enough requirement to be added to the module itself...) Do you have any objection to me submitting a doc patch with this code? (It's similar to what I came up with, but a lot cleaner). Paul From abarnert at yahoo.com Tue Mar 24 16:32:51 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Tue, 24 Mar 2015 08:32:51 -0700 Subject: [Python-ideas] os.path.commonprefix: Yes that old chestnut. In-Reply-To: References: <22b04160-2ff7-4c98-9d2c-97ac13b3bc25@googlegroups.com> <20150321124732.0b47730e@fsol> <239E08B3-8FD0-4744-95E7-691900325A1D@yahoo.com> <850BA2F2-AF08-4DE8-8661-ABF9A66A2D3A@yahoo.com> Message-ID: <6E7564BC-1D7D-4000-B70D-787BA93FCC7D@yahoo.com> On Mar 24, 2015, at 8:15 AM, Paul Moore wrote: > >> On 24 March 2015 at 14:44, Andrew Barnert wrote: >> From my other reply, looking over the functions used in some of the rosettacode examples, it looks like the generic iterable function, when it exists, and the language makes it feasible, often handles an arbitrary number of arguments, not just two. Which makes sense, now that I think about it. So: >> >> def common_prefix(*iterables): >> for first, *rest in zip(*iterables): >> if any(first != part for part in rest): >> return >> yield first > > Thanks for the research. This is probably something that could be > included as a recipe in the itertools documentation. (I doubt it would > be viewed as a common enough requirement to be added to the module > itself...) Do you have any objection to me submitting a doc patch with > this code? (It's similar to what I came up with, but a lot cleaner). I agree that it belongs as a recipe in itertools (and of course it can go into the third-party more-itertools library on PyPI for people who want to just import and use it, as with the other useful recipes). But you probably want to test it first, and make sure I thought through the stupid edge cases like no iterables or empty iterables, since that's just off the top of my head, and while tired and recovering from a cold to boot. :) Also, while I think this generic function is useful on its own merits, it doesn't solve the problem that started this thread--as pointed out by (I forget who), just using it on the string parts gives the wrong answer for Windows paths. Should there also be a recipe in the pathlib docs referencing the itertools recipe? (Is it even reasonable for a recipe in one module's docs to rely on a recipe in another's?) Or maybe, as Tal (I think) suggested, a better solution is to avoid the problem and use parents instead of parts, meaning you don't even need a common_prefix recipe, but a last_initial_match (although that's just last(common_prefix(*args)), so maybe that doesn't really matter.) From p.f.moore at gmail.com Tue Mar 24 16:45:01 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 24 Mar 2015 15:45:01 +0000 Subject: [Python-ideas] os.path.commonprefix: Yes that old chestnut. In-Reply-To: <6E7564BC-1D7D-4000-B70D-787BA93FCC7D@yahoo.com> References: <22b04160-2ff7-4c98-9d2c-97ac13b3bc25@googlegroups.com> <20150321124732.0b47730e@fsol> <239E08B3-8FD0-4744-95E7-691900325A1D@yahoo.com> <850BA2F2-AF08-4DE8-8661-ABF9A66A2D3A@yahoo.com> <6E7564BC-1D7D-4000-B70D-787BA93FCC7D@yahoo.com> Message-ID: On 24 March 2015 at 15:32, Andrew Barnert wrote: > But you probably want to test it first, and make sure I thought through the stupid edge cases like no iterables or empty iterables, since that's just off the top of my head, and while tired and recovering from a cold to boot. :) lol Understood. It's still cleaner than my one :-) > Also, while I think this generic function is useful on its own merits, it doesn't solve the problem that started this thread--as pointed out by (I forget who), just using it on the string parts gives the wrong answer for Windows paths. Should there also be a recipe in the pathlib docs referencing the itertools recipe? (Is it even reasonable for a recipe in one module's docs to rely on a recipe in another's?) Or maybe, as Tal (I think) suggested, a better solution is to avoid the problem and use parents instead of parts, meaning you don't even need a common_prefix recipe, but a last_initial_match (although that's just last(common_prefix(*args)), so maybe that doesn't really matter.) I still think there's a case for a pathlib.PurePath.common_prefix() method, if we get consensus on the behaviour, so if we did that, I'd probably grab the itertools recipe and use it as a private function in pathlib. It feels mildly clumsy to have a private implementation of an itertools recipe in a different stdlib module, but I don't think it's a big enough deal to justify pushing for the function to be actually added to itertools (which would involve rewriting in C, for a start). Paul From guido at python.org Tue Mar 24 17:42:19 2015 From: guido at python.org (Guido van Rossum) Date: Tue, 24 Mar 2015 09:42:19 -0700 Subject: [Python-ideas] PEP 484 (Type Hints) -- second draft In-Reply-To: <5510D31D.3080307@canterbury.ac.nz> References: <20150323214145.GA94958@cskk.homeip.net> <5510D31D.3080307@canterbury.ac.nz> Message-ID: On Mon, Mar 23, 2015 at 7:59 PM, Greg wrote: > On 24/03/2015 11:35 a.m., Guido van Rossum wrote: > >> when you use an >> if-expression, there's no way to shut it up! >> > > Maybe an if-expression should assume the union type, then? > This was flagged as a mypy bug, but the resolution is still up in the air: https://github.com/JukkaL/mypy/issues/622 -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From taleinat at gmail.com Tue Mar 24 19:00:37 2015 From: taleinat at gmail.com (Tal Einat) Date: Tue, 24 Mar 2015 20:00:37 +0200 Subject: [Python-ideas] pathlib.Path.parts - tuple of path objects rather than strings? In-Reply-To: References: <690A535E-4A78-4B0D-9BB5-153BFCF70757@unterwaditzer.net> Message-ID: On Tue, Mar 24, 2015 at 4:56 PM, Andrew Barnert wrote: > On Mar 24, 2015, at 6:44 AM, Tal Einat wrote: >> Path.parents already exists: >> >> Python 3.4.2 (default, Feb 23 2015, 21:16:28) >> [GCC 4.2.1 Compatible Apple LLVM 5.1 (clang-503.0.40)] on darwin >> Type "help", "copyright", "credits" or "license" for more information. >>>>> import pathlib >>>>> list(pathlib.Path("/usr/bin/true").parents) >> [PosixPath('/usr/bin'), PosixPath('/usr'), PosixPath('/')] > > Ooh, forgot about that! > > That implies a conceptually simpler implementation: it's just first(last(takewhile(all_equal(reversed(map(parents)))))), but I'm too tired to finish translating that from Haskelly terms to itertoolsy terms in my head. :) I'm not sure if that would be simpler (or more efficient) in practice, however. > > Also, it would give the opposite answer to is the 'true' bit a potential part of the result if you common-prefix /usr/bin/true with itself (or only pass one argument), which goes nicely with my intuition but not so nicely with the rosettacode page. But maybe that's just a matter of naming--with "common_parent" or "common_ancestor" instead of "common_prefix" it seems pretty obvious and unambiguous? Well, the function I wrote was "common_parent" since that was what I needed, and what it does is indeed clearer than "common_prefix". Off the top of my head, I can't think of use cases where one would want "common_prefix" instead of "common_parent" where Path.relative_to() wouldn't be better. So maybe just provide "common_parent"? - Tal Einat From storchaka at gmail.com Tue Mar 24 19:01:18 2015 From: storchaka at gmail.com (Serhiy Storchaka) Date: Tue, 24 Mar 2015 20:01:18 +0200 Subject: [Python-ideas] os.path.commonprefix: Yes that old chestnut. In-Reply-To: References: <22b04160-2ff7-4c98-9d2c-97ac13b3bc25@googlegroups.com> <20150321124732.0b47730e@fsol> Message-ID: On 24.03.15 13:56, Paul Moore wrote: > On 23 March 2015 at 21:33, Gregory P. Smith wrote: >> +1 pathlib would be the appropriate place for the correctly behaving >> function to appear. > > OK, so here's a question. What actual use cases exist for a > common_prefix function? The reason I ask is that I'm looking at some > of the edge cases, and the obvious behaviour isn't particularly clear > to me. > > For example, common_prefix('a/b/file.c', 'a/b/file.c'). The common > prefix is obviously 'a/b/file.c' - but I can imagine people *actually* > wanting the common *directory* containing both files. But taken > literally, that's only possible if you check the filesystem, so it > would no longer be a PurePath operation. > > And what about common_prefix('foo/bar', '../here/foo')? Or > common_prefix('bar/baz', 'foo/../bar/baz')? Pathlib avoids collapsing > ... because the meaning could change in the face of symlinks. I believe > the same applies here. Maybe you need to call resolve() before doing > the common prefix operation (but that gives an absolute path). > > With the above limitations, would a common_prefix function actually > help typical use cases? In my experience, doing list operations on > pathobj.parts is often simple enough that I don't need specialised > functions like common_prefix... > > Getting the edge cases right is fiddly enough for common_prefix that a > specialised function is a reasonable idea, but only if the "obvious" > behaviour is clear. If there's a lot of conflicting possibilities, > maybe a recipe in the docs would be a better option. Your are asking the same questions that were asked about os,path.commonpath(). You can look at discussions on the tracker, in Python-Dev and in Python-Ideas. http://bugs.python.org/issue10395 http://mail.python.org/pipermail/python-dev/2000-July/005897.html http://mail.python.org/pipermail/python-dev/2000-August/008385.html http://comments.gmane.org/gmane.comp.python.ideas/17719 From storchaka at gmail.com Tue Mar 24 19:06:09 2015 From: storchaka at gmail.com (Serhiy Storchaka) Date: Tue, 24 Mar 2015 20:06:09 +0200 Subject: [Python-ideas] os.path.commonprefix: Yes that old chestnut. In-Reply-To: <22b04160-2ff7-4c98-9d2c-97ac13b3bc25@googlegroups.com> References: <22b04160-2ff7-4c98-9d2c-97ac13b3bc25@googlegroups.com> Message-ID: On 21.03.15 06:41, Paddy3118 wrote: > I had to add a comment to an RC entry > that was > using os.path..commonprefix to compute the longest common prefix which > it does regardless of path separators, (which is what it should do). > > It has been discussed before - it > goes back to the nineties it seems, but we are still left with a > misplaced function, (it should be str.commonprefix), with an awful hack > of saying "yes we know its ballsed up" in the documentation > . > > I guess we missed a great opportunity to fix this when we moved to > Python 3 too!? > > The fix seems clear: deprecate os.path.commonprefix whilst creating a > true os.path.commonpath and str.commonprefix. The deprecated function > should hang around and the standard libs modified to switch to the new > function(s) > > I've heard that some religious people put obvious faults in their work > as only their god should get so close to perfection - maybe that's why > we still have this wart :-) OK, if there are no objections and comments, I'll commit the recent patch for issue10395 (with only the doc change as Raymond suggested). From p.f.moore at gmail.com Tue Mar 24 19:14:51 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 24 Mar 2015 18:14:51 +0000 Subject: [Python-ideas] os.path.commonprefix: Yes that old chestnut. In-Reply-To: References: <22b04160-2ff7-4c98-9d2c-97ac13b3bc25@googlegroups.com> Message-ID: On 24 March 2015 at 18:06, Serhiy Storchaka wrote: > OK, if there are no objections and comments, I'll commit the recent patch > for issue10395 (with only the doc change as Raymond suggested). +1 from me. Paul From guettliml at thomas-guettler.de Wed Mar 25 12:00:02 2015 From: guettliml at thomas-guettler.de (=?UTF-8?B?VGhvbWFzIEfDvHR0bGVy?=) Date: Wed, 25 Mar 2015 12:00:02 +0100 Subject: [Python-ideas] OrderedDict.insert() Message-ID: <55129532.8030502@thomas-guettler.de> I am missing OrderedDict.insert() Other people, too: http://stackoverflow.com/questions/16664874/how-can-i-add-the-element-at-the-top-of-ordereddict-in-python Why was it not implemented up to now? Why not implement it for the future? Regards, Thomas G?ttler From eliben at gmail.com Wed Mar 25 13:42:40 2015 From: eliben at gmail.com (Eli Bendersky) Date: Wed, 25 Mar 2015 05:42:40 -0700 Subject: [Python-ideas] gofmt for Python: standardized styling as a language feature In-Reply-To: References: <5508DBB0.2010209@stoneleaf.us> Message-ID: On Tue, Mar 17, 2015 at 8:29 PM, Eli Bendersky wrote: > Interesting coincidence :-) > > We (at Google) are just on the verge of releasing an open-source tool for > auto-formatting of Python code. It should be posted to Github within the > next few days or weeks, under a permissive license (Apache 2.0 or something > similar). > > The tool works like gofmt and clang-format (the C++/JS version provided as > part of the Clang toolchain) - it only acts on whitespace and performs > re-formatting of code (PEP-8 oriented for Python but it's configurable). It > does not rename variables/classes/functions to conform to any particular > style. The algorithm is based on clang-format, and lib2to3 is used for the > parsing side - so it's very solid. We use it on Python 2 code internally, > but Python 3 support is also planned and should be easy. > > We intend to keep actively developing the tool in the open and will be > happy to accept test cases, fixes and patches in general. > > So stay tuned! > > https://github.com/google/yapf Eli > P.S. I'm also against such a tool being a part of the Python distribution, > at least in the near term. A tool/library has to first prove itself outside > of core for a while, and only when/if it proves to be widely used and > stable, an incorporation into core should be considered. The stdlib is big > as it is, and any additional code is a burden on all core developers. > Incorporation into the stdlib also imposes a very slow pace of > development/release on said library/tool, and hence should only be > considered when it's been extremely stable for a while. > > Eli > > > > On Tue, Mar 17, 2015 at 7:17 PM, Nicholas Chammas < > nicholas.chammas at gmail.com> wrote: > >> First class means it's like a language feature. Actually, in my case I am >> using "first-class" and "included with the language" synonymously. Not sure >> if that's appropriate terminology, but that's what I mean. >> >> To explain by rough analogy, Twisted is to asyncio what autopep8 would be >> to a first-class auto-styler. (I say "rough" because I understand that >> Twisted is not supposed to be replaced entirely by asyncio, but I hope my >> point is clear.) >> >> Nick >> >> On Tue, Mar 17, 2015 at 9:58 PM Ethan Furman wrote: >> >>> On 03/17/2015 06:54 PM, Nicholas Chammas wrote: >>> >>> > I am not proposing putting |autopep8| in the standard library. I am >>> proposing that an auto-styler be included with >>> > Python as a first-class utility. The emphasis here is on /included/ >>> and /first-class/. >>> >>> Perhaps you could explain what you mean by a /first-class/ utility? >>> >>> -- >>> ~Ethan~ >>> >>> _______________________________________________ >>> Python-ideas mailing list >>> Python-ideas at python.org >>> https://mail.python.org/mailman/listinfo/python-ideas >>> Code of Conduct: http://python.org/psf/codeofconduct/ >> >> >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From markus at unterwaditzer.net Wed Mar 25 14:08:39 2015 From: markus at unterwaditzer.net (Markus Unterwaditzer) Date: Wed, 25 Mar 2015 14:08:39 +0100 Subject: [Python-ideas] OrderedDict.insert() In-Reply-To: <55129532.8030502@thomas-guettler.de> References: <55129532.8030502@thomas-guettler.de> Message-ID: <8772702B-D0F7-4CBA-9B69-C0C8292D42F3@unterwaditzer.net> As far as I can see, no concrete usecase is shown in the SO question. What is yours? -- Markus On 25 March 2015 12:00:02 CET, "Thomas G?ttler" wrote: >I am missing OrderedDict.insert() > >Other people, too: > >http://stackoverflow.com/questions/16664874/how-can-i-add-the-element-at-the-top-of-ordereddict-in-python > >Why was it not implemented up to now? > >Why not implement it for the future? > >Regards, > Thomas G?ttler > >_______________________________________________ >Python-ideas mailing list >Python-ideas at python.org >https://mail.python.org/mailman/listinfo/python-ideas >Code of Conduct: http://python.org/psf/codeofconduct/ From tjreedy at udel.edu Wed Mar 25 17:38:09 2015 From: tjreedy at udel.edu (Terry Reedy) Date: Wed, 25 Mar 2015 12:38:09 -0400 Subject: [Python-ideas] gofmt for Python: standardized styling as a language feature In-Reply-To: References: <5508DBB0.2010209@stoneleaf.us> Message-ID: On 3/25/2015 8:42 AM, Eli Bendersky wrote: > > > On Tue, Mar 17, 2015 at 8:29 PM, Eli Bendersky > > wrote: > > Interesting coincidence :-) > > We (at Google) are just on the verge of releasing an open-source > tool for auto-formatting of Python code. It should be posted to > Github within the next few days or weeks, under a permissive license > (Apache 2.0 or something similar). > > The tool works like gofmt and clang-format (the C++/JS version > provided as part of the Clang toolchain) - it only acts on > whitespace and performs re-formatting of code (PEP-8 oriented for > Python but it's configurable). It does not rename > variables/classes/functions to conform to any particular style. The > algorithm is based on clang-format, and lib2to3 is used for the > parsing side - so it's very solid. We use it on Python 2 code > internally, but Python 3 support is also planned and should be easy. > > We intend to keep actively developing the tool in the open and will > be happy to accept test cases, fixes and patches in general. > > So stay tuned! > > > https://github.com/google/yapf yet-another-python-formatter The so-called pep8 style in style.py has # The number of spaces required before a trailing comment. SPACES_BEFORE_COMMENT=1, PEP8 actually says "Inline comments should be separated by at least two spaces from the statement.", which is what you have in the google style. -- Terry Jan Reedy From tjreedy at udel.edu Wed Mar 25 17:46:04 2015 From: tjreedy at udel.edu (Terry Reedy) Date: Wed, 25 Mar 2015 12:46:04 -0400 Subject: [Python-ideas] OrderedDict.insert() In-Reply-To: <55129532.8030502@thomas-guettler.de> References: <55129532.8030502@thomas-guettler.de> Message-ID: On 3/25/2015 7:00 AM, Thomas G?ttler wrote: > I am missing OrderedDict.insert() > > Other people, too: > > > http://stackoverflow.com/questions/16664874/how-can-i-add-the-element-at-the-top-of-ordereddict-in-python I just added this answer, which I suspect covers move use cases. === If you know you will want a 'c' key, but do not know the value, insert 'c' with a dummy value when you create the dict. d1 = OrderedDict([('c', None), ('a', '1'), ('b', '2')]) and change the value later. d1['c'] = 3 -- Terry Jan Reedy From abarnert at yahoo.com Wed Mar 25 17:51:17 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Wed, 25 Mar 2015 09:51:17 -0700 Subject: [Python-ideas] OrderedDict.insert() In-Reply-To: <55129532.8030502@thomas-guettler.de> References: <55129532.8030502@thomas-guettler.de> Message-ID: On Mar 25, 2015, at 04:00, Thomas G?ttler wrote: > > I am missing OrderedDict.insert() With what interface? Does it take an index, like list.insert, even though other OrderedDict methods (including __getitem__ and friends) don't? Or does it take another key? If so, does it insert before or after that key? And what if that key doesn't exist? Meanwhile, if the new key was already in the dict, should it move to the new position, update values in place, or raise an exception? Do you expect O(1) performance? > Other people, too: > > http://stackoverflow.com/questions/16664874/how-can-i-add-the-element-at-the-top-of-ordereddict-in-python That other person is asking for some way to put a new element on the left instead of the right, which doesn't have any of those issues. (And which is already possible with move_to_end.) And an anonymous low-rep user asking a question, getting a handful of upvotes, and accepting an answer that says "You can't, just build a new dict" with no comment doesn't exactly show a burning desire within a wide community. In fact, I think this does a better job showing how OrderedDict is often an attractive nuisance than how it's incomplete--the guy can't explain why he wants this, and meanwhile he apparently thinks you have to build a new dict and call update to add a value (or at least that doing so should have a different result than just using __setitem__ with that new key and value). My guess is that either he didn't really want an OrderedDict in the first place, or Zagorulkin Dmitri is right that the real problem is that his program needs this feature and it should be rewritten so it doesn't. But without actually knowing the use case that's just a guess. > Why was it not implemented up to now? > > Why not implement it for the future? > > Regards, > Thomas G?ttler > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ From ron3200 at gmail.com Wed Mar 25 20:59:49 2015 From: ron3200 at gmail.com (Ron Adam) Date: Wed, 25 Mar 2015 15:59:49 -0400 Subject: [Python-ideas] OrderedDict.insert() In-Reply-To: References: <55129532.8030502@thomas-guettler.de> Message-ID: On 03/25/2015 12:51 PM, Andrew Barnert wrote: > On Mar 25, 2015, at 04:00, Thomas G?ttler > wrote: >>> I am missing OrderedDict.insert() > With what interface? Does it take an index, like list.insert, even > though other OrderedDict methods (including __getitem__ and friends) > don't? Or does it take another key? If so, does it insert before or > after that key? And what if that key doesn't exist? Meanwhile, if the > new key was already in the dict, should it move to the new position, > update values in place, or raise an exception? Do you expect O(1) > performance? >>> Other people, too: >>> http://stackoverflow.com/questions/16664874/how-can-i-add-the-element-at-the-top-of-ordereddict-in-python > That other person is asking for some way to put a new element on the > left instead of the right, which doesn't have any of those issues. (And > which is already possible with move_to_end.) And an anonymous low-rep > user asking a question, getting a handful of upvotes, and accepting an > answer that says "You can't, just build a new dict" with no comment > doesn't exactly show a burning desire within a wide community. > In fact, I think this does a better job showing how OrderedDict is often > an attractive nuisance than how it's incomplete--the guy can't explain > why he wants this, and meanwhile he apparently thinks you have to build > a new dict and call update to add a value (or at least that doing so > should have a different result than just using __setitem__ with that new > key and value). My guess is that either he didn't really want an > OrderedDict in the first place, or Zagorulkin Dmitri is right that the > real problem is that his program needs this feature and it should be > rewritten so it doesn't. But without actually knowing the use case > that's just a guess. As to OrderedDict being a an attractive nuisance, I'm not sure. I do think there are data structures that benefit from being index-able by both position and by name. Scheme uses association lists for this, and documents that they are a bit slower than mappings. But it is easy to convert one to the other and back again. In python you can convert to a list of items and back. But it can be expensive to keep two versions of the same data in memory or to constantly convert back and forth. An ordered dict insures the conversion to an items list will result in a dependable order. That may be the primary benefit. One way to do an insert is to mirror how the __init__ method works with items, but with an index for the first value. Dict.insert_items(index, items) Another possibility is to have both get_items and set_items methods than accept an index or slice object as the first argument. The __getitem__ and __setitem__ methods should not take an index in a dictionary. It is a dictionary first, and ordered data second. If it was an association list, then this would be reversed. Cheers, Ron From jeanpierreda at gmail.com Thu Mar 26 01:34:34 2015 From: jeanpierreda at gmail.com (Devin Jeanpierre) Date: Wed, 25 Mar 2015 17:34:34 -0700 Subject: [Python-ideas] gofmt for Python: standardized styling as a language feature In-Reply-To: References: <5508DBB0.2010209@stoneleaf.us> Message-ID: I'm not sure everyone that works on yapf actually reads this ML, so it'd be a good idea to use github's bug tracker when reporting problems in the future. I filed a ticket for you: https://github.com/google/yapf/issues/14 -- Devin On Wed, Mar 25, 2015 at 9:38 AM, Terry Reedy wrote: > On 3/25/2015 8:42 AM, Eli Bendersky wrote: >> >> >> >> On Tue, Mar 17, 2015 at 8:29 PM, Eli Bendersky >> > > wrote: >> >> Interesting coincidence :-) >> >> We (at Google) are just on the verge of releasing an open-source >> tool for auto-formatting of Python code. It should be posted to >> Github within the next few days or weeks, under a permissive license >> (Apache 2.0 or something similar). >> >> The tool works like gofmt and clang-format (the C++/JS version >> provided as part of the Clang toolchain) - it only acts on >> whitespace and performs re-formatting of code (PEP-8 oriented for >> Python but it's configurable). It does not rename >> variables/classes/functions to conform to any particular style. The >> algorithm is based on clang-format, and lib2to3 is used for the >> parsing side - so it's very solid. We use it on Python 2 code >> internally, but Python 3 support is also planned and should be easy. >> >> We intend to keep actively developing the tool in the open and will >> be happy to accept test cases, fixes and patches in general. >> >> So stay tuned! >> >> >> https://github.com/google/yapf > > > yet-another-python-formatter > > The so-called pep8 style in style.py has > > # The number of spaces required before a trailing comment. > SPACES_BEFORE_COMMENT=1, > > PEP8 actually says "Inline comments should be separated by at least two > spaces from the statement.", which is what you have in the google style. > > -- > Terry Jan Reedy > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ From eliben at gmail.com Thu Mar 26 02:00:09 2015 From: eliben at gmail.com (Eli Bendersky) Date: Wed, 25 Mar 2015 18:00:09 -0700 Subject: [Python-ideas] gofmt for Python: standardized styling as a language feature In-Reply-To: References: <5508DBB0.2010209@stoneleaf.us> Message-ID: On Wed, Mar 25, 2015 at 9:38 AM, Terry Reedy wrote: > On 3/25/2015 8:42 AM, Eli Bendersky wrote: > >> >> >> On Tue, Mar 17, 2015 at 8:29 PM, Eli Bendersky >> > > wrote: >> >> Interesting coincidence :-) >> >> We (at Google) are just on the verge of releasing an open-source >> tool for auto-formatting of Python code. It should be posted to >> Github within the next few days or weeks, under a permissive license >> (Apache 2.0 or something similar). >> >> The tool works like gofmt and clang-format (the C++/JS version >> provided as part of the Clang toolchain) - it only acts on >> whitespace and performs re-formatting of code (PEP-8 oriented for >> Python but it's configurable). It does not rename >> variables/classes/functions to conform to any particular style. The >> algorithm is based on clang-format, and lib2to3 is used for the >> parsing side - so it's very solid. We use it on Python 2 code >> internally, but Python 3 support is also planned and should be easy. >> >> We intend to keep actively developing the tool in the open and will >> be happy to accept test cases, fixes and patches in general. >> >> So stay tuned! >> >> >> https://github.com/google/yapf >> > > yet-another-python-formatter > > The so-called pep8 style in style.py has > > # The number of spaces required before a trailing comment. > SPACES_BEFORE_COMMENT=1, > > PEP8 actually says "Inline comments should be separated by at least two > spaces from the statement.", which is what you have in the google style. > > Thanks Terry. Devin opened an issue on Github and we'll fix this. Eli > -- > Terry Jan Reedy > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From probiner at gmail.com Thu Mar 26 02:45:29 2015 From: probiner at gmail.com (pedro santos) Date: Thu, 26 Mar 2015 01:45:29 +0000 Subject: [Python-ideas] list.pop() accepting a list of integers as argument Message-ID: Is this best way to remove multiple items from a list? Because if so I think passing a list of integers through pop would be cleaner. Code here: http://repl.it/f6E Cheers -- *------------------------------[image: http://i153.photobucket.com/albums/s202/animatics/probiner-sig.gif]Pedro Alpiar?a dos Santos Animator 3DModeler Illustrator >> http://probiner.x10.mx/ * -------------- next part -------------- An HTML attachment was scrubbed... URL: From probiner at gmail.com Thu Mar 26 03:39:38 2015 From: probiner at gmail.com (pedro santos) Date: Thu, 26 Mar 2015 02:39:38 +0000 Subject: [Python-ideas] list.pop() accepting a list of integers as argument In-Reply-To: References: Message-ID: Hi David Thanks for the idea. Just some considerations: - Del is a list with indices and not the items themselves, so if you replace the code with that line it won't output the same result. - *del *deletes all items with an equal value so if I have L=[2,2,2,1,2] and have Del=[2], I end up with L=[1], which useful, but not my goal here and the reason I went to pop() instead, so I can address indices. Same remark for *in*. - I wouldn't mind if* in* and *del* approach would work if you instead of checking the value would be checking if a list had such indices and then deleting all at the same time if so. Again the same code with comments. http://repl.it/f6E/3 Thanks On Thu, Mar 26, 2015 at 2:04 AM, David Blaschke wrote: > ist comprehension is straight forward IMHO, and possibly faster than > pop depending on whether the items are removed one at a time or just > marked for deletion later > new_list=[item for item in L if item not in Del] > > On 3/25/15, pedro santos wrote: > > Is this best way to remove multiple items from a list? > > Because if so I think passing a list of integers through pop would be > > cleaner. > > > > Code here: http://repl.it/f6E > > > > Cheers > > > > > > > > > > -- > > > > > > > > > > *------------------------------[image: > > http://i153.photobucket.com/albums/s202/animatics/probiner-sig.gif]Pedro > > Alpiar?a dos Santos Animator 3DModeler Illustrator >> > > http://probiner.x10.mx/ * > > > > > -- > Only when you actually get to the state where there is neither > delusion nor enlightenment are you finally comfortable...Foyan > -- *------------------------------[image: http://i153.photobucket.com/albums/s202/animatics/probiner-sig.gif]Pedro Alpiar?a dos Santos Animator 3DModeler Illustrator >> http://probiner.x10.mx/ * -------------- next part -------------- An HTML attachment was scrubbed... URL: From ron3200 at gmail.com Thu Mar 26 04:36:04 2015 From: ron3200 at gmail.com (Ron Adam) Date: Wed, 25 Mar 2015 23:36:04 -0400 Subject: [Python-ideas] list.pop() accepting a list of integers as argument In-Reply-To: References: Message-ID: On 03/25/2015 09:45 PM, pedro santos wrote: > Is this best way to remove multiple items from a list? > Because if so I think passing a list of integers through pop would be cleaner. This is probably the fastest way to remove N items from the end. >>> L = list("123456789") >>> L ['1', '2', '3', '4', '5', '6', '7', '8', '9'] >>> L[-3:] = [] >>> L ['1', '2', '3', '4', '5', '6'] Cheers, Ron From probiner at gmail.com Thu Mar 26 07:58:57 2015 From: probiner at gmail.com (pedro santos) Date: Thu, 26 Mar 2015 06:58:57 +0000 Subject: [Python-ideas] list.pop() accepting a list of integers as argument In-Reply-To: References: Message-ID: Hi Ron Wasn't looking for the n items from the end but good to know. Cheers On Thu, Mar 26, 2015 at 3:36 AM, Ron Adam wrote: > > > On 03/25/2015 09:45 PM, pedro santos wrote: > >> Is this best way to remove multiple items from a list? >> Because if so I think passing a list of integers through pop would be >> cleaner. >> > > This is probably the fastest way to remove N items from the end. > > >>> L = list("123456789") > >>> L > ['1', '2', '3', '4', '5', '6', '7', '8', '9'] > >>> L[-3:] = [] > >>> L > ['1', '2', '3', '4', '5', '6'] > > Cheers, > Ron > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- *------------------------------[image: http://i153.photobucket.com/albums/s202/animatics/probiner-sig.gif]Pedro Alpiar?a dos Santos Animator 3DModeler Illustrator >> http://probiner.x10.mx/ * -------------- next part -------------- An HTML attachment was scrubbed... URL: From storchaka at gmail.com Thu Mar 26 08:41:53 2015 From: storchaka at gmail.com (Serhiy Storchaka) Date: Thu, 26 Mar 2015 09:41:53 +0200 Subject: [Python-ideas] list.pop() accepting a list of integers as argument In-Reply-To: References: Message-ID: On 26.03.15 05:36, Ron Adam wrote: > This is probably the fastest way to remove N items from the end. > > >>> L = list("123456789") > >>> L > ['1', '2', '3', '4', '5', '6', '7', '8', '9'] > >>> L[-3:] = [] > >>> L > ['1', '2', '3', '4', '5', '6'] del L[-3:] From szport at gmail.com Thu Mar 26 06:17:48 2015 From: szport at gmail.com (Zaur Shibzukhov) Date: Wed, 25 Mar 2015 22:17:48 -0700 (PDT) Subject: [Python-ideas] recorarray: a mutable alternative to namedtuple In-Reply-To: References: Message-ID: <6b53b718-0220-48d5-b5b5-ded2fa9e867c@googlegroups.com> ???????, 17 ????? 2015 ?., 19:52:28 UTC+3 ???????????? Luciano Ramalho ???????: > > Sometimes we need a simple class to hold some mutable attributes, > provide a nice repr, support == for testing, and support iterable > unpacking, so you can write: > > >>> p = Point(3, 4) > >>> x, y = p > > That's very much like the classes built by namedtuple, but mutable. > > I propose we add to the collections module another class factory. I am > calling it plainclass, but perhaps we can think of a better name. Here > is how it would be used: > > >>> import collections > >>> Point = collections.plainclass('Point', 'x y') > > The signature of the plainclass function would be exactly the same as > namedtuple, supporting the same alternative ways of naming the > attributes. > > There is an attempt to make such alternative: recordarray . -------------- next part -------------- An HTML attachment was scrubbed... URL: From guettliml at thomas-guettler.de Thu Mar 26 09:59:06 2015 From: guettliml at thomas-guettler.de (=?UTF-8?B?VGhvbWFzIEfDvHR0bGVy?=) Date: Thu, 26 Mar 2015 09:59:06 +0100 Subject: [Python-ideas] OrderedDict.insert() In-Reply-To: References: <55129532.8030502@thomas-guettler.de> Message-ID: <5513CA5A.1020001@thomas-guettler.de> Am 25.03.2015 um 20:59 schrieb Ron Adam: > > > On 03/25/2015 12:51 PM, Andrew Barnert wrote: >> On Mar 25, 2015, at 04:00, Thomas G?ttler >> wrote: > >>>> I am missing OrderedDict.insert() > >> With what interface? Does it take an index, like list.insert, even >> though other OrderedDict methods (including __getitem__ and friends) >> don't? Or does it take another key? If so, does it insert before or >> after that key? And what if that key doesn't exist? Meanwhile, if the >> new key was already in the dict, should it move to the new position, >> update values in place, or raise an exception? Do you expect O(1) >> performance? My use case is that django drops support for SortedDict, and you should use collection.OrderedDict in the future: https://code.djangoproject.com/wiki/SortedDict We have one line where we use my_ordered_dict.insert() and this line is already updated. The issue is not very important for me. To your question: The interface of django was insert(index, key, value) but you are right. An interface which is relative to an existing key would be better. insert_before(other_key, new_key, new_value) insert_behind(other_key, new_key, new_value) Both methods should raise KeyError if other_key is not in the dict and should raise KeyError if new_key is already in the dict. Performance: I don't care. I store big data in a database. I don't carry it around in the memory if the python interpreter. Regards, Thomas G?ttler From abarnert at yahoo.com Thu Mar 26 11:08:56 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Thu, 26 Mar 2015 03:08:56 -0700 Subject: [Python-ideas] OrderedDict.insert() In-Reply-To: <5513CA5A.1020001@thomas-guettler.de> References: <55129532.8030502@thomas-guettler.de> <5513CA5A.1020001@thomas-guettler.de> Message-ID: <9A99AC22-980F-42F5-ACBE-89573F60189E@yahoo.com> On Mar 26, 2015, at 01:59, Thomas G?ttler wrote: > >> Am 25.03.2015 um 20:59 schrieb Ron Adam: >> >> >>> On 03/25/2015 12:51 PM, Andrew Barnert wrote: >>> On Mar 25, 2015, at 04:00, Thomas G?ttler >>> wrote: >> >>>>> I am missing OrderedDict.insert() >> >>> With what interface? Does it take an index, like list.insert, even >>> though other OrderedDict methods (including __getitem__ and friends) >>> don't? Or does it take another key? If so, does it insert before or >>> after that key? And what if that key doesn't exist? Meanwhile, if the >>> new key was already in the dict, should it move to the new position, >>> update values in place, or raise an exception? Do you expect O(1) >>> performance? > > My use case is that django drops support for SortedDict, and you should use > collection.OrderedDict in the future: > > https://code.djangoproject.com/wiki/SortedDict From a quick scan (see https://github.com/django/django/commit/7379d9aceab01e007966b5fe1f47ba7590deb887), it looks like Django deprecated insert in 1.5--two versions before deprecating SortedDict itself, while the type was still being heavily used all over Django itself. That implies that they didn't think it belonged there either (although I didn't find the issue or discussion where they made that decision, so don't take that as gospel--for all I know maybe they just discovered a bad bug in their implementation and decided deprecating it would break less code than changing it...). Meanwhile, that's not really a use case. Without any idea of why you used SortedDict.insert, I still have no clue why someone would use OrderedDict.insert. I don't think it's worth adding stuff to Python's stdlib just to make it easier for people porting pre-Django-1.5 code to Django 1.9 to work around an intentional change in Django. > We have one line where we use my_ordered_dict.insert() and this line is already updated. > The issue is not very important for me. > > To your question: > > The interface of django was insert(index, key, value) but you are right. An interface > which is relative to an existing key would be better. I'm not sure about that. I suspect inserting at the front would be one of the more common uses for this (again, just guessing, because I don't know of _any_ real-life uses... but that was what the SO question you linked to wanted...), which would be pretty clumsy with before-key instead of before-index. But that was part of the point. I don't think the meaning or interface of OrderedDict.insert is obvious, and you didn't say what you wanted it to do. In fact what you really wanted and what Django used to do are different, so what are the odds that whichever one we chose, some users would have guessed the opposite? > insert_before(other_key, new_key, new_value) > > insert_behind(other_key, new_key, new_value) > > Both methods should raise KeyError if other_key is not in the dict and > should raise KeyError if new_key is already in the dict. > > Performance: I don't care. I store big data in a database. I don't carry it > around in the memory if the python interpreter. > > Regards, > Thomas G?ttler > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ From ericsnowcurrently at gmail.com Thu Mar 26 23:06:06 2015 From: ericsnowcurrently at gmail.com (Eric Snow) Date: Thu, 26 Mar 2015 16:06:06 -0600 Subject: [Python-ideas] OrderedDict.insert() In-Reply-To: <55129532.8030502@thomas-guettler.de> References: <55129532.8030502@thomas-guettler.de> Message-ID: On Wed, Mar 25, 2015 at 5:00 AM, Thomas G?ttler wrote: > I am missing OrderedDict.insert() > > Other people, too: > > > http://stackoverflow.com/questions/16664874/how-can-i-add-the-element-at-the-top-of-ordereddict-in-python > > Why was it not implemented up to now? > > Why not implement it for the future? Ordered dict is a dict that preserves insertion order (and exposes via iteration-related methods like __iter__ and the views). That's it. It is not a dict that also implements Sequence. Keep in mind that OrderedDict is a fairly simply adaptation of dict. Unfortunately there is a lingering temptation to think of it as a sequence. -eric From liik.joonas at gmail.com Fri Mar 27 05:08:46 2015 From: liik.joonas at gmail.com (Joonas Liik) Date: Fri, 27 Mar 2015 06:08:46 +0200 Subject: [Python-ideas] recorarray: a mutable alternative to namedtuple In-Reply-To: <6b53b718-0220-48d5-b5b5-ded2fa9e867c@googlegroups.com> References: <6b53b718-0220-48d5-b5b5-ded2fa9e867c@googlegroups.com> Message-ID: namedlist perhaps? :) if you want:"namedtuple, but mutable. " then namedlist seems like the obvious alternative.. On 26 March 2015 at 07:17, Zaur Shibzukhov wrote: > > > ???????, 17 ????? 2015 ?., 19:52:28 UTC+3 ???????????? Luciano Ramalho > ???????: >> >> Sometimes we need a simple class to hold some mutable attributes, >> provide a nice repr, support == for testing, and support iterable >> unpacking, so you can write: >> >> >>> p = Point(3, 4) >> >>> x, y = p >> >> That's very much like the classes built by namedtuple, but mutable. >> >> I propose we add to the collections module another class factory. I am >> calling it plainclass, but perhaps we can think of a better name. Here >> is how it would be used: >> >> >>> import collections >> >>> Point = collections.plainclass('Point', 'x y') >> >> The signature of the plainclass function would be exactly the same as >> namedtuple, supporting the same alternative ways of naming the >> attributes. >> >> > There is an attempt to make such alternative: recordarray > . > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From szport at gmail.com Fri Mar 27 05:26:52 2015 From: szport at gmail.com (Zaur Shibzukhov) Date: Fri, 27 Mar 2015 07:26:52 +0300 Subject: [Python-ideas] recorarray: a mutable alternative to namedtuple In-Reply-To: References: <6b53b718-0220-48d5-b5b5-ded2fa9e867c@googlegroups.com> Message-ID: --- *Zaur Shibzukhov* 2015-03-27 7:08 GMT+03:00 Joonas Liik : > namedlist perhaps? :) > > if you want:"namedtuple, but mutable. " then namedlist seems like the > obvious alternative.. > The name 'namedlist' is already used in https://pypi.python.org/pypi/namedlist, 'mutabletuple' -- too in https://pypi.python.org/pypi/mutabletuple. The name recordarray is means that it 1) array of objects and 2) record with access to fields by attributes. > > On 26 March 2015 at 07:17, Zaur Shibzukhov wrote: > >> >> >> ???????, 17 ????? 2015 ?., 19:52:28 UTC+3 ???????????? Luciano Ramalho >> ???????: >>> >>> Sometimes we need a simple class to hold some mutable attributes, >>> provide a nice repr, support == for testing, and support iterable >>> unpacking, so you can write: >>> >>> >>> p = Point(3, 4) >>> >>> x, y = p >>> >>> That's very much like the classes built by namedtuple, but mutable. >>> >>> I propose we add to the collections module another class factory. I am >>> calling it plainclass, but perhaps we can think of a better name. Here >>> is how it would be used: >>> >>> >>> import collections >>> >>> Point = collections.plainclass('Point', 'x y') >>> >>> The signature of the plainclass function would be exactly the same as >>> namedtuple, supporting the same alternative ways of naming the >>> attributes. >>> >>> >> There is an attempt to make such alternative: recordarray >> . >> >> >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From abarnert at yahoo.com Fri Mar 27 05:32:45 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Thu, 26 Mar 2015 21:32:45 -0700 Subject: [Python-ideas] recorarray: a mutable alternative to namedtuple In-Reply-To: References: <6b53b718-0220-48d5-b5b5-ded2fa9e867c@googlegroups.com> Message-ID: <809CF78E-DA96-4D97-A1C4-6CCAD16ED50A@yahoo.com> On Mar 26, 2015, at 21:08, Joonas Liik wrote: > > namedlist perhaps? :) > > if you want:"namedtuple, but mutable. " then namedlist seems like the obvious alternative.. But, as discussed earlier in the thread, a list isn't right, or at least isn't obviously right, because lists can change size, and what would it mean for a namedlist to, say, delete element 3? And that's really the problem: Python's entire infrastructure is designed around things which are reshapable like lists, or immutable like tuples, and this doesn't fit either one. >> On 26 March 2015 at 07:17, Zaur Shibzukhov wrote: >> >> >> ???????, 17 ????? 2015 ?., 19:52:28 UTC+3 ???????????? Luciano Ramalho ???????: >>> >>> Sometimes we need a simple class to hold some mutable attributes, >>> provide a nice repr, support == for testing, and support iterable >>> unpacking, so you can write: >>> >>> >>> p = Point(3, 4) >>> >>> x, y = p >>> >>> That's very much like the classes built by namedtuple, but mutable. >>> >>> I propose we add to the collections module another class factory. I am >>> calling it plainclass, but perhaps we can think of a better name. Here >>> is how it would be used: >>> >>> >>> import collections >>> >>> Point = collections.plainclass('Point', 'x y') >>> >>> The signature of the plainclass function would be exactly the same as >>> namedtuple, supporting the same alternative ways of naming the >>> attributes. >> >> There is an attempt to make such alternative: recordarray The name "array" seems really unfortunate. It doesn't give you any clue that this thing is halfway between a tuple and a list. Also, we've already got a bytearray, array.array, and the NumPy array types, all of which hold homogeneous simple-value types and can be accessed as buffers. And bytearray and array.array are resizable. And np.ndarray--like a C array, in effect--returns views when sliced rather than copies. Just about everything this name implies is misleading. And as for recordarray, that doesn't exactly scream "like a namedtuple, but with array instead of tuple". Also, in the weeks this thing has been discussed, no one has yet come up with a use case where. Look at the motivating example, Point--do you think location[2] meaning the same thing as location.z is a good thing? That's neither easy nor common in C and its OO descendants, or SmallTalk and its, or most other languages, and I can't remember ever being bothered by that. The best anyone has come up with is that it might be more space-efficient than a SimpleNamespace or a standard __slots__ class, for all those times when you really need a billion points and can afford to needlessly waste 140 bytes per point instead of 12, but can't afford to waste 188. -------------- next part -------------- An HTML attachment was scrubbed... URL: From abarnert at yahoo.com Fri Mar 27 05:40:46 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Thu, 26 Mar 2015 21:40:46 -0700 Subject: [Python-ideas] recorarray: a mutable alternative to namedtuple In-Reply-To: References: <6b53b718-0220-48d5-b5b5-ded2fa9e867c@googlegroups.com> Message-ID: <0CF24FB3-2E88-4A44-840F-358A4498B4EF@yahoo.com> On Mar 26, 2015, at 21:26, Zaur Shibzukhov wrote: > 2015-03-27 7:08 GMT+03:00 Joonas Liik : >> namedlist perhaps? :) >> >> if you want:"namedtuple, but mutable. " then namedlist seems like the obvious alternative.. > > The name 'namedlist' is already used in https://pypi.python.org/pypi/namedlist, 'mutabletuple' -- too in https://pypi.python.org/pypi/mutabletuple. If you're trying to provide the same concept, why use a completely unrelated name? That's like saying 'I want an ordered set, but there's already an "orderedset" on PyPI so I went with "sortedsequence"'. > The name recordarray is means that it 1) array of objects and 2) record with access to fields by attributes. But how is being "an array of objects" any different from what a tuple, list, array.array, bytearray, bytes, str, etc. already are? What's specifically array-like about this type as opposed to all of those? And what's specifically record-like about your type compared to namedtuple, Struct, or SimpleNamespace? > >> >>> On 26 March 2015 at 07:17, Zaur Shibzukhov wrote: >>> >>> >>> ???????, 17 ????? 2015 ?., 19:52:28 UTC+3 ???????????? Luciano Ramalho ???????: >>>> >>>> Sometimes we need a simple class to hold some mutable attributes, >>>> provide a nice repr, support == for testing, and support iterable >>>> unpacking, so you can write: >>>> >>>> >>> p = Point(3, 4) >>>> >>> x, y = p >>>> >>>> That's very much like the classes built by namedtuple, but mutable. >>>> >>>> I propose we add to the collections module another class factory. I am >>>> calling it plainclass, but perhaps we can think of a better name. Here >>>> is how it would be used: >>>> >>>> >>> import collections >>>> >>> Point = collections.plainclass('Point', 'x y') >>>> >>>> The signature of the plainclass function would be exactly the same as >>>> namedtuple, supporting the same alternative ways of naming the >>>> attributes. >>> >>> There is an attempt to make such alternative: recordarray. >>> >>> >>> _______________________________________________ >>> Python-ideas mailing list >>> Python-ideas at python.org >>> https://mail.python.org/mailman/listinfo/python-ideas >>> Code of Conduct: http://python.org/psf/codeofconduct/ > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From guettliml at thomas-guettler.de Fri Mar 27 09:52:02 2015 From: guettliml at thomas-guettler.de (=?UTF-8?B?VGhvbWFzIEfDvHR0bGVy?=) Date: Fri, 27 Mar 2015 09:52:02 +0100 Subject: [Python-ideas] OrderedDict.insert() In-Reply-To: <9A99AC22-980F-42F5-ACBE-89573F60189E@yahoo.com> References: <55129532.8030502@thomas-guettler.de> <5513CA5A.1020001@thomas-guettler.de> <9A99AC22-980F-42F5-ACBE-89573F60189E@yahoo.com> Message-ID: <55151A32.3050003@thomas-guettler.de> I withdraw the idea. Thank you all for your answers. Am 26.03.2015 um 11:08 schrieb Andrew Barnert: > On Mar 26, 2015, at 01:59, Thomas G?ttler wrote: >> >>> Am 25.03.2015 um 20:59 schrieb Ron Adam: >>> >>> >>>> On 03/25/2015 12:51 PM, Andrew Barnert wrote: >>>> On Mar 25, 2015, at 04:00, Thomas G?ttler >>>> wrote: >>> >>>>>> I am missing OrderedDict.insert() >>> >>>> With what interface? Does it take an index, like list.insert, even >>>> though other OrderedDict methods (including __getitem__ and friends) >>>> don't? Or does it take another key? If so, does it insert before or >>>> after that key? And what if that key doesn't exist? Meanwhile, if the >>>> new key was already in the dict, should it move to the new position, >>>> update values in place, or raise an exception? Do you expect O(1) >>>> performance? >> >> My use case is that django drops support for SortedDict, and you should use >> collection.OrderedDict in the future: >> >> https://code.djangoproject.com/wiki/SortedDict > > From a quick scan (see https://github.com/django/django/commit/7379d9aceab01e007966b5fe1f47ba7590deb887), it looks like Django deprecated insert in 1.5--two versions before deprecating SortedDict itself, while the type was still being heavily used all over Django itself. That implies that they didn't think it belonged there either (although I didn't find the issue or discussion where they made that decision, so don't take that as gospel--for all I know maybe they just discovered a bad bug in their implementation and decided deprecating it would break less code than changing it...). > > Meanwhile, that's not really a use case. Without any idea of why you used SortedDict.insert, I still have no clue why someone would use OrderedDict.insert. I don't think it's worth adding stuff to Python's stdlib just to make it easier for people porting pre-Django-1.5 code to Django 1.9 to work around an intentional change in Django. > >> We have one line where we use my_ordered_dict.insert() and this line is already updated. >> The issue is not very important for me. >> >> To your question: >> >> The interface of django was insert(index, key, value) but you are right. An interface >> which is relative to an existing key would be better. > > I'm not sure about that. I suspect inserting at the front would be one of the more common uses for this (again, just guessing, because I don't know of _any_ real-life uses... but that was what the SO question you linked to wanted...), which would be pretty clumsy with before-key instead of before-index. > > But that was part of the point. I don't think the meaning or interface of OrderedDict.insert is obvious, and you didn't say what you wanted it to do. In fact what you really wanted and what Django used to do are different, so what are the odds that whichever one we chose, some users would have guessed the opposite? > >> insert_before(other_key, new_key, new_value) >> >> insert_behind(other_key, new_key, new_value) >> >> Both methods should raise KeyError if other_key is not in the dict and >> should raise KeyError if new_key is already in the dict. >> >> Performance: I don't care. I store big data in a database. I don't carry it >> around in the memory if the python interpreter. >> >> Regards, >> Thomas G?ttler >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ From toddrjen at gmail.com Fri Mar 27 10:03:26 2015 From: toddrjen at gmail.com (Todd) Date: Fri, 27 Mar 2015 10:03:26 +0100 Subject: [Python-ideas] recorarray: a mutable alternative to namedtuple In-Reply-To: <809CF78E-DA96-4D97-A1C4-6CCAD16ED50A@yahoo.com> References: <6b53b718-0220-48d5-b5b5-ded2fa9e867c@googlegroups.com> <809CF78E-DA96-4D97-A1C4-6CCAD16ED50A@yahoo.com> Message-ID: On Mar 27, 2015 5:33 AM, "Andrew Barnert" wrote: > > On Mar 26, 2015, at 21:08, Joonas Liik wrote: >> On 26 March 2015 at 07:17, Zaur Shibzukhov wrote: >>> >>> >>> >>> ???????, 17 ????? 2015 ?., 19:52:28 UTC+3 ???????????? Luciano Ramalho ???????: >>>> >>>> Sometimes we need a simple class to hold some mutable attributes, >>>> provide a nice repr, support == for testing, and support iterable >>>> unpacking, so you can write: >>>> >>>> >>> p = Point(3, 4) >>>> >>> x, y = p >>>> >>>> That's very much like the classes built by namedtuple, but mutable. >>>> >>>> I propose we add to the collections module another class factory. I am >>>> calling it plainclass, but perhaps we can think of a better name. Here >>>> is how it would be used: >>>> >>>> >>> import collections >>>> >>> Point = collections.plainclass('Point', 'x y') >>>> >>>> The signature of the plainclass function would be exactly the same as >>>> namedtuple, supporting the same alternative ways of naming the >>>> attributes. >>>> >>> >>> There is an attempt to make such alternative: recordarray > > > The name "array" seems really unfortunate. It doesn't give you any clue that this thing is halfway between a tuple and a list. Also, we've already got a bytearray, array.array, and the NumPy array types, all of which hold homogeneous simple-value types and can be accessed as buffers. And bytearray and array.array are resizable. And np.ndarray--like a C array, in effect--returns views when sliced rather than copies. Just about everything this name implies is misleading. > > And as for recordarray, that doesn't exactly scream "like a namedtuple, but with array instead of tuple". Numpy already has the concept of a "record array", which they call "recarray", which is a numpy array that is partially accessible in a similar manner to named tuples. However, otherwise they are the same as numpy arrays, which means they have a fixed size (usually) but mutable contents. Whether you consider this a point in favor or a point against, however, probably depends on your point of view. -------------- next part -------------- An HTML attachment was scrubbed... URL: From szport at gmail.com Fri Mar 27 14:17:22 2015 From: szport at gmail.com (Zaur Shibzukhov) Date: Fri, 27 Mar 2015 16:17:22 +0300 Subject: [Python-ideas] recorarray: a mutable alternative to namedtuple In-Reply-To: <0CF24FB3-2E88-4A44-840F-358A4498B4EF@yahoo.com> References: <6b53b718-0220-48d5-b5b5-ded2fa9e867c@googlegroups.com> <0CF24FB3-2E88-4A44-840F-358A4498B4EF@yahoo.com> Message-ID: --- *Zaur Shibzukhov* 2015-03-27 7:40 GMT+03:00 Andrew Barnert : > On Mar 26, 2015, at 21:26, Zaur Shibzukhov wrote: > > 2015-03-27 7:08 GMT+03:00 Joonas Liik : > >> namedlist perhaps? :) >> >> if you want:"namedtuple, but mutable. " then namedlist seems like the >> obvious alternative.. >> > > The name 'namedlist' is already used in > https://pypi.python.org/pypi/namedlist, 'mutabletuple' -- too in > https://pypi.python.org/pypi/mutabletuple. > > > If you're trying to provide the same concept, why use a completely > unrelated name? That's like saying 'I want an ordered set, but there's > already an "orderedset" on PyPI so I went with "sortedsequence"'. > > The name recordarray is means that it 1) array of objects and 2) record > with access to fields by attributes. > > > But how is being "an array of objects" any different from what a tuple, > list, array.array, bytearray, bytes, str, etc. already are? What's > specifically array-like about this type as opposed to all of those? And > what's specifically record-like about your type compared to namedtuple, > Struct, or SimpleNamespace? > > I am inclined to the fact that it's better to rename `objectarray` to 'mutabletuple' in order to be explicit about what is that. 'recordarray' is a factory function that make exactly the same as 'namedtuple' factory does (except that it create subclass of 'mutabletuple' and make '_replace' to update the 'self', not make a copy). So may be it's better to call it as 'record' or 'recordtype', or even 'recordfactory'? -------------- next part -------------- An HTML attachment was scrubbed... URL: From jsbueno at python.org.br Fri Mar 27 14:22:54 2015 From: jsbueno at python.org.br (Joao S. O. Bueno) Date: Fri, 27 Mar 2015 10:22:54 -0300 Subject: [Python-ideas] recorarray: a mutable alternative to namedtuple In-Reply-To: <0CF24FB3-2E88-4A44-840F-358A4498B4EF@yahoo.com> References: <6b53b718-0220-48d5-b5b5-ded2fa9e867c@googlegroups.com> <0CF24FB3-2E88-4A44-840F-358A4498B4EF@yahoo.com> Message-ID: On 27 March 2015 at 01:40, Andrew Barnert wrote: > But how is being "an array of objects" any different from what a tuple, > list, array.array, bytearray, bytes, str, etc. already are? What's > specifically array-like about this type as opposed to all of those? And > what's specifically record-like about your type compared to namedtuple, > Struct, or SimpleNamespace? Acutally, on my understanding, the request on this thread is for something that is quite concrete, existing in other languages, and that can be done in Python in a few lines, but is not in the stdlib: The Python equivalent of a C Struct. Just that. An easy to create class, with named fields, with possible type-enforcement for those fields. Or maybe it _does_ exist in Python, and it is a matter of having a nice example in the docs: for example a "blank" class with "__slots__" would do it. Or a blank class with slots that could serialize and deserialize itself to a sequence in a seamless way. class Base: __slots__ = () def __init__(self, seq=None): if not seq: return for attr, val in zip(self.slots, seq): setattr(self, attr, val) def __iter__(self): for attr in self.__slots__: yield getattr(self, attr) def NamedList(name, fields): ... # split string with space separated fields, and other niceities here return type(name, (Base,), dict(__slots__=fields)) And 10-15 more lines if one wants type-checking, default values, __repr__ into that. I think getting a proper recipe for this, and publicizing it on the documentation ecosystem is enough - maybe a Pypi module adding some more goodies - and if that would get any traction - the usual consideration for inclusion could apply. From rosuav at gmail.com Fri Mar 27 15:04:07 2015 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 28 Mar 2015 01:04:07 +1100 Subject: [Python-ideas] Geo coordinates conversion in stdlib In-Reply-To: References: <550F30FB.5010000@sdamon.com> <20150322223815.06082280@fsol> <21A308F0-5F2E-48BB-9F3F-4A3CFAB1D6DD@yahoo.com> Message-ID: On Sat, Mar 28, 2015 at 12:52 AM, anatoly techtonik wrote: > Why the wave module is there? Or ioctl module? Are they so widely used? > > This process is flawed. If anybody would sponsor the research on popularity > of stdlib, I believe that math will be excluded from top 50 as well. Will > stdlib > without math will still be ok? > > The measure to estimate the usefullness of standard library is not in > download score, but in coverage across different areas. There thousands of > people downloading stuff from youtube, so should we include youtube-dl > instead of math? The criteria for the *removal* from the stdlib would be insanely tight, compared to the criteria for *addition*. So some modules might now simply be grandfathered in (I suspect wave is one of them), due to the shift in ease of grabbing stuff off PyPI. But the math module is pretty important and fundamental, so it's not going anywhere. As to youtube-dl, though: Definitely not. No matter how popular it is, it still fails the criterion of release schedule synchronization - it needs to be agile enough to update as often as it needs, according to host site changes, and would be shackled rather badly by inclusion in the stdlib. (You probably don't want to get it from your OS repo either.) But none of this changes the recommendations I made earlier. You need to *demonstrate*, not simply state, a need; and one of the best ways to demonstrate support, API stability, usage, etc, is to stick a package up on PyPI. Not everything needs to be in the stdlib, but if you're going to argue that something should be, what better way than by pointing out what it's already doing? ChrisA From alexander.belopolsky at gmail.com Fri Mar 27 16:21:19 2015 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Fri, 27 Mar 2015 11:21:19 -0400 Subject: [Python-ideas] Top 10 Python modules that need a redesign Was: Geo coordinates conversion in stdlib In-Reply-To: References: Message-ID: On Fri, Mar 27, 2015 at 10:54 AM, anatoly techtonik wrote: > Where have you been when PEP 3108 was discussed? I have not seen any >> other list of Python modules that needed a redesign, so I cannot tell >> what's on your top ten list. >> > > http://sayspy.blogspot.com/2009/07/informal-poll-what-python-stdlib.html > Interesting. I did not see this back in the day. The top entry (urllib and friends) makes sense and there were heavily redesigned in Python 3. I am surprised that distutils redesign got less support than logging and datetime. I suspect that this may have to do with logging and datetime APIs not being PEP8 compliant. Popular votes tend to exaggerate the importance of trivial things such as the spelling of class and method names and brush off more subtle, but important design flaws. -------------- next part -------------- An HTML attachment was scrubbed... URL: From alexander.belopolsky at gmail.com Fri Mar 27 17:55:36 2015 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Fri, 27 Mar 2015 12:55:36 -0400 Subject: [Python-ideas] Top 10 Python modules that need a redesign Was: Geo coordinates conversion in stdlib In-Reply-To: References: Message-ID: On Fri, Mar 27, 2015 at 12:13 PM, anatoly techtonik wrote: > > Here is something that can be used as an example that it is not about PEP8 https://code.google.com/p/rainforce/wiki/WartsOfPython#measure_time And it takes a lot of energy to collect something like that for the reference. Well, in that document, I see the total of four "warts" related to the datetime module. If four warts bring a module to the top of the "worst designed stdlib modules" list, it can only mean that stdlib is almost perfect! For each "wart" the author has a "What can be done?" section, but no suggested solution involves a major redesign of the datetime module. The author complains about non-obviousness of strftime and indeed, for users without C background, neither name nor semantics is familiar. But the proposed solution >>> time.format('{{hours}}:{{minutes}}:{{seconds}}', 1090) '00:18:10' Does not look like a big win over the already available solution: >>> '{t.hour}:{t.minute}:{t.second}'.format(t=datetime.now()) '12:37:38' Overall, while I agree that there are a few warts in the datetime module, I have not seen any serious enough to warrant a major redesign or even any non backward compatible changes. Maintaining backward compatibility in the datetime module is indeed non-trivial, but this is the way I think it should be developed. -------------- next part -------------- An HTML attachment was scrubbed... URL: From szport at gmail.com Fri Mar 27 18:57:28 2015 From: szport at gmail.com (Zaur Shibzukhov) Date: Fri, 27 Mar 2015 20:57:28 +0300 Subject: [Python-ideas] A mutable alternative to namedtuple: recordarray -> recordclass: Message-ID: --- *Zaur Shibzukhov* 2015-03-27 16:17 GMT+03:00 Zaur Shibzukhov : > > > --- > *Zaur Shibzukhov* > > > 2015-03-27 7:40 GMT+03:00 Andrew Barnert : > >> On Mar 26, 2015, at 21:26, Zaur Shibzukhov wrote: >> >> 2015-03-27 7:08 GMT+03:00 Joonas Liik : >> >>> namedlist perhaps? :) >>> >>> if you want:"namedtuple, but mutable. " then namedlist seems like the >>> obvious alternative.. >>> >> >> The name 'namedlist' is already used in >> https://pypi.python.org/pypi/namedlist, 'mutabletuple' -- too in >> https://pypi.python.org/pypi/mutabletuple. >> >> >> If you're trying to provide the same concept, why use a completely >> unrelated name? That's like saying 'I want an ordered set, but there's >> already an "orderedset" on PyPI so I went with "sortedsequence"'. >> >> The name recordarray is means that it 1) array of objects and 2) record >> with access to fields by attributes. >> >> >> But how is being "an array of objects" any different from what a tuple, >> list, array.array, bytearray, bytes, str, etc. already are? What's >> specifically array-like about this type as opposed to all of those? And >> what's specifically record-like about your type compared to namedtuple, >> Struct, or SimpleNamespace? >> >> > I am inclined to the fact that it's better to rename `objectarray` to > 'mutabletuple' in order to be explicit about what is that. > 'recordarray' is a factory function that make exactly the same as > 'namedtuple' factory does (except that it create subclass of 'mutabletuple' > and make '_replace' to update the 'self', not make a copy). So may be it's > better to call it as 'record' or 'recordtype', or even 'recordfactory'? > > After consideration of all arguments recordarray project was renamed tand moved to recordclass . -------------- next part -------------- An HTML attachment was scrubbed... URL: From jw14896.2014 at my.bristol.ac.uk Fri Mar 27 22:36:34 2015 From: jw14896.2014 at my.bristol.ac.uk (Jamie Willis) Date: Fri, 27 Mar 2015 21:36:34 +0000 Subject: [Python-ideas] Proposal for new augmented assignment operator for method calls Message-ID: (As posted on python-list) I would like to propose a new piece of syntax for the python language; .= In short, the operator is form of syntactic sugar, for instance consider the following code: hello = "hello world " > hello = hello.strip() This could be written as: hello = "hello world " > hello .= strip() In this slightly contrived example, the programmer saved (a small amount of) time when writing the code. With code with longer variable names, or lots of similar statements all in a row, this helps to keep code more concise, and could cut down on potential bugs due to typos (thanks to Skip Montanaro for pointing this out). The operator would be constricted to one method or field on the right-hand side, which must belong to the object on the left hand side. To provide another example, consider this basic LinkedList class: class LinkedList: > > def __init__(self, data, head=None): > self.data = data > self.next = None > self.head = head if head != None else self > def append(self, data): new_node = LinkedList(data, self.head) > self.next = new_node > return new_node > linked_list = LinkedList(7) > linked_list = linked_list.append(8) > linked_list = linked_list.append(9) > loop_node = linked_list.head while loop_node != None: > print(loop_node.data) > loop_node = loop_node.next Taking a look at "*linked_list = linked_list.append(8)*", we could write this as: linked_list .= append(8) under the new syntax and then "*loop_node = loop_node.next*" could simply become: loop_node .= next Would this idea get much support if submitted as a PEP? What are the potential concerns I could consider with the syntax (Skip has already pointed out that the "dot" is pretty invisible in the operator, so perhaps a different character could be used)? Thanks, Jamie -------------- next part -------------- An HTML attachment was scrubbed... URL: From alexander.belopolsky at gmail.com Fri Mar 27 23:01:17 2015 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Fri, 27 Mar 2015 18:01:17 -0400 Subject: [Python-ideas] Proposal for new augmented assignment operator for method calls In-Reply-To: References: Message-ID: On Fri, Mar 27, 2015 at 5:36 PM, Jamie Willis wrote: > > I would like to propose a new piece of syntax for the python language; .= > > In short, the operator is form of syntactic sugar, for instance consider the following code: > >> hello = "hello world " >> hello = hello.strip() > > > This could be written as: > >> hello = "hello world " >> hello .= strip() "Syntax must not look like grit on Tim's screen!" -1 -------------- next part -------------- An HTML attachment was scrubbed... URL: From abarnert at yahoo.com Sat Mar 28 00:13:46 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Fri, 27 Mar 2015 16:13:46 -0700 Subject: [Python-ideas] recorarray: a mutable alternative to namedtuple In-Reply-To: References: <6b53b718-0220-48d5-b5b5-ded2fa9e867c@googlegroups.com> <0CF24FB3-2E88-4A44-840F-358A4498B4EF@yahoo.com> Message-ID: On Mar 27, 2015, at 06:22, Joao S. O. Bueno wrote: > > On 27 March 2015 at 01:40, Andrew Barnert > wrote: >> But how is being "an array of objects" any different from what a tuple, >> list, array.array, bytearray, bytes, str, etc. already are? What's >> specifically array-like about this type as opposed to all of those? And >> what's specifically record-like about your type compared to namedtuple, >> Struct, or SimpleNamespace? > > Acutally, on my understanding, the request on this thread is for > something that is quite concrete, > existing in other languages, and that can be done in Python in a few > lines, but is not in > the stdlib: > > The Python equivalent of a C Struct. But a C struct is not a subtype of, or substitutable for, a C array. It's not indexable. And the same is true with the equivalents in other languages. In fact, the dichotomy between struct--heterogeneous collection of fixed named fields--and array--homogeneous collection of indexed fields--goes back way before C. So, if you want the equivalent of a C struct, there's no reason to make it an iterable in Python. And a class already is the Python of a C struct, it's just that it can do _more_ than a C struct. A language like C++ that wants to share code and values with C has to bend over backward to make it possible to write a C++ struct (or class) that doesn't use any of its extra features and is therefore exactly equivalent to a C struct, but Python has no need for that. (Except when you actually do want to share values with C code--but for that case, we've got ctypes.Struct, which is exactly what you want in that situation.) > Just that. > An easy to create class, with named fields, Which is easy to do: just create a class, and create its fields in the __init__ method (or, in some cases, it's acceptable to use class attributes as "default values" for instance attributes). > with possible > type-enforcement for those fields. Of course namedtuple doesn't have type-enforcement for the fields. I'm not sure whether you're talking about MyPy static type checking, or runtime checking, but either way, it's easier to add onto a regular class than to a namedtuple-like class factory. > Or maybe it _does_ exist in Python, and it is a matter of having a > nice example in the docs: > for example a "blank" class with "__slots__" would do it. A blank class without __slots__ can also do it. There are times when __slots__ are useful, but usually you're fine with just a plain __dict__. Encouraging people to use it when they have no need for it just because it's more like idiomatic C would be a bad idea. (It's like encouraging people to use @property to get something more like idiomatic .NET or ObjC, when actually they should just be writing idiomatic Python and using attributes directly.) > Or a blank class with slots that could serialize and deserialize > itself to a sequence > in a seamless way. Why do you want to serialize and deserialize to a sequence? A C struct can't do that, and neither can equivalent types in other languages. > class Base: > __slots__ = () > def __init__(self, seq=None): > if not seq: return > for attr, val in zip(self.slots, seq): > setattr(self, attr, val) > def __iter__(self): > for attr in self.__slots__: > yield getattr(self, attr) > > > def NamedList(name, fields): > ... # split string with space separated fields, and other niceities here > return type(name, (Base,), dict(__slots__=fields)) > > And 10-15 more lines if one wants type-checking, default values, > __repr__ into that. Default values and __repr__ are _also_ not part of a C struct. So, again, if what you're looking for is the equivalent of a C struct, you can replace all of the above with: def Base: pass If you want other features that C structs don't have, then yes, you may have to write them, but the same is true in C (and, in fact, it's clumsier and more difficult in C). > Ithink getting a proper recipe for this, and > publicizing it on the documentation ecosystem is enough - maybe a > Pypi module adding some more goodies - and if that would get any > traction - the usual consideration for inclusion could apply. From mrocklin at gmail.com Sat Mar 28 00:19:54 2015 From: mrocklin at gmail.com (Matthew Rocklin) Date: Fri, 27 Mar 2015 16:19:54 -0700 Subject: [Python-ideas] History on proposals for Macros? Message-ID: I want Python to have macros. This is obviously a hard sell. I'm willing to do some legwork to demonstrate value. What would a good proposal include? Are there good examples of failed proposals on this topic? Is the core team open to this topic? Thank you for your time, - Mathew Rocklin -------------- next part -------------- An HTML attachment was scrubbed... URL: From carl at oddbird.net Sat Mar 28 00:23:41 2015 From: carl at oddbird.net (Carl Meyer) Date: Fri, 27 Mar 2015 17:23:41 -0600 Subject: [Python-ideas] History on proposals for Macros? In-Reply-To: References: Message-ID: <5515E67D.6010102@oddbird.net> On 03/27/2015 05:19 PM, Matthew Rocklin wrote: > I want Python to have macros. This is obviously a hard sell. I'm > willing to do some legwork to demonstrate value. > > What would a good proposal include? Are there good examples of failed > proposals on this topic? > > Is the core team open to this topic? You probably want to have a look at MacroPy [1]. I don't think it was ever seriously proposed for core, though. Carl [1] https://github.com/lihaoyi/macropy -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 819 bytes Desc: OpenPGP digital signature URL: From luciano at ramalho.org Sat Mar 28 00:24:22 2015 From: luciano at ramalho.org (Luciano Ramalho) Date: Fri, 27 Mar 2015 20:24:22 -0300 Subject: [Python-ideas] History on proposals for Macros? In-Reply-To: References: Message-ID: On Fri, Mar 27, 2015 at 8:19 PM, Matthew Rocklin wrote: > I want Python to have macros. This is obviously a hard sell. I'm willing > to do some legwork to demonstrate value. You're probably aware of this "prior art", but anyway, it's worth a link. Looks solid to me: https://github.com/lihaoyi/macropy Cheers, Luciano -- Luciano Ramalho | Author of Fluent Python (O'Reilly, 2015) | http://shop.oreilly.com/product/0636920032519.do | Professor em: http://python.pro.br | Twitter: @ramalhoorg From alexander.belopolsky at gmail.com Sat Mar 28 00:28:23 2015 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Fri, 27 Mar 2015 19:28:23 -0400 Subject: [Python-ideas] Proposal for new augmented assignment operator for method calls In-Reply-To: References: Message-ID: On Fri, Mar 27, 2015 at 5:36 PM, Jamie Willis wrote: > Would this idea get much support if submitted as a PEP? I should have read your post to the end before sending a witty reply. I think you are facing an uphill battle regardless of the syntax. In many ways even the augmented assignment operators that are already in the language are "un-pythonic" and many dark corners have been discovered since they were introduced. > What are the potential concerns I could consider with the syntax (Skip has > already pointed out that the "dot" is pretty invisible in the operator, so > perhaps a different character could be used)? @= would be a better choice, but @ has been taken for the matrix multiplication operator. -------------- next part -------------- An HTML attachment was scrubbed... URL: From luciano at ramalho.org Sat Mar 28 00:29:28 2015 From: luciano at ramalho.org (Luciano Ramalho) Date: Fri, 27 Mar 2015 20:29:28 -0300 Subject: [Python-ideas] recorarray: a mutable alternative to namedtuple In-Reply-To: References: <6b53b718-0220-48d5-b5b5-ded2fa9e867c@googlegroups.com> <0CF24FB3-2E88-4A44-840F-358A4498B4EF@yahoo.com> Message-ID: On Fri, Mar 27, 2015 at 8:13 PM, Andrew Barnert wrote: > So, if you want the equivalent of a C struct, there's no reason to make it an iterable in Python. Yes, there is: iterable unpacking. > Which is easy to do: just create a class, and create its fields in the __init__ method (or, in some cases, it's acceptable to use class attributes as "default values" for instance attributes). Boilerplate with lots of repetition, with little added value. For example, in a basic __init__ each attribute name usually occurs three times: as an argument name in the method declaration, and then twice when it's assigned to self. Ruby does much better, for example. Best, Luciano From abarnert at yahoo.com Sat Mar 28 01:21:05 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Fri, 27 Mar 2015 17:21:05 -0700 Subject: [Python-ideas] Top 10 Python modules that need a redesign Was: Geo coordinates conversion in stdlib In-Reply-To: References: Message-ID: <6F7DC05A-2563-4DD1-B20A-F8F1C9D37630@yahoo.com> On Mar 27, 2015, at 08:21, Alexander Belopolsky wrote: > > > On Fri, Mar 27, 2015 at 10:54 AM, anatoly techtonik wrote: >>> Where have you been when PEP 3108 was discussed? I have not seen any other list of Python modules that needed a redesign, so I cannot tell what's on your top ten list. >> >> http://sayspy.blogspot.com/2009/07/informal-poll-what-python-stdlib.html > > Interesting. I did not see this back in the day. The top entry (urllib and friends) makes sense and there were heavily redesigned in Python 3. I am surprised that distutils redesign got less support than logging and datetime. I'm not sure how much weight to put on an informal poll of 176 people self-selected as readers of a particular blog in the first place, but... Back in those days, I think most people had no idea what they wanted from a redesign of distutils, and until at least one of the competing projects to extend/fix/replace it was ready for prime time (at least design-wise) the idea of changing the stdlib to follow one of them was a bit scary. > I suspect that this may have to do with logging and datetime APIs not being PEP8 compliant. Popular votes tend to exaggerate the importance of trivial things such as the spelling of class and method names and brush off more subtle, but important design flaws. Interesting point. And I think you're right--but I think you can take it farther. Even beyond the PEP 8 renaming, what kinds of things did people really want from those modules? People also wanted the 90%-compatible formatting/parsing functions to be 100% Java-compatible for logging and 100% my-plafform's-C-lib-compatible for datetime. And they wanted to be able to plug pytz into datetime without making it so easy to write what looks like correct timezone-aware code but actually isn't. And they wanted easier conversion between datetime's types and those in time and elsewhere. Those changes aren't as trivial as PEP 8 renaming, but they're still simple to express, concrete, and unambiguous (while still potentially requiring a backward-incompatible change). Who wouldn't vote for that? -------------- next part -------------- An HTML attachment was scrubbed... URL: From abarnert at yahoo.com Sat Mar 28 02:09:03 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Fri, 27 Mar 2015 18:09:03 -0700 Subject: [Python-ideas] recorarray: a mutable alternative to namedtuple In-Reply-To: References: <6b53b718-0220-48d5-b5b5-ded2fa9e867c@googlegroups.com> <0CF24FB3-2E88-4A44-840F-358A4498B4EF@yahoo.com> Message-ID: <7ABB2084-83EC-4090-9B91-1A4459640FA8@yahoo.com> On Mar 27, 2015, at 16:29, Luciano Ramalho wrote: > > On Fri, Mar 27, 2015 at 8:13 PM, Andrew Barnert > wrote: >> So, if you want the equivalent of a C struct, there's no reason to make it an iterable in Python. > > Yes, there is: iterable unpacking. Why? You can't do the equivalent in C or any of its descendants (or most other languages with a struct/record type, or most pedagogical or theoretical struct/record concepts). Nor can you do anything even vaguely similar. So why would anyone expect that "the equivalent of a C struct" in Python should be able to do something that a C struct, and its equivalents in other languages, can't? Also, the desire to _not_ have to use iterable unpacking is why we have namedtuple (and structseq in the C API) in the first place: to make tuples that can be used as records, not the other way around. A namedtuple stat result allows your users to access the fields by name instead of by index, which not only makes their code more readable, it also means stat can return different sets of extra fields on different platforms and in new versions without breaking their code. Even in C, this is important: because you access the PyObject fields by name, I can hand you a PyList* cast to a PyObject* and you can use it; if C allowed you to access it by iterable unpacking and you did so, I'd instead have to copy the PyObject fields of the PyList into a new PyObject that didn't have any extra fields. >> Which is easy to do: just create a class, and create its fields in the __init__ method (or, in some cases, it's acceptable to use class attributes as "default values" for instance attributes). > > Boilerplate with lots of repetition, with little added value. For > example, in a basic __init__ each attribute name usually occurs three > times: as an argument name in the method declaration, and then twice > when it's assigned to self. Ruby does much better, for example. Let's compare some C code and the equivalent Python: struct Person { const char *name; int age; } struct Person person_make(const char *name, int age) { struct Person p; p.name = strdup(name); p.age = age; return p; } class Person def __init__(self, name: str, age: int): self.name = name self.age = age You really think that this is not like a C struct because it has too much boilerplate compared to the C equivalent? Of course it's trivial to wrap up that boilerplate if you're going to create 20 of these. And to add in other functionality that C structs (and, except for the first, Python namedtuples) don't have that your project needs, like a nice repr, default values, runtime type checking, a JSON serialization schema, an ORM mapping, an HTML form representation, etc. If you really want to add in being a sequence, you can add that too--but again, what's the use case for that? It's certainly not being more like a C struct. From abarnert at yahoo.com Sat Mar 28 02:31:12 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Fri, 27 Mar 2015 18:31:12 -0700 Subject: [Python-ideas] Proposal for new augmented assignment operator for method calls In-Reply-To: References: Message-ID: On Mar 27, 2015, at 14:36, Jamie Willis wrote: > > (As posted on python-list) > > I would like to propose a new piece of syntax for the python language; .= > > In short, the operator is form of syntactic sugar, for instance consider the following code: > >> hello = "hello world " >> hello = hello.strip() > > This could be written as: > >> hello = "hello world " >> hello .= strip() > > In this slightly contrived example, the programmer saved (a small amount of) time when writing the code. With code with longer variable names, or lots of similar statements all in a row, this helps to keep code more concise, and could cut down on potential bugs due to typos (thanks to Skip Montanaro for pointing this out). > > The operator would be constricted to one method or field on the right-hand side, which must belong to the object on the left hand side. > > To provide another example, consider this basic LinkedList class: > >> class LinkedList: >> >> def __init__(self, data, head=None): >> self.data = data >> self.next = None >> self.head = head if head != None else self >> >> def append(self, data): >> new_node = LinkedList(data, self.head) >> self.next = new_node >> return new_node >> >> linked_list = LinkedList(7) >> linked_list = linked_list.append(8) >> linked_list = linked_list.append(9) >> >> loop_node = linked_list.head >> while loop_node != None: >> print(loop_node.data) >> loop_node = loop_node.next This is a pretty weird variation on the linked list data structure that's hard to imagine anyone actually using. And it seems like it's only because you've made it so weird that you need an append function that (unpythonically) mutates self and returns some other but related object. And it's only because you've made that append function that you need this new syntax at all. It's certainly possible that a more reasonable class would also give you a use case for this feature, but it's hard to guess without seeing one. Also, a much more Pythonic way to write your loop would be to write a list iterating function (whether an __iter__ method, or an external function) and just loop over that, which means the ugly "node = node.next" would either never appear, or appear only once inside that iterator rather than all over user code. Your improved version is only marginally better: loop_node = linked_list.head while loop_node != None: print(loop_node.data) loop_node .= next While doing it right gives you: for loop_node in linked_list: print(loop_node.data) (Also, as a side note, you shouldn't compare to None with !=; use is not.) -------------- next part -------------- An HTML attachment was scrubbed... URL: From jsbueno at python.org.br Sat Mar 28 02:45:38 2015 From: jsbueno at python.org.br (Joao S. O. Bueno) Date: Fri, 27 Mar 2015 22:45:38 -0300 Subject: [Python-ideas] recorarray: a mutable alternative to namedtuple In-Reply-To: <7ABB2084-83EC-4090-9B91-1A4459640FA8@yahoo.com> References: <6b53b718-0220-48d5-b5b5-ded2fa9e867c@googlegroups.com> <0CF24FB3-2E88-4A44-840F-358A4498B4EF@yahoo.com> <7ABB2084-83EC-4090-9B91-1A4459640FA8@yahoo.com> Message-ID: On 27 March 2015 at 22:09, Andrew Barnert wrote: > Of course it's trivial to wrap up that boilerplate if you're going to create 20 of these. And to add in other functionality that C structs (and, except for the first, Python namedtuples) don't have that your project needs, like a nice repr, default values, runtime type checking, a JSON serialization schema, an ORM mapping, an HTML form representation, etc. So -that is the point - it is trivial to do away with the boiler plate, as I've shown on the other message, - but there is no way to do it in the stdlib, so it is a wheel that is reinvented everytime. NamedTuples are a way to do _almost_ that: make it tirvial to create a class with only this fixed set of attributes, with as little boilerplate as one can think of - but one can't change the attributes on an instance of it. That is why the start of the thread is about a "mutable named tuple" - not because it is a tuple - but because it creates a basic class with fixed attributes that works nicely, with minimal boiler plate. Rethinking at my example, I think it does fit exactly in "too small to be an external dependence in itself, and with too many subtle behaviors to get done right again and again in several projects". js -><- From rosuav at gmail.com Sat Mar 28 03:01:30 2015 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 28 Mar 2015 13:01:30 +1100 Subject: [Python-ideas] recorarray: a mutable alternative to namedtuple In-Reply-To: <7ABB2084-83EC-4090-9B91-1A4459640FA8@yahoo.com> References: <6b53b718-0220-48d5-b5b5-ded2fa9e867c@googlegroups.com> <0CF24FB3-2E88-4A44-840F-358A4498B4EF@yahoo.com> <7ABB2084-83EC-4090-9B91-1A4459640FA8@yahoo.com> Message-ID: On Sat, Mar 28, 2015 at 12:09 PM, Andrew Barnert wrote: > class Person > def __init__(self, name: str, age: int): > self.name = name > self.age = age > > Of course it's trivial to wrap up that boilerplate if you're going to create 20 of these. Here's a crazy thought: you could use functools.wraps() to abuse **kwargs. def make_attributes(func): @functools.wraps(func) def inner(self, **args): self.__dict__.update(args) inner(self, **args) return inner class Person: @make_attributes def __init__(self, *, name: str, age: int): pass Thanks to wraps(), you still have your parameter names for introspection and help() and so on. Thanks to **args, you can do bulk operations on all the args. It's a bit naughty (and it does preclude positional args, though a little bit more work in the decorator could support that too), but it would work..... ChrisA From abarnert at yahoo.com Sat Mar 28 03:22:58 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Fri, 27 Mar 2015 19:22:58 -0700 Subject: [Python-ideas] History on proposals for Macros? In-Reply-To: <5515E67D.6010102@oddbird.net> References: <5515E67D.6010102@oddbird.net> Message-ID: <662C946E-992A-46EB-8B45-CACCB69D5B57@yahoo.com> On Mar 27, 2015, at 16:23, Carl Meyer wrote: > >> On 03/27/2015 05:19 PM, Matthew Rocklin wrote: >> I want Python to have macros. This is obviously a hard sell. I'm >> willing to do some legwork to demonstrate value. >> >> What would a good proposal include? Are there good examples of failed >> proposals on this topic? >> >> Is the core team open to this topic? > > You probably want to have a look at MacroPy [1]. > > I don't think it was ever seriously proposed for core, though. I think it might be worth looking at what smaller core changes could allow MacroPy to be better or simpler than it is. Some of the changes to the import mechanism and the ast module that we've already had between 3.0 and 3.4 have done that, and also have the benefit of making a Python implementation easier to understand and to hack on. I suspect the same would be true for other possible changes. And then, once those changes are in and MacroPy is as simple, pleasant, and wart-free as it can be, it would probably be an easier sell (although still maybe not easy...) to get something like it incorporated into the core compiler and/or importer instead of as an external import hook. Of course you may look at MacroPy and think, "This is way too big and complicated, I just want to be able to do what Language X had without learning all this other stuff", but then it'll at least help focus the ideas on what you do and don't want, right? Anyway, I think people who proposed macros in the past have had two major problems. First, they didn't think through the design. Lisp-style macros don't work in a language with syntax more complex than s-expressions. And making them partially text-based doesn't help either, given the way whitespace works in Python. And once you get to the point where you're doing AST transforms with the ast module, that doesn't really feel like macros anymore. If you instead come at it from the perspective of Dylan or one of the ML-with-macros variants, where what you're looking for is a way to write AST transformers in a declarative language that fits into the host language, you might get a lot father. Second, they didn't think through the marketing. Ask any famous Lispers why macros are important, and the answer is that they let you reprogram the language first, into a language that's easier to write your actual program in. That's a blatantly anti-Pythonic thing to do. You often can't read good Lisp code until you first learn the new, project-specific language they've built on Lisp, and nobody wants that for Python. Even smaller versions of that, like being able to create new flow control syntax, etc. are an anti-selling-point for Python. So, you need to think of a new way to sell macros that explicitly disavows the idea of creating new syntax, but still shows how macros can usefully do things that look and feel Pythonic but can't be done in Python. From steve at pearwood.info Sat Mar 28 14:37:09 2015 From: steve at pearwood.info (Steven D'Aprano) Date: Sun, 29 Mar 2015 00:37:09 +1100 Subject: [Python-ideas] recorarray: a mutable alternative to namedtuple In-Reply-To: References: <6b53b718-0220-48d5-b5b5-ded2fa9e867c@googlegroups.com> <0CF24FB3-2E88-4A44-840F-358A4498B4EF@yahoo.com> Message-ID: <20150328133709.GE25453@ando.pearwood.info> On Fri, Mar 27, 2015 at 04:13:46PM -0700, Andrew Barnert wrote: > On Mar 27, 2015, at 06:22, Joao S. O. Bueno wrote: [...] > > The Python equivalent of a C Struct. > > But a C struct is not a subtype of, or substitutable for, a C array. > It's not indexable. And the same is true with the equivalents in other > languages. In fact, the dichotomy between struct--heterogeneous > collection of fixed named fields--and array--homogeneous collection of > indexed fields--goes back way before C. So, if you want the equivalent > of a C struct, there's no reason to make it an iterable in Python. Joao said "The Python equivalent of a C struct", not "a C struct". Python is not C, and Python data types are not limited to what C does. Python strings aren't limited to C null-delimited strings, and Python ints aren't limited to what C ints can do. I think the original thread was quite explicit about what is wanted: something like a mutable equivalent to namedtuple. Namedtuples are used in Python where C would use a struct, or Pascal a record, except that namedtuples (being tuples) are immutable. I think it's quite reasonable to want a mutable version. Effectively, namedtuple is just a convenience function for wrapping up a bunch of nice-to-have but not essential functionality around an immutable struct. Python got by with unnamed tuples for over a decade, so it's not like we *have* to have namedtuples. But having got them, would we go back to using regular tuples as a struct? Hell no. Having named fields is so much better. > And a class already is the Python of a C struct, it's just that it can > do _more_ than a C struct. This is why it is unfair to insist that a Python equivalent of a C struct be limited to what C structs do. > > Just that. > > An easy to create class, with named fields, > > Which is easy to do: just create a class, and create its fields in the > __init__ method (or, in some cases, it's acceptable to use class > attributes as "default values" for instance attributes). If this is so easy, why we have namedtuple *and* SimpleNamespace in the standard library. Are they both mistakes? SimpleNamespace is especially interesting. The docs say: "However, for a structured record type use namedtuple() instead." https://docs.python.org/3/library/types.html#types.SimpleNamespace which is great if you want an *immutable* structured record type, but not if you want a mutable one. Which brings us back to where this thread started: a request for a mutable version of namedtuple. That's trickier than namedtuple, because we don't have a mutable version of a tuple to inherit from. Lists won't do the job, because they have a whole lot of functionality that are inappropriate, e.g. sort, reverse, pop methods. That makes it harder to create a mutable structured record type, not simpler. Think about the functional requirements: - it should be semantically a struct, not a list or array; - with a fixed set of named fields; - fields should be ordered: a record with fields foo and bar is not the same as a record with fields bar and foo; - accessing fields by index would be a Nice To Have, but not essential; - but iteration is essential, for sequence unpacking; - values in the fields must be mutable; - it should support equality, but not hashing (since it is mutable); - it must have a nice repr and/or str; - being mutable, it may directly or indirectly contain a reference to itself (e.g. x.field = x) so it needs to deal with that correctly; - support for pickle; - like namedtuple, it may benefit from a handful of methods such as '_asdict', '_fields', '_make', '_replace' or similar. Does this sound easy to write? Well, sure, in the big picture, it's hardly a 100,000 line application. But it's not a trivial class. -- Steve From rob.cliffe at btinternet.com Sat Mar 28 15:56:02 2015 From: rob.cliffe at btinternet.com (Rob Cliffe) Date: Sat, 28 Mar 2015 14:56:02 +0000 Subject: [Python-ideas] Format character to center strings Message-ID: <5516C102.1040608@btinternet.com> As a string can be left-justified (within a given width) using e.g. '%-20s' % s and right-justified using '%+20s' % s # or '%20s' % s why not allow a string to be centered using '%=20s' %s Rob Cliffe From mrocklin at gmail.com Sat Mar 28 17:53:48 2015 From: mrocklin at gmail.com (Matthew Rocklin) Date: Sat, 28 Mar 2015 09:53:48 -0700 Subject: [Python-ideas] History on proposals for Macros? In-Reply-To: References: Message-ID: Responding to comments off list: I'm not referring to C-style preprocessor macros, I'm referring to macros historically found in functional languages and commonly found in many user-targeted languages built in the last few years. The goal is to create things that look like functions but have access to the expression that was passed in. Some examples where this is useful: plot(year, miles / gallon) # Plot with labels determined by input-expressions, e.g. miles/gallon assertRaises(ZeroDivisionError, 1/0) # Evaluate the rhs 1/0 within assertRaises function, not before run_concurrently(f(x), f(y), f(z)) # Run f three times in three threads controlled by run_concurrently Generally one constructs something that looks like a function but, rather than receiving a pre-evaluated input, receives a syntax tree along with the associated context. This allows that function-like-thing to manipulate the expression and to control the context in which the evaluation occurs. There are lots of arguments against this, mostly focused around potential misuse. I'm looking for history of such arguments and for a general "Yes, this is theoretically possible" or "Not a chance in hell" from the community. Both are fine. Cheers, Matthew On Fri, Mar 27, 2015 at 4:24 PM, Luciano Ramalho wrote: > On Fri, Mar 27, 2015 at 8:19 PM, Matthew Rocklin > wrote: > > I want Python to have macros. This is obviously a hard sell. I'm > willing > > to do some legwork to demonstrate value. > > You're probably aware of this "prior art", but anyway, it's worth a > link. Looks solid to me: > > https://github.com/lihaoyi/macropy > > Cheers, > > Luciano > > -- > Luciano Ramalho > | Author of Fluent Python (O'Reilly, 2015) > | http://shop.oreilly.com/product/0636920032519.do > | Professor em: http://python.pro.br > | Twitter: @ramalhoorg > -------------- next part -------------- An HTML attachment was scrubbed... URL: From flying-sheep at web.de Sat Mar 28 18:07:32 2015 From: flying-sheep at web.de (Philipp A.) Date: Sat, 28 Mar 2015 17:07:32 +0000 Subject: [Python-ideas] Format character to center strings In-Reply-To: <5516C102.1040608@btinternet.com> References: <5516C102.1040608@btinternet.com> Message-ID: because you should use '{:^20}'.format(s) instead of the obsolete % stuff. Rob Cliffe schrieb am Sa., 28. M?rz 2015 um 16:01 Uhr: > As a string can be left-justified (within a given width) using e.g. > '%-20s' % s > and right-justified using > '%+20s' % s # or '%20s' % s > why not allow a string to be centered using > '%=20s' %s > > Rob Cliffe > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dwblas at gmail.com Sat Mar 28 18:11:49 2015 From: dwblas at gmail.com (David Blaschke) Date: Sat, 28 Mar 2015 10:11:49 -0700 Subject: [Python-ideas] Format character to center strings In-Reply-To: <5516C102.1040608@btinternet.com> References: <5516C102.1040608@btinternet.com> Message-ID: Centered=same amount of white space on the left and to the right of the text. This depends on the font being used, especially whether it is a fixed or proportional font. So you would possibly have to also specify the font face and font size being used. Programming is not like a word processor where the font is specified within each document. Run this program to see the display difference between two "sentences" that are the same length, and between proportional and fixed fonts. #! /usr/bin/python try: import Tkinter as tk ## Python 2.x except ImportError: import tkinter as tk ## Python 3.x class DifferentFonts(): def __init__(self): self.top=tk.Tk() lit_1="Display lots of i's and l's-iiilll" lit_2="This is the same length as 1-wwwqq" ctr = 0 for each_font in (('Verdana', 12), ('Fixed', 12)): tk.Label(self.top, text=each_font[0]+"-"*30, font=each_font).grid(row=ctr) ctr += 1 tk.Label(self.top, text=lit_1, font=each_font).grid(row=ctr, sticky="w") ctr += 1 tk.Label(self.top, text=lit_2, font=each_font).grid(row=ctr, sticky="w") ctr += 1 tk.Button(self.top, text="Quit", bg="orange", command=self.top.quit).grid(row=20) self.top.mainloop() DifferentFonts() On 3/28/15, Rob Cliffe wrote: > As a string can be left-justified (within a given width) using e.g. > '%-20s' % s > and right-justified using > '%+20s' % s # or '%20s' % s > why not allow a string to be centered using > '%=20s' %s > > Rob Cliffe > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- Only when you actually get to the state where there is neither delusion nor enlightenment are you finally comfortable...Foyan From alexander.belopolsky at gmail.com Sat Mar 28 18:12:16 2015 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Sat, 28 Mar 2015 13:12:16 -0400 Subject: [Python-ideas] Format character to center strings In-Reply-To: References: <5516C102.1040608@btinternet.com> Message-ID: On Sat, Mar 28, 2015 at 1:07 PM, Philipp A. wrote: > because you should use '{:^20}'.format(s) instead of the obsolete % stuff. > "% stuff" is not "obsolete", but it follows C standard by design and any new features that do not come from C will likely be rejected. -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Sat Mar 28 18:26:24 2015 From: steve at pearwood.info (Steven D'Aprano) Date: Sun, 29 Mar 2015 04:26:24 +1100 Subject: [Python-ideas] History on proposals for Macros? In-Reply-To: References: Message-ID: <20150328172624.GG25453@ando.pearwood.info> On Sat, Mar 28, 2015 at 09:53:48AM -0700, Matthew Rocklin wrote: [...] > The goal is to create things that look like functions but have access to > the expression that was passed in. > assertRaises(ZeroDivisionError, 1/0) # Evaluate the rhs 1/0 within > assertRaises function, not before > Generally one constructs something that looks like a function but, rather > than receiving a pre-evaluated input, receives a syntax tree along with the > associated context. This allows that function-like-thing to manipulate the > expression and to control the context in which the evaluation occurs. How will the Python compiler determine that assertRaises should receive the syntax tree rather than the evaluated 1/0 expression (which of course will raise)? The information that assertRaises is a "macro" is not available at compile time. I really like the idea of delaying the evaluation of certain expressions until a time of the caller's choosing, but I don't see how to make that work automatically. Of course we can do it manually by wrapping the expression in a function, or by writing it as a string and compiling it with compile() for later eval'ing, but the sort of thing you show above strikes me as fundamentally incompatible with Python's execution model. Happy to be proven wrong though. -- Steve From steve at pearwood.info Sat Mar 28 18:37:51 2015 From: steve at pearwood.info (Steven D'Aprano) Date: Sun, 29 Mar 2015 04:37:51 +1100 Subject: [Python-ideas] Format character to center strings In-Reply-To: References: <5516C102.1040608@btinternet.com> Message-ID: <20150328173751.GH25453@ando.pearwood.info> On Sat, Mar 28, 2015 at 10:11:49AM -0700, David Blaschke wrote: > Centered=same amount of white space on the left and to the right of > the text. This depends on the font being used, especially whether it > is a fixed or proportional font. You're over-thinking it. Python already offers two ways of centering strings, with the distance measured in number-of-characters rather than pixels or points. py> "Hello world!".center(20) ' Hello world! ' py> "{:^20}".format("Goodbye everyone!") ' Goodbye everyone! ' Left and right justification also pad the string to a fixed width in characters, not in pixels. -- Steve From ericsnowcurrently at gmail.com Sat Mar 28 19:11:34 2015 From: ericsnowcurrently at gmail.com (Eric Snow) Date: Sat, 28 Mar 2015 12:11:34 -0600 Subject: [Python-ideas] recorarray: a mutable alternative to namedtuple In-Reply-To: <20150328133709.GE25453@ando.pearwood.info> References: <6b53b718-0220-48d5-b5b5-ded2fa9e867c@googlegroups.com> <0CF24FB3-2E88-4A44-840F-358A4498B4EF@yahoo.com> <20150328133709.GE25453@ando.pearwood.info> Message-ID: On Sat, Mar 28, 2015 at 7:37 AM, Steven D'Aprano wrote: > Effectively, namedtuple is just a convenience function for wrapping up a > bunch of nice-to-have but not essential functionality around an > immutable struct. Python got by with unnamed tuples for over a decade, > so it's not like we *have* to have namedtuples. But having got them, > would we go back to using regular tuples as a struct? Hell no. Having > named fields is so much better. +1, though it doesn't *necessarily* follow that a mutable equivalent is a good idea. This (and related threads) imply there is at least some support for the new type in principle. I haven't followed the threads too closely so I've missed any mention of solid pythonic use cases that would give the idea much more solid footing. However, I have seen references to prior art on the cheeseshop which may be used to provide harder evidence (both of support and of solid use cases). Regardless, I concur that there are many cases where types and functions have been added to the stdlib that weren't strictly necessary. Perhaps if those proposals had come from someone else or when the mood on python-dev was different then they would not have been added. That is what has happened with numerous other we-have-gotten-by-without-it-so-why-add-it ideas (which may also have proven themselves as namedtuple has). Ultimately we have to be careful in this space because, as Raymond often reminds us, it really is important to make the effort to keep Python small enough to fit in people's brains (and in *some* regard we've probably failed there already). With the great efforts in the last year to improve packaging, the cheeseshop is increasingly the better place for new types and helpers to live. With that in mind, perhaps we should start adding a section to the bottom of relevant docs that contains links to vetted PyPI packages (and recipes) that provide extended capabilities. We've done this already in a few select places (e.g. the 3.2+ namedtuple docs). > If this is so easy, why we have namedtuple *and* SimpleNamespace > in the standard library. Are they both mistakes? > > SimpleNamespace is especially interesting. The docs say: > > "However, for a structured record type use namedtuple() instead." > > https://docs.python.org/3/library/types.html#types.SimpleNamespace As the person who wrote that I'll point out that I added it to help make the distinction clearer between the two. At the time there were concerns about the similarities and with users getting confused about which to use. I will argue that "record type" implies an archive of data, ergo immutable. "Structured" refers to being ordered and having attribute access. IMHO that statement is clear and helpful, but if it has proven otherwise we should consider improving it. In contrast, I see the proposal here as somewhat of a middle ground. Folks are looking for a factory mechanism that produces classes with slots and have both iteration (for unpacking) and index lookup. So something like this: class FixedClassMeta(type): # Ideally this would be a "classonly" method (like classmethod but # class-only) method on FixedClass and not need a metaclass. def subclass(base, name, *fields): # XXX validate fields first args = ', '.join(fields) body = '\n '.join('self.{0} = {0}'.format(f) for f in fields) code = """def __init__(self, {}):\n {}""".format(args, body) ns = {} exec(code, ns) class X(base): __slots__ = fields __init__ = ns['__init__'] X.__name__ = name X.__qualname__ = X.__qualname__.replace('X', name, 1) X.__doc__ = """...""" return X class FixedClass(metaclass=FixedClassMeta): __slots__ = () def __repr__(self): items = ("{}={!r}".format(f, getattr(self, f)) for f in self.__slots__) return "{}({})".format(self.__class__.__name__, ', '.join(items)) def __iter__(self): # for unpacking return (getattr(self, f) for f in self.__slots__) def __getitem__(self, index): field = self.__slots__[index] try: return getattr(self, field) except AttributeError: raise IndexError(index) # Index lookup exists for convenience, but assignment & deletion # are not in scope. def fixedClass(name, field_names): """A factory that produces classes with fixed, ordered attributes. The returned class has __slots__ set to the field names, as well as __iter__ (for unpacking) and __getitem__ implemented. """ if isinstance(field_names, str): fields = field_names.replace(',', ' ').split() else: fields = field_names return FixedClass.subclass(name, *fields) That said, I'm still not clear on what the use cases are. > Which brings us back to where this thread started: a request for a > mutable version of namedtuple. That's trickier than namedtuple, because > we don't have a mutable version of a tuple to inherit from. Lists won't > do the job, because they have a whole lot of functionality that are > inappropriate, e.g. sort, reverse, pop methods. > > That makes it harder to create a mutable structured record type, not > simpler. > > Think about the functional requirements: > > - it should be semantically a struct, not a list or array; This is the key point. It is a fixed-size class with iteration for unpacking and index lookup for convenience. A full-fledged mutable namedtuple doesn't make sense (without clear use cases). > > - with a fixed set of named fields; > > - fields should be ordered: a record with fields foo and bar is not the > same as a record with fields bar and foo; Ah, my example above would have to grow __eq__ then. > > - accessing fields by index would be a Nice To Have, but not essential; Exactly. Not Essential. > > - but iteration is essential, for sequence unpacking; This brings to mind a different proposal that has come up in the past (a separate "dunder" method for unpacking). Iteration seems out of place for here, but we need it for sequence unpacking. > > - values in the fields must be mutable; > > - it should support equality, but not hashing (since it is mutable); > > - it must have a nice repr and/or str; > > - being mutable, it may directly or indirectly contain a reference to > itself (e.g. x.field = x) so it needs to deal with that correctly; Ah, yes. "RuntimeError: maximum recursion depth exceeded". :) > > - support for pickle; > > - like namedtuple, it may benefit from a handful of methods such as > '_asdict', '_fields', '_make', '_replace' or similar. Perhaps. I think there are a few things we can learn from namedtuple that can be applied for this hypothetical new type/factory. And to add to your list: - performance should be a consideration since the apparent use cases relate to handling many of these as "records". Again, I'm not sold on the benefit of this over the existing alternatives. For records use namedtuple (with the _replace method for "mutation"). -eric From breamoreboy at yahoo.co.uk Sat Mar 28 19:11:44 2015 From: breamoreboy at yahoo.co.uk (Mark Lawrence) Date: Sat, 28 Mar 2015 18:11:44 +0000 Subject: [Python-ideas] Format character to center strings In-Reply-To: <5516C102.1040608@btinternet.com> References: <5516C102.1040608@btinternet.com> Message-ID: On 28/03/2015 14:56, Rob Cliffe wrote: > As a string can be left-justified (within a given width) using e.g. > '%-20s' % s > and right-justified using > '%+20s' % s # or '%20s' % s > why not allow a string to be centered using > '%=20s' %s > > Rob Cliffe > This http://bugs.python.org/issue23624 was rejected so I see two chances of this happening, zero or none. -- My fellow Pythonistas, ask not what our language can do for you, ask what you can do for our language. Mark Lawrence From ericsnowcurrently at gmail.com Sat Mar 28 19:14:02 2015 From: ericsnowcurrently at gmail.com (Eric Snow) Date: Sat, 28 Mar 2015 12:14:02 -0600 Subject: [Python-ideas] recorarray: a mutable alternative to namedtuple In-Reply-To: References: <6b53b718-0220-48d5-b5b5-ded2fa9e867c@googlegroups.com> <0CF24FB3-2E88-4A44-840F-358A4498B4EF@yahoo.com> <20150328133709.GE25453@ando.pearwood.info> Message-ID: On Sat, Mar 28, 2015 at 12:11 PM, Eric Snow wrote: > On Sat, Mar 28, 2015 at 7:37 AM, Steven D'Aprano wrote: >> - but iteration is essential, for sequence unpacking; > > This brings to mind a different proposal that has come up in the past > (a separate "dunder" method for unpacking). What ever became of that proposal? If I recall correctly, it had moderate support but the champion wasn't able to continue pursuing it. -eric From abarnert at yahoo.com Sat Mar 28 19:57:46 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Sat, 28 Mar 2015 11:57:46 -0700 Subject: [Python-ideas] recorarray: a mutable alternative to namedtuple In-Reply-To: <20150328133709.GE25453@ando.pearwood.info> References: <6b53b718-0220-48d5-b5b5-ded2fa9e867c@googlegroups.com> <0CF24FB3-2E88-4A44-840F-358A4498B4EF@yahoo.com> <20150328133709.GE25453@ando.pearwood.info> Message-ID: <7694034E-55FA-423D-A953-955D0B6215E0@yahoo.com> On Mar 28, 2015, at 06:37, Steven D'Aprano wrote: > >> On Fri, Mar 27, 2015 at 04:13:46PM -0700, Andrew Barnert wrote: >> On Mar 27, 2015, at 06:22, Joao S. O. Bueno wrote: > [...] >>> The Python equivalent of a C Struct. >> >> But a C struct is not a subtype of, or substitutable for, a C array. >> It's not indexable. And the same is true with the equivalents in other >> languages. In fact, the dichotomy between struct--heterogeneous >> collection of fixed named fields--and array--homogeneous collection of >> indexed fields--goes back way before C. So, if you want the equivalent >> of a C struct, there's no reason to make it an iterable in Python. > > Joao said "The Python equivalent of a C struct", not "a C struct". > Python is not C, and Python data types are not limited to what C does. > Python strings aren't limited to C null-delimited strings, and Python > ints aren't limited to what C ints can do. Sure, but nobody just invents random new features to add to int and then justifies them by saying "I want the equivalent of a C int" even though C int doesn't have those features. People invent features (like bit_length) to solve actual use cases, and justify them based on those use cases. Multiple people have asked "what do you want this for?", and the best answer anyone's given has been "the equivalent of a C struct". (That, and to prematurely and badly optimize memory usage.) Even worse, when I ask why specifically anyone wants this thing to be iterable, the answer is "to be the equivalent of a C struct", and that doesn't answer the question. [snip] >> And a class already is the Python of a C struct, it's just that it can >> do _more_ than a C struct. > > This is why it is unfair to insist that a Python equivalent of a C > struct be limited to what C structs do. I'm not saying a record type shouldn't be allowed to have any features that C structs don't, just that equivalency with C structs isn't an argument for features that C structs don't have. Some of the extra features are so obviously desirable that they probably don't need any argument--if you're going to build this thing, having a nice repr or not breaking pickle seems hard to argue against. But iterability is not that kind of obvious win. Also, how is it "unfair" to suggest that this thing should be limited in some ways? For example, two instances of the same class can have completely different fields; presumably two instances of the same record type really shouldn't. There's no reason it _couldn't_ be completely open like a general class, it's just that you usually don't want it to be. Similarly, there's no reason it couldn't be a sequence, but I don't think you usually want it to be. [snip] > Which brings us back to where this thread started: a request for a > mutable version of namedtuple. That's trickier than namedtuple, because > we don't have a mutable version of a tuple to inherit Sent from my iPhone > do the job, because they have a whole lot of functionality that are > inappropriate, e.g. sort, reverse, pop methods. > > That makes it harder to create a mutable structured record type, not > simpler. Sure. And that's the problem. If you want something that's "just like a sequence whose elements can be replaced but whose shape is fixed, except that the elements are also named", you run into the problem that Python doesn't have such a sequence type. It's a perfectly coherent concept, and there's no reason you could design a language around immutable, fixed-shape-mutable, and mutable-shape sequences instead of just the first and last, but that's not the way Python was designed. Should that be changed? Or is the only use for such a type to underlie this new type? > Think about the functional requirements: > > - it should be semantically a struct, not a list or array; > > - with a fixed set of named fields; > > - fields should be ordered: a record with fields foo and bar is not the > same as a record with fields bar and foo; Note that namedtuples are nominatively typed, not structurally--a record with fields foo and bar is not necessarily the same as another record with fields foo and bar. Ordering doesn't enter into it; they were defined separately, so they're separate types. Do you want the same behavior here, or the behavior your description implies instead? > - accessing fields by index would be a Nice To Have, but not essential; Why would that be nice to have? The record/sequence dichotomy has been fundamental to the design of languages since the earliest days, and it's still there in almost all languages. Maybe every language in the world is wrong--but if so, surely you can explain why? For structseq, there was a good reason: a stat result is a 7-tuple as well as being a record with 13-odd fields, because there was a pre-existing mass of code that used stat results as 7-tuples, but people also wanted to be able to access the newer or not-100%-portable fields. That's a great use case. And people have used structseq in other similar examples to migrate users painlessly from an early API that turned out to be too simple and limited. And namedtuple gives you a way to write APIs in a similar style that previously could only be (easily) written with a C extension, which is an obvious win. That's clearly not the case here--nobody has existing APIs that use a fixed-length but mutable sequence that they want to expand into something more flexible, because Python doesn't come with such a sequence type. Of course that's not the only use anyone's ever found for, respectively, structseq and namedtuple--e.g., converting to namedtuple turns out to be handy for cases where you want a record but some external API like SQL gives you a sequence, and that would probably be a good enough justification for namedtuple too. But what is the use that justifies this addition? (For example, if you need to take SQL rows as a sequence, mutate them by name, and then do something I can't imagine with them that requires them to still be a sequence, that would be a pretty good answer.) > - but iteration is essential, for sequence unpacking; Again, why is that essential? TOOWTDI isn't an iron-clad rule, but it's a good rule of thumb; adding a second way to access the members of a record that's both unique to Python and less Pythonic seems like a bad idea, unless there's some good reason that overbalances it in the other direction. Think of stat code: it's a lot more readable when you access the fields by name instead of by unpacking. Why wouldn't the same be true for, say, a Person record, or an Address record, or an ImageHeader record, or almost anything else you can imagine? (I can think of one particular special case where it might be nice: small, homogenous, essentially-sequence-like records like a Vector or Point or... Well, really just a Vector or Point. And they're clearly special. Both in C and in Python, you're often torn between storing them as an array or as a sequence, and you'll find different apps doing it each way. That isn't true for a Person or Address etc.) > - values in the fields must be mutable; > > - it should support equality, but not hashing (since it is mutable); > > - it must have a nice repr and/or str; > > - being mutable, it may directly or indirectly contain a reference to > itself (e.g. x.field = x) so it needs to deal with that correctly; > > - support for pickle; > > - like namedtuple, it may benefit from a handful of methods such as > '_asdict', '_fields', '_make', '_replace' or similar. > > > Does this sound easy to write? Well, sure, in the big picture, it's > hardly a 100,000 line application. But it's not a trivial class. > > > > -- > Steve > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ From rob.cliffe at btinternet.com Sat Mar 28 19:59:01 2015 From: rob.cliffe at btinternet.com (Rob Cliffe) Date: Sat, 28 Mar 2015 18:59:01 +0000 Subject: [Python-ideas] Format character to center strings In-Reply-To: References: <5516C102.1040608@btinternet.com> Message-ID: <5516F9F5.2020100@btinternet.com> On 28/03/2015 18:11, Mark Lawrence wrote: > On 28/03/2015 14:56, Rob Cliffe wrote: >> As a string can be left-justified (within a given width) using e.g. >> '%-20s' % s >> and right-justified using >> '%+20s' % s # or '%20s' % s >> why not allow a string to be centered using >> '%=20s' %s >> >> Rob Cliffe >> > > This http://bugs.python.org/issue23624 was rejected so I see two > chances of this happening, zero or none. > Except that that referred to a proposed change that would break existing code. This is a proposed new feature which wouldn't. (And for what it's worth, I too would prefer it to add an odd pad character on the right.) From abarnert at yahoo.com Sat Mar 28 20:50:09 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Sat, 28 Mar 2015 12:50:09 -0700 Subject: [Python-ideas] History on proposals for Macros? In-Reply-To: <20150328172624.GG25453@ando.pearwood.info> References: <20150328172624.GG25453@ando.pearwood.info> Message-ID: <36A4F7D4-20A4-4BE9-9D86-B423C13A7320@yahoo.com> On Mar 28, 2015, at 10:26, Steven D'Aprano wrote: > >> On Sat, Mar 28, 2015 at 09:53:48AM -0700, Matthew Rocklin wrote: >> [...] >> The goal is to create things that look like functions but have access to >> the expression that was passed in. > >> assertRaises(ZeroDivisionError, 1/0) # Evaluate the rhs 1/0 within >> assertRaises function, not before > >> Generally one constructs something that looks like a function but, rather >> than receiving a pre-evaluated input, receives a syntax tree along with the >> associated context. This allows that function-like-thing to manipulate the >> expression and to control the context in which the evaluation occurs. > > How will the Python compiler determine that assertRaises should receive > the syntax tree rather than the evaluated 1/0 expression (which of > course will raise)? The information that assertRaises is a "macro" is > not available at compile time. Well, it _could_ be available. At the time you're compiling a scope (a function, class, or top-level module code), if it uses an identifier that's the name of a macro in scope, the compiler expands the macro instead of compiling in a function call. Python already has a few things you can do at runtime that affect subsequent compilation, like __future__ statements; this isn't impossible. Of course that doesn't mean it's a good idea. It would require some pretty significant changes that may not be immediately obvious (e.g., the whole .pyc mechanism no longer works as-is if you can import macros from other modules, which isn't an issue for __future__ statements...). And debugging could be a nightmare--you now have to know what names are defined at call time, and also what names were defined at definition time, to trace through a function. And so on. > I really like the idea of delaying the evaluation of certain expressions > until a time of the caller's choosing, but I don't see how to make that > work automatically. Of course we can do it manually by wrapping the > expression in a function, That doesn't work for all uses of macros--a macro can swap two variables, or break or return, etc., and a higher-order function whose arguments are delayed by wrapping them in a function can't do that. But it does work for _many_ uses of macros, like this one. And if you look at languages with light lambda syntax, a lot of things that you'd naturally write as a macro in Lisp, you instead naturally write as a plain higher-order function, and the result is often more readable, not less. (Explicit is better than implicit, but you sometimes don't realize it if the explicitness forces you to write more boilerplate than actual code...) Consider: assertRaises(ZeroDivisionError, :1/0) That ":1/0" means the same thing as "lambda: 1/0". And now "wrapping the expression in a function" doesn't seem so bad. If such a light-lambda syntax reduced the desire for macros down to the point where it could be ignored, and if that desire weren't _already_ low enough that it can be ignored, it would be worth adding. I think the second "if" is where it fails, not the first, but I could be wrong. From jheiv at jheiv.com Sat Mar 28 23:35:17 2015 From: jheiv at jheiv.com (James Edwards) Date: Sat, 28 Mar 2015 18:35:17 -0400 Subject: [Python-ideas] "Loose" descriptors Message-ID: (I apologize in advance if this was covered previously or elsewhere; a fairly thorough search didn't yield anything.) Currently, we can implement a descriptor as follows: class VerboseDescriptor(object): def __init__(self, init): self.val = init def __get__(self, obj, typ=None): print("Get is %s" % (self.val)) return self.val def __set__(self, obj, val): print("Set to %s" % (val)) self.val = val class Holder(object): val = VerboseDescriptor(2) foo = Holder() _ = foo.val foo.val = 4 Which of course works as expected, displaying: Get is 2 Set to 4 But we can't achieve the same indirection without a "Holder"-type class. That is, we can't do the following: class VerboseDescriptor(object): def __init__(self, init): self.val = init def __get__(self, obj, typ=None): print("Get is %s" % (self.val)) return self.val def __set__(self, obj, val): print("Set to %s" % (val)) self.val = val val = VerboseDescriptor(2) _ = val val = 4 Or rather, we can, but what might be expected (the same functionality / output), is not what occurs. My understanding of the resolution of the first example is that python looks for `val` in `foo.__dict__`, then in `Holder.__dict__`, where it finds it, identifies that `Holder.val` implements `__get__`, and handles the indirection accordingly. So `foo.val = 4` ends up being something like `Holder.val.__set__(foo, 4)`. But my question is would it make sense to be able to do this sort of indirection for "loose" objects also, not just attributes of classes or instances? In other words, could descriptors be extended so that they perform the same sort of indirection in the second example as the first? I envision (but with no real knowledge of the implications of this) that just as python identifies that `val` in `Holder.__dict__` implements `__get__`, python could similarly identify that `val` in `globals()` (or `sys.modules[__name__].__dict__`, or wherever is more appropriate) implements `__get__` and handle the indirection. Of course, a syntax issue would arise if this were possible -- notably the class definition of `Holder` would break. So some syntax adjustment would have to be made. But is this reasonable / of interest / attainable? Thanks for your time and consideration. From rosuav at gmail.com Sat Mar 28 23:45:18 2015 From: rosuav at gmail.com (Chris Angelico) Date: Sun, 29 Mar 2015 09:45:18 +1100 Subject: [Python-ideas] History on proposals for Macros? In-Reply-To: <36A4F7D4-20A4-4BE9-9D86-B423C13A7320@yahoo.com> References: <20150328172624.GG25453@ando.pearwood.info> <36A4F7D4-20A4-4BE9-9D86-B423C13A7320@yahoo.com> Message-ID: On Sun, Mar 29, 2015 at 6:50 AM, Andrew Barnert wrote: > Python already has a few things you can do at runtime that affect subsequent compilation, like __future__ statements; this isn't impossible. To be technically accurate, __future__ directives do not affect "subsequent compilation" - they affect the compilation of the one module they're at the top of, you can't do this: def func1(f): print "Hello, world!" f("Haha") from __future__ import print_function def func2(): func1(print) ChrisA From greg.ewing at canterbury.ac.nz Sat Mar 28 23:51:31 2015 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sun, 29 Mar 2015 11:51:31 +1300 Subject: [Python-ideas] "Loose" descriptors In-Reply-To: References: Message-ID: <55173073.9070600@canterbury.ac.nz> James Edwards wrote: > I envision (but with no real knowledge of the implications of this) > that just as python identifies that `val` in `Holder.__dict__` > implements `__get__`, python could similarly identify that `val` in > `globals()` (or `sys.modules[__name__].__dict__`, or wherever is more > appropriate) implements `__get__` and handle the indirection. This would slow down *all* lookups of global names in order to support a rarely-used feature. Since global name lookup is critical performance-wise, this is likely to meet a lot of resistance. Similar arguments have been made concerning the support of descriptors in module namespaces, which is a closely related idea. -- Greg From steve at pearwood.info Sun Mar 29 03:51:44 2015 From: steve at pearwood.info (Steven D'Aprano) Date: Sun, 29 Mar 2015 12:51:44 +1100 Subject: [Python-ideas] History on proposals for Macros? In-Reply-To: <36A4F7D4-20A4-4BE9-9D86-B423C13A7320@yahoo.com> References: <20150328172624.GG25453@ando.pearwood.info> <36A4F7D4-20A4-4BE9-9D86-B423C13A7320@yahoo.com> Message-ID: <20150329015144.GJ25453@ando.pearwood.info> On Sat, Mar 28, 2015 at 12:50:09PM -0700, Andrew Barnert wrote: > On Mar 28, 2015, at 10:26, Steven D'Aprano wrote: > > > >> On Sat, Mar 28, 2015 at 09:53:48AM -0700, Matthew Rocklin wrote: > >> [...] > >> The goal is to create things that look like functions but have access to > >> the expression that was passed in. > > > >> assertRaises(ZeroDivisionError, 1/0) # Evaluate the rhs 1/0 within > >> assertRaises function, not before > > > >> Generally one constructs something that looks like a function but, rather > >> than receiving a pre-evaluated input, receives a syntax tree along with the > >> associated context. This allows that function-like-thing to manipulate the > >> expression and to control the context in which the evaluation occurs. > > > > How will the Python compiler determine that assertRaises should receive > > the syntax tree rather than the evaluated 1/0 expression (which of > > course will raise)? The information that assertRaises is a "macro" is > > not available at compile time. > > Well, it _could_ be available. At the time you're compiling a scope (a > function, class, or top-level module code), if it uses an identifier > that's the name of a macro in scope, the compiler expands the macro > instead of compiling in a function call. Perhaps I trimmed out too much of Matthew's comment, but he did say he isn't talking about C-style preprocessor macros, so I think that if you are imagining "expanding the macro" C-style, you're barking up the wrong tree. From his description, I don't think Lisp macros are quite the right description either. (As always, I welcome correction if I'm the one who is mistaken.) C macros are more or less equivalent to a source-code rewriter: if you define a macro for "foo", whenever the compiler sees a token "foo", it replaces it with the body of the macro. More or less. Lisp macros are different, and more powerful: http://c2.com/cgi/wiki?LispMacro http://cl-cookbook.sourceforge.net/macros.html but I don't think that's what Matthew wants either. The critical phrase is, I think: "one constructs something that looks like a function but, rather than receiving a pre-evaluated input, receives a syntax tree along with the associated context" which I interpret in this way: Suppose we have this chunk of code creating then using a macro: macro mymacro(expr): # process expr somehow mymacro(x + 1) That is *roughly* equivalent (ignoring the part about context) to what we can do today: def myfunction(expr): assert isinstance(expr, ast.Expression) # process expr somehow tree = ast.parse('x + 1', mode='eval') myfunction(tree) The critical difference being that instead of the author writing code to manually generate the syntax tree from a string at runtime, the compiler automatically generates the tree from the source code at compile time. This is why I think that it can't be done by Python. What should the compiler do here? callables = [myfunction, mymacro] random.shuffle(callables) for f in callables: f(x + 1) If that strikes you as too artificial, how about a simpler case? from mymodule import name name(x + 1) If `name` will refer to a function at runtime, the compiler needs to generate code which evaluates x+1 and passes the result to `name`; but if `name` will refer to a macro, the compiler needs to generate an ast (plus context) and pass it without evaluating it. To do that, it needs to know at compile-time which objects are functions and which are macros, but that is not available until runtime. But we might be able to rescue this proposal by dropping the requirement that the compiler knows when to pass the syntax tree and when to evaluate it. Suppose instead we had a lightweight syntax for generating the AST plus grabbing the current context: x = 23 spam(x + 1, !(x+1)) # macro syntax !( ... ) Now the programmer is responsible for deciding when to use an AST and when to evaluate it, not the compiler, and "macros" become regular functions which just happen to expect an AST as their argument. [...] > If such a light-lambda syntax reduced the desire for macros down to > the point where it could be ignored, and if that desire weren't > _already_ low enough that it can be ignored, it would be worth adding. > I think the second "if" is where it fails, not the first, but I could > be wrong. I presume that Matthew wants the opportunity to post-process the AST, not merely evaluate it. If all you want is to wrap some code an an environment in a bundle for later evaluation, you are right, a function will do the job. But it's hard to manipulate byte code, hence the desire for a syntax tree. -- Steve From rosuav at gmail.com Sun Mar 29 04:13:41 2015 From: rosuav at gmail.com (Chris Angelico) Date: Sun, 29 Mar 2015 13:13:41 +1100 Subject: [Python-ideas] History on proposals for Macros? In-Reply-To: <20150329015144.GJ25453@ando.pearwood.info> References: <20150328172624.GG25453@ando.pearwood.info> <36A4F7D4-20A4-4BE9-9D86-B423C13A7320@yahoo.com> <20150329015144.GJ25453@ando.pearwood.info> Message-ID: On Sun, Mar 29, 2015 at 12:51 PM, Steven D'Aprano wrote: > But we might be able to rescue this proposal by dropping the requirement > that the compiler knows when to pass the syntax tree and when to > evaluate it. Suppose instead we had a lightweight syntax for generating > the AST plus grabbing the current context: > > x = 23 > spam(x + 1, !(x+1)) # macro syntax !( ... ) > > > Now the programmer is responsible for deciding when to use an AST and > when to evaluate it, not the compiler, and "macros" become regular > functions which just happen to expect an AST as their argument. This is actually a lot more plausible than most of the other theories (except the preprocessor, but anyone can do that, and it's not necessarily going to help you any). If the magic macro operator is given precedence equivalent to lambda, it would often be possible to do it without the parens, too. In the same way that "lambda: expr" yields an object (a function) rather than evaluating its argument, the macro syntax would yield an object (an AST tree or somesuch) without actually evaluating the expression. ChrisA From steve at pearwood.info Sun Mar 29 04:17:07 2015 From: steve at pearwood.info (Steven D'Aprano) Date: Sun, 29 Mar 2015 13:17:07 +1100 Subject: [Python-ideas] "Loose" descriptors In-Reply-To: <55173073.9070600@canterbury.ac.nz> References: <55173073.9070600@canterbury.ac.nz> Message-ID: <20150329021707.GK25453@ando.pearwood.info> On Sun, Mar 29, 2015 at 11:51:31AM +1300, Greg Ewing wrote: > James Edwards wrote: > >I envision (but with no real knowledge of the implications of this) > >that just as python identifies that `val` in `Holder.__dict__` > >implements `__get__`, python could similarly identify that `val` in > >`globals()` (or `sys.modules[__name__].__dict__`, or wherever is more > >appropriate) implements `__get__` and handle the indirection. > > This would slow down *all* lookups of global names in > order to support a rarely-used feature. Since global > name lookup is critical performance-wise, this is > likely to meet a lot of resistance. > > Similar arguments have been made concerning the > support of descriptors in module namespaces, which > is a closely related idea. I don't think it's a closely related idea, I think it is *exactly* the same idea. If I've understood him correctly, James wants to use descriptors in global variables, i.e. module namespaces, not just class attributes. You are right that making this the default behaviour would cause slow-down of all module-level lookups for very rare advantage, but what if we moved the descriptor logic into the global __dict__ instead of the compiler? Then, in thory at least, supporting the descriptor protocol for global variables will be opt-in on a per-module basis. Here's a sketch of how you might do it. class DescriptorDict(dict): def __getitem__(self, key): obj = super().__getitem__(key) if hasattr(type(obj), '__get__'): # I'm not sure what to pass as instance here. return type(obj).__get__(obj, instance) return obj __dict__ = DescriptorDict(__dict__) I'm sure there are complications I haven't even begun to imagine, but here are a few obvious ones: (1) Can modules use a dict subclass for __dict__? (2) Can you replace the __dict__ on the fly like this? (3) If not, how do you bootstrap the module to have a magic dict instead of an ordinary dict? (4) The descriptor protocol expects to pass the instance and/or class to the __get__ method, as well as `self`. self is the descriptor itself; what is instance? The module? I don't know if it can be done, but if it could be, it would certainly open the doors for a lot of interesting experiments. -- Steve From dw+python-ideas at hmmz.org Sun Mar 29 04:34:05 2015 From: dw+python-ideas at hmmz.org (David Wilson) Date: Sun, 29 Mar 2015 02:34:05 +0000 Subject: [Python-ideas] "Loose" descriptors In-Reply-To: References: Message-ID: <20150329023405.GA7521@k3> On Sat, Mar 28, 2015 at 06:35:17PM -0400, James Edwards wrote: > But my question is would it make sense to be able to do this sort of > indirection for "loose" objects also, not just attributes of classes > or instances? In other words, could descriptors be extended so that > they perform the same sort of indirection in the second example as the > first? FWIW you could probably experiment with this using PyPy's object spaces feature without modifying the interpreter: http://pypy.readthedocs.org/en/latest/objspace-proxies.html The 'exec' statement also indirectly supports a restrictive version of this, by allowing dict subclasses to be provided for globals/locals. Naturally the effect wouldn't carry over to function calls invoked by the statement. As a general comment, the ability to overload scope load/store seems like an expert feature of questionable applicability. It's fun to play with, but I'd hope to never encounter it in code intended for maintainability. David From abarnert at yahoo.com Sun Mar 29 05:58:25 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Sat, 28 Mar 2015 20:58:25 -0700 Subject: [Python-ideas] History on proposals for Macros? In-Reply-To: References: <20150328172624.GG25453@ando.pearwood.info> <36A4F7D4-20A4-4BE9-9D86-B423C13A7320@yahoo.com> Message-ID: <006D5417-D16B-41A7-8B6E-0395FEDA2D58@yahoo.com> On Mar 28, 2015, at 15:45, Chris Angelico wrote: > > On Sun, Mar 29, 2015 at 6:50 AM, Andrew Barnert > wrote: >> Python already has a few things you can do at runtime that affect subsequent compilation, like __future__ statements; this isn't impossible. > > To be technically accurate, __future__ directives do not affect > "subsequent compilation" - they affect the compilation of the one > module they're at the top of, you can't do this: > > def func1(f): > print "Hello, world!" > f("Haha") > > from __future__ import print_function > > def func2(): > func1(print) Except at the interactive interpreter, where you can write exactly that, and it will do exactly what you'd want/expect. From abarnert at yahoo.com Sun Mar 29 07:29:29 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Sat, 28 Mar 2015 22:29:29 -0700 Subject: [Python-ideas] History on proposals for Macros? In-Reply-To: <20150329015144.GJ25453@ando.pearwood.info> References: <20150328172624.GG25453@ando.pearwood.info> <36A4F7D4-20A4-4BE9-9D86-B423C13A7320@yahoo.com> <20150329015144.GJ25453@ando.pearwood.info> Message-ID: <9834EACC-EEA3-4F1B-B20D-4BD6F9399B92@yahoo.com> On Mar 28, 2015, at 18:51, Steven D'Aprano wrote: > >> On Sat, Mar 28, 2015 at 12:50:09PM -0700, Andrew Barnert wrote: >>> On Mar 28, 2015, at 10:26, Steven D'Aprano wrote: >>> >>>> On Sat, Mar 28, 2015 at 09:53:48AM -0700, Matthew Rocklin wrote: >>>> [...] >>>> The goal is to create things that look like functions but have access to >>>> the expression that was passed in. >>> >>>> assertRaises(ZeroDivisionError, 1/0) # Evaluate the rhs 1/0 within >>>> assertRaises function, not before >>> >>>> Generally one constructs something that looks like a function but, rather >>>> than receiving a pre-evaluated input, receives a syntax tree along with the >>>> associated context. This allows that function-like-thing to manipulate the >>>> expression and to control the context in which the evaluation occurs. >>> >>> How will the Python compiler determine that assertRaises should receive >>> the syntax tree rather than the evaluated 1/0 expression (which of >>> course will raise)? The information that assertRaises is a "macro" is >>> not available at compile time. >> >> Well, it _could_ be available. At the time you're compiling a scope (a >> function, class, or top-level module code), if it uses an identifier >> that's the name of a macro in scope, the compiler expands the macro >> instead of compiling in a function call. > > Perhaps I trimmed out too much of Matthew's comment, but he did say he > isn't talking about C-style preprocessor macros, so I think that if you > are imagining "expanding the macro" C-style, you're barking up the wrong > tree. No, I'm not imagining C style expansion. What I'm imagining is closer to Lisp-style, and closer still to Dylan-style.* I didn't want to get into details because they're complicated, and there even may be multiple complicated ways to do things that have to be chosen from, and none of that is relevant to your question. But if you're curious, let me give a more specific explanation: A macro is compiled by transforming its AST into AST-transforming bytecode, similar to a function, but then instead of embedding a BUILD_FUNCTION opcode into the defining scope's bytecode, you do that, and _also_ effectively call it in the current scope and bind the macro to the name there.** A macro is expanded by parsing the arguments into ASTs, calling the AST-transforming function on those ASTs, and substituting the result into the tree at the point of the macro call.*** You don't need to explicitly "pass" a context; the context in which the function is called (the compile-time scope, etc.) is implicitly available, just as for runtime functions. As I mentioned before, there are a number of additional issues you'd have to resolve (again, think import, and .pyc files, for an example), some of which may make the feature undesirable once you think them through, but I don't think any of them are relevant to your question, and I think something like this design is what he was asking about. * In Lisp, because there is no syntax and hence no separate step to turn a parenthesized token stream into an AST, it's ambiguous at which stage--before or after that non-existent step--macros are applied. In languages with syntax and grammars, the usual answer is to do it after parsing to AST. You can conceptually define macros at any stage in the pipeline, but that's where they turn out to be most useful. Also, I'm ignoring the issue of hygiene, but I think most people want macros to be hygienic by default and unhygienic only on explicit demand, rather than what Lisp does. ** The details here are tricky because the compiler's notion of "current scope" isn't defined by the language anyway and doesn't correspond to anything defined at runtime, but the intuitive idea is clear: while compiling a module or other scope, if you reach a def or class, a compiler has to do the equivalent of recursively calling itself or pushing a scope onto a stack manually; that stack defines the compile-time scope. So the name bound to the macro goes away when you exit that recursive call/pop that scope from the stack. The compiler currently keeps track of all variables assigned to in the current scope to determine local variables and closures; I believe macro assignments can be piled on top of that. If not, this is something completely new that you have to bolt on. *** In most languages with syntax and macros, a macro can only take an expression, and must return an expression--which is fine for most of those languages, where almost everything (flow control, variable binding, etc.) is an expression, but not so much in Python, where many of those things can only be done in a statement. But allowing a statement or an expression (or any arbitrary AST node) and allowing a macro to likewise "return" either (or any) type isn't straightforward once you think through some examples. > From his description, I don't think Lisp macros are quite the > right description either. (As always, I welcome correction if I'm the > one who is mistaken.) > > C macros are more or less equivalent to a source-code rewriter: if you > define a macro for "foo", whenever the compiler sees a token "foo", it > replaces it with the body of the macro. More or less. > > Lisp macros are different, and more powerful: > > http://c2.com/cgi/wiki?LispMacro > http://cl-cookbook.sourceforge.net/macros.html > > but I don't think that's what Matthew wants either. The critical phrase > is, I think: > > "one constructs something that looks like a function but, rather than > receiving a pre-evaluated input, receives a syntax tree along with the > associated context" > > which I interpret in this way: > > Suppose we have this chunk of code creating then using a macro: > > macro mymacro(expr): > # process expr somehow > > mymacro(x + 1) > > > That is *roughly* equivalent (ignoring the part about context) to what > we can do today: > > def myfunction(expr): > assert isinstance(expr, ast.Expression) > # process expr somehow > > tree = ast.parse('x + 1', mode='eval') > myfunction(tree) > > > The critical difference being that instead of the author writing code to > manually generate the syntax tree from a string at runtime, the compiler > automatically generates the tree from the source code at compile time. > > This is why I think that it can't be done by Python. What should the > compiler do here? > > callables = [myfunction, mymacro] > random.shuffle(callables) > for f in callables: > f(x + 1) Well, that depends on how much of Python you want to be available at compile time. One possibility is that only the defmacro statement is executed at compile time. You may want to add imports to that. You may want to add assignments. You may want to add some large, ridiculously-complex, but well-defined subset of Python (I would ask you to think C++ constexpr rules, but that request might be construed as torture.) Or you may even want the entire language. And any of the above could be modified by making some or all of those constructs compile-time only when explicitly required (again, like constexpr). Any of these is conceptually sensible (although none of them may be desirable...), and they give you different answers here. For example, if you only execute defmacro and import at compile time, then at compile time f is not expanded as a macro, it's just called as a function, which will probably raise a TypeError at runtime (because the current value of x + 1 is probably not an AST node...). > If that strikes you as too artificial, how about a simpler case? > > from mymodule import name > name(x + 1) > > > If `name` will refer to a function at runtime, the compiler needs to > generate code which evaluates x+1 and passes the result to `name`; but > if `name` will refer to a macro, the compiler needs to generate an ast > (plus context) and pass it without evaluating it. To do that, it needs > to know at compile-time which objects are functions and which are > macros, but that is not available until runtime. This is exactly what I was referring to when I said that you need to make significant changes to other parts of the language, such as revising the import machinery and .pyc files, and that this problem does not apply to future statements. I don't want to go over all the details in as much depth as the last question, so hopefully you'll just accept that the answer is the same: it's not available today, but it could be available, in a variety of different ways that you'd have to choose between, all of which would have different knock-on effects. > But we might be able to rescue this proposal by dropping the requirement > that the compiler knows when to pass the syntax tree and when to > evaluate it. Suppose instead we had a lightweight syntax for generating > the AST plus grabbing the current context: > > x = 23 > spam(x + 1, !(x+1)) # macro syntax !( ... ) That's a lot closer to what MacroPy does. And notice that if the !() syntax were added to the grammar, MacroPy or something like it could be significantly simpler.* Which means it might be easier to integrate it directly into the builtin compiler--but also means it might be less desirable to do so, as leaving it as an externally-supplied import hook has all the benefits of externally-supplied modules in general. (However, there is still the disadvantage that you have to apply an import hook before importing, effectively meaning you can't use or define macros in your top-level script. If necessary, you could fix that as well with special syntax that must come before anything but comments and future statements that adds an import hook. This is the kind of thing I was talking about in my first message, about finding smaller changes to the language that make MacroPy or something like it simpler and/or more flexible.) * I believe OCaml recently added something similar for similar purposes, but I haven't used it in a few years, so I may be misinterpreting what I saw skimming the what's new for the last major version. > Now the programmer is responsible for deciding when to use an AST and > when to evaluate it, not the compiler, and "macros" become regular > functions which just happen to expect an AST as their argument. No, not quite. What do you _do_ with the AST returned by the macro? And when do you do it? You still have to substitute it into the tree of the current compilation target in place of the macro call, which means it still has to be available at compile time. It does provide a simpler way to resolve some of the other issues, but it doesn't resolve the most fundamental one. > [...] >> If such a light-lambda syntax reduced the desire for macros down to >> the point where it could be ignored, and if that desire weren't >> _already_ low enough that it can be ignored, it would be worth adding. >> I think the second "if" is where it fails, not the first, but I could >> be wrong. > > I presume that Matthew wants the opportunity to post-process the AST, > not merely evaluate it. If all you want is to wrap some code an an > environment in a bundle for later evaluation, you are right, a function > will do the job. But it's hard to manipulate byte code, hence the desire > for a syntax tree. Sure, but most of the examples people give for wanting macros--including his example that you quoted--don't actually do anything that can't be done with a higher-order function. Which means they may not actually want macros, they just think they do. Ask a Haskell lover why Haskell doesn't need macros, and he'll tell you that it's because you don't need them, you only think you do because of your Lisp prejudices. Of course that isn't 100% true,* but it's true enough that most people are happy without macros in Haskell. Similarly, while it would be even farther from 100% true in Python,** it might still be true enough that most people are happy without macros in Python. (Except, as I said, most people are _already_ happy without macros in Python, which means we may have an even simpler option: just do nothing.) * For one reasonably well-known example (although I may be misremembering, so take this as a "this kind of thing" rather than "exactly this..."), if Haskell98 had macros, you could use them to simulate GADTs, which didn't exist until a later version of the language. For an equivalent example in Python: you could use macros to simulate with statements in Python 2.5. As long as non-silly cases for macros are rare enough, people are satisfied with evaluating them at language design time (a discussion on the list followed by an update to the language and a patch to GHCI/CPython) instead of compile time. :) ** The main reason it would be less true in Python is eager evaluation; a lazy language like Haskell (or, even better, a dataflow language) can replace even more uses of macros with HOFs than an eager language. For example, in Haskell, the equivalent of "def foo(x, y): return y if x else 0" doesn't need the value of y unless x is true, so it doesn't matter that it y is a value rather than an expression. But OCaml, for example, also doesn't have lazy evaluation, and people seem to have the same attitude toward macros there too. (Although it does have a powerful preprocessor, it's not that much different from what Python has with import hooks.) Well, despite trying to skim over some parts, I still wrote a whole book here; apologies for that, to anyone who's still reading. :) From abarnert at yahoo.com Sun Mar 29 07:36:37 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Sat, 28 Mar 2015 22:36:37 -0700 Subject: [Python-ideas] "Loose" descriptors In-Reply-To: <20150329021707.GK25453@ando.pearwood.info> References: <55173073.9070600@canterbury.ac.nz> <20150329021707.GK25453@ando.pearwood.info> Message-ID: <3E5CAD4F-A573-4403-BEF3-6434A145E592@yahoo.com> On Mar 28, 2015, at 19:17, Steven D'Aprano wrote: > >> On Sun, Mar 29, 2015 at 11:51:31AM +1300, Greg Ewing wrote: >> James Edwards wrote: >>> I envision (but with no real knowledge of the implications of this) >>> that just as python identifies that `val` in `Holder.__dict__` >>> implements `__get__`, python could similarly identify that `val` in >>> `globals()` (or `sys.modules[__name__].__dict__`, or wherever is more >>> appropriate) implements `__get__` and handle the indirection. >> >> This would slow down *all* lookups of global names in >> order to support a rarely-used feature. Since global >> name lookup is critical performance-wise, this is >> likely to meet a lot of resistance. >> >> Similar arguments have been made concerning the >> support of descriptors in module namespaces, which >> is a closely related idea. > > I don't think it's a closely related idea, I think it is *exactly* the > same idea. If I've understood him correctly, James wants to use > descriptors in global variables, i.e. module namespaces, not just class > attributes. Yeah, that's how I interpreted it too. The obvious way to do this is to provide some way for a module to define not only an instance of the module type, but a subclass and an instance of that subclass. The subclass then provides a natural place for the descriptors to live (and a natural answer to what gets passed for inst and typ). There was a proposal for something very much like that some time late last year, IIRC, which might be worth digging up. (Modules that don't define a subclass could still be easily optimized for fast global lookup, because the compiler knows that the base module class doesn't have any descriptors.) > You are right that making this the default behaviour would cause > slow-down of all module-level lookups for very rare advantage, but what > if we moved the descriptor logic into the global __dict__ instead of the > compiler? Then, in thory at least, supporting the descriptor protocol > for global variables will be opt-in on a per-module basis. > > Here's a sketch of how you might do it. > > class DescriptorDict(dict): > def __getitem__(self, key): > obj = super().__getitem__(key) > if hasattr(type(obj), '__get__'): > # I'm not sure what to pass as instance here. > return type(obj).__get__(obj, instance) > return obj > > > __dict__ = DescriptorDict(__dict__) > > > I'm sure there are complications I haven't even begun to imagine, but > here are a few obvious ones: > > (1) Can modules use a dict subclass for __dict__? > (2) Can you replace the __dict__ on the fly like this? > (3) If not, how do you bootstrap the module to have a magic dict instead > of an ordinary dict? > (4) The descriptor protocol expects to pass the instance and/or class > to the __get__ method, as well as `self`. self is the descriptor > itself; what is instance? The module? > > > I don't know if it can be done, but if it could be, it would certainly > open the doors for a lot of interesting experiments. > > > > -- > Steve > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ From techtonik at gmail.com Fri Mar 27 14:52:22 2015 From: techtonik at gmail.com (anatoly techtonik) Date: Fri, 27 Mar 2015 16:52:22 +0300 Subject: [Python-ideas] Geo coordinates conversion in stdlib In-Reply-To: References: <550F30FB.5010000@sdamon.com> <20150322223815.06082280@fsol> <21A308F0-5F2E-48BB-9F3F-4A3CFAB1D6DD@yahoo.com> Message-ID: On Mon, Mar 23, 2015 at 10:21 AM, Chris Angelico wrote: > On Mon, Mar 23, 2015 at 5:08 PM, anatoly techtonik > wrote: > > Handling geo coordinates in most used formats is as important as > > handling .zip archives in stdlib. > > That may or may not be the case, but if it is, the best way to argue > the point is: > > 1) Find or create a third-party package on PyPI > 2) Demonstrate that this package has a stable API > 3) Demonstrate or pledge support for the package > 4) Appeal for the package's inclusion in the stdlib > > You'll need to show that there's benefit to having it in the stdlib as > opposed to on PyPI - demonstrate the problem with "pip install > packagename" that means that you want it in the standard library. Just > *stating* that there is a need isn't sufficient - you need to prove > it. > Why the wave module is there? Or ioctl module? Are they so widely used? This process is flawed. If anybody would sponsor the research on popularity of stdlib, I believe that math will be excluded from top 50 as well. Will stdlib without math will still be ok? The measure to estimate the usefullness of standard library is not in download score, but in coverage across different areas. There thousands of people downloading stuff from youtube, so should we include youtube-dl instead of math? -- anatoly t. -------------- next part -------------- An HTML attachment was scrubbed... URL: From techtonik at gmail.com Fri Mar 27 15:30:09 2015 From: techtonik at gmail.com (anatoly techtonik) Date: Fri, 27 Mar 2015 17:30:09 +0300 Subject: [Python-ideas] Geo coordinates conversion in stdlib In-Reply-To: <019975C9-137D-4834-80E6-09F471A789B3@yahoo.com> References: <550F30FB.5010000@sdamon.com> <20150322223815.06082280@fsol> <21A308F0-5F2E-48BB-9F3F-4A3CFAB1D6DD@yahoo.com> <019975C9-137D-4834-80E6-09F471A789B3@yahoo.com> Message-ID: On Mon, Mar 23, 2015 at 11:29 AM, Andrew Barnert wrote: > On Mar 22, 2015, at 11:08 PM, anatoly techtonik > wrote: > > > > On Mon, Mar 23, 2015 at 1:37 AM, Andrew Barnert > > wrote: > >> On Mar 22, 2015, at 2:38 PM, Antoine Pitrou > wrote: > >>> > >>> On Sun, 22 Mar 2015 17:15:39 -0400 > >>> Alexander Walters > >>> wrote: > >>>>> On 3/21/2015 05:46, anatoly techtonik wrote: > >>>>> I think it should be in stdlib. > >>>>> > >>>>> 11? 18? 0? S, 2? 42? 0? E to -11.3, 2.7 and back. > >>>>> > >>>>> This is from Wikipedia. > >>>> Why does this need to be in the standard library? > >>> > >>> Because Anatoly needs it, of course :) > >> > >> Well, it would certainly be nice to have all of Wikipedia in the > stdlib, but I wonder how that would affect Python's release schedule, and > its use in resource-constrained systems. > >> > >> More seriously, if you're just looking to convert arcminutes and > arcseconds to and from fractional degrees, it's exactly the same algorithm > as converting minutes and seconds to fractional hours, which we already > have in the datetime.timedelta class. If you can't figure out the > complexity of multiplying and divmodding by 60 twice, you can borrow its > code, or just wrap it and delegate. That seems trivial enough to do in any > project that needs it. And that means projects that want > degrees-arcminutes-arcseconds-milliarcseconds (e.g., for astronomy) or just > degrees and fractional arcminutes with no arcseconds (e.g., for marine > navigation) or degrees, arcminutes, and eighth-arcminutes (e.g., for US > military scopes) can just as easily get what they need, instead of being > forced to use the variation surveyors prefer. > > > > That's nice to know, but IIRC datetime is from the top 10 Python > > modules that need a redesign. Things contained therein doesn't pass > > human usability check, and are not used as a result. > > There are thousands of projects that actually do use it; thousands of > people asking questions on StackOverflow, Python-tutor, etc. and getting > actual answers; etc. So your theoretical argument that it's not usable > doesn't really matter given that practically it is actually usable, and > used. > The fact that thousands of people are asking question on SO about datetime is the main indicator that the module fails usability check. If it passed, there was no need to ask stuff. If docs were good, the rate of question was much lower. And when I say about top 10 modules that need a redesign, it is not abstract. http://sayspy.blogspot.com/2009/07/informal-poll-what-python-stdlib.html > My story for that day is that I needed to process geodata from > > Wikipedia to see the distribution over the Earth surface, so I thought > > that if I know the Python it is not a problem. It appeared that the > > problem is there, and the problem is recurring (last time I had GPS > > points). > > I suspect that people who do a lot of geodata processing in Python already > have third-party libs that they use, and would have no interest in a very > small limited subset being duplicated in the stdlib. Actually, these people are journalists and we'd like to teach them some Python scripting. If regexps and file reading operations are there, it is still a problem to do an interesting case without installing a lot of stuff, meaning that they are not using Python, but some Python distribution. Maybe you're right - every type Python user should have a specialized Python distribution like Anaconda or WinPython, but the most basic entrypoint should be in stdlib. There should be evolution in learning, going from math -> SciPy, urlib2 -> requests, etc. The Python should come with toys for most major uses of scripting languages in modern world. It is already long past only "computer science" stuff. > > Python docs explain a lot about core OS concepts, but Python > > is not only a systems language. I'd say that it is now being replaced > > with Go for all system stuff, > > I don't know anyone outside Google who's using Go for any system (or > sysadmin, which seems to be more likely what you're talking about) stuff. > I'm sure there are exceptions, but the idea that Python is being > universally replaced by Go in that area is so ridiculous that it's hard to > believe you actually meant it seriously. It's like saying that Ruby is > being replaced by Scala for all web server stuff just because Scala is the > fastest-growing language in that area this week. Well, maybe it is only my impression, but I haven't seen any wide used user level app recently written in Python, and there is a plenty of tools written in Go for the areas that were once Python domain - system tools. Docker, LXD, Syncthing - these are the tools that I actually use almost daily, and I am looking at https://github.com/limetext/lime right now for some reason. > and more attempts are made to use is as > > a data processing language. But that is not possible if Python doesn't > > have a basic concepts of data representation from the Real Life (RL) > > (tm) in its stdlib. > > Most people I know who use Python as a serious data processing language > don't care much what's in the stdlib because they're using numpy and/or > pandas. Yes, and that's sad that the Python still doesn't have the basic building blocks of Pandas in stdlib. It is the proven de-facto popularity achievement that is required for packages to enter stdlib, but that still doesn't happen. Looks like the granularity of package is not sufficient for designing next stdlib - the need is to descent into the level of structures. > > Handling geo coordinates in most used formats is as important as > > handling .zip archives in stdlib. > > Not even close. Zip archives are used in a ridiculously wide range of > application areas; geoccordinates are used only in processing geocoordinate > data (and there only in small enough scales that you don't need numpy or > some specialized geodata library that I don't know about but that I'll bet > exists). There are indeed a lot of proliferation in geo standards, but for the basis stuff I trust the Wikipedia choice, because the practicality beats purity. -- anatoly t. -------------- next part -------------- An HTML attachment was scrubbed... URL: From techtonik at gmail.com Fri Mar 27 15:38:30 2015 From: techtonik at gmail.com (anatoly techtonik) Date: Fri, 27 Mar 2015 17:38:30 +0300 Subject: [Python-ideas] Geo coordinates conversion in stdlib In-Reply-To: <550FDA07.5020704@egenix.com> References: <550F30FB.5010000@sdamon.com> <20150322223815.06082280@fsol> <21A308F0-5F2E-48BB-9F3F-4A3CFAB1D6DD@yahoo.com> <550FDA07.5020704@egenix.com> Message-ID: On Mon, Mar 23, 2015 at 12:16 PM, M.-A. Lemburg wrote: > > BTW: If you expect the Python stdlib to make coffee, clean your house > and drive your kids to school, you're probably looking at the > wrong language - we only do batteries in the stdlib :-) > Well, I got your point, it is not about GPS module anymore, it is about inability of community decisions on the stuff that looks like highly subjective and personal. I don't think I can get convincing data for that, because nobody has the money to spend time on that data mining stuff, especially me. So the final argument would need to be extended to do the research on Python usage in non-computer science and system programming world to see if GPS conversion module in standard library provides any value. That means that we need to find the metrics that will be more useful than just a PyPI download index. -------------- next part -------------- An HTML attachment was scrubbed... URL: From techtonik at gmail.com Fri Mar 27 15:52:52 2015 From: techtonik at gmail.com (anatoly techtonik) Date: Fri, 27 Mar 2015 17:52:52 +0300 Subject: [Python-ideas] Geo coordinates conversion in stdlib In-Reply-To: References: <550F30FB.5010000@sdamon.com> <20150322223815.06082280@fsol> <21A308F0-5F2E-48BB-9F3F-4A3CFAB1D6DD@yahoo.com> Message-ID: On Fri, Mar 27, 2015 at 5:04 PM, Chris Angelico wrote: > On Sat, Mar 28, 2015 at 12:52 AM, anatoly techtonik > wrote: > > Why the wave module is there? Or ioctl module? Are they so widely used? > > > > This process is flawed. If anybody would sponsor the research on > popularity > > of stdlib, I believe that math will be excluded from top 50 as well. Will > > stdlib > > without math will still be ok? > > > > The measure to estimate the usefullness of standard library is not in > > download score, but in coverage across different areas. There thousands > of > > people downloading stuff from youtube, so should we include youtube-dl > > instead of math? > > The criteria for the *removal* from the stdlib would be insanely > tight, compared to the criteria for *addition*. So some modules might > now simply be grandfathered in (I suspect wave is one of them), due to > the shift in ease of grabbing stuff off PyPI. But the math module is > pretty important and fundamental, so it's not going anywhere. > > As to youtube-dl, though: Definitely not. No matter how popular it is, > it still fails the criterion of release schedule synchronization - it > needs to be agile enough to update as often as it needs, according to > host site changes, and would be shackled rather badly by inclusion in > the stdlib. (You probably don't want to get it from your OS repo > either.) > > But none of this changes the recommendations I made earlier. You need > to *demonstrate*, not simply state, a need; and one of the best ways > to demonstrate support, API stability, usage, etc, is to stick a > package up on PyPI. Not everything needs to be in the stdlib, but if > you're going to argue that something should be, what better way than > by pointing out what it's already doing? There is now quite a big lump of data for me that in social systems (such as community developed projects) the reputation is more important than rationale, so how do you imagine a demonstration? Well, I have this: https://en.wikipedia.org/wiki/List_of_solar_eclipses_in_the_21st_century here is the list of all sun eclipses in the World. How would you show them on map to paste into your student paper? You need to import the table into your spreadsheet, then transform coordinates into floats and then export the data for you map drawing tools. The problem - spreadsheets don't convert coordinates. You need a script for that. And that script you will need for much every data processing involving data from Wikipedia. Is that fair enough for demonstration of a need? -- anatoly t. -------------- next part -------------- An HTML attachment was scrubbed... URL: From techtonik at gmail.com Fri Mar 27 15:54:09 2015 From: techtonik at gmail.com (anatoly techtonik) Date: Fri, 27 Mar 2015 17:54:09 +0300 Subject: [Python-ideas] Top 10 Python modules that need a redesign Was: Geo coordinates conversion in stdlib In-Reply-To: References: Message-ID: On Mon, Mar 23, 2015 at 8:21 PM, Alexander Belopolsky < alexander.belopolsky at gmail.com> wrote: > > On Mon, Mar 23, 2015 at 2:08 AM, anatoly techtonik > wrote: > > > > That's nice to know, but IIRC datetime is from the top 10 Python > > modules that need a redesign. Things contained therein doesn't pass > > human usability check, and are not used as a result. > > > Where have you been when PEP 3108 was discussed? I have not seen any > other list of Python modules that needed a redesign, so I cannot tell > what's on your top ten list. > http://sayspy.blogspot.com/2009/07/informal-poll-what-python-stdlib.html -------------- next part -------------- An HTML attachment was scrubbed... URL: From techtonik at gmail.com Fri Mar 27 16:05:52 2015 From: techtonik at gmail.com (anatoly techtonik) Date: Fri, 27 Mar 2015 18:05:52 +0300 Subject: [Python-ideas] Top 10 Python modules that need a redesign Was: Geo coordinates conversion in stdlib In-Reply-To: References: Message-ID: On Mon, Mar 23, 2015 at 9:59 PM, Terry Reedy wrote: > On 3/23/2015 1:21 PM, Alexander Belopolsky wrote: > >> >> On Mon, Mar 23, 2015 at 2:08 AM, anatoly techtonik >> > > wrote: >> > >> > That's nice to know, but IIRC datetime is from the top 10 Python >> > modules that need a redesign. Things contained therein doesn't pass >> > human usability check, and are not used as a result. >> > > ... > >> in what sense does it not "pass human usability check"? >> > ... > > Please don't feed the troll. He is known for opinionated digs like the > above and historically, nothing good comes from trying to discuss them. Well, I admit that it takes a really bad mood and anger to write stuff about anything, and it is not very constructive state of mind. The rest of the things I could just probably silently enjoy (or spend time on something else). Whatever. To make anything constructive out of it, people need to accept the "usability" and "user experience" as a study disciplines, and that the studies could be and should be applied to Python (in this case to stdlib). -- anatoly t. -------------- next part -------------- An HTML attachment was scrubbed... URL: From techtonik at gmail.com Fri Mar 27 17:05:44 2015 From: techtonik at gmail.com (anatoly techtonik) Date: Fri, 27 Mar 2015 19:05:44 +0300 Subject: [Python-ideas] Python Research Insitute Message-ID: I am well aware of https://www.blender.org/institute/ and I like it. And recently I realized that my claims and accusations towards PSF are just wrong. PSF is a fund - it should process the money to ensure Python Community has at lest some degree of independence from sponsors. It is not its role to register domains, maintain servers, assign copyright to itself from contributors - it is not its role to do anything at all. PSF waits for initiatives to appear and then supports them, but the problem is - there are not many initiatives (except website, which is well, ). And that's the problem that PSF is not solving, because it can not do this. It can not just do what it wants - grab a bunch of people together to solve a particular problem, provide them with a place to do stuff, with access to funding from people interested in certain initiative, direct contact with them, being interested people from outside into the team and do other nasty things (like filming movies!). PSF holds a lot of responsibility and therefore is not effective for research works and experiments. This things that PSF can not do, and nobody can, because economy doesn't allow this: 1. Maintain a list of top 10 stdlib modules that need a redesign The one is gathered there: http://sayspy.blogspot.com/2009/07/informal-poll-what-python-stdlib.html Try with Python community a usability study to find out "why people feel these things need a redesign". 2. Do economic study on Python contributions How many people could escape the corporate "slavery" to do the necessary work for high quality patches. Are they allowed to work on Python, do they feel stressed, etc. How do they feel the economic model helps Python development, is there a way for alternative models that help more people to jump in to make good thing better. 3. Get data on Python services, develop participation tools bugs.python.org is not only a GSoC project. There are a lot of things that can be improved, should be improved and the issues are not trivial. 4. Publish a research what will happen if Python 2.8 is released Study the things like https://gist.github.com/techtonik/574401cebf5e17c33f5b with reports on income and outcomes that could actually be referenced (unlike mails, which are good, but hard to filter). 5. Get data on Python usage, a set of areas and most popular use cases This is came from Geo coordinates conversion module in stdlib. >> Most people I know who use Python as a serious data processing language don't care much what's in the stdlib because they're using numpy and/or pandas. > Yes, and that's sad that the Python still doesn't have the basic building blocks of Pandas in stdlib. Study the idea of making a stlib that consists from modules that cover the most common scenarios of using the scripting language, provide a "toy modules" that cover these common scenarios - a mini demos of more powerful tools, like a "swiss-knives toolbelt". 6. Get a tools on measuring Python stdlib usage Build a linked semantic AST from all packages on PyPI (vector data like OpenStreetMap, but from the language level objects). This will allow to grep how many times a particular module is used, and allow people to use that database to extend into their own research (like trying to guess context and usage scenarios for stdlib calls). 7. Record a summaries of Python evolution, build media materials Visual information requires much much much less time to be consumed, but requires a significant amount of time to produce. PyCon talks are exceptionally good for video material, but for the community coordination, a lot of HTML5 and dynamic design works needs to be done. PSF won't be able to make it interesting to HTML5 folks, because it is solely about Python itself, and the object of Python Research Institute include everything that goes side by side and influences the development of the project and community around it. From "cultural aspects of Monty Python in context of computer language programming" to "feasibility of designing an operating system in stackless" and "test of new economic models for sustainable Python development" etc. I just can't stop with ideas - research how appearance of startup scene affected open source contributions, research why open source project die and how to prevent that, what happened with former Python contributors and where their interest is now, how other languages compare to Python, what are the key problems of converting Python code to other languages, how to deal with complexity etc., what this "agile" is about and could it be useful for Python development etc. Please, tell me if that is a worthy utopia and add your ideas. -------------- next part -------------- An HTML attachment was scrubbed... URL: From techtonik at gmail.com Fri Mar 27 17:13:48 2015 From: techtonik at gmail.com (anatoly techtonik) Date: Fri, 27 Mar 2015 19:13:48 +0300 Subject: [Python-ideas] Top 10 Python modules that need a redesign Was: Geo coordinates conversion in stdlib In-Reply-To: References: Message-ID: On Fri, Mar 27, 2015 at 6:21 PM, Alexander Belopolsky < alexander.belopolsky at gmail.com> wrote: > > On Fri, Mar 27, 2015 at 10:54 AM, anatoly techtonik > wrote: > >> Where have you been when PEP 3108 was discussed? I have not seen any >>> other list of Python modules that needed a redesign, so I cannot tell >>> what's on your top ten list. >>> >> >> http://sayspy.blogspot.com/2009/07/informal-poll-what-python-stdlib.html >> > > Interesting. I did not see this back in the day. The top entry (urllib > and friends) makes sense and there were heavily redesigned in Python 3. I > am surprised that distutils redesign got less support than logging and > datetime. I suspect that this may have to do with logging and datetime > APIs not being PEP8 compliant. Popular votes tend to exaggerate the > importance of trivial things such as the spelling of class and method names > and brush off more subtle, but important design flaws. > I don't think PEP8 matters at all. Python is not PHP - it has namespaces and consistency between modules never was so important issue for me at least, and I am pretty picky. But it is only a guess. I let data change my mind if there will be some. My theory is that these modules are just full of warts for specific but very common scenarios. Here is something that can be used as an example that it is not about PEP8 https://code.google.com/p/rainforce/wiki/WartsOfPython#measure_time And it takes a lot of energy to collect something like that for the reference. Usually you're just left a feeling that something suxx and this feeling is reinforced every time you hit the wart again and again. It's like AI learning. -- anatoly t. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jsbfox at gmail.com Sun Mar 29 19:04:12 2015 From: jsbfox at gmail.com (Dmitry Kazakov) Date: Sun, 29 Mar 2015 18:04:12 +0100 Subject: [Python-ideas] Add a context manager to keep stream position unchanged Message-ID: I propose adding a context manager to contextlib module, which would restore the stream position of its argument at exit: with contextlib.closing(file_like_object) as file: file.read(1) old = file.tell() with contextlib.keep_stream_position(file) as f: assert file is f assert file_like_object is f # Do something with the file(-like) object e.g. feed it # to `PIL.Image.open` (will consume the whole stream) assert file.tell() == old This CM would allow us to reuse streams efficiently, without requiring to call seek() explicitly. I will open a new issue and submit a patch in case of favorable responses. From ron3200 at gmail.com Sun Mar 29 19:21:00 2015 From: ron3200 at gmail.com (Ron Adam) Date: Sun, 29 Mar 2015 13:21:00 -0400 Subject: [Python-ideas] History on proposals for Macros? In-Reply-To: <20150329015144.GJ25453@ando.pearwood.info> References: <20150328172624.GG25453@ando.pearwood.info> <36A4F7D4-20A4-4BE9-9D86-B423C13A7320@yahoo.com> <20150329015144.GJ25453@ando.pearwood.info> Message-ID: On 03/28/2015 09:51 PM, Steven D'Aprano wrote: > But we might be able to rescue this proposal by dropping the requirement > that the compiler knows when to pass the syntax tree and when to > evaluate it. Suppose instead we had a lightweight syntax for generating > the AST plus grabbing the current context: > > x = 23 > spam(x + 1, !(x+1)) # macro syntax !( ... ) > > > Now the programmer is responsible for deciding when to use an AST and > when to evaluate it, not the compiler, and "macros" become regular > functions which just happen to expect an AST as their argument. Something related to this that I've wanted to experiment with, but is hard to do in python to be able to split a function signature and body, and be able to use them independently. (But in a well defined way.) A signature object could have a default body that returns the closure. And a body (or code) could have a default signature that *takes* a namespace. Then a function becomes ... code(sig(...)) <---> function(...) The separate parts could be created with a decorator. @signature def sig_x(x): pass @code def inc_x(): x += 1 @code def dec_x(): x -= 1 In most cases it's best to think of applying code bodies to names spaces. names = sig_x(0) inc_x(names) dec_x(names) That is nicer than continuations as each code block is a well defined unit that executes to completion and doesn't require suspending the frame. (Yes, it can be done with dictionaries, but that wouldn't give the macro like functionality (see below) this would. And there may be other benifits to having it at a lower more efficient level.) To allow macro like ability a code block needs to be executable in the current scope. That can be done just by doing... code(locals()) # Dependable? And sugar to do that could be... if x < 10: ^^ inc_x #just an example syntax. else: ^^ dec_x # Note the ^^ looks like the M in Macro. ;-) Possibly the decorators could be used with lambda directly to get inline functionality. code(lambda : x + 1) And a bit of sugar to shorten the common uses if needed. spam(x + 1, code(lambda : x + 1)) spam(x + 1, ^^: x + 1) Cheers, Ron From skip.montanaro at gmail.com Sun Mar 29 23:00:28 2015 From: skip.montanaro at gmail.com (Skip Montanaro) Date: Sun, 29 Mar 2015 16:00:28 -0500 Subject: [Python-ideas] Add a context manager to keep stream position unchanged In-Reply-To: References: Message-ID: Interesting idea. It should raise an exception as early as possible if the stream doesn't support seeking... Skip On Mar 29, 2015 12:04 PM, "Dmitry Kazakov" wrote: > I propose adding a context manager to contextlib module, which would > restore the stream position of its argument at exit: > > with contextlib.closing(file_like_object) as file: > file.read(1) > old = file.tell() > with contextlib.keep_stream_position(file) as f: > assert file is f > assert file_like_object is f > # Do something with the file(-like) object e.g. feed it > # to `PIL.Image.open` (will consume the whole stream) > > assert file.tell() == old > > This CM would allow us to reuse streams efficiently, without requiring > to call seek() explicitly. I will open a new issue and submit a patch > in case of favorable responses. > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From skip.montanaro at gmail.com Sun Mar 29 23:01:25 2015 From: skip.montanaro at gmail.com (Skip Montanaro) Date: Sun, 29 Mar 2015 16:01:25 -0500 Subject: [Python-ideas] Add a context manager to keep stream position unchanged In-Reply-To: References: Message-ID: Ugh. Apologies for the top post. Stupid default in Gmail. S -------------- next part -------------- An HTML attachment was scrubbed... URL: From toddrjen at gmail.com Sun Mar 29 23:15:47 2015 From: toddrjen at gmail.com (Todd) Date: Sun, 29 Mar 2015 23:15:47 +0200 Subject: [Python-ideas] Geo coordinates conversion in stdlib In-Reply-To: References: <550F30FB.5010000@sdamon.com> <20150322223815.06082280@fsol> <21A308F0-5F2E-48BB-9F3F-4A3CFAB1D6DD@yahoo.com> <019975C9-137D-4834-80E6-09F471A789B3@yahoo.com> Message-ID: On Mar 29, 2015 6:59 PM, "anatoly techtonik" wrote: > > On Mon, Mar 23, 2015 at 11:29 AM, Andrew Barnert wrote: >> Most people I know who use Python as a serious data processing language don't care much what's in the stdlib because they're using numpy and/or pandas. > > > Yes, and that's sad that the Python still doesn't have the basic building blocks of Pandas in stdlib. It is the proven de-facto popularity achievement that is required for packages to enter stdlib, but that still doesn't happen. Looks like the granularity of package is not sufficient for designing next stdlib - the need is to descent into the level of structures. It isn't just granularity, it is pace of development. Once something is in stdlib, is pretty much impossible to make substantial changes to it. Even if all the other criteria were met, the pace of development of projects like numpy and pandas is far too fast for stdlib. Numpy devs, at least, have explicitly stated they do not want numpy in the stdlib for precisely this reason. "stdlib is where packages go to die". I don't know who originally said that, but it is very true. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ron3200 at gmail.com Sun Mar 29 23:47:59 2015 From: ron3200 at gmail.com (Ron Adam) Date: Sun, 29 Mar 2015 17:47:59 -0400 Subject: [Python-ideas] History on proposals for Macros? In-Reply-To: References: <20150328172624.GG25453@ando.pearwood.info> <36A4F7D4-20A4-4BE9-9D86-B423C13A7320@yahoo.com> <20150329015144.GJ25453@ando.pearwood.info> Message-ID: On 03/29/2015 01:21 PM, Ron Adam wrote: > To allow macro like ability a code block needs to be executable in the > current scope. That can be done just by doing... > > code(locals()) # Dependable? Just to clarify, this should have been... code_obj(locals()) Since I used "code" as the decorator name, this may have been a bit confusing. Cheers, Ron From timothy.c.delaney at gmail.com Mon Mar 30 00:01:52 2015 From: timothy.c.delaney at gmail.com (Tim Delaney) Date: Mon, 30 Mar 2015 09:01:52 +1100 Subject: [Python-ideas] Add a context manager to keep stream position unchanged In-Reply-To: References: Message-ID: On 30 March 2015 at 04:04, Dmitry Kazakov wrote: > I propose adding a context manager to contextlib module, which would > restore the stream position of its argument at exit: > > with contextlib.closing(file_like_object) as file: > file.read(1) > old = file.tell() > with contextlib.keep_stream_position(file) as f: > assert file is f > assert file_like_object is f > # Do something with the file(-like) object e.g. feed it > # to `PIL.Image.open` (will consume the whole stream) > > assert file.tell() == old > > This CM would allow us to reuse streams efficiently, without requiring > to call seek() explicitly. I will open a new issue and submit a patch > in case of favorable responses. > I like the idea. Might need some bikeshedding on the name (maybe restore_position?). To test for seeking, I would suggest having a seek to current position during construction. Not perfect (e.g. there may be streams that can only seek forwards, not backwards) but I think it's the best non-destructive option. Tim Delaney -------------- next part -------------- An HTML attachment was scrubbed... URL: From jheiv at jheiv.com Sat Mar 28 23:27:49 2015 From: jheiv at jheiv.com (jedwards) Date: Sat, 28 Mar 2015 15:27:49 -0700 (PDT) Subject: [Python-ideas] "Loose" descriptors? Message-ID: (I apologize in advance if this was covered previously or elsewhere; a fairly thorough search didn't yield anything.) Currently, we can implement a descriptor as follows: class VerboseDescriptor(object): def __init__(self, init): self.val = init def __get__(self, obj, typ=None): print("Get is %s" % (self.val)) return self.val def __set__(self, obj, val): print("Set to %s" % (val)) self.val = val class Holder(object): val = VerboseDescriptor(2) foo = Holder() _ = foo.val foo.val = 4 Which of course works as expected, displaying: > Get is 2 > Set to 4 But we can't achieve the same indirection without a "Holder"-type class. That is, we can't do the following: class VerboseDescriptor(object): def __init__(self, init): self.val = init def __get__(self, obj, typ=None): print("Get is %s" % (self.val)) return self.val def __set__(self, obj, val): print("Set to %s" % (val)) self.val = val val = VerboseDescriptor(2) _ = val val = 4 Or rather, we can, but what might be expected (the same functionality / output), is not what occurs. My understanding of the resolution of the first example is that python looks for `val` in `foo.__dict__`, then in `Holder.__dict__`, where it finds it, identifies that `Holder.val` implements `__get__`, and handles the indirection accordingly. So `foo.val = 4` ends up being something like `Holder.val.__set__(foo, 4)`. But my question is would it make sense to be able to do this sort of indirection for "loose" objects also, not just attributes of classes or instances? In other words, could descriptors be extended so that they perform the same sort of indirection in the second example as the first? I envision (but with no real knowledge of the implications of this) that just as python identifies that `val` in `Holder.__dict__` implements `__get__`, python could similarly identify that `val` in `globals()` (or `sys.modules[__name__].__dict__`, or wherever is more appropriate) implements `__get__` and handle the indirection. Of course, a syntax issue would arise if this were possible -- notably the class definition of `Holder` would break. So some syntax adjustment would have to be made. But is this reasonable / of interest / attainable? Thanks for your time and consideration. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jheiv at jheiv.com Mon Mar 30 01:19:10 2015 From: jheiv at jheiv.com (James Edwards) Date: Sun, 29 Mar 2015 19:19:10 -0400 Subject: [Python-ideas] "Loose" descriptors? In-Reply-To: References: Message-ID: I sincerely apologize for the duplicate, google-groups decided to push the email to the list as well, a day later. I should have stuck with mailman. On Sat, Mar 28, 2015 at 6:27 PM, jedwards wrote: > (I apologize in advance if this was covered previously or elsewhere; a > fairly thorough search didn't yield anything.) > > Currently, we can implement a descriptor as follows: > > class VerboseDescriptor(object): > def __init__(self, init): > self.val = init > > def __get__(self, obj, typ=None): > print("Get is %s" % (self.val)) > return self.val > > def __set__(self, obj, val): > print("Set to %s" % (val)) > self.val = val > > class Holder(object): > val = VerboseDescriptor(2) > > > foo = Holder() > _ = foo.val > foo.val = 4 > > Which of course works as expected, displaying: >> >> Get is 2 >> Set to 4 > > > But we can't achieve the same indirection without a "Holder"-type class. > > That is, we can't do the following: > > class VerboseDescriptor(object): > def __init__(self, init): > self.val = init > > def __get__(self, obj, typ=None): > print("Get is %s" % (self.val)) > return self.val > > def __set__(self, obj, val): > print("Set to %s" % (val)) > self.val = val > > > val = VerboseDescriptor(2) > _ = val > val = 4 > > Or rather, we can, but what might be expected (the same functionality / > output), is not what occurs. > > My understanding of the resolution of the first example is that python looks > for `val` in `foo.__dict__`, then in `Holder.__dict__`, where it finds it, > identifies that `Holder.val` implements `__get__`, and handles the > indirection accordingly. > > So `foo.val = 4` ends up being something like `Holder.val.__set__(foo, 4)`. > > But my question is would it make sense to be able to do this sort of > indirection for "loose" objects also, not just attributes of classes or > instances? In other words, could descriptors be extended so that they > perform the same sort of indirection in the second example as the first? > > I envision (but with no real knowledge of the implications of this) that > just as python identifies that `val` in `Holder.__dict__` implements > `__get__`, python could similarly identify that `val` in `globals()` (or > `sys.modules[__name__].__dict__`, or wherever is more appropriate) > implements `__get__` and handle the indirection. > > Of course, a syntax issue would arise if this were possible -- notably the > class definition of `Holder` would break. So some syntax adjustment would > have to be made. But is this reasonable / of interest / attainable? > > Thanks for your time and consideration. > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ From njs at pobox.com Mon Mar 30 01:23:17 2015 From: njs at pobox.com (Nathaniel Smith) Date: Sun, 29 Mar 2015 16:23:17 -0700 Subject: [Python-ideas] "Loose" descriptors In-Reply-To: <3E5CAD4F-A573-4403-BEF3-6434A145E592@yahoo.com> References: <55173073.9070600@canterbury.ac.nz> <20150329021707.GK25453@ando.pearwood.info> <3E5CAD4F-A573-4403-BEF3-6434A145E592@yahoo.com> Message-ID: On Sat, Mar 28, 2015 at 10:36 PM, Andrew Barnert wrote: > On Mar 28, 2015, at 19:17, Steven D'Aprano wrote: >> >>> On Sun, Mar 29, 2015 at 11:51:31AM +1300, Greg Ewing wrote: >>> James Edwards wrote: >>>> I envision (but with no real knowledge of the implications of this) >>>> that just as python identifies that `val` in `Holder.__dict__` >>>> implements `__get__`, python could similarly identify that `val` in >>>> `globals()` (or `sys.modules[__name__].__dict__`, or wherever is more >>>> appropriate) implements `__get__` and handle the indirection. >>> >>> This would slow down *all* lookups of global names in >>> order to support a rarely-used feature. Since global >>> name lookup is critical performance-wise, this is >>> likely to meet a lot of resistance. >>> >>> Similar arguments have been made concerning the >>> support of descriptors in module namespaces, which >>> is a closely related idea. >> >> I don't think it's a closely related idea, I think it is *exactly* the >> same idea. If I've understood him correctly, James wants to use >> descriptors in global variables, i.e. module namespaces, not just class >> attributes. > > Yeah, that's how I interpreted it too. > > The obvious way to do this is to provide some way for a module to define not only an instance of the module type, but a subclass and an instance of that subclass. The subclass then provides a natural place for the descriptors to live (and a natural answer to what gets passed for inst and typ). > > There was a proposal for something very much like that some time late last year, IIRC, which might be worth digging up. It got implemented. In 3.5, you can do: # mymodule.py: class MyModuleType(types.ModuleType): @property def someattr(self): print("Fetching someattr!") return 1 sys.modules[__name__].__class__ = MyModuleType and now accessing mymodule.someattr will print a message. This is fast -- it doesn't add any overhead for regular module attribute lookups. OTOH it only works for module attribute lookups, not for direct accesses to the global dict. So code executing inside the module namespace can't write a bare "someattr" to refer to the property; only external users of the module see "someattr". There's some code to make this available on all versions of CPython here: https://github.com/njsmith/metamodule/ which I'll clean up slightly and put up on pypi Any Day Now... -n -- Nathaniel J. Smith -- http://vorpus.org From abarnert at yahoo.com Mon Mar 30 02:36:00 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Sun, 29 Mar 2015 17:36:00 -0700 Subject: [Python-ideas] History on proposals for Macros? In-Reply-To: References: <20150328172624.GG25453@ando.pearwood.info> <36A4F7D4-20A4-4BE9-9D86-B423C13A7320@yahoo.com> <20150329015144.GJ25453@ando.pearwood.info> Message-ID: <4C42926A-2DAE-4BB3-86E6-FC248C5C729D@yahoo.com> > On Mar 29, 2015, at 10:21, Ron Adam wrote: > > > >> On 03/28/2015 09:51 PM, Steven D'Aprano wrote: >> But we might be able to rescue this proposal by dropping the requirement >> that the compiler knows when to pass the syntax tree and when to >> evaluate it. Suppose instead we had a lightweight syntax for generating >> the AST plus grabbing the current context: >> >> x = 23 >> spam(x + 1, !(x+1)) # macro syntax !( ... ) >> >> >> Now the programmer is responsible for deciding when to use an AST and >> when to evaluate it, not the compiler, and "macros" become regular >> functions which just happen to expect an AST as their argument. > > Something related to this that I've wanted to experiment with, but is hard to do in python to be able to split a function signature and body, and be able to use them independently. (But in a well defined way.) Almost everything you're asking for is already there. A function object contains, among other things, a sequence of closure cells, a local and global environment, default parameter values, and a code object. A code object contains, among other things, parameter names, a count of locals, and a bytecode string. You can see the attributes of these objects at runtime, and the inspect module docs describe what they mean. You can also construct these objects at runtime by using their constructors (you have to use types.FunctionType and types.CodeType; the built-in help can show you the parameters). You can also compile source code (or an AST) to a code object with the compile function. You can call a code object with the exec function, which takes a namespace (or, optionally, separate local and global namespaces--and, in a slightly hacky way, you can also override the builtin namespace). There are also Signature objects in the inspect module, but they're not "live" usable objects, they're nicely-organized-for-human-use representations of the signature of a function. So practically you'd use a dummy function object or just a dict or something, and create a new function from its attributes/members and the new code. So, except for minor spelling differences, that's exactly what you're asking for, and it's already there. But if I've guessed right about _why_ you want this, it doesn't do what you'd want it to, and I don't think there's any way it could. Bytecode accesses locals (including arguments), constants, and closure cells by index from the frame object, not by name from a locals dict (although the frame has one of those as well, in case you want to debug or introspect, or call locals()). So, when you call a function, Python sets up the frame object, matching positional and keyword arguments (and default values in the function object) up to parameters and building up the sequence of locals. The frame is also essential for returning and uncaught exceptions (it has a back pointer to the calling frame). The big thing you can't do directly is to create new closure cells programmatically from Python. The compiler has to know which of your locals will be used as closure variables by any embedded functions; it then stores these specially within your code object, so the MAKE_CLOSURE bytecode that creates a function object out of each embedded function can create matching closure cells to store in the embedded function object. This is the part that you need to add into what Python already has, and I'm not sure there's a clean way to do it. But you really should learn how all the existing stuff works (the inspect docs, the dis module, and the help for the constructors in the types module are actually sufficient for this, without having to read the C source code, in 3.4 and later) and find out for yourself, because if I'm wrong, you may come up with something cool. (Plus, it's fun and useful to learn about.) That only gets you through the first half of your message--enough to make inc_x work as a local function (well, almost--without a "nonlocal x" statement it's going to compile x as a local variable rather than a closure variable, and however you execute it, you're just going to get UnboundLocalError). What about the second part, where you execute code in an existing frame? That's even trickier. A frame already has its complete lists of locals, cells, and constants at construction time. If your code objects never used any new locals or constants, and never touched any nonlocal variables that weren't already touched by the calling function, all you need is some kind of "relocation" step that remaps the indices compiled into the bytecode into the indices in the calling frame (there's enough info in the code objects to do the mapping; for actually creating the relocated bytecode from the original you'll want something like the byteplay module, which unfortunately doesn't exist for 3.4--although I have an incomplete port that might be good enough to play with if you're interested). You can almost get away with the "no new locals or nonlocal cells" part, but "no new constants" is pretty restrictive. For example, if you compile inc_x into a fragment that can be executed inline, the number 1 is going to be constant #0 in its code object. And now, you try to "relocate" it to run in a frame with a different code object, and (unless that different code object happened to refer to 1 as a constant as well) there's nothing to match it up to. And again, I don't see a way around this without an even more drastic rearchitecting of how Python frames work--but again, I think it's worth looking for yourself in hopes that I'm wrong. There's another problem: every function body compiler to code ends with a return bytecode. If you didn't write one explicitly, you get the equivalent of "return None". That means that, unless you solve that in some way, executing a fragment inline is always going to return from the calling function. And what do you want to do about explicit return? Or break and continue? Of the two approaches, I think the first one seems cleaner. If you can make a closure cell out of x and then wrap inc_x's code in a normal closure that references it, that still feels like Python. (And having to make "nonlocal x" explicit seems like a good thing, not a limitation.) Fixing up fragments to run in a different frame, and modifying frames at runtime to allow them to be fixed up, seems a lot hackier. And the whole return issue is pretty serious, too. One last possibility to consider is something between the two: a different kind of object, defined differently, like a proc in Ruby (which is defined with a block rather than a def) might solve some of the problems with either approach. And stealing from Ruby again, procs have their own "mini-frames"; there's a two-level stack where every function stack frame has a proc stack frame, which allows a solution to the return-value problem that wouldn't be available with either closures or fragments. (However, note that the return-value problem is much more serious in Ruby, where everything is supposed to be an expression, with a value; in Python you can just say "fragment calls are statements, so they don't have values" if that's what you want.) One last note inline: > A signature object could have a default body that returns the closure. > > And a body (or code) could have a default signature that *takes* a namespace. > > > Then a function becomes ... > > code(sig(...)) <---> function(...) > > > > The separate parts could be created with a decorator. > > @signature > def sig_x(x): pass > > @code > def inc_x(): x += 1 > > @code > def dec_x(): x -= 1 > > > In most cases it's best to think of applying code bodies to names spaces. > > names = sig_x(0) > inc_x(names) > dec_x(names) > > That is nicer than continuations as each code block is a well defined unit that executes to completion and doesn't require suspending the frame. > > (Yes, it can be done with dictionaries, but that wouldn't give the macro like functionality (see below) this would. And there may be other benifits to having it at a lower more efficient level.) > > > To allow macro like ability a code block needs to be executable in the current scope. That can be done just by doing... > > code(locals()) # Dependable? > > > And sugar to do that could be... > > if x < 10: > ^^ inc_x #just an example syntax. > else: > ^^ dec_x # Note the ^^ looks like the M in Macro. ;-) > > > Possibly the decorators could be used with lambda directly to get inline functionality. > > code(lambda : x + 1) This is a very different thing from what you were doing above. A function that modifies a closure cell's value, like inc_x, can't be written as a lambda (because assignments are statements). And this lambda is completely pointless if you're going to use it in a context where you ignore its return value (like the way you used inc_x above). So, I'm not sure what you're trying to do here, but I think you may have another problem to solve on top of the ones I already mentioned. > And a bit of sugar to shorten the common uses if needed. > > spam(x + 1, code(lambda : x + 1)) > > spam(x + 1, ^^: x + 1) > > > Cheers, > Ron > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ From abarnert at yahoo.com Mon Mar 30 02:40:36 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Sun, 29 Mar 2015 17:40:36 -0700 Subject: [Python-ideas] Add a context manager to keep stream position unchanged In-Reply-To: References: Message-ID: <9D3E2056-73CD-4FA6-9D20-53C846BCD71F@yahoo.com> On Mar 29, 2015, at 10:04, Dmitry Kazakov wrote: > > I propose adding a context manager to contextlib module, which would > restore the stream position of its argument at exit: > > with contextlib.closing(file_like_object) as file: > file.read(1) > old = file.tell() > with contextlib.keep_stream_position(file) as f: > assert file is f > assert file_like_object is f > # Do something with the file(-like) object e.g. feed it > # to `PIL.Image.open` (will consume the whole stream) > > assert file.tell() == old > > This CM would allow us to reuse streams efficiently, without requiring > to call seek() explicitly. I will open a new issue and submit a patch > in case of favorable responses. I'm pretty sure an implementation of this is already on PyPI somewhere, although I can't remember exactly where, and it's possible that I just built it myself when needed rather than finding it... Anyway, if I'm right, that would be helpful for both bikeshedding the name, and spotting any potential problems that had to be worked around (the need for f.seek(f.tell()) to catch non-seekable files early is the only non-obvious one I can think of, but if someone has been using this for years in a wide range of projects, they may have found one that none of us will immediately think of). > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ From guido at python.org Mon Mar 30 02:53:43 2015 From: guido at python.org (Guido van Rossum) Date: Sun, 29 Mar 2015 17:53:43 -0700 Subject: [Python-ideas] History on proposals for Macros? In-Reply-To: References: Message-ID: On Sat, Mar 28, 2015 at 9:53 AM, Matthew Rocklin wrote: > Responding to comments off list: > > I'm not referring to C-style preprocessor macros, I'm referring to macros > historically found in functional languages and commonly found in many > user-targeted languages built in the last few years. > Do you have examples and references? IIRC there's something named macros in Scheme but Scheme, unlike Python, completely unifies code and data, and there is a standard in-memory representation for code. > The goal is to create things that look like functions but have access to > the expression that was passed in. > > Some examples where this is useful: > > plot(year, miles / gallon) # Plot with labels determined by > input-expressions, e.g. miles/gallon > > assertRaises(ZeroDivisionError, 1/0) # Evaluate the rhs 1/0 within > assertRaises function, not before > > run_concurrently(f(x), f(y), f(z)) # Run f three times in three > threads controlled by run_concurrently > > Generally one constructs something that looks like a function but, rather > than receiving a pre-evaluated input, receives a syntax tree along with the > associated context. This allows that function-like-thing to manipulate the > expression and to control the context in which the evaluation occurs. > None of the examples need the syntax tree though. The first wants the string, the last probably just want a way to turn an argument into a lambda. > There are lots of arguments against this, mostly focused around potential > misuse. I'm looking for history of such arguments and for a general "Yes, > this is theoretically possible" or "Not a chance in hell" from the > community. Both are fine. > I don't think this is a mainline need in Python, so it's probably both. :-) -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Mon Mar 30 03:27:39 2015 From: guido at python.org (Guido van Rossum) Date: Sun, 29 Mar 2015 18:27:39 -0700 Subject: [Python-ideas] Add a context manager to keep stream position unchanged In-Reply-To: References: Message-ID: On Sun, Mar 29, 2015 at 10:04 AM, Dmitry Kazakov wrote: > I propose adding a context manager to contextlib module, which would > restore the stream position of its argument at exit: > > with contextlib.closing(file_like_object) as file: > file.read(1) > old = file.tell() > with contextlib.keep_stream_position(file) as f: > assert file is f > assert file_like_object is f > # Do something with the file(-like) object e.g. feed it > # to `PIL.Image.open` (will consume the whole stream) > > assert file.tell() == old > > This CM would allow us to reuse streams efficiently, without requiring > to call seek() explicitly. I will open a new issue and submit a patch > in case of favorable responses. > I object to the use of the word "efficiently". You save a line of code and a try/finally block (though the latter probably rarely matters) but the cost of seeking around in the file (negligible though it is) is not reduced by using a CM. It also sounds like this would be a really simple exercise, hardly worth adding to the stdlib. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From ron3200 at gmail.com Mon Mar 30 06:12:03 2015 From: ron3200 at gmail.com (Ron Adam) Date: Mon, 30 Mar 2015 00:12:03 -0400 Subject: [Python-ideas] History on proposals for Macros? In-Reply-To: <4C42926A-2DAE-4BB3-86E6-FC248C5C729D@yahoo.com> References: <20150328172624.GG25453@ando.pearwood.info> <36A4F7D4-20A4-4BE9-9D86-B423C13A7320@yahoo.com> <20150329015144.GJ25453@ando.pearwood.info> <4C42926A-2DAE-4BB3-86E6-FC248C5C729D@yahoo.com> Message-ID: On 03/29/2015 08:36 PM, Andrew Barnert wrote: >>> Something related to this that I've wanted to experiment with, but >>> is hard to do in python to be able to split a function signature and >>> body, and be able to use them independently. (But in a well defined >>> way.) > Almost everything you're asking for is already there. Yes, I have looked into most of what you mention here. > A function object contains, among other things, a sequence of closure > cells, a local and global environment, default parameter values, and a > code object. > > A code object contains, among other things, parameter names, a count of > locals, and a bytecode string. > > You can see the attributes of these objects at runtime, and the inspect > module docs describe what they mean. You can also construct these > objects at runtime by using their constructors (you have to use > types.FunctionType and types.CodeType; the built-in help can show you > the parameters). > > You can also compile source code (or an AST) to a code object with the > compile function. > > You can call a code object with the exec function, which takes a > namespace (or, optionally, separate local and global namespaces--and, in > a slightly hacky way, you can also override the builtin namespace). > > There are also Signature objects in the inspect module, but they're not > "live" usable objects, they're nicely-organized-for-human-use > representations of the signature of a function. So practically you'd use > a dummy function object or just a dict or something, and create a new > function from its attributes/members and the new code. > > So, except for minor spelling differences, that's exactly what you're > asking for, and it's already there. I think it's more than minor spelling differences. :-) I've played around with de-constructing functions and using the constructors to put them back together again, enough to know it's actually quite hard to get everything right with other than the original parts. The exec function may be a good start to experimenting with this. I havn't used it with code objects enough to be familiar with what limits that has. It may not be that difficult to copy the C source for exec and create a new function more specific to this idea. (as a test) Even if that's slow, it may be good enough as a start. > But if I've guessed right about_why_ you want this, it doesn't do what > you'd want it to, and I don't think there's any way it could. > > Bytecode accesses locals (including arguments), constants, and closure > cells by index from the frame object, not by name from a locals dict > (although the frame has one of those as well, in case you want to debug > or introspect, or call locals()). So, when you call a function, Python > sets up the frame object, matching positional and keyword arguments (and > default values in the function object) up to parameters and building up > the sequence of locals. The frame is also essential for returning and > uncaught exceptions (it has a back pointer to the calling frame). This isn't a problem if the callable_code object creates a new frame. It is an issue when running a code block in the current frame. But I think there may be a way to get around that. > The big thing you can't do directly is to create new closure cells > programmatically from Python. The compiler has to know which of your > locals will be used as closure variables by any embedded functions; it > then stores these specially within your code object, so the MAKE_CLOSURE > bytecode that creates a function object out of each embedded function > can create matching closure cells to store in the embedded function > object. This is the part that you need to add into what Python already > has, and I'm not sure there's a clean way to do it. > But you really should learn how all the existing stuff works (the > inspect docs, the dis module, and the help for the constructors in the > types module are actually sufficient for this, without having to read > the C source code, in 3.4 and later) and find out for yourself, because > if I'm wrong, you may come up with something cool. (Plus, it's fun and > useful to learn about.) I'm familiar with most of how python works, and even hacked a bit on ceval.c. (for fun) I haven't played much with the AST side of things, but I do know generally how python is put together and works. > That only gets you through the first half of your message--enough to > make inc_x work as a local function (well, almost--without a "nonlocal > x" statement it's going to compile x as a local variable rather than a > closure variable, and however you execute it, you're just going to get > UnboundLocalError). > > What about the second part, where you execute code in an existing > frame? > > That's even trickier. Yes, I definitely agree. I think one of the tests of a good idea is that it makes something that is normally hard (or tricky), simple and easy. But actually doing that may be quite hard. (or even not possible.) > A frame already has its complete lists of locals, > cells, and constants at construction time. If your code objects never > used any new locals or constants, and never touched any nonlocal > variables that weren't already touched by the calling function, all you > need is some kind of "relocation" step that remaps the indices compiled > into the bytecode into the indices in the calling frame (there's enough > info in the code objects to do the mapping; for actually creating the > relocated bytecode from the original you'll want something like the > byteplay module, which unfortunately doesn't exist for 3.4--although I > have an incomplete port that might be good enough to play with if you're > interested). > You can almost get away with the "no new locals or nonlocal cells" part, > but "no new constants" is pretty restrictive. For example, if you > compile inc_x into a fragment that can be executed inline, the number 1 > is going to be constant #0 in its code object. And now, you try to > "relocate" it to run in a frame with a different code object, and > (unless that different code object happened to refer to 1 as a constant > as well) there's nothing to match it up to. > > And again, I don't see a way around this without an even more drastic > rearchitecting of how Python frames work--but again, I think it's worth > looking for yourself in hopes that I'm wrong. Think of these things as non_local_blocks. The difference is they would use dynamic scope instead of static scope. Or to put it another way, they would inherit the scope they are executed in. > There's another problem: every function body compiler to code ends with > a return bytecode. If you didn't write one explicitly, you get the > equivalent of "return None". That means that, unless you solve that in > some way, executing a fragment inline is always going to return from the > calling function. And what do you want to do about explicit return? Or > break and continue? As a non_local_block, a return could be a needed requirement. It would return the value to the current location in the current frame. Break and continue are harder. Probably give an error the same as what would happen if they are used outside a loop. A break or return would need to be local to the loop. So you can't have a break in a non_local_block unless the block also has the loop in it. That keeps the associated parts local to each other. > Of the two approaches, I think the first one seems cleaner. If you can > make a closure cell out of x and then wrap inc_x's code in a normal > closure that references it, that still feels like Python. (And having to > make "nonlocal x" explicit seems like a good thing, not a limitation.) Agree... I picture the first approach as a needed step to get to the second part. > Fixing up fragments to run in a different frame, and modifying frames at > runtime to allow them to be fixed up, seems a lot hackier. And the whole > return issue is pretty serious, too. > > One last possibility to consider is something between the two: a > different kind of object, defined differently, like a proc in Ruby > (which is defined with a block rather than a def) might solve some of > the problems with either approach. And stealing from Ruby again, procs > have their own "mini-frames"; there's a two-level stack where every > function stack frame has a proc stack frame, which allows a solution to > the return-value problem that wouldn't be available with either closures > or fragments. This may be closer to how I am thinking it would work. :-) > (However, note that the return-value problem is much more > serious in Ruby, where everything is supposed to be an expression, with > a value; in Python you can just say "fragment calls are statements, so > they don't have values" if that's what you want.) It seems to me they can be either. Python ignores None when it's returned by a function and not assigned to anything. And if a value is returned, than it's returned to the current position in the current frame. The return in this case is a non-local-block return. So I think it wouldn't be an issue. > > One last note inline: > >>> A signature object could have a default body that returns the >>> closure. >>> >>> And a body (or code) could have a default signature that*takes* a >>> namespace. >>> >>> >>> Then a function becomes ... >>> >>> code(sig(...)) <---> function(...) >>> >>> >>> >>> The separate parts could be created with a decorator. >>> >>> @signature def sig_x(x): pass >>> >>> @code def inc_x(): x += 1 >>> >>> @code def dec_x(): x -= 1 >>> >>> >>> In most cases it's best to think of applying code bodies to names >>> spaces. >>> >>> names = sig_x(0) inc_x(names) dec_x(names) >>> >>> That is nicer than continuations as each code block is a well >>> defined unit that executes to completion and doesn't require >>> suspending the frame. >>> >>> (Yes, it can be done with dictionaries, but that wouldn't give the >>> macro like functionality (see below) this would. And there may be >>> other benifits to having it at a lower more efficient level.) >>> >>> >>> To allow macro like ability a code block needs to be executable in >>> the current scope. That can be done just by doing... >>> >>> code(locals()) # Dependable? >>> >>> >>> And sugar to do that could be... >>> >>> if x < 10: ^^ inc_x #just an example syntax. else: ^^ dec_x # >>> Note the ^^ looks like the M in Macro.;-) >>> >>> >>> Possibly the decorators could be used with lambda directly to get >>> inline functionality. >>> >>> code(lambda : x + 1) > This is a very different thing from what you were doing above. A > function that modifies a closure cell's value, like inc_x, can't be > written as a lambda (because assignments are statements). And this > lambda is completely pointless if you're going to use it in a context > where you ignore its return value (like the way you used inc_x above). > So, I'm not sure what you're trying to do here, but I think you may have > another problem to solve on top of the ones I already mentioned. It was an incomplete example. It should have been... add_1_to_x = code(lambda: x + 1) and then later you could use it in the same way as above. x = ^^ add_1_to_x This is just an example to show how the first option above connects to the examples below. with "^^: x + 1" being equivalent to "code(lambda: x + 1)". Which would also be equivalent to ... @code def add_1_to_x(): return x + 1 x = ^^ add_1_to_x >>> And a bit of sugar to shorten the common uses if needed. >>> >>> spam(x + 1, code(lambda : x + 1)) >>> >>> spam(x + 1, ^^: x + 1) From jheiv at jheiv.com Mon Mar 30 07:21:30 2015 From: jheiv at jheiv.com (James Edwards) Date: Mon, 30 Mar 2015 01:21:30 -0400 Subject: [Python-ideas] "Loose" descriptors In-Reply-To: <20150329023405.GA7521@k3> References: <20150329023405.GA7521@k3> Message-ID: On Sat, Mar 28, 2015 at 10:34 PM, David Wilson wrote: > FWIW you could probably experiment with this using PyPy's object spaces > feature without modifying the interpreter: > > http://pypy.readthedocs.org/en/latest/objspace-proxies.html > That's David, I'll look into these. "Proxy" type objects were exactly the motivation for my suggestion. I'm working with very deep nested dictionaries -- not the most efficient data structure, but they're a very direct representation of what I'm trying to do. I wanted to create an ItemProxy object, for example: class ItemProxy: def __init__(self, obj, key): self.obj = obj self.key = key def get(self): return self.obj[self.key] def set(self, val): self.obj[self.key] = val d = { '0': {'a':'foo', 'b':'bar'}, '1': {'c':'car', 'd':'baz'}, } p = ItemProxy(d['0'], 'a') print(p.get()) p.set('abc') print(p.get()) But instead of writing print(p.get()), and p.set('abc'), I wanted to just write print(p), and p='abc'. w.r.t. using eval(), since posting I've dug up this Mar 2011 email to this list from Raymond Hettinger: "descriptors outside of classes" https://mail.python.org/pipermail/python-ideas/2011-March/009657.html Here he describes a method for simulating the object.__getattribute__ indirection that descriptors provide explicitly by using eval and passing a locals dictionary. On Sun, Mar 29, 2015 at 7:23 PM, Nathaniel Smith wrote: > It got implemented. In 3.5, you can do: > # mymodule.py: > class MyModuleType(types.ModuleType): > ... > sys.modules[__name__].__class__ = MyModuleType > and now accessing mymodule.someattr will print a message. I saw this approach also, (I believe, I can't test this in 3.5 currently and fails on 3.4) using slightly different process but with the same end result (and possible on 2.7, at least): http://www.dr-josiah.com/2013/12/properties-on-python-modules.html And while this certainly works, it is restricted to modules, and I'd imagine breaks traditional static checking. In order to not break static checking, you could use the approach as described in the link, but the process described there is certainly a workaround and not a great solution. On Sat, Mar 28, 2015 at 10:17 PM, Steven D'Aprano wrote: > (1) Can modules use a dict subclass for __dict__? > (2) Can you replace the __dict__ on the fly like this? > (3) If not, how do you bootstrap the module to have a magic dict instead > of an ordinary dict? If this were possible, it seems like it would provide the flexibility necessary to implement "global" descriptors, or at least module-level descriptors, but as far as I can tell, it is not. Trying to directly replace __dict__ results in AttributeError: readonly attribute Even if you try to replace the module reference in sys.modules with a class that implements a custom __dict__, it doesn't appear to be not used. For example: # mymod.py import sys class CustomDict(dict): def __getattribute__(self, *args, **kwargs): print(["getattribute", args, kwargs]) return dict.__getattribute__(self, *args, **kwargs) def __setattr__(self, *args, **kwargs): print(["setattr", args, kwargs]) return dict.__setattr__(*args, **kwargs) def __missing__(self, key): print("Missing:", key) return None class Module: pass bar = "bar" # Some module attribute, that will get carried into module module = Module() module.__dict__ = CustomDict(globals()) sys.modules[module.__name__] = module When inspected, the dict type is "correct": print(type(mymod.__dict__)) # But when module attributes are accessed/set, the expected print functions aren't shown: print(mymod.bar) # bar (and nothing else) mymod.bar = "baz" # (nothing) print(mymod.bar) # baz (and nothing else) # Attempt to access undefined module attribute, hoping to trigger __missing__ print(mymod.zzz) # AttributeError: 'Module' object has no attribute 'zzz' If it were possible to do this on a per-module basis, I think it would provide much added flexibility, while restricting the lookup penalty to modules that chose to implement it. From abarnert at yahoo.com Mon Mar 30 09:03:47 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Mon, 30 Mar 2015 00:03:47 -0700 Subject: [Python-ideas] History on proposals for Macros? In-Reply-To: References: <20150328172624.GG25453@ando.pearwood.info> <36A4F7D4-20A4-4BE9-9D86-B423C13A7320@yahoo.com> <20150329015144.GJ25453@ando.pearwood.info> <4C42926A-2DAE-4BB3-86E6-FC248C5C729D@yahoo.com> Message-ID: <710F1B95-B063-44C1-9BB4-E8AB9F1B9133@yahoo.com> On Mar 29, 2015, at 21:12, Ron Adam wrote: > > On 03/29/2015 08:36 PM, Andrew Barnert wrote: >>>> Something related to this that I've wanted to experiment with, but >>>> is hard to do in python to be able to split a function signature and >>>> body, and be able to use them independently. (But in a well defined >>>> way.) > >> Almost everything you're asking for is already there. > > Yes, I have looked into most of what you mention here. > >> A function object contains, among other things, a sequence of closure >> cells, a local and global environment, default parameter values, and a >> code object. >> >> A code object contains, among other things, parameter names, a count of >> locals, and a bytecode string. >> >> You can see the attributes of these objects at runtime, and the inspect >> module docs describe what they mean. You can also construct these >> objects at runtime by using their constructors (you have to use >> types.FunctionType and types.CodeType; the built-in help can show you >> the parameters). >> >> You can also compile source code (or an AST) to a code object with the >> compile function. >> >> You can call a code object with the exec function, which takes a >> namespace (or, optionally, separate local and global namespaces--and, in >> a slightly hacky way, you can also override the builtin namespace). >> >> There are also Signature objects in the inspect module, but they're not >> "live" usable objects, they're nicely-organized-for-human-use >> representations of the signature of a function. So practically you'd use >> a dummy function object or just a dict or something, and create a new >> function from its attributes/members and the new code. >> >> So, except for minor spelling differences, that's exactly what you're >> asking for, and it's already there. > > I think it's more than minor spelling differences. :-) The objects you're asking for already exist, but in some cases with slightly different names. There isn't neat calling syntax for things like cloning a function object with a different code object but the same other attributes is a bit painful, but it's just a few two-line wrapper functions that you only have to write once (I think you can even subclass CodeType and FunctionType to add the syntax; if not, you can wrap and delegate). So, what you're asking for really is already part of Python, except for minor spelling differences. The problem is that what you're asking for doesn't give you what you want, because other things (like "the environment") don't work the way you're assuming they do, or because of consequences you probably haven't thought of (like the return issue). > I've played around with de-constructing functions and using the constructors to put them back together again, enough to know it's actually quite hard to get everything right with other than the original parts. It really isn't, once you learn how they work. Getting the values to put into them (assembling bytecode, creating the lnotab, etc.) is a different story, but if your intention is to do that just by compiling function (or fragment) source, it's all done for you. > The exec function may be a good start to experimenting with this. I havn't used it with code objects enough to be familiar with what limits that has. It may not be that difficult to copy the C source for exec and create a new function more specific to this idea. (as a test) > > Even if that's slow, it may be good enough as a start. Why would it be slow? Also, what do you want your function to do differently from exec? What you described is wanting to run a code object in a specified environment; that's what exec does. I think you're thinking of Python environments as if they were scheme environments, but they aren't. In particular, Python closures don't work in terms of accessing variables from the environment by name; they work in terms of accessing cells from the function object. >> But if I've guessed right about_why_ you want this, it doesn't do what >> you'd want it to, and I don't think there's any way it could. >> >> Bytecode accesses locals (including arguments), constants, and closure >> cells by index from the frame object, not by name from a locals dict >> (although the frame has one of those as well, in case you want to debug >> or introspect, or call locals()). So, when you call a function, Python >> sets up the frame object, matching positional and keyword arguments (and >> default values in the function object) up to parameters and building up >> the sequence of locals. The frame is also essential for returning and >> uncaught exceptions (it has a back pointer to the calling frame). > > This isn't a problem if the callable_code object creates a new frame. Yes, it is, as I explained before. There's the minor problem that your code object needs to use nonlocal if it wants to reassign to the calling frame's x variable, and the major problem that there is no way to give it a closure that gives it access to that frame's x variable unless, when that calling frame's code was compiled, there was already an embedded function that needed to close over x. I suppose you could hack up the compiler to generate cells for every local variable instead of just those that are actually needed. Then you could sort of do what you want--instead of exec, you construct a function with the appropriate cells in the closure (by matching up the callee code's freevars with the calling code's cellvars). > It is an issue when running a code block in the current frame. But I think there may be a way to get around that. > > >> The big thing you can't do directly is to create new closure cells >> programmatically from Python. The compiler has to know which of your >> locals will be used as closure variables by any embedded functions; it >> then stores these specially within your code object, so the MAKE_CLOSURE >> bytecode that creates a function object out of each embedded function >> can create matching closure cells to store in the embedded function >> object. This is the part that you need to add into what Python already >> has, and I'm not sure there's a clean way to do it. > > >> But you really should learn how all the existing stuff works (the >> inspect docs, the dis module, and the help for the constructors in the >> types module are actually sufficient for this, without having to read >> the C source code, in 3.4 and later) and find out for yourself, because >> if I'm wrong, you may come up with something cool. (Plus, it's fun and >> useful to learn about.) > > I'm familiar with most of how python works, and even hacked a bit on ceval.c. (for fun) I haven't played much with the AST side of things, but I do know generally how python is put together and works. > > >> That only gets you through the first half of your message--enough to >> make inc_x work as a local function (well, almost--without a "nonlocal >> x" statement it's going to compile x as a local variable rather than a >> closure variable, and however you execute it, you're just going to get >> UnboundLocalError). >> >> What about the second part, where you execute code in an existing >> frame? > > >> That's even trickier. > > Yes, I definitely agree. I think one of the tests of a good idea is that it makes something that is normally hard (or tricky), simple and easy. But actually doing that may be quite hard. (or even not possible.) > >> A frame already has its complete lists of locals, >> cells, and constants at construction time. If your code objects never >> used any new locals or constants, and never touched any nonlocal >> variables that weren't already touched by the calling function, all you >> need is some kind of "relocation" step that remaps the indices compiled >> into the bytecode into the indices in the calling frame (there's enough >> info in the code objects to do the mapping; for actually creating the >> relocated bytecode from the original you'll want something like the >> byteplay module, which unfortunately doesn't exist for 3.4--although I >> have an incomplete port that might be good enough to play with if you're >> interested). > >> You can almost get away with the "no new locals or nonlocal cells" part, >> but "no new constants" is pretty restrictive. For example, if you >> compile inc_x into a fragment that can be executed inline, the number 1 >> is going to be constant #0 in its code object. And now, you try to >> "relocate" it to run in a frame with a different code object, and >> (unless that different code object happened to refer to 1 as a constant >> as well) there's nothing to match it up to. >> >> And again, I don't see a way around this without an even more drastic >> rearchitecting of how Python frames work--but again, I think it's worth >> looking for yourself in hopes that I'm wrong. > > Think of these things as non_local_blocks. The difference is they would use dynamic scope instead of static scope. Or to put it another way, they would inherit the scope they are executed in. That's a bit of a weird name, since it's a block that's executed locally on the current frame, as opposed to the normal non-local way, but... OK. It makes sense conceptually, but practically it doesn't fit in with Python's execution model. First, the scope is partly defined at compilation time--that's where the list of constants comes from, and the lists of local and free names. If the frame's code is still the calling function, none of these things are available in the frame. If, on the other hand, it's the non_local_block (what I was falling the "fragment"), then the calling scope's variables aren't available. (And just copying the caller's code's stuff into a new frame only gives you copies of the caller's variables, which doesn't let you do the main thing you wanted to do.) Unless you somehow make them both available (e.g., the Ruby-style two-level call stack I mentioned), I don't see a way around that. Second, remember that Python bytecode accesses locals by index; unless you do something like the relocation I described above, you have no way to access the calling scope's variables, and that's doable, but not at all trivial, and doesn't seem very Pythonic. Also, even if you do that, you've still got a problem. Python's LEGB rule that decides which scope to find a name in is handled mostly at compile time. The compiler decides whether to emit LOAD_FAST or one of the other LOAD_*, and likewise for STORE_*, based on the static lexical scope the name is defined in. That's going to make it very hard to hack in dynamic scope on top of Python bytecode--the "relocation" has to actually involve not just renumbering locals, but reproducing all the work the compiler does to decide what's local/global/etc. and replacing instructions as appropriate. As an alternative, maybe you don't want code objects at all, but rather AST objects. Then they can be compiled to bytecode in a given scope (with the builtin compile function) and then executed there (with exec). This solves most of the new problems added by the dynamic scoping/non_local_block/fragment idea while still allowing most of the benefits. The downside, of course, is that you're compiling stuff all over the place--but that's how dynamic code works in most Lisps, so... >> There's another problem: every function body compiler to code ends with >> a return bytecode. If you didn't write one explicitly, you get the >> equivalent of "return None". That means that, unless you solve that in >> some way, executing a fragment inline is always going to return from the >> calling function. And what do you want to do about explicit return? Or >> break and continue? > > As a non_local_block, a return could be a needed requirement. It would return the value to the current location in the current frame. Do you mean that a call to a non_local_block is an expression, and the value of the expression is the value returned by the block? If so, that makes sense, but then the RETURN_VALUE handler in the interpreter has to be sensitive to the context it's in and do two different things, or the compiler has to be sensitive to what it's compiling--which seems impossible, given that you want to def a regular function and then decorate it after the fact--and issue a different opcode for return (and for the implicit return None). > Break and continue are harder. Probably give an error the same as what would happen if they are used outside a loop. A break or return would need to be local to the loop. So you can't have a break in a non_local_block unless the block also has the loop in it. That keeps the associated parts local to each other. That definitely makes things simpler--and, I think, better. It will piss off people who expect (from Lisp or Ruby) to be able to do flow control across the boundaries of a non_local_block, but too bad for them. :) >> Of the two approaches, I think the first one seems cleaner. If you can >> make a closure cell out of x and then wrap inc_x's code in a normal >> closure that references it, that still feels like Python. (And having to >> make "nonlocal x" explicit seems like a good thing, not a limitation.) > > Agree... I picture the first approach as a needed step to get to the second part. I don't think it really is; most of what you need to solve for the first becomes irrelevant for the second. For example, figuring out a way to dynamically generate closure cells to share variables across frame's is useless when you switch to running with the parent's variables as locals in the same frame. >> Fixing up fragments to run in a different frame, and modifying frames at >> runtime to allow them to be fixed up, seems a lot hackier. And the whole >> return issue is pretty serious, too. >> >> One last possibility to consider is something between the two: a >> different kind of object, defined differently, like a proc in Ruby >> (which is defined with a block rather than a def) might solve some of >> the problems with either approach. And stealing from Ruby again, procs >> have their own "mini-frames"; there's a two-level stack where every >> function stack frame has a proc stack frame, which allows a solution to >> the return-value problem that wouldn't be available with either closures >> or fragments. > > This may be closer to how I am thinking it would work. :-) It sounds like it might be. I think it's a really clumsy solution, but obviously it can work or Ruby wouldn't work. :) Notice that Ruby also avoids a lot of the problems below by having completely different syntax and semantics for defining procs vs. functions, and not allowing you to convert them into each other. But if you don't want that, then you have to solve all the problems they got around this way. >> (However, note that the return-value problem is much more >> serious in Ruby, where everything is supposed to be an expression, with >> a value; in Python you can just say "fragment calls are statements, so >> they don't have values" if that's what you want.) > > It seems to me they can be either. Python ignores None when it's returned by a function and not assigned to anything. And if a value is returned, than it's returned to the current position in the current frame. No, that's not how things work. Python doesn't do anything special with None. A value is _always_ returned, whether None or otherwise. That value always becomes the value of the calling expression. And Python certainly doesn't care whether you assign it to something--e.g., you can use it inside a larger expression or return it or yield it without assigning it to anything. Of course if the outermost expression is part of an expression statement, the value that it evaluated to is ignored, but that has nothing to do with the value being None, or coming from a function call, or anything else; an expression statement just means "evaluate this expression, then throw away the results". Also, you're missing the bigger point: look at how RETURN_VALUE actually works. If you're running inside the caller's scope, it's going to return from the caller's scope unless you do something (which you need to figure out) to make that not true. > The return in this case is a non-local-block return. So I think it wouldn't be an issue. What does "a non-local-block return" mean at the implementation level? A different opcode from RETURN_VALUE? A RETURN_VALUE executed from within a non_local_block object's code? A RETURN_VALUE executed from a frame that has a non_local_block running? Whatever the answer, how does the compiler or interpreter (as appropriate) know to do that? >> One last note inline: >> >>>> A signature object could have a default body that returns the >>>> closure. >>>> >>>> And a body (or code) could have a default signature that*takes* a >>>> namespace. >>>> >>>> >>>> Then a function becomes ... >>>> >>>> code(sig(...)) <---> function(...) >>>> >>>> >>>> >>>> The separate parts could be created with a decorator. >>>> >>>> @signature def sig_x(x): pass >>>> >>>> @code def inc_x(): x += 1 >>>> >>>> @code def dec_x(): x -= 1 >>>> >>>> >>>> In most cases it's best to think of applying code bodies to names >>>> spaces. >>>> >>>> names = sig_x(0) inc_x(names) dec_x(names) >>>> >>>> That is nicer than continuations as each code block is a well >>>> defined unit that executes to completion and doesn't require >>>> suspending the frame. >>>> >>>> (Yes, it can be done with dictionaries, but that wouldn't give the >>>> macro like functionality (see below) this would. And there may be >>>> other benifits to having it at a lower more efficient level.) >>>> >>>> >>>> To allow macro like ability a code block needs to be executable in >>>> the current scope. That can be done just by doing... >>>> >>>> code(locals()) # Dependable? >>>> >>>> >>>> And sugar to do that could be... >>>> >>>> if x < 10: ^^ inc_x #just an example syntax. else: ^^ dec_x # >>>> Note the ^^ looks like the M in Macro.;-) >>>> >>>> >>>> Possibly the decorators could be used with lambda directly to get >>>> inline functionality. >>>> >>>> code(lambda : x + 1) > >> This is a very different thing from what you were doing above. A >> function that modifies a closure cell's value, like inc_x, can't be >> written as a lambda (because assignments are statements). And this >> lambda is completely pointless if you're going to use it in a context >> where you ignore its return value (like the way you used inc_x above). >> So, I'm not sure what you're trying to do here, but I think you may have >> another problem to solve on top of the ones I already mentioned. > > It was an incomplete example. It should have been... > > add_1_to_x = code(lambda: x + 1) > > and then later you could use it in the same way as above. > > x = ^^ add_1_to_x OK, it sounds like what you're really looking for here is that code(spam) returns a function that's just like spam, but all of its variables (although you still have to work out what that means--remember that Python has already decided local vs. cell vs. global at compile time, before you even get to this code function) will use dynamic rather than lexical scoping. All of the other stuff seems to be irrelevant. In fact, maybe it would be simpler to just do what Lisp does: explicitly define individual _variables_ as dynamically scoped, effectively the same way we can already define variables as global or nonlocal, instead of compiling a function and then trying to turn some of its variables into dynamic variables after the fact. And the good news is, I'm 99% sure someone already did this and wrote a blog post about it. I don't know where, and it may be a few years and versions out of date, but it would be nice if you could look at what he did, see that you're 90% of the way to what you want, and just have to solve the last 10%. Plus, you can experiment with this without hacking up anything, with a bit of clumsiness. It's pretty easy to create a class whose instances dynamically scope their attributes with an explicit stack. (If it isn't obvious how, let me know and I'll write it for you.) Then you just instantiate that class (globally, if you want), and have both the caller and the callee use an attribute of that instance instead of a normal variable whenever you want a dynamically-scoped variable, and you're done. You can write nice examples that actually work in Python today to show how this would be useful, and then compare to how much better it would look with real dynamic variable support. > This is just an example to show how the first option above connects to the examples below. with "^^: x + 1" being equivalent to "code(lambda: x + 1)". > > Which would also be equivalent to ... > > @code > def add_1_to_x(): > return x + 1 > > x = ^^ add_1_to_x > > > >>>> And a bit of sugar to shorten the common uses if needed. >>>> >>>> spam(x + 1, code(lambda : x + 1)) >>>> >>>> spam(x + 1, ^^: x + 1) > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ From jsbfox at gmail.com Mon Mar 30 10:35:17 2015 From: jsbfox at gmail.com (Dmitry Kazakov) Date: Mon, 30 Mar 2015 09:35:17 +0100 Subject: [Python-ideas] Add a context manager to keep stream position unchanged In-Reply-To: References: Message-ID: If the final resolution is "we don't need this in the library", that's fine - I agree that the proposal is trivial and doesn't gain much. OTOH the contextlib.closing can also be said to be a simple exercise, because its implementation is 22 lines long and it's equivalent to @contextmanager def closing(thing): try: yield thing finally: thing.close() (from the docs) -------------- next part -------------- An HTML attachment was scrubbed... URL: From abarnert at yahoo.com Mon Mar 30 11:03:37 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Mon, 30 Mar 2015 02:03:37 -0700 Subject: [Python-ideas] Add a context manager to keep stream position unchanged In-Reply-To: References: Message-ID: On Mar 30, 2015, at 01:35, Dmitry Kazakov wrote: > > If the final resolution is "we don't need this in the library", that's fine - I agree that the proposal is trivial and doesn't gain much. OTOH the contextlib.closing can also be said to be a simple exercise, because its implementation is 22 lines long and it's equivalent to > > @contextmanager > def closing(thing): > try: > yield thing > finally: > thing.close() > > (from the docs) I suspect that if closing were proposed today (or if you were designing Python from scratch today) it never would have made it into contextlib. At the time, nobody had any experience with context managers. And there used to be tons of types that it was useful for; now, most of those types are context managers and therefore don't need closing. Python didn't come with pip, so having things in the stdlib was a lot more important. And it was one of the motivating examples in the PEP, which made it a look like a no-brainer addition. That doesn't mean there might not be a good argument for this idea. For example, if it turns out to be something that _looks_ trivial to code but is very easy to get subtly wrong, so a lot of people are likely to do it themselves and end up with buggy code, then it sounds like a good candidate. (Well, it would to me, but then I'm not who you have to convince...) But if you agree that it really is trivial, I don't see why it needs to be in the stdlib. -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Mon Mar 30 13:11:11 2015 From: steve at pearwood.info (Steven D'Aprano) Date: Mon, 30 Mar 2015 22:11:11 +1100 Subject: [Python-ideas] History on proposals for Macros? In-Reply-To: <710F1B95-B063-44C1-9BB4-E8AB9F1B9133@yahoo.com> References: <20150328172624.GG25453@ando.pearwood.info> <36A4F7D4-20A4-4BE9-9D86-B423C13A7320@yahoo.com> <20150329015144.GJ25453@ando.pearwood.info> <4C42926A-2DAE-4BB3-86E6-FC248C5C729D@yahoo.com> <710F1B95-B063-44C1-9BB4-E8AB9F1B9133@yahoo.com> Message-ID: <20150330111110.GF15925@ando.pearwood.info> On Mon, Mar 30, 2015 at 12:03:47AM -0700, Andrew Barnert wrote: > On Mar 29, 2015, at 21:12, Ron Adam wrote: [snip stuff about dynamic scoping] Raymond Hettinger's ChainMap (now also in the std lib) may possibly be used to emulate different scoping rules, including dynamic. http://code.activestate.com/recipes/577434-nested-contexts-a-chain-of-mapping-objects/ http://code.activestate.com/recipes/305268-chained-map-lookups/ If you can build a function object with __globals__ set to a chained map, you may be able to accomplish most of what you are discussing. By memory, __globals__ needs to be a dict, so you have to use a dict subclass, something like this: class Context(ChainMap, dict): pass -- Steve From jsbueno at python.org.br Mon Mar 30 14:52:28 2015 From: jsbueno at python.org.br (Joao S. O. Bueno) Date: Mon, 30 Mar 2015 09:52:28 -0300 Subject: [Python-ideas] Add a context manager to keep stream position unchanged In-Reply-To: References: Message-ID: On 29 March 2015 at 19:01, Tim Delaney wrote: > On 30 March 2015 at 04:04, Dmitry Kazakov wrote: >> >> I propose adding a context manager to contextlib module, which would >> restore the stream position of its argument at exit: >> >> with contextlib.closing(file_like_object) as file: >> file.read(1) >> old = file.tell() >> with contextlib.keep_stream_position(file) as f: >> assert file is f >> assert file_like_object is f >> # Do something with the file(-like) object e.g. feed it >> # to `PIL.Image.open` (will consume the whole stream) >> >> assert file.tell() == old >> >> This CM would allow us to reuse streams efficiently, without requiring >> to call seek() explicitly. I will open a new issue and submit a patch >> in case of favorable responses. > > > I like the idea. Might need some bikeshedding on the name (maybe > restore_position?). > > To test for seeking, I would suggest having a seek to current position > during construction. Not perfect (e.g. there may be streams that can only > seek forwards, not backwards) but I think it's the best non-destructive > option. > If that is being done, anyway, I propose an optional parameter that would allow the context to internally use itertools.tee if the stream can't seek itself. Maybe with even three states: "strict": raises an exception if the stream is not [back]seekable "default": creates a tee cache if the stream is not back-seekable "cache": always cache. (And maybe a maxbuffer parameter to go along? ) > Tim Delaney > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ From abarnert at yahoo.com Mon Mar 30 15:52:16 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Mon, 30 Mar 2015 06:52:16 -0700 Subject: [Python-ideas] Add a context manager to keep stream position unchanged In-Reply-To: References: Message-ID: <1345887E-789E-4F74-B825-01E54A460135@yahoo.com> On Mar 30, 2015, at 05:52, Joao S. O. Bueno wrote: > >> On 29 March 2015 at 19:01, Tim Delaney wrote: >>> On 30 March 2015 at 04:04, Dmitry Kazakov wrote: >>> >>> I propose adding a context manager to contextlib module, which would >>> restore the stream position of its argument at exit: >>> >>> with contextlib.closing(file_like_object) as file: >>> file.read(1) >>> old = file.tell() >>> with contextlib.keep_stream_position(file) as f: >>> assert file is f >>> assert file_like_object is f >>> # Do something with the file(-like) object e.g. feed it >>> # to `PIL.Image.open` (will consume the whole stream) >>> >>> assert file.tell() == old >>> >>> This CM would allow us to reuse streams efficiently, without requiring >>> to call seek() explicitly. I will open a new issue and submit a patch >>> in case of favorable responses. >> >> >> I like the idea. Might need some bikeshedding on the name (maybe >> restore_position?). >> >> To test for seeking, I would suggest having a seek to current position >> during construction. Not perfect (e.g. there may be streams that can only >> seek forwards, not backwards) but I think it's the best non-destructive >> option. > > If that is being done, anyway, I propose an optional parameter that > would allow the context to internally use itertools.tee if the stream > can't seek itself. That sounds cool, but... If you have an actual file object with seek and tell, you're often using other methods besides iterating lines (like read), and tee won't help with those, so it might be a little misleading. > Maybe with even three states: > "strict": raises an exception if the stream is not [back]seekable > "default": creates a tee cache if the stream is not back-seekable > "cache": always cache. > > (And maybe a maxbuffer parameter to go along? ) > > > >> Tim Delaney >> >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ From rosuav at gmail.com Mon Mar 30 15:59:10 2015 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 31 Mar 2015 00:59:10 +1100 Subject: [Python-ideas] Add a context manager to keep stream position unchanged In-Reply-To: <1345887E-789E-4F74-B825-01E54A460135@yahoo.com> References: <1345887E-789E-4F74-B825-01E54A460135@yahoo.com> Message-ID: On Tue, Mar 31, 2015 at 12:52 AM, Andrew Barnert wrote: >> If that is being done, anyway, I propose an optional parameter that >> would allow the context to internally use itertools.tee if the stream >> can't seek itself. > > That sounds cool, but... If you have an actual file object with seek and tell, you're often using other methods besides iterating lines (like read), and tee won't help with those, so it might be a little misleading. > That shouldn't be an issue, though, as those will just use seek/tell. I'm more wondering about the ones that _do_ need the tee'ing; how, after the context manager completes, do you have the file object return something from the other half of the tee? I'm not sure this mode will work. If the stream can't use seek(), this context manager should simply let the exception bubble IMO. ChrisA From abarnert at yahoo.com Mon Mar 30 17:36:40 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Mon, 30 Mar 2015 08:36:40 -0700 Subject: [Python-ideas] Add a context manager to keep stream position unchanged In-Reply-To: References: <1345887E-789E-4F74-B825-01E54A460135@yahoo.com> Message-ID: On Mar 30, 2015, at 06:59, Chris Angelico wrote: > > On Tue, Mar 31, 2015 at 12:52 AM, Andrew Barnert > wrote: >>> If that is being done, anyway, I propose an optional parameter that >>> would allow the context to internally use itertools.tee if the stream >>> can't seek itself. >> >> That sounds cool, but... If you have an actual file object with seek and tell, you're often using other methods besides iterating lines (like read), and tee won't help with those, so it might be a little misleading. > > That shouldn't be an issue, though, as those will just use seek/tell. The whole point of this context manager is to seek at enter and tell at exit. If it's useful to people using files as iterators, it's at least as useful to people using them as files. More so, in fact, because most code that asks for a file-like iterator isn't going to want to assume it's actually a file, while code that asks for something that has file methods doesn't have that problem. > I'm more wondering about the ones that _do_ need the tee'ing; how, > after the context manager completes, do you have the file object > return something from the other half of the tee? You're right; tee doesn't do anything to the iterable you give it, it creates two new iterators that share the iterable and a cache. So the caller has to use one of those new iterators, which would be clumsy. Something like: with restoring_position(f, tee_if_needed=True) as f, flocal: # do stuff with flocal as an iterator of lines # do stuff with f, which is an iterator of the same lines Of course this only works if f only lives within this scope, or is a closure or global variable (if it was a normal parameter, as soon as you return to the caller, he's got the original file object seeked to the wrong place). On top of the issue with f now being an iterator rather than whatever kind of file-like object you started with, the potential for confusion and misuse is so high that I don't think I'd do this even internally to my own project, much less suggest it for the stdlib... But at least the implementation is still pretty simple: @contextmanager def restoring_position(t, tee_if_needed=False): try: pos = f.tell() f.seek(pos) except (AttributeError, IForgetTheExceptionForNotSeekable): if not tee_jf_needed: raise f, flocal = tee(f) frestore = None else: flocal = frestore = f try: yield f, flocal finally: if frestore is not None: frestore.seek(pos) A better solution might be to build an actual file tee--a BufferedReader or TextIOWrapper that wraps another object of the same type, doesn't discard used data from the buffer, and can therefore seek back to the start of its buffer without seeking the wrapped file. But even then, I don't think I'd use it this way. I'd construct the wrapper explicitly, and give it a method that returns a pos-restoring context manager. > I'm not sure this > mode will work. If the stream can't use seek(), this context manager > should simply let the exception bubble IMO. Agreed. If you really want to, just try the context manager, and fall back to tee if it fails. A couple lines of code, and a lot more explicit and readable, and probably not that commonly needed... From ron3200 at gmail.com Mon Mar 30 20:55:47 2015 From: ron3200 at gmail.com (Ron Adam) Date: Mon, 30 Mar 2015 14:55:47 -0400 Subject: [Python-ideas] History on proposals for Macros? In-Reply-To: <710F1B95-B063-44C1-9BB4-E8AB9F1B9133@yahoo.com> References: <20150328172624.GG25453@ando.pearwood.info> <36A4F7D4-20A4-4BE9-9D86-B423C13A7320@yahoo.com> <20150329015144.GJ25453@ando.pearwood.info> <4C42926A-2DAE-4BB3-86E6-FC248C5C729D@yahoo.com> <710F1B95-B063-44C1-9BB4-E8AB9F1B9133@yahoo.com> Message-ID: On 03/30/2015 03:03 AM, Andrew Barnert wrote: >>> Break and continue are harder. Probably give an error the same as >>> what would happen if they are used outside a loop. A break or return >>> would need to be local to the loop. So you can't have a break in a >>> non_local_block unless the block also has the loop in it. That keeps >>> the associated parts local to each other. > > That definitely makes things simpler--and, I think, better. It will piss > off people who expect (from Lisp or Ruby) to be able to do flow control > across the boundaries of a non_local_block, but too bad for them. Or in my own language I wrote to test ideas like this. In it, the keywords are objects so you can return a statement/keyword from a function and have it execute at that location. So for example a function can increment a counter and return a "nil" keyword, (a no-op like pythons pass keyword), and once a limit is reached, return a "break" keyword. Which breaks the loop. The memory model is by static hash-map lookup. Each defined function object gets a reference to the parents hash-map. So name resolution walks the hash-map tree until it finds the name. There are a number of ways to make that more efficient, but for now it keeps the language simple. Python on the other hand does many more checks at compile time, and pull's the name references into the code object. That allows for smaller and faster byte code execution. This seems to be where the main issues are. But as you noted, many of the pieces are already there. So it shouldn't be that difficult to come up with a working test implementation. For now, experimenting with exec may be the best way to test the idea. And possibly come back with some real code to discuss and see where that goes. I think until some real code is written we will go in circles pointing out things wrong with the idea. So lets wait a bit for some real code and examples. When people think of macro's they may be thinking of several different concepts. One is to redefine a statement or expression in a way that makes it easier to use. In that case the expression is transposed to another form at compile time. Lisp macro's work in this way. Another is to in-line a block of code defined in a single location to numerous other locations. Generally this is done with pre-processors. I think in-lining python functions has been discussed before and that idea would overlap this one. My interest in functions that can be taken apart and reused with signature/scope objects is a bit different. The idea of mutating the name space by applying code to it rather than calling code by applying values to it. (A normal function call.) This of course is what objects do, they have a state and methods are used to alter the state. But there are other things I think may be of interest in this idea that may relate back to other areas of python. (NOTE: Just read Stevens message about Raymond's chainmap. So I'm going to see it is useful.) And the reason for bringing this idea up here was I think it could be used to implement the lite macro behaviour that was suggested with a bit of added syntax. On the other hand it appears to me, that Python is going in the direction of making it easier to compile to C code. More dynamic features may not be helpful in the long run. >> >It was an incomplete example. It should have been... >> > >> > add_1_to_x = code(lambda: x + 1) >> > >> >and then later you could use it in the same way as above. >> > >> > x = ^^ add_1_to_x > OK, it sounds like what you're really looking for here is that code(spam) returns a function that's just like spam, but all of its variables (although you still have to work out what that means--remember that Python has already decided local vs. cell vs. global at compile time, before you even get to this code function) will use dynamic rather than lexical scoping. All of the other stuff seems to be irrelevant. > > In fact, maybe it would be simpler to just do what Lisp does: explicitly > define individual_variables_ as dynamically scoped, effectively the same > way we can already define variables as global or nonlocal, instead of > compiling a function and then trying to turn some of its variables into > dynamic variables after the fact. It's something to try, but if a code block needs boiler plate to work, or the function it's put in needs it, it really isn't go to be very nice. > And the good news is, I'm 99% sure someone already did this and wrote a > blog post about it. I don't know where, and it may be a few years and > versions out of date, but it would be nice if you could look at what he > did, see that you're 90% of the way to what you want, and just have to > solve the last 10%. > > Plus, you can experiment with this without hacking up anything, with a > bit of clumsiness. It's pretty easy to create a class whose instances > dynamically scope their attributes with an explicit stack. (If it isn't > obvious how, let me know and I'll write it for you.) Then you just > instantiate that class (globally, if you want), and have both the caller > and the callee use an attribute of that instance instead of a normal > variable whenever you want a dynamically-scoped variable, and you're done. > You can write nice examples that actually work in Python today to show how > this would be useful, and then compare to how much better it would look > with real dynamic variable support. I'm going to play with the idea a bit over the next few days. :-) Cheers, Ron >>> This is just an example to show how the first option above connects >>> tothe examples below. with "^^: x + 1" being equivalent to "code(lambda: x + 1)". >> > >> >Which would also be equivalent to ... >> > >> > @code >> > def add_1_to_x(): >> > return x + 1 >> > >> > x = ^^ add_1_to_x >> > >> > >> > >>>>> >>>>And a bit of sugar to shorten the common uses if needed. >>>>> >>>> >>>>> >>>>spam(x + 1, code(lambda : x + 1)) >>>>> >>>> >>>>> >>>>spam(x + 1, ^^: x + 1) From abarnert at yahoo.com Mon Mar 30 22:54:54 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Mon, 30 Mar 2015 13:54:54 -0700 Subject: [Python-ideas] History on proposals for Macros? In-Reply-To: References: <20150328172624.GG25453@ando.pearwood.info> <36A4F7D4-20A4-4BE9-9D86-B423C13A7320@yahoo.com> <20150329015144.GJ25453@ando.pearwood.info> <4C42926A-2DAE-4BB3-86E6-FC248C5C729D@yahoo.com> <710F1B95-B063-44C1-9BB4-E8AB9F1B9133@yahoo.com> Message-ID: <4E099DC0-B146-476E-833C-3CF73CD53B9A@yahoo.com> On Mar 30, 2015, at 11:55, Ron Adam wrote: > > On 03/30/2015 03:03 AM, Andrew Barnert wrote: [snip] > And the reason for bringing this idea up here was I think it could be used to implement the lite macro behaviour that was suggested with a bit of added syntax. > > On the other hand it appears to me, that Python is going in the direction of making it easier to compile to C code. More dynamic features may not be helpful in the long run. I don't think this is true. There definitely isn't any momentum in that direction--shedskin and its two competitors are dead, while PyPy and numba are alive and kicking ass. And I don't think it's desirable, or that the core developers think it's desirable. If you think Guido's and Jukka's static type annotations are a step in the direction of static compilation, read the two PEPs; that's very explicitly not a goal, at least in the near term. (And the impression I get from the list discussions is that Guido is skeptical that it will be in the long term, but he's willing to keep an open mind.) Anyway, usually, macros are used to generate a few functions that are called many times, not to generate a whole lot of functions that are only called once, so they don't cause any problems for a decent tracing JIT. >>> >It was an incomplete example. It should have been... >>> > >>> > add_1_to_x = code(lambda: x + 1) >>> > >>> >and then later you could use it in the same way as above. >>> > >>> > x = ^^ add_1_to_x >> OK, it sounds like what you're really looking for here is that code(spam) returns a function that's just like spam, but all of its variables (although you still have to work out what that means--remember that Python has already decided local vs. cell vs. global at compile time, before you even get to this code function) will use dynamic rather than lexical scoping. All of the other stuff seems to be irrelevant. >> >> In fact, maybe it would be simpler to just do what Lisp does: explicitly >> define individual_variables_ as dynamically scoped, effectively the same >> way we can already define variables as global or nonlocal, instead of >> compiling a function and then trying to turn some of its variables into >> dynamic variables after the fact. > > It's something to try, but if a code block needs boiler plate to work, or the function it's put in needs it, it really isn't go to be very nice. I think I didn't explain this very well. Take a step back. Instead of having functions whose variables are all lexically scoped, and blocks whose variables are all dynamically scoped, what if you had just one kind of function, but two kinds of variables? Exactly like defvar in Lisp (although hopefully with a better name...). That's obviously more flexible. And it should be much simpler to implement. (And it's dead simple to fake in standard CPython 3.4, without hacking anything--as Steven pointed out, ChainMap makes it even easier). Once you have dynamic variables, it's not that hard to add a new defblock statement that acts just like def except its variables are dynamic by default, instead of lexical. That skips over all the problems with your first version and immediately gets you your second version, and it avoids most of the problems there as well. The only thing you're missing in the end is the ability to convert functions to blocks and vice-versa at runtime. That's the part that causes all the problems, and I don't think is actually necessary for what you want (but if I'm wrong, I guess never mind :). From ron3200 at gmail.com Tue Mar 31 00:36:21 2015 From: ron3200 at gmail.com (Ron Adam) Date: Mon, 30 Mar 2015 18:36:21 -0400 Subject: [Python-ideas] History on proposals for Macros? In-Reply-To: <4E099DC0-B146-476E-833C-3CF73CD53B9A@yahoo.com> References: <20150328172624.GG25453@ando.pearwood.info> <36A4F7D4-20A4-4BE9-9D86-B423C13A7320@yahoo.com> <20150329015144.GJ25453@ando.pearwood.info> <4C42926A-2DAE-4BB3-86E6-FC248C5C729D@yahoo.com> <710F1B95-B063-44C1-9BB4-E8AB9F1B9133@yahoo.com> <4E099DC0-B146-476E-833C-3CF73CD53B9A@yahoo.com> Message-ID: On 03/30/2015 04:54 PM, Andrew Barnert wrote: > That skips over all the problems with your first version and immediately > gets you your second version, and it avoids most of the problems there > as well. The only thing you're missing in the end is the ability to > convert functions to blocks and vice-versa at runtime. That's the part > that causes all the problems, and I don't think is actually necessary > for what you want (but if I'm wrong, I guess never mind:). I don't think its necessary either and converting function blocks to an insertable code block isn't what I had in mind. If you can run a function block in specified frame, then it isn't needed. Just grab the current frame, execute the function block with it, as a function. Of course the programmer would be responsible for making sure the names in the code block are available in the name space used with it. Name errors should propagate normally. result = call_with_frame(function_code_object, sys.getframe(0)) But I think it will still runs into the name issues you mentioned earlier. And as you also mentioned it's quite possible someone has already done a call_with_frame function. I just haven't found it yet. Cheers, Ron From tjreedy at udel.edu Tue Mar 31 02:41:26 2015 From: tjreedy at udel.edu (Terry Reedy) Date: Mon, 30 Mar 2015 20:41:26 -0400 Subject: [Python-ideas] PEP 484: Generating test inputs from type hints Message-ID: PEP 484 proposes "a standard syntax for type annotations", principally for static analysis (and other static uses), with runtime checking or optimization mentioned as a possibility. Another (real) runtime use is generating unit test inputs. Today on python-list, David MacIver announced Hypothesis, a testing library that does just that. (The thread title is "Hypothesis 1.0: A production quality property-based testing library for Python".) Here is an example from http://hypothesis.readthedocs.org/en/latest/quickstart.html from hypothesis import given @given([int]) def test_reversing_twice_gives_same_list(xs): assert xs == list(reversed(reversed(xs))) When a test runner such as pytest calls the wrapper, it calls the real function with a variety of int lists (including [], but details not important here). @given() also works with unittest TestCase methods. When I mentioned PEP 484 and the alternative syntax List(int), he replied "Having something like that as standard would be great for Hypothesis and I intend to support it once it becomes available." I presume 'support' could and would include a version of given() that accessed .__annotations__. Of course, if every 'test_xyz' were to be decorated with the same decorator, then the decorator could be omitted and the fuzzing moved into the test runner. -- Terry Jan Reedy From guido at python.org Tue Mar 31 04:18:24 2015 From: guido at python.org (Guido van Rossum) Date: Mon, 30 Mar 2015 19:18:24 -0700 Subject: [Python-ideas] PEP 484: Generating test inputs from type hints In-Reply-To: References: Message-ID: On Mon, Mar 30, 2015 at 5:41 PM, Terry Reedy wrote: > PEP 484 proposes "a standard syntax for type annotations", principally for > static analysis (and other static uses), with runtime checking or > optimization mentioned as a possibility. > > Another (real) runtime use is generating unit test inputs. Today on > python-list, David MacIver announced Hypothesis, a testing library that > does just that. (The thread title is "Hypothesis 1.0: A production quality > property-based testing library for Python".) Here is an example from > http://hypothesis.readthedocs.org/en/latest/quickstart.html > > from hypothesis import given > @given([int]) > def test_reversing_twice_gives_same_list(xs): > assert xs == list(reversed(reversed(xs))) > > When a test runner such as pytest calls the wrapper, it calls the real > function with a variety of int lists (including [], but details not > important here). @given() also works with unittest TestCase methods. > > When I mentioned PEP 484 and the alternative syntax List(int), he replied > "Having something like that as standard would be great for Hypothesis and I > intend to support it once it becomes available." > > I presume 'support' could and would include a version of given() that > accessed .__annotations__. Of course, if every 'test_xyz' were to be > decorated with the same decorator, then the decorator could be omitted and > the fuzzing moved into the test runner. This sounds cool. Note that you shouldn't access __annotations__ directly -- it may contain string literals. The typing module exports a functio get_type_hints() that takes a function object and returns a dict like __annotations__ but with the string literals evaluated. It also honors the @no_type_check decorator. For details see here: https://github.com/ambv/typehinting/blob/master/prototyping/typing.py#L1143 -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From mrocklin at gmail.com Tue Mar 31 05:20:47 2015 From: mrocklin at gmail.com (Matthew Rocklin) Date: Mon, 30 Mar 2015 20:20:47 -0700 Subject: [Python-ideas] History on proposals for Macros? In-Reply-To: References: Message-ID: Lisps like Scheme do indeed have an easier time with these due to the whole code-is-data thing, it's quite doable in languages with real syntax though. R and Julia would be good examples of syntactic languages with full macros. The Julia implementation might be a good model for what Python could do. Their docs are also a nice read if anyone isn't familiar with the topic. Macropy represents unevaluated expressions with the objects from the ast module. This seems like a sane choice. To be a little pedantic I'll give a brief example loosely showing what a macro is, then I'll talk about assert statements as a use case where macros might help with a pain point in normal Python programming. *Brief Educational Blurb* We write code as text defmacro f(x): ... f(a + b) * sin(c) We then parse parts of that text into syntax trees. [image: Inline image 1] Usually we translate these trees into byte-code and evaluate bottom-up, starting with a + b, then applying f, etc... Macros stop this process. They capture the subtrees beneath them before execution. Whenever we see a macro (f), we don't evaluate its subtree (a + b). Instead we transform the subtree into an in-memory representation (perhaps ast.BinOp(a, ast.Add(), b)) and hand that to f to do with as it will. Lets see an example with assertions. *Use case with Assertions* When testing we often want to write statements like the following assert x == y assert x in y etc... When these statements fail we want to emit statements that are well informed of the full expression, e.g. 5 != 6 5 was not found in {1, 2, 3} In Python we can't do this; assert only gets True or False and doesn't understand what generated that value . We've come up with a couple of workarounds. The first is the venerable unittest.TestCase methods that take the two sides of the comparison explicitly e.g. assertEquals(a, b), assertContains(a, b). This was sufficiently uncomfortable that projects like py.test arose and gained adoption. Py.test goes through the trouble of parsing the python test_.py files in order to generate nicer error messages. Having macros around would allow users to write this kind of functionality directly in Python rather than resorting to full text parsing and code transformation. Macros provide an escape out of pure bottom-up evaluation. On Sun, Mar 29, 2015 at 5:53 PM, Guido van Rossum wrote: > On Sat, Mar 28, 2015 at 9:53 AM, Matthew Rocklin > wrote: > >> Responding to comments off list: >> >> I'm not referring to C-style preprocessor macros, I'm referring to macros >> historically found in functional languages and commonly found in many >> user-targeted languages built in the last few years. >> > > Do you have examples and references? IIRC there's something named macros > in Scheme but Scheme, unlike Python, completely unifies code and data, and > there is a standard in-memory representation for code. > > >> The goal is to create things that look like functions but have access to >> the expression that was passed in. >> >> Some examples where this is useful: >> >> plot(year, miles / gallon) # Plot with labels determined by >> input-expressions, e.g. miles/gallon >> >> assertRaises(ZeroDivisionError, 1/0) # Evaluate the rhs 1/0 within >> assertRaises function, not before >> >> run_concurrently(f(x), f(y), f(z)) # Run f three times in three >> threads controlled by run_concurrently >> >> Generally one constructs something that looks like a function but, rather >> than receiving a pre-evaluated input, receives a syntax tree along with the >> associated context. This allows that function-like-thing to manipulate the >> expression and to control the context in which the evaluation occurs. >> > > None of the examples need the syntax tree though. The first wants the > string, the last probably just want a way to turn an argument into a lambda. > > >> There are lots of arguments against this, mostly focused around potential >> misuse. I'm looking for history of such arguments and for a general "Yes, >> this is theoretically possible" or "Not a chance in hell" from the >> community. Both are fine. >> > > I don't think this is a mainline need in Python, so it's probably both. :-) > > -- > --Guido van Rossum (python.org/~guido) > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: macro.png Type: image/png Size: 17961 bytes Desc: not available URL: From mrocklin at gmail.com Tue Mar 31 05:27:21 2015 From: mrocklin at gmail.com (Matthew Rocklin) Date: Mon, 30 Mar 2015 20:27:21 -0700 Subject: [Python-ideas] History on proposals for Macros? In-Reply-To: References: Message-ID: Also, just to credentialize myself, I am not a huge Lisp lover. I don't want macros to do crazy logic programming or whatever. I write numeric code in the scientific Python ecosystem. I want macros to build better interfaces for downstream users. This seems to be the modern use case in user-focused languages rather than lisp-magic-hell. On Mon, Mar 30, 2015 at 8:20 PM, Matthew Rocklin wrote: > Lisps like Scheme do indeed have an easier time with these due to the > whole code-is-data thing, it's quite doable in languages with real syntax > though. R and Julia would be good examples of syntactic languages with > full macros. The Julia implementation might be a good model for what > Python could do. Their docs > are also > a nice read if anyone isn't familiar with the topic. > > Macropy represents unevaluated expressions with the objects from the ast > module. This seems like a sane choice. > > To be a little pedantic I'll give a brief example loosely showing what a > macro is, then I'll talk about assert statements as a use case where macros > might help with a pain point in normal Python programming. > > *Brief Educational Blurb* > > We write code as text > > defmacro f(x): > ... > > f(a + b) * sin(c) > > We then parse parts of that text into syntax trees. > > [image: Inline image 1] > Usually we translate these trees into byte-code and evaluate bottom-up, > starting with a + b, then applying f, etc... Macros stop this process. > They capture the subtrees beneath them before execution. Whenever we see a > macro (f), we don't evaluate its subtree (a + b). Instead we transform > the subtree into an in-memory representation (perhaps ast.BinOp(a, > ast.Add(), b)) and hand that to f to do with as it will. Lets see an > example with assertions. > > *Use case with Assertions* > > When testing we often want to write statements like the following > > assert x == y > assert x in y > etc... > > When these statements fail we want to emit statements that are well > informed of the full expression, e.g. > > 5 != 6 > 5 was not found in {1, 2, 3} > > In Python we can't do this; assert only gets True or False and doesn't > understand what generated that value . We've come up with a couple of > workarounds. The first is the venerable unittest.TestCase methods that > take the two sides of the comparison explicitly e.g. assertEquals(a, b), > assertContains(a, b). This was sufficiently uncomfortable that projects > like py.test arose and gained adoption. Py.test goes through the trouble > of parsing the python test_.py files in order to generate nicer error > messages. > > Having macros around would allow users to write this kind of functionality > directly in Python rather than resorting to full text parsing and code > transformation. Macros provide an escape out of pure bottom-up evaluation. > > > On Sun, Mar 29, 2015 at 5:53 PM, Guido van Rossum > wrote: > >> On Sat, Mar 28, 2015 at 9:53 AM, Matthew Rocklin >> wrote: >> >>> Responding to comments off list: >>> >>> I'm not referring to C-style preprocessor macros, I'm referring to >>> macros historically found in functional languages and commonly found in >>> many user-targeted languages built in the last few years. >>> >> >> Do you have examples and references? IIRC there's something named macros >> in Scheme but Scheme, unlike Python, completely unifies code and data, and >> there is a standard in-memory representation for code. >> >> >>> The goal is to create things that look like functions but have access to >>> the expression that was passed in. >>> >>> Some examples where this is useful: >>> >>> plot(year, miles / gallon) # Plot with labels determined by >>> input-expressions, e.g. miles/gallon >>> >>> assertRaises(ZeroDivisionError, 1/0) # Evaluate the rhs 1/0 within >>> assertRaises function, not before >>> >>> run_concurrently(f(x), f(y), f(z)) # Run f three times in three >>> threads controlled by run_concurrently >>> >>> Generally one constructs something that looks like a function but, >>> rather than receiving a pre-evaluated input, receives a syntax tree along >>> with the associated context. This allows that function-like-thing to >>> manipulate the expression and to control the context in which the >>> evaluation occurs. >>> >> >> None of the examples need the syntax tree though. The first wants the >> string, the last probably just want a way to turn an argument into a lambda. >> >> >>> There are lots of arguments against this, mostly focused around >>> potential misuse. I'm looking for history of such arguments and for a >>> general "Yes, this is theoretically possible" or "Not a chance in hell" >>> from the community. Both are fine. >>> >> >> I don't think this is a mainline need in Python, so it's probably both. >> :-) >> >> -- >> --Guido van Rossum (python.org/~guido) >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: macro.png Type: image/png Size: 17961 bytes Desc: not available URL: From mrocklin at gmail.com Tue Mar 31 06:26:05 2015 From: mrocklin at gmail.com (Matthew Rocklin) Date: Mon, 30 Mar 2015 21:26:05 -0700 Subject: [Python-ideas] History on proposals for Macros? In-Reply-To: References: Message-ID: > > Is something stopping you from exploring this? Do you have specific ideas > on how to improve on macropy? > Macropy is great but it requires an import-hook. Many scientific users work interactively. > It sounds almost as if you would like to implement this but you want some > kind of promise ahead of time that your work will be incorporated into the > language. But that's just not how it works. When you want to explore a big > idea like this, at some point you have to be willing to take the risk of > writing code without a guaranteed pay off. Haoyi didn't ask for macropy to > be incorporated into Python -- in fact he was surprised at the amount of > uptake it got. > The hard problem isn't building macros, it's deciding whether or not macros are good for Python. I'm trying to start a discussion. If this isn't the right place for that then I apologize. > You've received quite a bit of feedback (and, may I say, push back :-) > from a small number of python-ideas veterans -- you can take this or leave > it, but at this point I think you've gotten about as much mileage out of > the list as can be expected. > My apologies. I didn't realize that I was misusing this list. I also didn't realize that I was receiving push-back, the comments here seemed friendly and encouraging. Last year at SciPy the message I heard was "If you want to convince the core team then come to python-ideas armed with motivating use cases." Here I am :) Anyway, if there isn't any interest then I'll leave off. Thank you all for your time, -Matt -------------- next part -------------- An HTML attachment was scrubbed... URL: From mrocklin at gmail.com Tue Mar 31 06:30:19 2015 From: mrocklin at gmail.com (Matthew Rocklin) Date: Mon, 30 Mar 2015 21:30:19 -0700 Subject: [Python-ideas] History on proposals for Macros? In-Reply-To: References: Message-ID: Whoops, my apologies. Apparently I don't get e-mails sent only to python-ideas and not also to me. There was a lot of conversation to which I was ignorant. On Mon, Mar 30, 2015 at 9:26 PM, Matthew Rocklin wrote: > Is something stopping you from exploring this? Do you have specific ideas >> on how to improve on macropy? >> > > Macropy is great but it requires an import-hook. Many scientific users > work interactively. > > >> It sounds almost as if you would like to implement this but you want some >> kind of promise ahead of time that your work will be incorporated into the >> language. But that's just not how it works. When you want to explore a big >> idea like this, at some point you have to be willing to take the risk of >> writing code without a guaranteed pay off. Haoyi didn't ask for macropy to >> be incorporated into Python -- in fact he was surprised at the amount of >> uptake it got. >> > > The hard problem isn't building macros, it's deciding whether or not > macros are good for Python. I'm trying to start a discussion. If this > isn't the right place for that then I apologize. > > >> You've received quite a bit of feedback (and, may I say, push back :-) >> from a small number of python-ideas veterans -- you can take this or leave >> it, but at this point I think you've gotten about as much mileage out of >> the list as can be expected. >> > > My apologies. I didn't realize that I was misusing this list. I also > didn't realize that I was receiving push-back, the comments here seemed > friendly and encouraging. > > Last year at SciPy the message I heard was "If you want to convince the > core team then come to python-ideas armed with motivating use cases." Here > I am :) > > Anyway, if there isn't any interest then I'll leave off. Thank you all > for your time, > -Matt > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mrocklin at gmail.com Tue Mar 31 07:09:58 2015 From: mrocklin at gmail.com (Matthew Rocklin) Date: Mon, 30 Mar 2015 22:09:58 -0700 Subject: [Python-ideas] History on proposals for Macros? In-Reply-To: References: Message-ID: The !(x + y) solution would release some pressure. People have considered using lambda: to delay execution in Pandas queries. The result is a bit odd: https://github.com/pydata/pandas/issues/9229#issuecomment-69691738 On Mon, Mar 30, 2015 at 9:26 PM, Matthew Rocklin wrote: > Is something stopping you from exploring this? Do you have specific ideas >> on how to improve on macropy? >> > > Macropy is great but it requires an import-hook. Many scientific users > work interactively. > > >> It sounds almost as if you would like to implement this but you want some >> kind of promise ahead of time that your work will be incorporated into the >> language. But that's just not how it works. When you want to explore a big >> idea like this, at some point you have to be willing to take the risk of >> writing code without a guaranteed pay off. Haoyi didn't ask for macropy to >> be incorporated into Python -- in fact he was surprised at the amount of >> uptake it got. >> > > The hard problem isn't building macros, it's deciding whether or not > macros are good for Python. I'm trying to start a discussion. If this > isn't the right place for that then I apologize. > > >> You've received quite a bit of feedback (and, may I say, push back :-) >> from a small number of python-ideas veterans -- you can take this or leave >> it, but at this point I think you've gotten about as much mileage out of >> the list as can be expected. >> > > My apologies. I didn't realize that I was misusing this list. I also > didn't realize that I was receiving push-back, the comments here seemed > friendly and encouraging. > > Last year at SciPy the message I heard was "If you want to convince the > core team then come to python-ideas armed with motivating use cases." Here > I am :) > > Anyway, if there isn't any interest then I'll leave off. Thank you all > for your time, > -Matt > -------------- next part -------------- An HTML attachment was scrubbed... URL: From shoyer at gmail.com Tue Mar 31 08:21:50 2015 From: shoyer at gmail.com (Stephan Hoyer) Date: Mon, 30 Mar 2015 23:21:50 -0700 Subject: [Python-ideas] History on proposals for Macros? In-Reply-To: References: Message-ID: Macros would be an extremely useful feature for pandas, the main data analysis library for Python (for which I'm a core developer). Why? Well, right now, R has better syntax than Python for writing data analysis code. The difference comes down to two macros that R developers have written within the past few years. Here's an example borrowed from the documentation for the dplyr R package [1]: flights %>% group_by(year, month, day) %>% select(arr_delay, dep_delay) %>% summarise( arr = mean(arr_delay), dep = mean(dep_delay) ) %>% filter(arr > 30 | dep > 30) Here "flights" is a dataframe, similar to a table in spreadsheet. It is also the only global variables in the analysis -- variables like "year" and "arr_delay" are actually columns in the dataframe. R evaluates variables lazily, in the context of the provided frame. In Python, functions like groupby_by would need to be macros. The other macro is the "pipe" or chaining operator %>%. This operator is used to avoid the need many temporary or highly nested expressions. The result is quite readable, but again, it needs to be a macro, because group_by and filter are simply functions that take a dataframe as their first argument. The fact that chaining works with plain functions means that it works even on libraries that weren't designed for it. We could do function chaining in Python by abusing an exist binary operator like >> or |, but all the objects on which it works would need to be custom types. What does this example look using pandas? Well, it's not as nice, and there's not much we can do about it because of the limitations of Python syntax: (flights .group_by('year', 'month', 'day') .select('arr_delay', 'dep_delay') .summarize( arr = lambda df: mean(df.arr_delay)), dep = lambda df: mean(df.dep_delay))) .filter(lambda df: (df.arr > 30) | (df.dep > 30))) (Astute readers will note that I've taken a few liberties with pandas syntax to make more similar to dplyr.) Instead of evaluating expressions in the delayed context of a dataframes, we use strings or functions. With all the lambdas there's a lot more noise than the R example, and it's harder to keep track of what's on. In principle we could simplify the lambda expressions to not use any arguments (Matthew linked to the GitHub comment where I showed what that would look like [2]), but the code remains awkwardly verbose. For chaining, instead of using functions and the pipe operator, we use methods. This works fine as long as users are only using pandas, but it means that unlike R, the Python dataframe is a closed ecosystem. Python developers (rightly) frown upon monkey-patching, so there's no way for external libraries to add their own functions (e.g., for custom plotting or file formats) on an equal footing to the methods built-in to pandas. I hope these use cases are illustrative. I don't have strong opinions on the technical merits of particular proposals. The "light lambda" syntax described by Andrew Barnert would at least solve the delayed evaluation use-case nicely, though the colon character is not ideal because it would rule out using light lambdas inside indexing brackets. Best, Stephan [1] http://cran.rstudio.com/web/packages/dplyr/vignettes/introduction.html#chaining [2] https://github.com/pydata/pandas/issues/9229#issuecomment-69691738 -------------- next part -------------- An HTML attachment was scrubbed... URL: From songofacandy at gmail.com Tue Mar 31 08:35:49 2015 From: songofacandy at gmail.com (INADA Naoki) Date: Tue, 31 Mar 2015 15:35:49 +0900 Subject: [Python-ideas] Adding rule about trailing comma. Message-ID: What about adding rule about trailing comma? Yes: foo(aaa, bbb) or foo(aaa, bbb, ) No: foo(aaa, bbb,) # except one-value tuple. or foo(aaa, bbb ) -- INADA Naoki From david at drmaciver.com Tue Mar 31 09:10:59 2015 From: david at drmaciver.com (David MacIver) Date: Tue, 31 Mar 2015 09:10:59 +0200 Subject: [Python-ideas] PEP 484: Generating test inputs from type hints Message-ID: (Sorry for the manual thread continuation, I wasn't previously on the mailing list) I presume 'support' could and would include a version of given() that > accessed .__annotations__. This is on my list of planned features independently of PEP 484 FWIW. It's on the long list of nice to have but minor things that didn't make it into 1.0. One of the problems is that I am for the moment somewhat committed to supporting Python 2.7 as well as Python 3 (I really wish I wasn't), or I would have probably built Hypothesis on using annotations from the beginning. To a degree though it's fortunate that I didn't given PEP 484. It doesn't look like I would be able to build the full range of features Hypothesis needs for @given using only type annotations, and 484 suggests the goal is to move to those being the only valid ones. The reason for this is that Hypothesis deliberately lets you mix strategies in with the specifiers used for annotation. So for example you can do something like: Palindromes = strategy(str).map(lambda x: x + str(reversed(x))) This is now a SearchStrategy object that still returns strings, but only ones that are palindromes. You can now do something like @given([Palindromes]) def test_this_is_a_list_of_palindromes(xs): # I'm not actually sure what interesting things you can say about a list of palindromes... So the point is that you can use a SearchStrategy deep inside the "type" signatures that Hypothesis uses, despite SearchStrategy not actually being at all type-like. (It actually used to be the case that an earlier version of SearchStrategy allowed you to test whether a value could have come from that strategy, which would have made them a bit more "type-like", but this proved to be a bad idea and the work that let me remove it was one of the best decisions I made in the design of Hypothesis) So basically I would fully intend for Hypothesis to become a consumer of types and annotations, but I don't think they can be used as a primary building block of it. (Note: I don't consider this a problem with PEP 484, which looks like a good proposal. Hypothesis's requirements are just weird and specialized and don't quite match type checking) > Of course, if every 'test_xyz' were to be > decorated with the same decorator, then the decorator could be omitted > and the fuzzing moved into the test runner. I'd be moderately against this being the primary mode of using Hypothesis, even without the above caveats. One of the things I like about Hypothesis right now is that because it just gives you Python functions it can be very unopinionated about what you're using to run your tests (even if in practice basically everyone is using unittest or pytest). However it should be relatively easy to support alongside once I've done some much needed refactoring of the core example exploration code. David -------------- next part -------------- An HTML attachment was scrubbed... URL: From abarnert at yahoo.com Tue Mar 31 09:19:33 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Tue, 31 Mar 2015 00:19:33 -0700 Subject: [Python-ideas] History on proposals for Macros? In-Reply-To: References: Message-ID: <13487A22-3B2D-45F5-B950-FBC9FBB53FD4@yahoo.com> On Mar 30, 2015, at 23:21, Stephan Hoyer wrote: > > Macros would be an extremely useful feature for pandas, the main data analysis library for Python (for which I'm a core developer). > > Why? Well, right now, R has better syntax than Python for writing data analysis code. The difference comes down to two macros that R developers have written within the past few years. > > Here's an example borrowed from the documentation for the dplyr R package [1]: > > flights %>% > group_by(year, month, day) %>% > select(arr_delay, dep_delay) %>% > summarise( > arr = mean(arr_delay), > dep = mean(dep_delay) > ) %>% > filter(arr > 30 | dep > 30) > > Here "flights" is a dataframe, similar to a table in spreadsheet. It is also the only global variables in the analysis -- variables like "year" and "arr_delay" are actually columns in the dataframe. R evaluates variables lazily, in the context of the provided frame. In Python, functions like groupby_by would need to be macros. > > The other macro is the "pipe" or chaining operator %>%. This operator is used to avoid the need many temporary or highly nested expressions. The result is quite readable, but again, it needs to be a macro, because group_by and filter are simply functions that take a dataframe as their first argument. The fact that chaining works with plain functions means that it works even on libraries that weren't designed for it. We could do function chaining in Python by abusing an exist binary operator like >> or |, but all the objects on which it works would need to be custom types. > > What does this example look using pandas? Well, it's not as nice, and there's not much we can do about it because of the limitations of Python syntax: > > (flights > .group_by('year', 'month', 'day') > .select('arr_delay', 'dep_delay') > .summarize( > arr = lambda df: mean(df.arr_delay)), > dep = lambda df: mean(df.dep_delay))) > .filter(lambda df: (df.arr > 30) | (df.dep > 30))) > > (Astute readers will note that I've taken a few liberties with pandas syntax to make more similar to dplyr.) > > Instead of evaluating expressions in the delayed context of a dataframes, we use strings or functions. With all the lambdas there's a lot more noise than the R example, and it's harder to keep track of what's on. In principle we could simplify the lambda expressions to not use any arguments (Matthew linked to the GitHub comment where I showed what that would look like [2]), but the code remains awkwardly verbose. > > For chaining, instead of using functions and the pipe operator, we use methods. This works fine as long as users are only using pandas, but it means that unlike R, the Python dataframe is a closed ecosystem. Python developers (rightly) frown upon monkey-patching, so there's no way for external libraries to add their own functions (e.g., for custom plotting or file formats) on an equal footing to the methods built-in to pandas. One way around this is to provide a documented, clean method for hooking your types--e.g., a register classmethod that then makes the function appear as a method in all instances. Functionally this is the same as monkeypatching, but it looks a lot more inviting to the user. (And it also allows you to rewrite things under the covers in ways that would break direct monkeypatching, if you ever want to.) There are more examples of opening up modules this way than classes, but it's the same idea. > I hope these use cases are illustrative. I don't have strong opinions on the technical merits of particular proposals. The "light lambda" syntax described by Andrew Barnert would at least solve the delayed evaluation use-case nicely, though the colon character is not ideal because it would rule out using light lambdas inside indexing brackets. The bare colon was just one of multiple suggestions that came up last time the idea was discussed (last February or so), and in previous discussions (going back to the long-abandoned PEP 312). In many cases, it looks very nice, but in others it's either ugly or (at least to a human) ambiguous without parens (which obviously defeat the whole point). I don't think anyone noticed the indexing issue, but someone (Nick Coghlan, I think) pointed out essentially the same issue in dict displays (e.g., for making a dynamic jump table). If you seriously want to revive that discussion--which might be worth doing, if your use cases are sufficiently different from the ones that were discussed (Tkinter and RPC server callbacks were the primary motivating case)--somewhere I have notes on all the syntaxes people suggested that I could dump on you. For the strings, another possibility is a namespace object that can effectively defer attribute lookup and have it done by the real table when you get it, as done by various ORMs, appscript, etc. Instead of this: > (flights > .group_by('year', 'month', 'day') ... You write: > (flights > .group_by(c.year, c.month, c.day) That "c" is just an instance of a simple type that wraps its __getattr__ argument up so you can access it later--when get an argument to group_by that's one of those wrappers, you look it up on the table. It's the same number of characters, and looks a little more magical, but arguably it's more readable, at least once people are used to your framework. For example, here's a query expression to find broken tracks (with missing filenames) in iTunes via appscript that was used in a real application (until iTunes Match made this no longer work...): playlist.tracks[its.location == k.missing] Both its and k are special objects you import from appscript; its.__getattr__ returns a wrapper that's used to look up "location" in the members of whatever class playlist.tracks instances turn out to be at runtime, and k.missing returns a wrapper that's used to look up "missing" in the global keywords of whatever app playlist turns out to be part of at runtime. You can take this even further: if mean weren't a function that takes a column and returns its mean, but instead a function that takes a column name and returns a function that takes a table and returns the mean of the column with that name, then you could just write this: > .summarize( > arr = mean(c.arr_delay) But of course that isn't what mean is. But it can be what, say, f.mean is, if f is another one of those attribute-delaying objects: f.mean(c.arr_delay) returns a wrapper that summarize can use to call the function named "mean" on the column named "arr_delay". So, the whole thing reduces to: > (flights > .group_by(c.year, c.month, c.day) > .select(c.arr_delay, c.dep_delay) > .summarize( > arr = f.mean(c.arr_delay)), > dep = f.mean(c.dep_delay))) > .filter(c.arr > 30 | c.dep > 30) -------------- next part -------------- An HTML attachment was scrubbed... URL: From liik.joonas at gmail.com Tue Mar 31 09:22:11 2015 From: liik.joonas at gmail.com (Joonas Liik) Date: Tue, 31 Mar 2015 10:22:11 +0300 Subject: [Python-ideas] History on proposals for Macros? In-Reply-To: References: Message-ID: isn't this: lambda df: mean(df.arr_delay) the same as.. functools.partial(mean, df.arr_delay) I kind of like the idea of a pipe operator (tho %>% looks just terrible and | is already taken) But consider: if we could get functools.compose that too would be alleviated. (something like https://mathieularose.com/function-composition-in-python/ ) On 31 March 2015 at 09:21, Stephan Hoyer wrote: > Macros would be an extremely useful feature for pandas, the main data > analysis library for Python (for which I'm a core developer). > > Why? Well, right now, R has better syntax than Python for writing data > analysis code. The difference comes down to two macros that R developers > have written within the past few years. > > Here's an example borrowed from the documentation for the dplyr R package > [1]: > > flights %>% > group_by(year, month, day) %>% > select(arr_delay, dep_delay) %>% > summarise( > arr = mean(arr_delay), > dep = mean(dep_delay) > ) %>% > filter(arr > 30 | dep > 30) > > Here "flights" is a dataframe, similar to a table in spreadsheet. It is > also the only global variables in the analysis -- variables like "year" and > "arr_delay" are actually columns in the dataframe. R evaluates variables > lazily, in the context of the provided frame. In Python, functions like > groupby_by would need to be macros. > > The other macro is the "pipe" or chaining operator %>%. This operator is > used to avoid the need many temporary or highly nested expressions. The > result is quite readable, but again, it needs to be a macro, because > group_by and filter are simply functions that take a dataframe as their > first argument. The fact that chaining works with plain functions means > that it works even on libraries that weren't designed for it. We could do > function chaining in Python by abusing an exist binary operator like >> or > |, but all the objects on which it works would need to be custom types. > > What does this example look using pandas? Well, it's not as nice, and > there's not much we can do about it because of the limitations of Python > syntax: > > (flights > .group_by('year', 'month', 'day') > .select('arr_delay', 'dep_delay') > .summarize( > arr = lambda df: mean(df.arr_delay)), > dep = lambda df: mean(df.dep_delay))) > .filter(lambda df: (df.arr > 30) | (df.dep > 30))) > > (Astute readers will note that I've taken a few liberties with pandas > syntax to make more similar to dplyr.) > > Instead of evaluating expressions in the delayed context of a dataframes, > we use strings or functions. With all the lambdas there's a lot more noise > than the R example, and it's harder to keep track of what's on. In > principle we could simplify the lambda expressions to not use any arguments > (Matthew linked to the GitHub comment where I showed what that would look > like [2]), but the code remains awkwardly verbose. > > For chaining, instead of using functions and the pipe operator, we use > methods. This works fine as long as users are only using pandas, but it > means that unlike R, the Python dataframe is a closed ecosystem. Python > developers (rightly) frown upon monkey-patching, so there's no way for > external libraries to add their own functions (e.g., for custom plotting or > file formats) on an equal footing to the methods built-in to pandas. > > I hope these use cases are illustrative. I don't have strong opinions on > the technical merits of particular proposals. The "light lambda" syntax > described by Andrew Barnert would at least solve the delayed evaluation > use-case nicely, though the colon character is not ideal because it would > rule out using light lambdas inside indexing brackets. > > Best, > Stephan > > [1] > http://cran.rstudio.com/web/packages/dplyr/vignettes/introduction.html#chaining > [2] https://github.com/pydata/pandas/issues/9229#issuecomment-69691738 > > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From abarnert at yahoo.com Tue Mar 31 09:30:50 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Tue, 31 Mar 2015 00:30:50 -0700 Subject: [Python-ideas] Adding rule about trailing comma. In-Reply-To: References: Message-ID: On Mar 30, 2015, at 23:35, INADA Naoki wrote: > > What about adding rule about trailing comma? I don't understand the proposed rule from your example. Are you suggesting that a trailing comma be a syntax error unless it's following by a newline? That would explain both "yesses" and the first "no", but then what's wrong with the second "no"? You can always add newlines in the middle of any parenthesized expression; are you suggesting that you shouldn't be allowed to add one at the very end, unless it's immediately following a comma? If so, why? Also, you seem to realize you're adding a gratuitous inconsistency with one-element tuples. Haven't they been through enough teasing without having to make them even more different? Most of all, whatever your actual rule is: Why? What's the intended benefit here? > > Yes: > > foo(aaa, > bbb) > > or > > foo(aaa, > bbb, > ) > > No: > > foo(aaa, > bbb,) # except one-value tuple. > > or > > foo(aaa, > bbb > ) > > > > -- > INADA Naoki > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ From rosuav at gmail.com Tue Mar 31 09:36:58 2015 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 31 Mar 2015 18:36:58 +1100 Subject: [Python-ideas] Adding rule about trailing comma. In-Reply-To: References: Message-ID: On Tue, Mar 31, 2015 at 5:35 PM, INADA Naoki wrote: > What about adding rule about trailing comma? My crystal ball tells me that you're possibly talking about PEP 8 rather than about actual language changes. Is that correct? There's absolutely no need to put restrictions like this into the language itself. If the trailing comma is permitted, it should be permitted whether there's a newline after it or not. In your own style guide, you're most welcome to advocate what you will, without needing any approval from the Python core devs. If you want to say, for instance, "we follow PEP 8, with the following additional requirements", and then list your trailing comma rule and any others you want to require, people will understand. Actually, most people will do something like this, anyway. ChrisA From abarnert at yahoo.com Tue Mar 31 09:57:26 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Tue, 31 Mar 2015 00:57:26 -0700 Subject: [Python-ideas] History on proposals for Macros? In-Reply-To: References: Message-ID: On Mar 31, 2015, at 00:22, Joonas Liik wrote: > > isn't this: > lambda df: mean(df.arr_delay) > the same as.. > functools.partial(mean, df.arr_delay) No, because in the first one df is a parameter (which gets the value of whatever DataFrame this is run on) while in the second it's a free variable (which just raises NameError, if you're lucky). You could do this with operator.attrgetter and composition, of course: compose(mean, attrgetter('arr_delay')) But I don't think that's more readable. Even if composition were an infix operator: (mean . attrgetter('arr_delay')) > I kind of like the idea of a pipe operator (tho %>% looks just terrible and | is already taken) > But consider: if we could get functools.compose that too would be alleviated. > (something like https://mathieularose.com/function-composition-in-python/ ) Why do you need to "get functools.compose"? If you just want the trivial version from that blog post, it's two lines that any novice can write himself. If you want one of the more complex versions that he dismisses, then there might be a better argument, but the post you linked argues against that, not for it. And if you don't trust yourself to write the two lines yourself, you can always pip install funcy or toolz or functional3 or more-functools. As far as using compose again for piping... Well, it's backward from what you want, and it also requires you to stack up enough parens to choke a Lisp guru, not to mention all the repetitions of compose itself (that's why Haskell has infix compose and apply operators with the precedence and associativity they have, so you can avoid all the parens). >> On 31 March 2015 at 09:21, Stephan Hoyer wrote: >> Macros would be an extremely useful feature for pandas, the main data analysis library for Python (for which I'm a core developer). >> >> Why? Well, right now, R has better syntax than Python for writing data analysis code. The difference comes down to two macros that R developers have written within the past few years. >> >> Here's an example borrowed from the documentation for the dplyr R package [1]: >> >> flights %>% >> group_by(year, month, day) %>% >> select(arr_delay, dep_delay) %>% >> summarise( >> arr = mean(arr_delay), >> dep = mean(dep_delay) >> ) %>% >> filter(arr > 30 | dep > 30) >> >> Here "flights" is a dataframe, similar to a table in spreadsheet. It is also the only global variables in the analysis -- variables like "year" and "arr_delay" are actually columns in the dataframe. R evaluates variables lazily, in the context of the provided frame. In Python, functions like groupby_by would need to be macros. >> >> The other macro is the "pipe" or chaining operator %>%. This operator is used to avoid the need many temporary or highly nested expressions. The result is quite readable, but again, it needs to be a macro, because group_by and filter are simply functions that take a dataframe as their first argument. The fact that chaining works with plain functions means that it works even on libraries that weren't designed for it. We could do function chaining in Python by abusing an exist binary operator like >> or |, but all the objects on which it works would need to be custom types. >> >> What does this example look using pandas? Well, it's not as nice, and there's not much we can do about it because of the limitations of Python syntax: >> >> (flights >> .group_by('year', 'month', 'day') >> .select('arr_delay', 'dep_delay') >> .summarize( >> arr = lambda df: mean(df.arr_delay)), >> dep = lambda df: mean(df.dep_delay))) >> .filter(lambda df: (df.arr > 30) | (df.dep > 30))) >> >> (Astute readers will note that I've taken a few liberties with pandas syntax to make more similar to dplyr.) >> >> Instead of evaluating expressions in the delayed context of a dataframes, we use strings or functions. With all the lambdas there's a lot more noise than the R example, and it's harder to keep track of what's on. In principle we could simplify the lambda expressions to not use any arguments (Matthew linked to the GitHub comment where I showed what that would look like [2]), but the code remains awkwardly verbose. >> >> For chaining, instead of using functions and the pipe operator, we use methods. This works fine as long as users are only using pandas, but it means that unlike R, the Python dataframe is a closed ecosystem. Python developers (rightly) frown upon monkey-patching, so there's no way for external libraries to add their own functions (e.g., for custom plotting or file formats) on an equal footing to the methods built-in to pandas. >> >> I hope these use cases are illustrative. I don't have strong opinions on the technical merits of particular proposals. The "light lambda" syntax described by Andrew Barnert would at least solve the delayed evaluation use-case nicely, though the colon character is not ideal because it would rule out using light lambdas inside indexing brackets. >> >> Best, >> Stephan >> >> [1] http://cran.rstudio.com/web/packages/dplyr/vignettes/introduction.html#chaining >> [2] https://github.com/pydata/pandas/issues/9229#issuecomment-69691738 >> >> >> >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From songofacandy at gmail.com Tue Mar 31 10:28:31 2015 From: songofacandy at gmail.com (INADA Naoki) Date: Tue, 31 Mar 2015 17:28:31 +0900 Subject: [Python-ideas] Adding rule about trailing comma. In-Reply-To: References: Message-ID: On Tue, Mar 31, 2015 at 4:30 PM, Andrew Barnert wrote: > On Mar 30, 2015, at 23:35, INADA Naoki wrote: >> >> What about adding rule about trailing comma? > > I don't understand the proposed rule from your example. Are you suggesting that a trailing comma be a syntax error unless it's following by a newline? I'm sorry, I meant about adding rule to PEP 8. Not Python syntax. > That would explain both "yesses" and the first "no", but then what's wrong with the second "no"? You can always add newlines in the middle of any parenthesized expression; are you suggesting that you shouldn't be allowed to add one at the very end, unless it's immediately following a comma? If so, why? > > Also, you seem to realize you're adding a gratuitous inconsistency with one-element tuples. Haven't they been through enough teasing without having to make them even more different? > > Most of all, whatever your actual rule is: Why? What's the intended benefit here? I recently write Go program more than Python. I feel good about line continuation rule in Go: Some characters including comma means line continuation. Benefit of this style is: 1) More consistent 2) No garbage diff just for just adding comma > >> >> Yes: >> >> foo(aaa, >> bbb) >> >> or >> >> foo(aaa, >> bbb, >> ) >> >> No: >> >> foo(aaa, >> bbb,) # except one-value tuple. >> >> or >> >> foo(aaa, >> bbb >> ) >> >> >> >> -- >> INADA Naoki >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ -- INADA Naoki From rosuav at gmail.com Tue Mar 31 10:34:19 2015 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 31 Mar 2015 19:34:19 +1100 Subject: [Python-ideas] Adding rule about trailing comma. In-Reply-To: References: Message-ID: On Tue, Mar 31, 2015 at 7:28 PM, INADA Naoki wrote: > Benefit of this style is: > > 1) More consistent > 2) No garbage diff just for just adding comma So what you have there is a good reason for adopting this style. Is there strong reason for mentioning it in PEP 8? Remember, every recommendation in a style guide has a cost - too many rules and it just becomes onerous, not to mention the increased likelihood of the rule giving the wrong advice in some corner case (which leads intelligent people to disregard the rule, and unintelligent checkers to spew warnings which then have to be ignored - this is one of the fundamental difficulties of scripting PEP 8 or any other style guide); ideally, trust people to be intelligent, and don't enforce more than you have to. ChrisA From guido at python.org Tue Mar 31 05:47:17 2015 From: guido at python.org (Guido van Rossum) Date: Mon, 30 Mar 2015 20:47:17 -0700 Subject: [Python-ideas] History on proposals for Macros? In-Reply-To: References: Message-ID: Matthew, Is something stopping you from exploring this? Do you have specific ideas on how to improve on macropy? It sounds almost as if you would like to implement this but you want some kind of promise ahead of time that your work will be incorporated into the language. But that's just not how it works. When you want to explore a big idea like this, at some point you have to be willing to take the risk of writing code without a guaranteed pay off. Haoyi didn't ask for macropy to be incorporated into Python -- in fact he was surprised at the amount of uptake it got. You've received quite a bit of feedback (and, may I say, push back :-) from a small number of python-ideas veterans -- you can take this or leave it, but at this point I think you've gotten about as much mileage out of the list as can be expected. Good luck! --Guido On Mon, Mar 30, 2015 at 8:27 PM, Matthew Rocklin wrote: > Also, just to credentialize myself, I am not a huge Lisp lover. I don't > want macros to do crazy logic programming or whatever. I write numeric > code in the scientific Python ecosystem. I want macros to build better > interfaces for downstream users. This seems to be the modern use case in > user-focused languages rather than lisp-magic-hell. > > On Mon, Mar 30, 2015 at 8:20 PM, Matthew Rocklin > wrote: > >> Lisps like Scheme do indeed have an easier time with these due to the >> whole code-is-data thing, it's quite doable in languages with real syntax >> though. R and Julia would be good examples of syntactic languages with >> full macros. The Julia implementation might be a good model for what >> Python could do. Their docs >> are >> also a nice read if anyone isn't familiar with the topic. >> >> Macropy represents unevaluated expressions with the objects from the ast >> module. This seems like a sane choice. >> >> To be a little pedantic I'll give a brief example loosely showing what a >> macro is, then I'll talk about assert statements as a use case where macros >> might help with a pain point in normal Python programming. >> >> *Brief Educational Blurb* >> >> We write code as text >> >> defmacro f(x): >> ... >> >> f(a + b) * sin(c) >> >> We then parse parts of that text into syntax trees. >> >> [image: Inline image 1] >> Usually we translate these trees into byte-code and evaluate bottom-up, >> starting with a + b, then applying f, etc... Macros stop this process. >> They capture the subtrees beneath them before execution. Whenever we see a >> macro (f), we don't evaluate its subtree (a + b). Instead we transform >> the subtree into an in-memory representation (perhaps ast.BinOp(a, >> ast.Add(), b)) and hand that to f to do with as it will. Lets see an >> example with assertions. >> >> *Use case with Assertions* >> >> When testing we often want to write statements like the following >> >> assert x == y >> assert x in y >> etc... >> >> When these statements fail we want to emit statements that are well >> informed of the full expression, e.g. >> >> 5 != 6 >> 5 was not found in {1, 2, 3} >> >> In Python we can't do this; assert only gets True or False and doesn't >> understand what generated that value . We've come up with a couple of >> workarounds. The first is the venerable unittest.TestCase methods that >> take the two sides of the comparison explicitly e.g. assertEquals(a, b), >> assertContains(a, b). This was sufficiently uncomfortable that projects >> like py.test arose and gained adoption. Py.test goes through the trouble >> of parsing the python test_.py files in order to generate nicer error >> messages. >> >> Having macros around would allow users to write this kind of >> functionality directly in Python rather than resorting to full text parsing >> and code transformation. Macros provide an escape out of pure bottom-up >> evaluation. >> >> On Sun, Mar 29, 2015 at 5:53 PM, Guido van Rossum >> wrote: >> >>> On Sat, Mar 28, 2015 at 9:53 AM, Matthew Rocklin >>> wrote: >>> >>>> Responding to comments off list: >>>> >>>> I'm not referring to C-style preprocessor macros, I'm referring to >>>> macros historically found in functional languages and commonly found in >>>> many user-targeted languages built in the last few years. >>>> >>> >>> Do you have examples and references? IIRC there's something named macros >>> in Scheme but Scheme, unlike Python, completely unifies code and data, and >>> there is a standard in-memory representation for code. >>> >>> >>>> The goal is to create things that look like functions but have access >>>> to the expression that was passed in. >>>> >>>> Some examples where this is useful: >>>> >>>> plot(year, miles / gallon) # Plot with labels determined by >>>> input-expressions, e.g. miles/gallon >>>> >>>> assertRaises(ZeroDivisionError, 1/0) # Evaluate the rhs 1/0 within >>>> assertRaises function, not before >>>> >>>> run_concurrently(f(x), f(y), f(z)) # Run f three times in three >>>> threads controlled by run_concurrently >>>> >>>> Generally one constructs something that looks like a function but, >>>> rather than receiving a pre-evaluated input, receives a syntax tree along >>>> with the associated context. This allows that function-like-thing to >>>> manipulate the expression and to control the context in which the >>>> evaluation occurs. >>>> >>> >>> None of the examples need the syntax tree though. The first wants the >>> string, the last probably just want a way to turn an argument into a lambda. >>> >>> >>>> There are lots of arguments against this, mostly focused around >>>> potential misuse. I'm looking for history of such arguments and for a >>>> general "Yes, this is theoretically possible" or "Not a chance in hell" >>>> from the community. Both are fine. >>>> >>> >>> I don't think this is a mainline need in Python, so it's probably both. >>> :-) >>> >>> -- >>> --Guido van Rossum (python.org/~guido) >>> >> >> > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: macro.png Type: image/png Size: 17961 bytes Desc: not available URL: From guido at python.org Tue Mar 31 16:52:38 2015 From: guido at python.org (Guido van Rossum) Date: Tue, 31 Mar 2015 07:52:38 -0700 Subject: [Python-ideas] History on proposals for Macros? In-Reply-To: References: Message-ID: On Mon, Mar 30, 2015 at 9:26 PM, Matthew Rocklin wrote: > Is something stopping you from exploring this? Do you have specific ideas >> on how to improve on macropy? >> > > Macropy is great but it requires an import-hook. Many scientific users > work interactively. > > >> It sounds almost as if you would like to implement this but you want some >> kind of promise ahead of time that your work will be incorporated into the >> language. But that's just not how it works. When you want to explore a big >> idea like this, at some point you have to be willing to take the risk of >> writing code without a guaranteed pay off. Haoyi didn't ask for macropy to >> be incorporated into Python -- in fact he was surprised at the amount of >> uptake it got. >> > > The hard problem isn't building macros, it's deciding whether or not > macros are good for Python. I'm trying to start a discussion. If this > isn't the right place for that then I apologize. > This is the right place, and we're now at the point where it's your job to either show a concrete design spec that can actually be implemented, and have its tires kicked, or just go off and build something. In the latter case you'll probably learn about some practical issues that nobody might have thought of yet. > You've received quite a bit of feedback (and, may I say, push back :-) >> from a small number of python-ideas veterans -- you can take this or leave >> it, but at this point I think you've gotten about as much mileage out of >> the list as can be expected. >> > > My apologies. I didn't realize that I was misusing this list. I also > didn't realize that I was receiving push-back, the comments here seemed > friendly and encouraging. > You weren't misusing the list. Maybe (based on your next message) you weren't reading it though. :-) > Last year at SciPy the message I heard was "If you want to convince the > core team then come to python-ideas armed with motivating use cases." Here > I am :) > > Anyway, if there isn't any interest then I'll leave off. Thank you all > for your time, > I think you misunderstand. There's interest but there are also real concerns. I really do think that an implementation (if, as you say, that isn't the hard part) would be very helpful to judge whether it is a desirable feature. (Maybe you discover you can do it in a way that can be distributed via PYPI -- "pip install macros".) -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From guettliml at thomas-guettler.de Tue Mar 31 21:17:39 2015 From: guettliml at thomas-guettler.de (=?UTF-8?B?VGhvbWFzIEfDvHR0bGVy?=) Date: Tue, 31 Mar 2015 21:17:39 +0200 Subject: [Python-ideas] infinite dates: psycopg2 would support them Message-ID: <551AF2D3.7010102@thomas-guettler.de> Hi, any progress on infinite datetimes? I just got the commitment from psycopg2 (lib to connect to PostgreSQL) developers, that they will support them, if python would: https://github.com/psycopg/psycopg2/issues/283#issuecomment-88016783 I can't help you with code in this context. I have never started a crowd funding campaign. Do you think it would be successful? Regards, Thomas G?ttler -- http://www.thomas-guettler.de/