From g.brandl at gmx.net Mon Jul 1 08:33:38 2013 From: g.brandl at gmx.net (Georg Brandl) Date: Mon, 01 Jul 2013 08:33:38 +0200 Subject: [Python-Dev] End of the mystery "@README.txt Mercurial bug" In-Reply-To: References: <51CA4DB8.8060802@trueblade.com> Message-ID: Am 26.06.2013 16:24, schrieb Victor Stinner: > 2013/6/26 Eric V. Smith : >> I think that's exactly what's happening. >> >> From the bug report: >> >> find $(srcdir) '(' -name '*.fdc' -o -name '*~' \ >> -o -name '[@,#]*' -o -name '*.old' \ >> -o -name '*.orig' -o -name '*.rej' \ >> -o -name '*.bak' ')' \ >> -exec rm -f {} ';' >> >> Will find files beginning with '@' inside subdirectories of $(srcdir)/.hg. > > In my opinion, make distclean should only remove files generated by > configure and a build. It should not remove random files. > > *~, .orig, .rej, .back should be kept. They are not generated by > configure nor make. I think you want "make clean" then. Georg From rdmurray at bitdance.com Mon Jul 1 13:19:12 2013 From: rdmurray at bitdance.com (R. David Murray) Date: Mon, 01 Jul 2013 07:19:12 -0400 Subject: [Python-Dev] End of the mystery "@README.txt Mercurial bug" In-Reply-To: References: <51CA4DB8.8060802@trueblade.com> Message-ID: <20130701111913.5CA932505F1@webabinitio.net> On Mon, 01 Jul 2013 08:33:38 +0200, Georg Brandl wrote: > Am 26.06.2013 16:24, schrieb Victor Stinner: > > 2013/6/26 Eric V. Smith : > >> I think that's exactly what's happening. > >> > >> From the bug report: > >> > >> find $(srcdir) '(' -name '*.fdc' -o -name '*~' \ > >> -o -name '[@,#]*' -o -name '*.old' \ > >> -o -name '*.orig' -o -name '*.rej' \ > >> -o -name '*.bak' ')' \ > >> -exec rm -f {} ';' > >> > >> Will find files beginning with '@' inside subdirectories of $(srcdir)/.hg. > > > > In my opinion, make distclean should only remove files generated by > > configure and a build. It should not remove random files. > > > > *~, .orig, .rej, .back should be kept. They are not generated by > > configure nor make. > > I think you want "make clean" then. That's a good point. If the find were dropped, the target would have to be renamed. "make configureclean", maybe. But I think it is easier and less confusing just to leave things as they will be after Eric applies the fix proposed in http://bugs.python.org/issue18312. --David From greg at krypto.org Mon Jul 1 22:54:08 2013 From: greg at krypto.org (Gregory P. Smith) Date: Mon, 1 Jul 2013 13:54:08 -0700 Subject: [Python-Dev] End of the mystery "@README.txt Mercurial bug" In-Reply-To: <20130701111913.5CA932505F1@webabinitio.net> References: <51CA4DB8.8060802@trueblade.com> <20130701111913.5CA932505F1@webabinitio.net> Message-ID: If we disallowed builds *from in source tree* requiring all output to go into a separate build output directory instead (like any sane person does*) we wouldn't need a crazy find in the source tree to mess things up. ;) this can be done today: $ mkdir foo && cd foo && ../my-hg/2.7/configure --srcdir=../my-hg/2.7 && make -j12 I think all we'd need to do is disallow the cwd when configuring or building from being within srcdir. -gps * note: the author is normally too lazy to be sane because it involves slightly more typing. On Mon, Jul 1, 2013 at 4:19 AM, R. David Murray wrote: > On Mon, 01 Jul 2013 08:33:38 +0200, Georg Brandl wrote: > > Am 26.06.2013 16:24, schrieb Victor Stinner: > > > 2013/6/26 Eric V. Smith : > > >> I think that's exactly what's happening. > > >> > > >> From the bug report: > > >> > > >> find $(srcdir) '(' -name '*.fdc' -o -name '*~' \ > > >> -o -name '[@,#]*' -o -name '*.old' \ > > >> -o -name '*.orig' -o -name '*.rej' \ > > >> -o -name '*.bak' ')' \ > > >> -exec rm -f {} ';' > > >> > > >> Will find files beginning with '@' inside subdirectories of > $(srcdir)/.hg. > > > > > > In my opinion, make distclean should only remove files generated by > > > configure and a build. It should not remove random files. > > > > > > *~, .orig, .rej, .back should be kept. They are not generated by > > > configure nor make. > > > > I think you want "make clean" then. > > That's a good point. If the find were dropped, the target would have to > be renamed. "make configureclean", maybe. But I think it is easier and > less confusing just to leave things as they will be after Eric applies > the fix proposed in http://bugs.python.org/issue18312. > > --David > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/greg%40krypto.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: From victor.stinner at gmail.com Mon Jul 1 23:05:56 2013 From: victor.stinner at gmail.com (Victor Stinner) Date: Mon, 1 Jul 2013 23:05:56 +0200 Subject: [Python-Dev] End of the mystery "@README.txt Mercurial bug" In-Reply-To: References: <51CA4DB8.8060802@trueblade.com> Message-ID: Hi Georg, 2013/7/1 Georg Brandl : > Am 26.06.2013 16:24, schrieb Victor Stinner: >> 2013/6/26 Eric V. Smith : >>> I think that's exactly what's happening. >>> >>> From the bug report: >>> >>> find $(srcdir) '(' -name '*.fdc' -o -name '*~' \ >>> -o -name '[@,#]*' -o -name '*.old' \ >>> -o -name '*.orig' -o -name '*.rej' \ >>> -o -name '*.bak' ')' \ >>> -exec rm -f {} ';' >>> >>> Will find files beginning with '@' inside subdirectories of $(srcdir)/.hg. >> >> In my opinion, make distclean should only remove files generated by >> configure and a build. It should not remove random files. >> >> *~, .orig, .rej, .back should be kept. They are not generated by >> configure nor make. > > I think you want "make clean" then. I don't understand why you are suggesting me to use "make clean". I would like to start a fresh build, so remove configure and Makefile, but I also want to keep my local changes and local files not tracked by Mercurial. I need this when the build does not work because a new file was added or a build script was modified. "make clean" does not change anything for this use case. For example, I don't understand why "make distclean" removes the "tags" file. Generating this file takes 20 to 30 seconds on my slow laptop, and it is not generated by Python build system, but by the external ctags program. Don't you think that we need two different "distclean" commands? One GNU-style "distclean" which only removes configure and Makefile, and another "distclean" which is the GNU "distclean" + the extra find removing temporary files. Victor From rdmurray at bitdance.com Mon Jul 1 23:13:28 2013 From: rdmurray at bitdance.com (R. David Murray) Date: Mon, 01 Jul 2013 17:13:28 -0400 Subject: [Python-Dev] End of the mystery "@README.txt Mercurial bug" In-Reply-To: References: <51CA4DB8.8060802@trueblade.com> Message-ID: <20130701211328.ECF1D2500E3@webabinitio.net> On Mon, 01 Jul 2013 23:05:56 +0200, Victor Stinner wrote: > Hi Georg, > > 2013/7/1 Georg Brandl : > > Am 26.06.2013 16:24, schrieb Victor Stinner: > >> 2013/6/26 Eric V. Smith : > >>> I think that's exactly what's happening. > >>> > >>> From the bug report: > >>> > >>> find $(srcdir) '(' -name '*.fdc' -o -name '*~' \ > >>> -o -name '[@,#]*' -o -name '*.old' \ > >>> -o -name '*.orig' -o -name '*.rej' \ > >>> -o -name '*.bak' ')' \ > >>> -exec rm -f {} ';' > >>> > >>> Will find files beginning with '@' inside subdirectories of $(srcdir)/.hg. > >> > >> In my opinion, make distclean should only remove files generated by > >> configure and a build. It should not remove random files. > >> > >> *~, .orig, .rej, .back should be kept. They are not generated by > >> configure nor make. > > > > I think you want "make clean" then. > > I don't understand why you are suggesting me to use "make clean". I > would like to start a fresh build, so remove configure and Makefile, > but I also want to keep my local changes and local files not tracked > by Mercurial. I need this when the build does not work because a new > file was added or a build script was modified. "make clean" does not > change anything for this use case. > > For example, I don't understand why "make distclean" removes the > "tags" file. Generating this file takes 20 to 30 seconds on my slow > laptop, and it is not generated by Python build system, but by the > external ctags program. > > Don't you think that we need two different "distclean" commands? One > GNU-style "distclean" which only removes configure and Makefile, and > another "distclean" which is the GNU "distclean" + the extra find > removing temporary files. The command that does not remove the extra files is *not* a 'distclean' command. 'buildclean' or 'configclean', but not 'distclean'. distclean still needs to be fixed, so please open a new issue for adding buildclean or whatever you want to call it, as Eric requested in the existing issue. --David From eric at trueblade.com Mon Jul 1 23:19:55 2013 From: eric at trueblade.com (Eric V. Smith) Date: Mon, 01 Jul 2013 17:19:55 -0400 Subject: [Python-Dev] End of the mystery "@README.txt Mercurial bug" In-Reply-To: <20130701211328.ECF1D2500E3@webabinitio.net> References: <51CA4DB8.8060802@trueblade.com> <20130701211328.ECF1D2500E3@webabinitio.net> Message-ID: <51D1F27B.6070001@trueblade.com> On 7/1/2013 5:13 PM, R. David Murray wrote: > distclean still needs to be fixed, so please open a new issue for > adding buildclean or whatever you want to call it, as Eric requested > in the existing issue. I finally got around to testing it today, so I'm going to check in my change to 18312 sometime shortly. -- Eric. From g.brandl at gmx.net Tue Jul 2 00:06:37 2013 From: g.brandl at gmx.net (Georg Brandl) Date: Tue, 02 Jul 2013 00:06:37 +0200 Subject: [Python-Dev] End of the mystery "@README.txt Mercurial bug" In-Reply-To: References: <51CA4DB8.8060802@trueblade.com> Message-ID: Am 01.07.2013 23:05, schrieb Victor Stinner: > Hi Georg, > > 2013/7/1 Georg Brandl : >> Am 26.06.2013 16:24, schrieb Victor Stinner: >>> 2013/6/26 Eric V. Smith : >>>> I think that's exactly what's happening. >>>> >>>> From the bug report: >>>> >>>> find $(srcdir) '(' -name '*.fdc' -o -name '*~' \ >>>> -o -name '[@,#]*' -o -name '*.old' \ >>>> -o -name '*.orig' -o -name '*.rej' \ >>>> -o -name '*.bak' ')' \ >>>> -exec rm -f {} ';' >>>> >>>> Will find files beginning with '@' inside subdirectories of $(srcdir)/.hg. >>> >>> In my opinion, make distclean should only remove files generated by >>> configure and a build. It should not remove random files. >>> >>> *~, .orig, .rej, .back should be kept. They are not generated by >>> configure nor make. >> >> I think you want "make clean" then. > > I don't understand why you are suggesting me to use "make clean". I > would like to start a fresh build, so remove configure and Makefile, > but I also want to keep my local changes and local files not tracked > by Mercurial. I need this when the build does not work because a new > file was added or a build script was modified. "make clean" does not > change anything for this use case. Right, I had wrongly remembered that "clean" also removed the Makefile. Note that to add to the confusion, there's an additional target named "clobber" (which is called by distclean). Both "clobber" and "distclean" remove Python generated and potentially user generated files. IMO things could be rearranged without too much effort so that "distclean" only removes Python-build-generated files, whereas "clobber" removes also user generated files. But please don't introduce yet another target. Georg From agriff at tin.it Tue Jul 2 11:27:02 2013 From: agriff at tin.it (Andrea Griffini) Date: Tue, 2 Jul 2013 11:27:02 +0200 Subject: [Python-Dev] Why are captured parameters also listed in co_varnames? Message-ID: I'm trying to understand how CPython implements closure variable capture and there is one minor point I can't understand. When a local is captured it gets allocated in co_cellvars and is accessed with (LOAD|STORE)_DEREF, and this is clear. However when a local is coming from a parameter it gets ALSO allocated in co_varnames even if the local slot apparently is not accesible because *_FAST opcodes are not generated. Is there a technical reason for this? It happens in CPython 2, 3 and even in PyPy... Andrea Griffini -------------- next part -------------- An HTML attachment was scrubbed... URL: From amauryfa at gmail.com Tue Jul 2 11:45:27 2013 From: amauryfa at gmail.com (Amaury Forgeot d'Arc) Date: Tue, 2 Jul 2013 11:45:27 +0200 Subject: [Python-Dev] Why are captured parameters also listed in co_varnames? In-Reply-To: References: Message-ID: 2013/7/2 Andrea Griffini > I'm trying to understand how CPython implements closure variable capture > and there is one minor point I can't understand. > > When a local is captured it gets allocated in co_cellvars and is accessed > with (LOAD|STORE)_DEREF, and this is clear. > However when a local is coming from a parameter it gets ALSO allocated in > co_varnames even if the local slot apparently is not accesible because > *_FAST opcodes are not generated. > > Is there a technical reason for this? It happens in CPython 2, 3 and even > in PyPy... > > co_varnames is also used in error messages, for example in the following code: >>> def f(): ... def g(): ... x ... print x ... x = 1 ... return g ... >>> f() UnboundLocalError: local variable 'x' referenced before assignment This is also needed when x is a parameter of f(), for inspect.signature of course, but also because in python3 you can "del x". -- Amaury Forgeot d'Arc -------------- next part -------------- An HTML attachment was scrubbed... URL: From ronaldoussoren at mac.com Tue Jul 2 13:51:13 2013 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Tue, 2 Jul 2013 13:51:13 +0200 Subject: [Python-Dev] Hooking into super() attribute resolution Message-ID: Hi, Below is a very preliminary draft PEP for adding a special method that can be used to hook into the attribute resolution process of the super object. The primary usecase for using this special method are classes that perform custom logic in their __getattribute__ method, where the default behavior of super (peekin the the class __dict__) is not appropriate. The primary reason I wrote this proposal is PyObjC: it dynamicly looks up methods in its __getattribute__ and caches the result in the class __dict__, because of this super() will often not work correctly and therefore I'm currently shipping a custom subclass of super() that basicly contains an in-line implementation of the hook that would be used by PyObjC. I have a partial implementation of the hook system in issue 18181 and a PyObjC patch that uses it. The implementation currently does not contain tests, and I'm sure that I'll find edge cases that I haven't thought about yet when I add tests. Ronald PEP: TODO Title: Hooking into super attribute resolution Version: $Revision$ Last-Modified: $Date$ Author: Ronald Oussoren Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 12-Jun-2013 Post-History: 2-Jul-2013 Abstract ======== In current python releases the attribute resolution of the `super class`_ peeks in the ``__dict__`` attribute of classes on the MRO to look for attributes. This PEP introduces a hook that classes can use to override that behavior for specific classes. Rationale ========= Peeking in the class ``__dict__`` works for regular classes, but can cause problems when a class dynamicly looks up attributes in a ``__getattribute__`` method. The new hook makes it possible to introduce the same customization for attribute lookup through the `super class`_. The superclass attribute lookup hook ==================================== In C code --------- A new slot ``tp_getattro_super`` is added to the ``PyTypeObject`` struct. The ``tp_getattro`` slot for super will call this slot when it is not ``NULL``, otherwise it will peek in the class ``tp_dict``. The slot has the following prototype:: PyObject* (*getattrosuperfunc)(PyTypeObject* tp, PyObject* self, PyObject* name); The function should perform attribute lookup for *name*, but only looking in type *tp* (which will be one of the types on the MRO for *self*) and without looking in the instance *__dict__*. The function returns ``NULL`` when the attribute cannot be found, and raises and exception. Exception other than ``AttributeError`` will cause failure of super's attribute resolution. In Python code -------------- A Python class can contain a definition for a method ``__getattribute_super__`` with the following prototype:: def __getattribute_super__(self, cls, name): pass The method should perform attribute lookup for *name* on instance *self* while only looking at *cls* (it should not look in super classes or the instance *__dict__* Alternative proposals --------------------- Reuse ``tp_getattro`` ..................... It would be nice to avoid adding a new slot, thus keeping the API simpler and easier to understand. A comment on `Issue 18181`_ asked about reusing the ``tp_getattro`` slot, that is super could call the ``tp_getattro`` slot of all methods along the MRO. AFAIK that won't work because ``tp_getattro`` will look in the instance ``__dict__`` before it tries to resolve attributes using classes in the MRO. This would mean that using ``tp_getattro`` instead of peeking the class dictionaries changes the semantics of the `super class`_. Open Issues =========== * The names of the new slot and magic method are far from settled. * I'm not too happy with the prototype for the new hook. * Should ``__getattribute_super__`` be a class method instead? References ========== * `Issue 18181`_ contains a prototype implementation The prototype uses different names than this proposal. Copyright ========= This document has been placed in the public domain. .. _`Issue 18181`: http://bugs.python.org/issue18181 .. _`super class`: http://docs.python.org/3/library/functions.html?highlight=super#super From eric at trueblade.com Tue Jul 2 15:52:56 2013 From: eric at trueblade.com (Eric V. Smith) Date: Tue, 02 Jul 2013 09:52:56 -0400 Subject: [Python-Dev] Issue 18312 "fix" broke buildbots Message-ID: <51D2DB38.8090106@trueblade.com> I'm not sure how my change broke the buildbots, but apparently it did. I need to run to a meeting now, but I can roll this back in the next few hours. If someone else wants to roll it back before I get to it, feel free. Sorry about the problem. I tested it locally, I'm not sure how the buildbots are affected. Eric. From rdmurray at bitdance.com Tue Jul 2 16:33:35 2013 From: rdmurray at bitdance.com (R. David Murray) Date: Tue, 02 Jul 2013 10:33:35 -0400 Subject: [Python-Dev] Issue 18312 "fix" broke buildbots In-Reply-To: <51D2DB38.8090106@trueblade.com> References: <51D2DB38.8090106@trueblade.com> Message-ID: <20130702143335.ACD702500E3@webabinitio.net> On Tue, 02 Jul 2013 09:52:56 -0400, "Eric V. Smith" wrote: > I'm not sure how my change broke the buildbots, but apparently it did. I > need to run to a meeting now, but I can roll this back in the next few > hours. > > If someone else wants to roll it back before I get to it, feel free. > > Sorry about the problem. I tested it locally, I'm not sure how the > buildbots are affected. I'm no longer sure it was your patch. Anyone who wants to look at the buildbots please do: http://buildbot.python.org/all/builders/x86%20Tiger%202.7/builds/2030 http://buildbot.python.org/all/builders/x86%20Tiger%203.3/builds/742 http://buildbot.python.org/all/builders/x86%20Tiger%203.x/builds/6522 http://buildbot.python.org/all/builders/AMD64%20FreeBSD%209.0%20dtrace%2Bclang%203.3/builds/699 http://buildbot.python.org/all/builders/x86%20Windows7%202.7/builds/2125 http://buildbot.python.org/all/builders/bolen-dmg-3.x/builds/278 http://buildbot.python.org/all/builders/x86%20XP-4%203.3/builds/832 I haven't looked at all of these. The blamelist cites Richard Oudkerk, but at least one of the logs I looked at was about @test files, in a number of different tests. I don't really have time to look at it firther right now either, I've got a client with an issue that needs solved asap... --David From eric at trueblade.com Tue Jul 2 16:57:11 2013 From: eric at trueblade.com (Eric V. Smith) Date: Tue, 02 Jul 2013 10:57:11 -0400 Subject: [Python-Dev] Issue 18312 "fix" broke buildbots In-Reply-To: <20130702143335.ACD702500E3@webabinitio.net> References: <51D2DB38.8090106@trueblade.com> <20130702143335.ACD702500E3@webabinitio.net> Message-ID: <51D2EA47.4030004@trueblade.com> On 07/02/2013 10:33 AM, R. David Murray wrote: > On Tue, 02 Jul 2013 09:52:56 -0400, "Eric V. Smith" wrote: >> I'm not sure how my change broke the buildbots, but apparently it did. I >> need to run to a meeting now, but I can roll this back in the next few >> hours. >> >> If someone else wants to roll it back before I get to it, feel free. >> >> Sorry about the problem. I tested it locally, I'm not sure how the >> buildbots are affected. > > I'm no longer sure it was your patch. Anyone who wants to look > at the buildbots please do: > > http://buildbot.python.org/all/builders/x86%20Tiger%202.7/builds/2030 > http://buildbot.python.org/all/builders/x86%20Tiger%203.3/builds/742 > http://buildbot.python.org/all/builders/x86%20Tiger%203.x/builds/6522 > http://buildbot.python.org/all/builders/AMD64%20FreeBSD%209.0%20dtrace%2Bclang%203.3/builds/699 > http://buildbot.python.org/all/builders/x86%20Windows7%202.7/builds/2125 > http://buildbot.python.org/all/builders/bolen-dmg-3.x/builds/278 > http://buildbot.python.org/all/builders/x86%20XP-4%203.3/builds/832 > > I haven't looked at all of these. The blamelist cites Richard Oudkerk, > but at least one of the logs I looked at was about @test files, in > a number of different tests. > > I don't really have time to look at it firther right now either, I've > got a client with an issue that needs solved asap... I've looked at a few of these, and they don't seem related to my change. I'll review it some more this afternoon. Eric. From pythoniks at gmail.com Tue Jul 2 20:31:48 2013 From: pythoniks at gmail.com (Pascal Chambon) Date: Tue, 02 Jul 2013 20:31:48 +0200 Subject: [Python-Dev] Solving the import-deadlock case Message-ID: <51D31C94.3010203@gmail.com> Hello everyone, I'd like to bring your attention to this issue, since it touches the fundamentals of python's import workflow: http://bugs.python.org/issue17716 /I've tried to post it on the python-import ML for weeks, but it must still be blocked somewhere in a moderation queue, so here I come ^^/ TLDR version: because of the way import current works, if importing a package "temporarily" fails whereas importing one of its children succeeded, we reach an unusable state, all subsequent attempts at importing that package will fail if a "from...import" is used somewhere. Typically, it makes a web worker broken, even though the typical behaviour of such process woudl be to retry loading, again and again, the failing view. I agree that a module loading should be, as much as possible, "side effects free", and thus shouldn't have temporary errors. But well, in practice, module loading is typically the time where process-wide initialization are done (modifying sys.path, os.environ, instantiating connection or thread pools, registering atexit handler, starting maintenance threads...), so that case has chances to happen at a moment or another, especially if accesses to filesystem or network (SQL...) are done at module loading, due to the lack of initialization system at upper levels. That's why I propose modifying the behaviour of module import, so that submodules are deleted as well when a parent module import fails. True, it means they will be reloaded as well when importing the parent will start again, but anyway we already have a "double execution" problem with the reloading of the parent module, so it shouldn't make a big difference. The only other solution I'd see would be to SYSTEMATICALLY perform name (re)binding when processing a from...import statement, to recover from the previously failed initialization. Dunno if it's a good idea. On a (separate but related) topic, to be safer on module reimports or reloadings, it could be interesting to add some "idempotency" to common initialization tasks ; for example the "atexit" registration system, wouldn't it be worth adding a boolean flag to explicitely skip registration if a callable with same fully qualified name is already registered. Do you have opinions on these subjects ? thanks, regards, Pascal -------------- next part -------------- An HTML attachment was scrubbed... URL: From rdmurray at bitdance.com Tue Jul 2 21:41:46 2013 From: rdmurray at bitdance.com (R. David Murray) Date: Tue, 02 Jul 2013 15:41:46 -0400 Subject: [Python-Dev] Solving the import-deadlock case In-Reply-To: <51D31C94.3010203@gmail.com> References: <51D31C94.3010203@gmail.com> Message-ID: <20130702194146.780342500CB@webabinitio.net> On Tue, 02 Jul 2013 20:31:48 +0200, Pascal Chambon wrote: > I agree that a module loading should be, as much as possible, "side > effects free", and thus shouldn't have temporary errors. But well, in > practice, module loading is typically the time where process-wide > initialization are done (modifying sys.path, os.environ, instantiating > connection or thread pools, registering atexit handler, starting > maintenance threads...), so that case has chances to happen at a moment > or another, especially if accesses to filesystem or network (SQL...) are > done at module loading, due to the lack of initialization system at > upper levels. There may well be a bug that could be/should be fixed here, but...it seems to me that other than the sys.path modifications, doing any of that at module import time has a strong code smell. --David From ncoghlan at gmail.com Tue Jul 2 23:24:46 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 3 Jul 2013 07:24:46 +1000 Subject: [Python-Dev] Solving the import-deadlock case In-Reply-To: <20130702194146.780342500CB@webabinitio.net> References: <51D31C94.3010203@gmail.com> <20130702194146.780342500CB@webabinitio.net> Message-ID: On 3 Jul 2013 05:44, "R. David Murray" wrote: > > On Tue, 02 Jul 2013 20:31:48 +0200, Pascal Chambon wrote: > > I agree that a module loading should be, as much as possible, "side > > effects free", and thus shouldn't have temporary errors. But well, in > > practice, module loading is typically the time where process-wide > > initialization are done (modifying sys.path, os.environ, instantiating > > connection or thread pools, registering atexit handler, starting > > maintenance threads...), so that case has chances to happen at a moment > > or another, especially if accesses to filesystem or network (SQL...) are > > done at module loading, due to the lack of initialization system at > > upper levels. > > There may well be a bug that could be/should be fixed here, but...it > seems to me that other than the sys.path modifications, doing any of that > at module import time has a strong code smell. Unfortunately it's one of those "Your code is dubious, but so many people do it anyway we should handle it better than we do" cases. We could also be a lot more emphatic about "import side effects are what marks the boundary between a library and a framework. To stay on the library side of that fence provide a 'start' or 'configure' function instead of doing things implicitly on import". Heck, even *defining* library, framework and application would be a good thing. OTOH, it's hard to find motivation to work on improving the handling of things you think people shouldn't be doing in the first place (that's one of the reasons circular import handling has never been made more consistent). (That's not to dismiss the work Pascal's already done - just pointing out why it may sometimes feel like it's difficult to get interest and feedback on things like this). Cheers, Nick. > > --David > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Tue Jul 2 23:32:53 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 3 Jul 2013 07:32:53 +1000 Subject: [Python-Dev] Solving the import-deadlock case In-Reply-To: <51D31C94.3010203@gmail.com> References: <51D31C94.3010203@gmail.com> Message-ID: On 3 Jul 2013 04:34, "Pascal Chambon" wrote: > > Hello everyone, > > I'd like to bring your attention to this issue, since it touches the fundamentals of python's import workflow: > http://bugs.python.org/issue17716 > > I've tried to post it on the python-import ML for weeks, but it must still be blocked somewhere in a moderation queue, so here I come ^^ > > TLDR version: because of the way import current works, if importing a package "temporarily" fails whereas importing one of its children succeeded, we reach an unusable state, all subsequent attempts at importing that package will fail if a "from...import" is used somewhere. Typically, it makes a web worker broken, even though the typical behaviour of such process woudl be to retry loading, again and again, the failing view. > > I agree that a module loading should be, as much as possible, "side effects free", and thus shouldn't have temporary errors. But well, in practice, module loading is typically the time where process-wide initialization are done (modifying sys.path, os.environ, instantiating connection or thread pools, registering atexit handler, starting maintenance threads...), so that case has chances to happen at a moment or another, especially if accesses to filesystem or network (SQL...) are done at module loading, due to the lack of initialization system at upper levels. > > That's why I propose modifying the behaviour of module import, so that submodules are deleted as well when a parent module import fails. True, it means they will be reloaded as well when importing the parent will start again, but anyway we already have a "double execution" problem with the reloading of the parent module, so it shouldn't make a big difference. > The only other solution I'd see would be to SYSTEMATICALLY perform name (re)binding when processing a from...import statement, to recover from the previously failed initialization. Dunno if it's a good idea. > > On a (separate but related) topic, to be safer on module reimports or reloadings, it could be interesting to add some "idempotency" to common initialization tasks ; for example the "atexit" registration system, wouldn't it be worth adding a boolean flag to explicitely skip registration if a callable with same fully qualified name is already registered. > > Do you have opinions on these subjects ? Back on topic... As I stated on the issue, I think purging the whole subtree when a package implicitly imports child modules is the least bad of the available options, and better than leaving the child modules in place in violation of the "all parent packages can be assumed to be in sys.modules" invariant (which is what we do now). Cheers, Nick. > > thanks, > regards, > Pascal > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com > -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Wed Jul 3 05:49:23 2013 From: guido at python.org (Guido van Rossum) Date: Tue, 2 Jul 2013 20:49:23 -0700 Subject: [Python-Dev] Hooking into super() attribute resolution In-Reply-To: References: Message-ID: No time to read the PEP in detail but the motivation sound reasonable. --Guido van Rossum (sent from Android phone) On Jul 2, 2013 4:53 AM, "Ronald Oussoren" wrote: > Hi, > > Below is a very preliminary draft PEP for adding a special method that can > be used to hook into the attribute resolution process of the super object. > > The primary usecase for using this special method are classes that perform > custom logic in their __getattribute__ method, where the default behavior > of super (peekin the the class __dict__) is not appropriate. The primary > reason I wrote this proposal is PyObjC: it dynamicly looks up methods in > its __getattribute__ and caches the result in the class __dict__, because > of this super() will often not work correctly and therefore I'm currently > shipping a custom subclass of super() that basicly contains an in-line > implementation of the hook that would be used by PyObjC. > > I have a partial implementation of the hook system in issue 18181 and a > PyObjC patch that uses it. The implementation currently does not contain > tests, and I'm sure that I'll find edge cases that I haven't thought about > yet when I add tests. > > Ronald > > > > > PEP: TODO > Title: Hooking into super attribute resolution > Version: $Revision$ > Last-Modified: $Date$ > Author: Ronald Oussoren > Status: Draft > Type: Standards Track > Content-Type: text/x-rst > Created: 12-Jun-2013 > Post-History: 2-Jul-2013 > > > Abstract > ======== > > In current python releases the attribute resolution of the `super class`_ > peeks in the ``__dict__`` attribute of classes on the MRO to look > for attributes. This PEP introduces a hook that classes can use > to override that behavior for specific classes. > > > Rationale > ========= > > Peeking in the class ``__dict__`` works for regular classes, but can > cause problems when a class dynamicly looks up attributes in a > ``__getattribute__`` method. > > The new hook makes it possible to introduce the same customization for > attribute lookup through the `super class`_. > > > The superclass attribute lookup hook > ==================================== > > In C code > --------- > > A new slot ``tp_getattro_super`` is added to the ``PyTypeObject`` struct. > The > ``tp_getattro`` slot for super will call this slot when it is not ``NULL``, > otherwise it will peek in the class ``tp_dict``. > > The slot has the following prototype:: > > PyObject* (*getattrosuperfunc)(PyTypeObject* tp, PyObject* self, > PyObject* name); > > The function should perform attribute lookup for *name*, but only looking > in > type *tp* (which will be one of the types on the MRO for *self*) and > without looking > in the instance *__dict__*. > > The function returns ``NULL`` when the attribute cannot be found, and > raises and > exception. Exception other than ``AttributeError`` will cause failure of > super's > attribute resolution. > > > In Python code > -------------- > > A Python class can contain a definition for a method > ``__getattribute_super__`` with > the following prototype:: > > def __getattribute_super__(self, cls, name): pass > > The method should perform attribute lookup for *name* on instance *self* > while only > looking at *cls* (it should not look in super classes or the instance > *__dict__* > > > Alternative proposals > --------------------- > > Reuse ``tp_getattro`` > ..................... > > It would be nice to avoid adding a new slot, thus keeping the API simpler > and easier > to understand. A comment on `Issue 18181`_ asked about reusing the > ``tp_getattro`` slot, > that is super could call the ``tp_getattro`` slot of all methods along the > MRO. > > AFAIK that won't work because ``tp_getattro`` will look in the instance > ``__dict__`` before > it tries to resolve attributes using classes in the MRO. This would mean > that using > ``tp_getattro`` instead of peeking the class dictionaries changes the > semantics of > the `super class`_. > > > Open Issues > =========== > > * The names of the new slot and magic method are far from settled. > > * I'm not too happy with the prototype for the new hook. > > * Should ``__getattribute_super__`` be a class method instead? > > > References > ========== > > * `Issue 18181`_ contains a prototype implementation > > The prototype uses different names than this proposal. > > > Copyright > ========= > > This document has been placed in the public domain. > > .. _`Issue 18181`: http://bugs.python.org/issue18181 > > .. _`super class`: > http://docs.python.org/3/library/functions.html?highlight=super#super > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/guido%40python.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: From db3l.net at gmail.com Wed Jul 3 06:34:14 2013 From: db3l.net at gmail.com (David Bolen) Date: Wed, 03 Jul 2013 00:34:14 -0400 Subject: [Python-Dev] Issue 18312 "fix" broke buildbots References: <51D2DB38.8090106@trueblade.com> <20130702143335.ACD702500E3@webabinitio.net> <51D2EA47.4030004@trueblade.com> Message-ID: "Eric V. Smith" writes: >> http://buildbot.python.org/all/builders/x86%20Tiger%202.7/builds/2030 >> http://buildbot.python.org/all/builders/x86%20Tiger%203.3/builds/742 >> http://buildbot.python.org/all/builders/x86%20Tiger%203.x/builds/6522 >> http://buildbot.python.org/all/builders/bolen-dmg-3.x/builds/278 (...) > I've looked at a few of these, and they don't seem related to my change. > I'll review it some more this afternoon. The above set (which are all on the x86 Tiger buildbot) appear due to some sort of filesystem corruption on the buildbot machine, so it appears that the timing of the failures and particular changesets is coincidental. I've repaired the filesystem and the disk passes a surface scan, so hopefully it'll be cleaned up. I've also re-queued the most recent builds in each branch. -- David From larry at hastings.org Wed Jul 3 10:36:24 2013 From: larry at hastings.org (Larry Hastings) Date: Wed, 03 Jul 2013 10:36:24 +0200 Subject: [Python-Dev] Expose stack effects to Python? Message-ID: <51D3E288.6000904@hastings.org> I wrote my own assembler for Python bytecode called "Maynard". I had to statically compute the stack effects for each bytecode instruction by hand; what I did was copied and pasted opcode_stack_effect() (which is static) out of Python/compile.c and into my own driver program, then I probed it with test values to produce a table. I then coded up a function using that table, but hand-calculating the value sometimes as there are some opcodes whose stack effect varies based on the oparg. It sure would be nice if this information was simply available to the Python interpreter; theoretically it can change between point releases. Would anybody mind if I added it somewhere? I'd probably just expose opcode_stack_effect to Python, then add all this other junk to the dis module and make it available there. //arry/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From victor.stinner at gmail.com Wed Jul 3 12:06:08 2013 From: victor.stinner at gmail.com (Victor Stinner) Date: Wed, 3 Jul 2013 12:06:08 +0200 Subject: [Python-Dev] Expose stack effects to Python? In-Reply-To: <51D3E288.6000904@hastings.org> References: <51D3E288.6000904@hastings.org> Message-ID: Hi, For my registervm project (fork of CPython using register-based bytecode, instead of stack-based bytecode), I implemented a Instruction.use_stack() method which just checks if the stack is "used": read the stack, exchange values in the stack (like "ROT" instruction), push or pop a value. Instruction.use_stack(): http://hg.python.org/sandbox/registervm/file/ff24dfecc27d/Lib/registervm.py#l546 The method uses a dummy heuristic, just because it was quick to implement it, and it's enough for my use case. To answer your question: yes, I would like a opcode_stack_effect() function. registervm has its own disassembler which creates objects, rather than just generating a text representation of the bytecode. I'm using objects because I rewrite most instructions and I need more information and functions: * disassembler (raw bytes => list of instructions) * assembler (list of instructions => raw bytes) * format an instruction (human readable assembler) * is_terminal(): last instruction of a block * is_cond_jump(): the instruction is a conditional jump? hesitate to move this disassembler to CPython directly. I'm not sure that it would be useful, its API is maybe too specific to my registervm project. * is_reg_used(), is_reg_replaced(), is_reg_modified(), etc.: checks on registers * etc. Would it be useful to have such high-level API in Python? Victor 2013/7/3 Larry Hastings : > > > I wrote my own assembler for Python bytecode called "Maynard". I had to > statically compute the stack effects for each bytecode instruction by hand; > what I did was copied and pasted opcode_stack_effect() (which is static) out > of Python/compile.c and into my own driver program, then I probed it with > test values to produce a table. I then coded up a function using that > table, but hand-calculating the value sometimes as there are some opcodes > whose stack effect varies based on the oparg. > > It sure would be nice if this information was simply available to the Python > interpreter; theoretically it can change between point releases. Would > anybody mind if I added it somewhere? I'd probably just expose > opcode_stack_effect to Python, then add all this other junk to the dis > module and make it available there. > > > /arry > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/victor.stinner%40gmail.com > From ncoghlan at gmail.com Wed Jul 3 14:16:51 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 3 Jul 2013 22:16:51 +1000 Subject: [Python-Dev] Expose stack effects to Python? In-Reply-To: References: <51D3E288.6000904@hastings.org> Message-ID: On 3 July 2013 20:06, Victor Stinner wrote: > Hi, > > For my registervm project (fork of CPython using register-based > bytecode, instead of stack-based bytecode), I implemented a > Instruction.use_stack() method which just checks if the stack is > "used": read the stack, exchange values in the stack (like "ROT" > instruction), push or pop a value. > > Instruction.use_stack(): > > http://hg.python.org/sandbox/registervm/file/ff24dfecc27d/Lib/registervm.py#l546 > > The method uses a dummy heuristic, just because it was quick to > implement it, and it's enough for my use case. > > To answer your question: yes, I would like a opcode_stack_effect() > function. > > registervm has its own disassembler which creates objects, rather than > just generating a text representation of the bytecode. I'm using > objects because I rewrite most instructions and I need more > information and functions: > > * disassembler (raw bytes => list of instructions) > * assembler (list of instructions => raw bytes) > * format an instruction (human readable assembler) > * is_terminal(): last instruction of a block > * is_cond_jump(): the instruction is a conditional jump? hesitate to > move this disassembler to CPython directly. I'm not sure that it would > be useful, its API is maybe too specific to my registervm project. > * is_reg_used(), is_reg_replaced(), is_reg_modified(), etc.: checks on > registers > * etc. > > Would it be useful to have such high-level API in Python? > I finally committed a longstanding patch to add something like that a while ago for 3.4: http://docs.python.org/dev/library/dis#bytecode-analysis It's still fairly low level, but already far more programmatically useful than the old disassembler text. I'm still inclined to push higher level stuff out to external libraries - this patch was mostly about making some of our compiler tests a bit more maintainable, as well as giving third party libraries better building blocks without changing the dis module too much. To get back to Larry's question, though, I think exposing the stack effects through dis.Instruction would be a good idea (since that will have access to the oparg to calculate the variable effects). As far as how to expose the data to Python goes, I suggest adding an _opcode C module to back opcode.py and eliminate the manual duplication of the opcode values while you're at it. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia -------------- next part -------------- An HTML attachment was scrubbed... URL: From fijall at gmail.com Wed Jul 3 14:22:32 2013 From: fijall at gmail.com (Maciej Fijalkowski) Date: Wed, 3 Jul 2013 14:22:32 +0200 Subject: [Python-Dev] Expose stack effects to Python? In-Reply-To: References: <51D3E288.6000904@hastings.org> Message-ID: I wrote a thing that adds more structure to dis (but is not finished) https://bitbucket.org/pypy/pypy/src/15b0489c15d8150b22815312dd283aa5bafcdd67/lib_pypy/disassembler.py?at=default On Wed, Jul 3, 2013 at 2:16 PM, Nick Coghlan wrote: > On 3 July 2013 20:06, Victor Stinner wrote: >> >> Hi, >> >> For my registervm project (fork of CPython using register-based >> bytecode, instead of stack-based bytecode), I implemented a >> Instruction.use_stack() method which just checks if the stack is >> "used": read the stack, exchange values in the stack (like "ROT" >> instruction), push or pop a value. >> >> Instruction.use_stack(): >> >> http://hg.python.org/sandbox/registervm/file/ff24dfecc27d/Lib/registervm.py#l546 >> >> The method uses a dummy heuristic, just because it was quick to >> implement it, and it's enough for my use case. >> >> To answer your question: yes, I would like a opcode_stack_effect() >> function. >> >> registervm has its own disassembler which creates objects, rather than >> just generating a text representation of the bytecode. I'm using >> objects because I rewrite most instructions and I need more >> information and functions: >> >> * disassembler (raw bytes => list of instructions) >> * assembler (list of instructions => raw bytes) >> * format an instruction (human readable assembler) >> >> * is_terminal(): last instruction of a block >> * is_cond_jump(): the instruction is a conditional jump? hesitate to >> move this disassembler to CPython directly. I'm not sure that it would >> be useful, its API is maybe too specific to my registervm project. >> * is_reg_used(), is_reg_replaced(), is_reg_modified(), etc.: checks on >> registers >> * etc. >> >> Would it be useful to have such high-level API in Python? > > > I finally committed a longstanding patch to add something like that a while > ago for 3.4: http://docs.python.org/dev/library/dis#bytecode-analysis > > It's still fairly low level, but already far more programmatically useful > than the old disassembler text. > > I'm still inclined to push higher level stuff out to external libraries - > this patch was mostly about making some of our compiler tests a bit more > maintainable, as well as giving third party libraries better building blocks > without changing the dis module too much. > > To get back to Larry's question, though, I think exposing the stack effects > through dis.Instruction would be a good idea (since that will have access to > the oparg to calculate the variable effects). > > As far as how to expose the data to Python goes, I suggest adding an _opcode > C module to back opcode.py and eliminate the manual duplication of the > opcode values while you're at it. > > Cheers, > Nick. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/fijall%40gmail.com > From chambon.pascal at wanadoo.fr Wed Jul 3 21:55:57 2013 From: chambon.pascal at wanadoo.fr (Pascal Chambon) Date: Wed, 03 Jul 2013 21:55:57 +0200 Subject: [Python-Dev] Solving the import-deadlock case In-Reply-To: References: <51D31C94.3010203@gmail.com> Message-ID: <51D481CD.8040108@wanadoo.fr> Thanks for the comments, in my particular case we're actually on a provisioning /framework/, so we chose the easy (lazy?) way, i.e initializing miscellaneous modules at loading times (like Django or others do, I think), rather than building an proper initialization dispatcher to be called from eg. a wsgi launcher. It works pretty well actually, except that nasty (but fortunately very rare) import deadlock. ^^ Since module loading errors *might* occur for tons of reasons (i.e searching the disk for py files already IS a side effect...), and since the current behaviour (letting children module survive disconnected from their parent) is more harmful than useful, I guess that the cleanup that Nick evocated iwould be the path to follow, wouldn't it ? thanks, Regards, Pascal Le 02/07/2013 23:32, Nick Coghlan a ?crit : > > > On 3 Jul 2013 04:34, "Pascal Chambon" > wrote: > > > > Hello everyone, > > > > I'd like to bring your attention to this issue, since it touches the > fundamentals of python's import workflow: > > http://bugs.python.org/issue17716 > > > > I've tried to post it on the python-import ML for weeks, but it must > still be blocked somewhere in a moderation queue, so here I come ^^ > > > > TLDR version: because of the way import current works, if importing > a package "temporarily" fails whereas importing one of its children > succeeded, we reach an unusable state, all subsequent attempts at > importing that package will fail if a "from...import" is used > somewhere. Typically, it makes a web worker broken, even though the > typical behaviour of such process woudl be to retry loading, again and > again, the failing view. > > > > I agree that a module loading should be, as much as possible, "side > effects free", and thus shouldn't have temporary errors. But well, in > practice, module loading is typically the time where process-wide > initialization are done (modifying sys.path, os.environ, instantiating > connection or thread pools, registering atexit handler, starting > maintenance threads...), so that case has chances to happen at a > moment or another, especially if accesses to filesystem or network > (SQL...) are done at module loading, due to the lack of initialization > system at upper levels. > > > > That's why I propose modifying the behaviour of module import, so > that submodules are deleted as well when a parent module import fails. > True, it means they will be reloaded as well when importing the parent > will start again, but anyway we already have a "double execution" > problem with the reloading of the parent module, so it shouldn't make > a big difference. > > The only other solution I'd see would be to SYSTEMATICALLY perform > name (re)binding when processing a from...import statement, to recover > from the previously failed initialization. Dunno if it's a good idea. > > > > On a (separate but related) topic, to be safer on module reimports > or reloadings, it could be interesting to add some "idempotency" to > common initialization tasks ; for example the "atexit" registration > system, wouldn't it be worth adding a boolean flag to explicitely skip > registration if a callable with same fully qualified name is already > registered. > > > > Do you have opinions on these subjects ? > > Back on topic... > > As I stated on the issue, I think purging the whole subtree when a > package implicitly imports child modules is the least bad of the > available options, and better than leaving the child modules in place > in violation of the "all parent packages can be assumed to be in > sys.modules" invariant (which is what we do now). > > Cheers, > Nick. > > > > thanks, > > regards, > > Pascal > > > > _______________________________________________ > > Python-Dev mailing list > > Python-Dev at python.org > > http://mail.python.org/mailman/listinfo/python-dev > > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com > > > > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/chambon.pascal%40wanadoo.fr -------------- next part -------------- An HTML attachment was scrubbed... URL: From victor.stinner at gmail.com Thu Jul 4 13:03:06 2013 From: victor.stinner at gmail.com (Victor Stinner) Date: Thu, 4 Jul 2013 13:03:06 +0200 Subject: [Python-Dev] PEP 446: Add new parameters to configure the inherance of files and for non-blocking sockets Message-ID: HTML version: http://www.python.org/dev/peps/pep-0446/ PEP: 446 Title: Add new parameters to configure the inherance of files and for non-blocking sockets Version: $Revision$ Last-Modified: $Date$ Author: Victor Stinner Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 3-July-2013 Python-Version: 3.4 Abstract ======== This PEP proposes new portable parameters and functions to configure the inherance of file descriptors and the non-blocking flag of sockets. Rationale ========= Inherance of file descriptors ----------------------------- The inherance of file descriptors in child processes can be configured on each file descriptor using a *close-on-exec* flag. By default, the close-on-exec flag is not set. On Windows, file descriptors are not inherited if the ``bInheritHandles`` parameter of the ``CreateProcess()`` function is ``FALSE``, even if the close-on-exec flag is not set. On UNIX, file descriptors with the close-and-exec flag set are closed at the execution of a new program (ex: when calling ``execv()``). The flag has no effect on ``fork()``, all file descriptors are inherited by the child process. Issues of the inherance of file descriptors ------------------------------------------- Inherance of file descriptors causes issues. For example, closing a file descriptor in the parent process does not release the resource (file, socket, ...), because the file descriptor is still open in the child process. Leaking file descriptors is also a major security vulnerability. An untrusted child process can read sensitive data like passwords and take control of the parent process though leaked file descriptors. It is for example a known vulnerability to escape from a chroot. Non-blocking sockets -------------------- To handle multiple network clients in a single thread, a multiplexing function like ``select()`` can be used. For best performances, sockets must be configured as non-blocking. Operations like ``send()`` and ``recv()`` return an ``EAGAIN`` or ``EWOULDBLOCK`` error if the operation would block. By default, newly created sockets are blocking. Setting the non-blocking mode requires additional system calls. Setting flags at the creation of the file descriptor ---------------------------------------------------- Windows and recent versions of other operating systems like Linux support setting the close-on-exec flag directly at the creation of file descriptors, and close-on-exec and blocking flags at the creation of sockets. Setting these flags at the creation is atomic and avoids additional system calls. Proposal ======== New cloexec And blocking Parameters ----------------------------------- Add a new optional *cloexec* on functions creating file descriptors: * ``io.FileIO`` * ``io.open()`` * ``open()`` * ``os.dup()`` * ``os.dup2()`` * ``os.fdopen()`` * ``os.open()`` * ``os.openpty()`` * ``os.pipe()`` * ``select.devpoll()`` * ``select.epoll()`` * ``select.kqueue()`` Add new optional *cloexec* and *blocking* parameters to functions creating sockets: * ``asyncore.dispatcher.create_socket()`` * ``socket.socket()`` * ``socket.socket.accept()`` * ``socket.socket.dup()`` * ``socket.socket.fromfd`` * ``socket.socketpair()`` The default value of *cloexec* is ``False`` and the default value of *blocking* is ``True``. The atomicity is not guaranteed. If the platform does not support setting close-on-exec and blocking flags at the creation of the file descriptor or socket, the flags are set using additional system calls. New Functions ------------- Add new functions the get and set the close-on-exec flag of a file descriptor: * ``os.get_cloexec(fd:int) -> bool`` * ``os.set_cloexec(fd:int, cloexec: bool)`` Other Changes ------------- The ``subprocess.Popen`` class must clear the close-on-exec flag of file descriptors of the ``pass_fds`` parameter. The close-on-exec flag must also be set on private file descriptors and sockets in the Python standard library. For example, on UNIX, os.urandom() opens ``/dev/urandom`` to read some random bytes and the file descriptor is closed at function exit. The file descriptor is not expected to be inherited by child processes. Rejected Alternatives ===================== PEP 433 ------- The PEP 433 entitled "Easier suppression of file descriptor inheritance" is a previous attempt proposing various other alternatives, but no consensus could be reached. This PEP has a well defined behaviour (the default value of the new *cloexec* parameter is not configurable), is more conservative (no backward compatibility issue), and is much simpler. Add blocking parameter for file descriptors and Windows overlapped I/O ---------------------------------------------------------------------- Windows supports non-blocking operations on files using an extension of the Windows API called "Overlapped I/O". Using this extension requires to modify the Python standard library and applications to pass a ``OVERLAPPED`` structure and an event loop to wait for the completion of operations. This PEP only tries to expose portable flags on file descriptors and sockets. Supporting overlapped I/O requires an abstraction providing a high-level and portable API for asynchronous operations on files and sockets. Overlapped I/O are out of the scope of this PEP. UNIX supports non-blocking files, moreover recent versions of operating systems support setting the non-blocking flag at the creation of a file descriptor. It would be possible to add a new optional *blocking* parameter to Python functions creating file descriptors. On Windows, creating a file descriptor with ``blocking=False`` would raise a ``NotImplementedError``. This behaviour is not acceptable for the ``os`` module which is designed as a thin wrapper on the C functions of the operating system. If a platform does not support a function, the function should not be available on the platform. For example, the ``os.fork()`` function is not available on Windows. For all these reasons, this alternative was rejected. The PEP 3156 proposes an abstraction for asynchronous I/O supporting non-blocking files on Windows. Links ===== Python issues: * `#10115: Support accept4() for atomic setting of flags at socket creation `_ * `#12105: open() does not able to set flags, such as O_CLOEXEC `_ * `#12107: TCP listening sockets created without FD_CLOEXEC flag `_ * `#16850: Add "e" mode to open(): close-and-exec (O_CLOEXEC) / O_NOINHERIT `_ * `#16860: Use O_CLOEXEC in the tempfile module `_ * `#16946: subprocess: _close_open_fd_range_safe() does not set close-on-exec flag on Linux < 2.6.23 if O_CLOEXEC is defined `_ * `#17070: Use the new cloexec to improve security and avoid bugs `_ Other links: * `Secure File Descriptor Handling `_ (Ulrich Drepper, 2008) Copyright ========= This document has been placed into the public domain. From victor.stinner at gmail.com Thu Jul 4 13:19:01 2013 From: victor.stinner at gmail.com (Victor Stinner) Date: Thu, 4 Jul 2013 13:19:01 +0200 Subject: [Python-Dev] PEP 446: Add new parameters to configure the inherance of files and for non-blocking sockets In-Reply-To: References: Message-ID: 2013/7/4 Victor Stinner : > Add a new optional *cloexec* on functions creating file descriptors: > > * ``io.FileIO`` > * ``io.open()`` > * ``open()`` The PEP 433 proposes adding an "e" mode to open in alternatives. I didn't keep this idea because the fopen() function of the GNU libc library has no mode for the O_NONBLOCK flag. IMO it is not interesting to mention it in the PEP 466. Victor From victor.stinner at gmail.com Thu Jul 4 13:13:06 2013 From: victor.stinner at gmail.com (Victor Stinner) Date: Thu, 4 Jul 2013 13:13:06 +0200 Subject: [Python-Dev] PEP 446: Add new parameters to configure the inherance of files and for non-blocking sockets In-Reply-To: References: Message-ID: > PEP: 446 > Title: Add new parameters to configure the inherance of files and for > non-blocking sockets > (...) > Rejected Alternatives > ===================== > > PEP 433 > ------- > > The PEP 433 entitled "Easier suppression of file descriptor inheritance" > is a previous attempt proposing various other alternatives, but no > consensus could be reached. > > This PEP has a well defined behaviour (the default value of the new > *cloexec* parameter is not configurable), is more conservative (no > backward compatibility issue), and is much simpler. Even if the PEP 433 was not explicitly rejected, no consensus could be reached. I didn't want to loose all my work on this PEP and so I'm proposing something new which should make everbody agrees :-) The PEP 446 is written to supersed the PEP 433. Even if they are very close, many drawbacks of the PEP 433 (especially the configurable default value of the new cloexec parameter) goes away with the PEP 466, which explains why the PEP 466 is much shorter. I preferred to create a new PEP to keep the historic in the PEP 433. I also chose to not detail the implementation in the PEP 466, because the PEP 433 already contains a lot of information. Victor From chris at simplistix.co.uk Thu Jul 4 13:21:47 2013 From: chris at simplistix.co.uk (Chris Withers) Date: Thu, 04 Jul 2013 12:21:47 +0100 Subject: [Python-Dev] lament for the demise of unbound methods Message-ID: <51D55ACB.4050900@simplistix.co.uk> Hi All, In Python 2, I can figure out whether I have a method or a function, and, more importantly, for an unbound method, I can figure out what class the method belongs to: >>> class MyClass(object): ... def method(self): pass ... >>> MyClass.method >>> MyClass.method.im_class There doesn't appear to be any way in Python 3 to do this, which is a little surprising and frustrating... What am I missing here? Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk From ronaldoussoren at mac.com Thu Jul 4 13:52:59 2013 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Thu, 4 Jul 2013 13:52:59 +0200 Subject: [Python-Dev] PEP 446: Add new parameters to configure the inherance of files and for non-blocking sockets In-Reply-To: References: Message-ID: On 4 Jul, 2013, at 13:19, Victor Stinner wrote: > 2013/7/4 Victor Stinner : >> Add a new optional *cloexec* on functions creating file descriptors: >> >> * ``io.FileIO`` >> * ``io.open()`` >> * ``open()`` > > The PEP 433 proposes adding an "e" mode to open in alternatives. I > didn't keep this idea because the fopen() function of the GNU libc > library has no mode for the O_NONBLOCK flag. IMO it is not interesting > to mention it in the PEP 466. I don't understand your reasoning, that is what has GNU libc to do with adding "e" mode to io.open? BTW. I have no particular fondness for an "e" flag, adding a clo_exec flag would be fine and I'm just curious. Ronald From ronaldoussoren at mac.com Thu Jul 4 13:55:58 2013 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Thu, 4 Jul 2013 13:55:58 +0200 Subject: [Python-Dev] lament for the demise of unbound methods In-Reply-To: <51D55ACB.4050900@simplistix.co.uk> References: <51D55ACB.4050900@simplistix.co.uk> Message-ID: On 4 Jul, 2013, at 13:21, Chris Withers wrote: > Hi All, > > In Python 2, I can figure out whether I have a method or a function, and, more importantly, for an unbound method, I can figure out what class the method belongs to: > > >>> class MyClass(object): > ... def method(self): pass > ... > >>> MyClass.method > > >>> MyClass.method.im_class > > > There doesn't appear to be any way in Python 3 to do this, which is a little surprising and frustrating... > > What am I missing here? You can find the fully qualified name of the method with the qualname attribute: >>> class A: ... def method(self): pass ... >>> A.method.__qualname__ 'A.method' Ronald > > Chris > > -- > Simplistix - Content Management, Batch Processing & Python Consulting > - http://www.simplistix.co.uk > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/ronaldoussoren%40mac.com From christian at python.org Thu Jul 4 13:59:23 2013 From: christian at python.org (Christian Heimes) Date: Thu, 04 Jul 2013 13:59:23 +0200 Subject: [Python-Dev] lament for the demise of unbound methods In-Reply-To: <51D55ACB.4050900@simplistix.co.uk> References: <51D55ACB.4050900@simplistix.co.uk> Message-ID: <51D5639B.4060604@python.org> Am 04.07.2013 13:21, schrieb Chris Withers: > There doesn't appear to be any way in Python 3 to do this, which is a > little surprising and frustrating... > > What am I missing here? I removed unbound methods almost six years ago: http://hg.python.org/cpython/rev/48af6375207e Christian From victor.stinner at gmail.com Thu Jul 4 14:04:25 2013 From: victor.stinner at gmail.com (Victor Stinner) Date: Thu, 4 Jul 2013 14:04:25 +0200 Subject: [Python-Dev] PEP 446: Add new parameters to configure the inherance of files and for non-blocking sockets In-Reply-To: References: Message-ID: 2013/7/4 Ronald Oussoren : >> The PEP 433 proposes adding an "e" mode to open in alternatives. I >> didn't keep this idea because the fopen() function of the GNU libc >> library has no mode for the O_NONBLOCK flag. IMO it is not interesting >> to mention it in the PEP 466. > > I don't understand your reasoning, that is what has GNU libc to do with adding "e" mode to io.open? The GNU libc supports fopen(filename, "re") to set O_CLOEXEC flag on the file. I pick the idea into the PEP 433, in alternatives: open(filename, "re") sets O_CLOEXEC flag on the file. For the PEP 466, I only proposed the original API of the PEP 433: open(filename, "r", cloexec=True). Victor From chris at simplistix.co.uk Thu Jul 4 14:13:57 2013 From: chris at simplistix.co.uk (Chris Withers) Date: Thu, 04 Jul 2013 13:13:57 +0100 Subject: [Python-Dev] lament for the demise of unbound methods In-Reply-To: <51D5639B.4060604@python.org> References: <51D55ACB.4050900@simplistix.co.uk> <51D5639B.4060604@python.org> Message-ID: <51D56705.1000908@simplistix.co.uk> On 04/07/2013 12:59, Christian Heimes wrote: > Am 04.07.2013 13:21, schrieb Chris Withers: >> There doesn't appear to be any way in Python 3 to do this, which is a >> little surprising and frustrating... >> >> What am I missing here? > > I removed unbound methods almost six years ago: > > http://hg.python.org/cpython/rev/48af6375207e Not disputing when it happened, more the why... ...the recommended change doesn't work, for obvious reasons: >>> MyClass.method.__self__.__class__ Traceback (most recent call last): File "", line 1, in AttributeError: 'function' object has no attribute '__self__' The loss of the ability to figure out the class from an unbound method seems quite an annoying step back from an introspection point of view. cheers, Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk From chris at simplistix.co.uk Thu Jul 4 14:16:12 2013 From: chris at simplistix.co.uk (Chris Withers) Date: Thu, 04 Jul 2013 13:16:12 +0100 Subject: [Python-Dev] lament for the demise of unbound methods In-Reply-To: References: <51D55ACB.4050900@simplistix.co.uk> Message-ID: <51D5678C.9050205@simplistix.co.uk> On 04/07/2013 12:55, Ronald Oussoren wrote: > > You can find the fully qualified name of the method with the qualname attribute: > >>>> class A: > ... def method(self): pass > ... >>>> A.method.__qualname__ > 'A.method' That doesn't seem helpful as a sensible way to get back to the class object: >> globals()[MyClass.method.__qualname__.split('.')[0]] Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk From brett at python.org Thu Jul 4 14:34:05 2013 From: brett at python.org (Brett Cannon) Date: Thu, 4 Jul 2013 08:34:05 -0400 Subject: [Python-Dev] lament for the demise of unbound methods In-Reply-To: <51D56705.1000908@simplistix.co.uk> References: <51D55ACB.4050900@simplistix.co.uk> <51D5639B.4060604@python.org> <51D56705.1000908@simplistix.co.uk> Message-ID: On Thu, Jul 4, 2013 at 8:13 AM, Chris Withers wrote: > On 04/07/2013 12:59, Christian Heimes wrote: > >> Am 04.07.2013 13:21, schrieb Chris Withers: >> >>> There doesn't appear to be any way in Python 3 to do this, which is a >>> little surprising and frustrating... >>> >>> What am I missing here? >>> >> >> I removed unbound methods almost six years ago: >> >> http://hg.python.org/cpython/**rev/48af6375207e >> > > Not disputing when it happened, more the why... > > ...the recommended change doesn't work, for obvious reasons: > > >>> MyClass.method.__self__.__**class__ > Traceback (most recent call last): > File "", line 1, in > AttributeError: 'function' object has no attribute '__self__' > > The loss of the ability to figure out the class from an unbound method > seems quite an annoying step back from an introspection point of view. > It's only annoying if you take the perspective that methods are somehow special compared to functions. With the removal of bound class methods that makes methods == functions that are an attribute on a class. And when you take that perspective it makes having anything special about methods seem wrong. It also makes adding a function to a class post-class creation make more sense since there is no difference technically. -------------- next part -------------- An HTML attachment was scrubbed... URL: From victor.stinner at gmail.com Thu Jul 4 15:59:23 2013 From: victor.stinner at gmail.com (Victor Stinner) Date: Thu, 4 Jul 2013 15:59:23 +0200 Subject: [Python-Dev] lament for the demise of unbound methods In-Reply-To: <51D5678C.9050205@simplistix.co.uk> References: <51D55ACB.4050900@simplistix.co.uk> <51D5678C.9050205@simplistix.co.uk> Message-ID: 2013/7/4 Chris Withers : > That doesn't seem helpful as a sensible way to get back to the class object: > >>> globals()[MyClass.method.__qualname__.split('.')[0]] > globals() can only be used if MyClass is in the same module. Otherwise, you a more complex function: --------------- import types def get_function_class(func): obj = func for name in func.__qualname__.split('.')[:-1]: if name == "": raise ValueError("you lose") if isinstance(obj, types.FunctionType): obj = func.__globals__[name] else: # get a method of a class, or a class defined in a child obj = getattr(obj, name) return obj --------------- Victor From guido at python.org Thu Jul 4 18:25:31 2013 From: guido at python.org (Guido van Rossum) Date: Thu, 4 Jul 2013 09:25:31 -0700 Subject: [Python-Dev] lament for the demise of unbound methods In-Reply-To: <51D55ACB.4050900@simplistix.co.uk> References: <51D55ACB.4050900@simplistix.co.uk> Message-ID: Chris, what do you want to do with the knowledge you are seeking? --Guido van Rossum (sent from Android phone) On Jul 4, 2013 4:28 AM, "Chris Withers" wrote: > Hi All, > > In Python 2, I can figure out whether I have a method or a function, and, > more importantly, for an unbound method, I can figure out what class the > method belongs to: > > >>> class MyClass(object): > ... def method(self): pass > ... > >>> MyClass.method > > >>> MyClass.method.im_class > > > There doesn't appear to be any way in Python 3 to do this, which is a > little surprising and frustrating... > > What am I missing here? > > Chris > > -- > Simplistix - Content Management, Batch Processing & Python Consulting > - http://www.simplistix.co.uk > ______________________________**_________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/**mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/**mailman/options/python-dev/** > guido%40python.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris at simplistix.co.uk Thu Jul 4 18:42:07 2013 From: chris at simplistix.co.uk (Chris Withers) Date: Thu, 04 Jul 2013 17:42:07 +0100 Subject: [Python-Dev] lament for the demise of unbound methods In-Reply-To: References: <51D55ACB.4050900@simplistix.co.uk> Message-ID: <51D5A5DF.3000103@simplistix.co.uk> Hi Guido, I've bumped into this a couple of times. First time was when I wanted to know whether what I had was a classmethod, staticmethod or normal method here: https://github.com/Simplistix/testfixtures/blob/master/testfixtures/replace.py#L59 This resulted in having to trawl through __dict__ here: https://github.com/Simplistix/testfixtures/blob/master/testfixtures/resolve.py#L17 ...rather than just using getattr. I bumped into it again, yesterday, trying to add support for classes to this lightweight dependency injection framework I'm developing: https://github.com/Simplistix/mush/blob/master/tests/test_runner.py#L189 Here's my local copy of that test: https://gist.github.com/cjw296/db64279c69cdc0c5e112 The workaround I was playing with this morning is a wrapper so that I know I have a class method, although what I really want to write at this line is: https://gist.github.com/cjw296/db64279c69cdc0c5e112#file-gistfile1-txt-L40 runner = Runner(T0, C1.meth, C2.meth1, C2.meth2) ...but if I do that, how can the runner know that what it gets for its second argument is a class method of C1? (which is this case means that it should do C1().meth() rather than C1.meth()) cheers, Chris On 04/07/2013 17:25, Guido van Rossum wrote: > Chris, what do you want to do with the knowledge you are seeking? > > --Guido van Rossum (sent from Android phone) > > On Jul 4, 2013 4:28 AM, "Chris Withers" > wrote: > > Hi All, > > In Python 2, I can figure out whether I have a method or a function, > and, more importantly, for an unbound method, I can figure out what > class the method belongs to: > > >>> class MyClass(object): > ... def method(self): pass > ... > >>> MyClass.method > > >>> MyClass.method.im_class > > > There doesn't appear to be any way in Python 3 to do this, which is > a little surprising and frustrating... > > What am I missing here? > > Chris > > -- > Simplistix - Content Management, Batch Processing & Python Consulting > - http://www.simplistix.co.uk > _________________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/__mailman/listinfo/python-dev > > Unsubscribe: > http://mail.python.org/__mailman/options/python-dev/__guido%40python.org > > > > ______________________________________________________________________ > This email has been scanned by the Symantec Email Security.cloud service. > For more information please visit http://www.symanteccloud.com > ______________________________________________________________________ -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk From guido at python.org Thu Jul 4 19:00:46 2013 From: guido at python.org (Guido van Rossum) Date: Thu, 4 Jul 2013 10:00:46 -0700 Subject: [Python-Dev] lament for the demise of unbound methods In-Reply-To: <51D5A5DF.3000103@simplistix.co.uk> References: <51D55ACB.4050900@simplistix.co.uk> <51D5A5DF.3000103@simplistix.co.uk> Message-ID: Thanks for the code pointers. So it's all about monkeypatching. :-) I have only a little sympathy, as there still seems to be a way to do this, it's just less convenient. Too bad. --Guido On Thu, Jul 4, 2013 at 9:42 AM, Chris Withers wrote: > Hi Guido, > > I've bumped into this a couple of times. > > First time was when I wanted to know whether what I had was a classmethod, > staticmethod or normal method here: > > https://github.com/Simplistix/**testfixtures/blob/master/** > testfixtures/replace.py#L59 > > This resulted in having to trawl through __dict__ here: > > https://github.com/Simplistix/**testfixtures/blob/master/** > testfixtures/resolve.py#L17 > > ...rather than just using getattr. > > I bumped into it again, yesterday, trying to add support for classes to > this lightweight dependency injection framework I'm developing: > > https://github.com/Simplistix/**mush/blob/master/tests/test_** > runner.py#L189 > > Here's my local copy of that test: > > https://gist.github.com/**cjw296/db64279c69cdc0c5e112 > > The workaround I was playing with this morning is a wrapper so that I know > I have a class method, although what I really want to write at this line is: > > https://gist.github.com/**cjw296/db64279c69cdc0c5e112#** > file-gistfile1-txt-L40 > > runner = Runner(T0, C1.meth, C2.meth1, C2.meth2) > > ...but if I do that, how can the runner know that what it gets for its > second argument is a class method of C1? > (which is this case means that it should do C1().meth() rather than > C1.meth()) > > cheers, > > Chris > > > On 04/07/2013 17:25, Guido van Rossum wrote: > >> Chris, what do you want to do with the knowledge you are seeking? >> >> --Guido van Rossum (sent from Android phone) >> >> On Jul 4, 2013 4:28 AM, "Chris Withers" > > wrote: >> >> Hi All, >> >> In Python 2, I can figure out whether I have a method or a function, >> and, more importantly, for an unbound method, I can figure out what >> class the method belongs to: >> >> >>> class MyClass(object): >> ... def method(self): pass >> ... >> >>> MyClass.method >> >> >>> MyClass.method.im_class >> >> >> There doesn't appear to be any way in Python 3 to do this, which is >> a little surprising and frustrating... >> >> What am I missing here? >> >> Chris >> >> -- >> Simplistix - Content Management, Batch Processing & Python Consulting >> - http://www.simplistix.co.uk >> ______________________________**___________________ >> Python-Dev mailing list >> Python-Dev at python.org >> http://mail.python.org/__**mailman/listinfo/python-dev >> >> > >> Unsubscribe: >> http://mail.python.org/__**mailman/options/python-dev/__** >> guido%40python.org >> > guido%40python.org >> > >> >> >> ______________________________**______________________________** >> __________ >> This email has been scanned by the Symantec Email Security.cloud service. >> For more information please visit http://www.symanteccloud.com >> ______________________________**______________________________** >> __________ >> > > -- > Simplistix - Content Management, Batch Processing & Python Consulting > - http://www.simplistix.co.uk > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris at simplistix.co.uk Thu Jul 4 19:20:54 2013 From: chris at simplistix.co.uk (Chris Withers) Date: Thu, 04 Jul 2013 18:20:54 +0100 Subject: [Python-Dev] lament for the demise of unbound methods In-Reply-To: References: <51D55ACB.4050900@simplistix.co.uk> <51D5A5DF.3000103@simplistix.co.uk> Message-ID: <51D5AEF6.50804@simplistix.co.uk> On 04/07/2013 18:00, Guido van Rossum wrote: > Thanks for the code pointers. So it's all about monkeypatching. :-) Well, that's the testfixtures use case, but for mush it's about figuring out whether you need to instantiate a class before calling a callable. MyClass.a_method is a bit like a functools.partial in the mush case, if I can pass that object around and know what to do with it (which I can in Python 2) then I only have to pass that around. In Python3, I either have to pass around the class, the method and a flag to indicate that a class and method are being passed, or wrap my own unboundmethod equivalent, meaning mush users would have to write method(MyClass, 'a_method') under Python 3 when they can just write MyClass.a_method under Python 2. > I > have only a little sympathy, as there still seems to be a way to do > this, it's just less convenient. Too bad. I don't know that Victor's suggestion will actually work in all the cases that MyClass.a_method.im_class does :-S Chris > On Thu, Jul 4, 2013 at 9:42 AM, Chris Withers > Hi Guido, > > I've bumped into this a couple of times. > > First time was when I wanted to know whether what I had was a > classmethod, staticmethod or normal method here: > > https://github.com/Simplistix/__testfixtures/blob/master/__testfixtures/replace.py#L59 > > > This resulted in having to trawl through __dict__ here: > > https://github.com/Simplistix/__testfixtures/blob/master/__testfixtures/resolve.py#L17 > > > ...rather than just using getattr. > > I bumped into it again, yesterday, trying to add support for classes > to this lightweight dependency injection framework I'm developing: > > https://github.com/Simplistix/__mush/blob/master/tests/test___runner.py#L189 > > > Here's my local copy of that test: > > https://gist.github.com/__cjw296/db64279c69cdc0c5e112 > > > The workaround I was playing with this morning is a wrapper so that > I know I have a class method, although what I really want to write > at this line is: > > https://gist.github.com/__cjw296/db64279c69cdc0c5e112#__file-gistfile1-txt-L40 > > > runner = Runner(T0, C1.meth, C2.meth1, C2.meth2) > > ...but if I do that, how can the runner know that what it gets for > its second argument is a class method of C1? > (which is this case means that it should do C1().meth() rather than > C1.meth()) > > cheers, > > Chris > > > On 04/07/2013 17:25, Guido van Rossum wrote: > > Chris, what do you want to do with the knowledge you are seeking? > > --Guido van Rossum (sent from Android phone) > > On Jul 4, 2013 4:28 AM, "Chris Withers" > __>> wrote: > > Hi All, > > In Python 2, I can figure out whether I have a method or a > function, > and, more importantly, for an unbound method, I can figure > out what > class the method belongs to: > > >>> class MyClass(object): > ... def method(self): pass > ... > >>> MyClass.method > > >>> MyClass.method.im_class > > > There doesn't appear to be any way in Python 3 to do this, > which is > a little surprising and frustrating... > > What am I missing here? > > Chris > > -- > Simplistix - Content Management, Batch Processing & Python > Consulting > - http://www.simplistix.co.uk > ___________________________________________________ > Python-Dev mailing list > Python-Dev at python.org > > > http://mail.python.org/____mailman/listinfo/python-dev > > > > Unsubscribe: > http://mail.python.org/____mailman/options/python-dev/____guido%40python.org > > > > > > __________________________________________________________________________ > This email has been scanned by the Symantec Email Security.cloud > service. > For more information please visit http://www.symanteccloud.com > __________________________________________________________________________ > > > -- > Simplistix - Content Management, Batch Processing & Python Consulting > - http://www.simplistix.co.uk > > > > > -- > --Guido van Rossum (python.org/~guido ) > ______________________________________________________________________ > This email has been scanned by the Symantec Email Security.cloud service. > For more information please visit http://www.symanteccloud.com > ______________________________________________________________________ -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk From ericsnowcurrently at gmail.com Thu Jul 4 20:05:08 2013 From: ericsnowcurrently at gmail.com (Eric Snow) Date: Thu, 4 Jul 2013 12:05:08 -0600 Subject: [Python-Dev] lament for the demise of unbound methods In-Reply-To: <51D55ACB.4050900@simplistix.co.uk> References: <51D55ACB.4050900@simplistix.co.uk> Message-ID: On Thu, Jul 4, 2013 at 5:21 AM, Chris Withers wrote: > Hi All, > > In Python 2, I can figure out whether I have a method or a function, and, > more importantly, for an unbound method, I can figure out what class the > method belongs to: > > >>> class MyClass(object): > ... def method(self): pass > ... > >>> MyClass.method > > >>> MyClass.method.im_class > > > There doesn't appear to be any way in Python 3 to do this, which is a > little surprising and frustrating... > > What am I missing here? You could always monkeypatch builtins.__build_class__ to add an attribute to every "unbound method" pointing to the class. -eric -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Thu Jul 4 21:09:06 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 4 Jul 2013 21:09:06 +0200 Subject: [Python-Dev] Oddity in MISC/News on default Message-ID: <20130704210906.74208200@fsol> Hello, In 3.4's Misc/NEWS, there's a huge section entitled "What's New in Python 3.3.1 release candidate 1". It seems it shouldn't be there, should it? Regards Antoine. From guido at python.org Thu Jul 4 21:17:05 2013 From: guido at python.org (Guido van Rossum) Date: Thu, 4 Jul 2013 12:17:05 -0700 Subject: [Python-Dev] Oddity in MISC/News on default In-Reply-To: <20130704210906.74208200@fsol> References: <20130704210906.74208200@fsol> Message-ID: Why not? Presumably those news items were all merged into the default branch, and Misc/NEWS also has all the news for all 3.3.0 releases. Why pick on 3.3.1 rc1? On Thu, Jul 4, 2013 at 12:09 PM, Antoine Pitrou wrote: > > Hello, > > In 3.4's Misc/NEWS, there's a huge section entitled "What's New in > Python 3.3.1 release candidate 1". It seems it shouldn't be there, > should it? > > Regards > > Antoine. > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/guido%40python.org > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Thu Jul 4 21:19:59 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 04 Jul 2013 21:19:59 +0200 Subject: [Python-Dev] Oddity in MISC/News on default In-Reply-To: References: <20130704210906.74208200@fsol> Message-ID: <1372965599.2599.2.camel@fsol> Le jeudi 04 juillet 2013 ? 12:17 -0700, Guido van Rossum a ?crit : > Why not? Presumably those news items were all merged into the default > branch, and Misc/NEWS also has all the news for all 3.3.0 releases. > Why pick on 3.3.1 rc1? 3.3.1rc1 is the only post-3.3.0 release that's mentioned there (e.g. 3.3.2 isn't), which is what makes me thing a mistake was made along the way (probably when merging). Regards Antoine. From guido at python.org Thu Jul 4 21:36:20 2013 From: guido at python.org (Guido van Rossum) Date: Thu, 4 Jul 2013 12:36:20 -0700 Subject: [Python-Dev] Oddity in MISC/News on default In-Reply-To: <1372965599.2599.2.camel@fsol> References: <20130704210906.74208200@fsol> <1372965599.2599.2.camel@fsol> Message-ID: Maybe the mistake is that the others aren't mentioned? Or perhaps everything before 3.4a1 should be dropped? I forget what kind of policy we have for this -- is it all changes in this branch or only changes unique to this branch? On Thu, Jul 4, 2013 at 12:19 PM, Antoine Pitrou wrote: > Le jeudi 04 juillet 2013 ? 12:17 -0700, Guido van Rossum a ?crit : > > Why not? Presumably those news items were all merged into the default > > branch, and Misc/NEWS also has all the news for all 3.3.0 releases. > > Why pick on 3.3.1 rc1? > > 3.3.1rc1 is the only post-3.3.0 release that's mentioned there (e.g. > 3.3.2 isn't), which is what makes me thing a mistake was made along the > way (probably when merging). > > Regards > > Antoine. > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/guido%40python.org > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From benjamin at python.org Thu Jul 4 21:47:30 2013 From: benjamin at python.org (Benjamin Peterson) Date: Thu, 4 Jul 2013 12:47:30 -0700 Subject: [Python-Dev] Oddity in MISC/News on default In-Reply-To: <20130704210906.74208200@fsol> References: <20130704210906.74208200@fsol> Message-ID: Does annotate show it might have been added accidently in a merge? 2013/7/4 Antoine Pitrou : > > Hello, > > In 3.4's Misc/NEWS, there's a huge section entitled "What's New in > Python 3.3.1 release candidate 1". It seems it shouldn't be there, > should it? > > Regards > > Antoine. > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/benjamin%40python.org -- Regards, Benjamin From tseaver at palladion.com Thu Jul 4 21:47:38 2013 From: tseaver at palladion.com (Tres Seaver) Date: Thu, 04 Jul 2013 15:47:38 -0400 Subject: [Python-Dev] PEP 446: Add new parameters to configure the inherance of files and for non-blocking sockets In-Reply-To: References: Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 07/04/2013 07:03 AM, Victor Stinner wrote: > Title: Add new parameters to configure the inherance of files and for > non-blocking sockets Not commenting on either the form or the substance (pun intended), but the word you want is "inheritance" -- "inherence" is a valid term[1], but would a good deal stranger notion to apply to a file descriptor. ;) [1] https://en.wikipedia.org/wiki/Inherence Platonic'ly, Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with undefined - http://www.enigmail.net/ iEYEARECAAYFAlHV0VoACgkQ+gerLs4ltQ4YCQCgp6mFPxEVVoXAXib/jrChjRxu QkAAoLJQIfBCQezj61LCAgmVaE1kwNmM =yiPj -----END PGP SIGNATURE----- From benjamin at python.org Thu Jul 4 21:50:09 2013 From: benjamin at python.org (Benjamin Peterson) Date: Thu, 4 Jul 2013 12:50:09 -0700 Subject: [Python-Dev] lament for the demise of unbound methods In-Reply-To: References: <51D55ACB.4050900@simplistix.co.uk> Message-ID: 2013/7/4 Eric Snow : > > On Thu, Jul 4, 2013 at 5:21 AM, Chris Withers > wrote: >> >> Hi All, >> >> In Python 2, I can figure out whether I have a method or a function, and, >> more importantly, for an unbound method, I can figure out what class the >> method belongs to: >> >> >>> class MyClass(object): >> ... def method(self): pass >> ... >> >>> MyClass.method >> >> >>> MyClass.method.im_class >> >> >> There doesn't appear to be any way in Python 3 to do this, which is a >> little surprising and frustrating... >> >> What am I missing here? > > > You could always monkeypatch builtins.__build_class__ to add an attribute to > every "unbound method" pointing to the class. I would not reccomend that. __build_class__ is very internal and it's contract may change between versions. -- Regards, Benjamin From solipsis at pitrou.net Thu Jul 4 21:51:27 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 4 Jul 2013 21:51:27 +0200 Subject: [Python-Dev] Oddity in MISC/News on default In-Reply-To: References: <20130704210906.74208200@fsol> Message-ID: <20130704215127.1a92ffb9@fsol> On Thu, 4 Jul 2013 12:47:30 -0700 Benjamin Peterson wrote: > Does annotate show it might have been added accidently in a merge? Annotate shows the original changesets on the 3.3 branch, which doesn't help ;-) Regards Antoine. From chris at simplistix.co.uk Thu Jul 4 21:59:03 2013 From: chris at simplistix.co.uk (Chris Withers) Date: Thu, 04 Jul 2013 20:59:03 +0100 Subject: [Python-Dev] lament for the demise of unbound methods In-Reply-To: References: <51D55ACB.4050900@simplistix.co.uk> Message-ID: <51D5D407.1070609@simplistix.co.uk> On 04/07/2013 20:50, Benjamin Peterson wrote: > 2013/7/4 Eric Snow : >> >> You could always monkeypatch builtins.__build_class__ to add an attribute to >> every "unbound method" pointing to the class. > > I would not reccomend that. __build_class__ is very internal and it's > contract may change between versions. Right, not to mention the fact that there's no way a library can ensure it monkeypatches that before the users of the library have created any classes. cheers, Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk From victor.stinner at gmail.com Thu Jul 4 22:12:46 2013 From: victor.stinner at gmail.com (Victor Stinner) Date: Thu, 4 Jul 2013 22:12:46 +0200 Subject: [Python-Dev] PEP 446: Add new parameters to configure the inherance of files and for non-blocking sockets In-Reply-To: References: Message-ID: 2013/7/4 Tres Seaver : > Not commenting on either the form or the substance (pun intended), but > the word you want is "inheritance" -- "inherence" is a valid term[1], but > would a good deal stranger notion to apply to a file descriptor. ;) Oh... I don't know why I wrote "inherance", it was "inheritance" in the PEP 433. Thanks, I fixed the typo in the PEP 466. Victor From cs at zip.com.au Fri Jul 5 02:41:02 2013 From: cs at zip.com.au (Cameron Simpson) Date: Fri, 5 Jul 2013 10:41:02 +1000 Subject: [Python-Dev] PEP 446: Add new parameters to configure the inherance of files and for non-blocking sockets In-Reply-To: References: Message-ID: <20130705004102.GA19819@cskk.homeip.net> First up I broadly like this. You might want to make clear that the "blocking" parameter refers only to the file creation calls (eg socket.socket) and not to the file descriptor itself, and is not to be confused with the UNIX O_NONBLOCK file descriptor flag (and whatever equivalent flag may apply on Windows). This is deducable from your PEP, but I was at first confused, and initially expected get_blocking/set_blocking functions in New Functions. On 04Jul2013 13:03, Victor Stinner wrote: | Other Changes | ------------- | The ``subprocess.Popen`` class must clear the close-on-exec flag of file | descriptors of the ``pass_fds`` parameter. I would expect Popen and friends to need to both clear the flag to get the descriptors across the fork() call, and _possibly_ to set the flag again after the fork. Naively, I would expect the the flag to be as it was before the Popen call, after the call. This is not addressed. Cheers, -- Cameron Simpson Time is nature's way of keeping everything from happening at once. From cs at zip.com.au Fri Jul 5 07:20:36 2013 From: cs at zip.com.au (Cameron Simpson) Date: Fri, 5 Jul 2013 15:20:36 +1000 Subject: [Python-Dev] PEP 446: Add new parameters to configure the inherance of files and for non-blocking sockets In-Reply-To: <20130705004102.GA19819@cskk.homeip.net> References: <20130705004102.GA19819@cskk.homeip.net> Message-ID: <20130705052036.GA37545@cskk.homeip.net> On 05Jul2013 10:41, I wrote: | On 04Jul2013 13:03, Victor Stinner wrote: | | Other Changes | | ------------- | | The ``subprocess.Popen`` class must clear the close-on-exec flag of file | | descriptors of the ``pass_fds`` parameter. | | I would expect Popen and friends to need to both clear the flag to | get the descriptors across the fork() call, and _possibly_ to set | the flag again after the fork. Naively, I would expect the the flag | to be as it was before the Popen call, after the call. This is harder than I'd thought through. Descriptors go across a fork() anyway, and after an exec() there is nobody to change their state. In order for the parent to restore the prior close-on-exec state (if deemed the right thing to do) it would need to know when the exec was done in the child. You'd need something like an extra descriptor attached to a pipe marked close-on-exec so that the parent could see EOF on the read end after the exec, since the exec would close the write end. tractable, but baroque. So I'd have the PEP take a position on the post-exec() state of the passed file descriptors, and if state is not going to be restored it should outright say that passing an fd clears the close-on-exec flag. Which is it does, but I think the doco should be pretty overt about: - this side effect of clearing close-on-exec on these fds - that the file descriptor and its close-on-exec state is common between the parent and the child - that setting of close-on-exec on the fds in the parent could in principle comprimise the exec in the child if done too soon If you decide state should get restored, the implementation gets a bit fiddlier to detect the exec() in the child. I can't speak for what might be required on non-UNIX. Cheers, -- Cameron Simpson Agree, for Law is costly. -- Very good advice to litigious Persons, founded upon Reason and Experience; for many Times the Charges of a Suit exceed the Value of the Thing in Dispute. - Bailey's dictionary, 1736 From victor.stinner at gmail.com Fri Jul 5 08:24:11 2013 From: victor.stinner at gmail.com (Victor Stinner) Date: Fri, 5 Jul 2013 08:24:11 +0200 Subject: [Python-Dev] PEP 446: Add new parameters to configure the inherance of files and for non-blocking sockets In-Reply-To: <20130705004102.GA19819@cskk.homeip.net> References: <20130705004102.GA19819@cskk.homeip.net> Message-ID: 2013/7/5 Cameron Simpson : > You might want to make clear that the "blocking" parameter refers > only to the file creation calls (eg socket.socket) and not to the > file descriptor itself, and is not to be confused with the UNIX > O_NONBLOCK file descriptor flag (and whatever equivalent flag may > apply on Windows). The two following codes are the same: s = socket.socket(..., blocking=False) and s = socket.socket(...) s.setblocking(False) Both set O_NONBLOCK flag (UNIX) or clear HANDLE_FLAG_INHERIT (Windows) on the socket (which is a file descriptor). socket.socket(..., blocking=False) cannot fail with EAGAIN or EWOULDBLOCK (at least on Linux according to the manual page). > This is deducable from your PEP, but I was at first confused, and > initially expected get_blocking/set_blocking functions in New > Functions. socket objects already have get_blocking() and set_blocking() methods. If we supported the blocking flag on any file descriptor, it would make sense to add such function to the os module. But I explained in the "Add blocking parameter for file descriptors and Windows overlapped I/O" section why I only want to set/clear the blocking file on sockets. > I would expect Popen and friends to need to both clear the flag to > get the descriptors across the fork() call, and _possibly_ to set > the flag again after the fork. Naively, I would expect the the flag > to be as it was before the Popen call, after the call. As explained in the "Inheritance of file descriptors" section, the close-on-exec flag has no effect on fork: "The flag has no effect on fork(), all file descriptors are inherited by the child process." The close-on-exec flag is cleared after the fork and before the exec(). Pseudo-code for Popen: pid = os.fork() if pid: return pid # child process for fd in pass_fds: os.set_cloexec(fd, False) os.execv(...) Victor From tjreedy at udel.edu Fri Jul 5 09:10:39 2013 From: tjreedy at udel.edu (Terry Reedy) Date: Fri, 05 Jul 2013 03:10:39 -0400 Subject: [Python-Dev] Oddity in MISC/News on default In-Reply-To: References: <20130704210906.74208200@fsol> <1372965599.2599.2.camel@fsol> Message-ID: On 7/4/2013 3:36 PM, Guido van Rossum wrote: > Maybe the mistake is that the others aren't mentioned? Or perhaps > everything before 3.4a1 should be dropped? I forget what kind of policy > we have for this -- is it all changes in this branch or only changes > unique to this branch? It cannot be 'unique' because most bug fixes are not unique to any branch. News for 3.x should have all relevant changes to 3.x, and, I think, only those. The question is whether the entries under 3.3.1rc1 in 3.4 News are changes unique to 3.3.1rc1 that should not be in News 3.4; changes to both 3.3 and 3.4 that are now listed twice, and should only be listed once; or changes to 3.4 that are now listed only under 3.3 but should instead be listed as 3.4 changes. I am not looking at the moment because is it 3am here. -- Terry Jan Reedy From fuzzyman at voidspace.org.uk Fri Jul 5 11:23:36 2013 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Fri, 5 Jul 2013 11:23:36 +0200 Subject: [Python-Dev] lament for the demise of unbound methods In-Reply-To: References: <51D55ACB.4050900@simplistix.co.uk> <51D5A5DF.3000103@simplistix.co.uk> Message-ID: <761E2333-74A7-4F50-9943-E8148189507B@voidspace.org.uk> On 4 Jul 2013, at 19:00, Guido van Rossum wrote: > Thanks for the code pointers. So it's all about monkeypatching. :-) I have only a little sympathy, as there still seems to be a way to do this, it's just less convenient. Too bad. > I've also lamented the death of bound methods in Python 3 for mock "autospeccing". Autospec introspects objects and provides mock objects with the same attributes - and with the same method signatures. For methods it needs to trim the first argument (because instances are called externally without self of course). Not being able to tell the difference between a module level function and an unbound method caused some pain then. (I worked round it by flowing the information about where the object came from through the code but it did add ugliness). Michael > --Guido > > On Thu, Jul 4, 2013 at 9:42 AM, Chris Withers wrote: > Hi Guido, > > I've bumped into this a couple of times. > > First time was when I wanted to know whether what I had was a classmethod, staticmethod or normal method here: > > https://github.com/Simplistix/testfixtures/blob/master/testfixtures/replace.py#L59 > > This resulted in having to trawl through __dict__ here: > > https://github.com/Simplistix/testfixtures/blob/master/testfixtures/resolve.py#L17 > > ...rather than just using getattr. > > I bumped into it again, yesterday, trying to add support for classes to this lightweight dependency injection framework I'm developing: > > https://github.com/Simplistix/mush/blob/master/tests/test_runner.py#L189 > > Here's my local copy of that test: > > https://gist.github.com/cjw296/db64279c69cdc0c5e112 > > The workaround I was playing with this morning is a wrapper so that I know I have a class method, although what I really want to write at this line is: > > https://gist.github.com/cjw296/db64279c69cdc0c5e112#file-gistfile1-txt-L40 > > runner = Runner(T0, C1.meth, C2.meth1, C2.meth2) > > ...but if I do that, how can the runner know that what it gets for its second argument is a class method of C1? > (which is this case means that it should do C1().meth() rather than C1.meth()) > > cheers, > > Chris > > > On 04/07/2013 17:25, Guido van Rossum wrote: > Chris, what do you want to do with the knowledge you are seeking? > > --Guido van Rossum (sent from Android phone) > > On Jul 4, 2013 4:28 AM, "Chris Withers" > wrote: > > Hi All, > > In Python 2, I can figure out whether I have a method or a function, > and, more importantly, for an unbound method, I can figure out what > class the method belongs to: > > >>> class MyClass(object): > ... def method(self): pass > ... > >>> MyClass.method > > >>> MyClass.method.im_class > > > There doesn't appear to be any way in Python 3 to do this, which is > a little surprising and frustrating... > > What am I missing here? > > Chris > > -- > Simplistix - Content Management, Batch Processing & Python Consulting > - http://www.simplistix.co.uk > _________________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/__mailman/listinfo/python-dev > > Unsubscribe: > http://mail.python.org/__mailman/options/python-dev/__guido%40python.org > > > > ______________________________________________________________________ > This email has been scanned by the Symantec Email Security.cloud service. > For more information please visit http://www.symanteccloud.com > ______________________________________________________________________ > > -- > Simplistix - Content Management, Batch Processing & Python Consulting > - http://www.simplistix.co.uk > > > > -- > --Guido van Rossum (python.org/~guido) _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk -- http://www.voidspace.org.uk/ May you do good and not evil May you find forgiveness for yourself and forgive others May you share freely, never taking more than you give. -- the sqlite blessing http://www.sqlite.org/different.html From martin at v.loewis.de Fri Jul 5 12:07:15 2013 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Fri, 05 Jul 2013 12:07:15 +0200 Subject: [Python-Dev] lament for the demise of unbound methods In-Reply-To: <761E2333-74A7-4F50-9943-E8148189507B@voidspace.org.uk> References: <51D55ACB.4050900@simplistix.co.uk> <51D5A5DF.3000103@simplistix.co.uk> <761E2333-74A7-4F50-9943-E8148189507B@voidspace.org.uk> Message-ID: <51D69AD3.7050003@v.loewis.de> Am 05.07.13 11:23, schrieb Michael Foord: > I've also lamented the death of bound methods in Python 3 for mock > "autospeccing". Autospec introspects objects and provides mock > objects with the same attributes - and with the same method > signatures. I wonder why you need to figure out the signatures in advance. Can you just wait until the function is actually used, and then process the parameters as you get them? Regards, Martin From lukasz at langa.pl Fri Jul 5 12:26:44 2013 From: lukasz at langa.pl (=?utf-8?Q?=C5=81ukasz_Langa?=) Date: Fri, 5 Jul 2013 12:26:44 +0200 Subject: [Python-Dev] lament for the demise of unbound methods In-Reply-To: <51D69AD3.7050003@v.loewis.de> References: <51D55ACB.4050900@simplistix.co.uk> <51D5A5DF.3000103@simplistix.co.uk> <761E2333-74A7-4F50-9943-E8148189507B@voidspace.org.uk> <51D69AD3.7050003@v.loewis.de> Message-ID: <076A4BF7-C8CC-4325-985F-BB905B174F01@langa.pl> On 5 lip 2013, at 12:07, Martin v. L?wis wrote: > I wonder why you need to figure out the signatures in advance. > Can you just wait until the function is actually used, and then > process the parameters as you get them? > My guess is that Michael's design lets mock objects be introspected as well, i.e. they don't appear as magical as they really are to the user code. -- Best regards, ?ukasz Langa WWW: http://lukasz.langa.pl/ Twitter: @llanga IRC: ambv on #python-dev -------------- next part -------------- An HTML attachment was scrubbed... URL: From cf.natali at gmail.com Fri Jul 5 12:26:48 2013 From: cf.natali at gmail.com (=?ISO-8859-1?Q?Charles=2DFran=E7ois_Natali?=) Date: Fri, 5 Jul 2013 12:26:48 +0200 Subject: [Python-Dev] PEP 446: Add new parameters to configure the inherance of files and for non-blocking sockets In-Reply-To: References: Message-ID: 2013/7/4 Victor Stinner : > Even if the PEP 433 was not explicitly rejected, no consensus could be > reached. I didn't want to loose all my work on this PEP and so I'm > proposing something new which should make everbody agrees :-) Thanks Victor, I think this one is perfectly fine! cf From martin at v.loewis.de Fri Jul 5 12:26:06 2013 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Fri, 05 Jul 2013 12:26:06 +0200 Subject: [Python-Dev] lament for the demise of unbound methods In-Reply-To: <51D5A5DF.3000103@simplistix.co.uk> References: <51D55ACB.4050900@simplistix.co.uk> <51D5A5DF.3000103@simplistix.co.uk> Message-ID: <51D69F3E.8060103@v.loewis.de> Am 04.07.13 18:42, schrieb Chris Withers: > Hi Guido, > > I've bumped into this a couple of times. > > First time was when I wanted to know whether what I had was a > classmethod, staticmethod or normal method here: > > https://github.com/Simplistix/testfixtures/blob/master/testfixtures/replace.py#L59 > > > This resulted in having to trawl through __dict__ here: > > https://github.com/Simplistix/testfixtures/blob/master/testfixtures/resolve.py#L17 You could use __getattribute__ instead: >>> class A: ... @staticmethod ... def s(): ... pass ... @classmethod ... def c(cl): ... pass ... def r(self): ... pass ... >>> A.__getattribute__(A,'s') >>> A.__getattribute__(A,'c') >>> A.__getattribute__(A,'r') Regards, Martin From cs at zip.com.au Fri Jul 5 11:15:59 2013 From: cs at zip.com.au (Cameron Simpson) Date: Fri, 5 Jul 2013 19:15:59 +1000 Subject: [Python-Dev] PEP 446: Add new parameters to configure the inherance of files and for non-blocking sockets In-Reply-To: References: Message-ID: <20130705091559.GA65340@cskk.homeip.net> On 05Jul2013 08:24, Victor Stinner wrote: | 2013/7/5 Cameron Simpson : | > You might want to make clear that the "blocking" parameter refers | > only to the file creation calls (eg socket.socket) and not to the | > file descriptor itself, and is not to be confused with the UNIX | > O_NONBLOCK file descriptor flag (and whatever equivalent flag may | > apply on Windows). | | The two following codes are the same: | | s = socket.socket(..., blocking=False) | and | s = socket.socket(...) | s.setblocking(False) | | Both set O_NONBLOCK flag (UNIX) Oh, how embarassing. | or clear HANDLE_FLAG_INHERIT (Windows) | on the socket (which is a file descriptor). | | socket.socket(..., blocking=False) cannot fail with EAGAIN or | EWOULDBLOCK (at least on Linux according to the manual page). Fair enough. | > This is deducable from your PEP, but I was at first confused, and | > initially expected get_blocking/set_blocking functions in New | > Functions. | | socket objects already have get_blocking() and set_blocking() methods. Ah, ok then. As far as it goes. Shouldn't there be a general purpose os.get_blocking() and os.set_blocking() functions that operate on any file descriptor, paralleling os.get_cloexec() and os.set_cloexec()? They're not socket specific operations in principle, merely commonly used with sockets. | If we supported the blocking flag on any file descriptor, it would | make sense to add such function to the os module. But I explained in | the "Add blocking parameter for file descriptors and Windows | overlapped I/O" section why I only want to set/clear the blocking file | on sockets. But WHY? I think socket file decriptors should be treated little differently from other file descriptors. Certainly a stream socket connection is outstanding like other unseekable file descriptors in many respects. Just "only wanting it on sockets" seems a bit special purpose. If you're going address O_NONBLOCK in this proposal in addition to the close-on-exec flag, it should be general purpose. Otherwise, I think it should be separated out into a separate proposal if you're proposing it just for sockets; make this proposal just close-on-exec and forget the blocking stuff for this specific PEP. | > I would expect Popen and friends to need to both clear the flag to | > get the descriptors across the fork() call, and _possibly_ to set | > the flag again after the fork. Naively, I would expect the the flag | > to be as it was before the Popen call, after the call. | | As explained in the "Inheritance of file descriptors" section, the | close-on-exec flag has no effect on fork: | | "The flag has no effect on fork(), all file descriptors are inherited | by the child process." Agreed, see my second post where I explained I'd misthought it. However, as stated there, I think the side effects should be fairly overtly stated in the docuemntation. | The close-on-exec flag is cleared after the fork and before the | exec(). Pseudo-code for Popen: | | pid = os.fork() | if pid: | return pid | # child process | for fd in pass_fds: os.set_cloexec(fd, False) | os.execv(...) Fine. No state restore is fine with me. I think it should be as clear as possible in the doco. Cheers, -- Cameron Simpson Experience is what you have got after you needed it. From status at bugs.python.org Fri Jul 5 18:07:31 2013 From: status at bugs.python.org (Python tracker) Date: Fri, 5 Jul 2013 18:07:31 +0200 (CEST) Subject: [Python-Dev] Summary of Python tracker Issues Message-ID: <20130705160731.7655E56A3B@psf.upfronthosting.co.za> ACTIVITY SUMMARY (2013-06-28 - 2013-07-05) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 4072 ( +3) closed 26066 (+39) total 30138 (+42) Open issues with patches: 1803 Issues opened (32) ================== #5308: cannot marshal objects with more than 2**31 elements http://bugs.python.org/issue5308 reopened by christian.heimes #13153: IDLE crashes when pasting non-BMP unicode char on Py3 http://bugs.python.org/issue13153 reopened by serhiy.storchaka #18324: set_payload does not handle binary payloads correctly http://bugs.python.org/issue18324 opened by r.david.murray #18325: test_kqueue fails in OpenBSD http://bugs.python.org/issue18325 opened by Federico.Schwindt #18326: Not Clear Docs http://bugs.python.org/issue18326 opened by icedream91 #18327: swapped arguments in compatible_for_assignment()? http://bugs.python.org/issue18327 opened by christian.heimes #18329: for line in socket.makefile() speed degradation http://bugs.python.org/issue18329 opened by mmarkk #18330: Fix idlelib.PyShell.build_subprocess_arglist use of __import_ http://bugs.python.org/issue18330 opened by terry.reedy #18331: Document that runpy.run_path and run_module copy the module gl http://bugs.python.org/issue18331 opened by Drekin #18334: type(name, bases, dict) does not call metaclass' __prepare__ a http://bugs.python.org/issue18334 opened by Nikratio #18335: Add textwrap.dedent, .indent, as str methods. http://bugs.python.org/issue18335 opened by terry.reedy #18336: codecs: Link to readline module (history) instead of fd.readli http://bugs.python.org/issue18336 opened by guettli #18338: python --version should send output to STDOUT http://bugs.python.org/issue18338 opened by jaalto #18340: float related test has problem with Denormal Flush to Zero com http://bugs.python.org/issue18340 opened by V.E.O #18342: Use the repr of a module name for ModuleNotFoundError in ceval http://bugs.python.org/issue18342 opened by brett.cannon #18344: _bufferedreader_read_all() may leak reference to data http://bugs.python.org/issue18344 opened by christian.heimes #18345: logging: file creation options with FileHandler and friends http://bugs.python.org/issue18345 opened by pitrou #18348: Additional code pages for EBCDIC http://bugs.python.org/issue18348 opened by roskakori #18349: argparse usage should preserve () in metavars such as range(20 http://bugs.python.org/issue18349 opened by paul.j3 #18351: Incorrect variable name in importlib._bootstrap._get_sourcefil http://bugs.python.org/issue18351 opened by brett.cannon #18352: collections.Counter with added attributes are not deepcopied p http://bugs.python.org/issue18352 opened by Olivier.Gagnon #18353: PyUnicode_WRITE_CHAR macro definition missing http://bugs.python.org/issue18353 opened by Wolf.Ihlenfeldt #18354: http://www.python.org/doc/ has outdated note http://bugs.python.org/issue18354 opened by dholth #18355: Merge super() guide into documentation http://bugs.python.org/issue18355 opened by Russkel #18356: help(numpy) causes segfault on exit http://bugs.python.org/issue18356 opened by Leeward #18357: add tests for dictview set difference operations http://bugs.python.org/issue18357 opened by frasertweedale #18360: Won't install. Keeps telling me DLL is missing. http://bugs.python.org/issue18360 opened by daleastar #18361: Move dev-in-a-box to os.cpu_count() http://bugs.python.org/issue18361 opened by brett.cannon #18362: Make build_cpython.py from dev-in-a-box work outside of a box http://bugs.python.org/issue18362 opened by brett.cannon #18363: Change use of acronym tag in devinabox index.html to abbr http://bugs.python.org/issue18363 opened by brett.cannon #18364: Remove _not_found hack from importlib http://bugs.python.org/issue18364 opened by brett.cannon #18365: Idle: mock Text class and test thereof http://bugs.python.org/issue18365 opened by terry.reedy Most recent 15 issues with no replies (15) ========================================== #18364: Remove _not_found hack from importlib http://bugs.python.org/issue18364 #18363: Change use of acronym tag in devinabox index.html to abbr http://bugs.python.org/issue18363 #18362: Make build_cpython.py from dev-in-a-box work outside of a box http://bugs.python.org/issue18362 #18361: Move dev-in-a-box to os.cpu_count() http://bugs.python.org/issue18361 #18355: Merge super() guide into documentation http://bugs.python.org/issue18355 #18349: argparse usage should preserve () in metavars such as range(20 http://bugs.python.org/issue18349 #18348: Additional code pages for EBCDIC http://bugs.python.org/issue18348 #18345: logging: file creation options with FileHandler and friends http://bugs.python.org/issue18345 #18342: Use the repr of a module name for ModuleNotFoundError in ceval http://bugs.python.org/issue18342 #18336: codecs: Link to readline module (history) instead of fd.readli http://bugs.python.org/issue18336 #18330: Fix idlelib.PyShell.build_subprocess_arglist use of __import_ http://bugs.python.org/issue18330 #18325: test_kqueue fails in OpenBSD http://bugs.python.org/issue18325 #18324: set_payload does not handle binary payloads correctly http://bugs.python.org/issue18324 #18320: idle, pydoc search python on wrong path at make altinstall wit http://bugs.python.org/issue18320 #18319: gettext() cannot find translations with plural forms http://bugs.python.org/issue18319 Most recent 15 issues waiting for review (15) ============================================= #18357: add tests for dictview set difference operations http://bugs.python.org/issue18357 #18352: collections.Counter with added attributes are not deepcopied p http://bugs.python.org/issue18352 #18351: Incorrect variable name in importlib._bootstrap._get_sourcefil http://bugs.python.org/issue18351 #18349: argparse usage should preserve () in metavars such as range(20 http://bugs.python.org/issue18349 #18344: _bufferedreader_read_all() may leak reference to data http://bugs.python.org/issue18344 #18338: python --version should send output to STDOUT http://bugs.python.org/issue18338 #18334: type(name, bases, dict) does not call metaclass' __prepare__ a http://bugs.python.org/issue18334 #18324: set_payload does not handle binary payloads correctly http://bugs.python.org/issue18324 #18321: Multivolume support in tarfile module http://bugs.python.org/issue18321 #18317: gettext: DoS via crafted Plural-Forms http://bugs.python.org/issue18317 #18309: Make python slightly more relocatable http://bugs.python.org/issue18309 #18308: checkRecvmsgAddress wrong in test_socket.py (AIX failures) http://bugs.python.org/issue18308 #18305: [patch] Fast sum() for non-numbers http://bugs.python.org/issue18305 #18301: In itertools.chain.from_iterable() there is no cls argument http://bugs.python.org/issue18301 #18294: zlib module is not completly 64-bit safe http://bugs.python.org/issue18294 Top 10 most discussed issues (10) ================================= #18329: for line in socket.makefile() speed degradation http://bugs.python.org/issue18329 18 msgs #18305: [patch] Fast sum() for non-numbers http://bugs.python.org/issue18305 7 msgs #18340: float related test has problem with Denormal Flush to Zero com http://bugs.python.org/issue18340 7 msgs #18226: IDLE Unit test for FormatParagrah.py http://bugs.python.org/issue18226 6 msgs #18338: python --version should send output to STDOUT http://bugs.python.org/issue18338 6 msgs #13582: IDLE and pythonw.exe stderr problem http://bugs.python.org/issue13582 5 msgs #18335: Add textwrap.dedent, .indent, as str methods. http://bugs.python.org/issue18335 5 msgs #14455: plistlib unable to read json and binary plist files http://bugs.python.org/issue14455 4 msgs #17206: Py_XDECREF() expands its argument multiple times http://bugs.python.org/issue17206 4 msgs #18334: type(name, bases, dict) does not call metaclass' __prepare__ a http://bugs.python.org/issue18334 4 msgs Issues closed (37) ================== #4199: add shorthand global and nonlocal statements http://bugs.python.org/issue4199 closed by gregory.p.smith #7136: Idle File Menu Option Improvement http://bugs.python.org/issue7136 closed by terry.reedy #11016: Re-implementation of the stat module in C http://bugs.python.org/issue11016 closed by christian.heimes #11185: test_wait4 error on AIX http://bugs.python.org/issue11185 closed by pitrou #12520: spurious output in test_warnings http://bugs.python.org/issue12520 closed by haypo #12716: Reorganize os docs for files/dirs/fds http://bugs.python.org/issue12716 closed by akuchling #14206: multiprocessing.Queue documentation is lacking important detai http://bugs.python.org/issue14206 closed by sbt #17097: multiprocessing BaseManager serve_client() does not check EINT http://bugs.python.org/issue17097 closed by sbt #17261: multiprocessing.manager BaseManager cannot return proxies from http://bugs.python.org/issue17261 closed by sbt #17273: Pool methods can only be used by parent process. http://bugs.python.org/issue17273 closed by sbt #17914: add os.cpu_count() http://bugs.python.org/issue17914 closed by neologix #18081: test_logging failure in WarningsTest on buildbots http://bugs.python.org/issue18081 closed by terry.reedy #18103: Create a GUI test framework for Idle http://bugs.python.org/issue18103 closed by terry.reedy #18155: csv.Sniffer.has_header doesn't escape characters used in regex http://bugs.python.org/issue18155 closed by r.david.murray #18189: IDLE Improvements: Unit test for Delegator.py http://bugs.python.org/issue18189 closed by terry.reedy #18224: pyvenv pydoc.py script causing AttributeErrors on Windows http://bugs.python.org/issue18224 closed by python-dev #18237: unittest.assertRaisesRegex(p) example is wrong in docs http://bugs.python.org/issue18237 closed by terry.reedy #18240: hmac unnecessarily restricts input to "bytes" http://bugs.python.org/issue18240 closed by christian.heimes #18242: IDLE should not be replacing warnings.formatwarning http://bugs.python.org/issue18242 closed by terry.reedy #18244: Adopt C3-based linearization for improved ABC support in funct http://bugs.python.org/issue18244 closed by lukasz.langa #18288: Idle 2.7: Run Module does not set __file__ http://bugs.python.org/issue18288 closed by terry.reedy #18302: test_multiprocessing: test.support.import_module() does not ig http://bugs.python.org/issue18302 closed by brett.cannon #18312: "make distclean" deletes files under .hg directory http://bugs.python.org/issue18312 closed by python-dev #18315: Fix fileinput doc and docstrings (add 'buffer' in 2.7) http://bugs.python.org/issue18315 closed by terry.reedy #18322: test_stat nits http://bugs.python.org/issue18322 closed by pitrou #18328: Use after free in pystate.c http://bugs.python.org/issue18328 closed by christian.heimes #18332: _posix_listdir may leak FD http://bugs.python.org/issue18332 closed by christian.heimes #18333: Memory leak in _pickle.c:Unpickler_set_memo() http://bugs.python.org/issue18333 closed by christian.heimes #18337: codecs: StremReader readline() breaks on undocumented characte http://bugs.python.org/issue18337 closed by serhiy.storchaka #18339: Segfault in Unpickler_set_memo() http://bugs.python.org/issue18339 closed by christian.heimes #18341: enhancements zlib.compress/decompress to accept Py_buffer http://bugs.python.org/issue18341 closed by jcea #18343: faulthandler_register_py() may use _Py_sighandler_t previous w http://bugs.python.org/issue18343 closed by haypo #18346: Backporting of atomic file rename to Python 2.7 http://bugs.python.org/issue18346 closed by r.david.murray #18347: ElementTree corrupts cAse of closing tags when html method is http://bugs.python.org/issue18347 closed by christian.heimes #18350: Python 3 unittest framework broken? http://bugs.python.org/issue18350 closed by amaury.forgeotdarc #18358: update links to Apple Style Guide http://bugs.python.org/issue18358 closed by ronaldoussoren #18359: pickle fail to handle some class with __new__ http://bugs.python.org/issue18359 closed by amaury.forgeotdarc From victor.stinner at gmail.com Fri Jul 5 19:03:35 2013 From: victor.stinner at gmail.com (Victor Stinner) Date: Fri, 5 Jul 2013 19:03:35 +0200 Subject: [Python-Dev] PEP 446: Add new parameters to configure the inherance of files and for non-blocking sockets In-Reply-To: <20130705091559.GA65340@cskk.homeip.net> References: <20130705091559.GA65340@cskk.homeip.net> Message-ID: 2013/7/5 Cameron Simpson : > | Both set O_NONBLOCK flag (UNIX) > > Oh, how embarassing. You said that the PEP is not cristal clear. Do you have a suggestion to make it more clear? Should I mention that the close-on-exec flag is O_CLOEXEC on UNIX, and HANDLE_FLAG_INHERIT on Windows? (except that HANDLE_FLAG_INHERIT set means inheritable, whereas O_CLOEXEC set means *not* inheritable) > | > This is deducable from your PEP, but I was at first confused, and > | > initially expected get_blocking/set_blocking functions in New > | > Functions. > | > | socket objects already have get_blocking() and set_blocking() methods. > > Ah, ok then. As far as it goes. Shouldn't there be a general purpose > os.get_blocking() and os.set_blocking() functions that operate on > any file descriptor, paralleling os.get_cloexec() and os.set_cloexec()? I didn't propose to add these two functions, because I'm not sure that they are really useful. We can add os.get_blocking() and os.set_blocking() on UNIX, but these functions would not be available on Windows. On Windows, os.set_blocking() only makes sense for sockets, and sockets already have a set_blocking() method. On UNIX, it makes sense because set_blocking(fd, True) can be implemented as a single syscall on some platforms (using ioctl), whereas most developers implement it using two syscalls (fcntl to get current flags, and fcntl to set the O_NONBLOCK flag) maybe because it is more portable. But we cannot add a new "blocking" parameter to all functions creating file descriptors, like open(), because of Windows. Or do you like the "raise NotImplementedError on Windows" option? > But WHY? I think socket file decriptors should be treated little > differently from other file descriptors. Because of Windows. On Windows, sockets and files are two different things. sockets have no "file descriptor", they have a HANDLE. For example, sockobj.fileno() on Windows returns a huge number, not something like 3 or 10. The C type HANDLE is larger than a file descriptor on Windows 64-bit (sizeof(void*) > sizeof(int)) and so functions must be modified to use a wider type (to parse their argument), and must support the HANDLE type (_open_osfhandle() can be used to have a single version of the code). > Otherwise, I think it should be separated out into a separate > proposal if you're proposing it just for sockets; make this proposal > just close-on-exec and forget the blocking stuff for this specific > PEP. The reason for addressing close-on-exec and blocking parameters in the same PEP is that new versions of operating systems support setting the two flags at the creation a new file descriptor and a new socket. Read the article linked at the end of the PEP for the background: http://udrepper.livejournal.com/20407.html The Python call socket.socket(..., cloexec=True, blocking=False) only calls one syscall on Linux >= 2.6.27. > | > I would expect Popen and friends to need to both clear the flag to > | > get the descriptors across the fork() call, and _possibly_ to set > | > the flag again after the fork. Naively, I would expect the the flag > | > to be as it was before the Popen call, after the call. > | > | As explained in the "Inheritance of file descriptors" section, the > | close-on-exec flag has no effect on fork: > | > | "The flag has no effect on fork(), all file descriptors are inherited > | by the child process." > > Agreed, see my second post where I explained I'd misthought it. > However, as stated there, I think the side effects should be fairly > overtly stated in the docuemntation. Sorry, I don't understand. Which "side effect"? The close-on-exec flag only affects inheritance of file descriptors at the execv() syscall, not at fork(). And execv() can be called without fork(). Popen must clear close-on-exec flag on files from its pass_fds parameter for convinience. It would be surprising to not inherit a file descriptor passed to Popen in pass_fds, don't you think so? os.execv() has no pass_fds parameter, and it is a thin wrapper on the syscall. Popen is a high-level API, that's why it prepares more things before calling the new program. > | The close-on-exec flag is cleared after the fork and before the > | exec(). Pseudo-code for Popen: > | > | pid = os.fork() > | if pid: > | return pid > | # child process > | for fd in pass_fds: os.set_cloexec(fd, False) > | os.execv(...) > > Fine. No state restore is fine with me. I think it should be as > clear as possible in the doco. I don't understand. I already wrote "The flag has no effect on fork(), all file descriptors are inherited by the child process" in the PEP. It is not enough? Do you have a suggestion to explain it better? Victor From benoit at marmelune.net Fri Jul 5 09:38:36 2013 From: benoit at marmelune.net (=?UTF-8?B?QmVub8OudCBCcnlvbg==?=) Date: Fri, 05 Jul 2013 09:38:36 +0200 Subject: [Python-Dev] PEP 423 : naming conventions and recipes related to packaging In-Reply-To: <20120627125055.196efe9f@pitrou.net> References: <4FEACD9D.8090208@marmelune.net> <20120627125055.196efe9f@pitrou.net> Message-ID: <51D677FC.8050107@marmelune.net> Hi! Attached is a an updated proposal for PEP 423. You can also find it online at https://gist.github.com/benoitbryon/2815051 I am attending at EuroPython 2013 in Florence. Isn't it a great opportunity to get feedback and discuss about a PEP? I registered an open-space session and a lightning-talk today! Some notes about the update... The main point that was discussed in the previous proposal was the "top-level namespace relates to code ownership rule". Here is a quote from Antoine Pitrou: Le 27/06/2012 12:50, Antoine Pitrou a ?crit : > On Wed, 27 Jun 2012 11:08:45 +0200 > Beno?t Bryon wrote: >> Hi, >> >> Here is an informational PEP proposal: >> http://hg.python.org/peps/file/52767ab7e140/pep-0423.txt >> >> Could you review it for style, consistency and content? > There is one Zen principle this PEP is missing: > > Flat is better than nested. > > This PEP seems to promote the practice of having a top-level namespace > denote ownership. I think it should do the reverse: promote > meaningful top-level packages (e.g. "sphinx") as standard practice, and > allow an exception for when a piece of software is part of a larger > organizational body. So, the main change in the proposal I'm sending today is the removal of this "ownership" rule. It has been replaced by "Use a single namespace (except special cases)". Some additional changes have been performed, such as removal of some sections about "opportunity" or "promote migrations". I also added a "Rationale" section where I pointed out some issues related to naming. The PEP has been marked as "deferred" because it was inactive and it is partly related to PEP 426. I left this deferred state. I am aware that some links in the PEP are broken... I will fix them later. My very first motivation is to get feedback about the "big" changes in the PEP. I wanted the update to be sent before EuroPython-2013's open-space session. I guess a detailed review would be nice anyway, for links, style, grammar... Also, I wonder whether the PEP could be shortened or not. Sometimes I cannot find straightforward words to explain things, so perhaps someone with better skills in english language could help. Or maybe some parts, such as the "How to rename a project" section, could be moved in other documents. Regards, Beno?t -------------- next part -------------- PEP: 423 Title: Naming conventions and recipes related to packaging Version: $Revision$ Last-Modified: $Date$ Author: Beno?t Bryon Discussions-To: Status: Deferred Type: Informational Content-Type: text/x-rst Created: 24-May-2012 Post-History: 5-Jul-2013 Abstract ======== This document deals with: * names of Python projects, * names of Python packages or modules being distributed, * namespace packages. It provides guidelines and recipes for distribution authors: * new projects should follow the `guidelines <#overview>`_ below. * existing projects should be aware of these guidelines and can follow `specific recipes for existing projects <#how-to-apply-naming-guidelines-on-existing-projects>`_. PEP Deferral ============ Further consideration of this PEP has been deferred at least until after PEP 426 (package metadata 2.0) and related updates have been resolved. Rationale: issues related to names ================================== For a long time, there have been no official reference on the "how to choose names" topic in the Python community. As a consequence, the Python package index (`PyPI`_) contains many naming patterns. The fact is that this heterogeneity causes some issues. Some of them are described below. .. note:: Examples were taken on July 2013. Naming things is a hard task, and naming Python projects or packages is not an exception. The purpose of this PEP is to help project authors to avoid common traps about naming, and focus on valuable things. Clashes ------- Projects names are unique on `PyPI`_. But names of distributed things (packages, modules) are not. And there are clashes. As an example, "pysendfile" and "django-sendfile" projects both distribute a "sendfile" package. Users cannot use both in an environment. Deep nested hierarchies ----------------------- Deep nested namespaces mean deep nested hierarchies. It obfuscates valuable project contents. As an example, with "plone.app.content" you get a deeply nested directory hierarchy: .. code:: text plone/ ??? app/ ?? ??? command/ ??? ... valuable code is here... Whereas, with flat packages like "sphinx", you have valuable code near the top-level directory: .. code:: text sphinx/ ??? ... valuable code is here... Unrelated namespaces -------------------- When project names are made of nested namespaces, and these namespaces are not strongly related, then there is confusion. As an example, it is not obvious that "zc.rst2" project is a general Python project, not tied to "zc" (Zope Corporation), but related to docutils' reStructuredText. As a consequence, some users discard "zc.rst2" project, because they think it is specific to "zc". This issue occurs with branded namespaces, i.e. when top-level namespace relates to an organization and various projects are put into this namespace. This issue also occurs when namespaces are used for categorization. Inconsistent names ------------------ When project and distributed packages do not share a single name, users have to think when they use names (install project or import package). As an example, which package does "django-pipeline" project distributes? Is it "djangopipeline", "django_pipeline" or "pipeline"? The answer is not obvious and the pattern varies depending on project. Users have to remember or search for the name to use. There is no obvious bijection between project name and package/module name. It means that a user have to remember at least two names (project and package) whereas one could be enough. This fact is a cause of clashes, because when you register a project name on `PyPI`_, you have no idea about package name clashes, because package names are not predictable. Terminology =========== Reference of terminology used in this PEP is `packaging terminology in Python documentation`_. Relationship with other PEPs ============================ * `PEP 8`_ deals with code style guide, including names of Python packages and modules. It covers syntax of package/modules names. * `PEP 426`_ deals with packaging metadata, and defines name argument of the ``packaging.core.setup()`` function. * `PEP 420`_ deals with namespace packages. It brings support of namespace packages to Python core. Before, namespaces packages were implemented by external libraries. * `PEP 3108`_ deals with transition between Python 2.x and Python 3.x applied to standard library: some modules to be deleted, some to be renamed. It points out that naming conventions matter and is an example of transition plan. Overview ======== Here is a summarized list of recommendations for you to choose names: * `understand and respect namespace ownership <#understand-and-respect-namespace-ownership>`_. * if your project is related to another project or community, first search for conventions in main project's documentation, then: * `follow specific project or related community conventions <#follow-community-or-related-project-conventions-if-any>`_, if any. * else (there is no specific convention), `follow a standard naming pattern <#use-standard-pattern-for-community-contributions>`_. * make sure names are unique, i.e. avoid duplicates: * `check for name availability`_, * `register names with PyPI`_. Exception is when you explicitely want to distribute alternatives to existing packages or modules. * `Use a single name`_. It implies `a project distributes a single package or module <#multiple-packages-modules-should-be-rare>`_ * `distribute only one package or module at a time <#multiple-packages-modules-should-be-rare>`_, unless you are in a special case. * make it easy to discover and remember your project: * `pick memorable names`_, * `pick meaningful names`_, * `use packaging metadata`_. * `avoid deep nesting`_: * one single level is the recommended way to go. * two levels can be used to point out strict relationships: the second level is specific to the first one. Main use cases are community contributions related to one project, and vendor specific projects. * you should not need more than two levels. Having more than three levels is strongly discouraged. * `follow PEP 8`_ for syntax of package and module names. * if, for some reason, your project does not follow the recommendations above, `document specific naming policy`_. In particular, projects which are receiving community contributions should `organize community contributions`_. * `if still in doubt, ask <#if-in-doubt-ask>`_. If in doubt, ask ================ If you feel unsure after reading this document, ask `Python community`_ on IRC or on a mailing list. Understand and respect namespace ownership ========================================== On `PyPI`_, all projects are put at index root. There is no owner or user level. PyPI cannot host two projects with the exact same name, even if owners are different. One name relies to one project, which relies to one ownership. .. note:: A project's ownership can be hold by several users. The top-level namespace relates to ownership. As an example, `Django`_ is owned and maintained by the Django Software Fundation. Understand the purpose of namespace before you use it. Do not plug into a namespace you do not own, unless explicitely authorized. `If in doubt, ask`_. As an example, do not plug in "django.contrib" namespace because it is managed by Django's core contributors. Project owners may define exceptions. See `Organize community contributions`_ below. As an example, `flask`_ project explicitely invites contributors to release projects in "flask.ext" namespace. Also, whenever possible, try to consider non-Python projects. As an example, you should not use "apache" as top-level namespace: "Apache" is the name of another (non Python) project. This is more an advice than a strict rule, but it could help identify your project on the internet or prevent some trademark issues. Private projects may use a namespace ------------------------------------ For internal/customer projects, feel free to use the company or main project name as the namespace. But keep in mind that, if a project is general purpose (i.e. not specific to your company or to some main project), then one level should be enough. This rule applies to closed-source projects. As an example, if you are creating a "climbing" project that is specific to the "Python Sport" company: you may use "pythonsport.climbing" name, even if it is closed source. Use a single name ================= Distribute only one package (or only one module) per project, and use package (or module) name as project name. * It avoids possible confusion between project name and distributed package or module name. * It makes the name consistent. * It is explicit: when one sees project name, he guesses package/module name, and vice versa. * It also limits implicit clashes between package/module names. By using a single name, when you register a project name to `PyPI`_, you also perform a basic package/module name availability verification. As an example, `pipeline`_, `python-pipeline`_ and `django-pipeline`_ all distribute a package or module called "pipeline". So installing two of them leads to errors. This issue wouldn't have occurred if these distributions used a single name. * As a bonus, it allows easier setup of the project: you provide the name of the package/module you want to distribute, and you get other names. Example: * Yes: Package name is "kheops" and project name is "kheops". * Yes: Package name is "kheops.history", i.e. ``import kheops.history`` and project name is "kheops.history", i.e. ``pip install kheops.history``. * No: Package name is "kheops" and project name is "KheopsPyramid". .. note:: For historical reasons, `PyPI`_ contains many distributions where project and distributed package/module names differ. Multiple packages/modules should be rare ---------------------------------------- Technically, Python distributions can provide multiple packages and/or modules. See `setup script reference`_ for details. Some distributions actually do. As an example, `setuptools`_ and `distribute`_ are both declaring "pkg_resources", "easy_install" and "site" modules in addition to respective "setuptools" and "distribute" packages. Consider this use case as exceptional. In most cases, you do not need this feature and distributing a single package or module is enough. Distinct names should be rare ----------------------------- A notable exception to the `Use a single name`_ rule is when you explicitely need distinct names. As an example, the `Pillow`_ project provides an alternative to the original `PIL`_ distribution. Both projects distribute a "PIL" package. Consider this use case as exceptional. In most cases, you don't need this feature and naming the project after the distributed package/module is enough. Follow PEP 8 ============ `PEP 8`_ applies to names of Python packages and modules. If you `Use a single name`_, `PEP 8`_ also applies to project names. The exceptions are namespace packages, where dots are required in project name. Pick memorable names ==================== One important thing about a project name is that it be memorable. As an example, `celery`_ is not a meaningful name. At first, it is not obvious that it deals with message queuing. But it is memorable because it can be used to feed a `RabbitMQ`_ server. Pick meaningful names ===================== Ask yourself "how would I describe in one sentence what this name is for?", and then "could anyone have guessed that by looking at the name?". As an example, `DateUtils`_ is a meaningful name. It is obvious that it deals with utilities for dates. When you are using namespaces, try to make each part meaningful. .. note:: Sometimes, you cannot find a name that is both memorable and meaningful. In such a situation, consider "memorable" feature is more important than "meaningful" one. Generally, by choosing a memorable name, you tend to make the name unique, and simplicity is a key of success. Whereas a meaningful name tend to be similar to names of other projects that deal with same concepts, and you tend make the name out of keywords/buzzwords. Use packaging metadata ====================== Consider that **project names are unique identifiers on PyPI**, i.e. their primary purpose is to identify, not to classify or describe. **Classifiers and keywords metadata are made for categorization.** Summary and description metadata are meant to describe the project. As an example, there is a "`Framework :: Twisted`_" classifier. Names of projects that have this classifier are quite heterogeneous. They do not follow a particular pattern to claim relation with Twisted. But we get the list using the classifier and that is fine. In order to `Organize community contributions`_, conventions about names and namespaces matter, but conventions about metadata are important too. As an example, we can find Plone portlets in many places: * plone.portlet.* * collective.portlet.* * collective.portlets.* * collective.*.portlets * some vendor-related projects such as "quintagroup.portlet.cumulus" * and even projects where "portlet" pattern doesn't appear in the name. Even if Plone community has conventions, using the name to categorize distributions is inapropriate. It's impossible to get the full list of distributions that provide portlets for Plone by filtering on names. But it would be possible if all these distributions used "Framework :: Plone" classifier and "portlet" keyword. When you release a project on `PyPI`_, you obviously want your project to be visible, and findable. Keep in mind that the name is not the only mean to make a project discoverable. If you do care about your project's visibility, take care of package metadata: including keywords, classifiers, README... And you may also take care of project's documentation and some stuff not related to packaging. Avoid deep nesting ================== `The Zen of Python`_ says: Flat is better than nested. A single level is recommended ----------------------------- In most cases, one level is enough. So, unless you are in a special situation mentioned below, your project name should be made of a single namespace. Lower levels indicate strict relationship to upper levels --------------------------------------------------------- In nested namespaces, lower levels point out strict relationship to higher ones. It means the second level is specific to the first one. Main use cases are community contributions related to one project and vendor specific (mostly private) projects. Two levels is almost always enough ---------------------------------- Don't define everything in deeply nested hierarchies: you will end up with projects and packages like "pythonsport.common.maps.forest". This type of name is both verbose and cumbersome (e.g. if you have many imports from the package). Furthermore, big hierarchies tend to break down over time as the boundaries between different packages blur. The consensus is that two levels of nesting are preferred. For example, we have ``plone.principalsource`` instead of ``plone.source.principal`` or something like that. The name is shorter, the package structure is simpler, and there would be very little to gain from having three levels of nesting here. It would be impractical to try to put all "core Plone" sources (a source is kind of vocabulary) into the ``plone.source.*`` namespace, in part because some sources are part of other packages, and in part because sources already exist in other places. Had we made a new namespace, it would be inconsistently used from the start. Yes: "pythonsport.climbing" Yes: "pythonsport.forestmap" No: "pythonsport.maps.forest" Do not use namespace levels for categorization ---------------------------------------------- `Use packaging metadata`_ instead. Don't use more than 3 levels ---------------------------- Technically, you have the ability to create deeply nested hierarchies. However, it is strongly discouraged. Document specific naming policy =============================== A project that does not follow this PEP's recommendations should mention and explain it in documentation. This rule is the simplest way to make an existing project comply with this PEP, without a rename. Follow community or related project conventions, if any ======================================================= Projects or related communities can have specific naming conventions, which may differ from those explained in this document. Specific conventions override this PEP. This rule exists for backward-compatibility purpose: new projects should follow this PEP's conventions. In such a case, `they should declare specific conventions in documentation <#organize-community-contributions>`_. So, if your project belongs to another project or to a community, first look for specific conventions in main project's documentation. If there is no specific conventions, follow the ones declared in this document. As an example, `Plone community`_ releases community contributions in the "collective" namespace package. It differs from the `standard namespace for contributions <#use-standard-pattern-for-community-contributions>`_ proposed here. But since it is documented, there is no ambiguity and you should follow this specific convention. Use standard pattern for community contributions ================================================ When no specific rule is defined, use the ``{MAINPROJECT}contrib.{PROJECT}`` pattern to store community contributions for any product or framework, where: * ``{MAINPROJECT}`` is the name of the related project. "pyranha" in the example below. * ``{PROJECT}`` is the name of your project. "giantteeth" in the example below. As an example: * you are the author of "pyranha" project. * you didn't defined specific naming conventions for community contributions. * a third-party developer wants to publish a "giantteeth" project related to your "pyranha" project in a community namespace. So he should publish it as "pyranhacontrib.giantteeth". It is the simplest way to `Organize community contributions`_. .. note:: Why ``{MAINPROJECT}contrib.*`` pattern? * ``{MAINPROJECT}c.*`` is not explicit enough. As examples, "zc" belongs to "Zope Corporation" whereas "z3c" belongs to "Zope 3 community". * ``{MAINPROJECT}community`` is too long. * ``{MAINPROJECT}community`` conflicts with existing namespaces such as "iccommunity" or "PyCommunity". * ``{MAINPROJECT}.contrib.*`` is inside {MAINPROJECT} namespace, i.e. it is owned by ${MAINPROJECT} authors. * ``{MAINPROJECT}.contrib.*`` breaks the `Avoid deep nesting`_ rule. * names where ``{MAINPROJECT}`` does not appear are not explicit enough, i.e. nobody can guess they are related to ``{MAINPROJECT}``. As an example, it is not obvious that "collective.*" belongs to Plone community. * ``{DIST}contrib.*`` looks like existing ``sphinxcontrib-*`` packages. But ``sphinxcontrib-*`` is actually about Sphinx contrib, so this is not a real conflict... In fact, the "contrib" suffix was inspired by "sphinxcontrib". Organize community contributions ================================ This is the counterpart of the `follow community conventions <#follow-community-or-related-project-conventions-if-any>`_ and `standard pattern for contributions <#use-standard-pattern-for-community-contributions>`_ rules. Actions: * Choose a naming convention for community contributions. * If it is not `the default <#use-standard-pattern-for-community-contributions>`_, then document it. * if you use the `default convention <#use-standard-pattern-for-community-contributions>`_, then this document should be enough. Don't reapeat it. You may reference it. * else, tell users about custom conventions in project's "contribute" or "create modules" documentation. * Also recommend the use of additional metadata, such as `classifiers and keywords <#use-packaging-metadata>`_. Example: "pyranha" is your project name and package name. Tell contributors that: * pyranha-related distributions should use the "pyranha" keyword * pyranha-related distributions providing templates should also use "templates" keyword. * community contributions should be released under "pyranhacontrib" namespace (i.e. use "pyranhacontrib.*" pattern). Register names with PyPI ======================== `PyPI`_ is the central place for distributions in Python community. So, it is also the place where to register project and package names. See `Registering with the Package Index`_ for details. Check for name availability =========================== Make sure it project name has not already been registered on `PyPI`_. .. note:: `PyPI`_ is the only official place where to register names. Also make sure the names of distributed packages or modules have not already been registered: * in the `Python Standard Library`_. * inside projects at `PyPI`_. .. note:: The `use a single name`_ rule helps you avoid clashes with package names: if a project name is available, then the package name has good chances to be available too. How to rename a project? ======================== Renaming a project is possible, but it should be done with care. Pay particular attention to README and documentation, so that users understand what happened. #. First of all, **do not remove legacy distributions from PyPI**. Because some users may be using them. #. Copy the legacy project, then change names (project and package/module). Pay attention to, at least: * packaging files, * folder name that contains source files, * documentation, including README, * import statements in code. #. Assign ``obsoleted_by`` metadata to new distribution in setup.cfg file. See `PEP 426 about obsoleted_by`_ and `setup.cfg specification`_. #. Release a new version of the renamed project, then publish it. #. Edit legacy project: * add dependency to new project, * drop everything except packaging stuff, * add the ``Development Status :: 7 - Inactive`` classifier in setup script, * publish a new release. So, users of the legacy package: * can continue using the legacy distributions at a deprecated version, * can upgrade to last version of legacy distribution, which is empty... * ... and automatically download new distribution as a dependency of the legacy one. Users who discover the legacy project see it is inactive. How to apply naming guidelines on existing projects? ==================================================== **There is no obligation for existing projects to be renamed**. The choice is left to project authors and maintainers for obvious reasons. However, project authors are invited to: * at least, state about current naming, i.e., if necessary * explain the reason why the project has this name in documentation, * `document specific naming policy`_. * optionally, `rename existing project or distributed packages/modules <#how-to-rename-a-project>`_. Projects that are meant to receive contributions from community should also `organize community contributions`_. References ========== Additional background: * `Martin Aspeli's article about names`_. Some parts of this document are quotes from this article. * `in development official packaging documentation`_. * `The Hitchhiker's Guide to Packaging`_, which has an empty placeholder for "naming specification". References and footnotes: .. _`packaging terminology in Python documentation`: http://docs.python.org/dev/packaging/introduction.html#general-python-terminology .. _`PEP 8`: http://www.python.org/dev/peps/pep-0008/#package-and-module-names .. _`PEP 426`: http://www.python.org/dev/peps/pep-0426/ .. _`PEP 420`: http://www.python.org/dev/peps/pep-0420/ .. _`PEP 3108`: http://www.python.org/dev/peps/pep-3108/ .. _`Python community`: http://www.python.org/community/ .. _`gp.fileupload`: http://pypi.python.org/pypi/gp.fileupload/ .. _`zest.releaser`: http://pypi.python.org/pypi/zest.releaser/ .. _`django`: http://djangoproject.com/ .. _`flask`: http://flask.pocoo.org/ .. _`sphinx`: http://sphinx.pocoo.org .. _`pypi`: http://pypi.python.org .. _`collective.recaptcha`: http://pypi.python.org/pypi/collective.recaptcha/ .. _`pipeline`: http://pypi.python.org/pypi/pipeline/ .. _`python-pipeline`: http://pypi.python.org/pypi/python-pipeline/ .. _`django-pipeline`: http://pypi.python.org/pypi/django-pipeline/ .. _`setup script reference`: http://docs.python.org/dev/packaging/setupscript.html .. _`setuptools`: http://pypi.python.org/pypi/setuptools .. _`distribute`: http://packages.python.org/distribute/ .. _`Pillow`: http://pypi.python.org/pypi/Pillow/ .. _`PIL`: http://pypi.python.org/pypi/PIL/ .. _`celery`: http://pypi.python.org/pypi/celery/ .. _`RabbitMQ`: http://www.rabbitmq.com .. _`DateUtils`: http://pypi.python.org/pypi/DateUtils/ .. _`Framework :: Twisted`: http://pypi.python.org/pypi?:action=browse&show=all&c=525 .. _`The Zen of Python`: http://www.python.org/dev/peps/pep-0020/ .. _`Plone community`: http://plone.org/community/develop .. _`Registering with the Package Index`: http://docs.python.org/dev/packaging/packageindex.html .. _`Python Standard Library`: http://docs.python.org/library/index.html .. _`PEP 426 about obsoleted_by`: http://www.python.org/dev/peps/pep-0426/#obsoleted-by .. _`setup.cfg specification`: http://docs.python.org/dev/packaging/setupcfg.html .. _`Martin Aspeli's article about names`: http://www.martinaspeli.net/articles/the-naming-of-things-package-names-and-namespaces .. _`in development official packaging documentation`: http://docs.python.org/dev/packaging/ .. _`The Hitchhiker's Guide to Packaging`: http://guide.python-distribute.org/specification.html#naming-specification Copyright ========= This document has been placed in the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End: From markus at unterwaditzer.net Fri Jul 5 21:31:46 2013 From: markus at unterwaditzer.net (Markus Unterwaditzer) Date: Fri, 05 Jul 2013 21:31:46 +0200 Subject: [Python-Dev] PEP 423 : naming conventions and recipes related to packaging In-Reply-To: <51D677FC.8050107@marmelune.net> References: <4FEACD9D.8090208@marmelune.net> <20120627125055.196efe9f@pitrou.net> <51D677FC.8050107@marmelune.net> Message-ID: <0a300e6e-ca3a-44e5-a029-162c37faa44f@email.android.com> In your first plone example you first use plone.app.content, but then present the directory structure of plone.app.command. Apart from that, the PEP seems legit to me, contentwise. I think some parts are clumsily formulated, but IMO rewriting these parts wouldn't even decrease the text's length or improve readability. -- Markus (from phone) "Beno?t Bryon" wrote: >Hi! > >Attached is a an updated proposal for PEP 423. >You can also find it online at >https://gist.github.com/benoitbryon/2815051 > >I am attending at EuroPython 2013 in Florence. Isn't it a great >opportunity to get feedback and discuss about a PEP? I registered an >open-space session and a lightning-talk today! > >Some notes about the update... > >The main point that was discussed in the previous proposal was the >"top-level namespace relates to code ownership rule". Here is a quote >from Antoine Pitrou: > >Le 27/06/2012 12:50, Antoine Pitrou a ?crit : >> On Wed, 27 Jun 2012 11:08:45 +0200 >> Beno?t Bryon wrote: >>> Hi, >>> >>> Here is an informational PEP proposal: >>> http://hg.python.org/peps/file/52767ab7e140/pep-0423.txt >>> >>> Could you review it for style, consistency and content? >> There is one Zen principle this PEP is missing: >> >> Flat is better than nested. >> >> This PEP seems to promote the practice of having a top-level >namespace >> denote ownership. I think it should do the reverse: promote >> meaningful top-level packages (e.g. "sphinx") as standard practice, >and >> allow an exception for when a piece of software is part of a larger >> organizational body. > >So, the main change in the proposal I'm sending today is the removal of > >this "ownership" rule. >It has been replaced by "Use a single namespace (except special >cases)". > >Some additional changes have been performed, such as removal of some >sections about "opportunity" or "promote migrations". I also added a >"Rationale" section where I pointed out some issues related to naming. > >The PEP has been marked as "deferred" because it was inactive and it is > >partly related to PEP 426. I left this deferred state. > >I am aware that some links in the PEP are broken... I will fix them >later. My very first motivation is to get feedback about the "big" >changes in the PEP. I wanted the update to be sent before >EuroPython-2013's open-space session. I guess a detailed review would >be >nice anyway, for links, style, grammar... > >Also, I wonder whether the PEP could be shortened or not. Sometimes I >cannot find straightforward words to explain things, so perhaps someone > >with better skills in english language could help. Or maybe some parts, > >such as the "How to rename a project" section, could be moved in other >documents. > >Regards, > >Beno?t > > >------------------------------------------------------------------------ > >_______________________________________________ >Python-Dev mailing list >Python-Dev at python.org >http://mail.python.org/mailman/listinfo/python-dev >Unsubscribe: >http://mail.python.org/mailman/options/python-dev/markus%40unterwaditzer.net From cs at zip.com.au Sat Jul 6 02:43:52 2013 From: cs at zip.com.au (Cameron Simpson) Date: Sat, 6 Jul 2013 10:43:52 +1000 Subject: [Python-Dev] PEP 446: Add new parameters to configure the inherance of files and for non-blocking sockets In-Reply-To: References: Message-ID: <20130706004352.GA66371@cskk.homeip.net> On 05Jul2013 19:03, Victor Stinner wrote: | 2013/7/5 Cameron Simpson : | > | Both set O_NONBLOCK flag (UNIX) | > | > Oh, how embarassing. | | You said that the PEP is not cristal clear. Do you have a suggestion | to make it more clear? | | Should I mention that the close-on-exec flag is O_CLOEXEC on UNIX, and | HANDLE_FLAG_INHERIT on Windows? (except that HANDLE_FLAG_INHERIT set | means inheritable, whereas O_CLOEXEC set means *not* inheritable) It wouldn't hurt. Otherwise, colour me mostly convinced. I've read your "Rejected Alternatives" more closely and Ulrich Drepper's article, though I think the article also supports adding a blocking (default True) parameter to open() and os.open(). If you try to change that default on a platform where it doesn't work, an exception should be raised. | The close-on-exec flag only affects inheritance of file descriptors at | the execv() syscall, not at fork(). And execv() can be called without | fork(). Yes. Please forget I mentioned fork(); it is only relevant if you were offering some facility to undo the addition of cloexec to a Popen passed file descriptor. Which you are not. | Popen must clear close-on-exec flag on files from its pass_fds | parameter for convinience. It would be surprising to not inherit a | file descriptor passed to Popen in pass_fds, don't you think so? Yes yes yes. | I don't understand. I already wrote "The flag has no effect on fork(), | all file descriptors are inherited by the child process" in the PEP. | It is not enough? Do you have a suggestion to explain it better? My concerns are probably as well bundled with concerns about users not realising the a file descriptor state after a fork is shared, and then being confused if the other process did a seek() etc. I think I'm just concerned about a naive caller using this scenario: # gather up some already open fds to pass fds = (5,6,7) P = Popen(...., pass_fds=fds) # now restore the cloexec state from before for fd in fds: os.set_cloexec(fd) where the cleanup loop in the parent gets to run before the exec() in the child. We could just consider it one of the pitfalls of fork/exec situations in general an expect people being this fiddly to need to add some synchornisation. Please consider your argument won. Cheers, -- Cameron Simpson From ronaldoussoren at mac.com Sat Jul 6 09:45:53 2013 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Sat, 6 Jul 2013 09:45:53 +0200 Subject: [Python-Dev] Hooking into super() attribute resolution In-Reply-To: References: Message-ID: <46EC256E-9ACE-4C4C-BE3D-325D7DB87B85@mac.com> I've updated the implementation in issue 18181 while adding some tests, and have updated the proposal as well. The proposal has some open issues at the moment, most important of which is the actual signature for the new special method; in particular I haven't been able to decide if this should be an instance-, class- or static method. It is a static method in the proposal and prototype, but I'm not convinced that that is the right solution. Ronald PEP: TODO Title: Hooking into super attribute resolution Version: $Revision$ Last-Modified: $Date$ Author: Ronald Oussoren Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 12-Jun-2013 Post-History: 2-Jul-2013, ? Abstract ======== In current python releases the attribute resolution of the `super class`_ peeks in the ``__dict__`` attribute of classes on the MRO to look for attributes. This PEP introduces a hook that classes can use to override that behavior for specific classes. Rationale ========= Peeking in the class ``__dict__`` works for regular classes, but can cause problems when a class dynamicly looks up attributes in a ``__getattribute__`` method. The new hook makes it possible to introduce the same customization for attribute lookup through the `super class`_. The superclass attribute lookup hook ==================================== In C code --------- A new slot ``tp_getattro_super`` is added to the ``PyTypeObject`` struct. The ``tp_getattro`` slot for super will call this slot when it is not ``NULL``, and will raise an exception when it is not set (which shouldn't happen because the method is implemented for :class:`object`). The slot has the following prototype:: PyObject* (*getattrosuperfunc)(PyTypeObject* cls, PyObject* name, PyObject* object, PyObject* owner); The function should perform attribute lookup on *object* for *name*, but only looking in type *tp* (which will be one of the types on the MRO for *self*) and without looking in the instance *__dict__*. The function returns ``NULL`` when the attribute cannot be found, and raises and exception. Exception other than ``AttributeError`` will cause failure of super's attribute resolution. The implementation of the slot for the :class:`object` type is ``PyObject_GenericGetAttrSuper``, which peeks in the ``tp_dict`` for *cls*. Note that *owner* and *object* will be the same object when using a class-mode super. In Python code -------------- A Python class can contain a definition for a static method ``__getattribute_super__`` with the following prototype:: def __getattribute_super__(cls, name, object, owner): pass The method should perform attribute lookup for *name* on instance *self* while only looking at *cls* (it should not look in super classes or the instance *__dict__* XXX: I haven't got a clue at the moment if the method should be an instance-, class- or staticmethod. The prototype uses a staticmethod. XXX: My prototype automagicly makes this a static method, just like __new__ is made into a static method. That's more convenient, but also (too?) magical. XXX: Should this raise AttributeError or return a magic value to signal that an attribute cannot be found (such as NotImplemented, used in the comparison operators)? I'm currently using an exception, a magical return value would be slightly more efficient because the exception machinery is not invoked. Alternative proposals --------------------- Reuse ``tp_getattro`` ..................... It would be nice to avoid adding a new slot, thus keeping the API simpler and easier to understand. A comment on `Issue 18181`_ asked about reusing the ``tp_getattro`` slot, that is super could call the ``tp_getattro`` slot of all methods along the MRO. AFAIK that won't work because ``tp_getattro`` will look in the instance ``__dict__`` before it tries to resolve attributes using classes in the MRO. This would mean that using ``tp_getattro`` instead of peeking the class dictionaries changes the semantics of the `super class`_. Open Issues =========== * The names of the new slot and magic method are far from settled. * I'm not too happy with the prototype for the new hook. * Should ``__getattribute_super__`` be a class method instead? -> Yes? The method looks up a named attribute name of an object in a specific class. Is also likely needed to deal with @classmethod and super(Class, Class) * Should ``__getattribute_super__`` be defined on object? -> Yes: makes it easier to delegate to the default implementation * This doesn't necessarily work for class method super class (e.g. super(object, object))... References ========== * `Issue 18181`_ contains a prototype implementation Copyright ========= This document has been placed in the public domain. .. _`Issue 18181`: http://bugs.python.org/issue18181 .. _`super class`: http://docs.python.org/3/library/functions.html?highlight=super#super From cf.natali at gmail.com Sat Jul 6 11:23:26 2013 From: cf.natali at gmail.com (=?ISO-8859-1?Q?Charles=2DFran=E7ois_Natali?=) Date: Sat, 6 Jul 2013 11:23:26 +0200 Subject: [Python-Dev] PEP 446: Add new parameters to configure the inherance of files and for non-blocking sockets In-Reply-To: <20130706004352.GA66371@cskk.homeip.net> References: <20130706004352.GA66371@cskk.homeip.net> Message-ID: > I've read your "Rejected Alternatives" more closely and Ulrich > Drepper's article, though I think the article also supports adding > a blocking (default True) parameter to open() and os.open(). If you > try to change that default on a platform where it doesn't work, an > exception should be raised. Contrarily to close-on-exec, non-blocking only applies to a limited type of files (e.g. it doesn't work for regular files, which represent 90% of open() use cases). Also, one of the main reasons for exposing close-on-exec in open()/socket() etc is to make it possible to create file descriptors with the close-on-exec flag atomically, to prevent unwanted FD inheritance especially in multi-threaded code. And that's not necessary for the non-blocking parameter. Those are two reasons why IMO "blocking" doesn't have to receive the same treatment as close-on-exec (there's also the Windows issue but I'm not familiar with it). cf From rdmurray at bitdance.com Sat Jul 6 13:59:07 2013 From: rdmurray at bitdance.com (R. David Murray) Date: Sat, 06 Jul 2013 07:59:07 -0400 Subject: [Python-Dev] =?utf-8?q?=5BPython-checkins=5D_cpython_=283=2E3=29?= =?utf-8?q?=3A_Issue_=2317860=3A_explicitly_mention_that_std*_strea?= =?utf-8?q?ms_are_opened_in_binary_mode_by?= In-Reply-To: <3bnQwH1z6Vz7LjW@mail.python.org> References: <3bnQwH1z6Vz7LjW@mail.python.org> Message-ID: <20130706115908.2072B250185@webabinitio.net> On Sat, 06 Jul 2013 10:25:19 +0200, ronald.oussoren wrote: > http://hg.python.org/cpython/rev/a2c2ffa1a41c > changeset: 84453:a2c2ffa1a41c > branch: 3.3 > parent: 84449:df79735b21c1 > user: Ronald Oussoren > date: Sat Jul 06 10:23:59 2013 +0200 > summary: > Issue #17860: explicitly mention that std* streams are opened in binary mode by default. > > The documentation does mention that the streams are opened in text mode > when univeral_newlines is true, but not that that they are opened in > binary mode when that argument is false and that seems to confuse at > least some users. > > files: > Doc/library/subprocess.rst | 6 ++++-- > 1 files changed, 4 insertions(+), 2 deletions(-) > > > diff --git a/Doc/library/subprocess.rst b/Doc/library/subprocess.rst > --- a/Doc/library/subprocess.rst > +++ b/Doc/library/subprocess.rst > @@ -293,7 +293,8 @@ > If *universal_newlines* is ``True``, the file objects *stdin*, *stdout* and > *stderr* will be opened as text streams in :term:`universal newlines` mode > using the encoding returned by :func:`locale.getpreferredencoding(False) > - `. For *stdin*, line ending characters > + `, otherwise these streams will be opened > + as binary streams. For *stdin*, line ending characters > ``'\n'`` in the input will be converted to the default line separator > :data:`os.linesep`. For *stdout* and *stderr*, all line endings in the > output will be converted to ``'\n'``. For more information see the IMO, either the default should be mentioned first, or the default should be mentioned in a parenthetical. Otherwise it sounds like newline translation is being done in both modes. Logically that makes no sense, so the above construction will likely lead to, at a minimum, an interruption in the flow for the reader, and at worse even more confusion than not mentioning it at all. --David From victor.stinner at gmail.com Sat Jul 6 14:04:13 2013 From: victor.stinner at gmail.com (Victor Stinner) Date: Sat, 6 Jul 2013 14:04:13 +0200 Subject: [Python-Dev] PEP 446: Add new parameters to configure the inherance of files and for non-blocking sockets In-Reply-To: References: <20130706004352.GA66371@cskk.homeip.net> Message-ID: 2013/7/6 Charles-Fran?ois Natali : >> I've read your "Rejected Alternatives" more closely and Ulrich >> Drepper's article, though I think the article also supports adding >> a blocking (default True) parameter to open() and os.open(). If you >> try to change that default on a platform where it doesn't work, an >> exception should be raised. > > Contrarily to close-on-exec, non-blocking only applies to a limited > type of files (e.g. it doesn't work for regular files, which represent > 90% of open() use cases). What do you mean by "does not work"? On Linux, O_NONBLOCK flag can be set on regular files, sockets, pipes, etc. Example with a regular file on Linux 3.9: smithers$ python3 Python 3.3.0 (default, Sep 29 2012, 22:07:38) >>> from fcntl import * >>> import os >>> f=open("/etc/issue", "rb") >>> fd=f.fileno() >>> flags=fcntl(fd, F_GETFL) >>> fcntl(fd, F_SETFL, flags|os.O_NONBLOCK) 0 >>> fcntl(fd, F_GETFL) & os.O_NONBLOCK 2048 >>> f.read(10) b'Fedora rel' For example, asyncore.file_dispatcher() uses fcntl to set the O_NONBLOCK flag. (The asyncore module is probably the only module of the stdlib which would benefit of a new os.set_blocking() function.) > Also, one of the main reasons for exposing close-on-exec in > open()/socket() etc is to make it possible to create file descriptors > with the close-on-exec flag atomically, to prevent unwanted FD > inheritance especially in multi-threaded code. And that's not > necessary for the non-blocking parameter. In the review of PEP 433 on this mailing list, Martin von Loewis said that the PEP does guarantee the atomicity. The implementation of the PEP falls back on ioctl or fcntl to set the flag on old Linux versions (or if the OS does not support setting O_CLOEXEC flag when creating a file descriptor). The GIL is released during system calls, so another thread can call execv(). That's why I mentionned explicitly in the PEP 446 that it does not offer any atomicity guarantee. But open(filename, blocking=False) on Linux has at least one advantage over f=open(filename); os.set_blocking(f.fileno()): it only uses one syscall (versus 2 or 3 syscalls). Victor From ronaldoussoren at mac.com Sat Jul 6 14:09:38 2013 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Sat, 6 Jul 2013 14:09:38 +0200 Subject: [Python-Dev] [Python-checkins] cpython (3.3): Issue #17860: explicitly mention that std* streams are opened in binary mode by In-Reply-To: <20130706115908.2072B250185@webabinitio.net> References: <3bnQwH1z6Vz7LjW@mail.python.org> <20130706115908.2072B250185@webabinitio.net> Message-ID: On 6 Jul, 2013, at 13:59, R. David Murray wrote: > On Sat, 06 Jul 2013 10:25:19 +0200, ronald.oussoren wrote: >> http://hg.python.org/cpython/rev/a2c2ffa1a41c >> changeset: 84453:a2c2ffa1a41c >> branch: 3.3 >> parent: 84449:df79735b21c1 >> user: Ronald Oussoren >> date: Sat Jul 06 10:23:59 2013 +0200 >> summary: >> Issue #17860: explicitly mention that std* streams are opened in binary mode by default. >> >> The documentation does mention that the streams are opened in text mode >> when univeral_newlines is true, but not that that they are opened in >> binary mode when that argument is false and that seems to confuse at >> least some users. >> >> files: >> Doc/library/subprocess.rst | 6 ++++-- >> 1 files changed, 4 insertions(+), 2 deletions(-) >> >> >> diff --git a/Doc/library/subprocess.rst b/Doc/library/subprocess.rst >> --- a/Doc/library/subprocess.rst >> +++ b/Doc/library/subprocess.rst >> @@ -293,7 +293,8 @@ >> If *universal_newlines* is ``True``, the file objects *stdin*, *stdout* and >> *stderr* will be opened as text streams in :term:`universal newlines` mode >> using the encoding returned by :func:`locale.getpreferredencoding(False) >> - `. For *stdin*, line ending characters >> + `, otherwise these streams will be opened >> + as binary streams. For *stdin*, line ending characters >> ``'\n'`` in the input will be converted to the default line separator >> :data:`os.linesep`. For *stdout* and *stderr*, all line endings in the >> output will be converted to ``'\n'``. For more information see the > > IMO, either the default should be mentioned first, or the default > should be mentioned in a parenthetical. Otherwise it sounds like > newline translation is being done in both modes. Logically that makes > no sense, so the above construction will likely lead to, at a minimum, > an interruption in the flow for the reader, and at worse even more > confusion than not mentioning it at all. You've got a point there. Converting the next text (", otherwise ...") to a parententical seems to be the cleanest fix, creating a separate sentence for the ``False`` case introduces duplication unless I restructure the text. Ronald > > --David > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/ronaldoussoren%40mac.com From ronaldoussoren at mac.com Sat Jul 6 14:11:52 2013 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Sat, 6 Jul 2013 14:11:52 +0200 Subject: [Python-Dev] PEP 446: Add new parameters to configure the inherance of files and for non-blocking sockets In-Reply-To: References: <20130706004352.GA66371@cskk.homeip.net> Message-ID: <95ACDB99-FBD5-4289-A592-C892A6651106@mac.com> On 6 Jul, 2013, at 14:04, Victor Stinner wrote: > 2013/7/6 Charles-Fran?ois Natali : >>> I've read your "Rejected Alternatives" more closely and Ulrich >>> Drepper's article, though I think the article also supports adding >>> a blocking (default True) parameter to open() and os.open(). If you >>> try to change that default on a platform where it doesn't work, an >>> exception should be raised. >> >> Contrarily to close-on-exec, non-blocking only applies to a limited >> type of files (e.g. it doesn't work for regular files, which represent >> 90% of open() use cases). > > What do you mean by "does not work"? On Linux, O_NONBLOCK flag can be > set on regular files, sockets, pipes, etc. I guess he means that O_NONBLOCK doesn't actually do anything for regular files, regular files are always ready for I/O as far as select et. al. are concerned and I/O will block when data has to be read from disk (or in the case of a network filesystem, from another machine). Ronald From victor.stinner at gmail.com Sat Jul 6 14:43:24 2013 From: victor.stinner at gmail.com (Victor Stinner) Date: Sat, 6 Jul 2013 14:43:24 +0200 Subject: [Python-Dev] PEP 446: Add new parameters to configure the inherance of files and for non-blocking sockets In-Reply-To: <20130706004352.GA66371@cskk.homeip.net> References: <20130706004352.GA66371@cskk.homeip.net> Message-ID: 2013/7/6 Cameron Simpson : > Yes. Please forget I mentioned fork(); it is only relevant if you > were offering some facility to undo the addition of cloexec to a > Popen passed file descriptor. Which you are not. Oh... gotcha. I now understood your concern. There is a little "trick" here: at fork, file descriptors are duplicated in the child process and almost all properties (open state, flags, offset, etc.) are shared. There is one little exception: file attributes are not shared, and there is only one file attribute: O_CLOEXEC. Setting O_CLOEXEC in a child process does not affect the flag in the parent process ;-) I will add a unit test to check this. I modified the PEP to explain that, and I also mentioned the name of the close-on-exec and blocking flags: http://hg.python.org/peps/rev/425f831fddf7 Victor From fuzzyman at voidspace.org.uk Sat Jul 6 14:48:26 2013 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Sat, 6 Jul 2013 14:48:26 +0200 Subject: [Python-Dev] lament for the demise of unbound methods In-Reply-To: <51D69AD3.7050003@v.loewis.de> References: <51D55ACB.4050900@simplistix.co.uk> <51D5A5DF.3000103@simplistix.co.uk> <761E2333-74A7-4F50-9943-E8148189507B@voidspace.org.uk> <51D69AD3.7050003@v.loewis.de> Message-ID: <5C366E9D-507E-44C5-8C9A-AD14933ECC79@voidspace.org.uk> On 5 Jul 2013, at 12:07, "Martin v. L?wis" wrote: > Am 05.07.13 11:23, schrieb Michael Foord: >> I've also lamented the death of bound methods in Python 3 for mock >> "autospeccing". Autospec introspects objects and provides mock >> objects with the same attributes - and with the same method >> signatures. > > I wonder why you need to figure out the signatures in advance. > Can you just wait until the function is actually used, and then > process the parameters as you get them? > How does that solve the problem? Given a call and a reference to the original "function object" I need to know whether or not to trim the first argument from the original signature or not (remove self if the corresponding function object is actually a method). Michael > Regards, > Martin -- http://www.voidspace.org.uk/ May you do good and not evil May you find forgiveness for yourself and forgive others May you share freely, never taking more than you give. -- the sqlite blessing http://www.sqlite.org/different.html From fuzzyman at voidspace.org.uk Sat Jul 6 14:49:27 2013 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Sat, 6 Jul 2013 14:49:27 +0200 Subject: [Python-Dev] lament for the demise of unbound methods In-Reply-To: <076A4BF7-C8CC-4325-985F-BB905B174F01@langa.pl> References: <51D55ACB.4050900@simplistix.co.uk> <51D5A5DF.3000103@simplistix.co.uk> <761E2333-74A7-4F50-9943-E8148189507B@voidspace.org.uk> <51D69AD3.7050003@v.loewis.de> <076A4BF7-C8CC-4325-985F-BB905B174F01@langa.pl> Message-ID: <5647F6A3-787D-4A24-9769-61CAB99C27E7@voidspace.org.uk> On 5 Jul 2013, at 12:26, ?ukasz Langa wrote: > On 5 lip 2013, at 12:07, Martin v. L?wis wrote: > >> I wonder why you need to figure out the signatures in advance. >> Can you just wait until the function is actually used, and then >> process the parameters as you get them? >> > > My guess is that Michael's design lets mock objects be introspected as well, i.e. they don't appear as magical as they really are to the user code. > This is also true. Doing it up front has some conveniences - for example dir(...) works correctly. Michael > -- > Best regards, > ?ukasz Langa > > WWW: http://lukasz.langa.pl/ > Twitter: @llanga > IRC: ambv on #python-dev > -- http://www.voidspace.org.uk/ May you do good and not evil May you find forgiveness for yourself and forgive others May you share freely, never taking more than you give. -- the sqlite blessing http://www.sqlite.org/different.html From ronaldoussoren at mac.com Sat Jul 6 15:03:13 2013 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Sat, 6 Jul 2013 15:03:13 +0200 Subject: [Python-Dev] [Python-checkins] cpython (3.3): Issue #17860: explicitly mention that std* streams are opened in binary mode by In-Reply-To: References: <3bnQwH1z6Vz7LjW@mail.python.org> <20130706115908.2072B250185@webabinitio.net> Message-ID: <1ADEC7C5-5557-4DBC-B220-1EECF6DB9A58@mac.com> On 6 Jul, 2013, at 14:09, Ronald Oussoren wrote: > > On 6 Jul, 2013, at 13:59, R. David Murray wrote: >> >> IMO, either the default should be mentioned first, or the default >> should be mentioned in a parenthetical. Otherwise it sounds like >> newline translation is being done in both modes. Logically that makes >> no sense, so the above construction will likely lead to, at a minimum, >> an interruption in the flow for the reader, and at worse even more >> confusion than not mentioning it at all. > > You've got a point there. Converting the next text (", otherwise ...") to a parententical > seems to be the cleanest fix, creating a separate sentence for the ``False`` case introduces > duplication unless I restructure the text. I didn't like the parenthentical after all. Would this work for you?: - If *universal_newlines* is ``True``, the file objects *stdin*, *stdout* and - *stderr* will be opened as text streams in :term:`universal newlines` mode + If *universal_newlines* is ``False`` the file objects *stdin*, *stdout* and + *stderr* will be opened as binary streams, and no line ending conversion is done. + + If *universal_newlines* is ``True``, these file objects + will be opened as text streams in :term:`universal newlines` mode using the encoding returned by :func:`locale.getpreferredencoding(False) - `, otherwise these streams will be opened - as binary streams. For *stdin*, line ending characters + `. For *stdin*, line ending characters ``'\n'`` in the input will be converted to the default line separator :data:`os.linesep`. For *stdout* and *stderr*, all line endings in the output will be converted to ``'\n'``. For more information see the That is, a new paragraph is added before the existing one to explain the behavior of "not universal_newlines". Ronald > > Ronald > >> >> --David >> _______________________________________________ >> Python-Dev mailing list >> Python-Dev at python.org >> http://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe: http://mail.python.org/mailman/options/python-dev/ronaldoussoren%40mac.com > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/ronaldoussoren%40mac.com From greg at krypto.org Sat Jul 6 17:14:26 2013 From: greg at krypto.org (Gregory P. Smith) Date: Sat, 6 Jul 2013 08:14:26 -0700 Subject: [Python-Dev] [Python-checkins] cpython (3.3): Issue #17860: explicitly mention that std* streams are opened in binary mode by In-Reply-To: <1ADEC7C5-5557-4DBC-B220-1EECF6DB9A58@mac.com> References: <3bnQwH1z6Vz7LjW@mail.python.org> <20130706115908.2072B250185@webabinitio.net> <1ADEC7C5-5557-4DBC-B220-1EECF6DB9A58@mac.com> Message-ID: Please update the docstring in subprocess.py with the wording improvements that you settle on while you're at it. On Sat, Jul 6, 2013 at 6:03 AM, Ronald Oussoren wrote: > > On 6 Jul, 2013, at 14:09, Ronald Oussoren wrote: > > > > > On 6 Jul, 2013, at 13:59, R. David Murray wrote: > >> > >> IMO, either the default should be mentioned first, or the default > >> should be mentioned in a parenthetical. Otherwise it sounds like > >> newline translation is being done in both modes. Logically that makes > >> no sense, so the above construction will likely lead to, at a minimum, > >> an interruption in the flow for the reader, and at worse even more > >> confusion than not mentioning it at all. > > > > You've got a point there. Converting the next text (", otherwise ...") > to a parententical > > seems to be the cleanest fix, creating a separate sentence for the > ``False`` case introduces > > duplication unless I restructure the text. > > I didn't like the parenthentical after all. Would this work for you?: > > > - If *universal_newlines* is ``True``, the file objects *stdin*, > *stdout* and > - *stderr* will be opened as text streams in :term:`universal newlines` > mode > + If *universal_newlines* is ``False`` the file objects *stdin*, > *stdout* and > + *stderr* will be opened as binary streams, and no line ending > conversion is done. > + > + If *universal_newlines* is ``True``, these file objects > + will be opened as text streams in :term:`universal newlines` mode > using the encoding returned by > :func:`locale.getpreferredencoding(False) > - `, otherwise these streams will be opened > - as binary streams. For *stdin*, line ending characters > + `. For *stdin*, line ending characters > ``'\n'`` in the input will be converted to the default line separator > :data:`os.linesep`. For *stdout* and *stderr*, all line endings in the > output will be converted to ``'\n'``. For more information see the > > That is, a new paragraph is added before the existing one to explain the > behavior of > "not universal_newlines". > > Ronald > > > > > > Ronald > > > >> > >> --David > >> _______________________________________________ > >> Python-Dev mailing list > >> Python-Dev at python.org > >> http://mail.python.org/mailman/listinfo/python-dev > >> Unsubscribe: > http://mail.python.org/mailman/options/python-dev/ronaldoussoren%40mac.com > > > > _______________________________________________ > > Python-Dev mailing list > > Python-Dev at python.org > > http://mail.python.org/mailman/listinfo/python-dev > > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/ronaldoussoren%40mac.com > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/greg%40krypto.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: From larry at hastings.org Sat Jul 6 19:33:17 2013 From: larry at hastings.org (Larry Hastings) Date: Sat, 06 Jul 2013 19:33:17 +0200 Subject: [Python-Dev] Rough idea for adding introspection information for builtins In-Reply-To: <5148374D.6050402@hastings.org> References: <5147ED55.30605@hastings.org> <5148374D.6050402@hastings.org> Message-ID: <51D854DD.7080807@hastings.org> Revisiting a four-month-old discussion: On 03/19/2013 11:00 AM, Larry Hastings wrote: > As for handling optional argument groups, my gut feeling is that we're > better off not leaking it out of Argument Clinic--don't expose it in > this string we're talking about, and don't add support for it in the > inspect.Parameter object. I'm not going to debate range(), the syntax > of which predates one of our release managers. But I suggest option > groups are simply a misfeature of the curses module. There are some > other possible uses in builtins (I forgot to dig those out this > evening) but so far we're talking adding complexity to an array of > technologies (this representation, the parser, the Parameter object) > to support a handful of uses of something we shouldn't have done in > the first place, for consumers who I think won't care and won't > appreciate the added conceptual complexity. I'm sad to say I've just about changed my mind on this. This is what help(os.stat) looks like in my dev branch for Argument Clinic: >>> help(os.stat) Help on built-in function stat in module posix: stat(...) os.stat(path, *, dir_fd=None, follow_symlinks=True) ... Argument Clinic added the line starting with "os.stat(path, ". pydoc generated the "stat(...)" line. It doesn't have any info because of the lack of introspection information. Once builtins have introspection information, pydoc can do a better job, and Argument Clinic can stop generating its redundant prototype line. But if pydoc doesn't have argument group information, it won't be able to tell where one group ends and the next begins, and it won't be able to render the prototype for the help text correctly. I fear misleading text is even worse than no text at all. I also suggest that fancy editors (PyCharm etc) want as much information as we can give them. If we give them argument group information, they can flag malformed calls (e.g. "there's no way to legally call this function with exactly three arguments"). I therefore have two credible consumers of this information. That's enough for me: I propose we amend the Parameter object to add option group information for positional-only parameters. //arry/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Sat Jul 6 22:54:54 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sat, 6 Jul 2013 22:54:54 +0200 Subject: [Python-Dev] Accepting PEP 445 Message-ID: <20130706225454.0932f6ba@fsol> Hello, I'm accepting PEP 445 (A C API to customize memory allocators) by Victor. There is probably some grammar to correct here and there (neither Victor nor I are native English speakers), but I don't want this to hold back acceptance. The PEP is an obvious improvement in flexibility for everyone embedding Python in some external application, or even recompiling their own Python runtime for specific uses. The implementation is tracked in http://bugs.python.org/issue3329 Thanks to Victor for his patience :-) Regards Antoine. From a.cavallo at cavallinux.eu Sat Jul 6 23:07:40 2013 From: a.cavallo at cavallinux.eu (Antonio Cavallo) Date: Sat, 6 Jul 2013 22:07:40 +0100 Subject: [Python-Dev] Accepting PEP 445 In-Reply-To: <20130706225454.0932f6ba@fsol> References: <20130706225454.0932f6ba@fsol> Message-ID: Could that remove the need for the --with-pydebug flag? On 6 Jul 2013, at 21:54, Antoine Pitrou wrote: > > Hello, > > I'm accepting PEP 445 (A C API to customize memory allocators) by > Victor. There is probably some grammar to correct here and there > (neither Victor nor I are native English speakers), but I don't want > this to hold back acceptance. The PEP is an obvious improvement in > flexibility for everyone embedding Python in some external application, > or even recompiling their own Python runtime for specific uses. > > The implementation is tracked in http://bugs.python.org/issue3329 > > Thanks to Victor for his patience :-) > > Regards > > Antoine. > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/a.cavallo%40cavallinux.eu From solipsis at pitrou.net Sat Jul 6 23:43:26 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sat, 6 Jul 2013 23:43:26 +0200 Subject: [Python-Dev] Accepting PEP 445 In-Reply-To: References: <20130706225454.0932f6ba@fsol> Message-ID: <20130706234326.0b94776b@fsol> On Sat, 6 Jul 2013 22:07:40 +0100 Antonio Cavallo wrote: > Could that remove the need for the --with-pydebug flag? No, --with-pydebug enables much more than just memory debug checks. Regards Antoine. > > > > > On 6 Jul 2013, at 21:54, Antoine Pitrou wrote: > > > > > Hello, > > > > I'm accepting PEP 445 (A C API to customize memory allocators) by > > Victor. There is probably some grammar to correct here and there > > (neither Victor nor I are native English speakers), but I don't want > > this to hold back acceptance. The PEP is an obvious improvement in > > flexibility for everyone embedding Python in some external application, > > or even recompiling their own Python runtime for specific uses. > > > > The implementation is tracked in http://bugs.python.org/issue3329 > > > > Thanks to Victor for his patience :-) > > > > Regards > > > > Antoine. > > > > > > _______________________________________________ > > Python-Dev mailing list > > Python-Dev at python.org > > http://mail.python.org/mailman/listinfo/python-dev > > Unsubscribe: http://mail.python.org/mailman/options/python-dev/a.cavallo%40cavallinux.eu > From ncoghlan at gmail.com Sun Jul 7 00:18:52 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 7 Jul 2013 08:18:52 +1000 Subject: [Python-Dev] lament for the demise of unbound methods In-Reply-To: <5647F6A3-787D-4A24-9769-61CAB99C27E7@voidspace.org.uk> References: <51D55ACB.4050900@simplistix.co.uk> <51D5A5DF.3000103@simplistix.co.uk> <761E2333-74A7-4F50-9943-E8148189507B@voidspace.org.uk> <51D69AD3.7050003@v.loewis.de> <076A4BF7-C8CC-4325-985F-BB905B174F01@langa.pl> <5647F6A3-787D-4A24-9769-61CAB99C27E7@voidspace.org.uk> Message-ID: On 6 Jul 2013 22:52, "Michael Foord" wrote: > > > On 5 Jul 2013, at 12:26, ?ukasz Langa wrote: > > > On 5 lip 2013, at 12:07, Martin v. L?wis wrote: > > > >> I wonder why you need to figure out the signatures in advance. > >> Can you just wait until the function is actually used, and then > >> process the parameters as you get them? > >> > > > > My guess is that Michael's design lets mock objects be introspected as well, i.e. they don't appear as magical as they really are to the user code. > > > > This is also true. Doing it up front has some conveniences - for example dir(...) works correctly. Doesn't this really mean that in Py3, introspection APIs need a class *and* an instance of that class for robust analysis of expected descriptor results? (which was technically always true, the removal of unbound methods just makes it significantly more important to handle descriptors with no special behaviour on the class side of things). And yes, I'm aware that idea poses a significant challenge for correct handling of ABCs :P Cheers, Nick. > > Michael > > > -- > > Best regards, > > ?ukasz Langa > > > > WWW: http://lukasz.langa.pl/ > > Twitter: @llanga > > IRC: ambv on #python-dev > > > > > -- > http://www.voidspace.org.uk/ > > > May you do good and not evil > May you find forgiveness for yourself and forgive others > May you share freely, never taking more than you give. > -- the sqlite blessing > http://www.sqlite.org/different.html > > > > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Sun Jul 7 00:32:46 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 7 Jul 2013 08:32:46 +1000 Subject: [Python-Dev] Rough idea for adding introspection information for builtins In-Reply-To: <51D854DD.7080807@hastings.org> References: <5147ED55.30605@hastings.org> <5148374D.6050402@hastings.org> <51D854DD.7080807@hastings.org> Message-ID: On 7 Jul 2013 05:22, "Larry Hastings" wrote: > > > Revisiting a four-month-old discussion: > > > On 03/19/2013 11:00 AM, Larry Hastings wrote: >> >> As for handling optional argument groups, my gut feeling is that we're better off not leaking it out of Argument Clinic--don't expose it in this string we're talking about, and don't add support for it in the inspect.Parameter object. I'm not going to debate range(), the syntax of which predates one of our release managers. But I suggest option groups are simply a misfeature of the curses module. There are some other possible uses in builtins (I forgot to dig those out this evening) but so far we're talking adding complexity to an array of technologies (this representation, the parser, the Parameter object) to support a handful of uses of something we shouldn't have done in the first place, for consumers who I think won't care and won't appreciate the added conceptual complexity. > > > I'm sad to say I've just about changed my mind on this. > > This is what help(os.stat) looks like in my dev branch for Argument Clinic: >> >> >>> help(os.stat) >> Help on built-in function stat in module posix: >> >> stat(...) >> os.stat(path, *, dir_fd=None, follow_symlinks=True) >> ... > > > Argument Clinic added the line starting with "os.stat(path, ". pydoc generated the "stat(...)" line. It doesn't have any info because of the lack of introspection information. > > Once builtins have introspection information, pydoc can do a better job, and Argument Clinic can stop generating its redundant prototype line. But if pydoc doesn't have argument group information, it won't be able to tell where one group ends and the next begins, and it won't be able to render the prototype for the help text correctly. I fear misleading text is even worse than no text at all. > > I also suggest that fancy editors (PyCharm etc) want as much information as we can give them. If we give them argument group information, they can flag malformed calls (e.g. "there's no way to legally call this function with exactly three arguments"). > > I therefore have two credible consumers of this information. That's enough for me: I propose we amend the Parameter object to add option group information for positional-only parameters. Rather than perpetuating unwanted complexity, can't we just add a single "incomplete signature" flag to handle the legacy cases, and leave those to the docstrings? As in, if the flag is set, pydoc displays the "..." because it knows the signature data isn't quite right. Alternatively (and even more simply), is it really so bad if argument clinic doesn't support introspection of such functions at all, and avoids setting __signature__ for such cases? As a third option, we could add an "alternative signatures" attribute to capture multiple orthogonal signatures that should be presented on separate lines. All of those possibilities sound more appealing to me than adding direct support for parameter groups at the Python level (with my preference being to postpone the question to 3.5 by not allowing introspection of affected functions in this initial iteration). Cheers, Nick. > > > /arry > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Sun Jul 7 00:35:37 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 7 Jul 2013 08:35:37 +1000 Subject: [Python-Dev] Accepting PEP 445 In-Reply-To: <20130706225454.0932f6ba@fsol> References: <20130706225454.0932f6ba@fsol> Message-ID: On 7 Jul 2013 07:01, "Antoine Pitrou" wrote: > > > Hello, > > I'm accepting PEP 445 (A C API to customize memory allocators) by > Victor. There is probably some grammar to correct here and there > (neither Victor nor I are native English speakers), but I don't want > this to hold back acceptance. The PEP is an obvious improvement in > flexibility for everyone embedding Python in some external application, > or even recompiling their own Python runtime for specific uses. > > The implementation is tracked in http://bugs.python.org/issue3329 > > Thanks to Victor for his patience :-) Yay! And thanks to you for suggesting this API would benefit from being run through the PEP process - I think the end result is a significant improvement over what we originally planned to commit. Cheers, Nick. > > Regards > > Antoine. > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From victor.stinner at gmail.com Sun Jul 7 01:04:38 2013 From: victor.stinner at gmail.com (Victor Stinner) Date: Sun, 7 Jul 2013 01:04:38 +0200 Subject: [Python-Dev] Accepting PEP 445 In-Reply-To: References: <20130706225454.0932f6ba@fsol> Message-ID: 2013/7/6 Antonio Cavallo : > Could that remove the need for the --with-pydebug flag? With the PEP 445, you still have to recompile Python with --with-debug, but you don't have to recompile Python extensions anymore. Debug checks on memory allocators are now implemented as hooks, instead of using C macros. I see this as an improvment ;-) Victor From larry at hastings.org Sun Jul 7 02:22:46 2013 From: larry at hastings.org (Larry Hastings) Date: Sun, 07 Jul 2013 02:22:46 +0200 Subject: [Python-Dev] Rough idea for adding introspection information for builtins In-Reply-To: References: <5147ED55.30605@hastings.org> <5148374D.6050402@hastings.org> <51D854DD.7080807@hastings.org> Message-ID: <51D8B4D6.3080107@hastings.org> On 07/07/2013 12:32 AM, Nick Coghlan wrote: > > Rather than perpetuating unwanted complexity, can't we just add a > single "incomplete signature" flag to handle the legacy cases, and > leave those to the docstrings? > > As in, if the flag is set, pydoc displays the "..." because it knows > the signature data isn't quite right. > > Alternatively (and even more simply), is it really so bad if argument > clinic doesn't support introspection of such functions at all, and > avoids setting __signature__ for such cases? > > As a third option, we could add an "alternative signatures" attribute > to capture multiple orthogonal signatures that should be presented on > separate lines. > > All of those possibilities sound more appealing to me than adding > direct support for parameter groups at the Python level (with my > preference being to postpone the question to 3.5 by not allowing > introspection of affected functions in this initial iteration). > First, I think the PyCharm case is compelling enough on its own. I realized after I sent it that there's a related class of tools that are interested: PyFlakes, PyLint, and the like. I'm sure the static correctness analyzers would like to be able to automatically determine "this is an illegal number of parameters for this function" for builtins--particularly for third-party builtins! The fact that we wouldn't need to special-case pydoc suggests it's the superior approach. ("Special cases aren't special enough to break the rules.") Second, the added complexity would be a single new member on the Parameter object. Let me propose such a parameter here, in the style of the Parameter class documentation: group If not None, represents which "optional parameter group" this parameter belongs to. Optional parameter groups are contiguous sequences of parameters that must either all be specified or all be unspecified. For example, if a function takes four parameters but the last two are in an optional parameter group, you could specify either two or four arguments to that function--it would be illegal to specify three arguments. Parameter groups can only contain positional-only parameters; therefore group will only be a non-None value when kind is POSITIONAL_ONLY. I suggest that is a manageable level of complexity. And that the tooling projects would very much like to have this information. Third, your proposals are respectively: 1) a hack which fixes the docstring but doesn't fix the introspection information (so we'd be providing incorrect introspection information to tools), 2) a small cop-out (which I think would also probably require a hack to pydoc), and 3) way more complicated than doing it the right way (so I don't see how it's an improvement). Of your three suggestions I dislike 2) least. This facet of call signatures has existed in Python since the addition of range(). I concede that it's legacy, but it's not going away. Ever. I now think we're better off embracing this complexity than trying to sweep it under the rug. //arry/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From cs at zip.com.au Sun Jul 7 02:29:12 2013 From: cs at zip.com.au (Cameron Simpson) Date: Sun, 7 Jul 2013 10:29:12 +1000 Subject: [Python-Dev] PEP 446: Add new parameters to configure the inherance of files and for non-blocking sockets In-Reply-To: References: Message-ID: <20130707002912.GA49468@cskk.homeip.net> On 06Jul2013 14:43, Victor Stinner wrote: | 2013/7/6 Cameron Simpson : | > Yes. Please forget I mentioned fork(); it is only relevant if you | > were offering some facility to undo the addition of cloexec to a | > Popen passed file descriptor. Which you are not. | | Oh... gotcha. I now understood your concern. | | There is a little "trick" here: at fork, file descriptors are | duplicated in the child process and almost all properties (open state, | flags, offset, etc.) are shared. There is one little exception: file | attributes are not shared, and there is only one file attribute: | O_CLOEXEC. Setting O_CLOEXEC in a child process does not affect the | flag in the parent process ;-) I will add a unit test to check this. That's news to me. Interesting. I can't find UNIX doco to support this, though I haven't finished looking. None of the POSIX, Linux or MacOSX fork() manual pages seem to describe this. Can you point me at doco for this? I thought file descriptors were a direct index into a per-process table of file handle structures, and did not think the latter got part copies, part shared over a fork. Also, is O_CLOEXEC really the only file attribute? So O_NONBLOCK is a flag and not an attribute (guessing the terminology distinguishes these)? Now you mention it, ISTR noticing that there were distinct ioctls to adjust flags and something else. | I modified the PEP to explain that, and I also mentioned the name of | the close-on-exec and blocking flags: | http://hg.python.org/peps/rev/425f831fddf7 Thanks. Your change says: On Windows, the close-on-exec flag is ``HANDLE_FLAG_INHERIT``. I feel it would be clearer to say: On Windows, the close-on-exec flag is the inverse of ``HANDLE_FLAG_INHERIT``. or On Windows, the close-on-exec flag implemented using ``HANDLE_FLAG_INHERIT``. (This latter is fine because the rest of the paragraph explains the meaning of HANDLE_FLAG_INHERIT.) The bare "is" in your current wording suggests to me that the truthiness (ick!) of close-on-exec is the same as for HANDLE_FLAG_INHERIT, which it isn't. -- Cameron Simpson I've always been a big Greenaway fan - I've seen and enjoyed "The Falls" for crying out loud. - Peter Alexander Merel From cs at zip.com.au Sun Jul 7 02:30:56 2013 From: cs at zip.com.au (Cameron Simpson) Date: Sun, 7 Jul 2013 10:30:56 +1000 Subject: [Python-Dev] PEP 446: Add new parameters to configure the inherance of files and for non-blocking sockets In-Reply-To: References: Message-ID: <20130707003056.GA55329@cskk.homeip.net> On 06Jul2013 11:23, Charles-Fran?ois Natali wrote: | > I've read your "Rejected Alternatives" more closely and Ulrich | > Drepper's article, though I think the article also supports adding | > a blocking (default True) parameter to open() and os.open(). If you | > try to change that default on a platform where it doesn't work, an | > exception should be raised. | | Contrarily to close-on-exec, non-blocking only applies to a limited | type of files (e.g. it doesn't work for regular files, which represent | 90% of open() use cases). sockets, pipes, serial devices, ... And you can set it on anything. Just because some things don't block anyway isn't really a counter argument. Cheers, -- Cameron Simpson Hofstadter's Law: It always takes longer than you expect, even when you take into account Hofstadter's Law. - Douglas Hosfstadter, Godel, Escher, Bach: an Eternal Golden Braid From ncoghlan at gmail.com Sun Jul 7 03:04:12 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 7 Jul 2013 11:04:12 +1000 Subject: [Python-Dev] Rough idea for adding introspection information for builtins In-Reply-To: <51D8B4D6.3080107@hastings.org> References: <5147ED55.30605@hastings.org> <5148374D.6050402@hastings.org> <51D854DD.7080807@hastings.org> <51D8B4D6.3080107@hastings.org> Message-ID: On 7 Jul 2013 10:25, "Larry Hastings" wrote: > > On 07/07/2013 12:32 AM, Nick Coghlan wrote: >> >> Rather than perpetuating unwanted complexity, can't we just add a single "incomplete signature" flag to handle the legacy cases, and leave those to the docstrings? >> >> As in, if the flag is set, pydoc displays the "..." because it knows the signature data isn't quite right. >> >> Alternatively (and even more simply), is it really so bad if argument clinic doesn't support introspection of such functions at all, and avoids setting __signature__ for such cases? >> >> As a third option, we could add an "alternative signatures" attribute to capture multiple orthogonal signatures that should be presented on separate lines. >> >> All of those possibilities sound more appealing to me than adding direct support for parameter groups at the Python level (with my preference being to postpone the question to 3.5 by not allowing introspection of affected functions in this initial iteration). > > > First, I think the PyCharm case is compelling enough on its own. I realized after I sent it that there's a related class of tools that are interested: PyFlakes, PyLint, and the like. I'm sure the static correctness analyzers would like to be able to automatically determine "this is an illegal number of parameters for this function" for builtins--particularly for third-party builtins! The fact that we wouldn't need to special-case pydoc suggests it's the superior approach. ("Special cases aren't special enough to break the rules.") > > Second, the added complexity would be a single new member on the Parameter object. Let me propose such a parameter here, in the style of the Parameter class documentation: >> >> group >>> >>> If not None, represents which "optional parameter group" this parameter belongs to. Optional parameter groups are contiguous sequences of parameters that must either all be specified or all be unspecified. For example, if a function takes four parameters but the last two are in an optional parameter group, you could specify either two or four arguments to that function--it would be illegal to specify three arguments. Parameter groups can only contain positional-only parameters; therefore group will only be a non-None value when kind is POSITIONAL_ONLY. > > > I suggest that is a manageable level of complexity. And that the tooling projects would very much like to have this information. > > Third, your proposals are respectively: 1) a hack which fixes the docstring but doesn't fix the introspection information (so we'd be providing incorrect introspection information to tools), 2) a small cop-out (which I think would also probably require a hack to pydoc), and 3) way more complicated than doing it the right way (so I don't see how it's an improvement). Of your three suggestions I dislike 2) least. > > This facet of call signatures has existed in Python since the addition of range(). I concede that it's legacy, but it's not going away. Ever. I now think we're better off embracing this complexity than trying to sweep it under the rug. The "group" attribute sounds reasonable to me, with the proviso that we use "multiple signature lines" as the way to represent them in pydoc (rather than inventing a notation for showing them in a single signature line). It occurs to me that "type" is itself an example of this kind of dual signature. Cheers, Nick. > > > /arry > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com > -------------- next part -------------- An HTML attachment was scrubbed... URL: From larry at hastings.org Sun Jul 7 04:48:18 2013 From: larry at hastings.org (Larry Hastings) Date: Sun, 07 Jul 2013 04:48:18 +0200 Subject: [Python-Dev] Rough idea for adding introspection information for builtins In-Reply-To: References: <5147ED55.30605@hastings.org> <5148374D.6050402@hastings.org> <51D854DD.7080807@hastings.org> <51D8B4D6.3080107@hastings.org> Message-ID: <51D8D6F2.80007@hastings.org> On 07/07/2013 03:04 AM, Nick Coghlan wrote: > > On 7 Jul 2013 10:25, "Larry Hastings" > wrote: > >> > >> group > >>> > >>> If not None, represents which "optional parameter group" this > parameter belongs to. Optional parameter groups are contiguous > sequences of parameters that must either all be specified or all be > unspecified. For example, if a function takes four parameters but the > last two are in an optional parameter group, you could specify either > two or four arguments to that function--it would be illegal to specify > three arguments. Parameter groups can only contain positional-only > parameters; therefore group will only be a non-None value when kind is > POSITIONAL_ONLY. > > The "group" attribute sounds reasonable to me, with the proviso that > we use "multiple signature lines" as the way to represent them in > pydoc (rather than inventing a notation for showing them in a single > signature line). > > It occurs to me that "type" is itself an example of this kind of dual > signature. > We don't have to invent a notation--because we already have one. It's square brackets enclosing the optional parameter groups. This is the syntax Guido dictated for Argument Clinic to use in its input DSL back at PyCon US 2013. And the Python standard library documentation has been using this convention since the 90s. (Admittedly as a documentation convention, not in code. But what we're talking about is documentation so I claim it's totally relevant.) If we combine that with the admittedly-new "/" indicating "all previous parameters are positional-only", which we're also already using in the Argument Clinic input DSL syntax (at your suggestion!), we have a complete, crisp syntax. I suppose "/" isn't really necessary, the Python documentation has survived without it for a long time. But I think it'd would be a nice clarification; it would convey that you can't splat in **kwargs into functions specifying it, for example. I expect this to be the format of the signature-for-builtins static data too, as well as the Argument Clinic input syntax. Sure seems like a nice idea to use the same syntax everywhere. Particularly allowing as how it's so nice and readable. An admission: the Python standard library documentation actually uses *both* square-brackets-for-optional-groups and multiple lines. Sometimes even for the same function! An example: http://docs.python.org/3/library/curses.html#curses.window.addch Of the two, I believe the square brackets syntax is far more common. //arry/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From raymond.hettinger at gmail.com Sun Jul 7 05:30:44 2013 From: raymond.hettinger at gmail.com (Raymond Hettinger) Date: Sat, 6 Jul 2013 17:30:44 -1000 Subject: [Python-Dev] lament for the demise of unbound methods In-Reply-To: References: <51D55ACB.4050900@simplistix.co.uk> <51D5639B.4060604@python.org> <51D56705.1000908@simplistix.co.uk> Message-ID: <1F06580E-1BD5-4986-814C-0E2572F4F0C5@gmail.com> On Jul 4, 2013, at 2:34 AM, Brett Cannon wrote: > The loss of the ability to figure out the class from an unbound method seems quite an annoying step back from an introspection point of view. > > It's only annoying if you take the perspective that methods are somehow special compared to functions. With the removal of bound class methods that makes methods == functions that are an attribute on a class. And when you take that perspective it makes having anything special about methods seem wrong. It also makes adding a function to a class post-class creation make more sense since there is no difference technically. Well said, Brett. This is a nice summary. Raymond -------------- next part -------------- An HTML attachment was scrubbed... URL: From rdmurray at bitdance.com Sun Jul 7 06:59:18 2013 From: rdmurray at bitdance.com (R. David Murray) Date: Sun, 07 Jul 2013 00:59:18 -0400 Subject: [Python-Dev] [Python-checkins] cpython (3.3): Issue #17860: explicitly mention that std* streams are opened in binary mode by In-Reply-To: References: <3bnQwH1z6Vz7LjW@mail.python.org> <20130706115908.2072B250185@webabinitio.net> <1ADEC7C5-5557-4DBC-B220-1EECF6DB9A58@mac.com> Message-ID: <20130707045918.BF3BE2502CD@webabinitio.net> On Sat, 06 Jul 2013 08:14:26 -0700, "Gregory P. Smith" wrote: > Please update the docstring in subprocess.py with the wording improvements > that you settle on while you're at it. > > On Sat, Jul 6, 2013 at 6:03 AM, Ronald Oussoren wrote: > > I didn't like the parenthentical after all. Would this work for you?: > > > > - If *universal_newlines* is ``True``, the file objects *stdin*, > > *stdout* and > > - *stderr* will be opened as text streams in :term:`universal newlines` > > mode > > + If *universal_newlines* is ``False`` the file objects *stdin*, > > *stdout* and > > + *stderr* will be opened as binary streams, and no line ending > > conversion is done. > > + > > + If *universal_newlines* is ``True``, these file objects > > + will be opened as text streams in :term:`universal newlines` mode > > using the encoding returned by > > :func:`locale.getpreferredencoding(False) > > - `, otherwise these streams will be opened > > - as binary streams. For *stdin*, line ending characters > > + `. For *stdin*, line ending characters > > ``'\n'`` in the input will be converted to the default line separator > > :data:`os.linesep`. For *stdout* and *stderr*, all line endings in the > > output will be converted to ``'\n'``. For more information see the > > > > That is, a new paragraph is added before the existing one to explain the > > behavior of "not universal_newlines". Looks good to me. --David From ronaldoussoren at mac.com Sun Jul 7 07:19:11 2013 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Sun, 7 Jul 2013 07:19:11 +0200 Subject: [Python-Dev] Rough idea for adding introspection information for builtins In-Reply-To: <51D854DD.7080807@hastings.org> References: <5147ED55.30605@hastings.org> <5148374D.6050402@hastings.org> <51D854DD.7080807@hastings.org> Message-ID: On 6 Jul, 2013, at 19:33, Larry Hastings wrote: > > Once builtins have introspection information, pydoc can do a better job, and Argument Clinic can stop generating its redundant prototype line. Not entirely on topic, but close enough: pydoc currently doesn't use the __signature__ information at all. Adding such support would be easy enough, see #17053 for an implementation ;-) Ronald From rdmurray at bitdance.com Sun Jul 7 07:25:04 2013 From: rdmurray at bitdance.com (R. David Murray) Date: Sun, 07 Jul 2013 01:25:04 -0400 Subject: [Python-Dev] Rough idea for adding introspection information for builtins In-Reply-To: <51D8D6F2.80007@hastings.org> References: <5147ED55.30605@hastings.org> <5148374D.6050402@hastings.org> <51D854DD.7080807@hastings.org> <51D8B4D6.3080107@hastings.org> <51D8D6F2.80007@hastings.org> Message-ID: <20130707052505.0CAD32502D1@webabinitio.net> On Sun, 07 Jul 2013 04:48:18 +0200, Larry Hastings wrote: > On 07/07/2013 03:04 AM, Nick Coghlan wrote: > > > > On 7 Jul 2013 10:25, "Larry Hastings" > > wrote: > > >> > > >> group > > >>> > > >>> If not None, represents which "optional parameter group" this > > parameter belongs to. Optional parameter groups are contiguous > > sequences of parameters that must either all be specified or all be > > unspecified. For example, if a function takes four parameters but the > > last two are in an optional parameter group, you could specify either > > two or four arguments to that function--it would be illegal to specify > > three arguments. Parameter groups can only contain positional-only > > parameters; therefore group will only be a non-None value when kind is > > POSITIONAL_ONLY. > > > > The "group" attribute sounds reasonable to me, with the proviso that > > we use "multiple signature lines" as the way to represent them in > > pydoc (rather than inventing a notation for showing them in a single > > signature line). > > > > It occurs to me that "type" is itself an example of this kind of dual > > signature. > > > > We don't have to invent a notation--because we already have one. It's > square brackets enclosing the optional parameter groups. This is the > syntax Guido dictated for Argument Clinic to use in its input DSL back > at PyCon US 2013. And the Python standard library documentation has > been using this convention since the 90s. (Admittedly as a documentation > convention, not in code. But what we're talking about is documentation > so I claim it's totally relevant.) > > If we combine that with the admittedly-new "/" indicating "all previous > parameters are positional-only", which we're also already using in the > Argument Clinic input DSL syntax (at your suggestion!), we have a > complete, crisp syntax. I suppose "/" isn't really necessary, the > Python documentation has survived without it for a long time. But I > think it'd would be a nice clarification; it would convey that you can't > splat in **kwargs into functions specifying it, for example. > > I expect this to be the format of the signature-for-builtins static data > too, as well as the Argument Clinic input syntax. Sure seems like a > nice idea to use the same syntax everywhere. Particularly allowing as > how it's so nice and readable. > > > An admission: the Python standard library documentation actually uses > *both* square-brackets-for-optional-groups and multiple lines. Sometimes > even for the same function! An example: > > http://docs.python.org/3/library/curses.html#curses.window.addch > > Of the two, I believe the square brackets syntax is far more common. Sorry to make your life more complicated, but unless I'm misunderstanding something, issue 18220 (http://bugs.python.org/issue18220) throws another monkey-wrench in to this. If I'm understanding this discussion correctly, that example: islice(stop) islice(start, stop [, step]) requires the multiple-signature approach. Note also that the python3 documentation has moved away from the [] notation wherever possible. --David From ronaldoussoren at mac.com Sun Jul 7 07:25:23 2013 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Sun, 7 Jul 2013 07:25:23 +0200 Subject: [Python-Dev] Rough idea for adding introspection information for builtins In-Reply-To: <51D8D6F2.80007@hastings.org> References: <5147ED55.30605@hastings.org> <5148374D.6050402@hastings.org> <51D854DD.7080807@hastings.org> <51D8B4D6.3080107@hastings.org> <51D8D6F2.80007@hastings.org> Message-ID: On 7 Jul, 2013, at 4:48, Larry Hastings wrote: > > If we combine that with the admittedly-new "/" indicating "all previous parameters are positional-only", Signature objects use a name in angled brackets to indicate that a parameter is positional only, for example "input()". That might be an alternative to adding a "/" in the argument list in pydoc's output. Ronald From chris at simplistix.co.uk Sun Jul 7 08:40:01 2013 From: chris at simplistix.co.uk (Chris Withers) Date: Sun, 07 Jul 2013 07:40:01 +0100 Subject: [Python-Dev] lament for the demise of unbound methods In-Reply-To: <51D69F3E.8060103@v.loewis.de> References: <51D55ACB.4050900@simplistix.co.uk> <51D5A5DF.3000103@simplistix.co.uk> <51D69F3E.8060103@v.loewis.de> Message-ID: <51D90D41.1080302@simplistix.co.uk> On 05/07/2013 11:26, "Martin v. L?wis" wrote: > ... >>>> A.__getattribute__(A,'s') > >>>> A.__getattribute__(A,'c') > >>>> A.__getattribute__(A,'r') > Okay, but with this line: found = found.__getattribute__(found, n) I get a tonne of failures like this: File "", line 2, in r.replace('testfixtures.tests.sample1.z',test_z) File "/Users/chris/LocalGIT/testfixtures/testfixtures/replace.py", line 50, in replace container, method, attribute, t_obj = resolve(target) File "/Users/chris/LocalGIT/testfixtures/testfixtures/resolve.py", line 17, in resolve found = found.__getattribute__(found, n) TypeError: expected 1 arguments, got 2 If I change it to : found = found.__getattribute__(n) I get fewer failures, but still plenty, now of the form: File "", line 1, in r.replace('testfixtures.tests.sample1.X.y',test_y) File "/Users/chris/LocalGIT/testfixtures/testfixtures/replace.py", line 50, in replace container, method, attribute, t_obj = resolve(target) File "/Users/chris/LocalGIT/testfixtures/testfixtures/resolve.py", line 17, in resolve found = found.__getattribute__(n) TypeError: expected 1 arguments, got 0 ...so I'm back to: found = found.__dict__[n] ...and having to catch both KeyError and AttributeError. Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk From cf.natali at gmail.com Sun Jul 7 09:30:11 2013 From: cf.natali at gmail.com (=?ISO-8859-1?Q?Charles=2DFran=E7ois_Natali?=) Date: Sun, 7 Jul 2013 09:30:11 +0200 Subject: [Python-Dev] PEP 446: Add new parameters to configure the inherance of files and for non-blocking sockets In-Reply-To: <20130707003056.GA55329@cskk.homeip.net> References: <20130707003056.GA55329@cskk.homeip.net> Message-ID: 2013/7/7 Cameron Simpson : > On 06Jul2013 11:23, Charles-Fran?ois Natali wrote: > | > I've read your "Rejected Alternatives" more closely and Ulrich > | > Drepper's article, though I think the article also supports adding > | > a blocking (default True) parameter to open() and os.open(). If you > | > try to change that default on a platform where it doesn't work, an > | > exception should be raised. > | > | Contrarily to close-on-exec, non-blocking only applies to a limited > | type of files (e.g. it doesn't work for regular files, which represent > | 90% of open() use cases). > > sockets, pipes, serial devices, ... How do you use open() on a socket (which are already covered by socket(blocking=...)? Also, I said *regular files* - for which O_NONBLOCK doesn't make sense - represent 90% of io.open() use cases, and stand by this claim. Nothing prevents you from setting the FD non-blocking manually. > And you can set it on anything. Just because some things don't block > anyway isn't really a counter argument. Well, it complicates the signature and implementation. If we go the same way, why stop there and not expose O_DSYNC, O_SYNC, O_DIRECT... When using a high-level API like io.open(), I think we should only expose portable flags, which are supported both on all operating systems (like the 'x' O_EXCL flag added in 3.3) and file types. If you want precise control over the open() sementics, os.open() is the way to go (that's also the rationale behind io.open() opener argument, see http://bugs.python.org/issue12105) cf From a.cavallo at cavallinux.eu Sun Jul 7 12:17:20 2013 From: a.cavallo at cavallinux.eu (Antonio Cavallo) Date: Sun, 7 Jul 2013 11:17:20 +0100 Subject: [Python-Dev] Accepting PEP 445 In-Reply-To: References: <20130706225454.0932f6ba@fsol> Message-ID: Yes that is indeed a great news. Having to debug some binary only extension modules that will make my (rather selfish) life so much easier ;) Thanks On 7 Jul 2013, at 00:04, Victor Stinner wrote: > 2013/7/6 Antonio Cavallo : >> Could that remove the need for the --with-pydebug flag? > > With the PEP 445, you still have to recompile Python with > --with-debug, but you don't have to recompile Python extensions > anymore. Debug checks on memory allocators are now implemented as > hooks, instead of using C macros. > > I see this as an improvment ;-) > > Victor From larry at hastings.org Sun Jul 7 13:35:30 2013 From: larry at hastings.org (Larry Hastings) Date: Sun, 07 Jul 2013 13:35:30 +0200 Subject: [Python-Dev] Rough idea for adding introspection information for builtins In-Reply-To: References: <5147ED55.30605@hastings.org> <5148374D.6050402@hastings.org> <51D854DD.7080807@hastings.org> <51D8B4D6.3080107@hastings.org> <51D8D6F2.80007@hastings.org> Message-ID: <51D95282.8030401@hastings.org> On 07/07/2013 07:19 AM, Ronald Oussoren wrote: > Not entirely on topic, but close enough: pydoc currently doesn't use > the __signature__ information at all. Adding such support would be > easy enough, see #17053 for an implementation ;-) True, it doesn't use inspect.signature, it uses inspect.getfullargspec. Since I don't propose modifying inspect.getfullargspec to add the optional parameter group information, 17053 or something like it would have to happen. On 07/07/2013 07:25 AM, R. David Murray wrote: > Sorry to make your life more complicated, but unless I'm misunderstanding > something, issue 18220 (http://bugs.python.org/issue18220) throws another > monkey-wrench in to this. If I'm understanding this discussion correctly, > that example: > > islice(stop) > islice(start, stop [, step]) > > requires the multiple-signature approach. > > Note also that the python3 documentation has moved away from the [] > notation wherever possible. It depends on what problem you're addressing. In terms of the Argument Clinic DSL, and in terms of the static introspection information stored for builtins, someone (Nick?) suggested a refinement to the semantics: in the face of ambiguity, prefer the leftmost group(s) first. That means that range() and islice() could be specified as follows: range([start,] stop, [step]) In terms of the documentation, it might be better to preserve the multiple-lines approach, as perhaps that's more obvious to the reader. On the other hand: in Python 3, help(itertools.islice) uses solely the optional group syntax, on one line. On 07/07/2013 07:25 AM, Ronald Oussoren wrote: > Signature objects use a name in angled brackets to indicate that a parameter is positional only, for example "input()". That might be an alternative to adding a "/" in the argument list in pydoc's output. I wasn't aware that Signature objects currently had any support whatsoever for positional-only parameters. Yes, in theory they do, but in practice they have never seen one, because positional-only parameters only occur in builtins and Signature objects have no metadata for builtins. (The very problem Argument Clinic eventually hopes to solve!) Can you cite an example of this, so I may examine it? //arry/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From ronaldoussoren at mac.com Sun Jul 7 13:42:30 2013 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Sun, 7 Jul 2013 13:42:30 +0200 Subject: [Python-Dev] Rough idea for adding introspection information for builtins In-Reply-To: <51D95282.8030401@hastings.org> References: <5147ED55.30605@hastings.org> <5148374D.6050402@hastings.org> <51D854DD.7080807@hastings.org> <51D8B4D6.3080107@hastings.org> <51D8D6F2.80007@hastings.org> <51D95282.8030401@hastings.org> Message-ID: <5272EBDB-81E1-4B68-B899-242AC8A25D9F@mac.com> On 7 Jul, 2013, at 13:35, Larry Hastings wrote: > > On 07/07/2013 07:25 AM, Ronald Oussoren wrote: >> Signature objects use a name in angled brackets to indicate that a parameter is positional only, for example "input()". That might be an alternative to adding a "/" in the argument list in pydoc's output. >> > > I wasn't aware that Signature objects currently had any support whatsoever for positional-only parameters. Yes, in theory they do, but in practice they have never seen one, because positional-only parameters only occur in builtins and Signature objects have no metadata for builtins. (The very problem Argument Clinic eventually hopes to solve!) > > Can you cite an example of this, so I may examine it? I have a branch of PyObjC that uses this: . That branch isn't quite stable yet, but does add a __signature__ slot to objc.selector and objc.function (basicly methods of Cocoa classes and automaticly wrapped global functions), both of which only have positional-only arguments. With the patch for pydoc/inspect I mentioned earlier I can then generate somewhat useful documentation for Cocoa classes using pydoc. A word of warning though: the PyObjC source code isn't the most approachable, the code that generates the Signature object is actually in python (callable_signature in pyobjc-core/Lib/objc/_callable_docstr.py) Ronald > > > /arry From victor.stinner at gmail.com Sun Jul 7 15:21:01 2013 From: victor.stinner at gmail.com (Victor Stinner) Date: Sun, 7 Jul 2013 15:21:01 +0200 Subject: [Python-Dev] PEP 446: Add new parameters to configure the inherance of files and for non-blocking sockets In-Reply-To: References: <20130707003056.GA55329@cskk.homeip.net> Message-ID: 2013/7/7 Charles-Fran?ois Natali : > 2013/7/7 Cameron Simpson : >> On 06Jul2013 11:23, Charles-Fran?ois Natali wrote: >> | > I've read your "Rejected Alternatives" more closely and Ulrich >> | > Drepper's article, though I think the article also supports adding >> | > a blocking (default True) parameter to open() and os.open(). If you >> | > try to change that default on a platform where it doesn't work, an >> | > exception should be raised. >> | >> | Contrarily to close-on-exec, non-blocking only applies to a limited >> | type of files (e.g. it doesn't work for regular files, which represent >> | 90% of open() use cases). >> >> sockets, pipes, serial devices, ... > > How do you use open() on a socket (which are already covered by > socket(blocking=...)? Also, I said *regular files* - for which > O_NONBLOCK doesn't make sense - represent 90% of io.open() use cases, > and stand by this claim. Nothing prevents you from setting the FD > non-blocking manually. Ok, I think that the best consensus here is to add a new os.set_blocking() function. The function would not be available on Windows, but should be available on all UNIX platforms. The advantage of os.set_blocking() over fcntl is that it may be implemented using ioctl (FIONBIO) to use 1 syscall instead of 2. It is also easier to remember calling "os.set_blocking(fd, True)" instead of "fcntl.fcntl(fd, fcntl.F_SETFL, fcntl.fcntl(fcntl.F_GETFL, fd) | os.O_NONBLOCK)". See the diff: http://hg.python.org/peps/rev/ce61588d244c @Charles-Fran?ois and Cameron: Do you prefer to new version of the PEP? http://www.python.org/dev/peps/pep-0446/ (will be updated in a few minutes) I'm not sure that many people are concerned by the number of syscalls, but there is at least Peter Portante who opened a thread "Modules/socketmodule.c: avoiding second fcntl() call worth the effort?" on this mailing list, last January: http://mail.python.org/pipermail/python-dev/2013-January/123661.html "OpenStack Swift using the Eventlet module, which sets the accepted socket non-blocking, resulting in twice the number of fcntl() calls. Not a killer on performance, but it seems simple enough to save a system call here." (This specific issue will be solved by the new blocking parameter added to socket constructor ;-)) Victor From victor.stinner at gmail.com Sun Jul 7 16:49:20 2013 From: victor.stinner at gmail.com (Victor Stinner) Date: Sun, 7 Jul 2013 16:49:20 +0200 Subject: [Python-Dev] Accepting PEP 445 In-Reply-To: <20130706225454.0932f6ba@fsol> References: <20130706225454.0932f6ba@fsol> Message-ID: 2013/7/6 Antoine Pitrou : > I'm accepting PEP 445 (A C API to customize memory allocators) by > Victor. There is probably some grammar to correct here and there > (neither Victor nor I are native English speakers), but I don't want > this to hold back acceptance. The PEP is an obvious improvement in > flexibility for everyone embedding Python in some external application, > or even recompiling their own Python runtime for specific uses. > > The implementation is tracked in http://bugs.python.org/issue3329 Thanks Antoine for you review :-) I pushed the implementation of the PEP, splitted in different changesets: new functions and then changes to use these new functions. You can now play with the new hooks to hack your own memory debug tool for Python 3.4! You can easily implement your own tool in Python! (if you hook PYMEM_DOMAIN_MEM and PYMEM_DOMAIN_OBJ, but not PYMEM_DOMAIN_RAW) I will share with you my experiment of these new APIs. I'm preparing a new version of pytracemalloc to adapt it to the API of the PEP 445. https://pypi.python.org/pypi/pytracemalloc Victor From Steve.Dower at microsoft.com Sun Jul 7 17:17:21 2013 From: Steve.Dower at microsoft.com (Steve Dower) Date: Sun, 7 Jul 2013 15:17:21 +0000 Subject: [Python-Dev] Hooking into super() attribute resolution In-Reply-To: <46EC256E-9ACE-4C4C-BE3D-325D7DB87B85@mac.com> References: , <46EC256E-9ACE-4C4C-BE3D-325D7DB87B85@mac.com> Message-ID: <0fe31345247846658b331aa358c38c66@BLUPR03MB035.namprd03.prod.outlook.com> Could the same result be achieved by hooking the MRO that super uses and returning a list of proxy objects? And then wouldn't you only really need a __getattribute__ that doesn't recurse (__getlocalattribute__)? The end result may be conceptually simpler, but you've thought through the edge cases better than I have. (Apologies for the HTML top-post) Sent from my Windows Phone ________________________________ From: Ronald Oussoren Sent: ?7/?6/?2013 0:47 To: Ronald Oussoren Cc: python-dev at python.org Dev Subject: Re: [Python-Dev] Hooking into super() attribute resolution I've updated the implementation in issue 18181 while adding some tests, and have updated the proposal as well. The proposal has some open issues at the moment, most important of which is the actual signature for the new special method; in particular I haven't been able to decide if this should be an instance-, class- or static method. It is a static method in the proposal and prototype, but I'm not convinced that that is the right solution. Ronald PEP: TODO Title: Hooking into super attribute resolution Version: $Revision$ Last-Modified: $Date$ Author: Ronald Oussoren Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 12-Jun-2013 Post-History: 2-Jul-2013, ? Abstract ======== In current python releases the attribute resolution of the `super class`_ peeks in the ``__dict__`` attribute of classes on the MRO to look for attributes. This PEP introduces a hook that classes can use to override that behavior for specific classes. Rationale ========= Peeking in the class ``__dict__`` works for regular classes, but can cause problems when a class dynamicly looks up attributes in a ``__getattribute__`` method. The new hook makes it possible to introduce the same customization for attribute lookup through the `super class`_. The superclass attribute lookup hook ==================================== In C code --------- A new slot ``tp_getattro_super`` is added to the ``PyTypeObject`` struct. The ``tp_getattro`` slot for super will call this slot when it is not ``NULL``, and will raise an exception when it is not set (which shouldn't happen because the method is implemented for :class:`object`). The slot has the following prototype:: PyObject* (*getattrosuperfunc)(PyTypeObject* cls, PyObject* name, PyObject* object, PyObject* owner); The function should perform attribute lookup on *object* for *name*, but only looking in type *tp* (which will be one of the types on the MRO for *self*) and without looking in the instance *__dict__*. The function returns ``NULL`` when the attribute cannot be found, and raises and exception. Exception other than ``AttributeError`` will cause failure of super's attribute resolution. The implementation of the slot for the :class:`object` type is ``PyObject_GenericGetAttrSuper``, which peeks in the ``tp_dict`` for *cls*. Note that *owner* and *object* will be the same object when using a class-mode super. In Python code -------------- A Python class can contain a definition for a static method ``__getattribute_super__`` with the following prototype:: def __getattribute_super__(cls, name, object, owner): pass The method should perform attribute lookup for *name* on instance *self* while only looking at *cls* (it should not look in super classes or the instance *__dict__* XXX: I haven't got a clue at the moment if the method should be an instance-, class- or staticmethod. The prototype uses a staticmethod. XXX: My prototype automagicly makes this a static method, just like __new__ is made into a static method. That's more convenient, but also (too?) magical. XXX: Should this raise AttributeError or return a magic value to signal that an attribute cannot be found (such as NotImplemented, used in the comparison operators)? I'm currently using an exception, a magical return value would be slightly more efficient because the exception machinery is not invoked. Alternative proposals --------------------- Reuse ``tp_getattro`` ..................... It would be nice to avoid adding a new slot, thus keeping the API simpler and easier to understand. A comment on `Issue 18181`_ asked about reusing the ``tp_getattro`` slot, that is super could call the ``tp_getattro`` slot of all methods along the MRO. AFAIK that won't work because ``tp_getattro`` will look in the instance ``__dict__`` before it tries to resolve attributes using classes in the MRO. This would mean that using ``tp_getattro`` instead of peeking the class dictionaries changes the semantics of the `super class`_. Open Issues =========== * The names of the new slot and magic method are far from settled. * I'm not too happy with the prototype for the new hook. * Should ``__getattribute_super__`` be a class method instead? -> Yes? The method looks up a named attribute name of an object in a specific class. Is also likely needed to deal with @classmethod and super(Class, Class) * Should ``__getattribute_super__`` be defined on object? -> Yes: makes it easier to delegate to the default implementation * This doesn't necessarily work for class method super class (e.g. super(object, object))... References ========== * `Issue 18181`_ contains a prototype implementation Copyright ========= This document has been placed in the public domain. .. _`Issue 18181`: http://bugs.python.org/issue18181 .. _`super class`: http://docs.python.org/3/library/functions.html?highlight=super#super _______________________________________________ Python-Dev mailing list Python-Dev at python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/steve.dower%40microsoft.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From larry at hastings.org Sun Jul 7 19:20:02 2013 From: larry at hastings.org (Larry Hastings) Date: Sun, 07 Jul 2013 19:20:02 +0200 Subject: [Python-Dev] Rough idea for adding introspection information for builtins In-Reply-To: <5272EBDB-81E1-4B68-B899-242AC8A25D9F@mac.com> References: <5147ED55.30605@hastings.org> <5148374D.6050402@hastings.org> <51D854DD.7080807@hastings.org> <51D8B4D6.3080107@hastings.org> <51D8D6F2.80007@hastings.org> <51D95282.8030401@hastings.org> <5272EBDB-81E1-4B68-B899-242AC8A25D9F@mac.com> Message-ID: <51D9A342.4060404@hastings.org> On 07/07/2013 01:42 PM, Ronald Oussoren wrote: > On 7 Jul, 2013, at 13:35, Larry Hastings wrote: >> On 07/07/2013 07:25 AM, Ronald Oussoren wrote: >>> Signature objects use a name in angled brackets to indicate that a parameter is positional only, for example "input()". That might be an alternative to adding a "/" in the argument list in pydoc's output. >>> >> I wasn't aware that Signature objects currently had any support whatsoever for positional-only parameters. Yes, in theory they do, but in practice they have never seen one, because positional-only parameters only occur in builtins and Signature objects have no metadata for builtins. (The very problem Argument Clinic eventually hopes to solve!) >> >> Can you cite an example of this, so I may examine it? > I have a branch of PyObjC that uses this: . That branch isn't quite stable yet, but does add a __signature__ slot to objc.selector and objc.function (basicly methods of Cocoa classes and automaticly wrapped global functions), both of which only have positional-only arguments. With the patch for pydoc/inspect I mentioned earlier I can then generate somewhat useful documentation for Cocoa classes using pydoc. > > A word of warning though: the PyObjC source code isn't the most approachable, the code that generates the Signature object is actually in python (callable_signature in pyobjc-core/Lib/objc/_callable_docstr.py) Ah. In other words, you have proposed it yourself in an external project. I thought you were saying this was something Python itself already did. In that case, I think I will stick with Guido's suggested syntax. Consider window.border in the curses module: eight positional-only parameters, each in its own optional parameter group. Adding sixteen angle-brackets to that already unreadable morass will make it even worse. But with "/" we add only a single extra character, in an easy-to-find place (the end). //arry/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From ronaldoussoren at mac.com Sun Jul 7 21:15:23 2013 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Sun, 7 Jul 2013 21:15:23 +0200 Subject: [Python-Dev] Rough idea for adding introspection information for builtins In-Reply-To: <51D9A342.4060404@hastings.org> References: <5147ED55.30605@hastings.org> <5148374D.6050402@hastings.org> <51D854DD.7080807@hastings.org> <51D8B4D6.3080107@hastings.org> <51D8D6F2.80007@hastings.org> <51D95282.8030401@hastings.org> <5272EBDB-81E1-4B68-B899-242AC8A25D9F@mac.com> <51D9A342.4060404@hastings.org> Message-ID: <476B98A3-4701-4964-B2BE-AE0ABF75E29E@mac.com> On 7 Jul, 2013, at 19:20, Larry Hastings wrote: > On 07/07/2013 01:42 PM, Ronald Oussoren wrote: >> On 7 Jul, 2013, at 13:35, Larry Hastings >> wrote: >> >>> On 07/07/2013 07:25 AM, Ronald Oussoren wrote: >>> >>>> Signature objects use a name in angled brackets to indicate that a parameter is positional only, for example "input()". That might be an alternative to adding a "/" in the argument list in pydoc's output. >>>> >>>> >>> I wasn't aware that Signature objects currently had any support whatsoever for positional-only parameters. Yes, in theory they do, but in practice they have never seen one, because positional-only parameters only occur in builtins and Signature objects have no metadata for builtins. (The very problem Argument Clinic eventually hopes to solve!) >>> >>> Can you cite an example of this, so I may examine it? >>> >> I have a branch of PyObjC that uses this: >> . That branch isn't quite stable yet, but does add a __signature__ slot to objc.selector and objc.function (basicly methods of Cocoa classes and automaticly wrapped global functions), both of which only have positional-only arguments. With the patch for pydoc/inspect I mentioned earlier I can then generate somewhat useful documentation for Cocoa classes using pydoc. >> >> A word of warning though: the PyObjC source code isn't the most approachable, the code that generates the Signature object is actually in python (callable_signature in pyobjc-core/Lib/objc/_callable_docstr.py) >> > > Ah. In other words, you have proposed it yourself in an external project. I thought you were saying this was something Python itself already did. I wasn't clear enough in what I wrote. The stdlib contains support for positional-only arguments in Signature objects (see Lib/inspect.py, line 1472, which says "_POSITIONAL_ONLY = _ParameterKind(0, name='POSITIONAL_ONLY')". The __str__ of Parameter amongst other says: if kind == _POSITIONAL_ONLY: if formatted is None: formatted = '' formatted = '<{}>'.format(formatted) That is, it adds angled brackets around the names of positional-only parameters. I pointed to PyObjC as an example of code that actually creates Signature objects with positional-only arguments, as far as I know the stdlib never does this because the stdlib can only create signatures for plain python functions and those cannot have such arguments. > In that case, I think I will stick with Guido's suggested syntax. Consider window.border in the curses module: eight positional-only parameters, each in its own optional parameter group. Adding sixteen angle-brackets to that already unreadable morass will make it even worse. But with "/" we add only a single extra character, in an easy-to-find place (the end). Using Guido's suggestion is fine by me, I agree that there is a clear risk of angle-bracket overload for functions with a lot of arguments. I do think that the __str__ for Signatures should be changed to match the convention. And to be clear: I'm looking forward to having Argument Clinic and __signature__ objects on built-in functions, "funcname(...)" in the output pydoc is somewhat annoying, especially for extensions where the author hasn't bothered to provide a docstring. That's one reason I wrote the __signature__ support in PyObjC in the first place (and the patch for pydoc to actually use the signature information) Ronald From ronaldoussoren at mac.com Sun Jul 7 21:36:43 2013 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Sun, 7 Jul 2013 21:36:43 +0200 Subject: [Python-Dev] Hooking into super() attribute resolution In-Reply-To: <0fe31345247846658b331aa358c38c66@BLUPR03MB035.namprd03.prod.outlook.com> References: <46EC256E-9ACE-4C4C-BE3D-325D7DB87B85@mac.com> <0fe31345247846658b331aa358c38c66@BLUPR03MB035.namprd03.prod.outlook.com> Message-ID: On 7 Jul, 2013, at 17:17, Steve Dower wrote: > Could the same result be achieved by hooking the MRO that super uses and returning a list of proxy objects? What is the advantage over adding a hook to the class itself? That seems to be the right place to add such a hook, super already looks in the classes along the MRO and my proposal would add a formal interface for that instead of having super peek into the class __dict__. I have thought of using a custom mapping object for the tp_dict slot to intercept this, but that won't work because super assumes that tp_dict is an actual PyDictObject (and likely other parts of the interpreter do so as well). > And then wouldn't you only really need a __getattribute__ that doesn't recurse (__getlocalattribute__)? The end result may be conceptually simpler, but you've thought through the edge cases better than I have. __getattribute_super__ already is a kind of __getlocalattribute__, the primairy difference being __getattribute_super__ is a staticmethod instead of an instance method. To be honest I'm not sure if a staticmethod is the right solution, I'm having a hard time to determine if this should be a class, instance or static method. Currently super(StartClass, x) basicly does (assuming x is an instance method): def __getattribute__(self, name): mro = type(x).mro() idx = mro.index(StartClass) while idx < len(mro): dct = mro[idx].__dict__ try: result = dct[name] # deal with descriptors here return result except KeyError: continue return object.__getattribute__(self, name) With my proposal 'dct' would no longer be needed and 'result = dct[name]' would be 'mro[idx].__getattribute_super__(mro[idx], name, x, StartClass)' (I may have the last argument for the call to __getattribute_super__ wrong, but that's the idea). Given that the first argument of __get...super__ is the same as the object the method get getattr-ed from I guess the method should be a classmethod instead of an staticmethod. Changing that would be easy enough. I'm still interested in feedback on the basic idea, I'd love to here that my proposal isn't necessary because there is already a way to get the behavior I'm looking for although that's nog going to happen ;-). Ronald > > (Apologies for the HTML top-post) I don't mind. PS. Does anyone know if the pep editors are away (conferences, holidays, ...)? I could just check in my proposal in the peps repository, but as this is my first PEP I'd prefer to follow the documented procedure and have someone that knows what he's doing look at the metadata before checking in. > > Sent from my Windows Phone > From: Ronald Oussoren > Sent: ?7/?6/?2013 0:47 > To: Ronald Oussoren > Cc: python-dev at python.org Dev > Subject: Re: [Python-Dev] Hooking into super() attribute resolution > > I've updated the implementation in issue 18181 while adding some tests, and have updated the proposal as well. > > The proposal has some open issues at the moment, most important of which is the actual signature for the new special method; in particular I haven't been able to decide if this should be an instance-, class- or static method. It is a static method in the proposal and prototype, but I'm not convinced that that is the right solution. > > Ronald > > > > > PEP: TODO > Title: Hooking into super attribute resolution > Version: $Revision$ > Last-Modified: $Date$ > Author: Ronald Oussoren > Status: Draft > Type: Standards Track > Content-Type: text/x-rst > Created: 12-Jun-2013 > Post-History: 2-Jul-2013, ? > > > Abstract > ======== > > In current python releases the attribute resolution of the `super class`_ > peeks in the ``__dict__`` attribute of classes on the MRO to look > for attributes. This PEP introduces a hook that classes can use > to override that behavior for specific classes. > > > Rationale > ========= > > Peeking in the class ``__dict__`` works for regular classes, but can > cause problems when a class dynamicly looks up attributes in a > ``__getattribute__`` method. > > The new hook makes it possible to introduce the same customization for > attribute lookup through the `super class`_. > > > The superclass attribute lookup hook > ==================================== > > In C code > --------- > > A new slot ``tp_getattro_super`` is added to the ``PyTypeObject`` struct. The > ``tp_getattro`` slot for super will call this slot when it is not ``NULL``, > and will raise an exception when it is not set (which shouldn't happen because > the method is implemented for :class:`object`). > > The slot has the following prototype:: > > PyObject* (*getattrosuperfunc)(PyTypeObject* cls, PyObject* name, > PyObject* object, PyObject* owner); > > The function should perform attribute lookup on *object* for *name*, but only > looking in type *tp* (which will be one of the types on the MRO for *self*) > and without looking in the instance *__dict__*. > > The function returns ``NULL`` when the attribute cannot be found, and raises and > exception. Exception other than ``AttributeError`` will cause failure of super's > attribute resolution. > > The implementation of the slot for the :class:`object` type is > ``PyObject_GenericGetAttrSuper``, which peeks in the ``tp_dict`` for *cls*. > > Note that *owner* and *object* will be the same object when using a > class-mode super. > > > In Python code > -------------- > > A Python class can contain a definition for a static method > ``__getattribute_super__`` with the following prototype:: > > def __getattribute_super__(cls, name, object, owner): pass > > The method should perform attribute lookup for *name* on instance *self* while > only looking at *cls* (it should not look in super classes or the instance > *__dict__* > > XXX: I haven't got a clue at the moment if the method should be an > instance-, class- or staticmethod. The prototype uses a staticmethod. > > XXX: My prototype automagicly makes this a static method, just like __new__ is > made into a static method. That's more convenient, but also (too?) magical. > > XXX: Should this raise AttributeError or return a magic value to signal that > an attribute cannot be found (such as NotImplemented, used in the comparison > operators)? I'm currently using an exception, a magical return value would > be slightly more efficient because the exception machinery is not invoked. > > > Alternative proposals > --------------------- > > Reuse ``tp_getattro`` > ..................... > > It would be nice to avoid adding a new slot, thus keeping the API simpler and > easier to understand. A comment on `Issue 18181`_ asked about reusing the > ``tp_getattro`` slot, that is super could call the ``tp_getattro`` slot of all > methods along the MRO. > > AFAIK that won't work because ``tp_getattro`` will look in the instance > ``__dict__`` before it tries to resolve attributes using classes in the MRO. > This would mean that using ``tp_getattro`` instead of peeking the class > dictionaries changes the semantics of the `super class`_. > > > Open Issues > =========== > > * The names of the new slot and magic method are far from settled. > > * I'm not too happy with the prototype for the new hook. > > * Should ``__getattribute_super__`` be a class method instead? > > -> Yes? The method looks up a named attribute name of an object in > a specific class. Is also likely needed to deal with @classmethod > and super(Class, Class) > > * Should ``__getattribute_super__`` be defined on object? > > -> Yes: makes it easier to delegate to the default implementation > > * This doesn't necessarily work for class method super class > (e.g. super(object, object))... > > > References > ========== > > * `Issue 18181`_ contains a prototype implementation > > > Copyright > ========= > > This document has been placed in the public domain. > > .. _`Issue 18181`: http://bugs.python.org/issue18181 > > .. _`super class`: http://docs.python.org/3/library/functions.html?highlight=super#super > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/steve.dower%40microsoft.com > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/ronaldoussoren%40mac.com From tjreedy at udel.edu Sun Jul 7 23:37:55 2013 From: tjreedy at udel.edu (Terry Reedy) Date: Sun, 07 Jul 2013 17:37:55 -0400 Subject: [Python-Dev] Rough idea for adding introspection information for builtins In-Reply-To: <51D95282.8030401@hastings.org> References: <5147ED55.30605@hastings.org> <5148374D.6050402@hastings.org> <51D854DD.7080807@hastings.org> <51D8B4D6.3080107@hastings.org> <51D8D6F2.80007@hastings.org> <51D95282.8030401@hastings.org> Message-ID: On 7/7/2013 7:35 AM, Larry Hastings wrote: > > > On 07/07/2013 07:19 AM, Ronald Oussoren wrote: >> Not entirely on topic, but close enough: pydoc currently doesn't use >> the __signature__ information at all. Adding such support would be >> easy enough, see #17053 for an implementation ;-) > > True, it doesn't use inspect.signature, it uses inspect.getfullargspec. This is currently true of Idle calltips. But with 3.2 out of the way, I plan to switch sometime and not worry about 2.7. > Since I don't propose modifying inspect.getfullargspec to add the > optional parameter group information, 17053 or something like it would > have to happen. > > > On 07/07/2013 07:25 AM, R. David Murray wrote: >> Sorry to make your life more complicated, but unless I'm misunderstanding >> something, issue 18220 (http://bugs.python.org/issue18220) throws another >> monkey-wrench in to this. If I'm understanding this discussion correctly, >> that example: >> >> islice(stop) >> islice(start, stop [, step]) >> >> requires the multiple-signature approach. >> >> Note also that the python3 documentation has moved away from the [] >> notation wherever possible. > > It depends on what problem you're addressing. In terms of the Argument > Clinic DSL, and in terms of the static introspection information stored > for builtins, someone (Nick?) suggested a refinement to the semantics: > in the face of ambiguity, prefer the leftmost group(s) first. That > means that range() and islice() could be specified as follows: > > range([start,] stop, [step]) This is how it was until last September. See #15831, which also changed max, min, and slice entries to use two lines. The multiple lines for bytes and str signatures in the docstrings were another issue. > In terms of the documentation, it might be better to preserve the > multiple-lines approach, as perhaps that's more obvious to the reader. It seems that signatures that do not match what one can do with a def statement are confusing. The header line for the Python version of the above is def range(start_or_stop, stop=None, step=1): My suggestion to use this, which is the actual signature, was rejected in favor of using two lines. This is fine with me, as it documents the calling signatures rather than the hybrid definition signature used to implement the call signatures. > On the other hand: in Python 3, help(itertools.islice) uses solely the > optional group syntax, on one line. Because it has not yet been changed ;-). > On 07/07/2013 07:25 AM, Ronald Oussoren wrote: >> Signature objects use a name in angled brackets to indicate that a parameter is positional only, for example "input()". That might be an alternative to adding a "/" in the argument list in pydoc's output. > > I wasn't aware that Signature objects currently had any support > whatsoever for positional-only parameters. Yes, in theory they do, but > in practice they have never seen one, because positional-only parameters > only occur in builtins and Signature objects have no metadata for > builtins. (The very problem Argument Clinic eventually hopes to solve!) I look forward to the day when accurate (auto-generated) call data is also available for C-coded functions. -- Terry Jan Reedy From Steve.Dower at microsoft.com Mon Jul 8 17:19:42 2013 From: Steve.Dower at microsoft.com (Steve Dower) Date: Mon, 8 Jul 2013 15:19:42 +0000 Subject: [Python-Dev] Hooking into super() attribute resolution In-Reply-To: References: <46EC256E-9ACE-4C4C-BE3D-325D7DB87B85@mac.com> <0fe31345247846658b331aa358c38c66@BLUPR03MB035.namprd03.prod.outlook.com>, Message-ID: <5fe3649f4c3c41ba927ca62fd762699c@BLUPR03MB035.namprd03.prod.outlook.com> The only real advantage is a simpler signature and more easily explained use (assuming the person you're explaining it to is familiar with metaclasses, so most of the hard explaining has been done). I'm still not sure that this isn't simply a bug in super. If the superclass's metaclass provides a __getattr__ then it should probably use it and abandon it's own MRO traversal. I still haven't thought the edge cases through, and it seems like there'd be some with that change, so that's where __getattribute_super__ comes in - super can call it without abandoning its MRO traversal. AFAICT, the difference between that and __getlocalattribute__ is that the latter would be implemented on a metaclass while the former takes extra parameters. I think this functionality is advanced enough that requiring a metaclass isn't unreasonable. (The proxy objects idea was a red herring, sorry :) ) Steve Sent from my Windows Phone ________________________________ From: Ronald Oussoren Sent: ?7/?7/?2013 12:37 To: Steve Dower Cc: python-dev at python.org Subject: Re: [Python-Dev] Hooking into super() attribute resolution On 7 Jul, 2013, at 17:17, Steve Dower wrote: > Could the same result be achieved by hooking the MRO that super uses and returning a list of proxy objects? What is the advantage over adding a hook to the class itself? That seems to be the right place to add such a hook, super already looks in the classes along the MRO and my proposal would add a formal interface for that instead of having super peek into the class __dict__. I have thought of using a custom mapping object for the tp_dict slot to intercept this, but that won't work because super assumes that tp_dict is an actual PyDictObject (and likely other parts of the interpreter do so as well). > And then wouldn't you only really need a __getattribute__ that doesn't recurse (__getlocalattribute__)? The end result may be conceptually simpler, but you've thought through the edge cases better than I have. __getattribute_super__ already is a kind of __getlocalattribute__, the primairy difference being __getattribute_super__ is a staticmethod instead of an instance method. To be honest I'm not sure if a staticmethod is the right solution, I'm having a hard time to determine if this should be a class, instance or static method. Currently super(StartClass, x) basicly does (assuming x is an instance method): def __getattribute__(self, name): mro = type(x).mro() idx = mro.index(StartClass) while idx < len(mro): dct = mro[idx].__dict__ try: result = dct[name] # deal with descriptors here return result except KeyError: continue return object.__getattribute__(self, name) With my proposal 'dct' would no longer be needed and 'result = dct[name]' would be 'mro[idx].__getattribute_super__(mro[idx], name, x, StartClass)' (I may have the last argument for the call to __getattribute_super__ wrong, but that's the idea). Given that the first argument of __get...super__ is the same as the object the method get getattr-ed from I guess the method should be a classmethod instead of an staticmethod. Changing that would be easy enough. I'm still interested in feedback on the basic idea, I'd love to here that my proposal isn't necessary because there is already a way to get the behavior I'm looking for although that's nog going to happen ;-). Ronald > > (Apologies for the HTML top-post) I don't mind. PS. Does anyone know if the pep editors are away (conferences, holidays, ...)? I could just check in my proposal in the peps repository, but as this is my first PEP I'd prefer to follow the documented procedure and have someone that knows what he's doing look at the metadata before checking in. > > Sent from my Windows Phone > From: Ronald Oussoren > Sent: ?7/?6/?2013 0:47 > To: Ronald Oussoren > Cc: python-dev at python.org Dev > Subject: Re: [Python-Dev] Hooking into super() attribute resolution > > I've updated the implementation in issue 18181 while adding some tests, and have updated the proposal as well. > > The proposal has some open issues at the moment, most important of which is the actual signature for the new special method; in particular I haven't been able to decide if this should be an instance-, class- or static method. It is a static method in the proposal and prototype, but I'm not convinced that that is the right solution. > > Ronald > > > > > PEP: TODO > Title: Hooking into super attribute resolution > Version: $Revision$ > Last-Modified: $Date$ > Author: Ronald Oussoren > Status: Draft > Type: Standards Track > Content-Type: text/x-rst > Created: 12-Jun-2013 > Post-History: 2-Jul-2013, ? > > > Abstract > ======== > > In current python releases the attribute resolution of the `super class`_ > peeks in the ``__dict__`` attribute of classes on the MRO to look > for attributes. This PEP introduces a hook that classes can use > to override that behavior for specific classes. > > > Rationale > ========= > > Peeking in the class ``__dict__`` works for regular classes, but can > cause problems when a class dynamicly looks up attributes in a > ``__getattribute__`` method. > > The new hook makes it possible to introduce the same customization for > attribute lookup through the `super class`_. > > > The superclass attribute lookup hook > ==================================== > > In C code > --------- > > A new slot ``tp_getattro_super`` is added to the ``PyTypeObject`` struct. The > ``tp_getattro`` slot for super will call this slot when it is not ``NULL``, > and will raise an exception when it is not set (which shouldn't happen because > the method is implemented for :class:`object`). > > The slot has the following prototype:: > > PyObject* (*getattrosuperfunc)(PyTypeObject* cls, PyObject* name, > PyObject* object, PyObject* owner); > > The function should perform attribute lookup on *object* for *name*, but only > looking in type *tp* (which will be one of the types on the MRO for *self*) > and without looking in the instance *__dict__*. > > The function returns ``NULL`` when the attribute cannot be found, and raises and > exception. Exception other than ``AttributeError`` will cause failure of super's > attribute resolution. > > The implementation of the slot for the :class:`object` type is > ``PyObject_GenericGetAttrSuper``, which peeks in the ``tp_dict`` for *cls*. > > Note that *owner* and *object* will be the same object when using a > class-mode super. > > > In Python code > -------------- > > A Python class can contain a definition for a static method > ``__getattribute_super__`` with the following prototype:: > > def __getattribute_super__(cls, name, object, owner): pass > > The method should perform attribute lookup for *name* on instance *self* while > only looking at *cls* (it should not look in super classes or the instance > *__dict__* > > XXX: I haven't got a clue at the moment if the method should be an > instance-, class- or staticmethod. The prototype uses a staticmethod. > > XXX: My prototype automagicly makes this a static method, just like __new__ is > made into a static method. That's more convenient, but also (too?) magical. > > XXX: Should this raise AttributeError or return a magic value to signal that > an attribute cannot be found (such as NotImplemented, used in the comparison > operators)? I'm currently using an exception, a magical return value would > be slightly more efficient because the exception machinery is not invoked. > > > Alternative proposals > --------------------- > > Reuse ``tp_getattro`` > ..................... > > It would be nice to avoid adding a new slot, thus keeping the API simpler and > easier to understand. A comment on `Issue 18181`_ asked about reusing the > ``tp_getattro`` slot, that is super could call the ``tp_getattro`` slot of all > methods along the MRO. > > AFAIK that won't work because ``tp_getattro`` will look in the instance > ``__dict__`` before it tries to resolve attributes using classes in the MRO. > This would mean that using ``tp_getattro`` instead of peeking the class > dictionaries changes the semantics of the `super class`_. > > > Open Issues > =========== > > * The names of the new slot and magic method are far from settled. > > * I'm not too happy with the prototype for the new hook. > > * Should ``__getattribute_super__`` be a class method instead? > > -> Yes? The method looks up a named attribute name of an object in > a specific class. Is also likely needed to deal with @classmethod > and super(Class, Class) > > * Should ``__getattribute_super__`` be defined on object? > > -> Yes: makes it easier to delegate to the default implementation > > * This doesn't necessarily work for class method super class > (e.g. super(object, object))... > > > References > ========== > > * `Issue 18181`_ contains a prototype implementation > > > Copyright > ========= > > This document has been placed in the public domain. > > .. _`Issue 18181`: http://bugs.python.org/issue18181 > > .. _`super class`: http://docs.python.org/3/library/functions.html?highlight=super#super > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/steve.dower%40microsoft.com > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/ronaldoussoren%40mac.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From ronaldoussoren at mac.com Mon Jul 8 17:57:47 2013 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Mon, 8 Jul 2013 17:57:47 +0200 Subject: [Python-Dev] Hooking into super() attribute resolution In-Reply-To: <5fe3649f4c3c41ba927ca62fd762699c@BLUPR03MB035.namprd03.prod.outlook.com> References: <46EC256E-9ACE-4C4C-BE3D-325D7DB87B85@mac.com> <0fe31345247846658b331aa358c38c66@BLUPR03MB035.namprd03.prod.outlook.com> <5fe3649f4c3c41ba927ca62fd762699c@BLUPR03MB035.namprd03.prod.outlook.com> Message-ID: <7946F238-3DA7-4198-9CED-23B33246B26F@mac.com> On 8 Jul, 2013, at 17:19, Steve Dower wrote: > The only real advantage is a simpler signature and more easily explained use (assuming the person you're explaining it to is familiar with metaclasses, so most of the hard explaining has been done). The signature is as complex as it is to be able to call descr.__get__ with the correct arguments. I ended up with the current signature when I added __getattribute_super__ to object and removed the tp_dict peeking code from super's tp_getattro. A way to get a simpler interface again would be a method that returns an attribute *without* performing calls to descr.__get__. That could then be used for both __getattribute__ and super.__getattribute__, instead of peeking in a class' dictionary. I must admit that I haven't thought about the ramifactions of this (both functionally and performance wise). This might end up being easier to explain: both normal attribute resolution and super's resolution would end up using the same mechanism, with the differences being that super doesn't begin resolution at the start of the mro and ignores the instance __dict__. The disadvantage is introducing a new way to affect attribute resolution (do I use "__getattribute__" or this new method?). The new interface would be something like: @classmethod def __getlocalname__(cls, object, name): pass Or as you mentioned later as a __getlocalname__ method on the metaclass. The "object" argument wouldn't be necessary to reproduce current functionality, and isn't necessary for my usecase as well, but a hook for attribute resolution on an instance that doesn't have access to that instance feels wrong. > > I'm still not sure that this isn't simply a bug in super. If the superclass's metaclass provides a __getattr__ then it should probably use it and abandon it's own MRO traversal. I'd have to think about this, but on first glance this would mean a change in the semantics that a metaclass' __getattr__ currently has. > > I still haven't thought the edge cases through, and it seems like there'd be some with that change, so that's where __getattribute_super__ comes in - super can call it without abandoning its MRO traversal. > > AFAICT, the difference between that and __getlocalattribute__ is that the latter would be implemented on a metaclass while the former takes extra parameters. I think this functionality is advanced enough that requiring a metaclass isn't unreasonable. I'm not necessarily oppossed to a solution that requires using a metaclass, I already have metaclasses with custom metaclasses in PyObjC and this wouldn't add that much complexity to that :-) Ronald From ncoghlan at gmail.com Tue Jul 9 00:59:03 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 9 Jul 2013 08:59:03 +1000 Subject: [Python-Dev] Rough idea for adding introspection information for builtins In-Reply-To: <476B98A3-4701-4964-B2BE-AE0ABF75E29E@mac.com> References: <5147ED55.30605@hastings.org> <5148374D.6050402@hastings.org> <51D854DD.7080807@hastings.org> <51D8B4D6.3080107@hastings.org> <51D8D6F2.80007@hastings.org> <51D95282.8030401@hastings.org> <5272EBDB-81E1-4B68-B899-242AC8A25D9F@mac.com> <51D9A342.4060404@hastings.org> <476B98A3-4701-4964-B2BE-AE0ABF75E29E@mac.com> Message-ID: Since it's relevant: my recollection us that the current use of angle brackets in inspect.Signature is just the default use of them for "there is no canonical representation of this, but leaving them out would be misleading" (I haven't checked if the PEP says that explicitly). I previously forgot Larry, Guido & I discussed the appropriate use of square brackets and the slash in the definition format at PyCon, so I now think having the Argument Clinic PEP also cover their appropriate use in the inspect.Signature string output is a good idea. Cheers, Nick. -------------- next part -------------- An HTML attachment was scrubbed... URL: From Steve.Dower at microsoft.com Tue Jul 9 01:21:52 2013 From: Steve.Dower at microsoft.com (Steve Dower) Date: Mon, 8 Jul 2013 23:21:52 +0000 Subject: [Python-Dev] Hooking into super() attribute resolution In-Reply-To: <7946F238-3DA7-4198-9CED-23B33246B26F@mac.com> References: <46EC256E-9ACE-4C4C-BE3D-325D7DB87B85@mac.com> <0fe31345247846658b331aa358c38c66@BLUPR03MB035.namprd03.prod.outlook.com> <5fe3649f4c3c41ba927ca62fd762699c@BLUPR03MB035.namprd03.prod.outlook.com> <7946F238-3DA7-4198-9CED-23B33246B26F@mac.com> Message-ID: <5339a8b72f264faa99a8720b5d1d161d@BLUPR03MB035.namprd03.prod.outlook.com> > From: Ronald Oussoren [mailto:ronaldoussoren at mac.com] > Sent: Monday, July 8, 2013 0858 > > On 8 Jul, 2013, at 17:19, Steve Dower wrote: > > > The only real advantage is a simpler signature and more easily explained > use (assuming the person you're explaining it to is familiar with metaclasses, > so most of the hard explaining has been done). > > The signature is as complex as it is to be able to call descr.__get__ with the > correct arguments. I ended up with the current signature when I added > __getattribute_super__ to object and removed the tp_dict peeking code > from super's tp_getattro. > > A way to get a simpler interface again would be a method that returns an > attribute *without* performing calls to descr.__get__. That could then be > used for both __getattribute__ and super.__getattribute__, instead of > peeking in a class' dictionary. I must admit that I haven't thought about the > ramifactions of this (both functionally and performance wise). This might > end up being easier to explain: both normal attribute resolution and super's > resolution would end up using the same mechanism, with the differences > being that super doesn't begin resolution at the start of the mro and ignores > the instance __dict__. The disadvantage is introducing a new way to affect > attribute resolution (do I use "__getattribute__" or this new method?). > > The new interface would be something like: > > @classmethod > def __getlocalname__(cls, object, name): > pass > > Or as you mentioned later as a __getlocalname__ method on the metaclass. > The "object" argument wouldn't be necessary to reproduce current > functionality, and isn't necessary for my usecase as well, but a hook for > attribute resolution on an instance that doesn't have access to that instance > feels wrong. Except that if it's on a metaclass, the 'instance' it has access to is cls. The descriptor side of things is more interesting, but I see no reason why super can't do that itself, since it knows the actual instance to call __get__ with. (Presumably it already does this with the __dict__ lookup, since that won't call __get__ either.) Explaining the new method is easiest if the default implementation is (literally): def __getlocalname__(self, name): try: return self.__dict__[name] except KeyError: raise AttributeError(name) which does not do any descriptor resolution (and is only a small step from simply replacing __dict__ with a custom object, which is basically where we started). The only change I've really suggested is making it an instance method that can be implemented on a metaclass if you want it for class members. > > > > I'm still not sure that this isn't simply a bug in super. If the superclass's > metaclass provides a __getattr__ then it should probably use it and abandon > it's own MRO traversal. > > I'd have to think about this, but on first glance this would mean a change in > the semantics that a metaclass' __getattr__ currently has. Exactly. Probably not a great idea to change that. > > > > I still haven't thought the edge cases through, and it seems like there'd be > some with that change, so that's where __getattribute_super__ comes in - > super can call it without abandoning its MRO traversal. > > > > AFAICT, the difference between that and __getlocalattribute__ is that the > latter would be implemented on a metaclass while the former takes extra > parameters. I think this functionality is advanced enough that requiring a > metaclass isn't unreasonable. > > I'm not necessarily oppossed to a solution that requires using a metaclass, I > already have metaclasses with custom metaclasses in PyObjC and this > wouldn't add that much complexity to that :-) I assumed you were - when I was working on similar sort of code they made life extremely easy. > Ronald Steve From victor.stinner at gmail.com Tue Jul 9 01:39:14 2013 From: victor.stinner at gmail.com (Victor Stinner) Date: Tue, 9 Jul 2013 01:39:14 +0200 Subject: [Python-Dev] pyfailmalloc: new debug tool to test how your applications handles MemoryError Message-ID: Hi, The PEP 445 (Add new APIs to customize Python memory allocators) has been accepted, I commited its implementation. So it's time to have fun with this API. I developed a small Python module (150 lines of C code) to inject memory allocation failures: https://pypi.python.org/pypi/pyfailmalloc This module can be used to test your application under very low memory condition, test how your application handles MemoryError. I ran it in the Python test suite: Python core and stdlib do not handle MemoryError correctly everywhere... I opened the following issue to track all bugs found by pytracemalloc and related commits to fix these bugs: http://bugs.python.org/issue18408 I already fixed bugs in PyObject_GC_NewVar(), PyDict_New(), list.pop(), type.__bases__ setter, etc. I also found a bug in _PyMem_DebugRealloc(): a function supposed to detect bugs :-) Have fun with this new tool on your applications and Python modules. Tell me if you would like to help me on issue #18408, there are still a lot of bugs to fix. Victor From victor.stinner at gmail.com Tue Jul 9 01:41:23 2013 From: victor.stinner at gmail.com (Victor Stinner) Date: Tue, 9 Jul 2013 01:41:23 +0200 Subject: [Python-Dev] pyfailmalloc: new debug tool to test how your applications handles MemoryError In-Reply-To: References: Message-ID: 2013/7/9 Victor Stinner : > I developed a small Python module (150 lines of C code) to inject > memory allocation failures: > https://pypi.python.org/pypi/pyfailmalloc Bitbucket was down, so I was unable to give the link to its source code. The server is back, here is the C code: https://bitbucket.org/haypo/pyfailmalloc/src/tip/failmalloc.c You can see that it's now easy to develop such tools thanks to the PEP 445. Victor From ronaldoussoren at mac.com Tue Jul 9 10:25:41 2013 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Tue, 9 Jul 2013 10:25:41 +0200 Subject: [Python-Dev] Hooking into super() attribute resolution In-Reply-To: <5339a8b72f264faa99a8720b5d1d161d@BLUPR03MB035.namprd03.prod.outlook.com> References: <46EC256E-9ACE-4C4C-BE3D-325D7DB87B85@mac.com> <0fe31345247846658b331aa358c38c66@BLUPR03MB035.namprd03.prod.outlook.com> <5fe3649f4c3c41ba927ca62fd762699c@BLUPR03MB035.namprd03.prod.outlook.com> <7946F238-3DA7-4198-9CED-23B33246B26F@mac.com> <5339a8b72f264faa99a8720b5d1d161d@BLUPR03MB035.namprd03.prod.outlook.com> Message-ID: <07A27E22-AC02-46E9-8B02-1A25B94A061D@mac.com> On 9 Jul, 2013, at 1:21, Steve Dower wrote: >> > > Except that if it's on a metaclass, the 'instance' it has access to is cls. The descriptor side of things is more interesting, but I see no reason why super can't do that itself, since it knows the actual instance to call __get__ with. (Presumably it already does this with the __dict__ lookup, since that won't call __get__ either.) > > Explaining the new method is easiest if the default implementation is (literally): > > def __getlocalname__(self, name): > try: > return self.__dict__[name] > except KeyError: > raise AttributeError(name) > > which does not do any descriptor resolution (and is only a small step from simply replacing __dict__ with a custom object, which is basically where we started). The only change I've really suggested is making it an instance method that can be implemented on a metaclass if you want it for class members. I like this idea and will experiment with implementing this later this week. The only thing I'm not sure about is how to indicate that the name could not be found, raising an exception could end up being to expensive if the __getlocalname__ hook gets used in object.__getattribute__ as well. I guess I'll have to run benchmarks to determine if this really is a problem. Ronald From martin at v.loewis.de Tue Jul 9 20:15:13 2013 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 09 Jul 2013 20:15:13 +0200 Subject: [Python-Dev] Accepting PEP 445 In-Reply-To: References: <20130706225454.0932f6ba@fsol> Message-ID: <51DC5331.8010409@v.loewis.de> Am 07.07.13 01:04, schrieb Victor Stinner: > 2013/7/6 Antonio Cavallo : >> Could that remove the need for the --with-pydebug flag? > > With the PEP 445, you still have to recompile Python with > --with-debug, but you don't have to recompile Python extensions > anymore. Really? What about _PyObject_HEAD_EXTRA? The object layout still changes under --with-pydebug, AFAICT. Regards, Martin From christian at python.org Fri Jul 12 01:58:19 2013 From: christian at python.org (Christian Heimes) Date: Fri, 12 Jul 2013 01:58:19 +0200 Subject: [Python-Dev] Python 3.4 and Windows XP: just 45 days until EOL Message-ID: <51DF469B.6010007@python.org> Hi, how do you feel about dropping Windows XP support for Python 3.4? It would enable us to use some features that are only available on Windows Vista and newer, for example http://bugs.python.org/issue6926 and http://bugs.python.org/issue1763 . PEP 11 says: A new feature release X.Y.0 will support all Windows releases whose extended support phase is not yet expired. For Python 3.4 is going to be a very close call. According to PEP 429 3.4.0 final is scheduled for February 22, 2014. The extended support phase of Windows XP ends merely 45 days later on April 8, 2014. Do we really have to restrict ourselves to an API that is going to become deprecated 45 days after the estimated release of 3.4.0? Christian From Steve.Dower at microsoft.com Fri Jul 12 02:11:55 2013 From: Steve.Dower at microsoft.com (Steve Dower) Date: Fri, 12 Jul 2013 00:11:55 +0000 Subject: [Python-Dev] Python 3.4 and Windows XP: just 45 days until EOL In-Reply-To: <51DF469B.6010007@python.org> References: <51DF469B.6010007@python.org> Message-ID: +1. And maybe amend PEP 11 to specify "whose extended support phase does not expire within 6 months of release"? (I picked 6 for no particular reason.) I don't see any good reason for Python to support an OS that Microsoft doesn't, but once 3.4.0 has been released with XP support it can't really be taken out for 3.4.1. Since 3.4.1 is almost certainly going to be after the end of extended support, better to drop it for .0 Steve > From: Christian Heimes > Subject: [Python-Dev] Python 3.4 and Windows XP: just 45 days until EOL > > Hi, > > how do you feel about dropping Windows XP support for Python 3.4? It > would enable us to use some features that are only available on Windows > Vista and newer, for example http://bugs.python.org/issue6926 and > http://bugs.python.org/issue1763 . > > PEP 11 says: > A new feature release X.Y.0 will support all Windows releases > whose extended support phase is not yet expired. > > For Python 3.4 is going to be a very close call. According to PEP 429 > 3.4.0 final is scheduled for February 22, 2014. The extended support > phase of Windows XP ends merely 45 days later on April 8, 2014. Do we > really have to restrict ourselves to an API that is going to become > deprecated 45 days after the estimated release of 3.4.0? > > Christian From ethan at stoneleaf.us Fri Jul 12 02:06:33 2013 From: ethan at stoneleaf.us (Ethan Furman) Date: Thu, 11 Jul 2013 17:06:33 -0700 Subject: [Python-Dev] Python 3.4 and Windows XP: just 45 days until EOL In-Reply-To: <51DF469B.6010007@python.org> References: <51DF469B.6010007@python.org> Message-ID: <51DF4889.4030007@stoneleaf.us> On 07/11/2013 04:58 PM, Christian Heimes wrote: > > how do you feel about dropping Windows XP support for Python 3.4? It > would enable us to use some features that are only available on Windows > Vista and newer, for example http://bugs.python.org/issue6926 and > http://bugs.python.org/issue1763 . > > PEP 11 says: > A new feature release X.Y.0 will support all Windows releases > whose extended support phase is not yet expired. > > For Python 3.4 is going to be a very close call. According to PEP 429 > 3.4.0 final is scheduled for February 22, 2014. The extended support > phase of Windows XP ends merely 45 days later on April 8, 2014. Do we > really have to restrict ourselves to an API that is going to become > deprecated 45 days after the estimated release of 3.4.0? Just because Microsoft no longer supports it doesn't mean there aren't still a lot of machines using it. Of course, I suppose those folks can just stick with 3.3. ;) -- ~Ethan~ From benhoyt at gmail.com Fri Jul 12 03:49:54 2013 From: benhoyt at gmail.com (Ben Hoyt) Date: Fri, 12 Jul 2013 13:49:54 +1200 Subject: [Python-Dev] Python 3.4 and Windows XP: just 45 days until EOL In-Reply-To: <51DF469B.6010007@python.org> References: <51DF469B.6010007@python.org> Message-ID: I guess it has to be dropped at some stage, but with Windows XP it's a case of "XP is dead. Long live XP!" There are still an awful lot of XP boxes out there, and I'd kind hate to see support dropped completely. We still use it here at home. Wikipedia/Net Applications says that Windows XP has still has a full 37% of market share! (http://en.wikipedia.org/wiki/Usage_share_of_operating_systems ) What about just have these attributes/functions on OSes that support it, for example os.kill on Python 2.6 vs 2.7? -Ben On Fri, Jul 12, 2013 at 11:58 AM, Christian Heimes wrote: > Hi, > > how do you feel about dropping Windows XP support for Python 3.4? It > would enable us to use some features that are only available on Windows > Vista and newer, for example http://bugs.python.org/issue6926 and > http://bugs.python.org/issue1763 . > > PEP 11 says: > A new feature release X.Y.0 will support all Windows releases > whose extended support phase is not yet expired. > > For Python 3.4 is going to be a very close call. According to PEP 429 > 3.4.0 final is scheduled for February 22, 2014. The extended support > phase of Windows XP ends merely 45 days later on April 8, 2014. Do we > really have to restrict ourselves to an API that is going to become > deprecated 45 days after the estimated release of 3.4.0? > > Christian > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/benhoyt%40gmail.com > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rdmurray at bitdance.com Fri Jul 12 04:28:49 2013 From: rdmurray at bitdance.com (R. David Murray) Date: Thu, 11 Jul 2013 22:28:49 -0400 Subject: [Python-Dev] Python 3.4 and Windows XP: just 45 days until EOL In-Reply-To: References: <51DF469B.6010007@python.org> Message-ID: <20130712022850.CB07A250066@webabinitio.net> On Fri, 12 Jul 2013 13:49:54 +1200, Ben Hoyt wrote: > I guess it has to be dropped at some stage, but with Windows XP it's a case > of "XP is dead. Long live XP!" There are still an awful lot of XP boxes out > there, and I'd kind hate to see support dropped completely. We still use it > here at home. > > Wikipedia/Net Applications says that Windows XP has still has a full 37% of > market share! (http://en.wikipedia.org/wiki/Usage_share_of_operating_systems > ) > > What about just have these attributes/functions on OSes that support it, > for example os.kill on Python 2.6 vs 2.7? I'm afraid it's not that simple. The issue (as I understand it from Crys) is that we compile using setting that prevent the advanced features being used, and that's really the only way to do it. That is, you can only get the advanced features by using certain settings, and if you use those, the compiled code won't run on XP. So it is not practical to decide only at runtime to support the advanced feature, meaning there would have to be a differently compiled version of Python specifically for Windows XP (and doubtless new XP-specific ifdefs *as well*), and I doubt the core team is going to go there. The older versions of Python won't be going away. Those can still be used on XP. Of course, they won't get bug fixes...just like XP itself. --David From benhoyt at gmail.com Fri Jul 12 04:59:40 2013 From: benhoyt at gmail.com (Ben Hoyt) Date: Fri, 12 Jul 2013 14:59:40 +1200 Subject: [Python-Dev] Python 3.4 and Windows XP: just 45 days until EOL In-Reply-To: <20130712022850.CB07A250066@webabinitio.net> References: <51DF469B.6010007@python.org> <20130712022850.CB07A250066@webabinitio.net> Message-ID: Ah, yeah, that makes sense -- thanks for the further explanation. True about older versions of Python "not going away". > What about just have these attributes/functions on OSes that support it, > > for example os.kill on Python 2.6 vs 2.7? > > I'm afraid it's not that simple. The issue (as I understand it from Crys) > is that we compile using setting that prevent the advanced features being > used, and that's really the only way to do it. That is, you can only get > the advanced features by using certain settings, and if you use those, > the compiled code won't run on XP. So it is not practical to decide only > at runtime to support the advanced feature, meaning there would have to > be a differently compiled version of Python specifically for Windows XP > (and doubtless new XP-specific ifdefs *as well*), and I doubt the core > team is going to go there. > > The older versions of Python won't be going away. Those can still be > used on XP. Of course, they won't get bug fixes...just like XP itself. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From stephen at xemacs.org Fri Jul 12 05:16:31 2013 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Fri, 12 Jul 2013 12:16:31 +0900 Subject: [Python-Dev] Python 3.4 and Windows XP: just 45 days until EOL In-Reply-To: References: <51DF469B.6010007@python.org> Message-ID: <87k3kwtqls.fsf@uwakimon.sk.tsukuba.ac.jp> Steve Dower writes: > I don't see any good reason for Python to support an OS that > Microsoft doesn't, How about the *users* of that OS? I don't see any good reason to take into account what Microsoft does or doesn't support. If that lack of support leads to Python users dropping XP like hot potatoes, that will be visible in itself. I doubt it will, though. EOL for XP has been coming a long long time, far longer than Microsoft anticipated ;-), yet usage persists (most small businesses I know in Japan are still using XP-based apps, small sample, I admit). If Python support for XP leads to significant pain for the majority of Python users, or the majority on Windows, that's a good reason to drop it, (which needs to be balanced against users who still need support). Regards, From regebro at gmail.com Fri Jul 12 05:27:28 2013 From: regebro at gmail.com (Lennart Regebro) Date: Fri, 12 Jul 2013 05:27:28 +0200 Subject: [Python-Dev] Python 3.4 and Windows XP: just 45 days until EOL In-Reply-To: References: <51DF469B.6010007@python.org> Message-ID: On Fri, Jul 12, 2013 at 2:11 AM, Steve Dower wrote: > +1. And maybe amend PEP 11 to specify "whose extended support phase does not expire within 6 months of release"? (I picked 6 for no particular reason.) Why have the specification in PEP 11 if we feel we can change the rules arbitrarily when we feel like it? //Lennart From ben+python at benfinney.id.au Fri Jul 12 07:23:00 2013 From: ben+python at benfinney.id.au (Ben Finney) Date: Fri, 12 Jul 2013 15:23:00 +1000 Subject: [Python-Dev] Python 3.4 and Windows XP: just 45 days until EOL References: <51DF469B.6010007@python.org> <87k3kwtqls.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <7wmwps1he3.fsf@benfinney.id.au> "Stephen J. Turnbull" writes: > I don't see any good reason to take into account what Microsoft does > or doesn't support. It seems you're advocating a position quite ad odds with . Can you propose an amendment to PEP 11 that would remove that consideration? -- \ ?If you do not trust the source do not use this program.? | `\ ?Microsoft Vista security dialogue | _o__) | Ben Finney From ncoghlan at gmail.com Fri Jul 12 07:32:27 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 12 Jul 2013 15:32:27 +1000 Subject: [Python-Dev] Python 3.4 and Windows XP: just 45 days until EOL In-Reply-To: References: <51DF469B.6010007@python.org> Message-ID: On 12 July 2013 13:27, Lennart Regebro wrote: > On Fri, Jul 12, 2013 at 2:11 AM, Steve Dower > wrote: > > +1. And maybe amend PEP 11 to specify "whose extended support phase does > not expire within 6 months of release"? (I picked 6 for no particular > reason.) > > Why have the specification in PEP 11 if we feel we can change the > rules arbitrarily when we feel like it? > Because process PEPs are documentation of community practice, not an inviolable constraint (e.g. PEP 1 has frequently lagged behind what we *actually* do, and only been updated once we noticed we had drifted away from the nominal procedures). In this case, the question of "What do we do when a Windows version goes EOL shortly after a Python release?" hasn't come up before, so PEP 11 has never had to take it into account. That doesn't mean we *should* change it, it just means the option is one we have available to us. Fixing issue 6926 only requires setting the minimum API version to Windows *XP*, so it isn't actually relevant to the question of whether or not to drop support for XP (only W2k, which I thought we already dropped, but we mustn't have bumped the minimum Windows API version at the time). Issue 1763 looks like it could be better solved through pywin32 than through standard library changes. It certainly doesn't appear to be worth the cost of dropping Windows XP support. Unless there are more compelling examples of APIs that we can't access through Windows XP compatible interfaces, -1 on the change for 3.4. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Fri Jul 12 09:45:11 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 12 Jul 2013 09:45:11 +0200 Subject: [Python-Dev] Python 3.4 and Windows XP: just 45 days until EOL References: <51DF469B.6010007@python.org> <20130712022850.CB07A250066@webabinitio.net> Message-ID: <20130712094511.16be8157@fsol> On Thu, 11 Jul 2013 22:28:49 -0400 "R. David Murray" wrote: > On Fri, 12 Jul 2013 13:49:54 +1200, Ben Hoyt wrote: > > I guess it has to be dropped at some stage, but with Windows XP it's a case > > of "XP is dead. Long live XP!" There are still an awful lot of XP boxes out > > there, and I'd kind hate to see support dropped completely. We still use it > > here at home. > > > > Wikipedia/Net Applications says that Windows XP has still has a full 37% of > > market share! (http://en.wikipedia.org/wiki/Usage_share_of_operating_systems > > ) > > > > What about just have these attributes/functions on OSes that support it, > > for example os.kill on Python 2.6 vs 2.7? > > I'm afraid it's not that simple. The issue (as I understand it from Crys) > is that we compile using setting that prevent the advanced features being > used, and that's really the only way to do it. That is, you can only get > the advanced features by using certain settings, and if you use those, > the compiled code won't run on XP. So it is not practical to decide only > at runtime to support the advanced feature, It's not practical, but we already do it. See check_GetFinalPathNameByHandle in Modules/posixmodule.c. Regards Antoine. From christian at python.org Fri Jul 12 13:32:52 2013 From: christian at python.org (Christian Heimes) Date: Fri, 12 Jul 2013 13:32:52 +0200 Subject: [Python-Dev] Python 3.4 and Windows XP: just 45 days until EOL In-Reply-To: References: <51DF469B.6010007@python.org> Message-ID: <51DFE964.9020006@python.org> Am 12.07.2013 03:49, schrieb Ben Hoyt: > I guess it has to be dropped at some stage, but with Windows XP it's a > case of "XP is dead. Long live XP!" There are still an awful lot of XP > boxes out there, and I'd kind hate to see support dropped completely. We > still use it here at home. > > Wikipedia/Net Applications says that Windows XP has still has a full 37% > of market share! > (http://en.wikipedia.org/wiki/Usage_share_of_operating_systems) I'm not planing to shut Windows XP out from Python completely. Users of Python can still use Python 3.3 or 2.7. Python 3.3 will get a final bug fix release after the release of Python 3.4.0 and security fixes until 2017. Windows XP is really, *really* old. It has been released almost 12 years ago. Linux kernel 2.4.0 was released about the same time. Its mainstream support has ended 4 years ago. Do people really expect that they can run the latest version of a program on a decommissioned operating system? Christian From murman at gmail.com Fri Jul 12 16:05:59 2013 From: murman at gmail.com (Michael Urman) Date: Fri, 12 Jul 2013 09:05:59 -0500 Subject: [Python-Dev] Python 3.4 and Windows XP: just 45 days until EOL In-Reply-To: <51DF469B.6010007@python.org> References: <51DF469B.6010007@python.org> Message-ID: On Thu, Jul 11, 2013 at 6:58 PM, Christian Heimes wrote: > For Python 3.4 is going to be a very close call. According to PEP 429 > 3.4.0 final is scheduled for February 22, 2014. The extended support > phase of Windows XP ends merely 45 days later on April 8, 2014. Do we > really have to restrict ourselves to an API that is going to become > deprecated 45 days after the estimated release of 3.4.0? If your motivation is to ease the use of APIs only available on Windows Vista and later, you've got another year to wait: Windows Server 2003 R2 extended support lasts through until July 2015. http://support.microsoft.com/lifecycle/search/default.aspx?alpha=Windows+Server+2003+R2 Michael From guido at python.org Fri Jul 12 17:50:49 2013 From: guido at python.org (Guido van Rossum) Date: Fri, 12 Jul 2013 08:50:49 -0700 Subject: [Python-Dev] Python 3.4 and Windows XP: just 45 days until EOL In-Reply-To: <51DFE964.9020006@python.org> References: <51DF469B.6010007@python.org> <51DFE964.9020006@python.org> Message-ID: You underestimate the reach of XP. For older or underpowered hardware outside the developed world it is still the de facto choice. And it definitely is the best version of Windows ever. None of the Win98 crap and none of the Vista junk. Telling people to go install Ubuntu is not really fair if others around them don't or if they need certain software. --Guido van Rossum (sent from Android phone) On Jul 12, 2013 4:34 AM, "Christian Heimes" wrote: > Am 12.07.2013 03:49, schrieb Ben Hoyt: > > I guess it has to be dropped at some stage, but with Windows XP it's a > > case of "XP is dead. Long live XP!" There are still an awful lot of XP > > boxes out there, and I'd kind hate to see support dropped completely. We > > still use it here at home. > > > > Wikipedia/Net Applications says that Windows XP has still has a full 37% > > of market share! > > (http://en.wikipedia.org/wiki/Usage_share_of_operating_systems) > > I'm not planing to shut Windows XP out from Python completely. Users of > Python can still use Python 3.3 or 2.7. Python 3.3 will get a final bug > fix release after the release of Python 3.4.0 and security fixes until > 2017. > > Windows XP is really, *really* old. It has been released almost 12 years > ago. Linux kernel 2.4.0 was released about the same time. Its mainstream > support has ended 4 years ago. Do people really expect that they can run > the latest version of a program on a decommissioned operating system? > > Christian > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/guido%40python.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: From status at bugs.python.org Fri Jul 12 18:07:25 2013 From: status at bugs.python.org (Python tracker) Date: Fri, 12 Jul 2013 18:07:25 +0200 (CEST) Subject: [Python-Dev] Summary of Python tracker Issues Message-ID: <20130712160725.F31FD56A2D@psf.upfronthosting.co.za> ACTIVITY SUMMARY (2013-07-05 - 2013-07-12) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 4067 ( -5) closed 26140 (+74) total 30207 (+69) Open issues with patches: 1834 Issues opened (49) ================== #7457: Adding a read_pkg_file to DistributionMetadata http://bugs.python.org/issue7457 reopened by jason.coombs #18366: Look into using setuptools 0.8 with devinabox http://bugs.python.org/issue18366 opened by brett.cannon #18367: See if a venv setup can be used for devinabox for coverage http://bugs.python.org/issue18367 opened by brett.cannon #18368: PyOS_StdioReadline() leaks memory when realloc() fails http://bugs.python.org/issue18368 opened by christian.heimes #18369: X509 cert class for ssl module http://bugs.python.org/issue18369 opened by christian.heimes #18372: _Pickler_New() doesn't call PyObject_GC_Track(self) http://bugs.python.org/issue18372 opened by christian.heimes #18373: implement sys.get/setbyteswarningflag() http://bugs.python.org/issue18373 opened by dholth #18374: ast.parse gives wrong position (col_offset) for some BinOp-s http://bugs.python.org/issue18374 opened by Aivar.Annamaa #18375: python -m test --randseed 1234 does not randomize tests http://bugs.python.org/issue18375 opened by flox #18376: show the effective count of process when running the testsuite http://bugs.python.org/issue18376 opened by flox #18378: locale.getdefaultlocale() fails on Mac OS X with default langu http://bugs.python.org/issue18378 opened by Dmitry.Jemerov #18379: SSLSocket.getpeercert(): OCSP and CRL DP URIs http://bugs.python.org/issue18379 opened by christian.heimes #18381: unittest warnings counter http://bugs.python.org/issue18381 opened by giampaolo.rodola #18382: multiprocessing's overlapped PipeConnection issues on Windows http://bugs.python.org/issue18382 opened by wprins #18383: test_warnings modifies warnings.filters when running with "-W http://bugs.python.org/issue18383 opened by flox #18384: Add devhelp build instructions to the documentation makefile http://bugs.python.org/issue18384 opened by brouberol #18387: Add 'symbols' link to pydoc's html menu bar. http://bugs.python.org/issue18387 opened by ron_adam #18389: document that os.path.relpath does not interrogate the file sy http://bugs.python.org/issue18389 opened by jveldridge #18391: socket.fromfd()'s API is difficult or impossible to use correc http://bugs.python.org/issue18391 opened by cks #18392: Doc: PyObject_Malloc() is not documented http://bugs.python.org/issue18392 opened by haypo #18393: Gestalt() is deprecated on OSX 10.8, remove support? http://bugs.python.org/issue18393 opened by ronaldoussoren #18394: cgi.FieldStorage triggers ResourceWarning sometimes http://bugs.python.org/issue18394 opened by flox #18395: Make _Py_char2wchar() and _Py_wchar2char() public http://bugs.python.org/issue18395 opened by haypo #18396: test_signal.test_issue9324() fails on buildbot AMD64 Windows7 http://bugs.python.org/issue18396 opened by haypo #18397: Python with MinGW http://bugs.python.org/issue18397 opened by Friedrich.Spee.von.Langenfeld #18400: Minor increase to Pickle test coverage http://bugs.python.org/issue18400 opened by davidsackett #18401: Tests for pdb import ~/.pdbrc http://bugs.python.org/issue18401 opened by sptonkin #18402: Finding perl64 http://bugs.python.org/issue18402 opened by mjdorma #18404: Memory leak in gdbmmodule http://bugs.python.org/issue18404 opened by bkabrda #18405: crypt.mksalt() result has unnecessarily low entropy http://bugs.python.org/issue18405 opened by christian.heimes #18406: unicodedata.itergraphemes / str.itergraphemes / str.graphemes http://bugs.python.org/issue18406 opened by dpk #18407: Fix compiler warnings in pythoncore for Win64 http://bugs.python.org/issue18407 opened by jkloth #18408: Fixes crashes found by pyfailmalloc http://bugs.python.org/issue18408 opened by haypo #18409: IDLE Improvements: Unit test for AutoComplete.py http://bugs.python.org/issue18409 opened by philwebster #18410: IDLE Improvements: Unit test for SearchDialog.py http://bugs.python.org/issue18410 opened by philwebster #18416: Move to absolute file paths for module.__file__ http://bugs.python.org/issue18416 opened by brett.cannon #18417: urlopen() has a hidden default for its timeout argument http://bugs.python.org/issue18417 opened by barry #18418: Thread.isAlive() sometimes True after fork http://bugs.python.org/issue18418 opened by emptysquare #18421: Refactor call_with_frame() function of pyexpat.c http://bugs.python.org/issue18421 opened by haypo #18423: Document limitations on -m http://bugs.python.org/issue18423 opened by acooke #18425: IDLE Unit test for IdleHistory.py http://bugs.python.org/issue18425 opened by JayKrish #18426: Crash when extension does not use PyModule_Create() http://bugs.python.org/issue18426 opened by Padowan #18428: IDLE startup error http://bugs.python.org/issue18428 opened by mwei11 #18429: IDLE: Format Paragraph doesn't function with comment blocks http://bugs.python.org/issue18429 opened by Todd.Rovito #18430: gzip, bz2, lzma: peek advances file position of existing file http://bugs.python.org/issue18430 opened by nbargnesi #18431: Encoded words in structured headers are not decoded by the new http://bugs.python.org/issue18431 opened by r.david.murray #18432: sched modules queue property should return a list, not an iter http://bugs.python.org/issue18432 opened by rhettinger #18433: Clarify venv docs about which methods check for pre-existing p http://bugs.python.org/issue18433 opened by brett.cannon #18434: Update venv example to use setuptools http://bugs.python.org/issue18434 opened by brett.cannon Most recent 15 issues with no replies (15) ========================================== #18434: Update venv example to use setuptools http://bugs.python.org/issue18434 #18433: Clarify venv docs about which methods check for pre-existing p http://bugs.python.org/issue18433 #18431: Encoded words in structured headers are not decoded by the new http://bugs.python.org/issue18431 #18429: IDLE: Format Paragraph doesn't function with comment blocks http://bugs.python.org/issue18429 #18425: IDLE Unit test for IdleHistory.py http://bugs.python.org/issue18425 #18423: Document limitations on -m http://bugs.python.org/issue18423 #18421: Refactor call_with_frame() function of pyexpat.c http://bugs.python.org/issue18421 #18410: IDLE Improvements: Unit test for SearchDialog.py http://bugs.python.org/issue18410 #18409: IDLE Improvements: Unit test for AutoComplete.py http://bugs.python.org/issue18409 #18405: crypt.mksalt() result has unnecessarily low entropy http://bugs.python.org/issue18405 #18401: Tests for pdb import ~/.pdbrc http://bugs.python.org/issue18401 #18400: Minor increase to Pickle test coverage http://bugs.python.org/issue18400 #18395: Make _Py_char2wchar() and _Py_wchar2char() public http://bugs.python.org/issue18395 #18394: cgi.FieldStorage triggers ResourceWarning sometimes http://bugs.python.org/issue18394 #18392: Doc: PyObject_Malloc() is not documented http://bugs.python.org/issue18392 Most recent 15 issues waiting for review (15) ============================================= #18432: sched modules queue property should return a list, not an iter http://bugs.python.org/issue18432 #18430: gzip, bz2, lzma: peek advances file position of existing file http://bugs.python.org/issue18430 #18429: IDLE: Format Paragraph doesn't function with comment blocks http://bugs.python.org/issue18429 #18425: IDLE Unit test for IdleHistory.py http://bugs.python.org/issue18425 #18418: Thread.isAlive() sometimes True after fork http://bugs.python.org/issue18418 #18408: Fixes crashes found by pyfailmalloc http://bugs.python.org/issue18408 #18407: Fix compiler warnings in pythoncore for Win64 http://bugs.python.org/issue18407 #18405: crypt.mksalt() result has unnecessarily low entropy http://bugs.python.org/issue18405 #18404: Memory leak in gdbmmodule http://bugs.python.org/issue18404 #18402: Finding perl64 http://bugs.python.org/issue18402 #18400: Minor increase to Pickle test coverage http://bugs.python.org/issue18400 #18396: test_signal.test_issue9324() fails on buildbot AMD64 Windows7 http://bugs.python.org/issue18396 #18393: Gestalt() is deprecated on OSX 10.8, remove support? http://bugs.python.org/issue18393 #18389: document that os.path.relpath does not interrogate the file sy http://bugs.python.org/issue18389 #18387: Add 'symbols' link to pydoc's html menu bar. http://bugs.python.org/issue18387 Top 10 most discussed issues (10) ================================= #18408: Fixes crashes found by pyfailmalloc http://bugs.python.org/issue18408 14 msgs #10042: functools.total_ordering fails to handle NotImplemented correc http://bugs.python.org/issue10042 12 msgs #18257: Two copies of python-config http://bugs.python.org/issue18257 12 msgs #18378: locale.getdefaultlocale() fails on Mac OS X with default langu http://bugs.python.org/issue18378 10 msgs #18100: socket.sendall() cannot send buffers of 2GB or more http://bugs.python.org/issue18100 9 msgs #18365: Idle: mock Text class and test thereof http://bugs.python.org/issue18365 9 msgs #18430: gzip, bz2, lzma: peek advances file position of existing file http://bugs.python.org/issue18430 9 msgs #18389: document that os.path.relpath does not interrogate the file sy http://bugs.python.org/issue18389 8 msgs #18397: Python with MinGW http://bugs.python.org/issue18397 8 msgs #18426: Crash when extension does not use PyModule_Create() http://bugs.python.org/issue18426 8 msgs Issues closed (72) ================== #2554: test_ioctl failed Python 2.6a2 Solaris 10 SUN C http://bugs.python.org/issue2554 closed by christian.heimes #3093: Namespace pollution from multiprocessing http://bugs.python.org/issue3093 closed by christian.heimes #3266: Python-2.5.2/Modules/mmapmodule.c:915: error: `O_RDWR' undecla http://bugs.python.org/issue3266 closed by christian.heimes #3329: API for setting the memory allocator used by Python http://bugs.python.org/issue3329 closed by haypo #5060: gcc profile guided optimization http://bugs.python.org/issue5060 closed by christian.heimes #7573: Position independent include of Python.h http://bugs.python.org/issue7573 closed by christian.heimes #7686: redundant open modes 'rbb', 'wbb', 'abb' no longer work on Win http://bugs.python.org/issue7686 closed by christian.heimes #9838: Inadequate C-API to Python 3 I/O objects http://bugs.python.org/issue9838 closed by christian.heimes #11012: Add log1p(), exp1m(), gamma(), and lgamma() to cmath http://bugs.python.org/issue11012 closed by rhettinger #11366: xml.etree.ElementTree.Element: docs should mention immediate s http://bugs.python.org/issue11366 closed by christian.heimes #11908: Weird `slice.stop or sys.maxint` http://bugs.python.org/issue11908 closed by rhettinger #12115: some tests need to be skipped on threadless systems http://bugs.python.org/issue12115 closed by christian.heimes #12272: Python 2.7.1 version conflict for package "Tcl" on Windows 7 http://bugs.python.org/issue12272 closed by christian.heimes #12984: XML NamedNodeMap ( attribName in NamedNodeMap fails ) http://bugs.python.org/issue12984 closed by christian.heimes #12990: launcher can't work on path including tradition chinese char http://bugs.python.org/issue12990 closed by ronaldoussoren #13396: new method random.getrandbytes() http://bugs.python.org/issue13396 closed by rhettinger #13418: Embedded Python memory leak http://bugs.python.org/issue13418 closed by christian.heimes #13871: namedtuple does not normalize field names when checking for du http://bugs.python.org/issue13871 closed by rhettinger #14139: test_ftplib: segfault http://bugs.python.org/issue14139 closed by christian.heimes #14268: _move_file is broken because of a bad mock http://bugs.python.org/issue14268 closed by christian.heimes #14620: Fatal Python error: Cannot recover from stack overflow. http://bugs.python.org/issue14620 closed by christian.heimes #14748: spwd.getspall() is returning LDAP (non local) users too http://bugs.python.org/issue14748 closed by christian.heimes #15197: test_gettext failure on Win64 buildbot http://bugs.python.org/issue15197 closed by christian.heimes #15553: Segfault in test_6_daemon_threads() of test_threading, on Mac http://bugs.python.org/issue15553 closed by ronaldoussoren #17198: dbm.whichdb references unitialized 'ndbm' variable http://bugs.python.org/issue17198 closed by ezio.melotti #17221: Resort Misc/NEWS http://bugs.python.org/issue17221 closed by christian.heimes #17845: Clarify successful build message http://bugs.python.org/issue17845 closed by brett.cannon #17860: subprocess docs lack info how to use output result http://bugs.python.org/issue17860 closed by ronaldoussoren #17987: test.support.captured_stderr, captured_stdin not documented http://bugs.python.org/issue17987 closed by r.david.murray #18013: cgi.FieldStorage does not parse W3C sample http://bugs.python.org/issue18013 closed by flox #18020: html.escape 10x slower than cgi.escape http://bugs.python.org/issue18020 closed by ezio.melotti #18044: Email headers do not properly decode to unicode. http://bugs.python.org/issue18044 closed by r.david.murray #18101: Tk.split() doesn't work with nested Unicode strings http://bugs.python.org/issue18101 closed by serhiy.storchaka #18106: There are unused variables in Lib/test/test_collections.py http://bugs.python.org/issue18106 closed by ezio.melotti #18116: getpass.unix_getpass() always fallback to sys.stdin http://bugs.python.org/issue18116 closed by r.david.murray #18136: Put local build paths before system build paths in configure.a http://bugs.python.org/issue18136 closed by brett.cannon #18203: Replace direct calls to malloc() with PyMem_Malloc() or PyMem_ http://bugs.python.org/issue18203 closed by haypo #18231: What's new in Python should explain what's new in UCD http://bugs.python.org/issue18231 closed by belopolsky #18289: python.org Interactive interpreter linked with libedit can seg http://bugs.python.org/issue18289 closed by loewis #18308: checkRecvmsgAddress wrong in test_socket.py (AIX failures) http://bugs.python.org/issue18308 closed by neologix #18338: python --version should send output to STDOUT http://bugs.python.org/issue18338 closed by serhiy.storchaka #18342: Use the repr of a module name for ImportError in ceval.c http://bugs.python.org/issue18342 closed by brett.cannon #18351: Incorrect variable name in importlib._bootstrap._get_sourcefil http://bugs.python.org/issue18351 closed by brett.cannon #18352: collections.Counter with added attributes are not deepcopied p http://bugs.python.org/issue18352 closed by rhettinger #18354: http://www.python.org/doc/ has outdated note http://bugs.python.org/issue18354 closed by brian.curtin #18360: Won't install. Keeps telling me DLL is missing. http://bugs.python.org/issue18360 closed by terry.reedy #18361: Move dev-in-a-box to os.cpu_count() http://bugs.python.org/issue18361 closed by brett.cannon #18362: Make build_cpython.py from dev-in-a-box work outside of a box http://bugs.python.org/issue18362 closed by brett.cannon #18363: Change use of acronym tag in devinabox index.html to abbr http://bugs.python.org/issue18363 closed by brett.cannon #18364: Remove _not_found hack from importlib http://bugs.python.org/issue18364 closed by brett.cannon #18370: ast.parse produces wrong lineno and col_offset for triple-quot http://bugs.python.org/issue18370 closed by Aivar.Annamaa #18371: Should be able to specify SSL version for smtplib http://bugs.python.org/issue18371 closed by christian.heimes #18377: Python Launcher code cleanup http://bugs.python.org/issue18377 closed by ronaldoussoren #18380: email/quoprimime.py doesn't use re.sub correctly http://bugs.python.org/issue18380 closed by ezio.melotti #18385: Add devhelp build instructions to the documentation makefile ( http://bugs.python.org/issue18385 closed by ezio.melotti #18386: Better random number generator http://bugs.python.org/issue18386 closed by rhettinger #18388: Link to the contributor agreement from devinabox http://bugs.python.org/issue18388 closed by brett.cannon #18390: Better documenting of ``hg revert -r`` when merging with anoth http://bugs.python.org/issue18390 closed by brett.cannon #18398: Debug Python build fails on OS X: Fatal Python error: bad ID: http://bugs.python.org/issue18398 closed by haypo #18399: Fix grammar in comment in python.c http://bugs.python.org/issue18399 closed by r.david.murray #18403: Minor bug in The Python Tutorial http://bugs.python.org/issue18403 closed by ezio.melotti #18411: signal.SIGINT in windows cause process exit directly. http://bugs.python.org/issue18411 closed by r.david.murray #18412: Minor problem in PEP 8 http://bugs.python.org/issue18412 closed by barry #18413: logging.error('asdf', extra={'msg':123}) raise KeyError except http://bugs.python.org/issue18413 closed by christian.heimes #18414: random.choices(seq, k) http://bugs.python.org/issue18414 closed by christian.heimes #18415: Normalize single/double quote usage in importlib http://bugs.python.org/issue18415 closed by brett.cannon #18419: For reasons I can't explain Python in some cases says that get http://bugs.python.org/issue18419 closed by ezio.melotti #18420: os.rename FileNotFound complaining about dst, but it is src wh http://bugs.python.org/issue18420 closed by ned.deily #18422: is_package missing so can't use -m http://bugs.python.org/issue18422 closed by brett.cannon #18424: sum() does not return only the sum of a sequence of numbers + http://bugs.python.org/issue18424 closed by r.david.murray #18427: str.replace causes segfault for long strings http://bugs.python.org/issue18427 closed by ronaldoussoren #979407: urllib2 digest auth totally broken http://bugs.python.org/issue979407 closed by christian.heimes From v+python at g.nevcal.com Fri Jul 12 21:45:03 2013 From: v+python at g.nevcal.com (Glenn Linderman) Date: Fri, 12 Jul 2013 12:45:03 -0700 Subject: [Python-Dev] Python 3.4 and Windows XP: just 45 days until EOL In-Reply-To: References: <51DF469B.6010007@python.org> <51DFE964.9020006@python.org> Message-ID: <51E05CBF.9080107@g.nevcal.com> On 7/12/2013 8:50 AM, Guido van Rossum wrote: > > You underestimate the reach of XP. For older or underpowered hardware > outside the developed world it is still the de facto choice. And it > definitely is the best version of Windows ever. None of the Win98 crap > and none of the Vista junk. > > Telling people to go install Ubuntu is not really fair if others > around them don't or if they need certain software. > > --Guido van Rossum (sent from Android phone) > Well said. +100 -------------- next part -------------- An HTML attachment was scrubbed... URL: From cs at zip.com.au Sat Jul 13 04:57:02 2013 From: cs at zip.com.au (Cameron Simpson) Date: Sat, 13 Jul 2013 12:57:02 +1000 Subject: [Python-Dev] PEP 446: Add new parameters to configure the inherance of files and for non-blocking sockets In-Reply-To: References: Message-ID: <20130713025702.GA23103@cskk.homeip.net> Sorry for the delay; work got in the way. On 07Jul2013 15:21, Victor Stinner wrote: | Ok, I think that the best consensus here is to add a new | os.set_blocking() function. The function would not be available on | Windows, but should be available on all UNIX platforms. Thanks. | See the diff: | http://hg.python.org/peps/rev/ce61588d244c This sentence: The flag is cleared in the child process before executing the program, the change does not change the flag in the parent process. needs a semicolon, not a comma. | @Charles-Fran?ois and Cameron: Do you prefer to new version of the PEP? | http://www.python.org/dev/peps/pep-0446/ | (will be updated in a few minutes) I'm happy with it. Thank you, -- Cameron Simpson From ericsnowcurrently at gmail.com Sat Jul 13 06:10:34 2013 From: ericsnowcurrently at gmail.com (Eric Snow) Date: Fri, 12 Jul 2013 22:10:34 -0600 Subject: [Python-Dev] PLY in stdlib (was cffi in stdlib) In-Reply-To: References: Message-ID: On Feb 27, 2013 4:31 AM, "Michael Foord" wrote: > > > On 27 Feb 2013, at 11:00, David Beazley wrote: > > >> > >> From: Eli Bendersky > >> > >> I'll be the first one to admit that pycparser is almost certainly not > >> generally useful enough to be exposed in the stdlib. So just using it as an > >> implementation detail is absolutely fine. PLY is a more interesting > >> question, however, since PLY is somewhat more generally useful. That said, > >> I see all this as implementation details that shouldn't distract us from > >> the main point of whether cffi should be added. > >> > > > > Regarding the inclusion of PLY or some subcomponent of it in the standard library, it's not an entirely crazy idea in my opinion. > > +1 PLY is capable and well tried-and-tested. We used it in Resolver One to implement a pretty large grammar and it is (in my opinion) best of breed in the Python parser generator world. Being stable and widely used, with an "available maintainer", makes it an ideal candidate for standard library inclusion. Is this still on the table? -eric -------------- next part -------------- An HTML attachment was scrubbed... URL: From tjreedy at udel.edu Sat Jul 13 06:41:10 2013 From: tjreedy at udel.edu (Terry Reedy) Date: Sat, 13 Jul 2013 00:41:10 -0400 Subject: [Python-Dev] PLY in stdlib (was cffi in stdlib) In-Reply-To: References: Message-ID: On 7/13/2013 12:10 AM, Eric Snow wrote: > > On Feb 27, 2013 4:31 AM, "Michael Foord" > +1 PLY is capable and well tried-and-tested. We used it in Resolver > One to implement a pretty large grammar and it is (in my opinion) best > of breed in the Python parser generator world. Being stable and widely > used, with an "available maintainer", makes it an ideal candidate for > standard library inclusion. > > Is this still on the table? Who is the maintainer and what is his opinion? -- Terry Jan Reedy From jdunck at gmail.com Sat Jul 13 06:57:20 2013 From: jdunck at gmail.com (Jeremy Dunck) Date: Fri, 12 Jul 2013 21:57:20 -0700 Subject: [Python-Dev] PLY in stdlib (was cffi in stdlib) In-Reply-To: References: Message-ID: David Beasley; see earlier in this same thread: http://mail.python.org/pipermail/python-dev/2013-February/thread.html#124389 On Fri, Jul 12, 2013 at 9:41 PM, Terry Reedy wrote: > On 7/13/2013 12:10 AM, Eric Snow wrote: > >> >> On Feb 27, 2013 4:31 AM, "Michael Foord" > > > > +1 PLY is capable and well tried-and-tested. We used it in Resolver >> One to implement a pretty large grammar and it is (in my opinion) best >> of breed in the Python parser generator world. Being stable and widely >> used, with an "available maintainer", makes it an ideal candidate for >> standard library inclusion. >> >> Is this still on the table? >> > > Who is the maintainer and what is his opinion? > > -- > Terry Jan Reedy > > > ______________________________**_________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/**mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/**mailman/options/python-dev/** > jdunck%40gmail.com > -------------- next part -------------- An HTML attachment was scrubbed... URL: From agriff at tin.it Sat Jul 13 08:41:55 2013 From: agriff at tin.it (Andrea Griffini) Date: Sat, 13 Jul 2013 08:41:55 +0200 Subject: [Python-Dev] Doubt about closure/function generation in bytecode Message-ID: Is there any semantic difference between BUILD_TUPLE 0 LOAD_CONST MAKE_CLOSURE 0 and LOAD_CONST MAKE_FUNCTION 0 ? In other words is there any difference between a function and a closure that doesn't capture anything? Is it just a speed optimization for a common case? Andrea Griffini -------------- next part -------------- An HTML attachment was scrubbed... URL: From fuzzyman at voidspace.org.uk Sat Jul 13 10:24:02 2013 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Sat, 13 Jul 2013 11:24:02 +0300 Subject: [Python-Dev] PLY in stdlib (was cffi in stdlib) In-Reply-To: References: Message-ID: <96746EE7-E96A-4363-B0F7-D7950207C702@voidspace.org.uk> On 13 Jul 2013, at 07:41, Terry Reedy wrote: > On 7/13/2013 12:10 AM, Eric Snow wrote: >> >> On Feb 27, 2013 4:31 AM, "Michael Foord" >> > +1 PLY is capable and well tried-and-tested. We used it in Resolver >> One to implement a pretty large grammar and it is (in my opinion) best >> of breed in the Python parser generator world. Being stable and widely >> used, with an "available maintainer", makes it an ideal candidate for >> standard library inclusion. >> >> Is this still on the table? > > Who is the maintainer and what is his opinion? The maintainer is David Beazley and as far as I recall he has not expressed an opinion on this particular question. It would obviously need his agreement (and maintenance commitment) if it is to fly. Michael > > -- > Terry Jan Reedy > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk -- http://www.voidspace.org.uk/ May you do good and not evil May you find forgiveness for yourself and forgive others May you share freely, never taking more than you give. -- the sqlite blessing http://www.sqlite.org/different.html From brett at python.org Sat Jul 13 15:12:42 2013 From: brett at python.org (Brett Cannon) Date: Sat, 13 Jul 2013 09:12:42 -0400 Subject: [Python-Dev] PLY in stdlib (was cffi in stdlib) In-Reply-To: <96746EE7-E96A-4363-B0F7-D7950207C702@voidspace.org.uk> References: <96746EE7-E96A-4363-B0F7-D7950207C702@voidspace.org.uk> Message-ID: On Sat, Jul 13, 2013 at 4:24 AM, Michael Foord wrote: > > On 13 Jul 2013, at 07:41, Terry Reedy wrote: > > > On 7/13/2013 12:10 AM, Eric Snow wrote: > >> > >> On Feb 27, 2013 4:31 AM, "Michael Foord" > > >> > +1 PLY is capable and well tried-and-tested. We used it in Resolver > >> One to implement a pretty large grammar and it is (in my opinion) best > >> of breed in the Python parser generator world. Being stable and widely > >> used, with an "available maintainer", makes it an ideal candidate for > >> standard library inclusion. > >> > >> Is this still on the table? > > > > Who is the maintainer and what is his opinion? > > > The maintainer is David Beazley and as far as I recall he has not > expressed an opinion on this particular question. It would obviously need > his agreement (and maintenance commitment) if it is to fly. > Just because we have now had two conflicting replies on this: David is down with PLY being added, but Alex Gaynor was working on a cleanup called RPLY for RPython. Basically David said the two of them should work together to clean up PLY and then it should be good to proposing for the stdlib (e.g. there are some backwards-compatibility hacks which should be removed). Below is David's original email on the topic from Feb 27: ------------------------------- Regarding the inclusion of PLY or some subcomponent of it in the standard library, it's not an entirely crazy idea in my opinion. LALR(1) parsers have been around for a long time, are generally known to anyone who's used yacc/bison, and would be useful outside the context of cffi or pycparser. PLY has also been around for about 12 years and is what I would call stable. It gets an update about every year or two, but that's about it. PLY is also relatively small--just two files and about 4300 lines of code (much of which could probably be scaled down a bit). The only downside to including PLY might be the fact that there are very few people walking around who've actually had to *implement* an LALR(1) parser generator. Some of the code for that is extremely hairy and mathematical. At this time, I don't think there are any bugs in it, but it's not the sort of thing that one wants to wander into casually. Also, there are some horrible hacks in PLY that I'd really like to get rid of, but am currently stuck with due to backwards compatibility issues. Alex Gaynor has been working on a PLY variant (RPLY) geared at RPython and which has a slightly different programming interface. I'd say if we were to go down this route, he and I should work together to put together some kind of more general "parsing.lalr" package (or similar) that cleans it up and makes it more suitable as a library for building different kinds of parsing tools on top of. Cheers, Dave -------------- next part -------------- An HTML attachment was scrubbed... URL: From dave at dabeaz.com Sat Jul 13 15:26:23 2013 From: dave at dabeaz.com (David Beazley) Date: Sat, 13 Jul 2013 08:26:23 -0500 Subject: [Python-Dev] PLY in stdlib (was cffi in stdlib) In-Reply-To: References: <96746EE7-E96A-4363-B0F7-D7950207C702@voidspace.org.uk> Message-ID: I'm in favor of PLY going into stdlib with the caveat that there are some things about it that should probably be cleaned up and modernized. For instance, the method by which it writes the cached parsing tables needs to be cleaned up. I still think putting the LALR(1) generator code into a common library usable by both PLY/RPLY would be a useful thing to do. That code is really hairy and non-trivial to understand without something like the Dragon book nearby (and even then it's not easy). So, if I were to make any kind of proposal, I would say, make a standard library module for just the LALR(1) generator code. If the PLY interface is needed to add pycparser or cffi to the standard library, that can be added too, but as a separate module that uses the more general LALR(1) library. Cheers, Dave On Jul 13, 2013, at 8:12 AM, Brett Cannon wrote: > > > > On Sat, Jul 13, 2013 at 4:24 AM, Michael Foord wrote: > > On 13 Jul 2013, at 07:41, Terry Reedy wrote: > > > On 7/13/2013 12:10 AM, Eric Snow wrote: > >> > >> On Feb 27, 2013 4:31 AM, "Michael Foord" > > >> > +1 PLY is capable and well tried-and-tested. We used it in Resolver > >> One to implement a pretty large grammar and it is (in my opinion) best > >> of breed in the Python parser generator world. Being stable and widely > >> used, with an "available maintainer", makes it an ideal candidate for > >> standard library inclusion. > >> > >> Is this still on the table? > > > > Who is the maintainer and what is his opinion? > > > The maintainer is David Beazley and as far as I recall he has not expressed an opinion on this particular question. It would obviously need his agreement (and maintenance commitment) if it is to fly. > > Just because we have now had two conflicting replies on this: David is down with PLY being added, but Alex Gaynor was working on a cleanup called RPLY for RPython. Basically David said the two of them should work together to clean up PLY and then it should be good to proposing for the stdlib (e.g. there are some backwards-compatibility hacks which should be removed). > > Below is David's original email on the topic from Feb 27: > > ------------------------------- > > Regarding the inclusion of PLY or some subcomponent of it in the standard library, it's not an entirely crazy idea in my opinion. LALR(1) parsers have been around for a long time, are generally known to anyone who's used yacc/bison, and would be useful outside the context of cffi or pycparser. PLY has also been around for about 12 years and is what I would call stable. It gets an update about every year or two, but that's about it. PLY is also relatively small--just two files and about 4300 lines of code (much of which could probably be scaled down a bit). > > The only downside to including PLY might be the fact that there are very few people walking around who've actually had to *implement* an LALR(1) parser generator. Some of the code for that is extremely hairy and mathematical. At this time, I don't think there are any bugs in it, but it's not the sort of thing that one wants to wander into casually. Also, there are some horrible hacks in PLY that I'd really like to get rid of, but am currently stuck with due to backwards compatibility issues. > > Alex Gaynor has been working on a PLY variant (RPLY) geared at RPython and which has a slightly different programming interface. I'd say if we were to go down this route, he and I should work together to put together some kind of more general "parsing.lalr" package (or similar) that cleans it up and makes it more suitable as a library for building different kinds of parsing tools on top of. > > Cheers, > Dave -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Sun Jul 14 08:54:33 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 14 Jul 2013 16:54:33 +1000 Subject: [Python-Dev] PLY in stdlib (was cffi in stdlib) In-Reply-To: References: <96746EE7-E96A-4363-B0F7-D7950207C702@voidspace.org.uk> Message-ID: On 13 July 2013 23:26, David Beazley wrote: > I'm in favor of PLY going into stdlib with the caveat that there are some > things about it that should probably be cleaned up and modernized. For > instance, the method by which it writes the cached parsing tables needs to > be cleaned up. I still think putting the LALR(1) generator code into a > common library usable by both PLY/RPLY would be a useful thing to do. That > code is really hairy and non-trivial to understand without something like > the Dragon book nearby (and even then it's not easy). > > So, if I were to make any kind of proposal, I would say, make a standard > library module for just the LALR(1) generator code. If the PLY interface > is needed to add pycparser or cffi to the standard library, that can be > added too, but as a separate module that uses the more general LALR(1) > library. > lrparsing is a more recent entry in the LR parsing stakes: https://pypi.python.org/pypi/lrparsing (although, as Russell put it in his PyCon AU lightning talk, if PLY had shown up ranked higher than 506 in his PyPI search for "parser", he probably would have just used that: http://pyvideo.org/video/2222/sunday-lightning-talks at about 2:15) (I plan to bug Russell about putting that up on one of the DVCS hosting sites next time I see him at BrisPy - for the moment, the source is available through the tarball/sdist) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Sun Jul 14 10:11:18 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 14 Jul 2013 18:11:18 +1000 Subject: [Python-Dev] Tweaking PEP 8 guidelines for use of leading underscores Message-ID: Currently, the naming section of PEP 8 doesn't say very much about what a leading underscore *means* in the Python standard library. I would like to add a new "Private interfaces" subsection under "Naming Conventions" to say the following: ================= Private interfaces Unless explicitly documented otherwise, a leading underscore on any name indicates that it is an internal implementation detail and backwards compatibility guarantees do not apply. It is strongly encouraged that private APIs (whether modules, classes, functions, attributes or other names) be clearly marked in this way, as many Python users rely on introspection to identify available functionality and may be mislead into believing an API without the leading underscore is in fact a public API with the standard backwards compatibility guarantees. All test modules are also considered private interfaces. Even though they typically lack the leading underscore, modules imported by another module are also considered an implementation detail. Other modules *should* not rely on indirect access to such modules unless they are an explicitly documented part of the API (such as ``os.path``). ================= Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia -------------- next part -------------- An HTML attachment was scrubbed... URL: From stephen at xemacs.org Sun Jul 14 10:30:22 2013 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Sun, 14 Jul 2013 17:30:22 +0900 Subject: [Python-Dev] Python 3.4 and Windows XP: just 45 days until EOL In-Reply-To: <7wmwps1he3.fsf@benfinney.id.au> References: <51DF469B.6010007@python.org> <87k3kwtqls.fsf@uwakimon.sk.tsukuba.ac.jp> <7wmwps1he3.fsf@benfinney.id.au> Message-ID: <87bo65tug1.fsf@uwakimon.sk.tsukuba.ac.jp> Ben Finney writes: > "Stephen J. Turnbull" writes: > > > I don't see any good reason to take into account what Microsoft does > > or doesn't support. > > It seems you're advocating a position quite ad odds with > . Not at all. The first thing that the PEP says about unsupporting code is: Unsupporting platforms If a certain platform that currently has special code in it is deemed to be without Python users, What a vendor supports is only a heuristic. Existence of users comes first. Note that the policy says that some Windows platforms *will* be supported. It doesn't say others will be unsupported (except implicitly: 3 years after the last version of Visual Studio capable of building releases for that platform goes out of extended support, the build infrastructure will be removed). I don't see a good reason to change the PEP. From ncoghlan at gmail.com Sun Jul 14 13:09:02 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 14 Jul 2013 21:09:02 +1000 Subject: [Python-Dev] Tweaking PEP 8 guidelines for use of leading underscores In-Reply-To: References: Message-ID: On 14 July 2013 18:11, Nick Coghlan wrote: > Currently, the naming section of PEP 8 doesn't say very much about what a > leading underscore *means* in the Python standard library. > > I would like to add a new "Private interfaces" subsection under "Naming > Conventions" to say the following: > > ================= > Private interfaces > > Unless explicitly documented otherwise, a leading underscore on any name > indicates that it is an internal implementation detail and backwards > compatibility guarantees do not apply. It is strongly encouraged that > private APIs (whether modules, classes, functions, attributes or other > names) be clearly marked in this way, as many Python users rely on > introspection to identify available functionality and may be mislead into > believing an API without the leading underscore is in fact a public API > with the standard backwards compatibility guarantees. > > All test modules are also considered private interfaces. > > Even though they typically lack the leading underscore, modules imported > by another module are also considered an implementation detail. Other > modules *should* not rely on indirect access to such modules unless they > are an explicitly documented part of the API (such as ``os.path``). > ================= > Slight adjustment to the proposed wording to ensure completely undocumented modules are also considered private: ================= Private interfaces Unless explicitly documented otherwise, a leading underscore on any name indicates that it is an internal implementation detail and backwards compatibility guarantees do not apply. It is strongly encouraged that private APIs (whether modules, classes, functions, attributes or other names) be clearly marked in this way, as Python users may rely on introspection to identify available functionality and may be misled into believing an API without a leading underscore is in fact a public API with the standard backwards compatibility guarantees. Even when their names do not start with a leading underscore, all test modules and all modules that are not covered in the documentation are also considered private interfaces. Similarly, the specific modules and external APIs imported by a module are always considered an implementation detail. Other modules should not rely on indirect access to such imported interfaces unless they are an explicitly documented part of the containing module's API (such as ``os.path`` or a package ``__init__`` module exposing functionality from submodules). ================= -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Sun Jul 14 15:13:13 2013 From: brett at python.org (Brett Cannon) Date: Sun, 14 Jul 2013 09:13:13 -0400 Subject: [Python-Dev] PLY in stdlib (was cffi in stdlib) In-Reply-To: References: <96746EE7-E96A-4363-B0F7-D7950207C702@voidspace.org.uk> Message-ID: On Sun, Jul 14, 2013 at 2:54 AM, Nick Coghlan wrote: > On 13 July 2013 23:26, David Beazley wrote: > >> I'm in favor of PLY going into stdlib with the caveat that there are some >> things about it that should probably be cleaned up and modernized. For >> instance, the method by which it writes the cached parsing tables needs to >> be cleaned up. I still think putting the LALR(1) generator code into a >> common library usable by both PLY/RPLY would be a useful thing to do. That >> code is really hairy and non-trivial to understand without something like >> the Dragon book nearby (and even then it's not easy). >> >> So, if I were to make any kind of proposal, I would say, make a standard >> library module for just the LALR(1) generator code. If the PLY interface >> is needed to add pycparser or cffi to the standard library, that can be >> added too, but as a separate module that uses the more general LALR(1) >> library. >> > > lrparsing is a more recent entry in the LR parsing stakes: > https://pypi.python.org/pypi/lrparsing (although, as Russell put it in > his PyCon AU lightning talk, if PLY had shown up ranked higher than 506 in > his PyPI search for "parser", he probably would have just used that: > http://pyvideo.org/video/2222/sunday-lightning-talks at about 2:15) > > (I plan to bug Russell about putting that up on one of the DVCS hosting > sites next time I see him at BrisPy - for the moment, the source is > available through the tarball/sdist) > It seems a bit new compared to PLY's 15 years of existence to be considered in the running. Plus Russell would need to change the license from GPL. -------------- next part -------------- An HTML attachment was scrubbed... URL: From dave at dabeaz.com Sun Jul 14 15:32:28 2013 From: dave at dabeaz.com (David Beazley) Date: Sun, 14 Jul 2013 08:32:28 -0500 Subject: [Python-Dev] PLY in stdlib (was cffi in stdlib) In-Reply-To: References: <96746EE7-E96A-4363-B0F7-D7950207C702@voidspace.org.uk> Message-ID: <8595C679-3D16-4312-98C8-146309FDF825@dabeaz.com> On Jul 14, 2013, at 8:13 AM, Brett Cannon wrote: > > > > On Sun, Jul 14, 2013 at 2:54 AM, Nick Coghlan wrote: > On 13 July 2013 23:26, David Beazley wrote: > I'm in favor of PLY going into stdlib with the caveat that there are some things about it that should probably be cleaned up and modernized. For instance, the method by which it writes the cached parsing tables needs to be cleaned up. I still think putting the LALR(1) generator code into a common library usable by both PLY/RPLY would be a useful thing to do. That code is really hairy and non-trivial to understand without something like the Dragon book nearby (and even then it's not easy). > > So, if I were to make any kind of proposal, I would say, make a standard library module for just the LALR(1) generator code. If the PLY interface is needed to add pycparser or cffi to the standard library, that can be added too, but as a separate module that uses the more general LALR(1) library. > > lrparsing is a more recent entry in the LR parsing stakes:https://pypi.python.org/pypi/lrparsing (although, as Russell put it in his PyCon AU lightning talk, if PLY had shown up ranked higher than 506 in his PyPI search for "parser", he probably would have just used that: http://pyvideo.org/video/2222/sunday-lightning-talks at about 2:15) > > (I plan to bug Russell about putting that up on one of the DVCS hosting sites next time I see him at BrisPy - for the moment, the source is available through the tarball/sdist) > > It seems a bit new compared to PLY's 15 years of existence to be considered in the running. Plus Russell would need to change the license from GPL. I honestly don't have any particular thoughts about PLY vs. other parser generators and the merits of their inclusion (or not) in the standard library. My impression has always been that the main interest in PLY was due to interest in seeing CFFI in the standard library. I'd say my main desire on the PLY side is that if it does go into the standard library, perhaps I could make it slightly less of mysterious black box and clean up a few bits. Cheers, Dave -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Sun Jul 14 15:49:21 2013 From: brett at python.org (Brett Cannon) Date: Sun, 14 Jul 2013 09:49:21 -0400 Subject: [Python-Dev] Tweaking PEP 8 guidelines for use of leading underscores In-Reply-To: References: Message-ID: On Sun, Jul 14, 2013 at 7:09 AM, Nick Coghlan wrote: > On 14 July 2013 18:11, Nick Coghlan wrote: > >> Currently, the naming section of PEP 8 doesn't say very much about what a >> leading underscore *means* in the Python standard library. >> >> I would like to add a new "Private interfaces" subsection under "Naming >> Conventions" to say the following: >> >> ================= >> Private interfaces >> >> Unless explicitly documented otherwise, a leading underscore on any name >> indicates that it is an internal implementation detail and backwards >> compatibility guarantees do not apply. It is strongly encouraged that >> private APIs (whether modules, classes, functions, attributes or other >> names) be clearly marked in this way, as many Python users rely on >> introspection to identify available functionality and may be mislead into >> believing an API without the leading underscore is in fact a public API >> with the standard backwards compatibility guarantees. >> >> All test modules are also considered private interfaces. >> >> Even though they typically lack the leading underscore, modules imported >> by another module are also considered an implementation detail. Other >> modules *should* not rely on indirect access to such modules unless they >> are an explicitly documented part of the API (such as ``os.path``). >> ================= >> > > Slight adjustment to the proposed wording to ensure completely > undocumented modules are also considered private: > > ================= > Private interfaces > > Unless explicitly documented otherwise, a leading underscore on any name > indicates that it is an internal implementation detail and backwards > compatibility guarantees do not apply. It is strongly encouraged that > private APIs (whether modules, classes, functions, attributes or other > names) be clearly marked in this way, as Python users may rely on > introspection to identify available functionality and may be misled into > believing an API without a leading underscore is in fact a public API with > the standard backwards compatibility guarantees. > > Even when their names do not start with a leading underscore, all test > modules and all modules that are not covered in the documentation are also > considered private interfaces. > > Similarly, the specific modules and external APIs imported by a module are > always considered an implementation detail. Other modules should not rely > on indirect access to such imported interfaces unless they are an > explicitly documented part of the containing module's API (such as > ``os.path`` or a package ``__init__`` module exposing functionality from > submodules). > ================= > +1 -------------- next part -------------- An HTML attachment was scrubbed... URL: From greg at krypto.org Sun Jul 14 18:15:23 2013 From: greg at krypto.org (Gregory P. Smith) Date: Sun, 14 Jul 2013 09:15:23 -0700 Subject: [Python-Dev] Tweaking PEP 8 guidelines for use of leading underscores In-Reply-To: References: Message-ID: +1 This is already how we've been behaving for years. On Sun, Jul 14, 2013 at 4:09 AM, Nick Coghlan wrote: > On 14 July 2013 18:11, Nick Coghlan wrote: > >> Currently, the naming section of PEP 8 doesn't say very much about what a >> leading underscore *means* in the Python standard library. >> >> I would like to add a new "Private interfaces" subsection under "Naming >> Conventions" to say the following: >> >> > Slight adjustment to the proposed wording to ensure completely > undocumented modules are also considered private: > > ================= > Private interfaces > > Unless explicitly documented otherwise, a leading underscore on any name > indicates that it is an internal implementation detail and backwards > compatibility guarantees do not apply. It is strongly encouraged that > private APIs (whether modules, classes, functions, attributes or other > names) be clearly marked in this way, > > as Python users may rely on introspection to identify available > functionality and may be misled into believing an API without a leading > underscore is in fact a public API with the standard backwards > compatibility guarantees. > While true, I'm not sure the last part of the sentence is necessary. Once we've established that a leading _ indicates something is private there isn't much point in explaining the various ways people might find them. I'm happy regardless of this bit being there. > Even when their names do not start with a leading underscore, all test > modules and all modules that are not covered in the documentation are also > considered private interfaces. > > Similarly, the specific modules and external APIs imported by a module are > always considered an implementation detail. Other modules should not rely > on indirect access to such imported interfaces unless they are an > explicitly documented part of the containing module's API (such as > ``os.path`` or a package ``__init__`` module exposing functionality from > submodules). > ================= > -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Mon Jul 15 01:48:33 2013 From: steve at pearwood.info (Steven D'Aprano) Date: Mon, 15 Jul 2013 09:48:33 +1000 Subject: [Python-Dev] Tweaking PEP 8 guidelines for use of leading underscores In-Reply-To: References: Message-ID: <51E338D1.5070900@pearwood.info> On 14/07/13 21:09, Nick Coghlan wrote: > Slight adjustment to the proposed wording to ensure completely undocumented > modules are also considered private: -1 on this adjustment. If somebody cannot be bothered writing a one-line doc string: "This module is private, don't touch." then they certainly shouldn't be allowed to use up a public name for a private module. I don't think we should be encouraging more private, undocumented modules. (Documentation is valuable even for private modules.) I'd go further, and say that no more private modules should be accepted for the std lib unless they have a leading underscore. I suppose for backwards compatibility reasons, we probably can't go through the std lib and rename private modules to make it clear they are private, but we don't have to accept new ones without the underscore. -- Steven From cs at zip.com.au Mon Jul 15 02:01:17 2013 From: cs at zip.com.au (Cameron Simpson) Date: Mon, 15 Jul 2013 10:01:17 +1000 Subject: [Python-Dev] Tweaking PEP 8 guidelines for use of leading underscores In-Reply-To: <51E338D1.5070900@pearwood.info> References: <51E338D1.5070900@pearwood.info> Message-ID: <20130715000117.GA46795@cskk.homeip.net> On 15Jul2013 09:48, Steven D'Aprano wrote: | On 14/07/13 21:09, Nick Coghlan wrote: | | >Slight adjustment to the proposed wording to ensure completely undocumented | >modules are also considered private: | | -1 on this adjustment. If somebody cannot be bothered writing a one-line doc string: | | "This module is private, don't touch." | | then they certainly shouldn't be allowed to use up a public name for a private module. I don't think we should be encouraging more private, undocumented modules. (Documentation is valuable even for private modules.) | | I'd go further, and say that no more private modules should be accepted for the std lib unless they have a leading underscore. I suppose for backwards compatibility reasons, we probably can't go through the std lib and rename private modules to make it clear they are private, but we don't have to accept new ones without the underscore. I disagree. A private module is a perfectly sane way to implement the internals of something, especially if it is subject to implementation change in the future. Clarification: is Nick classifying a module with docstrings by no content in the "modules" section of the Python doco as undocumented? That is what I would presume; I'd expect the code to be littered with docstrings anyway, but the module as a whole is not presented in the documentation and so should be private and not relied upon. Cheers, -- Cameron Simpson I sometimes wish that people would put a little more emphasis upon the observance of the law than they do upon its enforcement. - Calvin Coolidge From benhoyt at gmail.com Mon Jul 15 02:21:38 2013 From: benhoyt at gmail.com (Ben Hoyt) Date: Mon, 15 Jul 2013 12:21:38 +1200 Subject: [Python-Dev] Why is the return value of __contains__ coerced to boolean, but that of __lt__ and the like is not? Message-ID: I'm curious why the return value of __contains__ is coerced to True or False, whereas the return value of "normal" comparison operators like __lt__ and the like are not. The latter return the value directly without forcing it to be True or False. This makes overriding __contains__ significantly less flexible, so I'm wondering why it's designed or implemented that way. (I believe it's the cmp_outcome() function in Python/ceval.c that does this: http://hg.python.org/cpython/file/db9fe49069ed/Python/ceval.c#l4545) For example, the "peewee" ORM overloads __lt__ and the like so it can map Python expressions to SQL. But it can't do this with the "in" operator due to the result of "x in y" always returning True or False in Python. So it (ab)uses the "<<" operator to do this instead (See the peewee docs at http://peewee.readthedocs.org/en/latest/peewee/querying.html#column-lookups ). I'm sure there's a good reason for why "in" is different here, but I can't see why right now. -Ben -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Mon Jul 15 02:27:16 2013 From: brett at python.org (Brett Cannon) Date: Sun, 14 Jul 2013 20:27:16 -0400 Subject: [Python-Dev] Tweaking PEP 8 guidelines for use of leading underscores In-Reply-To: <20130715000117.GA46795@cskk.homeip.net> References: <51E338D1.5070900@pearwood.info> <20130715000117.GA46795@cskk.homeip.net> Message-ID: On Sun, Jul 14, 2013 at 8:01 PM, Cameron Simpson wrote: > On 15Jul2013 09:48, Steven D'Aprano wrote: > | On 14/07/13 21:09, Nick Coghlan wrote: > | > | >Slight adjustment to the proposed wording to ensure completely > undocumented > | >modules are also considered private: > | > | -1 on this adjustment. If somebody cannot be bothered writing a one-line > doc string: > | > | "This module is private, don't touch." > | > | then they certainly shouldn't be allowed to use up a public name for a > private module. I don't think we should be encouraging more private, > undocumented modules. (Documentation is valuable even for private modules.) > | > | I'd go further, and say that no more private modules should be accepted > for the std lib unless they have a leading underscore. I suppose for > backwards compatibility reasons, we probably can't go through the std lib > and rename private modules to make it clear they are private, but we don't > have to accept new ones without the underscore. > > I disagree. > > A private module is a perfectly sane way to implement the internals > of something, especially if it is subject to implementation change > in the future. > > Clarification: is Nick classifying a module with docstrings by no > content in the "modules" section of the Python doco as undocumented? > Yes. This has nothing to do with docstrings, just the official documentation at docs.python.org. > That is what I would presume; I'd expect the code to be littered > with docstrings anyway, but the module as a whole is not presented > in the documentation and so should be private and not relied upon. > Exactly. -------------- next part -------------- An HTML attachment was scrubbed... URL: From benhoyt at gmail.com Mon Jul 15 02:29:46 2013 From: benhoyt at gmail.com (Ben Hoyt) Date: Mon, 15 Jul 2013 12:29:46 +1200 Subject: [Python-Dev] Why is the return value of __contains__ coerced to boolean, but that of __lt__ and the like is not? In-Reply-To: References: Message-ID: Oh, just to show what I mean here with some code (same in both Python 2.x and 3.x): ----- from __future__ import print_function class C(object): def __contains__(self, x): print('__contains__({0!r})'.format(x)) return x def __lt__(self, x): print('__lt__({0!r})'.format(x)) return x c = C() print(42 in c) print(0 in c) print(c < 42) print(c < 0) ----- This prints the following: __contains__(42) True __contains__(0) False __lt__(42) 42 __lt__(0) 0 Whereas I'd kinda expect it to print: __contains__(42) 42 __contains__(0) 0 __lt__(42) 42 __lt__(0) 0 -Ben On Mon, Jul 15, 2013 at 12:21 PM, Ben Hoyt wrote: > I'm curious why the return value of __contains__ is coerced to True or > False, whereas the return value of "normal" comparison operators like > __lt__ and the like are not. The latter return the value directly without > forcing it to be True or False. > > This makes overriding __contains__ significantly less flexible, so I'm > wondering why it's designed or implemented that way. (I believe it's the > cmp_outcome() function in Python/ceval.c that does this: > http://hg.python.org/cpython/file/db9fe49069ed/Python/ceval.c#l4545) > > For example, the "peewee" ORM overloads __lt__ and the like so it can map > Python expressions to SQL. But it can't do this with the "in" operator due > to the result of "x in y" always returning True or False in Python. So it > (ab)uses the "<<" operator to do this instead (See the peewee docs at > http://peewee.readthedocs.org/en/latest/peewee/querying.html#column-lookups > ). > > I'm sure there's a good reason for why "in" is different here, but I can't > see why right now. > > -Ben > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Mon Jul 15 02:30:02 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 15 Jul 2013 10:30:02 +1000 Subject: [Python-Dev] Tweaking PEP 8 guidelines for use of leading underscores In-Reply-To: <51E338D1.5070900@pearwood.info> References: <51E338D1.5070900@pearwood.info> Message-ID: On 15 July 2013 09:48, Steven D'Aprano wrote: > On 14/07/13 21:09, Nick Coghlan wrote: > >> Slight adjustment to the proposed wording to ensure completely >> undocumented >> modules are also considered private: > > > -1 on this adjustment. If somebody cannot be bothered writing a one-line doc > string: > > "This module is private, don't touch." > > then they certainly shouldn't be allowed to use up a public name for a > private module. I don't think we should be encouraging more private, > undocumented modules. (Documentation is valuable even for private modules.) For context, this arose when I checked PEP 8 after the point was raised on distutils-sig that pip uses a public name for its implementation module (also pip, same as the CLI), but officially exposes no stable public programmatic API. At the moment, according to what little PEP 8 has to say about public vs private interfaces, that means a bundled pip would represent a new public API, despite the fact that the only documented interface for pip is the CLI, not the Python module API. Since we've been burned by people assuming private APIs are public in the past, I figured it made sense to get the actual standards we follow in practice documented properly, *before* anyone further gets the wrong idea from "import pip; help(pip)". Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Mon Jul 15 02:45:20 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 15 Jul 2013 10:45:20 +1000 Subject: [Python-Dev] Why is the return value of __contains__ coerced to boolean, but that of __lt__ and the like is not? In-Reply-To: References: Message-ID: On 15 July 2013 10:21, Ben Hoyt wrote: > I'm sure there's a good reason for why "in" is different here, but I can't > see why right now. It depends on what you mean by "good reason" - PEP 207 (which is what allows arbitrary objects to be returned from comparison operations) was entirely about replacing __cmp__ with the rich comparison methods, it doesn't mention __contains__ at all. At this point the main limitations are backwards compatibility (having existing containment tests suddenly start returning anything other than True or False would be problematic), along with the signature of CPython's sq_contains slot (it returns an integer rather than a PyObject pointer). Accordingly, to convert containment testing to a rich comparison operation would require a new protocol. That said, there is potential value in redefining containment in terms of a symmetric protocol (rather than the current only-controlled-by-the-container behaviour), so such a PEP may be worth writing. (it would initially be a topic for python-ideas rather than python-dev, though) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From tjreedy at udel.edu Mon Jul 15 04:10:16 2013 From: tjreedy at udel.edu (Terry Reedy) Date: Sun, 14 Jul 2013 22:10:16 -0400 Subject: [Python-Dev] Tweaking PEP 8 guidelines for use of leading underscores In-Reply-To: References: Message-ID: On 7/14/2013 7:09 AM, Nick Coghlan wrote: > Slight adjustment to the proposed wording to ensure completely > undocumented modules are also considered private: > > ================= > Private interfaces > > Unless explicitly documented otherwise, a leading underscore on any name > indicates that it is an internal implementation detail and backwards > compatibility guarantees do not apply. It is strongly encouraged that > private APIs (whether modules, classes, functions, attributes or other > names) be clearly marked in this way, as Python users may rely on > introspection to identify available functionality and may be misled into > believing an API without a leading underscore is in fact a public API > with the standard backwards compatibility guarantees. > > Even when their names do not start with a leading underscore, all test > modules and all modules that are not covered in the documentation are > also considered private interfaces. I was going to suggest adding 'and most idlelib ' between 'test' and 'modules', but the broader addition covers idlelib, which is not mentioned yet in the docs, even in the (unindexed) Idle chapter. When it is, I will try to remember to make explicit which names and interfaces are public and that the rest are private. -- Terry Jan Reedy From benhoyt at gmail.com Mon Jul 15 05:34:08 2013 From: benhoyt at gmail.com (Ben Hoyt) Date: Mon, 15 Jul 2013 15:34:08 +1200 Subject: [Python-Dev] Why is the return value of __contains__ coerced to boolean, but that of __lt__ and the like is not? In-Reply-To: References: Message-ID: Thanks, Nick -- that's helpful info. Writing such a PEP is a nice idea, but I think it'd be beyond me (I'm not familiar enough with CPython internals, protocols, etc). Can you explain what you mean by "symmetric protocol rather than the current only-controlled-by-the-container behaviour"? -Ben On Mon, Jul 15, 2013 at 12:45 PM, Nick Coghlan wrote: > On 15 July 2013 10:21, Ben Hoyt wrote: > > I'm sure there's a good reason for why "in" is different here, but I > can't > > see why right now. > > It depends on what you mean by "good reason" - PEP 207 (which is what > allows arbitrary objects to be returned from comparison operations) > was entirely about replacing __cmp__ with the rich comparison methods, > it doesn't mention __contains__ at all. > > At this point the main limitations are backwards compatibility (having > existing containment tests suddenly start returning anything other > than True or False would be problematic), along with the signature of > CPython's sq_contains slot (it returns an integer rather than a > PyObject pointer). > > Accordingly, to convert containment testing to a rich comparison > operation would require a new protocol. That said, there is potential > value in redefining containment in terms of a symmetric protocol > (rather than the current only-controlled-by-the-container behaviour), > so such a PEP may be worth writing. (it would initially be a topic for > python-ideas rather than python-dev, though) > > Cheers, > Nick. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Mon Jul 15 05:37:15 2013 From: steve at pearwood.info (Steven D'Aprano) Date: Mon, 15 Jul 2013 13:37:15 +1000 Subject: [Python-Dev] Tweaking PEP 8 guidelines for use of leading underscores In-Reply-To: References: <51E338D1.5070900@pearwood.info> Message-ID: <20130715033714.GA21977@ando> On Mon, Jul 15, 2013 at 10:30:02AM +1000, Nick Coghlan wrote: > On 15 July 2013 09:48, Steven D'Aprano wrote: > > On 14/07/13 21:09, Nick Coghlan wrote: > > > >> Slight adjustment to the proposed wording to ensure completely > >> undocumented > >> modules are also considered private: > > > > > > -1 on this adjustment. If somebody cannot be bothered writing a one-line doc > > string: > > > > "This module is private, don't touch." > > > > then they certainly shouldn't be allowed to use up a public name for a > > private module. I don't think we should be encouraging more private, > > undocumented modules. (Documentation is valuable even for private modules.) > > For context, this arose when I checked PEP 8 after the point was > raised on distutils-sig that pip uses a public name for its > implementation module (also pip, same as the CLI), but officially > exposes no stable public programmatic API. I don't think we should bless this as official policy, certainly not for the standard library. We can't tell external modules what to do, and possibly there's nothing we can do about existing undocumented private modules, short of renaming them, but I think that at least for new additions to the std lib module names should follow the same single-underscore convention as functions, classes etc. A single leading underscore introduces the private namespace, everything else is public. I've seen too much undocumented public code to ever assume that lack of documentation implies it is private. > At the moment, according to what little PEP 8 has to say about public > vs private interfaces, that means a bundled pip would represent a new > public API, despite the fact that the only documented interface for > pip is the CLI, not the Python module API. > > Since we've been burned by people assuming private APIs are public in > the past, I figured it made sense to get the actual standards we > follow in practice documented properly, *before* anyone further gets > the wrong idea from "import pip; help(pip)". I can tell you that people will do that regardless of what PEP 8 has to say about "not documented == private". Especially as worded. I'm sure I won't be the only person to reason that if a module has a docstring, it is therefore documented. It takes one character to explicitly document a module as private. That's much better than expecting people to treat their failure to find documentation as an implicit warning. -- Steven From steve at pearwood.info Mon Jul 15 05:44:16 2013 From: steve at pearwood.info (Steven D'Aprano) Date: Mon, 15 Jul 2013 13:44:16 +1000 Subject: [Python-Dev] Tweaking PEP 8 guidelines for use of leading underscores In-Reply-To: <20130715000117.GA46795@cskk.homeip.net> References: <51E338D1.5070900@pearwood.info> <20130715000117.GA46795@cskk.homeip.net> Message-ID: <20130715034416.GB21977@ando> On Mon, Jul 15, 2013 at 10:01:17AM +1000, Cameron Simpson wrote: > On 15Jul2013 09:48, Steven D'Aprano wrote: > | I'd go further, and say that no more private modules should be > | accepted for the std lib unless they have a leading underscore. I > | suppose for backwards compatibility reasons, we probably can't go > | through the std lib and rename private modules to make it clear they > | are private, but we don't have to accept new ones without the > | underscore. > > I disagree. > > A private module is a perfectly sane way to implement the internals > of something, especially if it is subject to implementation change > in the future. Of course private modules are sane. I never suggested "no new private modules at all". But putting them in the same namespace as public modules is not, just to save a leading underscore in the file name. You don't even have to use the underscore in your own code: import _stuff as stuff is allowed, and doesn't make _stuff.py public since imported modules are considered implementation details by default. -- Steven From steve at pearwood.info Mon Jul 15 05:47:56 2013 From: steve at pearwood.info (Steven D'Aprano) Date: Mon, 15 Jul 2013 13:47:56 +1000 Subject: [Python-Dev] Why is the return value of __contains__ coerced to boolean, but that of __lt__ and the like is not? In-Reply-To: References: Message-ID: <20130715034756.GC21977@ando> On Mon, Jul 15, 2013 at 03:34:08PM +1200, Ben Hoyt wrote: > Thanks, Nick -- that's helpful info. Writing such a PEP is a nice idea, but > I think it'd be beyond me (I'm not familiar enough with CPython internals, > protocols, etc). > > Can you explain what you mean by "symmetric protocol rather than the > current only-controlled-by-the-container behaviour"? Most operators can be controlled by either the left-hand or right-hand operand. For example, x + y can end up calling either x.__add__(y) or y.__radd_(x). The `in` operator is an exception, it only ever calls the container: x in y => y.__contains__(x) but never x.__contained_by__(y) -- Steven From ncoghlan at gmail.com Mon Jul 15 05:53:56 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 15 Jul 2013 13:53:56 +1000 Subject: [Python-Dev] Tweaking PEP 8 guidelines for use of leading underscores In-Reply-To: <20130715033714.GA21977@ando> References: <51E338D1.5070900@pearwood.info> <20130715033714.GA21977@ando> Message-ID: On 15 Jul 2013 13:43, "Steven D'Aprano" wrote: > > On Mon, Jul 15, 2013 at 10:30:02AM +1000, Nick Coghlan wrote: > > On 15 July 2013 09:48, Steven D'Aprano wrote: > > > On 14/07/13 21:09, Nick Coghlan wrote: > > > > > >> Slight adjustment to the proposed wording to ensure completely > > >> undocumented > > >> modules are also considered private: > > > > > > > > > -1 on this adjustment. If somebody cannot be bothered writing a one-line doc > > > string: > > > > > > "This module is private, don't touch." > > > > > > then they certainly shouldn't be allowed to use up a public name for a > > > private module. I don't think we should be encouraging more private, > > > undocumented modules. (Documentation is valuable even for private modules.) > > > > For context, this arose when I checked PEP 8 after the point was > > raised on distutils-sig that pip uses a public name for its > > implementation module (also pip, same as the CLI), but officially > > exposes no stable public programmatic API. > > I don't think we should bless this as official policy, certainly not for > the standard library. We can't tell external modules what to do, and > possibly there's nothing we can do about existing undocumented private > modules, short of renaming them, but I think that at least for new > additions to the std lib module names should follow the same > single-underscore convention as functions, classes etc. A single leading > underscore introduces the private namespace, everything else is public. > > I've seen too much undocumented public code to ever assume that lack of > documentation implies it is private. > > > > At the moment, according to what little PEP 8 has to say about public > > vs private interfaces, that means a bundled pip would represent a new > > public API, despite the fact that the only documented interface for > > pip is the CLI, not the Python module API. > > > > Since we've been burned by people assuming private APIs are public in > > the past, I figured it made sense to get the actual standards we > > follow in practice documented properly, *before* anyone further gets > > the wrong idea from "import pip; help(pip)". > > I can tell you that people will do that regardless of what PEP 8 has to > say about "not documented == private". Especially as worded. I'm sure I > won't be the only person to reason that if a module has a docstring, it > is therefore documented. > > It takes one character to explicitly document a module as private. > That's much better than expecting people to treat their failure to > find documentation as an implicit warning. We're considering bundling the pip CLI for the convenience of educators and beginners, not adding their implementation module to the standard library. That means we need a bright shining line for what constitutes "in the standard library". Yes, it's better if people properly use a leading underscore, but the fact is many don't (including idlelib, as Terry noted) and it becomes a disruptive change to add it later if people don't start with it. The pip and IDLE developers have better things to be doing than changing the names of undocumented APIs just because the application is distributed along with CPython. Cheers, Nick. > > > > -- > Steven > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Mon Jul 15 05:55:37 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 15 Jul 2013 13:55:37 +1000 Subject: [Python-Dev] Tweaking PEP 8 guidelines for use of leading underscores In-Reply-To: <20130715034416.GB21977@ando> References: <51E338D1.5070900@pearwood.info> <20130715000117.GA46795@cskk.homeip.net> <20130715034416.GB21977@ando> Message-ID: On 15 Jul 2013 13:46, "Steven D'Aprano" wrote: > > On Mon, Jul 15, 2013 at 10:01:17AM +1000, Cameron Simpson wrote: > > On 15Jul2013 09:48, Steven D'Aprano wrote: > > > | I'd go further, and say that no more private modules should be > > | accepted for the std lib unless they have a leading underscore. I > > | suppose for backwards compatibility reasons, we probably can't go > > | through the std lib and rename private modules to make it clear they > > | are private, but we don't have to accept new ones without the > > | underscore. > > > > I disagree. > > > > A private module is a perfectly sane way to implement the internals > > of something, especially if it is subject to implementation change > > in the future. > > Of course private modules are sane. I never suggested "no new private > modules at all". But putting them in the same namespace as public > modules is not, just to save a leading underscore in the file name. It's not to save a leading underscore - it's to avoid renaming existing packages like pip and idlelib. Cheers, Nick. > > You don't even have to use the underscore in your own code: > > import _stuff as stuff > > is allowed, and doesn't make _stuff.py public since imported modules are > considered implementation details by default. > > > -- > Steven > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Mon Jul 15 06:17:55 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 15 Jul 2013 14:17:55 +1000 Subject: [Python-Dev] Tweaking PEP 8 guidelines for use of leading underscores In-Reply-To: References: Message-ID: On 15 July 2013 02:15, Gregory P. Smith wrote: > On Sun, Jul 14, 2013 at 4:09 AM, Nick Coghlan wrote: >> as Python users may rely on introspection to identify available >> functionality and may be misled into believing an API without a leading >> underscore is in fact a public API with the standard backwards compatibility >> guarantees. > > > While true, I'm not sure the last part of the sentence is necessary. Once > we've established that a leading _ indicates something is private there > isn't much point in explaining the various ways people might find them. I'm > happy regardless of this bit being there. You'd be surprised how many non-core devs react with astonishment when I suggest that not documenting something isn't enough to avoid having users consider it a supported public API - they usually get it after I point out how far you can usually get just by using dir() and help() to play with a new library at the interactive prompt instead of looking at out-of-band docs. I figure including the second part will help prevent some "But why?" reactions in the future. I think Steven has a reasonable point about being clearer that an explicit leading underscore is the preferred solution for any new private modules, though, so here's an updated proposal: ================= Private interfaces Unless explicitly documented otherwise, a leading underscore on any name indicates that it is an internal implementation detail and any backwards compatibility guarantees do not apply. It is strongly encouraged that private APIs (whether modules, classes, functions, attributes or other names) be clearly marked in this way, as Python users may rely on introspection to identify available functionality and may be misled into believing an API without a leading underscore is in fact a public API with the standard backwards compatibility guarantees. While the explicit use of a leading underscore in the names of private modules is preferred, all test modules and all modules that are not explicitly covered in the documentation are also considered private interfaces, even when their names do not start with a leading underscore and even if they include a module level documentation string. This includes submodules of packages that are documented as if they were a single module. Similarly, the specific modules and external APIs imported by a module are always considered an implementation detail. Other modules should not rely on indirect access to such imported interfaces unless they are an explicitly documented part of the containing module's API (such as ``os.path`` or a package ``__init__`` module that exposes functionality from submodules). ================= -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From regebro at gmail.com Mon Jul 15 07:12:50 2013 From: regebro at gmail.com (Lennart Regebro) Date: Mon, 15 Jul 2013 07:12:50 +0200 Subject: [Python-Dev] Tweaking PEP 8 guidelines for use of leading underscores In-Reply-To: References: Message-ID: On Mon, Jul 15, 2013 at 6:17 AM, Nick Coghlan wrote: > ================= > Private interfaces > > Unless explicitly documented otherwise, a leading underscore on any > name indicates that it is an internal implementation detail and any > backwards compatibility guarantees do not apply. It is strongly > encouraged that private APIs (whether modules, classes, functions, > attributes or other names) be clearly marked in this way, as Python > users may rely on introspection to identify available functionality > and may be misled into believing an API without a leading underscore > is in fact a public API with the standard backwards compatibility > guarantees. Very good. > While the explicit use of a leading underscore in the names of private > modules is preferred, all test modules and all modules that are not > explicitly covered in the documentation are also considered private > interfaces, even when their names do not start with a leading > underscore and even if they include a module level documentation > string. This includes submodules of packages that are documented as if > they were a single module. But wait, aren't this about how to use other peoples modules, more than a style guide of how to write your own modules? I don't think this belongs in PEP 8 at all. //Lennart From stephen at xemacs.org Mon Jul 15 07:16:37 2013 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Mon, 15 Jul 2013 14:16:37 +0900 Subject: [Python-Dev] Tweaking PEP 8 guidelines for use of leading underscores In-Reply-To: References: <51E338D1.5070900@pearwood.info> <20130715000117.GA46795@cskk.homeip.net> <20130715034416.GB21977@ando> Message-ID: <871u70tnbe.fsf@uwakimon.sk.tsukuba.ac.jp> Nick Coghlan writes: >> Of course private modules are sane. I never suggested "no new private >> modules at all". But putting them in the same namespace as public >> modules is not, just to save a leading underscore in the file name. > It's not to save a leading underscore - it's to avoid renaming > existing packages like pip and idlelib. "existing" is not a subset of "new." Am I missing something? What Steven is suggesting is that there are two kinds of private package in the stdlib: those that can conveniently be given names that indicate that they are private (generally, "new" packages), and those that can't ("existing" packages). For the latter IIRC he suggested adding a line at the top of the module documentation "This module is *private*, and you cannot depend on API compatibility with future or past releases of this module. The public API is in ... [alternatively, There is no public API]." I really don't see how one can object to this suggestion, given an appropriate definition of "convenient enough to get a _name". From raymond.hettinger at gmail.com Mon Jul 15 08:01:21 2013 From: raymond.hettinger at gmail.com (Raymond Hettinger) Date: Sun, 14 Jul 2013 23:01:21 -0700 Subject: [Python-Dev] PLY in stdlib (was cffi in stdlib) In-Reply-To: <8595C679-3D16-4312-98C8-146309FDF825@dabeaz.com> References: <96746EE7-E96A-4363-B0F7-D7950207C702@voidspace.org.uk> <8595C679-3D16-4312-98C8-146309FDF825@dabeaz.com> Message-ID: <268FFB76-3125-4BB1-9706-4BFE86696652@gmail.com> On Jul 14, 2013, at 6:32 AM, David Beazley wrote: > I honestly don't have any particular thoughts about PLY vs. other parser generators and the merits of their inclusion (or not) in the standard library. I would love to have PLY in the standard library. It would open up a whole new world to some users and be the basis for tool generation for others. Raymond -------------- next part -------------- An HTML attachment was scrubbed... URL: From benjamin at python.org Mon Jul 15 08:06:28 2013 From: benjamin at python.org (Benjamin Peterson) Date: Sun, 14 Jul 2013 23:06:28 -0700 Subject: [Python-Dev] Doubt about closure/function generation in bytecode In-Reply-To: References: Message-ID: 2013/7/12 Andrea Griffini : > Is there any semantic difference between > > BUILD_TUPLE 0 > LOAD_CONST > MAKE_CLOSURE 0 > > and > > LOAD_CONST > MAKE_FUNCTION 0 > > ? > > In other words is there any difference between a function and a closure that > doesn't capture anything? Is it just a speed optimization for a common case? It probably has more to do with the fact that nested scopes with added much later in Python's history than functions. > -- Regards, Benjamin From tjreedy at udel.edu Mon Jul 15 09:31:03 2013 From: tjreedy at udel.edu (Terry Reedy) Date: Mon, 15 Jul 2013 03:31:03 -0400 Subject: [Python-Dev] Tweaking PEP 8 guidelines for use of leading underscores In-Reply-To: References: Message-ID: On 7/15/2013 12:17 AM, Nick Coghlan wrote: > You'd be surprised how many non-core devs react with astonishment > when I suggest that not documenting something isn't enough to avoid > having users consider it a supported public API - they usually get it > after I point out how far you can usually get just by using dir() and > help() to play with a new library at the interactive prompt instead of > looking at out-of-band docs. I figure including the second part will > help prevent some "But why?" reactions in the future. I agree with the goal of preventing such reactions, but I suggest slightly different means to do so. I just looked and idlelib.__init__.py consists of """# Dummy file to make this a package. """ help(idlelib) prints the comment (which surprised me, but it seems to be smart enough to look for a 'doc comment' when it does not find a doc string). I now plan to change the comment to a docstring so 'idlelib.__doc__' also works and change the text to something like """A package of private modules that implement the IDLE integrated shell and editor application. See the manual for the public entry points.""" (or maybe list them briefly here in the doc string). This should properly inform people who introspect. Also, the manual needs a small, indexed, section on idlelib that says much the same thing along with documenting what *is* public (which is not completely clear to me at the moment). > I think Steven has a reasonable point about being clearer that an > explicit leading underscore is the preferred solution for any new > private modules, though, so here's an updated proposal: The public/private distinction is not always as clean-cut as seems to be assumed in this discussion. Test and idlelib both have public entry points to an application based on private files. ('Running the Python test suite' is a batch application.) So I do not think either should be _ prefixed. For both, and anything similar, a proper docstring and manual entry explaining the nuances should be completely acceptable as a substitute for a blunt, all-or-nothing flag. I also think that marking something private, either way, or partially private with words, should extend to its contents. In particular, the 100s of implementation files in test and idlelib need not be _ marked as as they would be covered once test and idlelib are properly documented. The special _ mark makes sense when it is the exception, but not in a context where it would be the overwhelming norm, because everything but one or two files is private. > ================= > Private interfaces > > Unless explicitly documented otherwise, a leading underscore on any > name indicates that it is an internal implementation detail and any > backwards compatibility guarantees do not apply. It is strongly > encouraged that private APIs (whether modules, classes, functions, > attributes or other names) be clearly marked in this way, as Python > users may rely on introspection to identify available functionality > and may be misled into believing an API without a leading underscore > is in fact a public API with the standard backwards compatibility > guarantees. > > While the explicit use of a leading underscore in the names of private > modules is preferred, all test modules and all modules that are not > explicitly covered in the documentation are also considered private > interfaces, even when their names do not start with a leading > underscore and even if they include a module level documentation > string. This includes submodules of packages that are documented as if > they were a single module. Since I am planning to explicitly cover idlelib in the doc, to list the public interface, just as is done for test, I would like it explicitly mentioned along with test. I do not think absence of documentation is a good signal. I would rather say that any private or partially private package that does not have a _ name *must* have a docstring saying that it is all or mostly private. (Or be a subpackage or module of something so marked.) And presuming that it is associated with something that is documented, then the doc should state that the implementation files are private. In other words, I suggest that "modules that are not explicitly covered in the documentation ... [whose] names do not start with a leading underscore" should be empty. And I think "even if they include a module level documentation string." is backwards. Such files *should* have a docstring that serves as a substitute for a _ prefix. > Similarly, the specific modules and external APIs imported by a module > are always considered an implementation detail. Other modules should > not rely on indirect access to such imported interfaces unless they > are an explicitly documented part of the containing module's API (such > as ``os.path`` or a package ``__init__`` module that exposes > functionality from submodules). > ================= -- Terry Jan Reedy From ncoghlan at gmail.com Mon Jul 15 09:32:53 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 15 Jul 2013 17:32:53 +1000 Subject: [Python-Dev] PLY in stdlib (was cffi in stdlib) In-Reply-To: <268FFB76-3125-4BB1-9706-4BFE86696652@gmail.com> References: <96746EE7-E96A-4363-B0F7-D7950207C702@voidspace.org.uk> <8595C679-3D16-4312-98C8-146309FDF825@dabeaz.com> <268FFB76-3125-4BB1-9706-4BFE86696652@gmail.com> Message-ID: On 15 July 2013 16:01, Raymond Hettinger wrote: > > On Jul 14, 2013, at 6:32 AM, David Beazley wrote: > > I honestly don't have any particular thoughts about PLY vs. other parser > generators and the merits of their inclusion (or not) in the standard > library. > > > I would love to have PLY in the standard library. > It would open up a whole new world to some users > and be the basis for tool generation for others. Agreed (and the fact it's a prerequisite for pycparsing and cffi just seals the deal). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From p.f.moore at gmail.com Mon Jul 15 09:37:01 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Mon, 15 Jul 2013 08:37:01 +0100 Subject: [Python-Dev] PLY in stdlib (was cffi in stdlib) In-Reply-To: <268FFB76-3125-4BB1-9706-4BFE86696652@gmail.com> References: <96746EE7-E96A-4363-B0F7-D7950207C702@voidspace.org.uk> <8595C679-3D16-4312-98C8-146309FDF825@dabeaz.com> <268FFB76-3125-4BB1-9706-4BFE86696652@gmail.com> Message-ID: On 15 July 2013 07:01, Raymond Hettinger wrote: > I would love to have PLY in the standard library. > It would open up a whole new world to some users > and be the basis for tool generation for others. > +1. Parser generators are useful tools - parsers are right on the boundary of "easy enough to understand why you'd like one, but hard enough to put you off implementing our own". And there is value in tools like this being in the stdlib as opposed to a 3rd party dependency. Paul -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Mon Jul 15 09:48:59 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 15 Jul 2013 17:48:59 +1000 Subject: [Python-Dev] Tweaking PEP 8 guidelines for use of leading underscores In-Reply-To: References: Message-ID: On 15 July 2013 17:31, Terry Reedy wrote: > I do not think absence of documentation is a good signal. I would rather say > that any private or partially private package that does not have a _ name > *must* have a docstring saying that it is all or mostly private. (Or be a > subpackage or module of something so marked.) And presuming that it is > associated with something that is documented, then the doc should state that > the implementation files are private. > > In other words, I suggest that "modules that are not explicitly covered in > the documentation ... [whose] names do not start with a leading underscore" > should be empty. And I think "even if they include a module level > documentation string." is backwards. Such files *should* have a docstring > that serves as a substitute for a _ prefix. I (now) agree - a disclaimer in the docstring is a better signal than an absence of documentation, and one we can reasonably make a requirement even for external applications that we bundle or otherwise bless as the "obvious way to do it" (as is happening with pip). ================= Private interfaces Unless explicitly documented otherwise, a leading underscore on any name indicates that it is an internal implementation detail and any backwards compatibility guarantees do not apply. It is strongly encouraged that private APIs (whether modules, classes, functions, attributes or other names) be clearly marked in this way, as Python users may rely on introspection to identify available functionality and may be misled into believing an API without a leading underscore is in fact a public API with the standard backwards compatibility guarantees. While the explicit use of a leading underscore is the preferred solution, the names of some private (or partially private) modules (such as ``test`` and ``idlelib``) lack the leading underscore either for historical reasons or because they expose a public command line interface through the ``-m`` switch. Such modules should include an explicit disclaimer in their module docstring, indicating that they do not use the leading underscore convention and noting where the definition of the public API (if any) can be found (the public API definition may also be part of the module docstring). As a general principle, the specific modules and external APIs imported by a module are always considered an implementation detail. Other modules should not rely on indirect access to such imported interfaces unless they are an explicitly documented part of the containing module's API (such as ``os.path`` or a package ``__init__`` module that exposes functionality from submodules). ================= Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From p.f.moore at gmail.com Mon Jul 15 10:26:50 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Mon, 15 Jul 2013 09:26:50 +0100 Subject: [Python-Dev] Adding Python scripts to PATHEXT on Windows Message-ID: With the addition of the Python launcher (PEP 397), Python scripts on Windows are executable in much the same way as on Unix. However, "out of the box" it is still not quite possible to use Python scripts. This is because the .py extension is not registered with Windows in the PATHEXT environment variable (which holds all file extensions which should be looked for on PATH at the command line). There are two main consequences to this: 1. Under powershell, ".py" files on PATH are not found at all (this is not the case with cmd.exe, and may be technically a bug in powershell - but it's unlikely to change nevertheless). 2. Under both powershell and cmd, users have to explicitly include the extension when invoking the command. I would like to propose that the .py extension be added to PATHEXT as part of the "Register Python Extensions" option in the Python MSI installer. This would mean that Python users had a clean way of writing cross-platform "wrapper scripts" and command-line applications out of the box, just by writing a normal Python script with a #! line at the top. Does anyone have any objections to this? I could try to write a patch, but I know next to nothing about building MSIs, so if any of the installer experts could help that would be fantastic. Paul PS There would still be one difference, in that scripts intended to be used as commands would be named with '.py' on Windows and with no extension on Unix, but that's a pretty trivial difference, and reflects fundamental platform differences rather than configuration. So I don't propose worrying about this. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Mon Jul 15 10:40:15 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 15 Jul 2013 18:40:15 +1000 Subject: [Python-Dev] Adding Python scripts to PATHEXT on Windows In-Reply-To: References: Message-ID: Sounds good to me - in the long run it may allow us to drop the Windows exe wrappers when installing scripts on Windows. Cheers, Nick. -------------- next part -------------- An HTML attachment was scrubbed... URL: From python.00 at klix.ch Mon Jul 15 10:32:12 2013 From: python.00 at klix.ch (Gerald Klix) Date: Mon, 15 Jul 2013 10:32:12 +0200 Subject: [Python-Dev] Adding Python scripts to PATHEXT on Windows In-Reply-To: References: Message-ID: <8FC77D8F-B447-4ACD-87DF-42A04ABB29C4@klix.ch> +1 BTW: I use lots of "*.py-executables" on Linux and OSX, so different extension don't bother me. Gerald Am 15.07.2013 um 10:26 schrieb Paul Moore : > With the addition of the Python launcher (PEP 397), Python scripts on > Windows are executable in much the same way as on Unix. However, "out of > the box" it is still not quite possible to use Python scripts. This is > because the .py extension is not registered with Windows in the PATHEXT > environment variable (which holds all file extensions which should be > looked for on PATH at the command line). > > There are two main consequences to this: > > 1. Under powershell, ".py" files on PATH are not found at all (this is not > the case with cmd.exe, and may be technically a bug in powershell - but > it's unlikely to change nevertheless). > 2. Under both powershell and cmd, users have to explicitly include the > extension when invoking the command. > > I would like to propose that the .py extension be added to PATHEXT as part > of the "Register Python Extensions" option in the Python MSI installer. > This would mean that Python users had a clean way of writing cross-platform > "wrapper scripts" and command-line applications out of the box, just by > writing a normal Python script with a #! line at the top. > > Does anyone have any objections to this? I could try to write a patch, but > I know next to nothing about building MSIs, so if any of the installer > experts could help that would be fantastic. > > Paul > > PS There would still be one difference, in that scripts intended to be used > as commands would be named with '.py' on Windows and with no extension on > Unix, but that's a pretty trivial difference, and reflects fundamental > platform differences rather than configuration. So I don't propose worrying > about this. > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/python.00%40klix.ch From p.f.moore at gmail.com Mon Jul 15 10:42:33 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Mon, 15 Jul 2013 09:42:33 +0100 Subject: [Python-Dev] Adding Python scripts to PATHEXT on Windows In-Reply-To: References: Message-ID: On 15 July 2013 09:40, Nick Coghlan wrote: > Sounds good to me - in the long run it may allow us to drop the Windows > exe wrappers when installing scripts on Windows. That's my motivation. In particular removing pip's exe wrappers to ease some of the issues being flagged on distutils-sig. Paul -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Mon Jul 15 12:04:54 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 15 Jul 2013 20:04:54 +1000 Subject: [Python-Dev] Adding Python scripts to PATHEXT on Windows In-Reply-To: References: Message-ID: On 15 July 2013 18:42, Paul Moore wrote: > On 15 July 2013 09:40, Nick Coghlan wrote: >> >> Sounds good to me - in the long run it may allow us to drop the Windows >> exe wrappers when installing scripts on Windows. > > > That's my motivation. In particular removing pip's exe wrappers to ease some > of the issues being flagged on distutils-sig. Ah, true. For some reason I was thinking we kept __main__.__file__ open, but we don't - we close it as soon as we have read and compiled it. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Mon Jul 15 12:22:57 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 15 Jul 2013 20:22:57 +1000 Subject: [Python-Dev] Refactoring test.support into a subpackage Message-ID: At the PyCon AU sprints, some of the sprinters worked on a plan Chris Jerdonek and I came up with months ago to convert test.support from a module into a subpackage. This plan arose from some nasty test suite hacks in the lead up to the release of Python 3.3, where some of the pkgutil tests ended up in test.test_runpy so they could use the package creation infrastructure I have in there without needing to extract a helper module with a release imminent. The scope of the plan covers a few distinct steps: 1. Move Lib/test/support.py to Lib/test/support/__init__.py (http://bugs.python.org/issue15494) 2. Add a new (documented) test.support.pkg_helper submodule with the package creation support code (http://bugs.python.org/issue15403 and http://bugs.python.org/issue15376) 3. Move the pkgutil tests to test.test_pkgutil where they belong (http://bugs.python.org/issue15358) 4. Move the existing test.script_helper and test.bytecode_helper support modules into the test.support subpackage and document them (no issue for that as yet) These changes would be made on both 3.3 and default, to avoid future merge conflicts (note that Indra also posted an updated patch for one pending test suite update to cope with the relocation: http://bugs.python.org/issue15415) My main motivation for doing this is to give a bit more visibility to the helpers that already exist, as well as making them easier to find when we want to look into the details of what a test is doing. At the moment, they tend to get lost in the sea of actual test files in the main test directory. (Chris and I initially held off on this while waiting for the final 3.2 release, and then that got delayed and it wasn't until Indra, Michael and Stephen started looking into it at the sprints that I started paying attention again) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ronaldoussoren at mac.com Mon Jul 15 13:13:13 2013 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Mon, 15 Jul 2013 13:13:13 +0200 Subject: [Python-Dev] Hooking into super() attribute resolution In-Reply-To: <5339a8b72f264faa99a8720b5d1d161d@BLUPR03MB035.namprd03.prod.outlook.com> References: <46EC256E-9ACE-4C4C-BE3D-325D7DB87B85@mac.com> <0fe31345247846658b331aa358c38c66@BLUPR03MB035.namprd03.prod.outlook.com> <5fe3649f4c3c41ba927ca62fd762699c@BLUPR03MB035.namprd03.prod.outlook.com> <7946F238-3DA7-4198-9CED-23B33246B26F@mac.com> <5339a8b72f264faa99a8720b5d1d161d@BLUPR03MB035.namprd03.prod.outlook.com> Message-ID: On 9 Jul, 2013, at 1:21, Steve Dower wrote: > > Except that if it's on a metaclass, the 'instance' it has access to is cls. The descriptor side of things is more interesting, but I see no reason why super can't do that itself, since it knows the actual instance to call __get__ with. (Presumably it already does this with the __dict__ lookup, since that won't call __get__ either.) > > Explaining the new method is easiest if the default implementation is (literally): > > def __getlocalname__(self, name): > try: > return self.__dict__[name] > except KeyError: > raise AttributeError(name) > > which does not do any descriptor resolution (and is only a small step from simply replacing __dict__ with a custom object, which is basically where we started). The only change I've really suggested is making it an instance method that can be implemented on a metaclass if you want it for class members. I've documented this (with a different name) in the current PEP draf () and am working on an implementation. Using a lookup method on the metaclass has a nice side-effect: the same method could be used by object.__getattribute__ (aka PyObject_GenericGetAttr) as well which could simplify my primary usecase for this new API. There is a problem with that though: the type attribute cache in Object/typeobject.c, using that cache isn't valid if _PyType_Lookup calls a method instead of peeking in tp_dict (the cache cannot be in the default __getlocalname__ implementation because a primary usecase for the cache appears to be avoiding walking the MRO [1]). I haven't decided yet what to do about this, a number of options: * Don't use __getlocalname__ in _PyType_Lookup * Use __getlocalname__ as a fallback after peeking in tp_dict (that is, use it like __getattr__ instead of __getattribute__) and only cache the attribute when it is fetched from tp_dict. * Don't add a default __getlocalname__ and disable the attribute cache for types with a metatype that does have this method (this might be non-trivial because a metatype might grow a __getlocalname__ slot after instances of the metatype have been created; that would be ugly code but is possbile) Ronald [1] "appears" because I haven't found documentation for the cache yet From ronaldoussoren at mac.com Mon Jul 15 17:12:45 2013 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Mon, 15 Jul 2013 17:12:45 +0200 Subject: [Python-Dev] PEP 447: hooking into super() attribute resolution Message-ID: <2055CD40-2081-45DC-8B9A-889085FC7CEE@mac.com> Hi, I've posted a new update of my proposal to add a way to override the attribute resolution proces by super(). I've rewritten the PEP and implementation based on feedback by Steve Dower. In the current edition of the proposal the hook is a method on the type, defined in a metaclass for types. A simple demo: class meta (type): def __locallookup__(cls, name): return type.__locallookup__(cls, name.upper()) class S (metaclass=meta): def m(self): return 42 def M(self): return "fortytwo" class SS (S): def m(self): return 21 def M(self): return "twentyone" o = SS() print("self", o.m()) print("super", super(SS, o).m()) The last line currently prints "super 42" and would print "super fortytwo" with my proposal. A major open issue: the __locallookup__ method could also be used for normal attribute resolution, but that probably causes issues with attribute caching (see the PEP for details). I haven't worked out yet if it is worthwhile to tweak the proposal to fix the caching issues (even though the proposal currently says that PyObject_GenericGetAttr will use the new method). Fixing the caching issue definitly would help in my primary use case by reducing code duplication, but might end up making the API unnecessarily complex. Ronald PEP: 447 Title: Hooking into super attribute resolution Version: $Revision$ Last-Modified: $Date$ Author: Ronald Oussoren Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 12-Jun-2013 Post-History: 2-Jul-2013, ? Abstract ======== In current python releases the attribute resolution of the `super class`_ peeks in the ``__dict__`` attribute of classes on the MRO to look for attributes. This PEP introduces a hook that classes can use to override that behavior for specific classes. Rationale ========= Peeking in the class ``__dict__`` works for regular classes, but can cause problems when a class dynamicly looks up attributes in a ``__getattribute__`` method. This new hook makes it possible to affect attribute lookup for both normal attribute lookup and lookup through the `super class`_. The superclass attribute lookup hook ==================================== Both ``super.__getattribute__`` and ``object.__getattribute__`` (or `PyObject_GenericGetAttr`_ in C code) walk an object's MRO and peek in the class' ``__dict__`` to look up attributes. A way to affect this lookup is using a method on the meta class for the type, that by default looks up the name in the class ``__dict__``. In Python code -------------- A meta type can define a method ``__locallookup__`` that is called during attribute resolution by both ``super.__getattribute__`` and ``object.__getattribute``:: class MetaType (type): def __locallookup__(cls, name): try: return cls.__dict__[name] except KeyError: raise AttributeError(name) from None The example method above is pseudo code for the implementation of this method on `type`_ (the actual implementation is in C, and doesn't suffer from the recursion problem in this example). The ``__locallookup__`` method has as its arguments a class and the name of the attribute that is looked up. It should return the value of the attribute without invoking descriptors, or raise `AttributeError`_ when the name cannot be found. In C code --------- A new slot ``tp_locallookup`` is added to the ``PyTypeObject`` struct, this slot corresponds to the ``__locallookup__`` method on `type`_. The slot has the following prototype:: PyObject* (*locallookupfunc)(PyTypeObject* cls, PyObject* name); This method should lookup *name* in the namespace of *cls*, without looking at superclasses, and should not invoke descriptors. The method returns ``NULL`` without setting an exception when the *name* cannot be found, and returns a new reference otherwise (not a borrowed reference). Usage of this hook ------------------ The new method will be defined for `type`_, and will peek in the class dictionary:: static PyObject* type_locallookup(PyTypeObject* cls, PyObject* name) { PyObject* res; if (!cls->tp_dict) { return NULL; } res = PyDict_GetItem(cls, name); Py_XINCREF(res); return res; } The new method will be used by both `PyObject_GenericGetAttr`_ and ``super.__getattribute__`` instead of peeking in a type's ``tp_dict``. Alternative proposals --------------------- ``__getattribute_super__`` .......................... An earlier version of this PEP used the following static method on classes:: def __getattribute_super__(cls, name, object, owner): pass This method performed name lookup as well as invoking descriptors and was necessarily limited to working only with ``super.__getattribute__``. Reuse ``tp_getattro`` ..................... It would be nice to avoid adding a new slot, thus keeping the API simpler and easier to understand. A comment on `Issue 18181`_ asked about reusing the ``tp_getattro`` slot, that is super could call the ``tp_getattro`` slot of all methods along the MRO. That won't work because ``tp_getattro`` will look in the instance ``__dict__`` before it tries to resolve attributes using classes in the MRO. This would mean that using ``tp_getattro`` instead of peeking the class dictionaries changes the semantics of the `super class`_. Open Issues =========== * The names of the new slot and magic method are far from settled. * Should the python method raise an exception or return a magic value (such as the `NotImplemented`_ return value used by comparison operators). The latter could be slightly faster because it doesn't have to overhead of setting up exception state, but makes it impossible to use that value as an attribute on a class. * The proposed change to `PyObject_GenericGetAttr`_ will probably cause problems with the attribute lookup cache (MCACHE): 1. That code stores borrowed references, which won't work when the hook is present. That is mostly fixable, but at the cost of possibly keeping garbage alive. 2. Caching isn't an option when a hook might execute arbitrary code (and there hence is no reason to assume that the hooks return value won't change later on). The only workaround I could find for this is to make the hook a fallback (that is, more like ``__getattr__`` than ``__getattribute__``). References ========== * `Issue 18181`_ contains a prototype implementation Copyright ========= This document has been placed in the public domain. .. _`Issue 18181`: http://bugs.python.org/issue18181 .. _`super class`: http://docs.python.org/3/library/functions.html#super .. _`NotImplemented`: http://docs.python.org/3/library/constants.html#NotImplemented .. _`PyObject_GenericGetAttr`: http://docs.python.org/3/c-api/object.html#PyObject_GenericGetAttr .. _`type`: http://docs.python.org/3/library/functions.html#type .. _`AttributeError`: http://docs.python.org/3/library/exceptions.html#AttributeError From guido at python.org Mon Jul 15 18:49:04 2013 From: guido at python.org (Guido van Rossum) Date: Mon, 15 Jul 2013 09:49:04 -0700 Subject: [Python-Dev] PEP 447: hooking into super() attribute resolution In-Reply-To: <2055CD40-2081-45DC-8B9A-889085FC7CEE@mac.com> References: <2055CD40-2081-45DC-8B9A-889085FC7CEE@mac.com> Message-ID: On Mon, Jul 15, 2013 at 8:12 AM, Ronald Oussoren wrote: > I've posted a new update of my proposal to add a way to override the attribute resolution proces by super(). I've rewritten the PEP and implementation based on feedback by Steve Dower. > > In the current edition of the proposal the hook is a method on the type, defined in a metaclass for types. A simple demo: > > class meta (type): > def __locallookup__(cls, name): > return type.__locallookup__(cls, name.upper()) > > class S (metaclass=meta): > def m(self): > return 42 > > def M(self): > return "fortytwo" > > class SS (S): > def m(self): > return 21 > > def M(self): > return "twentyone" > > o = SS() > print("self", o.m()) > print("super", super(SS, o).m()) > > The last line currently prints "super 42" and would print "super fortytwo" with my proposal. I would suggest adding that motivation to the PEP, since you clearly thought it was necessary to get people to consider your proposal. :-) > A major open issue: the __locallookup__ method could also be used for normal attribute resolution, but that probably causes issues with attribute caching (see the PEP for details). I haven't worked out yet if it is worthwhile to tweak the proposal to fix the caching issues (even though the proposal currently says that PyObject_GenericGetAttr will use the new method). Fixing the caching issue definitly would help in my primary use case by reducing code duplication, but might end up making the API unnecessarily complex. Hm. It looks like the functionality you actually want to hook into is in _PyType_Lookup(). I think that in this case the PEP's acceptance will be highly correlated with your ability to produce an actual patch that (a) implements exactly the functionality you want (never mind whether it matches the exact API currently proposed), and (b) doesn't slow down classes that don't provide this hook. Other than that, I think that it's a laudable attempt at generalizing, and I hope you solve the implementation conundrum. -- --Guido van Rossum (python.org/~guido) From ronaldoussoren at mac.com Mon Jul 15 18:56:56 2013 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Mon, 15 Jul 2013 18:56:56 +0200 Subject: [Python-Dev] PEP 447: hooking into super() attribute resolution In-Reply-To: References: <2055CD40-2081-45DC-8B9A-889085FC7CEE@mac.com> Message-ID: <23580FF6-CE74-4461-85BE-4333A894B552@mac.com> On 15 Jul, 2013, at 18:49, Guido van Rossum wrote: > > >> A major open issue: the __locallookup__ method could also be used for normal attribute resolution, but that probably causes issues with attribute caching (see the PEP for details). I haven't worked out yet if it is worthwhile to tweak the proposal to fix the caching issues (even though the proposal currently says that PyObject_GenericGetAttr will use the new method). Fixing the caching issue definitly would help in my primary use case by reducing code duplication, but might end up making the API unnecessarily complex. > > Hm. It looks like the functionality you actually want to hook into is > in _PyType_Lookup(). That's right. I didn't want to get too technical, but forgot to consider who are reading this :-) > > I think that in this case the PEP's acceptance will be highly > correlated with your ability to produce an actual patch that (a) > implements exactly the functionality you want (never mind whether it > matches the exact API currently proposed), and (b) doesn't slow down > classes that don't provide this hook. I'd only reall need the functional change to super(), but I am working on a patch that also changes _PyType_Lookup. I think I can avoid slowdown by making the tp_locallookup optional and only "punishing" those classes that use the new slot. A minor complication is that I'll have to change the interface of _PyType_Lookup, it currently returns a borrowed reference and will return a new reference. That's just careful bookkeeping though. > > Other than that, I think that it's a laudable attempt at generalizing, > and I hope you solve the implementation conundrum. I was pleasantly surprised in how the changed API was cleaner and applicable to _PyType_Lookup as well. I guess that means I'm on the right path. Ronald From zachary.ware at gmail.com Mon Jul 15 18:43:46 2013 From: zachary.ware at gmail.com (Zachary Ware) Date: Mon, 15 Jul 2013 11:43:46 -0500 Subject: [Python-Dev] [Python-checkins] cpython: Issue #18393: Remove use of deprecated API on OSX In-Reply-To: <3bv9J80Cjxz7LjQ@mail.python.org> References: <3bv9J80Cjxz7LjQ@mail.python.org> Message-ID: On Mon, Jul 15, 2013 at 11:32 AM, ronald.oussoren wrote: > http://hg.python.org/cpython/rev/ccbaf6762b54 > changeset: 84634:ccbaf6762b54 > user: Ronald Oussoren > date: Mon Jul 15 18:32:09 2013 +0200 > summary: > Issue #18393: Remove use of deprecated API on OSX > > The "Gestalt" function on OSX is deprecated (starting with OSX 10.8), > remove its usage from the stdlib. The patch removes a number of private I believe this means that Lib/test/leakers/test_gestalt.py can be removed as well. > functions and a private module, but does not change the public API. > > The removed code was effectively dead, the platform module has used > other code to fetch the OSX version for years and could only use > on the Gestalt-based code as a fallback. That fallback can only trigger > on broken OSX installs (that is, someone has removed parts of the system > install) > > files: > Lib/platform.py | 64 +------------------------- > Misc/NEWS | 4 + > Modules/_gestalt.c | 84 ---------------------------------- > setup.py | 4 - > 4 files changed, 5 insertions(+), 151 deletions(-) > > > > > -- > Repository URL: http://hg.python.org/cpython > -- Zach From guido at python.org Mon Jul 15 19:14:10 2013 From: guido at python.org (Guido van Rossum) Date: Mon, 15 Jul 2013 10:14:10 -0700 Subject: [Python-Dev] PEP 447: hooking into super() attribute resolution In-Reply-To: <23580FF6-CE74-4461-85BE-4333A894B552@mac.com> References: <2055CD40-2081-45DC-8B9A-889085FC7CEE@mac.com> <23580FF6-CE74-4461-85BE-4333A894B552@mac.com> Message-ID: On Mon, Jul 15, 2013 at 9:56 AM, Ronald Oussoren wrote: > > On 15 Jul, 2013, at 18:49, Guido van Rossum wrote: >> >> >>> A major open issue: the __locallookup__ method could also be used for normal attribute resolution, but that probably causes issues with attribute caching (see the PEP for details). I haven't worked out yet if it is worthwhile to tweak the proposal to fix the caching issues (even though the proposal currently says that PyObject_GenericGetAttr will use the new method). Fixing the caching issue definitly would help in my primary use case by reducing code duplication, but might end up making the API unnecessarily complex. >> >> Hm. It looks like the functionality you actually want to hook into is >> in _PyType_Lookup(). > > That's right. I didn't want to get too technical, but forgot to consider > who are reading this :-) Heh. :-) >> I think that in this case the PEP's acceptance will be highly >> correlated with your ability to produce an actual patch that (a) >> implements exactly the functionality you want (never mind whether it >> matches the exact API currently proposed), and (b) doesn't slow down >> classes that don't provide this hook. > > I'd only reall need the functional change to super(), but I am working on > a patch that also changes _PyType_Lookup. I think I can avoid slowdown > by making the tp_locallookup optional and only "punishing" those classes > that use the new slot. A minor complication is that I'll have to change > the interface of _PyType_Lookup, it currently returns a borrowed reference > and will return a new reference. That's just careful bookkeeping though. Hm. That sounds scary. I'd rename it, because this is actually an important (and scary) signature change, that type checkers otherwise won't see. Don't trust that you can find all call sites (there may be some in 3rd party code even though it starts with _). You don't want to suddenly start causing leaks all over the place. FWIW I agree that the change should affect _PyType_Lookup, because it would be very surprising if super() used a different override than regular attribute lookup. >> Other than that, I think that it's a laudable attempt at generalizing, >> and I hope you solve the implementation conundrum. > > I was pleasantly surprised in how the changed API was cleaner and applicable > to _PyType_Lookup as well. I guess that means I'm on the right path. There's a Zen of Python line about that. :-) -- --Guido van Rossum (python.org/~guido) From barry at python.org Mon Jul 15 21:51:31 2013 From: barry at python.org (Barry Warsaw) Date: Mon, 15 Jul 2013 15:51:31 -0400 Subject: [Python-Dev] Tweaking PEP 8 guidelines for use of leading underscores In-Reply-To: References: Message-ID: <20130715155131.43c7a519@anarchist> On Jul 14, 2013, at 06:11 PM, Nick Coghlan wrote: >Private interfaces PEP 8 does say: _single_leading_underscore: weak "internal use" indicator. E.g. from M import * does not import objects whose name starts with an underscore. I'm in favor of making this a stronger recommendation, but I have a small semantic quibble. Instead of "private interface" it should be "non-public interface". The two aren't quite the same thing; for example, often single-leading underscores are used for methods that subclasses are supposed to override, e.g. akin to "protected" in C++. Besides, Python doesn't really have any strong notion of privateness, so saying "non-public" means "just because you can, doesn't mean you should". -Barry From tjreedy at udel.edu Mon Jul 15 21:54:59 2013 From: tjreedy at udel.edu (Terry Reedy) Date: Mon, 15 Jul 2013 15:54:59 -0400 Subject: [Python-Dev] Refactoring test.support into a subpackage In-Reply-To: References: Message-ID: On 7/15/2013 6:22 AM, Nick Coghlan wrote: > At the PyCon AU sprints, some of the sprinters worked on a plan Chris > Jerdonek and I came up with months ago to convert test.support from a > module into a subpackage. > > This plan arose from some nasty test suite hacks in the lead up to the > release of Python 3.3, where some of the pkgutil tests ended up in > test.test_runpy so they could use the package creation infrastructure > I have in there without needing to extract a helper module with a > release imminent. > > The scope of the plan covers a few distinct steps: > > 1. Move Lib/test/support.py to Lib/test/support/__init__.py > (http://bugs.python.org/issue15494) > 2. Add a new (documented) test.support.pkg_helper submodule with the > package creation support code (http://bugs.python.org/issue15403 and > http://bugs.python.org/issue15376) > 3. Move the pkgutil tests to test.test_pkgutil where they belong > (http://bugs.python.org/issue15358) > 4. Move the existing test.script_helper and test.bytecode_helper > support modules into the test.support subpackage and document them (no > issue for that as yet) > > These changes would be made on both 3.3 and default, to avoid future > merge conflicts (note that Indra also posted an updated patch for one > pending test suite update to cope with the relocation: > http://bugs.python.org/issue15415) > > My main motivation for doing this is to give a bit more visibility to > the helpers that already exist, as well as making them easier to find Bundling together all the support code I should know about sounds good to me. I only became aware of script_helper, recently, when I saw it imported in a test_xxx file. I still have no idea what it does. > when we want to look into the details of what a test is doing. At the > moment, they tend to get lost in the sea of actual test files in the > main test directory. -- Terry Jan Reedy From solipsis at pitrou.net Mon Jul 15 21:56:45 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 15 Jul 2013 21:56:45 +0200 Subject: [Python-Dev] Tweaking PEP 8 guidelines for use of leading underscores References: <20130715155131.43c7a519@anarchist> Message-ID: <20130715215645.421c2eed@fsol> On Mon, 15 Jul 2013 15:51:31 -0400 Barry Warsaw wrote: > On Jul 14, 2013, at 06:11 PM, Nick Coghlan wrote: > > >Private interfaces > > PEP 8 does say: > > _single_leading_underscore: weak "internal use" indicator. E.g. from M > import * does not import objects whose name starts with an underscore. > > I'm in favor of making this a stronger recommendation, but I have a small > semantic quibble. Instead of "private interface" it should be "non-public > interface". The two aren't quite the same thing; for example, often > single-leading underscores are used for methods that subclasses are supposed > to override, e.g. akin to "protected" in C++. C++ doesn't have a monopoly over the definition of "private". > Besides, Python doesn't really have any strong notion of privateness, so > saying "non-public" means "just because you can, doesn't mean you should". If it doesn't have any notion of privateness, then it can't have a notion of "publicness" either. If you really want another word (I am personally fine with "private"), "internal" it should be. Regards Antoine. From barry at python.org Mon Jul 15 22:10:45 2013 From: barry at python.org (Barry Warsaw) Date: Mon, 15 Jul 2013 16:10:45 -0400 Subject: [Python-Dev] Tweaking PEP 8 guidelines for use of leading underscores In-Reply-To: <20130715215645.421c2eed@fsol> References: <20130715155131.43c7a519@anarchist> <20130715215645.421c2eed@fsol> Message-ID: <20130715161045.7a0c6bcd@anarchist> On Jul 15, 2013, at 09:56 PM, Antoine Pitrou wrote: >If you really want another word (I am personally fine with "private"), >"internal" it should be. I would be fine with "internal", since that's how PEP 8 already classifies such names. :) -Barry From ncoghlan at gmail.com Tue Jul 16 00:28:36 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 16 Jul 2013 08:28:36 +1000 Subject: [Python-Dev] Tweaking PEP 8 guidelines for use of leading underscores In-Reply-To: <20130715161045.7a0c6bcd@anarchist> References: <20130715155131.43c7a519@anarchist> <20130715215645.421c2eed@fsol> <20130715161045.7a0c6bcd@anarchist> Message-ID: On 16 Jul 2013 06:13, "Barry Warsaw" wrote: > > On Jul 15, 2013, at 09:56 PM, Antoine Pitrou wrote: > > >If you really want another word (I am personally fine with "private"), > >"internal" it should be. > > I would be fine with "internal", since that's how PEP 8 already classifies > such names. :) Yeah, internal is a better word for this. Cheers, Nick. > > -Barry > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From barry at python.org Tue Jul 16 00:40:39 2013 From: barry at python.org (Barry Warsaw) Date: Mon, 15 Jul 2013 18:40:39 -0400 Subject: [Python-Dev] Tweaking PEP 8 guidelines for use of leading underscores In-Reply-To: References: Message-ID: <20130715184039.60590e62@anarchist> Working from what I think is the latest version. In general, i'd rather be prescriptive of future conventions than descriptive of current conventions. It's okay to exempt existing code, and as a general rule we've never been fond of rewriting existing code to update it to new standards or APIs. We don't need to do so here either. On Jul 15, 2013, at 05:48 PM, Nick Coghlan wrote: >Private interfaces "Internal" or "Non-public" >Unless explicitly documented otherwise, a leading underscore on any >name indicates that it is an internal implementation detail and any >backwards compatibility guarantees do not apply. It is strongly >encouraged that private APIs (whether modules, classes, functions, >attributes or other names) be clearly marked in this way, as Python >users may rely on introspection to identify available functionality >and may be misled into believing an API without a leading underscore >is in fact a public API with the standard backwards compatibility >guarantees. How about: "All internal interfaces (modules, classes, functions, attributes or other names) should be prefixed with a single leading underscore. Such names are internal implementation details for which no backward compatibility guarantees are made, unless otherwise specified. Existing code and other narrowly accepted exceptions may override this recommendation, in which case the docstrings and/or documentation for such code must clearly and explicitly state the internal status of the APIs. Imported names should always be considered an implementation detail. Other modules must not rely on indirect access to such imported names unless they are an explicitly documented part of the containing module's API, such as ``os.path`` or a package's ``__init__`` module that exposes functionality from submodules. Public names exported by a module should be include in the module's ``__all__`` attribute." >While the explicit use of a leading underscore is the preferred solution, >the names of some private (or partially private) modules (such as ``test`` >and ``idlelib``) lack the leading underscore either for historical reasons >or because they expose a public command line interface through the >``-m`` switch. Such modules should include an explicit disclaimer in >their module docstring, indicating that they do not use the leading >underscore convention and noting where the definition of the public >API (if any) can be found (the public API definition may also be part >of the module docstring). > >As a general principle, the specific modules and external APIs imported by >a module are always considered an implementation detail. Other modules >should not rely on indirect access to such imported interfaces unless they >are an explicitly documented part of the containing module's API (such >as ``os.path`` or a package ``__init__`` module that exposes functionality >from submodules). Cheers, -Barry From guido at python.org Tue Jul 16 01:14:53 2013 From: guido at python.org (Guido van Rossum) Date: Mon, 15 Jul 2013 16:14:53 -0700 Subject: [Python-Dev] Misc re.match() complaint Message-ID: In a discussion about mypy I discovered that the Python 3 version of the re module's Match object behaves subtly different from the Python 2 version when the target string (i.e. the haystack, not the needle) is a buffer object. In Python 2, the type of the return value of group() is always either a Unicode string or an 8-bit string, and the type is determined by looking at the target string -- if the target is unicode, group() returns a unicode string, otherwise, group() returns an 8-bit string. In particular, if the target is a buffer object, group() returns an 8-bit string. I think this is the appropriate behavior: otherwise using regular expression matching to extract a small substring from a large target string would unnecessarily keep the large target string alive as long as the substring is alive. But in Python 3, the behavior of group() has changed so that its return type always matches that of the target string. I think this is bad -- apart from the lifetime concern, it means that if your target happens to be a bytearray, the return value isn't even hashable! Does anyone remember whether this was a conscious decision? Is it too late to fix? -- --Guido van Rossum (python.org/~guido) From greg at krypto.org Tue Jul 16 01:30:17 2013 From: greg at krypto.org (Gregory P. Smith) Date: Mon, 15 Jul 2013 16:30:17 -0700 Subject: [Python-Dev] Misc re.match() complaint In-Reply-To: References: Message-ID: On Mon, Jul 15, 2013 at 4:14 PM, Guido van Rossum wrote: > In a discussion about mypy I discovered that the Python 3 version of > the re module's Match object behaves subtly different from the Python > 2 version when the target string (i.e. the haystack, not the needle) > is a buffer object. > > In Python 2, the type of the return value of group() is always either > a Unicode string or an 8-bit string, and the type is determined by > looking at the target string -- if the target is unicode, group() > returns a unicode string, otherwise, group() returns an 8-bit string. > In particular, if the target is a buffer object, group() returns an > 8-bit string. I think this is the appropriate behavior: otherwise > using regular expression matching to extract a small substring from a > large target string would unnecessarily keep the large target string > alive as long as the substring is alive. > > But in Python 3, the behavior of group() has changed so that its > return type always matches that of the target string. I think this is > bad -- apart from the lifetime concern, it means that if your target > happens to be a bytearray, the return value isn't even hashable! > > Does anyone remember whether this was a conscious decision? Is it too > late to fix? Hmm, that is not what I'd expect either. I would never expect it to return a bytearray; I'd normally assume that .group() returned a bytes object if the input was binary data and a str object if the input was unicode data (str) regardless of specific types containing the input target data. I'm going to hazard a guess that not much, if anything, would be depending on getting a bytearray out of that. Fix this in 3.4? 3.3 and earlier users are stuck with an extra bytes() call and data copy in these cases I guess. -gps -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Tue Jul 16 01:37:27 2013 From: guido at python.org (Guido van Rossum) Date: Mon, 15 Jul 2013 16:37:27 -0700 Subject: [Python-Dev] Misc re.match() complaint In-Reply-To: References: Message-ID: Ok, created http://bugs.python.org/issue18468. On Mon, Jul 15, 2013 at 4:30 PM, Gregory P. Smith wrote: > > On Mon, Jul 15, 2013 at 4:14 PM, Guido van Rossum wrote: >> >> In a discussion about mypy I discovered that the Python 3 version of >> the re module's Match object behaves subtly different from the Python >> 2 version when the target string (i.e. the haystack, not the needle) >> is a buffer object. >> >> In Python 2, the type of the return value of group() is always either >> a Unicode string or an 8-bit string, and the type is determined by >> looking at the target string -- if the target is unicode, group() >> returns a unicode string, otherwise, group() returns an 8-bit string. >> In particular, if the target is a buffer object, group() returns an >> 8-bit string. I think this is the appropriate behavior: otherwise >> using regular expression matching to extract a small substring from a >> large target string would unnecessarily keep the large target string >> alive as long as the substring is alive. >> >> But in Python 3, the behavior of group() has changed so that its >> return type always matches that of the target string. I think this is >> bad -- apart from the lifetime concern, it means that if your target >> happens to be a bytearray, the return value isn't even hashable! >> >> Does anyone remember whether this was a conscious decision? Is it too >> late to fix? > > > Hmm, that is not what I'd expect either. I would never expect it to return a > bytearray; I'd normally assume that .group() returned a bytes object if the > input was binary data and a str object if the input was unicode data (str) > regardless of specific types containing the input target data. > > I'm going to hazard a guess that not much, if anything, would be depending > on getting a bytearray out of that. Fix this in 3.4? 3.3 and earlier users > are stuck with an extra bytes() call and data copy in these cases I guess. > > -gps > -- --Guido van Rossum (python.org/~guido) From python at mrabarnett.plus.com Tue Jul 16 02:10:51 2013 From: python at mrabarnett.plus.com (MRAB) Date: Tue, 16 Jul 2013 01:10:51 +0100 Subject: [Python-Dev] Misc re.match() complaint In-Reply-To: References: Message-ID: <51E48F8B.1090905@mrabarnett.plus.com> On 16/07/2013 00:30, Gregory P. Smith wrote: > > On Mon, Jul 15, 2013 at 4:14 PM, Guido van Rossum > wrote: > > In a discussion about mypy I discovered that the Python 3 version of > the re module's Match object behaves subtly different from the Python > 2 version when the target string (i.e. the haystack, not the needle) > is a buffer object. > > In Python 2, the type of the return value of group() is always either > a Unicode string or an 8-bit string, and the type is determined by > looking at the target string -- if the target is unicode, group() > returns a unicode string, otherwise, group() returns an 8-bit string. > In particular, if the target is a buffer object, group() returns an > 8-bit string. I think this is the appropriate behavior: otherwise > using regular expression matching to extract a small substring from a > large target string would unnecessarily keep the large target string > alive as long as the substring is alive. > > But in Python 3, the behavior of group() has changed so that its > return type always matches that of the target string. I think this is > bad -- apart from the lifetime concern, it means that if your target > happens to be a bytearray, the return value isn't even hashable! > > Does anyone remember whether this was a conscious decision? Is it too > late to fix? > > > Hmm, that is not what I'd expect either. I would never expect it to > return a bytearray; I'd normally assume that .group() returned a bytes > object if the input was binary data and a str object if the input was > unicode data (str) regardless of specific types containing the input > target data. > > I'm going to hazard a guess that not much, if anything, would be > depending on getting a bytearray out of that. Fix this in 3.4? 3.3 and > earlier users are stuck with an extra bytes() call and data copy in > these cases I guess. > I'm not sure I understand the complaint. I get this for Python 2.7: Python 2.7.5 (default, May 15 2013, 22:43:36) [MSC v.1500 32 bit (Intel)] on win 32 Type "help", "copyright", "credits" or "license" for more information. >>> import array >>> import re >>> re.match(r"a", array.array("b", "a")).group() array('b', [97]) It's the same even in Python 2.4. From ncoghlan at gmail.com Tue Jul 16 02:10:40 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 16 Jul 2013 10:10:40 +1000 Subject: [Python-Dev] Misc re.match() complaint In-Reply-To: References: Message-ID: On 16 Jul 2013 09:17, "Guido van Rossum" wrote: > > Does anyone remember whether this was a conscious decision? I doubt it was a conscious decision - an unfortunate amount of the standard library's handling of the text model change falls into the category of "implementation accident" :( > Is it too > late to fix? Like Greg, I'm comfortable with the idea of calling "bug" on this one, fixing it in 3.4 and making a note in the "Porting to Python 3.4" section of the What's New guide. Cheers, Nick. > > -- > --Guido van Rossum (python.org/~guido) > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Tue Jul 16 02:25:11 2013 From: guido at python.org (Guido van Rossum) Date: Mon, 15 Jul 2013 17:25:11 -0700 Subject: [Python-Dev] Misc re.match() complaint In-Reply-To: <51E48F8B.1090905@mrabarnett.plus.com> References: <51E48F8B.1090905@mrabarnett.plus.com> Message-ID: On Mon, Jul 15, 2013 at 5:10 PM, MRAB wrote: > On 16/07/2013 00:30, Gregory P. Smith wrote: >> >> >> On Mon, Jul 15, 2013 at 4:14 PM, Guido van Rossum > > wrote: >> >> In a discussion about mypy I discovered that the Python 3 version of >> the re module's Match object behaves subtly different from the Python >> 2 version when the target string (i.e. the haystack, not the needle) >> is a buffer object. >> >> In Python 2, the type of the return value of group() is always either >> a Unicode string or an 8-bit string, and the type is determined by >> looking at the target string -- if the target is unicode, group() >> returns a unicode string, otherwise, group() returns an 8-bit string. >> In particular, if the target is a buffer object, group() returns an >> 8-bit string. I think this is the appropriate behavior: otherwise >> using regular expression matching to extract a small substring from a >> large target string would unnecessarily keep the large target string >> alive as long as the substring is alive. >> >> But in Python 3, the behavior of group() has changed so that its >> return type always matches that of the target string. I think this is >> bad -- apart from the lifetime concern, it means that if your target >> happens to be a bytearray, the return value isn't even hashable! >> >> Does anyone remember whether this was a conscious decision? Is it too >> late to fix? >> >> >> Hmm, that is not what I'd expect either. I would never expect it to >> return a bytearray; I'd normally assume that .group() returned a bytes >> object if the input was binary data and a str object if the input was >> unicode data (str) regardless of specific types containing the input >> target data. >> >> I'm going to hazard a guess that not much, if anything, would be >> depending on getting a bytearray out of that. Fix this in 3.4? 3.3 and >> earlier users are stuck with an extra bytes() call and data copy in >> these cases I guess. >> > I'm not sure I understand the complaint. > > I get this for Python 2.7: > > Python 2.7.5 (default, May 15 2013, 22:43:36) [MSC v.1500 32 bit (Intel)] on > win > 32 > Type "help", "copyright", "credits" or "license" for more information. >>>> import array >>>> import re >>>> re.match(r"a", array.array("b", "a")).group() > array('b', [97]) > > It's the same even in Python 2.4. Ah, but now try it with buffer(): >> re.search('yz+', buffer('xyzzy')).group() 'yzz' >>> The equivalent in Python 3 (using memoryview) returns a memoryview: >>> re.search(b'yz+', memoryview(b'xyzzy')).group() >>> And I still think that any return type for group() except bytes or str is wrong. (Except possibly a subclass of these.) -- --Guido van Rossum (python.org/~guido) From victor.stinner at gmail.com Tue Jul 16 02:34:49 2013 From: victor.stinner at gmail.com (Victor Stinner) Date: Tue, 16 Jul 2013 02:34:49 +0200 Subject: [Python-Dev] Is it safe to call PyEval_EvalFrameEx() with an exception set? Message-ID: Hi, I'm working on the issue #18408 (fix issues found by my pyfailmalloc tool). To analyze some bugs, I have to add debug traces in various functions to find which function returned NULL without setting an error, or the opposite: returned a valid object, but with an exception set. I would like to add assertions in Python/ceval.c to detect such bugs earlier. The problem is that some functions rely on the ability to call PyEval_EvalFrameEx() with an exception set. Is it expected? Should it be avoided? The current exception can be replaced with a new exception. Example 1: module = PyImport_Import(module_name); if (module == NULL) { PyErr_Format(PicklingError, "Can't pickle %R: import of module %R failed", obj, module_name); goto error; } Formatting the obj argument calls indirectly PyEval_EvalFrameEx(), whereas an ImportError was raised. The original ImportError is replaced with a new PickleError exception. To not call PyEval_EvalFrameEx() with an exception set, PyErr_Format() can call PyErr_Clear() before calling PyUnicode_FromFormatV(), maybe only in debug mode. Example 2: test_sqlite uses sqlite.connection.create_aggregate() to create a Python callback called by the C sqlite3 library. The "callback" is a class with 3 methods: constructor, step() and finalize(). The test AggregateTests.CheckAggrNoStep() uses a class with no step() method: it ensures that an AttributeError is raised. The problem is that the finalize() method is called even if the class has no step method, and so there is a current AttributeError exception. I don't know if finalize() should be called in this case. It may be a bug in the sqlite module. More generally, Victor From chrism at plope.com Tue Jul 16 02:23:13 2013 From: chrism at plope.com (Chris McDonough) Date: Mon, 15 Jul 2013 20:23:13 -0400 Subject: [Python-Dev] Tweaking PEP 8 guidelines for use of leading underscores In-Reply-To: <20130715184039.60590e62@anarchist> References: <20130715184039.60590e62@anarchist> Message-ID: <1373934193.30382.59.camel@thinko> On Mon, 2013-07-15 at 18:40 -0400, Barry Warsaw wrote: > Working from what I think is the latest version. > > In general, i'd rather be prescriptive of future conventions than descriptive > of current conventions. It's okay to exempt existing code, and as a general > rule we've never been fond of rewriting existing code to update it to new > standards or APIs. We don't need to do so here either. FWIW, I'm very skeptical of a PEP 8 guideline that would try to proscribe that the "non-internal" API of a module or class would be defined solely by a naming convention. If what's being described here does become a rule, there is reason to believe that future users who treat this PEP as the word-of-god (and there are a *lot* of them; I hear from people literally every week who want to "PEP8-ify" my code in some limited-value-added way) will be harmed. They'll be living in a fantasy world where every non-underscore-prefixed thing is now a defacto API. But I have lived in a world where that has not been the case since 1998, and the chance that I'll go back and change all my public code to satisfy a questionable introspection convention is pretty slim. - C > > On Jul 15, 2013, at 05:48 PM, Nick Coghlan wrote: > > >Private interfaces > > "Internal" or "Non-public" > > >Unless explicitly documented otherwise, a leading underscore on any > >name indicates that it is an internal implementation detail and any > >backwards compatibility guarantees do not apply. It is strongly > >encouraged that private APIs (whether modules, classes, functions, > >attributes or other names) be clearly marked in this way, as Python > >users may rely on introspection to identify available functionality > >and may be misled into believing an API without a leading underscore > >is in fact a public API with the standard backwards compatibility > >guarantees. > > How about: > > "All internal interfaces (modules, classes, functions, attributes or other > names) should be prefixed with a single leading underscore. Such names are > internal implementation details for which no backward compatibility guarantees > are made, unless otherwise specified. > > Existing code and other narrowly accepted exceptions may override this > recommendation, in which case the docstrings and/or documentation for such > code must clearly and explicitly state the internal status of the APIs. > > Imported names should always be considered an implementation detail. Other > modules must not rely on indirect access to such imported names unless they > are an explicitly documented part of the containing module's API, such as > ``os.path`` or a package's ``__init__`` module that exposes functionality from > submodules. Public names exported by a module should be include in the > module's ``__all__`` attribute." > > >While the explicit use of a leading underscore is the preferred solution, > >the names of some private (or partially private) modules (such as ``test`` > >and ``idlelib``) lack the leading underscore either for historical reasons > >or because they expose a public command line interface through the > >``-m`` switch. Such modules should include an explicit disclaimer in > >their module docstring, indicating that they do not use the leading > >underscore convention and noting where the definition of the public > >API (if any) can be found (the public API definition may also be part > >of the module docstring). > > > >As a general principle, the specific modules and external APIs imported by > >a module are always considered an implementation detail. Other modules > >should not rely on indirect access to such imported interfaces unless they > >are an explicitly documented part of the containing module's API (such > >as ``os.path`` or a package ``__init__`` module that exposes functionality > >from submodules). > > Cheers, > -Barry > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/lists%40plope.com From steve at pearwood.info Tue Jul 16 03:25:03 2013 From: steve at pearwood.info (Steven D'Aprano) Date: Tue, 16 Jul 2013 11:25:03 +1000 Subject: [Python-Dev] Tweaking PEP 8 guidelines for use of leading underscores In-Reply-To: <1373934193.30382.59.camel@thinko> References: <20130715184039.60590e62@anarchist> <1373934193.30382.59.camel@thinko> Message-ID: <51E4A0EF.8070205@pearwood.info> On 16/07/13 10:23, Chris McDonough wrote: > If what's being described here does become a rule, there is reason to > believe that future users who treat this PEP as the word-of-god (and > there are a *lot* of them; I hear from people literally every week who > want to "PEP8-ify" my code in some limited-value-added way) will be > harmed. I sympathise with your pain, but is that not true of every PEP 8 naming convention? > They'll be living in a fantasy world where every > non-underscore-prefixed thing is now a defacto API. If your code has no obvious, documented convention at all for what's internal and what is not, they are no worse off. If you do have a documented convention for internal implementation details, then you are no worse off. "I have better things to do than PEP8-ify old, working, stable code" is a perfectly acceptable answer. "I have better things to do than PEP9-ify old, working, stable code, but if you want to provide regression tests and a working patch, I'll let you do so" might be an even better one :-) -- Steven From ncoghlan at gmail.com Tue Jul 16 03:30:40 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 16 Jul 2013 11:30:40 +1000 Subject: [Python-Dev] Tweaking PEP 8 guidelines for use of leading underscores In-Reply-To: <1373934193.30382.59.camel@thinko> References: <20130715184039.60590e62@anarchist> <1373934193.30382.59.camel@thinko> Message-ID: On 16 July 2013 10:23, Chris McDonough wrote: > On Mon, 2013-07-15 at 18:40 -0400, Barry Warsaw wrote: >> Working from what I think is the latest version. >> >> In general, i'd rather be prescriptive of future conventions than descriptive >> of current conventions. It's okay to exempt existing code, and as a general >> rule we've never been fond of rewriting existing code to update it to new >> standards or APIs. We don't need to do so here either. > > FWIW, I'm very skeptical of a PEP 8 guideline that would try to > proscribe that the "non-internal" API of a module or class would be > defined solely by a naming convention. > > If what's being described here does become a rule, there is reason to > believe that future users who treat this PEP as the word-of-god (and > there are a *lot* of them; I hear from people literally every week who > want to "PEP8-ify" my code in some limited-value-added way) will be > harmed. They'll be living in a fantasy world where every > non-underscore-prefixed thing is now a defacto API. But I have lived in > a world where that has not been the case since 1998, and the chance that > I'll go back and change all my public code to satisfy a questionable > introspection convention is pretty slim. Hence the "just say it's an internal API in the docstring" escape clause. That's a pretty low bar to meet for PEP 8 compliance, and one we need, too (for things like test, idlelib and a bundled pip). I do need to figure out how to tweak the wording to make it clear that internal nature of an API applies to all contained APIs as well. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From stephen at xemacs.org Tue Jul 16 04:03:17 2013 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Tue, 16 Jul 2013 11:03:17 +0900 Subject: [Python-Dev] Misc re.match() complaint In-Reply-To: References: <51E48F8B.1090905@mrabarnett.plus.com> Message-ID: <87vc4bs1lm.fsf@uwakimon.sk.tsukuba.ac.jp> Guido van Rossum writes: > And I still think that any return type for group() except bytes or str > is wrong. (Except possibly a subclass of these.) I'm not sure I understand. Do you mean in the context of the match object API, where constructing "(target, match.start(), match.end())" to get a group-like object that refers to the target rather than copying the text is simple? (Such objects are very useful in the restricted application of constructing a programmable text editor.) Or is this something deeper, that a group *is* a new object in principle? From python at mrabarnett.plus.com Tue Jul 16 04:18:34 2013 From: python at mrabarnett.plus.com (MRAB) Date: Tue, 16 Jul 2013 03:18:34 +0100 Subject: [Python-Dev] Misc re.match() complaint In-Reply-To: References: <51E48F8B.1090905@mrabarnett.plus.com> Message-ID: <51E4AD7A.2050505@mrabarnett.plus.com> On 16/07/2013 01:25, Guido van Rossum wrote: > On Mon, Jul 15, 2013 at 5:10 PM, MRAB wrote: >> On 16/07/2013 00:30, Gregory P. Smith wrote: >>> >>> >>> On Mon, Jul 15, 2013 at 4:14 PM, Guido van Rossum >> > wrote: >>> >>> In a discussion about mypy I discovered that the Python 3 version of >>> the re module's Match object behaves subtly different from the Python >>> 2 version when the target string (i.e. the haystack, not the needle) >>> is a buffer object. >>> >>> In Python 2, the type of the return value of group() is always either >>> a Unicode string or an 8-bit string, and the type is determined by >>> looking at the target string -- if the target is unicode, group() >>> returns a unicode string, otherwise, group() returns an 8-bit string. >>> In particular, if the target is a buffer object, group() returns an >>> 8-bit string. I think this is the appropriate behavior: otherwise >>> using regular expression matching to extract a small substring from a >>> large target string would unnecessarily keep the large target string >>> alive as long as the substring is alive. >>> >>> But in Python 3, the behavior of group() has changed so that its >>> return type always matches that of the target string. I think this is >>> bad -- apart from the lifetime concern, it means that if your target >>> happens to be a bytearray, the return value isn't even hashable! >>> >>> Does anyone remember whether this was a conscious decision? Is it too >>> late to fix? >>> >>> >>> Hmm, that is not what I'd expect either. I would never expect it to >>> return a bytearray; I'd normally assume that .group() returned a bytes >>> object if the input was binary data and a str object if the input was >>> unicode data (str) regardless of specific types containing the input >>> target data. >>> >>> I'm going to hazard a guess that not much, if anything, would be >>> depending on getting a bytearray out of that. Fix this in 3.4? 3.3 and >>> earlier users are stuck with an extra bytes() call and data copy in >>> these cases I guess. >>> >> I'm not sure I understand the complaint. >> >> I get this for Python 2.7: >> >> Python 2.7.5 (default, May 15 2013, 22:43:36) [MSC v.1500 32 bit (Intel)] on >> win >> 32 >> Type "help", "copyright", "credits" or "license" for more information. >>>>> import array >>>>> import re >>>>> re.match(r"a", array.array("b", "a")).group() >> array('b', [97]) >> >> It's the same even in Python 2.4. > > Ah, but now try it with buffer(): > >>> re.search('yz+', buffer('xyzzy')).group() > 'yzz' >>>> > > The equivalent in Python 3 (using memoryview) returns a memoryview: > >>>> re.search(b'yz+', memoryview(b'xyzzy')).group() > >>>> > > And I still think that any return type for group() except bytes or str > is wrong. (Except possibly a subclass of these.) > On the other hand, I think that it's not unreasonable that the output is the same type as the input. You could reason that what it's doing is returning a slice of the input, and that slice should be the same type as its source. Incidentally, the regex module does what Python 3's re module currently does, even in Python 2. Nobody's complained! From chrism at plope.com Tue Jul 16 04:20:35 2013 From: chrism at plope.com (Chris McDonough) Date: Mon, 15 Jul 2013 22:20:35 -0400 Subject: [Python-Dev] Tweaking PEP 8 guidelines for use of leading underscores In-Reply-To: <51E4A0EF.8070205@pearwood.info> References: <20130715184039.60590e62@anarchist> <1373934193.30382.59.camel@thinko> <51E4A0EF.8070205@pearwood.info> Message-ID: <1373941235.30382.62.camel@thinko> On Tue, 2013-07-16 at 11:25 +1000, Steven D'Aprano wrote: > > If your code has no obvious, documented convention at all for what's internal and what is not, they are no worse off. > > If you do have a documented convention for internal implementation details, then you are no worse off. "I have better things to do than PEP8-ify old, working, stable code" is a perfectly acceptable answer. "I have better things to do than PEP9-ify old, working, stable code, but if you want to provide regression tests and a working patch, I'll let you do so" might be an even better one :-) Welp, I guess I'm logically boxed in then. Thanks for showing me the errors in my thinking. Should be no problem to manage the updating of that 500K lines of public code. /scarcasm - C From guido at python.org Tue Jul 16 04:20:29 2013 From: guido at python.org (Guido van Rossum) Date: Mon, 15 Jul 2013 19:20:29 -0700 Subject: [Python-Dev] Misc re.match() complaint In-Reply-To: <87vc4bs1lm.fsf@uwakimon.sk.tsukuba.ac.jp> References: <51E48F8B.1090905@mrabarnett.plus.com> <87vc4bs1lm.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: On Mon, Jul 15, 2013 at 7:03 PM, Stephen J. Turnbull wrote: > Guido van Rossum writes: > > > And I still think that any return type for group() except bytes or str > > is wrong. (Except possibly a subclass of these.) > > I'm not sure I understand. Do you mean in the context of the match > object API, where constructing "(target, match.start(), match.end())" > to get a group-like object that refers to the target rather than > copying the text is simple? (Such objects are very useful in the > restricted application of constructing a programmable text editor.) I'm not sure I understand you. :-( The group() method on the match object returned by re.match() and re.search() returns a string-ish object representing the matched substring. (I'm using "string-ish" to allow for both unicode and bytes, which are exactly the two matching modes supported be the re module.) In most contexts (text editors excluded) the program will use this string just as it would use any other string, perhaps using it to open a file, perhaps as a key into some cache, and so on. I can clearly see the reasons why you want the target string to allow other types besides str and bytes, in particular other things that are known to represent sequences of bytes, such as bytearray and memoryview. These reasons primarily have to do with optimizing the representation of the target string in case it takes up a large amount of memory, or other situations where we'd like to reduce the number of times each byte is copied before we see it. But I don't see as much of a use case for group() returning an object of the same type as the target string. In particular in the case of a target string that is a bytearray, group() has to copy the bytes regardless of whether it creates a bytes or a bytearray instance. And I do see a use case for group() returning an immutable object. > Or is this something deeper, that a group *is* a new object in > principle? No, I just think of it as returning "a string" and I think it's most useful if that is always an immutable object, even if the target string is some other bytes buffer. FWIW, it feels as if the change in behavior is probably just due to how slices work. -- --Guido van Rossum (python.org/~guido) From guido at python.org Tue Jul 16 04:23:55 2013 From: guido at python.org (Guido van Rossum) Date: Mon, 15 Jul 2013 19:23:55 -0700 Subject: [Python-Dev] Misc re.match() complaint In-Reply-To: <51E4AD7A.2050505@mrabarnett.plus.com> References: <51E48F8B.1090905@mrabarnett.plus.com> <51E4AD7A.2050505@mrabarnett.plus.com> Message-ID: On Mon, Jul 15, 2013 at 7:18 PM, MRAB wrote: > On the other hand, I think that it's not unreasonable that the output > is the same type as the input. You could reason that what it's doing is > returning a slice of the input, and that slice should be the same type > as its source. By now I'm pretty sure that is why it changed. But I am challenging how useful that is, compared to always returning something immutable. > Incidentally, the regex module does what Python 3's re module currently > does, even in Python 2. Nobody's complained! Well, you'd only see complaints from folks who (a) use the regex module, (b) use it with a buffer object as the target string, and (c) try to use the group() return value as a dict key. Each of these is probably a small majority of all users. -- --Guido van Rossum (python.org/~guido) From ncoghlan at gmail.com Tue Jul 16 04:34:00 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 16 Jul 2013 12:34:00 +1000 Subject: [Python-Dev] Tweaking PEP 8 guidelines for use of leading underscores In-Reply-To: <1373941235.30382.62.camel@thinko> References: <20130715184039.60590e62@anarchist> <1373934193.30382.59.camel@thinko> <51E4A0EF.8070205@pearwood.info> <1373941235.30382.62.camel@thinko> Message-ID: On 16 July 2013 12:20, Chris McDonough wrote: > On Tue, 2013-07-16 at 11:25 +1000, Steven D'Aprano wrote: > >> >> If your code has no obvious, documented convention at all for what's internal and what is not, they are no worse off. >> >> If you do have a documented convention for internal implementation details, then you are no worse off. "I have better things to do than PEP8-ify old, working, stable code" is a perfectly acceptable answer. "I have better things to do than PEP9-ify old, working, stable code, but if you want to provide regression tests and a working patch, I'll let you do so" might be an even better one :-) > > Welp, I guess I'm logically boxed in then. Thanks for showing me the > errors in my thinking. Should be no problem to manage the updating of > that 500K lines of public code. > > /scarcasm How do get from "If this doesn't apply to a module, just add something like 'This is an internal API' or 'This module includes internal APIs, consult the documentation for the public API' to the module docstring" to "updating 500k lines of public code"? The version in Barry's email that you replied to has that escape clause in it, so the fact it was missing from my original text doesn't justify this reaction. Cheers, Nick. P.S. Note that, while I'm trying to account for it in this particular case, we're never going to let the fact that many people misuse PEP 8 by considering it as a holy standard that should be followed by all Python code everywhere stop us from including updates that are valid specifically for the standard library. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From victor.stinner at gmail.com Tue Jul 16 04:38:23 2013 From: victor.stinner at gmail.com (Victor Stinner) Date: Tue, 16 Jul 2013 04:38:23 +0200 Subject: [Python-Dev] [Python-checkins] cpython (merge default -> 3.3): merge 3.3 (#18470) In-Reply-To: <3bvQHF6dScz7Lp9@mail.python.org> References: <3bvQHF6dScz7Lp9@mail.python.org> Message-ID: 2013/7/16 benjamin.peterson : > http://hg.python.org/cpython/rev/8889c9b5dd3a > changeset: 84654:8889c9b5dd3a > branch: 3.3 > parent: 84653:c3a510b22218 > parent: 84652:8a078bf3cf14 > user: Benjamin Peterson > date: Mon Jul 15 19:15:49 2013 -0700 > summary: > merge 3.3 (#18470) This commit merges branches default and 3.3 into 3.3. Usually, we merge branches 3.3 and default into default. Because of this changeset, when I try to merge 3.3 into default, I get a long list of modifications... Victor From victor.stinner at gmail.com Tue Jul 16 04:40:49 2013 From: victor.stinner at gmail.com (Victor Stinner) Date: Tue, 16 Jul 2013 04:40:49 +0200 Subject: [Python-Dev] [Python-checkins] cpython (3.3): check the return value of new_string() (closes #18470) In-Reply-To: <3bvQHD4yjMz7LlT@mail.python.org> References: <3bvQHD4yjMz7LlT@mail.python.org> Message-ID: 2013/7/16 benjamin.peterson : > http://hg.python.org/cpython/rev/c3a510b22218 > changeset: 84653:c3a510b22218 > branch: 3.3 > parent: 84651:e22dd5fda5a8 > user: Benjamin Peterson > date: Mon Jul 15 19:15:34 2013 -0700 > summary: > check the return value of new_string() (closes #18470) > > ... > > > diff --git a/Parser/tokenizer.c b/Parser/tokenizer.c > --- a/Parser/tokenizer.c > +++ b/Parser/tokenizer.c > @@ -250,17 +253,21 @@ > t++; > > if (begin < t) { > - char* r = new_string(begin, t - begin); > + char* r = new_string(begin, t - begin, tok); > + if (!r) > + return 0; > char* q = get_normal_name(r); > ... Visual Studio does support instructions between declarations, and so this changeset broke Windows buildbots. Should we add the "-Werror=declaration-after-statement" compiler flag to the 3.3 branch? (in debug mode?) Victor From barry at python.org Tue Jul 16 04:48:15 2013 From: barry at python.org (Barry Warsaw) Date: Mon, 15 Jul 2013 22:48:15 -0400 Subject: [Python-Dev] Tweaking PEP 8 guidelines for use of leading underscores In-Reply-To: <1373934193.30382.59.camel@thinko> References: <20130715184039.60590e62@anarchist> <1373934193.30382.59.camel@thinko> Message-ID: <20130715224815.07b7a8f3@anarchist> On Jul 15, 2013, at 08:23 PM, Chris McDonough wrote: >On Mon, 2013-07-15 at 18:40 -0400, Barry Warsaw wrote: >> Working from what I think is the latest version. >> >> In general, i'd rather be prescriptive of future conventions than >> descriptive of current conventions. It's okay to exempt existing code, and >> as a general rule we've never been fond of rewriting existing code to >> update it to new standards or APIs. We don't need to do so here either. > >FWIW, I'm very skeptical of a PEP 8 guideline that would try to >proscribe that the "non-internal" API of a module or class would be >defined solely by a naming convention. > >If what's being described here does become a rule, there is reason to >believe that future users who treat this PEP as the word-of-god (and >there are a *lot* of them; I hear from people literally every week who >want to "PEP8-ify" my code in some limited-value-added way) will be >harmed. They'll be living in a fantasy world where every >non-underscore-prefixed thing is now a defacto API. But I have lived in >a world where that has not been the case since 1998, and the chance that >I'll go back and change all my public code to satisfy a questionable >introspection convention is pretty slim. I can sympathize with this. What I tried to get at, but probably didn't succeed, is to say "if it starts with a single underscore, treat it as internal" but not necessarily the converse, i.o.w. if it *doesn't* start with an underscore that does not imply that it's public. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From chrism at plope.com Tue Jul 16 05:02:42 2013 From: chrism at plope.com (Chris McDonough) Date: Mon, 15 Jul 2013 23:02:42 -0400 Subject: [Python-Dev] Tweaking PEP 8 guidelines for use of leading underscores In-Reply-To: References: <20130715184039.60590e62@anarchist> <1373934193.30382.59.camel@thinko> <51E4A0EF.8070205@pearwood.info> <1373941235.30382.62.camel@thinko> Message-ID: <1373943762.30382.88.camel@thinko> On Tue, 2013-07-16 at 12:34 +1000, Nick Coghlan wrote: > How do get from "If this doesn't apply to a module, just add something > like 'This is an internal API' or 'This module includes internal APIs, > consult the documentation for the public API' to the module docstring" > to "updating 500k lines of public code"? The version in Barry's email > that you replied to has that escape clause in it, so the fact it was > missing from my original text doesn't justify this reaction. I may be wrong, but as I see it this is a decision to help (although only conditionally!) a theoretical IDE user or an agressive user of dir() at the expense of effectively either 1) making existing maintainers revisit existing code, or 2 ) making maintainers continually explain to users that their codebase is not "PEP8 compliant". > P.S. Note that, while I'm trying to account for it in this particular > case, we're never going to let the fact that many people misuse PEP 8 > by considering it as a holy standard that should be followed by all > Python code everywhere stop us from including updates that are valid > specifically for the standard library. Fair enough, technically. But historical reality is that a lot of people just haven't read much past the title: "Style Guide for Python Code". Or at least they appear to have no concept that it's about code *in the stdlib only*. People take it way, way too seriously for non-stdlib code. But they do, and it takes time to manage that. Given that misunderstanding, is there a way to divorce stdlib-centric guidelines like the one being discussed now from "PEP8"-ness? I don't think any amount of marketing effort or reasoned explanation is going to separate "PEP8" from "correct code" in people's minds at this point. - C ps.. the real irritant behind my minirant is this: OSS developers have spent many months jumping through bw incompat hoops in Python over the last few years, and it has taken time away from doing things that provide value. The less I can do of that, the better, and Python gets more value too. That said, I realize that I'm in the minority because I happen to have a metric ton of public code out there. But it'd be nice if that was encouraged rather than effectively punished on the hunch that it might provide some benefit for a theoretical new user. From ncoghlan at gmail.com Tue Jul 16 05:03:29 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 16 Jul 2013 13:03:29 +1000 Subject: [Python-Dev] Misc re.match() complaint In-Reply-To: References: <51E48F8B.1090905@mrabarnett.plus.com> <87vc4bs1lm.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: On 16 July 2013 12:20, Guido van Rossum wrote: > On Mon, Jul 15, 2013 at 7:03 PM, Stephen J. Turnbull wrote: >> Or is this something deeper, that a group *is* a new object in >> principle? > > No, I just think of it as returning "a string" and I think it's most > useful if that is always an immutable object, even if the target > string is some other bytes buffer. > > FWIW, it feels as if the change in behavior is probably just due to > how slices work. I took a look at the way the 2.7 re code works, and the change does indeed appear to be due to the difference in the way slices work for buffer and memoryview objects: Slicing a buffer creates an 8-bit string: >>> buffer(b"abc")[0:1] 'a' Slicing a memoryview creates another memoryview: >>> memoryview(b"abc")[0:1] Unfortunately, memoryview doesn't currently allow subclasses, so it isn't easy to create a derivative that coerces to bytes on slicing :( Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Tue Jul 16 05:11:49 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 16 Jul 2013 13:11:49 +1000 Subject: [Python-Dev] Tweaking PEP 8 guidelines for use of leading underscores In-Reply-To: <1373943762.30382.88.camel@thinko> References: <20130715184039.60590e62@anarchist> <1373934193.30382.59.camel@thinko> <51E4A0EF.8070205@pearwood.info> <1373941235.30382.62.camel@thinko> <1373943762.30382.88.camel@thinko> Message-ID: On 16 July 2013 13:02, Chris McDonough wrote: > OSS developers have spent many months jumping through bw incompat hoops > in Python over the last few years, and it has taken time away from doing > things that provide value. The less I can do of that, the better, and > Python gets more value too. That said, I realize that I'm in the > minority because I happen to have a metric ton of public code out there. > But it'd be nice if that was encouraged rather than effectively punished > on the hunch that it might provide some benefit for a theoretical new > user. You, Armin and everyone else that works on the bytes/text boundary are indeed the hardest hit by the Python 3 transition, and I appreciate the hard work you have all done to help make that transition as successful as it has been so far. However, the fact that people abuse PEP 8 by treating it as "all Python code in the world should follow these rules" cannot, and will not, stop us from continuing to use it to set appropriate guidelines *for the standard library*. I'll look into adding some stronger wording at the top making it clear that while PEP 8 is a useful starting point and a good default if a project doesn't have a defined style guide of it's own, it is *not* the be-all-and-end-all for Python style guides. Treating it as such as an abuse of the PEP, pure and simple. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From chrism at plope.com Tue Jul 16 05:16:46 2013 From: chrism at plope.com (Chris McDonough) Date: Mon, 15 Jul 2013 23:16:46 -0400 Subject: [Python-Dev] Tweaking PEP 8 guidelines for use of leading underscores In-Reply-To: References: <20130715184039.60590e62@anarchist> <1373934193.30382.59.camel@thinko> <51E4A0EF.8070205@pearwood.info> <1373941235.30382.62.camel@thinko> <1373943762.30382.88.camel@thinko> Message-ID: <1373944606.30382.92.camel@thinko> On Tue, 2013-07-16 at 13:11 +1000, Nick Coghlan wrote: > On 16 July 2013 13:02, Chris McDonough wrote: > > OSS developers have spent many months jumping through bw incompat hoops > > in Python over the last few years, and it has taken time away from doing > > things that provide value. The less I can do of that, the better, and > > Python gets more value too. That said, I realize that I'm in the > > minority because I happen to have a metric ton of public code out there. > > But it'd be nice if that was encouraged rather than effectively punished > > on the hunch that it might provide some benefit for a theoretical new > > user. > > You, Armin and everyone else that works on the bytes/text boundary are > indeed the hardest hit by the Python 3 transition, and I appreciate > the hard work you have all done to help make that transition as > successful as it has been so far. > > However, the fact that people abuse PEP 8 by treating it as "all > Python code in the world should follow these rules" cannot, and will > not, stop us from continuing to use it to set appropriate guidelines > *for the standard library*. > > I'll look into adding some stronger wording at the top making it clear > that while PEP 8 is a useful starting point and a good default if a > project doesn't have a defined style guide of it's own, it is *not* > the be-all-and-end-all for Python style guides. Treating it as such as > an abuse of the PEP, pure and simple. I understand that. Unfortunately the remainder of the world does not. The same IDEs that would be helped via this proposed change have "PEP8 modes" turned on *by default*! http://blog.jetbrains.com/pycharm/2013/02/long-awaited-pep-8-checks-on-the-fly-improved-doctest-support-and-more-in-pycharm-2-7/ It seems like an unwise step to continue stuffing things into the PEP8 brand bag as a result if that stuff is only meant to apply to the stdlib. - C From benjamin at python.org Tue Jul 16 05:55:16 2013 From: benjamin at python.org (Benjamin Peterson) Date: Mon, 15 Jul 2013 20:55:16 -0700 Subject: [Python-Dev] [Python-checkins] cpython (3.3): check the return value of new_string() (closes #18470) In-Reply-To: References: <3bvQHD4yjMz7LlT@mail.python.org> Message-ID: 2013/7/15 Victor Stinner : > 2013/7/16 benjamin.peterson : >> http://hg.python.org/cpython/rev/c3a510b22218 >> changeset: 84653:c3a510b22218 >> branch: 3.3 >> parent: 84651:e22dd5fda5a8 >> user: Benjamin Peterson >> date: Mon Jul 15 19:15:34 2013 -0700 >> summary: >> check the return value of new_string() (closes #18470) >> >> ... >> >> >> diff --git a/Parser/tokenizer.c b/Parser/tokenizer.c >> --- a/Parser/tokenizer.c >> +++ b/Parser/tokenizer.c >> @@ -250,17 +253,21 @@ >> t++; >> >> if (begin < t) { >> - char* r = new_string(begin, t - begin); >> + char* r = new_string(begin, t - begin, tok); >> + if (!r) >> + return 0; >> char* q = get_normal_name(r); >> ... > > Visual Studio does support instructions between declarations, and so > this changeset broke Windows buildbots. Fixed. > > Should we add the "-Werror=declaration-after-statement" compiler flag > to the 3.3 branch? (in debug mode?) We've lived with this limitation for years. I'm sure we can wait until 3.4 becomes maintenance. :) -- Regards, Benjamin From guido at python.org Tue Jul 16 06:53:42 2013 From: guido at python.org (Guido van Rossum) Date: Mon, 15 Jul 2013 21:53:42 -0700 Subject: [Python-Dev] Misc re.match() complaint In-Reply-To: References: <51E48F8B.1090905@mrabarnett.plus.com> <87vc4bs1lm.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: Hm. I'd still like to change this, but I understand it's debatable... Is the group() method written in C or Python? If it's in C it should be simple enough to let it just do a little bit of pointer math and construct a bytes object from the given area of memory -- after all, it must have a pointer to that memory area in order to do the matching in the first place (although I realize the code may be separated by a gulf of abstraction :-). --Guido On Mon, Jul 15, 2013 at 8:03 PM, Nick Coghlan wrote: > On 16 July 2013 12:20, Guido van Rossum wrote: >> On Mon, Jul 15, 2013 at 7:03 PM, Stephen J. Turnbull wrote: >>> Or is this something deeper, that a group *is* a new object in >>> principle? >> >> No, I just think of it as returning "a string" and I think it's most >> useful if that is always an immutable object, even if the target >> string is some other bytes buffer. >> >> FWIW, it feels as if the change in behavior is probably just due to >> how slices work. > > I took a look at the way the 2.7 re code works, and the change does > indeed appear to be due to the difference in the way slices work for > buffer and memoryview objects: > > Slicing a buffer creates an 8-bit string: > >>>> buffer(b"abc")[0:1] > 'a' > > Slicing a memoryview creates another memoryview: > >>>> memoryview(b"abc")[0:1] > > > Unfortunately, memoryview doesn't currently allow subclasses, so it > isn't easy to create a derivative that coerces to bytes on slicing :( > > Cheers, > Nick. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia -- --Guido van Rossum (python.org/~guido) From ncoghlan at gmail.com Tue Jul 16 07:44:47 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 16 Jul 2013 15:44:47 +1000 Subject: [Python-Dev] Tweaking PEP 8 guidelines for use of leading underscores In-Reply-To: <1373944606.30382.92.camel@thinko> References: <20130715184039.60590e62@anarchist> <1373934193.30382.59.camel@thinko> <51E4A0EF.8070205@pearwood.info> <1373941235.30382.62.camel@thinko> <1373943762.30382.88.camel@thinko> <1373944606.30382.92.camel@thinko> Message-ID: On 16 July 2013 13:16, Chris McDonough wrote: > I understand that. Unfortunately the remainder of the world does not. > The same IDEs that would be helped via this proposed change have "PEP8 > modes" turned on *by default*! > http://blog.jetbrains.com/pycharm/2013/02/long-awaited-pep-8-checks-on-the-fly-improved-doctest-support-and-more-in-pycharm-2-7/ > > It seems like an unwise step to continue stuffing things into the PEP8 > brand bag as a result if that stuff is only meant to apply to the > stdlib. Clarifying what constitutes an internal interface in a way that doesn't require renaming anything is a necessary prerequisite for bundling or bootstrapping the pip CLI in Python 3.4 (as pip exposes its internal implemetnation API as "import pip" rather than "import _pip" and renaming it would lead to a lot of pointless code churn). Without that concern, the topic never would have come up. Yes, this new section may give the PEP 8 prescriptivists more ammunition to complain about existing projects with different coding standards not complying with the latest standard library guidelines. We're not unfamiliar with the complaint, given the number of standard library modules (some dating from the 20th century) that don't comply with the current incarnation of PEP 8 circa 2013. However, PEP 8 is already quite explicit about what we think of that kind of behaviour: http://www.python.org/dev/peps/pep-0008/#a-foolish-consistency-is-the-hobgoblin-of-little-minds PEP 8 is a tool. Like many of the other tools we provide as part of Python, it can be abused. However, the likelihood of such abuse is *not* an excuse to avoid using the tool appropriately, it's a reason to continue the thankless task of trying to educate people on what's appropriate and what isn't (and making wholesale changes to working code just to comply with a style guide generally isn't appropriate). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From stephen at xemacs.org Tue Jul 16 07:59:11 2013 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Tue, 16 Jul 2013 14:59:11 +0900 Subject: [Python-Dev] Misc re.match() complaint In-Reply-To: References: <51E48F8B.1090905@mrabarnett.plus.com> <87vc4bs1lm.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <87ppujrqog.fsf@uwakimon.sk.tsukuba.ac.jp> Guido van Rossum writes: > I'm not sure I understand you. :-( My apologies. This: > > Or is this something deeper, that a group *is* a new object in > > principle? > > No, I just think of it as returning "a string" and I think it's most > useful if that is always an immutable object, even if the target > string is some other bytes buffer. is exactly the kind of answer I was looking for. From chrism at plope.com Tue Jul 16 08:01:22 2013 From: chrism at plope.com (Chris McDonough) Date: Tue, 16 Jul 2013 02:01:22 -0400 Subject: [Python-Dev] Tweaking PEP 8 guidelines for use of leading underscores In-Reply-To: References: <20130715184039.60590e62@anarchist> <1373934193.30382.59.camel@thinko> <51E4A0EF.8070205@pearwood.info> <1373941235.30382.62.camel@thinko> <1373943762.30382.88.camel@thinko> <1373944606.30382.92.camel@thinko> Message-ID: <1373954482.30382.111.camel@thinko> On Tue, 2013-07-16 at 15:44 +1000, Nick Coghlan wrote: > On 16 July 2013 13:16, Chris McDonough wrote: > > I understand that. Unfortunately the remainder of the world does not. > > The same IDEs that would be helped via this proposed change have "PEP8 > > modes" turned on *by default*! > > http://blog.jetbrains.com/pycharm/2013/02/long-awaited-pep-8-checks-on-the-fly-improved-doctest-support-and-more-in-pycharm-2-7/ > > > > It seems like an unwise step to continue stuffing things into the PEP8 > > brand bag as a result if that stuff is only meant to apply to the > > stdlib. > > Clarifying what constitutes an internal interface in a way that > doesn't require renaming anything is a necessary prerequisite for > bundling or bootstrapping the pip CLI in Python 3.4 (as pip exposes > its internal implemetnation API as "import pip" rather than "import > _pip" and renaming it would lead to a lot of pointless code churn). > Without that concern, the topic never would have come up. > > Yes, this new section may give the PEP 8 prescriptivists more > ammunition to complain about existing projects with different coding > standards not complying with the latest standard library guidelines. > We're not unfamiliar with the complaint, given the number of standard > library modules (some dating from the 20th century) that don't comply > with the current incarnation of PEP 8 circa 2013. > > However, PEP 8 is already quite explicit about what we think of that > kind of behaviour: > http://www.python.org/dev/peps/pep-0008/#a-foolish-consistency-is-the-hobgoblin-of-little-minds > > PEP 8 is a tool. Like many of the other tools we provide as part of > Python, it can be abused. However, the likelihood of such abuse is > *not* an excuse to avoid using the tool appropriately, it's a reason > to continue the thankless task of trying to educate people on what's > appropriate and what isn't (and making wholesale changes to working > code just to comply with a style guide generally isn't appropriate). Well, I know it's arguing with the umpire at this point, but PEP8 prescriptionism (great word by the way!) is de rigeur these days. It's a python-dev perogative to ignore that, but it has consequences that reverberate further than this maillist. - C From ncoghlan at gmail.com Tue Jul 16 08:10:31 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 16 Jul 2013 16:10:31 +1000 Subject: [Python-Dev] Tweaking PEP 8 guidelines for use of leading underscores In-Reply-To: <1373954482.30382.111.camel@thinko> References: <20130715184039.60590e62@anarchist> <1373934193.30382.59.camel@thinko> <51E4A0EF.8070205@pearwood.info> <1373941235.30382.62.camel@thinko> <1373943762.30382.88.camel@thinko> <1373944606.30382.92.camel@thinko> <1373954482.30382.111.camel@thinko> Message-ID: On 16 July 2013 16:01, Chris McDonough wrote: > Well, I know it's arguing with the umpire at this point, but PEP8 > prescriptionism (great word by the way!) is de rigeur these days. It's > a python-dev perogative to ignore that, but it has consequences that > reverberate further than this maillist. Yeah, we should probably be a bit more vocal in pointing out that PEP 8 is something to be applied judiciously, *not* treated as "sacred writ, never to be questioned". It's very tempting to just sigh and ignore it, instead of explicitly pointing out the problems with taking it too literally for people's personal projects (or, worse, using it to abuse developers of projects that long predate the recommendations in more recent versions) :P Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Tue Jul 16 08:20:38 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 16 Jul 2013 16:20:38 +1000 Subject: [Python-Dev] Misc re.match() complaint In-Reply-To: References: <51E48F8B.1090905@mrabarnett.plus.com> <87vc4bs1lm.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: On 16 July 2013 14:53, Guido van Rossum wrote: > Hm. I'd still like to change this, but I understand it's debatable... > Is the group() method written in C or Python? If it's in C it should > be simple enough to let it just do a little bit of pointer math and > construct a bytes object from the given area of memory -- after all, > it must have a pointer to that memory area in order to do the matching > in the first place (although I realize the code may be separated by a > gulf of abstraction :-). It shouldn't be too bad - I tracked it down through sre_compile, and everything seems to funnel into match_getslice_by_index [1], so it should be possible to detect the non-bytes, non-strings there and coerce them to bytes. OTOH, you can already get the same effect by explicitly wrapping the input in memoryview before passing it to re, and then converting the output to bytes to release the reference to the underlying data, and doing that doesn't raise ugly backwards compatibility concerns.... Cheers, Nick. [1] http://hg.python.org/cpython/file/daf9ea42b610/Modules/_sre.c#l3198 -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ronaldoussoren at mac.com Tue Jul 16 08:32:41 2013 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Tue, 16 Jul 2013 08:32:41 +0200 Subject: [Python-Dev] [Python-checkins] cpython: Issue #18393: Remove use of deprecated API on OSX In-Reply-To: References: <3bv9J80Cjxz7LjQ@mail.python.org> Message-ID: On 15 Jul, 2013, at 18:43, Zachary Ware wrote: > On Mon, Jul 15, 2013 at 11:32 AM, ronald.oussoren > wrote: >> http://hg.python.org/cpython/rev/ccbaf6762b54 >> changeset: 84634:ccbaf6762b54 >> user: Ronald Oussoren >> date: Mon Jul 15 18:32:09 2013 +0200 >> summary: >> Issue #18393: Remove use of deprecated API on OSX >> >> The "Gestalt" function on OSX is deprecated (starting with OSX 10.8), >> remove its usage from the stdlib. The patch removes a number of private > > I believe this means that Lib/test/leakers/test_gestalt.py can be > removed as well. Interesting... test_gestalt.py cannot have worked in Py3k at all. I've removed the file. Thanks, Ronald From solipsis at pitrou.net Tue Jul 16 09:55:51 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 16 Jul 2013 09:55:51 +0200 Subject: [Python-Dev] Misc re.match() complaint References: <51E48F8B.1090905@mrabarnett.plus.com> <87vc4bs1lm.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <20130716095551.5aca0285@pitrou.net> Le Mon, 15 Jul 2013 21:53:42 -0700, Guido van Rossum a ?crit : > Hm. I'd still like to change this, but I understand it's debatable... > Is the group() method written in C or Python? Is there a strong enough use case to change it? I can't say the current behaviour seems very useful either, but some people may depend on it. I already find it a bit weird that you're passing a bytearray or memoryview to re.match(), to be honest :-) Regards Antoine. > If it's in C it should > be simple enough to let it just do a little bit of pointer math and > construct a bytes object from the given area of memory -- after all, > it must have a pointer to that memory area in order to do the matching > in the first place (although I realize the code may be separated by a > gulf of abstraction :-). > > --Guido > > On Mon, Jul 15, 2013 at 8:03 PM, Nick Coghlan > wrote: > > On 16 July 2013 12:20, Guido van Rossum wrote: > >> On Mon, Jul 15, 2013 at 7:03 PM, Stephen J. Turnbull > >> wrote: > >>> Or is this something deeper, that a group *is* a new object in > >>> principle? > >> > >> No, I just think of it as returning "a string" and I think it's > >> most useful if that is always an immutable object, even if the > >> target string is some other bytes buffer. > >> > >> FWIW, it feels as if the change in behavior is probably just due to > >> how slices work. > > > > I took a look at the way the 2.7 re code works, and the change does > > indeed appear to be due to the difference in the way slices work for > > buffer and memoryview objects: > > > > Slicing a buffer creates an 8-bit string: > > > >>>> buffer(b"abc")[0:1] > > 'a' > > > > Slicing a memoryview creates another memoryview: > > > >>>> memoryview(b"abc")[0:1] > > > > > > Unfortunately, memoryview doesn't currently allow subclasses, so it > > isn't easy to create a derivative that coerces to bytes on > > slicing :( > > > > Cheers, > > Nick. > > > > -- > > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > > > From solipsis at pitrou.net Tue Jul 16 09:57:50 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 16 Jul 2013 09:57:50 +0200 Subject: [Python-Dev] Is it safe to call PyEval_EvalFrameEx() with an exception set? References: Message-ID: <20130716095750.77bdf874@pitrou.net> Le Tue, 16 Jul 2013 02:34:49 +0200, Victor Stinner a ?crit : > Hi, > > I'm working on the issue #18408 (fix issues found by my pyfailmalloc > tool). To analyze some bugs, I have to add debug traces in various > functions to find which function returned NULL without setting an > error, or the opposite: returned a valid object, but with an exception > set. > > I would like to add assertions in Python/ceval.c to detect such bugs > earlier. The problem is that some functions rely on the ability to > call PyEval_EvalFrameEx() with an exception set. Is it expected? > Should it be avoided? Well, if your frame is a generator frame, you must be able to throw() into it, i.e. resume it with an exception set. The difficulty here is that there are different meanings for "an exception is set": - PyErr_Occurred() is true - the f_exc_type field and friends are set - possibly another one that I'm forgetting about Regards Antoine. From tjreedy at udel.edu Tue Jul 16 10:41:55 2013 From: tjreedy at udel.edu (Terry Reedy) Date: Tue, 16 Jul 2013 04:41:55 -0400 Subject: [Python-Dev] Tweaking PEP 8 guidelines for use of leading underscores In-Reply-To: References: <20130715184039.60590e62@anarchist> <1373934193.30382.59.camel@thinko> <51E4A0EF.8070205@pearwood.info> <1373941235.30382.62.camel@thinko> <1373943762.30382.88.camel@thinko> Message-ID: On 7/15/2013 11:11 PM, Nick Coghlan wrote: > I'll look into adding some stronger wording at the top making it clear > that while PEP 8 is a useful starting point and a good default if a > project doesn't have a defined style guide of it's own, it is *not* > the be-all-and-end-all for Python style guides. Treating it as such as > an abuse of the PEP, pure and simple. How about (re)naming it Style Guide for Python Standard Library Code Without 'Standard Library' in the title, I can see how some people might think it meant to apply to all code. I do not think we need to worry about the above being too narrow. -- Terry Jan Reedy From tjreedy at udel.edu Tue Jul 16 11:18:28 2013 From: tjreedy at udel.edu (Terry Reedy) Date: Tue, 16 Jul 2013 05:18:28 -0400 Subject: [Python-Dev] Misc re.match() complaint In-Reply-To: References: Message-ID: On 7/15/2013 7:14 PM, Guido van Rossum wrote: > In a discussion about mypy I discovered that the Python 3 version of > the re module's Match object behaves subtly different from the Python > 2 version when the target string (i.e. the haystack, not the needle) > is a buffer object. > > In Python 2, the type of the return value of group() is always either > a Unicode string or an 8-bit string, and the type is determined by > looking at the target string -- if the target is unicode, group() > returns a unicode string, otherwise, group() returns an 8-bit string. > In particular, if the target is a buffer object, group() returns an > 8-bit string. I think this is the appropriate behavior: otherwise > using regular expression matching to extract a small substring from a > large target string would unnecessarily keep the large target string > alive as long as the substring is alive. > > But in Python 3, the behavior of group() has changed so that its > return type always matches that of the target string. I think this is > bad -- apart from the lifetime concern, it means that if your target > happens to be a bytearray, the return value isn't even hashable! > > Does anyone remember whether this was a conscious decision? Is it too > late to fix? In both Python 2 and Python 3, the second sentence of the docs is "Both patterns and strings to be searched can be Unicode strings as well as 8-bit strings." The Python 3 version goes on to say that patterns and targets must match. "However, Unicode strings and 8-bit strings cannot be mixed." I normally consider '8-bit string' to mean 'bytes'. It certainly meant that in Python 2. We use 'buffer object' or 'object satisfying the buffer protocol' to mean 'bytes, byte_arrays, or memoryviews'. I wonder if the change was an artifact of changing the code to prohibit mixing Unicode and bytes. Going on "match.group([group1, ...]) Returns one or more subgroups of the match. If there is a single argument, the result is a single string;" In both 2.x and 3.x docs, I usually understand generic 'string' to mean 'Unicode or bytes'. In any case, The sentence and a half from 'Returns' to 'string' is *exactly the same* as in the 2.x docs. As near as I could tell looking by the, the rest of the entry for match.group is unchanged from 2.x to 3.x. So it is easy to think that the behavior change is an unintended regression. -- Terry Jan Reedy From tjreedy at udel.edu Tue Jul 16 11:23:54 2013 From: tjreedy at udel.edu (Terry Reedy) Date: Tue, 16 Jul 2013 05:23:54 -0400 Subject: [Python-Dev] Misc re.match() complaint In-Reply-To: References: <51E48F8B.1090905@mrabarnett.plus.com> <87vc4bs1lm.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: On 7/15/2013 10:20 PM, Guido van Rossum wrote: >> Or is this something deeper, that a group *is* a new object in >> principle? > > No, I just think of it as returning "a string" That is exactly what the doc says it does. See my other post. -- Terry Jan Reedy From victor.stinner at gmail.com Tue Jul 16 12:08:11 2013 From: victor.stinner at gmail.com (Victor Stinner) Date: Tue, 16 Jul 2013 12:08:11 +0200 Subject: [Python-Dev] Is it safe to call PyEval_EvalFrameEx() with an exception set? In-Reply-To: <20130716095750.77bdf874@pitrou.net> References: <20130716095750.77bdf874@pitrou.net> Message-ID: 2013/7/16 Antoine Pitrou : > Le Tue, 16 Jul 2013 02:34:49 +0200, > Victor Stinner a ?crit : >> I would like to add assertions in Python/ceval.c to detect such bugs >> earlier. The problem is that some functions rely on the ability to >> call PyEval_EvalFrameEx() with an exception set. Is it expected? >> Should it be avoided? > > Well, if your frame is a generator frame, you must be able to throw() > into it, i.e. resume it with an exception set. > > The difficulty here is that there are different meanings for "an > exception is set": > - PyErr_Occurred() is true > - the f_exc_type field and friends are set > - possibly another one that I'm forgetting about Ah yes, a generator can call PyEval_EvalFrameEx() with an exception set (PyErr_Occurred() is non-NULL), but it sets throwflag to 1. My question is when an exception is set (PyErr_Occurred() is non-NULL) at the beginning of the bytecode evaluation loop. Just after the "goto error;" in this extract of ceval.c: if (throwflag) /* support for generator.throw() */ goto error; for (;;) { ... } Victor From shibturn at gmail.com Tue Jul 16 12:28:28 2013 From: shibturn at gmail.com (Richard Oudkerk) Date: Tue, 16 Jul 2013 11:28:28 +0100 Subject: [Python-Dev] Tweaking PEP 8 guidelines for use of leading underscores In-Reply-To: References: <20130715184039.60590e62@anarchist> <1373934193.30382.59.camel@thinko> <51E4A0EF.8070205@pearwood.info> <1373941235.30382.62.camel@thinko> <1373943762.30382.88.camel@thinko> <1373944606.30382.92.camel@thinko> Message-ID: On 16/07/2013 6:44am, Nick Coghlan wrote: > Clarifying what constitutes an internal interface in a way that > doesn't require renaming anything is a necessary prerequisite for > bundling or bootstrapping the pip CLI in Python 3.4 (as pip exposes > its internal implemetnation API as "import pip" rather than "import > _pip" and renaming it would lead to a lot of pointless code churn). > Without that concern, the topic never would have come up. BTW, how does the use of __all__ effect things? Somewhere I got the idea that if a module uses __all__ then anything not listed is internal. I take it that is wrong? -- Richard From ncoghlan at gmail.com Tue Jul 16 13:03:51 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 16 Jul 2013 21:03:51 +1000 Subject: [Python-Dev] Misc re.match() complaint In-Reply-To: References: Message-ID: On 16 July 2013 19:18, Terry Reedy wrote: > I wonder if the change was an artifact of changing the code to prohibit > mixing Unicode and bytes. I'm pretty sure we the only thing we changed in 3.x is to migrate re to the PEP 3118 buffer API, and the behavioural change Guido is seeing is actually the one between the 2.x buffer (which returns 8-bit strings when sliced) and other types (including memoryview) which return instances of themselves. Getting the old buffer behaviour in 3.x without an extra copy operation should just be a matter of wrapping the input with memoryview (to avoid copying the group elements in the match object) and the output with bytes (to avoid keeping the entire original object alive just to reference a few small pieces of it that were matched by the regex): >>> import re >>> data = bytearray(b"aaabbbcccddd") >>> re.match(b"(a*)b*c*(d*)", data).group(2) bytearray(b'ddd') >>> bytes(re.match(b"(a*)b*c*(d*)", memoryview(data)).group(2)) b'ddd' Given that, I'm inclined to keep the existing behaviour on backwards compatibility grounds. To make the above code work on both 2.x *and* 3.x without making an extra copy, it's possible to keep the bytes call (it should be a no-op on 2.x) and dynamically switch the type used to wrap the input between buffer in 2.x and memoryview in 3.x (unfortunately, the 2.x memoryview doesn't work for this case, as the 2.x re API doesn't accept it as valid input). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Tue Jul 16 13:09:23 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 16 Jul 2013 21:09:23 +1000 Subject: [Python-Dev] Tweaking PEP 8 guidelines for use of leading underscores In-Reply-To: References: <20130715184039.60590e62@anarchist> <1373934193.30382.59.camel@thinko> <51E4A0EF.8070205@pearwood.info> <1373941235.30382.62.camel@thinko> <1373943762.30382.88.camel@thinko> Message-ID: On 16 July 2013 18:41, Terry Reedy wrote: > On 7/15/2013 11:11 PM, Nick Coghlan wrote: > >> I'll look into adding some stronger wording at the top making it clear >> that while PEP 8 is a useful starting point and a good default if a >> project doesn't have a defined style guide of it's own, it is *not* >> the be-all-and-end-all for Python style guides. Treating it as such as >> an abuse of the PEP, pure and simple. > > > How about (re)naming it > Style Guide for Python Standard Library Code > > Without 'Standard Library' in the title, I can see how some people might > think it meant to apply to all code. I do not think we need to worry about > the above being too narrow. I reread the whole thing today - it turns out we *have* tweaked it over time to account for other people using it for their projects as well. The one genuinely standard library specific element still in it (the prohibition on the use of function annotations) is explicitly qualified as only applying to the standard library and not anything else. So I think we need to continue supporting that use case. I did find it interesting that we *don't* explicitly advise against the use of "import *" for anything other than optional accelerator modules or republishing internal interfaces through a public API, even though we advice against the practice in the tutorial. Perhaps another oversight worth correcting? Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Tue Jul 16 13:10:50 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 16 Jul 2013 21:10:50 +1000 Subject: [Python-Dev] Tweaking PEP 8 guidelines for use of leading underscores In-Reply-To: References: <20130715184039.60590e62@anarchist> <1373934193.30382.59.camel@thinko> <51E4A0EF.8070205@pearwood.info> <1373941235.30382.62.camel@thinko> <1373943762.30382.88.camel@thinko> <1373944606.30382.92.camel@thinko> Message-ID: On 16 July 2013 20:28, Richard Oudkerk wrote: > On 16/07/2013 6:44am, Nick Coghlan wrote: >> >> Clarifying what constitutes an internal interface in a way that >> doesn't require renaming anything is a necessary prerequisite for >> bundling or bootstrapping the pip CLI in Python 3.4 (as pip exposes >> its internal implemetnation API as "import pip" rather than "import >> _pip" and renaming it would lead to a lot of pointless code churn). >> Without that concern, the topic never would have come up. > > > BTW, how does the use of __all__ effect things? Somewhere I got the idea > that if a module uses __all__ then anything not listed is internal. I take > it that is wrong? Gah, you're right. I did consider that, then forgot to include it in the proposed text. Rather than throwing more versions of the text at the list, I'll create a tracker issue and post a proposed patch for PEP 8. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Tue Jul 16 13:40:00 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 16 Jul 2013 21:40:00 +1000 Subject: [Python-Dev] Why does PEP 8 advise against explicit relative imports? Message-ID: PEP 8 advises developers to use absolute imports rather than explicit relative imports. Why? Using absolute imports couple the internal implementation of a package to its public name - you can't just change the top level directory name any more, you have to go through and change all the absolute imports as well. You also can't easily vendor a package that uses absolute imports inside another project either, since all the absolute imports will break. What's the *concrete* benefit of using absolute imports that overcomes these significant shortcomings in maintainability and composability? Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From thomas at python.org Tue Jul 16 14:02:28 2013 From: thomas at python.org (Thomas Wouters) Date: Tue, 16 Jul 2013 14:02:28 +0200 Subject: [Python-Dev] Why does PEP 8 advise against explicit relative imports? In-Reply-To: References: Message-ID: On Tue, Jul 16, 2013 at 1:40 PM, Nick Coghlan wrote: > PEP 8 advises developers to use absolute imports rather than explicit > relative imports. > > Why? Using absolute imports couple the internal implementation of a > package to its public name - you can't just change the top level > directory name any more, you have to go through and change all the > absolute imports as well. You also can't easily vendor a package that > uses absolute imports inside another project either, since all the > absolute imports will break. > The problem with relative imports (both explicit and implicit) is that it makes it less obvious when you are importing a module under the wrong name. If a package ends up in sys.path directly (for example, by executing something that lives inside it directly) then an explicit relative import directly in the package will fail, but an explicit relative import in a sub-package won't, and you can end up with the subtly confusing mess of multiple imports of the same .py file under different names. > What's the *concrete* benefit of using absolute imports that overcomes > these significant shortcomings in maintainability and composability? > In my experience changing the name of a package is problematic for more reasons than just 'import'. We used to do the put-in-a-package thing for third-party packages at Google, and we quit doing it because it was just too painful: it's hard to find all the places that reference the package by name (think of __import__, or mucking in sys.modules), and it's extra problematic with packages that contain extension modules. You can't rename a package, even for 'vendoring a package', without carefully checking whether it'll work -- unless you don't care if it'll work, of course :) The alternative is to include a package *without* changing its name (by adding the right directory, without __init__.py, to sys.path) and that has worked out a lot better for us. Even so, there's no reason for *the standard library* to use explicit relative imports, and that's what PEP 8 is supposed to cover, right? :) -- Thomas Wouters Hi! I'm an email virus! Think twice before sending your email to help me spread! -------------- next part -------------- An HTML attachment was scrubbed... URL: From martin at v.loewis.de Tue Jul 16 14:17:39 2013 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 16 Jul 2013 14:17:39 +0200 Subject: [Python-Dev] Python 3.4 and Windows XP: just 45 days until EOL In-Reply-To: <51DF469B.6010007@python.org> References: <51DF469B.6010007@python.org> Message-ID: <51E539E3.2020104@v.loewis.de> Am 12.07.13 01:58, schrieb Christian Heimes: > For Python 3.4 is going to be a very close call. According to PEP 429 > 3.4.0 final is scheduled for February 22, 2014. The extended support > phase of Windows XP ends merely 45 days later on April 8, 2014. Do we > really have to restrict ourselves to an API that is going to become > deprecated 45 days after the estimated release of 3.4.0? I suggest to stick to the words of the PEP, which means that XP needs to be supported in 3.4. For 3.5, according to the PEP, XP support *may* be dropped - it doesn't have to be dropped. Notice that you cannot yet know for sure that XP support ends on April 8 - even though it's likely that it will. Microsoft might (and, in the past did) extend an earlier end-of-support date (although they probably won't do so for XP anymore). The most significant reason for dropping XP support in Python will likely be the desire to switch to new VC versions: Newer CRTs don't support XP anymore. Regards, Martin From ncoghlan at gmail.com Tue Jul 16 14:24:17 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 16 Jul 2013 22:24:17 +1000 Subject: [Python-Dev] Tweaking PEP 8 guidelines for use of leading underscores In-Reply-To: References: <20130715184039.60590e62@anarchist> <1373934193.30382.59.camel@thinko> <51E4A0EF.8070205@pearwood.info> <1373941235.30382.62.camel@thinko> <1373943762.30382.88.camel@thinko> <1373944606.30382.92.camel@thinko> Message-ID: On 16 July 2013 21:10, Nick Coghlan wrote: > On 16 July 2013 20:28, Richard Oudkerk wrote: >> On 16/07/2013 6:44am, Nick Coghlan wrote: >>> >>> Clarifying what constitutes an internal interface in a way that >>> doesn't require renaming anything is a necessary prerequisite for >>> bundling or bootstrapping the pip CLI in Python 3.4 (as pip exposes >>> its internal implemetnation API as "import pip" rather than "import >>> _pip" and renaming it would lead to a lot of pointless code churn). >>> Without that concern, the topic never would have come up. >> >> >> BTW, how does the use of __all__ effect things? Somewhere I got the idea >> that if a module uses __all__ then anything not listed is internal. I take >> it that is wrong? > > Gah, you're right. I did consider that, then forgot to include it in > the proposed text. > > Rather than throwing more versions of the text at the list, I'll > create a tracker issue and post a proposed patch for PEP 8. Tracker issue: http://bugs.python.org/issue18472 When I went to write this in context, I realised it made more sense to focus on defining what a *public* interface was, and how that differed from an internal interface. WIth that approach, the natural flow of the text was: * the public interface is preferentially defined by the documentation * __all__ is the main programmatic interface * leading underscores are then mostly a helper that makes it easier to remember which is which while actually *working* on the code (but still a good practice) I also spotted a few other oddities as I went through. The obviously wrong ones I just fixed, but there were a couple relating to exceptions that seem unnecessary or ill-advised given PEP 352 and 3151, plus the one about absolute vs explicit relative imports that I started a separate thread for. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Tue Jul 16 14:27:03 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 16 Jul 2013 22:27:03 +1000 Subject: [Python-Dev] Why does PEP 8 advise against explicit relative imports? In-Reply-To: References: Message-ID: On 16 July 2013 22:02, Thomas Wouters wrote: > On Tue, Jul 16, 2013 at 1:40 PM, Nick Coghlan wrote: >> >> PEP 8 advises developers to use absolute imports rather than explicit >> relative imports. >> >> Why? Using absolute imports couple the internal implementation of a >> package to its public name - you can't just change the top level >> directory name any more, you have to go through and change all the >> absolute imports as well. You also can't easily vendor a package that >> uses absolute imports inside another project either, since all the >> absolute imports will break. > > > The problem with relative imports (both explicit and implicit) is that it > makes it less obvious when you are importing a module under the wrong name. > If a package ends up in sys.path directly (for example, by executing > something that lives inside it directly) then an explicit relative import > directly in the package will fail, but an explicit relative import in a > sub-package won't, and you can end up with the subtly confusing mess of > multiple imports of the same .py file under different names. > >> >> What's the *concrete* benefit of using absolute imports that overcomes >> these significant shortcomings in maintainability and composability? > > In my experience changing the name of a package is problematic for more > reasons than just 'import'. We used to do the put-in-a-package thing for > third-party packages at Google, and we quit doing it because it was just too > painful: it's hard to find all the places that reference the package by name > (think of __import__, or mucking in sys.modules), and it's extra problematic > with packages that contain extension modules. You can't rename a package, > even for 'vendoring a package', without carefully checking whether it'll > work -- unless you don't care if it'll work, of course :) The alternative is > to include a package *without* changing its name (by adding the right > directory, without __init__.py, to sys.path) and that has worked out a lot > better for us. These are good reasons, but they're not the reasons the PEP currently gives. "Better failure modes" is a good practical benefit :) > Even so, there's no reason for *the standard library* to use explicit > relative imports, and that's what PEP 8 is supposed to cover, right? :) I realised in re-reading the whole thing that we actually admitted defeat on that front back during the last function annotations discussion :) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Tue Jul 16 14:35:08 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 16 Jul 2013 22:35:08 +1000 Subject: [Python-Dev] Why does PEP 8 advise against explicit relative imports? In-Reply-To: References: Message-ID: On 16 July 2013 22:27, Nick Coghlan wrote: > These are good reasons, but they're not the reasons the PEP currently > gives. "Better failure modes" is a good practical benefit :) I have now included an updated rationale for this rule in the patch attached to http://bugs.python.org/issue18472 (the recommendation to use absolute imports remains unchanged). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From mail at timgolden.me.uk Tue Jul 16 14:42:10 2013 From: mail at timgolden.me.uk (Tim Golden) Date: Tue, 16 Jul 2013 13:42:10 +0100 Subject: [Python-Dev] Python 3.4 and Windows XP: just 45 days until EOL In-Reply-To: <51DF469B.6010007@python.org> References: <51DF469B.6010007@python.org> Message-ID: <51E53FA2.4030102@timgolden.me.uk> On 12/07/2013 00:58, Christian Heimes wrote: > Hi, > > how do you feel about dropping Windows XP support for Python 3.4? It > would enable us to use some features that are only available on Windows > Vista and newer, for example http://bugs.python.org/issue6926 and > http://bugs.python.org/issue1763 . > > PEP 11 says: > A new feature release X.Y.0 will support all Windows releases > whose extended support phase is not yet expired. > > For Python 3.4 is going to be a very close call. According to PEP 429 > 3.4.0 final is scheduled for February 22, 2014. The extended support > phase of Windows XP ends merely 45 days later on April 8, 2014. Do we > really have to restrict ourselves to an API that is going to become > deprecated 45 days after the estimated release of 3.4.0? I would like to continue support for WinXP. It's still widely, widely used -- MS support notwithstanding. The situation might have been different if Vista had been a viable corporate desktop, but I suspect that many outfits have waited (as we did here) until Win7 before moving forward. Win7 is now our default desktop for new machines, but we're still running a slight majority of WinXP machines. TJG From steve at pearwood.info Tue Jul 16 15:19:21 2013 From: steve at pearwood.info (Steven D'Aprano) Date: Tue, 16 Jul 2013 23:19:21 +1000 Subject: [Python-Dev] Tweaking PEP 8 guidelines for use of leading underscores In-Reply-To: References: <20130715184039.60590e62@anarchist> <1373934193.30382.59.camel@thinko> <51E4A0EF.8070205@pearwood.info> <1373941235.30382.62.camel@thinko> <1373943762.30382.88.camel@thinko> <1373944606.30382.92.camel@thinko> Message-ID: <51E54859.4070901@pearwood.info> On 16/07/13 20:28, Richard Oudkerk wrote: > On 16/07/2013 6:44am, Nick Coghlan wrote: >> Clarifying what constitutes an internal interface in a way that >> doesn't require renaming anything is a necessary prerequisite for >> bundling or bootstrapping the pip CLI in Python 3.4 (as pip exposes >> its internal implemetnation API as "import pip" rather than "import >> _pip" and renaming it would lead to a lot of pointless code churn). >> Without that concern, the topic never would have come up. > > BTW, how does the use of __all__ effect things? Somewhere I got the idea that if a module uses __all__ then anything not listed is internal. I take it that is wrong? That is not how I interpret __all__. In the absence of any explicit documentation, I interpret __all__ as nothing more than a list of names which wildcard imports will bring in, without necessarily meaning that other names are private. For example, I might have a module explicitly designed for wildcard imports at the interactive interpreter: from module import * brings in the functions which I expect will be useful interactively, not necessarily the entire public API. For example, pkgutil includes classes with single-underscore methods, which I take as private. It also has a function simplegeneric, which is undocumented and not listed in __all__. In in the absence of even a comment saying "Don't use this", I take it as an oversight, not policy that simplegeneric is private. -- Steven From rdmurray at bitdance.com Tue Jul 16 15:39:12 2013 From: rdmurray at bitdance.com (R. David Murray) Date: Tue, 16 Jul 2013 09:39:12 -0400 Subject: [Python-Dev] Tweaking PEP 8 guidelines for use of leading underscores In-Reply-To: <51E54859.4070901@pearwood.info> References: <20130715184039.60590e62@anarchist> <1373934193.30382.59.camel@thinko> <51E4A0EF.8070205@pearwood.info> <1373941235.30382.62.camel@thinko> <1373943762.30382.88.camel@thinko> <1373944606.30382.92.camel@thinko> <51E54859.4070901@pearwood.info> Message-ID: <20130716133912.C532325007E@webabinitio.net> On Tue, 16 Jul 2013 23:19:21 +1000, Steven D'Aprano wrote: > On 16/07/13 20:28, Richard Oudkerk wrote: > > On 16/07/2013 6:44am, Nick Coghlan wrote: > >> Clarifying what constitutes an internal interface in a way that > >> doesn't require renaming anything is a necessary prerequisite for > >> bundling or bootstrapping the pip CLI in Python 3.4 (as pip exposes > >> its internal implemetnation API as "import pip" rather than "import > >> _pip" and renaming it would lead to a lot of pointless code churn). > >> Without that concern, the topic never would have come up. > > > > BTW, how does the use of __all__ effect things? Somewhere I got the idea that if a module uses __all__ then anything not listed is internal. I take it that is wrong? > > > That is not how I interpret __all__. In the absence of any explicit documentation, I interpret __all__ as nothing more than a list of names which wildcard imports will bring in, without necessarily meaning that other names are private. For example, I might have a module explicitly designed for wildcard imports at the interactive interpreter: > > from module import * > > brings in the functions which I expect will be useful interactively, not necessarily the entire public API. > > For example, pkgutil includes classes with single-underscore methods, which I take as private. It also has a function simplegeneric, which is undocumented and not listed in __all__. In in the absence of even a comment saying "Don't use this", I take it as an oversight, not policy that simplegeneric is private. I think you'd be wrong about that, though. simplegeneric should really be treated as private. I'm speaking here not about the general principle of the thing, but about my understanding of simplegeneric's specific history. --David From ncoghlan at gmail.com Tue Jul 16 15:48:06 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 16 Jul 2013 23:48:06 +1000 Subject: [Python-Dev] Tweaking PEP 8 guidelines for use of leading underscores In-Reply-To: <20130716133912.C532325007E@webabinitio.net> References: <20130715184039.60590e62@anarchist> <1373934193.30382.59.camel@thinko> <51E4A0EF.8070205@pearwood.info> <1373941235.30382.62.camel@thinko> <1373943762.30382.88.camel@thinko> <1373944606.30382.92.camel@thinko> <51E54859.4070901@pearwood.info> <20130716133912.C532325007E@webabinitio.net> Message-ID: On 16 July 2013 23:39, R. David Murray wrote: > On Tue, 16 Jul 2013 23:19:21 +1000, Steven D'Aprano wrote: >> On 16/07/13 20:28, Richard Oudkerk wrote: >> > On 16/07/2013 6:44am, Nick Coghlan wrote: >> >> Clarifying what constitutes an internal interface in a way that >> >> doesn't require renaming anything is a necessary prerequisite for >> >> bundling or bootstrapping the pip CLI in Python 3.4 (as pip exposes >> >> its internal implemetnation API as "import pip" rather than "import >> >> _pip" and renaming it would lead to a lot of pointless code churn). >> >> Without that concern, the topic never would have come up. >> > >> > BTW, how does the use of __all__ effect things? Somewhere I got the idea that if a module uses __all__ then anything not listed is internal. I take it that is wrong? >> >> >> That is not how I interpret __all__. In the absence of any explicit documentation, I interpret __all__ as nothing more than a list of names which wildcard imports will bring in, without necessarily meaning that other names are private. For example, I might have a module explicitly designed for wildcard imports at the interactive interpreter: >> >> from module import * >> >> brings in the functions which I expect will be useful interactively, not necessarily the entire public API. >> >> For example, pkgutil includes classes with single-underscore methods, which I take as private. It also has a function simplegeneric, which is undocumented and not listed in __all__. In in the absence of even a comment saying "Don't use this", I take it as an oversight, not policy that simplegeneric is private. > > I think you'd be wrong about that, though. simplegeneric should really be > treated as private. I'm speaking here not about the general principle of > the thing, but about my understanding of simplegeneric's specific history. And, indeed, you're correct (the issue that eventually became the functools.singledispatch PEP started life with a title like "move simplegeneric to functools and make it public"). For the general case, the patch I posted to the issue tracker sets the precedence as docs -> __all__ -> leading underscore. If a module has public APIs that aren't in __all__, it should cover them in the docs, otherwise people should assume they're private. It's OK if there are exceptions to that general rule - there's a reason PEP 8 has the great big qualifier pointing out that it isn't universally applicable (even if we might sometimes wish otherwise). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From barry at python.org Tue Jul 16 16:48:55 2013 From: barry at python.org (Barry Warsaw) Date: Tue, 16 Jul 2013 10:48:55 -0400 Subject: [Python-Dev] Why does PEP 8 advise against explicit relative imports? In-Reply-To: References: Message-ID: <20130716104855.4c2cbecb@anarchist> On Jul 16, 2013, at 02:02 PM, Thomas Wouters wrote: >In my experience changing the name of a package is problematic for more >reasons than just 'import'. We used to do the put-in-a-package thing for >third-party packages at Google, and we quit doing it because it was just >too painful: it's hard to find all the places that reference the package by >name (think of __import__, or mucking in sys.modules), and it's extra >problematic with packages that contain extension modules. You can't rename >a package, even for 'vendoring a package', without carefully checking >whether it'll work -- unless you don't care if it'll work, of course :) The >alternative is to include a package *without* changing its name (by adding >the right directory, without __init__.py, to sys.path) and that has worked >out a lot better for us. Vendoring also gives no end of headaches to distro packagers because we have to un-vendor that stuff, which can lead to a mess of patches on top of upstream. >Even so, there's no reason for *the standard library* to use explicit >relative imports, and that's what PEP 8 is supposed to cover, right? :) Right! It doesn't bother me at all that this particular recommendation may be followed by third parties too. :) -Barry From barry at python.org Tue Jul 16 17:14:03 2013 From: barry at python.org (Barry Warsaw) Date: Tue, 16 Jul 2013 11:14:03 -0400 Subject: [Python-Dev] Tweaking PEP 8 guidelines for use of leading underscores In-Reply-To: <1373943762.30382.88.camel@thinko> References: <20130715184039.60590e62@anarchist> <1373934193.30382.59.camel@thinko> <51E4A0EF.8070205@pearwood.info> <1373941235.30382.62.camel@thinko> <1373943762.30382.88.camel@thinko> Message-ID: <20130716111403.0afd68ac@anarchist> On Jul 15, 2013, at 11:02 PM, Chris McDonough wrote: >Given that misunderstanding, is there a way to divorce stdlib-centric >guidelines like the one being discussed now from "PEP8"-ness? I don't >think any amount of marketing effort or reasoned explanation is going to >separate "PEP8" from "correct code" in people's minds at this point. The standard seems to be "if the pep8 program complains, I forward that upstream", so it's probably most effective to discuss with them. OTOH, we've had to clarify PEP 8 in order to get the pep8 developers to adjust their tool, most recently e.g. the closing brace/bracket/paren for multiline collections. -Barry From barry at python.org Tue Jul 16 17:19:17 2013 From: barry at python.org (Barry Warsaw) Date: Tue, 16 Jul 2013 11:19:17 -0400 Subject: [Python-Dev] Tweaking PEP 8 guidelines for use of leading underscores In-Reply-To: References: <20130715184039.60590e62@anarchist> <1373934193.30382.59.camel@thinko> <51E4A0EF.8070205@pearwood.info> <1373941235.30382.62.camel@thinko> <1373943762.30382.88.camel@thinko> <1373944606.30382.92.camel@thinko> Message-ID: <20130716111917.654718b0@anarchist> On Jul 16, 2013, at 11:28 AM, Richard Oudkerk wrote: >BTW, how does the use of __all__ effect things? Somewhere I got the idea >that if a module uses __all__ then anything not listed is internal. I take >it that is wrong? Purely technically, __all__ is there to affect how from-import-* works. I personally think it's a good idea to include all your public names, and none of your non-public names, in __all__, but it's not always easy to keep up-to-date. pyflakes has the nice benefit of complaining when something is named in __all__ that doesn't exist in the module, but that's only one part of the keeping-things-up-to-date problem. -Barry From barry at python.org Tue Jul 16 17:21:14 2013 From: barry at python.org (Barry Warsaw) Date: Tue, 16 Jul 2013 11:21:14 -0400 Subject: [Python-Dev] Tweaking PEP 8 guidelines for use of leading underscores In-Reply-To: References: <20130715184039.60590e62@anarchist> <1373934193.30382.59.camel@thinko> <51E4A0EF.8070205@pearwood.info> <1373941235.30382.62.camel@thinko> <1373943762.30382.88.camel@thinko> <1373944606.30382.92.camel@thinko> <51E54859.4070901@pearwood.info> <20130716133912.C532325007E@webabinitio.net> Message-ID: <20130716112114.3f04711b@anarchist> On Jul 16, 2013, at 11:48 PM, Nick Coghlan wrote: >For the general case, the patch I posted to the issue tracker sets the >precedence as docs -> __all__ -> leading underscore. +1 -Barry From barry at python.org Tue Jul 16 17:26:55 2013 From: barry at python.org (Barry Warsaw) Date: Tue, 16 Jul 2013 11:26:55 -0400 Subject: [Python-Dev] Tweaking PEP 8 guidelines for use of leading underscores In-Reply-To: References: <20130715184039.60590e62@anarchist> <1373934193.30382.59.camel@thinko> <51E4A0EF.8070205@pearwood.info> <1373941235.30382.62.camel@thinko> <1373943762.30382.88.camel@thinko> <1373944606.30382.92.camel@thinko> <1373954482.30382.111.camel@thinko> Message-ID: <20130716112655.42281637@anarchist> On Jul 16, 2013, at 04:10 PM, Nick Coghlan wrote: >Yeah, we should probably be a bit more vocal in pointing out that PEP >8 is something to be applied judiciously, *not* treated as "sacred >writ, never to be questioned". It's very tempting to just sigh and >ignore it, instead of explicitly pointing out the problems with taking >it too literally for people's personal projects (or, worse, using it >to abuse developers of projects that long predate the recommendations >in more recent versions) :P In my experience, most PEP 8 violation bugs that I see for various upstreams (mine and others) comes from running the pep8 tool over the code. A lot of upstreams do this as part of their normal QA (e.g. you can set up unittests that run pep8, pyflakes, and other linters). Those will generally be fine because they're already committed to conformance to pep8's notion of what PEP 8 says [1]. In other cases, Someone Out There runs pep8 over your code, and files bugs on all the complaints that get produced. That can definitely lead to churn and waste, especially on big, established code bases. -Barry [1] which I do not always agree with anyway, as evidenced by the recent discussion on closing paren indentation for multi-line collections. From guido at python.org Tue Jul 16 19:21:36 2013 From: guido at python.org (Guido van Rossum) Date: Tue, 16 Jul 2013 10:21:36 -0700 Subject: [Python-Dev] Misc re.match() complaint In-Reply-To: <20130716095551.5aca0285@pitrou.net> References: <51E48F8B.1090905@mrabarnett.plus.com> <87vc4bs1lm.fsf@uwakimon.sk.tsukuba.ac.jp> <20130716095551.5aca0285@pitrou.net> Message-ID: On Tue, Jul 16, 2013 at 12:55 AM, Antoine Pitrou wrote: > Is there a strong enough use case to change it? I can't say the current > behaviour seems very useful either, but some people may depend on it. This is the crucial question. I personally see the current behavior as an artifact of the (lack of) design process, not as a conscious decision. Given that we also have m.string, m.start(grp) and m.end(grp), those who need something matching the original type (or even something that is known to be a reference into the original object) can use that API; for most use cases, all you care about is is the selected group as a string, and it is more useful if that is always an immutable string (bytes or str). The situation is most egregious if the target string is a bytearray, where there is currently no way to get the result as an immutable bytes object without an extra copy. (There's no API that lets you create a bytes object directly from a slice of a bytearray.) In terms of backwards compatibility, I wouldn't want to do this in a bugfix release, but for a feature release I think it's fine -- the number of applications that could be bitten by this must be extremely small (and the work-around is backward-compatible: just use m.string[m.start() : m.stop()]). > I already find it a bit weird that you're passing a bytearray or > memoryview to re.match(), to be honest :-) Yes, this is somewhat of an odd corner, but actually most built-in APIs taking bytes also take anything else that can be coerced to bytes (io.open() seems to be the exception, and it feels like an accident -- os.open() *does* accept bytearray and friends). This is quite useful for code that interacts with C code or system calls -- often you have a large buffer shared between C and Python code for efficiency, and being able to do pretty much anything to the buffer that you can do to a bytes object (apart from using it as a dict key) helps a lot. -- --Guido van Rossum (python.org/~guido) From tjreedy at udel.edu Tue Jul 16 23:27:37 2013 From: tjreedy at udel.edu (Terry Reedy) Date: Tue, 16 Jul 2013 17:27:37 -0400 Subject: [Python-Dev] Why does PEP 8 advise against explicit relative imports? In-Reply-To: References: Message-ID: On 7/16/2013 7:40 AM, Nick Coghlan wrote: > PEP 8 advises developers to use absolute imports rather than explicit > relative imports. > > Why? Using absolute imports couple the internal implementation of a > package to its public name - you can't just change the top level > directory name any more, you have to go through and change all the > absolute imports as well. You also can't easily vendor a package that > uses absolute imports inside another project either, since all the > absolute imports will break. > > What's the *concrete* benefit of using absolute imports that overcomes > these significant shortcomings in maintainability and composability? From my viewpoint, absolute imports always work, on all python versions, whereas people have reported various problems with relative imports on python-list. At least some of the complaints were valid, at least at the time, but I have not paid too much attention. Even if idlelib files used relative imports, 'idlelib' cannot change and I see no reason to ever change 'idle_test'. I would like to change camelCaseFileNames.py, but if ever done, that would affect both types of imports. -- Terry Jan Reedy From tjreedy at udel.edu Tue Jul 16 23:46:34 2013 From: tjreedy at udel.edu (Terry Reedy) Date: Tue, 16 Jul 2013 17:46:34 -0400 Subject: [Python-Dev] Tweaking PEP 8 guidelines for use of leading underscores In-Reply-To: <20130716133912.C532325007E@webabinitio.net> References: <20130715184039.60590e62@anarchist> <1373934193.30382.59.camel@thinko> <51E4A0EF.8070205@pearwood.info> <1373941235.30382.62.camel@thinko> <1373943762.30382.88.camel@thinko> <1373944606.30382.92.camel@thinko> <51E54859.4070901@pearwood.info> <20130716133912.C532325007E@webabinitio.net> Message-ID: On 7/16/2013 9:39 AM, R. David Murray wrote: > On Tue, 16 Jul 2013 23:19:21 +1000, Steven D'Aprano wrote: >> For example, pkgutil includes classes with single-underscore methods, which I take as private. It also has a function simplegeneric, which is undocumented and not listed in __all__. In in the absence of even a comment saying "Don't use this", I take it as an oversight, not policy that simplegeneric is private. > > I think you'd be wrong about that, though. simplegeneric should really be > treated as private. I'm speaking here not about the general principle of > the thing, but about my understanding of simplegeneric's specific history. I think Steven (valid) point is "Why not, then, say it is internal either in docs or name?"-- which in this case would be in the docs. -- Terry Jan Reedy From storchaka at gmail.com Tue Jul 16 23:56:00 2013 From: storchaka at gmail.com (Serhiy Storchaka) Date: Wed, 17 Jul 2013 00:56:00 +0300 Subject: [Python-Dev] cpython: Issue #18408: Fix fileio_read() on _PyBytes_Resize() failure In-Reply-To: <3bvvP70WNXz7Ljg@mail.python.org> References: <3bvvP70WNXz7Ljg@mail.python.org> Message-ID: 17.07.13 00:09, victor.stinner ???????(??): > http://hg.python.org/cpython/rev/533eb9ab895a > changeset: 84669:533eb9ab895a > user: Victor Stinner > date: Tue Jul 16 21:36:02 2013 +0200 > summary: > Issue #18408: Fix fileio_read() on _PyBytes_Resize() failure > > bytes is NULL on _PyBytes_Resize() failure > > files: > Modules/_io/fileio.c | 2 +- > 1 files changed, 1 insertions(+), 1 deletions(-) > > > diff --git a/Modules/_io/fileio.c b/Modules/_io/fileio.c > --- a/Modules/_io/fileio.c > +++ b/Modules/_io/fileio.c > @@ -739,7 +739,7 @@ > > if (n != size) { > if (_PyBytes_Resize(&bytes, n) < 0) { > - Py_DECREF(bytes); > + Py_CLEAR(bytes); > return NULL; > } > } Why not Py_DECREF? From victor.stinner at gmail.com Wed Jul 17 00:03:14 2013 From: victor.stinner at gmail.com (Victor Stinner) Date: Wed, 17 Jul 2013 00:03:14 +0200 Subject: [Python-Dev] cpython: Issue #18408: Fix fileio_read() on _PyBytes_Resize() failure In-Reply-To: References: <3bvvP70WNXz7Ljg@mail.python.org> Message-ID: 2013/7/16 Serhiy Storchaka : >> http://hg.python.org/cpython/rev/533eb9ab895a >> summary: >> Issue #18408: Fix fileio_read() on _PyBytes_Resize() failure >> >> bytes is NULL on _PyBytes_Resize() failure > > Why not Py_DECREF? Because Py_DECREF(NULL) does crash. Victor From storchaka at gmail.com Wed Jul 17 00:25:15 2013 From: storchaka at gmail.com (Serhiy Storchaka) Date: Wed, 17 Jul 2013 01:25:15 +0300 Subject: [Python-Dev] cpython: Issue #18408: Fix fileio_read() on _PyBytes_Resize() failure In-Reply-To: References: <3bvvP70WNXz7Ljg@mail.python.org> Message-ID: 17.07.13 01:03, Victor Stinner ???????(??): > 2013/7/16 Serhiy Storchaka : >>> http://hg.python.org/cpython/rev/533eb9ab895a >>> summary: >>> Issue #18408: Fix fileio_read() on _PyBytes_Resize() failure >>> >>> bytes is NULL on _PyBytes_Resize() failure >> >> Why not Py_DECREF? > > Because Py_DECREF(NULL) does crash. Oh, I meaned Py_XDECREF. From christian at python.org Wed Jul 17 00:29:05 2013 From: christian at python.org (Christian Heimes) Date: Wed, 17 Jul 2013 00:29:05 +0200 Subject: [Python-Dev] cpython: Issue #18408: Fix fileio_read() on _PyBytes_Resize() failure In-Reply-To: References: <3bvvP70WNXz7Ljg@mail.python.org> Message-ID: Am 17.07.2013 00:03, schrieb Victor Stinner: > 2013/7/16 Serhiy Storchaka : >>> http://hg.python.org/cpython/rev/533eb9ab895a >>> summary: >>> Issue #18408: Fix fileio_read() on _PyBytes_Resize() failure >>> >>> bytes is NULL on _PyBytes_Resize() failure >> >> Why not Py_DECREF? > > Because Py_DECREF(NULL) does crash. Why not Py_XDECREF() then? From victor.stinner at gmail.com Wed Jul 17 00:35:17 2013 From: victor.stinner at gmail.com (Victor Stinner) Date: Wed, 17 Jul 2013 00:35:17 +0200 Subject: [Python-Dev] cpython: Issue #18408: Fix fileio_read() on _PyBytes_Resize() failure In-Reply-To: References: <3bvvP70WNXz7Ljg@mail.python.org> Message-ID: 2013/7/17 Serhiy Storchaka : > Oh, I meaned Py_XDECREF. Ah ok :-) Well, for this specific code, it can probably be replaced with: if (_PyBytes_Resize(&bytes, n) < 0) return NULL; I'm not sure that _PyBytes_Resize() *always* decref bytes and then set bytes to NULL. I was too lazy to check this. If someone wants to audit the code of _PyBytes_Resize(), there are many other places where Py_XDECREF / Py_CLEAR can be removed on _PyBytes_Resize() failure. And I'm quite sure that there are still places where Py_DECREF() is still used on _PyBytes_Resize() failure ;-) Victor From victor.stinner at gmail.com Wed Jul 17 01:08:31 2013 From: victor.stinner at gmail.com (Victor Stinner) Date: Wed, 17 Jul 2013 01:08:31 +0200 Subject: [Python-Dev] Is it safe to call PyEval_EvalFrameEx() with an exception set? In-Reply-To: References: Message-ID: 2013/7/16 Victor Stinner : > I would like to add assertions in Python/ceval.c to detect such bugs > earlier. The problem is that some functions rely on the ability to > call PyEval_EvalFrameEx() with an exception set. Is it expected? > Should it be avoided? The current exception can be replaced with a new > exception. Another example. Py_ReprLeave() function can be called while an exception is set. I'm not sure that it is "safe" to call new code while an exception is set. The function can raise a new exception. For example, PyList_SetSlice() can fail because of a MemoryError. I modified Py_ReprLeave() to save and then restore the current exception: http://hg.python.org/cpython/rev/28ff7ac91477 If Py_ReprLeave() raises a new exception, it is clears (replaced by the previous exception context). Victor From tseaver at palladion.com Wed Jul 17 01:34:03 2013 From: tseaver at palladion.com (Tres Seaver) Date: Tue, 16 Jul 2013 19:34:03 -0400 Subject: [Python-Dev] Tweaking PEP 8 guidelines for use of leading underscores In-Reply-To: References: <20130715184039.60590e62@anarchist> <1373934193.30382.59.camel@thinko> <51E4A0EF.8070205@pearwood.info> <1373941235.30382.62.camel@thinko> <1373943762.30382.88.camel@thinko> Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 07/16/2013 07:09 AM, Nick Coghlan wrote: > I did find it interesting that we *don't* explicitly advise against > the use of "import *" for anything other than optional accelerator > modules or republishing internal interfaces through a public API, > even though we advice against the practice in the tutorial. Perhaps > another oversight worth correcting? +1. 'from foo import *' (from any source other than another module in the same package) is a code stench far worse than anything else PEP8 proscribes. Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with undefined - http://www.enigmail.net/ iEYEARECAAYFAlHl2GsACgkQ+gerLs4ltQ6+1QCgmu5k6p5otzPxvzGh5mA1Ch7t 2f0AoK2a0/m144bnIwBFLLuY9l2bdWMN =878p -----END PGP SIGNATURE----- From barry at python.org Wed Jul 17 03:07:33 2013 From: barry at python.org (Barry Warsaw) Date: Tue, 16 Jul 2013 21:07:33 -0400 Subject: [Python-Dev] Tweaking PEP 8 guidelines for use of leading underscores In-Reply-To: References: <20130715184039.60590e62@anarchist> <1373934193.30382.59.camel@thinko> <51E4A0EF.8070205@pearwood.info> <1373941235.30382.62.camel@thinko> <1373943762.30382.88.camel@thinko> Message-ID: <20130716210733.2ccd5192@limelight.wooz.org> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 On Jul 16, 2013, at 07:34 PM, Tres Seaver wrote: >On 07/16/2013 07:09 AM, Nick Coghlan wrote: > >> I did find it interesting that we *don't* explicitly advise against >> the use of "import *" for anything other than optional accelerator >> modules or republishing internal interfaces through a public API, >> even though we advice against the practice in the tutorial. Perhaps >> another oversight worth correcting? > >+1. 'from foo import *' (from any source other than another module in >the same package) is a code stench far worse than anything else PEP8 >proscribes. Maybe we should disable it for anything that isn't at the interactive prompt <0.5 wink>. - -Barry -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.20 (GNU/Linux) iQIcBAEBCAAGBQJR5e5VAAoJEBJutWOnSwa/SK4P/22KvWNRQ7sjDWJ+jebgpXyD mkUeO4r905uqp5Bvedlyj4GoFhdFPQoeQEGgQeNwiMkFxCGJDKiY1XoJAgqj2zOE vUgoyLE5YSAnz8xG/Ib3ZZPXhDJY14gu/czbX/cc83Q+cQJd0+RpnhUt5jQkxAYR k6xXF7hwTHaTYlqzdTDQUtimg21AUb0lgI5XqCMGL4aac4H69JiN+3ydI3BQxqxY MXe4h4rzDxnaBnekHYCouM1jsN6CsLeIHafJR8W4Zgm9vuxZYIha+kAQd+tILFXG d/0hpvLIGP+ubNPIvS1Ay+aYsH7PiL4keCoP0pi+llMTQv+W5wDeCumLpvXjaEFj 46c7HaRaxbBqpactsG98rZOtmBwSgEwzzPSQak+sWHDCX4RFOS3qDh3WhtY0Lw7n NdEkJ0f+6bEt2ot+emDpth/SAxgOEUjNOhZ8/glYtvZcz2wBWRGkCAIEwSJtP1VZ YgRHFsoUUbS6KoI9cZWhwynmIVQf4Wn9a2Zx5/YUbHIM2mY1s60MxcOn3KgPpEU4 Zjp/gyPHsoIORrZU2Q2pwzdWEmSzorZBP/NqRqlhwBTexKdfqyRFztsFMVcnLPDv Cl0t04BGXH8/0fZ+peEOccqnuPsHumr9VqmhMCcLChRPRQyBD9ETSPVwHhprdtka qkLcyWl/M3ijyuAkdYbl =Q+Du -----END PGP SIGNATURE----- From steve at pearwood.info Wed Jul 17 03:55:42 2013 From: steve at pearwood.info (Steven D'Aprano) Date: Wed, 17 Jul 2013 11:55:42 +1000 Subject: [Python-Dev] Why does PEP 8 advise against explicit relative imports? In-Reply-To: References: Message-ID: <51E5F99E.4080301@pearwood.info> On 16/07/13 22:02, Thomas Wouters wrote: > On Tue, Jul 16, 2013 at 1:40 PM, Nick Coghlan wrote: > >> PEP 8 advises developers to use absolute imports rather than explicit >> relative imports. >> >> Why? Using absolute imports couple the internal implementation of a >> package to its public name - you can't just change the top level >> directory name any more, you have to go through and change all the >> absolute imports as well. You also can't easily vendor a package that >> uses absolute imports inside another project either, since all the >> absolute imports will break. >> > > The problem with relative imports (both explicit and implicit) is that it > makes it less obvious when you are importing a module under the wrong name. > If a package ends up in sys.path directly (for example, by executing > something that lives inside it directly) then an explicit relative import > directly in the package will fail, but an explicit relative import in a > sub-package won't, and you can end up with the subtly confusing mess of > multiple imports of the same .py file under different names. I don't understand this objection. What do you mean by "importing a module under the wrong name"? As I see it, if relative imports are a bad idea for the std lib, they're a bad idea for everyone. (What am I missing?) So why do we have relative imports? Was it a mistake? I don't think so. It seems to me that you're giving some fairly obscure edge cases, if I've understood you correctly. If adding a package directory to sys.path breaks relative imports, isn't that an argument against adding the package directory to sys.path rather than relative imports? >> What's the *concrete* benefit of using absolute imports that overcomes >> these significant shortcomings in maintainability and composability? > > In my experience changing the name of a package is problematic for more > reasons than just 'import'. We used to do the put-in-a-package thing for > third-party packages at Google, and we quit doing it because it was just > too painful: it's hard to find all the places that reference the package by > name (think of __import__, or mucking in sys.modules), Again, these are some fairly odd edge cases. I don't mean to imply that they never occur in practice, obviously they do, but perhaps you're overly sensitive to them. It seems strange to me that PEP 8 should recommend against using relative imports for the 49 packages that would benefit from it just to avoid problems with the 1 that does weird things with __import__ or manually mucks about with sys.modules. Have I missed something? > Even so, there's no reason for *the standard library* to use explicit > relative imports, and that's what PEP 8 is supposed to cover, right? :) If a third-party package uses relative imports, are you suggesting that those rel imports should be turned into absolute imports as a precondition of it being moved into the std lib? -- Steven From guido at python.org Wed Jul 17 04:15:45 2013 From: guido at python.org (Guido van Rossum) Date: Tue, 16 Jul 2013 19:15:45 -0700 Subject: [Python-Dev] Why does PEP 8 advise against explicit relative imports? In-Reply-To: <51E5F99E.4080301@pearwood.info> References: <51E5F99E.4080301@pearwood.info> Message-ID: I have to say that a long time ago, when you couldn't tell a relative import from a top-level import, the ban on relative imports made more sense. -- --Guido van Rossum (python.org/~guido) From ncoghlan at gmail.com Wed Jul 17 04:38:54 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 17 Jul 2013 12:38:54 +1000 Subject: [Python-Dev] Why does PEP 8 advise against explicit relative imports? In-Reply-To: <51E5F99E.4080301@pearwood.info> References: <51E5F99E.4080301@pearwood.info> Message-ID: On 17 July 2013 11:55, Steven D'Aprano wrote: > On 16/07/13 22:02, Thomas Wouters wrote: > As I see it, if relative imports are a bad idea for the std lib, they're a > bad idea for everyone. (What am I missing?) So why do we have relative > imports? Was it a mistake? I don't think so. > > It seems to me that you're giving some fairly obscure edge cases, if I've > understood you correctly. If adding a package directory to sys.path breaks > relative imports, isn't that an argument against adding the package > directory to sys.path rather than relative imports? What Thomas is saying is that, given the current state of the import system, using absolute imports usually give you better behaviour (or at least better error messages) than explicit relative imports do when things go wrong. At present, there are still too many ways to get your import configuration into a bad state (most notably running a module inside a package by filepath rather than the -m switch), and if that happens, using absolute imports is likely to make it a bit easier to find your way out again. I actually agree with that rationale - while I prefer explicit relative imports myself, I'm also more experienced than most when it comes to identifying and resolving import configuration problems (see http://python-notes.curiousefficiency.org/en/latest/python_concepts/import_traps.html) So, for now, avoiding even explicit relative imports is still a good default position. People who feel qualified to reject PEP 8's advice on this topic should also know enough to deal with the potential consequences. I'll be proposing a rewording of this admonition to give a better rationale, but I won't be proposing that the recommendation be changed :) Once we get sys.path initialisation to a better place (amongst other things), then it will be time to revisit the recommendation itself. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From guido at python.org Wed Jul 17 05:48:03 2013 From: guido at python.org (Guido van Rossum) Date: Tue, 16 Jul 2013 20:48:03 -0700 Subject: [Python-Dev] Why does PEP 8 advise against explicit relative imports? In-Reply-To: References: <51E5F99E.4080301@pearwood.info> Message-ID: If sounds like the main problem with relative imports (even explicit) is caused by ignoring the package structure and running modules inside a package as a script, without using -m. Maybe we should update the PEP to make this abundantly clear? I note that an additional problem with informational PEPs like PEP 8 is that they aren't tied to a particular Python version, so even if we solved the above issue perfectly in Python 3.4 there'd still be tons of people applying PEP 8 to older code who would benefit from the warning against relative imports. So maybe we should also add that to the PEP. -- --Guido van Rossum (python.org/~guido) From stephen at xemacs.org Wed Jul 17 06:15:16 2013 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Wed, 17 Jul 2013 13:15:16 +0900 Subject: [Python-Dev] Misc re.match() complaint In-Reply-To: References: <51E48F8B.1090905@mrabarnett.plus.com> <87vc4bs1lm.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <87li55styj.fsf@uwakimon.sk.tsukuba.ac.jp> Terry Reedy writes: > On 7/15/2013 10:20 PM, Guido van Rossum wrote: > > >> Or is this something deeper, that a group *is* a new object in > >> principle? > > > > No, I just think of it as returning "a string" > > That is exactly what the doc says it does. See my other post. The problem is that IIUC '"a string"' is intentionally *not* referring to the usual "str or bytes objects" (at least that's one of the standard uses for scare quotes, to indicate an unusual usage). Either the docstring is using "string" in a similarly ambiguous way, or else it's incorrect under the interpretation that buffer objects are *not* "strings", so they should be inadmissible as targets. Something should be fixed, and I suppose it should be the return type of group(). BTW, I suggest that Terry's usage of "string" (to mean "str or bytes" in 3.x, "unicode or str" in 2.x) be adopted, and Guido's "stringish" be given expanded meaning, including buffer objects. Then we can say informally that in searching and matching a target is a stringish, the pattern is a stringish (?) or compiled re, but the group method returns a string. Steve From ronaldoussoren at mac.com Wed Jul 17 07:58:03 2013 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Wed, 17 Jul 2013 07:58:03 +0200 Subject: [Python-Dev] Why does PEP 8 advise against explicit relative imports? In-Reply-To: References: Message-ID: <741654DE-CA5C-42D7-AB3A-D950EAEB28BC@mac.com> On 16 Jul, 2013, at 14:02, Thomas Wouters wrote: > On Tue, Jul 16, 2013 at 1:40 PM, Nick Coghlan wrote: > >> PEP 8 advises developers to use absolute imports rather than explicit >> relative imports. >> >> Why? Using absolute imports couple the internal implementation of a >> package to its public name - you can't just change the top level >> directory name any more, you have to go through and change all the >> absolute imports as well. You also can't easily vendor a package that >> uses absolute imports inside another project either, since all the >> absolute imports will break. >> > > The problem with relative imports (both explicit and implicit) is that it > makes it less obvious when you are importing a module under the wrong name. > If a package ends up in sys.path directly (for example, by executing > something that lives inside it directly) then an explicit relative import > directly in the package will fail, but an explicit relative import in a > sub-package won't, and you can end up with the subtly confusing mess of > multiple imports of the same .py file under different names. That's only a problem for subpackages (that is, for example when distutils.commands ends up being importable as commands), because explicit relative imports (``from .other import attr``) don't work in toplevel modules or scripts because the leading dot means "the current package". I tend to use explicit relative imports in my code when that increases readability by reducing unnecessary duplication. That is, readability tends to suffer when you have a number of lines like with "from package.module1 import name". Ronald From tjreedy at udel.edu Wed Jul 17 11:05:50 2013 From: tjreedy at udel.edu (Terry Reedy) Date: Wed, 17 Jul 2013 05:05:50 -0400 Subject: [Python-Dev] Misc re.match() complaint In-Reply-To: <87li55styj.fsf@uwakimon.sk.tsukuba.ac.jp> References: <51E48F8B.1090905@mrabarnett.plus.com> <87vc4bs1lm.fsf@uwakimon.sk.tsukuba.ac.jp> <87li55styj.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: On 7/17/2013 12:15 AM, Stephen J. Turnbull wrote: > Terry Reedy writes: > > On 7/15/2013 10:20 PM, Guido van Rossum wrote: > > > > >> Or is this something deeper, that a group *is* a new object in > > >> principle? > > > > > > No, I just think of it as returning "a string" > > > > That is exactly what the doc says it does. See my other post. > > The problem is that IIUC '"a string"' is intentionally *not* referring > to the usual "str or bytes objects" (at least that's one of the > standard uses for scare quotes, to indicate an unusual usage). There are no 'scare quotes' in the doc. I put quote marks on things to indicated that I was quoting. I do not know how Guido regarded his marks. > Either > the docstring is using "string" in a similarly ambiguous way, or else > it's incorrect under the interpretation that buffer objects are *not* > "strings", so they should be inadmissible as targets. Saying that input arguments can be "Unicode strings as well as 8-bit strings' (the wording is from 2.x, carried over to 3.x) does not necessary exclude other inputs. CPython is somethimes more more permissive than the doc requires. If the doc said str, bytes, butearray, or memoryview, then other implementations would have to do the same to be conforming. I do not know if that is intended or not. The question is whether CPython should be just as permissive as to the output types of .group(). (And what, if any requirement should be imposed on other implementations.) > Something > should be fixed, and I suppose it should be the return type of group(). > > BTW, I suggest that Terry's usage of "string" (to mean "str or bytes" > in 3.x, "unicode or str" in 2.x) be adopted, and Guido's "stringish" This word is an adjective, not a noun. > be given expanded meaning, including buffer objects. Then we can say > informally that in searching and matching a target is a stringish, the > pattern is a stringish (?) or compiled re, but the group method > returns a string. Guido's idea to fix (tighten up) the output in 3.4 is fine with me. -- Terry Jan Reedy From stephen at xemacs.org Wed Jul 17 13:15:38 2013 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Wed, 17 Jul 2013 20:15:38 +0900 Subject: [Python-Dev] Misc re.match() complaint In-Reply-To: References: <51E48F8B.1090905@mrabarnett.plus.com> <87vc4bs1lm.fsf@uwakimon.sk.tsukuba.ac.jp> <87li55styj.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <87ehaxsahx.fsf@uwakimon.sk.tsukuba.ac.jp> Terry Reedy writes: > > "stringish" > > This word is an adjective, not a noun. Ah, a strict grammarian. That trumps any cards I could possibly play. From victor.stinner at gmail.com Wed Jul 17 13:16:27 2013 From: victor.stinner at gmail.com (Victor Stinner) Date: Wed, 17 Jul 2013 13:16:27 +0200 Subject: [Python-Dev] PEP 446: Add new parameters to configure the inherance of files and for non-blocking sockets In-Reply-To: References: <20130707003056.GA55329@cskk.homeip.net> Message-ID: 2013/7/7 Charles-Fran?ois Natali : > Well, it complicates the signature and implementation. > If we go the same way, why stop there and not expose O_DSYNC, O_SYNC, > O_DIRECT... I added this counter-argument to the PEP. > If you want precise control over the open() sementics, os.open() is > the way to go (that's also the rationale behind io.open() opener > argument, see http://bugs.python.org/issue12105) Right. Victor From victor.stinner at gmail.com Wed Jul 17 13:31:14 2013 From: victor.stinner at gmail.com (Victor Stinner) Date: Wed, 17 Jul 2013 13:31:14 +0200 Subject: [Python-Dev] PEP 446: Add new parameters to configure the inherance of files and for non-blocking sockets In-Reply-To: <20130707002912.GA49468@cskk.homeip.net> References: <20130707002912.GA49468@cskk.homeip.net> Message-ID: 2013/7/7 Cameron Simpson : > On 06Jul2013 14:43, Victor Stinner wrote: > | 2013/7/6 Cameron Simpson : > | > Yes. Please forget I mentioned fork(); it is only relevant if you > | > were offering some facility to undo the addition of cloexec to a > | > Popen passed file descriptor. Which you are not. > | > | Oh... gotcha. I now understood your concern. > | > | There is a little "trick" here: at fork, file descriptors are > | duplicated in the child process and almost all properties (open state, > | flags, offset, etc.) are shared. There is one little exception: file > | attributes are not shared, and there is only one file attribute: > | O_CLOEXEC. Setting O_CLOEXEC in a child process does not affect the > | flag in the parent process ;-) I will add a unit test to check this. > > That's news to me. Interesting. > > I can't find UNIX doco to support this, though I haven't finished > looking. None of the POSIX, Linux or MacOSX fork() manual pages > seem to describe this. > > Can you point me at doco for this? I thought file descriptors were > a direct index into a per-process table of file handle structures, > and did not think the latter got part copies, part shared over a > fork. Sorry, I found this fact my mistake, not in the doc :-) You can verify this behaviour using the follownig script: hg.python.org/peps/file/tip/pep-0466/test_cloexec.py Output on Linux: initial state: fd=3, cloexec=False child process after fork, set cloexec: cloexec=True child process after exec: fd=3, cloexec= parent process after fork: cloexec=False parent process after exec: fd=3, cloexec=False "parent process after fork: cloexec=False" means that the flag was not modified in the parent process. > Also, is O_CLOEXEC really the only file attribute? So O_NONBLOCK > is a flag and not an attribute (guessing the terminology distinguishes > these)? Now you mention it, ISTR noticing that there were distinct > ioctls to adjust flags and something else. See the manual page: http://linux.die.net/man/2/fcntl There are "File descriptor flags " and "File status flags", it looks like they are inherited/shared differently between the parent and the child process. > Your change says: > > On Windows, the close-on-exec flag is ``HANDLE_FLAG_INHERIT``. > > I feel it would be clearer to say: > > On Windows, the close-on-exec flag is the inverse of ``HANDLE_FLAG_INHERIT``. Ok, fixed in the PEP. Victor From victor.stinner at gmail.com Wed Jul 17 13:31:40 2013 From: victor.stinner at gmail.com (Victor Stinner) Date: Wed, 17 Jul 2013 13:31:40 +0200 Subject: [Python-Dev] PEP 446: Add new parameters to configure the inherance of files and for non-blocking sockets In-Reply-To: <20130713025702.GA23103@cskk.homeip.net> References: <20130713025702.GA23103@cskk.homeip.net> Message-ID: 2013/7/13 Cameron Simpson : > This sentence: > > The flag is cleared in the child process before executing the > program, the change does not change the flag in the parent process. > > needs a semicolon, not a comma. Fixed. Victor From steve at pearwood.info Wed Jul 17 13:50:42 2013 From: steve at pearwood.info (Steven D'Aprano) Date: Wed, 17 Jul 2013 21:50:42 +1000 Subject: [Python-Dev] String terminology [was Re: Misc re.match() complaint] In-Reply-To: References: <51E48F8B.1090905@mrabarnett.plus.com> <87vc4bs1lm.fsf@uwakimon.sk.tsukuba.ac.jp> <87li55styj.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <51E68512.1000506@pearwood.info> On 17/07/13 19:05, Terry Reedy wrote: > Saying that input arguments can be "Unicode strings as well as 8-bit strings' (the wording is from 2.x, carried over to 3.x) does not necessary exclude other inputs. "8-bit strings" seems somewhat ambiguous to me. In UTF-8, many Unicode strings are 8-bit, as they can be with Python 3.3's flexible string format. I prefer to stick to Unicode or text string, versus byte string. Pedants who point out that "byte" does not necessarily mean 8-bits, and therefore we should talk about octets, will be slapped with a large halibut :-) -- Steven From storchaka at gmail.com Wed Jul 17 15:12:34 2013 From: storchaka at gmail.com (Serhiy Storchaka) Date: Wed, 17 Jul 2013 16:12:34 +0300 Subject: [Python-Dev] Misc re.match() complaint In-Reply-To: References: <51E48F8B.1090905@mrabarnett.plus.com> <87vc4bs1lm.fsf@uwakimon.sk.tsukuba.ac.jp> <20130716095551.5aca0285@pitrou.net> Message-ID: 16.07.13 20:21, Guido van Rossum ???????(??): > The situation is most egregious if the target string is a bytearray, > where there is currently no way to get the result as an immutable > bytes object without an extra copy. (There's no API that lets you > create a bytes object directly from a slice of a bytearray.) m = memoryview(data) if m: return m.cast('B')[low:high].tobytes() else: # cast() doesn't work for empty memoryview return b'' From brett at python.org Wed Jul 17 15:35:59 2013 From: brett at python.org (Brett Cannon) Date: Wed, 17 Jul 2013 09:35:59 -0400 Subject: [Python-Dev] Tweaking PEP 8 guidelines for use of leading underscores In-Reply-To: <20130716210733.2ccd5192@limelight.wooz.org> References: <20130715184039.60590e62@anarchist> <1373934193.30382.59.camel@thinko> <51E4A0EF.8070205@pearwood.info> <1373941235.30382.62.camel@thinko> <1373943762.30382.88.camel@thinko> <20130716210733.2ccd5192@limelight.wooz.org> Message-ID: On Tue, Jul 16, 2013 at 9:07 PM, Barry Warsaw wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA256 > > On Jul 16, 2013, at 07:34 PM, Tres Seaver wrote: > > >On 07/16/2013 07:09 AM, Nick Coghlan wrote: > > > >> I did find it interesting that we *don't* explicitly advise against > >> the use of "import *" for anything other than optional accelerator > >> modules or republishing internal interfaces through a public API, > >> even though we advice against the practice in the tutorial. Perhaps > >> another oversight worth correcting? > > > >+1. 'from foo import *' (from any source other than another module in > >the same package) is a code stench far worse than anything else PEP8 > >proscribes. > > Maybe we should disable it for anything that isn't at the interactive > prompt > <0.5 wink>. > Or how about dropping the whole ``from ... import ...`` format entirely? <0.5 wink; seriously, it's implementation is a pain and it has wonky semantics> -------------- next part -------------- An HTML attachment was scrubbed... URL: From python at mrabarnett.plus.com Wed Jul 17 15:56:44 2013 From: python at mrabarnett.plus.com (MRAB) Date: Wed, 17 Jul 2013 14:56:44 +0100 Subject: [Python-Dev] Misc re.match() complaint In-Reply-To: <87li55styj.fsf@uwakimon.sk.tsukuba.ac.jp> References: <51E48F8B.1090905@mrabarnett.plus.com> <87vc4bs1lm.fsf@uwakimon.sk.tsukuba.ac.jp> <87li55styj.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <51E6A29C.1070001@mrabarnett.plus.com> On 17/07/2013 05:15, Stephen J. Turnbull wrote: > Terry Reedy writes: > > On 7/15/2013 10:20 PM, Guido van Rossum wrote: > > > > >> Or is this something deeper, that a group *is* a new object in > > >> principle? > > > > > > No, I just think of it as returning "a string" > > > > That is exactly what the doc says it does. See my other post. > > The problem is that IIUC '"a string"' is intentionally *not* referring > to the usual "str or bytes objects" (at least that's one of the > standard uses for scare quotes, to indicate an unusual usage). Either > the docstring is using "string" in a similarly ambiguous way, or else > it's incorrect under the interpretation that buffer objects are *not* > "strings", so they should be inadmissible as targets. Something > should be fixed, and I suppose it should be the return type of group(). > > BTW, I suggest that Terry's usage of "string" (to mean "str or bytes" > in 3.x, "unicode or str" in 2.x) be adopted, and Guido's "stringish" > be given expanded meaning, including buffer objects. Then we can say > informally that in searching and matching a target is a stringish, the > pattern is a stringish (?) or compiled re, but the group method > returns a string. > Instead of "stringish", how about "stringoid"? To me, "stringish" is an adjective, but "stringoid" can be a noun or an adjective. According to http://dictionary.reference.com: """ -oid ?suffix forming adjectives, ?suffix forming nouns indicating likeness, resemblance, or similarity: anthropoid """ From barry at python.org Wed Jul 17 15:56:09 2013 From: barry at python.org (Barry Warsaw) Date: Wed, 17 Jul 2013 09:56:09 -0400 Subject: [Python-Dev] Tweaking PEP 8 guidelines for use of leading underscores In-Reply-To: References: <20130715184039.60590e62@anarchist> <1373934193.30382.59.camel@thinko> <51E4A0EF.8070205@pearwood.info> <1373941235.30382.62.camel@thinko> <1373943762.30382.88.camel@thinko> <20130716210733.2ccd5192@limelight.wooz.org> Message-ID: <20130717095609.6824d790@anarchist> On Jul 17, 2013, at 09:35 AM, Brett Cannon wrote: >Or how about dropping the whole ``from ... import ...`` format entirely? ><0.5 wink; seriously, it's implementation is a pain and it has wonky >semantics> Yes, let's break *everything* :) -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From guido at python.org Wed Jul 17 16:55:13 2013 From: guido at python.org (Guido van Rossum) Date: Wed, 17 Jul 2013 07:55:13 -0700 Subject: [Python-Dev] String terminology [was Re: Misc re.match() complaint] In-Reply-To: <51E68512.1000506@pearwood.info> References: <51E48F8B.1090905@mrabarnett.plus.com> <87vc4bs1lm.fsf@uwakimon.sk.tsukuba.ac.jp> <87li55styj.fsf@uwakimon.sk.tsukuba.ac.jp> <51E68512.1000506@pearwood.info> Message-ID: When precision is needed I say things like 'a str object' or 'a bytes object'. There is no shame in a bit of verbosity around such issues, especially in the reference docs (tutorials are a different issue). On Wed, Jul 17, 2013 at 4:50 AM, Steven D'Aprano wrote: > On 17/07/13 19:05, Terry Reedy wrote: > >> Saying that input arguments can be "Unicode strings as well as 8-bit >> strings' (the wording is from 2.x, carried over to 3.x) does not necessary >> exclude other inputs. > > > "8-bit strings" seems somewhat ambiguous to me. In UTF-8, many Unicode > strings are 8-bit, as they can be with Python 3.3's flexible string format. > I prefer to stick to Unicode or text string, versus byte string. > > Pedants who point out that "byte" does not necessarily mean 8-bits, and > therefore we should talk about octets, will be slapped with a large halibut > :-) > > > -- > Steven > > > > > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/guido%40python.org -- --Guido van Rossum (python.org/~guido) From rdmurray at bitdance.com Wed Jul 17 18:20:34 2013 From: rdmurray at bitdance.com (R. David Murray) Date: Wed, 17 Jul 2013 12:20:34 -0400 Subject: [Python-Dev] Tweaking PEP 8 guidelines for use of leading underscores In-Reply-To: References: <20130715184039.60590e62@anarchist> <1373934193.30382.59.camel@thinko> <51E4A0EF.8070205@pearwood.info> <1373941235.30382.62.camel@thinko> <1373943762.30382.88.camel@thinko> <1373944606.30382.92.camel@thinko> <51E54859.4070901@pearwood.info> <20130716133912.C532325007E@webabinitio.net> Message-ID: <20130717162034.601F325014C@webabinitio.net> On Tue, 16 Jul 2013 17:46:34 -0400, Terry Reedy wrote: > On 7/16/2013 9:39 AM, R. David Murray wrote: > > On Tue, 16 Jul 2013 23:19:21 +1000, Steven D'Aprano wrote: > > >> For example, pkgutil includes classes with single-underscore methods, which I take as private. It also has a function simplegeneric, which is undocumented and not listed in __all__. In in the absence of even a comment saying "Don't use this", I take it as an oversight, not policy that simplegeneric is private. > > > > I think you'd be wrong about that, though. simplegeneric should really be > > treated as private. I'm speaking here not about the general principle of > > the thing, but about my understanding of simplegeneric's specific history. > > I think Steven (valid) point is "Why not, then, say it is internal > either in docs or name?"-- which in this case would be in the docs. I don't think that's what he was saying; but yes, we should do that :) --David From dholth at gmail.com Wed Jul 17 19:47:55 2013 From: dholth at gmail.com (Daniel Holth) Date: Wed, 17 Jul 2013 13:47:55 -0400 Subject: [Python-Dev] Tweaking PEP 8 guidelines for use of leading underscores In-Reply-To: <20130717162034.601F325014C@webabinitio.net> References: <20130715184039.60590e62@anarchist> <1373934193.30382.59.camel@thinko> <51E4A0EF.8070205@pearwood.info> <1373941235.30382.62.camel@thinko> <1373943762.30382.88.camel@thinko> <1373944606.30382.92.camel@thinko> <51E54859.4070901@pearwood.info> <20130716133912.C532325007E@webabinitio.net> <20130717162034.601F325014C@webabinitio.net> Message-ID: https://pypi.python.org/pypi/apipkg provides a much more effective way to denote API than an _ On Wed, Jul 17, 2013 at 12:20 PM, R. David Murray wrote: > On Tue, 16 Jul 2013 17:46:34 -0400, Terry Reedy wrote: >> On 7/16/2013 9:39 AM, R. David Murray wrote: >> > On Tue, 16 Jul 2013 23:19:21 +1000, Steven D'Aprano wrote: >> >> >> For example, pkgutil includes classes with single-underscore methods, which I take as private. It also has a function simplegeneric, which is undocumented and not listed in __all__. In in the absence of even a comment saying "Don't use this", I take it as an oversight, not policy that simplegeneric is private. >> > >> > I think you'd be wrong about that, though. simplegeneric should really be >> > treated as private. I'm speaking here not about the general principle of >> > the thing, but about my understanding of simplegeneric's specific history. >> >> I think Steven (valid) point is "Why not, then, say it is internal >> either in docs or name?"-- which in this case would be in the docs. > > I don't think that's what he was saying; but yes, we should do that :) > > --David > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/dholth%40gmail.com From ezio.melotti at gmail.com Thu Jul 18 15:15:44 2013 From: ezio.melotti at gmail.com (Ezio Melotti) Date: Thu, 18 Jul 2013 15:15:44 +0200 Subject: [Python-Dev] Misc re.match() complaint In-Reply-To: <87li55styj.fsf@uwakimon.sk.tsukuba.ac.jp> References: <51E48F8B.1090905@mrabarnett.plus.com> <87vc4bs1lm.fsf@uwakimon.sk.tsukuba.ac.jp> <87li55styj.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: Hi, On Wed, Jul 17, 2013 at 6:15 AM, Stephen J. Turnbull wrote: > > BTW, I suggest that Terry's usage of "string" (to mean "str or bytes" > in 3.x, "unicode or str" in 2.x) be adopted, and Guido's "stringish" > be given expanded meaning, including buffer objects. "string" means "str", "bytes" means "bytes", "bytes-like object" means "any object that supports the buffer protocol" [0] (including bytes). "string and bytes-like object" includes all of them. I don't think we need to introduce new terms. Best Regards, Ezio Melotti [0]: http://docs.python.org/3/glossary.html#term-bytes-like-object > Then we can say > informally that in searching and matching a target is a stringish, the > pattern is a stringish (?) or compiled re, but the group method > returns a string. > > Steve From guido at python.org Thu Jul 18 17:02:27 2013 From: guido at python.org (Guido van Rossum) Date: Thu, 18 Jul 2013 08:02:27 -0700 Subject: [Python-Dev] Misc re.match() complaint In-Reply-To: References: <51E48F8B.1090905@mrabarnett.plus.com> <87vc4bs1lm.fsf@uwakimon.sk.tsukuba.ac.jp> <87li55styj.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: On Thu, Jul 18, 2013 at 6:15 AM, Ezio Melotti wrote: > I don't think we need to introduce new terms. +1 -- --Guido van Rossum (python.org/~guido) From storchaka at gmail.com Thu Jul 18 19:38:32 2013 From: storchaka at gmail.com (Serhiy Storchaka) Date: Thu, 18 Jul 2013 20:38:32 +0300 Subject: [Python-Dev] Dash Message-ID: What type of dash is preferable in the documentation? The en dash (?) or the em dash (?)? From techtonik at gmail.com Thu Jul 18 19:43:07 2013 From: techtonik at gmail.com (anatoly techtonik) Date: Thu, 18 Jul 2013 20:43:07 +0300 Subject: [Python-Dev] Fwd: Issue 1229 in nativeclient: Get Python to work under Native Client In-Reply-To: <8-11067354921584041717-18302888370781815686-nativeclient=googlecode.com@googlecode.com> References: <0-11067354921584041717-18302888370781815686-nativeclient=googlecode.com@googlecode.com> <7-11067354921584041717-18302888370781815686-nativeclient=googlecode.com@googlecode.com> <8-11067354921584041717-18302888370781815686-nativeclient=googlecode.com@googlecode.com> Message-ID: Does Python build system support cross-compiling? NaCl projects seems to have problem with that. Just thought you might be interested to know about it. ---------- Forwarded message ---------- From: Date: Tue, Jul 16, 2013 at 9:24 PM Subject: Re: Issue 1229 in nativeclient: Get Python to work under Native Client To: techtonik at gmail.com Updates: Status: Fixed Comment #8 on issue 1229 by mseaborn at chromium.org: Get Python to work under Native Client http://code.google.com/p/**nativeclient/issues/detail?id=**1229 This is done in naclports. Sam Clegg added initial support for building against nacl-glibc in https://code.google.com/p/** naclports/source/detail?r=802 . Sam found a way to work around the Python build's lack of support for cross-compiling. The workaround is to build a native "pgen" first and inject it into the later cross-build. Building Python with nacl-newlib and PNaCl is a work in progress: https://codereview.chromium.**org/18112024/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From tjreedy at udel.edu Thu Jul 18 18:44:39 2013 From: tjreedy at udel.edu (Terry Reedy) Date: Thu, 18 Jul 2013 12:44:39 -0400 Subject: [Python-Dev] Misc re.match() complaint In-Reply-To: References: <51E48F8B.1090905@mrabarnett.plus.com> <87vc4bs1lm.fsf@uwakimon.sk.tsukuba.ac.jp> <87li55styj.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <51E81B77.6020209@udel.edu> On 7/18/2013 9:15 AM, Ezio Melotti wrote: In 3.x > "string" means "str", "bytes" means "bytes", "bytes-like object" means > "any object that supports the buffer protocol" [0] (including bytes). > "string and bytes-like object" includes all of them. > I don't think we need to introduce new terms. I agree. We just need to use them consistently, and update docs carried over without change from 2.x (like re doc), where 'string' meant 'unicode or str (bytes)' or even 'unicode and bytes-like'. From guido at python.org Thu Jul 18 19:48:37 2013 From: guido at python.org (Guido van Rossum) Date: Thu, 18 Jul 2013 10:48:37 -0700 Subject: [Python-Dev] Dash In-Reply-To: References: Message-ID: I believe there are only a few places where en-dashes should be used, for most things you should use either em-dash or hyphen. Consult your trusted typography source (for US English, please, punctuation preferences vary by locale). E.g. Google for "em dash en dash". On Thu, Jul 18, 2013 at 10:38 AM, Serhiy Storchaka wrote: > What type of dash is preferable in the documentation? The en dash (?) or the > em dash (?)? -- --Guido van Rossum (python.org/~guido) From storchaka at gmail.com Thu Jul 18 20:46:09 2013 From: storchaka at gmail.com (Serhiy Storchaka) Date: Thu, 18 Jul 2013 21:46:09 +0300 Subject: [Python-Dev] Dash In-Reply-To: References: Message-ID: 18.07.13 20:48, Guido van Rossum ???????(??): > I believe there are only a few places where en-dashes should be used, > for most things you should use either em-dash or hyphen. Consult your > trusted typography source (for US English, please, punctuation > preferences vary by locale). E.g. Google for "em dash en dash". Currently Python documentation in most cases uses en-dashes. Should we replace them to em-dashes? Should we remove spaces around dashes? Or we should replace a half-dozen of em-dashes found in Python documentation to en-dashes? I believe all hypens used in place of dash should be replaced to dash (but to en- or em- dash?) in any case. From brian at python.org Thu Jul 18 20:54:50 2013 From: brian at python.org (Brian Curtin) Date: Thu, 18 Jul 2013 13:54:50 -0500 Subject: [Python-Dev] Dash In-Reply-To: References: Message-ID: On Jul 18, 2013 1:46 PM, "Serhiy Storchaka" wrote: > > 18.07.13 20:48, Guido van Rossum ???????(??): > >> I believe there are only a few places where en-dashes should be used, >> for most things you should use either em-dash or hyphen. Consult your >> trusted typography source (for US English, please, punctuation >> preferences vary by locale). E.g. Google for "em dash en dash". > > > Currently Python documentation in most cases uses en-dashes. Should we replace them to em-dashes? Should we remove spaces around dashes? > > Or we should replace a half-dozen of em-dashes found in Python documentation to en-dashes? > > I believe all hypens used in place of dash should be replaced to dash (but to en- or em- dash?) in any case. Besides visual consistency in a couple of places, is there a reason to care enough to make a wholesale change? -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Thu Jul 18 20:55:29 2013 From: guido at python.org (Guido van Rossum) Date: Thu, 18 Jul 2013 11:55:29 -0700 Subject: [Python-Dev] Dash In-Reply-To: References: Message-ID: On Thu, Jul 18, 2013 at 11:46 AM, Serhiy Storchaka wrote: > 18.07.13 20:48, Guido van Rossum ???????(??): > >> I believe there are only a few places where en-dashes should be used, >> for most things you should use either em-dash or hyphen. Consult your >> trusted typography source (for US English, please, punctuation >> preferences vary by locale). E.g. Google for "em dash en dash". > > > Currently Python documentation in most cases uses en-dashes. Should we > replace them to em-dashes? Should we remove spaces around dashes? This may well be a relic from times when in typewritten text there were really only two types of dashes: a short one, typed as a hyphen, used for both hyphen and en-dash (and minus sign), and a long one, typed as two hyphens, used for em-dash. I suspect at some point the conversion to HTML or typeset text changed so that two hyphens mean en-dash and three mean em-dash, and few people noticed (and nobody really wants to type three hyphens except for a handful of typographical nuts). > Or we should replace a half-dozen of em-dashes found in Python documentation > to en-dashes? If my theory is right that makes sense. Especially if it's only a half-dozen. > I believe all hypens used in place of dash should be replaced to dash (but > to en- or em- dash?) in any case. But I wouldn't go change hundreds of hyphens -- chances are that you get bored reviewing the scripted automation you're using and break a few places. I'd say there are more important things to fix in the docs (like writing better overviews for many modules and classes). If we have a writing guide for the Python docs it would be useful to add something about dash types though. And if we don't have a writing guide, perhaps it makes sense to start one? -- --Guido van Rossum (python.org/~guido) From shibturn at gmail.com Thu Jul 18 21:16:01 2013 From: shibturn at gmail.com (Richard Oudkerk) Date: Thu, 18 Jul 2013 20:16:01 +0100 Subject: [Python-Dev] Dash In-Reply-To: References: Message-ID: On 18/07/2013 7:55pm, Guido van Rossum wrote: > This may well be a relic from times when in typewritten text there > were really only two types of dashes: a short one, typed as a hyphen, > used for both hyphen and en-dash (and minus sign), and a long one, > typed as two hyphens, used for em-dash. > > I suspect at some point the conversion to HTML or typeset text changed > so that two hyphens mean en-dash and three mean em-dash, and few > people noticed (and nobody really wants to type three hyphens except > for a handful of typographical nuts). Weren't the docs originally done LaTeX? In LaTeX "--" is also an en-dash and "---" is an em-dash. -- Richard From guido at python.org Thu Jul 18 21:50:27 2013 From: guido at python.org (Guido van Rossum) Date: Thu, 18 Jul 2013 12:50:27 -0700 Subject: [Python-Dev] Dash In-Reply-To: References: Message-ID: On Thu, Jul 18, 2013 at 12:16 PM, Richard Oudkerk wrote: > On 18/07/2013 7:55pm, Guido van Rossum wrote: >> >> This may well be a relic from times when in typewritten text there >> were really only two types of dashes: a short one, typed as a hyphen, >> used for both hyphen and en-dash (and minus sign), and a long one, >> typed as two hyphens, used for em-dash. >> >> I suspect at some point the conversion to HTML or typeset text changed >> so that two hyphens mean en-dash and three mean em-dash, and few >> people noticed (and nobody really wants to type three hyphens except >> for a handful of typographical nuts). > > > Weren't the docs originally done LaTeX? In LaTeX "--" is also an en-dash > and "---" is an em-dash. Right -- but I've been typing two hyphens to mean an em-dash all my life. :-) -- --Guido van Rossum (python.org/~guido) From ezio.melotti at gmail.com Thu Jul 18 23:49:18 2013 From: ezio.melotti at gmail.com (Ezio Melotti) Date: Thu, 18 Jul 2013 23:49:18 +0200 Subject: [Python-Dev] Dash In-Reply-To: References: Message-ID: Hi, On Thu, Jul 18, 2013 at 7:38 PM, Serhiy Storchaka wrote: > What type of dash is preferable in the documentation? The en dash (?) or the > em dash (?)? > Both should be used where appropriate [0]. The em dash is more common, however "--" (commonly used to indicate the em dash in e.g. emails) gets converted to an en dash by Sphinx [1]. I noticed this a while ago, and I started using "---" in the documentation whenever I wanted an em dash. If this is not documented it should be added to the "documenting" page of the devguide, so that people start using the right ones and convert the wrong ones when they come across them. Best Regards, Ezio Melotti [0]: http://en.wikipedia.org/wiki/Dash [1]: https://bitbucket.org/birkenfeld/sphinx/src/default/sphinx/util/smartypants.py#cl-261 From steve at pearwood.info Fri Jul 19 06:51:39 2013 From: steve at pearwood.info (Steven D'Aprano) Date: Fri, 19 Jul 2013 14:51:39 +1000 Subject: [Python-Dev] Dash In-Reply-To: References: Message-ID: <51E8C5DB.9010701@pearwood.info> On 19/07/13 04:46, Serhiy Storchaka wrote: > 18.07.13 20:48, Guido van Rossum ???????(??): >> I believe there are only a few places where en-dashes should be used, >> for most things you should use either em-dash or hyphen. Consult your >> trusted typography source (for US English, please, punctuation >> preferences vary by locale). E.g. Google for "em dash en dash". > > Currently Python documentation in most cases uses en-dashes. Should we replace them to em-dashes? Should we remove spaces around dashes? > > Or we should replace a half-dozen of em-dashes found in Python documentation to en-dashes? > > I believe all hypens used in place of dash should be replaced to dash (but to en- or em- dash?) in any case. It depends on the context, and I don't believe you could completely automate the process (at least not without using something that understands natural language, like NLTK, and probably not even then). I think it will require a human reader to review them, like any other style and grammar edit. Wikipedia has a good overview which mostly agrees with my typesetting and style books: https://en.wikipedia.org/wiki/Dash Hyphens are commonly used for compound words, although in practice hyphenated words gradually lose the hyphen. E.g. we usually write "inbox" rather than "in-box", although we still write "in-tray". Hyphens are also used at the end of the line to break a word to the next line. En-dashes are used for durations and ranges (sometimes with a thin space on either side, otherwise a regular space can be used). E.g. "October?December". En-dash is also used when making a compound word from words which themselves are compound words, e.g. "The pre?World War II economy" joins "pre-" with "World War II", not just "World". Em-dashes are used for parenthetical asides, or to indicate a break in speech or thought. They are often used when a comma is too weak and a period is too strong?a bit like a colon. Different sources give different recommendations regarding spaces around dashes. The Chicago Manual of Style agrees with most British sources that em-dashes should never have spaces around them, but the New York Times style guide sets hair-spaces around them. Unusually for me, I tend to agree with the NY Times on this one. A regular space is usually too wide. Many of these conventions are style conventions, rather than strictly grammatical. For example, although English grammar says we can use an en-dash to make ranges of numbers, the SI standard recommends against expressions like "10?50 volts" since it can be mistaken for subtraction, and recommends "10 to 50 volts". Optimistically, I think it would probably be safe[1] to replace " -- " or " --- " in text with "\N{THIN SPACE}\N{EM DASH}\N{THIN SPACE}" (or \N{HAIR SPACE} if you prefer) without human review, but for any other changes, I wouldn't even try to automate it. [1] Famous last words. -- Steven From ben+python at benfinney.id.au Fri Jul 19 09:25:39 2013 From: ben+python at benfinney.id.au (Ben Finney) Date: Fri, 19 Jul 2013 17:25:39 +1000 Subject: [Python-Dev] Dash References: Message-ID: <7wmwpjqado.fsf@benfinney.id.au> Serhiy Storchaka writes: > What type of dash is preferable in the documentation? The en dash (?) > or the em dash (?)? They have different purposes; use whichever is appropriate for the context. This isn't an official Python developer position. But all of ?-?, ???, ???, ???, etc. have distinct and common meanings in English text. So it's a mistaken idea to think of ?which dash is preferable? since they are *all* preferable for their distinct uses. -- \ ?Science doesn't work by vote and it doesn't work by | `\ authority.? ?Richard Dawkins, _Big Mistake_ (The Guardian, | _o__) 2006-12-27) | Ben Finney From ncoghlan at gmail.com Fri Jul 19 10:38:05 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 19 Jul 2013 18:38:05 +1000 Subject: [Python-Dev] Dash In-Reply-To: <7wmwpjqado.fsf@benfinney.id.au> References: <7wmwpjqado.fsf@benfinney.id.au> Message-ID: On 19 July 2013 17:25, Ben Finney wrote: > Serhiy Storchaka writes: > >> What type of dash is preferable in the documentation? The en dash (-) >> or the em dash (--)? > > They have different purposes; use whichever is appropriate for the > context. > > This isn't an official Python developer position. But all of '-', '-', > '--', '-', etc. have distinct and common meanings in English text. So > it's a mistaken idea to think of "which dash is preferable" since they > are *all* preferable for their distinct uses. I don't know about "common". I had no idea there were 3 dash types until a couple of weeks ago. I thought there were only two (short/hyphen/minus-sign and long/em-dash). I still don't really know what the difference is between the two long ones (en-dash vs em-dash). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From solipsis at pitrou.net Fri Jul 19 10:46:20 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 19 Jul 2013 10:46:20 +0200 Subject: [Python-Dev] Dash References: <7wmwpjqado.fsf@benfinney.id.au> Message-ID: <20130719104620.3c5ce104@pitrou.net> Le Fri, 19 Jul 2013 18:38:05 +1000, Nick Coghlan a ?crit : > On 19 July 2013 17:25, Ben Finney wrote: > > Serhiy Storchaka writes: > > > >> What type of dash is preferable in the documentation? The en dash > >> (-) or the em dash (--)? > > > > They have different purposes; use whichever is appropriate for the > > context. > > > > This isn't an official Python developer position. But all of '-', > > '-', '--', '-', etc. have distinct and common meanings in English > > text. So it's a mistaken idea to think of "which dash is > > preferable" since they are *all* preferable for their distinct uses. > > I don't know about "common". I had no idea there were 3 dash types > until a couple of weeks ago. I thought there were only two > (short/hyphen/minus-sign and long/em-dash). I still don't really know > what the difference is between the two long ones (en-dash vs em-dash). But there aren't 3 dash types. There are at least 5 of them! https://en.wikipedia.org/wiki/Dash#Common_dashes (not to mention that Twisted has/had its own dash, IIRC) cheers Antoine. From steve at pearwood.info Fri Jul 19 11:53:10 2013 From: steve at pearwood.info (Steven D'Aprano) Date: Fri, 19 Jul 2013 19:53:10 +1000 Subject: [Python-Dev] Dash In-Reply-To: References: <7wmwpjqado.fsf@benfinney.id.au> Message-ID: <51E90C86.90205@pearwood.info> On 19/07/13 18:38, Nick Coghlan wrote: > I don't know about "common". I had no idea there were 3 dash types *Way* more than three. hyphen minus sign (not the same as a hyphen!) en-dash em-dash two-em-dash three-em-dash figure dash horizontal bar swung dash plus another half-dozen or so non-English hyphens, all of which have different typographic (and sometimes grammatical) purposes. Wikipedia has a good description: https://en.wikipedia.org/wiki/Dash but a brief summary is: ? hyphen - is the smallest and is used for compound words like "out-tray"; ? minus sign ? is normally raised compared to the others; ? en-dash ? should be the width of "n" and is used for ranges, like Mon?Fri; ? em-dash ? should be the width of "m" and is used for a break stronger than a comma but weaker than a full stop; ? horizontal bar ? should be longer than the em-dash and is used for quotations in some European languages; ? swung dash ? is normally found in dictionaries (paper, not Python) to stand in for the word being defined. But of course different fonts follow these rules to a greater or lesser degree. -- Steven From storchaka at gmail.com Fri Jul 19 11:55:01 2013 From: storchaka at gmail.com (Serhiy Storchaka) Date: Fri, 19 Jul 2013 12:55:01 +0300 Subject: [Python-Dev] Dash In-Reply-To: References: Message-ID: 18.07.13 21:54, Brian Curtin ???????(??): > Besides visual consistency in a couple of places, is there a reason to > care enough to make a wholesale change? Single hyphen instead of a dash just looks too ugly to me. Trying to fix this I noticed that the documentation is inconsistent regarding the dash. Sometimes en and em dashes meet in the same line and in the same sentence. I.e.: """ The :mod:`weakref` module also allows creating proxy objects which behave like weak references --- an object referenced only by proxy objects is deallocated -- but instead of requiring an explicit call to retrieve the object, the proxy """ From storchaka at gmail.com Fri Jul 19 12:03:59 2013 From: storchaka at gmail.com (Serhiy Storchaka) Date: Fri, 19 Jul 2013 13:03:59 +0300 Subject: [Python-Dev] Dash In-Reply-To: References: Message-ID: 18.07.13 21:55, Guido van Rossum ???????(??): > On Thu, Jul 18, 2013 at 11:46 AM, Serhiy Storchaka wrote: >> Or we should replace a half-dozen of em-dashes found in Python documentation >> to en-dashes? > > If my theory is right that makes sense. Especially if it's only a half-dozen. I'm sorry, I were wrong. Actually em-dash used much more then en-dash in Python documentation. From storchaka at gmail.com Fri Jul 19 12:16:45 2013 From: storchaka at gmail.com (Serhiy Storchaka) Date: Fri, 19 Jul 2013 13:16:45 +0300 Subject: [Python-Dev] Dash In-Reply-To: References: Message-ID: 19.07.13 00:49, Ezio Melotti ???????(??): > On Thu, Jul 18, 2013 at 7:38 PM, Serhiy Storchaka wrote: >> What type of dash is preferable in the documentation? The en dash (?) or the >> em dash (?)? >> > > Both should be used where appropriate [0]. Of course I looked in Wikipedia before asking on this list. And I were surprised that en-dash allowed in same places as em-dashes: """ Either version may be used to denote a break in a sentence or to set off parenthetical statements, although writers are generally cautioned to use a single form consistently within their work. In this function, en dashes are used with spaces and em dashes are used without them:[1] [Em dash:] In matters of grave importance, style?not sincerity?is the vital thing. [En dash:] In matters of grave importance, style ? not sincerity ? is the vital thing. """ I'm asking only about this case, when the dash is used to denote a break in a sentence or to set off parenthetical statements. Currently Python documentation uses fourth variants: 1. A single hyphen "-" surrounded with spaces. "In matters of grave importance, style - not sincerity - is the vital thing." I think this case should be converted to one of followed, but to what of them? 2. A double hyphen "--" (converted to en-dash) surrounded with spaces. "In matters of grave importance, style -- not sincerity -- is the vital thing." 3. A triple hyphen "---" (converted to em-dash) surrounded with spaces. "In matters of grave importance, style --- not sincerity --- is the vital thing." 4. A triple hyphen "---" (converted to em-dash) which isn't surrounded with spaces. "In matters of grave importance, style---not sincerity---is the vital thing." > The em dash is more > common, however "--" (commonly used to indicate the em dash in e.g. > emails) gets converted to an en dash by Sphinx [1]. I noticed this a > while ago, and I started using "---" in the documentation whenever I > wanted an em dash. This is common in TeX. From storchaka at gmail.com Fri Jul 19 12:20:30 2013 From: storchaka at gmail.com (Serhiy Storchaka) Date: Fri, 19 Jul 2013 13:20:30 +0300 Subject: [Python-Dev] Dash In-Reply-To: <7wmwpjqado.fsf@benfinney.id.au> References: <7wmwpjqado.fsf@benfinney.id.au> Message-ID: 19.07.13 10:25, Ben Finney ???????(??): > Serhiy Storchaka writes: >> What type of dash is preferable in the documentation? The en dash (?) >> or the em dash (?)? > > They have different purposes; use whichever is appropriate for the > context. I mean only a context where a dash is used to denote a break in a sentence or to set off parenthetical statements. Currently Python documentation uses four different variants (see my answer to Ezio). From storchaka at gmail.com Fri Jul 19 12:32:34 2013 From: storchaka at gmail.com (Serhiy Storchaka) Date: Fri, 19 Jul 2013 13:32:34 +0300 Subject: [Python-Dev] Dash In-Reply-To: <51E8C5DB.9010701@pearwood.info> References: <51E8C5DB.9010701@pearwood.info> Message-ID: 19.07.13 07:51, Steven D'Aprano ???????(??): > Optimistically, I think it would probably be safe[1] to replace " -- " > or " --- " in text with "\N{THIN SPACE}\N{EM DASH}\N{THIN SPACE}" (or > \N{HAIR SPACE} if you prefer) without human review, but for any other > changes, I wouldn't even try to automate it. No, it is not safe. " -- " used in code examples in which it doesn't converted to en-dash. Also we should correct tables and underlines in titles. I have a manually written and multiple times checked patch (modifies about 640 lines) which safely replaces " -- " with " --- " but I doubt what actually should be a preferable dash. From status at bugs.python.org Fri Jul 19 18:07:37 2013 From: status at bugs.python.org (Python tracker) Date: Fri, 19 Jul 2013 18:07:37 +0200 (CEST) Subject: [Python-Dev] Summary of Python tracker Issues Message-ID: <20130719160737.0EE1A56A99@psf.upfronthosting.co.za> ACTIVITY SUMMARY (2013-07-12 - 2013-07-19) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 4092 (+25) closed 26185 (+45) total 30277 (+70) Open issues with patches: 1846 Issues opened (46) ================== #18436: Add mapping of symbol to function to operator module http://bugs.python.org/issue18436 opened by zach.ware #18438: Obsolete url in comment inside decimal module http://bugs.python.org/issue18438 opened by vajrasky #18439: Patchcheck for ACKS, NEWS does not work on Windows. http://bugs.python.org/issue18439 opened by terry.reedy #18441: Idle: Make test.support.requires('gui') skip when it should. http://bugs.python.org/issue18441 opened by terry.reedy #18443: Misc/Readme still documents TextMate http://bugs.python.org/issue18443 opened by iwontbecreative #18444: IDLE: Mac OS X pressing ctrl-A in Python shell moved cursor be http://bugs.python.org/issue18444 opened by Todd.Rovito #18445: Tools/Script/Readme is outdated http://bugs.python.org/issue18445 opened by iwontbecreative #18451: Omit test files in devinabox coverage run http://bugs.python.org/issue18451 opened by brett.cannon #18453: There are unused variables inside DateTimeTestCase class in te http://bugs.python.org/issue18453 opened by vajrasky #18454: distutils crashes when uploading to PyPI having only the usern http://bugs.python.org/issue18454 opened by jgehrcke #18456: Documentation for PyDict_Update is incorrect http://bugs.python.org/issue18456 opened by ncoghlan #18458: libedit history offset workaround http://bugs.python.org/issue18458 opened by ronaldoussoren #18459: readline: libedit support on non-apple platforms http://bugs.python.org/issue18459 opened by ronaldoussoren #18460: .chm documentation files advertised in download.html but not t http://bugs.python.org/issue18460 opened by iwontbecreative #18461: X Error in tkinter http://bugs.python.org/issue18461 opened by serhiy.storchaka #18465: There are unused variables and unused import in Lib/test/test_ http://bugs.python.org/issue18465 opened by vajrasky #18466: Spelling mistakes in various code comments. http://bugs.python.org/issue18466 opened by iwontbecreative #18467: argparse - problematic 'default' behavior http://bugs.python.org/issue18467 opened by matej.tyc #18468: re.group() should never return a bytearray http://bugs.python.org/issue18468 opened by gvanrossum #18472: PEP 8 updates - internal interfaces and import * http://bugs.python.org/issue18472 opened by ncoghlan #18473: some objects pickled by Python 3.x are not unpicklable in Pyth http://bugs.python.org/issue18473 opened by kmike #18476: No way to pass custom arguments for loggers and formatters. http://bugs.python.org/issue18476 opened by piotr.dobrogost #18477: fix_import (2to3 fixer) is not case-sensitive http://bugs.python.org/issue18477 opened by Valentin.Lorentz #18478: Class bodies: when does a name become local? http://bugs.python.org/issue18478 opened by terry.reedy #18481: lcov report http://bugs.python.org/issue18481 opened by christian.heimes #18483: Add hour-24 type of time to test_http2time_formats in test_htt http://bugs.python.org/issue18483 opened by vajrasky #18484: No unit test for iso2time function from http.cookiejar module http://bugs.python.org/issue18484 opened by vajrasky #18485: mingw: configure for shared build http://bugs.python.org/issue18485 opened by rpetrov #18486: mingw: dynamic loading support http://bugs.python.org/issue18486 opened by rpetrov #18487: mingw implement exec prefix http://bugs.python.org/issue18487 opened by rpetrov #18488: sqlite: finalize() method of user function may be called with http://bugs.python.org/issue18488 opened by haypo #18489: IDLE Unit test for SearchEngine.py http://bugs.python.org/issue18489 opened by philwebster #18491: Add "exe wrapper" functionality to Windows launcher http://bugs.python.org/issue18491 opened by pmoore #18492: Add test.support.regrtest_run flag, simplify support.requires http://bugs.python.org/issue18492 opened by zach.ware #18493: make profile-opt fails with pre-existing python2.7 in path http://bugs.python.org/issue18493 opened by Claudio.Freire #18494: PyType_GenericSet/GetDict functions misnamed in docs? http://bugs.python.org/issue18494 opened by Wolf.Ihlenfeldt #18495: mingw: ignore main program for frozen scripts http://bugs.python.org/issue18495 opened by rpetrov #18496: mingw: setup exclude termios module http://bugs.python.org/issue18496 opened by rpetrov #18497: mingw: setup _multiprocessing module http://bugs.python.org/issue18497 opened by rpetrov #18498: mingw: setup select module http://bugs.python.org/issue18498 opened by rpetrov #18499: mingw: setup _ctypes module with system libffi http://bugs.python.org/issue18499 opened by rpetrov #18500: mingw: defect winsock2 and setup _socket module http://bugs.python.org/issue18500 opened by rpetrov #18501: _elementtree.c calls Python callbacks while a Python exception http://bugs.python.org/issue18501 opened by haypo #18502: CDLL does not use same paths as util.find_library http://bugs.python.org/issue18502 opened by shjohnson.pi #18503: No assertion in test_del_param_on_nonexistent_header function http://bugs.python.org/issue18503 opened by vajrasky #18504: IDLE:Improvements- Improving Mock_Text http://bugs.python.org/issue18504 opened by JayKrish Most recent 15 issues with no replies (15) ========================================== #18504: IDLE:Improvements- Improving Mock_Text http://bugs.python.org/issue18504 #18502: CDLL does not use same paths as util.find_library http://bugs.python.org/issue18502 #18500: mingw: defect winsock2 and setup _socket module http://bugs.python.org/issue18500 #18499: mingw: setup _ctypes module with system libffi http://bugs.python.org/issue18499 #18498: mingw: setup select module http://bugs.python.org/issue18498 #18497: mingw: setup _multiprocessing module http://bugs.python.org/issue18497 #18496: mingw: setup exclude termios module http://bugs.python.org/issue18496 #18495: mingw: ignore main program for frozen scripts http://bugs.python.org/issue18495 #18494: PyType_GenericSet/GetDict functions misnamed in docs? http://bugs.python.org/issue18494 #18493: make profile-opt fails with pre-existing python2.7 in path http://bugs.python.org/issue18493 #18489: IDLE Unit test for SearchEngine.py http://bugs.python.org/issue18489 #18487: mingw implement exec prefix http://bugs.python.org/issue18487 #18486: mingw: dynamic loading support http://bugs.python.org/issue18486 #18485: mingw: configure for shared build http://bugs.python.org/issue18485 #18484: No unit test for iso2time function from http.cookiejar module http://bugs.python.org/issue18484 Most recent 15 issues waiting for review (15) ============================================= #18500: mingw: defect winsock2 and setup _socket module http://bugs.python.org/issue18500 #18499: mingw: setup _ctypes module with system libffi http://bugs.python.org/issue18499 #18498: mingw: setup select module http://bugs.python.org/issue18498 #18497: mingw: setup _multiprocessing module http://bugs.python.org/issue18497 #18496: mingw: setup exclude termios module http://bugs.python.org/issue18496 #18495: mingw: ignore main program for frozen scripts http://bugs.python.org/issue18495 #18492: Add test.support.regrtest_run flag, simplify support.requires http://bugs.python.org/issue18492 #18491: Add "exe wrapper" functionality to Windows launcher http://bugs.python.org/issue18491 #18489: IDLE Unit test for SearchEngine.py http://bugs.python.org/issue18489 #18487: mingw implement exec prefix http://bugs.python.org/issue18487 #18486: mingw: dynamic loading support http://bugs.python.org/issue18486 #18485: mingw: configure for shared build http://bugs.python.org/issue18485 #18481: lcov report http://bugs.python.org/issue18481 #18477: fix_import (2to3 fixer) is not case-sensitive http://bugs.python.org/issue18477 #18472: PEP 8 updates - internal interfaces and import * http://bugs.python.org/issue18472 Top 10 most discussed issues (10) ================================= #18408: Fixes crashes found by pyfailmalloc http://bugs.python.org/issue18408 34 msgs #18373: implement sys.get/setbyteswarningflag() http://bugs.python.org/issue18373 9 msgs #18491: Add "exe wrapper" functionality to Windows launcher http://bugs.python.org/issue18491 9 msgs #11874: argparse assertion failure with brackets in metavars http://bugs.python.org/issue11874 8 msgs #17214: http.client.HTTPConnection.putrequest encode error http://bugs.python.org/issue17214 7 msgs #17933: test_ftp failure / ftplib error formatting issue http://bugs.python.org/issue17933 7 msgs #18258: Fix test discovery for test_codecmaps*.py http://bugs.python.org/issue18258 7 msgs #18444: IDLE: Mac OS X pressing ctrl-A in Python shell moved cursor be http://bugs.python.org/issue18444 7 msgs #18305: [patch] Fast sum() for non-numbers http://bugs.python.org/issue18305 6 msgs #18387: Add 'symbols' link to pydoc's html menu bar. http://bugs.python.org/issue18387 6 msgs Issues closed (42) ================== #17482: functools.update_wrapper mishandles __wrapped__ http://bugs.python.org/issue17482 closed by python-dev #17767: Fix test discovery for test_locale.py http://bugs.python.org/issue17767 closed by serhiy.storchaka #17778: Fix test discovery for test_multiprocessing.py http://bugs.python.org/issue17778 closed by sbt #18021: Update broken link to Apple Publication Style Guide http://bugs.python.org/issue18021 closed by ezio.melotti #18042: Provide enum.unique class decorator http://bugs.python.org/issue18042 closed by python-dev #18266: Fix test discovery for test_largefile.py http://bugs.python.org/issue18266 closed by serhiy.storchaka #18279: IDLE Unit test for RstripExtension.py http://bugs.python.org/issue18279 closed by terry.reedy #18336: codecs: Link to readline module (history) instead of fd.readli http://bugs.python.org/issue18336 closed by serhiy.storchaka #18344: _bufferedreader_read_all() may leak reference to data http://bugs.python.org/issue18344 closed by sbt #18365: Idle: mock Text class and test thereof http://bugs.python.org/issue18365 closed by terry.reedy #18366: Look into using setuptools 0.8 with devinabox http://bugs.python.org/issue18366 closed by brett.cannon #18393: Gestalt() is deprecated on OSX 10.8, remove support? http://bugs.python.org/issue18393 closed by ronaldoussoren #18428: IDLE startup error http://bugs.python.org/issue18428 closed by ned.deily #18429: IDLE: Format Paragraph doesn't function with comment blocks http://bugs.python.org/issue18429 closed by Todd.Rovito #18431: Encoded words in structured headers are not decoded by the new http://bugs.python.org/issue18431 closed by r.david.murray #18432: sched modules queue property should return a list, not an iter http://bugs.python.org/issue18432 closed by rhettinger #18433: Clarify venv docs about which methods check for pre-existing p http://bugs.python.org/issue18433 closed by python-dev #18434: Update venv example to use setuptools http://bugs.python.org/issue18434 closed by python-dev #18435: Move venv from a custom Context class to types.SimpleNamespace http://bugs.python.org/issue18435 closed by python-dev #18437: Typo in comment inside class Charset about output_charset http://bugs.python.org/issue18437 closed by r.david.murray #18440: hash() and __hash__() do not document their size constraints http://bugs.python.org/issue18440 closed by barry #18442: Typo in developers.rst line 27 http://bugs.python.org/issue18442 closed by ned.deily #18446: Dead link in documenting.rst http://bugs.python.org/issue18446 closed by ezio.melotti #18447: Pydoc crashes on large contents http://bugs.python.org/issue18447 closed by ned.deily #18448: Tools/demo/eiffel.py fails to run tests. http://bugs.python.org/issue18448 closed by serhiy.storchaka #18450: Typos in developers.rst http://bugs.python.org/issue18450 closed by python-dev #18452: Typo in Doc/howto/argparse.rst http://bugs.python.org/issue18452 closed by python-dev #18455: Multiprocessing connection SocketClient retries connection on http://bugs.python.org/issue18455 closed by sbt #18457: Fix and cleanup the ss1.py demo http://bugs.python.org/issue18457 closed by serhiy.storchaka #18462: Download > Other documentation page contains a link to a frenc http://bugs.python.org/issue18462 closed by ned.deily #18464: Typo in the name of function in test__encoded_words.py http://bugs.python.org/issue18464 closed by python-dev #18469: PyStructSequence does not handle exceptions correctly http://bugs.python.org/issue18469 closed by python-dev #18470: tokenizer.c does not handle new_string() failure http://bugs.python.org/issue18470 closed by python-dev #18471: Typo in heapq documentation http://bugs.python.org/issue18471 closed by python-dev #18474: Lambda assigned to object does not automatically get self http://bugs.python.org/issue18474 closed by amaury.forgeotdarc #18475: No unittest.main in Lib/test/test_email/test_inversion.py http://bugs.python.org/issue18475 closed by python-dev #18479: Improvements to Powershell activate script for venv http://bugs.python.org/issue18479 closed by python-dev #18480: _elementtree: missing PyType_Ready call http://bugs.python.org/issue18480 closed by ronaldoussoren #18482: dis.dis fails on one letter strings. http://bugs.python.org/issue18482 closed by r.david.murray #18490: Error embedding datetime in C++ http://bugs.python.org/issue18490 closed by christian.heimes #18449: Demo 'ss1.py' crash in python3 http://bugs.python.org/issue18449 closed by serhiy.storchaka #18463: Download > Other documentation page outdated/dead links http://bugs.python.org/issue18463 closed by ned.deily From ben+python at benfinney.id.au Fri Jul 19 21:26:30 2013 From: ben+python at benfinney.id.au (Ben Finney) Date: Sat, 20 Jul 2013 05:26:30 +1000 Subject: [Python-Dev] Dash References: <7wmwpjqado.fsf@benfinney.id.au> Message-ID: <7wwqom4ahl.fsf@benfinney.id.au> Serhiy Storchaka writes: > 19.07.13 10:25, Ben Finney ???????(??): > > Serhiy Storchaka writes: > >> What type of dash is preferable in the documentation? The en dash > >> (?) or the em dash (?)? > > > > They have different purposes; use whichever is appropriate for the > > context. > > I mean only a context where a dash is used to denote a break in a > sentence or to set off parenthetical statements. Yes, and both of those are used for different purposes, so ?what type is preferable (for all the purposes you mention)? is a poor way of looking at it, IMO. Local style guides have rules about when and how to use the different dashes for specific contexts. (For myself, I prefer parenthetic dashes to be en dashes ???, and a break in a sentence to be marked by an em dash ???. But I also prefer to surround the dash with space for the latter purpose, so my preference isn't universal.) Wikipedia's article (already referred) discusses the matter fairly well . > Currently Python documentation uses four different variants (see my > answer to Ezio). I'd recommend that the Python documentation BDFL should choose and dictate an existing well-groomed style guide for Python documentation, preferably one which explicitly talks about when to use each of the commonly-used dashes. -- \ ?If you have the facts on your side, pound the facts. If you | `\ have the law on your side, pound the law. If you have neither | _o__) on your side, pound the table.? ?anonymous | Ben Finney From ben+python at benfinney.id.au Fri Jul 19 21:29:13 2013 From: ben+python at benfinney.id.au (Ben Finney) Date: Sat, 20 Jul 2013 05:29:13 +1000 Subject: [Python-Dev] Dash References: Message-ID: <7wsiza4ad2.fsf@benfinney.id.au> Serhiy Storchaka writes: > I'm sorry, I were wrong. Actually em-dash used much more then en-dash > in Python documentation. But, as you've also discovered, many authors will type two hyphens ?--? when they want an em dash ???, yet this is rendered to an en dash ???. So the prevalence of the latter in the documentation probably does not reflect the preferences of the authors. -- \ ?Nature hath given men one tongue but two ears, that we may | `\ hear from others twice as much as we speak.? ?Epictetus, | _o__) _Fragments_ | Ben Finney From ben+python at benfinney.id.au Fri Jul 19 21:32:38 2013 From: ben+python at benfinney.id.au (Ben Finney) Date: Sat, 20 Jul 2013 05:32:38 +1000 Subject: [Python-Dev] Dash References: Message-ID: <7wob9y4a7d.fsf@benfinney.id.au> Serhiy Storchaka writes: > I'm asking only about this case, when the dash is used to denote a > break in a sentence or to set off parenthetical statements. That's two separate cases: * denote a break in a sentence * set off parenthetical statements In my experience, an em dash is commonly used for the former, and en dashes commonly used for the latter. Using the same dash for both is unusual (and IMO needlessly ambiguous to the reader). > Currently Python documentation uses fourth variants: All your examples are only for parenthetical statements. Can you find examples of the former, where a break (not parenthetical) in the sentence is intended? -- \ ?When I was a kid I used to pray every night for a new bicycle. | `\ Then I realised that the Lord doesn't work that way so I stole | _o__) one and asked Him to forgive me.? ?Emo Philips | Ben Finney From ethan at stoneleaf.us Sat Jul 20 01:47:36 2013 From: ethan at stoneleaf.us (Ethan Furman) Date: Fri, 19 Jul 2013 16:47:36 -0700 Subject: [Python-Dev] dict __contains__ raises TypeError on unhashable input Message-ID: <51E9D018.4030806@stoneleaf.us> While working on issue #18508 I stumbled across this: Traceback (most recent call last): ... File "/usr/local/lib/python3.4/enum.py", line 417, in __new__ if value in cls._value2member_map: TypeError: unhashable type: 'list' I'll wrap it in a try-except block, but I must admit I was surprised the answer wasn't False. After all, if the input is unhashable then obviously it's not in the dict; furthermore, if I were to compare the number 5 with a set() I would get False, not a TypeMismatch error, and dict keys are basically done by equality, the hash is just (?) a speed-up. -- ~Ethan~ From ethan at stoneleaf.us Sat Jul 20 03:30:23 2013 From: ethan at stoneleaf.us (Ethan Furman) Date: Fri, 19 Jul 2013 18:30:23 -0700 Subject: [Python-Dev] dict __contains__ raises TypeError on unhashable input In-Reply-To: <51E9D018.4030806@stoneleaf.us> References: <51E9D018.4030806@stoneleaf.us> Message-ID: <51E9E82F.2060804@stoneleaf.us> http://bugs.python.org/issue18510 Commenting further: some_key in dict is conceptually the same as some_key in dict.keys() which /would/ return False for an unhashable key -- at least it did in 2.x; for 3.x you have to say some_key in list(dict.keys()) which seems like a step backwards. Is it worth changing __contains__ and keys() to be in line with equality? -- ~Ethan~ From benhoyt at gmail.com Sat Jul 20 05:11:44 2013 From: benhoyt at gmail.com (Ben Hoyt) Date: Sat, 20 Jul 2013 15:11:44 +1200 Subject: [Python-Dev] dict __contains__ raises TypeError on unhashable input In-Reply-To: <51E9E82F.2060804@stoneleaf.us> References: <51E9D018.4030806@stoneleaf.us> <51E9E82F.2060804@stoneleaf.us> Message-ID: "unhashable key" doesn't make sense to me. If it's a "key", it has to be hashable, and I'd think it'd often be a sign of a bug if you were trying to do "unhashable_value in mapping", so to me it seems good that it raises a TypeError. It totally depends on what's on the RHS of the in -- for instance, you get a TypeError if you do: 42 in 'xyz' Whereas by your reasoning, it should return False, because the integer 42 can never be "in" that string. But really you want the TypeError, because that's a bug waiting to happen. This is something of an aside, but I like the fact that Python is (relatively) strongly typed, and I think this is an example of that. Just the other day I was surprised to learn that in both Java and C# you can do: "foo" + 3 + 4 And get "foo34" without any compile-time or run-time error (in a statically typed language like Java or C# I would have expected a compile-time error). A friend had written code equivalent to that, except with the 3 and 4 being integer variables. He was expecting to see "foo7" because that was his intent, but then realized what was going on -- numbers are automatically converted to strings in this context, and the + operators are applied left-to-right. Whereas if you try that in Python, bomp -- TypeError: cannot concatenate 'str' and 'int' objects! -Ben On Sat, Jul 20, 2013 at 1:30 PM, Ethan Furman wrote: > http://bugs.python.org/**issue18510 > > Commenting further: > > some_key in dict > > is conceptually the same as > > some_key in dict.keys() > > which /would/ return False for an unhashable key -- at least it did in > 2.x; for 3.x you have to say > > some_key in list(dict.keys()) > > which seems like a step backwards. > > Is it worth changing __contains__ and keys() to be in line with equality? > > > -- > ~Ethan~ > ______________________________**_________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/**mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/**mailman/options/python-dev/** > benhoyt%40gmail.com > -------------- next part -------------- An HTML attachment was scrubbed... URL: From benjamin at python.org Sat Jul 20 09:13:47 2013 From: benjamin at python.org (Benjamin Peterson) Date: Sat, 20 Jul 2013 00:13:47 -0700 Subject: [Python-Dev] dict __contains__ raises TypeError on unhashable input In-Reply-To: <51E9D018.4030806@stoneleaf.us> References: <51E9D018.4030806@stoneleaf.us> Message-ID: 2013/7/19 Ethan Furman : > While working on issue #18508 I stumbled across this: > > Traceback (most recent call last): > ... > File "/usr/local/lib/python3.4/enum.py", line 417, in __new__ > if value in cls._value2member_map: > TypeError: unhashable type: 'list' > > I'll wrap it in a try-except block, but I must admit I was surprised the > answer wasn't False. After all, if the input is unhashable then obviously > it's not in the dict; furthermore, if I were to compare the number 5 with a > set() I would get False, not a TypeMismatch error, and dict keys are > basically done by equality, the hash is just (?) a speed-up. I'm not exactly sure what the last part of that sentence means. Anyway, it's well established that operations on a key in a dict are going to involve looking up the key, and thus hashing it. You wouldn't expect, {}.get(unhashable, None) not to raise, right? -- Regards, Benjamin From ncoghlan at gmail.com Sat Jul 20 09:39:29 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 20 Jul 2013 17:39:29 +1000 Subject: [Python-Dev] dict __contains__ raises TypeError on unhashable input In-Reply-To: <51E9D018.4030806@stoneleaf.us> References: <51E9D018.4030806@stoneleaf.us> Message-ID: On 20 July 2013 09:47, Ethan Furman wrote: > While working on issue #18508 I stumbled across this: > > Traceback (most recent call last): > ... > File "/usr/local/lib/python3.4/enum.py", line 417, in __new__ > if value in cls._value2member_map: > TypeError: unhashable type: 'list' > > I'll wrap it in a try-except block, but I must admit I was surprised the > answer wasn't False. After all, if the input is unhashable then obviously > it's not in the dict; furthermore, if I were to compare the number 5 with a > set() I would get False, not a TypeMismatch error, and dict keys are > basically done by equality, the hash is just (?) a speed-up. "in" is controlled entirely by the container. Both set() and dict() consider passing them an unhashable object to be a categorical type error: >>> [] in set() Traceback (most recent call last): File "", line 1, in TypeError: unhashable type: 'list' >>> [] in {} Traceback (most recent call last): File "", line 1, in TypeError: unhashable type: 'list' Equality is different, because people do equality comparisons between unlike types all the time, so we try *very* hard to ensure "I do not understand this type" is reported as False rather than "I don't know" (as indicated by an exception). It's basically the only place where we do that - pretty much everywhere else, we err on the side of "I do not understand what this even means" translating to an exception (Python 3 took another big step in that direction by making sure ordered comparisons between unlike types threw TypeError by default, instead of returning a dubious answer). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From solipsis at pitrou.net Sat Jul 20 12:21:41 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sat, 20 Jul 2013 12:21:41 +0200 Subject: [Python-Dev] dict __contains__ raises TypeError on unhashable input References: <51E9D018.4030806@stoneleaf.us> Message-ID: <20130720122141.148c8df8@fsol> On Fri, 19 Jul 2013 16:47:36 -0700 Ethan Furman wrote: > While working on issue #18508 I stumbled across this: > > Traceback (most recent call last): > ... > File "/usr/local/lib/python3.4/enum.py", line 417, in __new__ > if value in cls._value2member_map: > TypeError: unhashable type: 'list' > > I'll wrap it in a try-except block, but I must admit I was surprised the answer wasn't False. After all, if the input > is unhashable then obviously it's not in the dict; furthermore, if I were to compare the number 5 with a set() I would > get False, not a TypeMismatch error, and dict keys are basically done by equality, the hash is just (?) a speed-up. Testing containment of an unhashable key in a dict is a programming error, so I'd prefer that we keep the TypeError here. Same reasoning as for: >>> 5 in "abc" Traceback (most recent call last): File "", line 1, in TypeError: 'in ' requires string as left operand, not int Sure, 5 technically isn't in "abc" and therefore the "in" operator could return False, but it's better to highlight the programming error. Regards Antoine. From ronaldoussoren at mac.com Sat Jul 20 12:21:53 2013 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Sat, 20 Jul 2013 12:21:53 +0200 Subject: [Python-Dev] dict __contains__ raises TypeError on unhashable input In-Reply-To: <51E9D018.4030806@stoneleaf.us> References: <51E9D018.4030806@stoneleaf.us> Message-ID: <94771240-354C-4240-BFCF-BDFCE6FE2E9C@mac.com> On 20 Jul, 2013, at 1:47, Ethan Furman wrote: > While working on issue #18508 I stumbled across this: > > Traceback (most recent call last): > ... > File "/usr/local/lib/python3.4/enum.py", line 417, in __new__ > if value in cls._value2member_map: > TypeError: unhashable type: 'list' > > I'll wrap it in a try-except block, but I must admit I was surprised the answer wasn't False. After all, if the input is unhashable then obviously it's not in the dict; furthermore, if I were to compare the number 5 with a set() I would get False, not a TypeMismatch error, and dict keys are basically done by equality, the hash is just (?) a speed-up. Not quite, there are some objects that compare equal without both of them being hashable: >>> frozenset([1,2]) == set([1,2]) True >>> dct = { frozenset([1,2]): 1 } >>> frozenset([1,2]) in dct True >>> set([1,2]) in dct Traceback (most recent call last): File "", line 1, in TypeError: unhashable type: 'set' It would be strange if the last test would return False instead of raising an error. Ronald > > -- > ~Ethan~ > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/ronaldoussoren%40mac.com From storchaka at gmail.com Sat Jul 20 14:18:02 2013 From: storchaka at gmail.com (Serhiy Storchaka) Date: Sat, 20 Jul 2013 15:18:02 +0300 Subject: [Python-Dev] Dash In-Reply-To: <7wob9y4a7d.fsf@benfinney.id.au> References: <7wob9y4a7d.fsf@benfinney.id.au> Message-ID: 19.07.13 22:32, Ben Finney ???????(??): > Serhiy Storchaka writes: >> I'm asking only about this case, when the dash is used to denote a >> break in a sentence or to set off parenthetical statements. > > That's two separate cases: > > * denote a break in a sentence > * set off parenthetical statements In Wikipedia they considered as one case. > In my experience, an em dash is commonly used for the former, and en > dashes commonly used for the latter. Using the same dash for both is > unusual (and IMO needlessly ambiguous to the reader). This is definitely should be described in the official guide. I never heard about this. >> Currently Python documentation uses fourth variants: > > All your examples are only for parenthetical statements. Can you find > examples of the former, where a break (not parenthetical) in the > sentence is intended? (2) and (4) are from Wikipedia (which consider it as one case). Here are real random examples from Python documentation: 1. "Common installation tools such as ``Distribute`` and ``pip`` work as expected with venvs - i.e. when a venv is active, they install Python packages into the venv without needing to be told to do so explicitly." 2. "Enter *string* in the table of "interned" strings and return the interned string which is *string* itself or a copy." 3. "The :class:`set` type is mutable --- the contents can be changed using methods like :meth:`add` and :meth:`remove`." 4. "Three locking mechanisms are used---dot locking and, if available, the :c:func:`flock` and :c:func:`lockf` system calls." From storchaka at gmail.com Sat Jul 20 14:23:46 2013 From: storchaka at gmail.com (Serhiy Storchaka) Date: Sat, 20 Jul 2013 15:23:46 +0300 Subject: [Python-Dev] cpython: Use strncat() instead of strcat() to silence some warnings. In-Reply-To: <3by7Hm4W8dz7LjM@mail.python.org> References: <3by7Hm4W8dz7LjM@mail.python.org> Message-ID: 20.07.13 15:12, christian.heimes ???????(??): > http://hg.python.org/cpython/rev/c92f4172d122 > changeset: 84723:c92f4172d122 > user: Christian Heimes > date: Sat Jul 20 14:11:28 2013 +0200 > summary: > Use strncat() instead of strcat() to silence some warnings. > CID 486616, CID 486617, CID 486615 > > files: > Modules/ossaudiodev.c | 6 +++--- > 1 files changed, 3 insertions(+), 3 deletions(-) > > > diff --git a/Modules/ossaudiodev.c b/Modules/ossaudiodev.c > --- a/Modules/ossaudiodev.c > +++ b/Modules/ossaudiodev.c > @@ -245,7 +245,7 @@ > int arg; > > assert(strlen(fname) <= 30); > - strcat(argfmt, fname); > + strncat(argfmt, fname, 30); > if (!PyArg_ParseTuple(args, argfmt, &arg)) > return NULL; > > @@ -270,7 +270,7 @@ > int arg = 0; > > assert(strlen(fname) <= 30); > - strcat(argfmt, fname); > + strncat(argfmt, fname, 30); > if (!PyArg_ParseTuple(args, argfmt, &arg)) > return NULL; > > @@ -290,7 +290,7 @@ > int rv; > > assert(strlen(fname) <= 30); > - strcat(argfmt, fname); > + strncat(argfmt, fname, 30); > if (!PyArg_ParseTuple(args, argfmt)) > return NULL; This will wrong when strlen(fname) is 30. strncat() will copy only 30 bytes, without terminal NUL. From solipsis at pitrou.net Sat Jul 20 14:36:04 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sat, 20 Jul 2013 14:36:04 +0200 Subject: [Python-Dev] cpython: Use strncat() instead of strcat() to silence some warnings. References: <3by7Hm4W8dz7LjM@mail.python.org> Message-ID: <20130720143604.596881f0@fsol> On Sat, 20 Jul 2013 15:23:46 +0300 Serhiy Storchaka wrote: > 20.07.13 15:12, christian.heimes ???????(??): > > http://hg.python.org/cpython/rev/c92f4172d122 > > changeset: 84723:c92f4172d122 > > user: Christian Heimes > > date: Sat Jul 20 14:11:28 2013 +0200 > > summary: > > Use strncat() instead of strcat() to silence some warnings. > > CID 486616, CID 486617, CID 486615 > > [...] > > This will wrong when strlen(fname) is 30. strncat() will copy only 30 > bytes, without terminal NUL. So, for the record, this is roughly how Rasmus Lerdorf introduced a security hole in PHP 5.3.7: "For people asking me out-of-band what the screw-up was, it was pretty simple. I changed this code: memcpy(passwd, MD5_MAGIC, MD5_MAGIC_LEN); strlcpy(passwd + MD5_MAGIC_LEN, sp, sl + 1); strcat(passwd, "$"); to: memcpy(passwd, MD5_MAGIC, MD5_MAGIC_LEN); strlcpy(passwd + MD5_MAGIC_LEN, sp, sl + 1); strlcat(passwd, "$", 1); **because the Coverity static analyzer warned about using strcat** [emphasis mine] and we generally try to avoid naked strcat/strcpy in the codebase even though in this case it is safe to do." https://plus.google.com/113641248237520845183/posts/g68d9RvRA1i Regards Antoine. From storchaka at gmail.com Sat Jul 20 14:48:09 2013 From: storchaka at gmail.com (Serhiy Storchaka) Date: Sat, 20 Jul 2013 15:48:09 +0300 Subject: [Python-Dev] cpython: Use strncat() instead of strcat() to silence some warnings. In-Reply-To: <20130720143604.596881f0@fsol> References: <3by7Hm4W8dz7LjM@mail.python.org> <20130720143604.596881f0@fsol> Message-ID: 20.07.13 15:36, Antoine Pitrou ???????(??): > On Sat, 20 Jul 2013 15:23:46 +0300 > Serhiy Storchaka wrote: >> 20.07.13 15:12, christian.heimes ???????(??): >>> http://hg.python.org/cpython/rev/c92f4172d122 >>> changeset: 84723:c92f4172d122 >>> user: Christian Heimes >>> date: Sat Jul 20 14:11:28 2013 +0200 >>> summary: >>> Use strncat() instead of strcat() to silence some warnings. >>> CID 486616, CID 486617, CID 486615 >>> > [...] >> >> This will wrong when strlen(fname) is 30. strncat() will copy only 30 >> bytes, without terminal NUL. > > So, for the record, this is roughly how Rasmus Lerdorf introduced a > security hole in PHP 5.3.7: > > > "For people asking me out-of-band what the screw-up was, it was pretty > simple. I changed this code: > > memcpy(passwd, MD5_MAGIC, MD5_MAGIC_LEN); > strlcpy(passwd + MD5_MAGIC_LEN, sp, sl + 1); > strcat(passwd, "$"); > > to: > > memcpy(passwd, MD5_MAGIC, MD5_MAGIC_LEN); > strlcpy(passwd + MD5_MAGIC_LEN, sp, sl + 1); > strlcat(passwd, "$", 1); > > **because the Coverity static analyzer warned about using strcat** > [emphasis mine] and we generally try to avoid naked strcat/strcpy in the > codebase even though in this case it is safe to do." > > https://plus.google.com/113641248237520845183/posts/g68d9RvRA1i strlcat != strncat. strlcat(dst, src, 1) actually do nothing. From solipsis at pitrou.net Sat Jul 20 15:00:49 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sat, 20 Jul 2013 15:00:49 +0200 Subject: [Python-Dev] cpython: Use strncat() instead of strcat() to silence some warnings. References: <3by7Hm4W8dz7LjM@mail.python.org> <20130720143604.596881f0@fsol> Message-ID: <20130720150049.5403ad1f@fsol> On Sat, 20 Jul 2013 15:48:09 +0300 Serhiy Storchaka wrote: > 20.07.13 15:36, Antoine Pitrou ???????(??): > > On Sat, 20 Jul 2013 15:23:46 +0300 > > Serhiy Storchaka wrote: > >> 20.07.13 15:12, christian.heimes ???????(??): > >>> http://hg.python.org/cpython/rev/c92f4172d122 > >>> changeset: 84723:c92f4172d122 > >>> user: Christian Heimes > >>> date: Sat Jul 20 14:11:28 2013 +0200 > >>> summary: > >>> Use strncat() instead of strcat() to silence some warnings. > >>> CID 486616, CID 486617, CID 486615 > >>> > > [...] > >> > >> This will wrong when strlen(fname) is 30. strncat() will copy only 30 > >> bytes, without terminal NUL. > > > > So, for the record, this is roughly how Rasmus Lerdorf introduced a > > security hole in PHP 5.3.7: > > > > > > "For people asking me out-of-band what the screw-up was, it was pretty > > simple. I changed this code: > > > > memcpy(passwd, MD5_MAGIC, MD5_MAGIC_LEN); > > strlcpy(passwd + MD5_MAGIC_LEN, sp, sl + 1); > > strcat(passwd, "$"); > > > > to: > > > > memcpy(passwd, MD5_MAGIC, MD5_MAGIC_LEN); > > strlcpy(passwd + MD5_MAGIC_LEN, sp, sl + 1); > > strlcat(passwd, "$", 1); > > > > **because the Coverity static analyzer warned about using strcat** > > [emphasis mine] and we generally try to avoid naked strcat/strcpy in the > > codebase even though in this case it is safe to do." > > > > https://plus.google.com/113641248237520845183/posts/g68d9RvRA1i > > strlcat != strncat. strlcat(dst, src, 1) actually do nothing. This is true. But the trigger in the "fix" was the same (trying to suppress a Coverity warning about strcat). Regards Antoine. From christian at python.org Sat Jul 20 15:27:01 2013 From: christian at python.org (Christian Heimes) Date: Sat, 20 Jul 2013 15:27:01 +0200 Subject: [Python-Dev] cpython: Use strncat() instead of strcat() to silence some warnings. In-Reply-To: References: <3by7Hm4W8dz7LjM@mail.python.org> Message-ID: > This will wrong when strlen(fname) is 30. strncat() will copy only 30 > bytes, without terminal NUL. That's not how strncat() works. strncat(dest, src, n) writes n+1 chars to the end of dest: n chars from src and +1 for the final NUL char. For this reason dest must be large enough to hold strlen(dest) + n + 1 chars. From christian at python.org Sat Jul 20 15:29:45 2013 From: christian at python.org (Christian Heimes) Date: Sat, 20 Jul 2013 15:29:45 +0200 Subject: [Python-Dev] cpython: Use strncat() instead of strcat() to silence some warnings. In-Reply-To: References: <3by7Hm4W8dz7LjM@mail.python.org> Message-ID: Am 20.07.2013 14:23, schrieb Serhiy Storchaka: > This will wrong when strlen(fname) is 30. strncat() will copy only 30 > bytes, without terminal NUL. http://linux.die.net/man/3/strncat The strncat() function is similar, except that * it will use at most n bytes from src; and * src does not need to be null-terminated if it contains n or more bytes. If src contains n or more bytes, strncat() writes n+1 bytes to dest (n from src plus the terminating null byte). Therefore, the size of dest must be at least strlen(dest)+n+1. From storchaka at gmail.com Sat Jul 20 15:56:25 2013 From: storchaka at gmail.com (Serhiy Storchaka) Date: Sat, 20 Jul 2013 16:56:25 +0300 Subject: [Python-Dev] cpython: Use strncat() instead of strcat() to silence some warnings. In-Reply-To: References: <3by7Hm4W8dz7LjM@mail.python.org> Message-ID: 20.07.13 16:27, Christian Heimes ???????(??): >> This will wrong when strlen(fname) is 30. strncat() will copy only 30 >> bytes, without terminal NUL. > > That's not how strncat() works. strncat(dest, src, n) writes n+1 chars > to the end of dest: n chars from src and +1 for the final NUL char. For > this reason dest must be large enough to hold strlen(dest) + n + 1 chars. Oh, true. strncat() always results NUL-terminated string. It's strncpy() can produce not NUL-terminated string. Sorry for noise. From christian at python.org Sat Jul 20 16:04:08 2013 From: christian at python.org (Christian Heimes) Date: Sat, 20 Jul 2013 16:04:08 +0200 Subject: [Python-Dev] cpython: Use strncat() instead of strcat() to silence some warnings. In-Reply-To: References: <3by7Hm4W8dz7LjM@mail.python.org> Message-ID: Am 20.07.2013 15:56, schrieb Serhiy Storchaka: > Oh, true. strncat() always results NUL-terminated string. It's strncpy() > can produce not NUL-terminated string. Sorry for noise. You're welcome! Better safe than sorry. :) From ethan at stoneleaf.us Sun Jul 21 03:37:57 2013 From: ethan at stoneleaf.us (Ethan Furman) Date: Sat, 20 Jul 2013 18:37:57 -0700 Subject: [Python-Dev] dict __contains__ raises TypeError on unhashable input In-Reply-To: <94771240-354C-4240-BFCF-BDFCE6FE2E9C@mac.com> References: <51E9D018.4030806@stoneleaf.us> <94771240-354C-4240-BFCF-BDFCE6FE2E9C@mac.com> Message-ID: <51EB3B75.1070701@stoneleaf.us> On 07/20/2013 03:21 AM, Ronald Oussoren wrote: > > On 20 Jul, 2013, at 1:47, Ethan Furman wrote: > >> While working on issue #18508 I stumbled across this: >> >> Traceback (most recent call last): >> ... >> File "/usr/local/lib/python3.4/enum.py", line 417, in __new__ >> if value in cls._value2member_map: >> TypeError: unhashable type: 'list' >> >> I'll wrap it in a try-except block, but I must admit I was surprised the answer wasn't False. After all, if the input is unhashable then obviously it's not in the dict; furthermore, if I were to compare the number 5 with a set() I would get False, not a TypeMismatch error, and dict keys are basically done by equality, the hash is just (?) a speed-up. > > Not quite, there are some objects that compare equal without both of them being hashable: > >>>> frozenset([1,2]) == set([1,2]) > True >>>> dct = { frozenset([1,2]): 1 } >>>> frozenset([1,2]) in dct > True >>>> set([1,2]) in dct > Traceback (most recent call last): > File "", line 1, in > TypeError: unhashable type: 'set' > > It would be strange if the last test would return False instead of raising an error. Yeah, that would be unfortunate. Okay, I can see dict.__contains__ and friends raising a TypeError, but what about dict.keys() ? In 2.x it returned a list, in 3.x it returns a dict_keys, and dict_keys also raises an error instead of returning False. -- ~Ethan~ From guido at python.org Sun Jul 21 04:37:02 2013 From: guido at python.org (Guido van Rossum) Date: Sat, 20 Jul 2013 19:37:02 -0700 Subject: [Python-Dev] dict __contains__ raises TypeError on unhashable input In-Reply-To: <51EB3B75.1070701@stoneleaf.us> References: <51E9D018.4030806@stoneleaf.us> <94771240-354C-4240-BFCF-BDFCE6FE2E9C@mac.com> <51EB3B75.1070701@stoneleaf.us> Message-ID: On Sat, Jul 20, 2013 at 6:37 PM, Ethan Furman wrote: > On 07/20/2013 03:21 AM, Ronald Oussoren wrote: >> >> >> On 20 Jul, 2013, at 1:47, Ethan Furman wrote: >> >>> While working on issue #18508 I stumbled across this: >>> >>> Traceback (most recent call last): >>> ... >>> File "/usr/local/lib/python3.4/enum.py", line 417, in __new__ >>> if value in cls._value2member_map: >>> TypeError: unhashable type: 'list' >>> >>> I'll wrap it in a try-except block, but I must admit I was surprised the >>> answer wasn't False. After all, if the input is unhashable then obviously >>> it's not in the dict; furthermore, if I were to compare the number 5 with a >>> set() I would get False, not a TypeMismatch error, and dict keys are >>> basically done by equality, the hash is just (?) a speed-up. >> >> >> Not quite, there are some objects that compare equal without both of them >> being hashable: >> >>>>> frozenset([1,2]) == set([1,2]) >> >> True >>>>> >>>>> dct = { frozenset([1,2]): 1 } >>>>> frozenset([1,2]) in dct >> >> True >>>>> >>>>> set([1,2]) in dct >> >> Traceback (most recent call last): >> File "", line 1, in >> TypeError: unhashable type: 'set' >> >> It would be strange if the last test would return False instead of raising >> an error. > > > Yeah, that would be unfortunate. > > Okay, I can see dict.__contains__ and friends raising a TypeError, but what > about dict.keys() ? In 2.x it returned a list, in 3.x it returns a > dict_keys, and dict_keys also raises an error instead of returning False. The same reasoning applies. And the Python 2 and 3 keys() methods are so different that the backwards compatibility argument doesn't really have any weight. -- --Guido van Rossum (python.org/~guido) From raymond.hettinger at gmail.com Mon Jul 22 01:36:35 2013 From: raymond.hettinger at gmail.com (Raymond Hettinger) Date: Sun, 21 Jul 2013 16:36:35 -0700 Subject: [Python-Dev] Building a Faster Python Message-ID: <252C50D8-C23D-438D-BAE1-B22E0D65A21A@gmail.com> Our current Mac OS X builds use GCC-4.2. On Python2.7, I ran a comparison of gcc-4.2.1 builds versus gcc-4.8.1 and found that the latter makes a much faster Python. PyBench2.0 shows the total running time dropping from 5653ms to 4571ms. The code is uniformly better in just about every category. Raymond ------------------------------------------------------------------------------- PYBENCH 2.0 ------------------------------------------------------------------------------- * using CPython 2.7.5+ (2.7:f8df7c50132f, Jul 21 2013, 15:57:46) [GCC 4.2.1 (Based on Apple Inc. build 5658) (LLVM build 2336.11.00)] * disabled garbage collection * system check interval set to maximum: 2147483647 * using timer: time.time Calibrating tests. Please wait... done. Running 10 round(s) of the suite at warp factor 10: * Round 1 done in 5.775 seconds. * Round 2 done in 5.794 seconds. * Round 3 done in 5.743 seconds. * Round 4 done in 5.736 seconds. * Round 5 done in 5.758 seconds. * Round 6 done in 5.787 seconds. * Round 7 done in 5.779 seconds. * Round 8 done in 5.781 seconds. * Round 9 done in 5.782 seconds. * Round 10 done in 5.811 seconds. ------------------------------------------------------------------------------- Benchmark: 2013-07-21 16:02:39 ------------------------------------------------------------------------------- Rounds: 10 Warp: 10 Timer: time.time Machine Details: Platform ID: Darwin-12.4.0-x86_64-i386-64bit Processor: i386 Python: Implementation: CPython Executable: /Users/raymondhettinger/py27/python.exe Version: 2.7.5+ Compiler: GCC 4.2.1 (Based on Apple Inc. build 5658) (LLVM build 2336.11.00) Bits: 64bit Build: Jul 21 2013 15:57:46 (#2.7:f8df7c50132f) Unicode: UCS2 Test minimum average operation overhead ------------------------------------------------------------------------------- BuiltinFunctionCalls: 112ms 114ms 0.22us 0.178ms BuiltinMethodLookup: 76ms 83ms 0.08us 0.207ms CompareFloats: 78ms 80ms 0.07us 0.237ms CompareFloatsIntegers: 73ms 74ms 0.08us 0.178ms CompareIntegers: 86ms 87ms 0.05us 0.354ms CompareInternedStrings: 85ms 86ms 0.06us 0.895ms CompareLongs: 72ms 72ms 0.07us 0.207ms CompareStrings: 66ms 66ms 0.07us 0.603ms CompareUnicode: 75ms 76ms 0.10us 0.455ms ComplexPythonFunctionCalls: 96ms 98ms 0.49us 0.298ms ConcatStrings: 166ms 179ms 0.36us 0.346ms ConcatUnicode: 80ms 82ms 0.27us 0.231ms CreateInstances: 111ms 115ms 1.03us 0.242ms CreateNewInstances: 84ms 87ms 1.04us 0.250ms CreateStringsWithConcat: 95ms 97ms 0.10us 0.602ms CreateUnicodeWithConcat: 70ms 72ms 0.18us 0.236ms DictCreation: 77ms 77ms 0.19us 0.237ms DictWithFloatKeys: 81ms 82ms 0.09us 0.442ms DictWithIntegerKeys: 84ms 85ms 0.07us 0.591ms DictWithStringKeys: 78ms 79ms 0.07us 0.590ms ForLoops: 69ms 70ms 2.81us 0.042ms IfThenElse: 73ms 74ms 0.06us 0.445ms ListSlicing: 69ms 70ms 4.98us 0.079ms NestedForLoops: 83ms 84ms 0.06us 0.016ms NestedListComprehensions: 103ms 105ms 8.78us 0.061ms NormalClassAttribute: 85ms 86ms 0.07us 0.298ms NormalInstanceAttribute: 81ms 81ms 0.07us 0.299ms PythonFunctionCalls: 94ms 95ms 0.29us 0.179ms PythonMethodCalls: 109ms 111ms 0.49us 0.093ms Recursion: 134ms 137ms 2.74us 0.297ms SecondImport: 110ms 111ms 1.11us 0.119ms SecondPackageImport: 113ms 115ms 1.15us 0.159ms SecondSubmoduleImport: 143ms 146ms 1.46us 0.119ms SimpleComplexArithmetic: 70ms 70ms 0.08us 0.237ms SimpleDictManipulation: 84ms 86ms 0.07us 0.297ms SimpleFloatArithmetic: 75ms 75ms 0.06us 0.355ms SimpleIntFloatArithmetic: 67ms 67ms 0.05us 0.355ms SimpleIntegerArithmetic: 68ms 68ms 0.05us 0.354ms SimpleListComprehensions: 93ms 94ms 7.82us 0.061ms SimpleListManipulation: 69ms 71ms 0.06us 0.386ms SimpleLongArithmetic: 81ms 82ms 0.12us 0.178ms SmallLists: 94ms 98ms 0.14us 0.237ms SmallTuples: 84ms 86ms 0.16us 0.267ms SpecialClassAttribute: 84ms 85ms 0.07us 0.299ms SpecialInstanceAttribute: 100ms 102ms 0.08us 0.299ms StringMappings: 440ms 446ms 1.77us 0.304ms StringPredicates: 150ms 156ms 0.22us 1.453ms StringSlicing: 108ms 115ms 0.21us 0.500ms TryExcept: 70ms 70ms 0.03us 0.444ms TryFinally: 102ms 103ms 0.64us 0.238ms TryRaiseExcept: 79ms 81ms 1.27us 0.256ms TupleSlicing: 95ms 98ms 0.37us 0.051ms UnicodeMappings: 75ms 76ms 2.10us 0.411ms UnicodePredicates: 77ms 80ms 0.15us 1.729ms UnicodeProperties: 122ms 123ms 0.31us 1.449ms UnicodeSlicing: 100ms 105ms 0.21us 0.442ms WithFinally: 102ms 105ms 0.65us 0.238ms WithRaiseExcept: 155ms 157ms 1.96us 0.297ms ------------------------------------------------------------------------------- Totals: 5653ms 5774ms ------------------------------------------------------------------------------- PYBENCH 2.0 ------------------------------------------------------------------------------- * using CPython 2.7.5+ (2.7:f8df7c50132f, Jul 21 2013, 16:11:35) [GCC 4.8.1] * disabled garbage collection * system check interval set to maximum: 2147483647 * using timer: time.time Calibrating tests. Please wait... done. Running 10 round(s) of the suite at warp factor 10: * Round 1 done in 4.654 seconds. * Round 2 done in 4.681 seconds. * Round 3 done in 4.660 seconds. * Round 4 done in 4.694 seconds. * Round 5 done in 4.674 seconds. * Round 6 done in 4.673 seconds. * Round 7 done in 4.672 seconds. * Round 8 done in 4.679 seconds. * Round 9 done in 4.692 seconds. * Round 10 done in 4.680 seconds. ------------------------------------------------------------------------------- Benchmark: 2013-07-21 16:15:45 ------------------------------------------------------------------------------- Rounds: 10 Warp: 10 Timer: time.time Machine Details: Platform ID: Darwin-12.4.0-x86_64-i386-64bit Processor: i386 Python: Implementation: CPython Executable: /Users/raymondhettinger/py27/python.exe Version: 2.7.5+ Compiler: GCC 4.8.1 Bits: 64bit Build: Jul 21 2013 16:11:35 (#2.7:f8df7c50132f) Unicode: UCS2 Test minimum average operation overhead ------------------------------------------------------------------------------- BuiltinFunctionCalls: 90ms 92ms 0.18us 0.185ms BuiltinMethodLookup: 53ms 55ms 0.05us 0.146ms CompareFloats: 60ms 60ms 0.05us 0.167ms CompareFloatsIntegers: 57ms 57ms 0.06us 0.125ms CompareIntegers: 67ms 67ms 0.04us 0.250ms CompareInternedStrings: 67ms 67ms 0.04us 0.631ms CompareLongs: 60ms 62ms 0.06us 0.146ms CompareStrings: 52ms 53ms 0.05us 0.425ms CompareUnicode: 62ms 62ms 0.08us 0.350ms ComplexPythonFunctionCalls: 85ms 86ms 0.43us 0.210ms ConcatStrings: 160ms 173ms 0.35us 0.252ms ConcatUnicode: 74ms 75ms 0.25us 0.172ms CreateInstances: 81ms 84ms 0.75us 0.170ms CreateNewInstances: 62ms 64ms 0.77us 0.192ms CreateStringsWithConcat: 75ms 76ms 0.08us 0.418ms CreateUnicodeWithConcat: 64ms 65ms 0.16us 0.167ms DictCreation: 56ms 57ms 0.14us 0.167ms DictWithFloatKeys: 60ms 61ms 0.07us 0.313ms DictWithIntegerKeys: 61ms 62ms 0.05us 0.418ms DictWithStringKeys: 55ms 56ms 0.05us 0.418ms ForLoops: 43ms 43ms 1.72us 0.032ms IfThenElse: 60ms 61ms 0.04us 0.313ms ListSlicing: 72ms 90ms 6.40us 0.041ms NestedForLoops: 59ms 61ms 0.04us 0.013ms NestedListComprehensions: 79ms 81ms 6.77us 0.043ms NormalClassAttribute: 62ms 63ms 0.05us 0.210ms NormalInstanceAttribute: 55ms 56ms 0.05us 0.210ms PythonFunctionCalls: 80ms 81ms 0.25us 0.126ms PythonMethodCalls: 84ms 84ms 0.38us 0.065ms Recursion: 97ms 97ms 1.95us 0.209ms SecondImport: 82ms 83ms 0.83us 0.083ms SecondPackageImport: 88ms 90ms 0.90us 0.083ms SecondSubmoduleImport: 114ms 115ms 1.15us 0.083ms SimpleComplexArithmetic: 52ms 53ms 0.06us 0.167ms SimpleDictManipulation: 65ms 65ms 0.05us 0.209ms SimpleFloatArithmetic: 50ms 51ms 0.04us 0.251ms SimpleIntFloatArithmetic: 52ms 52ms 0.04us 0.251ms SimpleIntegerArithmetic: 52ms 52ms 0.04us 0.251ms SimpleListComprehensions: 67ms 70ms 5.82us 0.043ms SimpleListManipulation: 49ms 50ms 0.04us 0.272ms SimpleLongArithmetic: 48ms 48ms 0.07us 0.125ms SmallLists: 80ms 83ms 0.12us 0.167ms SmallTuples: 72ms 72ms 0.13us 0.221ms SpecialClassAttribute: 62ms 62ms 0.05us 0.210ms SpecialInstanceAttribute: 127ms 129ms 0.11us 0.210ms StringMappings: 396ms 399ms 1.58us 0.233ms StringPredicates: 125ms 127ms 0.18us 1.117ms StringSlicing: 95ms 101ms 0.18us 0.355ms TryExcept: 47ms 48ms 0.02us 0.313ms TryFinally: 60ms 61ms 0.38us 0.167ms TryRaiseExcept: 62ms 64ms 0.99us 0.167ms TupleSlicing: 85ms 88ms 0.33us 0.028ms UnicodeMappings: 93ms 93ms 2.59us 0.355ms UnicodePredicates: 66ms 66ms 0.12us 1.342ms UnicodeProperties: 86ms 88ms 0.22us 1.125ms UnicodeSlicing: 84ms 91ms 0.19us 0.315ms WithFinally: 91ms 93ms 0.58us 0.167ms WithRaiseExcept: 127ms 129ms 1.61us 0.209ms ------------------------------------------------------------------------------- Totals: 4571ms 4676ms From benhoyt at gmail.com Mon Jul 22 01:46:40 2013 From: benhoyt at gmail.com (Ben Hoyt) Date: Mon, 22 Jul 2013 11:46:40 +1200 Subject: [Python-Dev] Building a Faster Python In-Reply-To: <252C50D8-C23D-438D-BAE1-B22E0D65A21A@gmail.com> References: <252C50D8-C23D-438D-BAE1-B22E0D65A21A@gmail.com> Message-ID: > PyBench2.0 shows the total running time dropping from 5653ms to 4571ms. That's very cool -- a significant improvement. Is this the kind of change that could go into 2.7.6 binaries? As a Windows user, it makes me wonder if compiling with the latest version of the Microsoft compiler would improve things similarly? (Though updating project files to that is almost certainly a bigger project than the gcc update.) -Ben -------------- next part -------------- An HTML attachment was scrubbed... URL: From brian at python.org Mon Jul 22 02:21:23 2013 From: brian at python.org (Brian Curtin) Date: Sun, 21 Jul 2013 19:21:23 -0500 Subject: [Python-Dev] Building a Faster Python In-Reply-To: References: <252C50D8-C23D-438D-BAE1-B22E0D65A21A@gmail.com> Message-ID: On Sun, Jul 21, 2013 at 6:46 PM, Ben Hoyt wrote: >> PyBench2.0 shows the total running time dropping from 5653ms to 4571ms. > > That's very cool -- a significant improvement. Is this the kind of change > that could go into 2.7.6 binaries? > > As a Windows user, it makes me wonder if compiling with the latest version > of the Microsoft compiler would improve things similarly? (Though updating > project files to that is almost certainly a bigger project than the gcc > update.) I think I have a 3.3 build on VS2012 somewhere - maybe I'll refresh it for default/3.4 and run the same benchmarks on it. The changes couldn't go into 2.7 as far as I'm aware, at least when it comes to changing Visual Studio versions. From Steve.Dower at microsoft.com Mon Jul 22 02:24:45 2013 From: Steve.Dower at microsoft.com (Steve Dower) Date: Mon, 22 Jul 2013 00:24:45 +0000 Subject: [Python-Dev] Building a Faster Python In-Reply-To: References: <252C50D8-C23D-438D-BAE1-B22E0D65A21A@gmail.com> Message-ID: >From: Ben Hoyt >> PyBench2.0 shows the total running time dropping from 5653ms to 4571ms. > > That's very cool -- a significant improvement. Is this the kind of change that could go into 2.7.6 binaries? > > As a Windows user, it makes me wonder if compiling with the latest version of the Microsoft compiler > would improve things similarly? I'd expect to see some improvement, based solely on the bugs fixed recently by the optimizer team. No idea how much, but I know that Martin builds with PGO, and that's also been improved recently. > (Though updating project files to that is almost certainly a bigger project than the gcc update.) No idea what the gcc update involved, but this is only accurate if it took less than two minutes :-) I've upgraded the CPython projects before and everything worked fine. Steve From nad at acm.org Mon Jul 22 02:32:51 2013 From: nad at acm.org (Ned Deily) Date: Sun, 21 Jul 2013 17:32:51 -0700 Subject: [Python-Dev] Building a Faster Python References: <252C50D8-C23D-438D-BAE1-B22E0D65A21A@gmail.com> Message-ID: In article <252C50D8-C23D-438D-BAE1-B22E0D65A21A at gmail.com>, Raymond Hettinger wrote: > Our current Mac OS X builds use GCC-4.2. > > On Python2.7, I ran a comparison of gcc-4.2.1 builds > versus gcc-4.8.1 and found that the latter makes a much > faster Python. PyBench2.0 shows the total running time > dropping from 5653ms to 4571ms. The code is uniformly > better in just about every category. Interesting. A note about compilers: it looks like the benchmark used the Apple gcc4.2-frontend / LLVM-backend hybrid compiler that is shipped with Xcode 4. We don't use that compiler for building any OS X binary installers, in fact, we blacklist it for Python 3.3+ because we couldn't get it to produce a working 3.3 interpreter: http://bugs.python.org/issue13241 Apple has deprecated use of that compiler (it hasn't been patched in ages) and has announced that it will no longer be shipped starting with the next major release of Xcode, which is expected with the upcoming release of OS X 10.9 (Mavericks). What would be interesting is to run the benchmark with Apple's currently supported compiler for OS X, clang, which is the default "cc" in the most recent versions of Xcode 4 (4.6.3) for OS X 10.7 and 10.8. $ cc --version Apple LLVM version 4.2 (clang-425.0.28) (based on LLVM 3.2svn) Clang is expected to be the only compiler available via Xcode for 10.9. Note that the most recent Python binary installers default to using clang to build user extension modules with Xcode 4 on 10.7 and 10.8. Currently, for the Intel-only (64-bit/32-bit) installers we instead use the final Apple-supplied gcc-4.2 (gcc front- and backend, no LLVM) shipped with Xcode 3.2.6 on OS X 10.6: [GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin] This makes it easier to provide one Python that can be installed and works on multiple OS X versions, for the Intel-only variant: 10.6 through 10.8. As of Xcode 4, Apple no longer ships this compiler. Because of the post-4.2 switch to GPL-3 licensing for gcc, it seems that Apple has no plans to look at, much less ship, anything newer than 4.2 from the gcc code base. At some point, we will probably switch to using clang for the Python in the installer but, at the moment, there appear to still be a large number of systems out there running OS X 10.6. (Also, we and most third-party open-source package managers, like MacPorts and Homebrew, stick to using Apple-supplied compilers for most things rather than vanilla Gnu gcc 4.x because, among other things, Apple includes additional support within their compiler build chain for features like universal binaries and SDK support.) -- Ned Deily, nad at acm.org From raymond.hettinger at gmail.com Mon Jul 22 03:04:16 2013 From: raymond.hettinger at gmail.com (Raymond Hettinger) Date: Sun, 21 Jul 2013 18:04:16 -0700 Subject: [Python-Dev] Building a Faster Python In-Reply-To: References: <252C50D8-C23D-438D-BAE1-B22E0D65A21A@gmail.com> Message-ID: On Jul 21, 2013, at 5:32 PM, Ned Deily wrote: > In article <252C50D8-C23D-438D-BAE1-B22E0D65A21A at gmail.com>, > Raymond Hettinger wrote: >> Our current Mac OS X builds use GCC-4.2. >> >> On Python2.7, I ran a comparison of gcc-4.2.1 builds >> versus gcc-4.8.1 and found that the latter makes a much >> faster Python. PyBench2.0 shows the total running time >> dropping from 5653ms to 4571ms. The code is uniformly >> better in just about every category. > > Interesting. A note about compilers: it looks like the benchmark used > the Apple gcc4.2-frontend / LLVM-backend hybrid compiler that is shipped > with Xcode 4. We don't use that compiler for building any OS X binary > installers, in fact, we blacklist it for Python 3.3+ because we couldn't > get it to produce a working 3.3 interpreter: The results aren't much different using the build published on python.org. Using a fresh OS X install from our website, PyBench clocks in at 5376ms Raymond --------------------------- ------------------------------------------------------------------------------- PYBENCH 2.0 ------------------------------------------------------------------------------- * using CPython 2.7.5 (v2.7.5:ab05e7dd2788, May 13 2013, 13:18:45) [GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] * disabled garbage collection * system check interval set to maximum: 2147483647 * using timer: time.time Calibrating tests. Please wait... done. Running 10 round(s) of the suite at warp factor 10: * Round 1 done in 5.459 seconds. * Round 2 done in 5.510 seconds. * Round 3 done in 5.476 seconds. * Round 4 done in 5.473 seconds. * Round 5 done in 5.497 seconds. * Round 6 done in 5.533 seconds. * Round 7 done in 5.520 seconds. * Round 8 done in 5.518 seconds. * Round 9 done in 5.515 seconds. * Round 10 done in 5.515 seconds. ------------------------------------------------------------------------------- Benchmark: 2013-07-21 17:54:13 ------------------------------------------------------------------------------- Rounds: 10 Warp: 10 Timer: time.time Machine Details: Platform ID: Darwin-12.4.0-x86_64-i386-64bit Processor: i386 Python: Implementation: CPython Executable: /Library/Frameworks/Python.framework/Versions/2.7/Resources/Python.app/Contents/MacOS/Python Version: 2.7.5 Compiler: GCC 4.2.1 (Apple Inc. build 5666) (dot 3) Bits: 64bit Build: May 13 2013 13:18:45 (#v2.7.5:ab05e7dd2788) Unicode: UCS2 Test minimum average operation overhead ------------------------------------------------------------------------------- BuiltinFunctionCalls: 108ms 109ms 0.21us 0.166ms BuiltinMethodLookup: 74ms 77ms 0.07us 0.193ms CompareFloats: 73ms 73ms 0.06us 0.229ms CompareFloatsIntegers: 71ms 71ms 0.08us 0.167ms CompareIntegers: 78ms 78ms 0.04us 0.334ms CompareInternedStrings: 76ms 77ms 0.05us 0.842ms CompareLongs: 69ms 70ms 0.07us 0.196ms CompareStrings: 61ms 61ms 0.06us 0.566ms CompareUnicode: 66ms 68ms 0.09us 0.431ms ComplexPythonFunctionCalls: 98ms 101ms 0.51us 0.280ms ConcatStrings: 166ms 179ms 0.36us 0.324ms ConcatUnicode: 79ms 81ms 0.27us 0.220ms CreateInstances: 105ms 107ms 0.96us 0.232ms CreateNewInstances: 84ms 88ms 1.04us 0.233ms CreateStringsWithConcat: 95ms 96ms 0.10us 0.552ms CreateUnicodeWithConcat: 70ms 72ms 0.18us 0.222ms DictCreation: 79ms 80ms 0.20us 0.221ms DictWithFloatKeys: 83ms 84ms 0.09us 0.414ms DictWithIntegerKeys: 87ms 88ms 0.07us 0.552ms DictWithStringKeys: 65ms 66ms 0.06us 0.552ms ForLoops: 58ms 58ms 2.34us 0.039ms IfThenElse: 65ms 66ms 0.05us 0.414ms ListSlicing: 72ms 89ms 6.38us 0.055ms NestedForLoops: 79ms 82ms 0.05us 0.014ms NestedListComprehensions: 94ms 96ms 7.98us 0.056ms NormalClassAttribute: 81ms 81ms 0.07us 0.280ms NormalInstanceAttribute: 73ms 75ms 0.06us 0.274ms PythonFunctionCalls: 85ms 86ms 0.26us 0.166ms PythonMethodCalls: 100ms 103ms 0.46us 0.086ms Recursion: 110ms 113ms 2.25us 0.276ms SecondImport: 104ms 105ms 1.05us 0.112ms SecondPackageImport: 111ms 111ms 1.11us 0.110ms SecondSubmoduleImport: 138ms 139ms 1.39us 0.110ms SimpleComplexArithmetic: 71ms 71ms 0.08us 0.221ms SimpleDictManipulation: 91ms 92ms 0.08us 0.276ms SimpleFloatArithmetic: 71ms 72ms 0.05us 0.331ms SimpleIntFloatArithmetic: 64ms 64ms 0.05us 0.331ms SimpleIntegerArithmetic: 64ms 64ms 0.05us 0.337ms SimpleListComprehensions: 81ms 83ms 6.93us 0.056ms SimpleListManipulation: 61ms 61ms 0.05us 0.359ms SimpleLongArithmetic: 65ms 66ms 0.10us 0.168ms SmallLists: 90ms 94ms 0.14us 0.221ms SmallTuples: 77ms 79ms 0.15us 0.248ms SpecialClassAttribute: 79ms 79ms 0.07us 0.281ms SpecialInstanceAttribute: 96ms 98ms 0.08us 0.281ms StringMappings: 435ms 436ms 1.73us 0.291ms StringPredicates: 142ms 143ms 0.20us 1.499ms StringSlicing: 106ms 114ms 0.20us 0.466ms TryExcept: 68ms 68ms 0.03us 0.414ms TryFinally: 72ms 73ms 0.46us 0.221ms TryRaiseExcept: 77ms 78ms 1.23us 0.221ms TupleSlicing: 90ms 96ms 0.37us 0.036ms UnicodeMappings: 100ms 101ms 2.79us 0.402ms UnicodePredicates: 77ms 78ms 0.15us 1.820ms UnicodeProperties: 95ms 96ms 0.24us 1.503ms UnicodeSlicing: 91ms 98ms 0.20us 0.412ms WithFinally: 98ms 100ms 0.62us 0.230ms WithRaiseExcept: 159ms 163ms 2.04us 0.282ms ------------------------------------------------------------------------------- Totals: 5376ms 5502ms -------------- next part -------------- An HTML attachment was scrubbed... URL: From larry at hastings.org Mon Jul 22 03:01:29 2013 From: larry at hastings.org (Larry Hastings) Date: Sun, 21 Jul 2013 18:01:29 -0700 Subject: [Python-Dev] Building a Faster Python In-Reply-To: <252C50D8-C23D-438D-BAE1-B22E0D65A21A@gmail.com> References: <252C50D8-C23D-438D-BAE1-B22E0D65A21A@gmail.com> Message-ID: <51EC8469.1030608@hastings.org> On 07/21/2013 04:36 PM, Raymond Hettinger wrote: > Our current Mac OS X builds use GCC-4.2. > > On Python2.7, I ran a comparison of gcc-4.2.1 builds > versus gcc-4.8.1 and found that the latter makes a much > faster Python. PyBench2.0 shows the total running time > dropping from 5653ms to 4571ms. The code is uniformly > better in just about every category. I know that newer Microsoft compilers tend to drop support for older operating systems, and often that means the binaries simply won't work on those operating systems. (Yes, technically it's the libraries that break the older code.) Are Apple's compilers the same? Or do they tend to support all the old versions of OS X? //arry/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From rdmurray at bitdance.com Mon Jul 22 04:11:42 2013 From: rdmurray at bitdance.com (R. David Murray) Date: Sun, 21 Jul 2013 22:11:42 -0400 Subject: [Python-Dev] =?utf-8?q?=5BPython-checkins=5D_cpython_=282=2E7=29?= =?utf-8?q?=3A_Issue_=2318441=3A_Make_test=2Esupport=2Erequires=28?= =?utf-8?q?=27gui=27=29_skip_when_it_should=2E?= In-Reply-To: <3bz3Fl3m9SzSdZ@mail.python.org> References: <3bz3Fl3m9SzSdZ@mail.python.org> Message-ID: <20130722021142.CC2862500A8@webabinitio.net> On Mon, 22 Jul 2013 02:13:47 +0200, terry.reedy wrote: > +# If buildbot improperly sets gui resource (#18365, #18441), remove it > +# so requires('gui') tests are skipped while non-gui tests still run. > +if use_resources and 'gui' in use_resources: Note that the buildbot cannot "improperly" set the GUI resource. Setting a resource on the regrtest command line says "please try to run these tests". If they can't be run, they should then be skipped for whatever reason they can't run. --David From nad at acm.org Mon Jul 22 07:35:45 2013 From: nad at acm.org (Ned Deily) Date: Sun, 21 Jul 2013 22:35:45 -0700 Subject: [Python-Dev] cpython (2.7): Issue #18441: Make test.support.requires('gui') skip when it should. References: <3bz3Fl3m9SzSdZ@mail.python.org> <20130722021142.CC2862500A8@webabinitio.net> <51ECAE41.5060403@udel.edu> Message-ID: In article <51ECAE41.5060403 at udel.edu>, Terry Reedy wrote: > On 7/21/2013 10:11 PM, R. David Murray wrote: > > On Mon, 22 Jul 2013 02:13:47 +0200, terry.reedy > > wrote: > >> +# If buildbot improperly sets gui resource (#18365, #18441), remove it > >> +# so requires('gui') tests are skipped while non-gui tests still run. > >> +if use_resources and 'gui' in use_resources: > > > > Note that the buildbot cannot "improperly" set the GUI resource. > > I disagree. I do understand though, that what I call improper setting on > the command line cannot be prevented. So I think regrtest (and unittest > if and when it supports resources) should intercept 'gui', test once and > for all whether it was set 'properly' (or 'appropriately', if you > prefer), which is to say, test whether there is a usable GUI, and ignore > 'gui' if not. > > > Setting a resource on the regrtest command line says "please try to run > > these tests". Right. > When doing so causes a failure at best and a crash at worst, and those > consequences and the means of avoiding them are not documented, that is > not a very considerate request. It has cost me over a day of my life. Unfortunately, that's the way regrtest resources work. > The printed out -h help says > "gui - Run tests that require a running GUI." > Not 'try', but 'run'. That implies to me that there *is* 'a running GUI' > and that is it all right to run tests that *require* such. That was also > what I understood from the discussion on > http://bugs.python.org/issue18103 Create a GUI test framework for Idle This is exactly what Issue8716 was about. The buildbot has no way of knowing ahead of time whether a test will cause a crash or not. Yes, Tk should not crash but it does in some cases. Speaking of #8716, that reminds me that there is an open issue with it (documented in Issue17496). There is the start of a patch there to use a more general approach to testing for a working Tk that might be applicable on all platforms. -- Ned Deily, nad at acm.org From ronaldoussoren at mac.com Mon Jul 22 08:04:23 2013 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Mon, 22 Jul 2013 08:04:23 +0200 Subject: [Python-Dev] Building a Faster Python In-Reply-To: <51EC8469.1030608@hastings.org> References: <252C50D8-C23D-438D-BAE1-B22E0D65A21A@gmail.com> <51EC8469.1030608@hastings.org> Message-ID: <9B1C8E04-4DA8-41CA-8FAA-A73F18EE4532@mac.com> On 22 Jul, 2013, at 3:01, Larry Hastings wrote: > On 07/21/2013 04:36 PM, Raymond Hettinger wrote: >> Our current Mac OS X builds use GCC-4.2. >> >> On Python2.7, I ran a comparison of gcc-4.2.1 builds >> versus gcc-4.8.1 and found that the latter makes a much >> faster Python. PyBench2.0 shows the total running time >> dropping from 5653ms to 4571ms. The code is uniformly >> better in just about every category. >> > > I know that newer Microsoft compilers tend to drop support for older operating systems, and often that means the binaries simply won't work on those operating systems. (Yes, technically it's the libraries that break the older code.) > > Are Apple's compilers the same? Or do they tend to support all the old versions of OS X? The compiler's included with Xcode 4 cannot generate code for PowerPC machines, other than that it should be possible to generate code for older machines with the latest Xcode version. I have a Python installation on my machine that I use to generate app distributions for a 10.5 machine, and AFAIK this would also work for deploying to 10.4. The hard part is usually convincing autotools-using projects to not use APIs that are available on the build machine but not on the deployment machine. Ronald From ronaldoussoren at mac.com Mon Jul 22 07:59:33 2013 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Mon, 22 Jul 2013 07:59:33 +0200 Subject: [Python-Dev] Building a Faster Python In-Reply-To: <252C50D8-C23D-438D-BAE1-B22E0D65A21A@gmail.com> References: <252C50D8-C23D-438D-BAE1-B22E0D65A21A@gmail.com> Message-ID: <22243275-1AC3-4BC2-B112-D082134E10F7@mac.com> On 22 Jul, 2013, at 1:36, Raymond Hettinger wrote: > Our current Mac OS X builds use GCC-4.2. > > On Python2.7, I ran a comparison of gcc-4.2.1 builds > versus gcc-4.8.1 and found that the latter makes a much > faster Python. PyBench2.0 shows the total running time > dropping from 5653ms to 4571ms. The code is uniformly > better in just about every category. Have you tried using clang from the latest devtools as well? The OSX binary installers are build using the developer tools from Apple, that happen to use gcc 4.2 on the machine used to build at least the 32-bit binary installer (as that's the latest Xcode that includes PPC support). FWIW I'd like to test with "clang -O4" as well (this performs link-time optimization), I've seen some speedup with other projects and this might help with CPython's speed as well. Ronald From ronaldoussoren at mac.com Mon Jul 22 08:08:51 2013 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Mon, 22 Jul 2013 08:08:51 +0200 Subject: [Python-Dev] Building a Faster Python In-Reply-To: References: <252C50D8-C23D-438D-BAE1-B22E0D65A21A@gmail.com> Message-ID: <38C7767E-E24C-42B8-A953-5FC989596373@mac.com> On 22 Jul, 2013, at 1:46, Ben Hoyt wrote: > > PyBench2.0 shows the total running time dropping from 5653ms to 4571ms. > > That's very cool -- a significant improvement. Is this the kind of change that could go into 2.7.6 binaries? I'd prefer not to do that (but don't build the installers anymore). The installers for OSX are build using the system compiler, using a different compiler makes it harder to build the installer. I don't even know if upstream GCC could easily be used for the binary installers, does GCC 4.8 support building FAT binaries in its compiler driver? Ronald From ronaldoussoren at mac.com Mon Jul 22 08:13:49 2013 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Mon, 22 Jul 2013 08:13:49 +0200 Subject: [Python-Dev] cpython (2.7): Issue #18441: Make test.support.requires('gui') skip when it should. In-Reply-To: References: <3bz3Fl3m9SzSdZ@mail.python.org> <20130722021142.CC2862500A8@webabinitio.net> <51ECAE41.5060403@udel.edu> Message-ID: On 22 Jul, 2013, at 7:35, Ned Deily wrote: > In article <51ECAE41.5060403 at udel.edu>, Terry Reedy > wrote: >> > > This is exactly what Issue8716 was about. The buildbot has no way of > knowing ahead of time whether a test will cause a crash or not. Yes, Tk > should not crash but it does in some cases. Speaking of #8716, that > reminds me that there is an open issue with it (documented in > Issue17496). There is the start of a patch there to use a more general > approach to testing for a working Tk that might be applicable on all > platforms. Issue17496 contains a patch that might fix the crashing problem for OSX (and possibly other Unix-y platforms). I haven't been able to test this in an environment that's simular enough to the buildbots (that is, a system where the user running the test does not have GUI access). That patch starts wish (the TCL shell) in a subprocess, and skips tests when that doesn't work (on OSX that would be because Tk crashed). Ronald From solipsis at pitrou.net Mon Jul 22 08:15:34 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 22 Jul 2013 08:15:34 +0200 Subject: [Python-Dev] Building a Faster Python References: <252C50D8-C23D-438D-BAE1-B22E0D65A21A@gmail.com> Message-ID: <20130722081534.7cef6df0@fsol> On Sun, 21 Jul 2013 16:36:35 -0700 Raymond Hettinger wrote: > Our current Mac OS X builds use GCC-4.2. > > On Python2.7, I ran a comparison of gcc-4.2.1 builds > versus gcc-4.8.1 and found that the latter makes a much > faster Python. PyBench2.0 shows the total running time > dropping from 5653ms to 4571ms. The code is uniformly > better in just about every category. You could try running the benchmarks suite to see what that gives: http://hg.python.org/benchmarks/ Regards Antoine. From fijall at gmail.com Mon Jul 22 09:32:01 2013 From: fijall at gmail.com (Maciej Fijalkowski) Date: Mon, 22 Jul 2013 09:32:01 +0200 Subject: [Python-Dev] Building a Faster Python In-Reply-To: <20130722081534.7cef6df0@fsol> References: <252C50D8-C23D-438D-BAE1-B22E0D65A21A@gmail.com> <20130722081534.7cef6df0@fsol> Message-ID: On Mon, Jul 22, 2013 at 8:15 AM, Antoine Pitrou wrote: > On Sun, 21 Jul 2013 16:36:35 -0700 > Raymond Hettinger wrote: >> Our current Mac OS X builds use GCC-4.2. >> >> On Python2.7, I ran a comparison of gcc-4.2.1 builds >> versus gcc-4.8.1 and found that the latter makes a much >> faster Python. PyBench2.0 shows the total running time >> dropping from 5653ms to 4571ms. The code is uniformly >> better in just about every category. > > You could try running the benchmarks suite to see what that gives: > http://hg.python.org/benchmarks/ > > Regards > > Antoine. or pypy benchmark suite which is more comprehensive for python 2.7 (http://bitbucket.org/pypy/benchmarks) From fijall at gmail.com Mon Jul 22 09:32:25 2013 From: fijall at gmail.com (Maciej Fijalkowski) Date: Mon, 22 Jul 2013 09:32:25 +0200 Subject: [Python-Dev] Building a Faster Python In-Reply-To: References: <252C50D8-C23D-438D-BAE1-B22E0D65A21A@gmail.com> <20130722081534.7cef6df0@fsol> Message-ID: On Mon, Jul 22, 2013 at 9:32 AM, Maciej Fijalkowski wrote: > On Mon, Jul 22, 2013 at 8:15 AM, Antoine Pitrou wrote: >> On Sun, 21 Jul 2013 16:36:35 -0700 >> Raymond Hettinger wrote: >>> Our current Mac OS X builds use GCC-4.2. >>> >>> On Python2.7, I ran a comparison of gcc-4.2.1 builds >>> versus gcc-4.8.1 and found that the latter makes a much >>> faster Python. PyBench2.0 shows the total running time >>> dropping from 5653ms to 4571ms. The code is uniformly >>> better in just about every category. >> >> You could try running the benchmarks suite to see what that gives: >> http://hg.python.org/benchmarks/ >> >> Regards >> >> Antoine. > > or pypy benchmark suite which is more comprehensive for python 2.7 > (http://bitbucket.org/pypy/benchmarks) Besides, is there any reason not to use clang by default on OS X? From ethan at stoneleaf.us Mon Jul 22 09:15:56 2013 From: ethan at stoneleaf.us (Ethan Furman) Date: Mon, 22 Jul 2013 00:15:56 -0700 Subject: [Python-Dev] dict __contains__ raises TypeError on unhashable input In-Reply-To: <51E9D018.4030806@stoneleaf.us> References: <51E9D018.4030806@stoneleaf.us> Message-ID: <51ECDC2C.4090808@stoneleaf.us> Thanks for the insights, everyone. -- ~Ethan~ From ronaldoussoren at mac.com Mon Jul 22 09:50:53 2013 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Mon, 22 Jul 2013 09:50:53 +0200 Subject: [Python-Dev] Building a Faster Python In-Reply-To: References: <252C50D8-C23D-438D-BAE1-B22E0D65A21A@gmail.com> <20130722081534.7cef6df0@fsol> Message-ID: On 22 Jul, 2013, at 9:32, Maciej Fijalkowski wrote: > On Mon, Jul 22, 2013 at 9:32 AM, Maciej Fijalkowski wrote: >> On Mon, Jul 22, 2013 at 8:15 AM, Antoine Pitrou wrote: >>> On Sun, 21 Jul 2013 16:36:35 -0700 >>> Raymond Hettinger wrote: >>>> Our current Mac OS X builds use GCC-4.2. >>>> >>>> On Python2.7, I ran a comparison of gcc-4.2.1 builds >>>> versus gcc-4.8.1 and found that the latter makes a much >>>> faster Python. PyBench2.0 shows the total running time >>>> dropping from 5653ms to 4571ms. The code is uniformly >>>> better in just about every category. >>> >>> You could try running the benchmarks suite to see what that gives: >>> http://hg.python.org/benchmarks/ >>> >>> Regards >>> >>> Antoine. >> >> or pypy benchmark suite which is more comprehensive for python 2.7 >> (http://bitbucket.org/pypy/benchmarks) > > Besides, is there any reason not to use clang by default on OS X? The 32-bit installer contains binaries that work on PPC, that's why those are built using an older version of Xcode. I'd have to check if that version of Xcode supports clang, and if that version of clang is good enough. The "intel" installer can, and should, be build with clang (and preferably with the most recent Xcode release to ensure that the latest supported compiler is used). Note that the CPython configure script, and distutils, already use clang by default if you a recent Xcode but that's primarily because gcc is llvm-gcc when you use Xcode and llvm-gcc is broken (it miscompiles at least the unicode implementation in Python 3.3), this overrides the default behavior of configure (using gcc whenever it is available unless the user explictly overrides). Ronald From martin at v.loewis.de Mon Jul 22 14:24:01 2013 From: martin at v.loewis.de (martin at v.loewis.de) Date: Mon, 22 Jul 2013 14:24:01 +0200 Subject: [Python-Dev] Building a Faster Python In-Reply-To: References: <252C50D8-C23D-438D-BAE1-B22E0D65A21A@gmail.com> Message-ID: <20130722142401.Horde.nqeJmFp2uS2W98k7RhbP4Q1@webmail.df.eu> Quoting Steve Dower : >> As a Windows user, it makes me wonder if compiling with the latest >> version of the Microsoft compiler >> would improve things similarly? > > I'd expect to see some improvement, based solely on the bugs fixed > recently by the optimizer team. No idea how much, but I know that > Martin builds with PGO, and that's also been improved recently. I would not expect to see any improvement, unfortunately. I used to build with PGO, but the recent VS releases miscompiled code, which was very hard to track down. So PGO has lost in my view for at least 3 VS releases :-( >> (Though updating project files to that is almost certainly a bigger >> project than the gcc update.) > > No idea what the gcc update involved, but this is only accurate if > it took less than two minutes :-) I've upgraded the CPython projects > before and everything worked fine. Still, we couldn't release 2.7 with any other compiler because of the CRT dependency. Regards, Martin From storchaka at gmail.com Mon Jul 22 15:30:41 2013 From: storchaka at gmail.com (Serhiy Storchaka) Date: Mon, 22 Jul 2013 16:30:41 +0300 Subject: [Python-Dev] Dash In-Reply-To: References: Message-ID: I have opened an issue (http://bugs.python.org/issue18529) for patches. From eric at trueblade.com Mon Jul 22 16:09:15 2013 From: eric at trueblade.com (Eric V. Smith) Date: Mon, 22 Jul 2013 10:09:15 -0400 Subject: [Python-Dev] [Python-checkins] cpython (3.3): let's not return NULL from functions that should return ints In-Reply-To: <3byyDG02GdzQlq@mail.python.org> References: <3byyDG02GdzQlq@mail.python.org> Message-ID: <51ED3D0B.9080803@trueblade.com> [re-sending to python-dev] On 7/21/2013 4:27 PM, benjamin.peterson wrote: > @@ -267,7 +267,7 @@ > Py_DECREF(io); > Py_DECREF(binary); > PyMem_FREE(found_encoding); > - return PyErr_SetFromErrnoWithFilenameObject(PyExc_IOError, filename); > + return 0; > } > fob = _PyObject_CallMethodId(io, &PyId_TextIOWrapper, "Os", binary, encoding); > Py_DECREF(io); Did you mean to remove the call to PyErr_SetFromErrnoWithFilenameObject? Or just call it, then ignore its return value and return 0? -- Eric. _______________________________________________ Python-checkins mailing list Python-checkins at python.org http://mail.python.org/mailman/listinfo/python-checkins From benjamin at python.org Mon Jul 22 16:58:07 2013 From: benjamin at python.org (Benjamin Peterson) Date: Mon, 22 Jul 2013 07:58:07 -0700 Subject: [Python-Dev] [Python-checkins] cpython (3.3): let's not return NULL from functions that should return ints In-Reply-To: <51ED3D0B.9080803@trueblade.com> References: <3byyDG02GdzQlq@mail.python.org> <51ED3D0B.9080803@trueblade.com> Message-ID: Yes, I meant to remove it. This function gnores errors in general. 2013/7/22 Eric V. Smith : > [re-sending to python-dev] > > On 7/21/2013 4:27 PM, benjamin.peterson wrote: > >> @@ -267,7 +267,7 @@ >> Py_DECREF(io); >> Py_DECREF(binary); >> PyMem_FREE(found_encoding); >> - return PyErr_SetFromErrnoWithFilenameObject(PyExc_IOError, filename); >> + return 0; >> } >> fob = _PyObject_CallMethodId(io, &PyId_TextIOWrapper, "Os", binary, encoding); >> Py_DECREF(io); > > Did you mean to remove the call to PyErr_SetFromErrnoWithFilenameObject? > Or just call it, then ignore its return value and return 0? > > > -- > Eric. > _______________________________________________ > Python-checkins mailing list > Python-checkins at python.org > http://mail.python.org/mailman/listinfo/python-checkins > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/benjamin%40python.org -- Regards, Benjamin From dmalcolm at redhat.com Mon Jul 22 17:08:32 2013 From: dmalcolm at redhat.com (David Malcolm) Date: Mon, 22 Jul 2013 11:08:32 -0400 Subject: [Python-Dev] Building a Faster Python In-Reply-To: References: <252C50D8-C23D-438D-BAE1-B22E0D65A21A@gmail.com> <20130722081534.7cef6df0@fsol> Message-ID: <1374505712.3898.13.camel@surprise> On Mon, 2013-07-22 at 09:32 +0200, Maciej Fijalkowski wrote: > On Mon, Jul 22, 2013 at 9:32 AM, Maciej Fijalkowski wrote: > > On Mon, Jul 22, 2013 at 8:15 AM, Antoine Pitrou wrote: > >> On Sun, 21 Jul 2013 16:36:35 -0700 > >> Raymond Hettinger wrote: > >>> Our current Mac OS X builds use GCC-4.2. > >>> > >>> On Python2.7, I ran a comparison of gcc-4.2.1 builds > >>> versus gcc-4.8.1 and found that the latter makes a much > >>> faster Python. PyBench2.0 shows the total running time > >>> dropping from 5653ms to 4571ms. The code is uniformly > >>> better in just about every category. > >> > >> You could try running the benchmarks suite to see what that gives: > >> http://hg.python.org/benchmarks/ > >> > >> Regards > >> > >> Antoine. > > > > or pypy benchmark suite which is more comprehensive for python 2.7 > > (http://bitbucket.org/pypy/benchmarks) > > Besides, is there any reason not to use clang by default on OS X? How did this thread go from: "for OS X, GCC 4.8.1 gives you significantly faster machine code than the system GCC 4.2.1" to "let's just use clang" ? Presumably if you want the faster possible machine code for the platform the correct approach is to benchmark the available compilers trying various options and to see which does best. Note that the meaning of e.g. "-O3" varies from compiler to compiler (I see someone else already made the Apples-to-Apples pun). (I should declare that I've been hacking on GCC for the last few months, so I have an interest in this) Hope this is constructive Dave From solipsis at pitrou.net Mon Jul 22 17:15:08 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 22 Jul 2013 17:15:08 +0200 Subject: [Python-Dev] Building a Faster Python References: <252C50D8-C23D-438D-BAE1-B22E0D65A21A@gmail.com> <20130722081534.7cef6df0@fsol> <1374505712.3898.13.camel@surprise> Message-ID: <20130722171508.43fa2ff7@pitrou.net> Le Mon, 22 Jul 2013 11:08:32 -0400, David Malcolm a ?crit : > > How did this thread go from: > "for OS X, GCC 4.8.1 gives you significantly faster machine code > than the system GCC 4.2.1" > to > "let's just use clang" > ? > > Presumably if you want the faster possible machine code for the > platform the correct approach is to benchmark the available compilers > trying various options and to see which does best. Depends whether the goal is to choose the fastest compiler, or to use the platform's official compiler. Linux distributions don't compile packages with icc, AFAIK. Regards Antoine. From dmalcolm at redhat.com Mon Jul 22 17:24:34 2013 From: dmalcolm at redhat.com (David Malcolm) Date: Mon, 22 Jul 2013 11:24:34 -0400 Subject: [Python-Dev] Building a Faster Python In-Reply-To: <20130722171508.43fa2ff7@pitrou.net> References: <252C50D8-C23D-438D-BAE1-B22E0D65A21A@gmail.com> <20130722081534.7cef6df0@fsol> <1374505712.3898.13.camel@surprise> <20130722171508.43fa2ff7@pitrou.net> Message-ID: <1374506674.3898.17.camel@surprise> On Mon, 2013-07-22 at 17:15 +0200, Antoine Pitrou wrote: > Le Mon, 22 Jul 2013 11:08:32 -0400, > David Malcolm a ?crit : > > > > How did this thread go from: > > "for OS X, GCC 4.8.1 gives you significantly faster machine code > > than the system GCC 4.2.1" > > to > > "let's just use clang" > > ? > > > > Presumably if you want the faster possible machine code for the > > platform the correct approach is to benchmark the available compilers > > trying various options and to see which does best. > > Depends whether the goal is to choose the fastest compiler, or to > use the platform's official compiler. Linux distributions don't compile > packages with icc, AFAIK. If Intel put icc under a Free Software/Open Source license, maybe we would ;-) (only my own opinion, of course) Dave From ronaldoussoren at mac.com Mon Jul 22 18:09:40 2013 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Mon, 22 Jul 2013 18:09:40 +0200 Subject: [Python-Dev] Building a Faster Python In-Reply-To: <1374505712.3898.13.camel@surprise> References: <252C50D8-C23D-438D-BAE1-B22E0D65A21A@gmail.com> <20130722081534.7cef6df0@fsol> <1374505712.3898.13.camel@surprise> Message-ID: On 22 Jul, 2013, at 17:08, David Malcolm wrote: > On Mon, 2013-07-22 at 09:32 +0200, Maciej Fijalkowski wrote: >> On Mon, Jul 22, 2013 at 9:32 AM, Maciej Fijalkowski wrote: >>> On Mon, Jul 22, 2013 at 8:15 AM, Antoine Pitrou wrote: >>>> On Sun, 21 Jul 2013 16:36:35 -0700 >>>> Raymond Hettinger wrote: >>>>> Our current Mac OS X builds use GCC-4.2. >>>>> >>>>> On Python2.7, I ran a comparison of gcc-4.2.1 builds >>>>> versus gcc-4.8.1 and found that the latter makes a much >>>>> faster Python. PyBench2.0 shows the total running time >>>>> dropping from 5653ms to 4571ms. The code is uniformly >>>>> better in just about every category. >>>> >>>> You could try running the benchmarks suite to see what that gives: >>>> http://hg.python.org/benchmarks/ >>>> >>>> Regards >>>> >>>> Antoine. >>> >>> or pypy benchmark suite which is more comprehensive for python 2.7 >>> (http://bitbucket.org/pypy/benchmarks) >> >> Besides, is there any reason not to use clang by default on OS X? > > How did this thread go from: > "for OS X, GCC 4.8.1 gives you significantly faster machine code > than the system GCC 4.2.1" > to > "let's just use clang" > ? Because we use the system compiler for building the official binary packages. I'm not looking forward to bootstrapping GCC multiple times[*] just to be able to build a slightly faster python. And more so because you have to be very careful when using a alternative compiler when building the installer, it is very easy to end up with a build that others cannot use to build extension because they don't have /Users/ronald/Tools/Compiler/gcc-4.8/bin/gcc. > > (I should declare that I've been hacking on GCC for the last few months, > so I have an interest in this) It would still be interesting to know which compiler would generate the fastest code for CPython. Apple tends to claim that clang generates better code than GCC, buit AFAIK they compare the latest clang with the latest version of GCC that they used to ship, which is ancient by now. Ronald [*] multiple times due to fat binaries. From ben at redfrontdoor.org Mon Jul 22 23:44:28 2013 From: ben at redfrontdoor.org (Ben North) Date: Mon, 22 Jul 2013 22:44:28 +0100 Subject: [Python-Dev] Avoiding error from repr() of recursive dictview Message-ID: Hi, A friend of mine, Ruadhan O'Flanagan, came across a bug which turned out to be the one noted in [http://bugs.python.org/issue18019], i.e.: >>> d={} >>> d[42]=d.viewvalues() >>> d This issue has been fixed in hg; the behaviour now is that a RuntimeError is produced for a recursive dictionary view: >>> d={} >>> d[42]=d.viewvalues() >>> d # (output line-broken:) {42: Traceback (most recent call last): File "", line 1, in RuntimeError: maximum recursion depth exceeded while getting the repr of a list Before finding this, though, I'd investigated and made a patch which produces a similar "..." output to a recursive dictionary. Reworking against current 2.7, the behaviour would be: >>> x={} >>> x[42]=x >>> x # existing behaviour for dictionaries: {42: {...}} >>> d={} >>> d[42]=d.viewvalues() >>> d # new behaviour: {42: dict_values([...])} >>> d[43]=d.viewitems() >>> d # (output line-broken:) {42: dict_values([..., dict_items([(42, ...), (43, ...)])]), 43: dict_items([(42, dict_values([..., ...])), (43, ...)])} Attached is the patch, against current 2.7 branch. If there is interest in applying this, I will create a proper patch (changelog entry, fix to Lib/test/test_dictviews.py, etc.). Thanks, Ben. -------------- next part -------------- A non-text attachment was scrubbed... Name: non-error-recursive-dictview.patch Type: application/octet-stream Size: 974 bytes Desc: not available URL: From greg at krypto.org Tue Jul 23 00:01:38 2013 From: greg at krypto.org (Gregory P. Smith) Date: Mon, 22 Jul 2013 15:01:38 -0700 Subject: [Python-Dev] Avoiding error from repr() of recursive dictview In-Reply-To: References: Message-ID: On Mon, Jul 22, 2013 at 2:44 PM, Ben North wrote: > Hi, > > A friend of mine, Ruadhan O'Flanagan, came across a bug which turned out > to be the one noted in [http://bugs.python.org/issue18019], i.e.: > > >>> d={} > >>> d[42]=d.viewvalues() > >>> d > > > This issue has been fixed in hg; the behaviour now is that a > RuntimeError is produced for a recursive dictionary view: > > >>> d={} > >>> d[42]=d.viewvalues() > >>> d # (output line-broken:) > {42: Traceback (most recent call last): > File "", line 1, in > RuntimeError: maximum recursion depth exceeded > while getting the repr of a list > > Before finding this, though, I'd investigated and made a patch which > produces a similar "..." output to a recursive dictionary. Reworking > against current 2.7, the behaviour would be: > > >>> x={} > >>> x[42]=x > >>> x # existing behaviour for dictionaries: > {42: {...}} > > >>> d={} > >>> d[42]=d.viewvalues() > >>> d # new behaviour: > {42: dict_values([...])} > >>> d[43]=d.viewitems() > >>> d # (output line-broken:) > {42: dict_values([..., dict_items([(42, ...), (43, ...)])]), > 43: dict_items([(42, dict_values([..., ...])), (43, ...)])} > > Attached is the patch, against current 2.7 branch. If there is interest > in applying this, I will create a proper patch (changelog entry, fix to > Lib/test/test_dictviews.py, etc.). > Mailing lists are where patches go to get lost and die. :) Post it on an issue on bugs.python.org. Given that the RuntimeError fix has been released, your proposed ... behavior is arguably a new feature so I'd only expect this to make sense for consideration in 3.4, not 2.7. (if accepted at all) -gps > > Thanks, > > Ben. > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/greg%40krypto.org > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ben at redfrontdoor.org Tue Jul 23 00:26:46 2013 From: ben at redfrontdoor.org (Ben North) Date: Mon, 22 Jul 2013 23:26:46 +0100 Subject: [Python-Dev] Avoiding error from repr() of recursive dictview In-Reply-To: References: Message-ID: Hi Greg, Thanks for the feedback. On 22 July 2013 23:01, Gregory P. Smith wrote: > On Mon, Jul 22, 2013 at 2:44 PM, Ben North wrote: >> [... proposed change of behaviour such that a recursive >> dictview gives a repr() with "..." rather than a RuntimeError ...] > > [...] Post it on an issue on bugs.python.org. http://bugs.python.org/issue18533 Ben. From zuo at chopin.edu.pl Tue Jul 23 00:30:12 2013 From: zuo at chopin.edu.pl (Jan Kaliszewski) Date: Tue, 23 Jul 2013 00:30:12 +0200 Subject: [Python-Dev] Avoiding error from repr() of recursive dictview In-Reply-To: References: Message-ID: <0f288d04d15c509860c14e6524468a69@chopin.edu.pl> 23.07.2013 00:01, Gregory P. Smith wrote: > On Mon, Jul 22, 2013 at 2:44 PM, Ben North > wrote: > >> A friend of mine, Ruadhan O'Flanagan, came across a bug which turned >> out >> to be the one noted in [http://bugs.python.org/issue18019 [1]], >> i.e.: >> >> >>> d={} >> >>> d[42]=d.viewvalues() >> >>> d >> >> >> This issue has been fixed in hg; the behaviour now is that a >> RuntimeError is produced for a recursive dictionary view: >> >> >>> d={} >> >>> d[42]=d.viewvalues() >> >>> d # (output line-broken:) >> {42: Traceback (most recent call last): >> ? File "", line 1, in >> RuntimeError: maximum recursion depth exceeded >> ? while getting the repr of a list >> >> Before finding this, though, I'd investigated and made a patch which >> produces a similar "..." output to a recursive dictionary. >> ?Reworking >> against current 2.7, the behaviour would be: >> >> >>> x={} >> >>> x[42]=x >> >>> x # existing behaviour for dictionaries: >> {42: {...}} >> >> >>> d={} >> >>> d[42]=d.viewvalues() >> >>> d # new behaviour: >> {42: dict_values([...])} >> >>> d[43]=d.viewitems() >> >>> d # (output line-broken:) >> {42: dict_values([..., dict_items([(42, ...), (43, ...)])]), >> ?43: dict_items([(42, dict_values([..., ...])), (43, ...)])} >> >> Attached is the patch, against current 2.7 branch. ?If there is >> interest >> in applying this, I will create a proper patch (changelog entry, fix >> to >> Lib/test/test_dictviews.py, etc.). > > Mailing lists are where patches go to get lost and die. :) ?Post it > on an issue on bugs.python.org [4]. ?Given that the RuntimeError fix > has been released, your proposed ... behavior is arguably a new > feature so I'd only expect this to make sense for consideration in > 3.4, not 2.7. ?(if accepted at all) IMHO it's still a bug (even though not so painful as segfault) that should also be fixed in 2.7 and 3.2/3.3. In other cases (such as `d={}; d[42]=d; repr(d)`) Python does its best to avoid an error -- why in this case (`d={}; d[42]=d.values(); repr(d)`) should it raise an exception? IMHO it's an obvious oversight in implementation, not a feature that anybody would expect. Regards. *j From victor.stinner at gmail.com Tue Jul 23 01:36:47 2013 From: victor.stinner at gmail.com (Victor Stinner) Date: Tue, 23 Jul 2013 01:36:47 +0200 Subject: [Python-Dev] [Python-checkins] cpython: Fix #18530. Remove extra stat call from posixpath.ismount In-Reply-To: <3bzW5x3YpTz7Lks@mail.python.org> References: <3bzW5x3YpTz7Lks@mail.python.org> Message-ID: Could you please keep the comment "# A symlink can never be a mount point" ? It is useful. (I didn't know that, I'm not a windows developer.) Victor Le 22 juil. 2013 20:08, "brian.curtin" a ?crit : > http://hg.python.org/cpython/rev/240adc564539 > changeset: 84791:240adc564539 > parent: 84788:84d6c1c0665e > user: Brian Curtin > date: Mon Jul 22 13:07:52 2013 -0500 > summary: > Fix #18530. Remove extra stat call from posixpath.ismount > > files: > Lib/posixpath.py | 22 ++++++++++++++-------- > Misc/NEWS | 3 +++ > 2 files changed, 17 insertions(+), 8 deletions(-) > > > diff --git a/Lib/posixpath.py b/Lib/posixpath.py > --- a/Lib/posixpath.py > +++ b/Lib/posixpath.py > @@ -182,18 +182,24 @@ > > def ismount(path): > """Test whether a path is a mount point""" > - if islink(path): > - # A symlink can never be a mount point > - return False > try: > s1 = os.lstat(path) > - if isinstance(path, bytes): > - parent = join(path, b'..') > - else: > - parent = join(path, '..') > + except OSError: > + # It doesn't exist -- so not a mount point. :-) > + return False > + else: > + if stat.S_ISLNK(s1.st_mode): > + return False > + > + if isinstance(path, bytes): > + parent = join(path, b'..') > + else: > + parent = join(path, '..') > + try: > s2 = os.lstat(parent) > except OSError: > - return False # It doesn't exist -- so not a mount point :-) > + return False > + > dev1 = s1.st_dev > dev2 = s2.st_dev > if dev1 != dev2: > diff --git a/Misc/NEWS b/Misc/NEWS > --- a/Misc/NEWS > +++ b/Misc/NEWS > @@ -162,6 +162,9 @@ > Library > ------- > > +- Issue #18530: Remove additional stat call from posixpath.ismount. > + Patch by Alex Gaynor. > + > - Issue #18514: Fix unreachable Py_DECREF() call in PyCData_FromBaseObj() > > - Issue #9177: Calling read() or write() now raises ValueError, not > > -- > Repository URL: http://hg.python.org/cpython > > _______________________________________________ > Python-checkins mailing list > Python-checkins at python.org > http://mail.python.org/mailman/listinfo/python-checkins > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From victor.stinner at gmail.com Tue Jul 23 01:43:30 2013 From: victor.stinner at gmail.com (Victor Stinner) Date: Tue, 23 Jul 2013 01:43:30 +0200 Subject: [Python-Dev] [Python-checkins] cpython: Issue #18520: Add a new PyStructSequence_InitType2() function, same than In-Reply-To: <3bzcCt6wttz7Ljj@mail.python.org> References: <3bzcCt6wttz7Ljj@mail.python.org> Message-ID: "Add a new PyStructSequence_InitType2()" I added a new function because I guess that it would break the API (and ABI) to change the return type of a function in a minor release. Tell me if you have a better name than PyStructSequence_InitType2() ;-) "Ex" suffix is usually used when parameters are added. It is not the case here. Victor Le 22 juil. 2013 23:59, "victor.stinner" a ?crit : > http://hg.python.org/cpython/rev/fc718c177ee6 > changeset: 84793:fc718c177ee6 > user: Victor Stinner > date: Mon Jul 22 22:24:54 2013 +0200 > summary: > Issue #18520: Add a new PyStructSequence_InitType2() function, same than > PyStructSequence_InitType() except that it has a return value (0 on > success, > -1 on error). > > * PyStructSequence_InitType2() now raises MemoryError on memory > allocation failure > * Fix also some calls to PyDict_SetItemString(): handle error > > files: > Include/pythonrun.h | 2 +- > Include/structseq.h | 2 + > Misc/NEWS | 4 +++ > Modules/_lsprof.c | 10 ++++--- > Modules/grpmodule.c | 11 ++++++-- > Modules/posixmodule.c | 24 ++++++++++++------ > Modules/pwdmodule.c | 5 ++- > Modules/resource.c | 9 ++++-- > Modules/signalmodule.c | 7 +++-- > Modules/spwdmodule.c | 8 ++++-- > Modules/timemodule.c | 5 ++- > Objects/floatobject.c | 9 ++++-- > Objects/longobject.c | 6 +++- > Objects/structseq.c | 37 +++++++++++++++++++++-------- > Python/pythonrun.c | 3 +- > Python/sysmodule.c | 23 ++++++++++++----- > Python/thread.c | 6 +++- > 17 files changed, 117 insertions(+), 54 deletions(-) > > > diff --git a/Include/pythonrun.h b/Include/pythonrun.h > --- a/Include/pythonrun.h > +++ b/Include/pythonrun.h > @@ -197,7 +197,7 @@ > PyAPI_FUNC(void) _PyExc_Init(PyObject * bltinmod); > PyAPI_FUNC(void) _PyImportHooks_Init(void); > PyAPI_FUNC(int) _PyFrame_Init(void); > -PyAPI_FUNC(void) _PyFloat_Init(void); > +PyAPI_FUNC(int) _PyFloat_Init(void); > PyAPI_FUNC(int) PyByteArray_Init(void); > PyAPI_FUNC(void) _PyRandom_Init(void); > #endif > diff --git a/Include/structseq.h b/Include/structseq.h > --- a/Include/structseq.h > +++ b/Include/structseq.h > @@ -24,6 +24,8 @@ > #ifndef Py_LIMITED_API > PyAPI_FUNC(void) PyStructSequence_InitType(PyTypeObject *type, > PyStructSequence_Desc *desc); > +PyAPI_FUNC(int) PyStructSequence_InitType2(PyTypeObject *type, > + PyStructSequence_Desc *desc); > #endif > PyAPI_FUNC(PyTypeObject*) PyStructSequence_NewType(PyStructSequence_Desc > *desc); > > diff --git a/Misc/NEWS b/Misc/NEWS > --- a/Misc/NEWS > +++ b/Misc/NEWS > @@ -10,6 +10,10 @@ > Core and Builtins > ----------------- > > +- Issue #18520: Add a new PyStructSequence_InitType2() function, same than > + PyStructSequence_InitType() except that it has a return value (0 on > success, > + -1 on error). > + > - Issue #15905: Fix theoretical buffer overflow in handling of > sys.argv[0], > prefix and exec_prefix if the operation system does not obey MAXPATHLEN. > > diff --git a/Modules/_lsprof.c b/Modules/_lsprof.c > --- a/Modules/_lsprof.c > +++ b/Modules/_lsprof.c > @@ -884,10 +884,12 @@ > PyDict_SetItemString(d, "Profiler", (PyObject *)&PyProfiler_Type); > > if (!initialized) { > - PyStructSequence_InitType(&StatsEntryType, > - &profiler_entry_desc); > - PyStructSequence_InitType(&StatsSubEntryType, > - &profiler_subentry_desc); > + if (PyStructSequence_InitType2(&StatsEntryType, > + &profiler_entry_desc) < 0) > + return NULL; > + if (PyStructSequence_InitType2(&StatsSubEntryType, > + &profiler_subentry_desc) < 0) > + return NULL; > } > Py_INCREF((PyObject*) &StatsEntryType); > Py_INCREF((PyObject*) &StatsSubEntryType); > diff --git a/Modules/grpmodule.c b/Modules/grpmodule.c > --- a/Modules/grpmodule.c > +++ b/Modules/grpmodule.c > @@ -210,9 +210,14 @@ > if (m == NULL) > return NULL; > d = PyModule_GetDict(m); > - if (!initialized) > - PyStructSequence_InitType(&StructGrpType, > &struct_group_type_desc); > - PyDict_SetItemString(d, "struct_group", (PyObject *) &StructGrpType); > + if (!initialized) { > + if (PyStructSequence_InitType2(&StructGrpType, > + &struct_group_type_desc) < 0) > + return NULL; > + } > + if (PyDict_SetItemString(d, "struct_group", > + (PyObject *)&StructGrpType) < 0) > + return NULL; > initialized = 1; > return m; > } > diff --git a/Modules/posixmodule.c b/Modules/posixmodule.c > --- a/Modules/posixmodule.c > +++ b/Modules/posixmodule.c > @@ -11518,19 +11518,23 @@ > if (!initialized) { > #if defined(HAVE_WAITID) && !defined(__APPLE__) > waitid_result_desc.name = MODNAME ".waitid_result"; > - PyStructSequence_InitType(&WaitidResultType, &waitid_result_desc); > + if (PyStructSequence_InitType2(&WaitidResultType, > &waitid_result_desc) < 0) > + return NULL; > #endif > > stat_result_desc.name = MODNAME ".stat_result"; > stat_result_desc.fields[7].name = PyStructSequence_UnnamedField; > stat_result_desc.fields[8].name = PyStructSequence_UnnamedField; > stat_result_desc.fields[9].name = PyStructSequence_UnnamedField; > - PyStructSequence_InitType(&StatResultType, &stat_result_desc); > + if (PyStructSequence_InitType2(&StatResultType, > &stat_result_desc) < 0) > + return NULL; > structseq_new = StatResultType.tp_new; > StatResultType.tp_new = statresult_new; > > statvfs_result_desc.name = MODNAME ".statvfs_result"; > - PyStructSequence_InitType(&StatVFSResultType, > &statvfs_result_desc); > + if (PyStructSequence_InitType2(&StatVFSResultType, > + &statvfs_result_desc) < 0) > + return NULL; > #ifdef NEED_TICKS_PER_SECOND > # if defined(HAVE_SYSCONF) && defined(_SC_CLK_TCK) > ticks_per_second = sysconf(_SC_CLK_TCK); > @@ -11543,12 +11547,15 @@ > > #if defined(HAVE_SCHED_SETPARAM) || defined(HAVE_SCHED_SETSCHEDULER) > sched_param_desc.name = MODNAME ".sched_param"; > - PyStructSequence_InitType(&SchedParamType, &sched_param_desc); > + if (PyStructSequence_InitType2(&SchedParamType, > &sched_param_desc) < 0) > + return NULL; > SchedParamType.tp_new = sched_param_new; > #endif > > /* initialize TerminalSize_info */ > - PyStructSequence_InitType(&TerminalSizeType, &TerminalSize_desc); > + if (PyStructSequence_InitType2(&TerminalSizeType, > + &TerminalSize_desc) < 0) > + return NULL; > } > #if defined(HAVE_WAITID) && !defined(__APPLE__) > Py_INCREF((PyObject*) &WaitidResultType); > @@ -11566,11 +11573,13 @@ > #endif > > times_result_desc.name = MODNAME ".times_result"; > - PyStructSequence_InitType(&TimesResultType, ×_result_desc); > + if (PyStructSequence_InitType2(&TimesResultType, ×_result_desc) > < 0) > + return NULL; > PyModule_AddObject(m, "times_result", (PyObject *)&TimesResultType); > > uname_result_desc.name = MODNAME ".uname_result"; > - PyStructSequence_InitType(&UnameResultType, &uname_result_desc); > + if (PyStructSequence_InitType2(&UnameResultType, &uname_result_desc) > < 0) > + return NULL; > PyModule_AddObject(m, "uname_result", (PyObject *)&UnameResultType); > > #ifdef __APPLE__ > @@ -11648,7 +11657,6 @@ > initialized = 1; > > return m; > - > } > > #ifdef __cplusplus > diff --git a/Modules/pwdmodule.c b/Modules/pwdmodule.c > --- a/Modules/pwdmodule.c > +++ b/Modules/pwdmodule.c > @@ -216,8 +216,9 @@ > return NULL; > > if (!initialized) { > - PyStructSequence_InitType(&StructPwdType, > - &struct_pwd_type_desc); > + if (PyStructSequence_InitType2(&StructPwdType, > + &struct_pwd_type_desc) < 0) > + return NULL; > initialized = 1; > } > Py_INCREF((PyObject *) &StructPwdType); > diff --git a/Modules/resource.c b/Modules/resource.c > --- a/Modules/resource.c > +++ b/Modules/resource.c > @@ -263,9 +263,12 @@ > /* Add some symbolic constants to the module */ > Py_INCREF(PyExc_OSError); > PyModule_AddObject(m, "error", PyExc_OSError); > - if (!initialized) > - PyStructSequence_InitType(&StructRUsageType, > - &struct_rusage_desc); > + if (!initialized) { > + if (PyStructSequence_InitType2(&StructRUsageType, > + &struct_rusage_desc) < 0) > + return NULL; > + } > + > Py_INCREF(&StructRUsageType); > PyModule_AddObject(m, "struct_rusage", > (PyObject*) &StructRUsageType); > diff --git a/Modules/signalmodule.c b/Modules/signalmodule.c > --- a/Modules/signalmodule.c > +++ b/Modules/signalmodule.c > @@ -978,9 +978,10 @@ > return NULL; > > #if defined(HAVE_SIGWAITINFO) || defined(HAVE_SIGTIMEDWAIT) > - if (!initialized) > - PyStructSequence_InitType(&SiginfoType, &struct_siginfo_desc); > - > + if (!initialized) { > + if (PyStructSequence_InitType2(&SiginfoType, > &struct_siginfo_desc) < 0) > + return NULL; > + } > Py_INCREF((PyObject*) &SiginfoType); > PyModule_AddObject(m, "struct_siginfo", (PyObject*) &SiginfoType); > initialized = 1; > diff --git a/Modules/spwdmodule.c b/Modules/spwdmodule.c > --- a/Modules/spwdmodule.c > +++ b/Modules/spwdmodule.c > @@ -196,9 +196,11 @@ > m=PyModule_Create(&spwdmodule); > if (m == NULL) > return NULL; > - if (!initialized) > - PyStructSequence_InitType(&StructSpwdType, > - &struct_spwd_type_desc); > + if (!initialized) { > + if (PyStructSequence_InitType2(&StructSpwdType, > + &struct_spwd_type_desc) < 0) > + return NULL; > + } > Py_INCREF((PyObject *) &StructSpwdType); > PyModule_AddObject(m, "struct_spwd", (PyObject *) &StructSpwdType); > initialized = 1; > diff --git a/Modules/timemodule.c b/Modules/timemodule.c > --- a/Modules/timemodule.c > +++ b/Modules/timemodule.c > @@ -1476,8 +1476,9 @@ > PyInit_timezone(m); > > if (!initialized) { > - PyStructSequence_InitType(&StructTimeType, > - &struct_time_type_desc); > + if (PyStructSequence_InitType2(&StructTimeType, > + &struct_time_type_desc) < 0) > + return NULL; > > #ifdef MS_WINDOWS > winver.dwOSVersionInfoSize = sizeof(winver); > diff --git a/Objects/floatobject.c b/Objects/floatobject.c > --- a/Objects/floatobject.c > +++ b/Objects/floatobject.c > @@ -1853,7 +1853,7 @@ > float_new, /* tp_new */ > }; > > -void > +int > _PyFloat_Init(void) > { > /* We attempt to determine if this machine is using IEEE > @@ -1903,8 +1903,11 @@ > float_format = detected_float_format; > > /* Init float info */ > - if (FloatInfoType.tp_name == 0) > - PyStructSequence_InitType(&FloatInfoType, &floatinfo_desc); > + if (FloatInfoType.tp_name == NULL) { > + if (PyStructSequence_InitType2(&FloatInfoType, &floatinfo_desc) < > 0) > + return 0; > + } > + return 1; > } > > int > diff --git a/Objects/longobject.c b/Objects/longobject.c > --- a/Objects/longobject.c > +++ b/Objects/longobject.c > @@ -5059,8 +5059,10 @@ > } > #endif > /* initialize int_info */ > - if (Int_InfoType.tp_name == 0) > - PyStructSequence_InitType(&Int_InfoType, &int_info_desc); > + if (Int_InfoType.tp_name == NULL) { > + if (PyStructSequence_InitType2(&Int_InfoType, &int_info_desc) < 0) > + return 0; > + } > > return 1; > } > diff --git a/Objects/structseq.c b/Objects/structseq.c > --- a/Objects/structseq.c > +++ b/Objects/structseq.c > @@ -320,12 +320,13 @@ > structseq_new, /* tp_new */ > }; > > -void > -PyStructSequence_InitType(PyTypeObject *type, PyStructSequence_Desc *desc) > +int > +PyStructSequence_InitType2(PyTypeObject *type, PyStructSequence_Desc > *desc) > { > PyObject *dict; > PyMemberDef* members; > int n_members, n_unnamed_members, i, k; > + PyObject *v; > > #ifdef Py_TRACE_REFS > /* if the type object was chained, unchain it first > @@ -347,8 +348,10 @@ > type->tp_doc = desc->doc; > > members = PyMem_NEW(PyMemberDef, n_members-n_unnamed_members+1); > - if (members == NULL) > - return; > + if (members == NULL) { > + PyErr_NoMemory(); > + return -1; > + } > > for (i = k = 0; i < n_members; ++i) { > if (desc->fields[i].name == PyStructSequence_UnnamedField) > @@ -366,22 +369,33 @@ > type->tp_members = members; > > if (PyType_Ready(type) < 0) > - return; > + return -1; > Py_INCREF(type); > > dict = type->tp_dict; > #define SET_DICT_FROM_INT(key, value) \ > do { \ > - PyObject *v = PyLong_FromLong((long) value); \ > - if (v != NULL) { \ > - PyDict_SetItemString(dict, key, v); \ > + v = PyLong_FromLong((long) value); \ > + if (v == NULL) \ > + return -1; \ > + if (PyDict_SetItemString(dict, key, v) < 0) { \ > Py_DECREF(v); \ > + return -1; \ > } \ > + Py_DECREF(v); \ > } while (0) > > SET_DICT_FROM_INT(visible_length_key, desc->n_in_sequence); > SET_DICT_FROM_INT(real_length_key, n_members); > SET_DICT_FROM_INT(unnamed_fields_key, n_unnamed_members); > + > + return 0; > +} > + > +void > +PyStructSequence_InitType(PyTypeObject *type, PyStructSequence_Desc *desc) > +{ > + (void)PyStructSequence_InitType2(type, desc); > } > > PyTypeObject* > @@ -390,8 +404,11 @@ > PyTypeObject *result; > > result = (PyTypeObject*)PyType_GenericAlloc(&PyType_Type, 0); > - if (result != NULL) { > - PyStructSequence_InitType(result, desc); > + if (result == NULL) > + return NULL; > + if (PyStructSequence_InitType2(result, desc) < 0) { > + Py_DECREF(result); > + return NULL; > } > return result; > } > diff --git a/Python/pythonrun.c b/Python/pythonrun.c > --- a/Python/pythonrun.c > +++ b/Python/pythonrun.c > @@ -328,7 +328,8 @@ > if (!PyByteArray_Init()) > Py_FatalError("Py_Initialize: can't init bytearray"); > > - _PyFloat_Init(); > + if (!_PyFloat_Init()) > + Py_FatalError("Py_Initialize: can't init float"); > > interp->modules = PyDict_New(); > if (interp->modules == NULL) > diff --git a/Python/sysmodule.c b/Python/sysmodule.c > --- a/Python/sysmodule.c > +++ b/Python/sysmodule.c > @@ -1634,8 +1634,10 @@ > SET_SYS_FROM_STRING("int_info", > PyLong_GetInfo()); > /* initialize hash_info */ > - if (Hash_InfoType.tp_name == 0) > - PyStructSequence_InitType(&Hash_InfoType, &hash_info_desc); > + if (Hash_InfoType.tp_name == NULL) { > + if (PyStructSequence_InitType2(&Hash_InfoType, &hash_info_desc) < > 0) > + return NULL; > + } > SET_SYS_FROM_STRING("hash_info", > get_hash_info()); > SET_SYS_FROM_STRING("maxunicode", > @@ -1676,8 +1678,11 @@ > } > > /* version_info */ > - if (VersionInfoType.tp_name == 0) > - PyStructSequence_InitType(&VersionInfoType, &version_info_desc); > + if (VersionInfoType.tp_name == NULL) { > + if (PyStructSequence_InitType2(&VersionInfoType, > + &version_info_desc) < 0) > + return NULL; > + } > version_info = make_version_info(); > SET_SYS_FROM_STRING("version_info", version_info); > /* prevent user from creating new instances */ > @@ -1688,8 +1693,10 @@ > SET_SYS_FROM_STRING("implementation", make_impl_info(version_info)); > > /* flags */ > - if (FlagsType.tp_name == 0) > - PyStructSequence_InitType(&FlagsType, &flags_desc); > + if (FlagsType.tp_name == 0) { > + if (PyStructSequence_InitType2(&FlagsType, &flags_desc) < 0) > + return NULL; > + } > SET_SYS_FROM_STRING("flags", make_flags()); > /* prevent user from creating new instances */ > FlagsType.tp_init = NULL; > @@ -1699,7 +1706,9 @@ > #if defined(MS_WINDOWS) > /* getwindowsversion */ > if (WindowsVersionType.tp_name == 0) > - PyStructSequence_InitType(&WindowsVersionType, > &windows_version_desc); > + if (PyStructSequence_InitType2(&WindowsVersionType, > + &windows_version_desc) < 0) > + return NULL; > /* prevent user from creating new instances */ > WindowsVersionType.tp_init = NULL; > WindowsVersionType.tp_new = NULL; > diff --git a/Python/thread.c b/Python/thread.c > --- a/Python/thread.c > +++ b/Python/thread.c > @@ -399,8 +399,10 @@ > int len; > #endif > > - if (ThreadInfoType.tp_name == 0) > - PyStructSequence_InitType(&ThreadInfoType, &threadinfo_desc); > + if (ThreadInfoType.tp_name == 0) { > + if (PyStructSequence_InitType2(&ThreadInfoType, &threadinfo_desc) > < 0) > + return NULL; > + } > > threadinfo = PyStructSequence_New(&ThreadInfoType); > if (threadinfo == NULL) > > -- > Repository URL: http://hg.python.org/cpython > > _______________________________________________ > Python-checkins mailing list > Python-checkins at python.org > http://mail.python.org/mailman/listinfo/python-checkins > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From benjamin at python.org Tue Jul 23 02:01:46 2013 From: benjamin at python.org (Benjamin Peterson) Date: Mon, 22 Jul 2013 17:01:46 -0700 Subject: [Python-Dev] [Python-checkins] cpython: Issue #18520: Add a new PyStructSequence_InitType2() function, same than In-Reply-To: References: <3bzcCt6wttz7Ljj@mail.python.org> Message-ID: We've cheerfully broken the ABI before on minor releases, though if it's part of the stable ABI, we can't be cavaliar about that anymore. 2013/7/22 Victor Stinner : > "Add a new PyStructSequence_InitType2()" > > I added a new function because I guess that it would break the API (and ABI) > to change the return type of a function in a minor release. > > Tell me if you have a better name than PyStructSequence_InitType2() ;-) > > "Ex" suffix is usually used when parameters are added. It is not the case > here. > > Victor > > Le 22 juil. 2013 23:59, "victor.stinner" a > ?crit : >> >> http://hg.python.org/cpython/rev/fc718c177ee6 >> changeset: 84793:fc718c177ee6 >> user: Victor Stinner >> date: Mon Jul 22 22:24:54 2013 +0200 >> summary: >> Issue #18520: Add a new PyStructSequence_InitType2() function, same than >> PyStructSequence_InitType() except that it has a return value (0 on >> success, >> -1 on error). >> >> * PyStructSequence_InitType2() now raises MemoryError on memory >> allocation failure >> * Fix also some calls to PyDict_SetItemString(): handle error >> >> files: >> Include/pythonrun.h | 2 +- >> Include/structseq.h | 2 + >> Misc/NEWS | 4 +++ >> Modules/_lsprof.c | 10 ++++--- >> Modules/grpmodule.c | 11 ++++++-- >> Modules/posixmodule.c | 24 ++++++++++++------ >> Modules/pwdmodule.c | 5 ++- >> Modules/resource.c | 9 ++++-- >> Modules/signalmodule.c | 7 +++-- >> Modules/spwdmodule.c | 8 ++++-- >> Modules/timemodule.c | 5 ++- >> Objects/floatobject.c | 9 ++++-- >> Objects/longobject.c | 6 +++- >> Objects/structseq.c | 37 +++++++++++++++++++++-------- >> Python/pythonrun.c | 3 +- >> Python/sysmodule.c | 23 ++++++++++++----- >> Python/thread.c | 6 +++- >> 17 files changed, 117 insertions(+), 54 deletions(-) >> >> >> diff --git a/Include/pythonrun.h b/Include/pythonrun.h >> --- a/Include/pythonrun.h >> +++ b/Include/pythonrun.h >> @@ -197,7 +197,7 @@ >> PyAPI_FUNC(void) _PyExc_Init(PyObject * bltinmod); >> PyAPI_FUNC(void) _PyImportHooks_Init(void); >> PyAPI_FUNC(int) _PyFrame_Init(void); >> -PyAPI_FUNC(void) _PyFloat_Init(void); >> +PyAPI_FUNC(int) _PyFloat_Init(void); >> PyAPI_FUNC(int) PyByteArray_Init(void); >> PyAPI_FUNC(void) _PyRandom_Init(void); >> #endif >> diff --git a/Include/structseq.h b/Include/structseq.h >> --- a/Include/structseq.h >> +++ b/Include/structseq.h >> @@ -24,6 +24,8 @@ >> #ifndef Py_LIMITED_API >> PyAPI_FUNC(void) PyStructSequence_InitType(PyTypeObject *type, >> PyStructSequence_Desc *desc); >> +PyAPI_FUNC(int) PyStructSequence_InitType2(PyTypeObject *type, >> + PyStructSequence_Desc *desc); >> #endif >> PyAPI_FUNC(PyTypeObject*) PyStructSequence_NewType(PyStructSequence_Desc >> *desc); >> >> diff --git a/Misc/NEWS b/Misc/NEWS >> --- a/Misc/NEWS >> +++ b/Misc/NEWS >> @@ -10,6 +10,10 @@ >> Core and Builtins >> ----------------- >> >> +- Issue #18520: Add a new PyStructSequence_InitType2() function, same >> than >> + PyStructSequence_InitType() except that it has a return value (0 on >> success, >> + -1 on error). >> + >> - Issue #15905: Fix theoretical buffer overflow in handling of >> sys.argv[0], >> prefix and exec_prefix if the operation system does not obey >> MAXPATHLEN. >> >> diff --git a/Modules/_lsprof.c b/Modules/_lsprof.c >> --- a/Modules/_lsprof.c >> +++ b/Modules/_lsprof.c >> @@ -884,10 +884,12 @@ >> PyDict_SetItemString(d, "Profiler", (PyObject *)&PyProfiler_Type); >> >> if (!initialized) { >> - PyStructSequence_InitType(&StatsEntryType, >> - &profiler_entry_desc); >> - PyStructSequence_InitType(&StatsSubEntryType, >> - &profiler_subentry_desc); >> + if (PyStructSequence_InitType2(&StatsEntryType, >> + &profiler_entry_desc) < 0) >> + return NULL; >> + if (PyStructSequence_InitType2(&StatsSubEntryType, >> + &profiler_subentry_desc) < 0) >> + return NULL; >> } >> Py_INCREF((PyObject*) &StatsEntryType); >> Py_INCREF((PyObject*) &StatsSubEntryType); >> diff --git a/Modules/grpmodule.c b/Modules/grpmodule.c >> --- a/Modules/grpmodule.c >> +++ b/Modules/grpmodule.c >> @@ -210,9 +210,14 @@ >> if (m == NULL) >> return NULL; >> d = PyModule_GetDict(m); >> - if (!initialized) >> - PyStructSequence_InitType(&StructGrpType, >> &struct_group_type_desc); >> - PyDict_SetItemString(d, "struct_group", (PyObject *) &StructGrpType); >> + if (!initialized) { >> + if (PyStructSequence_InitType2(&StructGrpType, >> + &struct_group_type_desc) < 0) >> + return NULL; >> + } >> + if (PyDict_SetItemString(d, "struct_group", >> + (PyObject *)&StructGrpType) < 0) >> + return NULL; >> initialized = 1; >> return m; >> } >> diff --git a/Modules/posixmodule.c b/Modules/posixmodule.c >> --- a/Modules/posixmodule.c >> +++ b/Modules/posixmodule.c >> @@ -11518,19 +11518,23 @@ >> if (!initialized) { >> #if defined(HAVE_WAITID) && !defined(__APPLE__) >> waitid_result_desc.name = MODNAME ".waitid_result"; >> - PyStructSequence_InitType(&WaitidResultType, >> &waitid_result_desc); >> + if (PyStructSequence_InitType2(&WaitidResultType, >> &waitid_result_desc) < 0) >> + return NULL; >> #endif >> >> stat_result_desc.name = MODNAME ".stat_result"; >> stat_result_desc.fields[7].name = PyStructSequence_UnnamedField; >> stat_result_desc.fields[8].name = PyStructSequence_UnnamedField; >> stat_result_desc.fields[9].name = PyStructSequence_UnnamedField; >> - PyStructSequence_InitType(&StatResultType, &stat_result_desc); >> + if (PyStructSequence_InitType2(&StatResultType, >> &stat_result_desc) < 0) >> + return NULL; >> structseq_new = StatResultType.tp_new; >> StatResultType.tp_new = statresult_new; >> >> statvfs_result_desc.name = MODNAME ".statvfs_result"; >> - PyStructSequence_InitType(&StatVFSResultType, >> &statvfs_result_desc); >> + if (PyStructSequence_InitType2(&StatVFSResultType, >> + &statvfs_result_desc) < 0) >> + return NULL; >> #ifdef NEED_TICKS_PER_SECOND >> # if defined(HAVE_SYSCONF) && defined(_SC_CLK_TCK) >> ticks_per_second = sysconf(_SC_CLK_TCK); >> @@ -11543,12 +11547,15 @@ >> >> #if defined(HAVE_SCHED_SETPARAM) || defined(HAVE_SCHED_SETSCHEDULER) >> sched_param_desc.name = MODNAME ".sched_param"; >> - PyStructSequence_InitType(&SchedParamType, &sched_param_desc); >> + if (PyStructSequence_InitType2(&SchedParamType, >> &sched_param_desc) < 0) >> + return NULL; >> SchedParamType.tp_new = sched_param_new; >> #endif >> >> /* initialize TerminalSize_info */ >> - PyStructSequence_InitType(&TerminalSizeType, &TerminalSize_desc); >> + if (PyStructSequence_InitType2(&TerminalSizeType, >> + &TerminalSize_desc) < 0) >> + return NULL; >> } >> #if defined(HAVE_WAITID) && !defined(__APPLE__) >> Py_INCREF((PyObject*) &WaitidResultType); >> @@ -11566,11 +11573,13 @@ >> #endif >> >> times_result_desc.name = MODNAME ".times_result"; >> - PyStructSequence_InitType(&TimesResultType, ×_result_desc); >> + if (PyStructSequence_InitType2(&TimesResultType, ×_result_desc) >> < 0) >> + return NULL; >> PyModule_AddObject(m, "times_result", (PyObject *)&TimesResultType); >> >> uname_result_desc.name = MODNAME ".uname_result"; >> - PyStructSequence_InitType(&UnameResultType, &uname_result_desc); >> + if (PyStructSequence_InitType2(&UnameResultType, &uname_result_desc) >> < 0) >> + return NULL; >> PyModule_AddObject(m, "uname_result", (PyObject *)&UnameResultType); >> >> #ifdef __APPLE__ >> @@ -11648,7 +11657,6 @@ >> initialized = 1; >> >> return m; >> - >> } >> >> #ifdef __cplusplus >> diff --git a/Modules/pwdmodule.c b/Modules/pwdmodule.c >> --- a/Modules/pwdmodule.c >> +++ b/Modules/pwdmodule.c >> @@ -216,8 +216,9 @@ >> return NULL; >> >> if (!initialized) { >> - PyStructSequence_InitType(&StructPwdType, >> - &struct_pwd_type_desc); >> + if (PyStructSequence_InitType2(&StructPwdType, >> + &struct_pwd_type_desc) < 0) >> + return NULL; >> initialized = 1; >> } >> Py_INCREF((PyObject *) &StructPwdType); >> diff --git a/Modules/resource.c b/Modules/resource.c >> --- a/Modules/resource.c >> +++ b/Modules/resource.c >> @@ -263,9 +263,12 @@ >> /* Add some symbolic constants to the module */ >> Py_INCREF(PyExc_OSError); >> PyModule_AddObject(m, "error", PyExc_OSError); >> - if (!initialized) >> - PyStructSequence_InitType(&StructRUsageType, >> - &struct_rusage_desc); >> + if (!initialized) { >> + if (PyStructSequence_InitType2(&StructRUsageType, >> + &struct_rusage_desc) < 0) >> + return NULL; >> + } >> + >> Py_INCREF(&StructRUsageType); >> PyModule_AddObject(m, "struct_rusage", >> (PyObject*) &StructRUsageType); >> diff --git a/Modules/signalmodule.c b/Modules/signalmodule.c >> --- a/Modules/signalmodule.c >> +++ b/Modules/signalmodule.c >> @@ -978,9 +978,10 @@ >> return NULL; >> >> #if defined(HAVE_SIGWAITINFO) || defined(HAVE_SIGTIMEDWAIT) >> - if (!initialized) >> - PyStructSequence_InitType(&SiginfoType, &struct_siginfo_desc); >> - >> + if (!initialized) { >> + if (PyStructSequence_InitType2(&SiginfoType, >> &struct_siginfo_desc) < 0) >> + return NULL; >> + } >> Py_INCREF((PyObject*) &SiginfoType); >> PyModule_AddObject(m, "struct_siginfo", (PyObject*) &SiginfoType); >> initialized = 1; >> diff --git a/Modules/spwdmodule.c b/Modules/spwdmodule.c >> --- a/Modules/spwdmodule.c >> +++ b/Modules/spwdmodule.c >> @@ -196,9 +196,11 @@ >> m=PyModule_Create(&spwdmodule); >> if (m == NULL) >> return NULL; >> - if (!initialized) >> - PyStructSequence_InitType(&StructSpwdType, >> - &struct_spwd_type_desc); >> + if (!initialized) { >> + if (PyStructSequence_InitType2(&StructSpwdType, >> + &struct_spwd_type_desc) < 0) >> + return NULL; >> + } >> Py_INCREF((PyObject *) &StructSpwdType); >> PyModule_AddObject(m, "struct_spwd", (PyObject *) &StructSpwdType); >> initialized = 1; >> diff --git a/Modules/timemodule.c b/Modules/timemodule.c >> --- a/Modules/timemodule.c >> +++ b/Modules/timemodule.c >> @@ -1476,8 +1476,9 @@ >> PyInit_timezone(m); >> >> if (!initialized) { >> - PyStructSequence_InitType(&StructTimeType, >> - &struct_time_type_desc); >> + if (PyStructSequence_InitType2(&StructTimeType, >> + &struct_time_type_desc) < 0) >> + return NULL; >> >> #ifdef MS_WINDOWS >> winver.dwOSVersionInfoSize = sizeof(winver); >> diff --git a/Objects/floatobject.c b/Objects/floatobject.c >> --- a/Objects/floatobject.c >> +++ b/Objects/floatobject.c >> @@ -1853,7 +1853,7 @@ >> float_new, /* tp_new */ >> }; >> >> -void >> +int >> _PyFloat_Init(void) >> { >> /* We attempt to determine if this machine is using IEEE >> @@ -1903,8 +1903,11 @@ >> float_format = detected_float_format; >> >> /* Init float info */ >> - if (FloatInfoType.tp_name == 0) >> - PyStructSequence_InitType(&FloatInfoType, &floatinfo_desc); >> + if (FloatInfoType.tp_name == NULL) { >> + if (PyStructSequence_InitType2(&FloatInfoType, &floatinfo_desc) < >> 0) >> + return 0; >> + } >> + return 1; >> } >> >> int >> diff --git a/Objects/longobject.c b/Objects/longobject.c >> --- a/Objects/longobject.c >> +++ b/Objects/longobject.c >> @@ -5059,8 +5059,10 @@ >> } >> #endif >> /* initialize int_info */ >> - if (Int_InfoType.tp_name == 0) >> - PyStructSequence_InitType(&Int_InfoType, &int_info_desc); >> + if (Int_InfoType.tp_name == NULL) { >> + if (PyStructSequence_InitType2(&Int_InfoType, &int_info_desc) < >> 0) >> + return 0; >> + } >> >> return 1; >> } >> diff --git a/Objects/structseq.c b/Objects/structseq.c >> --- a/Objects/structseq.c >> +++ b/Objects/structseq.c >> @@ -320,12 +320,13 @@ >> structseq_new, /* tp_new */ >> }; >> >> -void >> -PyStructSequence_InitType(PyTypeObject *type, PyStructSequence_Desc >> *desc) >> +int >> +PyStructSequence_InitType2(PyTypeObject *type, PyStructSequence_Desc >> *desc) >> { >> PyObject *dict; >> PyMemberDef* members; >> int n_members, n_unnamed_members, i, k; >> + PyObject *v; >> >> #ifdef Py_TRACE_REFS >> /* if the type object was chained, unchain it first >> @@ -347,8 +348,10 @@ >> type->tp_doc = desc->doc; >> >> members = PyMem_NEW(PyMemberDef, n_members-n_unnamed_members+1); >> - if (members == NULL) >> - return; >> + if (members == NULL) { >> + PyErr_NoMemory(); >> + return -1; >> + } >> >> for (i = k = 0; i < n_members; ++i) { >> if (desc->fields[i].name == PyStructSequence_UnnamedField) >> @@ -366,22 +369,33 @@ >> type->tp_members = members; >> >> if (PyType_Ready(type) < 0) >> - return; >> + return -1; >> Py_INCREF(type); >> >> dict = type->tp_dict; >> #define SET_DICT_FROM_INT(key, value) \ >> do { \ >> - PyObject *v = PyLong_FromLong((long) value); \ >> - if (v != NULL) { \ >> - PyDict_SetItemString(dict, key, v); \ >> + v = PyLong_FromLong((long) value); \ >> + if (v == NULL) \ >> + return -1; \ >> + if (PyDict_SetItemString(dict, key, v) < 0) { \ >> Py_DECREF(v); \ >> + return -1; \ >> } \ >> + Py_DECREF(v); \ >> } while (0) >> >> SET_DICT_FROM_INT(visible_length_key, desc->n_in_sequence); >> SET_DICT_FROM_INT(real_length_key, n_members); >> SET_DICT_FROM_INT(unnamed_fields_key, n_unnamed_members); >> + >> + return 0; >> +} >> + >> +void >> +PyStructSequence_InitType(PyTypeObject *type, PyStructSequence_Desc >> *desc) >> +{ >> + (void)PyStructSequence_InitType2(type, desc); >> } >> >> PyTypeObject* >> @@ -390,8 +404,11 @@ >> PyTypeObject *result; >> >> result = (PyTypeObject*)PyType_GenericAlloc(&PyType_Type, 0); >> - if (result != NULL) { >> - PyStructSequence_InitType(result, desc); >> + if (result == NULL) >> + return NULL; >> + if (PyStructSequence_InitType2(result, desc) < 0) { >> + Py_DECREF(result); >> + return NULL; >> } >> return result; >> } >> diff --git a/Python/pythonrun.c b/Python/pythonrun.c >> --- a/Python/pythonrun.c >> +++ b/Python/pythonrun.c >> @@ -328,7 +328,8 @@ >> if (!PyByteArray_Init()) >> Py_FatalError("Py_Initialize: can't init bytearray"); >> >> - _PyFloat_Init(); >> + if (!_PyFloat_Init()) >> + Py_FatalError("Py_Initialize: can't init float"); >> >> interp->modules = PyDict_New(); >> if (interp->modules == NULL) >> diff --git a/Python/sysmodule.c b/Python/sysmodule.c >> --- a/Python/sysmodule.c >> +++ b/Python/sysmodule.c >> @@ -1634,8 +1634,10 @@ >> SET_SYS_FROM_STRING("int_info", >> PyLong_GetInfo()); >> /* initialize hash_info */ >> - if (Hash_InfoType.tp_name == 0) >> - PyStructSequence_InitType(&Hash_InfoType, &hash_info_desc); >> + if (Hash_InfoType.tp_name == NULL) { >> + if (PyStructSequence_InitType2(&Hash_InfoType, &hash_info_desc) < >> 0) >> + return NULL; >> + } >> SET_SYS_FROM_STRING("hash_info", >> get_hash_info()); >> SET_SYS_FROM_STRING("maxunicode", >> @@ -1676,8 +1678,11 @@ >> } >> >> /* version_info */ >> - if (VersionInfoType.tp_name == 0) >> - PyStructSequence_InitType(&VersionInfoType, &version_info_desc); >> + if (VersionInfoType.tp_name == NULL) { >> + if (PyStructSequence_InitType2(&VersionInfoType, >> + &version_info_desc) < 0) >> + return NULL; >> + } >> version_info = make_version_info(); >> SET_SYS_FROM_STRING("version_info", version_info); >> /* prevent user from creating new instances */ >> @@ -1688,8 +1693,10 @@ >> SET_SYS_FROM_STRING("implementation", make_impl_info(version_info)); >> >> /* flags */ >> - if (FlagsType.tp_name == 0) >> - PyStructSequence_InitType(&FlagsType, &flags_desc); >> + if (FlagsType.tp_name == 0) { >> + if (PyStructSequence_InitType2(&FlagsType, &flags_desc) < 0) >> + return NULL; >> + } >> SET_SYS_FROM_STRING("flags", make_flags()); >> /* prevent user from creating new instances */ >> FlagsType.tp_init = NULL; >> @@ -1699,7 +1706,9 @@ >> #if defined(MS_WINDOWS) >> /* getwindowsversion */ >> if (WindowsVersionType.tp_name == 0) >> - PyStructSequence_InitType(&WindowsVersionType, >> &windows_version_desc); >> + if (PyStructSequence_InitType2(&WindowsVersionType, >> + &windows_version_desc) < 0) >> + return NULL; >> /* prevent user from creating new instances */ >> WindowsVersionType.tp_init = NULL; >> WindowsVersionType.tp_new = NULL; >> diff --git a/Python/thread.c b/Python/thread.c >> --- a/Python/thread.c >> +++ b/Python/thread.c >> @@ -399,8 +399,10 @@ >> int len; >> #endif >> >> - if (ThreadInfoType.tp_name == 0) >> - PyStructSequence_InitType(&ThreadInfoType, &threadinfo_desc); >> + if (ThreadInfoType.tp_name == 0) { >> + if (PyStructSequence_InitType2(&ThreadInfoType, &threadinfo_desc) >> < 0) >> + return NULL; >> + } >> >> threadinfo = PyStructSequence_New(&ThreadInfoType); >> if (threadinfo == NULL) >> >> -- >> Repository URL: http://hg.python.org/cpython >> >> _______________________________________________ >> Python-checkins mailing list >> Python-checkins at python.org >> http://mail.python.org/mailman/listinfo/python-checkins >> > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/benjamin%40python.org > -- Regards, Benjamin From brian at python.org Tue Jul 23 02:06:59 2013 From: brian at python.org (Brian Curtin) Date: Mon, 22 Jul 2013 19:06:59 -0500 Subject: [Python-Dev] [Python-checkins] cpython: Fix #18530. Remove extra stat call from posixpath.ismount In-Reply-To: References: <3bzW5x3YpTz7Lks@mail.python.org> Message-ID: On Mon, Jul 22, 2013 at 6:36 PM, Victor Stinner wrote: > Could you please keep the comment "# A symlink can never be a mount point" ? > It is useful. (I didn't know that, I'm not a windows developer.) I don't think that's specific to Windows, but I added it back in d6213012d87b. From dwightdhutto at gmail.com Tue Jul 23 07:39:03 2013 From: dwightdhutto at gmail.com (David Hutto) Date: Tue, 23 Jul 2013 01:39:03 -0400 Subject: [Python-Dev] Building a Faster Python In-Reply-To: References: <252C50D8-C23D-438D-BAE1-B22E0D65A21A@gmail.com> <20130722081534.7cef6df0@fsol> <1374505712.3898.13.camel@surprise> Message-ID: Just a suggestion from working with an assembly language stepper from a while back with Intel x86...lost to an HD crash, but couldn't you disassemble the binary, run through the assembly, and look for specific instructions that you could refine into a simpler, smaller cycling time to improve upon with a stepper that counts cycles? I've seen other assembly steppers after I started mine for electrical engineering of PCB's, but I don't know if they're counting cycles, and suggesting other actions in an editor typ/optimization mode that the compiler might miss. On Mon, Jul 22, 2013 at 12:09 PM, Ronald Oussoren wrote: > > On 22 Jul, 2013, at 17:08, David Malcolm wrote: > > > On Mon, 2013-07-22 at 09:32 +0200, Maciej Fijalkowski wrote: > >> On Mon, Jul 22, 2013 at 9:32 AM, Maciej Fijalkowski > wrote: > >>> On Mon, Jul 22, 2013 at 8:15 AM, Antoine Pitrou > wrote: > >>>> On Sun, 21 Jul 2013 16:36:35 -0700 > >>>> Raymond Hettinger wrote: > >>>>> Our current Mac OS X builds use GCC-4.2. > >>>>> > >>>>> On Python2.7, I ran a comparison of gcc-4.2.1 builds > >>>>> versus gcc-4.8.1 and found that the latter makes a much > >>>>> faster Python. PyBench2.0 shows the total running time > >>>>> dropping from 5653ms to 4571ms. The code is uniformly > >>>>> better in just about every category. > >>>> > >>>> You could try running the benchmarks suite to see what that gives: > >>>> http://hg.python.org/benchmarks/ > >>>> > >>>> Regards > >>>> > >>>> Antoine. > >>> > >>> or pypy benchmark suite which is more comprehensive for python 2.7 > >>> (http://bitbucket.org/pypy/benchmarks) > >> > >> Besides, is there any reason not to use clang by default on OS X? > > > > How did this thread go from: > > "for OS X, GCC 4.8.1 gives you significantly faster machine code > > than the system GCC 4.2.1" > > to > > "let's just use clang" > > ? > > Because we use the system compiler for building the official binary > packages. > > I'm not looking forward to bootstrapping GCC multiple times[*] just to be > able > to build a slightly faster python. And more so because you have to be very > careful when using a alternative compiler when building the installer, it > is > very easy to end up with a build that others cannot use to build extension > because they don't have /Users/ronald/Tools/Compiler/gcc-4.8/bin/gcc. > > > > > (I should declare that I've been hacking on GCC for the last few months, > > so I have an interest in this) > > It would still be interesting to know which compiler would generate the > fastest code for CPython. Apple tends to claim that clang generates better > code than GCC, buit AFAIK they compare the latest clang with the latest > version of GCC that they used to ship, which is ancient by now. > > Ronald > > [*] multiple times due to fat binaries. > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/dwightdhutto%40gmail.com > -- Best Regards, David Hutto *CEO:* *http://www.hitwebdevelopment.com* -------------- next part -------------- An HTML attachment was scrubbed... URL: From ronaldoussoren at mac.com Tue Jul 23 08:15:29 2013 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Tue, 23 Jul 2013 08:15:29 +0200 Subject: [Python-Dev] [Python-checkins] cpython: Issue #18520: Add a new PyStructSequence_InitType2() function, same than In-Reply-To: References: <3bzcCt6wttz7Ljj@mail.python.org> Message-ID: On 23 Jul, 2013, at 2:01, Benjamin Peterson wrote: > We've cheerfully broken the ABI before on minor releases, though if > it's part of the stable ABI, we can't be cavaliar about that anymore. It is not part of the stable ABI. Given that the implementation of PyStructSequence_InitType() in the patch just calls PyStructSequence_InitType2() and ignores the return value you could change the return value of ..InitType(). This may or may not break existing extensions using the function (depending on platform ABI details, AFAIK this is not a problem on x86/x86_64), but reusing extensions across python feature releases is not supported anyway. There are no problems when compiling code, most C compilers won't even warn about ignored return values unless you explicitly ask for it. Ronald > > 2013/7/22 Victor Stinner : >> "Add a new PyStructSequence_InitType2()" >> >> I added a new function because I guess that it would break the API (and ABI) >> to change the return type of a function in a minor release. >> >> Tell me if you have a better name than PyStructSequence_InitType2() ;-) >> >> "Ex" suffix is usually used when parameters are added. It is not the case >> here. >> >> Victor >> >> Le 22 juil. 2013 23:59, "victor.stinner" a >> ?crit : >>> >>> http://hg.python.org/cpython/rev/fc718c177ee6 >>> changeset: 84793:fc718c177ee6 >>> user: Victor Stinner >>> date: Mon Jul 22 22:24:54 2013 +0200 >>> summary: >>> Issue #18520: Add a new PyStructSequence_InitType2() function, same than >>> PyStructSequence_InitType() except that it has a return value (0 on >>> success, >>> -1 on error). >>> >>> * PyStructSequence_InitType2() now raises MemoryError on memory >>> allocation failure >>> * Fix also some calls to PyDict_SetItemString(): handle error >>> >>> files: >>> Include/pythonrun.h | 2 +- >>> Include/structseq.h | 2 + >>> Misc/NEWS | 4 +++ >>> Modules/_lsprof.c | 10 ++++--- >>> Modules/grpmodule.c | 11 ++++++-- >>> Modules/posixmodule.c | 24 ++++++++++++------ >>> Modules/pwdmodule.c | 5 ++- >>> Modules/resource.c | 9 ++++-- >>> Modules/signalmodule.c | 7 +++-- >>> Modules/spwdmodule.c | 8 ++++-- >>> Modules/timemodule.c | 5 ++- >>> Objects/floatobject.c | 9 ++++-- >>> Objects/longobject.c | 6 +++- >>> Objects/structseq.c | 37 +++++++++++++++++++++-------- >>> Python/pythonrun.c | 3 +- >>> Python/sysmodule.c | 23 ++++++++++++----- >>> Python/thread.c | 6 +++- >>> 17 files changed, 117 insertions(+), 54 deletions(-) >>> >>> >>> diff --git a/Include/pythonrun.h b/Include/pythonrun.h >>> --- a/Include/pythonrun.h >>> +++ b/Include/pythonrun.h >>> @@ -197,7 +197,7 @@ >>> PyAPI_FUNC(void) _PyExc_Init(PyObject * bltinmod); >>> PyAPI_FUNC(void) _PyImportHooks_Init(void); >>> PyAPI_FUNC(int) _PyFrame_Init(void); >>> -PyAPI_FUNC(void) _PyFloat_Init(void); >>> +PyAPI_FUNC(int) _PyFloat_Init(void); >>> PyAPI_FUNC(int) PyByteArray_Init(void); >>> PyAPI_FUNC(void) _PyRandom_Init(void); >>> #endif >>> diff --git a/Include/structseq.h b/Include/structseq.h >>> --- a/Include/structseq.h >>> +++ b/Include/structseq.h >>> @@ -24,6 +24,8 @@ >>> #ifndef Py_LIMITED_API >>> PyAPI_FUNC(void) PyStructSequence_InitType(PyTypeObject *type, >>> PyStructSequence_Desc *desc); >>> +PyAPI_FUNC(int) PyStructSequence_InitType2(PyTypeObject *type, >>> + PyStructSequence_Desc *desc); >>> #endif >>> PyAPI_FUNC(PyTypeObject*) PyStructSequence_NewType(PyStructSequence_Desc >>> *desc); >>> >>> diff --git a/Misc/NEWS b/Misc/NEWS >>> --- a/Misc/NEWS >>> +++ b/Misc/NEWS >>> @@ -10,6 +10,10 @@ >>> Core and Builtins >>> ----------------- >>> >>> +- Issue #18520: Add a new PyStructSequence_InitType2() function, same >>> than >>> + PyStructSequence_InitType() except that it has a return value (0 on >>> success, >>> + -1 on error). >>> + >>> - Issue #15905: Fix theoretical buffer overflow in handling of >>> sys.argv[0], >>> prefix and exec_prefix if the operation system does not obey >>> MAXPATHLEN. >>> >>> diff --git a/Modules/_lsprof.c b/Modules/_lsprof.c >>> --- a/Modules/_lsprof.c >>> +++ b/Modules/_lsprof.c >>> @@ -884,10 +884,12 @@ >>> PyDict_SetItemString(d, "Profiler", (PyObject *)&PyProfiler_Type); >>> >>> if (!initialized) { >>> - PyStructSequence_InitType(&StatsEntryType, >>> - &profiler_entry_desc); >>> - PyStructSequence_InitType(&StatsSubEntryType, >>> - &profiler_subentry_desc); >>> + if (PyStructSequence_InitType2(&StatsEntryType, >>> + &profiler_entry_desc) < 0) >>> + return NULL; >>> + if (PyStructSequence_InitType2(&StatsSubEntryType, >>> + &profiler_subentry_desc) < 0) >>> + return NULL; >>> } >>> Py_INCREF((PyObject*) &StatsEntryType); >>> Py_INCREF((PyObject*) &StatsSubEntryType); >>> diff --git a/Modules/grpmodule.c b/Modules/grpmodule.c >>> --- a/Modules/grpmodule.c >>> +++ b/Modules/grpmodule.c >>> @@ -210,9 +210,14 @@ >>> if (m == NULL) >>> return NULL; >>> d = PyModule_GetDict(m); >>> - if (!initialized) >>> - PyStructSequence_InitType(&StructGrpType, >>> &struct_group_type_desc); >>> - PyDict_SetItemString(d, "struct_group", (PyObject *) &StructGrpType); >>> + if (!initialized) { >>> + if (PyStructSequence_InitType2(&StructGrpType, >>> + &struct_group_type_desc) < 0) >>> + return NULL; >>> + } >>> + if (PyDict_SetItemString(d, "struct_group", >>> + (PyObject *)&StructGrpType) < 0) >>> + return NULL; >>> initialized = 1; >>> return m; >>> } >>> diff --git a/Modules/posixmodule.c b/Modules/posixmodule.c >>> --- a/Modules/posixmodule.c >>> +++ b/Modules/posixmodule.c >>> @@ -11518,19 +11518,23 @@ >>> if (!initialized) { >>> #if defined(HAVE_WAITID) && !defined(__APPLE__) >>> waitid_result_desc.name = MODNAME ".waitid_result"; >>> - PyStructSequence_InitType(&WaitidResultType, >>> &waitid_result_desc); >>> + if (PyStructSequence_InitType2(&WaitidResultType, >>> &waitid_result_desc) < 0) >>> + return NULL; >>> #endif >>> >>> stat_result_desc.name = MODNAME ".stat_result"; >>> stat_result_desc.fields[7].name = PyStructSequence_UnnamedField; >>> stat_result_desc.fields[8].name = PyStructSequence_UnnamedField; >>> stat_result_desc.fields[9].name = PyStructSequence_UnnamedField; >>> - PyStructSequence_InitType(&StatResultType, &stat_result_desc); >>> + if (PyStructSequence_InitType2(&StatResultType, >>> &stat_result_desc) < 0) >>> + return NULL; >>> structseq_new = StatResultType.tp_new; >>> StatResultType.tp_new = statresult_new; >>> >>> statvfs_result_desc.name = MODNAME ".statvfs_result"; >>> - PyStructSequence_InitType(&StatVFSResultType, >>> &statvfs_result_desc); >>> + if (PyStructSequence_InitType2(&StatVFSResultType, >>> + &statvfs_result_desc) < 0) >>> + return NULL; >>> #ifdef NEED_TICKS_PER_SECOND >>> # if defined(HAVE_SYSCONF) && defined(_SC_CLK_TCK) >>> ticks_per_second = sysconf(_SC_CLK_TCK); >>> @@ -11543,12 +11547,15 @@ >>> >>> #if defined(HAVE_SCHED_SETPARAM) || defined(HAVE_SCHED_SETSCHEDULER) >>> sched_param_desc.name = MODNAME ".sched_param"; >>> - PyStructSequence_InitType(&SchedParamType, &sched_param_desc); >>> + if (PyStructSequence_InitType2(&SchedParamType, >>> &sched_param_desc) < 0) >>> + return NULL; >>> SchedParamType.tp_new = sched_param_new; >>> #endif >>> >>> /* initialize TerminalSize_info */ >>> - PyStructSequence_InitType(&TerminalSizeType, &TerminalSize_desc); >>> + if (PyStructSequence_InitType2(&TerminalSizeType, >>> + &TerminalSize_desc) < 0) >>> + return NULL; >>> } >>> #if defined(HAVE_WAITID) && !defined(__APPLE__) >>> Py_INCREF((PyObject*) &WaitidResultType); >>> @@ -11566,11 +11573,13 @@ >>> #endif >>> >>> times_result_desc.name = MODNAME ".times_result"; >>> - PyStructSequence_InitType(&TimesResultType, ×_result_desc); >>> + if (PyStructSequence_InitType2(&TimesResultType, ×_result_desc) >>> < 0) >>> + return NULL; >>> PyModule_AddObject(m, "times_result", (PyObject *)&TimesResultType); >>> >>> uname_result_desc.name = MODNAME ".uname_result"; >>> - PyStructSequence_InitType(&UnameResultType, &uname_result_desc); >>> + if (PyStructSequence_InitType2(&UnameResultType, &uname_result_desc) >>> < 0) >>> + return NULL; >>> PyModule_AddObject(m, "uname_result", (PyObject *)&UnameResultType); >>> >>> #ifdef __APPLE__ >>> @@ -11648,7 +11657,6 @@ >>> initialized = 1; >>> >>> return m; >>> - >>> } >>> >>> #ifdef __cplusplus >>> diff --git a/Modules/pwdmodule.c b/Modules/pwdmodule.c >>> --- a/Modules/pwdmodule.c >>> +++ b/Modules/pwdmodule.c >>> @@ -216,8 +216,9 @@ >>> return NULL; >>> >>> if (!initialized) { >>> - PyStructSequence_InitType(&StructPwdType, >>> - &struct_pwd_type_desc); >>> + if (PyStructSequence_InitType2(&StructPwdType, >>> + &struct_pwd_type_desc) < 0) >>> + return NULL; >>> initialized = 1; >>> } >>> Py_INCREF((PyObject *) &StructPwdType); >>> diff --git a/Modules/resource.c b/Modules/resource.c >>> --- a/Modules/resource.c >>> +++ b/Modules/resource.c >>> @@ -263,9 +263,12 @@ >>> /* Add some symbolic constants to the module */ >>> Py_INCREF(PyExc_OSError); >>> PyModule_AddObject(m, "error", PyExc_OSError); >>> - if (!initialized) >>> - PyStructSequence_InitType(&StructRUsageType, >>> - &struct_rusage_desc); >>> + if (!initialized) { >>> + if (PyStructSequence_InitType2(&StructRUsageType, >>> + &struct_rusage_desc) < 0) >>> + return NULL; >>> + } >>> + >>> Py_INCREF(&StructRUsageType); >>> PyModule_AddObject(m, "struct_rusage", >>> (PyObject*) &StructRUsageType); >>> diff --git a/Modules/signalmodule.c b/Modules/signalmodule.c >>> --- a/Modules/signalmodule.c >>> +++ b/Modules/signalmodule.c >>> @@ -978,9 +978,10 @@ >>> return NULL; >>> >>> #if defined(HAVE_SIGWAITINFO) || defined(HAVE_SIGTIMEDWAIT) >>> - if (!initialized) >>> - PyStructSequence_InitType(&SiginfoType, &struct_siginfo_desc); >>> - >>> + if (!initialized) { >>> + if (PyStructSequence_InitType2(&SiginfoType, >>> &struct_siginfo_desc) < 0) >>> + return NULL; >>> + } >>> Py_INCREF((PyObject*) &SiginfoType); >>> PyModule_AddObject(m, "struct_siginfo", (PyObject*) &SiginfoType); >>> initialized = 1; >>> diff --git a/Modules/spwdmodule.c b/Modules/spwdmodule.c >>> --- a/Modules/spwdmodule.c >>> +++ b/Modules/spwdmodule.c >>> @@ -196,9 +196,11 @@ >>> m=PyModule_Create(&spwdmodule); >>> if (m == NULL) >>> return NULL; >>> - if (!initialized) >>> - PyStructSequence_InitType(&StructSpwdType, >>> - &struct_spwd_type_desc); >>> + if (!initialized) { >>> + if (PyStructSequence_InitType2(&StructSpwdType, >>> + &struct_spwd_type_desc) < 0) >>> + return NULL; >>> + } >>> Py_INCREF((PyObject *) &StructSpwdType); >>> PyModule_AddObject(m, "struct_spwd", (PyObject *) &StructSpwdType); >>> initialized = 1; >>> diff --git a/Modules/timemodule.c b/Modules/timemodule.c >>> --- a/Modules/timemodule.c >>> +++ b/Modules/timemodule.c >>> @@ -1476,8 +1476,9 @@ >>> PyInit_timezone(m); >>> >>> if (!initialized) { >>> - PyStructSequence_InitType(&StructTimeType, >>> - &struct_time_type_desc); >>> + if (PyStructSequence_InitType2(&StructTimeType, >>> + &struct_time_type_desc) < 0) >>> + return NULL; >>> >>> #ifdef MS_WINDOWS >>> winver.dwOSVersionInfoSize = sizeof(winver); >>> diff --git a/Objects/floatobject.c b/Objects/floatobject.c >>> --- a/Objects/floatobject.c >>> +++ b/Objects/floatobject.c >>> @@ -1853,7 +1853,7 @@ >>> float_new, /* tp_new */ >>> }; >>> >>> -void >>> +int >>> _PyFloat_Init(void) >>> { >>> /* We attempt to determine if this machine is using IEEE >>> @@ -1903,8 +1903,11 @@ >>> float_format = detected_float_format; >>> >>> /* Init float info */ >>> - if (FloatInfoType.tp_name == 0) >>> - PyStructSequence_InitType(&FloatInfoType, &floatinfo_desc); >>> + if (FloatInfoType.tp_name == NULL) { >>> + if (PyStructSequence_InitType2(&FloatInfoType, &floatinfo_desc) < >>> 0) >>> + return 0; >>> + } >>> + return 1; >>> } >>> >>> int >>> diff --git a/Objects/longobject.c b/Objects/longobject.c >>> --- a/Objects/longobject.c >>> +++ b/Objects/longobject.c >>> @@ -5059,8 +5059,10 @@ >>> } >>> #endif >>> /* initialize int_info */ >>> - if (Int_InfoType.tp_name == 0) >>> - PyStructSequence_InitType(&Int_InfoType, &int_info_desc); >>> + if (Int_InfoType.tp_name == NULL) { >>> + if (PyStructSequence_InitType2(&Int_InfoType, &int_info_desc) < >>> 0) >>> + return 0; >>> + } >>> >>> return 1; >>> } >>> diff --git a/Objects/structseq.c b/Objects/structseq.c >>> --- a/Objects/structseq.c >>> +++ b/Objects/structseq.c >>> @@ -320,12 +320,13 @@ >>> structseq_new, /* tp_new */ >>> }; >>> >>> -void >>> -PyStructSequence_InitType(PyTypeObject *type, PyStructSequence_Desc >>> *desc) >>> +int >>> +PyStructSequence_InitType2(PyTypeObject *type, PyStructSequence_Desc >>> *desc) >>> { >>> PyObject *dict; >>> PyMemberDef* members; >>> int n_members, n_unnamed_members, i, k; >>> + PyObject *v; >>> >>> #ifdef Py_TRACE_REFS >>> /* if the type object was chained, unchain it first >>> @@ -347,8 +348,10 @@ >>> type->tp_doc = desc->doc; >>> >>> members = PyMem_NEW(PyMemberDef, n_members-n_unnamed_members+1); >>> - if (members == NULL) >>> - return; >>> + if (members == NULL) { >>> + PyErr_NoMemory(); >>> + return -1; >>> + } >>> >>> for (i = k = 0; i < n_members; ++i) { >>> if (desc->fields[i].name == PyStructSequence_UnnamedField) >>> @@ -366,22 +369,33 @@ >>> type->tp_members = members; >>> >>> if (PyType_Ready(type) < 0) >>> - return; >>> + return -1; >>> Py_INCREF(type); >>> >>> dict = type->tp_dict; >>> #define SET_DICT_FROM_INT(key, value) \ >>> do { \ >>> - PyObject *v = PyLong_FromLong((long) value); \ >>> - if (v != NULL) { \ >>> - PyDict_SetItemString(dict, key, v); \ >>> + v = PyLong_FromLong((long) value); \ >>> + if (v == NULL) \ >>> + return -1; \ >>> + if (PyDict_SetItemString(dict, key, v) < 0) { \ >>> Py_DECREF(v); \ >>> + return -1; \ >>> } \ >>> + Py_DECREF(v); \ >>> } while (0) >>> >>> SET_DICT_FROM_INT(visible_length_key, desc->n_in_sequence); >>> SET_DICT_FROM_INT(real_length_key, n_members); >>> SET_DICT_FROM_INT(unnamed_fields_key, n_unnamed_members); >>> + >>> + return 0; >>> +} >>> + >>> +void >>> +PyStructSequence_InitType(PyTypeObject *type, PyStructSequence_Desc >>> *desc) >>> +{ >>> + (void)PyStructSequence_InitType2(type, desc); >>> } >>> >>> PyTypeObject* >>> @@ -390,8 +404,11 @@ >>> PyTypeObject *result; >>> >>> result = (PyTypeObject*)PyType_GenericAlloc(&PyType_Type, 0); >>> - if (result != NULL) { >>> - PyStructSequence_InitType(result, desc); >>> + if (result == NULL) >>> + return NULL; >>> + if (PyStructSequence_InitType2(result, desc) < 0) { >>> + Py_DECREF(result); >>> + return NULL; >>> } >>> return result; >>> } >>> diff --git a/Python/pythonrun.c b/Python/pythonrun.c >>> --- a/Python/pythonrun.c >>> +++ b/Python/pythonrun.c >>> @@ -328,7 +328,8 @@ >>> if (!PyByteArray_Init()) >>> Py_FatalError("Py_Initialize: can't init bytearray"); >>> >>> - _PyFloat_Init(); >>> + if (!_PyFloat_Init()) >>> + Py_FatalError("Py_Initialize: can't init float"); >>> >>> interp->modules = PyDict_New(); >>> if (interp->modules == NULL) >>> diff --git a/Python/sysmodule.c b/Python/sysmodule.c >>> --- a/Python/sysmodule.c >>> +++ b/Python/sysmodule.c >>> @@ -1634,8 +1634,10 @@ >>> SET_SYS_FROM_STRING("int_info", >>> PyLong_GetInfo()); >>> /* initialize hash_info */ >>> - if (Hash_InfoType.tp_name == 0) >>> - PyStructSequence_InitType(&Hash_InfoType, &hash_info_desc); >>> + if (Hash_InfoType.tp_name == NULL) { >>> + if (PyStructSequence_InitType2(&Hash_InfoType, &hash_info_desc) < >>> 0) >>> + return NULL; >>> + } >>> SET_SYS_FROM_STRING("hash_info", >>> get_hash_info()); >>> SET_SYS_FROM_STRING("maxunicode", >>> @@ -1676,8 +1678,11 @@ >>> } >>> >>> /* version_info */ >>> - if (VersionInfoType.tp_name == 0) >>> - PyStructSequence_InitType(&VersionInfoType, &version_info_desc); >>> + if (VersionInfoType.tp_name == NULL) { >>> + if (PyStructSequence_InitType2(&VersionInfoType, >>> + &version_info_desc) < 0) >>> + return NULL; >>> + } >>> version_info = make_version_info(); >>> SET_SYS_FROM_STRING("version_info", version_info); >>> /* prevent user from creating new instances */ >>> @@ -1688,8 +1693,10 @@ >>> SET_SYS_FROM_STRING("implementation", make_impl_info(version_info)); >>> >>> /* flags */ >>> - if (FlagsType.tp_name == 0) >>> - PyStructSequence_InitType(&FlagsType, &flags_desc); >>> + if (FlagsType.tp_name == 0) { >>> + if (PyStructSequence_InitType2(&FlagsType, &flags_desc) < 0) >>> + return NULL; >>> + } >>> SET_SYS_FROM_STRING("flags", make_flags()); >>> /* prevent user from creating new instances */ >>> FlagsType.tp_init = NULL; >>> @@ -1699,7 +1706,9 @@ >>> #if defined(MS_WINDOWS) >>> /* getwindowsversion */ >>> if (WindowsVersionType.tp_name == 0) >>> - PyStructSequence_InitType(&WindowsVersionType, >>> &windows_version_desc); >>> + if (PyStructSequence_InitType2(&WindowsVersionType, >>> + &windows_version_desc) < 0) >>> + return NULL; >>> /* prevent user from creating new instances */ >>> WindowsVersionType.tp_init = NULL; >>> WindowsVersionType.tp_new = NULL; >>> diff --git a/Python/thread.c b/Python/thread.c >>> --- a/Python/thread.c >>> +++ b/Python/thread.c >>> @@ -399,8 +399,10 @@ >>> int len; >>> #endif >>> >>> - if (ThreadInfoType.tp_name == 0) >>> - PyStructSequence_InitType(&ThreadInfoType, &threadinfo_desc); >>> + if (ThreadInfoType.tp_name == 0) { >>> + if (PyStructSequence_InitType2(&ThreadInfoType, &threadinfo_desc) >>> < 0) >>> + return NULL; >>> + } >>> >>> threadinfo = PyStructSequence_New(&ThreadInfoType); >>> if (threadinfo == NULL) >>> >>> -- >>> Repository URL: http://hg.python.org/cpython >>> >>> _______________________________________________ >>> Python-checkins mailing list >>> Python-checkins at python.org >>> http://mail.python.org/mailman/listinfo/python-checkins >>> >> >> _______________________________________________ >> Python-Dev mailing list >> Python-Dev at python.org >> http://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe: >> http://mail.python.org/mailman/options/python-dev/benjamin%40python.org >> > > > > -- > Regards, > Benjamin > _______________________________________________ > Python-checkins mailing list > Python-checkins at python.org > http://mail.python.org/mailman/listinfo/python-checkins From solipsis at pitrou.net Tue Jul 23 08:27:27 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 23 Jul 2013 08:27:27 +0200 Subject: [Python-Dev] cpython: Add modeling file for Coverity Scan. References: <3bzfGN573Hz7Lkf@mail.python.org> Message-ID: <20130723082727.634ee6cb@fsol> On Tue, 23 Jul 2013 01:31:24 +0200 (CEST) christian.heimes wrote: > + > +typedef int sdigit; > +typedef long Py_ssize_t; Can't you write "typedef ssize_t Py_ssize_t" instead? Regards Antoine. From solipsis at pitrou.net Tue Jul 23 08:23:45 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 23 Jul 2013 08:23:45 +0200 Subject: [Python-Dev] cpython: Issue #18520: Add a new PyStructSequence_InitType2() function, same than References: <3bzcCt6wttz7Ljj@mail.python.org> <20130723082230.57a5f5b2@fsol> Message-ID: <20130723082345.037d0921@fsol> On Tue, 23 Jul 2013 08:22:30 +0200 Antoine Pitrou wrote: > On Tue, 23 Jul 2013 08:15:29 +0200 > Ronald Oussoren wrote: > > > > On 23 Jul, 2013, at 2:01, Benjamin Peterson wrote: > > > > > We've cheerfully broken the ABI before on minor releases, though if > > > it's part of the stable ABI, we can't be cavaliar about that anymore. > > > > It is not part of the stable ABI. Given that the implementation of > > PyStructSequence_InitType() in the patch just calls PyStructSequence_InitType2() > > and ignores the return value you could change the return value of ..InitType(). > > > > This may or may not break existing extensions using the function (depending on > > platform ABI details, AFAIK this is not a problem on x86/x86_64), but reusing > > extensions across python feature releases is not supported anyway. > > Not supported? It should certainly be supported accross bugfix > versions. Otherwise, installing a new bugfix version would force you to > recompile all independently-installed extension modules. Ah, but I see that "minor release" was used in the sense of "feature release". My bad. (Grrr!) Regards Antoine. From christian at python.org Tue Jul 23 12:00:51 2013 From: christian at python.org (Christian Heimes) Date: Tue, 23 Jul 2013 12:00:51 +0200 Subject: [Python-Dev] cpython (3.3): return NULL here In-Reply-To: <3bznlW6d69z7LkD@mail.python.org> References: <3bznlW6d69z7LkD@mail.python.org> Message-ID: <51EE5453.4010209@python.org> Am 23.07.2013 07:08, schrieb benjamin.peterson: > http://hg.python.org/cpython/rev/042ff9325c5e > changeset: 84804:042ff9325c5e > branch: 3.3 > parent: 84789:bb63f813a00f > user: Benjamin Peterson > date: Mon Jul 22 22:08:09 2013 -0700 > summary: > return NULL here > > files: > Python/dynload_shlib.c | 3 ++- > 1 files changed, 2 insertions(+), 1 deletions(-) > > > diff --git a/Python/dynload_shlib.c b/Python/dynload_shlib.c > --- a/Python/dynload_shlib.c > +++ b/Python/dynload_shlib.c > @@ -91,7 +91,8 @@ > int i; > struct stat statb; > if (fstat(fileno(fp), &statb) == -1) { > - return PyErr_SetFromErrno(PyExc_IOError); > + PyErr_SetFromErrno(PyExc_IOError); > + return NULL; > } PyErr_SetFromErrno() already and always returns NULL. Or do you prefer to return NULL explicitly? Christian From benjamin at python.org Tue Jul 23 17:10:40 2013 From: benjamin at python.org (Benjamin Peterson) Date: Tue, 23 Jul 2013 08:10:40 -0700 Subject: [Python-Dev] [Python-checkins] cpython (3.3): return NULL here In-Reply-To: <51EE5453.4010209@python.org> References: <3bznlW6d69z7LkD@mail.python.org> <51EE5453.4010209@python.org> Message-ID: 2013/7/23 Christian Heimes : > Am 23.07.2013 07:08, schrieb benjamin.peterson: >> http://hg.python.org/cpython/rev/042ff9325c5e >> changeset: 84804:042ff9325c5e >> branch: 3.3 >> parent: 84789:bb63f813a00f >> user: Benjamin Peterson >> date: Mon Jul 22 22:08:09 2013 -0700 >> summary: >> return NULL here >> >> files: >> Python/dynload_shlib.c | 3 ++- >> 1 files changed, 2 insertions(+), 1 deletions(-) >> >> >> diff --git a/Python/dynload_shlib.c b/Python/dynload_shlib.c >> --- a/Python/dynload_shlib.c >> +++ b/Python/dynload_shlib.c >> @@ -91,7 +91,8 @@ >> int i; >> struct stat statb; >> if (fstat(fileno(fp), &statb) == -1) { >> - return PyErr_SetFromErrno(PyExc_IOError); >> + PyErr_SetFromErrno(PyExc_IOError); >> + return NULL; >> } > > PyErr_SetFromErrno() already and always returns NULL. Or do you prefer > to return NULL explicitly? It might always return NULL, but the compiler sees (PyObject *)NULL when this function returns dl_funcptr. -- Regards, Benjamin From pelhamny at gmail.com Tue Jul 23 12:17:20 2013 From: pelhamny at gmail.com (Nicholas Hart) Date: Tue, 23 Jul 2013 06:17:20 -0400 Subject: [Python-Dev] tuple index out of range Message-ID: Hi, I am new to this list and to troubleshooting python. I hope someone can help me. I am getting this tuple index out of range error while running a test call to my python code. Not sure what this error really means and was hoping someone might shed some light on how to fix this. Also was wondering why much of my .py files are not getting compiled to .pyc upon first run.. is this unusual or need I not worry? Running python 2.5.2 on fedora. Thanks, Nick -------------- next part -------------- An HTML attachment was scrubbed... URL: From ronaldoussoren at mac.com Tue Jul 23 17:30:09 2013 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Tue, 23 Jul 2013 17:30:09 +0200 Subject: [Python-Dev] tuple index out of range In-Reply-To: References: Message-ID: <7E7D3ED0-8306-4F53-9D67-D1D7CACAAA87@mac.com> On 23 Jul, 2013, at 12:17, Nicholas Hart wrote: > Hi, > > I am new to this list and to troubleshooting python. I hope someone can help me. I am getting this tuple index out of range error while running a test call to my python code. Not sure what this error really means and was hoping someone might shed some light on how to fix this. Also was wondering why much of my .py files are not getting compiled to .pyc upon first run.. is this unusual or need I not worry? > > Running python 2.5.2 on fedora. Nick, This list is focussed on the development of Python, not on development with Python. The python-list list is more appropriate for asking questions like yours. Regards, Ronald > > Thanks, > Nick > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/ronaldoussoren%40mac.com From christian at python.org Tue Jul 23 17:36:34 2013 From: christian at python.org (Christian Heimes) Date: Tue, 23 Jul 2013 17:36:34 +0200 Subject: [Python-Dev] cpython (3.3): return NULL here In-Reply-To: References: <3bznlW6d69z7LkD@mail.python.org> <51EE5453.4010209@python.org> Message-ID: <51EEA302.7070008@python.org> Am 23.07.2013 17:10, schrieb Benjamin Peterson: >> PyErr_SetFromErrno() already and always returns NULL. Or do you prefer >> to return NULL explicitly? > > It might always return NULL, but the compiler sees (PyObject *)NULL > when this function returns dl_funcptr. Oh, you are right. I must have missed the compiler warning. How about we turn type return and type assignment warnings into fatal errors? Christian From ronaldoussoren at mac.com Tue Jul 23 17:46:26 2013 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Tue, 23 Jul 2013 17:46:26 +0200 Subject: [Python-Dev] cpython (3.3): return NULL here In-Reply-To: <51EEA302.7070008@python.org> References: <3bznlW6d69z7LkD@mail.python.org> <51EE5453.4010209@python.org> <51EEA302.7070008@python.org> Message-ID: <9C38BD1B-24FF-4360-A729-6DC215238A6C@mac.com> On 23 Jul, 2013, at 17:36, Christian Heimes wrote: > Am 23.07.2013 17:10, schrieb Benjamin Peterson: >>> PyErr_SetFromErrno() already and always returns NULL. Or do you prefer >>> to return NULL explicitly? >> >> It might always return NULL, but the compiler sees (PyObject *)NULL >> when this function returns dl_funcptr. > > Oh, you are right. I must have missed the compiler warning. How about we > turn type return and type assignment warnings into fatal errors? That's probably possible with a '-Werror=....' argument. But please consider issue 18211 before unconditionally adding such a flag, as that issue mentions new compiler flags also get used when compiling 3th-party extensions. I guess there needs to be (yet) another CFLAGS_xxx variable in the Makefile that gets added to $(CFLAGS) during the build of Python itself, but is ignored by distutils and sysconfig. Ronald > > Christian > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/ronaldoussoren%40mac.com From martin at v.loewis.de Tue Jul 23 18:11:26 2013 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 23 Jul 2013 18:11:26 +0200 Subject: [Python-Dev] Adding Python scripts to PATHEXT on Windows In-Reply-To: References: Message-ID: <51EEAB2E.9030804@v.loewis.de> Am 15.07.13 10:26, schrieb Paul Moore: > Does anyone have any objections to this? I could try to write a patch, > but I know next to nothing about building MSIs, so if any of the > installer experts could help that would be fantastic. It's fine with me. I could write the patch, but will likely forget - so please remind me with every alpha that misses this. Also, contributions are welcome. Regards, Martin From p.f.moore at gmail.com Tue Jul 23 18:33:15 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 23 Jul 2013 17:33:15 +0100 Subject: [Python-Dev] Adding Python scripts to PATHEXT on Windows In-Reply-To: <51EEAB2E.9030804@v.loewis.de> References: <51EEAB2E.9030804@v.loewis.de> Message-ID: On 23 July 2013 17:11, "Martin v. L?wis" wrote: > Am 15.07.13 10:26, schrieb Paul Moore: > > Does anyone have any objections to this? I could try to write a patch, > > but I know next to nothing about building MSIs, so if any of the > > installer experts could help that would be fantastic. > > It's fine with me. I could write the patch, but will likely forget - so > please remind me with every alpha that misses this. Also, contributions > are welcome. > Thanks. I think I have the basic idea of what's needed, so I'll write an initial patch. If you can check it that'd be great. Paul -------------- next part -------------- An HTML attachment was scrubbed... URL: From christian at python.org Tue Jul 23 21:52:47 2013 From: christian at python.org (Christian Heimes) Date: Tue, 23 Jul 2013 21:52:47 +0200 Subject: [Python-Dev] cpython: Add modeling file for Coverity Scan. In-Reply-To: <20130723082727.634ee6cb@fsol> References: <3bzfGN573Hz7Lkf@mail.python.org> <20130723082727.634ee6cb@fsol> Message-ID: <51EEDF0F.8020105@python.org> Am 23.07.2013 08:27, schrieb Antoine Pitrou: > On Tue, 23 Jul 2013 01:31:24 +0200 (CEST) > christian.heimes wrote: >> + >> +typedef int sdigit; >> +typedef long Py_ssize_t; > > Can't you write "typedef ssize_t Py_ssize_t" instead? No, but it really doesn't matter. Coverity just needs a similar type for modeling. "modeling_file.c", line 23: error #20: identifier "ssize_t" is undefined typedef ssize_t Py_ssize_t; ^ 1 error detected in the compilation of "modeling_file.c". ERROR: cov-translate returned with code 2 From victor.stinner at gmail.com Wed Jul 24 00:45:22 2013 From: victor.stinner at gmail.com (Victor Stinner) Date: Wed, 24 Jul 2013 00:45:22 +0200 Subject: [Python-Dev] Inherance of file descriptor and handles on Windows (PEP 446) Message-ID: Hi, Guido van Rossum and others asked me details on how file descriptors and handles are inherited on Windows, for the PEP 446. http://www.python.org/dev/peps/pep-0446/ I hacked Python 3.4 to add a os.get_cloexec() function (extracted from my implementation of the PEP 433), here are some results. Python functions open(), os.open() and os.dup() create file descriptors with the HANDLE_FLAG_INHERIT flag set (cloexec=False), whereas os.pipe() creates 2 file descriptors with the HANDLE_FLAG_INHERIT flag unset (cloexec=False, see also issue #4708). Even if the HANDLE_FLAG_INHERIT flag is set, all handles are closed if subprocess is used with close_fds=True (which is the default value of the parameter), and all file descriptors are closed except 0, 1 and 2. If close_fds=False, handles with the HANDLE_FLAG_INHERIT flag set are inherited, but all file descriptors are still closed except 0, 1 and 2. (I didn't check if file descriptors 0, 1 and 2 are inherited, duplicated or new file descriptors.) The PEP 466 allows you to control which handles are inherited to child process when you use subprocess with close_fds=False. (The subprocess parameter should be called "close_handles" on Windows to avoid confusion.) Said differently: the HANDLE_FLAG_INHERIT flag only has an effect on *handles*, as indicated in its name. On Windows, file *descriptors* are never inherited (are always closed) in child processes. I don't think that it is possible to inherit file descriptors on Windows. By the way, using pass_fds on Windows raises an assertion error ("pass_fds not supported on Windows"). Another example in Python: --- import subprocess, sys code = """ import os, sys fd = int(sys.argv[1]) f = os.fdopen(fd, "rb") print(f.read()) """ f = open(__file__, "rb") fd = f.fileno() subprocess.call([sys.executable, "-c", code, str(fd)], close_fds=False) --- On Unix, the child process will write the script into stdout. On Windows, you just get an OSError(9, "Bad file descriptor") exception. To fix this example on Windows, you have to: * Retrieve the handle of the file using msvcrt.get_osfhandle() ; * Pass the handle, instead of the file descriptor, to the child ; * Create a file descriptor from the handle using msvcrt.open_osfhandle() in the child. The fix would be simpler if Python would provide the handle of a file object (ex: in a method) and if open() supported opening a handle as it does with file descriptors on UNIX. Victor From shibturn at gmail.com Wed Jul 24 01:21:18 2013 From: shibturn at gmail.com (Richard Oudkerk) Date: Wed, 24 Jul 2013 00:21:18 +0100 Subject: [Python-Dev] Inherance of file descriptor and handles on Windows (PEP 446) In-Reply-To: References: Message-ID: On 23/07/2013 11:45pm, Victor Stinner wrote: > Said differently: the HANDLE_FLAG_INHERIT flag only has an effect on > *handles*, as indicated in its name. On Windows, file*descriptors* > are never inherited (are always closed) in child processes. I don't > think that it is possible to inherit file descriptors on Windows. Actually, you can inherit fds if you use os.spawnv() instead of subprocess.Popen(). -- Richard From victor.stinner at gmail.com Wed Jul 24 01:40:18 2013 From: victor.stinner at gmail.com (Victor Stinner) Date: Wed, 24 Jul 2013 01:40:18 +0200 Subject: [Python-Dev] Inherance of file descriptor and handles on Windows (PEP 446) In-Reply-To: References: Message-ID: The multiprocessing module is an example of use case relying on inherance of handles. It calls CreateProcess() with bInheritHandles=TRUE to share a pipe between the manager (parent) and the worker (child process). Note: subprocess and multiprocess have their own function to set the HANDLE_FLAG_INHERIT flag: they use DuplicateHandle(), whereas SetHandleInformation() could be used (to reuse the existing handle instead of creating a new handle). 2013/7/24 Victor Stinner : > Python functions open(), os.open() and os.dup() create file > descriptors with the HANDLE_FLAG_INHERIT flag set (cloexec=False), > whereas os.pipe() creates 2 file descriptors with the > HANDLE_FLAG_INHERIT flag unset (cloexec=False, see also issue #4708). > (...) > If close_fds=False, handles with the HANDLE_FLAG_INHERIT flag set are > inherited, but all file descriptors are still closed except 0, 1 and > 2. Leaking handles in child processes is also an issue on Windows. Random examples. http://bugs.python.org/issue17634 "Win32: shutil.copy leaks file handles to child processes" "Win32's native CopyFile API call doesn't leak file handles to child processes." http://ghc.haskell.org/trac/ghc/ticket/2650 "The case in which I originally ran into this was System.Directory.copyFile intermittently reporting a "permission denied" error for a temp file it was using. I think it was trying to delete it, but failing because a child process somewhere was hanging on to the Handle." According to the issue, GHC calls CreateProcess with bInheritHandles=TRUE (as Python did until Python 3.2). http://support.microsoft.com/kb/315939 "This behavior can occur if two threads simultaneously create child processes and redirect the STD handles through pipes. In this scenario, there is a race condition during the creation of the pipes and processes, in which it is possible for one child to inherit file handles intended for the other child." => Python looks to be correct, it uses the (StartupInfo) "STARTF_USESTDHANDLES" flag http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6428742 Java is still calling CreateProcess() with bInheritHandles=TRUE which causes issues like "6347873: (so) Ports opened with ServerSocketChannel blocks when using Runtime.exec". Interesting comment: "Submitter has a point. Very risky to fix." :-/ For the record, the default value of close_fds parameter was also set to True on Windows by the following changeset: changeset: 66889:59b43dc34158 user: Gregory P. Smith date: Tue Dec 14 13:43:30 2010 +0000 files: Doc/library/subprocess.rst Lib/subprocess.py Lib/test/test_subprocess.py Misc/NEWS Modules/_posixsubprocess.c description: Issue #6559: fix the subprocess.Popen pass_fds implementation. Add a unittest. Issue #7213: Change the close_fds default on Windows to better match the new default on POSIX. True when possible (False if stdin/stdout/stderr are supplied). Update the documentation to reflect all of the above. The changeset was written to fix http://bugs.python.org/issue7213 ; another example of leak of handles. > The PEP 466 allows you to control which handles are inherited to child > process when you use subprocess with close_fds=False. (The subprocess > parameter should be called "close_handles" on Windows to avoid > confusion.) Another advantage of the PEP 446 is that most of the time, the HANDLE_FLAG_INHERIT flag value can be set atomatically at the creation of the file descriptor (creation of the handle). It is a nice enhancement to fight against race conditions with threads ;-) "Most of the time": for example, socket are inherited by default, WSA_FLAG_NO_HANDLE_INHERIT flag was only added to Windows Vista. Victor From p.f.moore at gmail.com Wed Jul 24 08:29:44 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 24 Jul 2013 07:29:44 +0100 Subject: [Python-Dev] Inherance of file descriptor and handles on Windows (PEP 446) In-Reply-To: References: Message-ID: On 23 July 2013 23:45, Victor Stinner wrote: > Said differently: the HANDLE_FLAG_INHERIT flag only has an effect on > *handles*, as indicated in its name. On Windows, file *descriptors* > are never inherited (are always closed) in child processes. I don't > think that it is possible to inherit file descriptors on Windows. > That is correct - handles are the OS-level concept, fds are implemented in the CRT. So code that uses raw Windows APIs to create a new process won't have any means to inherit fds. > The fix would be simpler if Python would provide the handle of a file > object (ex: in a method) and if open() supported opening a handle as > it does with file descriptors on UNIX. > That would give a similar level of functionality to Unix. Whether it is used sufficiently often to be worth it, is a separate question, of course... Paul -------------- next part -------------- An HTML attachment was scrubbed... URL: From greg at krypto.org Wed Jul 24 08:43:59 2013 From: greg at krypto.org (Gregory P. Smith) Date: Tue, 23 Jul 2013 23:43:59 -0700 Subject: [Python-Dev] [Python-checkins] cpython (3.3): return NULL here In-Reply-To: <9C38BD1B-24FF-4360-A729-6DC215238A6C@mac.com> References: <3bznlW6d69z7LkD@mail.python.org> <51EE5453.4010209@python.org> <51EEA302.7070008@python.org> <9C38BD1B-24FF-4360-A729-6DC215238A6C@mac.com> Message-ID: On Tue, Jul 23, 2013 at 8:46 AM, Ronald Oussoren wrote: > > On 23 Jul, 2013, at 17:36, Christian Heimes wrote: > > > Am 23.07.2013 17:10, schrieb Benjamin Peterson: > >>> PyErr_SetFromErrno() already and always returns NULL. Or do you prefer > >>> to return NULL explicitly? > >> > >> It might always return NULL, but the compiler sees (PyObject *)NULL > >> when this function returns dl_funcptr. > > > > Oh, you are right. I must have missed the compiler warning. How about we > > turn type return and type assignment warnings into fatal errors? > > That's probably possible with a '-Werror=....' argument. But please > consider > issue 18211 before unconditionally adding such a flag, as that issue > mentions > new compiler flags also get used when compiling 3th-party extensions. > > I guess there needs to be (yet) another CFLAGS_xxx variable in the > Makefile that > gets added to $(CFLAGS) during the build of Python itself, but is ignored > by > distutils and sysconfig. > It seems fair to turn those on in 3.4 and require that third party extensions clean up their code when porting from 3.3 to 3.4. > Ronald > > > > > Christian > > > > _______________________________________________ > > Python-Dev mailing list > > Python-Dev at python.org > > http://mail.python.org/mailman/listinfo/python-dev > > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/ronaldoussoren%40mac.com > > _______________________________________________ > Python-checkins mailing list > Python-checkins at python.org > http://mail.python.org/mailman/listinfo/python-checkins > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ronaldoussoren at mac.com Wed Jul 24 09:01:30 2013 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Wed, 24 Jul 2013 09:01:30 +0200 Subject: [Python-Dev] [Python-checkins] cpython (3.3): return NULL here In-Reply-To: References: <3bznlW6d69z7LkD@mail.python.org> <51EE5453.4010209@python.org> <51EEA302.7070008@python.org> <9C38BD1B-24FF-4360-A729-6DC215238A6C@mac.com> Message-ID: <568DFFEE-F257-4EA2-A749-DA8BCECCDDF0@mac.com> On 24 Jul, 2013, at 8:43, Gregory P. Smith wrote: > > On Tue, Jul 23, 2013 at 8:46 AM, Ronald Oussoren wrote: > > On 23 Jul, 2013, at 17:36, Christian Heimes wrote: > > > Am 23.07.2013 17:10, schrieb Benjamin Peterson: > >>> PyErr_SetFromErrno() already and always returns NULL. Or do you prefer > >>> to return NULL explicitly? > >> > >> It might always return NULL, but the compiler sees (PyObject *)NULL > >> when this function returns dl_funcptr. > > > > Oh, you are right. I must have missed the compiler warning. How about we > > turn type return and type assignment warnings into fatal errors? > > That's probably possible with a '-Werror=....' argument. But please consider > issue 18211 before unconditionally adding such a flag, as that issue mentions > new compiler flags also get used when compiling 3th-party extensions. > > I guess there needs to be (yet) another CFLAGS_xxx variable in the Makefile that > gets added to $(CFLAGS) during the build of Python itself, but is ignored by > distutils and sysconfig. > > It seems fair to turn those on in 3.4 and require that third party extensions clean up their code when porting from 3.3 to 3.4. In this case its "just" code cleanup, the issue I filed (see above) is for another -Werror flag that causes compile errors with some valid C99 code that isn't valid C89. That's good for CPython itself because its source code is explicitly C89, but is not good when building 3th-party extensions. A proper fix requires tweaking the configure script, Makefile and distutils and that's not really a fun prospect ;-) Ronald > > > Ronald > > > > > Christian > > > > _______________________________________________ > > Python-Dev mailing list > > Python-Dev at python.org > > http://mail.python.org/mailman/listinfo/python-dev > > Unsubscribe: http://mail.python.org/mailman/options/python-dev/ronaldoussoren%40mac.com > > _______________________________________________ > Python-checkins mailing list > Python-checkins at python.org > http://mail.python.org/mailman/listinfo/python-checkins > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/ronaldoussoren%40mac.com From solipsis at pitrou.net Wed Jul 24 10:17:59 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 24 Jul 2013 10:17:59 +0200 Subject: [Python-Dev] [Python-checkins] cpython (3.3): return NULL here References: <3bznlW6d69z7LkD@mail.python.org> <51EE5453.4010209@python.org> <51EEA302.7070008@python.org> <9C38BD1B-24FF-4360-A729-6DC215238A6C@mac.com> <568DFFEE-F257-4EA2-A749-DA8BCECCDDF0@mac.com> Message-ID: <20130724101759.7f608f12@pitrou.net> Le Wed, 24 Jul 2013 09:01:30 +0200, Ronald Oussoren a ?crit : > > On 24 Jul, 2013, at 8:43, Gregory P. Smith wrote: > > > > > On Tue, Jul 23, 2013 at 8:46 AM, Ronald Oussoren > > wrote: > > > > On 23 Jul, 2013, at 17:36, Christian Heimes > > wrote: > > > > > Am 23.07.2013 17:10, schrieb Benjamin Peterson: > > >>> PyErr_SetFromErrno() already and always returns NULL. Or do you > > >>> prefer to return NULL explicitly? > > >> > > >> It might always return NULL, but the compiler sees (PyObject > > >> *)NULL when this function returns dl_funcptr. > > > > > > Oh, you are right. I must have missed the compiler warning. How > > > about we turn type return and type assignment warnings into fatal > > > errors? > > > > That's probably possible with a '-Werror=....' argument. But please > > consider issue 18211 before unconditionally adding such a flag, as > > that issue mentions new compiler flags also get used when compiling > > 3th-party extensions. > > > > I guess there needs to be (yet) another CFLAGS_xxx variable in the > > Makefile that gets added to $(CFLAGS) during the build of Python > > itself, but is ignored by distutils and sysconfig. > > > > It seems fair to turn those on in 3.4 and require that third party > > extensions clean up their code when porting from 3.3 to 3.4. > > In this case its "just" code cleanup, the issue I filed (see above) > is for another -Werror flag that causes compile errors with some > valid C99 code that isn't valid C89. That's good for CPython itself > because its source code is explicitly C89, but is not good when > building 3th-party extensions. Agreed. We shouldn't impose specific error flags on 3rd-party extension writers. Regards Antoine. From bkabrda at redhat.com Wed Jul 24 11:12:59 2013 From: bkabrda at redhat.com (Bohuslav Kabrda) Date: Wed, 24 Jul 2013 05:12:59 -0400 (EDT) Subject: [Python-Dev] Python 3 as a Default in Linux Distros In-Reply-To: <731599098.5891653.1374652024043.JavaMail.root@redhat.com> Message-ID: <1903981217.6011829.1374657179869.JavaMail.root@redhat.com> Hi all, in recent days, there has been a discussion on fedora-devel (see thread [1]) about moving to Python 3 as a default. I'd really love to hear opinions on the matter from the upstream, mainly regarding these two points (that are not that clearly defined in my original proposal and have been formed during the discussion): - Should we point /usr/bin/python to Python 3 when we make the move? I know that pep 394 [2] deals with this and it says that /usr/bin/python may refer to Python 3 on some bleeding edge distributions - supposedly, this was added to the pep because of what Arch Linux did, not the other way round. As the pep says, the recommendation of pointing /usr/bin/python to Python 2 may be changed after the Python 3 ecosystem is sufficiently mature. I'm wondering if there are any more specific criteria - list of big projects migrated/ported or something like that - or will this be judged by what I'd call "overall spirit" in Python community (I hope you know what I mean by this)? In Fedora, we have two concerns that clash in this decision - being "First" (e.g. actively promote and use new technologies and also suggest them to our users) vs. not breaking user expectations. So we figured it'd be a good idea to ask upstream to get more opinions on this. - What should user get after using "yum install python"? There are basically few ways of coping with this: 1) Just keep doing what we do, eventually far in the future drop "python" package and never provide it again (= go on only with python3/python4/... while having "yum install python" do nothing). 2) Do what is in 1), but when "python" is dropped, use virtual provide (*) "python" for python3 package, so that "yum install python" installs python3. 3), 4) Rename python to python2 and {don't add, add} virtual provide "python" in the same way that is in 1), 2) 5) Rename python to python2 and python3 to python at one point. This makes sense to me from the traditional "one version in distro + possibly compat package shipping the old" approach in Linux, but some say that Python 2 and Python 3 are just different languages [3] and this should never be done. All of the approaches have their pros and cons, but generally it is all about what user should get when he tries to install python - either nothing or python2 for now and python3 in future - and how we as a distro cope with that on the technical side (and when we should actually do the switch). Just as a sidenote, IMO the package that gets installed as "python" (if any) should point to /usr/bin/python, which makes consider these two points very closely coupled. Thank you all for your suggestions and opinions. Slavek. -- Regards, Bohuslav "Slavek" Kabrda. (*) You can think of a virtual provide as of a "packaging symlink" - adding "Provides: foo" to package makes "yum install foo" install this package, even if this package name is "bar". [1] https://lists.fedoraproject.org/pipermail/devel/2013-July/186098.html [2] http://www.python.org/dev/peps/pep-0394/ [3] https://lists.fedoraproject.org/pipermail/devel/2013-July/186186.html From rosuav at gmail.com Wed Jul 24 12:16:33 2013 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 24 Jul 2013 20:16:33 +1000 Subject: [Python-Dev] Python 3 as a Default in Linux Distros In-Reply-To: <1903981217.6011829.1374657179869.JavaMail.root@redhat.com> References: <731599098.5891653.1374652024043.JavaMail.root@redhat.com> <1903981217.6011829.1374657179869.JavaMail.root@redhat.com> Message-ID: On Wed, Jul 24, 2013 at 7:12 PM, Bohuslav Kabrda wrote: > - What should user get after using "yum install python"? > There are basically few ways of coping with this: > 1) Just keep doing what we do, eventually far in the future drop "python" package and never provide it again (= go on only with python3/python4/... while having "yum install python" do nothing). > 2) Do what is in 1), but when "python" is dropped, use virtual provide (*) "python" for python3 package, so that "yum install python" installs python3. > 3), 4) Rename python to python2 and {don't add, add} virtual provide "python" in the same way that is in 1), 2) > 5) Rename python to python2 and python3 to python at one point. This makes sense to me from the traditional "one version in distro + possibly compat package shipping the old" approach in Linux, but some say that Python 2 and Python 3 are just different languages [3] and this should never be done. If a user's (rather than dev's) perspective is of value, I'm a Debian and Ubuntu user who wields Python as a multi-context weapon. Typing "packagemanager install python" (for me that's apt-get, for you yum, same diff) should get the one against which most other packages are going to depend. Since your policy is to move people forward, that's most likely going to be Python 3, unless you have a lot of packages that simply haven't yet moved. Your fifth option seems messy. I'd rather see python2 and python3 as viable packages, and then use the virtual provide to symlink 'python' to one of them, probably py3. So I'd vote for option 4. Python 2 and 3 aren't so different as to be called separate languages, there just happen to be some backward-incompatible changes. ChrisA From dirkjan at ochtman.nl Wed Jul 24 12:33:07 2013 From: dirkjan at ochtman.nl (Dirkjan Ochtman) Date: Wed, 24 Jul 2013 12:33:07 +0200 Subject: [Python-Dev] Python 3 as a Default in Linux Distros In-Reply-To: <1903981217.6011829.1374657179869.JavaMail.root@redhat.com> References: <731599098.5891653.1374652024043.JavaMail.root@redhat.com> <1903981217.6011829.1374657179869.JavaMail.root@redhat.com> Message-ID: On Wed, Jul 24, 2013 at 11:12 AM, Bohuslav Kabrda wrote: > - What should user get after using "yum install python"? > There are basically few ways of coping with this: > 1) Just keep doing what we do, eventually far in the future drop "python" package and never provide it again (= go on only with python3/python4/... while having "yum install python" do nothing). > 2) Do what is in 1), but when "python" is dropped, use virtual provide (*) "python" for python3 package, so that "yum install python" installs python3. > 3), 4) Rename python to python2 and {don't add, add} virtual provide "python" in the same way that is in 1), 2) > 5) Rename python to python2 and python3 to python at one point. This makes sense to me from the traditional "one version in distro + possibly compat package shipping the old" approach in Linux, but some say that Python 2 and Python 3 are just different languages [3] and this should never be done. > All of the approaches have their pros and cons, but generally it is all about what user should get when he tries to install python - either nothing or python2 for now and python3 in future - and how we as a distro cope with that on the technical side (and when we should actually do the switch). > Just as a sidenote, IMO the package that gets installed as "python" (if any) should point to /usr/bin/python, which makes consider these two points very closely coupled. On Gentoo we get python2 and python3 executables and have user-level tools to change what the 'python' symlink points to. I think we default to only having Python 3 installed (and Python 3 is even made the default Python), though it's currently always the case that Python 2 gets pulled in by some core dependencies. Cheers, Dirkjan From p.f.moore at gmail.com Wed Jul 24 12:36:25 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 24 Jul 2013 11:36:25 +0100 Subject: [Python-Dev] Python 3 as a Default in Linux Distros In-Reply-To: <1903981217.6011829.1374657179869.JavaMail.root@redhat.com> References: <731599098.5891653.1374652024043.JavaMail.root@redhat.com> <1903981217.6011829.1374657179869.JavaMail.root@redhat.com> Message-ID: On 24 July 2013 10:12, Bohuslav Kabrda wrote: > - What should user get after using "yum install python"? > There are basically few ways of coping with this: > 1) Just keep doing what we do, eventually far in the future drop "python" > package and never provide it again (= go on only with python3/python4/... > while having "yum install python" do nothing). > 2) Do what is in 1), but when "python" is dropped, use virtual provide (*) > "python" for python3 package, so that "yum install python" installs python3. > 3), 4) Rename python to python2 and {don't add, add} virtual provide > "python" in the same way that is in 1), 2) > 5) Rename python to python2 and python3 to python at one point. This makes > sense to me from the traditional "one version in distro + possibly compat > package shipping the old" approach in Linux, but some say that Python 2 and > Python 3 are just different languages [3] and this should never be done. > I'm not a Unix user, but IMO any approach that has the simple package name and command "python" disappear or become deprecated, makes me sad. I think that "yum install python" should install Python, and "python" should run Python. Frankly, the differences between Python 2 and Python 3 are *not* big enough that users writing simple scripts and applications for personal or local use can't cope with this. Not everyone writes major libraries, and it's the casual users you should be looking at. There may be a debate over *when* this usage should give the user Python 3 rather than Python 2 (I'm very much an early adopter by instinct, so I'd say sooner rather than later, but others may differ) but that's secondary. Paul -------------- next part -------------- An HTML attachment was scrubbed... URL: From fijall at gmail.com Wed Jul 24 13:53:22 2013 From: fijall at gmail.com (Maciej Fijalkowski) Date: Wed, 24 Jul 2013 13:53:22 +0200 Subject: [Python-Dev] Python 3 as a Default in Linux Distros In-Reply-To: References: <731599098.5891653.1374652024043.JavaMail.root@redhat.com> <1903981217.6011829.1374657179869.JavaMail.root@redhat.com> Message-ID: On Wed, Jul 24, 2013 at 12:36 PM, Paul Moore wrote: > On 24 July 2013 10:12, Bohuslav Kabrda wrote: >> >> - What should user get after using "yum install python"? >> There are basically few ways of coping with this: >> 1) Just keep doing what we do, eventually far in the future drop "python" >> package and never provide it again (= go on only with python3/python4/... >> while having "yum install python" do nothing). >> 2) Do what is in 1), but when "python" is dropped, use virtual provide (*) >> "python" for python3 package, so that "yum install python" installs python3. >> 3), 4) Rename python to python2 and {don't add, add} virtual provide >> "python" in the same way that is in 1), 2) >> 5) Rename python to python2 and python3 to python at one point. This makes >> sense to me from the traditional "one version in distro + possibly compat >> package shipping the old" approach in Linux, but some say that Python 2 and >> Python 3 are just different languages [3] and this should never be done. > > > I'm not a Unix user, but IMO any approach that has the simple package name > and command "python" disappear or become deprecated, makes me sad. I think > that "yum install python" should install Python, and "python" should run > Python. Frankly, the differences between Python 2 and Python 3 are *not* big > enough that users writing simple scripts and applications for personal or > local use can't cope with this. Not everyone writes major libraries, and > it's the casual users you should be looking at. the problem is not that. The problem is that people *use* major libraries a lot. From brett at python.org Wed Jul 24 15:34:11 2013 From: brett at python.org (Brett Cannon) Date: Wed, 24 Jul 2013 09:34:11 -0400 Subject: [Python-Dev] Python 3 as a Default in Linux Distros In-Reply-To: <1903981217.6011829.1374657179869.JavaMail.root@redhat.com> References: <731599098.5891653.1374652024043.JavaMail.root@redhat.com> <1903981217.6011829.1374657179869.JavaMail.root@redhat.com> Message-ID: On Wed, Jul 24, 2013 at 5:12 AM, Bohuslav Kabrda wrote: > Hi all, > in recent days, there has been a discussion on fedora-devel (see thread > [1]) about moving to Python 3 as a default. > I'd really love to hear opinions on the matter from the upstream, mainly > regarding these two points (that are not that clearly defined in my > original proposal and have been formed during the discussion): > > - Should we point /usr/bin/python to Python 3 when we make the move? > I know that pep 394 [2] deals with this and it says that /usr/bin/python > may refer to Python 3 on some bleeding edge distributions - supposedly, > this was added to the pep because of what Arch Linux did, not the other way > round. > As the pep says, the recommendation of pointing /usr/bin/python to Python > 2 may be changed after the Python 3 ecosystem is sufficiently mature. I'm > wondering if there are any more specific criteria - list of big projects > migrated/ported or something like that - or will this be judged by what I'd > call "overall spirit" in Python community (I hope you know what I mean by > this)? > In Fedora, we have two concerns that clash in this decision - being > "First" (e.g. actively promote and use new technologies and also suggest > them to our users) vs. not breaking user expectations. So we figured it'd > be a good idea to ask upstream to get more opinions on this. > > - What should user get after using "yum install python"? > There are basically few ways of coping with this: > 1) Just keep doing what we do, eventually far in the future drop "python" > package and never provide it again (= go on only with python3/python4/... > while having "yum install python" do nothing). > 2) Do what is in 1), but when "python" is dropped, use virtual provide (*) > "python" for python3 package, so that "yum install python" installs python3. > 3), 4) Rename python to python2 and {don't add, add} virtual provide > "python" in the same way that is in 1), 2) > 5) Rename python to python2 and python3 to python at one point. This makes > sense to me from the traditional "one version in distro + possibly compat > package shipping the old" approach in Linux, but some say that Python 2 and > Python 3 are just different languages [3] and this should never be done. > All of the approaches have their pros and cons, but generally it is all > about what user should get when he tries to install python - either nothing > or python2 for now and python3 in future - and how we as a distro cope with > that on the technical side (and when we should actually do the switch). > Just as a sidenote, IMO the package that gets installed as "python" (if > any) should point to /usr/bin/python, which makes consider these two points > very closely coupled. > A similar discussion broke out when Arch Linux switched python to point to python3. This led to http://www.python.org/dev/peps/pep-0394/ which says have python2/python3, and have python point at whatever makes the most sense to you based on your users and version uptake (option 3/4). The key, though, is adding python2 and getting your code to use that binary specifically so that shifting the default name is more of a convenience than something which might break existing code not ready for the switch. -------------- next part -------------- An HTML attachment was scrubbed... URL: From regebro at gmail.com Wed Jul 24 15:54:29 2013 From: regebro at gmail.com (Lennart Regebro) Date: Wed, 24 Jul 2013 15:54:29 +0200 Subject: [Python-Dev] Python 3 as a Default in Linux Distros In-Reply-To: <1903981217.6011829.1374657179869.JavaMail.root@redhat.com> References: <731599098.5891653.1374652024043.JavaMail.root@redhat.com> <1903981217.6011829.1374657179869.JavaMail.root@redhat.com> Message-ID: On Wed, Jul 24, 2013 at 11:12 AM, Bohuslav Kabrda wrote: > - Should we point /usr/bin/python to Python 3 when we make the move? No. > - What should user get after using "yum install python"? Will a base install include Python 3? If it does, I think yum install python should mean python3, and hence already be installed. //Lennart From dwightdhutto at gmail.com Wed Jul 24 16:09:11 2013 From: dwightdhutto at gmail.com (David Hutto) Date: Wed, 24 Jul 2013 10:09:11 -0400 Subject: [Python-Dev] Python 3 as a Default in Linux Distros In-Reply-To: References: <731599098.5891653.1374652024043.JavaMail.root@redhat.com> <1903981217.6011829.1374657179869.JavaMail.root@redhat.com> Message-ID: Haven't been over to yum land in a while, but usually they stabilize the distro in synergy to the python version. Ubuntu came with 2.7, and I just use sudo-apt get install, for 3.2(or a gui package manager), and then can use 3.2 with using the command python3.2 in command line, or you can compile from source with a newer python not available with your distro's package manager/repository. It's really up to your distro's base linux, and python version decided on top of a specific distro they define as compatible with the distros system. On Wed, Jul 24, 2013 at 9:54 AM, Lennart Regebro wrote: > On Wed, Jul 24, 2013 at 11:12 AM, Bohuslav Kabrda > wrote: > > - Should we point /usr/bin/python to Python 3 when we make the move? > > No. > > > - What should user get after using "yum install python"? > > Will a base install include Python 3? If it does, I think yum install > python should mean python3, and hence already be installed. > > //Lennart > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/dwightdhutto%40gmail.com > -- Best Regards, David Hutto *CEO:* *http://www.hitwebdevelopment.com* -------------- next part -------------- An HTML attachment was scrubbed... URL: From martin at v.loewis.de Wed Jul 24 17:10:27 2013 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed, 24 Jul 2013 17:10:27 +0200 Subject: [Python-Dev] Python 3 as a Default in Linux Distros In-Reply-To: <1903981217.6011829.1374657179869.JavaMail.root@redhat.com> References: <1903981217.6011829.1374657179869.JavaMail.root@redhat.com> Message-ID: <51EFEE63.5080702@v.loewis.de> Am 24.07.13 11:12, schrieb Bohuslav Kabrda: > - Should we point /usr/bin/python to Python 3 when we make the move? This should depend on the answer to this question: - for how long have you been providing /usr/bin/python2 binaries? Users "should" have been explicit in declaring scripts as /usr/bin/python2 for quite some time now, except that this would break on distros which don't provide a python2 symlink. Ideally, you "should" have warned users to be explicit with python2 if the script wouldn't work on python3. But I'd wave this requirement, as there is already the upstream PEP. So (IMO) if it the last three Fedora releases had been providing python2 binaries, it should be allowed to switch python to be python3. The exact number can be debated; it should depend on what releases are still in active use so that a script would run on multiple releases. Then you can tell users that possible breakage is easy to fix: just install python2, and change the shebang line. Regards, Martin From a.badger at gmail.com Wed Jul 24 17:26:44 2013 From: a.badger at gmail.com (Toshio Kuratomi) Date: Wed, 24 Jul 2013 08:26:44 -0700 Subject: [Python-Dev] Python 3 as a Default in Linux Distros In-Reply-To: References: <731599098.5891653.1374652024043.JavaMail.root@redhat.com> <1903981217.6011829.1374657179869.JavaMail.root@redhat.com> Message-ID: <20130724152644.GI11402@unaka.lan> Note: I'm the opposite number to bkabrda in the discussion on the Fedora Lists about how quickly we should be breaking end-user expectations of what "python" means. On Wed, Jul 24, 2013 at 09:34:11AM -0400, Brett Cannon wrote: > > > > On Wed, Jul 24, 2013 at 5:12 AM, Bohuslav Kabrda wrote: > > Hi all, > in recent days, there has been a discussion on fedora-devel (see thread > [1]) about moving to Python 3 as a default. > I'd really love to hear opinions on the matter from the upstream, mainly > regarding these two points (that are not that clearly defined in my > original proposal and have been formed during the discussion): > Note that the proposal is for Fedora 22. So the timeframe for making the switch in development is approximately 8 months from now. Timeframe for that release to be public is October 2014. > - Should we point /usr/bin/python to Python 3 when we make the move? > I know that pep 394 [2] deals with this and it says that /usr/bin/python > may refer to Python 3 on some bleeding edge distributions - supposedly, > this was added to the pep because of what Arch Linux did, not the other way > round. > As the pep says, the recommendation of pointing /usr/bin/python to Python 2 > may be changed after the Python 3 ecosystem is sufficiently mature. I'm > wondering if there are any more specific criteria - list of big projects > migrated/ported or something like that - or will this be judged by what I'd > call "overall spirit" in Python community (I hope you know what I mean by > this)? > In Fedora, we have two concerns that clash in this decision - being "First" > (e.g. actively promote and use new technologies and also suggest them to > our users) vs. not breaking user expectations. So we figured it'd be a good > idea to ask upstream to get more opinions on this. > > - What should user get after using "yum install python"? > There are basically few ways of coping with this: > 1) Just keep doing what we do, eventually far in the future drop "python" > package and never provide it again (= go on only with python3/python4/... > while having "yum install python" do nothing). > 2) Do what is in 1), but when "python" is dropped, use virtual provide (*) > "python" for python3 package, so that "yum install python" installs > python3. > 3), 4) Rename python to python2 and {don't add, add} virtual provide > "python" in the same way that is in 1), 2) > 4) Is my preference: python package becomes python2; Virtual Provide: python means you'd get the python package is what I'd promote for now. Users still expect python2 when they talk about "python". At some point in the future, people will come to pycon and talks will apply to python3 unless otherwise specified. People writing new blog posts will say "python" and the code they use as samples won't run on the python2 interpreter. Expecting for that to be the case in 12 months seems premature. > 5) Rename python to python2 and python3 to python at one point. This makes > sense to me from the traditional "one version in distro + possibly compat > package shipping the old" approach in Linux, but some say that Python 2 and > Python 3 are just different languages [3] and this should never be done. > All of the approaches have their pros and cons, but generally it is all > about what user should get when he tries to install python - either nothing > or python2 for now and python3 in future - and how we as a distro cope with > that on the technical side (and when we should actually do the switch). > Just as a sidenote, IMO the package that gets installed as "python" (if > any) should point to /usr/bin/python, which makes consider these two points > very closely coupled. > > > A similar discussion broke out when Arch Linux switched python to point to > python3. This led to?http://www.python.org/dev/peps/pep-0394/?which says have > python2/python3, and have python point at whatever makes the most sense to you > based on your users and version uptake (option 3/4). I think bkabrda is looking for some clarification on PEP-394. My reading and participation in the previous discussions lead me to believe that while PEP-394 wants to be diplomatic, the message it wants to get across is: 1) warn distributions that what Arch was doing was premature. 2) provide a means to get them to switch at roughly the same time (when the recommendation in the PEP is flipped to suggest linking /usr/bin/python to /usr/bin/python3) This is especially my reading from the Recommendations section of the PEP. Unfortunately, we're getting stuck in the Abstract section which has this bullet point: * python should refer to the same target as python2 but may refer to python3 on some bleeding edge distributions Knowing the history, I read this in two parts: * Recommendation to distributions: "python should refer to the same target as python2". * Statement of fact: "but may refer to python3 on some bleeding edge *ahem*Arch*ahem* distributions" However, other people are reading this as one recommendation for distributions: If you are a conservative, slow moving distro (like RHEL or Debian Stable) then you should point to python2. If you are a fast moving distro like Arch or Fedora or Ubuntu then you should point to python3. So -- is there some opinion on which of these is correct? > The key, though, is adding > python2 and getting your code to use that binary??specifically so that shifting > the default name is more of a convenience than something which might break > existing code not ready for the switch. yeah, this is something that I think should be in the distribution's one year plan regardless of whether the binary is switched now or not. Note that I (and from Nick's last email on the fedora thread, I think he would agree) consider this a prerequisite but not the primary gating factor for shifting /usr/bin/python to point to /usr/bin/python3. In addition to the applications managed by the distribution there's also the mass of local scripts written for each site that will break if the interpreter is switched. I proposed above some of the things that would indicate user expectation has switched and make it appropriate to break scripts that haven't ported (because those users and admins would look at the world around them and see that they were overdue on changing their expectation that python meant python2 instead of python3.) Barry Warsaw, on IRC, also proposed that letting the upstream python core team make the decision by switching at the distribution level when the core team switched the recommendation in PEP-394 made sense and I can agree with that as well. -Toshio -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 198 bytes Desc: not available URL: From ncoghlan at gmail.com Wed Jul 24 17:41:33 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 25 Jul 2013 01:41:33 +1000 Subject: [Python-Dev] Python 3 as a Default in Linux Distros In-Reply-To: <20130724152644.GI11402@unaka.lan> References: <731599098.5891653.1374652024043.JavaMail.root@redhat.com> <1903981217.6011829.1374657179869.JavaMail.root@redhat.com> <20130724152644.GI11402@unaka.lan> Message-ID: On 25 July 2013 01:26, Toshio Kuratomi wrote: > On Wed, Jul 24, 2013 at 09:34:11AM -0400, Brett Cannon wrote: >> A similar discussion broke out when Arch Linux switched python to point to >> python3. This led to http://www.python.org/dev/peps/pep-0394/ which says have >> python2/python3, and have python point at whatever makes the most sense to you >> based on your users and version uptake (option 3/4). > > I think bkabrda is looking for some clarification on PEP-394. My > reading and participation in the previous discussions lead me to believe > that while PEP-394 wants to be diplomatic, the message it wants to get > across is: > > 1) warn distributions that what Arch was doing was premature. > 2) provide a means to get them to switch at roughly the same time (when the > recommendation in the PEP is flipped to suggest linking /usr/bin/python > to /usr/bin/python3) > > This is especially my reading from the Recommendations section of the PEP. > Unfortunately, we're getting stuck in the Abstract section which has this > bullet point: > > * python should refer to the same target as python2 but may refer to python3 > on some bleeding edge distributions > > Knowing the history, I read this in two parts: > * Recommendation to distributions: "python should refer to the same target > as python2". > * Statement of fact: "but may refer to python3 on some bleeding edge > *ahem*Arch*ahem* distributions" > > However, other people are reading this as one recommendation for > distributions: > > If you are a conservative, slow moving distro (like RHEL or Debian Stable) > then you should point to python2. If you are a fast moving distro like > Arch or Fedora or Ubuntu then you should point to python3. > > So -- is there some opinion on which of these is correct? It's the first one, but I can definitely see how it could be read as the second one. How's this for an updated wording in the abstract: * for the time being, all distributions should ensure that python refers to the same target as python2 * however, users should be aware that python refers to python3 on at least Arch Linux (that change is what prompted the creation of this PEP), so "python" should be used in the shebang line only for scripts that are source compatible with both Python 2 and 3 That's what the previous bullet point was *trying* to say, but it wasn't very clear, since the first half of the sentence was aimed at distributions, while the second was aimed at end users. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Wed Jul 24 17:45:28 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 25 Jul 2013 01:45:28 +1000 Subject: [Python-Dev] Python 3 as a Default in Linux Distros In-Reply-To: References: <731599098.5891653.1374652024043.JavaMail.root@redhat.com> <1903981217.6011829.1374657179869.JavaMail.root@redhat.com> <20130724152644.GI11402@unaka.lan> Message-ID: On 25 July 2013 01:41, Nick Coghlan wrote: > How's this for an updated wording in the abstract: > > * for the time being, all distributions should ensure that python > refers to the same target as python2 > * however, users should be aware that python refers to python3 on at > least Arch Linux (that change is > what prompted the creation of this PEP), so "python" should be > used in the shebang line only for > scripts that are source compatible with both Python 2 and 3 > > That's what the previous bullet point was *trying* to say, but it > wasn't very clear, since the first half of the sentence was aimed at > distributions, while the second was aimed at end users. I'll also note that this is deliberately consistent with the behaviour of the "py" launcher on Windows (as defined in PEP 397): if you invoke that from the command with both Python 2.x and Python 3.x installed, it will default to using the most recently released version of Python 2, *unless* you pass a "-3" on the command line, or the shebang line indicates the script is a Python 3 script. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From regebro at gmail.com Wed Jul 24 17:56:08 2013 From: regebro at gmail.com (Lennart Regebro) Date: Wed, 24 Jul 2013 17:56:08 +0200 Subject: [Python-Dev] Python 3 as a Default in Linux Distros In-Reply-To: References: <731599098.5891653.1374652024043.JavaMail.root@redhat.com> <1903981217.6011829.1374657179869.JavaMail.root@redhat.com> Message-ID: On Wed, Jul 24, 2013 at 3:54 PM, Lennart Regebro wrote: > On Wed, Jul 24, 2013 at 11:12 AM, Bohuslav Kabrda wrote: >> - Should we point /usr/bin/python to Python 3 when we make the move? > > No. To be more explicit. I think it's perfectly fine to not provide a /usr/bin/python at all, but I think pointing it to Python 3 will provide unhelpful error messages. Also having a /usr/bin/python that does nothing but say "You should use either /usr/bin/python2 or /usr/bin/python3" would work. But pointing it to Python 3 will make for unhelpful and confusing error messages for anyone having Python scripts using /usr/bin/python, which is the majority of people having python scripts at all. //Lennart From barry at python.org Wed Jul 24 18:08:20 2013 From: barry at python.org (Barry Warsaw) Date: Wed, 24 Jul 2013 12:08:20 -0400 Subject: [Python-Dev] Python 3 as a Default in Linux Distros In-Reply-To: <1903981217.6011829.1374657179869.JavaMail.root@redhat.com> References: <731599098.5891653.1374652024043.JavaMail.root@redhat.com> <1903981217.6011829.1374657179869.JavaMail.root@redhat.com> Message-ID: <20130724120820.342b6075@anarchist> On Jul 24, 2013, at 05:12 AM, Bohuslav Kabrda wrote: >in recent days, there has been a discussion on fedora-devel (see thread [1]) >about moving to Python 3 as a default. I've been lurking via Gmane. :) >- Should we point /usr/bin/python to Python 3 when we make the move? Over in Debian (inherited by Ubuntu), we have no plans to do this. Currently, /usr/bin/python points to pythonX.Y which for us is 2.7. Our /usr/bin/python2 symlink also points to python2.7, and our /usr/bin/python3 points to python3.3. I'm very much in favor of promoting Python 3 adoption through the entire ecosystem. We've been working in Ubuntu to banish Python 2 off the images, eventually only in universe by the next LTS (14.04 - in April 2014). For practical reasons, I don't think we'll quite get there, but it's good to have goals, and I've been really impressed with the progress made by distro developers and upstreams. "Porting is one honking great idea -- let's do more of those!" We have a mandate that all new work (certainly where Canonical is the upstream for Ubuntu packages) should be Python 3 unless there are specific dependencies for which porting is impractical and there are no alternatives. We want no Python 2 on the Ubuntu Touch images. Some of the new work will retain Python 2 compatibility for now, again for practical reasons. Code we control is the easy part. There is just so much out there that isn't even in the distro, stuff that no distro maintainer will ever even know about (unless we break stuff ;), and all of that is expecting /usr/bin/python to be Python 2. If they want Python 3, they are explicitly shebanging /usr/bin/python3. I think Nick in one of the Fedora threads correctly identified the vast number of one-off scripts out there. At this time, I think keeping /usr/bin/python pointing to Python 2 (2.7 preferably) strikes the right balance between what we in Python land would like to see happen, with the least amount of pain, and friendly encouragement to our distro users. I hope someday we can remember Python 2 as fondly as we remember 1.5.2 (1.6.1). I think there's a ton of things we can do to get us there, but forcing our users through unnecessary pain too early won't help the cause. The last year or so has really shown a very positive shift toward Python 3, in code, mindshare, and enthusiasm. >I know that pep 394 [2] deals with this and it says that /usr/bin/python may >refer to Python 3 on some bleeding edge distributions - supposedly, this was >added to the pep because of what Arch Linux did, not the other way round. Correct. I think we should consider rewording the text to make this more clear. >As the pep says, the recommendation of pointing /usr/bin/python to Python 2 >may be changed after the Python 3 ecosystem is sufficiently mature. I'm >wondering if there are any more specific criteria - list of big projects >migrated/ported or something like that - or will this be judged by what I'd >call "overall spirit" in Python community (I hope you know what I mean by >this)? In the discussions I've had at Pycon and various other forum, it seems like Python 2.7's scheduled security-only transition in May 2015 would be an appropriate time to make the switch. If there's broad agreement to do this, we have plenty of time to prepare our users, and all of our software stacks. >- What should user get after using "yum install python"? I can only say that for Debian and Ubuntu, `apt-get install python` will get you the default Python 2 version (e.g. 2.7). `apt-get install python3` will get you the default Python 3 version (e.g. 3.3). I don't think changing this really provides much user benefit, and I've not seen any complaints/bugs indicating user confusion or frustration. I'd advocate for being user driven here, i.e. IIABDFI[1]. >All of the approaches have their pros and cons, but generally it is all about >what user should get when he tries to install python - either nothing or >python2 for now and python3 in future - and how we as a distro cope with that >on the technical side (and when we should actually do the switch). I'm glad Arch is out there hacking at the vines. For whatever part I can play and for whatever my opinion matters, I would like to keep the Fedora/RH and Debian/Ubuntu ecosystem in sync. PEP 394 is the instrument we should rally around. Cheers, -Barry [1] The secret 20th Zen of Python? If It Ain't Broke Don't Fix It. From lgautier at gmail.com Wed Jul 24 18:21:54 2013 From: lgautier at gmail.com (Laurent Gautier) Date: Wed, 24 Jul 2013 18:21:54 +0200 Subject: [Python-Dev] Python 3 as a Default in Linux Distros In-Reply-To: References: <731599098.5891653.1374652024043.JavaMail.root@redhat.com> <1903981217.6011829.1374657179869.JavaMail.root@redhat.com> Message-ID: <51EFFF22.1070603@gmail.com> On 07/24/2013 05:56 PM, Lennart Regebro wrote: > On Wed, Jul 24, 2013 at 3:54 PM, Lennart Regebro wrote: >> On Wed, Jul 24, 2013 at 11:12 AM, Bohuslav Kabrda wrote: >>> - Should we point /usr/bin/python to Python 3 when we make the move? >> No. > To be more explicit. I think it's perfectly fine to not provide a > /usr/bin/python at all, but I think pointing it to Python 3 will > provide unhelpful error messages. Also having a /usr/bin/python that > does nothing but say "You should use either /usr/bin/python2 or > /usr/bin/python3" would work. > > But pointing it to Python 3 will make for unhelpful and confusing > error messages for anyone having Python scripts using /usr/bin/python, > which is the majority of people having python scripts at all. I guess that the underlying idea is to make the majority of python scripts working with Python 3. What about the following idea ? - errors that are typical of "Python 2 script running with Python 3"-specific are probably limited (e.g., use of unicode, use of xrange, etc...) - /usr/bin/python could be a wrapper that tries to run it with Python 3 and display an informative message if the errors above are occurring (e.g., "Python 3 is the current release, and this script might be designed for the older Python 2 - please explicitly use python2 if you need it" - /usr/bin/python could issue a warning in any case about the need to be explicit about python2/python3 > > //Lennart > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/lgautier%40gmail.com From martin at v.loewis.de Wed Jul 24 18:05:26 2013 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed, 24 Jul 2013 18:05:26 +0200 Subject: [Python-Dev] Python 3 as a Default in Linux Distros In-Reply-To: References: <731599098.5891653.1374652024043.JavaMail.root@redhat.com> <1903981217.6011829.1374657179869.JavaMail.root@redhat.com> Message-ID: <51EFFB46.2040300@v.loewis.de> Am 24.07.13 17:56, schrieb Lennart Regebro:> On Wed, Jul 24, 2013 at 3:54 PM, Lennart Regebro wrote: >> On Wed, Jul 24, 2013 at 11:12 AM, Bohuslav Kabrda wrote: >>> - Should we point /usr/bin/python to Python 3 when we make the move? >> >> No. > > To be more explicit. I think it's perfectly fine to not provide a > /usr/bin/python at all I disagree. For interactive use, it's good if there is something called "python" in the path. Regards, Martin From rosuav at gmail.com Wed Jul 24 18:30:51 2013 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 25 Jul 2013 02:30:51 +1000 Subject: [Python-Dev] Python 3 as a Default in Linux Distros In-Reply-To: <51EFFF22.1070603@gmail.com> References: <731599098.5891653.1374652024043.JavaMail.root@redhat.com> <1903981217.6011829.1374657179869.JavaMail.root@redhat.com> <51EFFF22.1070603@gmail.com> Message-ID: On Thu, Jul 25, 2013 at 2:21 AM, Laurent Gautier wrote: > - errors that are typical of "Python 2 script running with Python > 3"-specific are probably limited (e.g., use of unicode, use of xrange, > etc...) > The most common, in interactive scripts at least, is likely to be: >>> print "Hello, world!" SyntaxError: invalid syntax How helpful it's possible to make that one, I don't know. Is it safe to presume that it's more likely a syntax error will come from an interpreter version mismatch than a code bug? ChrisA From brian at python.org Wed Jul 24 18:34:58 2013 From: brian at python.org (Brian Curtin) Date: Wed, 24 Jul 2013 11:34:58 -0500 Subject: [Python-Dev] Python 3 as a Default in Linux Distros In-Reply-To: References: <731599098.5891653.1374652024043.JavaMail.root@redhat.com> <1903981217.6011829.1374657179869.JavaMail.root@redhat.com> Message-ID: On Wed, Jul 24, 2013 at 10:56 AM, Lennart Regebro wrote: > On Wed, Jul 24, 2013 at 3:54 PM, Lennart Regebro wrote: >> On Wed, Jul 24, 2013 at 11:12 AM, Bohuslav Kabrda wrote: >>> - Should we point /usr/bin/python to Python 3 when we make the move? >> >> No. > > To be more explicit. I think it's perfectly fine to not provide a > /usr/bin/python at all, but I think pointing it to Python 3 will > provide unhelpful error messages. -1. For one example, this is going to add non-productive and potentially confusing steps before any educator can use Python to teach anything. From barry at python.org Wed Jul 24 18:40:15 2013 From: barry at python.org (Barry Warsaw) Date: Wed, 24 Jul 2013 12:40:15 -0400 Subject: [Python-Dev] Python 3 as a Default in Linux Distros In-Reply-To: <20130724152644.GI11402@unaka.lan> References: <731599098.5891653.1374652024043.JavaMail.root@redhat.com> <1903981217.6011829.1374657179869.JavaMail.root@redhat.com> <20130724152644.GI11402@unaka.lan> Message-ID: <20130724124015.05d79275@anarchist> On Jul 24, 2013, at 08:26 AM, Toshio Kuratomi wrote: > I think bkabrda is looking for some clarification on PEP-394. My >reading and participation in the previous discussions lead me to believe >that while PEP-394 wants to be diplomatic, the message it wants to get >across is: > >1) warn distributions that what Arch was doing was premature. >2) provide a means to get them to switch at roughly the same time (when the > recommendation in the PEP is flipped to suggest linking /usr/bin/python > to /usr/bin/python3) +1 >This is especially my reading from the Recommendations section of the PEP. >Unfortunately, we're getting stuck in the Abstract section which has this >bullet point: > >* python should refer to the same target as python2 but may refer to python3 >on some bleeding edge distributions Yes, that wording is confusing and should be fixed. >Knowing the history, I read this in two parts: >* Recommendation to distributions: "python should refer to the same target > as python2". >* Statement of fact: "but may refer to python3 on some bleeding edge > *ahem*Arch*ahem* distributions" > >However, other people are reading this as one recommendation for >distributions: > >If you are a conservative, slow moving distro (like RHEL or Debian Stable) >then you should point to python2. If you are a fast moving distro like >Arch or Fedora or Ubuntu then you should point to python3. > >So -- is there some opinion on which of these is correct? You know how I'm going to answer. :) I like the way you've split that out into recommendations and statements-of-fact. >> The key, though, is adding python2 and getting your code to use that >> binary??specifically so that shifting the default name is more of a >> convenience than something which might break existing code not ready for >> the switch. > > yeah, this is something that I think should be in the distribution's >one year plan regardless of whether the binary is switched now or not. We need to do that in Debian. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From barry at python.org Wed Jul 24 18:42:09 2013 From: barry at python.org (Barry Warsaw) Date: Wed, 24 Jul 2013 12:42:09 -0400 Subject: [Python-Dev] Python 3 as a Default in Linux Distros In-Reply-To: References: <731599098.5891653.1374652024043.JavaMail.root@redhat.com> <1903981217.6011829.1374657179869.JavaMail.root@redhat.com> <20130724152644.GI11402@unaka.lan> Message-ID: <20130724124209.61f8a8ed@anarchist> On Jul 25, 2013, at 01:41 AM, Nick Coghlan wrote: >How's this for an updated wording in the abstract: > > * for the time being, all distributions should ensure that python >refers to the same target as python2 > * however, users should be aware that python refers to python3 on at >least Arch Linux (that change is > what prompted the creation of this PEP), so "python" should be >used in the shebang line only for > scripts that are source compatible with both Python 2 and 3 +1 -Barry From tjreedy at udel.edu Wed Jul 24 19:04:43 2013 From: tjreedy at udel.edu (Terry Reedy) Date: Wed, 24 Jul 2013 13:04:43 -0400 Subject: [Python-Dev] Python 3 as a Default in Linux Distros In-Reply-To: <1903981217.6011829.1374657179869.JavaMail.root@redhat.com> References: <731599098.5891653.1374652024043.JavaMail.root@redhat.com> <1903981217.6011829.1374657179869.JavaMail.root@redhat.com> Message-ID: On 7/24/2013 5:12 AM, Bohuslav Kabrda wrote: > Hi all, in recent days, there has been a discussion on fedora-devel > (see thread [1]) about moving to Python 3 as a default. Default-shift is a known natural language phenomenon. https://en.wikipedia.org/wiki/Retronym It is inevitably messy in the middle. Even worse for us, "Python' is both a natural and artificial language term and the default meaning will not necessarily be synchronized across the two domains. Not being a current *nix user, I have no opinion on the specific alternatives. However, I am concerned about newcomers being mis-served by being mislead into thinking that 'Python' is python 2 rather python 3. We see this in some newbie posts on Python list. (This is especially a concern when default Python is still, say, 2.5.) I have no idea, though, how to avoid breaking scripts while directing newcomers in the proper direction. -- Terry Jan Reedy From gjcarneiro at gmail.com Wed Jul 24 19:16:00 2013 From: gjcarneiro at gmail.com (Gustavo Carneiro) Date: Wed, 24 Jul 2013 18:16:00 +0100 Subject: [Python-Dev] Python 3 as a Default in Linux Distros In-Reply-To: References: <731599098.5891653.1374652024043.JavaMail.root@redhat.com> <1903981217.6011829.1374657179869.JavaMail.root@redhat.com> Message-ID: On Wed, Jul 24, 2013 at 6:04 PM, Terry Reedy wrote: > On 7/24/2013 5:12 AM, Bohuslav Kabrda wrote: > >> Hi all, in recent days, there has been a discussion on fedora-devel >> (see thread [1]) about moving to Python 3 as a default. >> > > Default-shift is a known natural language phenomenon. > https://en.wikipedia.org/wiki/**Retronym > It is inevitably messy in the middle. Even worse for us, "Python' is both > a natural and artificial language term and the default meaning will not > necessarily be synchronized across the two domains. > > Not being a current *nix user, I have no opinion on the specific > alternatives. However, I am concerned about newcomers being mis-served by > being mislead into thinking that 'Python' is python 2 rather python 3. We > see this in some newbie posts on Python list. (This is especially a concern > when default Python is still, say, 2.5.) I have no idea, though, how to > avoid breaking scripts while directing newcomers in the proper direction. +1 Maybe we could make interactive python2 scripts print a big warning advising users to use python 3 instead of 2, if possible? I mean, if stdout is a TTY. This way, even if 'python' is python2, newbies will see the warning. I think every developer (newbie or not) sooner or later will try a python interactive shell, and so will see the warning. It is important also for experienced python developers to be told explicitly that they should be thinking about switching to python 3, at some point. > > > -- > Terry Jan Reedy > > > ______________________________**_________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/**mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/**mailman/options/python-dev/** > gjcarneiro%40gmail.com > -- Gustavo J. A. M. Carneiro "The universe is always one step beyond logic." -- Frank Herbert -------------- next part -------------- An HTML attachment was scrubbed... URL: From lgautier at gmail.com Wed Jul 24 19:22:35 2013 From: lgautier at gmail.com (Laurent Gautier) Date: Wed, 24 Jul 2013 19:22:35 +0200 Subject: [Python-Dev] Python 3 as a Default in Linux Distros In-Reply-To: References: <731599098.5891653.1374652024043.JavaMail.root@redhat.com> <1903981217.6011829.1374657179869.JavaMail.root@redhat.com> <51EFFF22.1070603@gmail.com> Message-ID: <51F00D5B.9060005@gmail.com> On 07/24/2013 06:30 PM, Chris Angelico wrote: > On Thu, Jul 25, 2013 at 2:21 AM, Laurent Gautier wrote: >> - errors that are typical of "Python 2 script running with Python >> 3"-specific are probably limited (e.g., use of unicode, use of xrange, >> etc...) >> > The most common, in interactive scripts at least, is likely to be: > >>>> print "Hello, world!" > SyntaxError: invalid syntax > > How helpful it's possible to make that one, I don't know. Is it safe > to presume that it's more likely a syntax error will come from an > interpreter version mismatch than a code bug? The wrapper in /usr/bin/python: - could use what is in 2to3. I think that most of the cases are solved there. - whenever interactive, could have an intermediate layer between the input in the console and execution. > > ChrisA > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/lgautier%40gmail.com From stefan_ml at behnel.de Wed Jul 24 19:33:42 2013 From: stefan_ml at behnel.de (Stefan Behnel) Date: Wed, 24 Jul 2013 19:33:42 +0200 Subject: [Python-Dev] Python 3 as a Default in Linux Distros In-Reply-To: References: <731599098.5891653.1374652024043.JavaMail.root@redhat.com> <1903981217.6011829.1374657179869.JavaMail.root@redhat.com> Message-ID: Gustavo Carneiro, 24.07.2013 19:16: > On Wed, Jul 24, 2013 at 6:04 PM, Terry Reedy wrote: >> On 7/24/2013 5:12 AM, Bohuslav Kabrda wrote: >>> Hi all, in recent days, there has been a discussion on fedora-devel >>> (see thread [1]) about moving to Python 3 as a default. >> >> Not being a current *nix user, I have no opinion on the specific >> alternatives. However, I am concerned about newcomers being mis-served by >> being mislead into thinking that 'Python' is python 2 rather python 3. We >> see this in some newbie posts on Python list. (This is especially a concern >> when default Python is still, say, 2.5.) I have no idea, though, how to >> avoid breaking scripts while directing newcomers in the proper direction. > > Maybe we could make interactive python2 scripts print a big warning > advising users to use python 3 instead of 2, if possible? Well, what I think could be done is to have the Py2 interpreter output a warning if an unpostfixed shebang is found in an executed script. That would eventually push users into either using the explicit "/usr/bin/env python2", or port to Py3 straight away (or both). Stefan From guido at python.org Wed Jul 24 19:56:05 2013 From: guido at python.org (Guido van Rossum) Date: Wed, 24 Jul 2013 10:56:05 -0700 Subject: [Python-Dev] Inherance of file descriptor and handles on Windows (PEP 446) In-Reply-To: References: Message-ID: On Tue, Jul 23, 2013 at 4:21 PM, Richard Oudkerk wrote: > On 23/07/2013 11:45pm, Victor Stinner wrote: >> >> Said differently: the HANDLE_FLAG_INHERIT flag only has an effect on >> *handles*, as indicated in its name. On Windows, file*descriptors* >> are never inherited (are always closed) in child processes. I don't >> think that it is possible to inherit file descriptors on Windows. > > > Actually, you can inherit fds if you use os.spawnv() instead of > subprocess.Popen(). Wow. Indeed you can -- I just tested this myself. How is this accomplished? I guess the CRT has a backchannel to talk to itself when it creates a process using spawn*? This is the only reason I can think of for the odd default in the CRT of opening file descriptors inheritable by default, which Victor discovered. (But it doesn't explain why os.pipe() creates uninheritable fds.) If it weren't for this I would definitely vote to change the default on Windows throughout the stdlib to create file descriptors whose handles aren't inheritable. (Perhaps with a different policy for stdin/stdout/stderr, which seem to be treated specially at the handle level.) I'm about ready to give up hope that we'll ever have a decent way to deal with this. But I'm also ready to propose that all this is such a mess that we *should* change the default fd/handle inheritance to False, *across platforms*, and damn the torpedoes -- i.e. accept breaking all existing 3rd party UNIX code for subprocess creation that bypasses the subprocess module, as well as breaking uses of os.spawn*() on both platforms that depend on FD inheritance beyond stdin/stdout/stderr). With the new, sane default, all we need instead of PEP 446 is a way to make an FD inheritable after it's been created, which can be a single os.make_inheritable(fd) call that you must apply to the fileno() of the stream or socket object you want inherited (or directly to a FD you created otherwise, e.g. with os.pipe()). On Windows, this should probably only work with os.spawn*(), since otherwise you need *handle* inheritance, not *FD* inheritance, and that's a non-portable concept anyway. We can fix multiprocessing any anything else in the stdlib that this breaks, I presume. To reduce the need for 3rd party subprocess creation code, we should have better daemon creation code in the stdlib -- I wrote some damn robust code for this purpose in my previous job, but it never saw the light of day. -- --Guido van Rossum (python.org/~guido) From shibturn at gmail.com Wed Jul 24 20:13:58 2013 From: shibturn at gmail.com (Richard Oudkerk) Date: Wed, 24 Jul 2013 19:13:58 +0100 Subject: [Python-Dev] Inherance of file descriptor and handles on Windows (PEP 446) In-Reply-To: References: Message-ID: <51F01966.7080908@gmail.com> > Wow. Indeed you can -- I just tested this myself. How is this > accomplished? I guess the CRT has a backchannel to talk to itself when > it creates a process using spawn*? CreateProcess() takes a STARTUPINFO argument with undocumented fields cbReserved2, lpReserved2. They are used to pass an array of fds. > But I'm also ready to propose that all this is such a mess that we > *should* change the default fd/handle inheritance to False, *across > platforms*, and damn the torpedoes -- i.e. accept breaking all > existing 3rd party UNIX code for subprocess creation that bypasses the > subprocess module, as well as breaking uses of os.spawn*() on both > platforms that depend on FD inheritance beyond stdin/stdout/stderr). +1 > We can fix multiprocessing any anything else in the stdlib that this > breaks, I presume. In the experimental branch of multiprocessing, child processes no longer inherit unnecessary handles. -- Richard From guido at python.org Wed Jul 24 20:17:13 2013 From: guido at python.org (Guido van Rossum) Date: Wed, 24 Jul 2013 11:17:13 -0700 Subject: [Python-Dev] Inherance of file descriptor and handles on Windows (PEP 446) In-Reply-To: <51F01966.7080908@gmail.com> References: <51F01966.7080908@gmail.com> Message-ID: On Wed, Jul 24, 2013 at 11:13 AM, Richard Oudkerk wrote: >> Wow. Indeed you can -- I just tested this myself. How is this >> accomplished? I guess the CRT has a backchannel to talk to itself when >> it creates a process using spawn*? > > CreateProcess() takes a STARTUPINFO argument with undocumented fields > cbReserved2, lpReserved2. They are used to pass an array of fds. Does it also inherit sockets (which take up a different namespace than regular FDs in CRT, unlike UNIX)? >> But I'm also ready to propose that all this is such a mess that we >> >> *should* change the default fd/handle inheritance to False, *across >> platforms*, and damn the torpedoes -- i.e. accept breaking all >> existing 3rd party UNIX code for subprocess creation that bypasses the >> subprocess module, as well as breaking uses of os.spawn*() on both >> platforms that depend on FD inheritance beyond stdin/stdout/stderr). > > +1 Thanks! This was a difficult conclusion to come to. "Damn the torpedoes" is occasionally a nice meme. :-( >> We can fix multiprocessing any anything else in the stdlib that this >> breaks, I presume. > > In the experimental branch of multiprocessing, child processes no longer > inherit unnecessary handles. And if we default individual handles to uninheritable, we can presumably fix the ones that multiprocessing creates with the express purpose of being inherited easily. (If it even uses that -- I haven't read the source code, maybe it uses named pipes? :-) -- --Guido van Rossum (python.org/~guido) From shibturn at gmail.com Wed Jul 24 20:36:48 2013 From: shibturn at gmail.com (Richard Oudkerk) Date: Wed, 24 Jul 2013 19:36:48 +0100 Subject: [Python-Dev] Inherance of file descriptor and handles on Windows (PEP 446) In-Reply-To: References: <51F01966.7080908@gmail.com> Message-ID: On 24/07/2013 7:17pm, Guido van Rossum wrote: > Does it also inherit sockets (which take up a different namespace than > regular FDs in CRT, unlike UNIX)? Not reliably. Processes created with CreateProcess() seems to inherit socket handles just like normal handles on my computer, but on some other computers -- with the same Windows version! -- it appears not to work. See http://bugs.python.org/issue17399. I think WSADuplicateSocket() should be used instead. > And if we default individual handles to uninheritable, we can > presumably fix the ones that multiprocessing creates with the express > purpose of being inherited easily. (If it even uses that -- I haven't > read the source code, maybe it uses named pipes? multiprocessing never really needs to create any inheritable handles: it can use DuplicateHandle() to transfer each handle directly to the child process. -- Richard From a.badger at gmail.com Wed Jul 24 21:27:16 2013 From: a.badger at gmail.com (Toshio Kuratomi) Date: Wed, 24 Jul 2013 12:27:16 -0700 Subject: [Python-Dev] Python 3 as a Default in Linux Distros In-Reply-To: <20130724124209.61f8a8ed@anarchist> References: <731599098.5891653.1374652024043.JavaMail.root@redhat.com> <1903981217.6011829.1374657179869.JavaMail.root@redhat.com> <20130724152644.GI11402@unaka.lan> <20130724124209.61f8a8ed@anarchist> Message-ID: <20130724192716.GJ11402@unaka.lan> On Wed, Jul 24, 2013 at 12:42:09PM -0400, Barry Warsaw wrote: > On Jul 25, 2013, at 01:41 AM, Nick Coghlan wrote: > > >How's this for an updated wording in the abstract: > > > > * for the time being, all distributions should ensure that python > >refers to the same target as python2 > > * however, users should be aware that python refers to python3 on at > >least Arch Linux (that change is > > what prompted the creation of this PEP), so "python" should be > >used in the shebang line only for > > scripts that are source compatible with both Python 2 and 3 > > +1 > +1 as well. Much clearer. -Toshio -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 198 bytes Desc: not available URL: From regebro at gmail.com Wed Jul 24 21:34:01 2013 From: regebro at gmail.com (Lennart Regebro) Date: Wed, 24 Jul 2013 21:34:01 +0200 Subject: [Python-Dev] Python 3 as a Default in Linux Distros In-Reply-To: <51F00D5B.9060005@gmail.com> References: <731599098.5891653.1374652024043.JavaMail.root@redhat.com> <1903981217.6011829.1374657179869.JavaMail.root@redhat.com> <51EFFF22.1070603@gmail.com> <51F00D5B.9060005@gmail.com> Message-ID: On Wed, Jul 24, 2013 at 7:22 PM, Laurent Gautier wrote: > The wrapper in /usr/bin/python: > - could use what is in 2to3. I think that most of the cases are solved > there. Only the most trivial cases are solved completely by 2to3, and the error messages you would get would be hard to understand, and running the scripts would be very slow. //Lennart From victor.stinner at gmail.com Wed Jul 24 23:50:08 2013 From: victor.stinner at gmail.com (Victor Stinner) Date: Wed, 24 Jul 2013 23:50:08 +0200 Subject: [Python-Dev] Inherance of file descriptor and handles on Windows (PEP 446) In-Reply-To: <51F01966.7080908@gmail.com> References: <51F01966.7080908@gmail.com> Message-ID: 2013/7/24 Richard Oudkerk : >> Wow. Indeed you can -- I just tested this myself. How is this >> accomplished? I guess the CRT has a backchannel to talk to itself when >> it creates a process using spawn*? > > CreateProcess() takes a STARTUPINFO argument with undocumented fields > cbReserved2, lpReserved2. They are used to pass an array of fds. So would it be possible to implement the pass_fds parameter of subprocess using spawnl() or the undocumented fields? And is it possible to close all handles except one (implement "pass_handles")? The idea would be to use something subprocess.Popen(cmd, pass_fds=[pipe_rfd], close_fds=True) or subprocess.Popen(cmd, pass_handles=[pipe_rhandle], close_handles=True) instead of subprocess.Popen(cmd, close_fds=False). > In the experimental branch of multiprocessing, child processes > no longer inherit unnecessary handles. Where is this branch? How did you create the channel between the manager and the worker? Victor From solipsis at pitrou.net Wed Jul 24 23:57:33 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 24 Jul 2013 23:57:33 +0200 Subject: [Python-Dev] Inherance of file descriptor and handles on Windows (PEP 446) References: Message-ID: <20130724235733.3505fab0@fsol> On Wed, 24 Jul 2013 10:56:05 -0700 Guido van Rossum wrote: > > But I'm also ready to propose that all this is such a mess that we > *should* change the default fd/handle inheritance to False, *across > platforms*, and damn the torpedoes -- i.e. accept breaking all > existing 3rd party UNIX code for subprocess creation that bypasses the > subprocess module, as well as breaking uses of os.spawn*() on both > platforms that depend on FD inheritance beyond stdin/stdout/stderr). So I suppose you mean "change it to False except for stdin/stdout/stderr"? > With the new, sane default, all we need instead of PEP 446 is a way to > make an FD inheritable after it's been created, which can be a single > os.make_inheritable(fd) call that you must apply to the fileno() of > the stream or socket object you want inherited (or directly to a FD > you created otherwise, e.g. with os.pipe()). On Windows, this should > probably only work with os.spawn*(), since otherwise you need *handle* > inheritance, not *FD* inheritance, and that's a non-portable concept > anyway. I'm not sure how *fd* inheritance could work without *handle* inheritance. (since a fd seems to just be a proxy for a handle) > To reduce the need for 3rd party subprocess creation code, we should > have better daemon creation code in the stdlib -- I wrote some damn > robust code for this purpose in my previous job, but it never saw the > light of day. What do you call "daemon"? An actual Unix-like daemon? Regards Antoine. From guido at python.org Thu Jul 25 00:25:50 2013 From: guido at python.org (Guido van Rossum) Date: Wed, 24 Jul 2013 15:25:50 -0700 Subject: [Python-Dev] Inherance of file descriptor and handles on Windows (PEP 446) In-Reply-To: <20130724235733.3505fab0@fsol> References: <20130724235733.3505fab0@fsol> Message-ID: On Wed, Jul 24, 2013 at 2:57 PM, Antoine Pitrou wrote: > On Wed, 24 Jul 2013 10:56:05 -0700 > Guido van Rossum wrote: >> >> But I'm also ready to propose that all this is such a mess that we >> *should* change the default fd/handle inheritance to False, *across >> platforms*, and damn the torpedoes -- i.e. accept breaking all >> existing 3rd party UNIX code for subprocess creation that bypasses the >> subprocess module, as well as breaking uses of os.spawn*() on both >> platforms that depend on FD inheritance beyond stdin/stdout/stderr). > > So I suppose you mean "change it to False except for > stdin/stdout/stderr"? Hm, that's a tricky detail. I expect that on UNIX those are pre-opened and inherited from the shell, and we should not change their cloexec status; IIRC on Windows inheritance of the stdin/out/err handles is guided by a separate flag to CreateProcess(). >> With the new, sane default, all we need instead of PEP 446 is a way to >> make an FD inheritable after it's been created, which can be a single >> os.make_inheritable(fd) call that you must apply to the fileno() of >> the stream or socket object you want inherited (or directly to a FD >> you created otherwise, e.g. with os.pipe()). On Windows, this should >> probably only work with os.spawn*(), since otherwise you need *handle* >> inheritance, not *FD* inheritance, and that's a non-portable concept >> anyway. > > I'm not sure how *fd* inheritance could work without *handle* > inheritance. > (since a fd seems to just be a proxy for a handle) The MS spawn() implementation takes care of this -- I presume they make the handles inheritable (I think some of the bugs probably refer to this fact) and then use the "internal" interface to pass the mapping from FDs to handles to the subprocess. >> To reduce the need for 3rd party subprocess creation code, we should >> have better daemon creation code in the stdlib -- I wrote some damn >> robust code for this purpose in my previous job, but it never saw the >> light of day. > > What do you call "daemon"? An actual Unix-like daemon? Yeah, a background process with parent PID 1 and not associated with any terminal group. -- --Guido van Rossum (python.org/~guido) From solipsis at pitrou.net Thu Jul 25 00:35:15 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 25 Jul 2013 00:35:15 +0200 Subject: [Python-Dev] Inherance of file descriptor and handles on Windows (PEP 446) In-Reply-To: References: <20130724235733.3505fab0@fsol> Message-ID: <20130725003515.4b79ff17@fsol> On Wed, 24 Jul 2013 15:25:50 -0700 Guido van Rossum wrote: > >> To reduce the need for 3rd party subprocess creation code, we should > >> have better daemon creation code in the stdlib -- I wrote some damn > >> robust code for this purpose in my previous job, but it never saw the > >> light of day. > > > > What do you call "daemon"? An actual Unix-like daemon? > > Yeah, a background process with parent PID 1 and not associated with > any terminal group. But is that relevant to the PEP? A daemon only uses fork(), not exec(). Or have I misunderstood your concern? Regards Antoine. From shibturn at gmail.com Thu Jul 25 00:52:29 2013 From: shibturn at gmail.com (Richard Oudkerk) Date: Wed, 24 Jul 2013 23:52:29 +0100 Subject: [Python-Dev] Inherance of file descriptor and handles on Windows (PEP 446) In-Reply-To: References: <51F01966.7080908@gmail.com> Message-ID: On 24/07/2013 10:50pm, Victor Stinner wrote: > So would it be possible to implement the pass_fds parameter of > subprocess using spawnl() or the undocumented fields? Not in a non-racy way. spawnv() calls CreateProcess() with bInheritHandles=TRUE, so *all* inheritable handles are inherited by the child. The passing of the arrays of fds just makes the fds in the child process match the fds in the parent. If you have a Visual Studio installed then the relevant code is in .../Microsoft Visual Studio 10.0/VC/crt/src/dospawn.c > And is it possible to close all handles except one (implement "pass_handles")? I don't know how to do that. >> In the experimental branch of multiprocessing, child processes >> no longer inherit unnecessary handles. > > Where is this branch? How did you create the channel between the > manager and the worker? http://hg.python.org/sandbox/sbt/ The parent creates a pipe and starts the child process. The pid of the parent, and the handle for the read end of the pipe are passed on the command line. Then the child "steals" the handle from the parent using OpenProcess() and DuplicateHandle() using the DUPLICATE_CLOSE_SOURCE flag. -- Richard From greg.ewing at canterbury.ac.nz Thu Jul 25 01:04:33 2013 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 25 Jul 2013 11:04:33 +1200 Subject: [Python-Dev] Python 3 as a Default in Linux Distros In-Reply-To: References: <731599098.5891653.1374652024043.JavaMail.root@redhat.com> <1903981217.6011829.1374657179869.JavaMail.root@redhat.com> <51EFFF22.1070603@gmail.com> Message-ID: <51F05D81.3020901@canterbury.ac.nz> Chris Angelico wrote: >>>>print "Hello, world!" > SyntaxError: invalid syntax > > Is it safe > to presume that it's more likely a syntax error will come from an > interpreter version mismatch than a code bug? Maybe look at sys.args[0], and if it ends in "python" with no version number, add something to the error message about possibly using the wrong version? -- Greg From eric at trueblade.com Thu Jul 25 01:12:02 2013 From: eric at trueblade.com (Eric V. Smith) Date: Wed, 24 Jul 2013 19:12:02 -0400 Subject: [Python-Dev] Inherance of file descriptor and handles on Windows (PEP 446) In-Reply-To: References: <20130724235733.3505fab0@fsol> Message-ID: <51F05F42.6070200@trueblade.com> On 7/24/2013 6:25 PM, Guido van Rossum wrote: >>> To reduce the need for 3rd party subprocess creation code, we should >>> have better daemon creation code in the stdlib -- I wrote some damn >>> robust code for this purpose in my previous job, but it never saw the >>> light of day. >> >> What do you call "daemon"? An actual Unix-like daemon? > > Yeah, a background process with parent PID 1 and not associated with > any terminal group. There's PEP 3143 and https://pypi.python.org/pypi/python-daemon. I've used it often, with great success. -- Eric. From ncoghlan at gmail.com Thu Jul 25 02:06:11 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 25 Jul 2013 10:06:11 +1000 Subject: [Python-Dev] Python 3 as a Default in Linux Distros In-Reply-To: <20130724192716.GJ11402@unaka.lan> References: <731599098.5891653.1374652024043.JavaMail.root@redhat.com> <1903981217.6011829.1374657179869.JavaMail.root@redhat.com> <20130724152644.GI11402@unaka.lan> <20130724124209.61f8a8ed@anarchist> <20130724192716.GJ11402@unaka.lan> Message-ID: On 25 Jul 2013 05:30, "Toshio Kuratomi" wrote: > > On Wed, Jul 24, 2013 at 12:42:09PM -0400, Barry Warsaw wrote: > > On Jul 25, 2013, at 01:41 AM, Nick Coghlan wrote: > > > > >How's this for an updated wording in the abstract: > > > > > > * for the time being, all distributions should ensure that python > > >refers to the same target as python2 > > > * however, users should be aware that python refers to python3 on at > > >least Arch Linux (that change is > > > what prompted the creation of this PEP), so "python" should be > > >used in the shebang line only for > > > scripts that are source compatible with both Python 2 and 3 > > > > +1 > > > +1 as well. Much clearer. OK, unless someone gets to it before me, I'll update it tonight. Cheers, Nick. > > -Toshio > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com > -------------- next part -------------- An HTML attachment was scrubbed... URL: From cs at zip.com.au Thu Jul 25 01:54:24 2013 From: cs at zip.com.au (Cameron Simpson) Date: Thu, 25 Jul 2013 09:54:24 +1000 Subject: [Python-Dev] Inherance of file descriptor and handles on Windows (PEP 446) In-Reply-To: <20130725003515.4b79ff17@fsol> References: <20130725003515.4b79ff17@fsol> Message-ID: <20130724235424.GA50053@cskk.homeip.net> On 25Jul2013 00:35, Antoine Pitrou wrote: | On Wed, 24 Jul 2013 15:25:50 -0700 | Guido van Rossum wrote: | > >> To reduce the need for 3rd party subprocess creation code, we should | > >> have better daemon creation code in the stdlib -- I wrote some damn | > >> robust code for this purpose in my previous job, but it never saw the | > >> light of day. | > > | > > What do you call "daemon"? An actual Unix-like daemon? | > | > Yeah, a background process with parent PID 1 and not associated with | > any terminal group. | | But is that relevant to the PEP? A daemon only uses fork(), not exec(). Not necessarily. I routinely run other commands in a daemonlike mode, without the command itself having "make myself a daemon" functionality. If I were writing the launcher in Python (and my "setsid" script is Python already), I would be doing all the daemon bits in Python including the fork, then execing the other tool. Cheers, -- If it can't be turned off, it's not a feature. - Karl Heuer From guido at python.org Thu Jul 25 02:41:22 2013 From: guido at python.org (Guido van Rossum) Date: Wed, 24 Jul 2013 17:41:22 -0700 Subject: [Python-Dev] Inherance of file descriptor and handles on Windows (PEP 446) In-Reply-To: <20130725003515.4b79ff17@fsol> References: <20130724235733.3505fab0@fsol> <20130725003515.4b79ff17@fsol> Message-ID: On Wed, Jul 24, 2013 at 3:35 PM, Antoine Pitrou wrote: > On Wed, 24 Jul 2013 15:25:50 -0700 > Guido van Rossum wrote: >> >> To reduce the need for 3rd party subprocess creation code, we should >> >> have better daemon creation code in the stdlib -- I wrote some damn >> >> robust code for this purpose in my previous job, but it never saw the >> >> light of day. >> > >> > What do you call "daemon"? An actual Unix-like daemon? >> >> Yeah, a background process with parent PID 1 and not associated with >> any terminal group. > > But is that relevant to the PEP? A daemon only uses fork(), not exec(). > Or have I misunderstood your concern? Actually it's common for "daemonization" libraries to run an arbitrary executable as a daemon, and then it would be relevant. E.g. zdaemon (https://pypi.python.org/pypi/zdaemon) does this. (Disclosure: I wrote the first version of zdaemon when I worked for Zope Corp over a decade ago. :-) -- --Guido van Rossum (python.org/~guido) From ben+python at benfinney.id.au Thu Jul 25 04:08:18 2013 From: ben+python at benfinney.id.au (Ben Finney) Date: Thu, 25 Jul 2013 12:08:18 +1000 Subject: [Python-Dev] Daemon creation code in the standard library (was: Inherance of file descriptor and handles on Windows (PEP 446)) References: Message-ID: <7wy58vs86l.fsf_-_@benfinney.id.au> Guido van Rossum writes: > To reduce the need for 3rd party subprocess creation code, we should > have better daemon creation code in the stdlib -- I wrote some damn > robust code for this purpose in my previous job, but it never saw the > light of day. Work continues on the PEP 3143-compatible ?python-daemon?, porting it to Python 3 and aiming for inclusion in the standard library. Interested parties are invited to join us on the discussion forums . -- \ ?Politics is not the art of the possible. It consists in | `\ choosing between the disastrous and the unpalatable.? ?John | _o__) Kenneth Galbraith, 1962-03-02 | Ben Finney From ben+python at benfinney.id.au Thu Jul 25 04:18:38 2013 From: ben+python at benfinney.id.au (Ben Finney) Date: Thu, 25 Jul 2013 12:18:38 +1000 Subject: [Python-Dev] Daemon creation code in the standard library References: <7wy58vs86l.fsf_-_@benfinney.id.au> Message-ID: <7w1u6njsap.fsf@benfinney.id.au> Ben Finney writes: > Work continues on the PEP 3143-compatible ?python-daemon?, porting it to > Python 3 and aiming for inclusion in the standard library. At PyPI , and development co-ordinated at Alioth . > Interested parties are invited to join us on the discussion forums The correct link for the ?python-daemon-devel? forum is . For announcements only, we have . -- \ ?This sentence contradicts itself ? no actually it doesn't.? | `\ ?Douglas Hofstadter | _o__) | Ben Finney From bkabrda at redhat.com Thu Jul 25 07:50:07 2013 From: bkabrda at redhat.com (Bohuslav Kabrda) Date: Thu, 25 Jul 2013 01:50:07 -0400 (EDT) Subject: [Python-Dev] Python 3 as a Default in Linux Distros In-Reply-To: References: <731599098.5891653.1374652024043.JavaMail.root@redhat.com> <1903981217.6011829.1374657179869.JavaMail.root@redhat.com> Message-ID: <872938422.6471844.1374731407688.JavaMail.root@redhat.com> ----- Original Message ----- > On Wed, Jul 24, 2013 at 11:12 AM, Bohuslav Kabrda wrote: > > - Should we point /usr/bin/python to Python 3 when we make the move? > > No. > > > - What should user get after using "yum install python"? > > Will a base install include Python 3? If it does, I think yum install > python should mean python3, and hence already be installed. > Yep, base will include Python 3. This is not just about python package, but also all other python extension modules (e.g. python-foo vs. python2-foo vs. python3-foo...). > //Lennart > -- Regards, Bohuslav "Slavek" Kabrda. From bkabrda at redhat.com Thu Jul 25 08:47:08 2013 From: bkabrda at redhat.com (Bohuslav Kabrda) Date: Thu, 25 Jul 2013 02:47:08 -0400 (EDT) Subject: [Python-Dev] Python 3 as a Default in Linux Distros In-Reply-To: <51EFEE63.5080702@v.loewis.de> References: <1903981217.6011829.1374657179869.JavaMail.root@redhat.com> <51EFEE63.5080702@v.loewis.de> Message-ID: <2124741396.6500012.1374734828603.JavaMail.root@redhat.com> ----- Original Message ----- > Am 24.07.13 11:12, schrieb Bohuslav Kabrda: > > - Should we point /usr/bin/python to Python 3 when we make the move? > > This should depend on the answer to this question: > - for how long have you been providing /usr/bin/python2 binaries? > Huh, I don't know exactly, but quick git blame of our python.spec shows that /usr/bin/python2 has been there since July 2009, but probably even longer. > Users "should" have been explicit in declaring scripts as > /usr/bin/python2 for quite some time now, except that this would break > on distros which don't provide a python2 symlink. > > Ideally, you "should" have warned users to be explicit with python2 if > the script wouldn't work on python3. But I'd wave this requirement, as > there is already the upstream PEP. > > So (IMO) if it the last three Fedora releases had been providing python2 > binaries, it should be allowed to switch python to be python3. The exact > number can be debated; it should depend on what releases are still in > active use so that a script would run on multiple releases. > Yep, it seems like 6+ fedora releases and certainly the currently active ones have that. We will however need to make it clear that everyone, especially package maintainers, points to /usr/bin/python{2,3}. > Then you can tell users that possible breakage is easy to fix: just > install python2, and change the shebang line. > > Regards, > Martin > > -- Regards, Bohuslav "Slavek" Kabrda. From fijall at gmail.com Thu Jul 25 11:45:10 2013 From: fijall at gmail.com (Maciej Fijalkowski) Date: Thu, 25 Jul 2013 11:45:10 +0200 Subject: [Python-Dev] Python 3 as a Default in Linux Distros In-Reply-To: <51F00D5B.9060005@gmail.com> References: <731599098.5891653.1374652024043.JavaMail.root@redhat.com> <1903981217.6011829.1374657179869.JavaMail.root@redhat.com> <51EFFF22.1070603@gmail.com> <51F00D5B.9060005@gmail.com> Message-ID: On Wed, Jul 24, 2013 at 7:22 PM, Laurent Gautier wrote: > On 07/24/2013 06:30 PM, Chris Angelico wrote: >> >> On Thu, Jul 25, 2013 at 2:21 AM, Laurent Gautier >> wrote: >>> >>> - errors that are typical of "Python 2 script running with Python >>> 3"-specific are probably limited (e.g., use of unicode, use of xrange, >>> etc...) >>> >> The most common, in interactive scripts at least, is likely to be: >> >>>>> print "Hello, world!" >> >> SyntaxError: invalid syntax >> >> How helpful it's possible to make that one, I don't know. Is it safe >> to presume that it's more likely a syntax error will come from an >> interpreter version mismatch than a code bug? > > > The wrapper in /usr/bin/python: > - could use what is in 2to3. I think that most of the cases are solved > there. > - whenever interactive, could have an intermediate layer between the input > in the console and execution. So you suggest that instead of a clear SyntaxError you should end up with a confusing error (something has no attribute xyz or so) after a while (if say somone tries to load twisted via 2to3). From a.badger at gmail.com Thu Jul 25 12:38:46 2013 From: a.badger at gmail.com (Toshio Kuratomi) Date: Thu, 25 Jul 2013 03:38:46 -0700 Subject: [Python-Dev] Python 3 as a Default in Linux Distros In-Reply-To: References: <731599098.5891653.1374652024043.JavaMail.root@redhat.com> <1903981217.6011829.1374657179869.JavaMail.root@redhat.com> Message-ID: On Jul 24, 2013 6:37 AM, "Brett Cannon" wrote: > The key, though, is adding python2 and getting your code to use that binary specifically so that shifting the default name is more of a convenience than something which might break existing code not ready for the switch. > Applicable to this, does anyone know whether distutils, setuptools, distlib, or any of the other standard build+install tools are doing shebang requiring? Are they doing the right thing wrt python vs python2? -Toshio -------------- next part -------------- An HTML attachment was scrubbed... URL: From lgautier at gmail.com Thu Jul 25 12:41:35 2013 From: lgautier at gmail.com (Laurent Gautier) Date: Thu, 25 Jul 2013 12:41:35 +0200 Subject: [Python-Dev] Python 3 as a Default in Linux Distros In-Reply-To: References: <731599098.5891653.1374652024043.JavaMail.root@redhat.com> <1903981217.6011829.1374657179869.JavaMail.root@redhat.com> <51EFFF22.1070603@gmail.com> <51F00D5B.9060005@gmail.com> Message-ID: <51F100DF.70204@gmail.com> On 07/25/2013 11:45 AM, Maciej Fijalkowski wrote: > On Wed, Jul 24, 2013 at 7:22 PM, Laurent Gautier wrote: >> On 07/24/2013 06:30 PM, Chris Angelico wrote: >>> On Thu, Jul 25, 2013 at 2:21 AM, Laurent Gautier >>> wrote: >>>> - errors that are typical of "Python 2 script running with Python >>>> 3"-specific are probably limited (e.g., use of unicode, use of xrange, >>>> etc...) >>>> >>> The most common, in interactive scripts at least, is likely to be: >>> >>>>>> print "Hello, world!" >>> SyntaxError: invalid syntax >>> >>> How helpful it's possible to make that one, I don't know. Is it safe >>> to presume that it's more likely a syntax error will come from an >>> interpreter version mismatch than a code bug? >> >> The wrapper in /usr/bin/python: >> - could use what is in 2to3. I think that most of the cases are solved >> there. >> - whenever interactive, could have an intermediate layer between the input >> in the console and execution. > So you suggest that instead of a clear SyntaxError you should end up > with a confusing error (something has no attribute xyz or so) after a > while (if say somone tries to load twisted via 2to3). In a sense, yes. I think that the a priori (expectations) a user has plays a role in whether something is confusing or not. Consider the two following situations: - a programmer is working on code, and is testing it. If the program fails, this is because he/she made a mistake and the error message should indicate where the error originates from - a user is running a python script (he expects to be working), and is using the default /usr/bin/python (formerly Python 2, now Python 3). If the program fails because of obvious Python 2-only idioms, reporting this rather that the SyntaxError is much less confusing. Having that said, the comments are pointing out that this is may be not a completely good idea (most notably because of how slow 2to3 is). A wrapper producing an unconditional warning about the default python being changed to Python 3, and differences between Python 2 and Python 3 the possible source of errors, (as I think it was suggested) is going to be a better idea. From regebro at gmail.com Thu Jul 25 13:19:29 2013 From: regebro at gmail.com (Lennart Regebro) Date: Thu, 25 Jul 2013 13:19:29 +0200 Subject: [Python-Dev] Python 3 as a Default in Linux Distros In-Reply-To: <51F100DF.70204@gmail.com> References: <731599098.5891653.1374652024043.JavaMail.root@redhat.com> <1903981217.6011829.1374657179869.JavaMail.root@redhat.com> <51EFFF22.1070603@gmail.com> <51F00D5B.9060005@gmail.com> <51F100DF.70204@gmail.com> Message-ID: On Thu, Jul 25, 2013 at 12:41 PM, Laurent Gautier wrote: > - a user is running a python script (he expects to be working), and is using > the default /usr/bin/python (formerly Python 2, now Python 3). If the > program fails because of obvious Python 2-only idioms, reporting this rather > that the SyntaxError is much less confusing. 2to3 will not report this. It will get rid of the SyntaxError (which at least is fairly clear) and give you a completely different and even more obscure error. You have replaced a somewhat unclear error with a very unclear error. If we want to report a problem, then /usr/bin/python should just report that you should use /usr/bin/python2 or /usr/bin/python3. That's clear. //Lennart From lgautier at gmail.com Thu Jul 25 13:53:33 2013 From: lgautier at gmail.com (Laurent Gautier) Date: Thu, 25 Jul 2013 13:53:33 +0200 Subject: [Python-Dev] Python 3 as a Default in Linux Distros In-Reply-To: References: <731599098.5891653.1374652024043.JavaMail.root@redhat.com> <1903981217.6011829.1374657179869.JavaMail.root@redhat.com> <51EFFF22.1070603@gmail.com> <51F00D5B.9060005@gmail.com> <51F100DF.70204@gmail.com> Message-ID: <51F111BD.5030008@gmail.com> On 07/25/2013 01:19 PM, Lennart Regebro wrote: > On Thu, Jul 25, 2013 at 12:41 PM, Laurent Gautier wrote: >> - a user is running a python script (he expects to be working), and is using >> the default /usr/bin/python (formerly Python 2, now Python 3). If the >> program fails because of obvious Python 2-only idioms, reporting this rather >> that the SyntaxError is much less confusing. > 2to3 will not report this. My meaning was the use of 2to3's machinery (and fire a message if a translation occurs) not 2to3 itself, obviously. > It will get rid of the SyntaxError (which > at least is fairly clear) and give you a completely different and even > more obscure error. You have replaced a somewhat unclear error with a > very unclear error. > > If we want to report a problem, then /usr/bin/python should just > report that you should use /usr/bin/python2 or /usr/bin/python3. > > That's clear. > > //Lennart From ncoghlan at gmail.com Thu Jul 25 14:25:26 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 25 Jul 2013 22:25:26 +1000 Subject: [Python-Dev] Python 3 as a Default in Linux Distros In-Reply-To: References: <731599098.5891653.1374652024043.JavaMail.root@redhat.com> <1903981217.6011829.1374657179869.JavaMail.root@redhat.com> Message-ID: On 25 July 2013 20:38, Toshio Kuratomi wrote: > > On Jul 24, 2013 6:37 AM, "Brett Cannon" wrote: >> The key, though, is adding python2 and getting your code to use that >> binary specifically so that shifting the default name is more of a >> convenience than something which might break existing code not ready for the >> switch. >> > Applicable to this, does anyone know whether distutils, setuptools, distlib, > or any of the other standard build+install tools are doing shebang > requiring? Are they doing the right thing wrt python vs python2? It occurs to me they're almost certainly using "sys.executable" to set the shebang line, so probably not :( distutils-sig could probably offer a better answer though, most of the packaging folks don't hang out here. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Thu Jul 25 14:35:28 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 25 Jul 2013 22:35:28 +1000 Subject: [Python-Dev] Python 3 as a Default in Linux Distros In-Reply-To: References: <731599098.5891653.1374652024043.JavaMail.root@redhat.com> <1903981217.6011829.1374657179869.JavaMail.root@redhat.com> <20130724152644.GI11402@unaka.lan> <20130724124209.61f8a8ed@anarchist> <20130724192716.GJ11402@unaka.lan> Message-ID: On 25 July 2013 10:06, Nick Coghlan wrote: > > On 25 Jul 2013 05:30, "Toshio Kuratomi" wrote: >> >> On Wed, Jul 24, 2013 at 12:42:09PM -0400, Barry Warsaw wrote: >> > On Jul 25, 2013, at 01:41 AM, Nick Coghlan wrote: >> > >> > >How's this for an updated wording in the abstract: >> > > >> > > * for the time being, all distributions should ensure that python >> > >refers to the same target as python2 >> > > * however, users should be aware that python refers to python3 on at >> > >least Arch Linux (that change is >> > > what prompted the creation of this PEP), so "python" should be >> > >used in the shebang line only for >> > > scripts that are source compatible with both Python 2 and 3 >> > >> > +1 >> > >> +1 as well. Much clearer. > > OK, unless someone gets to it before me, I'll update it tonight. Update is here: http://hg.python.org/peps/rev/4b379a690ae2 I added one final bullet point to the abstract with advice for users: * in preparation for an eventual change in the default version of Python, Python 2 only scripts should either be updated to be source compatible with Python 3 or else to use python2 in the shebang line. I also rewrote the first two bullet points in the Migration Notes section, as I realised they contributed heavily to the notion that distros like Fedora should switch aggressively, while only the conservative distros like RHEL and Debian stable should preserve the old behaviour (which wasn't the intent at all). Finally, I added a couple of other notes: * one pointing out the kind of time frame we're thinking about (i.e. years) by mentioning the 2.7 switch to security fix only mode * one noting that the Python launcher for Windows also default to Python 2 if both 2.x and 3.x are installed, so we're also aiming for cross platform consistency here Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From eliben at gmail.com Thu Jul 25 16:29:29 2013 From: eliben at gmail.com (Eli Bendersky) Date: Thu, 25 Jul 2013 07:29:29 -0700 Subject: [Python-Dev] Official github mirror for CPython? Message-ID: Hi all, I've been looking for a Github mirror for Python, and found two: * https://github.com/python-git/python has a lot of forks/watches/starts but seems to be very out of date (last updated 4 years ago) * https://github.com/python-mirror/python doesn't appear to be very popular but is updated daily Are some of you the owners of these repositories? Should we consolidate to a single "semi-official" mirror? Eli -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Thu Jul 25 16:33:07 2013 From: rosuav at gmail.com (Chris Angelico) Date: Fri, 26 Jul 2013 00:33:07 +1000 Subject: [Python-Dev] Official github mirror for CPython? In-Reply-To: References: Message-ID: On Fri, Jul 26, 2013 at 12:29 AM, Eli Bendersky wrote: > Hi all, > > I've been looking for a Github mirror for Python, and found two: > > * https://github.com/python-git/python has a lot of forks/watches/starts but > seems to be very out of date (last updated 4 years ago) > * https://github.com/python-mirror/python doesn't appear to be very popular > but is updated daily Also https://github.com/akheron/cpython seems to be the same again. ChrisA From christian at python.org Thu Jul 25 16:37:02 2013 From: christian at python.org (Christian Heimes) Date: Thu, 25 Jul 2013 16:37:02 +0200 Subject: [Python-Dev] Official github mirror for CPython? In-Reply-To: References: Message-ID: <51F1380E.9070608@python.org> Am 25.07.2013 16:29, schrieb Eli Bendersky: > Hi all, > > I've been looking for a Github mirror for Python, and found two: > > * https://github.com/python-git/python has a lot of forks/watches/starts > but seems to be very out of date (last updated 4 years ago) > * https://github.com/python-mirror/python doesn't appear to be very > popular but is updated daily > > Are some of you the owners of these repositories? Should we consolidate > to a single "semi-official" mirror? +1 Does the PSF have an official account on github? We have one on bitbucket... Christian From brian at python.org Thu Jul 25 16:48:40 2013 From: brian at python.org (Brian Curtin) Date: Thu, 25 Jul 2013 09:48:40 -0500 Subject: [Python-Dev] Official github mirror for CPython? In-Reply-To: <51F1380E.9070608@python.org> References: <51F1380E.9070608@python.org> Message-ID: On Thu, Jul 25, 2013 at 9:37 AM, Christian Heimes wrote: > Am 25.07.2013 16:29, schrieb Eli Bendersky: >> Hi all, >> >> I've been looking for a Github mirror for Python, and found two: >> >> * https://github.com/python-git/python has a lot of forks/watches/starts >> but seems to be very out of date (last updated 4 years ago) >> * https://github.com/python-mirror/python doesn't appear to be very >> popular but is updated daily >> >> Are some of you the owners of these repositories? Should we consolidate >> to a single "semi-official" mirror? > > +1 > > Does the PSF have an official account on github? We have one on bitbucket... I don't remember who runs this, and I thought I was in it (maybe just on BB), but: https://github.com/python From christian at python.org Thu Jul 25 16:58:46 2013 From: christian at python.org (Christian Heimes) Date: Thu, 25 Jul 2013 16:58:46 +0200 Subject: [Python-Dev] Official github mirror for CPython? In-Reply-To: References: <51F1380E.9070608@python.org> Message-ID: <51F13D26.9090603@python.org> Am 25.07.2013 16:48, schrieb Brian Curtin: > On Thu, Jul 25, 2013 at 9:37 AM, Christian Heimes wrote: >> Am 25.07.2013 16:29, schrieb Eli Bendersky: >>> Hi all, >>> >>> I've been looking for a Github mirror for Python, and found two: >>> >>> * https://github.com/python-git/python has a lot of forks/watches/starts >>> but seems to be very out of date (last updated 4 years ago) >>> * https://github.com/python-mirror/python doesn't appear to be very >>> popular but is updated daily >>> >>> Are some of you the owners of these repositories? Should we consolidate >>> to a single "semi-official" mirror? >> >> +1 >> >> Does the PSF have an official account on github? We have one on bitbucket... > > I don't remember who runs this, and I thought I was in it (maybe just > on BB), but: https://github.com/python Jesse might know more about it. He is an admin of https://bitbucket.org/PSF/, too. Christian From eliben at gmail.com Thu Jul 25 17:18:34 2013 From: eliben at gmail.com (Eli Bendersky) Date: Thu, 25 Jul 2013 08:18:34 -0700 Subject: [Python-Dev] Official github mirror for CPython? In-Reply-To: References: <51F1380E.9070608@python.org> Message-ID: On Thu, Jul 25, 2013 at 7:48 AM, Brian Curtin wrote: > On Thu, Jul 25, 2013 at 9:37 AM, Christian Heimes > wrote: > > Am 25.07.2013 16:29, schrieb Eli Bendersky: > >> Hi all, > >> > >> I've been looking for a Github mirror for Python, and found two: > >> > >> * https://github.com/python-git/python has a lot of > forks/watches/starts > >> but seems to be very out of date (last updated 4 years ago) > >> * https://github.com/python-mirror/python doesn't appear to be very > >> popular but is updated daily > >> > >> Are some of you the owners of these repositories? Should we consolidate > >> to a single "semi-official" mirror? > > > > +1 > > > > Does the PSF have an official account on github? We have one on > bitbucket... > > I don't remember who runs this, and I thought I was in it (maybe just > on BB), but: https://github.com/python > This appears to be a good place, but it doesn't have an actual mirror (yet?) ;-) -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Thu Jul 25 17:26:58 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 25 Jul 2013 17:26:58 +0200 Subject: [Python-Dev] Daemon creation code in the standard library (was: Inherance of file descriptor and handles on Windows (PEP 446)) References: <7wy58vs86l.fsf_-_@benfinney.id.au> Message-ID: <20130725172658.60fe7982@pitrou.net> Le Thu, 25 Jul 2013 12:08:18 +1000, Ben Finney a ?crit : > Guido van Rossum writes: > > > To reduce the need for 3rd party subprocess creation code, we should > > have better daemon creation code in the stdlib -- I wrote some damn > > robust code for this purpose in my previous job, but it never saw > > the light of day. > > Work continues on the PEP 3143-compatible ?python-daemon?, porting it > to Python 3 and aiming for inclusion in the standard library. The PEP hasn't been formally accepted yet, however. Skimming back through the archives, one sticking point was the default value of the "umask" parameter. Setting the umask to 0 if the user didn't ask for something else is a disaster, security-wise. Another problem I've had when using it is that the `pidfile` combines the notion of pidfile and process lock in one unique attribute. This is quite inflexible when you're using something else than Skip Montanaro's "lockfile" library. I'm using a separate lock based on locket.py: https://github.com/mwilliamson/locket.py because it is based on POSIX advisory locks, and therefore doesn't suffer from the "stale pid file" issues you get when a process (or the whole system) crashes. Therefore I'd be -1 on the PEP until those issues are alleviated. Regards Antoine. From brett at python.org Thu Jul 25 17:30:45 2013 From: brett at python.org (Brett Cannon) Date: Thu, 25 Jul 2013 11:30:45 -0400 Subject: [Python-Dev] Official github mirror for CPython? In-Reply-To: References: <51F1380E.9070608@python.org> Message-ID: On Thu, Jul 25, 2013 at 11:18 AM, Eli Bendersky wrote: > > > > On Thu, Jul 25, 2013 at 7:48 AM, Brian Curtin wrote: > >> On Thu, Jul 25, 2013 at 9:37 AM, Christian Heimes >> wrote: >> > Am 25.07.2013 16:29, schrieb Eli Bendersky: >> >> Hi all, >> >> >> >> I've been looking for a Github mirror for Python, and found two: >> >> >> >> * https://github.com/python-git/python has a lot of >> forks/watches/starts >> >> but seems to be very out of date (last updated 4 years ago) >> >> * https://github.com/python-mirror/python doesn't appear to be very >> >> popular but is updated daily >> >> >> >> Are some of you the owners of these repositories? Should we consolidate >> >> to a single "semi-official" mirror? >> > >> > +1 >> > >> > Does the PSF have an official account on github? We have one on >> bitbucket... >> >> I don't remember who runs this, and I thought I was in it (maybe just >> on BB), but: https://github.com/python >> > > This appears to be a good place, but it doesn't have an actual mirror > (yet?) ;-) > Based on the list of people who are members of github.com/python it's as official as it's going to get (depends on who of that group owns it). But assuming whomever owns it is okay with hosting a mirror, what exactly is going to be required to have it updated regularly? Someone is going to have to write the scripts to pull from the hg repo and then push up to github else it's just going to end up out-of-date on top of maintaining whatever setup is devised. -------------- next part -------------- An HTML attachment was scrubbed... URL: From regebro at gmail.com Thu Jul 25 18:03:42 2013 From: regebro at gmail.com (Lennart Regebro) Date: Thu, 25 Jul 2013 18:03:42 +0200 Subject: [Python-Dev] Python 3 as a Default in Linux Distros In-Reply-To: <51F111BD.5030008@gmail.com> References: <731599098.5891653.1374652024043.JavaMail.root@redhat.com> <1903981217.6011829.1374657179869.JavaMail.root@redhat.com> <51EFFF22.1070603@gmail.com> <51F00D5B.9060005@gmail.com> <51F100DF.70204@gmail.com> <51F111BD.5030008@gmail.com> Message-ID: On Thu, Jul 25, 2013 at 1:53 PM, Laurent Gautier wrote: > My meaning was the use of 2to3's machinery (and fire a message if a > translation occurs) not 2to3 itself, obviously. I don't understand what you mean is the difference. //Lennart From a.badger at gmail.com Thu Jul 25 18:08:14 2013 From: a.badger at gmail.com (Toshio Kuratomi) Date: Thu, 25 Jul 2013 09:08:14 -0700 Subject: [Python-Dev] Python 3 as a Default in Linux Distros In-Reply-To: References: <731599098.5891653.1374652024043.JavaMail.root@redhat.com> <1903981217.6011829.1374657179869.JavaMail.root@redhat.com> Message-ID: <20130725160814.GM11402@unaka.lan> On Thu, Jul 25, 2013 at 10:25:26PM +1000, Nick Coghlan wrote: > On 25 July 2013 20:38, Toshio Kuratomi wrote: > > > > On Jul 24, 2013 6:37 AM, "Brett Cannon" wrote: > >> The key, though, is adding python2 and getting your code to use that > >> binary specifically so that shifting the default name is more of a > >> convenience than something which might break existing code not ready for the > >> switch. > >> > > Applicable to this, does anyone know whether distutils, setuptools, distlib, > > or any of the other standard build+install tools are doing shebang > > requiring? Are they doing the right thing wrt python vs python2? > > It occurs to me they're almost certainly using "sys.executable" to set > the shebang line, so probably not :( > > distutils-sig could probably offer a better answer though, most of the > packaging folks don't hang out here. > Thanks! For other Linux distributors following along, here's my message to distutils-sig: http://mail.python.org/pipermail/distutils-sig/2013-July/022001.html -Toshio -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 198 bytes Desc: not available URL: From fijall at gmail.com Thu Jul 25 18:59:17 2013 From: fijall at gmail.com (Maciej Fijalkowski) Date: Thu, 25 Jul 2013 18:59:17 +0200 Subject: [Python-Dev] Official github mirror for CPython? In-Reply-To: References: <51F1380E.9070608@python.org> Message-ID: On Thu, Jul 25, 2013 at 5:30 PM, Brett Cannon wrote: > > > > On Thu, Jul 25, 2013 at 11:18 AM, Eli Bendersky wrote: >> >> >> >> >> On Thu, Jul 25, 2013 at 7:48 AM, Brian Curtin wrote: >>> >>> On Thu, Jul 25, 2013 at 9:37 AM, Christian Heimes >>> wrote: >>> > Am 25.07.2013 16:29, schrieb Eli Bendersky: >>> >> Hi all, >>> >> >>> >> I've been looking for a Github mirror for Python, and found two: >>> >> >>> >> * https://github.com/python-git/python has a lot of >>> >> forks/watches/starts >>> >> but seems to be very out of date (last updated 4 years ago) >>> >> * https://github.com/python-mirror/python doesn't appear to be very >>> >> popular but is updated daily >>> >> >>> >> Are some of you the owners of these repositories? Should we >>> >> consolidate >>> >> to a single "semi-official" mirror? >>> > >>> > +1 >>> > >>> > Does the PSF have an official account on github? We have one on >>> > bitbucket... >>> >>> I don't remember who runs this, and I thought I was in it (maybe just >>> on BB), but: https://github.com/python >> >> >> This appears to be a good place, but it doesn't have an actual mirror >> (yet?) ;-) > > > Based on the list of people who are members of github.com/python it's as > official as it's going to get (depends on who of that group owns it). > > But assuming whomever owns it is okay with hosting a mirror, what exactly is > going to be required to have it updated regularly? Someone is going to have > to write the scripts to pull from the hg repo and then push up to github > else it's just going to end up out-of-date on top of maintaining whatever > setup is devised. > It's slightly more work than this. You need to keep all the features relying on hg work, like the revision version it was compiled from etc. From christian at python.org Thu Jul 25 20:48:02 2013 From: christian at python.org (Christian Heimes) Date: Thu, 25 Jul 2013 20:48:02 +0200 Subject: [Python-Dev] Coverity Scan Message-ID: Hello, this is an update on my work and the current status of Coverity Scan. Maybe you have noticed a checkins made be me that end with the line "CID #". These are checkins that fix an issue that was discovered by the static code analyzer Coverity. Coverity is a commercial product but it's a free service for some Open Source projects. Python has been analyzed by Coverity since about 2007. Guido, Neal, Brett, Stefan and some other developers have used Coverity before I took over. I fixed a couple of issues before 3.3 reached the RC phase and more bugs in the last couple of months. Coverity is really great and its web GUI is fun to use, too. I was able to identify and fix resource leaks, NULL pointer issues, buffer overflows and missing checks all over the place. Because it's a static analyzer that follows data-flows and control-flows the tool can detect issues in error paths that are hardly visited at all. I have started to document Coverity here: http://docs.python.org/devguide/coverity.html Interview --------- A week ago I was contacted by Coverity. They have started a series of articles and press releases about Open Source projects that use their free service Coverity Scan, see http://www.coverity.com/company/press-releases/read/coverity-introduces-monthly-spotlight-series-for-coverity-scan-open-source-projects Two days ago I had a lovely phone interview about my involvement in the Python project and our development style. They are going to release a nice article in a couple of weeks. In the mean time we have time to fix the remaining couple issues. We *might* be able to reach the highest coverity integrity level! I have dealt with all major issues so we just have to fix a couple of issues. Current stats ------------- Lines of Code: 396,179 Defect Density: 0.05 Total defects: 1,054 Outstanding: 21 (Coverity Connect shows less) Dismissed: 222 Fixed: 811 http://i.imgur.com/NoELjcj.jpg http://i.imgur.com/eJSzTUX.jpg open issues ----------- http://bugs.python.org/issue17899 http://bugs.python.org/issue18556 http://bugs.python.org/issue18555 http://bugs.python.org/issue18552 http://bugs.python.org/issue18551 http://bugs.python.org/issue18550 http://bugs.python.org/issue18528 Christian From lgautier at gmail.com Thu Jul 25 21:16:56 2013 From: lgautier at gmail.com (Laurent Gautier) Date: Thu, 25 Jul 2013 21:16:56 +0200 Subject: [Python-Dev] Python 3 as a Default in Linux Distros In-Reply-To: References: <731599098.5891653.1374652024043.JavaMail.root@redhat.com> <1903981217.6011829.1374657179869.JavaMail.root@redhat.com> <51EFFF22.1070603@gmail.com> <51F00D5B.9060005@gmail.com> <51F100DF.70204@gmail.com> <51F111BD.5030008@gmail.com> Message-ID: <51F179A8.3030505@gmail.com> On 07/25/2013 06:03 PM, Lennart Regebro wrote: > On Thu, Jul 25, 2013 at 1:53 PM, Laurent Gautier wrote: >> My meaning was the use of 2to3's machinery (and fire a message if a >> translation occurs) not 2to3 itself, obviously. > I don't understand what you mean is the difference. > > //Lennart bullet-point version: - 2to3 is a code translator - to function, it needs to identify Python 2-only idioms so they can be translated - the idea _was_ (*) to re-use that engine (identification-only, not translation) so the presence of Python 2-only idioms would cause a "Python 3 is now the default for "python", buddy. This script was seemingly written for Python 2, so you should run it with the command python2" sort of error message. The speed issue could be smaller than an full run of 2to3 since the error would be triggered at the first snippet of Python 2-only code encountered, but *. (*: I withdrew the suggestion few emails back) L. From tjreedy at udel.edu Fri Jul 26 00:00:55 2013 From: tjreedy at udel.edu (Terry Reedy) Date: Thu, 25 Jul 2013 18:00:55 -0400 Subject: [Python-Dev] Coverity Scan In-Reply-To: References: Message-ID: On 7/25/2013 2:48 PM, Christian Heimes wrote: > Hello, > > this is an update on my work and the current status of Coverity Scan. Great work. > > Maybe you have noticed a checkins made be me that end with the line "CID > #". These are checkins that fix an issue that was discovered by the > static code analyzer Coverity. Coverity is a commercial product but it's > a free service for some Open Source projects. Python has been analyzed > by Coverity since about 2007. Guido, Neal, Brett, Stefan and some other > developers have used Coverity before I took over. I fixed a couple of > issues before 3.3 reached the RC phase and more bugs in the last couple > of months. The benefit for us is not just improving Python having external verification of its excellence in relation both to other open-source projects and commercial software. > Coverity is really great and its web GUI is fun to use, too. I was able > to identify and fix resource leaks, NULL pointer issues, buffer > overflows and missing checks all over the place. Because it's a static > analyzer that follows data-flows and control-flows the tool can detect > issues in error paths that are hardly visited at all. I have started to > document Coverity here: > > http://docs.python.org/devguide/coverity.html > > > Interview > --------- > > A week ago I was contacted by Coverity. They have started a series of > articles and press releases about Open Source projects that use their > free service Coverity Scan, see > > http://www.coverity.com/company/press-releases/read/coverity-introduces-monthly-spotlight-series-for-coverity-scan-open-source-projects The intention is to promote the best of open source to industry. > Two days ago I had a lovely phone interview about my involvement in the > Python project and our development style. They are going to release a > nice article in a couple of weeks. In the mean time we have time to fix > the remaining couple issues. We *might* be able to reach the highest > coverity integrity level! I have dealt with all major issues so we just > have to fix a couple of issues. > Current stats > ------------- > > Lines of Code: 396,179 C only? or does Python code now count as 'source code'? > Defect Density: 0.05 = defects per thousand lines = 20/400 Anything under 1 is good. The release above reports Samba now at .6. http://www.pcworld.com/article/2038244/linux-code-is-the-benchmark-of-quality-study-concludes.html reports Linux 3.8 as having the same for 7.6 million lines. > Total defects: 1,054 > Outstanding: 21 (Coverity Connect shows less) > Dismissed: 222 This implies that they accept our designation of some things as False Positives or Intentional. Does Coverity do any review of such designations, so a project cannot cheat? > Fixed: 811 > > http://i.imgur.com/NoELjcj.jpg > http://i.imgur.com/eJSzTUX.jpg > > > open issues > ----------- > > http://bugs.python.org/issue17899 > http://bugs.python.org/issue18556 > http://bugs.python.org/issue18555 > http://bugs.python.org/issue18552 > http://bugs.python.org/issue18551 > http://bugs.python.org/issue18550 > http://bugs.python.org/issue18528 -- Terry Jan Reedy From tjreedy at udel.edu Fri Jul 26 00:32:08 2013 From: tjreedy at udel.edu (Terry Reedy) Date: Thu, 25 Jul 2013 18:32:08 -0400 Subject: [Python-Dev] Coverity Scan In-Reply-To: References: Message-ID: On 7/25/2013 6:00 PM, Terry Reedy wrote: >> Defect Density: 0.05 > > = defects per thousand lines = 20/400 > > Anything under 1 is good. The release above reports Samba now at .6. > http://www.pcworld.com/article/2038244/linux-code-is-the-benchmark-of-quality-study-concludes.html > > reports Linux 3.8 as having the same for 7.6 million lines. > >> Total defects: 1,054 >> Outstanding: 21 (Coverity Connect shows less) >> Dismissed: 222 > > This implies that they accept our designation of some things as False > Positives or Intentional. Does Coverity do any review of such > designations, so a project cannot cheat? I found the answer here https://docs.google.com/file/d/0B5wQCOK_TiRiMWVqQ0xPaDEzbkU/edit Coverity Integrity Level 1 is 1 (defect/1000 lines) Level 2 is .1 (we have passed that) Level 3 is .01 + no major defects + <20% (all all defects?) false positives as that is their normal rate.# A higher false positive rates requires auditing by Coverity. They claim "A higher false positive rate indicates misconfiguration, usage of unusual idioms, or incorrect diagnosis of a large number of defects." They else add "or a flaw in our analysis." # Since false positives should stay constant as true positives are reduced toward 0, false / all should tend toward 1 (100%) if I understand the ratio correctly. > >> Fixed: 811 >> >> http://i.imgur.com/NoELjcj.jpg >> http://i.imgur.com/eJSzTUX.jpg -- Terry Jan Reedy From solipsis at pitrou.net Fri Jul 26 00:50:13 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 26 Jul 2013 00:50:13 +0200 Subject: [Python-Dev] Coverity Scan References: Message-ID: <20130726005013.022300ff@fsol> On Thu, 25 Jul 2013 18:00:55 -0400 Terry Reedy wrote: > On 7/25/2013 2:48 PM, Christian Heimes wrote: > > Hello, > > > > this is an update on my work and the current status of Coverity Scan. > > Great work. > > > > > Maybe you have noticed a checkins made be me that end with the line "CID > > #". These are checkins that fix an issue that was discovered by the > > static code analyzer Coverity. Coverity is a commercial product but it's > > a free service for some Open Source projects. Python has been analyzed > > by Coverity since about 2007. Guido, Neal, Brett, Stefan and some other > > developers have used Coverity before I took over. I fixed a couple of > > issues before 3.3 reached the RC phase and more bugs in the last couple > > of months. > > The benefit for us is not just improving Python having external > verification of its excellence in relation both to other open-source > projects and commercial software. "Excellence"? The term is too weak, I would say "perfection" at least, but perhaps we should go as far as "divinity". Regards Antoine. From christian at python.org Fri Jul 26 00:56:30 2013 From: christian at python.org (Christian Heimes) Date: Fri, 26 Jul 2013 00:56:30 +0200 Subject: [Python-Dev] Coverity Scan In-Reply-To: References: Message-ID: <51F1AD1E.70309@python.org> Am 26.07.2013 00:32, schrieb Terry Reedy: > I found the answer here > https://docs.google.com/file/d/0B5wQCOK_TiRiMWVqQ0xPaDEzbkU/edit > Coverity Integrity Level 1 is 1 (defect/1000 lines) > Level 2 is .1 (we have passed that) > Level 3 is .01 + no major defects + <20% (all all defects?) false > positives as that is their normal rate.# > > A higher false positive rates requires auditing by Coverity. They claim > "A higher false positive rate indicates misconfiguration, usage of > unusual idioms, or incorrect diagnosis of a large number of defects." > They else add "or a flaw in our analysis." > > # Since false positives should stay constant as true positives are > reduced toward 0, false / all should tend toward 1 (100%) if I > understand the ratio correctly. About 40% of the dismissed cases are cause by a handful of issues. I have documented these issues as "known limitations" http://docs.python.org/devguide/coverity.html#known-limitations . For example about 35 false positives are related to PyLong_FromLong() and our small integer optimization. A correct modeling file would eliminate the false positive defects. My attempts don't work as hoped and I don't have access to all professional coverity tools to debug my trials. Nearly 20 false positives are caused by Py_BuildValue("N"). I'm still astonished that Coverity understands Python's reference counting most of the time. :) Did I mention that we have almost reached Level 3? All major defects have been dealt with (one of them locally on the test machine until Larry pushes his patch soonish), 4 of 7 minor issues must be closed and our dismissed rate is just little over 20% (222 out of 1054 = 21%). Christian From christian at python.org Fri Jul 26 01:02:12 2013 From: christian at python.org (Christian Heimes) Date: Fri, 26 Jul 2013 01:02:12 +0200 Subject: [Python-Dev] Coverity Scan In-Reply-To: <20130726005013.022300ff@fsol> References: <20130726005013.022300ff@fsol> Message-ID: <51F1AE74.8050401@python.org> Am 26.07.2013 00:50, schrieb Antoine Pitrou: > "Excellence"? The term is too weak, I would say "perfection" at least, > but perhaps we should go as far as "divinity". Don't forget that Python can offer lots of places to keep your bike clean and dry ... *scnr* From christian at python.org Fri Jul 26 01:20:01 2013 From: christian at python.org (Christian Heimes) Date: Fri, 26 Jul 2013 01:20:01 +0200 Subject: [Python-Dev] Coverity Scan In-Reply-To: References: Message-ID: Am 26.07.2013 00:00, schrieb Terry Reedy: >> http://www.coverity.com/company/press-releases/read/coverity-introduces-monthly-spotlight-series-for-coverity-scan-open-source-projects >> > > The intention is to promote the best of open source to industry. I think it's also a marketing tool. They like to sell their product. I don't have a problem with that. After all Coverity provides a useful service for free that supplements our own debugging tools. >> Lines of Code: 396,179 > > C only? or does Python code now count as 'source code'? It's just C code and headers. Coverity doesn't analyze Python code. According to cloc Python has 296707 + 78126 == 374833 lines of code in C and header files. I'm not sure why Coverity detects more. > >> Defect Density: 0.05 > > = defects per thousand lines = 20/400 > > Anything under 1 is good. The release above reports Samba now at .6. > http://www.pcworld.com/article/2038244/linux-code-is-the-benchmark-of-quality-study-concludes.html > > reports Linux 3.8 as having the same for 7.6 million lines. These are amazing numbers. Python is much smaller. > >> Total defects: 1,054 >> Outstanding: 21 (Coverity Connect shows less) >> Dismissed: 222 > > This implies that they accept our designation of some things as False > Positives or Intentional. Does Coverity do any review of such > designations, so a project cannot cheat? What's the point of cheating? :) I could dismiss any remaining defect as intentionally or false positive but that would only harm ourselves. As you already pointed out Coverity reserves the right to inspect dismissed bugs for their highest ranking. I'm in the process of looking through all dismissed defects. Some of them are relics of deleted files and removed code. Some other may go away with proper modeling. Christian From tjreedy at udel.edu Fri Jul 26 01:24:34 2013 From: tjreedy at udel.edu (Terry Reedy) Date: Thu, 25 Jul 2013 19:24:34 -0400 Subject: [Python-Dev] Coverity Scan In-Reply-To: <51F1AD1E.70309@python.org> References: <51F1AD1E.70309@python.org> Message-ID: On 7/25/2013 6:56 PM, Christian Heimes wrote: > Am 26.07.2013 00:32, schrieb Terry Reedy: >> # Since false positives should stay constant as true positives are >> reduced toward 0, false / all should tend toward 1 (100%) if I >> understand the ratio correctly. Which I did not ;-). > About 40% of the dismissed cases are cause by a handful of issues. I > have documented these issues as "known limitations" > http://docs.python.org/devguide/coverity.html#known-limitations . > > For example about 35 false positives are related to PyLong_FromLong() > and our small integer optimization. A correct modeling file would > eliminate the false positive defects. My attempts don't work as hoped > and I don't have access to all professional coverity tools to debug my > trials. Perhaps Coverity will help when doing an audit. > Nearly 20 false positives are caused by Py_BuildValue("N"). I'm still > astonished that Coverity understands Python's reference counting most of > the time. :) > > Did I mention that we have almost reached Level 3? All major defects It is hard to measure the benefit of preventitive medicine, but I imagine that we should see fewer mysterious crashes and heisenbugs than we would have. In any case, Level 3 certification should help people promoting the use of Python in organizational settings, whether as employees or consultants. > have been dealt with (one of them locally on the test machine until > Larry pushes his patch soonish), 4 of 7 minor issues must be closed and .1 * 390 allows 3 defects (or 4 if they round up) -- astonishingly good! > our dismissed rate is just little over 20% (222 out of 1054 = 21%). So merely verifying the 35 PyLong_FromLong dismissals will put us under. Thanks for clarifying the proper denominator -- all defects ever found. It seems obvious in retrospect, but I was focused on current stats, not the history. -- Terry Jan Reedy From ncoghlan at gmail.com Fri Jul 26 01:31:50 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 26 Jul 2013 09:31:50 +1000 Subject: [Python-Dev] Official github mirror for CPython? In-Reply-To: References: <51F1380E.9070608@python.org> Message-ID: On 26 Jul 2013 03:02, "Maciej Fijalkowski" wrote: > > On Thu, Jul 25, 2013 at 5:30 PM, Brett Cannon wrote: > > > > > > > > On Thu, Jul 25, 2013 at 11:18 AM, Eli Bendersky wrote: > >> > >> > >> > >> > >> On Thu, Jul 25, 2013 at 7:48 AM, Brian Curtin wrote: > >>> > >>> On Thu, Jul 25, 2013 at 9:37 AM, Christian Heimes < christian at python.org> > >>> wrote: > >>> > Am 25.07.2013 16:29, schrieb Eli Bendersky: > >>> >> Hi all, > >>> >> > >>> >> I've been looking for a Github mirror for Python, and found two: > >>> >> > >>> >> * https://github.com/python-git/python has a lot of > >>> >> forks/watches/starts > >>> >> but seems to be very out of date (last updated 4 years ago) > >>> >> * https://github.com/python-mirror/python doesn't appear to be very > >>> >> popular but is updated daily > >>> >> > >>> >> Are some of you the owners of these repositories? Should we > >>> >> consolidate > >>> >> to a single "semi-official" mirror? > >>> > > >>> > +1 > >>> > > >>> > Does the PSF have an official account on github? We have one on > >>> > bitbucket... > >>> > >>> I don't remember who runs this, and I thought I was in it (maybe just > >>> on BB), but: https://github.com/python > >> > >> > >> This appears to be a good place, but it doesn't have an actual mirror > >> (yet?) ;-) > > > > > > Based on the list of people who are members of github.com/python it's as > > official as it's going to get (depends on who of that group owns it). > > > > But assuming whomever owns it is okay with hosting a mirror, what exactly is > > going to be required to have it updated regularly? Someone is going to have > > to write the scripts to pull from the hg repo and then push up to github > > else it's just going to end up out-of-date on top of maintaining whatever > > setup is devised. > > > > It's slightly more work than this. You need to keep all the features > relying on hg work, like the revision version it was compiled from > etc. To be honest, if people are going to spend time tinkering with our VCS infrastructure, one of the most interesting things we could do is explore what would be involved in setting up RhodeCode on hg.python.org :) (For those that haven't seen it, RhodeCode seems broadly comparable to BitBucket feature wise, but because of the way it is licensed, the source code is freely available to all, and running your own instance is free-as-in-beer for non-profits and open source projects). Cheers, Nick. > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From cs at zip.com.au Fri Jul 26 05:48:14 2013 From: cs at zip.com.au (Cameron Simpson) Date: Fri, 26 Jul 2013 13:48:14 +1000 Subject: [Python-Dev] Daemon creation code in the standard library (was: Inherance of file descriptor and handles on Windows (PEP 446)) In-Reply-To: <20130725172658.60fe7982@pitrou.net> References: <20130725172658.60fe7982@pitrou.net> Message-ID: <20130726034814.GA58699@cskk.homeip.net> On 25Jul2013 17:26, Antoine Pitrou wrote: | Le Thu, 25 Jul 2013 12:08:18 +1000, | Ben Finney a ?crit : | > Work continues on the PEP 3143-compatible ?python-daemon?, porting it | > to Python 3 and aiming for inclusion in the standard library. | | The PEP hasn't been formally accepted yet, however. | Skimming back through the archives, one sticking point was the default | value of the "umask" parameter. Setting the umask to 0 if the user | didn't ask for something else is a disaster, security-wise. I have to say, +10 here. I have always found the convention that daemons have a umask of 0 to be utterly bogus, because almost all library code relies on the umask to set default security policy for initial file permissions. Prone to rant on this at length if required... Cheers, -- Cameron Simpson Cordless hoses have been around for quite some time. They're called buckets. - Dan Prener From ben+python at benfinney.id.au Fri Jul 26 09:11:35 2013 From: ben+python at benfinney.id.au (Ben Finney) Date: Fri, 26 Jul 2013 17:11:35 +1000 Subject: [Python-Dev] Daemon creation code in the standard library References: <7wy58vs86l.fsf_-_@benfinney.id.au> <20130725172658.60fe7982@pitrou.net> Message-ID: <7wsiz1bxso.fsf@benfinney.id.au> Antoine Pitrou writes: > Therefore I'd be -1 on [PEP 3143] until those issues are alleviated. Cameron Simpson writes: > I have always found the convention that daemons have a umask of 0 > to be utterly bogus, because almost all library code relies on the > umask to set default security policy for initial file permissions. > > Prone to rant on this at length if required... Thanks folks. We'd love to have this discussion over at the ?python-daemon-devel? discussion forum if you want to have it in more detail. -- \ ?We are all agreed that your theory is crazy. The question that | `\ divides us is whether it is crazy enough to have a chance of | _o__) being correct.? ?Niels Bohr (to Wolfgang Pauli), 1958 | Ben Finney From ronaldoussoren at mac.com Fri Jul 26 09:38:10 2013 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Fri, 26 Jul 2013 09:38:10 +0200 Subject: [Python-Dev] Daemon creation code in the standard library In-Reply-To: <7w1u6njsap.fsf@benfinney.id.au> References: <7wy58vs86l.fsf_-_@benfinney.id.au> <7w1u6njsap.fsf@benfinney.id.au> Message-ID: <54C49643-17F7-4CAE-AAB4-1FD143DC22E5@mac.com> On 25 Jul, 2013, at 4:18, Ben Finney wrote: > Ben Finney writes: > >> Work continues on the PEP 3143-compatible ?python-daemon?, porting it to >> Python 3 and aiming for inclusion in the standard library. At first glance the library appears to close all open files, with an option to exclude some specific file descriptors (that is, you need to pass a list of files that shouldn't be closed). That makes it a lot harder to do some initialization before daemonizing. I prefer to perform at least some initialization early in program startup to be able to give sensible error messages. I've had too many initscripts that claimed to have started a daemon sucessfully, only to have that daemon stop right away because it noticed a problem right after it detached itself. Ronald > > At PyPI , and > development co-ordinated at Alioth > . > >> Interested parties are invited to join us on the discussion forums > > The correct link for the ?python-daemon-devel? forum is > . > For announcements only, we have > . > > -- > \ ?This sentence contradicts itself ? no actually it doesn't.? | > `\ ?Douglas Hofstadter | > _o__) | > Ben Finney > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/ronaldoussoren%40mac.com From solipsis at pitrou.net Fri Jul 26 09:50:47 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 26 Jul 2013 09:50:47 +0200 Subject: [Python-Dev] Official github mirror for CPython? References: <51F1380E.9070608@python.org> Message-ID: <20130726095047.7c55e7d2@pitrou.net> Le Fri, 26 Jul 2013 09:31:50 +1000, Nick Coghlan a ?crit : > > To be honest, if people are going to spend time tinkering with our VCS > infrastructure, one of the most interesting things we could do is > explore what would be involved in setting up RhodeCode on > hg.python.org :) > > (For those that haven't seen it, RhodeCode seems broadly comparable to > BitBucket feature wise, but because of the way it is licensed, the > source code is freely available to all, and running your own instance > is free-as-in-beer for non-profits and open source projects). By "freely available", do you mean actual open source / free software? Regards Antoine. From p.f.moore at gmail.com Fri Jul 26 10:04:39 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Fri, 26 Jul 2013 09:04:39 +0100 Subject: [Python-Dev] Official github mirror for CPython? In-Reply-To: <20130726095047.7c55e7d2@pitrou.net> References: <51F1380E.9070608@python.org> <20130726095047.7c55e7d2@pitrou.net> Message-ID: On 26 July 2013 08:50, Antoine Pitrou wrote: > > (For those that haven't seen it, RhodeCode seems broadly comparable to > > BitBucket feature wise, but because of the way it is licensed, the > > source code is freely available to all, and running your own instance > > is free-as-in-beer for non-profits and open source projects). > > By "freely available", do you mean actual open source / free software? > Looks like it - from the website: "Open source source control management system for Mercurial and GIT with code-reviews, built in push/pull server, LDAP/AD, permissions system and full text search." Paul -------------- next part -------------- An HTML attachment was scrubbed... URL: From ronaldoussoren at mac.com Fri Jul 26 10:05:28 2013 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Fri, 26 Jul 2013 10:05:28 +0200 Subject: [Python-Dev] Official github mirror for CPython? In-Reply-To: <20130726095047.7c55e7d2@pitrou.net> References: <51F1380E.9070608@python.org> <20130726095047.7c55e7d2@pitrou.net> Message-ID: <3A462190-3B65-44ED-B699-9B0B8B1BC775@mac.com> On 26 Jul, 2013, at 9:50, Antoine Pitrou wrote: > Le Fri, 26 Jul 2013 09:31:50 +1000, > Nick Coghlan a ?crit : >> >> To be honest, if people are going to spend time tinkering with our VCS >> infrastructure, one of the most interesting things we could do is >> explore what would be involved in setting up RhodeCode on >> hg.python.org :) >> >> (For those that haven't seen it, RhodeCode seems broadly comparable to >> BitBucket feature wise, but because of the way it is licensed, the >> source code is freely available to all, and running your own instance >> is free-as-in-beer for non-profits and open source projects). > > By "freely available", do you mean actual open source / free software? It appears to be GPLv3, with a for-pay enterprise edition. The latter is free-as-in-beer for non-profits and open source projects. Ronald (See ) From dirkjan at ochtman.nl Fri Jul 26 10:03:26 2013 From: dirkjan at ochtman.nl (Dirkjan Ochtman) Date: Fri, 26 Jul 2013 10:03:26 +0200 Subject: [Python-Dev] Official github mirror for CPython? In-Reply-To: <20130726095047.7c55e7d2@pitrou.net> References: <51F1380E.9070608@python.org> <20130726095047.7c55e7d2@pitrou.net> Message-ID: On Fri, Jul 26, 2013 at 9:50 AM, Antoine Pitrou wrote: >> (For those that haven't seen it, RhodeCode seems broadly comparable to >> BitBucket feature wise, but because of the way it is licensed, the >> source code is freely available to all, and running your own instance >> is free-as-in-beer for non-profits and open source projects). > > By "freely available", do you mean actual open source / free software? It seems to be licensed under the GPLv3. https://secure.rhodecode.org/rhodecode/files/433d6385b216da52f68fa871ed1ff99f8d618613/COPYING https://rhodecode.com/blog/25/new-rhodecode-licensing Cheers, Dirkjan From solipsis at pitrou.net Fri Jul 26 10:14:27 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 26 Jul 2013 10:14:27 +0200 Subject: [Python-Dev] Official github mirror for CPython? References: <51F1380E.9070608@python.org> <20130726095047.7c55e7d2@pitrou.net> Message-ID: <20130726101427.6ecb58be@pitrou.net> Le Fri, 26 Jul 2013 10:03:26 +0200, Dirkjan Ochtman a ?crit : > On Fri, Jul 26, 2013 at 9:50 AM, Antoine Pitrou > wrote: > >> (For those that haven't seen it, RhodeCode seems broadly > >> comparable to BitBucket feature wise, but because of the way it is > >> licensed, the source code is freely available to all, and running > >> your own instance is free-as-in-beer for non-profits and open > >> source projects). > > > > By "freely available", do you mean actual open source / free > > software? > > It seems to be licensed under the GPLv3. > > https://secure.rhodecode.org/rhodecode/files/433d6385b216da52f68fa871ed1ff99f8d618613/COPYING > https://rhodecode.com/blog/25/new-rhodecode-licensing Ah, right. Apparently a certain @ncoghlan_dev contributed to that... Regards Antoine. From ncoghlan at gmail.com Fri Jul 26 12:51:16 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 26 Jul 2013 20:51:16 +1000 Subject: [Python-Dev] Official github mirror for CPython? In-Reply-To: <20130726101427.6ecb58be@pitrou.net> References: <51F1380E.9070608@python.org> <20130726095047.7c55e7d2@pitrou.net> <20130726101427.6ecb58be@pitrou.net> Message-ID: On 26 July 2013 18:14, Antoine Pitrou wrote: > Le Fri, 26 Jul 2013 10:03:26 +0200, > Dirkjan Ochtman a ?crit : >> On Fri, Jul 26, 2013 at 9:50 AM, Antoine Pitrou >> wrote: >> >> (For those that haven't seen it, RhodeCode seems broadly >> >> comparable to BitBucket feature wise, but because of the way it is >> >> licensed, the source code is freely available to all, and running >> >> your own instance is free-as-in-beer for non-profits and open >> >> source projects). >> > >> > By "freely available", do you mean actual open source / free >> > software? >> >> It seems to be licensed under the GPLv3. >> >> https://secure.rhodecode.org/rhodecode/files/433d6385b216da52f68fa871ed1ff99f8d618613/COPYING >> https://rhodecode.com/blog/25/new-rhodecode-licensing > > Ah, right. Apparently a certain @ncoghlan_dev contributed to that... Yeah, I've been publicly lamenting the closed source nature of GitHub and BitBucket for a while now, dislike the Allura UI enough that "but it's open source!" wasn't enough to win me over, and most of the other free hosting options don't support Mercurial (or have anything resembling a coherent business model). When RhodeCode brought their hosted version to my attention, it aligned so well with what I had been saying I wanted (a code hosting service making a genuine commitment to openness, with a good user experience and an apparently viable business model) that I felt *obliged* to check it out. While I haven't migrated all my personal repos yet (due to some domain name issues specific to the Hosted version), I was quite happy with the user experience of the tool itself and plan to do that migration eventually :) As far as the licensing goes, rereading the comments below the licensing announcement as well as the licenses themselves: 1. Versions prior to 2.0 (latest: 1.7.1) are already published under GPLv3 2. RhodeCode 2.0 (aka "RhodeCode Enterprise") will be the first published using a Business Source license 3. The Business Source licensing means that 2.0 *will* become open source under "GPLv2 or later" terms in August 2015 - RhodeCode won't have the option of changing their mind and deciding not to open source it after all. For the next two years, even though the source will be made available, it's not technically an open source project yet, since free usage under the business source license is limited to installations with no more than 20 Active users. More users than that requires a commercial license, which you either pay for, or is offered for free to schools and universities, public open source projects and also for development, environment and human rights NGOs. I believe Hosted is already running 2.0, so that's the one that impressed me. I'm not sure what 1.7.1 is like, since I have never used it :) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From solipsis at pitrou.net Fri Jul 26 13:46:14 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 26 Jul 2013 13:46:14 +0200 Subject: [Python-Dev] Official github mirror for CPython? References: <51F1380E.9070608@python.org> <20130726095047.7c55e7d2@pitrou.net> <20130726101427.6ecb58be@pitrou.net> Message-ID: <20130726134614.5de465b6@pitrou.net> Le Fri, 26 Jul 2013 20:51:16 +1000, Nick Coghlan a ?crit : > > I believe Hosted is already running 2.0, so that's the one that > impressed me. Strange. Apparently, free trial accounts get a 1.7.1 version: "RhodeCode 1.7.1.hosted.4356". From my POV, it would need a ton of UI polish before being pleasant to use... Also, it's not obvious whether you can integrate your repo with a bug tracker and a code review tool. Regards Antoine. From ncoghlan at gmail.com Fri Jul 26 14:06:55 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 26 Jul 2013 22:06:55 +1000 Subject: [Python-Dev] Official github mirror for CPython? In-Reply-To: <20130726134614.5de465b6@pitrou.net> References: <51F1380E.9070608@python.org> <20130726095047.7c55e7d2@pitrou.net> <20130726101427.6ecb58be@pitrou.net> <20130726134614.5de465b6@pitrou.net> Message-ID: On 26 July 2013 21:46, Antoine Pitrou wrote: > Le Fri, 26 Jul 2013 20:51:16 +1000, > Nick Coghlan a ?crit : >> >> I believe Hosted is already running 2.0, so that's the one that >> impressed me. > > Strange. Apparently, free trial accounts get a 1.7.1 version: > "RhodeCode 1.7.1.hosted.4356". > From my POV, it would need a ton of UI polish before being pleasant > to use... I tolerate Bugzilla and Gerrit every work day, which may have affected my standards for what I judge to be an acceptable user interface ;) > Also, it's not obvious whether you can integrate your repo with a > bug tracker and a code review tool. A self-hosted instance still has Mercurial on the backend, so any Mercurial based integration should still work, even if it isn't configurable through the front end. However, I hadn't actually brought it up previously because I know there would be a *lot* of work in taking it from "vague notion" to "concrete infrastructure update proposal with a solid rationale behind it" and I'm not in a position to do that work myself. I just couldn't resist mentioning it once the idea of improved GitHub syncing was brought up :) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From victor.stinner at gmail.com Fri Jul 26 14:08:35 2013 From: victor.stinner at gmail.com (Victor Stinner) Date: Fri, 26 Jul 2013 14:08:35 +0200 Subject: [Python-Dev] Inherance of file descriptor and handles on Windows (PEP 446) In-Reply-To: References: Message-ID: 2013/7/24 Guido van Rossum : > But I'm also ready to propose that all this is such a mess that we > *should* change the default fd/handle inheritance to False, *across > platforms*, and damn the torpedoes -- i.e. accept breaking all > existing 3rd party UNIX code for subprocess creation that bypasses the > subprocess module, as well as breaking uses of os.spawn*() on both > platforms that depend on FD inheritance beyond stdin/stdout/stderr). > > With the new, sane default, all we need instead of PEP 446 is a way to > make an FD inheritable after it's been created, which can be a single > os.make_inheritable(fd) call that you must apply to the fileno() of > the stream or socket object you want inherited (or directly to a FD > you created otherwise, e.g. with os.pipe()). On Windows, this should > probably only work with os.spawn*(), since otherwise you need *handle* > inheritance, not *FD* inheritance, and that's a non-portable concept > anyway. After having written 2 PEP on the topic, I slowly agree that make all file descriptors non-inheritable is the best *compromise*. It solves most, or all, issues. The main drawback is the additionnal syscalls: on some platforms, 2 additional syscalls are need to make a file descriptor non-inheritable for each creation of file descriptor. According to my benchmark on the implementation of the PEP 433: the overhead of making a file descriptor non-inheritable is between 1% and 3% (7.8 ?s => 7.9 or 8.0 ?s) on Linux 3.6. http://www.python.org/dev/peps/pep-0433/#performances Having to make a file descriptor inheritable after creating it non-inheritable is also not optimal. Making it first non-inheritable requires 0, 1 or 2 extra syscalls, and making it inheritable again require also 1 or 2 syscalls. So f=open(...); os.make_inheritable(f.fileno()) can take up to 5 syscalls (1 open + 4 fnctl), whereas it can be done in only 1 syscall (1 open). One of the motivation of the PEP 433 an 446 is to reduce the number of syscalls, even if the use case was to make sockets *non-inheritable*. If we consider that the most common case is to use non-inheritable file descriptors, having to call os.make_inheritable() may be acceptable. Windows and recent operating syscalls support creating file descriptor directly non-inheritable in a single syscalls. ioctl() can be also be used instead of fcntl() to use 1 syscall instead of 2. > We can fix multiprocessing any anything else in the stdlib that this > breaks, I presume. The CGI code rely on inheritance of file descriptors 0, 1 and 2 which are pipes. The file descriptors 0, 1 and 2 are replaced with the pipes using os.dup2(). Victor From solipsis at pitrou.net Fri Jul 26 14:25:25 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 26 Jul 2013 14:25:25 +0200 Subject: [Python-Dev] Inherance of file descriptor and handles on Windows (PEP 446) References: Message-ID: <20130726142525.79448c82@pitrou.net> Le Fri, 26 Jul 2013 14:08:35 +0200, Victor Stinner a ?crit : > > After having written 2 PEP on the topic, I slowly agree that make all > file descriptors non-inheritable is the best *compromise*. It solves > most, or all, issues. Even stdin/stdout/stderr? I think inheriting them is the sane default. > The main drawback is the additionnal syscalls: on some platforms, 2 > additional syscalls are need to make a file descriptor non-inheritable > for each creation of file descriptor. According to my benchmark on the > implementation of the PEP 433: the overhead of making a file > descriptor non-inheritable is between 1% and 3% (7.8 ?s => 7.9 or 8.0 > ?s) on Linux 3.6. 1% and 3% of what? You're telling us there's a 0.1?s overhead. It's positively tiny. Regards Antoine. From eliben at gmail.com Fri Jul 26 14:56:25 2013 From: eliben at gmail.com (Eli Bendersky) Date: Fri, 26 Jul 2013 05:56:25 -0700 Subject: [Python-Dev] Coverity Scan In-Reply-To: References: Message-ID: Just a quick question - is there a chance to convince Coverity to detect Python refcounting leaks in C API code :-) ? This could be useful not only for Python but for extensions too. As it stands now, Coverity's leak detection is Python must be pretty weak because almost everything is done via PyObject refcounts. Eli On Thu, Jul 25, 2013 at 11:48 AM, Christian Heimes wrote: > Hello, > > this is an update on my work and the current status of Coverity Scan. > > Maybe you have noticed a checkins made be me that end with the line "CID > #". These are checkins that fix an issue that was discovered by the > static code analyzer Coverity. Coverity is a commercial product but it's > a free service for some Open Source projects. Python has been analyzed > by Coverity since about 2007. Guido, Neal, Brett, Stefan and some other > developers have used Coverity before I took over. I fixed a couple of > issues before 3.3 reached the RC phase and more bugs in the last couple > of months. > > Coverity is really great and its web GUI is fun to use, too. I was able > to identify and fix resource leaks, NULL pointer issues, buffer > overflows and missing checks all over the place. Because it's a static > analyzer that follows data-flows and control-flows the tool can detect > issues in error paths that are hardly visited at all. I have started to > document Coverity here: > > http://docs.python.org/devguide/coverity.html > > > Interview > --------- > > A week ago I was contacted by Coverity. They have started a series of > articles and press releases about Open Source projects that use their > free service Coverity Scan, see > > > > http://www.coverity.com/company/press-releases/read/coverity-introduces-monthly-spotlight-series-for-coverity-scan-open-source-projects > > Two days ago I had a lovely phone interview about my involvement in the > Python project and our development style. They are going to release a > nice article in a couple of weeks. In the mean time we have time to fix > the remaining couple issues. We *might* be able to reach the highest > coverity integrity level! I have dealt with all major issues so we just > have to fix a couple of issues. > > > Current stats > ------------- > > Lines of Code: 396,179 > Defect Density: 0.05 > Total defects: 1,054 > Outstanding: 21 (Coverity Connect shows less) > Dismissed: 222 > Fixed: 811 > > http://i.imgur.com/NoELjcj.jpg > http://i.imgur.com/eJSzTUX.jpg > > > open issues > ----------- > > http://bugs.python.org/issue17899 > http://bugs.python.org/issue18556 > http://bugs.python.org/issue18555 > http://bugs.python.org/issue18552 > http://bugs.python.org/issue18551 > http://bugs.python.org/issue18550 > http://bugs.python.org/issue18528 > > > Christian > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/eliben%40gmail.com > -------------- next part -------------- An HTML attachment was scrubbed... URL: From christian at python.org Fri Jul 26 16:29:59 2013 From: christian at python.org (Christian Heimes) Date: Fri, 26 Jul 2013 16:29:59 +0200 Subject: [Python-Dev] Coverity Scan In-Reply-To: References: Message-ID: <51F287E7.6020702@python.org> Am 26.07.2013 14:56, schrieb Eli Bendersky: > Just a quick question - is there a chance to convince Coverity to detect > Python refcounting leaks in C API code :-) ? This could be useful not > only for Python but for extensions too. As it stands now, Coverity's > leak detection is Python must be pretty weak because almost everything > is done via PyObject refcounts. Coverity is able to detect some cases of refcount leaks. I don't know if the software is able to keep track of all reference counts. But it understands missing Py_DECREF() in error branches. For example: PyObject *n = PyLong_FromLong(0); PyObject *u = PyUnicode_FromString("example"); if (u == NULL) { return NULL; /* Coverity detects that 'n' leaks memory */ } Christian From brett at python.org Fri Jul 26 16:29:40 2013 From: brett at python.org (Brett Cannon) Date: Fri, 26 Jul 2013 10:29:40 -0400 Subject: [Python-Dev] Coverity Scan In-Reply-To: <51F1AD1E.70309@python.org> References: <51F1AD1E.70309@python.org> Message-ID: On Thu, Jul 25, 2013 at 6:56 PM, Christian Heimes wrote: > Am 26.07.2013 00:32, schrieb Terry Reedy: > > I found the answer here > > https://docs.google.com/file/d/0B5wQCOK_TiRiMWVqQ0xPaDEzbkU/edit > > Coverity Integrity Level 1 is 1 (defect/1000 lines) > > Level 2 is .1 (we have passed that) > > Level 3 is .01 + no major defects + <20% (all all defects?) false > > positives as that is their normal rate.# > > > > A higher false positive rates requires auditing by Coverity. They claim > > "A higher false positive rate indicates misconfiguration, usage of > > unusual idioms, or incorrect diagnosis of a large number of defects." > > They else add "or a flaw in our analysis." > > > > # Since false positives should stay constant as true positives are > > reduced toward 0, false / all should tend toward 1 (100%) if I > > understand the ratio correctly. > > About 40% of the dismissed cases are cause by a handful of issues. I > have documented these issues as "known limitations" > http://docs.python.org/devguide/coverity.html#known-limitations . > > For example about 35 false positives are related to PyLong_FromLong() > and our small integer optimization. A correct modeling file would > eliminate the false positive defects. My attempts don't work as hoped > and I don't have access to all professional coverity tools to debug my > trials. > Have you tried asking for help from Coverity? They have been rather nice so far and they may be willing to just give us free help in getting the modeling file set up properly. -Brett > > Nearly 20 false positives are caused by Py_BuildValue("N"). I'm still > astonished that Coverity understands Python's reference counting most of > the time. :) > > Did I mention that we have almost reached Level 3? All major defects > have been dealt with (one of them locally on the test machine until > Larry pushes his patch soonish), 4 of 7 minor issues must be closed and > our dismissed rate is just little over 20% (222 out of 1054 = 21%). > -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Fri Jul 26 16:30:58 2013 From: brett at python.org (Brett Cannon) Date: Fri, 26 Jul 2013 10:30:58 -0400 Subject: [Python-Dev] Coverity Scan In-Reply-To: References: Message-ID: On Fri, Jul 26, 2013 at 8:56 AM, Eli Bendersky wrote: > Just a quick question - is there a chance to convince Coverity to detect > Python refcounting leaks in C API code :-) ? > You can always ask. =) > This could be useful not only for Python but for extensions too. As it > stands now, Coverity's leak detection is Python must be pretty weak because > almost everything is done via PyObject refcounts. > Just an FYI (mostly for others since I think Eli was at PyCon in the relevant talk), David Malcolm has his work with gcc and refleak detection. But yes, it would be nice if it was in Coverity as it would then be part of the daily check. -Brett > > Eli > > > On Thu, Jul 25, 2013 at 11:48 AM, Christian Heimes wrote: > >> Hello, >> >> this is an update on my work and the current status of Coverity Scan. >> >> Maybe you have noticed a checkins made be me that end with the line "CID >> #". These are checkins that fix an issue that was discovered by the >> static code analyzer Coverity. Coverity is a commercial product but it's >> a free service for some Open Source projects. Python has been analyzed >> by Coverity since about 2007. Guido, Neal, Brett, Stefan and some other >> developers have used Coverity before I took over. I fixed a couple of >> issues before 3.3 reached the RC phase and more bugs in the last couple >> of months. >> >> Coverity is really great and its web GUI is fun to use, too. I was able >> to identify and fix resource leaks, NULL pointer issues, buffer >> overflows and missing checks all over the place. Because it's a static >> analyzer that follows data-flows and control-flows the tool can detect >> issues in error paths that are hardly visited at all. I have started to >> document Coverity here: >> >> http://docs.python.org/devguide/coverity.html >> >> >> Interview >> --------- >> >> A week ago I was contacted by Coverity. They have started a series of >> articles and press releases about Open Source projects that use their >> free service Coverity Scan, see >> >> >> >> http://www.coverity.com/company/press-releases/read/coverity-introduces-monthly-spotlight-series-for-coverity-scan-open-source-projects >> >> Two days ago I had a lovely phone interview about my involvement in the >> Python project and our development style. They are going to release a >> nice article in a couple of weeks. In the mean time we have time to fix >> the remaining couple issues. We *might* be able to reach the highest >> coverity integrity level! I have dealt with all major issues so we just >> have to fix a couple of issues. >> >> >> Current stats >> ------------- >> >> Lines of Code: 396,179 >> Defect Density: 0.05 >> Total defects: 1,054 >> Outstanding: 21 (Coverity Connect shows less) >> Dismissed: 222 >> Fixed: 811 >> >> http://i.imgur.com/NoELjcj.jpg >> http://i.imgur.com/eJSzTUX.jpg >> >> >> open issues >> ----------- >> >> http://bugs.python.org/issue17899 >> http://bugs.python.org/issue18556 >> http://bugs.python.org/issue18555 >> http://bugs.python.org/issue18552 >> http://bugs.python.org/issue18551 >> http://bugs.python.org/issue18550 >> http://bugs.python.org/issue18528 >> >> >> Christian >> >> _______________________________________________ >> Python-Dev mailing list >> Python-Dev at python.org >> http://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe: >> http://mail.python.org/mailman/options/python-dev/eliben%40gmail.com >> > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/brett%40python.org > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Fri Jul 26 16:48:03 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 26 Jul 2013 16:48:03 +0200 Subject: [Python-Dev] Coverity Scan References: <51F287E7.6020702@python.org> Message-ID: <20130726164803.66f9887e@pitrou.net> Le Fri, 26 Jul 2013 16:29:59 +0200, Christian Heimes a ?crit : > Coverity is able to detect some cases of refcount leaks. I don't know > if the software is able to keep track of all reference counts. But it > understands missing Py_DECREF() in error branches. > > For example: > > PyObject *n = PyLong_FromLong(0); > PyObject *u = PyUnicode_FromString("example"); > > if (u == NULL) { > return NULL; > /* Coverity detects that 'n' leaks memory */ > } But 'n' doesn't leak memory since PyLong_FromLong(0) is statically allocated ;-) More generally, in similar cases (e.g. replace "0" with a non-small integer), you don't need any knowledge of reference counts to infer that there is a memory leak. When the code discards the only existing pointer to a heap-allocated memory area, there's a leak. What we call "refcount leaks" is generally when an area is still pointer-accessible, but failure to decrement the reference count appropriately means it will never be released. Regards Antoine. From eliben at gmail.com Fri Jul 26 17:13:03 2013 From: eliben at gmail.com (Eli Bendersky) Date: Fri, 26 Jul 2013 08:13:03 -0700 Subject: [Python-Dev] Coverity Scan In-Reply-To: <51F287E7.6020702@python.org> References: <51F287E7.6020702@python.org> Message-ID: On Fri, Jul 26, 2013 at 7:29 AM, Christian Heimes wrote: > Am 26.07.2013 14:56, schrieb Eli Bendersky: > > Just a quick question - is there a chance to convince Coverity to detect > > Python refcounting leaks in C API code :-) ? This could be useful not > > only for Python but for extensions too. As it stands now, Coverity's > > leak detection is Python must be pretty weak because almost everything > > is done via PyObject refcounts. > > Coverity is able to detect some cases of refcount leaks. I don't know if > the software is able to keep track of all reference counts. But it > understands missing Py_DECREF() in error branches. > > For example: > > PyObject *n = PyLong_FromLong(0); > PyObject *u = PyUnicode_FromString("example"); > > if (u == NULL) { > return NULL; > /* Coverity detects that 'n' leaks memory */ > } > Interesting. I was thinking of something more general though. Especially if we can mark function arguments and return values as stealing references / creating new ones / etc, many many common refcount bugs can be detected with static analysis. This is definitely research-y, probably too much for our current stage of relationship with Coverity :) Eli -------------- next part -------------- An HTML attachment was scrubbed... URL: From status at bugs.python.org Fri Jul 26 18:07:38 2013 From: status at bugs.python.org (Python tracker) Date: Fri, 26 Jul 2013 18:07:38 +0200 (CEST) Subject: [Python-Dev] Summary of Python tracker Issues Message-ID: <20130726160738.291AD56A1A@psf.upfronthosting.co.za> ACTIVITY SUMMARY (2013-07-19 - 2013-07-26) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 4114 (+22) closed 26220 (+35) total 30334 (+57) Open issues with patches: 1856 Issues opened (37) ================== #18505: Duplicate function names in test_email.py http://bugs.python.org/issue18505 opened by vajrasky #18507: import_init() should not use Py_FatalError() but return an err http://bugs.python.org/issue18507 opened by haypo #18509: CJK decoders should return MBERR_EXCEPTION on PyUnicodeWriter http://bugs.python.org/issue18509 opened by haypo #18512: sys.stdout.write does not allow bytes in Python 3.x http://bugs.python.org/issue18512 opened by ntrrgc #18515: zipfile._ZipDecryptor generates wasteful crc32 table on import http://bugs.python.org/issue18515 opened by dholth #18516: Typos in Lib/email/generator.py and Lib/email/architecture.rst http://bugs.python.org/issue18516 opened by vajrasky #18517: "xxlimited" extension declared incorrectly in setup.py http://bugs.python.org/issue18517 opened by ned.deily #18518: return-ing within code timed with timeit.timeit causes wrong r http://bugs.python.org/issue18518 opened by icedream91 #18519: test_sqlite crashes on OS X tiger 3.x buildbot http://bugs.python.org/issue18519 opened by ned.deily #18521: [cppcheck] Full report http://bugs.python.org/issue18521 opened by serval2412 #18526: Add resource management/guarding to unittest http://bugs.python.org/issue18526 opened by zach.ware #18527: Upgrade Modules/zlib to 1.2.8 http://bugs.python.org/issue18527 opened by christian.heimes #18528: Possible fd leak in socketmodule http://bugs.python.org/issue18528 opened by christian.heimes #18529: Use long dash http://bugs.python.org/issue18529 opened by serhiy.storchaka #18530: posixpath.ismount performs extra lstat calls http://bugs.python.org/issue18530 opened by alex #18531: Undocumented different between METH_KEYWORDS and **kws http://bugs.python.org/issue18531 opened by barry #18532: hashlib.HASH objects should officially expose the hash name http://bugs.python.org/issue18532 opened by jason.coombs #18533: Avoid error from repr() of recursive dictview http://bugs.python.org/issue18533 opened by bennorth #18534: State clearly that open() 'file' param is "name" attr of the r http://bugs.python.org/issue18534 opened by ncoghlan #18535: termios.tcgetattr should return namedtuple http://bugs.python.org/issue18535 opened by techtonik #18538: `python -m dis ` relying on argparse http://bugs.python.org/issue18538 opened by maker #18539: Arguments tooltip wrong if def contains fractional default val http://bugs.python.org/issue18539 opened by ariel_bruner #18540: imaplib.IMAP4() ends with "Name or service not known" on Fedor http://bugs.python.org/issue18540 opened by sYnfo #18543: urllib.parse.urlopen shouldn't ignore installed opener when ca http://bugs.python.org/issue18543 opened by moritzs #18544: subprocess.Popen support for redirection of arbitrary file des http://bugs.python.org/issue18544 opened by ropoctorix #18546: ssl.get_server_certificate like addition for cert chain http://bugs.python.org/issue18546 opened by underrun #18548: In unittest doc change WidgetTestCase to SimpleWidgetTestCase http://bugs.python.org/issue18548 opened by py.user #18550: internal_setblocking() doesn't check return value of fcntl() http://bugs.python.org/issue18550 opened by christian.heimes #18551: child_exec() doesn't check return value of fcntl() http://bugs.python.org/issue18551 opened by christian.heimes #18552: obj2ast_object() doesn't check return value of PyArena_AddPyOb http://bugs.python.org/issue18552 opened by christian.heimes #18553: os.isatty() is not Unix only http://bugs.python.org/issue18553 opened by techtonik #18554: os.__all__ is incomplete http://bugs.python.org/issue18554 opened by ronaldoussoren #18555: type_set_bases() doesn't check return value of PyArg_UnpackTu http://bugs.python.org/issue18555 opened by christian.heimes #18558: Iterable glossary entry needs clarification http://bugs.python.org/issue18558 opened by Zero #18559: _pickle: NULL ptr dereference when PyLong_FromSsize_t() fails http://bugs.python.org/issue18559 opened by christian.heimes #18560: builtin_sum() doesn't check return value of PyLong_FromLong() http://bugs.python.org/issue18560 opened by christian.heimes #18561: ctypes _build_callargs() doesn't check name for NULL http://bugs.python.org/issue18561 opened by christian.heimes Most recent 15 issues with no replies (15) ========================================== #18561: ctypes _build_callargs() doesn't check name for NULL http://bugs.python.org/issue18561 #18560: builtin_sum() doesn't check return value of PyLong_FromLong() http://bugs.python.org/issue18560 #18559: _pickle: NULL ptr dereference when PyLong_FromSsize_t() fails http://bugs.python.org/issue18559 #18554: os.__all__ is incomplete http://bugs.python.org/issue18554 #18553: os.isatty() is not Unix only http://bugs.python.org/issue18553 #18552: obj2ast_object() doesn't check return value of PyArena_AddPyOb http://bugs.python.org/issue18552 #18551: child_exec() doesn't check return value of fcntl() http://bugs.python.org/issue18551 #18550: internal_setblocking() doesn't check return value of fcntl() http://bugs.python.org/issue18550 #18548: In unittest doc change WidgetTestCase to SimpleWidgetTestCase http://bugs.python.org/issue18548 #18544: subprocess.Popen support for redirection of arbitrary file des http://bugs.python.org/issue18544 #18543: urllib.parse.urlopen shouldn't ignore installed opener when ca http://bugs.python.org/issue18543 #18539: Arguments tooltip wrong if def contains fractional default val http://bugs.python.org/issue18539 #18527: Upgrade Modules/zlib to 1.2.8 http://bugs.python.org/issue18527 #18509: CJK decoders should return MBERR_EXCEPTION on PyUnicodeWriter http://bugs.python.org/issue18509 #18507: import_init() should not use Py_FatalError() but return an err http://bugs.python.org/issue18507 Most recent 15 issues waiting for review (15) ============================================= #18560: builtin_sum() doesn't check return value of PyLong_FromLong() http://bugs.python.org/issue18560 #18559: _pickle: NULL ptr dereference when PyLong_FromSsize_t() fails http://bugs.python.org/issue18559 #18554: os.__all__ is incomplete http://bugs.python.org/issue18554 #18552: obj2ast_object() doesn't check return value of PyArena_AddPyOb http://bugs.python.org/issue18552 #18548: In unittest doc change WidgetTestCase to SimpleWidgetTestCase http://bugs.python.org/issue18548 #18540: imaplib.IMAP4() ends with "Name or service not known" on Fedor http://bugs.python.org/issue18540 #18538: `python -m dis ` relying on argparse http://bugs.python.org/issue18538 #18533: Avoid error from repr() of recursive dictview http://bugs.python.org/issue18533 #18530: posixpath.ismount performs extra lstat calls http://bugs.python.org/issue18530 #18529: Use long dash http://bugs.python.org/issue18529 #18528: Possible fd leak in socketmodule http://bugs.python.org/issue18528 #18526: Add resource management/guarding to unittest http://bugs.python.org/issue18526 #18519: test_sqlite crashes on OS X tiger 3.x buildbot http://bugs.python.org/issue18519 #18517: "xxlimited" extension declared incorrectly in setup.py http://bugs.python.org/issue18517 #18515: zipfile._ZipDecryptor generates wasteful crc32 table on import http://bugs.python.org/issue18515 Top 10 most discussed issues (10) ================================= #15805: Add stdout redirection tool to contextlib http://bugs.python.org/issue15805 24 msgs #17899: os.listdir() leaks FDs if invoked on FD pointing to a non-dire http://bugs.python.org/issue17899 13 msgs #18535: termios.tcgetattr should return namedtuple http://bugs.python.org/issue18535 8 msgs #18519: test_sqlite crashes on OS X tiger 3.x buildbot http://bugs.python.org/issue18519 7 msgs #18530: posixpath.ismount performs extra lstat calls http://bugs.python.org/issue18530 7 msgs #18441: Idle: Make test.support.requires('gui') skip when it should. http://bugs.python.org/issue18441 6 msgs #18533: Avoid error from repr() of recursive dictview http://bugs.python.org/issue18533 6 msgs #18078: threading.Condition to allow notify on a specific waiter http://bugs.python.org/issue18078 5 msgs #18324: set_payload does not handle binary payloads correctly http://bugs.python.org/issue18324 5 msgs #18408: Fixes crashes found by pyfailmalloc http://bugs.python.org/issue18408 5 msgs Issues closed (34) ================== #8079: make install fails with -j8 with python2.6/config on FreeBSD http://bugs.python.org/issue8079 closed by christian.heimes #9177: ssl.read/write on closed socket raises AttributeError http://bugs.python.org/issue9177 closed by pitrou #14853: test_file.py depends on sys.stdin being unseekable http://bugs.python.org/issue14853 closed by ezio.melotti #15130: remove redundant paragraph in socket howto http://bugs.python.org/issue15130 closed by r.david.murray #15905: Copy to fixed size buffer w/o check in sys_update_path http://bugs.python.org/issue15905 closed by christian.heimes #16937: -u (unbuffered I/O) command line option documentation mismatch http://bugs.python.org/issue16937 closed by ezio.melotti #17818: aifc.Aifc_read/Aifc_write.getparams can return a namedtuple http://bugs.python.org/issue17818 closed by r.david.murray #17944: Refactor test_zipfile http://bugs.python.org/issue17944 closed by serhiy.storchaka #18327: swapped arguments in compatible_for_assignment()? http://bugs.python.org/issue18327 closed by christian.heimes #18439: Patchcheck for ACKS, NEWS does not work on Windows. http://bugs.python.org/issue18439 closed by terry.reedy #18460: .chm documentation files advertised in download.html but not t http://bugs.python.org/issue18460 closed by loewis #18488: sqlite: finalize() method of user function may be called with http://bugs.python.org/issue18488 closed by haypo #18491: Add "exe wrapper" functionality to Windows launcher http://bugs.python.org/issue18491 closed by python-dev #18503: No assertion in test_del_param_on_nonexistent_header function http://bugs.python.org/issue18503 closed by r.david.murray #18506: DISABLEd Tkinter.Listbox still fires <> for mou http://bugs.python.org/issue18506 closed by ned.deily #18508: enum.Enum population of _value2member_map does not match fallb http://bugs.python.org/issue18508 closed by python-dev #18510: dict.__contains__ and dict.keys().__contains__ raises exceptio http://bugs.python.org/issue18510 closed by ronaldoussoren #18511: random.shuffle could be faster http://bugs.python.org/issue18511 closed by rhettinger #18513: test.cmath fails on OS/X with gcc-4.8 in non-debug build http://bugs.python.org/issue18513 closed by mark.dickinson #18514: Unreachable Py_DECREF() in ctypes's PyCData_FromBaseObj() http://bugs.python.org/issue18514 closed by christian.heimes #18520: Fixes bugs found by pyfailmalloc during Python initialization http://bugs.python.org/issue18520 closed by haypo #18522: Error creating a raw string of r'\\?\' http://bugs.python.org/issue18522 closed by pitrou #18523: test_signal failure under Windows http://bugs.python.org/issue18523 closed by haypo #18524: BufferedReader.read1() documentation/implementation difference http://bugs.python.org/issue18524 closed by pitrou #18525: Shutil cannot import WindowsError on windows http://bugs.python.org/issue18525 closed by ronaldoussoren #18536: ????????? Google+ ???????????????????????? http://bugs.python.org/issue18536 closed by pitrou #18537: bool.toggle() http://bugs.python.org/issue18537 closed by eric.smith #18541: LoggerAdapter example is counter-productive http://bugs.python.org/issue18541 closed by python-dev #18542: httpd memory consuption increasing continously due to mod_pyth http://bugs.python.org/issue18542 closed by ronaldoussoren #18545: enum always runs member_type when use_args is True http://bugs.python.org/issue18545 closed by python-dev #18547: os.path.dirname() does not behave as expected for path without http://bugs.python.org/issue18547 closed by r.david.murray #18549: Dead code in socket_ntohl() http://bugs.python.org/issue18549 closed by christian.heimes #18556: ctypes' U_set() doesn't check return value of PyUnicode_AsWide http://bugs.python.org/issue18556 closed by brett.cannon #18557: email address parser desired http://bugs.python.org/issue18557 closed by r.david.murray From solipsis at pitrou.net Fri Jul 26 19:07:35 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 26 Jul 2013 19:07:35 +0200 Subject: [Python-Dev] Daemon creation code in the standard library References: <7wy58vs86l.fsf_-_@benfinney.id.au> <7w1u6njsap.fsf@benfinney.id.au> <54C49643-17F7-4CAE-AAB4-1FD143DC22E5@mac.com> Message-ID: <20130726190735.4c4582e2@fsol> On Fri, 26 Jul 2013 09:38:10 +0200 Ronald Oussoren wrote: > > On 25 Jul, 2013, at 4:18, Ben Finney wrote: > > > Ben Finney writes: > > > >> Work continues on the PEP 3143-compatible ?python-daemon?, porting it to > >> Python 3 and aiming for inclusion in the standard library. > > At first glance the library appears to close all open files, with an option > to exclude some specific file descriptors (that is, you need to pass a list > of files that shouldn't be closed). Indeed, it's annoying when you want to setup logging before daemonization starts. I had to hack my way through logging handlers to find the fd I had to keep open. > That makes it a lot harder to do some initialization before daemonizing. > I prefer to perform at least some initialization early in program startup to > be able to give sensible error messages. I've had too many initscripts that > claimed to have started a daemon sucessfully, only to have that daemon stop > right away because it noticed a problem right after it detached itself. Agreed. Regards Antoine. From christian at python.org Fri Jul 26 22:09:44 2013 From: christian at python.org (Christian Heimes) Date: Fri, 26 Jul 2013 22:09:44 +0200 Subject: [Python-Dev] Coverity Scan In-Reply-To: References: <51F1AD1E.70309@python.org> Message-ID: Am 26.07.2013 16:29, schrieb Brett Cannon: > Have you tried asking for help from Coverity? They have been rather nice > so far and they may be willing to just give us free help in getting the > modeling file set up properly. Yes, I'm in contact with Dakshesh. I was able to figure out one model for a false positive on my own. Dakshesh is helping me with another. Christian From victor.stinner at gmail.com Fri Jul 26 22:17:47 2013 From: victor.stinner at gmail.com (Victor Stinner) Date: Fri, 26 Jul 2013 22:17:47 +0200 Subject: [Python-Dev] Inherance of file descriptor and handles on Windows (PEP 446) In-Reply-To: <20130726142525.79448c82@pitrou.net> References: <20130726142525.79448c82@pitrou.net> Message-ID: 2013/7/26 Antoine Pitrou : >> The main drawback is the additionnal syscalls: on some platforms, 2 >> additional syscalls are need to make a file descriptor non-inheritable >> for each creation of file descriptor. According to my benchmark on the >> implementation of the PEP 433: the overhead of making a file >> descriptor non-inheritable is between 1% and 3% (7.8 ?s => 7.9 or 8.0 >> ?s) on Linux 3.6. > > 1% and 3% of what? > You're telling us there's a 0.1?s overhead. It's positively tiny. Copy-paste of the link: """ On Linux, setting the close-on-flag has a low overhead on performances. Results of bench_cloexec.py on Linux 3.6: - close-on-flag not set: 7.8 us - O_CLOEXEC: 1% slower (7.9 us) - ioctl(): 3% slower (8.0 us) - fcntl(): 3% slower (8.0 us) """ The overhead is between 0.1 and 0.2 ?s (100 and 200 ns) according to my micro-benchmark. "python -c pass" takes 19,000 ?s (0.019 sec) on my PC. It uses 207 syscalls creating file descriptors (open() and openat()): 67 are successful, 140 are failing with ENOENT. The estimated overhead on "python -c pass" is 0.2*67=13.4 ?s (0.07%). Victor From solipsis at pitrou.net Fri Jul 26 23:24:32 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 26 Jul 2013 23:24:32 +0200 Subject: [Python-Dev] Inherance of file descriptor and handles on Windows (PEP 446) In-Reply-To: References: <20130726142525.79448c82@pitrou.net> Message-ID: <20130726232432.3651f1d5@fsol> On Fri, 26 Jul 2013 22:17:47 +0200 Victor Stinner wrote: > 2013/7/26 Antoine Pitrou : > >> The main drawback is the additionnal syscalls: on some platforms, 2 > >> additional syscalls are need to make a file descriptor non-inheritable > >> for each creation of file descriptor. According to my benchmark on the > >> implementation of the PEP 433: the overhead of making a file > >> descriptor non-inheritable is between 1% and 3% (7.8 ?s => 7.9 or 8.0 > >> ?s) on Linux 3.6. > > > > 1% and 3% of what? > > You're telling us there's a 0.1?s overhead. It's positively tiny. > > Copy-paste of the link: > > """ > On Linux, setting the close-on-flag has a low overhead on > performances. Results of bench_cloexec.py on Linux 3.6: > > - close-on-flag not set: 7.8 us > - O_CLOEXEC: 1% slower (7.9 us) > - ioctl(): 3% slower (8.0 us) > - fcntl(): 3% slower (8.0 us) > """ You aren't answering my question: slower than what? Benchmarking is useless if you aren't telling us what exactly you are benchmarking. > The overhead is between 0.1 and 0.2 ?s (100 and 200 ns) according to > my micro-benchmark. That's what I figured out (see above). It's tiny. Antoine. From guido at python.org Sat Jul 27 00:12:18 2013 From: guido at python.org (Guido van Rossum) Date: Fri, 26 Jul 2013 15:12:18 -0700 Subject: [Python-Dev] Inherance of file descriptor and handles on Windows (PEP 446) In-Reply-To: References: Message-ID: On Fri, Jul 26, 2013 at 5:08 AM, Victor Stinner wrote: > After having written 2 PEP on the topic, I slowly agree that make all > file descriptors non-inheritable is the best *compromise*. It solves > most, or all, issues. Right. > The main drawback is the additionnal syscalls: on some platforms, 2 > additional syscalls are need to make a file descriptor non-inheritable > for each creation of file descriptor. According to my benchmark on the > implementation of the PEP 433: the overhead of making a file > descriptor non-inheritable is between 1% and 3% (7.8 ?s => 7.9 or 8.0 > ?s) on Linux 3.6. > http://www.python.org/dev/peps/pep-0433/#performances Remember that this is going to be Python 3.4 and newer. AFAICT setting O_CLOEXEC on open works on OSX (at least the man page on OSX 10.8 has it), on newer Linuxes, and the equivalent on Windows. So even if it does cost an extra syscall on older systems, those systems will be obsolete before Python 3.4 becomes mainstream there. And it does look like the syscalls are pretty cheap. I also don't think I'm not particularly worried about the cost of syscalls for making a socket (non)blocking -- although we should probably avoid the second fcntl() call if the first call shows the flag is already set the way we want it. -- --Guido van Rossum (python.org/~guido) From victor.stinner at gmail.com Sat Jul 27 00:18:40 2013 From: victor.stinner at gmail.com (Victor Stinner) Date: Sat, 27 Jul 2013 00:18:40 +0200 Subject: [Python-Dev] Inherance of file descriptor and handles on Windows (PEP 446) In-Reply-To: <20130726232432.3651f1d5@fsol> References: <20130726142525.79448c82@pitrou.net> <20130726232432.3651f1d5@fsol> Message-ID: 2013/7/26 Antoine Pitrou : > On Fri, 26 Jul 2013 22:17:47 +0200 >> """ >> On Linux, setting the close-on-flag has a low overhead on >> performances. Results of bench_cloexec.py on Linux 3.6: >> >> - close-on-flag not set: 7.8 us >> - O_CLOEXEC: 1% slower (7.9 us) >> - ioctl(): 3% slower (8.0 us) >> - fcntl(): 3% slower (8.0 us) >> """ > > You aren't answering my question: slower than what? Ah, you didn't understand the labels. bench_cloexec.py runs a benchmark on os.open(path, os.O_RDONLY, cloexec=False) and os.open(path, os.O_RDONLY, cloexec=True) with different implementation of making the file descriptor non-inheritable. close-on-flag not set: 7.8 us => C code: open(path, O_RDONLY) O_CLOEXEC: 1% slower (7.9 us) => C code: open(path, O_RDONLY|CLOEXEC) => 1% slower than open(path, O_RDONLY) ioctl(): 3% slower (8.0 us) => C code: fd=open(path, O_RDONLY); ioctl(fd, FIOCLEX, 0) => 3% slower than open(path, O_RDONLY) fcntl(): 3% slower (8.0 us) => C code: fd=open(path, O_RDONLY); flags = fcntl(fd, F_GETFD); fcntl(fd, F_SETFD, flags | FD_CLOEXEC) => 3% slower than open(path, O_RDONLY) Victor From solipsis at pitrou.net Sat Jul 27 00:23:59 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sat, 27 Jul 2013 00:23:59 +0200 Subject: [Python-Dev] Inherance of file descriptor and handles on Windows (PEP 446) In-Reply-To: References: <20130726142525.79448c82@pitrou.net> <20130726232432.3651f1d5@fsol> Message-ID: <20130727002359.2631ea8a@fsol> On Sat, 27 Jul 2013 00:18:40 +0200 Victor Stinner wrote: > 2013/7/26 Antoine Pitrou : > > On Fri, 26 Jul 2013 22:17:47 +0200 > >> """ > >> On Linux, setting the close-on-flag has a low overhead on > >> performances. Results of bench_cloexec.py on Linux 3.6: > >> > >> - close-on-flag not set: 7.8 us > >> - O_CLOEXEC: 1% slower (7.9 us) > >> - ioctl(): 3% slower (8.0 us) > >> - fcntl(): 3% slower (8.0 us) > >> """ > > > > You aren't answering my question: slower than what? > > Ah, you didn't understand the labels. bench_cloexec.py runs a > benchmark on os.open(path, os.O_RDONLY, cloexec=False) and > os.open(path, os.O_RDONLY, cloexec=True) with different implementation > of making the file descriptor non-inheritable. > > close-on-flag not set: 7.8 us > => C code: open(path, O_RDONLY) > > O_CLOEXEC: 1% slower (7.9 us) > => C code: open(path, O_RDONLY|CLOEXEC) > => 1% slower than open(path, O_RDONLY) > > ioctl(): 3% slower (8.0 us) > => C code: fd=open(path, O_RDONLY); ioctl(fd, FIOCLEX, 0) > => 3% slower than open(path, O_RDONLY) > > fcntl(): 3% slower (8.0 us) > => C code: fd=open(path, O_RDONLY); flags = fcntl(fd, F_GETFD); > fcntl(fd, F_SETFD, flags | FD_CLOEXEC) > => 3% slower than open(path, O_RDONLY) Ok, so I think this it is a totally reasonable compromise. People who bother about a 3% slowdown when calling os.open() can optimize the hell out of their code using Cython for all I care :-) Regards Antoine. From greg at krypto.org Sat Jul 27 06:26:00 2013 From: greg at krypto.org (Gregory P. Smith) Date: Fri, 26 Jul 2013 21:26:00 -0700 Subject: [Python-Dev] Inherance of file descriptor and handles on Windows (PEP 446) In-Reply-To: <20130727002359.2631ea8a@fsol> References: <20130726142525.79448c82@pitrou.net> <20130726232432.3651f1d5@fsol> <20130727002359.2631ea8a@fsol> Message-ID: On Fri, Jul 26, 2013 at 3:23 PM, Antoine Pitrou wrote: > On Sat, 27 Jul 2013 00:18:40 +0200 > Victor Stinner wrote: > > 2013/7/26 Antoine Pitrou : > > > On Fri, 26 Jul 2013 22:17:47 +0200 > > >> """ > > >> On Linux, setting the close-on-flag has a low overhead on > > >> performances. Results of bench_cloexec.py on Linux 3.6: > > >> > > >> - close-on-flag not set: 7.8 us > > >> - O_CLOEXEC: 1% slower (7.9 us) > > >> - ioctl(): 3% slower (8.0 us) > > >> - fcntl(): 3% slower (8.0 us) > > >> """ > > > > > > You aren't answering my question: slower than what? > > > > Ah, you didn't understand the labels. bench_cloexec.py runs a > > benchmark on os.open(path, os.O_RDONLY, cloexec=False) and > > os.open(path, os.O_RDONLY, cloexec=True) with different implementation > > of making the file descriptor non-inheritable. > > > > close-on-flag not set: 7.8 us > > => C code: open(path, O_RDONLY) > > > > O_CLOEXEC: 1% slower (7.9 us) > > => C code: open(path, O_RDONLY|CLOEXEC) > > => 1% slower than open(path, O_RDONLY) > > > > ioctl(): 3% slower (8.0 us) > > => C code: fd=open(path, O_RDONLY); ioctl(fd, FIOCLEX, 0) > > => 3% slower than open(path, O_RDONLY) > > > > fcntl(): 3% slower (8.0 us) > > => C code: fd=open(path, O_RDONLY); flags = fcntl(fd, F_GETFD); > > fcntl(fd, F_SETFD, flags | FD_CLOEXEC) > > => 3% slower than open(path, O_RDONLY) > > Ok, so I think this it is a totally reasonable compromise. > > People who bother about a 3% slowdown when calling os.open() can > optimize the hell out of their code using Cython for all I care :-) > > +1 ;) and +1 for making the sane default of noinherit / cloexec / whatever-others-call-it by default for all fds/handles ever opened by Python. It stops ignoring the issue (ie: the status quo of matching the default behavior of C as defined in the 1970s)... That is a GOOD thing. :) -gps -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Sat Jul 27 06:36:01 2013 From: guido at python.org (Guido van Rossum) Date: Fri, 26 Jul 2013 21:36:01 -0700 Subject: [Python-Dev] Inherance of file descriptor and handles on Windows (PEP 446) In-Reply-To: References: <20130726142525.79448c82@pitrou.net> <20130726232432.3651f1d5@fsol> <20130727002359.2631ea8a@fsol> Message-ID: On Fri, Jul 26, 2013 at 9:26 PM, Gregory P. Smith wrote: > > On Fri, Jul 26, 2013 at 3:23 PM, Antoine Pitrou wrote: >> >> On Sat, 27 Jul 2013 00:18:40 +0200 >> Victor Stinner wrote: >> > 2013/7/26 Antoine Pitrou : >> > > On Fri, 26 Jul 2013 22:17:47 +0200 >> > >> """ >> > >> On Linux, setting the close-on-flag has a low overhead on >> > >> performances. Results of bench_cloexec.py on Linux 3.6: >> > >> >> > >> - close-on-flag not set: 7.8 us >> > >> - O_CLOEXEC: 1% slower (7.9 us) >> > >> - ioctl(): 3% slower (8.0 us) >> > >> - fcntl(): 3% slower (8.0 us) >> > >> """ >> > > >> > > You aren't answering my question: slower than what? >> > >> > Ah, you didn't understand the labels. bench_cloexec.py runs a >> > benchmark on os.open(path, os.O_RDONLY, cloexec=False) and >> > os.open(path, os.O_RDONLY, cloexec=True) with different implementation >> > of making the file descriptor non-inheritable. >> > >> > close-on-flag not set: 7.8 us >> > => C code: open(path, O_RDONLY) >> > >> > O_CLOEXEC: 1% slower (7.9 us) >> > => C code: open(path, O_RDONLY|CLOEXEC) >> > => 1% slower than open(path, O_RDONLY) >> > >> > ioctl(): 3% slower (8.0 us) >> > => C code: fd=open(path, O_RDONLY); ioctl(fd, FIOCLEX, 0) >> > => 3% slower than open(path, O_RDONLY) >> > >> > fcntl(): 3% slower (8.0 us) >> > => C code: fd=open(path, O_RDONLY); flags = fcntl(fd, F_GETFD); >> > fcntl(fd, F_SETFD, flags | FD_CLOEXEC) >> > => 3% slower than open(path, O_RDONLY) >> >> Ok, so I think this it is a totally reasonable compromise. >> >> People who bother about a 3% slowdown when calling os.open() can >> optimize the hell out of their code using Cython for all I care :-) >> > > +1 ;) > > and +1 for making the sane default of noinherit / cloexec / > whatever-others-call-it by default for all fds/handles ever opened by > Python. It stops ignoring the issue (ie: the status quo of matching the > default behavior of C as defined in the 1970s)... That is a GOOD thing. :) Do we even need a new PEP, or should we just do it? Or can we adapt Victor's PEP 446? -- --Guido van Rossum (python.org/~guido) From ncoghlan at gmail.com Sat Jul 27 07:04:31 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 27 Jul 2013 15:04:31 +1000 Subject: [Python-Dev] Inherance of file descriptor and handles on Windows (PEP 446) In-Reply-To: References: <20130726142525.79448c82@pitrou.net> <20130726232432.3651f1d5@fsol> <20130727002359.2631ea8a@fsol> Message-ID: On 27 July 2013 14:36, Guido van Rossum wrote: > On Fri, Jul 26, 2013 at 9:26 PM, Gregory P. Smith wrote: >> >> On Fri, Jul 26, 2013 at 3:23 PM, Antoine Pitrou wrote: >>> >>> On Sat, 27 Jul 2013 00:18:40 +0200 >>> Victor Stinner wrote: >>> > 2013/7/26 Antoine Pitrou : >>> > > On Fri, 26 Jul 2013 22:17:47 +0200 >>> > >> """ >>> > >> On Linux, setting the close-on-flag has a low overhead on >>> > >> performances. Results of bench_cloexec.py on Linux 3.6: >>> > >> >>> > >> - close-on-flag not set: 7.8 us >>> > >> - O_CLOEXEC: 1% slower (7.9 us) >>> > >> - ioctl(): 3% slower (8.0 us) >>> > >> - fcntl(): 3% slower (8.0 us) >>> > >> """ >>> > > >>> > > You aren't answering my question: slower than what? >>> > >>> > Ah, you didn't understand the labels. bench_cloexec.py runs a >>> > benchmark on os.open(path, os.O_RDONLY, cloexec=False) and >>> > os.open(path, os.O_RDONLY, cloexec=True) with different implementation >>> > of making the file descriptor non-inheritable. >>> > >>> > close-on-flag not set: 7.8 us >>> > => C code: open(path, O_RDONLY) >>> > >>> > O_CLOEXEC: 1% slower (7.9 us) >>> > => C code: open(path, O_RDONLY|CLOEXEC) >>> > => 1% slower than open(path, O_RDONLY) >>> > >>> > ioctl(): 3% slower (8.0 us) >>> > => C code: fd=open(path, O_RDONLY); ioctl(fd, FIOCLEX, 0) >>> > => 3% slower than open(path, O_RDONLY) >>> > >>> > fcntl(): 3% slower (8.0 us) >>> > => C code: fd=open(path, O_RDONLY); flags = fcntl(fd, F_GETFD); >>> > fcntl(fd, F_SETFD, flags | FD_CLOEXEC) >>> > => 3% slower than open(path, O_RDONLY) >>> >>> Ok, so I think this it is a totally reasonable compromise. >>> >>> People who bother about a 3% slowdown when calling os.open() can >>> optimize the hell out of their code using Cython for all I care :-) >>> >> >> +1 ;) >> >> and +1 for making the sane default of noinherit / cloexec / >> whatever-others-call-it by default for all fds/handles ever opened by >> Python. It stops ignoring the issue (ie: the status quo of matching the >> default behavior of C as defined in the 1970s)... That is a GOOD thing. :) > > Do we even need a new PEP, or should we just do it? Or can we adapt > Victor's PEP 446? Adapting the PEP sounds good - while I agree with switching to a sane default, I think the daemonisation thread suggests there may need to be a supporting API to help force FDs created by nominated logging handlers to be inherited. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From p.f.moore at gmail.com Sat Jul 27 10:11:32 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Sat, 27 Jul 2013 09:11:32 +0100 Subject: [Python-Dev] Adding Python scripts to PATHEXT on Windows In-Reply-To: References: <51EEAB2E.9030804@v.loewis.de> Message-ID: On 23 July 2013 17:33, Paul Moore wrote: > On 23 July 2013 17:11, "Martin v. L?wis" wrote: > >> Am 15.07.13 10:26, schrieb Paul Moore: >> > Does anyone have any objections to this? I could try to write a patch, >> > but I know next to nothing about building MSIs, so if any of the >> > installer experts could help that would be fantastic. >> >> It's fine with me. I could write the patch, but will likely forget - so >> please remind me with every alpha that misses this. Also, contributions >> are welcome. >> > > Thanks. > > I think I have the basic idea of what's needed, so I'll write an initial > patch. If you can check it that'd be great. > OK, see http://bugs.python.org/issue18569 As I mention in the patch, I have yet to successfully manage to build an MSI installer (I tried a while ago, and the instructions didn't work - I'm not 100% sure why and I'm afraid I don't have the time right now to try again) so the patch will definitely need checking by someone who can build the MSI. When I get some time (probably not for a few weeks) I'll try the "how to build the installer" instructions again and see if I can work out what went wrong and either fix the issue or report back. Paul -------------- next part -------------- An HTML attachment was scrubbed... URL: From victor.stinner at gmail.com Sat Jul 27 15:44:20 2013 From: victor.stinner at gmail.com (Victor Stinner) Date: Sat, 27 Jul 2013 15:44:20 +0200 Subject: [Python-Dev] Inherance of file descriptor and handles on Windows (PEP 446) In-Reply-To: References: <20130726142525.79448c82@pitrou.net> <20130726232432.3651f1d5@fsol> <20130727002359.2631ea8a@fsol> Message-ID: 2013/7/27 Guido van Rossum : > Do we even need a new PEP, or should we just do it? Or can we adapt > Victor's PEP 446? I can rewrite the PEP 446 to: * make all file descriptors and handles non-inheritable * remove the cloexec parameter * remove everything about non-blocking sockets (O_NONBLOCK), it should be discussed in a new PEP (it's no more related to O_CLOEXEC / HANDLE_INHERIT_FLAG) Should I rename os.set_cloexec(fd, cloexec) to os.set_inheritable(fd, inheritable), and os.get_cloexec(fd) to os.get_inheritable(fd)? Or do you prefer a simple os.make_inheritable(fd) with no inheritable parameter? I prefer an explicit parameter, so it's also possible to force again non-inheritable, which also makes sense if the file descriptor was not created by Python. Victor From guido at python.org Sat Jul 27 16:36:19 2013 From: guido at python.org (Guido van Rossum) Date: Sat, 27 Jul 2013 07:36:19 -0700 Subject: [Python-Dev] Inherance of file descriptor and handles on Windows (PEP 446) In-Reply-To: References: <20130726142525.79448c82@pitrou.net> <20130726232432.3651f1d5@fsol> <20130727002359.2631ea8a@fsol> Message-ID: On Saturday, July 27, 2013, Victor Stinner wrote: > 2013/7/27 Guido van Rossum >: > > Do we even need a new PEP, or should we just do it? Or can we adapt > > Victor's PEP 446? > > I can rewrite the PEP 446 to: > > * make all file descriptors and handles non-inheritable > * remove the cloexec parameter > * remove everything about non-blocking sockets (O_NONBLOCK), it should > be discussed in a new PEP (it's no more related to O_CLOEXEC / > HANDLE_INHERIT_FLAG) Sounds good. > > Should I rename os.set_cloexec(fd, cloexec) to os.set_inheritable(fd, > inheritable), and os.get_cloexec(fd) to os.get_inheritable(fd)? Yes. > > Or do you prefer a simple os.make_inheritable(fd) with no inheritable > parameter? I prefer an explicit parameter, so it's also possible to > force again non-inheritable, which also makes sense if the file > descriptor was not created by Python. Agreed. > > Victor > -- --Guido van Rossum (on iPad) -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Sat Jul 27 16:41:30 2013 From: guido at python.org (Guido van Rossum) Date: Sat, 27 Jul 2013 07:41:30 -0700 Subject: [Python-Dev] Inherance of file descriptor and handles on Windows (PEP 446) In-Reply-To: References: <20130726142525.79448c82@pitrou.net> <20130726232432.3651f1d5@fsol> <20130727002359.2631ea8a@fsol> Message-ID: P.S. perhaps more important than a PEP rewrite is a working patch to see how realistic this is. Could you make the alpha 1 release? On Saturday, July 27, 2013, Guido van Rossum wrote: > > > On Saturday, July 27, 2013, Victor Stinner wrote: > >> 2013/7/27 Guido van Rossum : >> > Do we even need a new PEP, or should we just do it? Or can we adapt >> > Victor's PEP 446? >> >> I can rewrite the PEP 446 to: >> >> * make all file descriptors and handles non-inheritable >> * remove the cloexec parameter >> * remove everything about non-blocking sockets (O_NONBLOCK), it should >> be discussed in a new PEP (it's no more related to O_CLOEXEC / >> HANDLE_INHERIT_FLAG) > > > Sounds good. > >> >> Should I rename os.set_cloexec(fd, cloexec) to os.set_inheritable(fd, >> inheritable), and os.get_cloexec(fd) to os.get_inheritable(fd)? > > > Yes. > >> >> Or do you prefer a simple os.make_inheritable(fd) with no inheritable >> parameter? I prefer an explicit parameter, so it's also possible to >> force again non-inheritable, which also makes sense if the file >> descriptor was not created by Python. > > > Agreed. > >> >> Victor >> > > > -- > --Guido van Rossum (on iPad) > -- --Guido van Rossum (on iPad) -------------- next part -------------- An HTML attachment was scrubbed... URL: From victor.stinner at gmail.com Sat Jul 27 19:56:10 2013 From: victor.stinner at gmail.com (Victor Stinner) Date: Sat, 27 Jul 2013 19:56:10 +0200 Subject: [Python-Dev] Inherance of file descriptor and handles on Windows (PEP 446) In-Reply-To: References: <20130726142525.79448c82@pitrou.net> <20130726232432.3651f1d5@fsol> <20130727002359.2631ea8a@fsol> Message-ID: 2013/7/27 Guido van Rossum : > P.S. perhaps more important than a PEP rewrite is a working patch to see how > realistic this is. Could you make the alpha 1 release? I already ran the whole Python test suite with non-inheritable file descriptors when I developed the PEP 433: it just works. So I'm confident :-) I "just" had to fix the cgi module, and some tests. For example, test_socket checks the exact type of sockets, whereas SOCK_CLOEXEC flag is present in sockobj.type for non-inheritable sockets created with this flag. I implemented the *new* PEP 446 (not written yet :-)) in a new repository: http://hg.python.org/features/pep-446 I had to invert the value of cloexec (inheritable value is just the opposite). The implementation works but it is not completed: * The doc should be reviewed * test_swap_fds() of test_subprocess fails * The implementation should be tested on Windows, FreeBSD and Solaris * I have to check if _Py_try_set_inheritable() can/must be replaced with _Py_set_inheritable() The implementation can be seen as a patch and reviewed using the following new issue: http://bugs.python.org/issue18571 Victor From Steve.Dower at microsoft.com Sat Jul 27 22:14:49 2013 From: Steve.Dower at microsoft.com (Steve Dower) Date: Sat, 27 Jul 2013 20:14:49 +0000 Subject: [Python-Dev] Adding Python scripts to PATHEXT on Windows In-Reply-To: References: <51EEAB2E.9030804@v.loewis.de> , Message-ID: Any chance of this being made optional when installing? It provides no benefit for people who prefer to associate scripts with an editor and may be a source of confusion/complaints. Env var changes typically require a reboot to take full effect, which I've also seen cause confusion. Also, it should include .pyw, .pyz and whatever was decided instead of .pywz (.pwz, I think?). Steve (Apologies for the top post.) Sent from my Windows Phone ________________________________ From: Paul Moore Sent: ?7/?27/?2013 1:13 To: Martin v. L?wis Cc: Python Dev Subject: Re: [Python-Dev] Adding Python scripts to PATHEXT on Windows On 23 July 2013 17:33, Paul Moore > wrote: On 23 July 2013 17:11, "Martin v. L?wis" > wrote: Am 15.07.13 10:26, schrieb Paul Moore: > Does anyone have any objections to this? I could try to write a patch, > but I know next to nothing about building MSIs, so if any of the > installer experts could help that would be fantastic. It's fine with me. I could write the patch, but will likely forget - so please remind me with every alpha that misses this. Also, contributions are welcome. Thanks. I think I have the basic idea of what's needed, so I'll write an initial patch. If you can check it that'd be great. OK, see http://bugs.python.org/issue18569 As I mention in the patch, I have yet to successfully manage to build an MSI installer (I tried a while ago, and the instructions didn't work - I'm not 100% sure why and I'm afraid I don't have the time right now to try again) so the patch will definitely need checking by someone who can build the MSI. When I get some time (probably not for a few weeks) I'll try the "how to build the installer" instructions again and see if I can work out what went wrong and either fix the issue or report back. Paul -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Sun Jul 28 00:36:19 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Sat, 27 Jul 2013 23:36:19 +0100 Subject: [Python-Dev] Adding Python scripts to PATHEXT on Windows In-Reply-To: References: <51EEAB2E.9030804@v.loewis.de> Message-ID: On 27 July 2013 21:14, Steve Dower wrote: > Any chance of this being made optional when installing? It provides no > benefit for people who prefer to associate scripts with an editor and may > be a source of confusion/complaints. > Personally, I don't know how to do this so someone else would have to - but if someone does, I won't particularly object. I'd prefer it to be on by default, though (just like associating py.exe with scripts is) because otherwise it's not reasonable to assume that a .py script will work as a command in the same way that a bat file does - which is really the point of wanting this in the first place. I'm a bit confused by your comment about "people who prefer to associate scripts with an editor", though. This is only done in precisely those situations when the installer associates .py scripts with py.exe (the launcher). So if you switch that off, you don't get PATHEXT either. And if you change the association after the fact, you're presumably just as capable of changing PATHEXT. The key point here is that I want the *default* behaviour of Python when installed using the standard msi to be that if I put a script foo.py on PATH somewhere, then typing "foo" at the command line will run that script. For that to be the case, PATHEXT needs to be set, and I view that as a (minor) omission which was made when the association with py.exe was implemented. (Which reminds me - it would be nice if the standalone installer for the py launcher included setting PATHEXT as well). > Env var changes typically require a reboot to take full effect, which I've > also seen cause confusion. > Really? If I add python.exe to PATH in the installer do I need to reboot? (I never do this, so I don't know - it's a genuine question). Also, it should include .pyw, .pyz and whatever was decided instead of > .pywz (.pwz, I think?). > It's basically a command line thing - I don't see why you'd want to launch a GUI script from the command line in the same way. But again, if someone wants to do it, I don't really mind (in this case I know what to do, so if there's consensus that it's needed, I can add it to the patch myself). The PEP for pyz and pwz hasn't been accepted yet. If it does, then yes pyz should be treated the same way (and we can have the same debate about pwz as we do about pyw :-)) Paul -------------- next part -------------- An HTML attachment was scrubbed... URL: From Steve.Dower at microsoft.com Sun Jul 28 01:30:47 2013 From: Steve.Dower at microsoft.com (Steve Dower) Date: Sat, 27 Jul 2013 23:30:47 +0000 Subject: [Python-Dev] Adding Python scripts to PATHEXT on Windows In-Reply-To: References: <51EEAB2E.9030804@v.loewis.de> , Message-ID: <70f1d786f4e8472c9cd29b38f20675fe@BLUPR03MB199.namprd03.prod.outlook.com> From: Paul Moore > I'm a bit confused by your comment about "people who prefer to associate scripts > with an editor", though. This is only done in precisely those situations when > the installer associates .py scripts with py.exe (the launcher). So if you > switch that off, you don't get PATHEXT either. That's optional enough for me - I always disable that (because I typically have 4-10 different versions installed at once). > And if you change the association after the fact, you're presumably just as capable > of changing PATHEXT. Not if the association is changed by another installer (presumably with the user's explicit permission). It would be very easy for people to go a long time without noticing this until some program does ShellExecute(...) on a name that happens to be a Python script and Visual Studio pops open... >> Env var changes typically require a reboot to take full effect, which I've also seen cause confusion. > Really? If I add python.exe to PATH in the installer do I need to reboot? (I > never do this, so I don't know - it's a genuine question). Env vars are inherited by child processes, so until you create a process without inheriting them, you won't see the new value. Depending on the shortcut used, cmd.exe is likely to inherit the variables from explorer.exe, which won't see the new ones until you log off/on again. Certainly any running prompts will need to be closed and restarted after installing. That said, on my Win8 machine, cmd.exe seems to be okay after changing the variables. However, PowerShell did not see the change immediately, so cmd.exe may be getting special treatment (or just ignoring the default environment). >> Also, it should include .pyw, .pyz and whatever was decided instead >> of .pywz (.pwz, I think?). > > It's basically a command line thing - I don't see why you'd want to launch > a GUI script from the command line in the same way. That's fair enough, though it may be seen as inconsistent. TBH I've never actually seen anything in the wild other than .py. My suspicion is that the .exe wrappers would remain for tools like pip anyway, rather than breaking people who choose to deselect that option. Of course, it's easy enough to replace them. Another issue to consider is that the modification to PATHEXT can't be undone when Python is uninstalled, unless each installation adds another ".PY" and each uninstall removes only one (so my PATHEXT would look like ...;.PY;.PY;.PY;.PY;.PY;.PY;.PY;.PY;.PY;.PY). I haven't checked, but this could seriously hurt performance of anything that uses PATHEXT to search PATH for anything ending in this extension. I hope that cmd.exe and PowerShell check for duplicate extensions, but I wouldn't be betting any money that they do. Steve From victor.stinner at gmail.com Sun Jul 28 02:39:19 2013 From: victor.stinner at gmail.com (Victor Stinner) Date: Sun, 28 Jul 2013 02:39:19 +0200 Subject: [Python-Dev] PEP 446: Open issues/questions Message-ID: Hi, I have a few more questions on the PEP 446: (A) How should we support support where os.set_inheritable() is not supported? Can we announce that os.set_inheritable() is always available or not? Does such platform exist? (B) Should subprocess make the file descriptors of pass_fds inheritable? If yes, should it be done before or after the fork? If it is done after the fork and before exec, it only affects the child process, at least on Linux (the file descriptor is still non-inheritable in the parent process). (C) Should we handle standard streams (0: stdin, 1: stdout, 2: stderr) differently? For example, os.dup2(fd, 0) should make the file descriptor 0 (stdin) inheritable or non-inheritable? On Windows, os.set_inheritable(fd, False) fails (error 87, invalid argument) on standard streams (0, 1, 2) and copies of the standard streams (created by os.dup()). Victor From p.f.moore at gmail.com Sun Jul 28 09:53:18 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Sun, 28 Jul 2013 08:53:18 +0100 Subject: [Python-Dev] Adding Python scripts to PATHEXT on Windows In-Reply-To: <70f1d786f4e8472c9cd29b38f20675fe@BLUPR03MB199.namprd03.prod.outlook.com> References: <51EEAB2E.9030804@v.loewis.de> <70f1d786f4e8472c9cd29b38f20675fe@BLUPR03MB199.namprd03.prod.outlook.com> Message-ID: On 28 July 2013 00:30, Steve Dower wrote: > Another issue to consider is that the modification to PATHEXT can't be > undone when Python is uninstalled, unless each installation adds another > ".PY" and each uninstall removes only one (so my PATHEXT would look like > ...;.PY;.PY;.PY;.PY;.PY;.PY;.PY;.PY;.PY;.PY). I haven't checked, but this > could seriously hurt performance of anything that uses PATHEXT to search > PATH for anything ending in this extension. I hope that cmd.exe and > PowerShell check for duplicate extensions, but I wouldn't be betting any > money that they do. > I used the same mechanism in msi.py (which is MSI functionality, I believe) that is used to add Python's directory to PATH - this adds on install and removes on ininstall. So it should work just as well as that does. Paul -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Sun Jul 28 09:58:51 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Sun, 28 Jul 2013 08:58:51 +0100 Subject: [Python-Dev] Adding Python scripts to PATHEXT on Windows In-Reply-To: <70f1d786f4e8472c9cd29b38f20675fe@BLUPR03MB199.namprd03.prod.outlook.com> References: <51EEAB2E.9030804@v.loewis.de> <70f1d786f4e8472c9cd29b38f20675fe@BLUPR03MB199.namprd03.prod.outlook.com> Message-ID: On 28 July 2013 00:30, Steve Dower wrote: > > And if you change the association after the fact, you're presumably just > as capable > > of changing PATHEXT. > > Not if the association is changed by another installer (presumably with > the user's explicit permission). It would be very easy for people to go a > long time without noticing this until some program does ShellExecute(...) > on a name that happens to be a Python script and Visual Studio pops open... Mph. I'd argue that as Python "owns" the .py extension, any other application that changes the default action of .py scripts without both making that behaviour optional and making it off by default, is at least unfriendly and arguably even broken. But that's just a matter of opinion... As regards ShellExecute, I don't know how that is affected by PATHEXT. I looked quite hard and couldn't find complete "official" documentation on how PATHEXT is supposed to work, so I'm working a bit on experiment and best guesses here :-( I'll go and check what the ShellExecute documentation says on the matter (but I'm surprised - I got the impression that PATHEXT was purely a console mechanism, so I don't quite understand why ShellExecute would use it). Paul -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Sun Jul 28 10:11:21 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 28 Jul 2013 18:11:21 +1000 Subject: [Python-Dev] Adding Python scripts to PATHEXT on Windows In-Reply-To: References: <51EEAB2E.9030804@v.loewis.de> Message-ID: On 28 July 2013 08:36, Paul Moore wrote: > The PEP for pyz and pwz hasn't been accepted yet. If it does, then yes pyz > should be treated the same way (and we can have the same debate about pwz as > we do about pyw :-)) Oops, thanks for the reminder. I'll nudge Daniel about that :) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From solipsis at pitrou.net Sun Jul 28 11:43:41 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sun, 28 Jul 2013 11:43:41 +0200 Subject: [Python-Dev] PEP 446: Open issues/questions References: Message-ID: <20130728114341.23ec940f@fsol> On Sun, 28 Jul 2013 02:39:19 +0200 Victor Stinner wrote: > Hi, > > I have a few more questions on the PEP 446: > > (A) How should we support support where os.set_inheritable() is not > supported? Can we announce that os.set_inheritable() is always > available or not? Does such platform exist? FD_CLOEXEC is POSIX: http://pubs.opengroup.org/onlinepubs/9699919799/functions/fcntl.html > (B) Should subprocess make the file descriptors of pass_fds > inheritable? If yes, should it be done before or after the fork? If it > is done after the fork and before exec, it only affects the child > process, at least on Linux (the file descriptor is still > non-inheritable in the parent process). If it is done, it should definitely be done after the fork, IMO. subprocess shouldn't have any long-lasting effects on the process. > (C) Should we handle standard streams (0: stdin, 1: stdout, 2: stderr) > differently? For example, os.dup2(fd, 0) should make the file > descriptor 0 (stdin) inheritable or non-inheritable? On Windows, > os.set_inheritable(fd, False) fails (error 87, invalid argument) on > standard streams (0, 1, 2) and copies of the standard streams (created > by os.dup()). I have been advocating for that, but I now realize that special-casing these three descriptors in a myriad of fd-creating functions isn't very attractive. (if a standard stream fd has been closed, any fd-creating function can re-create that fd: including socket.socket(), etc.) So perhaps only the *original* standard streams should be left inheritable? The Windows error is a bit more worrying: will it bubble up to Python code when set_inheritable(fd, False) is called implicitly by a fd-creating function? Should set_inheritable() be best effort and silence that error? Regards Antoine. From cf.natali at gmail.com Sun Jul 28 12:18:55 2013 From: cf.natali at gmail.com (=?ISO-8859-1?Q?Charles=2DFran=E7ois_Natali?=) Date: Sun, 28 Jul 2013 12:18:55 +0200 Subject: [Python-Dev] PEP 446: Open issues/questions In-Reply-To: <20130728114341.23ec940f@fsol> References: <20130728114341.23ec940f@fsol> Message-ID: 2013/7/28 Antoine Pitrou : >> (C) Should we handle standard streams (0: stdin, 1: stdout, 2: stderr) >> differently? For example, os.dup2(fd, 0) should make the file >> descriptor 0 (stdin) inheritable or non-inheritable? On Windows, >> os.set_inheritable(fd, False) fails (error 87, invalid argument) on >> standard streams (0, 1, 2) and copies of the standard streams (created >> by os.dup()). > > I have been advocating for that, but I now realize that special-casing > these three descriptors in a myriad of fd-creating functions isn't very > attractive. > (if a standard stream fd has been closed, any fd-creating function can > re-create that fd: including socket.socket(), etc.) > > So perhaps only the *original* standard streams should be left > inheritable? Having stdin/stdout/stderr cloexec (e.g. after a dup() to redirect to a log file, a socket...) will likely break a lot of code, e.g. code using os.system(), or code calling exec manually (and I'm sure there's a bunch of it). Also, it'll be puzzling to have syscalls automatically set the cloexec flag. I guess a lot of people doing system programming with Python will get bitten, but that's a discussion we already had months ago... cf From ncoghlan at gmail.com Sun Jul 28 12:20:46 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 28 Jul 2013 20:20:46 +1000 Subject: [Python-Dev] Refactoring test.support into a subpackage In-Reply-To: References: Message-ID: On 15 July 2013 20:22, Nick Coghlan wrote: > At the PyCon AU sprints, some of the sprinters worked on a plan Chris > Jerdonek and I came up with months ago to convert test.support from a > module into a subpackage. > > This plan arose from some nasty test suite hacks in the lead up to the > release of Python 3.3, where some of the pkgutil tests ended up in > test.test_runpy so they could use the package creation infrastructure > I have in there without needing to extract a helper module with a > release imminent. > > The scope of the plan covers a few distinct steps: > > 1. Move Lib/test/support.py to Lib/test/support/__init__.py > (http://bugs.python.org/issue15494) > 2. Add a new (documented) test.support.pkg_helper submodule with the > package creation support code (http://bugs.python.org/issue15403 and > http://bugs.python.org/issue15376) > 3. Move the pkgutil tests to test.test_pkgutil where they belong > (http://bugs.python.org/issue15358) Just a heads up that I'm working on these first three steps this evening, so the Lib/test/support.py -> Lib/test/support/__init__.py move will happen shortly. I'll also be landing the patch to add test.support.temp_dir() and test.support.change_cwd() This shouldn't affect any other patches, unless they're adding or change things in test.support. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From chris at python.org Sun Jul 28 13:23:16 2013 From: chris at python.org (Chris Withers) Date: Sun, 28 Jul 2013 12:23:16 +0100 Subject: [Python-Dev] Official github mirror for CPython? In-Reply-To: References: <51F1380E.9070608@python.org> Message-ID: <51F4FF24.2040600@python.org> On 25/07/2013 16:30, Brett Cannon wrote: > Based on the list of people who are members of github.com/python > it's as official as it's going to get > (depends on who of that group owns it). > > But assuming whomever owns it is okay with hosting a mirror, what > exactly is going to be required to have it updated regularly? Someone is > going to have to write the scripts to pull from the hg repo and then > push up to github else it's just going to end up out-of-date on top of > maintaining whatever setup is devised. Mike Bayer has set up some pretty slick stuff in this area for SQLAlchemy and Alembic. I'm sure he could give pointers to anyone interested in getting it going... Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk From ncoghlan at gmail.com Sun Jul 28 14:41:51 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 28 Jul 2013 22:41:51 +1000 Subject: [Python-Dev] Refactoring test.support into a subpackage In-Reply-To: References: Message-ID: On 28 July 2013 20:20, Nick Coghlan wrote: >> The scope of the plan covers a few distinct steps: >> >> 1. Move Lib/test/support.py to Lib/test/support/__init__.py >> (http://bugs.python.org/issue15494) >> 2. Add a new (documented) test.support.pkg_helper submodule with the >> package creation support code (http://bugs.python.org/issue15403 and >> http://bugs.python.org/issue15376) >> 3. Move the pkgutil tests to test.test_pkgutil where they belong >> (http://bugs.python.org/issue15358) > > Just a heads up that I'm working on these first three steps this > evening, so the Lib/test/support.py -> Lib/test/support/__init__.py > move will happen shortly. > > I'll also be landing the patch to add test.support.temp_dir() and > test.support.change_cwd() The test.support switch (#15494) and the factoring out of the temporary directory helpers to test.support (#15415) are done for both 3.3 and default (so it should be OK to rely on them without causing merge conflicts in the tests). #15403, #15376 and #15358 (the package testing cleanups) are still open, but are also simpler to deal with now that test.support is a package. I also created #18576 to cover moving and documenting script_helper, and #18578 to cover moving and documenting bytecode_helper. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From Steve.Dower at microsoft.com Sun Jul 28 16:54:54 2013 From: Steve.Dower at microsoft.com (Steve Dower) Date: Sun, 28 Jul 2013 14:54:54 +0000 Subject: [Python-Dev] Adding Python scripts to PATHEXT on Windows In-Reply-To: References: <51EEAB2E.9030804@v.loewis.de> <70f1d786f4e8472c9cd29b38f20675fe@BLUPR03MB199.namprd03.prod.outlook.com>, Message-ID: <65bbfabbef24446db8be71d231ad9afe@BLUPR03MB199.namprd03.prod.outlook.com> Windows fully supports *user* customization of file associations and there is no concept of ownership. The first program installed that sets up an association will normally get it, and as of Win8 the next application will be blocked (and suggested to the user next time they open a file). Whether the other application stole the extension is irrelevant: a user is allowed to associate .py with something other than py.exe. I'm also working off experience with Windows here, I haven't done any special research. It may be that the APIs will assume .exe if no extension is specified, but I'd be surprised if backwards compatibility with with DOS, which I assume included PATHEXT, hadn't been a consideration. The Run dialog uses ShellExecute, so you can test things there easily. Sent from my Windows Phone ________________________________ From: Paul Moore Sent: ?7/?28/?2013 1:00 To: Steve Dower Cc: Martin v. L?wis; Python Dev Subject: Re: [Python-Dev] Adding Python scripts to PATHEXT on Windows On 28 July 2013 00:30, Steve Dower > wrote: > And if you change the association after the fact, you're presumably just as capable > of changing PATHEXT. Not if the association is changed by another installer (presumably with the user's explicit permission). It would be very easy for people to go a long time without noticing this until some program does ShellExecute(...) on a name that happens to be a Python script and Visual Studio pops open... Mph. I'd argue that as Python "owns" the .py extension, any other application that changes the default action of .py scripts without both making that behaviour optional and making it off by default, is at least unfriendly and arguably even broken. But that's just a matter of opinion... As regards ShellExecute, I don't know how that is affected by PATHEXT. I looked quite hard and couldn't find complete "official" documentation on how PATHEXT is supposed to work, so I'm working a bit on experiment and best guesses here :-( I'll go and check what the ShellExecute documentation says on the matter (but I'm surprised - I got the impression that PATHEXT was purely a console mechanism, so I don't quite understand why ShellExecute would use it). Paul -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Mon Jul 29 10:11:30 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 29 Jul 2013 10:11:30 +0200 Subject: [Python-Dev] PEP 446: Open issues/questions References: <20130728114341.23ec940f@fsol> Message-ID: <20130729101130.3b12d0fb@pitrou.net> Le Sun, 28 Jul 2013 12:18:55 +0200, Charles-Fran?ois Natali a ?crit : > 2013/7/28 Antoine Pitrou : > >> (C) Should we handle standard streams (0: stdin, 1: stdout, 2: > >> stderr) differently? For example, os.dup2(fd, 0) should make the > >> file descriptor 0 (stdin) inheritable or non-inheritable? On > >> Windows, os.set_inheritable(fd, False) fails (error 87, invalid > >> argument) on standard streams (0, 1, 2) and copies of the standard > >> streams (created by os.dup()). > > > > I have been advocating for that, but I now realize that > > special-casing these three descriptors in a myriad of fd-creating > > functions isn't very attractive. > > (if a standard stream fd has been closed, any fd-creating function > > can re-create that fd: including socket.socket(), etc.) > > > > So perhaps only the *original* standard streams should be left > > inheritable? > > Having stdin/stdout/stderr cloexec (e.g. after a dup() to redirect to > a log file, a socket...) will likely break a lot of code, e.g. code > using os.system(), or code calling exec manually (and I'm sure there's > a bunch of it). Hmm. os.exec*() could easily make standard streams non-CLOEXEC before calling the underlying C library function. Things are more annoying for os.system(), though. > Also, it'll be puzzling to have syscalls automatically set the cloexec > flag. I guess a lot of people doing system programming with Python > will get bitten, but that's a discussion we already had months ago... Perhaps this advocates for a global flag, e.g. sys.set_default_fd_inheritance(), with False (non-inheritable) being the default for sanity and security. Regards Antoine. From ethan at stoneleaf.us Mon Jul 29 09:55:53 2013 From: ethan at stoneleaf.us (Ethan Furman) Date: Mon, 29 Jul 2013 00:55:53 -0700 Subject: [Python-Dev] MyOpenID.com no longer supported Message-ID: <51F62009.7080304@stoneleaf.us> Excerpt from http://meta.stackoverflow.com/q/190442/176681: Janrain no longer actively supports MyOpenID, and announced on Twitter that their users should proceed with caution. This decision was made by Janrain, [snip] I know the Python bug tracker allows MyOpenID logins; if that is your only login method you should add another that doesn't depend on MyOpenID. -- ~Ethan~ From ronaldoussoren at mac.com Mon Jul 29 12:30:34 2013 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Mon, 29 Jul 2013 12:30:34 +0200 Subject: [Python-Dev] PEP 446: cloexec flag on os.open Message-ID: Victor, PEP 446 mentions that a cloexec flag gets added to os.open. This API already has a way to specify this: the O_CLOEXEC bit in the flags argument. A new cloexec parameter is nicely consistent with the other APIs, but introcudes a second way to set that flag. What will the following calls do?: os.open(path, os.O_RDONLY|os.O_CLOEXEC, cloexec=False) os.open(path, os.O_RDONLY, cloexec=True) The PEP doesn't specify this, but the implementation for PEP 443 in issue 17036 basicly ignores the cloexec argument in the first call and adds O_CLOEXEC in the second call. That can lead to confusing behavior when the flags argument to os.open is passed from elsewhere (e.g. a wrapper around os.open that passes in arguments and just overrides the cloexec argument). It might be better to just drop the cloexec flag to os.open and make os.O_CLOEXEC an alias for os.O_NOINHERIT on Windows. Ronald From ronaldoussoren at mac.com Mon Jul 29 14:49:18 2013 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Mon, 29 Jul 2013 14:49:18 +0200 Subject: [Python-Dev] PEP 447: add type.__locallookup__ Message-ID: <75030FAC-6918-4E94-95DA-67A88D53E6F5@mac.com> Hi, This PEP proposed to add a __locallookup__ slot to type objects, which is used by _PyType_Lookup and super_getattro instead of peeking in the tp_dict of classes. The PEP text explains why this is needed. Differences with the previous version: * Better explanation of why this is a useful addition * type.__locallookup__ is no longer optional. * I've added benchmarking results using pybench. (using the patch attached to issue 18181) Ronald PEP: 447 Title: Add __locallookup__ method to metaclass Version: $Revision$ Last-Modified: $Date$ Author: Ronald Oussoren Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 12-Jun-2013 Post-History: 2-Jul-2013, 15-Jul-2013, 29-Jul-2013 Abstract ======== Currently ``object.__getattribute__`` and ``super.__getattribute__`` peek in the ``__dict__`` of classes on the MRO for a class when looking for an attribute. This PEP adds an optional ``__locallookup__`` method to a metaclass that can be used to override this behavior. Rationale ========= It is currently not possible to influence how the `super class`_ looks up attributes (that is, ``super.__getattribute__`` unconditionally peeks in the class ``__dict__``), and that can be problematic for dynamic classes that can grow new methods on demand. The ``__locallookup__`` method makes it possible to dynamicly add attributes even when looking them up using the `super class`_. The new method affects ``object.__getattribute__`` (and `PyObject_GenericGetAttr`_) as well for consistency. Background ---------- The current behavior of ``super.__getattribute__`` causes problems for classes that are dynamic proxies for other (non-Python) classes or types, an example of which is `PyObjC`_. PyObjC creates a Python class for every class in the Objective-C runtime, and looks up methods in the Objective-C runtime when they are used. This works fine for normal access, but doesn't work for access with ``super`` objects. Because of this PyObjC currently includes a custom ``super`` that must be used with its classes. The API in this PEP makes it possible to remove the custom ``super`` and simplifies the implementation because the custom lookup behavior can be added in a central location. The superclass attribute lookup hook ==================================== Both ``super.__getattribute__`` and ``object.__getattribute__`` (or `PyObject_GenericGetAttr`_ in C code) walk an object's MRO and peek in the class' ``__dict__`` to look up attributes. A way to affect this lookup is using a method on the meta class for the type, that by default looks up the name in the class ``__dict__``. In Python code -------------- A meta type can define a method ``__locallookup__`` that is called during attribute resolution by both ``super.__getattribute__`` and ``object.__getattribute``:: class MetaType(type): def __locallookup__(cls, name): try: return cls.__dict__[name] except KeyError: raise AttributeError(name) from None The ``__locallookup__`` method has as its arguments a class and the name of the attribute that is looked up. It should return the value of the attribute without invoking descriptors, or raise `AttributeError`_ when the name cannot be found. The `type`_ class provides a default implementation for ``__locallookup__``, that looks up the name in the class dictionary. Example usage ............. The code below implements a silly metaclass that redirects attribute lookup to uppercase versions of names:: class UpperCaseAccess (type): def __locallookup__(cls, name): return cls.__dict__[name.upper()] class SillyObject (metaclass=UpperCaseAccess): def m(self): return 42 def M(self): return "fourtytwo" obj = SillyObject() assert obj.m() == "fortytwo" In C code --------- A new slot ``tp_locallookup`` is added to the ``PyTypeObject`` struct, this slot corresponds to the ``__locallookup__`` method on `type`_. The slot has the following prototype:: PyObject* (*locallookupfunc)(PyTypeObject* cls, PyObject* name); This method should lookup *name* in the namespace of *cls*, without looking at superclasses, and should not invoke descriptors. The method returns ``NULL`` without setting an exception when the *name* cannot be found, and returns a new reference otherwise (not a borrowed reference). Use of this hook by the interpreter ----------------------------------- The new method is required for metatypes and as such is defined on `type_`. Both ``super.__getattribute__`` and ``object.__getattribute__``/`PyObject_GenericGetAttr`_ (through ``_PyType_Lookup``) use the this ``__locallookup__`` method when walking the MRO. Other changes to the implementation ----------------------------------- The change for `PyObject_GenericGetAttr`_ will be done by changing the private function ``_PyType_Lookup``. This currently returns a borrowed reference, but must return a new reference when the ``__locallookup__`` method is present. Because of this ``_PyType_Lookup`` will be renamed to ``_PyType_LookupName``, this will cause compile-time errors for all out-of-tree users of this private API. The attribute lookup cache in ``Objects/typeobject.c`` is disabled for classes that have a metaclass that overrides ``__locallookup__``, because using the cache might not be valid for such classes. Performance impact ------------------ The pybench output below compares an implementation of this PEP with the regular source tree, both based on changeset a5681f50bae2, run on an idle machine an Core i7 processor running Centos 6.4. Even though the machine was idle there were clear differences between runs, I've seen difference in "minimum time" vary from -0.1% to +1.5%, with simular (but slightly smaller) differences in the "average time" difference. .. :: ------------------------------------------------------------------------------- PYBENCH 2.1 ------------------------------------------------------------------------------- * using CPython 3.4.0a0 (default, Jul 29 2013, 13:01:34) [GCC 4.4.7 20120313 (Red Hat 4.4.7-3)] * disabled garbage collection * system check interval set to maximum: 2147483647 * using timer: time.perf_counter * timer: resolution=1e-09, implementation=clock_gettime(CLOCK_MONOTONIC) ------------------------------------------------------------------------------- Benchmark: pep447.pybench ------------------------------------------------------------------------------- Rounds: 10 Warp: 10 Timer: time.perf_counter Machine Details: Platform ID: Linux-2.6.32-358.114.1.openstack.el6.x86_64-x86_64-with-centos-6.4-Final Processor: x86_64 Python: Implementation: CPython Executable: /tmp/default-pep447/bin/python3 Version: 3.4.0a0 Compiler: GCC 4.4.7 20120313 (Red Hat 4.4.7-3) Bits: 64bit Build: Jul 29 2013 14:09:12 (#default) Unicode: UCS4 ------------------------------------------------------------------------------- Comparing with: default.pybench ------------------------------------------------------------------------------- Rounds: 10 Warp: 10 Timer: time.perf_counter Machine Details: Platform ID: Linux-2.6.32-358.114.1.openstack.el6.x86_64-x86_64-with-centos-6.4-Final Processor: x86_64 Python: Implementation: CPython Executable: /tmp/default/bin/python3 Version: 3.4.0a0 Compiler: GCC 4.4.7 20120313 (Red Hat 4.4.7-3) Bits: 64bit Build: Jul 29 2013 13:01:34 (#default) Unicode: UCS4 Test minimum run-time average run-time this other diff this other diff ------------------------------------------------------------------------------- BuiltinFunctionCalls: 45ms 44ms +1.3% 45ms 44ms +1.3% BuiltinMethodLookup: 26ms 27ms -2.4% 27ms 27ms -2.2% CompareFloats: 33ms 34ms -0.7% 33ms 34ms -1.1% CompareFloatsIntegers: 66ms 67ms -0.9% 66ms 67ms -0.8% CompareIntegers: 51ms 50ms +0.9% 51ms 50ms +0.8% CompareInternedStrings: 34ms 33ms +0.4% 34ms 34ms -0.4% CompareLongs: 29ms 29ms -0.1% 29ms 29ms -0.0% CompareStrings: 43ms 44ms -1.8% 44ms 44ms -1.8% ComplexPythonFunctionCalls: 44ms 42ms +3.9% 44ms 42ms +4.1% ConcatStrings: 33ms 33ms -0.4% 33ms 33ms -1.0% CreateInstances: 47ms 48ms -2.9% 47ms 49ms -3.4% CreateNewInstances: 35ms 36ms -2.5% 36ms 36ms -2.5% CreateStringsWithConcat: 69ms 70ms -0.7% 69ms 70ms -0.9% DictCreation: 52ms 50ms +3.1% 52ms 50ms +3.0% DictWithFloatKeys: 40ms 44ms -10.1% 43ms 45ms -5.8% DictWithIntegerKeys: 32ms 36ms -11.2% 35ms 37ms -4.6% DictWithStringKeys: 29ms 34ms -15.7% 35ms 40ms -11.0% ForLoops: 30ms 29ms +2.2% 30ms 29ms +2.2% IfThenElse: 38ms 41ms -6.7% 38ms 41ms -6.9% ListSlicing: 36ms 36ms -0.7% 36ms 37ms -1.3% NestedForLoops: 43ms 45ms -3.1% 43ms 45ms -3.2% NestedListComprehensions: 39ms 40ms -1.7% 39ms 40ms -2.1% NormalClassAttribute: 86ms 82ms +5.1% 86ms 82ms +5.0% NormalInstanceAttribute: 42ms 42ms +0.3% 42ms 42ms +0.0% PythonFunctionCalls: 39ms 38ms +3.5% 39ms 38ms +2.8% PythonMethodCalls: 51ms 49ms +3.0% 51ms 50ms +2.8% Recursion: 67ms 68ms -1.4% 67ms 68ms -1.4% SecondImport: 41ms 36ms +12.5% 41ms 36ms +12.6% SecondPackageImport: 45ms 40ms +13.1% 45ms 40ms +13.2% SecondSubmoduleImport: 92ms 95ms -2.4% 95ms 98ms -3.6% SimpleComplexArithmetic: 28ms 28ms -0.1% 28ms 28ms -0.2% SimpleDictManipulation: 57ms 57ms -1.0% 57ms 58ms -1.0% SimpleFloatArithmetic: 29ms 28ms +4.7% 29ms 28ms +4.9% SimpleIntFloatArithmetic: 37ms 41ms -8.5% 37ms 41ms -8.7% SimpleIntegerArithmetic: 37ms 41ms -9.4% 37ms 42ms -10.2% SimpleListComprehensions: 33ms 33ms -1.9% 33ms 34ms -2.9% SimpleListManipulation: 28ms 30ms -4.3% 29ms 30ms -4.1% SimpleLongArithmetic: 26ms 26ms +0.5% 26ms 26ms +0.5% SmallLists: 40ms 40ms +0.1% 40ms 40ms +0.1% SmallTuples: 46ms 47ms -2.4% 46ms 48ms -3.0% SpecialClassAttribute: 126ms 120ms +4.7% 126ms 121ms +4.4% SpecialInstanceAttribute: 42ms 42ms +0.6% 42ms 42ms +0.8% StringMappings: 94ms 91ms +3.9% 94ms 91ms +3.8% StringPredicates: 48ms 49ms -1.7% 48ms 49ms -2.1% StringSlicing: 45ms 45ms +1.4% 46ms 45ms +1.5% TryExcept: 23ms 22ms +4.9% 23ms 22ms +4.8% TryFinally: 32ms 32ms -0.1% 32ms 32ms +0.1% TryRaiseExcept: 17ms 17ms +0.9% 17ms 17ms +0.5% TupleSlicing: 49ms 48ms +1.1% 49ms 49ms +1.0% WithFinally: 48ms 47ms +2.3% 48ms 47ms +2.4% WithRaiseExcept: 45ms 44ms +0.8% 45ms 45ms +0.5% ------------------------------------------------------------------------------- Totals: 2284ms 2287ms -0.1% 2306ms 2308ms -0.1% (this=pep447.pybench, other=default.pybench) Alternative proposals --------------------- ``__getattribute_super__`` .......................... An earlier version of this PEP used the following static method on classes:: def __getattribute_super__(cls, name, object, owner): pass This method performed name lookup as well as invoking descriptors and was necessarily limited to working only with ``super.__getattribute__``. Reuse ``tp_getattro`` ..................... It would be nice to avoid adding a new slot, thus keeping the API simpler and easier to understand. A comment on `Issue 18181`_ asked about reusing the ``tp_getattro`` slot, that is super could call the ``tp_getattro`` slot of all methods along the MRO. That won't work because ``tp_getattro`` will look in the instance ``__dict__`` before it tries to resolve attributes using classes in the MRO. This would mean that using ``tp_getattro`` instead of peeking the class dictionaries changes the semantics of the `super class`_. References ========== * `Issue 18181`_ contains a prototype implementation Copyright ========= This document has been placed in the public domain. .. _`Issue 18181`: http://bugs.python.org/issue18181 .. _`super class`: http://docs.python.org/3/library/functions.html#super .. _`NotImplemented`: http://docs.python.org/3/library/constants.html#NotImplemented .. _`PyObject_GenericGetAttr`: http://docs.python.org/3/c-api/object.html#PyObject_GenericGetAttr .. _`type`: http://docs.python.org/3/library/functions.html#type .. _`AttributeError`: http://docs.python.org/3/library/exceptions.html#AttributeError .. _`PyObjC`: http://pyobjc.sourceforge.net/ .. _`classmethod`: http://docs.python.org/3/library/functions.html#classmethod From solipsis at pitrou.net Mon Jul 29 14:58:30 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 29 Jul 2013 14:58:30 +0200 Subject: [Python-Dev] PEP 447: add type.__locallookup__ References: <75030FAC-6918-4E94-95DA-67A88D53E6F5@mac.com> Message-ID: <20130729145830.2d0317e7@pitrou.net> Hi, Le Mon, 29 Jul 2013 14:49:18 +0200, Ronald Oussoren a ?crit : > Hi, > > This PEP proposed to add a __locallookup__ slot to type objects, > which is used by _PyType_Lookup and super_getattro instead of peeking > in the tp_dict of classes. The PEP text explains why this is needed. > > Differences with the previous version: > > * Better explanation of why this is a useful addition > > * type.__locallookup__ is no longer optional. > > * I've added benchmarking results using pybench. > (using the patch attached to issue 18181) Could you please run the whole benchmark suite? http://hg.python.org/benchmarks/ Thanks Antoine. From ronaldoussoren at mac.com Mon Jul 29 15:07:37 2013 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Mon, 29 Jul 2013 15:07:37 +0200 Subject: [Python-Dev] PEP 447: add type.__locallookup__ In-Reply-To: <20130729145830.2d0317e7@pitrou.net> References: <75030FAC-6918-4E94-95DA-67A88D53E6F5@mac.com> <20130729145830.2d0317e7@pitrou.net> Message-ID: <16097E4E-CF34-4660-AF70-FF67C0BDC393@mac.com> On 29 Jul, 2013, at 14:58, Antoine Pitrou wrote: > > Hi, > > Le Mon, 29 Jul 2013 14:49:18 +0200, > Ronald Oussoren a ?crit : >> Hi, >> >> This PEP proposed to add a __locallookup__ slot to type objects, >> which is used by _PyType_Lookup and super_getattro instead of peeking >> in the tp_dict of classes. The PEP text explains why this is needed. >> >> Differences with the previous version: >> >> * Better explanation of why this is a useful addition >> >> * type.__locallookup__ is no longer optional. >> >> * I've added benchmarking results using pybench. >> (using the patch attached to issue 18181) > > Could you please run the whole benchmark suite? > http://hg.python.org/benchmarks/ Sure. Ronald From ronaldoussoren at mac.com Mon Jul 29 15:26:41 2013 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Mon, 29 Jul 2013 15:26:41 +0200 Subject: [Python-Dev] PEP 447: add type.__locallookup__ In-Reply-To: <16097E4E-CF34-4660-AF70-FF67C0BDC393@mac.com> References: <75030FAC-6918-4E94-95DA-67A88D53E6F5@mac.com> <20130729145830.2d0317e7@pitrou.net> <16097E4E-CF34-4660-AF70-FF67C0BDC393@mac.com> Message-ID: On 29 Jul, 2013, at 15:07, Ronald Oussoren wrote: > > On 29 Jul, 2013, at 14:58, Antoine Pitrou wrote: > >> >> Hi, >> >> Le Mon, 29 Jul 2013 14:49:18 +0200, >> Ronald Oussoren a ?crit : >>> Hi, >>> >>> This PEP proposed to add a __locallookup__ slot to type objects, >>> which is used by _PyType_Lookup and super_getattro instead of peeking >>> in the tp_dict of classes. The PEP text explains why this is needed. >>> >>> Differences with the previous version: >>> >>> * Better explanation of why this is a useful addition >>> >>> * type.__locallookup__ is no longer optional. >>> >>> * I've added benchmarking results using pybench. >>> (using the patch attached to issue 18181) >> >> Could you please run the whole benchmark suite? >> http://hg.python.org/benchmarks/ > > Sure. That's harder than I had expected, when I use the "make_perf3.sh" to create a python 3 compatible version of the benchmark suite and then run the suite it craps out because it cannnot find "spitfire", which isn't translated (as are several other benchmarks). I'll have to investigate why the suite doesn't work. Ronald > > Ronald > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/ronaldoussoren%40mac.com From solipsis at pitrou.net Mon Jul 29 15:41:25 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 29 Jul 2013 15:41:25 +0200 Subject: [Python-Dev] PEP 447: add type.__locallookup__ References: <75030FAC-6918-4E94-95DA-67A88D53E6F5@mac.com> <20130729145830.2d0317e7@pitrou.net> <16097E4E-CF34-4660-AF70-FF67C0BDC393@mac.com> Message-ID: <20130729154125.28c3e7c2@pitrou.net> Le Mon, 29 Jul 2013 15:26:41 +0200, Ronald Oussoren a ?crit : > > On 29 Jul, 2013, at 15:07, Ronald Oussoren > wrote: > > > > > On 29 Jul, 2013, at 14:58, Antoine Pitrou > > wrote: > > > >> > >> Hi, > >> > >> Le Mon, 29 Jul 2013 14:49:18 +0200, > >> Ronald Oussoren a ?crit : > >>> Hi, > >>> > >>> This PEP proposed to add a __locallookup__ slot to type objects, > >>> which is used by _PyType_Lookup and super_getattro instead of > >>> peeking in the tp_dict of classes. The PEP text explains why > >>> this is needed. > >>> > >>> Differences with the previous version: > >>> > >>> * Better explanation of why this is a useful addition > >>> > >>> * type.__locallookup__ is no longer optional. > >>> > >>> * I've added benchmarking results using pybench. > >>> (using the patch attached to issue 18181) > >> > >> Could you please run the whole benchmark suite? > >> http://hg.python.org/benchmarks/ > > > > Sure. > > That's harder than I had expected, when I use the "make_perf3.sh" to > create a python 3 compatible version of the benchmark suite and then > run the suite it craps out because it cannnot find "spitfire", which > isn't translated (as are several other benchmarks). Ah, sorry. I think it's enough to run the "2n3" subsuite here, though. Regards Antoine. From guido at python.org Mon Jul 29 17:08:30 2013 From: guido at python.org (Guido van Rossum) Date: Mon, 29 Jul 2013 08:08:30 -0700 Subject: [Python-Dev] MyOpenID.com no longer supported In-Reply-To: <51F62009.7080304@stoneleaf.us> References: <51F62009.7080304@stoneleaf.us> Message-ID: On Mon, Jul 29, 2013 at 12:55 AM, Ethan Furman wrote: > Excerpt from http://meta.stackoverflow.com/q/190442/176681: > > Janrain no longer actively supports MyOpenID, and announced on Twitter that > their users should proceed with caution. > > This decision was made by Janrain, [snip] > > I know the Python bug tracker allows MyOpenID logins; if that is your only > login method you should add another that doesn't depend on MyOpenID. I imagine plenty of people don't read either of the two lists you posted to. Would it be possible to dig through the database and find out which users use MyOpenID and email them individually? -- --Guido van Rossum (python.org/~guido) From rdmurray at bitdance.com Mon Jul 29 18:56:27 2013 From: rdmurray at bitdance.com (R. David Murray) Date: Mon, 29 Jul 2013 12:56:27 -0400 Subject: [Python-Dev] MyOpenID.com no longer supported In-Reply-To: References: <51F62009.7080304@stoneleaf.us> Message-ID: <20130729165628.4CBB02502CD@webabinitio.net> On Mon, 29 Jul 2013 08:08:30 -0700, Guido van Rossum wrote: > On Mon, Jul 29, 2013 at 12:55 AM, Ethan Furman wrote: > > Excerpt from http://meta.stackoverflow.com/q/190442/176681: > > > > Janrain no longer actively supports MyOpenID, and announced on Twitter that > > their users should proceed with caution. > > > > This decision was made by Janrain, [snip] > > > > I know the Python bug tracker allows MyOpenID logins; if that is your only > > login method you should add another that doesn't depend on MyOpenID. > > I imagine plenty of people don't read either of the two lists you > posted to. Would it be possible to dig through the database and find > out which users use MyOpenID and email them individually? I don't know for sure that it is possible, but I presume that it is. However, when I tried to do it, I couldn't figure out how to get from the _openid_discovery table (which has 22 entries that have myopenid as the endpoint server) to the corresponding user record. Either Martin needs to clue me in, or I'll have to find time to read his openid code :) --David From christian at python.org Mon Jul 29 19:15:40 2013 From: christian at python.org (Christian Heimes) Date: Mon, 29 Jul 2013 19:15:40 +0200 Subject: [Python-Dev] C code coverage report with lcov Message-ID: Hi, I have done some experiments with GCC's gcov and lcov to get the C code coverage of our unit test suite. You may find today's report at http://tiran.bitbucket.org/python-lcov/ I'm working on a patch for our Makefile to include all steps in one simple make tag. http://bugs.python.org/issue18481 Christian From brett at python.org Mon Jul 29 19:58:55 2013 From: brett at python.org (Brett Cannon) Date: Mon, 29 Jul 2013 13:58:55 -0400 Subject: [Python-Dev] C code coverage report with lcov In-Reply-To: References: Message-ID: On Mon, Jul 29, 2013 at 1:15 PM, Christian Heimes wrote: > Hi, > > I have done some experiments with GCC's gcov and lcov to get the C code > coverage of our unit test suite. You may find today's report at > > http://tiran.bitbucket.org/python-lcov/ Thanks! I took a quick poke around and it seems some things are legitimately not being executed, while others are error conditions that we wouldn't expect to occur (e.g. memory exhaustion). If we ever decide to get serious about code coverage (both C and Python code) we may need to have a discussion as a group about our feelings related to pragmas dictating when code should be left out of coverage reports. -------------- next part -------------- An HTML attachment was scrubbed... URL: From rdmurray at bitdance.com Mon Jul 29 20:11:17 2013 From: rdmurray at bitdance.com (R. David Murray) Date: Mon, 29 Jul 2013 14:11:17 -0400 Subject: [Python-Dev] C code coverage report with lcov In-Reply-To: References: Message-ID: <20130729181118.8450025014C@webabinitio.net> On Mon, 29 Jul 2013 13:58:55 -0400, Brett Cannon wrote: > On Mon, Jul 29, 2013 at 1:15 PM, Christian Heimes wrote: > > > Hi, > > > > I have done some experiments with GCC's gcov and lcov to get the C code > > coverage of our unit test suite. You may find today's report at > > > > http://tiran.bitbucket.org/python-lcov/ > > > Thanks! > > I took a quick poke around and it seems some things are legitimately not > being executed, while others are error conditions that we wouldn't expect > to occur (e.g. memory exhaustion). If we ever decide to get serious about > code coverage (both C and Python code) we may need to have a discussion as > a group about our feelings related to pragmas dictating when code should be > left out of coverage reports. I suspect Victor will eventually have something for us for exercising the memory exhaustion code :) --David From christian at python.org Mon Jul 29 21:31:02 2013 From: christian at python.org (Christian Heimes) Date: Mon, 29 Jul 2013 21:31:02 +0200 Subject: [Python-Dev] C code coverage report with lcov In-Reply-To: References: Message-ID: <51F6C2F6.8040704@python.org> Am 29.07.2013 19:58, schrieb Brett Cannon: > I took a quick poke around and it seems some things are legitimately not > being executed, while others are error conditions that we wouldn't > expect to occur (e.g. memory exhaustion). If we ever decide to get > serious about code coverage (both C and Python code) we may need to have > a discussion as a group about our feelings related to pragmas dictating > when code should be left out of coverage reports. Yeah, object allocation and creation checks as well as verification of operating system API call are reason for lots branch and line misses. I'm not sure what we can do about that. I don't want to plaster the code with LCOV_EXCL_LINE comments. As first action we should look into function coverage. We may not be able to execute every branch but at least we should be able to call and verify each function. Christian From brett at python.org Mon Jul 29 21:38:54 2013 From: brett at python.org (Brett Cannon) Date: Mon, 29 Jul 2013 15:38:54 -0400 Subject: [Python-Dev] C code coverage report with lcov In-Reply-To: <51F6C2F6.8040704@python.org> References: <51F6C2F6.8040704@python.org> Message-ID: On Mon, Jul 29, 2013 at 3:31 PM, Christian Heimes wrote: > Am 29.07.2013 19:58, schrieb Brett Cannon: > > I took a quick poke around and it seems some things are legitimately not > > being executed, while others are error conditions that we wouldn't > > expect to occur (e.g. memory exhaustion). If we ever decide to get > > serious about code coverage (both C and Python code) we may need to have > > a discussion as a group about our feelings related to pragmas dictating > > when code should be left out of coverage reports. > > Yeah, object allocation and creation checks as well as verification of > operating system API call are reason for lots branch and line misses. > I'm not sure what we can do about that. I don't want to plaster the code > with LCOV_EXCL_LINE comments. > > As first action we should look into function coverage. We may not be > able to execute every branch but at least we should be able to call and > verify each function. > If there's a way to report just function coverage then I think that's a great place to start. -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Mon Jul 29 21:48:33 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 29 Jul 2013 21:48:33 +0200 Subject: [Python-Dev] C code coverage report with lcov References: <51F6C2F6.8040704@python.org> Message-ID: <20130729214833.5c20e9aa@fsol> On Mon, 29 Jul 2013 21:31:02 +0200 Christian Heimes wrote: > Am 29.07.2013 19:58, schrieb Brett Cannon: > > I took a quick poke around and it seems some things are legitimately not > > being executed, while others are error conditions that we wouldn't > > expect to occur (e.g. memory exhaustion). If we ever decide to get > > serious about code coverage (both C and Python code) we may need to have > > a discussion as a group about our feelings related to pragmas dictating > > when code should be left out of coverage reports. > > Yeah, object allocation and creation checks as well as verification of > operating system API call are reason for lots branch and line misses. > I'm not sure what we can do about that. I don't want to plaster the code > with LCOV_EXCL_LINE comments. Ideally, we should run coverage runs on different systems (Unices, Windows...) and merge the results together, so that we know which paths are really uncovered. Regards Antoine. From christian at python.org Mon Jul 29 22:05:36 2013 From: christian at python.org (Christian Heimes) Date: Mon, 29 Jul 2013 22:05:36 +0200 Subject: [Python-Dev] C code coverage report with lcov In-Reply-To: References: <51F6C2F6.8040704@python.org> Message-ID: <51F6CB10.6020204@python.org> Am 29.07.2013 21:38, schrieb Brett Cannon: > If there's a way to report just function coverage then I think > that's a great place to start. lcov's genhtml command doesn't support just function coverage. But I have removed branch coverage. It makes the report a little bit more readable. Christian From victor.stinner at gmail.com Mon Jul 29 23:42:36 2013 From: victor.stinner at gmail.com (Victor Stinner) Date: Mon, 29 Jul 2013 23:42:36 +0200 Subject: [Python-Dev] PEP 446: Open issues/questions In-Reply-To: References: <20130728114341.23ec940f@fsol> Message-ID: 2013/7/28 Charles-Fran?ois Natali : > 2013/7/28 Antoine Pitrou : >>> (C) Should we handle standard streams (0: stdin, 1: stdout, 2: stderr) >>> differently? For example, os.dup2(fd, 0) should make the file >>> descriptor 0 (stdin) inheritable or non-inheritable? On Windows, >>> os.set_inheritable(fd, False) fails (error 87, invalid argument) on >>> standard streams (0, 1, 2) and copies of the standard streams (created >>> by os.dup()). >> >> I have been advocating for that, but I now realize that special-casing >> these three descriptors in a myriad of fd-creating functions isn't very >> attractive. >> (if a standard stream fd has been closed, any fd-creating function can >> re-create that fd: including socket.socket(), etc.) >> So perhaps only the *original* standard streams should be left >> inheritable? I plan to only change functions *creating* (and replacing) new file descriptors. Existing file descriptors (like 0, 1, 2) are unchanged. > Having stdin/stdout/stderr cloexec (e.g. after a dup() to redirect to > a log file, a socket...) will likely break a lot of code, e.g. code > using os.system(), or code calling exec manually (and I'm sure there's > a bunch of it). I propose to add just one special case os.dup2(fd, fd2): if fd2 < 3, fd2 is set to inheritable (otherwise, fd2 is set to non-inheritable). os.dup2() is commonly used to replace stdin, stdout and stderr between fork and exec. The http.server (cgi server) and pty modules use dup2() for example http://hg.python.org/features/pep-446/rev/f8a52518be4c Victor From solipsis at pitrou.net Mon Jul 29 23:49:49 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 29 Jul 2013 23:49:49 +0200 Subject: [Python-Dev] PEP 446: Open issues/questions In-Reply-To: References: <20130728114341.23ec940f@fsol> Message-ID: <20130729234949.3eb52577@fsol> On Mon, 29 Jul 2013 23:42:36 +0200 Victor Stinner wrote: > > >> So perhaps only the *original* standard streams should be left > >> inheritable? > > I plan to only change functions *creating* (and replacing) new file > descriptors. Existing file descriptors (like 0, 1, 2) are unchanged. You can create fds 0, 1 and 2 if they were closed before. > > Having stdin/stdout/stderr cloexec (e.g. after a dup() to redirect to > > a log file, a socket...) will likely break a lot of code, e.g. code > > using os.system(), or code calling exec manually (and I'm sure there's > > a bunch of it). > > I propose to add just one special case os.dup2(fd, fd2): if fd2 < 3, > fd2 is set to inheritable (otherwise, fd2 is set to non-inheritable). > os.dup2() is commonly used to replace stdin, stdout and stderr between > fork and exec. The http.server (cgi server) and pty modules use dup2() > for example Mmh. Doing it only in dup2() sounds like too much special casing for me. I would prefer adding a new explicit parameter: os.dup2(fd, fd2, inheritable=False) (as a bonus, it can be implemented using dup3() under (GNU/)Linux) Regards Antoine. From syedk at pacificloud.com Mon Jul 29 23:57:08 2013 From: syedk at pacificloud.com (syed khalid) Date: Mon, 29 Jul 2013 11:57:08 -1000 Subject: [Python-Dev] importing modules Message-ID: I am attempting to import modules from Shogun to python from a non-standard python directory ie from my /home/xxx directory. is there a way on ubuntu to selectively some modules, scripts, data from one directory and others modules, scripts from another directory. In other words, is there a file(s) that provide pointers to where these modules are located. regards Sy -- -------------- next part -------------- An HTML attachment was scrubbed... URL: From victor.stinner at gmail.com Tue Jul 30 00:34:01 2013 From: victor.stinner at gmail.com (Victor Stinner) Date: Tue, 30 Jul 2013 00:34:01 +0200 Subject: [Python-Dev] PEP 446: Open issues/questions In-Reply-To: <20130728114341.23ec940f@fsol> References: <20130728114341.23ec940f@fsol> Message-ID: 2013/7/28 Antoine Pitrou : >> (B) Should subprocess make the file descriptors of pass_fds >> inheritable? If yes, should it be done before or after the fork? If it >> is done after the fork and before exec, it only affects the child >> process, at least on Linux (the file descriptor is still >> non-inheritable in the parent process). > > If it is done, it should definitely be done after the fork, IMO. > subprocess shouldn't have any long-lasting effects on the process. I modified the subprocess module to make fds of pass_fds inheritable. http://hg.python.org/features/pep-446/rev/75e5d34898aa If we don't do that, it will probably break all applications using pass_fds (and so the backward compatibility). Victor From steve at pearwood.info Tue Jul 30 00:53:13 2013 From: steve at pearwood.info (Steven D'Aprano) Date: Tue, 30 Jul 2013 08:53:13 +1000 Subject: [Python-Dev] importing modules In-Reply-To: References: Message-ID: <51F6F259.106@pearwood.info> Hi Syed, On 30/07/13 07:57, syed khalid wrote: > I am attempting to import modules from Shogun to python from a non-standard > python directory ie from my /home/xxx directory. is there a way on ubuntu > to selectively some modules, scripts, data from one directory and others > modules, scripts from another directory. In other words, is there a file(s) > that provide pointers to where these modules are located. This mailing list is for development OF Python, not development WITH Python. Questions like yours are best asked on the python-list at python.org mailing list, or if you prefer Usenet, comp.lang.python. http://mail.python.org/mailman/listinfo/python-list My (brief) answer to your question, is you probably should set sys.path appropriately. If that's not enough to point you in the right direction to solve the problem, please take any further discussion to python-list rather than ask here. Regards, -- Steven From cf.natali at gmail.com Tue Jul 30 09:09:38 2013 From: cf.natali at gmail.com (=?ISO-8859-1?Q?Charles=2DFran=E7ois_Natali?=) Date: Tue, 30 Jul 2013 09:09:38 +0200 Subject: [Python-Dev] PEP 446: Open issues/questions In-Reply-To: <20130729101130.3b12d0fb@pitrou.net> References: <20130728114341.23ec940f@fsol> <20130729101130.3b12d0fb@pitrou.net> Message-ID: >> Having stdin/stdout/stderr cloexec (e.g. after a dup() to redirect to >> a log file, a socket...) will likely break a lot of code, e.g. code >> using os.system(), or code calling exec manually (and I'm sure there's >> a bunch of it). > > Hmm. os.exec*() could easily make standard streams non-CLOEXEC before > calling the underlying C library function. Things are more annoying for > os.system(), though. > >> Also, it'll be puzzling to have syscalls automatically set the cloexec >> flag. I guess a lot of people doing system programming with Python >> will get bitten, but that's a discussion we already had months ago... > > Perhaps this advocates for a global flag, e.g. > sys.set_default_fd_inheritance(), with False (non-inheritable) being > the default for sanity and security. This looks more and more like PEP 433 :-) And honestly, when I think about it, I think that this whole mess is a solution looking for a problem. If we don't want to inherit file descriptors in child processes, the answer is simple: the subprocess module (this fact is not even mentioned in the PEP). If a user wants to use the execve() syscall directly, then he should be aware of the implications. I don't think that patching half the stdlib and complicating the API of many functions is the right way to do this. The stdlib should be updated to replace the handful of places where exec() is called explicitly by subprocess (the only one I can think on top of my head is http.server.CGIHTTPRequestHandler (issue #16945)), otherwise that's about it. cf > > Regards > > Antoine. > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/cf.natali%40gmail.com From victor.stinner at gmail.com Tue Jul 30 11:24:42 2013 From: victor.stinner at gmail.com (Victor Stinner) Date: Tue, 30 Jul 2013 11:24:42 +0200 Subject: [Python-Dev] PEP 446: Open issues/questions In-Reply-To: References: <20130728114341.23ec940f@fsol> Message-ID: 2013/7/28 Charles-Fran?ois Natali : > Also, it'll be puzzling to have syscalls automatically set the cloexec > flag. I guess a lot of people doing system programming with Python > will get bitten, but that's a discussion we already had months ago... The inheritance of file descriptors (and Windows handles) is discussed since january 2013. Thanks to all exchanges on python-dev, we now know well the perimeter of such changes. Each option has been discussed, and advantages and drawbacks of each option were listed. Going against the POSIX standard (clear inheritable flag when creating a fd) is a drawback, but it's less important than issues fixed by such change (don't "leak" fd or handle to child processes which fixes many other issues). Victor -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Tue Jul 30 11:29:12 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 30 Jul 2013 11:29:12 +0200 Subject: [Python-Dev] PEP 446: Open issues/questions References: <20130728114341.23ec940f@fsol> <20130729101130.3b12d0fb@pitrou.net> Message-ID: <20130730112912.53e108a7@pitrou.net> Le Tue, 30 Jul 2013 09:09:38 +0200, Charles-Fran?ois Natali a ?crit : > > > > Perhaps this advocates for a global flag, e.g. > > sys.set_default_fd_inheritance(), with False (non-inheritable) being > > the default for sanity and security. > > This looks more and more like PEP 433 :-) > > And honestly, when I think about it, I think that this whole mess is a > solution looking for a problem. > If we don't want to inherit file descriptors in child processes, the > answer is simple: the subprocess module (this fact is not even > mentioned in the PEP). This is a good point. Are there any reasons (other than fd inheritance) not to use subprocess? If there are, perhaps we should try to eliminate them by improving subprocess. Regards Antoine. From victor.stinner at gmail.com Tue Jul 30 11:37:56 2013 From: victor.stinner at gmail.com (Victor Stinner) Date: Tue, 30 Jul 2013 11:37:56 +0200 Subject: [Python-Dev] PEP 446: Open issues/questions In-Reply-To: References: <20130728114341.23ec940f@fsol> <20130729101130.3b12d0fb@pitrou.net> Message-ID: Le 30 juil. 2013 09:11, "Charles-Fran?ois Natali" a ?crit : > > Perhaps this advocates for a global flag, e.g. > > sys.set_default_fd_inheritance(), with False (non-inheritable) being > > the default for sanity and security. > > This looks more and more like PEP 433 :-) I don't like the global modifiable default, for the reasons Charles Francois gave some months ago. I did have time yet to rewrite the PEP 446, I'm writing first its implementation. Basically, the new PEP 446 is: - functions creating file descriptors and Windows handle now clear the inheritable flag (disable inheritance by default) - add new functions os.get/set_inheritable() - subprocess makes fds of pass_fds inheritable - (maybe) os.dup2(fd, fd2) makes fd2 inheritable for fd 0, 1, 2 That's all! No more new cloexec parameter to +15 functions and classes, no more global variable (default inheritavle values). Victor -------------- next part -------------- An HTML attachment was scrubbed... URL: From shibturn at gmail.com Tue Jul 30 11:52:38 2013 From: shibturn at gmail.com (Richard Oudkerk) Date: Tue, 30 Jul 2013 10:52:38 +0100 Subject: [Python-Dev] PEP 446: Open issues/questions In-Reply-To: References: <20130728114341.23ec940f@fsol> <20130729101130.3b12d0fb@pitrou.net> Message-ID: On 30/07/2013 8:09am, Charles-Fran?ois Natali wrote: > If we don't want to inherit file descriptors in child processes, the > answer is simple: the subprocess module (this fact is not even > mentioned in the PEP). Note that on Windows subprocess has no equivalent of a passfds argument, and if you redirect the standard streams then you are forced to inherit all inheritable handles. -- Richard From victor.stinner at gmail.com Tue Jul 30 12:52:55 2013 From: victor.stinner at gmail.com (Victor Stinner) Date: Tue, 30 Jul 2013 12:52:55 +0200 Subject: [Python-Dev] PEP 446: Open issues/questions In-Reply-To: References: <20130728114341.23ec940f@fsol> <20130729101130.3b12d0fb@pitrou.net> Message-ID: 2013/7/30 Richard Oudkerk : > Note that on Windows subprocess has no equivalent of a passfds argument, and > if you redirect the standard streams then you are forced to inherit all > inheritable handles. You can redirect standard streams (stdin, stdout, stderr) using the startup info structure: startupinfo.dwFlags |= _winapi.STARTF_USESTDHANDLES startupinfo.hStdInput = p2cread startupinfo.hStdOutput = c2pwrite startupinfo.hStdError = errwrite But I don't know how to add another streams (ex: pipe). I would be nice to have a "pass_handles" on Windows. Victor From shibturn at gmail.com Tue Jul 30 14:12:12 2013 From: shibturn at gmail.com (Richard Oudkerk) Date: Tue, 30 Jul 2013 13:12:12 +0100 Subject: [Python-Dev] PEP 446: Open issues/questions In-Reply-To: References: <20130728114341.23ec940f@fsol> <20130729101130.3b12d0fb@pitrou.net> Message-ID: On 30/07/2013 11:52am, Victor Stinner wrote: > You can redirect standard streams (stdin, stdout, stderr) using the > startup info structure: > > startupinfo.dwFlags |= _winapi.STARTF_USESTDHANDLES > startupinfo.hStdInput = p2cread > startupinfo.hStdOutput = c2pwrite > startupinfo.hStdError = errwrite The documentation for STARTUPINFO says this about STARTF_USESTDHANDLES: If this flag is specified when calling one of the process creation functions, the handles must be inheritable and the function's bInheritHandles parameter must be set to TRUE. So, as I said, if you redirect the streams then you inherit all inheritable handles. -- Richard From martin at v.loewis.de Tue Jul 30 19:02:44 2013 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 30 Jul 2013 19:02:44 +0200 Subject: [Python-Dev] MyOpenID.com no longer supported In-Reply-To: <20130729165628.4CBB02502CD@webabinitio.net> References: <51F62009.7080304@stoneleaf.us> <20130729165628.4CBB02502CD@webabinitio.net> Message-ID: <51F7F1B4.50908@v.loewis.de> Am 29.07.13 18:56, schrieb R. David Murray: > Either Martin needs to clue me in, or I'll have to find time to read > his openid code :) If you want to read the code, this issue can be discovered from schema.py. The openids are stored per user in a single field; multiple openids are simply space-separated. To find all myopenid users, do select id,_username,_address,_openids from _user where _openids like '%myopenid.com/%'; You could filter out those with multiple openids (and not _openids like '% %'). I'd appreciate if you could contact them; if anybody loses access and registered a working email address, the password reset procedure should still work for them. FWIW, 204 out of 13581 users only have a myopenid registered; how many of these also have a working password, I cannot tell. 49 more have a second OpenID registered (which atleast must have worked when they registered it - but it could be an alias for the myopenid one). Regards, Martin From martin at v.loewis.de Tue Jul 30 19:24:49 2013 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 30 Jul 2013 19:24:49 +0200 Subject: [Python-Dev] Adding Python scripts to PATHEXT on Windows In-Reply-To: References: <51EEAB2E.9030804@v.loewis.de> Message-ID: <51F7F6E1.4040002@v.loewis.de> Am 28.07.13 00:36, schrieb Paul Moore: > On 27 July 2013 21:14, Steve Dower > wrote: > > Any chance of this being made optional when installing? It provides > no benefit for people who prefer to associate scripts with an editor > and may be a source of confusion/complaints. > > > Personally, I don't know how to do this so someone else would have to - It seems that this was settled as fine as-is; if you ever wanted to allow this to be specifically enabled, you'd have to do three things 1. create a new "feature" in the feature tree. Your patch currently uses the "Extensions" feature (child of the default feature); you could make your feature a subfeature of Extensions. 2. create a new "component". This should work similar to REGISTRY.path. 3. Add a "FeatureComponents" entry saying that the "REGISTRY.pathext" component (say) belongs to the "PathExt" feature (say). Instead of hooking the environment variable to the REGISTRY.def component, it should then go to the REGISTRY.pathext component. The nice thing about this feature tree is that it allows convenient user configuration yet powerful scripted installation, see http://msdn.microsoft.com/en-us/library/windows/desktop/aa368585(v=vs.85).aspx For example, you could set the msidbFeatureAttributesFollowParent flag, which would make it be selected/unselected if the parent feature is, yet still allow it to be independent of the parent for people who really want to. Regards, Martin From rdmurray at bitdance.com Tue Jul 30 19:31:01 2013 From: rdmurray at bitdance.com (R. David Murray) Date: Tue, 30 Jul 2013 13:31:01 -0400 Subject: [Python-Dev] =?utf-8?q?=5BPython-checkins=5D_cpython_=282=2E7=29?= =?utf-8?q?=3A_Issue_=2318441=3A_Make_test=2Esupport=2Erequires=28?= =?utf-8?q?=27gui=27=29_skip_when_it_should=2E?= In-Reply-To: <3c2jY31SQ0z7LjW@mail.python.org> References: <3c2jY31SQ0z7LjW@mail.python.org> Message-ID: <20130730173102.1195025007D@webabinitio.net> On Sun, 28 Jul 2013 01:09:43 +0200, terry.reedy wrote: > http://hg.python.org/cpython/rev/dd9941f5fcda > changeset: 84870:dd9941f5fcda > branch: 2.7 > parent: 84865:c0788ee86a65 > user: Terry Jan Reedy > date: Sun Jul 21 20:13:24 2013 -0400 > summary: > Issue #18441: Make test.support.requires('gui') skip when it should. > (Consolidating this check and various checks in tkinter files and moving them > to test.support and test.regrtest will be another issue.) > > files: > Lib/idlelib/idle_test/test_text.py | 5 +--- > Lib/test/test_idle.py | 20 ++++++++++++++--- > 2 files changed, 17 insertions(+), 8 deletions(-) > > > diff --git a/Lib/idlelib/idle_test/test_text.py b/Lib/idlelib/idle_test/test_text.py > --- a/Lib/idlelib/idle_test/test_text.py > +++ b/Lib/idlelib/idle_test/test_text.py > @@ -216,10 +216,7 @@ > requires('gui') > from Tkinter import Tk, Text > cls.Text = Text > - try: > - cls.root = Tk() > - except TclError as msg: > - raise unittest.SkipTest('TclError: %s' % msg) > + cls.root = Tk() > > @classmethod > def tearDownClass(cls): > diff --git a/Lib/test/test_idle.py b/Lib/test/test_idle.py > --- a/Lib/test/test_idle.py > +++ b/Lib/test/test_idle.py > @@ -1,9 +1,21 @@ > -# Skip test if _tkinter or _thread wasn't built or idlelib was deleted. > -from test.test_support import import_module > -import_module('Tkinter') > -import_module('threading') # imported by PyShell, imports _thread > +# Skip test if _thread or _tkinter wasn't built or idlelib was deleted. > +from test.test_support import import_module, use_resources > +import_module('threading') # imported by idlelib.PyShell, imports _thread > +tk = import_module('Tkinter') > idletest = import_module('idlelib.idle_test') > > +# If buildbot improperly sets gui resource (#18365, #18441), remove it > +# so requires('gui') tests are skipped while non-gui tests still run. > +if use_resources and 'gui' in use_resources: > + try: > + root = tk.Tk() > + root.destroy() > + except TclError: > + while True: > + use_resources.delete('gui') > + if 'gui' not in use_resources: > + break I believe that this logic is incorrect. If regrtest is run with "-u gui", given this code the skip message will be "requires gui resource"...but the caller specified the gui resource, which will make the skip message completely confusing. Instead, if it is true that 'gui' always means 'tk must work', then the _is_gui_available function should do the Tk() check. Currently as far as I can see it is indeed the case that requires('gui') always means tk must work. So, I believe that the correct fix is to move check_tk_availability to test.support, and call it from _is_gui_available. If we ever add another gui toolkit, we can deal with moving the tk check out into a separate 'tk' resource at that time. > + > # Without test_main present, regrtest.runtest_inner (line1219) calls > # unittest.TestLoader().loadTestsFromModule(this_module) which calls > # load_tests() if it finds it. (Unittest.main does the same.) > > -- > Repository URL: http://hg.python.org/cpython > _______________________________________________ > Python-checkins mailing list > Python-checkins at python.org > http://mail.python.org/mailman/listinfo/python-checkins From solipsis at pitrou.net Tue Jul 30 20:42:00 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 30 Jul 2013 20:42:00 +0200 Subject: [Python-Dev] PEP 442 aftermath: module globals at shutdown Message-ID: <20130730204200.6d77df8b@fsol> Hello, PEP 442 has now been committed in time for testing in Alpha 1. This paves the way for the removal of another well-known annoyance: the behaviour of module globals at shutdown. Now that reference cycles aren't a barrier to object finalization anymore, we shouldn't need to set module globals to None before trying to reclaim modules. (and then, we don't need to cache global functions for use in finalizers, either) I have a patch to suppress the hack in http://bugs.python.org/issue18214 Once I get to add some tests, I would like to commit it soon too! Regards Antoine. From p.f.moore at gmail.com Tue Jul 30 20:56:46 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 30 Jul 2013 19:56:46 +0100 Subject: [Python-Dev] Adding Python scripts to PATHEXT on Windows In-Reply-To: <51F7F6E1.4040002@v.loewis.de> References: <51EEAB2E.9030804@v.loewis.de> <51F7F6E1.4040002@v.loewis.de> Message-ID: On 30 July 2013 18:24, "Martin v. L?wis" wrote: > > Personally, I don't know how to do this so someone else would have to - > > It seems that this was settled as fine as-is; if you ever wanted to > allow this to be specifically enabled, you'd have to do three things > Thanks, Martin - MSI is a mystery to me, so this is very useful. Saved for future reference. Paul -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Tue Jul 30 21:58:58 2013 From: guido at python.org (Guido van Rossum) Date: Tue, 30 Jul 2013 12:58:58 -0700 Subject: [Python-Dev] PEP 442 aftermath: module globals at shutdown In-Reply-To: <20130730204200.6d77df8b@fsol> References: <20130730204200.6d77df8b@fsol> Message-ID: I'm very excited to see this happening! It's been a constant pain and one of the things I've always regretted. Thanks Antoine! --Guido On Tue, Jul 30, 2013 at 11:42 AM, Antoine Pitrou wrote: > > Hello, > > PEP 442 has now been committed in time for testing in Alpha 1. > > This paves the way for the removal of another well-known annoyance: the > behaviour of module globals at shutdown. Now that reference cycles > aren't a barrier to object finalization anymore, we shouldn't need > to set module globals to None before trying to reclaim modules. > (and then, we don't need to cache global functions for use in > finalizers, either) > > I have a patch to suppress the hack in > http://bugs.python.org/issue18214 > Once I get to add some tests, I would like to commit it soon too! > > Regards > > Antoine. -- --Guido van Rossum (python.org/~guido) From pjenvey at underboss.org Tue Jul 30 22:39:02 2013 From: pjenvey at underboss.org (Philip Jenvey) Date: Tue, 30 Jul 2013 13:39:02 -0700 Subject: [Python-Dev] PyPy3 2.1 beta 1 released Message-ID: ================ PyPy3 2.1 beta 1 ================ We're pleased to announce the first beta of the upcoming 2.1 release of PyPy3. This is the first release of PyPy which targets Python 3 (3.2.3) compatibility. We would like to thank all of the people who donated_ to the `py3k proposal`_ for supporting the work that went into this and future releases. You can download the PyPy3 2.1 beta 1 release here: http://pypy.org/download.html#pypy3-2-1-beta-1 Highlights ========== * The first release of PyPy3: support for Python 3, targetting CPython 3.2.3! - There are some `known issues`_ including performance regressions (issues `#1540`_ & `#1541`_) slated to be resolved before the final release. What is PyPy? ============== PyPy is a very compliant Python interpreter, almost a drop-in replacement for CPython 2.7.3 or 3.2.3. It's fast due to its integrated tracing JIT compiler. This release supports x86 machines running Linux 32/64, Mac OS X 64 or Windows 32. Also this release supports ARM machines running Linux 32bit - anything with ``ARMv6`` (like the Raspberry Pi) or ``ARMv7`` (like Beagleboard, Chromebook, Cubieboard, etc.) that supports ``VFPv3`` should work. Cheers, the PyPy team From tseaver at palladion.com Tue Jul 30 23:18:44 2013 From: tseaver at palladion.com (Tres Seaver) Date: Tue, 30 Jul 2013 17:18:44 -0400 Subject: [Python-Dev] PyPy3 2.1 beta 1 released In-Reply-To: References: Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 07/30/2013 04:39 PM, Philip Jenvey wrote: > ================ PyPy3 2.1 beta 1 ================ > > We're pleased to announce the first beta of the upcoming 2.1 release > of PyPy3. This is the first release of PyPy which targets Python 3 > (3.2.3) compatibility. > > We would like to thank all of the people who donated_ to the `py3k > proposal`_ for supporting the work that went into this and future > releases. > > You can download the PyPy3 2.1 beta 1 release here: > > http://pypy.org/download.html#pypy3-2-1-beta-1 > > Highlights ========== > > * The first release of PyPy3: support for Python 3, targetting CPython > 3.2.3! > > - There are some `known issues`_ including performance regressions > (issues `#1540`_ & `#1541`_) slated to be resolved before the final > release. > > What is PyPy? ============== > > PyPy is a very compliant Python interpreter, almost a drop-in > replacement for CPython 2.7.3 or 3.2.3. It's fast due to its > integrated tracing JIT compiler. > > This release supports x86 machines running Linux 32/64, Mac OS X 64 or > Windows 32. Also this release supports ARM machines running Linux > 32bit - anything with ``ARMv6`` (like the Raspberry Pi) or ``ARMv7`` > (like Beagleboard, Chromebook, Cubieboard, etc.) that supports > ``VFPv3`` should work. Wow -- congratulations! Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with undefined - http://www.enigmail.net/ iEYEARECAAYFAlH4LbQACgkQ+gerLs4ltQ655ACfXIlTNAldCJdCSjY0Os8xIhLu vl4AoIi8eXqB6Ef/RO2W46iIbQvlX41j =R2vL -----END PGP SIGNATURE----- From solipsis at pitrou.net Tue Jul 30 23:32:23 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 30 Jul 2013 23:32:23 +0200 Subject: [Python-Dev] PEP 442 aftermath: module globals at shutdown In-Reply-To: References: <20130730204200.6d77df8b@fsol> Message-ID: <20130730233223.1e472996@fsol> On Tue, 30 Jul 2013 12:58:58 -0700 Guido van Rossum wrote: > I'm very excited to see this happening! It's been a constant pain and > one of the things I've always regretted. Thanks Antoine! Note this is currently imperfect. I've identified two reasons why a pure Python module could stay alive even after being removed from sys.modules: - it is held alive by a C extension: the main example is the locale module, which is held alive by _io and in turn keeps alive other Python modules (such as collections or re). But there is also the readline module, whose completer can keep alive other stuff (this is aggravated by the fact the readline module currently doesn't have a proper dealloc routine). - it is held alive through builtins: the site module patches builtins with additional objects, which themselves keep references to the site module's globals, and with it other modules Regards Antoine. > > --Guido > > On Tue, Jul 30, 2013 at 11:42 AM, Antoine Pitrou wrote: > > > > Hello, > > > > PEP 442 has now been committed in time for testing in Alpha 1. > > > > This paves the way for the removal of another well-known annoyance: the > > behaviour of module globals at shutdown. Now that reference cycles > > aren't a barrier to object finalization anymore, we shouldn't need > > to set module globals to None before trying to reclaim modules. > > (and then, we don't need to cache global functions for use in > > finalizers, either) > > > > I have a patch to suppress the hack in > > http://bugs.python.org/issue18214 > > Once I get to add some tests, I would like to commit it soon too! > > > > Regards > > > > Antoine. > From christian at python.org Wed Jul 31 02:20:52 2013 From: christian at python.org (Christian Heimes) Date: Wed, 31 Jul 2013 02:20:52 +0200 Subject: [Python-Dev] C code coverage report with lcov In-Reply-To: References: Message-ID: Am 29.07.2013 19:15, schrieb Christian Heimes: > Hi, > > I have done some experiments with GCC's gcov and lcov to get the C code > coverage of our unit test suite. You may find today's report at > > http://tiran.bitbucket.org/python-lcov/ > > I'm working on a patch for our Makefile to include all steps in one > simple make tag. http://bugs.python.org/issue18481 Have fun with make coverage-report as documented in the devguide http://docs.python.org/devguide/coverage.html#measuring-coverage-of-c-code-with-gcov-and-lcov Christian From christian at python.org Wed Jul 31 02:28:39 2013 From: christian at python.org (Christian Heimes) Date: Wed, 31 Jul 2013 02:28:39 +0200 Subject: [Python-Dev] C code coverage report with lcov In-Reply-To: <20130729214833.5c20e9aa@fsol> References: <51F6C2F6.8040704@python.org> <20130729214833.5c20e9aa@fsol> Message-ID: <51F85A37.3040903@python.org> Am 29.07.2013 21:48, schrieb Antoine Pitrou: > Ideally, we should run coverage runs on different systems (Unices, > Windows...) and merge the results together, so that we know which paths > are really uncovered. I don't that is easily possible. The coverage report depends on GCC and its gcov extension -- so much for Windows. I also don't know if gcov supports cross-profiling on varying platforms and operating systems. By the way gcov understands preprocessor output. It doesn't report lines as uncovered when the lines or functions are #ifdef-ed out. Christian From tjreedy at udel.edu Wed Jul 31 02:52:27 2013 From: tjreedy at udel.edu (Terry Reedy) Date: Tue, 30 Jul 2013 20:52:27 -0400 Subject: [Python-Dev] cpython (2.7): Issue #18441: Make test.support.requires('gui') skip when it should. In-Reply-To: <20130730173102.1195025007D@webabinitio.net> References: <3c2jY31SQ0z7LjW@mail.python.org> <20130730173102.1195025007D@webabinitio.net> Message-ID: <51F85FCB.8090008@udel.edu> On 7/30/2013 1:31 PM, R. David Murray wrote: > On Sun, 28 Jul 2013 01:09:43 +0200, terry.reedy wrote: > >> Issue #18441: Make test.support.requires('gui') skip when it should. >> (Consolidating this check and various checks in tkinter files and moving them >> to test.support and test.regrtest will be another issue.) >> +# Skip test if _thread or _tkinter wasn't built or idlelib was deleted. >> +from test.test_support import import_module, use_resources >> +import_module('threading') # imported by idlelib.PyShell, imports _thread >> +tk = import_module('Tkinter') >> idletest = import_module('idlelib.idle_test') >> >> +# If buildbot improperly sets gui resource (#18365, #18441), remove it >> +# so requires('gui') tests are skipped while non-gui tests still run. >> +if use_resources and 'gui' in use_resources: >> + try: >> + root = tk.Tk() >> + root.destroy() >> + except TclError: >> + while True: >> + use_resources.delete('gui') >> + if 'gui' not in use_resources: >> + break > > I believe that this logic is incorrect. It works for the intended purpose. > If regrtest is run with "-u gui", given this code the skip message will > be "requires gui resource" but the caller specified the gui resource, > which will make the skip message completely confusing. The message is accurate; string 'gui' does not create or enable the required physical graphics subsystem. Anyway, the message is not in any of the current buildbot reports I looked at. This patch has no effect on normal machines with graphics and nor on most of the buildbots (at least the stable one). On the 3 stable buildbots it is written for, the message will only appear if test_idle fails (in one of the text tests that do run) and is rerun or displayed in verbose mode. I doubt that such a rare occurrence will seriously confuse any of the few developers who will read the verbose idle_test output. > Instead, if it is true that 'gui' always means 'tk must work', then the > _is_gui_available function should do the Tk() check. Currently as far > as I can see it is indeed the case that requires('gui') always means tk > must work. So, I believe that the correct fix is to move > check_tk_availability to test.support, and call it from > _is_gui_available. Ned and I agreed in the issue discussion that gui checks (also in tkinter files) should be consolidated into test.support. See http://bugs.python.org/issue18604 The check in this patch will be modified or simply removed once that is done. But I need it here now to continue pushing new idle tests. A consolidated gui check could report to the user something like "'gui' is being ignored because graphics are not usable". > If we ever add another gui toolkit, we can deal with > moving the tk check out into a separate 'tk' resource at that time. The above check is intended mostly as a gui check, not a tk check, because it is only performed after successfully importing tkinter, which imports _tkinter, which initializes tcl/tk. -- Terry Jan Reedy From greg.ewing at canterbury.ac.nz Wed Jul 31 03:30:58 2013 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Wed, 31 Jul 2013 13:30:58 +1200 Subject: [Python-Dev] PEP 442 aftermath: module globals at shutdown In-Reply-To: <20130730233223.1e472996@fsol> References: <20130730204200.6d77df8b@fsol> <20130730233223.1e472996@fsol> Message-ID: <51F868D2.7040601@canterbury.ac.nz> Antoine Pitrou wrote: > - it is held alive through builtins: the site module patches builtins > with additional objects, which themselves keep references to the site > module's globals, The module probably *should* stay alive in that case, since it's still accessible via those patched builtins. -- Greg From rdmurray at bitdance.com Wed Jul 31 04:08:31 2013 From: rdmurray at bitdance.com (R. David Murray) Date: Tue, 30 Jul 2013 22:08:31 -0400 Subject: [Python-Dev] cpython (2.7): Issue #18441: Make test.support.requires('gui') skip when it should. In-Reply-To: <51F85FCB.8090008@udel.edu> References: <3c2jY31SQ0z7LjW@mail.python.org> <20130730173102.1195025007D@webabinitio.net> <51F85FCB.8090008@udel.edu> Message-ID: <20130731020832.07CF62500B9@webabinitio.net> On Tue, 30 Jul 2013 20:52:27 -0400, Terry Reedy wrote: > On 7/30/2013 1:31 PM, R. David Murray wrote: > > On Sun, 28 Jul 2013 01:09:43 +0200, terry.reedy wrote: > > I believe that this logic is incorrect. > > It works for the intended purpose. > > > If regrtest is run with "-u gui", given this code the skip message will > > be "requires gui resource" but the caller specified the gui resource, > > which will make the skip message completely confusing. > > The message is accurate; string 'gui' does not create or enable the > required physical graphics subsystem. Anyway, the message is not in any > of the current buildbot reports I looked at. This patch has no effect on > normal machines with graphics and nor on most of the buildbots (at least > the stable one). On the 3 stable buildbots it is written for, the > message will only appear if test_idle fails (in one of the text tests > that do run) and is rerun or displayed in verbose mode. I doubt that > such a rare occurrence will seriously confuse any of the few developers > who will read the verbose idle_test output. OK, I see what you are saying, but I really do think the skip message should differentiate between the flag not being specified and the physical resource not existing. > > Instead, if it is true that 'gui' always means 'tk must work', then the > > _is_gui_available function should do the Tk() check. Currently as far > > as I can see it is indeed the case that requires('gui') always means tk > > must work. So, I believe that the correct fix is to move > > check_tk_availability to test.support, and call it from > > _is_gui_available. > > Ned and I agreed in the issue discussion that gui checks (also in > tkinter files) should be consolidated into test.support. See > http://bugs.python.org/issue18604 > The check in this patch will be modified or simply removed once that is > done. But I need it here now to continue pushing new idle tests. > > A consolidated gui check could report to the user something like "'gui' > is being ignored because graphics are not usable". As long as this is the end result, I'm fine with it. --David From ronaldoussoren at mac.com Wed Jul 31 08:15:50 2013 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Wed, 31 Jul 2013 08:15:50 +0200 Subject: [Python-Dev] to rename or not... Message-ID: <28A29D72-0EFC-4B77-98B9-B416E522549B@mac.com> Hi, The relevant issue for this question is Plistlib contains object serialization logic that is in spirit simular to json and pickle, but has a different output format and different limitations. The functions in the module don't reflect that though, they are "readPlist" and "writePlist" instead of "load" and "dump". While working on the issue I noticed something uglier than that: plistlib in py3k inherits a design decision that was necessary in py2 and feels decidedly odd in py3k. It represents binary data objects of type plistlib.Data instead of bytes. Those objects have an attribute with the actual data. The distinction was necessary in Python 2 to make it possible to keep binary data and strings apart, but is no longer necessary in Python 3. Because of this I'd like to introduce a new API in plistlib that fixes both problems. In particular: * Add 'load', 'loads', 'dump' and 'dumps', those use "bytes" for binary data by default * Keep and deprecate "readPlist", "writePlist" and the their string equivalents, those still use Data objects (and call the new API to do the actual work). I'd like some feedback on this change. On the one hand the new APIs make it possible to clean up the API of plistlib, on the other hand this is a big API change. Ronald P.S. The issue itself is about adding support for binary plist files, I got a bit carried away while testing and refactoring the original patch :-( From ronaldoussoren at mac.com Wed Jul 31 08:17:55 2013 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Wed, 31 Jul 2013 08:17:55 +0200 Subject: [Python-Dev] PEP 447: add type.__locallookup__ In-Reply-To: <75030FAC-6918-4E94-95DA-67A88D53E6F5@mac.com> References: <75030FAC-6918-4E94-95DA-67A88D53E6F5@mac.com> Message-ID: <7D6D4488-2855-41F8-A9F2-1FF23B9E3A56@mac.com> On 29 Jul, 2013, at 14:49, Ronald Oussoren wrote: > Hi, > > This PEP proposed to add a __locallookup__ slot to type objects, > which is used by _PyType_Lookup and super_getattro instead of peeking > in the tp_dict of classes. The PEP text explains why this is needed. > > Differences with the previous version: > > * Better explanation of why this is a useful addition > > * type.__locallookup__ is no longer optional. > > * I've added benchmarking results using pybench. > (using the patch attached to issue 18181) > > Ronald And something I forgot to ask: is anyone willing to be the BDFL-Delegate for PEP 447? Ronald From solipsis at pitrou.net Wed Jul 31 08:21:50 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 31 Jul 2013 08:21:50 +0200 Subject: [Python-Dev] PEP 442 aftermath: module globals at shutdown References: <20130730204200.6d77df8b@fsol> <20130730233223.1e472996@fsol> <51F868D2.7040601@canterbury.ac.nz> Message-ID: <20130731082150.293a5a74@fsol> On Wed, 31 Jul 2013 13:30:58 +1200 Greg Ewing wrote: > Antoine Pitrou wrote: > > - it is held alive through builtins: the site module patches builtins > > with additional objects, which themselves keep references to the site > > module's globals, > > The module probably *should* stay alive in that case, since > it's still accessible via those patched builtins. Of course. I'm not gonna kill reachable modules like a cowboy on the loose :-) The problem is that its global objects will get finalized very late. Anyway, the solution is to unpatch builtins through an atexit handler in the site module. (my patch still has a last step where surviving modules get their globals wiped, though) Regards Antoine. From solipsis at pitrou.net Wed Jul 31 10:09:46 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 31 Jul 2013 10:09:46 +0200 Subject: [Python-Dev] to rename or not... References: <28A29D72-0EFC-4B77-98B9-B416E522549B@mac.com> Message-ID: <20130731100946.752a3c43@pitrou.net> Le Wed, 31 Jul 2013 08:15:50 +0200, Ronald Oussoren a ?crit : > > Because of this I'd like to introduce a new API in plistlib that > fixes both problems. In particular: > > * Add 'load', 'loads', 'dump' and 'dumps', those use "bytes" for > binary data by default > > * Keep and deprecate "readPlist", "writePlist" and the their string > equivalents, those still use Data objects (and call the new API to do > the actual work). > > I'd like some feedback on this change. On the one hand the new APIs > make it possible to clean up the API of plistlib, on the other hand > this is a big API change. +1 on the principle. I can't say more, not being a Mac user. Regards Antoine. From greg.ewing at canterbury.ac.nz Wed Jul 31 10:39:10 2013 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Wed, 31 Jul 2013 20:39:10 +1200 Subject: [Python-Dev] to rename or not... In-Reply-To: <28A29D72-0EFC-4B77-98B9-B416E522549B@mac.com> References: <28A29D72-0EFC-4B77-98B9-B416E522549B@mac.com> Message-ID: <51F8CD2E.4010703@canterbury.ac.nz> Ronald Oussoren wrote: > * Add 'load', 'loads', 'dump' and 'dumps', those use "bytes" for binary data > by default > > * Keep and deprecate "readPlist", "writePlist" and the their string > equivalents, those still use Data objects +1, makes sense to me. -- Greg From rdmurray at bitdance.com Wed Jul 31 12:52:25 2013 From: rdmurray at bitdance.com (R. David Murray) Date: Wed, 31 Jul 2013 06:52:25 -0400 Subject: [Python-Dev] to rename or not... In-Reply-To: <51F8CD2E.4010703@canterbury.ac.nz> References: <28A29D72-0EFC-4B77-98B9-B416E522549B@mac.com> <51F8CD2E.4010703@canterbury.ac.nz> Message-ID: <20130731105226.E842025007D@webabinitio.net> On Wed, 31 Jul 2013 20:39:10 +1200, Greg Ewing wrote: > Ronald Oussoren wrote: > > * Add 'load', 'loads', 'dump' and 'dumps', those use "bytes" for binary data > > by default > > > > * Keep and deprecate "readPlist", "writePlist" and the their string > > equivalents, those still use Data objects > > +1, makes sense to me. Yes, +1. Presumably the plistlib module was "ported" to py3k just enough to make it pass its tests, without reconsidering the design (and API) the way you have now done. So I think it is appropriate to do so. But, then, I would, considering I'm doing the same thing to the email module for much the same reason :). --David From ronaldoussoren at mac.com Wed Jul 31 13:12:42 2013 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Wed, 31 Jul 2013 13:12:42 +0200 Subject: [Python-Dev] to rename or not... In-Reply-To: <20130731100946.752a3c43@pitrou.net> References: <28A29D72-0EFC-4B77-98B9-B416E522549B@mac.com> <20130731100946.752a3c43@pitrou.net> Message-ID: <9F88D258-8B93-40DA-95F6-2BD7909E2CF0@mac.com> On 31 Jul, 2013, at 10:09, Antoine Pitrou wrote: > Le Wed, 31 Jul 2013 08:15:50 +0200, > Ronald Oussoren a ?crit : >> >> Because of this I'd like to introduce a new API in plistlib that >> fixes both problems. In particular: >> >> * Add 'load', 'loads', 'dump' and 'dumps', those use "bytes" for >> binary data by default >> >> * Keep and deprecate "readPlist", "writePlist" and the their string >> equivalents, those still use Data objects (and call the new API to do >> the actual work). >> >> I'd like some feedback on this change. On the one hand the new APIs >> make it possible to clean up the API of plistlib, on the other hand >> this is a big API change. > > +1 on the principle. I can't say more, not being a Mac user. plistlib is the portable part of the stdlib :-) But anyway, I've got enough positive feedback that I'll introduce the new API, although it probably won't make the first alpha release. Ronald From brett at python.org Wed Jul 31 15:37:04 2013 From: brett at python.org (Brett Cannon) Date: Wed, 31 Jul 2013 09:37:04 -0400 Subject: [Python-Dev] C code coverage report with lcov In-Reply-To: <51F85A37.3040903@python.org> References: <51F6C2F6.8040704@python.org> <20130729214833.5c20e9aa@fsol> <51F85A37.3040903@python.org> Message-ID: On Tue, Jul 30, 2013 at 8:28 PM, Christian Heimes wrote: > Am 29.07.2013 21:48, schrieb Antoine Pitrou: > > Ideally, we should run coverage runs on different systems (Unices, > > Windows...) and merge the results together, so that we know which paths > > are really uncovered. > > I don't that is easily possible. The coverage report depends on GCC and > its gcov extension -- so much for Windows. I also don't know if gcov > supports cross-profiling on varying platforms and operating systems. > > By the way gcov understands preprocessor output. It doesn't report lines > as uncovered when the lines or functions are #ifdef-ed out. > In case are curious to try and make this work using clang, I found the flags to use at http://clang-developers.42468.n3.nabble.com/Code-coverage-on-clang-td4033066.html and how to make it work with lcov. I also discovered clang itself uses https://github.com/ddunbar/zcov instead of lcov. This is all untested, but I didn't want to forget the links and in case someone has clang installed with compiler-rt and wants to see how it works. -------------- next part -------------- An HTML attachment was scrubbed... URL: