From solipsis at pitrou.net Fri Feb 1 00:50:01 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 1 Feb 2013 00:50:01 +0100 Subject: [Python-Dev] cpython (2.7): - Issue #17086: Backport the patches from the 3.3 branch to cross-build References: <3YxxXq44T7zQ28@mail.python.org> Message-ID: <20130201005001.5f73e15b@pitrou.net> On Thu, 31 Jan 2013 23:52:27 +0100 (CET) matthias.klose wrote: > http://hg.python.org/cpython/rev/8ee6d96a1019 > changeset: 81859:8ee6d96a1019 > branch: 2.7 > parent: 81855:df9f8feb7444 > user: doko at python.org > date: Thu Jan 31 23:52:03 2013 +0100 > summary: > - Issue #17086: Backport the patches from the 3.3 branch to cross-build > the package. You aren't supposed to add new features to bugfix branches. Did you have a specific reason to do this? Regards Antoine. From victor.stinner at gmail.com Fri Feb 1 01:07:16 2013 From: victor.stinner at gmail.com (Victor Stinner) Date: Fri, 1 Feb 2013 01:07:16 +0100 Subject: [Python-Dev] Release or not release the GIL Message-ID: Hi, While working on the implementation of the PEP 433, I saw different places where Python asks for operating resources without releasing the GIL. In my implementation, I released the GIL in the following places. Is it correct? - os.dup() - os.dup2() - os.pipe() - socket.socketpair() os.listdir() releases the GIL to call dup(), whereas os.dup() doesn't. Victor From ncoghlan at gmail.com Fri Feb 1 01:42:05 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 1 Feb 2013 10:42:05 +1000 Subject: [Python-Dev] importlib.find_loader In-Reply-To: References: Message-ID: Yep, looks like a bug in the bootstrapping, failing to set __loader__ properly. However, I also think the current handling of the "no __loader__ attribute" case is a separate bug - since we generally aim to tolerate non-modules in sys.modules (albeit somewhat grudgingly), importlib should probably be using "getattr(mod, '__loader__', None)" rather than assuming the attribute will always be present. Cheers, Nick. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Fri Feb 1 02:00:24 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 1 Feb 2013 11:00:24 +1000 Subject: [Python-Dev] cpython (2.7): - Issue #17086: Backport the patches from the 3.3 branch to cross-build In-Reply-To: <20130201005001.5f73e15b@pitrou.net> References: <3YxxXq44T7zQ28@mail.python.org> <20130201005001.5f73e15b@pitrou.net> Message-ID: On Fri, Feb 1, 2013 at 9:50 AM, Antoine Pitrou wrote: > On Thu, 31 Jan 2013 23:52:27 +0100 (CET) > matthias.klose wrote: >> http://hg.python.org/cpython/rev/8ee6d96a1019 >> changeset: 81859:8ee6d96a1019 >> branch: 2.7 >> parent: 81855:df9f8feb7444 >> user: doko at python.org >> date: Thu Jan 31 23:52:03 2013 +0100 >> summary: >> - Issue #17086: Backport the patches from the 3.3 branch to cross-build >> the package. > > You aren't supposed to add new features to bugfix branches. Did you > have a specific reason to do this? One of the reasons for the long maintenance period on 2.7 is to keep it building as the underlying platforms change. With the rise of ARM systems, being able to reliably cross-build Python 2.7 for ARM from an x86_64 system is fairly important. That said, as a procedural point for build related new features in 2.7, I agree they should be proposed, with an explicit rationale, on python-dev before proceeding with the commit. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From guido at python.org Fri Feb 1 02:44:35 2013 From: guido at python.org (Guido van Rossum) Date: Thu, 31 Jan 2013 17:44:35 -0800 Subject: [Python-Dev] Release or not release the GIL In-Reply-To: References: Message-ID: I'm guessing those system calls are just implemented by the kernel and cannot block for I/O, so it was deemed unnecessary to release the GIL around them. I don't mind changing that though, you can never know what happens when you make a system call. On Thu, Jan 31, 2013 at 4:07 PM, Victor Stinner wrote: > Hi, > > While working on the implementation of the PEP 433, I saw different > places where Python asks for operating resources without releasing the > GIL. > > In my implementation, I released the GIL in the following places. Is it correct? > > - os.dup() > - os.dup2() > - os.pipe() > - socket.socketpair() > > os.listdir() releases the GIL to call dup(), whereas os.dup() doesn't. > > Victor > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org -- --Guido van Rossum (python.org/~guido) From ncoghlan at gmail.com Fri Feb 1 05:15:45 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 1 Feb 2013 14:15:45 +1000 Subject: [Python-Dev] [Python-checkins] cpython (2.7): #16128: clarify that instances of user-defined classes compare equal with In-Reply-To: <3Yy3VL2XyVzRKB@mail.python.org> References: <3Yy3VL2XyVzRKB@mail.python.org> Message-ID: On 1 Feb 2013 13:22, "ezio.melotti" wrote: > > http://hg.python.org/cpython/rev/79a021beaf58 > changeset: 81861:79a021beaf58 > branch: 2.7 > parent: 81859:8ee6d96a1019 > user: Ezio Melotti > date: Fri Feb 01 05:18:44 2013 +0200 > summary: > #16128: clarify that instances of user-defined classes compare equal with themselves. > > files: > Doc/glossary.rst | 3 ++- > 1 files changed, 2 insertions(+), 1 deletions(-) > > > diff --git a/Doc/glossary.rst b/Doc/glossary.rst > --- a/Doc/glossary.rst > +++ b/Doc/glossary.rst > @@ -330,7 +330,8 @@ > All of Python's immutable built-in objects are hashable, while no mutable > containers (such as lists or dictionaries) are. Objects which are > instances of user-defined classes are hashable by default; they all > - compare unequal, and their hash value is their :func:`id`. > + compare unequal (except with themselves), and their hash value is their > + :func:`id`. The hash(x) == id(x) behaviour is a CPython implementation detail. It shouldn't be mentioned here. > > IDLE > An Integrated Development Environment for Python. IDLE is a basic editor > > -- > Repository URL: http://hg.python.org/cpython > > _______________________________________________ > Python-checkins mailing list > Python-checkins at python.org > http://mail.python.org/mailman/listinfo/python-checkins > -------------- next part -------------- An HTML attachment was scrubbed... URL: From theller at ctypes.org Fri Feb 1 07:56:46 2013 From: theller at ctypes.org (Thomas Heller) Date: Fri, 01 Feb 2013 07:56:46 +0100 Subject: [Python-Dev] importlib.find_loader In-Reply-To: References: Message-ID: Am 01.02.2013 01:42, schrieb Nick Coghlan: > Yep, looks like a bug in the bootstrapping, failing to set __loader__ > properly. It also has the effect that reload does not work: Type "help", "copyright", "credits" or "license" for more information. >>> import imp >>> import math >>> imp.reload(math) >>> import signal >>> imp.reload(signal) Traceback (most recent call last): File "", line 1, in File "C:\Python33-64\lib\imp.py", line 252, in reload return module.__loader__.load_module(name) AttributeError: 'module' object has no attribute '__loader__' >>> > However, I also think the current handling of the "no __loader__ > attribute" case is a separate bug - since we generally aim to tolerate > non-modules in sys.modules (albeit somewhat grudgingly), importlib > should probably be using "getattr(mod, '__loader__', None)" rather than > assuming the attribute will always be present. Thomas From solipsis at pitrou.net Fri Feb 1 08:10:15 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 1 Feb 2013 08:10:15 +0100 Subject: [Python-Dev] cpython (merge 3.3 -> default): #16128: merge with 3.3. References: <3Yy3VQ355CzRJ5@mail.python.org> Message-ID: <20130201081015.4a773871@pitrou.net> On Fri, 1 Feb 2013 04:20:46 +0100 (CET) ezio.melotti wrote: > > diff --git a/Doc/glossary.rst b/Doc/glossary.rst > --- a/Doc/glossary.rst > +++ b/Doc/glossary.rst > @@ -320,7 +320,8 @@ > All of Python's immutable built-in objects are hashable, while no mutable > containers (such as lists or dictionaries) are. Objects which are > instances of user-defined classes are hashable by default; they all > - compare unequal, and their hash value is their :func:`id`. > + compare unequal (except with themselves), and their hash value is their > + :func:`id`. Python 3.3.0+ (3.3:127abd16168a, Sep 29 2012, 18:46:33) [GCC 4.5.2] on linux Type "help", "copyright", "credits" or "license" for more information. >>> o = object() >>> hash(o) == id(o) False Regards Antoine. From solipsis at pitrou.net Fri Feb 1 08:14:25 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 1 Feb 2013 08:14:25 +0100 Subject: [Python-Dev] cpython (2.7): - Issue #17086: Backport the patches from the 3.3 branch to cross-build In-Reply-To: References: <3YxxXq44T7zQ28@mail.python.org> <20130201005001.5f73e15b@pitrou.net> Message-ID: <20130201081425.39120441@pitrou.net> On Fri, 1 Feb 2013 11:00:24 +1000 Nick Coghlan wrote: > On Fri, Feb 1, 2013 at 9:50 AM, Antoine Pitrou wrote: > > On Thu, 31 Jan 2013 23:52:27 +0100 (CET) > > matthias.klose wrote: > >> http://hg.python.org/cpython/rev/8ee6d96a1019 > >> changeset: 81859:8ee6d96a1019 > >> branch: 2.7 > >> parent: 81855:df9f8feb7444 > >> user: doko at python.org > >> date: Thu Jan 31 23:52:03 2013 +0100 > >> summary: > >> - Issue #17086: Backport the patches from the 3.3 branch to cross-build > >> the package. > > > > You aren't supposed to add new features to bugfix branches. Did you > > have a specific reason to do this? > > One of the reasons for the long maintenance period on 2.7 is to keep > it building as the underlying platforms change. With the rise of ARM > systems, being able to reliably cross-build Python 2.7 for ARM from an > x86_64 system is fairly important. I would like to see a better argument for this. The rise of ARM systems is the rise of ARM systems powerful enough to build Python without cross-compiling (which is why we *now* have ARM buildbots). The very small ARM systems which need cross-compiling have existed for decades. > That said, as a procedural point for build related new features in > 2.7, I agree they should be proposed, with an explicit rationale, on > python-dev before proceeding with the commit. I think this huge changeset should be reverted. It's a complete failure in terms of procedure and following the rules. "Just because it can be useful" is not a good enough reason to violate our release model without even asking. Regards Antoine. From lvroyce at linux.vnet.ibm.com Fri Feb 1 11:58:35 2013 From: lvroyce at linux.vnet.ibm.com (Royce Lv) Date: Fri, 01 Feb 2013 18:58:35 +0800 Subject: [Python-Dev] issue17097:baseManager serve_client() not check EINTR when recv request In-Reply-To: <20130131091702.GB3217@redhat.com> References: <20130123084901.GR6145@redhat.com> <51013C65.3050703@redhat.com> <51020A6F.6040201@linux.vnet.ibm.com> <20130131091702.GB3217@redhat.com> Message-ID: <510B9FDB.6080602@linux.vnet.ibm.com> Hi All, We create our customised manager which will fork child process with baseManager. Because we registered SIGCHLD to reap the zombie for manager, we found this causes baseManager raise RemoteError when called twice.(Test script attached.) After look at baseManager.py we found recv() in handling the request has been interrupted by comming SIGCHLD, not retry recv(), but raise to client side directly. try: methodname = obj = None request = recv()<------------------this line been interrupted by SIGCHLD ident, methodname, args, kwds = request obj, exposed, gettypeid = id_to_obj[ident] if methodname not in exposed: raise AttributeError( 'method %r of %r object is not in exposed=%r' % (methodname, type(obj), exposed) ) function = getattr(obj, methodname) try: res = function(*args, **kwds) except Exception, e: msg = ('#ERROR', e) else: typeid = gettypeid and gettypeid.get(methodname, None) if typeid: rident, rexposed = self.create(conn, typeid, res) token = Token(typeid, self.address, rident) msg = ('#PROXY', (rexposed, token)) else: msg = ('#RETURN', res) except AttributeError: if methodname is None: msg = ('#TRACEBACK', format_exc()) else: try: fallback_func = self.fallback_mapping[methodname] result = fallback_func( self, conn, ident, obj, *args, **kwds ) msg = ('#RETURN', result) except Exception: msg = ('#TRACEBACK', format_exc()) except EOFError: util.debug('got EOF -- exiting thread serving %r', threading.current_thread().name) sys.exit(0) except Exception:<------does not handle IOError,INTR here should retry recv() msg = ('#TRACEBACK', format_exc()) REF: http://bugs.python.org/issue17097 -------------- next part -------------- A non-text attachment was scrubbed... Name: fakesupervdsm.py Type: text/x-python Size: 1571 bytes Desc: not available URL: From shibturn at gmail.com Fri Feb 1 14:08:01 2013 From: shibturn at gmail.com (Richard Oudkerk) Date: Fri, 01 Feb 2013 13:08:01 +0000 Subject: [Python-Dev] Release or not release the GIL In-Reply-To: References: Message-ID: On 01/02/2013 1:44am, Guido van Rossum wrote: > I'm guessing those system calls are just implemented by the kernel and > cannot block for I/O, so it was deemed unnecessary to release the GIL > around them. I don't mind changing that though, you can never know > what happens when you make a system call. close() can block, so I assume the same is true of dup2(), since dup2(oldfd, newfd) closes oldfd. -- Richard From victor.stinner at gmail.com Fri Feb 1 14:20:02 2013 From: victor.stinner at gmail.com (Victor Stinner) Date: Fri, 1 Feb 2013 14:20:02 +0100 Subject: [Python-Dev] Release or not release the GIL In-Reply-To: References: Message-ID: > dup2(oldfd, newfd) closes oldfd. No, it doesn't close oldfd. It may close newfd if it was already open. Victor From cf.natali at gmail.com Fri Feb 1 14:37:35 2013 From: cf.natali at gmail.com (=?ISO-8859-1?Q?Charles=2DFran=E7ois_Natali?=) Date: Fri, 1 Feb 2013 14:37:35 +0100 Subject: [Python-Dev] Release or not release the GIL In-Reply-To: References: Message-ID: >> dup2(oldfd, newfd) closes oldfd. > > No, it doesn't close oldfd. > > It may close newfd if it was already open. (I guess that's what he meant). Anyway, only dup2() should probably release the GIL. One reasonable heuristic is to check the man page: if the syscall can return EINTR, then the GIL should be released. From brett at python.org Fri Feb 1 14:46:15 2013 From: brett at python.org (Brett Cannon) Date: Fri, 1 Feb 2013 08:46:15 -0500 Subject: [Python-Dev] importlib.find_loader In-Reply-To: References: Message-ID: Bugs #17098 and #17099 filed. On Fri, Feb 1, 2013 at 1:56 AM, Thomas Heller wrote: > Am 01.02.2013 01:42, schrieb Nick Coghlan: > > Yep, looks like a bug in the bootstrapping, failing to set __loader__ >> properly. >> > > It also has the effect that reload does not work: > > > Type "help", "copyright", "credits" or "license" for more information. > >>> import imp > >>> import math > >>> imp.reload(math) > > >>> import signal > >>> imp.reload(signal) > Traceback (most recent call last): > File "", line 1, in > File "C:\Python33-64\lib\imp.py", line 252, in reload > return module.__loader__.load_module(**name) > > AttributeError: 'module' object has no attribute '__loader__' > >>> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From amauryfa at gmail.com Fri Feb 1 15:18:39 2013 From: amauryfa at gmail.com (Amaury Forgeot d'Arc) Date: Fri, 1 Feb 2013 15:18:39 +0100 Subject: [Python-Dev] Release or not release the GIL In-Reply-To: References: Message-ID: 2013/2/1 Charles-Fran?ois Natali > >> dup2(oldfd, newfd) closes oldfd. > > > > No, it doesn't close oldfd. > > > > It may close newfd if it was already open. > > (I guess that's what he meant). > > Anyway, only dup2() should probably release the GIL. > > One reasonable heuristic is to check the man page: if the syscall can > return EINTR, then the GIL should be released. Should the call be retried in the EINTR case? (After a PyErr_CheckSignals) -- Amaury Forgeot d'Arc -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Fri Feb 1 16:22:42 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 1 Feb 2013 16:22:42 +0100 Subject: [Python-Dev] Release or not release the GIL References: Message-ID: <20130201162242.77522b26@pitrou.net> Le Fri, 1 Feb 2013 15:18:39 +0100, "Amaury Forgeot d'Arc" a ?crit : > 2013/2/1 Charles-Fran?ois Natali > > > >> dup2(oldfd, newfd) closes oldfd. > > > > > > No, it doesn't close oldfd. > > > > > > It may close newfd if it was already open. > > > > (I guess that's what he meant). > > > > Anyway, only dup2() should probably release the GIL. > > > > One reasonable heuristic is to check the man page: if the syscall > > can return EINTR, then the GIL should be released. > > > Should the call be retried in the EINTR case? > (After a PyErr_CheckSignals) I don't think we want to retry low-level system calls (but I'm not sure we're currently consistent in that regard). Regards Antoine. From status at bugs.python.org Fri Feb 1 18:07:31 2013 From: status at bugs.python.org (Python tracker) Date: Fri, 1 Feb 2013 18:07:31 +0100 (CET) Subject: [Python-Dev] Summary of Python tracker Issues Message-ID: <20130201170731.99B3856917@psf.upfronthosting.co.za> ACTIVITY SUMMARY (2013-01-25 - 2013-02-01) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 3848 (+11) closed 25028 (+57) total 28876 (+68) Open issues with patches: 1681 Issues opened (52) ================== #17034: Use Py_CLEAR() in stringobject.c and bytesobject.c http://bugs.python.org/issue17034 opened by serhiy.storchaka #17035: Use new style classes in {class, static}method examples http://bugs.python.org/issue17035 opened by berker.peksag #17036: Implementation of the PEP 433: Easier suppression of file desc http://bugs.python.org/issue17036 opened by haypo #17037: Use a test.support helper to wrap the PEP 399 boilerplate http://bugs.python.org/issue17037 opened by eric.snow #17043: Invalid read in test_codecs http://bugs.python.org/issue17043 opened by skrah #17044: Implement PEP 422: Simple class initialisation hook http://bugs.python.org/issue17044 opened by daniel.urban #17045: Improve C-API doc for PyTypeObject http://bugs.python.org/issue17045 opened by eric.snow #17046: test_subprocess test_executable_without_cwd fails when run wit http://bugs.python.org/issue17046 opened by ned.deily #17047: Fix double double words words http://bugs.python.org/issue17047 opened by serhiy.storchaka #17048: calendar should understand full- vs. half-width characters http://bugs.python.org/issue17048 opened by nneonneo #17050: argparse.REMAINDER doesn't work as first argument http://bugs.python.org/issue17050 opened by chris.jerdonek #17052: unittest discovery should use self.testLoader http://bugs.python.org/issue17052 opened by chris.jerdonek #17053: pydoc should use inspect.signature instead of inspect.getfulla http://bugs.python.org/issue17053 opened by ronaldoussoren #17056: Support Visual Studio 2012 http://bugs.python.org/issue17056 opened by cowwoc #17057: Data model documentation grammar http://bugs.python.org/issue17057 opened by Hobson.Lane #17060: IDLE -- jump to home should not go past the PS1 and PS2 prompt http://bugs.python.org/issue17060 opened by rhettinger #17061: tokenize unconditionally emits NL after comment lines & blank http://bugs.python.org/issue17061 opened by takluyver #17062: An os.walk inspired replacement for pkgutil.walk_packages http://bugs.python.org/issue17062 opened by ncoghlan #17063: assert_called_with could be more powerful if it allowed placeh http://bugs.python.org/issue17063 opened by pitrou #17064: Fix sporadic buildbot failures for test_mailbox http://bugs.python.org/issue17064 opened by jkloth #17065: Fix sporadic buildbot failures for test_winreg http://bugs.python.org/issue17065 opened by jkloth #17066: Fix test discovery for test_robotparser.py http://bugs.python.org/issue17066 opened by zach.ware #17067: Examples in documentation for 3.x in 23.1.3.4. Deferred transl http://bugs.python.org/issue17067 opened by carsten.klein at axn-software.de #17068: peephole optimization for constant strings http://bugs.python.org/issue17068 opened by nnorwitz #17069: HTTP result code in urllib2.urlopen() file object undocumented http://bugs.python.org/issue17069 opened by tazle #17070: PEP 433: Use the new cloexec to improve security and avoid bug http://bugs.python.org/issue17070 opened by haypo #17073: Integer overflow in sqlite module http://bugs.python.org/issue17073 opened by serhiy.storchaka #17076: shutil.copytree failing on xattr-less filesystems (like NFS) http://bugs.python.org/issue17076 opened by twouters #17077: Fix test_tools hangs http://bugs.python.org/issue17077 opened by jkloth #17078: string.Template.safe_substitute hard-wires "braces" as {} http://bugs.python.org/issue17078 opened by Adam.Kellas #17079: Fix test discovery for test_ctypes.py http://bugs.python.org/issue17079 opened by zach.ware #17080: A better error message for float() http://bugs.python.org/issue17080 opened by Jonathan.Livni #17082: Fix test discovery for test_dbm*.py http://bugs.python.org/issue17082 opened by zach.ware #17083: can't specify newline string for readline for binary files http://bugs.python.org/issue17083 opened by susurrus #17084: nntplib.NNTP.xover does not always return results as documente http://bugs.python.org/issue17084 opened by dholm #17085: test_socket crashes the whole test suite http://bugs.python.org/issue17085 opened by ddvento at ucar.edu #17086: backport cross-build patches to the 2.7 branch http://bugs.python.org/issue17086 opened by doko #17087: Improve the repr for regular expression match objects http://bugs.python.org/issue17087 opened by rhettinger #17088: ElementTree incorrectly refuses to write attributes without na http://bugs.python.org/issue17088 opened by silverbacknet #17089: Expat parser parses strings only when XML encoding is UTF-8 http://bugs.python.org/issue17089 opened by serhiy.storchaka #17091: thread.lock.acquire docstring bug http://bugs.python.org/issue17091 opened by arigo #17092: Disable TIPC in configure http://bugs.python.org/issue17092 opened by ddvento at ucar.edu #17093: Make importlib.abc more inheritance-friendly http://bugs.python.org/issue17093 opened by brett.cannon #17094: sys._current_frames() reports too many/wrong stack frames http://bugs.python.org/issue17094 opened by Ringding #17095: Modules/Setup *shared* support broken http://bugs.python.org/issue17095 opened by twouters #17096: the system keyring should be used instead of ~/.pypirc http://bugs.python.org/issue17096 opened by graingert #17097: baseManager serve_client() not check EINTR when recv request http://bugs.python.org/issue17097 opened by lvroyce #17098: Set __loader__ on modules imported by the C level http://bugs.python.org/issue17098 opened by brett.cannon #17099: Raise ValueError when __loader__ not defined for importlib.fin http://bugs.python.org/issue17099 opened by brett.cannon #17100: rotating an ordereddict http://bugs.python.org/issue17100 opened by pitrou #17101: __main__.py Multiprocessing on Windows http://bugs.python.org/issue17101 opened by pthiem #17055: ftplib.ftpcp test http://bugs.python.org/issue17055 opened by inc0 Most recent 15 issues with no replies (15) ========================================== #17101: __main__.py Multiprocessing on Windows http://bugs.python.org/issue17101 #17100: rotating an ordereddict http://bugs.python.org/issue17100 #17097: baseManager serve_client() not check EINTR when recv request http://bugs.python.org/issue17097 #17096: the system keyring should be used instead of ~/.pypirc http://bugs.python.org/issue17096 #17095: Modules/Setup *shared* support broken http://bugs.python.org/issue17095 #17094: sys._current_frames() reports too many/wrong stack frames http://bugs.python.org/issue17094 #17091: thread.lock.acquire docstring bug http://bugs.python.org/issue17091 #17089: Expat parser parses strings only when XML encoding is UTF-8 http://bugs.python.org/issue17089 #17088: ElementTree incorrectly refuses to write attributes without na http://bugs.python.org/issue17088 #17085: test_socket crashes the whole test suite http://bugs.python.org/issue17085 #17084: nntplib.NNTP.xover does not always return results as documente http://bugs.python.org/issue17084 #17082: Fix test discovery for test_dbm*.py http://bugs.python.org/issue17082 #17079: Fix test discovery for test_ctypes.py http://bugs.python.org/issue17079 #17073: Integer overflow in sqlite module http://bugs.python.org/issue17073 #17070: PEP 433: Use the new cloexec to improve security and avoid bug http://bugs.python.org/issue17070 Most recent 15 issues waiting for review (15) ============================================= #17095: Modules/Setup *shared* support broken http://bugs.python.org/issue17095 #17094: sys._current_frames() reports too many/wrong stack frames http://bugs.python.org/issue17094 #17089: Expat parser parses strings only when XML encoding is UTF-8 http://bugs.python.org/issue17089 #17086: backport cross-build patches to the 2.7 branch http://bugs.python.org/issue17086 #17082: Fix test discovery for test_dbm*.py http://bugs.python.org/issue17082 #17080: A better error message for float() http://bugs.python.org/issue17080 #17079: Fix test discovery for test_ctypes.py http://bugs.python.org/issue17079 #17077: Fix test_tools hangs http://bugs.python.org/issue17077 #17076: shutil.copytree failing on xattr-less filesystems (like NFS) http://bugs.python.org/issue17076 #17073: Integer overflow in sqlite module http://bugs.python.org/issue17073 #17070: PEP 433: Use the new cloexec to improve security and avoid bug http://bugs.python.org/issue17070 #17066: Fix test discovery for test_robotparser.py http://bugs.python.org/issue17066 #17065: Fix sporadic buildbot failures for test_winreg http://bugs.python.org/issue17065 #17064: Fix sporadic buildbot failures for test_mailbox http://bugs.python.org/issue17064 #17057: Data model documentation grammar http://bugs.python.org/issue17057 Top 10 most discussed issues (10) ================================= #17037: Use a test.support helper to wrap the PEP 399 boilerplate http://bugs.python.org/issue17037 30 msgs #16997: subtests http://bugs.python.org/issue16997 14 msgs #14468: Update cloning guidelines in devguide http://bugs.python.org/issue14468 12 msgs #3871: cross and native build of python for mingw* hosts http://bugs.python.org/issue3871 9 msgs #17036: Implementation of the PEP 433: Easier suppression of file desc http://bugs.python.org/issue17036 8 msgs #16743: mmap on Windows can mishandle files larger than sys.maxsize http://bugs.python.org/issue16743 6 msgs #17063: assert_called_with could be more powerful if it allowed placeh http://bugs.python.org/issue17063 6 msgs #17080: A better error message for float() http://bugs.python.org/issue17080 6 msgs #16968: Fix test discovery for test_concurrent_futures.py http://bugs.python.org/issue16968 5 msgs #17015: mock could be smarter and inspect the spec's signature http://bugs.python.org/issue17015 5 msgs Issues closed (54) ================== #1783: nonexistent data items declared as exports in sysmodule.h http://bugs.python.org/issue1783 closed by serhiy.storchaka #3718: environment variable MACHDEP and python build system http://bugs.python.org/issue3718 closed by doko #4844: ZipFile doesn't range check in _EndRecData() http://bugs.python.org/issue4844 closed by serhiy.storchaka #7576: Avoid warnings in PyModuleDef_HEAD_INIT http://bugs.python.org/issue7576 closed by serhiy.storchaka #9290: IDLE and Command line present different behavior for sys.stdin http://bugs.python.org/issue9290 closed by serhiy.storchaka #10156: Initialization of globals in unicodeobject.c http://bugs.python.org/issue10156 closed by skrah #10588: imp.find_module raises unexpected SyntaxError http://bugs.python.org/issue10588 closed by brett.cannon #11049: add tests for test.support http://bugs.python.org/issue11049 closed by brett.cannon #11676: Improve imp.load_module and submodules doc http://bugs.python.org/issue11676 closed by brett.cannon #12004: PyZipFile.writepy gives internal error on syntax errors http://bugs.python.org/issue12004 closed by serhiy.storchaka #12397: re match object methods have no docstrings http://bugs.python.org/issue12397 closed by serhiy.storchaka #12583: More detailed ImportError messages http://bugs.python.org/issue12583 closed by brett.cannon #12648: Wrong import module search order on Windows http://bugs.python.org/issue12648 closed by brett.cannon #12693: test.support.transient_internet prints to stderr when verbose http://bugs.python.org/issue12693 closed by brett.cannon #12701: Apple's clang 2.1 (xcode 4.1, OSX 10.7) optimizer miscompiles http://bugs.python.org/issue12701 closed by brett.cannon #13048: Handling of paths in first argument of imp.find_module() http://bugs.python.org/issue13048 closed by brett.cannon #13539: Return value missing in calendar.TimeEncoding.__enter__ http://bugs.python.org/issue13539 closed by serhiy.storchaka #13590: extension module builds fail with python.org OS X installers o http://bugs.python.org/issue13590 closed by ned.deily #13886: readline-related test_builtin failure http://bugs.python.org/issue13886 closed by nadeem.vawda #13997: Clearly explain the bare minimum Python 3 users should know ab http://bugs.python.org/issue13997 closed by ezio.melotti #15505: unittest.installHandler incorrectly assumes SIGINT handler is http://bugs.python.org/issue15505 closed by python-dev #16096: Get rid of dangerous integer overflow tricks http://bugs.python.org/issue16096 closed by serhiy.storchaka #16128: hashable documentation error http://bugs.python.org/issue16128 closed by ezio.melotti #16235: Add python-config.sh for use during cross compilation. http://bugs.python.org/issue16235 closed by doko #16769: Remove some old Visual Studio versions from PC/ directory http://bugs.python.org/issue16769 closed by brian.curtin #16772: int() accepts float number base http://bugs.python.org/issue16772 closed by mark.dickinson #16899: Add support for C99 complex type (_Complex) as ctypes.c_comple http://bugs.python.org/issue16899 closed by mark.dickinson #16971: Refleaks in charmap decoder http://bugs.python.org/issue16971 closed by serhiy.storchaka #16972: Useless function call in site.py http://bugs.python.org/issue16972 closed by brett.cannon #16975: Broken error handling in codecs.escape_decode() http://bugs.python.org/issue16975 closed by serhiy.storchaka #16979: Broken error handling in codecs.unicode_escape_decode() http://bugs.python.org/issue16979 closed by serhiy.storchaka #16980: SystemError in codecs.unicode_escape_decode() http://bugs.python.org/issue16980 closed by serhiy.storchaka #17020: random.random() generating values >= 1.0 http://bugs.python.org/issue17020 closed by r.david.murray #17027: support for drag and drop of files to tkinter application http://bugs.python.org/issue17027 closed by ned.deily #17028: launcher does not read shebang line when arguments are given http://bugs.python.org/issue17028 closed by python-dev #17031: fix running regen in cross builds http://bugs.python.org/issue17031 closed by doko #17038: multiprocessing only use one core when C module are imported http://bugs.python.org/issue17038 closed by neologix #17039: socket.SocketIO hides socket timeouts as blocking errors http://bugs.python.org/issue17039 closed by Ronny.Pfannschmidt #17040: Document context manager support for shelf objects http://bugs.python.org/issue17040 closed by ezio.melotti #17041: Fix tests for build --without-doc-strings http://bugs.python.org/issue17041 closed by serhiy.storchaka #17042: Example in C-API memory management doc has confusing order http://bugs.python.org/issue17042 closed by ezio.melotti #17049: calendar throws UnicodeEncodeError when locale is specified http://bugs.python.org/issue17049 closed by serhiy.storchaka #17054: cStringIO.StringIO aborted when more then INT_MAX bytes writte http://bugs.python.org/issue17054 closed by serhiy.storchaka #17059: tarfile.is_tarfile without read permissions raises AttributeEr http://bugs.python.org/issue17059 closed by serhiy.storchaka #17071: Signature.bind() fails with a keyword argument named "self" http://bugs.python.org/issue17071 closed by pitrou #17072: Decimal, quantize, round and negative value http://bugs.python.org/issue17072 closed by mark.dickinson #17074: (docs) Consistent formatting of constants http://bugs.python.org/issue17074 closed by chris.jerdonek #17075: logging documentation for library cleanup http://bugs.python.org/issue17075 closed by chris.jerdonek #17081: documentation http://bugs.python.org/issue17081 closed by chris.jerdonek #17090: io.TextIOWrapper does not handle UTF-8 encoded streams correct http://bugs.python.org/issue17090 closed by r.david.murray #1777134: minidom pretty xml output improvement http://bugs.python.org/issue1777134 closed by ezio.melotti #17051: Memory leak in os.path.isdir under Windows http://bugs.python.org/issue17051 closed by serhiy.storchaka #17058: Python crashes with error 0xc0000417 http://bugs.python.org/issue17058 closed by brian.curtin #1681974: mkdtemp fails on Windows if username has non-ASCII character http://bugs.python.org/issue1681974 closed by ezio.melotti From greg.ewing at canterbury.ac.nz Fri Feb 1 23:04:43 2013 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sat, 02 Feb 2013 11:04:43 +1300 Subject: [Python-Dev] Release or not release the GIL In-Reply-To: References: Message-ID: <510C3BFB.7040503@canterbury.ac.nz> Charles-Fran?ois Natali wrote: > Anyway, only dup2() should probably release the GIL. Depending on how your kernel is implemented, both pipe() and socketpair() could create file system entries, and therefore could block, although probably only for a very short time. -- Greg From greg.ewing at canterbury.ac.nz Fri Feb 1 23:16:26 2013 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sat, 02 Feb 2013 11:16:26 +1300 Subject: [Python-Dev] Release or not release the GIL In-Reply-To: References: Message-ID: <510C3EBA.8020206@canterbury.ac.nz> Amaury Forgeot d'Arc wrote: > One reasonable heuristic is to check the man page: if the syscall can > return EINTR, then the GIL should be released. Hmmm, curious. According to the MacOSX man pages, both dup() and dup2() can return EINTR, but pipe() and socketpair() can't. I'm particularly surprised about dup(), as that's the one I would have thought couldn't possibly block -- it only needs to manipulate in-memory data structures. For Linux, the only one returning EINTR is dup2(). FreeBSD has none of them returning EINTR. So it looks like there's a fair bit of variation between unices here. -- Greg From greg at krypto.org Sat Feb 2 00:25:27 2013 From: greg at krypto.org (Gregory P. Smith) Date: Fri, 1 Feb 2013 15:25:27 -0800 Subject: [Python-Dev] Release or not release the GIL In-Reply-To: <20130201162242.77522b26@pitrou.net> References: <20130201162242.77522b26@pitrou.net> Message-ID: On Fri, Feb 1, 2013 at 7:22 AM, Antoine Pitrou wrote: > Le Fri, 1 Feb 2013 15:18:39 +0100, > "Amaury Forgeot d'Arc" a ?crit : > > 2013/2/1 Charles-Fran?ois Natali > > > > > >> dup2(oldfd, newfd) closes oldfd. > > > > > > > > No, it doesn't close oldfd. > > > > > > > > It may close newfd if it was already open. > > > > > > (I guess that's what he meant). > > > > > > Anyway, only dup2() should probably release the GIL. > > > > > > One reasonable heuristic is to check the man page: if the syscall > > > can return EINTR, then the GIL should be released. > > > > > > Should the call be retried in the EINTR case? > > (After a PyErr_CheckSignals) > > I don't think we want to retry low-level system calls (but I'm not sure > we're currently consistent in that regard). > I think this is what you meant but to be clear: Anywhere we're using them within a library for a good purpose, we do need to retry. If we're merely exposing them via the os module such as os.dup, its up to the caller to deal with the retry. -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Sat Feb 2 00:47:25 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sat, 2 Feb 2013 00:47:25 +0100 Subject: [Python-Dev] Release or not release the GIL In-Reply-To: References: <20130201162242.77522b26@pitrou.net> Message-ID: <20130202004725.6fd1c32c@pitrou.net> On Fri, 1 Feb 2013 15:25:27 -0800 "Gregory P. Smith" wrote: > On Fri, Feb 1, 2013 at 7:22 AM, Antoine Pitrou wrote: > > > Le Fri, 1 Feb 2013 15:18:39 +0100, > > "Amaury Forgeot d'Arc" a ?crit : > > > 2013/2/1 Charles-Fran?ois Natali > > > > > > > >> dup2(oldfd, newfd) closes oldfd. > > > > > > > > > > No, it doesn't close oldfd. > > > > > > > > > > It may close newfd if it was already open. > > > > > > > > (I guess that's what he meant). > > > > > > > > Anyway, only dup2() should probably release the GIL. > > > > > > > > One reasonable heuristic is to check the man page: if the syscall > > > > can return EINTR, then the GIL should be released. > > > > > > > > > Should the call be retried in the EINTR case? > > > (After a PyErr_CheckSignals) > > > > I don't think we want to retry low-level system calls (but I'm not sure > > we're currently consistent in that regard). > > > > I think this is what you meant but to be clear: Anywhere we're using them > within a library for a good purpose, we do need to retry. If we're merely > exposing them via the os module such as os.dup, its up to the caller to > deal with the retry. Indeed, that's what I meant. Sorry for not being very clear. Regards Antoine. From a.cavallo at cavallinux.eu Sat Feb 2 16:55:49 2013 From: a.cavallo at cavallinux.eu (Antonio Cavallo) Date: Sat, 02 Feb 2013 15:55:49 +0000 Subject: [Python-Dev] cpython (2.7): - Issue #17086: Backport the patches from the 3.3 branch to cross-build In-Reply-To: <20130201081425.39120441@pitrou.net> References: <3YxxXq44T7zQ28@mail.python.org> <20130201005001.5f73e15b@pitrou.net> <20130201081425.39120441@pitrou.net> Message-ID: <510D3705.7020304@cavallinux.eu> I personally agree that huge changesets aren't good. But I can also see the problem of cross compiling should be sooner or later addressed. I'm fairly interested in cross building python on android too. A point to support cross compiling (instead native build arm-on-arm) could be android doesn't provide (to my knowledge) a native toolchain: that's a quite large target. And assuming arm equals unix so there's always an installed or installable compiler on it is not always the case. If I can suggest a simple solution that'd be a simple makefile to compile python (I have one on my own): no "configure" magic involved more akin to the windows build style (yes, I've just said the W* word). I hope this helps, antonio >>>> - Issue #17086: Backport the patches from the 3.3 branch to cross-build >>>> the package. >>> You aren't supposed to add new features to bugfix branches. Did you >>> have a specific reason to do this? >> One of the reasons for the long maintenance period on 2.7 is to keep >> it building as the underlying platforms change. With the rise of ARM >> systems, being able to reliably cross-build Python 2.7 for ARM from an >> x86_64 system is fairly important. > > I would like to see a better argument for this. The rise of ARM systems > is the rise of ARM systems powerful enough to build Python without > cross-compiling (which is why we *now* have ARM buildbots). > The very small ARM systems which need cross-compiling have existed for > decades. > >> That said, as a procedural point for build related new features in >> 2.7, I agree they should be proposed, with an explicit rationale, on >> python-dev before proceeding with the commit. > > I think this huge changeset should be reverted. It's a complete failure > in terms of procedure and following the rules. "Just because it can be > useful" is not a good enough reason to violate our release model > without even asking. > > Regards > > Antoine. > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/a.cavallo%40cavallinux.eu From ncoghlan at gmail.com Sun Feb 3 07:04:32 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 3 Feb 2013 16:04:32 +1000 Subject: [Python-Dev] BDFL delegation for PEP 426 (PyPI metadata 1.3) Message-ID: In doing the detailed review of PEP 426 as BDFL-Delegate, I keep noticing confusing problems with the current spec that mean I want to become a *co-author* on the spec, rather than explaining to the current authors the aspects I object to, until they produce a version that I'm happy with (this is frustrating for the authors as well, since several of the problems have been inherited from the previous version of the spec rather than being new in the current version). Now, Guido's authored and accepted his own PEPs in the past, but to date we've avoided letting anyone else do that. Since I *definitely* want to co-author the new metadata PEP (mostly to address issues with the version specifier section and to include the *rationale* for changes, rather than merely listing them as previous versions of the metadata PEPs have done), that means one of the following needs to happen: - someone else volunteers to be BDFL-Delegate for PEP 426 (MvL, perhaps?) - I get clear approval (perhaps from Guido?) to be both co-author *and* BDFL-Delegate for PEP 426 Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From guido at python.org Sun Feb 3 07:37:54 2013 From: guido at python.org (Guido van Rossum) Date: Sat, 2 Feb 2013 22:37:54 -0800 Subject: [Python-Dev] BDFL delegation for PEP 426 (PyPI metadata 1.3) In-Reply-To: References: Message-ID: On Sat, Feb 2, 2013 at 10:04 PM, Nick Coghlan wrote: > In doing the detailed review of PEP 426 as BDFL-Delegate, I keep > noticing confusing problems with the current spec that mean I want to > become a *co-author* on the spec, rather than explaining to the > current authors the aspects I object to, until they produce a version > that I'm happy with (this is frustrating for the authors as well, > since several of the problems have been inherited from the previous > version of the spec rather than being new in the current version). > > Now, Guido's authored and accepted his own PEPs in the past, but to > date we've avoided letting anyone else do that. Since I *definitely* > want to co-author the new metadata PEP (mostly to address issues with > the version specifier section and to include the *rationale* for > changes, rather than merely listing them as previous versions of the > metadata PEPs have done), that means one of the following needs to > happen: > > - someone else volunteers to be BDFL-Delegate for PEP 426 (MvL, perhaps?) > - I get clear approval (perhaps from Guido?) to be both co-author > *and* BDFL-Delegate for PEP 426 I don't know or care much about PyPI metadata, so do what you feel is right. If you are uncomfortable being PEP-uncle *and* -author, find another author or another uncle. But since it doesn't affect the language or library, it's fine with me if you are both. :-) -- --Guido van Rossum (python.org/~guido) From ncoghlan at gmail.com Sun Feb 3 07:44:33 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 3 Feb 2013 16:44:33 +1000 Subject: [Python-Dev] BDFL delegation for PEP 426 (PyPI metadata 1.3) In-Reply-To: References: Message-ID: On Sun, Feb 3, 2013 at 4:37 PM, Guido van Rossum wrote: > I don't know or care much about PyPI metadata, so do what you feel is > right. If you are uncomfortable being PEP-uncle *and* -author, find > another author or another uncle. But since it doesn't affect the > language or library, it's fine with me if you are both. :-) To clarify what will happen if I handle both jobs: I'll bug a few specific people like Vinay Sajip (distlib), and MvL or Richard Jones (PyPI) to at least glance over it, as well as giving distutils-sig, catalog-sig and python-dev one last chance to comment on my revised draft. Then, wearing my BDFL-Delegate hat, I'd decide what feedback I considered worth addressing and what could be safely ignored (or postponed to the next time the metadata gets updated). I don't expect anything I want to do to be particularly controversial, but I think it's worth trying to get it right (even if it delays wheel support in pip for a few more weeks). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From solipsis at pitrou.net Sun Feb 3 12:27:54 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sun, 3 Feb 2013 12:27:54 +0100 Subject: [Python-Dev] BDFL delegation for PEP 426 (PyPI metadata 1.3) References: Message-ID: <20130203122754.7745a1be@pitrou.net> On Sun, 3 Feb 2013 16:44:33 +1000 Nick Coghlan wrote: > On Sun, Feb 3, 2013 at 4:37 PM, Guido van Rossum wrote: > > I don't know or care much about PyPI metadata, so do what you feel is > > right. If you are uncomfortable being PEP-uncle *and* -author, find > > another author or another uncle. But since it doesn't affect the > > language or library, it's fine with me if you are both. :-) > > To clarify what will happen if I handle both jobs: I'll bug a few > specific people like Vinay Sajip (distlib), and MvL or Richard Jones > (PyPI) to at least glance over it, as well as giving distutils-sig, > catalog-sig and python-dev one last chance to comment on my revised > draft. > > Then, wearing my BDFL-Delegate hat, I'd decide what feedback I > considered worth addressing and what could be safely ignored (or > postponed to the next time the metadata gets updated). > > I don't expect anything I want to do to be particularly controversial, > but I think it's worth trying to get it right (even if it delays wheel > support in pip for a few more weeks). Will wheel be implemented in distutils? Regards Antoine. From p.f.moore at gmail.com Sun Feb 3 13:34:36 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Sun, 3 Feb 2013 12:34:36 +0000 Subject: [Python-Dev] BDFL delegation for PEP 426 (PyPI metadata 1.3) In-Reply-To: <20130203122754.7745a1be@pitrou.net> References: <20130203122754.7745a1be@pitrou.net> Message-ID: On 3 February 2013 11:27, Antoine Pitrou wrote: >> I don't expect anything I want to do to be particularly controversial, >> but I think it's worth trying to get it right (even if it delays wheel >> support in pip for a few more weeks). > > Will wheel be implemented in distutils? There are no plans to do so (and I don't think it's necessary, particularly given the general "distutils is frozen" principle). To clarify: - The wheel project provides a bdist_wheel command to build wheels, but this depends on setuptools/distribute. I don't know how hard it would be to write a version that didn't depend on them. - The wheel project provides standalone commands to install wheels, and to convert eggs and wininst installers to wheels. The pip integration is basically to allow pip to find wheels on PyPI or any local indexes you have, and to install them via the "pip install" command. It also incidentally includes pip support for the new dist-info metadata format (I can't recall which PEP that is, sorry) simply because that's the format that wheel uses rather than the legacy egg-info format. So it's perfectly possible to use wheels right now, without the pip integration. But the pip developers don't want to integrate the wheel format just because it exists - they want the assurance that it's an accepted format supported by PEPs, hence the interest in getting the 3 wheel PEPs (of which the metadata PEP is the first) accepted. Paul From storchaka at gmail.com Sun Feb 3 13:39:20 2013 From: storchaka at gmail.com (Serhiy Storchaka) Date: Sun, 03 Feb 2013 14:39:20 +0200 Subject: [Python-Dev] 2.7.4 In-Reply-To: References: Message-ID: There are crashers for which patches were proposed but do not reviewed yet: Issue #6083: Reference counting bug in PyArg_ParseTuple and PyArg_ParseTupleAndKeywords. Issue #7358: cStringIO not 64-bit safe. Issue #16137: Using time.asctime() with an array with negative tm_hour causes Python Crash. Issue #16686: audioop overflow issues. #8865 is reviewed but not committed. From solipsis at pitrou.net Sun Feb 3 13:48:53 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sun, 3 Feb 2013 13:48:53 +0100 Subject: [Python-Dev] BDFL delegation for PEP 426 (PyPI metadata 1.3) In-Reply-To: References: <20130203122754.7745a1be@pitrou.net> Message-ID: <20130203134853.098aadaa@pitrou.net> On Sun, 3 Feb 2013 12:34:36 +0000 Paul Moore wrote: > > So it's perfectly possible to use wheels right now, without the pip > integration. But the pip developers don't want to integrate the wheel > format just because it exists - they want the assurance that it's an > accepted format supported by PEPs, hence the interest in getting the 3 > wheel PEPs (of which the metadata PEP is the first) accepted. But it's an "accepted format" by whom? If we accept the PEP but refuse to integrate support in the stdlib, it doesn't make much sense. I vote for removing the "distutils is frozen" principle. It was already destructive enough when packaging was still alive - bug fixes were actively turned down for the dubious reason that fixing bugs would break compatibility. Its rationale has become completely obsolete now that packaging has failed. Regards Antoine. From ncoghlan at gmail.com Sun Feb 3 14:04:01 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 3 Feb 2013 23:04:01 +1000 Subject: [Python-Dev] BDFL delegation for PEP 426 (PyPI metadata 1.3) In-Reply-To: References: <20130203122754.7745a1be@pitrou.net> Message-ID: On Sun, Feb 3, 2013 at 10:34 PM, Paul Moore wrote: > So it's perfectly possible to use wheels right now, without the pip > integration. But the pip developers don't want to integrate the wheel > format just because it exists - they want the assurance that it's an > accepted format supported by PEPs, hence the interest in getting the 3 > wheel PEPs (of which the metadata PEP is the first) accepted. And they're right to be concerned - I've just made it clear to Daniel that before PEP 427 will be accepted, it must either switch to using S/MIME for signatures and drop support for Java Web Signatures completely, or else it must contain a compelling rationale for why we should even be considering a signature scheme that isn't yet an IETF standard. I take the disclaimer the IETF put on their drafts seriously: "It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." MvL raised this concern last time the wheel format was discussed, and, to date, nothing has happened to address it. JWS *does* look like a neat piece of technology, but it's just too young to be basing our binary distribution infrastructure on it (especially as new crypto is, by default, bad crypto - that's why NIST/NSA hold their multi-year competitions when they need to come up with new crypto related algorithms). The other two PEPs (the new metadata and the version compatibility tags) are in a much better place. Most of the issues with PEP 426 have been inherited from the previous version of the metadata, rather than being related to the changes Daniel needed for the wheel format, and I've just completed a new draft that should address most of those problems. It's been a while since I looked closely at the compatibility tag PEP, but I don't recall their being any significant problems with it last time it was discussed. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Sun Feb 3 14:08:04 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 3 Feb 2013 23:08:04 +1000 Subject: [Python-Dev] BDFL delegation for PEP 426 (PyPI metadata 1.3) In-Reply-To: <20130203134853.098aadaa@pitrou.net> References: <20130203122754.7745a1be@pitrou.net> <20130203134853.098aadaa@pitrou.net> Message-ID: On Sun, Feb 3, 2013 at 10:48 PM, Antoine Pitrou wrote: > On Sun, 3 Feb 2013 12:34:36 +0000 > Paul Moore wrote: >> >> So it's perfectly possible to use wheels right now, without the pip >> integration. But the pip developers don't want to integrate the wheel >> format just because it exists - they want the assurance that it's an >> accepted format supported by PEPs, hence the interest in getting the 3 >> wheel PEPs (of which the metadata PEP is the first) accepted. > > But it's an "accepted format" by whom? If we accept the PEP but refuse > to integrate support in the stdlib, it doesn't make much sense. > > I vote for removing the "distutils is frozen" principle. It was already > destructive enough when packaging was still alive - bug fixes were > actively turned down for the dubious reason that fixing bugs would > break compatibility. Its rationale has become completely obsolete now > that packaging has failed. The rationale for the distutils freeze is "don't break setuptools". That rationale still holds. The most likely current candidate for integration into Python 3.4 is distlib, which, as was proposed when packaging was dropped, is built incrementally as components are standardised, rather than being a mish-mash of updated code and flawed design inherited from distutils. Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From solipsis at pitrou.net Sun Feb 3 14:09:01 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sun, 3 Feb 2013 14:09:01 +0100 Subject: [Python-Dev] BDFL delegation for PEP 426 (PyPI metadata 1.3) In-Reply-To: References: <20130203122754.7745a1be@pitrou.net> <20130203134853.098aadaa@pitrou.net> Message-ID: <20130203140901.678b32b6@pitrou.net> On Sun, 3 Feb 2013 23:08:04 +1000 Nick Coghlan wrote: > On Sun, Feb 3, 2013 at 10:48 PM, Antoine Pitrou wrote: > > On Sun, 3 Feb 2013 12:34:36 +0000 > > Paul Moore wrote: > >> > >> So it's perfectly possible to use wheels right now, without the pip > >> integration. But the pip developers don't want to integrate the wheel > >> format just because it exists - they want the assurance that it's an > >> accepted format supported by PEPs, hence the interest in getting the 3 > >> wheel PEPs (of which the metadata PEP is the first) accepted. > > > > But it's an "accepted format" by whom? If we accept the PEP but refuse > > to integrate support in the stdlib, it doesn't make much sense. > > > > I vote for removing the "distutils is frozen" principle. It was already > > destructive enough when packaging was still alive - bug fixes were > > actively turned down for the dubious reason that fixing bugs would > > break compatibility. Its rationale has become completely obsolete now > > that packaging has failed. > > The rationale for the distutils freeze is "don't break setuptools". > That rationale still holds. I'm sure it is perfectly possible to evolve and bugfix distutils without breaking distribute. What's more, distribute is actually maintained and can evolve to accomodate the fixes. As for setuptools (as opposed to distribute), I don't think we should care anymore. Regards Antoine. From ncoghlan at gmail.com Sun Feb 3 14:20:03 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 3 Feb 2013 23:20:03 +1000 Subject: [Python-Dev] BDFL delegation for PEP 426 (PyPI metadata 1.3) In-Reply-To: References: <20130203122754.7745a1be@pitrou.net> Message-ID: On Sun, Feb 3, 2013 at 11:04 PM, Nick Coghlan wrote: > MvL raised this concern last time the wheel format was discussed, and, to date, > nothing has happened to address it. My apologies to Daniel, it appears I misremembered this part of the previous discussion. Daniel assures me MvL was objecting primarily to the use of one particular crypto algorithm (which was subsequently removed from the PEP), rather than the use of JWS in general. That said, my own concerns about relying on an Internet Draft (when the IETF specifically say not to do that) remain. It just means they're a new issue, rather than one continuing over from the previous discussion. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From tseaver at palladion.com Sun Feb 3 14:27:05 2013 From: tseaver at palladion.com (Tres Seaver) Date: Sun, 03 Feb 2013 08:27:05 -0500 Subject: [Python-Dev] BDFL delegation for PEP 426 (PyPI metadata 1.3) In-Reply-To: <20130203140901.678b32b6@pitrou.net> References: <20130203122754.7745a1be@pitrou.net> <20130203134853.098aadaa@pitrou.net> <20130203140901.678b32b6@pitrou.net> Message-ID: <510E65A9.90600@palladion.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 02/03/2013 08:09 AM, Antoine Pitrou wrote: > I'm sure it is perfectly possible to evolve and bugfix distutils > without breaking distribute. What's more, distribute is actually > maintained and can evolve to accomodate the fixes. I wouldn't be on that proposition: there are a lot of burnouts down that rabbit hole. > As for setuptools (as opposed to distribute), I don't think we should > care anymore. Yes, you need to care. It is *still* true today that distribute and setuptools remain largely interchangeable, which is the only thing that makes distribute viable, in the ecosystem sense. Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with undefined - http://www.enigmail.net/ iEYEARECAAYFAlEOZakACgkQ+gerLs4ltQ5N7ACgv/v0GTs7xyQ7I/jvrcu/s2i5 kHcAoJUKigWKZTiVr1ARMLt2xtQF+TAL =mjAm -----END PGP SIGNATURE----- From brett at python.org Sun Feb 3 15:17:41 2013 From: brett at python.org (Brett Cannon) Date: Sun, 3 Feb 2013 09:17:41 -0500 Subject: [Python-Dev] [Python-checkins] cpython (3.3): Issue #17098: Be more stringent of setting __loader__ on early imported In-Reply-To: <3YyWrW1hC2zSTB@mail.python.org> References: <3YyWrW1hC2zSTB@mail.python.org> Message-ID: FYI, Raymond let me know that this is causing the buildbots to occasionally fail thanks to xml.parsers.expat.errors somehow circumventing import. I'm going to try and figure out what's going on. On Fri, Feb 1, 2013 at 4:38 PM, brett.cannon wrote: > http://hg.python.org/cpython/rev/19ea454ccdf7 > changeset: 81884:19ea454ccdf7 > branch: 3.3 > parent: 81882:4a4688b865ff > user: Brett Cannon > date: Fri Feb 01 15:31:49 2013 -0500 > summary: > Issue #17098: Be more stringent of setting __loader__ on early imported > modules. Also made test more rigorous. > > files: > Lib/importlib/_bootstrap.py | 7 +- > Lib/test/test_importlib/test_api.py | 6 + > Python/importlib.h | 591 ++++++++------- > 3 files changed, 310 insertions(+), 294 deletions(-) > > > diff --git a/Lib/importlib/_bootstrap.py b/Lib/importlib/_bootstrap.py > --- a/Lib/importlib/_bootstrap.py > +++ b/Lib/importlib/_bootstrap.py > @@ -1704,10 +1704,13 @@ > BYTECODE_SUFFIXES = DEBUG_BYTECODE_SUFFIXES > > module_type = type(sys) > - for module in sys.modules.values(): > + for name, module in sys.modules.items(): > if isinstance(module, module_type): > if not hasattr(module, '__loader__'): > - module.__loader__ = BuiltinImporter > + if name in sys.builtin_module_names: > + module.__loader__ = BuiltinImporter > + elif _imp.is_frozen(name): > + module.__loader__ = FrozenImporter > > self_module = sys.modules[__name__] > for builtin_name in ('_io', '_warnings', 'builtins', 'marshal'): > diff --git a/Lib/test/test_importlib/test_api.py > b/Lib/test/test_importlib/test_api.py > --- a/Lib/test/test_importlib/test_api.py > +++ b/Lib/test/test_importlib/test_api.py > @@ -184,6 +184,12 @@ > if isinstance(module, types.ModuleType): > self.assertTrue(hasattr(module, '__loader__'), > '{!r} lacks a __loader__ attribute'.format(name)) > + if name in sys.builtin_module_names: > + self.assertEqual(importlib.machinery.BuiltinImporter, > + module.__loader__) > + elif imp.is_frozen(name): > + self.assertEqual(importlib.machinery.FrozenImporter, > + module.__loader__) > > def test_main(): > from test.support import run_unittest > diff --git a/Python/importlib.h b/Python/importlib.h > --- a/Python/importlib.h > +++ b/Python/importlib.h > [stripped] > > -- > Repository URL: http://hg.python.org/cpython > > _______________________________________________ > Python-checkins mailing list > Python-checkins at python.org > http://mail.python.org/mailman/listinfo/python-checkins > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From benjamin at python.org Sun Feb 3 15:38:39 2013 From: benjamin at python.org (Benjamin Peterson) Date: Sun, 3 Feb 2013 09:38:39 -0500 Subject: [Python-Dev] 2.7.4 In-Reply-To: References: Message-ID: 2013/2/3 Serhiy Storchaka : > There are crashers for which patches were proposed but do not reviewed yet: > > Issue #6083: Reference counting bug in PyArg_ParseTuple and > PyArg_ParseTupleAndKeywords. > > Issue #7358: cStringIO not 64-bit safe. > > Issue #16137: Using time.asctime() with an array with negative tm_hour > causes Python Crash. > > Issue #16686: audioop overflow issues. > > #8865 is reviewed but not committed. Thanks. In the future, this should be raised to "release blocker" priority to get my attention. -- Regards, Benjamin From dholth at gmail.com Sun Feb 3 15:41:29 2013 From: dholth at gmail.com (Daniel Holth) Date: Sun, 3 Feb 2013 09:41:29 -0500 Subject: [Python-Dev] BDFL delegation for PEP 426 (PyPI metadata 1.3) In-Reply-To: <20130203134853.098aadaa@pitrou.net> References: <20130203122754.7745a1be@pitrou.net> <20130203134853.098aadaa@pitrou.net> Message-ID: The neat thing about wheel is that you can install them without having the software used to build them. So we might try to provide a very simple wheel installer script with Python that did not even depend on DistUtils. You would be able to install pip etc with that tool. There is no need to put wheel directly into DistUtils. On Feb 3, 2013 7:54 AM, "Antoine Pitrou" wrote: > On Sun, 3 Feb 2013 12:34:36 +0000 > Paul Moore wrote: > > > > So it's perfectly possible to use wheels right now, without the pip > > integration. But the pip developers don't want to integrate the wheel > > format just because it exists - they want the assurance that it's an > > accepted format supported by PEPs, hence the interest in getting the 3 > > wheel PEPs (of which the metadata PEP is the first) accepted. > > But it's an "accepted format" by whom? If we accept the PEP but refuse > to integrate support in the stdlib, it doesn't make much sense. > > I vote for removing the "distutils is frozen" principle. It was already > destructive enough when packaging was still alive - bug fixes were > actively turned down for the dubious reason that fixing bugs would > break compatibility. Its rationale has become completely obsolete now > that packaging has failed. > > Regards > > Antoine. > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/dholth%40gmail.com > -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Sun Feb 3 15:43:41 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sun, 3 Feb 2013 15:43:41 +0100 Subject: [Python-Dev] BDFL delegation for PEP 426 (PyPI metadata 1.3) In-Reply-To: References: <20130203122754.7745a1be@pitrou.net> <20130203134853.098aadaa@pitrou.net> Message-ID: <20130203154341.598b063c@pitrou.net> On Sun, 3 Feb 2013 09:41:29 -0500 Daniel Holth wrote: > The neat thing about wheel is that you can install them without having the > software used to build them. So we might try to provide a very simple wheel > installer script with Python that did not even depend on DistUtils. You > would be able to install pip etc with that tool. There is no need to put > wheel directly into DistUtils. But you still need to *produce* wheels, don't you? It would be more useful for everyone if that facility was bundled with Python. Regards Antoine. From p.f.moore at gmail.com Sun Feb 3 15:51:45 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Sun, 3 Feb 2013 14:51:45 +0000 Subject: [Python-Dev] BDFL delegation for PEP 426 (PyPI metadata 1.3) In-Reply-To: References: <20130203122754.7745a1be@pitrou.net> <20130203134853.098aadaa@pitrou.net> Message-ID: On 3 February 2013 14:41, Daniel Holth wrote: > The neat thing about wheel is that you can install them without having the > software used to build them. So we might try to provide a very simple wheel > installer script with Python that did not even depend on DistUtils. You > would be able to install pip etc with that tool. There is no need to put > wheel directly into DistUtils. The other side of the equation, adding a bdist_wheel command to distutils should *also* be simple. As command classes are essentially independent code, it's not even really going to violate whatever remains of the "distutils is frozen" philosophy. Is there any mileage to submitting a patch to the Python stdlib that implements a bdist_wheel distutils subcommand and a wheel installer in the stdlib? I'm willing to write that patch based on Daniel's existing wheel code if no-one else is willing to, but only if there's likely to be some level of support from python-dev (for example, if Antoine and Nick are in general agreement on the principle, I'd be willing to write the code and we could see where the subsequent debate leads). Paul From ncoghlan at gmail.com Sun Feb 3 16:33:29 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 4 Feb 2013 01:33:29 +1000 Subject: [Python-Dev] BDFL delegation for PEP 426 (PyPI metadata 1.3) In-Reply-To: References: <20130203122754.7745a1be@pitrou.net> <20130203134853.098aadaa@pitrou.net> Message-ID: On 4 Feb 2013 00:54, "Paul Moore" wrote: > > On 3 February 2013 14:41, Daniel Holth wrote: > > The neat thing about wheel is that you can install them without having the > > software used to build them. So we might try to provide a very simple wheel > > installer script with Python that did not even depend on DistUtils. You > > would be able to install pip etc with that tool. There is no need to put > > wheel directly into DistUtils. > > The other side of the equation, adding a bdist_wheel command to > distutils should *also* be simple. As command classes are essentially > independent code, it's not even really going to violate whatever > remains of the "distutils is frozen" philosophy. > > Is there any mileage to submitting a patch to the Python stdlib that > implements a bdist_wheel distutils subcommand and a wheel installer in > the stdlib? I'm willing to write that patch based on Daniel's existing > wheel code if no-one else is willing to, but only if there's likely to > be some level of support from python-dev (for example, if Antoine and > Nick are in general agreement on the principle, I'd be willing to > write the code and we could see where the subsequent debate leads). Adding a bdist_wheel command makes sense to me, and I agree it is the kind of change that shouldn't cause setuptools any grief. The original trigger for the distutils freeze was a ban on internal refactoring, so new features that don't alter internal APIs should be fine. My one real concern at this point is the signature scheme. We definitely need one, so dropping it from the PEP isn't an option, but the choice of JWS over S/MIME worries me. The two acceptable options I currently see are to wait until JWS is approved by the IETF, or wait until an S/MIME based wheel implementation is available. I'd love to be able to just accept the wheel format as it stands, but absent a compelling pro-JWS story in the PEP, it isn't going to happen :( I'd also hold off on a wheel downloader for now, that side of things is likely to be affected if/when distlib gets proposed for 3.4. > > Paul > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From barry at python.org Sun Feb 3 18:08:41 2013 From: barry at python.org (Barry Warsaw) Date: Sun, 3 Feb 2013 12:08:41 -0500 Subject: [Python-Dev] BDFL delegation for PEP 426 (PyPI metadata 1.3) In-Reply-To: References: Message-ID: <20130203120841.6f807f2c@limelight.wooz.org> On Feb 03, 2013, at 04:04 PM, Nick Coghlan wrote: >- someone else volunteers to be BDFL-Delegate for PEP 426 (MvL, perhaps?) On principle, I think it's a good idea to try to recruit another PEP czar. I'm not volunteering though, due to lack of time. Cheers, -Barry From merwok at netwok.org Sun Feb 3 19:33:23 2013 From: merwok at netwok.org (=?UTF-8?B?w4lyaWMgQXJhdWpv?=) Date: Sun, 03 Feb 2013 13:33:23 -0500 Subject: [Python-Dev] BDFL delegation for PEP 426 + distutils freeze In-Reply-To: <20130203134853.098aadaa@pitrou.net> References: <20130203122754.7745a1be@pitrou.net> <20130203134853.098aadaa@pitrou.net> Message-ID: <510EAD73.7090406@netwok.org> Hi, I don?t have a problem with Nick improving the PEP and deciding on its acceptance. Le 03/02/2013 07:48, Antoine Pitrou a ?crit : > On Sun, 3 Feb 2013 12:34:36 +0000 > Paul Moore wrote: >> So it's perfectly possible to use wheels right now, without the pip >> integration. But the pip developers don't want to integrate the wheel >> format just because it exists - they want the assurance that it's an >> accepted format supported by PEPs, hence the interest in getting the 3 >> wheel PEPs (of which the metadata PEP is the first) accepted. > > But it's an "accepted format" by whom? If we accept the PEP but refuse > to integrate support in the stdlib, it doesn't make much sense. There?s a chicken and egg problem: python-dev could have waited a year or two to see the wheel format being used and fixed/improved before deciding to do something in the stdlib, but the pip maintainers wanted an official blessing first. Accepting the wheel PEP would mean that python-dev and the community think wheels are an interesting way forward, and after some time needed to find out what fixes and improvements are needed, python-dev is ready to add stable, official support in the stdlib. (I know Daniel Holth is frustrated by the wait for what he sees as a simple format that?s been stable since last year, but it?s really not long for stdlib inclusion, and there *are* things to sort out.) > I vote for removing the "distutils is frozen" principle. I?ve also been thinking about that. There have been two exceptions to the freeze, for ABI flags in extension module names and for pycache directories. When the stable ABI was added and MvL wanted to change distutils (I don?t know to do what exactly), Tarek stood firm on the freeze and asked for any improvement to go into distutils2, and after MvL said that he would not contibute to an outside project, we merged d2 into the stdlib. Namespace packages did not impact distutils either. Now that we?ve removed packaging from the stdlib, we have two Python features that are not supported in the standard packaging system, and I agree that it is a bad thing for our users. I?d like to propose a reformulation of the freeze: - refactorings for the sake of cleanup are still shunned - fixes to really old bugs that have become the expected behavior are still avoided - fixes to follow OS changes are still allowed (we?ve had a number for Debian multiarch, Apple moving stuff around, Windows manifest options changes) - support for Python evolutions that involve totally new code, commands or setup parameters are now possible (this enables stable API support as well as a new bdist format) - behavior changes to track Python behavior changes are now possible (this enables recognizing namespace packages, unless we decide they need a new setup parameter) We?ll probably need to talk this over at PyCon (FYI I won?t be at the language summit but I?ll take part in the packaging mini-summit planned thanks to Nick). Regards From dholth at gmail.com Sun Feb 3 19:57:39 2013 From: dholth at gmail.com (Daniel Holth) Date: Sun, 3 Feb 2013 13:57:39 -0500 Subject: [Python-Dev] BDFL delegation for PEP 426 (PyPI metadata 1.3) In-Reply-To: <20130203154341.598b063c@pitrou.net> References: <20130203122754.7745a1be@pitrou.net> <20130203134853.098aadaa@pitrou.net> <20130203154341.598b063c@pitrou.net> Message-ID: My position is that these days distutils doesn't belong in the standard library any more than Django does. So I am mildly opposed to supporting it when you should be using better designed third party tools like Bento or setuptools. Wheel makes it possible for Python to get out of the build tool business. Just install your preferred tools with a concise bootstrap installer. On Sun, 3 Feb 2013 09:41:29 -0500 Daniel Holth wrote: > The neat thing about wheel is that you can install them without having the > software used to build them. So we might try to provide a very simple wheel > installer script with Python that did not even depend on DistUtils. You > would be able to install pip etc with that tool. There is no need to put > wheel directly into DistUtils. But you still need to *produce* wheels, don't you? It would be more useful for everyone if that facility was bundled with Python. Regards Antoine. _______________________________________________ Python-Dev mailing list Python-Dev at python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/dholth%40gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From merwok at netwok.org Sun Feb 3 20:06:57 2013 From: merwok at netwok.org (=?UTF-8?B?w4lyaWMgQXJhdWpv?=) Date: Sun, 03 Feb 2013 14:06:57 -0500 Subject: [Python-Dev] BDFL delegation for PEP 426 (PyPI metadata 1.3) In-Reply-To: References: <20130203122754.7745a1be@pitrou.net> <20130203134853.098aadaa@pitrou.net> <20130203154341.598b063c@pitrou.net> Message-ID: <510EB551.8050705@netwok.org> Hi, Le 03/02/2013 13:57, Daniel Holth a ?crit : > My position is that these days distutils doesn't belong in the standard > library any more than Django does. You can install anything you want, but first you need an installer. I think that a language needs packaging formats and basic build and install tools in its stdlib. You can use different tools for specialized needs or tastes, but I think years and years of frustration show that people need a basic thing working out of the box. > So I am mildly opposed to supporting it when you should be using better > designed third party tools like Bento Aside: can we keep the discussion civil instead of constantly bashing distutils and heaping praise on one tool? (It happened with easy_install before bento.) I?m the first one to recognize that distutils has deep flaws, but I respect the time and energy that many people have put into it. > Wheel makes it possible for Python to get out of the build tool > business. Just install your preferred tools with a concise bootstrap > installer. Looks like we agree that a basic tool able to bootstrap the packaging story is needed :) Regards From ralf at systemexit.de Sun Feb 3 20:08:48 2013 From: ralf at systemexit.de (Ralf Schmitt) Date: Sun, 03 Feb 2013 20:08:48 +0100 Subject: [Python-Dev] BDFL delegation for PEP 426 (PyPI metadata 1.3) In-Reply-To: (Daniel Holth's message of "Sun, 3 Feb 2013 13:57:39 -0500") References: <20130203122754.7745a1be@pitrou.net> <20130203134853.098aadaa@pitrou.net> <20130203154341.598b063c@pitrou.net> Message-ID: <8738xdb4fz.fsf@myhost.lan> Daniel Holth writes: > Wheel makes it possible for Python to get out of the build tool > business. Just install your preferred tools with a concise bootstrap > installer. If this is true, it would also have been possible with eggs, yet it didn't happen. Why do you think it will happen now or am I missing something? From chris.jerdonek at gmail.com Sun Feb 3 20:40:41 2013 From: chris.jerdonek at gmail.com (Chris Jerdonek) Date: Sun, 3 Feb 2013 11:40:41 -0800 Subject: [Python-Dev] BDFL delegation for PEP 426 + distutils freeze In-Reply-To: <510EAD73.7090406@netwok.org> References: <20130203122754.7745a1be@pitrou.net> <20130203134853.098aadaa@pitrou.net> <510EAD73.7090406@netwok.org> Message-ID: On Sun, Feb 3, 2013 at 10:33 AM, ?ric Araujo wrote: > Le 03/02/2013 07:48, Antoine Pitrou a ?crit : >> I vote for removing the "distutils is frozen" principle. > I?ve also been thinking about that. There have been two exceptions to > the freeze, for ABI flags in extension module names and for pycache > directories. When the stable ABI was added and MvL wanted to change > distutils (I don?t know to do what exactly), Tarek stood firm on the > freeze and asked for any improvement to go into distutils2, and after > MvL said that he would not contibute to an outside project, we merged d2 > into the stdlib. Namespace packages did not impact distutils either. > Now that we?ve removed packaging from the stdlib, we have two Python > features that are not supported in the standard packaging system, and I > agree that it is a bad thing for our users. > > I?d like to propose a reformulation of the freeze: This could be common knowledge, but is the current formulation of the freeze spelled out somewhere? --Chris From dholth at gmail.com Sun Feb 3 20:42:39 2013 From: dholth at gmail.com (Daniel Holth) Date: Sun, 3 Feb 2013 14:42:39 -0500 Subject: [Python-Dev] BDFL delegation for PEP 426 (PyPI metadata 1.3) In-Reply-To: <8738xdb4fz.fsf@myhost.lan> References: <20130203122754.7745a1be@pitrou.net> <20130203134853.098aadaa@pitrou.net> <20130203154341.598b063c@pitrou.net> <8738xdb4fz.fsf@myhost.lan> Message-ID: Bento is the only available packaging tool to heap praise onto and it is impressive. I am reacting to all the hate heaped on setup tools when I think the underlying DistUtils design is a big part of the problem. My feeling is that stdlib packaging tools should be for bootstrapping and reference, more like wsgiref than django. Wheel is very similar to egg but tries to solve the original setup tools sin of installing something you don't want (the entire setuptools build system) by omitting the runtime component. The result is a very reviewable couple hundred lines bootstrap installer or "unzip" in a pinch. Its also built upon Python standards that were not available when easy_install was introduced. On Feb 3, 2013 2:09 PM, "Ralf Schmitt" wrote: > Daniel Holth writes: > > > Wheel makes it possible for Python to get out of the build tool > > business. Just install your preferred tools with a concise bootstrap > > installer. > > If this is true, it would also have been possible with eggs, yet it > didn't happen. Why do you think it will happen now or am I missing > something? > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/dholth%40gmail.com > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hodgestar+pythondev at gmail.com Sun Feb 3 21:00:29 2013 From: hodgestar+pythondev at gmail.com (Simon Cross) Date: Sun, 3 Feb 2013 22:00:29 +0200 Subject: [Python-Dev] BDFL delegation for PEP 426 (PyPI metadata 1.3) In-Reply-To: References: <20130203122754.7745a1be@pitrou.net> <20130203134853.098aadaa@pitrou.net> <20130203154341.598b063c@pitrou.net> <8738xdb4fz.fsf@myhost.lan> Message-ID: On Sun, Feb 3, 2013 at 9:42 PM, Daniel Holth wrote: > Bento is the only available packaging tool to heap praise onto and it is > impressive. If Bento is cool, is there some way we can help it gain more traction in the Python ecosystem? Not necessarily by incorporating it into stdlib, but perhaps by officially sanctioning it in other ways (documentation, PyPI, perhaps getting some helpful hooks / tweaks to Python itself)? I don't know the answer to these questions, but if there is a good solution out there, it would be cool to throw our community weight behind it. Schiavo Simon From brian at python.org Sun Feb 3 21:07:21 2013 From: brian at python.org (Brian Curtin) Date: Sun, 3 Feb 2013 14:07:21 -0600 Subject: [Python-Dev] BDFL delegation for PEP 426 (PyPI metadata 1.3) In-Reply-To: References: <20130203122754.7745a1be@pitrou.net> <20130203134853.098aadaa@pitrou.net> <20130203154341.598b063c@pitrou.net> <8738xdb4fz.fsf@myhost.lan> Message-ID: On Sun, Feb 3, 2013 at 2:00 PM, Simon Cross wrote: > On Sun, Feb 3, 2013 at 9:42 PM, Daniel Holth wrote: >> Bento is the only available packaging tool to heap praise onto and it is >> impressive. > > If Bento is cool, is there some way we can help it gain more traction > in the Python ecosystem? Not necessarily by incorporating it into > stdlib, but perhaps by officially sanctioning it in other ways > (documentation, PyPI, perhaps getting some helpful hooks / tweaks to > Python itself)? > > I don't know the answer to these questions, but if there is a good > solution out there, it would be cool to throw our community weight > behind it. I don't think we, as in python-dev, should do this. If people want to start telling others to use bento on their own, that's fine. For the core team to get behind it would probably require a lot of work to safely stamp it as "the new way...that we don't actually have anything to do with" If python-dev officially says "hey, use bento", then it has all sorts of problems and ends up dying in 6 months, we will look like idiots. From qwcode at gmail.com Sun Feb 3 21:16:18 2013 From: qwcode at gmail.com (Marcus Smith) Date: Sun, 3 Feb 2013 12:16:18 -0800 Subject: [Python-Dev] BDFL delegation for PEP 426 (PyPI metadata 1.3) In-Reply-To: References: <20130203122754.7745a1be@pitrou.net> Message-ID: > The pip integration is basically to allow pip to find wheels on PyPI > or any local indexes you have, and to install them via the "pip > install" command. it also offers "pip wheel" for building wheels (using bdist_wheel) locally for your requirements, since wheels wouldn't be pervasive on PyPI for a while so, let's say you have a large app that has a requirements file, that ultimately installs 90 packages. run: pip wheel --wheel-dir=/my_wheels -r requirements.txt this will produce 90 wheel archives in a local directory, that you can then install against. Marcus -------------- next part -------------- An HTML attachment was scrubbed... URL: From pje at telecommunity.com Sun Feb 3 21:04:04 2013 From: pje at telecommunity.com (PJ Eby) Date: Sun, 3 Feb 2013 15:04:04 -0500 Subject: [Python-Dev] BDFL delegation for PEP 426 (PyPI metadata 1.3) In-Reply-To: References: <20130203122754.7745a1be@pitrou.net> <20130203134853.098aadaa@pitrou.net> Message-ID: On Sun, Feb 3, 2013 at 8:08 AM, Nick Coghlan wrote: > The rationale for the distutils freeze is "don't break setuptools". > That rationale still holds. IIRC, the historical issue that triggered the freeze was not that the distutils refactoring broke setuptools, but that it did so in what was supposed to be a bugfix-only release of Python, via a change to what appeared to be a public/documented method. This rationale doesn't apply to major/minor releases of Python, and the freeze was only supposed to apply to 2.x in any event. (IIRC, some of the breakage was the result of backporting 3.x distutils changes to 2.x.) Anyway, adding features or refactoring distutils is not the problem: doing it in bugfix releases is. With normal releases, third-party extenders of distutils at least have the opportunity to test and issue updates in a timely fashion. From mjainit at gmail.com Sun Feb 3 21:51:41 2013 From: mjainit at gmail.com (Jainit Purohit) Date: Mon, 4 Feb 2013 02:21:41 +0530 Subject: [Python-Dev] Interested in GSoC 2013 Message-ID: Hi, I'm Jainit and I'm planning to apply for GSoC 2013 for the PSF. I was also part of GSoC 2012 in interface ecology lab, Texas A&M university. I just gone though Python developer's guide and how to become core contributor document. And I just compiled CPython on my machine. Since I'm new to this community, Can anyone assign me some task/issues/bug to work on to get the better idea of how things works. I would also like to know if any of you have any ideas which can be implemented this summer. Thanks!, Jainit -------------- next part -------------- An HTML attachment was scrubbed... URL: From dholth at gmail.com Sun Feb 3 22:14:17 2013 From: dholth at gmail.com (Daniel Holth) Date: Sun, 3 Feb 2013 16:14:17 -0500 Subject: [Python-Dev] BDFL delegation for PEP 426 + distutils freeze In-Reply-To: References: <20130203122754.7745a1be@pitrou.net> <20130203134853.098aadaa@pitrou.net> <510EAD73.7090406@netwok.org> Message-ID: I did think that updating distutils to have basic support for the packaging PEPs was a decent idea, but then it wound up being more or less rewritten entirely like I've been cajoled into doing with PEP 426 (Metadata). I don't know whether distutils(1) can survive the minimum changes required for wheel (.dist-info folders and a bdist_wheel command) without breaking a significant number of its clients or not. History suggests no. It may still be worth a shot. bdist_wheel started out as a distutils2 plugin but it was inconvenient to develop it inside the CPython repository and it was not practical to convert projects over from setuptools to distutils2. The pypi metadata has had a standardlib generator for a long time without having anything in the standard library to make use of the metadata. Wheels are just .zip files of one package's share of the site-packages directory. Why not support them asymmetrically. I value the PEP and consensus for the format more than the implementation. One idea that's sure to be controversial would be to include a bootstrap installer as a script without committing to a Python API (just a command-line API). Since Python 2.6, wheel's own wheel archive can be used to install wheels: $ python wheel-0.14.0-py2.py3-none-any.whl/wheel -h usage: wheel [-h] {keygen,sign,verify,unpack,install,convert,help} ... On Sun, Feb 3, 2013 at 2:40 PM, Chris Jerdonek wrote: > On Sun, Feb 3, 2013 at 10:33 AM, ?ric Araujo wrote: > > Le 03/02/2013 07:48, Antoine Pitrou a ?crit : > >> I vote for removing the "distutils is frozen" principle. > > I?ve also been thinking about that. There have been two exceptions to > > the freeze, for ABI flags in extension module names and for pycache > > directories. When the stable ABI was added and MvL wanted to change > > distutils (I don?t know to do what exactly), Tarek stood firm on the > > freeze and asked for any improvement to go into distutils2, and after > > MvL said that he would not contibute to an outside project, we merged d2 > > into the stdlib. Namespace packages did not impact distutils either. > > Now that we?ve removed packaging from the stdlib, we have two Python > > features that are not supported in the standard packaging system, and I > > agree that it is a bad thing for our users. > > > > I?d like to propose a reformulation of the freeze: > > This could be common knowledge, but is the current formulation of the > freeze spelled out somewhere? > > --Chris > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/dholth%40gmail.com > -------------- next part -------------- An HTML attachment was scrubbed... URL: From greg at krypto.org Sun Feb 3 22:20:31 2013 From: greg at krypto.org (Gregory P. Smith) Date: Sun, 3 Feb 2013 13:20:31 -0800 Subject: [Python-Dev] cpython (2.7): - Issue #17086: Backport the patches from the 3.3 branch to cross-build In-Reply-To: <20130201081425.39120441@pitrou.net> References: <3YxxXq44T7zQ28@mail.python.org> <20130201005001.5f73e15b@pitrou.net> <20130201081425.39120441@pitrou.net> Message-ID: On Thu, Jan 31, 2013 at 11:14 PM, Antoine Pitrou wrote: > On Fri, 1 Feb 2013 11:00:24 +1000 > Nick Coghlan wrote: > > On Fri, Feb 1, 2013 at 9:50 AM, Antoine Pitrou > wrote: > > > On Thu, 31 Jan 2013 23:52:27 +0100 (CET) > > > matthias.klose wrote: > > >> http://hg.python.org/cpython/rev/8ee6d96a1019 > > >> changeset: 81859:8ee6d96a1019 > > >> branch: 2.7 > > >> parent: 81855:df9f8feb7444 > > >> user: doko at python.org > > >> date: Thu Jan 31 23:52:03 2013 +0100 > > >> summary: > > >> - Issue #17086: Backport the patches from the 3.3 branch to > cross-build > > >> the package. > > > > > > You aren't supposed to add new features to bugfix branches. Did you > > > have a specific reason to do this? > > > > One of the reasons for the long maintenance period on 2.7 is to keep > > it building as the underlying platforms change. With the rise of ARM > > systems, being able to reliably cross-build Python 2.7 for ARM from an > > x86_64 system is fairly important. > > I would like to see a better argument for this. The rise of ARM systems > is the rise of ARM systems powerful enough to build Python without > cross-compiling (which is why we *now* have ARM buildbots). > The very small ARM systems which need cross-compiling have existed for > decades. > It is quite common for developers to build a single code base on a single workstation targeting a plethora of platforms all at once. Requiring native systems with self hosting tool-chains for builds is a non-starter as those often don't exist. Making Python 2.7's configure+makefiles easier to cross compile out of the box is a good thing. Side note: we really need a cross compiling build-bot host + target system or we'll inevitably break this. > > That said, as a procedural point for build related new features in > > 2.7, I agree they should be proposed, with an explicit rationale, on > > python-dev before proceeding with the commit. > > I think this huge changeset should be reverted. It's a complete failure > in terms of procedure and following the rules. "Just because it can be > useful" is not a good enough reason to violate our release model > without even asking. > That's up to the 2.7 release manager. Yes, this could have been done better by asking first. But IMNSHO I'd prefer to see it stay in. -gps > > Regards > > Antoine. > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/greg%40krypto.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: From vinay_sajip at yahoo.co.uk Sun Feb 3 22:52:08 2013 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Sun, 3 Feb 2013 21:52:08 +0000 (UTC) Subject: [Python-Dev] BDFL delegation for PEP 426 (PyPI metadata 1.3) References: <20130203122754.7745a1be@pitrou.net> Message-ID: Paul Moore gmail.com> writes: > So it's perfectly possible to use wheels right now, without the pip > integration. But the pip developers don't want to integrate the wheel > format just because it exists - they want the assurance that it's an > accepted format supported by PEPs, hence the interest in getting the 3 > wheel PEPs (of which the metadata PEP is the first) accepted. Likewise, I will look at the possibility of providing wheel support in distlib, once it has been accepted as a standard and the open issues (such as signature scheme) have been resolved. Regards, Vinay Sajip From dholth at gmail.com Sun Feb 3 23:13:24 2013 From: dholth at gmail.com (Daniel Holth) Date: Sun, 3 Feb 2013 17:13:24 -0500 Subject: [Python-Dev] BDFL delegation for PEP 426 (PyPI metadata 1.3) In-Reply-To: References: <20130203122754.7745a1be@pitrou.net> Message-ID: They can be signed with pypi detached signatures already. It works now exactly as for sdist. The innovation was supposed to be in convenience for the signer, in allowing keys to be trusted per package and for a list of dependencies and the expected signing keys to be shared easily. Does anyone have a concise pure py3 s/mime implementation? On Feb 3, 2013 4:52 PM, "Vinay Sajip" wrote: > Paul Moore gmail.com> writes: > > > So it's perfectly possible to use wheels right now, without the pip > > integration. But the pip developers don't want to integrate the wheel > > format just because it exists - they want the assurance that it's an > > accepted format supported by PEPs, hence the interest in getting the 3 > > wheel PEPs (of which the metadata PEP is the first) accepted. > > Likewise, I will look at the possibility of providing wheel support in > distlib, > once it has been accepted as a standard and the open issues (such as > signature > scheme) have been resolved. > > Regards, > > Vinay Sajip > > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/dholth%40gmail.com > -------------- next part -------------- An HTML attachment was scrubbed... URL: From vinay_sajip at yahoo.co.uk Sun Feb 3 23:16:28 2013 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Sun, 3 Feb 2013 22:16:28 +0000 (UTC) Subject: [Python-Dev] BDFL delegation for PEP 426 (PyPI metadata 1.3) References: <20130203122754.7745a1be@pitrou.net> <20130203134853.098aadaa@pitrou.net> <20130203154341.598b063c@pitrou.net> <8738xdb4fz.fsf@myhost.lan> Message-ID: Daniel Holth gmail.com> writes: > Bento is the only available packaging tool to heap praise onto and it is > impressive. I am reacting to all the hate heaped on setup tools when I think > the underlying DistUtils design is a big part of the problem. My feeling is > that stdlib packaging tools should be for bootstrapping and reference, more > like wsgiref than django. Bento is interesting, but I wouldn't jump to heap praise onto it. Apart from the somewhat idiosyncratic source style, David Cournapeau himself points to what he regards as weaknesses in it[1]. It is no doubt a better build tool than distutils for NumPy/SciPy users, and that's great. However, from what I could see, the documentation is not very detailed, making it hard to evaluate, and much of the build internals seem to be in some other tool called yaku, for which I couldn't find much documentation. When I see a sentiment suggesting that "Python should get out of the build tools business" (paraphrasing what Daniel said, not quoting him), I can certainly see why the distutils compiler infrastructure would make one think that, if it was impossible to come up with something better. However, I do feel there is room for a battery included in this area, as long as it is simple and easier to build on than distutils has been. Although I have not started to add anything in this area to distlib, I have spent some time thinking about the problems, and I have some experimental work in mind. I don't believe that "hate" is the right word when talking about why setuptools isn't loved more by python-dev. Since it was extending the flawed distutils design, it doesn't make sense to have it getting any closer to the stdlib, and it could perhaps be argued that distutils2/packaging also failed to make sufficient progress for the same reason. Certainly, with distlib I've tried to incorporate a lot of the setuptools functionality which developers find useful - in-package data, package exports (entry points), wrapping callables with scripts and so on. Regards, Vinay Sajip [1] http://bento.readthedocs.org/en/latest/faq.html#why-shouldn-t-i-use-bento From hodgestar+pythondev at gmail.com Sun Feb 3 23:23:40 2013 From: hodgestar+pythondev at gmail.com (Simon Cross) Date: Mon, 4 Feb 2013 00:23:40 +0200 Subject: [Python-Dev] BDFL delegation for PEP 426 (PyPI metadata 1.3) In-Reply-To: References: <20130203122754.7745a1be@pitrou.net> <20130203134853.098aadaa@pitrou.net> <20130203154341.598b063c@pitrou.net> <8738xdb4fz.fsf@myhost.lan> Message-ID: On Mon, Feb 4, 2013 at 12:16 AM, Vinay Sajip wrote: > Bento is interesting, but I wouldn't jump to heap praise onto it. Apart from the > somewhat idiosyncratic source style, David Cournapeau himself points to what he > regards as weaknesses in it[1]. For the record, all the reasons listed at [1] appear trivial. [1] http://bento.readthedocs.org/en/latest/faq.html#why-shouldn-t-i-use-bento Schiavo Simon From vinay_sajip at yahoo.co.uk Sun Feb 3 23:24:57 2013 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Sun, 3 Feb 2013 22:24:57 +0000 (UTC) Subject: [Python-Dev] BDFL delegation for PEP 426 (PyPI metadata 1.3) References: <20130203122754.7745a1be@pitrou.net> <20130203134853.098aadaa@pitrou.net> <20130203154341.598b063c@pitrou.net> <510EB551.8050705@netwok.org> Message-ID: ?ric Araujo netwok.org> writes: > Looks like we agree that a basic tool able to bootstrap the packaging > story is needed :) Agreed. Just because distutils can't easily/reliably build things that are better built with SCons/WAF/tup/whatever, doesn't mean that we shouldn't have the ability to build pure-Python distributions and distributions including C libs and extensions, with the ability to extend easily by third-party tools. It just needs to be done in a way which is easy to build on, so the included battery stays small and simple. Easier said than done, I know :-) Regards, Vinay Sajip From vinay_sajip at yahoo.co.uk Sun Feb 3 23:34:37 2013 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Sun, 3 Feb 2013 22:34:37 +0000 (UTC) Subject: [Python-Dev] BDFL delegation for PEP 426 (PyPI metadata 1.3) References: <20130203122754.7745a1be@pitrou.net> <20130203134853.098aadaa@pitrou.net> <20130203154341.598b063c@pitrou.net> <8738xdb4fz.fsf@myhost.lan> Message-ID: Simon Cross gmail.com> writes: > For the record, all the reasons listed at [1] appear trivial. In Bento's author's own words - "Weak documentation", "Mediocre code quality", "at a lower level, a lot of code leaves to be desired" may be trivial if David is just being self-deprecating, but what if he isn't? Or perhaps that part of the page is out of date, and needs updating? I can certainly agree with the "Weak documentation" part of the assessment, but this makes it hard to assess the package as a whole. Note that I'm not sniping - writing good documentation is hard work. Regards, Vinay Sajip From dholth at gmail.com Sun Feb 3 23:48:47 2013 From: dholth at gmail.com (Daniel Holth) Date: Sun, 3 Feb 2013 17:48:47 -0500 Subject: [Python-Dev] BDFL delegation for PEP 426 (PyPI metadata 1.3) In-Reply-To: References: <20130203122754.7745a1be@pitrou.net> <20130203134853.098aadaa@pitrou.net> <20130203154341.598b063c@pitrou.net> <8738xdb4fz.fsf@myhost.lan> Message-ID: He is being self deprecating. Its also true that python dev can't recommend bento wholesale. That is fine with me. On Feb 3, 2013 5:36 PM, "Vinay Sajip" wrote: > Simon Cross gmail.com> writes: > > > For the record, all the reasons listed at [1] appear trivial. > > In Bento's author's own words - "Weak documentation", "Mediocre code > quality", > "at a lower level, a lot of code leaves to be desired" may be trivial if > David > is just being self-deprecating, but what if he isn't? Or perhaps that part > of > the page is out of date, and needs updating? I can certainly agree with the > "Weak documentation" part of the assessment, but this makes it hard to > assess > the package as a whole. Note that I'm not sniping - writing good > documentation > is hard work. > > Regards, > > Vinay Sajip > > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/dholth%40gmail.com > -------------- next part -------------- An HTML attachment was scrubbed... URL: From nadeem.vawda at gmail.com Mon Feb 4 00:08:28 2013 From: nadeem.vawda at gmail.com (Nadeem Vawda) Date: Mon, 4 Feb 2013 00:08:28 +0100 Subject: [Python-Dev] 2.7.4 In-Reply-To: References: Message-ID: Just to clarify, the release branch hasn't been created yet, correct? - Nadeem On Sun, Feb 3, 2013 at 3:38 PM, Benjamin Peterson wrote: > 2013/2/3 Serhiy Storchaka : > > There are crashers for which patches were proposed but do not reviewed > yet: > > > > Issue #6083: Reference counting bug in PyArg_ParseTuple and > > PyArg_ParseTupleAndKeywords. > > > > Issue #7358: cStringIO not 64-bit safe. > > > > Issue #16137: Using time.asctime() with an array with negative tm_hour > > causes Python Crash. > > > > Issue #16686: audioop overflow issues. > > > > #8865 is reviewed but not committed. > > Thanks. In the future, this should be raised to "release blocker" > priority to get my attention. > > -- > Regards, > Benjamin > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/nadeem.vawda%40gmail.com > -------------- next part -------------- An HTML attachment was scrubbed... URL: From cournape at gmail.com Mon Feb 4 00:20:05 2013 From: cournape at gmail.com (David Cournapeau) Date: Sun, 3 Feb 2013 23:20:05 +0000 Subject: [Python-Dev] BDFL delegation for PEP 426 (PyPI metadata 1.3) In-Reply-To: References: <20130203122754.7745a1be@pitrou.net> <20130203134853.098aadaa@pitrou.net> <20130203154341.598b063c@pitrou.net> <8738xdb4fz.fsf@myhost.lan> Message-ID: On Sun, Feb 3, 2013 at 10:34 PM, Vinay Sajip wrote: > Simon Cross gmail.com> writes: > >> For the record, all the reasons listed at [1] appear trivial. > > In Bento's author's own words - "Weak documentation", "Mediocre code quality", > "at a lower level, a lot of code leaves to be desired" may be trivial if David > is just being self-deprecating, but what if he isn't? Or perhaps that part of > the page is out of date, and needs updating? I can certainly agree with the > "Weak documentation" part of the assessment, but this makes it hard to assess > the package as a whole. Note that I'm not sniping - writing good documentation > is hard work. You are putting the words out of the context in which those were written: it is stated that the focus is on the general architecture and low-coupling which are the main issues I saw with distutils. Bento is designed to use multiple build backends (it can use distutils to build C extensions, or waf, the latter being how numpy/scipy is being built with bento). FWIW, I am not in favor of having bento blessed (or any other tool for that matter). The fundamental mistake of the previous attempts at packaging has been to formalize too early, or impose de-facto standards without much specification. That's why wheel and similar efforts are the way forward: they tackle a narrow but well defined sub-problem of packaging. Thus, they can be reused by other libraries to build higher abstractions. They are also less prone to the 'fatigue' that often arise in packaging efforts. David From brett at python.org Mon Feb 4 00:36:56 2013 From: brett at python.org (Brett Cannon) Date: Sun, 3 Feb 2013 18:36:56 -0500 Subject: [Python-Dev] 2.7.4 In-Reply-To: References: Message-ID: On Sun, Feb 3, 2013 at 6:08 PM, Nadeem Vawda wrote: > Just to clarify, the release branch hasn't been created yet, correct? > > It's by tag and no: http://hg.python.org/cpython/tags -Brett > - Nadeem > > > On Sun, Feb 3, 2013 at 3:38 PM, Benjamin Peterson wrote: > >> 2013/2/3 Serhiy Storchaka : >> > There are crashers for which patches were proposed but do not reviewed >> yet: >> > >> > Issue #6083: Reference counting bug in PyArg_ParseTuple and >> > PyArg_ParseTupleAndKeywords. >> > >> > Issue #7358: cStringIO not 64-bit safe. >> > >> > Issue #16137: Using time.asctime() with an array with negative tm_hour >> > causes Python Crash. >> > >> > Issue #16686: audioop overflow issues. >> > >> > #8865 is reviewed but not committed. >> >> Thanks. In the future, this should be raised to "release blocker" >> priority to get my attention. >> >> -- >> Regards, >> Benjamin >> _______________________________________________ >> Python-Dev mailing list >> Python-Dev at python.org >> http://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe: >> http://mail.python.org/mailman/options/python-dev/nadeem.vawda%40gmail.com >> > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/brett%40python.org > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Mon Feb 4 00:55:22 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 4 Feb 2013 09:55:22 +1000 Subject: [Python-Dev] BDFL delegation for PEP 426 (PyPI metadata 1.3) In-Reply-To: References: <20130203122754.7745a1be@pitrou.net> <20130203134853.098aadaa@pitrou.net> <20130203154341.598b063c@pitrou.net> <8738xdb4fz.fsf@myhost.lan> Message-ID: On 4 Feb 2013 09:22, "David Cournapeau" wrote: > > On Sun, Feb 3, 2013 at 10:34 PM, Vinay Sajip wrote: > > Simon Cross gmail.com> writes: > > > >> For the record, all the reasons listed at [1] appear trivial. > > > > In Bento's author's own words - "Weak documentation", "Mediocre code quality", > > "at a lower level, a lot of code leaves to be desired" may be trivial if David > > is just being self-deprecating, but what if he isn't? Or perhaps that part of > > the page is out of date, and needs updating? I can certainly agree with the > > "Weak documentation" part of the assessment, but this makes it hard to assess > > the package as a whole. Note that I'm not sniping - writing good documentation > > is hard work. > > You are putting the words out of the context in which those were > written: it is stated that the focus is on the general architecture > and low-coupling which are the main issues I saw with distutils. Bento > is designed to use multiple build backends (it can use distutils to > build C extensions, or waf, the latter being how numpy/scipy is being > built with bento). > > FWIW, I am not in favor of having bento blessed (or any other tool for > that matter). The fundamental mistake of the previous attempts at > packaging has been to formalize too early, or impose de-facto > standards without much specification. That's why wheel and similar > efforts are the way forward: they tackle a narrow but well defined > sub-problem of packaging. Thus, they can be reused by other libraries > to build higher abstractions. They are also less prone to the > 'fatigue' that often arise in packaging efforts. Another couple of key pieces are "Setup-Requires-Dist" and the extension mechanism in PEP 426 - they're intended to make it easier to communicate the use of third party tools for building, packaging and installation, as well as making it easier to provide additional metadata for the benefit of those tools. Cheers, Nick. > > David > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From benjamin at python.org Mon Feb 4 01:11:36 2013 From: benjamin at python.org (Benjamin Peterson) Date: Sun, 3 Feb 2013 19:11:36 -0500 Subject: [Python-Dev] update on 2.7.4 Message-ID: As you may have noticed, no 2.7.4 rc has been created yet. Yesterday, the buildbots were all red, and release blocker issues had to be dealt with. Today, I was not as availabIe and people were fixing important-looking crashers. In general, there seems to have been a lot more last-minute scrambling around than usual for a bugfix release. I'm afraid I'm still going to have to delay longer to see if we can get a few security patches in. It could be next week. Benjamin From greg at krypto.org Mon Feb 4 02:50:13 2013 From: greg at krypto.org (Gregory P. Smith) Date: Sun, 3 Feb 2013 17:50:13 -0800 Subject: [Python-Dev] Interested in GSoC 2013 In-Reply-To: References: Message-ID: First, welcome to Python. For people just starting out contributing we have setup a core-mentorship mailing list ideally suited for this type of question. http://mail.python.org/mailman/listinfo/core-mentorship general tip: look for open issues marked with the 'easy' on bugs.python.org. On Sun, Feb 3, 2013 at 12:51 PM, Jainit Purohit wrote: > Hi, > > > I'm Jainit and I'm planning to apply for GSoC 2013 > for the PSF. I was also part of GSoC 2012 in interface ecology lab, Texas > A&M university. I just gone though Python developer's guide and how to > become core contributor document. And I just compiled CPython on my > machine. Since I'm new to this community, Can anyone assign me some > task/issues/bug to work on to get the better idea of how things works. I > would also like to know if any of you have any ideas which can be > implemented this summer. > > > Thanks!, > > Jainit > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/greg%40krypto.org > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From vinay_sajip at yahoo.co.uk Mon Feb 4 03:01:02 2013 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Mon, 4 Feb 2013 02:01:02 +0000 (UTC) Subject: [Python-Dev] BDFL delegation for PEP 426 (PyPI metadata 1.3) References: <20130203122754.7745a1be@pitrou.net> <20130203134853.098aadaa@pitrou.net> <20130203154341.598b063c@pitrou.net> <8738xdb4fz.fsf@myhost.lan> Message-ID: David Cournapeau gmail.com> writes: > You are putting the words out of the context in which those were > written: it is stated that the focus is on the general architecture OK, no offence was meant. Thanks for the clarification. Regards, Vinay Sajip From eliben at gmail.com Mon Feb 4 05:54:31 2013 From: eliben at gmail.com (Eli Bendersky) Date: Sun, 3 Feb 2013 20:54:31 -0800 Subject: [Python-Dev] update on 2.7.4 In-Reply-To: References: Message-ID: On Sun, Feb 3, 2013 at 4:11 PM, Benjamin Peterson wrote: > As you may have noticed, no 2.7.4 rc has been created yet. Yesterday, > the buildbots were all red, and release blocker issues had to be dealt > with. Today, I was not as availabIe and people were fixing > important-looking crashers. In general, there seems to have been a lot > more last-minute scrambling around than usual for a bugfix release. > > I'm afraid I'm still going to have to delay longer to see if we can > get a few security patches in. It could be next week. > Thanks for the update, Benjamin. Given the recent discussions and events, I would feel even safer if more than a week passed. I don't see how waiting a bit longer is going to hurt anyone. At this point in 2.7's life, stability is perhaps the most important thing. Perhaps this is the right time to specifically ask all contributors to consider what may be blocking such a release - any bugs you recall seeing that appear important and must be fixed? Eli -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Mon Feb 4 10:27:47 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 4 Feb 2013 10:27:47 +0100 Subject: [Python-Dev] update on 2.7.4 References: Message-ID: <20130204102747.53aa84f1@pitrou.net> Le Sun, 3 Feb 2013 19:11:36 -0500, Benjamin Peterson a ?crit : > As you may have noticed, no 2.7.4 rc has been created yet. Yesterday, > the buildbots were all red, and release blocker issues had to be dealt > with. Today, I was not as availabIe and people were fixing > important-looking crashers. In general, there seems to have been a lot > more last-minute scrambling around than usual for a bugfix release. I don't know for others, but I had forgotten you were going to do a release. Sorry. Regards Antoine. From cournape at gmail.com Mon Feb 4 10:42:38 2013 From: cournape at gmail.com (David Cournapeau) Date: Mon, 4 Feb 2013 09:42:38 +0000 Subject: [Python-Dev] BDFL delegation for PEP 426 (PyPI metadata 1.3) In-Reply-To: References: <20130203122754.7745a1be@pitrou.net> <20130203134853.098aadaa@pitrou.net> <20130203154341.598b063c@pitrou.net> <8738xdb4fz.fsf@myhost.lan> Message-ID: On Mon, Feb 4, 2013 at 2:01 AM, Vinay Sajip wrote: > David Cournapeau gmail.com> writes: > >> You are putting the words out of the context in which those were >> written: it is stated that the focus is on the general architecture > > OK, no offence was meant. Thanks for the clarification. No worries, none taken :) David From g.brandl at gmx.net Mon Feb 4 15:51:21 2013 From: g.brandl at gmx.net (Georg Brandl) Date: Mon, 04 Feb 2013 15:51:21 +0100 Subject: [Python-Dev] update on 2.7.4 In-Reply-To: References: Message-ID: Am 04.02.2013 01:11, schrieb Benjamin Peterson: > As you may have noticed, no 2.7.4 rc has been created yet. Yesterday, > the buildbots were all red, and release blocker issues had to be dealt > with. Today, I was not as availabIe and people were fixing > important-looking crashers. In general, there seems to have been a lot > more last-minute scrambling around than usual for a bugfix release. > > I'm afraid I'm still going to have to delay longer to see if we can > get a few security patches in. It could be next week. FWIW, the same goes for 3.2.4. Georg From brett at python.org Mon Feb 4 16:48:07 2013 From: brett at python.org (Brett Cannon) Date: Mon, 4 Feb 2013 10:48:07 -0500 Subject: [Python-Dev] Interested in GSoC 2013 In-Reply-To: References: Message-ID: On Sun, Feb 3, 2013 at 8:50 PM, Gregory P. Smith wrote: > First, welcome to Python. > > For people just starting out contributing we have setup a core-mentorship > mailing list ideally suited for this type of question. > http://mail.python.org/mailman/listinfo/core-mentorship > > general tip: look for open issues marked with the 'easy' on > bugs.python.org. > Actually there is an entire section to the devguide dedicated to helping you find something to do: http://docs.python.org/devguide/#contributing. Handling "easy" issues are part of the advanced section. =) -Brett > > > On Sun, Feb 3, 2013 at 12:51 PM, Jainit Purohit wrote: > >> Hi, >> >> >> I'm Jainit and I'm planning to apply for GSoC 2013 >> for the PSF. I was also part of GSoC 2012 in interface ecology lab, Texas >> A&M university. I just gone though Python developer's guide and how to >> become core contributor document. And I just compiled CPython on my >> machine. Since I'm new to this community, Can anyone assign me some >> task/issues/bug to work on to get the better idea of how things works. I >> would also like to know if any of you have any ideas which can be >> implemented this summer. >> >> >> Thanks!, >> >> Jainit >> >> _______________________________________________ >> Python-Dev mailing list >> Python-Dev at python.org >> http://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe: >> http://mail.python.org/mailman/options/python-dev/greg%40krypto.org >> >> > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/brett%40python.org > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From benjamin at python.org Mon Feb 4 17:23:49 2013 From: benjamin at python.org (Benjamin Peterson) Date: Mon, 4 Feb 2013 11:23:49 -0500 Subject: [Python-Dev] update on 2.7.4 In-Reply-To: References: Message-ID: 2013/2/3 Eli Bendersky : > > On Sun, Feb 3, 2013 at 4:11 PM, Benjamin Peterson > wrote: >> >> As you may have noticed, no 2.7.4 rc has been created yet. Yesterday, >> the buildbots were all red, and release blocker issues had to be dealt >> with. Today, I was not as availabIe and people were fixing >> important-looking crashers. In general, there seems to have been a lot >> more last-minute scrambling around than usual for a bugfix release. >> >> I'm afraid I'm still going to have to delay longer to see if we can >> get a few security patches in. It could be next week. > > > Thanks for the update, Benjamin. Given the recent discussions and events, I > would feel even safer if more than a week passed. I don't see how waiting a > bit longer is going to hurt anyone. At this point in 2.7's life, stability > is perhaps the most important thing. > > Perhaps this is the right time to specifically ask all contributors to > consider what may be blocking such a release - any bugs you recall seeing > that appear important and must be fixed? Friendly reminder: Please set such bugs to the "release blocker" priority. Otherwise, I will not see them. -- Regards, Benjamin From mal at egenix.com Mon Feb 4 19:30:03 2013 From: mal at egenix.com (M.-A. Lemburg) Date: Mon, 04 Feb 2013 19:30:03 +0100 Subject: [Python-Dev] BDFL delegation for PEP 426 + distutils freeze In-Reply-To: <510EAD73.7090406@netwok.org> References: <20130203122754.7745a1be@pitrou.net> <20130203134853.098aadaa@pitrou.net> <510EAD73.7090406@netwok.org> Message-ID: <510FFE2B.8010004@egenix.com> On 03.02.2013 19:33, ?ric Araujo wrote: >> I vote for removing the "distutils is frozen" principle. > I?ve also been thinking about that. There have been two exceptions to > the freeze, for ABI flags in extension module names and for pycache > directories. When the stable ABI was added and MvL wanted to change > distutils (I don?t know to do what exactly), Tarek stood firm on the > freeze and asked for any improvement to go into distutils2, and after > MvL said that he would not contibute to an outside project, we merged d2 > into the stdlib. Namespace packages did not impact distutils either. > Now that we?ve removed packaging from the stdlib, we have two Python > features that are not supported in the standard packaging system, and I > agree that it is a bad thing for our users. > > I?d like to propose a reformulation of the freeze: > - refactorings for the sake of cleanup are still shunned > - fixes to really old bugs that have become the expected behavior are > still avoided > - fixes to follow OS changes are still allowed (we?ve had a number for > Debian multiarch, Apple moving stuff around, Windows manifest options > changes) > - support for Python evolutions that involve totally new code, commands > or setup parameters are now possible (this enables stable API support > as well as a new bdist format) > - behavior changes to track Python behavior changes are now possible > (this enables recognizing namespace packages, unless we decide they > need a new setup parameter) > > We?ll probably need to talk this over at PyCon (FYI I won?t be at the > language summit but I?ll take part in the packaging mini-summit planned > thanks to Nick). +1 on lifting the freeze from me. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Feb 04 2013) >>> Python Projects, Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From doko at ubuntu.com Mon Feb 4 21:04:39 2013 From: doko at ubuntu.com (Matthias Klose) Date: Mon, 04 Feb 2013 21:04:39 +0100 Subject: [Python-Dev] cpython (2.7): - Issue #17086: Backport the patches from the 3.3 branch to cross-build In-Reply-To: References: <3YxxXq44T7zQ28@mail.python.org> <20130201005001.5f73e15b@pitrou.net> <20130201081425.39120441@pitrou.net> Message-ID: <51101457.8090505@ubuntu.com> Am 03.02.2013 22:20, schrieb Gregory P. Smith: > On Thu, Jan 31, 2013 at 11:14 PM, Antoine Pitrou wrote: > >> On Fri, 1 Feb 2013 11:00:24 +1000 >> Nick Coghlan wrote: >>> On Fri, Feb 1, 2013 at 9:50 AM, Antoine Pitrou >> wrote: >>>> On Thu, 31 Jan 2013 23:52:27 +0100 (CET) >>>> matthias.klose wrote: >>>>> http://hg.python.org/cpython/rev/8ee6d96a1019 >>>>> changeset: 81859:8ee6d96a1019 >>>>> branch: 2.7 >>>>> parent: 81855:df9f8feb7444 >>>>> user: doko at python.org >>>>> date: Thu Jan 31 23:52:03 2013 +0100 >>>>> summary: >>>>> - Issue #17086: Backport the patches from the 3.3 branch to >> cross-build >>>>> the package. >>>> >>>> You aren't supposed to add new features to bugfix branches. Did you >>>> have a specific reason to do this? >>> >>> One of the reasons for the long maintenance period on 2.7 is to keep >>> it building as the underlying platforms change. With the rise of ARM >>> systems, being able to reliably cross-build Python 2.7 for ARM from an >>> x86_64 system is fairly important. >> >> I would like to see a better argument for this. The rise of ARM systems >> is the rise of ARM systems powerful enough to build Python without >> cross-compiling (which is why we *now* have ARM buildbots). >> The very small ARM systems which need cross-compiling have existed for >> decades. >> > > It is quite common for developers to build a single code base on a single > workstation targeting a plethora of platforms all at once. Requiring native > systems with self hosting tool-chains for builds is a non-starter as those > often don't exist. Making Python 2.7's configure+makefiles easier to cross > compile out of the box is a good thing. > > Side note: we really need a cross compiling build-bot host + target system > or we'll inevitably break this. > > >>> That said, as a procedural point for build related new features in >>> 2.7, I agree they should be proposed, with an explicit rationale, on >>> python-dev before proceeding with the commit. >> >> I think this huge changeset should be reverted. It's a complete failure >> in terms of procedure and following the rules. "Just because it can be >> useful" is not a good enough reason to violate our release model >> without even asking. >> > > That's up to the 2.7 release manager. Yes, this could have been done > better by asking first. But IMNSHO I'd prefer to see it stay in. I filed Issue #17086, and got feedback from the release manager, which I maybe interpreted too easily as not objecting. And it looked like a nice opportunity to get this into a release (hoping not to listen to another PyCon talk how to hack cross-builds). Even for the trunk, getting feed-back for cross-build related issues is difficult. Maybe reviewers are turned off by impolite comments in some of the cross-build related issues in the bug tracker, but that doesn't explain that you don't get feedback for most of the cross-build issues. So what I do understand, build-related issues like an arm64 or mingw32 port are ok for 2.7, if they are stable on the trunk, and communicated on python-dev? I'll see that I setup a cross buildd building in a cross-build environment and testing in a chroot with qemu installed. I hope that the buildd setup is able to support this. Talking about the release model, yes I think there are some issues: - the 2.7 branch is the only branch which doesn't have expected release dates on the calendar. And from a distributor/vendor point of view, I think yearly releases are too seldom. Such a release could end up only up to 24 months later in a (linux) distribution. - there were way too may regressions checked in on at least the 2.7 branch. Is it just our vcs merge model that we first have to check in on the branches, and then merge to the trunk? Afaics python is the only project to have such an approach. Others have trunk first, maybe with immediate backport, maybe with later backport. Matthias PS: Antoine: Please CC the committer for such emails. From tjreedy at udel.edu Tue Feb 5 07:13:04 2013 From: tjreedy at udel.edu (Terry Reedy) Date: Tue, 05 Feb 2013 01:13:04 -0500 Subject: [Python-Dev] cpython (2.7): - Issue #17086: Backport the patches from the 3.3 branch to cross-build In-Reply-To: <51101457.8090505@ubuntu.com> References: <3YxxXq44T7zQ28@mail.python.org> <20130201005001.5f73e15b@pitrou.net> <20130201081425.39120441@pitrou.net> <51101457.8090505@ubuntu.com> Message-ID: On 2/4/2013 3:04 PM, Matthias Klose wrote: > - the 2.7 branch is the only branch which doesn't have expected release > dates on the calendar. Which calendar? I see 2.7.4, 3.2.4 (its final release), and 3.3.1 on the Release Schecule at python.org. > And from a distributor/vendor point of view, I > think yearly releases are too seldom. Such a release could end up > only up to 24 months later in a (linux) distribution. Some subset of 'we' proposed 5 instead of 2 years of bugfix support for 2.7, released 2010 July 4, but no particular interval, especially after the first 2 years. > - there were way too may regressions checked in on at least the 2.7 > branch. Regressions? That normally means adding a bug as part of a patch, especially one that was previously fixed. We continually add tests to try to prevent that. What period of time do you mean 'were' to refer to? > Is it just our vcs merge model that we first have to check in > on the branches, and then merge to the trunk? Mercurial works best if merges are done in the forward direction. However, this is not relevant to 2.7 patches as they are pushed to tip but *not* merged. The repository is a two-headed monster ;=). > Afaics python is the > only project to have such an approach. Others have trunk first, maybe > with immediate backport, maybe with later backport. If a patch is first commited to tip (for 3.4) and then backported to 3.3, then when the backport is pushed, a null merge is needed to avoid making a third head, and the dag looks messier. (I believe 'null merge' is the right term, but I have never done this.) My impression is that most 3.x bugfix patches merge forward with little or no change. -- Terry Jan Reedy From theller at ctypes.org Wed Feb 6 10:26:38 2013 From: theller at ctypes.org (Thomas Heller) Date: Wed, 06 Feb 2013 10:26:38 +0100 Subject: [Python-Dev] importlib hooray! Message-ID: I have become a fan of the new python 3.3 importlib in the last few days. It has allowed me to write a ModuleMapper which I put into sys.metapath (in sitecustomize.py, for Python 3.3). This mapper currently does rename modules like 'Queue' or '_winreg' to the Python3 modules 'queue' or 'winreg' without the need to change my 2.7 sourcecode (changing the sourcecode is required when using six.moves to write 2.x/3.x compatible code). The six.moves approach has another problem with freeze tools (py2exe for example): ModuleFinder does not find the moved modules because it cannot track __import__(). I have also started a new modulefinder which uses importlib to find modules; this was quite easy since I could copy a lot of code from importlib._bootstrap. The great thing is that this new modulefinder is much better than the old one: It finds modules in zipped eggs or other zip-archives which are on sys.path; also it finds the renamed modules correctly that my ModuleMapper has mapped. The only thing that I am missing is that it is a bit of guesswork to find out the type of the module that importlib.find_loader() has found. Previously imp.find_module returned a tuple containing the type of module found (like imp.PY_SOURCE), I have to poke around in some attributes of the result of importlib.find_loader() to guess the type. Thomas From chris.jerdonek at gmail.com Wed Feb 6 11:03:27 2013 From: chris.jerdonek at gmail.com (Chris Jerdonek) Date: Wed, 6 Feb 2013 02:03:27 -0800 Subject: [Python-Dev] BDFL delegation for PEP 426 + distutils freeze In-Reply-To: References: <20130203122754.7745a1be@pitrou.net> <20130203134853.098aadaa@pitrou.net> <510EAD73.7090406@netwok.org> Message-ID: On Sun, Feb 3, 2013 at 11:40 AM, Chris Jerdonek wrote: > On Sun, Feb 3, 2013 at 10:33 AM, ?ric Araujo wrote: >> Le 03/02/2013 07:48, Antoine Pitrou a ?crit : >>> I vote for removing the "distutils is frozen" principle. >> I?ve also been thinking about that. There have been two exceptions to >> the freeze, for ABI flags in extension module names and for pycache >> directories. When the stable ABI was added and MvL wanted to change >> distutils (I don?t know to do what exactly), Tarek stood firm on the >> freeze and asked for any improvement to go into distutils2, and after >> MvL said that he would not contibute to an outside project, we merged d2 >> into the stdlib. Namespace packages did not impact distutils either. >> Now that we?ve removed packaging from the stdlib, we have two Python >> features that are not supported in the standard packaging system, and I >> agree that it is a bad thing for our users. >> >> I?d like to propose a reformulation of the freeze: > > This could be common knowledge, but is the current formulation of the > freeze spelled out somewhere? I asked this earlier, but didn't see a response. Is the freeze stated somewhere like in a PEP? If not, can someone state it precisely (e.g. what's allowed to change and what's not)? Thanks, --Chris From doko at ubuntu.com Wed Feb 6 14:16:13 2013 From: doko at ubuntu.com (Matthias Klose) Date: Wed, 06 Feb 2013 14:16:13 +0100 Subject: [Python-Dev] cpython (2.7): - Issue #17086: Backport the patches from the 3.3 branch to cross-build In-Reply-To: References: <3YxxXq44T7zQ28@mail.python.org> <20130201005001.5f73e15b@pitrou.net> <20130201081425.39120441@pitrou.net> <51101457.8090505@ubuntu.com> Message-ID: <5112579D.5050701@ubuntu.com> Am 05.02.2013 07:13, schrieb Terry Reedy: > On 2/4/2013 3:04 PM, Matthias Klose wrote: > >> - the 2.7 branch is the only branch which doesn't have expected release >> dates on the calendar. > > Which calendar? I see 2.7.4, 3.2.4 (its final release), and 3.3.1 on the Release > Schecule at python.org. maybe these are now updated. until recently 3.3.1 was targeted for Nov/Dec 2012, and 2.7.4 wasn't found on the calendar. >> - there were way too may regressions checked in on at least the 2.7 >> branch. > > Regressions? That normally means adding a bug as part of a patch, especially one > that was previously fixed. We continually add tests to try to prevent that. What > period of time do you mean 'were' to refer to? - http://bugs.python.org/issue9374 - http://bugs.python.org/issue15906 - http://bugs.python.org/issue16012 - http://bugs.python.org/issue10182 - http://bugs.python.org/issue16828 not a complete list, all these are past the 2.7.3 release. >> Is it just our vcs merge model that we first have to check in >> on the branches, and then merge to the trunk? > > Mercurial works best if merges are done in the forward direction. However, this > is not relevant to 2.7 patches as they are pushed to tip but *not* merged. The > repository is a two-headed monster ;=). so it should be possible to delay patches for 2.7 until they don't show issues in 3.2 or 3.3. >> Afaics python is the >> only project to have such an approach. Others have trunk first, maybe >> with immediate backport, maybe with later backport. > > If a patch is first commited to tip (for 3.4) and then backported to 3.3, then > when the backport is pushed, a null merge is needed to avoid making a third > head, and the dag looks messier. (I believe 'null merge' is the right term, but > I have never done this.) My impression is that most 3.x bugfix patches merge > forward with little or no change. so is it only this technical limitation, why the bugfix process is defined this way? From solipsis at pitrou.net Wed Feb 6 14:38:21 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 6 Feb 2013 14:38:21 +0100 Subject: [Python-Dev] cpython (2.7): - Issue #17086: Backport the patches from the 3.3 branch to cross-build References: <3YxxXq44T7zQ28@mail.python.org> <20130201005001.5f73e15b@pitrou.net> <20130201081425.39120441@pitrou.net> <51101457.8090505@ubuntu.com> Message-ID: <20130206143821.06b010ae@pitrou.net> Le Mon, 04 Feb 2013 21:04:39 +0100, Matthias Klose a ?crit : > So what I do understand, build-related issues like an arm64 or > mingw32 port are ok for 2.7, if they are stable on the trunk, and > communicated on python-dev? Making Python build on a new platform is, AFAICT, considered a new feature, not a bugfix. For example, we support new MSVC versions in the main development branch, not in bugfix branches. > - there were way too may regressions checked in on at least the 2.7 > branch. Is it just our vcs merge model that we first have to > check in on the branches, and then merge to the trunk? Afaics python > is the only project to have such an approach. Others have trunk > first, maybe with immediate backport, maybe with later backport. I don't think this has anything to do with the merge model. Regressions can always happen for things that are not unit-tested. Also, looking at the regressions you listed in another message, none of them appeared serious enough. Regards Antoine. From brett at python.org Wed Feb 6 15:17:49 2013 From: brett at python.org (Brett Cannon) Date: Wed, 6 Feb 2013 09:17:49 -0500 Subject: [Python-Dev] importlib hooray! In-Reply-To: References: Message-ID: On Wed, Feb 6, 2013 at 4:26 AM, Thomas Heller wrote: > I have become a fan of the new python 3.3 importlib > in the last few days. > Glad it's working out for you! > > It has allowed me to write a ModuleMapper which I put into > sys.metapath (in sitecustomize.py, for Python 3.3). > > This mapper currently does rename modules like 'Queue' or '_winreg' > to the Python3 modules 'queue' or 'winreg' without the need to change > my 2.7 sourcecode (changing the sourcecode is required when using > six.moves to write 2.x/3.x compatible code). > > The six.moves approach has another problem with freeze tools (py2exe > for example): ModuleFinder does not find the moved modules because > it cannot track __import__(). > > > I have also started a new modulefinder which uses importlib to find > modules; this was quite easy since I could copy a lot of code from > importlib._bootstrap. The great thing is that this new modulefinder > is much better than the old one: It finds modules in zipped eggs > or other zip-archives which are on sys.path; also it finds the renamed > modules correctly that my ModuleMapper has mapped. > > > The only thing that I am missing is that it is a bit of guesswork to > find out the type of the module that importlib.find_loader() has found. > > Previously imp.find_module returned a tuple containing the type of > module found (like imp.PY_SOURCE), I have to poke around in some > attributes of the result of importlib.find_loader() to guess the type. > You should be able to tell with an isinstance check on the returned loader and not have to look at any attributes (e.g. source will subclass SourceLoader, frozen will subclass FrozenImporter, etc.). If it continues to be an issue just let me know and we can see what we can do (this is obviously a niche need and so I would prefer not to change the loader API to explicitly accommodate this, but it can be done). -------------- next part -------------- An HTML attachment was scrubbed... URL: From brian at python.org Wed Feb 6 21:29:16 2013 From: brian at python.org (Brian Curtin) Date: Wed, 6 Feb 2013 14:29:16 -0600 Subject: [Python-Dev] PyCon Tickets Almost Sold Out Message-ID: Since the Language Summit is held at PyCon I think this counts as on-topic... If you're interested in going to the conference, there are under 50 tickets remaining: https://us.pycon.org/2013/registration/ From merwok at netwok.org Wed Feb 6 21:45:18 2013 From: merwok at netwok.org (=?UTF-8?B?w4lyaWMgQXJhdWpv?=) Date: Wed, 06 Feb 2013 15:45:18 -0500 Subject: [Python-Dev] BDFL delegation for PEP 426 + distutils freeze In-Reply-To: References: <20130203122754.7745a1be@pitrou.net> <20130203134853.098aadaa@pitrou.net> <510EAD73.7090406@netwok.org> Message-ID: <5112C0DE.6040705@netwok.org> Le 06/02/2013 05:03, Chris Jerdonek a ?crit : > I asked this earlier, but didn't see a response. Is the freeze > stated somewhere like in a PEP? It?s part in notes from the PyCon 2010 Language Summit, part in unwritten policy in the heads of people involved in distutils bugs these last years. > If not, can someone state it precisely (e.g. > what's allowed to change and what's not)? Bug fixes are possible; changes to keep working with Python (i.e. pyc files are generated in pycache directories after 3.2); changes to be able to build on current OSes (e.g. following Mac dev tools location change, introduction of Debian multiarch, etc.). Some bugs have been here for so long that everybody depends on them or work around them, or they would be extremely painful to fix (e.g. in the option parsing code) for little benefit, so they are wontfix. Cleanups, refactorings and improvements were banned by the feature freeze. Regards From barry at python.org Wed Feb 6 22:08:39 2013 From: barry at python.org (Barry Warsaw) Date: Wed, 6 Feb 2013 16:08:39 -0500 Subject: [Python-Dev] cpython (2.7): - Issue #17086: Backport the patches from the 3.3 branch to cross-build In-Reply-To: <20130206143821.06b010ae@pitrou.net> References: <3YxxXq44T7zQ28@mail.python.org> <20130201005001.5f73e15b@pitrou.net> <20130201081425.39120441@pitrou.net> <51101457.8090505@ubuntu.com> <20130206143821.06b010ae@pitrou.net> Message-ID: <20130206160839.52b17caa@anarchist.wooz.org> On Feb 06, 2013, at 02:38 PM, Antoine Pitrou wrote: >Le Mon, 04 Feb 2013 21:04:39 +0100, >Matthias Klose a ?crit : >> So what I do understand, build-related issues like an arm64 or >> mingw32 port are ok for 2.7, if they are stable on the trunk, and >> communicated on python-dev? > >Making Python build on a new platform is, AFAICT, considered a new >feature, not a bugfix. For example, we support new MSVC versions in the >main development branch, not in bugfix branches. Except that 2.7 is an exception to that since it's the last of the Python 2 series, and has a much longer lifespan than normal releases. I'm pretty sure we agreed that there would be some exceptions for issues like new platforms for 2.7. -Barry From tjreedy at udel.edu Thu Feb 7 01:19:50 2013 From: tjreedy at udel.edu (Terry Reedy) Date: Wed, 06 Feb 2013 19:19:50 -0500 Subject: [Python-Dev] cpython (2.7): - Issue #17086: Backport the patches from the 3.3 branch to cross-build In-Reply-To: <5112579D.5050701@ubuntu.com> References: <3YxxXq44T7zQ28@mail.python.org> <20130201005001.5f73e15b@pitrou.net> <20130201081425.39120441@pitrou.net> <51101457.8090505@ubuntu.com> <5112579D.5050701@ubuntu.com> Message-ID: <5112F326.4050801@udel.edu> On 2/6/2013 8:16 AM, Matthias Klose wrote: > Am 05.02.2013 07:13, schrieb Terry Reedy: >> On 2/4/2013 3:04 PM, Matthias Klose wrote: >>> - there were way too may regressions checked in on at least the 2.7 >>> branch. I think you are using 'regression' too freely. But that aside, I think you are pointing to a legitimate issue, in the first and maybe second example of how to handle 2.7. >> Regressions? That normally means adding a bug as part of a patch, especially one >> that was previously fixed. We continually add tests to try to prevent that. What >> period of time do you mean 'were' to refer to? > > - http://bugs.python.org/issue9374 From what I read (but it was a long discussion), this patch fixed behavior that was agreed to be a bug. The disagreement was whether the bug was clear or obscure. Since the bug was long-standing, and since much code had accommodated to the bug, and even depended on it, the bugfix broke more code than usual. This is not a regression, but is a concern, and if I understand correctly, the fix was removed or revised for existing versions. > - http://bugs.python.org/issue15906 I did not understand the issue, except to note that it affected all versions, involved apparent ambiguity in the doc, and was caught by ubuntu testing. > - http://bugs.python.org/issue16012 This reported a clear regression, on all versions, not just 2.7, that was caught before release. As the person responsible for the regression said "Too bad it [the regressed behavior] wasn't tested for :(". We still need more patches to improve test coverage. > - http://bugs.python.org/issue10182 This fixed an all-versions bug in _sre. The initial patch was fine for 3.x but not sufficiently adjusted for 2.7. Py_BuildValue needed to be replaced in certain places with PyInt_FromSsize_t rather than PyLong_FromSsize_t to maintain the external API. This was not caught because by the stdlib test but was by a third party who tests their package with tip. Yay for them. The process worked. > - http://bugs.python.org/issue16828 Again, an all-versions issue, not 2.7 specific. http://bugs.python.org/issue14398 disabled compression of '' in the process of enabling compression of >4GB strings. Not a bad tradeoff, really, but not best. Again, this was not caught initially because there *was* no stdlib test for ''. But there was in a 2.7 ubuntu package. > not a complete list, all these are past the 2.7.3 release. In only 1 of the 5 was there a 2.7-specific regression. I am not sure what specific change you would like. In the last three examples, would you want 2.7 left with the bug? >>> Is it just our vcs merge model that we first have to check in >>> on the branches, and then merge to the trunk? >> >> Mercurial works best if merges are done in the forward direction. However, this >> is not relevant to 2.7 patches as they are pushed to tip but *not* merged. The >> repository is a two-headed monster ;=). > > so it should be possible to delay patches for 2.7 until they don't show issues > in 3.2 or 3.3. Yes, however 1. Later may mean never. 2. If the regression is 2.7 specific, as with #10182, then not showing an issue for 3.x is irrelevant. 3. If the regression is general, then *at present*, the 2.7 regression is the most likely to be caught by external testing (mostly by unbuntu). That says to me that bug fixes for 2.7 *should* go into tip along with the others. I could even argue that the 2.7 fix should go in first. 4. If bugfixes are not applied to 2.7, that increases the distance between 2.7 and 3.x and possibly increases the difficulty of writing 2&3 code. If we do leave a fixed-in-3.x bug in 2.7, then perhaps, if it is possible to detect that the bug is being depended upon, there should be a deprecation warning in the code. -- Terry Jan Reedy From ncoghlan at gmail.com Thu Feb 7 08:36:20 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 7 Feb 2013 17:36:20 +1000 Subject: [Python-Dev] Moderating the severity of the distutils freeze Message-ID: On Thu, Feb 7, 2013 at 6:45 AM, ?ric Araujo wrote: > Bug fixes are possible; changes to keep working with Python (i.e. pyc > files are generated in pycache directories after 3.2); changes to be > able to build on current OSes (e.g. following Mac dev tools location > change, introduction of Debian multiarch, etc.). > > Some bugs have been here for so long that everybody depends on them or > work around them, or they would be extremely painful to fix (e.g. in the > option parsing code) for little benefit, so they are wontfix. > > Cleanups, refactorings and improvements were banned by the feature freeze. I think the points to be clarified are that *new* internal APIs (including new commands, like bdist_wheel) are legitimate in a feature release, and internal changes which don't affect APIs (like some of the tweaks Dave Malcolm and I made to try to make building Python 3 RPMs on RHEL 6 with bdist_rpm less broken) are legitimate in maintenance releases. distutils is going to be around for a long time for backwards compatibility reasons, so if people want to improve it within those constraints they should be allowed to - we just need to be extra careful with maintenance releases, by treating even nominally internal APIs as if they were public. The current interpretation of the situation as a complete "no few features" freeze is overkill, when the setuptools/distribute developers will have plenty of time to send up warning flares during the alpha/beta/rc cycle. An appropriate rule would be more like "no major refactorings of distutils internals" (which may indeed mean distutils never supports metadata versions past 1.1). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From regebro at gmail.com Thu Feb 7 09:45:37 2013 From: regebro at gmail.com (Lennart Regebro) Date: Thu, 7 Feb 2013 09:45:37 +0100 Subject: [Python-Dev] PEP-0431 update 4 Message-ID: Here is the update of PEP-0431. The major changes are that the new class now has a lowercase name, to conform to the rest of the datetime modeule: dsttimezone (i decided against dst_timezone, as non of the other classes have underscores). Also the timezone name sets are gone. Thanks to Nick Coghlan for finding a whole bunch of minor mistakes. //Lennart ---- PEP: 431 Title: Time zone support improvements Version: $Revision$ Last-Modified: $Date$ Author: Lennart Regebro BDFL-Delegate: Barry Warsaw Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 11-Dec-2012 Post-History: 11-Dec-2012, 28-Dec-2012, 28-Jan-2013 Abstract ======== This PEP proposes the implementation of concrete time zone support in the Python standard library, and also improvements to the time zone API to deal with ambiguous time specifications during DST changes. Proposal ======== Concrete time zone support -------------------------- The time zone support in Python has no concrete implementation in the standard library outside of a tzinfo baseclass that supports fixed offsets. To properly support time zones you need to include a database over all time zones, both current and historical, including daylight saving changes. But such information changes frequently, so even if we include the last information in a Python release, that information would be outdated just a few months later. Time zone support has therefore only been available through two third-party modules, ``pytz`` and ``dateutil``, both who include and wrap the "zoneinfo" database. This database, also called "tz" or "The Olsen database", is the de-facto standard time zone database over time zones, and it is included in most Unix and Unix-like operating systems, including OS X. This gives us the opportunity to include the code that supports the zoneinfo data in the standard library, but by default use the operating system's copy of the data, which typically will be kept updated by the updating mechanism of the operating system or distribution. For those who have an operating system that does not include the zoneinfo database, for example Windows, the Python source distribution will include a copy of the zoneinfo database, and a distribution containing the latest zoneinfo database will also be available at the Python Package Index, so it can be easily installed with the Python packaging tools such as ``easy_install`` or ``pip``. This could also be done on Unices that are no longer receiving updates and therefore have an outdated database. With such a mechanism Python would have full time zone support in the standard library on any platform, and a simple package installation would provide an updated time zone database on those platforms where the zoneinfo database isn't included, such as Windows, or on platforms where OS updates are no longer provided. The time zone support will be implemented by making the ``datetime`` module into a package, and adding time zone support to ``datetime`` based on Stuart Bishop's ``pytz`` module. Getting the local time zone --------------------------- On Unix there is no standard way of finding the name of the time zone that is being used. All the information that is available is the time zone abbreviations, such as ``EST`` and ``PDT``, but many of those abbreviations are ambiguous and therefore you can't rely on them to figure out which time zone you are located in. There is however a standard for finding the compiled time zone information since it's located in ``/etc/localtime``. Therefore it is possible to create a local time zone object with the correct time zone information even though you don't know the name of the time zone. A function in ``datetime`` should be provided to return the local time zone. The support for this will be made by integrating Lennart Regebro's ``tzlocal`` module into the new ``datetime`` module. For Windows it will look up the local Windows time zone name, and use a mapping between Windows time zone names and zoneinfo time zone names provided by the Unicode consortium to convert that to a zoneinfo time zone. The mapping should be updated before each major or bugfix release, scripts for doing so will be provided in the ``Tools/`` directory. Ambiguous times --------------- When changing over from daylight savings time (DST) the clock is turned back one hour. This means that the times during that hour happens twice, once with DST and then once without DST. Similarly, when changing to daylight savings time, one hour goes missing. The current time zone API can not differentiate between the two ambiguous times during a change from DST. For example, in Stockholm the time of 2012-11-28 02:00:00 happens twice, both at UTC 2012-11-28 00:00:00 and also at 2012-11-28 01:00:00. The current time zone API can not disambiguate this and therefore it's unclear which time should be returned:: # This could be either 00:00 or 01:00 UTC: >>> dt = datetime(2012, 10, 28, 2, 0, tzinfo=zoneinfo('Europe/Stockholm')) # But we can not specify which: >>> dt.astimezone(zoneinfo('UTC')) datetime.datetime(2012, 10, 28, 1, 0, tzinfo=) ``pytz`` solved this problem by adding ``is_dst`` parameters to several methods of the tzinfo objects to make it possible to disambiguate times when this is desired. This PEP proposes to add these ``is_dst`` parameters to the relevant methods of the ``datetime`` API, and therefore add this functionality directly to ``datetime``. This is likely the hardest part of this PEP as this involves updating the C version of the ``datetime`` library with this functionality, as this involved writing new code, and not just reorganizing existing external libraries. Implementation API ================== The zoneinfo database --------------------- The latest version of the zoneinfo database should exist in the ``Lib/tzdata`` directory of the Python source control system. This copy of the database should be updated before every Python feature and bug-fix release, but not for releases of Python versions that are in security-fix-only-mode. Scripts to update the database will be provided in ``Tools/``, and the release instructions will be updated to include this update. New configure options ``--enable-internal-timezone-database`` and ``--disable-internal-timezone-database`` will be implemented to enable and disable the installation of this database when installing from source. A source install will default to installing them. Binary installers for systems that have a system-provided zoneinfo database may skip installing the included database since it would never be used for these platforms. For other platforms, for example Windows, binary installers must install the included database. Changes in the ``datetime``-module ---------------------------------- The public API of the new time zone support contains one new class, one new function, one new exception and four new collections. In addition to this, several methods on the datetime object gets a new ``is_dst`` parameter. New class ``dsttimezone`` ^^^^^^^^^^^^^^^^^^^^^^^^ This class provides a concrete implementation of the ``tzinfo`` base class that implements DST support. New function ``zoneinfo(name=None, db_path=None)`` ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ This function takes a name string that must be a string specifying a valid zoneinfo time zone, i.e. "US/Eastern", "Europe/Warsaw" or "Etc/GMT". If not given, the local time zone will be looked up. If an invalid zone name is given, or the local time zone can not be retrieved, the function raises `UnknownTimeZoneError`. The function also takes an optional path to the location of the zoneinfo database which should be used. If not specified, the function will look for databases in the following order: 1. Check if the `tzdata-update` module is installed, and then use that database. 2. Use the database in ``/usr/share/zoneinfo``, if it exists. 3. Use the Python-provided database in ``Lib/tzdata``. If no database is found an ``UnknownTimeZoneError`` or subclass thereof will be raised with a message explaining that no zoneinfo database can be found, but that you can install one with the ``tzdata-update`` package. New parameter ``is_dst`` ^^^^^^^^^^^^^^^^^^^^^^^^ A new ``is_dst`` parameter is added to several methods to handle time ambiguity during DST changeovers. * ``tzinfo.utcoffset(dt, is_dst=False)`` * ``tzinfo.dst(dt, is_dst=False)`` * ``tzinfo.tzname(dt, is_dst=False)`` * ``datetime.astimezone(tz, is_dst=False)`` The ``is_dst`` parameter can be ``False`` (default), ``True``, or ``None``. ``False`` will specify that the given datetime should be interpreted as not happening during daylight savings time, i.e. that the time specified is after the change from DST. This is default to preserve existing behavior. ``True`` will specify that the given datetime should be interpreted as happening during daylight savings time, i.e. that the time specified is before the change from DST. ``None`` will raise an ``AmbiguousTimeError`` exception if the time specified was during a DST change over. It will also raise a ``NonExistentTimeError`` if a time is specified during the "missing time" in a change to DST. New exceptions ^^^^^^^^^^^^^^ * ``UnknownTimeZoneError`` This exception is a subclass of KeyError and raised when giving a time zone specification that can't be found:: >>> datetime.zoneinfo('Europe/New_York') Traceback (most recent call last): ... UnknownTimeZoneError: There is no time zone called 'Europe/New_York' * ``InvalidTimeError`` This exception serves as a base for ``AmbiguousTimeError`` and ``NonExistentTimeError``, to enable you to trap these two separately. It will subclass from ValueError, so that you can catch these errors together with inputs like the 29th of February 2011. * ``AmbiguousTimeError`` This exception is raised when giving a datetime specification that is ambiguous while setting ``is_dst`` to None:: >>> datetime(2012, 11, 28, 2, 0, tzinfo=zoneinfo('Europe/Stockholm'), is_dst=None) >>> Traceback (most recent call last): ... AmbiguousTimeError: 2012-10-28 02:00:00 is ambiguous in time zone Europe/Stockholm * ``NonExistentTimeError`` This exception is raised when giving a datetime specification for a time that due to daylight saving does not exist, while setting ``is_dst`` to None:: >>> datetime(2012, 3, 25, 2, 0, tzinfo=zoneinfo('Europe/Stockholm'), is_dst=None) >>> Traceback (most recent call last): ... NonExistentTimeError: 2012-03-25 02:00:00 does not exist in time zone Europe/Stockholm New collections ^^^^^^^^^^^^^^^ * ``all_timezones`` is the exhaustive list of the time zone names that can be used, listed alphabethically. * ``common_timezones`` is a list of useful, current time zones, listed alphabethically. The ``tzdata-update``-package ----------------------------- The zoneinfo database will be packaged for easy installation with ``easy_install``/``pip``/``buildout``. This package will not install any Python code, and will not contain any Python code except that which is needed for installation. It will be kept updated with the same tools as the internal database, but released whenever the ``zoneinfo``-database is updated, and use the same version schema. Differences from the ``pytz`` API ================================= * ``pytz`` has the functions ``localize()`` and ``normalize()`` to work around that ``tzinfo`` doesn't have is_dst. When ``is_dst`` is implemented directly in ``datetime.tzinfo`` they are no longer needed. * The ``timezone()`` function is called ``zoneinfo()`` to avoid clashing with the ``timezone`` class introduced in Python 3.2. * ``zoneinfo()`` will return the local time zone if called without arguments. * The class ``pytz.StaticTzInfo`` is there to provide the ``is_dst`` support for static time zones. When ``is_dst`` support is included in ``datetime.tzinfo`` it is no longer needed. * ``InvalidTimeError`` subclasses from ``ValueError``. Resources ========= * http://pytz.sourceforge.net/ * http://pypi.python.org/pypi/tzlocal * http://pypi.python.org/pypi/python-dateutil * http://unicode.org/cldr/data/common/supplemental/windowsZones.xml Copyright ========= This document has been placed in the public domain. From solipsis at pitrou.net Thu Feb 7 10:10:29 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 7 Feb 2013 10:10:29 +0100 Subject: [Python-Dev] cpython (2.7): - Issue #17086: Backport the patches from the 3.3 branch to cross-build References: <3YxxXq44T7zQ28@mail.python.org> <20130201005001.5f73e15b@pitrou.net> <20130201081425.39120441@pitrou.net> <51101457.8090505@ubuntu.com> <20130206143821.06b010ae@pitrou.net> <20130206160839.52b17caa@anarchist.wooz.org> Message-ID: <20130207101029.730a0aeb@pitrou.net> Le Wed, 6 Feb 2013 16:08:39 -0500, Barry Warsaw a ?crit : > On Feb 06, 2013, at 02:38 PM, Antoine Pitrou wrote: > > >Le Mon, 04 Feb 2013 21:04:39 +0100, > >Matthias Klose a ?crit : > >> So what I do understand, build-related issues like an arm64 or > >> mingw32 port are ok for 2.7, if they are stable on the trunk, and > >> communicated on python-dev? > > > >Making Python build on a new platform is, AFAICT, considered a new > >feature, not a bugfix. For example, we support new MSVC versions in > >the main development branch, not in bugfix branches. > > Except that 2.7 is an exception to that since it's the last of the > Python 2 series, and has a much longer lifespan than normal > releases. I'm pretty sure we agreed that there would be some > exceptions for issues like new platforms for 2.7. Well, apparently we didn't make such an exception for MSVC :-) Regards Antoine. From ncoghlan at gmail.com Thu Feb 7 11:02:39 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 7 Feb 2013 20:02:39 +1000 Subject: [Python-Dev] cpython (2.7): - Issue #17086: Backport the patches from the 3.3 branch to cross-build In-Reply-To: <20130207101029.730a0aeb@pitrou.net> References: <3YxxXq44T7zQ28@mail.python.org> <20130201005001.5f73e15b@pitrou.net> <20130201081425.39120441@pitrou.net> <51101457.8090505@ubuntu.com> <20130206143821.06b010ae@pitrou.net> <20130206160839.52b17caa@anarchist.wooz.org> <20130207101029.730a0aeb@pitrou.net> Message-ID: On 7 Feb 2013 19:13, "Antoine Pitrou" wrote: > > Le Wed, 6 Feb 2013 16:08:39 -0500, > Barry Warsaw a ?crit : > > On Feb 06, 2013, at 02:38 PM, Antoine Pitrou wrote: > > > > >Le Mon, 04 Feb 2013 21:04:39 +0100, > > >Matthias Klose a ?crit : > > >> So what I do understand, build-related issues like an arm64 or > > >> mingw32 port are ok for 2.7, if they are stable on the trunk, and > > >> communicated on python-dev? > > > > > >Making Python build on a new platform is, AFAICT, considered a new > > >feature, not a bugfix. For example, we support new MSVC versions in > > >the main development branch, not in bugfix branches. > > > > Except that 2.7 is an exception to that since it's the last of the > > Python 2 series, and has a much longer lifespan than normal > > releases. I'm pretty sure we agreed that there would be some > > exceptions for issues like new platforms for 2.7. > > Well, apparently we didn't make such an exception for MSVC :-) Python 2.7 still runs on Windows just fine, and changing the C runtime used is *not* purely a change to the build process (it effectively breaks the ABI). I'm not sure how you see that decision being remotely related to Benjamin's decision to allow the cross-compilation patches. Cheers, Nick. > > Regards > > Antoine. > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Thu Feb 7 11:23:53 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 7 Feb 2013 11:23:53 +0100 Subject: [Python-Dev] cpython (2.7): - Issue #17086: Backport the patches from the 3.3 branch to cross-build References: <3YxxXq44T7zQ28@mail.python.org> <20130201005001.5f73e15b@pitrou.net> <20130201081425.39120441@pitrou.net> <51101457.8090505@ubuntu.com> <20130206143821.06b010ae@pitrou.net> <20130206160839.52b17caa@anarchist.wooz.org> <20130207101029.730a0aeb@pitrou.net> Message-ID: <20130207112353.5f9298bf@pitrou.net> Le Thu, 7 Feb 2013 20:02:39 +1000, Nick Coghlan a ?crit : > On 7 Feb 2013 19:13, "Antoine Pitrou" wrote: > > > > Le Wed, 6 Feb 2013 16:08:39 -0500, > > Barry Warsaw a ?crit : > > > On Feb 06, 2013, at 02:38 PM, Antoine Pitrou wrote: > > > > > > >Le Mon, 04 Feb 2013 21:04:39 +0100, > > > >Matthias Klose a ?crit : > > > >> So what I do understand, build-related issues like an arm64 or > > > >> mingw32 port are ok for 2.7, if they are stable on the trunk, > > > >> and communicated on python-dev? > > > > > > > >Making Python build on a new platform is, AFAICT, considered a > > > >new feature, not a bugfix. For example, we support new MSVC > > > >versions in the main development branch, not in bugfix branches. > > > > > > Except that 2.7 is an exception to that since it's the last of the > > > Python 2 series, and has a much longer lifespan than normal > > > releases. I'm pretty sure we agreed that there would be some > > > exceptions for issues like new platforms for 2.7. > > > > Well, apparently we didn't make such an exception for MSVC :-) > > Python 2.7 still runs on Windows just fine, and changing the C > runtime used is *not* purely a change to the build process (it > effectively breaks the ABI). There is a difference between supporting several MSVCs and changing the MSVC used for the official binary builds. People did ask us to support newer MSVCs. > I'm not sure how you see that decision being remotely related to > Benjamin's decision to allow the cross-compilation patches. I was merely replying to the idea that "we allow new build features in 2.7". Non-trivial build patches have a tendency to produce unexpected breakage (for the record, the initial cross-compiling commits on 3.x brought a lot of breakage at first). Our build system is complicated and *very* poorly documented, I am surprised you think it is a good idea to accept large changes in bugfix releases. I'm still of the advice that non-trivial build enhancements shouldn't be treated differently from any new feature. Patches for bugfix branches can be maintained externally, as they already are (e.g. by Linux distributions). (btw, nothing to do with this discussion, Nick, but it appears your RHEL buildbot is offline) Regards Antoine. From a.cavallo at cavallinux.eu Thu Feb 7 11:44:10 2013 From: a.cavallo at cavallinux.eu (a.cavallo at cavallinux.eu) Date: Thu, 7 Feb 2013 11:44:10 +0100 Subject: [Python-Dev] cpython (2.7): - Issue #17086: Backport thepatches from the 3.3 branch to cross-build Message-ID: <64467.1360233850@cavallinux.eu> > (btw, nothing to do with this discussion, Nick, but it appears your > RHEL buildbot is offline) > > Regards Btw rpm build and packaging can be done on the suse build service: it is a very complete (and free) service. https://build.opensuse.org From brett at python.org Thu Feb 7 15:18:59 2013 From: brett at python.org (Brett Cannon) Date: Thu, 7 Feb 2013 09:18:59 -0500 Subject: [Python-Dev] PyCon Tickets Almost Sold Out In-Reply-To: References: Message-ID: On Wed, Feb 6, 2013 at 3:29 PM, Brian Curtin wrote: > Since the Language Summit is held at PyCon I think this counts as > on-topic... > > If you're interested in going to the conference, there are under 50 > tickets remaining: https://us.pycon.org/2013/registration/ ... and then there were none. Tutorials are still open, though. -Brett -------------- next part -------------- An HTML attachment was scrubbed... URL: From littleq0903 at gmail.com Thu Feb 7 16:33:46 2013 From: littleq0903 at gmail.com (LittleQ) Date: Thu, 7 Feb 2013 23:33:46 +0800 Subject: [Python-Dev] PyConTW is calling for proposals! Message-ID: (link: PyConTW is calling for proposals!) http://tw.pycon.org/2013/en/blog/2012/11/21/call-for-proposals-en/ Hello, everyone! I'm Colin Su (LittleQ), Public Relation @ PyConTW. PyCon Taiwan is calling for proposals now, Python is becoming one of famous programming languages in Taiwan, We need every Python developers' participating. Go to the link above, and submit your proposal right now! :D Feel free to send me an email if you have any question. Have a nice Day, Colin, PR @ PyConTW (if this mail is not appropriate on this mailing list (python-dev), just let me know, then I won't bother everybody anymore, thanks.) -- { greeting: "Have a nice day!", name: "Colin Su", nickname: "LittleQ", email: "littleq0903 at gmail.com", website: "http://about.me/littleq", title: [ ["Genie Capital", "Web Developer"], ["National Chengchi University", "Student", "Computer Science Dept." ] ] } -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Thu Feb 7 20:47:51 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 7 Feb 2013 20:47:51 +0100 Subject: [Python-Dev] PyConTW is calling for proposals! References: Message-ID: <20130207204751.7591cee3@pitrou.net> Hi, On Thu, 7 Feb 2013 23:33:46 +0800 LittleQ wrote: > > PyCon Taiwan is calling for proposals now, Python is becoming one of famous > programming languages in Taiwan, We need every Python developers' > participating. Go to the link above, and submit your proposal right now! :D > > Feel free to send me an email if you have any question. > > Have a nice Day, > > Colin, PR @ PyConTW > > (if this mail is not appropriate on this mailing list (python-dev), just > let me know, then I won't bother everybody anymore, thanks.) It sounds totally legitimate to me. cheers Antoine. From chris at simplistix.co.uk Fri Feb 8 11:58:36 2013 From: chris at simplistix.co.uk (Chris Withers) Date: Fri, 08 Feb 2013 10:58:36 +0000 Subject: [Python-Dev] help with knowledge on how to find which release a fix will land in Message-ID: <5114DA5C.3090801@simplistix.co.uk> Hi All, Where would I look to find out which release a fix for an issue (http://bugs.python.org/issue15822 if you're interested ;-)) will land in? cheers, Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk From solipsis at pitrou.net Fri Feb 8 12:17:03 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 8 Feb 2013 12:17:03 +0100 Subject: [Python-Dev] help with knowledge on how to find which release a fix will land in References: <5114DA5C.3090801@simplistix.co.uk> Message-ID: <20130208121703.2c146e6c@pitrou.net> Le Fri, 08 Feb 2013 10:58:36 +0000, Chris Withers a ?crit : > Hi All, > > Where would I look to find out which release a fix for an issue > (http://bugs.python.org/issue15822 if you're interested ;-)) will > land in? Just read that issue's comments and you'll know in which branches the fix was committed. Regards Antoine. From chris at simplistix.co.uk Fri Feb 8 13:45:48 2013 From: chris at simplistix.co.uk (Chris Withers) Date: Fri, 08 Feb 2013 12:45:48 +0000 Subject: [Python-Dev] help with knowledge on how to find which release a fix will land in In-Reply-To: <20130208121703.2c146e6c@pitrou.net> References: <5114DA5C.3090801@simplistix.co.uk> <20130208121703.2c146e6c@pitrou.net> Message-ID: <5114F37C.8070704@simplistix.co.uk> On 08/02/2013 11:17, Antoine Pitrou wrote: > Le Fri, 08 Feb 2013 10:58:36 +0000, > Chris Withers a ?crit : >> Hi All, >> >> Where would I look to find out which release a fix for an issue >> (http://bugs.python.org/issue15822 if you're interested ;-)) will >> land in? > > Just read that issue's comments and you'll know in which branches the > fix was committed. Indeed, so it's on the 2.7, 3.2 and default (3.3) branches, but how would I find out which release it will be in? I would hope 3.3.1, but how can I check? What's the next 3.2.x release? cheers, Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk From solipsis at pitrou.net Fri Feb 8 14:16:04 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 8 Feb 2013 14:16:04 +0100 Subject: [Python-Dev] help with knowledge on how to find which release a fix will land in References: <5114DA5C.3090801@simplistix.co.uk> <20130208121703.2c146e6c@pitrou.net> <5114F37C.8070704@simplistix.co.uk> Message-ID: <20130208141604.7a4525a3@pitrou.net> Le Fri, 08 Feb 2013 12:45:48 +0000, Chris Withers a ?crit : > On 08/02/2013 11:17, Antoine Pitrou wrote: > > Le Fri, 08 Feb 2013 10:58:36 +0000, > > Chris Withers a ?crit : > >> Hi All, > >> > >> Where would I look to find out which release a fix for an issue > >> (http://bugs.python.org/issue15822 if you're interested ;-)) will > >> land in? > > > > Just read that issue's comments and you'll know in which branches > > the fix was committed. > > Indeed, so it's on the 2.7, 3.2 and default (3.3) branches, but how > would I find out which release it will be in? The next release, quite naturally. > I would hope 3.3.1, but how can I check? > What's the next 3.2.x release? I suppose you can take a look at http://python.org/download/releases/ and increment the respective minor numbers. Regards Antoine. From ronaldoussoren at mac.com Fri Feb 8 14:16:49 2013 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Fri, 8 Feb 2013 14:16:49 +0100 Subject: [Python-Dev] help with knowledge on how to find which release a fix will land in In-Reply-To: <5114F37C.8070704@simplistix.co.uk> References: <5114DA5C.3090801@simplistix.co.uk> <20130208121703.2c146e6c@pitrou.net> <5114F37C.8070704@simplistix.co.uk> Message-ID: On 8 Feb, 2013, at 13:45, Chris Withers wrote: > On 08/02/2013 11:17, Antoine Pitrou wrote: >> Le Fri, 08 Feb 2013 10:58:36 +0000, >> Chris Withers a ?crit : >>> Hi All, >>> >>> Where would I look to find out which release a fix for an issue >>> (http://bugs.python.org/issue15822 if you're interested ;-)) will >>> land in? >> >> Just read that issue's comments and you'll know in which branches the >> fix was committed. > > Indeed, so it's on the 2.7, 3.2 and default (3.3) branches, but how would I find out which release it will be in? > > I would hope 3.3.1, but how can I check? In general bugfixes are mentioned in Misc/NEWS in the sections for the releases where the bugfix ends up. > What's the next 3.2.x release? According to the calendar 3.2.4 will be released on Feb 16. Ronald > > cheers, > > Chris > > -- > Simplistix - Content Management, Batch Processing & Python Consulting > - http://www.simplistix.co.uk > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/ronaldoussoren%40mac.com From tshepang at gmail.com Fri Feb 8 14:17:47 2013 From: tshepang at gmail.com (Tshepang Lekhonkhobe) Date: Fri, 8 Feb 2013 15:17:47 +0200 Subject: [Python-Dev] help with knowledge on how to find which release a fix will land in In-Reply-To: <5114F37C.8070704@simplistix.co.uk> References: <5114DA5C.3090801@simplistix.co.uk> <20130208121703.2c146e6c@pitrou.net> <5114F37C.8070704@simplistix.co.uk> Message-ID: On Fri, Feb 8, 2013 at 2:45 PM, Chris Withers wrote: > On 08/02/2013 11:17, Antoine Pitrou wrote: >> >> Le Fri, 08 Feb 2013 10:58:36 +0000, >> Chris Withers a ?crit : >>> >>> Hi All, >>> >>> Where would I look to find out which release a fix for an issue >>> (http://bugs.python.org/issue15822 if you're interested ;-)) will >>> land in? >> >> >> Just read that issue's comments and you'll know in which branches the >> fix was committed. > > > Indeed, so it's on the 2.7, 3.2 and default (3.3) branches, but how would I > find out which release it will be in? > > I would hope 3.3.1, but how can I check? > What's the next 3.2.x release? Have a look at http://docs.python.org/devguide/triaging.html#priority. That page says that the issue you are interested in has too low a priority (high) to determine which release it will be in; that is if you are just looking at the issue report, but you can always poke those involved in that issue. From tshepang at gmail.com Fri Feb 8 14:22:06 2013 From: tshepang at gmail.com (Tshepang Lekhonkhobe) Date: Fri, 8 Feb 2013 15:22:06 +0200 Subject: [Python-Dev] help with knowledge on how to find which release a fix will land in In-Reply-To: References: <5114DA5C.3090801@simplistix.co.uk> <20130208121703.2c146e6c@pitrou.net> <5114F37C.8070704@simplistix.co.uk> Message-ID: On Fri, Feb 8, 2013 at 3:17 PM, Tshepang Lekhonkhobe wrote: > On Fri, Feb 8, 2013 at 2:45 PM, Chris Withers wrote: >> On 08/02/2013 11:17, Antoine Pitrou wrote: >>> >>> Le Fri, 08 Feb 2013 10:58:36 +0000, >>> Chris Withers a ?crit : >>>> >>>> Hi All, >>>> >>>> Where would I look to find out which release a fix for an issue >>>> (http://bugs.python.org/issue15822 if you're interested ;-)) will >>>> land in? >>> >>> >>> Just read that issue's comments and you'll know in which branches the >>> fix was committed. >> >> >> Indeed, so it's on the 2.7, 3.2 and default (3.3) branches, but how would I >> find out which release it will be in? >> >> I would hope 3.3.1, but how can I check? >> What's the next 3.2.x release? > > Have a look at http://docs.python.org/devguide/triaging.html#priority. > That page says that the issue you are interested in has too low a > priority (high) to determine which release it will be in; that is if > you are just looking at the issue report, but you can always poke > those involved in that issue. Oh, I did not notice that it was already fixed. In that case, it will be in the next releases of the stated Versions (i.e. 2.7, 3.2, and 3.3). From stefan_ml at behnel.de Fri Feb 8 15:09:07 2013 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 08 Feb 2013 15:09:07 +0100 Subject: [Python-Dev] Why does Signature.from_function() have to check the type of its argument? Message-ID: Hi, I'm wondering about the purpose of this code in inspect.Signature.from_function(): """ if not isinstance(func, types.FunctionType): raise TypeError('{!r} is not a Python function'.format(func)) """ Is there any reason why this method would have to explicitly check the type of its argument? Why can't it just accept any object that quacks like a function? The reason why I ran into this is that I was trying to implement PEP 362 for Cython functions (which are not of Python function type types.FunctionType), and these two lines currently force me to copy the entire method body over into my code, just to build a Signature object from introspection. I find that a really bad design. Could this check please be removed completely? Or at least be changed to use isfunction() instead, so that it becomes a bit more easy to monkey patch it? (which is hard enough to explain to users already, BTW) Oh, and that's for 3.3.1, please. Stefan From benjamin at python.org Fri Feb 8 16:06:09 2013 From: benjamin at python.org (Benjamin Peterson) Date: Fri, 8 Feb 2013 10:06:09 -0500 Subject: [Python-Dev] Why does Signature.from_function() have to check the type of its argument? In-Reply-To: References: Message-ID: 2013/2/8 Stefan Behnel : > Hi, > > I'm wondering about the purpose of this code in > inspect.Signature.from_function(): > > """ > if not isinstance(func, types.FunctionType): > raise TypeError('{!r} is not a Python function'.format(func)) > """ > > Is there any reason why this method would have to explicitly check the type > of its argument? Why can't it just accept any object that quacks like a > function? The signature() function checks for types.FunctionType in order to call Signature.from_function(). How would you reimplement that? -- Regards, Benjamin From ncoghlan at gmail.com Fri Feb 8 16:08:16 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 9 Feb 2013 01:08:16 +1000 Subject: [Python-Dev] Why does Signature.from_function() have to check the type of its argument? In-Reply-To: References: Message-ID: On Sat, Feb 9, 2013 at 12:09 AM, Stefan Behnel wrote: > Hi, > > I'm wondering about the purpose of this code in > inspect.Signature.from_function(): > > """ > if not isinstance(func, types.FunctionType): > raise TypeError('{!r} is not a Python function'.format(func)) > """ > > Is there any reason why this method would have to explicitly check the type > of its argument? Why can't it just accept any object that quacks like a > function? Primarily because the expected approach for alternative objects that want to expose signature in 3.3 was for them to provide the __signature__ attribute directly, rather than providing the far messier grab bag of eclectic attributes exposed by native Python functions (there's also a tentative plan kicking around somewhere to have a string version for 3.4, so we can start doing something useful with functions implemented in C). I suspect the specific reason for the lack of ducktyping is merely to provide a nice error message when people call the class method rather than the module level function, so removing the check sounds reasonable to me. The ultimate decision will be Georg's as RM, though. Probably the quickest way to get a decision from him would be a tracker issue set to release blocker, with a patch removing those two lines and a test confirming that non-functions with the right attributes are correctly accepted (e.g. by copying the attributes from a real function to a custom object and checking Signature.from_function gives the same answer). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Fri Feb 8 16:20:41 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 9 Feb 2013 01:20:41 +1000 Subject: [Python-Dev] Why does Signature.from_function() have to check the type of its argument? In-Reply-To: References: Message-ID: On Sat, Feb 9, 2013 at 1:06 AM, Benjamin Peterson wrote: > 2013/2/8 Stefan Behnel : >> Hi, >> >> I'm wondering about the purpose of this code in >> inspect.Signature.from_function(): >> >> """ >> if not isinstance(func, types.FunctionType): >> raise TypeError('{!r} is not a Python function'.format(func)) >> """ >> >> Is there any reason why this method would have to explicitly check the type >> of its argument? Why can't it just accept any object that quacks like a >> function? > > The signature() function checks for types.FunctionType in order to > call Signature.from_function(). How would you reimplement that? I assumed Stefan was wanting to use Signature.from_function() to set __signature__ (as soon as you do that, inspect.signature will do the right thing). At the moment, you would have to build the signature by hand (which was the original intent of the design), but we simply didn't think about projects like Cython that might be providing the full types.FunctionType API without being ordinary function instances. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From chris at simplistix.co.uk Fri Feb 8 16:39:20 2013 From: chris at simplistix.co.uk (Chris Withers) Date: Fri, 08 Feb 2013 15:39:20 +0000 Subject: [Python-Dev] why do we allow this syntax? Message-ID: <51151C28.3020101@simplistix.co.uk> Hi All, Just had a bit of an embarrassing incident in some code where I did: sometotal =+ somevalue I'm curious why this syntax is allowed? I'm sure there are good reasons, but thought I'd ask... Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk From donald.stufft at gmail.com Fri Feb 8 16:41:52 2013 From: donald.stufft at gmail.com (Donald Stufft) Date: Fri, 8 Feb 2013 10:41:52 -0500 Subject: [Python-Dev] why do we allow this syntax? In-Reply-To: <51151C28.3020101@simplistix.co.uk> References: <51151C28.3020101@simplistix.co.uk> Message-ID: <98E0DC4A0B7C405ABBCEBD4AAE7C0902@gmail.com> On Friday, February 8, 2013 at 10:39 AM, Chris Withers wrote: > Hi All, > > Just had a bit of an embarrassing incident in some code where I did: > > sometotal =+ somevalue I'm guessing this gets parsed as sometotal = +somevalue > > I'm curious why this syntax is allowed? I'm sure there are good reasons, > but thought I'd ask... > > Chris > > -- > Simplistix - Content Management, Batch Processing & Python Consulting > - http://www.simplistix.co.uk > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org (mailto:Python-Dev at python.org) > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/donald.stufft%40gmail.com > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From benjamin at python.org Fri Feb 8 16:42:39 2013 From: benjamin at python.org (Benjamin Peterson) Date: Fri, 8 Feb 2013 10:42:39 -0500 Subject: [Python-Dev] why do we allow this syntax? In-Reply-To: <51151C28.3020101@simplistix.co.uk> References: <51151C28.3020101@simplistix.co.uk> Message-ID: 2013/2/8 Chris Withers : > Hi All, > > Just had a bit of an embarrassing incident in some code where I did: > > sometotal =+ somevalue That's just a strange way of expressing sometotal = +somevalue > > I'm curious why this syntax is allowed? I'm sure there are good reasons, but > thought I'd ask... -- Regards, Benjamin From rosuav at gmail.com Fri Feb 8 16:42:57 2013 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 9 Feb 2013 02:42:57 +1100 Subject: [Python-Dev] why do we allow this syntax? In-Reply-To: <51151C28.3020101@simplistix.co.uk> References: <51151C28.3020101@simplistix.co.uk> Message-ID: On Sat, Feb 9, 2013 at 2:39 AM, Chris Withers wrote: > Hi All, > > Just had a bit of an embarrassing incident in some code where I did: > > sometotal =+ somevalue > > I'm curious why this syntax is allowed? I'm sure there are good reasons, but > thought I'd ask... For the same reason that you can negate a value with: sometotal = -somevalue The unary + and - operators are seldom problematic. ChrisA From catch-all at masklinn.net Fri Feb 8 16:50:42 2013 From: catch-all at masklinn.net (Xavier Morel) Date: Fri, 8 Feb 2013 16:50:42 +0100 Subject: [Python-Dev] why do we allow this syntax? In-Reply-To: <51151C28.3020101@simplistix.co.uk> References: <51151C28.3020101@simplistix.co.uk> Message-ID: <0D9FF447-C507-42FC-B6C6-B80A9B8B8E04@masklinn.net> On 2013-02-08, at 16:39 , Chris Withers wrote: > Hi All, > > Just had a bit of an embarrassing incident in some code where I did: > > sometotal =+ somevalue > > I'm curious why this syntax is allowed? I'm sure there are good reasons, but thought I'd ask? sometotal = (expression) is valid syntax, and +value is valid syntax. Thus what you wrote is perfectly normal syntax, it's the assignment of a pos'd value, badly formatted. pep8.py will warn against it (it'll complain that the whitespace around `+` is wonky). But I see no justification for disallowing this, anymore than for disallowing the rougly equivalent (and just as error-prone) `sometotal = -somevalue`. From stefan_ml at behnel.de Fri Feb 8 16:52:47 2013 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 08 Feb 2013 16:52:47 +0100 Subject: [Python-Dev] Why does Signature.from_function() have to check the type of its argument? In-Reply-To: References: Message-ID: Nick Coghlan, 08.02.2013 16:08: > On Sat, Feb 9, 2013 at 12:09 AM, Stefan Behnel wrote: >> I'm wondering about the purpose of this code in >> inspect.Signature.from_function(): >> >> """ >> if not isinstance(func, types.FunctionType): >> raise TypeError('{!r} is not a Python function'.format(func)) >> """ >> >> Is there any reason why this method would have to explicitly check the type >> of its argument? Why can't it just accept any object that quacks like a >> function? > > Primarily because the expected approach for alternative objects that > want to expose signature in 3.3 was for them to provide the > __signature__ attribute directly, rather than providing the far > messier grab bag of eclectic attributes exposed by native Python > functions (there's also a tentative plan kicking around somewhere to > have a string version for 3.4, so we can start doing something useful > with functions implemented in C). > > I suspect the specific reason for the lack of ducktyping is merely to > provide a nice error message when people call the class method rather > than the module level function, so removing the check sounds > reasonable to me. The ultimate decision will be Georg's as RM, > though. Probably the quickest way to get a decision from him would be > a tracker issue set to release blocker, with a patch removing those > two lines and a test confirming that non-functions with the right > attributes are correctly accepted (e.g. by copying the attributes from > a real function to a custom object and checking > Signature.from_function gives the same answer). Thanks for the feedback, patch is up. https://bugs.python.org/issue17159 Stefan From oscar.j.benjamin at gmail.com Fri Feb 8 16:52:46 2013 From: oscar.j.benjamin at gmail.com (Oscar Benjamin) Date: Fri, 8 Feb 2013 15:52:46 +0000 Subject: [Python-Dev] why do we allow this syntax? In-Reply-To: <51151C28.3020101@simplistix.co.uk> References: <51151C28.3020101@simplistix.co.uk> Message-ID: On 8 February 2013 15:39, Chris Withers wrote: > Hi All, > > Just had a bit of an embarrassing incident in some code where I did: > > sometotal =+ somevalue > > I'm curious why this syntax is allowed? I'm sure there are good reasons, but > thought I'd ask... Because '+' can represent an unary prefix operator just the same as '-': >>> + + -- ++- +- -+ ++3 -3 Oscar From stefan_ml at behnel.de Fri Feb 8 16:54:36 2013 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 08 Feb 2013 16:54:36 +0100 Subject: [Python-Dev] Why does Signature.from_function() have to check the type of its argument? In-Reply-To: References: Message-ID: Nick Coghlan, 08.02.2013 16:20: > On Sat, Feb 9, 2013 at 1:06 AM, Benjamin Peterson wrote: >> 2013/2/8 Stefan Behnel: >>> I'm wondering about the purpose of this code in >>> inspect.Signature.from_function(): >>> >>> """ >>> if not isinstance(func, types.FunctionType): >>> raise TypeError('{!r} is not a Python function'.format(func)) >>> """ >>> >>> Is there any reason why this method would have to explicitly check the type >>> of its argument? Why can't it just accept any object that quacks like a >>> function? >> >> The signature() function checks for types.FunctionType in order to >> call Signature.from_function(). How would you reimplement that? It should call isfunction() instead of running an explicit type check. > I assumed Stefan was wanting to use Signature.from_function() to set > __signature__ (as soon as you do that, inspect.signature will do the > right thing). Absolutely. Stefan From storchaka at gmail.com Fri Feb 8 17:03:52 2013 From: storchaka at gmail.com (Serhiy Storchaka) Date: Fri, 08 Feb 2013 18:03:52 +0200 Subject: [Python-Dev] why do we allow this syntax? In-Reply-To: <51151C28.3020101@simplistix.co.uk> References: <51151C28.3020101@simplistix.co.uk> Message-ID: On 08.02.13 17:39, Chris Withers wrote: > Just had a bit of an embarrassing incident in some code where I did: > > sometotal =+ somevalue > > I'm curious why this syntax is allowed? I'm sure there are good reasons, > but thought I'd ask... And why this syntax is allowed? pi = 3,14 And this: fruits = [ 'apple', 'pear', 'banana' 'ananas', 'mango' ] From chris at simplistix.co.uk Fri Feb 8 17:05:54 2013 From: chris at simplistix.co.uk (Chris Withers) Date: Fri, 08 Feb 2013 16:05:54 +0000 Subject: [Python-Dev] why do we allow this syntax? In-Reply-To: References: <51151C28.3020101@simplistix.co.uk> Message-ID: <51152262.3080105@simplistix.co.uk> On 08/02/2013 15:42, Benjamin Peterson wrote: > 2013/2/8 Chris Withers: >> Hi All, >> >> Just had a bit of an embarrassing incident in some code where I did: >> >> sometotal =+ somevalue > > That's just a strange way of expressing > > sometotal = +somevalue Indeed, but why should this be possible? When could it do something useful? :-) Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk From benjamin at python.org Fri Feb 8 17:10:21 2013 From: benjamin at python.org (Benjamin Peterson) Date: Fri, 8 Feb 2013 11:10:21 -0500 Subject: [Python-Dev] why do we allow this syntax? In-Reply-To: <51152262.3080105@simplistix.co.uk> References: <51151C28.3020101@simplistix.co.uk> <51152262.3080105@simplistix.co.uk> Message-ID: 2013/2/8 Chris Withers : > On 08/02/2013 15:42, Benjamin Peterson wrote: >> >> 2013/2/8 Chris Withers: >>> >>> Hi All, >>> >>> Just had a bit of an embarrassing incident in some code where I did: >>> >>> sometotal =+ somevalue >> >> >> That's just a strange way of expressing >> >> sometotal = +somevalue > > > Indeed, but why should this be possible? When could it do something useful? > :-) + is a normal overridable operator. -- Regards, Benjamin From oscar.j.benjamin at gmail.com Fri Feb 8 17:17:55 2013 From: oscar.j.benjamin at gmail.com (Oscar Benjamin) Date: Fri, 8 Feb 2013 16:17:55 +0000 Subject: [Python-Dev] why do we allow this syntax? In-Reply-To: References: <51151C28.3020101@simplistix.co.uk> <51152262.3080105@simplistix.co.uk> Message-ID: On 8 February 2013 16:10, Benjamin Peterson wrote: > 2013/2/8 Chris Withers : >> On 08/02/2013 15:42, Benjamin Peterson wrote: >>> >>> 2013/2/8 Chris Withers: >>>> >>>> Hi All, >>>> >>>> Just had a bit of an embarrassing incident in some code where I did: >>>> >>>> sometotal =+ somevalue >>> >>> >>> That's just a strange way of expressing >>> >>> sometotal = +somevalue >> >> >> Indeed, but why should this be possible? When could it do something useful? >> :-) > > + is a normal overridable operator. Decimal.__pos__ uses it to return a Decimal instance that has the default precision of the current Decimal context: >>> from decimal import Decimal >>> d = Decimal('0.33333333333333333333333333333333333333') >>> d Decimal('0.33333333333333333333333333333333333333') >>> +d Decimal('0.3333333333333333333333333333') Oscar From status at bugs.python.org Fri Feb 8 18:07:28 2013 From: status at bugs.python.org (Python tracker) Date: Fri, 8 Feb 2013 18:07:28 +0100 (CET) Subject: [Python-Dev] Summary of Python tracker Issues Message-ID: <20130208170728.4C12556911@psf.upfronthosting.co.za> ACTIVITY SUMMARY (2013-02-01 - 2013-02-08) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 3847 ( -1) closed 25088 (+60) total 28935 (+59) Open issues with patches: 1674 Issues opened (45) ================== #13886: readline-related test_builtin failure http://bugs.python.org/issue13886 reopened by nadeem.vawda #17102: tarfile extract can write files outside the destination path http://bugs.python.org/issue17102 opened by gregory.p.smith #17103: ampersand "&" in path prevents compilation of Python http://bugs.python.org/issue17103 opened by gheorghe.mosteoru #17104: Tk() not defined in Tkinter module http://bugs.python.org/issue17104 opened by zephyr #17108: import silently prefers package over module when both availabl http://bugs.python.org/issue17108 opened by shai #17110: sys.argv docs should explaining how to handle encoding issues http://bugs.python.org/issue17110 opened by ncoghlan #17111: test_surrogates of test_fileio fails sometimes on OS X 10.4 http://bugs.python.org/issue17111 opened by ned.deily #17112: Some doctest-based tests fail when run with python -OO http://bugs.python.org/issue17112 opened by ned.deily #17113: argparse.RawDescriptionHelpFormatter should not delete blank l http://bugs.python.org/issue17113 opened by roysmith #17115: __loader__ = None should be fine http://bugs.python.org/issue17115 opened by brett.cannon #17116: xml.parsers.expat.(errors|model) don't set the __loader__ attr http://bugs.python.org/issue17116 opened by brett.cannon #17117: Update importlib.util.module_for_loader/set_loader to set when http://bugs.python.org/issue17117 opened by brett.cannon #17119: Integer overflow when passing large string to Tkinter http://bugs.python.org/issue17119 opened by serhiy.storchaka #17120: Mishandled _POSIX_C_SOURCE and _XOPEN_SOURCE in pyconfig.h http://bugs.python.org/issue17120 opened by RAW #17121: SSH upload for distutils http://bugs.python.org/issue17121 opened by christian.heimes #17123: Add OCSP support to ssl module http://bugs.python.org/issue17123 opened by christian.heimes #17126: test_gdb fails http://bugs.python.org/issue17126 opened by ddvento at ucar.edu #17127: multiprocessing.dummy.Pool does not accept maxtasksperchild ar http://bugs.python.org/issue17127 opened by Noah.Yetter #17128: OS X system openssl deprecated - installer should build local http://bugs.python.org/issue17128 opened by ned.deily #17130: Add runcall() function to profile.py and cProfile.py http://bugs.python.org/issue17130 opened by gvanrossum #17131: subprocess.Popen.terminate can raise exception on Posix http://bugs.python.org/issue17131 opened by siona #17134: Use Windows' certificate store for CA certs http://bugs.python.org/issue17134 opened by christian.heimes #17135: imp doc should direct to importlib http://bugs.python.org/issue17135 opened by eric.araujo #17136: ctypes tests fail with clang on non-OS X http://bugs.python.org/issue17136 opened by djc #17137: Malfunctioning compiled code in Python 3.3 x64 http://bugs.python.org/issue17137 opened by pepalogik #17138: XPath error in xml.etree.ElementTree http://bugs.python.org/issue17138 opened by Antoine2008 #17140: Provide a more obvious public ThreadPool API http://bugs.python.org/issue17140 opened by ncoghlan #17141: random.vonmisesvariate() hangs for large kappa http://bugs.python.org/issue17141 opened by serhiy.storchaka #17143: trace.py uses _warn without importing it http://bugs.python.org/issue17143 opened by Dmitry.Jemerov #17144: Distutils: "sdist register upload" ignores -r argument http://bugs.python.org/issue17144 opened by gwrtheyrn #17145: memoryview(array.array) http://bugs.python.org/issue17145 opened by dbrecht #17146: Improve test.support.import_fresh_module() http://bugs.python.org/issue17146 opened by eric.snow #17147: BytesIO should be mentioned in SpooledTemporaryFile documentat http://bugs.python.org/issue17147 opened by serhiy.storchaka #17148: mingw: nt thread model detection http://bugs.python.org/issue17148 opened by rpetrov #17149: random.vonmisesvariate() results range is inconsistent for sma http://bugs.python.org/issue17149 opened by serhiy.storchaka #17150: pprint could use line continuation for long string literals http://bugs.python.org/issue17150 opened by pitrou #17152: Array module should support "boolean" natively http://bugs.python.org/issue17152 opened by jcea #17153: tarfile extract fails when Unicode in pathname http://bugs.python.org/issue17153 opened by vinay.sajip #17154: the 'ignore' pdb command raises IndexError http://bugs.python.org/issue17154 opened by xdegaye #17155: Logging throwing UnicodeEncodeError exception http://bugs.python.org/issue17155 opened by Kronuz #17156: Tools/i18n/pygettext.py doesn't parse unicode string. http://bugs.python.org/issue17156 opened by umedoblock #17157: issubclass() should accept iterables in 2nd arg http://bugs.python.org/issue17157 opened by ramchandra.apte #17158: help() module searcher text is misleading http://bugs.python.org/issue17158 opened by ramchandra.apte #17159: Remove explicit type check from inspect.Signature.from_functio http://bugs.python.org/issue17159 opened by scoder #17160: test_urllib2net fails http://bugs.python.org/issue17160 opened by ddvento at ucar.edu Most recent 15 issues with no replies (15) ========================================== #17160: test_urllib2net fails http://bugs.python.org/issue17160 #17158: help() module searcher text is misleading http://bugs.python.org/issue17158 #17155: Logging throwing UnicodeEncodeError exception http://bugs.python.org/issue17155 #17154: the 'ignore' pdb command raises IndexError http://bugs.python.org/issue17154 #17153: tarfile extract fails when Unicode in pathname http://bugs.python.org/issue17153 #17148: mingw: nt thread model detection http://bugs.python.org/issue17148 #17147: BytesIO should be mentioned in SpooledTemporaryFile documentat http://bugs.python.org/issue17147 #17140: Provide a more obvious public ThreadPool API http://bugs.python.org/issue17140 #17138: XPath error in xml.etree.ElementTree http://bugs.python.org/issue17138 #17131: subprocess.Popen.terminate can raise exception on Posix http://bugs.python.org/issue17131 #17130: Add runcall() function to profile.py and cProfile.py http://bugs.python.org/issue17130 #17127: multiprocessing.dummy.Pool does not accept maxtasksperchild ar http://bugs.python.org/issue17127 #17120: Mishandled _POSIX_C_SOURCE and _XOPEN_SOURCE in pyconfig.h http://bugs.python.org/issue17120 #17119: Integer overflow when passing large string to Tkinter http://bugs.python.org/issue17119 #17116: xml.parsers.expat.(errors|model) don't set the __loader__ attr http://bugs.python.org/issue17116 Most recent 15 issues waiting for review (15) ============================================= #17159: Remove explicit type check from inspect.Signature.from_functio http://bugs.python.org/issue17159 #17156: Tools/i18n/pygettext.py doesn't parse unicode string. http://bugs.python.org/issue17156 #17148: mingw: nt thread model detection http://bugs.python.org/issue17148 #17147: BytesIO should be mentioned in SpooledTemporaryFile documentat http://bugs.python.org/issue17147 #17145: memoryview(array.array) http://bugs.python.org/issue17145 #17143: trace.py uses _warn without importing it http://bugs.python.org/issue17143 #17141: random.vonmisesvariate() hangs for large kappa http://bugs.python.org/issue17141 #17137: Malfunctioning compiled code in Python 3.3 x64 http://bugs.python.org/issue17137 #17111: test_surrogates of test_fileio fails sometimes on OS X 10.4 http://bugs.python.org/issue17111 #17095: Modules/Setup *shared* support broken http://bugs.python.org/issue17095 #17094: sys._current_frames() reports too many/wrong stack frames http://bugs.python.org/issue17094 #17089: Expat parser parses strings only when XML encoding is UTF-8 http://bugs.python.org/issue17089 #17086: backport cross-build patches to the 2.7 branch http://bugs.python.org/issue17086 #17082: Fix test discovery for test_dbm*.py http://bugs.python.org/issue17082 #17080: A better error message for float() http://bugs.python.org/issue17080 Top 10 most discussed issues (10) ================================= #17137: Malfunctioning compiled code in Python 3.3 x64 http://bugs.python.org/issue17137 18 msgs #17121: SSH upload for distutils http://bugs.python.org/issue17121 15 msgs #5308: cannot marshal objects with more than 2**31 elements http://bugs.python.org/issue5308 12 msgs #17047: Fix double double words words http://bugs.python.org/issue17047 11 msgs #13355: random.triangular error when low = high=mode http://bugs.python.org/issue13355 10 msgs #16038: ftplib: unlimited readline() from connection http://bugs.python.org/issue16038 10 msgs #17128: OS X system openssl deprecated - installer should build local http://bugs.python.org/issue17128 10 msgs #17108: import silently prefers package over module when both availabl http://bugs.python.org/issue17108 9 msgs #16555: Add es_cu to locale aliases http://bugs.python.org/issue16555 8 msgs #17100: rotating an ordereddict http://bugs.python.org/issue17100 8 msgs Issues closed (58) ================== #2840: Expat parser locks XML source file if ContentHandler raises an http://bugs.python.org/issue2840 closed by serhiy.storchaka #3754: cross-compilation support for python build http://bugs.python.org/issue3754 closed by rpetrov #3871: cross and native build of python for mingw* hosts http://bugs.python.org/issue3871 closed by doko #4011: Create DAG for PEP 101 http://bugs.python.org/issue4011 closed by georg.brandl #5289: ctypes.util.find_library does not work under Solaris http://bugs.python.org/issue5289 closed by python-dev #6972: zipfile.ZipFile overwrites files outside destination path http://bugs.python.org/issue6972 closed by gregory.p.smith #8754: quote bad module name in ImportError-like messages http://bugs.python.org/issue8754 closed by brett.cannon #11159: Sax parser crashes if given unicode file name http://bugs.python.org/issue11159 closed by serhiy.storchaka #12177: re.match raises MemoryError http://bugs.python.org/issue12177 closed by serhiy.storchaka #12268: file readline, readlines & readall methods can lose data on EI http://bugs.python.org/issue12268 closed by gregory.p.smith #12779: Update packaging documentation http://bugs.python.org/issue12779 closed by eric.araujo #13192: ImportError silences low-level OS errors http://bugs.python.org/issue13192 closed by pitrou #13541: HTTPResponse (urllib) has no attribute read1 needed for TextIO http://bugs.python.org/issue13541 closed by serhiy.storchaka #14263: switch_index_if_fails fails on py2 http://bugs.python.org/issue14263 closed by serhiy.storchaka #14340: Update embedded copy of expat - fix security & crash issues http://bugs.python.org/issue14340 closed by gregory.p.smith #14580: imp.reload can fail for sub-modules http://bugs.python.org/issue14580 closed by brett.cannon #14724: kill imp.load_dynamic's third argument http://bugs.python.org/issue14724 closed by brett.cannon #14940: Usage documentation for pysetup http://bugs.python.org/issue14940 closed by eric.araujo #15116: remove out-of-date Mac application scripting documentation http://bugs.python.org/issue15116 closed by ned.deily #15298: _sysconfigdata is generated in srcdir, not builddir http://bugs.python.org/issue15298 closed by ned.deily #15344: devinabox: failure when running make_a_box multiple times http://bugs.python.org/issue15344 closed by brett.cannon #15359: Sockets support for CAN_BCM http://bugs.python.org/issue15359 closed by neologix #15483: CROSS: initialise include and library paths in setup.py http://bugs.python.org/issue15483 closed by rpetrov #15633: httplib.response is not closed after all data has been read http://bugs.python.org/issue15633 closed by pitrou #15881: multiprocessing 'NoneType' object is not callable http://bugs.python.org/issue15881 closed by benjamin.peterson #16137: Using time.asctime() with an array with negative tm_hour cause http://bugs.python.org/issue16137 closed by belopolsky #16256: permissions wrong on Mac doc dir http://bugs.python.org/issue16256 closed by ned.deily #16526: Python does not cross compile properly http://bugs.python.org/issue16526 closed by doko #16698: test_posix.test_getgroups fails on some systems http://bugs.python.org/issue16698 closed by ned.deily #16723: io.TextIOWrapper on urllib.request.urlopen terminates prematur http://bugs.python.org/issue16723 closed by serhiy.storchaka #16811: email.message.Message flatten dies of list index out of range http://bugs.python.org/issue16811 closed by r.david.murray #16903: subprocess.Popen.communicate with universal_newlines=True does http://bugs.python.org/issue16903 closed by serhiy.storchaka #16948: email.mime.text.MIMEText: QP encoding broken with charset!=ISO http://bugs.python.org/issue16948 closed by r.david.murray #17015: mock could be smarter and inspect the spec's signature http://bugs.python.org/issue17015 closed by pitrou #17034: Use Py_CLEAR() in stringobject.c and bytesobject.c http://bugs.python.org/issue17034 closed by serhiy.storchaka #17069: HTTP result code in urllib2.urlopen() file object undocumented http://bugs.python.org/issue17069 closed by orsenthil #17073: Integer overflow in sqlite module http://bugs.python.org/issue17073 closed by serhiy.storchaka #17076: shutil.copytree failing on xattr-less filesystems (like NFS) http://bugs.python.org/issue17076 closed by hynek #17077: Fix test_tools hangs http://bugs.python.org/issue17077 closed by serhiy.storchaka #17091: thread.lock.acquire docstring bug http://bugs.python.org/issue17091 closed by r.david.murray #17092: Disable TIPC in configure http://bugs.python.org/issue17092 closed by ddvento at ucar.edu #17098: Set __loader__ on modules imported by the C level http://bugs.python.org/issue17098 closed by brett.cannon #17105: Python 3.2 segfault http://bugs.python.org/issue17105 closed by skrah #17106: assertion error in IO reading text file as binary http://bugs.python.org/issue17106 closed by serhiy.storchaka #17107: test_sni in test_urllib2net could be enabled http://bugs.python.org/issue17107 closed by pitrou #17109: unittest.mock has wrong heading levels http://bugs.python.org/issue17109 closed by python-dev #17114: Python IDLE GUI does not open in Ubuntu 12.04 http://bugs.python.org/issue17114 closed by serhiy.storchaka #17118: Add tests for testing Python-Tcl interaction http://bugs.python.org/issue17118 closed by serhiy.storchaka #17122: Fix and cleanup test_functools http://bugs.python.org/issue17122 closed by serhiy.storchaka #17124: import subprocess hangs for ~25 seconds, time.py file in dir - http://bugs.python.org/issue17124 closed by neologix #17125: tokenizer.tokenize passes a bytes object to str.startswith http://bugs.python.org/issue17125 closed by r.david.murray #17129: Include CA bundle and provide access to system's CA http://bugs.python.org/issue17129 closed by eric.araujo #17132: yield_arg missing from symbol.sym_name http://bugs.python.org/issue17132 closed by python-dev #17133: ssl.wrap_socket doesn't take server_hostname as a kwarg http://bugs.python.org/issue17133 closed by pitrou #17139: dateTime.now() return format missing decimal seconds. http://bugs.python.org/issue17139 closed by mark.dickinson #17142: test_any calls all() instead of any() http://bugs.python.org/issue17142 closed by r.david.murray #17151: Python 3 changement of behavior with __ne__: documentation not http://bugs.python.org/issue17151 closed by mark.dickinson #1597850: Cross compiling patches for MINGW http://bugs.python.org/issue1597850 closed by hanwen From pje at telecommunity.com Fri Feb 8 19:46:08 2013 From: pje at telecommunity.com (PJ Eby) Date: Fri, 8 Feb 2013 13:46:08 -0500 Subject: [Python-Dev] Why does Signature.from_function() have to check the type of its argument? In-Reply-To: References: Message-ID: On Fri, Feb 8, 2013 at 10:54 AM, Stefan Behnel wrote: > Nick Coghlan, 08.02.2013 16:20: >> On Sat, Feb 9, 2013 at 1:06 AM, Benjamin Peterson wrote: >>> 2013/2/8 Stefan Behnel: >>>> I'm wondering about the purpose of this code in >>>> inspect.Signature.from_function(): >>>> >>>> """ >>>> if not isinstance(func, types.FunctionType): >>>> raise TypeError('{!r} is not a Python function'.format(func)) >>>> """ >>>> >>>> Is there any reason why this method would have to explicitly check the type >>>> of its argument? Why can't it just accept any object that quacks like a >>>> function? >>> >>> The signature() function checks for types.FunctionType in order to >>> call Signature.from_function(). How would you reimplement that? > > It should call isfunction() instead of running an explicit type check. Isn't it possible now for an object to implement __instancecheck__ and claim to be an instance of FunctionType, anyway? (For that matter, shouldn't there be some ABCs for this?) From chris at simplistix.co.uk Fri Feb 8 18:45:28 2013 From: chris at simplistix.co.uk (Chris Withers) Date: Fri, 08 Feb 2013 17:45:28 +0000 Subject: [Python-Dev] why do we allow this syntax? In-Reply-To: References: <51151C28.3020101@simplistix.co.uk> <51152262.3080105@simplistix.co.uk> Message-ID: <511539B8.7010701@simplistix.co.uk> On 08/02/2013 16:17, Oscar Benjamin wrote: > Decimal.__pos__ uses it to return a Decimal instance that has the > default precision of the current Decimal context: > >>>> from decimal import Decimal >>>> d = Decimal('0.33333333333333333333333333333333333333') >>>> d > Decimal('0.33333333333333333333333333333333333333') >>>> +d > Decimal('0.3333333333333333333333333333') That's the answer I was hoping wouldn't be coming... Oh well, guess I'll have to improve on my sloppy typing. Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk From catch-all at masklinn.net Fri Feb 8 21:36:35 2013 From: catch-all at masklinn.net (Xavier Morel) Date: Fri, 8 Feb 2013 21:36:35 +0100 Subject: [Python-Dev] why do we allow this syntax? In-Reply-To: <511539B8.7010701@simplistix.co.uk> References: <51151C28.3020101@simplistix.co.uk> <51152262.3080105@simplistix.co.uk> <511539B8.7010701@simplistix.co.uk> Message-ID: <55EBC98F-E03E-4EF8-A2B7-18FE6B58BBF9@masklinn.net> On 2013-02-08, at 18:45 , Chris Withers wrote: > On 08/02/2013 16:17, Oscar Benjamin wrote: >> Decimal.__pos__ uses it to return a Decimal instance that has the >> default precision of the current Decimal context: >> >>>>> from decimal import Decimal >>>>> d = Decimal('0.33333333333333333333333333333333333333') >>>>> d >> Decimal('0.33333333333333333333333333333333333333') >>>>> +d >> Decimal('0.3333333333333333333333333333') > > That's the answer I was hoping wouldn't be coming... > > Oh well, guess I'll have to improve on my sloppy typing. Or just run flake8 (with a bit of configuration to disable the annoying stuff). From stefan_ml at behnel.de Fri Feb 8 22:14:28 2013 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 08 Feb 2013 22:14:28 +0100 Subject: [Python-Dev] Why does Signature.from_function() have to check the type of its argument? In-Reply-To: References: Message-ID: PJ Eby, 08.02.2013 19:46: > On Fri, Feb 8, 2013 at 10:54 AM, Stefan Behnel wrote: >> Nick Coghlan, 08.02.2013 16:20: >>> On Sat, Feb 9, 2013 at 1:06 AM, Benjamin Peterson wrote: >>>> 2013/2/8 Stefan Behnel: >>>>> I'm wondering about the purpose of this code in >>>>> inspect.Signature.from_function(): >>>>> >>>>> """ >>>>> if not isinstance(func, types.FunctionType): >>>>> raise TypeError('{!r} is not a Python function'.format(func)) >>>>> """ >>>>> >>>>> Is there any reason why this method would have to explicitly check the type >>>>> of its argument? Why can't it just accept any object that quacks like a >>>>> function? >>>> >>>> The signature() function checks for types.FunctionType in order to >>>> call Signature.from_function(). How would you reimplement that? >> >> It should call isfunction() instead of running an explicit type check. > > Isn't it possible now for an object to implement __instancecheck__ and > claim to be an instance of FunctionType, anyway? (For that matter, > shouldn't there be some ABCs for this?) Wow, good call. Providing an __instancecheck__() method that simply says yes when it's asked for PyFunction_Type really works. Thanks! Stefan From graffatcolmingov at gmail.com Fri Feb 8 22:35:59 2013 From: graffatcolmingov at gmail.com (Ian Cordasco) Date: Fri, 8 Feb 2013 16:35:59 -0500 Subject: [Python-Dev] why do we allow this syntax? In-Reply-To: <55EBC98F-E03E-4EF8-A2B7-18FE6B58BBF9@masklinn.net> References: <51151C28.3020101@simplistix.co.uk> <51152262.3080105@simplistix.co.uk> <511539B8.7010701@simplistix.co.uk> <55EBC98F-E03E-4EF8-A2B7-18FE6B58BBF9@masklinn.net> Message-ID: On Feb 8, 2013 3:37 PM, "Xavier Morel" wrote: > > On 2013-02-08, at 18:45 , Chris Withers wrote: > > > On 08/02/2013 16:17, Oscar Benjamin wrote: > >> Decimal.__pos__ uses it to return a Decimal instance that has the > >> default precision of the current Decimal context: > >> > >>>>> from decimal import Decimal > >>>>> d = Decimal('0.33333333333333333333333333333333333333') > >>>>> d > >> Decimal('0.33333333333333333333333333333333333333') > >>>>> +d > >> Decimal('0.3333333333333333333333333333') > > > > That's the answer I was hoping wouldn't be coming... > > > > Oh well, guess I'll have to improve on my sloppy typing. > > Or just run flake8 (with a bit of configuration to disable the annoying stuff). As flake8's maintainer, I second this. -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefan_ml at behnel.de Fri Feb 8 23:44:11 2013 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 08 Feb 2013 23:44:11 +0100 Subject: [Python-Dev] Why does Signature.from_function() have to check the type of its argument? In-Reply-To: References: Message-ID: Stefan Behnel, 08.02.2013 22:14: > PJ Eby, 08.02.2013 19:46: >> On Fri, Feb 8, 2013 at 10:54 AM, Stefan Behnel wrote: >>> Nick Coghlan, 08.02.2013 16:20: >>>> On Sat, Feb 9, 2013 at 1:06 AM, Benjamin Peterson wrote: >>>>> 2013/2/8 Stefan Behnel: >>>>>> I'm wondering about the purpose of this code in >>>>>> inspect.Signature.from_function(): >>>>>> >>>>>> """ >>>>>> if not isinstance(func, types.FunctionType): >>>>>> raise TypeError('{!r} is not a Python function'.format(func)) >>>>>> """ >>>>>> >>>>>> Is there any reason why this method would have to explicitly check the type >>>>>> of its argument? Why can't it just accept any object that quacks like a >>>>>> function? >>>>> >>>>> The signature() function checks for types.FunctionType in order to >>>>> call Signature.from_function(). How would you reimplement that? >>> >>> It should call isfunction() instead of running an explicit type check. >> >> Isn't it possible now for an object to implement __instancecheck__ and >> claim to be an instance of FunctionType, anyway? (For that matter, >> shouldn't there be some ABCs for this?) > > Wow, good call. Providing an __instancecheck__() method that simply says > yes when it's asked for PyFunction_Type really works. Argh - sorry, got it all wrong. "__instancecheck__()" works exactly the other way round. In the type check above, it's the FunctionType type that gets asked for an instance check, which doesn't help at all in this case because it simply doesn't know about the desired subtype relation. It would work if type(func).__instancecheck__() was used, but that doesn't happen. So, no help from that side, sadly. Stefan From tom at scrace.org Sat Feb 9 10:59:13 2013 From: tom at scrace.org (Thomas Scrace) Date: Sat, 9 Feb 2013 09:59:13 +0000 Subject: [Python-Dev] os.path.join failure mode Message-ID: If a function (or other non-string object) is accidentally passed as an argument to os.path.join() the result is an AttributeError: In [3]: os.path.join(fn, "path") > --------------------------------------------------------------------------- > AttributeError Traceback (most recent call last) > /home/tom/ in () > ----> 1 os.path.join(fn, "path") > /usr/lib/python2.7/posixpath.pyc in join(a, *p) > 66 if b.startswith('/'): > 67 path = b > ---> 68 elif path == '' or path.endswith('/'): > 69 path += b > 70 else: > AttributeError: 'function' object has no attribute 'endswith' It's relatively easy to run into this if you mean to pass the return value of a function (fn()) as the argument but accidentally forget to append parens (()) to the callable, thus passing the function itself instead. Would it not be more helpful to raise a TypeError("Argument must be a string") than the slightly more mysterious AttributeError? It's not the most difficult error in the world to figure out, I admit, but a TypeError just seems like the more correct thing to do here. Thanks, Tom -------------- next part -------------- An HTML attachment was scrubbed... URL: From rdmurray at bitdance.com Sat Feb 9 14:31:00 2013 From: rdmurray at bitdance.com (R. David Murray) Date: Sat, 09 Feb 2013 08:31:00 -0500 Subject: [Python-Dev] os.path.join failure mode In-Reply-To: References: Message-ID: <20130209133100.D8D64250BC8@webabinitio.net> On Sat, 09 Feb 2013 09:59:13 +0000, Thomas Scrace wrote: > If a function (or other non-string object) is accidentally passed as an > argument to os.path.join() the result is an AttributeError: > > > In [3]: os.path.join(fn, "path") > > --------------------------------------------------------------------------- > > AttributeError Traceback (most recent call last) > > /home/tom/ in () > > ----> 1 os.path.join(fn, "path") > > /usr/lib/python2.7/posixpath.pyc in join(a, *p) > > 66 if b.startswith('/'): > > 67 path = b > > ---> 68 elif path == '' or path.endswith('/'): > > 69 path += b > > 70 else: > > AttributeError: 'function' object has no attribute 'endswith' > > > > It's relatively easy to run into this if you mean to pass the return value > of a function (fn()) as the argument but accidentally forget to append > parens (()) to the callable, thus passing the function itself instead. > > Would it not be more helpful to raise a TypeError("Argument must be a > string") than the slightly more mysterious AttributeError? > > It's not the most difficult error in the world to figure out, I admit, but > a TypeError just seems like the more correct thing to do here. We don't generally do that kind of type checking in Python. Sometimes the error that results is obscure enough that adding an early check of some sort is worth it, but this one is very clear, so I don't see that an additional check would add much value here. The reason we avoid such type checks is that we prefer to operate via "duck typing", which means that if an object behaves like the expected input, it is accepted. Here, if we did an explicit type check for str, it would prevent join from working on an "act alike" object that had just enough str like methods to work correctly in os.join (for example, some specialized object that was among other things a filename proxy). Granted, this is pretty far-fetched for os.path.join, but the general principle applies :) --David From andrew.svetlov at gmail.com Sat Feb 9 14:52:20 2013 From: andrew.svetlov at gmail.com (Andrew Svetlov) Date: Sat, 9 Feb 2013 15:52:20 +0200 Subject: [Python-Dev] Force to use Py_CLEAR and Py_VISIT in example for Python extending Message-ID: For now http://docs.python.org/3/extending/newtypes.html#supporting-cyclic-garbage-collection at first the doc demonstrate long way than note Py_CLEAR and Py_VISIT macroses. I like to remove code similar to if (self->first) { vret = visit(self->first, arg); if (vret != 0) return vret; } and tmp = self->first; self->first = NULL; Py_XDECREF(tmp); and replace those to Py_VISIT and Py_CLEAN. I think we have to demonstrate best practices in our examples. Let's py2.7 branch live untouched, change only docs for python 3.x Any objections? -- Thanks, Andrew Svetlov From ncoghlan at gmail.com Sat Feb 9 14:53:55 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 9 Feb 2013 23:53:55 +1000 Subject: [Python-Dev] os.path.join failure mode In-Reply-To: <20130209133100.D8D64250BC8@webabinitio.net> References: <20130209133100.D8D64250BC8@webabinitio.net> Message-ID: On Sat, Feb 9, 2013 at 11:31 PM, R. David Murray wrote: > The reason we avoid such type checks is that we prefer to operate via > "duck typing", which means that if an object behaves like the expected > input, it is accepted. Here, if we did an explicit type check for str, > it would prevent join from working on an "act alike" object that had > just enough str like methods to work correctly in os.join (for example, > some specialized object that was among other things a filename proxy). > Granted, this is pretty far-fetched for os.path.join, but the general > principle applies :) I occasionally ponder the possibility of a time where we have a string.Text ABC, and can stop hardcoding checks for isinstance(str) everywhere :P (No, I'm not volunteering to write it. And if anyone does want to discuss it further, please start a new thread on python-ideas, as python-dev isn't the right venue) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From benjamin at python.org Sat Feb 9 14:54:20 2013 From: benjamin at python.org (Benjamin Peterson) Date: Sat, 9 Feb 2013 08:54:20 -0500 Subject: [Python-Dev] Force to use Py_CLEAR and Py_VISIT in example for Python extending In-Reply-To: References: Message-ID: 2013/2/9 Andrew Svetlov : > For now http://docs.python.org/3/extending/newtypes.html#supporting-cyclic-garbage-collection > at first the doc demonstrate long way than note Py_CLEAR and Py_VISIT macroses. > > I like to remove code similar to > > if (self->first) { > vret = visit(self->first, arg); > if (vret != 0) > return vret; > } > > and > > tmp = self->first; > self->first = NULL; > Py_XDECREF(tmp); > > and replace those to Py_VISIT and Py_CLEAN. > > I think we have to demonstrate best practices in our examples. > Let's py2.7 branch live untouched, change only docs for python 3.x > > Any objections? IMO, it's fine if you change 2.7, too. -- Regards, Benjamin From ncoghlan at gmail.com Sat Feb 9 15:01:55 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 10 Feb 2013 00:01:55 +1000 Subject: [Python-Dev] Force to use Py_CLEAR and Py_VISIT in example for Python extending In-Reply-To: References: Message-ID: On Sat, Feb 9, 2013 at 11:52 PM, Andrew Svetlov wrote: > I think we have to demonstrate best practices in our examples. > Let's py2.7 branch live untouched, change only docs for python 3.x > > Any objections? For debugging purposes, I think extension developers do need to understand what those macros are doing under the covers. However, the order of presentation should definitely be reversed to "here is how you should write it", followed by an explanation of what that actually means with the macros expanded. For Py_CLEAR, it would be OK to just link to the macro's entry in the C API docs, but for Py_VISIT, I think it's worth expanding out the full example. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From tom at scrace.org Sat Feb 9 15:35:33 2013 From: tom at scrace.org (Thomas Scrace) Date: Sat, 9 Feb 2013 14:35:33 +0000 (UTC) Subject: [Python-Dev] os.path.join failure mode References: <20130209133100.D8D64250BC8@webabinitio.net> Message-ID: R. David Murray bitdance.com> writes: > The reason we avoid such type checks is that we prefer to operate via > "duck typing", which means that if an object behaves like the expected > input, it is accepted. Here, if we did an explicit type check for str, > it would prevent join from working on an "act alike" object that had > just enough str like methods to work correctly in os.join (for example, > some specialized object that was among other things a filename proxy). I see, that makes sense. Thanks. I guess this actually goes to the heart of the flexibility of dynamic/weakly-typed languages. If we wanted to strictly enforce the type of a function's arguments we would use a strong type system. From rdmurray at bitdance.com Sat Feb 9 18:15:28 2013 From: rdmurray at bitdance.com (R. David Murray) Date: Sat, 09 Feb 2013 12:15:28 -0500 Subject: [Python-Dev] os.path.join failure mode In-Reply-To: References: <20130209133100.D8D64250BC8@webabinitio.net> Message-ID: <20130209171528.BCC11250030@webabinitio.net> On Sat, 09 Feb 2013 14:35:33 +0000, Thomas Scrace wrote: > R. David Murray bitdance.com> writes: > > > The reason we avoid such type checks is that we prefer to operate via > > "duck typing", which means that if an object behaves like the expected > > input, it is accepted. Here, if we did an explicit type check for str, > > it would prevent join from working on an "act alike" object that had > > just enough str like methods to work correctly in os.join (for example, > > some specialized object that was among other things a filename proxy). > > I see, that makes sense. Thanks. I guess this actually goes to the heart of the > flexibility of dynamic/weakly-typed languages. If we wanted to strictly enforce > the type of a function's arguments we would use a strong type system. No, it is more the difference between *statically* typed and dynamically typed. Python is a strongly typed language (every object has a specific type). --David From tom at scrace.org Sat Feb 9 18:57:14 2013 From: tom at scrace.org (Thomas Scrace) Date: Sat, 9 Feb 2013 17:57:14 +0000 Subject: [Python-Dev] os.path.join failure mode In-Reply-To: <20130209171528.BCC11250030@webabinitio.net> References: <20130209133100.D8D64250BC8@webabinitio.net> <20130209171528.BCC11250030@webabinitio.net> Message-ID: On 9 February 2013 17:15, R. David Murray wrote: > > No, it is more the difference between *statically* typed and dynamically > typed. Python is a strongly typed language (every object has a specific > type). > > > Yes, sorry, I think I probably have my terminology confused. What I really meant by 'strongly typed' was something like "a language where values passed to functions must be of a specified type". That is to say, something more along the lines of C, where function definitions explicitly specify the type of each argument, and where it is an error to pass to a function an argument whose type does not match that in the definition. Sorry to turn this into programming design 101, I know it's not really the place for it. -------------- next part -------------- An HTML attachment was scrubbed... URL: From tjreedy at udel.edu Sat Feb 9 19:57:41 2013 From: tjreedy at udel.edu (Terry Reedy) Date: Sat, 09 Feb 2013 13:57:41 -0500 Subject: [Python-Dev] os.path.join failure mode In-Reply-To: <20130209133100.D8D64250BC8@webabinitio.net> References: <20130209133100.D8D64250BC8@webabinitio.net> Message-ID: On 2/9/2013 8:31 AM, R. David Murray wrote: > On Sat, 09 Feb 2013 09:59:13 +0000, Thomas Scrace wrote: >> If a function (or other non-string object) is accidentally passed as an >> argument to os.path.join() the result is an AttributeError: >> >> >> In [3]: os.path.join(fn, "path") >>> --------------------------------------------------------------------------- >>> AttributeError Traceback (most recent call last) >>> /home/tom/ in () >>> ----> 1 os.path.join(fn, "path") >>> /usr/lib/python2.7/posixpath.pyc in join(a, *p) >>> 66 if b.startswith('/'): >>> 67 path = b >>> ---> 68 elif path == '' or path.endswith('/'): >>> 69 path += b >>> 70 else: >>> AttributeError: 'function' object has no attribute 'endswith' >> >> It's relatively easy to run into this if you mean to pass the return value >> of a function (fn()) as the argument but accidentally forget to append >> parens (()) to the callable, thus passing the function itself instead. >> >> Would it not be more helpful to raise a TypeError("Argument must be a >> string") than the slightly more mysterious AttributeError? >> >> It's not the most difficult error in the world to figure out, I admit, but >> a TypeError just seems like the more correct thing to do here. I agree. Since the exception type is not documented and since no one should intentionally pass anything but strings, and therefore should not be writing try: os.path.join(a,b) except AttributeError: barf() I think it would be acceptable to make a change in 3.4. > We don't generally do that kind of type checking in Python. Sometimes the > error that results is obscure enough that adding an early check of some > sort is worth it, but this one is very clear, so I don't see that an > additional check would add much value here. Changing AttributeError to TypeError only requires try-except, which is cheap if there is no error, not an early type check. > The reason we avoid such type checks is that we prefer to operate via > "duck typing", which means that if an object behaves like the expected > input, it is accepted. And catching the existing AttributeError allows work-alikes. I agree that an isinstance check is bad as well as unnecessary. As it is, the core code for path, given above, is *already* wrapped in try: except TypeError, (to give a more friendly error message!, as TypeError is most like from a unicode-bytes mixing). So there would be no extra cost for correct calls and all that is needed is another except clause. except AttributeError as e: bad = e.args[0].split()[0] msg = "all components must be strings; one is a {}".format(bad) raise TypeError("all components must be strings") Thomas, unless David or someone else shoots down this idea here, I say go ahead and open an enhancement issue and add terry.reedy as nosy. -- Terry Jan Reedy From rdmurray at bitdance.com Sat Feb 9 21:08:45 2013 From: rdmurray at bitdance.com (R. David Murray) Date: Sat, 09 Feb 2013 15:08:45 -0500 Subject: [Python-Dev] os.path.join failure mode In-Reply-To: References: <20130209133100.D8D64250BC8@webabinitio.net> Message-ID: <20130209200845.87BB125003E@webabinitio.net> On Sat, 09 Feb 2013 13:57:41 -0500, Terry Reedy wrote: > On 2/9/2013 8:31 AM, R. David Murray wrote: > Changing AttributeError to TypeError only requires try-except, which is > cheap if there is no error, not an early type check. > > > The reason we avoid such type checks is that we prefer to operate via > > "duck typing", which means that if an object behaves like the expected > > input, it is accepted. > > And catching the existing AttributeError allows work-alikes. I agree > that an isinstance check is bad as well as unnecessary. > > As it is, the core code for path, given above, is *already* wrapped in > try: except TypeError, (to give a more friendly error message!, as > TypeError is most like from a unicode-bytes mixing). So there would be > no extra cost for correct calls and all that is needed is another except > clause. > > except AttributeError as e: > bad = e.args[0].split()[0] > msg = "all components must be strings; one is a {}".format(bad) > raise TypeError("all components must be strings") > > Thomas, unless David or someone else shoots down this idea here, I say > go ahead and open an enhancement issue and add terry.reedy as nosy. Especially given that there is already a try/except there, this seems fine to me. I think perhaps the error text would be better if it referred only to the type that is invalid, not to str. So something like: TypeError("object of type {} is not valid as a path "component".format(type(bad))) --David From tom at scrace.org Sat Feb 9 22:18:15 2013 From: tom at scrace.org (Thomas Scrace) Date: Sat, 9 Feb 2013 21:18:15 +0000 Subject: [Python-Dev] os.path.join failure mode In-Reply-To: <20130209200845.87BB125003E@webabinitio.net> References: <20130209133100.D8D64250BC8@webabinitio.net> <20130209200845.87BB125003E@webabinitio.net> Message-ID: On 9 February 2013 20:08, R. David Murray wrote: > > > Especially given that there is already a try/except there, this seems > fine to me. I think perhaps the error text would be better if it > referred only to the type that is invalid, not to str. So something > like: > > TypeError("object of type {} is not valid as a path > "component".format(type(bad))) > > --David I agree that this is a better error message. I have raised an enhancement issue as Terry suggested: http://bugs.python.org/issue17174 From greg.ewing at canterbury.ac.nz Sun Feb 10 00:23:17 2013 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sun, 10 Feb 2013 12:23:17 +1300 Subject: [Python-Dev] os.path.join failure mode In-Reply-To: References: <20130209133100.D8D64250BC8@webabinitio.net> Message-ID: <5116DA65.6090302@canterbury.ac.nz> Terry Reedy wrote: > I agree. Since the exception type is not documented and since no one > should intentionally pass anything but strings, and therefore should not > be writing > > try: > os.path.join(a,b) > except AttributeError: > barf() > > I think it would be acceptable to make a change in 3.4. Why should we go out of our way to address this particular case, and not any of the infinitely many other situations where you could forget to add parens to a function call? -- Greg From eric at trueblade.com Sun Feb 10 00:45:51 2013 From: eric at trueblade.com (Eric V. Smith) Date: Sat, 09 Feb 2013 18:45:51 -0500 Subject: [Python-Dev] os.path.join failure mode In-Reply-To: <5116DA65.6090302@canterbury.ac.nz> References: <20130209133100.D8D64250BC8@webabinitio.net> <5116DA65.6090302@canterbury.ac.nz> Message-ID: <5116DFAF.90802@trueblade.com> On 2/9/2013 6:23 PM, Greg Ewing wrote: > Terry Reedy wrote: >> I agree. Since the exception type is not documented and since no one >> should intentionally pass anything but strings, and therefore should >> not be writing >> >> try: >> os.path.join(a,b) >> except AttributeError: >> barf() >> >> I think it would be acceptable to make a change in 3.4. > > Why should we go out of our way to address this particular > case, and not any of the infinitely many other situations > where you could forget to add parens to a function call? > I agree we do not want to go down this slippery slope. The number of such places in the standard library is huge. The existing error message is one that a user should be able to understand, since it's so common. -- Eric. From senthil at uthcode.com Sun Feb 10 08:39:01 2013 From: senthil at uthcode.com (Senthil Kumaran) Date: Sat, 9 Feb 2013 23:39:01 -0800 Subject: [Python-Dev] os.path.join failure mode In-Reply-To: <5116DFAF.90802@trueblade.com> References: <20130209133100.D8D64250BC8@webabinitio.net> <5116DA65.6090302@canterbury.ac.nz> <5116DFAF.90802@trueblade.com> Message-ID: And this is not just with Python. Try any other dynamic language (Ruby), send a function in place of a string and see failure msg. And if the question is really about path joins and path manipulations, then I believe PEP 428 ( http://www.python.org/dev/peps/pep-0428/) would be better candidate to address these scenarios. On Sat, Feb 9, 2013 at 3:45 PM, Eric V. Smith wrote: > On 2/9/2013 6:23 PM, Greg Ewing wrote: >> Terry Reedy wrote: >>> I agree. Since the exception type is not documented and since no one >>> should intentionally pass anything but strings, and therefore should >>> not be writing >>> >>> try: >>> os.path.join(a,b) >>> except AttributeError: >>> barf() >>> >>> I think it would be acceptable to make a change in 3.4. >> >> Why should we go out of our way to address this particular >> case, and not any of the infinitely many other situations >> where you could forget to add parens to a function call? >> > > I agree we do not want to go down this slippery slope. The number of > such places in the standard library is huge. The existing error message > is one that a user should be able to understand, since it's so common. > > -- > Eric. > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/senthil%40uthcode.com From chris at simplistix.co.uk Sun Feb 10 11:36:15 2013 From: chris at simplistix.co.uk (Chris Withers) Date: Sun, 10 Feb 2013 10:36:15 +0000 Subject: [Python-Dev] checking what atexit handlers are registered in Python 3 Message-ID: <5117781F.2080401@simplistix.co.uk> Hi All, I'm finally getting around to porting some of the packages I maintain over to Python 3. One rough edge I've hit: I see the atexit module has moved to be C-based and, as far as I can tell, no longer allows you to introspect what atexit functions have been registered. If I'm writing tests for code that registers atexit handlers, how can I check that they've been correctly registered? cheers, Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk From ncoghlan at gmail.com Sun Feb 10 13:32:50 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 10 Feb 2013 22:32:50 +1000 Subject: [Python-Dev] Submitting PEP 422 (Simple class initialization hook) for pronouncement Message-ID: I wrote PEP 422 (simple class initialization hook) last year after PJE pointed out that Python 3 style metaclasses made some Python 2 code impossible to migrate (since the class body could no longer request modifications to be made to the class after initialization was complete). It then languished, as I never found the time to actually implement it. Fortunately, Daniel Urban has now agreed to be a co-author on the PEP, and has fixed a couple of lingering technical errors in the PEP, as well as providing a reference implementation at http://bugs.python.org/issue17044 For those that don't recall the original discussion, the proposal is to add a new __init_class__ hook, invoked after the class object is created, but before the class decorators are applied. This provides a simple approach to inherited post-creation modification of classes, without the need for a custom metaclass. There's currently no BDFL delegate assigned, so this is a request for a pronouncement from Guido. Regards, Nick. ======================================= PEP: 422 Title: Simple class initialisation hook Version: $Revision$ Last-Modified: $Date$ Author: Nick Coghlan , Daniel Urban Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 5-Jun-2012 Python-Version: 3.4 Post-History: 5-Jun-2012, 10-Feb-2012 Abstract ======== In Python 2, the body of a class definition could modify the way a class was created (or simply arrange to run other code after the class was created) by setting the ``__metaclass__`` attribute in the class body. While doing this implicitly from called code required the use of an implementation detail (specifically, ``sys._getframes()``), it could also be done explicitly in a fully supported fashion (for example, by passing ``locals()`` to a function that calculated a suitable ``__metaclass__`` value) There is currently no corresponding mechanism in Python 3 that allows the code executed in the class body to directly influence how the class object is created. Instead, the class creation process is fully defined by the class header, before the class body even begins executing. This PEP proposes a mechanism that will once again allow the body of a class definition to more directly influence the way a class is created (albeit in a more constrained fashion), as well as replacing some current uses of metaclasses with a simpler, easier to understand alternative. Background ========== For an already created class ``cls``, the term "metaclass" has a clear meaning: it is the value of ``type(cls)``. *During* class creation, it has another meaning: it is also used to refer to the metaclass hint that may be provided as part of the class definition. While in many cases these two meanings end up referring to one and the same object, there are two situations where that is not the case: * If the metaclass hint refers to an instance of ``type``, then it is considered as a candidate metaclass along with the metaclasses of all of the parents of the class being defined. If a more appropriate metaclass is found amongst the candidates, then it will be used instead of the one given in the metaclass hint. * Otherwise, an explicit metaclass hint is assumed to be a factory function and is called directly to create the class object. In this case, the final metaclass will be determined by the factory function definition. In the typical case (where the factory functions just calls ``type``, or, in Python 3.3 or later, ``types.new_class``) the actual metaclass is then determined based on the parent classes. It is notable that only the actual metaclass is inherited - a factory function used as a metaclass hook sees only the class currently being defined, and is not invoked for any subclasses. In Python 3, the metaclass hint is provided using the ``metaclass=Meta`` keyword syntax in the class header. This allows the ``__prepare__`` method on the metaclass to be used to create the ``locals()`` namespace used during execution of the class body (for example, specifying the use of ``collections.OrderedDict`` instead of a regular ``dict``). In Python 2, there was no ``__prepare__`` method (that API was added for Python 3 by PEP 3115). Instead, a class body could set the ``__metaclass__`` attribute, and the class creation process would extract that value from the class namespace to use as the metaclass hint. There is `published code`_ that makes use of this feature. Another new feature in Python 3 is the zero-argument form of the ``super()`` builtin, introduced by PEP 3135. This feature uses an implicit ``__class__`` reference to the class being defined to replace the "by name" references required in Python 2. Just as code invoked during execution of a Python 2 metaclass could not call methods that referenced the class by name (as the name had not yet been bound in the containing scope), similarly, Python 3 metaclasses cannot call methods that rely on the implicit ``__class__`` reference (as it is not populated until after the metaclass has returned control to the class creation machiner). Proposal ======== This PEP proposes that a mechanism be added to Python 3 that meets the following criteria: 1. Restores the ability for class namespaces to have some influence on the class creation process (above and beyond populating the namespace itself), but potentially without the full flexibility of the Python 2 style ``__metaclass__`` hook 2. Integrates nicely with class inheritance structures (including mixins and multiple inheritance) 3. Integrates nicely with the implicit ``__class__`` reference and zero-argument ``super()`` syntax introduced by PEP 3135 4. Can be added to an existing base class without a significant risk of introducing backwards compatibility problems One mechanism that can achieve this goal is to add a new class initialisation hook, modelled directly on the existing instance initialisation hook, but with the signature constrained to match that of an ordinary class decorator. Specifically, it is proposed that class definitions be able to provide a class initialisation hook as follows:: class Example: @classmethod def __init_class__(cls): # This is invoked after the class is created, but before any # explicit decorators are called # The usual super() mechanisms are used to correctly support # multiple inheritance. The class decorator style signature helps # ensure that invoking the parent class is as simple as possible. If present on the created object, this new hook will be called by the class creation machinery *after* the ``__class__`` reference has been initialised. For ``types.new_class()``, it will be called as the last step before returning the created class object. If a metaclass wishes to block class initialisation for some reason, it must arrange for ``cls.__init_class__`` to trigger ``AttributeError``. Note, that when ``__init_class__`` is called, the name of the class is not bound to the new class object yet. As a consequence, the two argument form of ``super()`` cannot be used to call methods (e.g., ``super(Example, cls)`` wouldn't work in the example above). However, the zero argument form of ``super()`` works as expected, since the ``__class__`` reference is already initialised. This general proposal is not a new idea (it was first suggested for inclusion in the language definition `more than 10 years ago`_, and a similar mechanism has long been supported by `Zope's ExtensionClass`_), but the situation has changed sufficiently in recent years that the idea is worth reconsidering. Key Benefits ============ Replaces many use cases for dynamic setting of ``__metaclass__`` ----------------------------------------------------------------- For use cases that don't involve completely replacing the defined class, Python 2 code that dynamically set ``__metaclass__`` can now dynamically set ``__init_class__`` instead. For more advanced use cases, introduction of an explicit metaclass (possibly made available as a required base class) will still be necessary in order to support Python 3. Easier inheritance of definition time behaviour ----------------------------------------------- Understanding Python's metaclasses requires a deep understanding of the type system and the class construction process. This is legitimately seen as challenging, due to the need to keep multiple moving parts (the code, the metaclass hint, the actual metaclass, the class object, instances of the class object) clearly distinct in your mind. Even when you know the rules, it's still easy to make a mistake if you're not being extremely careful. An earlier version of this PEP actually included such a mistake: it stated "subclass of type" for a constraint that is actually "instance of type". Understanding the proposed class initialisation hook only requires understanding decorators and ordinary method inheritance, which isn't quite as daunting a task. The new hook provides a more gradual path towards understanding all of the phases involved in the class definition process. Reduced chance of metaclass conflicts ------------------------------------- One of the big issues that makes library authors reluctant to use metaclasses (even when they would be appropriate) is the risk of metaclass conflicts. These occur whenever two unrelated metaclasses are used by the desired parents of a class definition. This risk also makes it very difficult to *add* a metaclass to a class that has previously been published without one. By contrast, adding an ``__init_class__`` method to an existing type poses a similar level of risk to adding an ``__init__`` method: technically, there is a risk of breaking poorly implemented subclasses, but when that occurs, it is recognised as a bug in the subclass rather than the library author breaching backwards compatibility guarantees. In fact, due to the constrained signature of ``__init_class__``, the risk in this case is actually even lower than in the case of ``__init__``. Integrates cleanly with \PEP 3135 --------------------------------- Unlike code that runs as part of the metaclass, code that runs as part of the new hook will be able to freely invoke class methods that rely on the implicit ``__class__`` reference introduced by PEP 3135, including methods that use the zero argument form of ``super()``. Alternatives ============ The Python 3 Status Quo ----------------------- The Python 3 status quo already offers a great deal of flexibility. For changes which only affect a single class definition and which can be specified at the time the code is written, then class decorators can be used to modify a class explicitly. Class decorators largely ignore class inheritance and can make full use of methods that rely on the ``__class__`` reference being populated. Using a custom metaclass provides the same level of power as it did in Python 2. However, it's notable that, unlike class decorators, a metaclass cannot call any methods that rely on the ``__class__`` reference, as that reference is not populated until after the metaclass constructor returns control to the class creation code. One major use case for metaclasses actually closely resembles the use of class decorators. It occurs whenever a metaclass has an implementation that uses the following pattern:: class Metaclass(type): def __new__(meta, *args, **kwds): cls = super(Metaclass, meta).__new__(meta, *args, **kwds) # Do something with cls return cls The key difference between this pattern and a class decorator is that it is automatically inherited by subclasses. However, it also comes with a major disadvantage: Python does not allow you to inherit from classes with unrelated metaclasses. Thus, the status quo requires that developers choose between the following two alternatives: * Use a class decorator, meaning that behaviour is not inherited and must be requested explicitly on every subclass * Use a metaclass, meaning that behaviour is inherited, but metaclass conflicts may make integration with other libraries and frameworks more difficult than it otherwise would be If this PEP is ultimately rejected, then this is the existing design that will remain in place by default. Restoring the Python 2 metaclass hook ------------------------------------- One simple alternative would be to restore support for a Python 2 style ``metaclass`` hook in the class body. This would be checked after the class body was executed, potentially overwriting the metaclass hint provided in the class header. The main attraction of such an approach is that it would simplify porting Python 2 applications that make use of this hook (especially those that do so dynamically). However, this approach does nothing to simplify the process of adding *inherited* class definition time behaviour, nor does it interoperate cleanly with the PEP 3135 ``__class__`` and ``super()`` semantics (as with any metaclass based solution, the ``__metaclass__`` hook would have to run before the ``__class__`` reference has been populated. Dynamic class decorators ------------------------ The original version of this PEP was called "Dynamic class decorators" and focused solely on a significantly more complicated proposal than that presented in the current version. As with the current version, it proposed that a new step be added to the class creation process, after the metaclass invocation to construct the class instance and before the application of lexical decorators. However, instead of a simple process of calling a single class method that relies on normal inheritance mechanisms, it proposed a far more complicated procedure that walked the class MRO looking for decorators stored in iterable ``__decorators__`` attributes. Using the current version of the PEP, the scheme originally proposed could be implemented as:: class DynamicDecorators(Base): @classmethod def __init_class__(cls): # Process any classes later in the MRO try: mro_chain = super().__init_class__ except AttributeError: pass else: mro_chain() # Process any __decorators__ attributes in the MRO for entry in reversed(cls.mro()): decorators = entry.__dict__.get("__decorators__", ()) for deco in reversed(decorators): cls = deco(cls) Any subclasses of ``DynamicDecorators`` would then automatically have the contents of any ``__decorators__`` attributes processed and invoked. The mechanism in the current PEP is considered superior, as many issues to do with ordering and the same decorator being invoked multiple times just go away, as that kind of thing is taken care of through the use of an ordinary class method invocation. Automatic metaclass derivation ------------------------------ When no appropriate metaclass is found, it's theoretically possible to automatically derive a metaclass for a new type based on the metaclass hint and the metaclasses of the bases. While adding such a mechanism would reduce the risk of spurious metaclass conflicts, it would do nothing to improve integration with PEP 3135, would not help with porting Python 2 code that set ``__metaclass__`` dynamically and would not provide a more straightforward inherited mechanism for invoking additional operations after the class invocation is complete. In addition, there would still be a risk of metaclass conflicts in cases where the base metaclasses were not written with multiple inheritance in mind. In such situations, there's a chance of introducing latent defects if one or more metaclasses are not invoked correctly. Calling the new hook from ``type.__init__`` ------------------------------------------- Calling the new hook automatically from ``type.__init__``, would achieve most of the goals of this PEP. However, using that approach would mean that ``__init_class__`` implementations would be unable to call any methods that relied on the ``__class__`` reference (or used the zero-argument form of ``super()``), and could not make use of those features themselves. Reference Implementation ======================== The reference implementation has been posted to the `issue tracker`_. References ========== .. _published code: http://mail.python.org/pipermail/python-dev/2012-June/119878.html .. _more than 10 years ago: http://mail.python.org/pipermail/python-dev/2001-November/018651.html .. _Zope's ExtensionClass: http://docs.zope.org/zope_secrets/extensionclass.html .. _issue tracker: http://bugs.python.org/issue17044 Copyright ========= This document has been placed in the public domain. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From solipsis at pitrou.net Sun Feb 10 13:47:00 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sun, 10 Feb 2013 13:47:00 +0100 Subject: [Python-Dev] Submitting PEP 422 (Simple class initialization hook) for pronouncement References: Message-ID: <20130210134700.7dd822df@pitrou.net> On Sun, 10 Feb 2013 22:32:50 +1000 Nick Coghlan wrote: > > Replaces many use cases for dynamic setting of ``__metaclass__`` > ----------------------------------------------------------------- > > For use cases that don't involve completely replacing the defined class, > Python 2 code that dynamically set ``__metaclass__`` can now dynamically > set ``__init_class__`` instead. For more advanced use cases, introduction of > an explicit metaclass (possibly made available as a required base class) will > still be necessary in order to support Python 3. So, what are the use cases? I probably write metaclasses once a year, I wonder how much incentive there is to bring an additional complication to the already complicated class construction process. Regards Antoine. From ncoghlan at gmail.com Sun Feb 10 14:17:00 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 10 Feb 2013 23:17:00 +1000 Subject: [Python-Dev] Submitting PEP 422 (Simple class initialization hook) for pronouncement In-Reply-To: <20130210134700.7dd822df@pitrou.net> References: <20130210134700.7dd822df@pitrou.net> Message-ID: On Sun, Feb 10, 2013 at 10:47 PM, Antoine Pitrou wrote: > On Sun, 10 Feb 2013 22:32:50 +1000 > Nick Coghlan wrote: >> >> Replaces many use cases for dynamic setting of ``__metaclass__`` >> ----------------------------------------------------------------- >> >> For use cases that don't involve completely replacing the defined class, >> Python 2 code that dynamically set ``__metaclass__`` can now dynamically >> set ``__init_class__`` instead. For more advanced use cases, introduction of >> an explicit metaclass (possibly made available as a required base class) will >> still be necessary in order to support Python 3. At least PEAK uses it for interface declarations, I believe Zope do something similar. However, just as I championed PEP 414 to make porting Unicode-aware projects easier, I'm championing this one because I don't consider it acceptable for Python 3 to be a less capable language than Python 2 in this regard. Having come up with what I consider to be an elegant design to restore that capability, I'm not especially interested in passing judgment on the worthiness of what PJE is doing with Python 2 metaclasses that cannot currently be replicated in Python 3. > So, what are the use cases? > I probably write metaclasses once a year, I wonder how much incentive > there is to bring an additional complication to the already > complicated class construction process. One of the main goals of the PEP is actually to give people even *less* reason to write custom metaclasses, as you won't need to write one any more if all you want to do is adjust the class attributes after execution of the class body completes, and automatically do the same for any subclasses. (The latter part is key - if you don't want inheritance, then class decorators fill the role just fine. That's why the design in the PEP is the way it us - __init_class__ effectively becomes the innermost class decorator, which is called even for subclasses) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From hodgestar+pythondev at gmail.com Sun Feb 10 14:33:10 2013 From: hodgestar+pythondev at gmail.com (Simon Cross) Date: Sun, 10 Feb 2013 15:33:10 +0200 Subject: [Python-Dev] Submitting PEP 422 (Simple class initialization hook) for pronouncement In-Reply-To: References: Message-ID: On Sun, Feb 10, 2013 at 2:32 PM, Nick Coghlan wrote: > For those that don't recall the original discussion, the proposal is > to add a new __init_class__ hook, invoked after the class object is > created, but before the class decorators are applied. This provides a > simple approach to inherited post-creation modification of classes, > without the need for a custom metaclass. Couldn't one just write a metaclass that calls __init_class__ for you? Schiavo Simon From solipsis at pitrou.net Sun Feb 10 14:34:42 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sun, 10 Feb 2013 14:34:42 +0100 Subject: [Python-Dev] Submitting PEP 422 (Simple class initialization hook) for pronouncement In-Reply-To: References: <20130210134700.7dd822df@pitrou.net> Message-ID: <20130210143442.5e3cafcc@pitrou.net> On Sun, 10 Feb 2013 23:17:00 +1000 Nick Coghlan wrote: > On Sun, Feb 10, 2013 at 10:47 PM, Antoine Pitrou wrote: > > On Sun, 10 Feb 2013 22:32:50 +1000 > > Nick Coghlan wrote: > >> > >> Replaces many use cases for dynamic setting of ``__metaclass__`` > >> ----------------------------------------------------------------- > >> > >> For use cases that don't involve completely replacing the defined class, > >> Python 2 code that dynamically set ``__metaclass__`` can now dynamically > >> set ``__init_class__`` instead. For more advanced use cases, introduction of > >> an explicit metaclass (possibly made available as a required base class) will > >> still be necessary in order to support Python 3. > > At least PEAK uses it for interface declarations, I believe Zope do > something similar. zope.interface has been ported to Python 3, so the annoyance can't be very blocking. > However, just as I championed PEP 414 to make porting Unicode-aware > projects easier, I'm championing this one because I don't consider it > acceptable for Python 3 to be a less capable language than Python 2 in > this regard. This sounds like a theoretical concern rather than a practical one. > Having come up with what I consider to be an elegant > design to restore that capability, I'm not especially interested in > passing judgment on the worthiness of what PJE is doing with Python 2 > metaclasses that cannot currently be replicated in Python 3. It may be locally elegant, but it still brings additional complication to the global model. Reading your proposal reminds me of the various ways of influencing the pickle protocol: http://docs.python.org/dev/library/pickle.html#pickling-class-instances Nobody can claim this is simple and easy to wrap your head around. It is a maintenance burden, and it discourages understanding of the underlying model by anyone but language experts. Regards Antoine. From ncoghlan at gmail.com Sun Feb 10 14:52:33 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 10 Feb 2013 23:52:33 +1000 Subject: [Python-Dev] Submitting PEP 422 (Simple class initialization hook) for pronouncement In-Reply-To: References: Message-ID: On Sun, Feb 10, 2013 at 11:33 PM, Simon Cross wrote: > On Sun, Feb 10, 2013 at 2:32 PM, Nick Coghlan wrote: >> For those that don't recall the original discussion, the proposal is >> to add a new __init_class__ hook, invoked after the class object is >> created, but before the class decorators are applied. This provides a >> simple approach to inherited post-creation modification of classes, >> without the need for a custom metaclass. > > Couldn't one just write a metaclass that calls __init_class__ for you? And, indeed, that's what Zope did more than 10 years ago. The problem is that you're now locked in to that metaclass - metaclass conflicts become a permanent risk. You also can't *add* a metaclass to public classes, the risk of introducing metaclass conflicts means that such a change is always backwards incompatible. Adding __init_class__ is fine though, because that's no higher risk than adding a __new__ or __init__ method (it's actually lower risk, since the signature is always consistent). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Sun Feb 10 15:09:55 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 11 Feb 2013 00:09:55 +1000 Subject: [Python-Dev] Submitting PEP 422 (Simple class initialization hook) for pronouncement In-Reply-To: <20130210143442.5e3cafcc@pitrou.net> References: <20130210134700.7dd822df@pitrou.net> <20130210143442.5e3cafcc@pitrou.net> Message-ID: On Sun, Feb 10, 2013 at 11:34 PM, Antoine Pitrou wrote: > Nobody can claim this is simple and easy to wrap your head around. It > is a maintenance burden, and it discourages understanding of the > underlying model by anyone but language experts. You think anyone but language experts fully understands the metaclass mechanism now? *I* got the description of some details wrong in the PEP, and I'm probably in the top half dozen people on the planet when it comes to understanding how Python 3's class model works. Plenty of people understand decorators though, and that's all __init_class__ is: a special class decorator that is applied before any other decorators, and is automatically inherited by subclasses. It massively lowers the barrier to entry for inherited post-modification of classes. Will it mean people will only bother to understand metaclasses if they actually *need* metaclass.__prepare__ or to control the identity of the object returned by metaclass.__call__? Absolutely. However, I don't see that as any different from the fact that vastly more people understand how to use instance __init__ methods correctly than understand the ins and outs of using __new__ to control the *creation* of the object, rather than merely initialising it after it is already created. In some respects, cleanly separating out the two steps of controlling initialisation and controlling creation can actually make the two easier to grasp (since you can tackle them individually as distinct concepts, rather than having to wrap your head around both of them at the same time). As far as the maintenance burden goes, the patch to enable PEP 422 for types.new_class() is trivial: - return meta(name, bases, ns, **kwds) + cls = meta(name, bases, ns, **kwds) + try: + initcl = cls.__init_class__ + except AttributeError: + pass + else: + initcl() + return cls The change in the builtin.__build_class__ implementation is similarly trivial (and has the same semantics), it's just more verbose due to its being written in C. The documentation changes are quite straightforward (moving __init_class__ and decorator invocation out to their own subsection), and the tests Daniel has provided are extensive. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From solipsis at pitrou.net Sun Feb 10 15:28:14 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sun, 10 Feb 2013 15:28:14 +0100 Subject: [Python-Dev] Submitting PEP 422 (Simple class initialization hook) for pronouncement In-Reply-To: References: <20130210134700.7dd822df@pitrou.net> <20130210143442.5e3cafcc@pitrou.net> Message-ID: <20130210152814.66c26f76@pitrou.net> On Mon, 11 Feb 2013 00:09:55 +1000 Nick Coghlan wrote: > > As far as the maintenance burden goes, the patch to enable PEP 422 for > types.new_class() is trivial: > > - return meta(name, bases, ns, **kwds) > + cls = meta(name, bases, ns, **kwds) > + try: > + initcl = cls.__init_class__ > + except AttributeError: > + pass > + else: > + initcl() > + return cls I didn't know types.new_class(). I suppose that's because I'm not part of the top half dozen people on the planet :-) I've stopped trying to understand when I saw about the __build_class__ additions and whatnot. I now consider that part of the language totally black magic. I'm not sure it's a good thing when even some maintainers of the language don't understand some of its (important) corners. > The change in the builtin.__build_class__ implementation is similarly > trivial (and has the same semantics), it's just more verbose due to > its being written in C. Sure, every little addition is "trivial". At the end you have a scary monster made of many little trivial additions along the years, and everyone has to take care not to break it. Regards Antoine. From stefan_ml at behnel.de Sun Feb 10 17:48:41 2013 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sun, 10 Feb 2013 17:48:41 +0100 Subject: [Python-Dev] Submitting PEP 422 (Simple class initialization hook) for pronouncement In-Reply-To: <20130210152814.66c26f76@pitrou.net> References: <20130210134700.7dd822df@pitrou.net> <20130210143442.5e3cafcc@pitrou.net> <20130210152814.66c26f76@pitrou.net> Message-ID: Antoine Pitrou, 10.02.2013 15:28: > On Mon, 11 Feb 2013 00:09:55 +1000 Nick Coghlan wrote: >> As far as the maintenance burden goes, the patch to enable PEP 422 for >> types.new_class() is trivial: >> >> - return meta(name, bases, ns, **kwds) >> + cls = meta(name, bases, ns, **kwds) >> + try: >> + initcl = cls.__init_class__ >> + except AttributeError: >> + pass >> + else: >> + initcl() >> + return cls > > I didn't know types.new_class(). I suppose that's because I'm not part > of the top half dozen people on the planet :-) I've stopped trying to > understand when I saw about the __build_class__ additions and whatnot. > I now consider that part of the language totally black magic. > > I'm not sure it's a good thing when even some maintainers of the > language don't understand some of its (important) corners. I, for one, only learned most of this stuff when we implemented it for Cython and I'm sure there are lots of little details that I'm not aware of. Use cases are too rare to fully learn it and the matter is certainly complex. >> The change in the builtin.__build_class__ implementation is similarly >> trivial (and has the same semantics), it's just more verbose due to >> its being written in C. > > Sure, every little addition is "trivial". At the end you have a scary > monster made of many little trivial additions along the years, and > everyone has to take care not to break it. FWIW, I agree with Nick that this actually kills some use cases of metaclasses and could thus lower the barrier for "doing stuff to classes" for those who don't want to dig into metaclasses for one reason or another. However, it's hard to say if this "new way of doing it" doesn't come with its own can of worms. For example, would cooperative calls to "__init_class__" work if a superclass already defines it? Do implementors have to remember that? And is it clear how this should be done, e.g. what should normally go first, my own code or the superclass call? Supporting cooperative __init_class__() calls properly might actually be a good thing. Currently, there's only one metaclass, plus a sequence of decorators, which makes the setup rather static and sometimes tedious for subclasses that need to do certain things by themselves, but in addition to what happens already. So, the way to explain it to users would be 1) don't use it, 2) if you really need to do something to a class, use a decorator, 3) if you need to decide dynamically what to do, define __init_class__() and 4) don't forget to call super's __init_class__() in that case, and 5) only if you need to do something substantially more involved and know what you're doing, use a metaclass. Sounds a bit smoother than the current metaclass hole that we throw people into directly when a decorator won't catch it for them. It's usually not a good idea to add complexity for something that's essentially a rare use case. Only the assumption that it might actually make it easier for users to handle certain other use cases where they'd currently end up with their own home-grown metaclass is a point that makes it worth considering for me. But then again, maybe these use cases are really rare enough to just let people figure out how to do it with metaclasses, or even a combination of metaclasses and decorators. Stefan From urban.dani+py at gmail.com Sun Feb 10 21:51:39 2013 From: urban.dani+py at gmail.com (Daniel Urban) Date: Sun, 10 Feb 2013 21:51:39 +0100 Subject: [Python-Dev] Submitting PEP 422 (Simple class initialization hook) for pronouncement In-Reply-To: References: <20130210134700.7dd822df@pitrou.net> <20130210143442.5e3cafcc@pitrou.net> <20130210152814.66c26f76@pitrou.net> Message-ID: On Sun, Feb 10, 2013 at 5:48 PM, Stefan Behnel wrote: > However, it's hard to say if this "new way of doing it" doesn't come with > its own can of worms. For example, would cooperative calls to > "__init_class__" work if a superclass already defines it? Do implementors > have to remember that? And is it clear how this should be done, e.g. what > should normally go first, my own code or the superclass call? Supporting > cooperative __init_class__() calls properly might actually be a good thing. > Currently, there's only one metaclass, plus a sequence of decorators, which > makes the setup rather static and sometimes tedious for subclasses that > need to do certain things by themselves, but in addition to what happens > already. Cooperative multiple inheritance of __init_class__ would work exactly the same way as it works e.g., for __init__ of any other method (actually it is easier, since as Nick mentioned, the signature is always the same): __init_class__ can simply use the zero argument form of super. There is a simple example in the tests at http://bugs.python.org/issue17044. Daniel From pje at telecommunity.com Sun Feb 10 22:27:49 2013 From: pje at telecommunity.com (PJ Eby) Date: Sun, 10 Feb 2013 16:27:49 -0500 Subject: [Python-Dev] Submitting PEP 422 (Simple class initialization hook) for pronouncement In-Reply-To: References: Message-ID: On Sun, Feb 10, 2013 at 7:32 AM, Nick Coghlan wrote: > class Example: > @classmethod > def __init_class__(cls): Is the @classmethod required? What happens if it's not present? Second, will type have a default __init_class__? (IMO, it should, otherwise it will be impossible to write co-operative __init_class__ functions.) Only other comment is that the PEP could use a more concrete use case, e.g.: class Record: __fields = {} @classmethod def __init_class__(cls): cls.__fields = dict(cls.__fields) # inherited fields cls.__fields.update({attr:val for attr, val in cls.__dict__.iteritems() if isinstance(val, Field)}) super().__init_class__() # be co-operative # ...other methods that use the __fields attribute class SomeRecord(Record): foo = Field(int) bar = Field(str) Putting something like this early on might help to demonstrate the usefulness of the feature on its own merits, independent of the porting issue, etc. ;-) From pje at telecommunity.com Sun Feb 10 22:41:31 2013 From: pje at telecommunity.com (PJ Eby) Date: Sun, 10 Feb 2013 16:41:31 -0500 Subject: [Python-Dev] Submitting PEP 422 (Simple class initialization hook) for pronouncement In-Reply-To: References: <20130210134700.7dd822df@pitrou.net> <20130210143442.5e3cafcc@pitrou.net> <20130210152814.66c26f76@pitrou.net> Message-ID: On Sun, Feb 10, 2013 at 11:48 AM, Stefan Behnel wrote: > So, the way to explain it to users would be 1) don't use it, 2) if you > really need to do something to a class, use a decorator, 3) if you need to > decide dynamically what to do, define __init_class__() and 4) don't forget > to call super's __init_class__() in that case, and 5) only if you need to > do something substantially more involved and know what you're doing, use a > metaclass. I'd revise that to: 1) if there's no harm in forgetting to decorate a subclass, use a class decorator 2) if you want to ensure that a modification is applied to every subclass of a single common base class, define __init_class__ (and always call its super) 3) If you need to make the class object *act* differently (not just initialize it or trigger some other side-effect at creation time), or if you want the class suite to return some other kind of object, you'll need a metaclass. Essentially, this change fixes a hole in class decorators that doesn't exist with function decorators: if you need the decoration applied to subclasses, you can end up with silent failures right now. Conversely, if you try prevent such failures using a metaclass, you not only have a big hill to climb, but the resulting code will be vulnerable to metaclass conflicts. The proposed solution neatly fixes both of these problems, providing One Obvious Way to do subclass initialization. (An alternative, I suppose, would be to let you do something like @@someClassDecorator to have inheritable class decorators, but as Nick has pointed out, implementing inheritable decorators is a lot more complicated.) From pje at telecommunity.com Sun Feb 10 22:46:06 2013 From: pje at telecommunity.com (PJ Eby) Date: Sun, 10 Feb 2013 16:46:06 -0500 Subject: [Python-Dev] Why does Signature.from_function() have to check the type of its argument? In-Reply-To: References: Message-ID: On Fri, Feb 8, 2013 at 5:44 PM, Stefan Behnel wrote: > Argh - sorry, got it all wrong. "__instancecheck__()" works exactly the > other way round. In the type check above, it's the FunctionType type that > gets asked for an instance check, which doesn't help at all in this case > because it simply doesn't know about the desired subtype relation. It would > work if type(func).__instancecheck__() was used, but that doesn't happen. > > So, no help from that side, sadly. How about you return FunctionType as your __class__ attribute? ;-) Your type() will still be different, but isinstance() also considers the __class__ if it's different from the type(). (At least it does in 2.x, I've not tried it in any 3.x versions yet...) From ethan at stoneleaf.us Sun Feb 10 22:59:50 2013 From: ethan at stoneleaf.us (Ethan Furman) Date: Sun, 10 Feb 2013 13:59:50 -0800 Subject: [Python-Dev] Why does Signature.from_function() have to check the type of its argument? In-Reply-To: References: Message-ID: <51181856.6060704@stoneleaf.us> On 02/10/2013 01:46 PM, PJ Eby wrote: > How about you return FunctionType as your __class__ attribute? ;-) > > Your type() will still be different, but isinstance() also considers > the __class__ if it's different from the type(). (At least it does in > 2.x, I've not tried it in any 3.x versions yet...) Python 3.2.3 (default, Oct 19 2012, 19:53:16) [GCC 4.7.2] on linux2 Type "help", "copyright", "credits" or "license" for more information. --> class Test(): ... pass ... --> t = Test() --> t.__class__ --> import types --> class Test(): ... __class__ = types.FunctionType ... --> t = Test() --> t.__class__ --> isinstance(t, types.FunctionType) True -- ~Ethan~ From eric at trueblade.com Mon Feb 11 00:06:35 2013 From: eric at trueblade.com (Eric V. Smith) Date: Sun, 10 Feb 2013 18:06:35 -0500 Subject: [Python-Dev] [Python-checkins] cpython (3.3): Reject float as uid or gid. In-Reply-To: <3Z43DT257WzScR@mail.python.org> References: <3Z43DT257WzScR@mail.python.org> Message-ID: <511827FB.9040905@trueblade.com> On 2/10/2013 4:29 PM, serhiy.storchaka wrote: > http://hg.python.org/cpython/rev/4ef048f4834e > changeset: 82147:4ef048f4834e > branch: 3.3 > parent: 82145:b322655a4a88 > user: Serhiy Storchaka > date: Sun Feb 10 23:28:02 2013 +0200 > summary: > Reject float as uid or gid. > A regression was introduced in the commit for issue issue #4591. > > files: > Modules/posixmodule.c | 16 ++++++++++++++-- > 1 files changed, 14 insertions(+), 2 deletions(-) > > > diff --git a/Modules/posixmodule.c b/Modules/posixmodule.c > --- a/Modules/posixmodule.c > +++ b/Modules/posixmodule.c > @@ -437,7 +437,13 @@ > _Py_Uid_Converter(PyObject *obj, void *p) > { > int overflow; > - long result = PyLong_AsLongAndOverflow(obj, &overflow); > + long result; > + if (PyFloat_Check(obj)) { > + PyErr_SetString(PyExc_TypeError, > + "integer argument expected, got float"); > + return 0; > + } > + result = PyLong_AsLongAndOverflow(obj, &overflow); > if (overflow < 0) > goto OverflowDown; > if (!overflow && result == -1) { Instead of special-casing float, isn't using __index__ the preferred way to do this? -- Eric. From solipsis at pitrou.net Mon Feb 11 09:02:43 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 11 Feb 2013 09:02:43 +0100 Subject: [Python-Dev] Submitting PEP 422 (Simple class initialization hook) for pronouncement References: Message-ID: <20130211090243.30af7e88@pitrou.net> On Sun, 10 Feb 2013 16:27:49 -0500 PJ Eby wrote: > On Sun, Feb 10, 2013 at 7:32 AM, Nick Coghlan wrote: > > class Example: > > @classmethod > > def __init_class__(cls): > > Is the @classmethod required? What happens if it's not present? > > Second, will type have a default __init_class__? (IMO, it should, > otherwise it will be impossible to write co-operative __init_class__ > functions.) > > Only other comment is that the PEP could use a more concrete use case, e.g.: > > class Record: > __fields = {} > > @classmethod > def __init_class__(cls): > cls.__fields = dict(cls.__fields) # inherited fields > cls.__fields.update({attr:val for attr, val in > cls.__dict__.iteritems() if isinstance(val, Field)}) > super().__init_class__() # be co-operative > > # ...other methods that use the __fields attribute > > class SomeRecord(Record): > foo = Field(int) > bar = Field(str) > > Putting something like this early on might help to demonstrate the > usefulness of the feature on its own merits, independent of the > porting issue, etc. ;-) Can you explain what the example does / is supposed to do? From chris at simplistix.co.uk Mon Feb 11 09:08:02 2013 From: chris at simplistix.co.uk (Chris Withers) Date: Mon, 11 Feb 2013 08:08:02 +0000 Subject: [Python-Dev] _not_found attribute on ImportError Message-ID: <5118A6E2.5010105@simplistix.co.uk> Hi All, I see in Python 3, some ImportErrors have grown a '_not_found' attribute. What's the significance of this attribute and where/how is it added? The only way I can seem to create this attribute is: ex = ImportError ex._not_found = True cheers, Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk From ncoghlan at gmail.com Mon Feb 11 11:33:24 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 11 Feb 2013 20:33:24 +1000 Subject: [Python-Dev] Submitting PEP 422 (Simple class initialization hook) for pronouncement In-Reply-To: References: <20130210134700.7dd822df@pitrou.net> <20130210143442.5e3cafcc@pitrou.net> <20130210152814.66c26f76@pitrou.net> Message-ID: On Mon, Feb 11, 2013 at 7:41 AM, PJ Eby wrote: > On Sun, Feb 10, 2013 at 11:48 AM, Stefan Behnel wrote: >> So, the way to explain it to users would be 1) don't use it, 2) if you >> really need to do something to a class, use a decorator, 3) if you need to >> decide dynamically what to do, define __init_class__() and 4) don't forget >> to call super's __init_class__() in that case, and 5) only if you need to >> do something substantially more involved and know what you're doing, use a >> metaclass. > > I'd revise that to: > > 1) if there's no harm in forgetting to decorate a subclass, use a > class decorator > 2) if you want to ensure that a modification is applied to every > subclass of a single common base class, define __init_class__ (and > always call its super) > 3) If you need to make the class object *act* differently (not just > initialize it or trigger some other side-effect at creation time), or > if you want the class suite to return some other kind of object, > you'll need a metaclass. I like that. Perhaps the PEP should propose some additional guidance in PEP 8 regarding class based metaprogramming? > Essentially, this change fixes a hole in class decorators that doesn't > exist with function decorators: if you need the decoration applied to > subclasses, you can end up with silent failures right now. > Conversely, if you try prevent such failures using a metaclass, you > not only have a big hill to climb, but the resulting code will be > vulnerable to metaclass conflicts. > > The proposed solution neatly fixes both of these problems, providing > One Obvious Way to do subclass initialization. I also realised last night that one significant benefit of cleanly separating class creation from class initialisation (as __new__ and __init__ separate instance creation and initialisation) is the ability to create a shared metaclass that just changes the namespace type with __prepare__, and then use __init_class__ to control what you do with it. Here's the more extended example I'm now considering adding to the PEP in order to show the improved composability the PEP offers (writing the below example with only metaclasses would be... challenging). It's still a toy example, but I don't believe there is any non-toy use case for metaclass composition that is going to be short enough to fit in a PEP: # Define a metaclass as in Python 3.3 and earlier import collections class OrderedMeta(type): def __prepare__(self, *args, **kwds): return collections.OrderedDict() # Won't be needed if we add a noop __init_class__ to type def __init_class__(cls): pass class OrderedClass(metaclass=OrderedMeta): pass # Easily supplement the metaclass behaviour in a class definition class SimpleRecord(OrderedClass): """Simple ordered record type (inheritance not supported)""" @classmethod def __init_class__(cls): super().__init_class__() cls.__fields = fields = [] for attr, obj in cls.__dict__.items(): if attr.startswith("_") or callable(obj): continue fields.append(attr) def __init__(self, *values): super().__init__(*values) for attr, obj in zip(self.__fields, values): setattr(self, attr, obj) def to_dict(self): fields = ((k, getattr(self, k)) for k in self.__fields) return collections.OrderedDict(fields) # Supplement the metaclass differently in another class definition class InheritableRecord(OrderedClass): """More complex record type that supports inheritance""" @classmethod def __init_class__(cls): super().__init_class__() cls.__fields = fields = [] for mro_cls in cls.mro(): for attr, obj in cls.__dict__.items(): if attr.startswith("_") or callable(obj): continue fields.append(attr) def __init__(self, *values): super().__init__(*values) for attr, obj in zip(self.__fields, values): setattr(self, attr, obj) def to_dict(self): fields = ((k, getattr(self, k)) for k in self.__fields) return collections.OrderedDict(fields) # Compared to custom metaclasses, composition is much simpler class ConfusedRecord(InheritableRecord, SimpleRecord): """Odd record type, only included to demonstrate composition""" # to_dict is inherited from InheritableRecord def to_simple_dict(self): return SimpleRecord.to_dict(self) Perhaps it would sweeten the deal if the PEP also provided types.OrderedMeta and types.OrderedClass, such that inheriting from types.OrderedClass and defining __init_class__ became the one-obvious-way to do order dependent class bodies? (I checked, we can make types depend on collections without a circular dependency. We would need to fix the increasingly inaccurate docstring, though) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Mon Feb 11 11:54:26 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 11 Feb 2013 20:54:26 +1000 Subject: [Python-Dev] _not_found attribute on ImportError In-Reply-To: <5118A6E2.5010105@simplistix.co.uk> References: <5118A6E2.5010105@simplistix.co.uk> Message-ID: On Mon, Feb 11, 2013 at 6:08 PM, Chris Withers wrote: > Hi All, > > I see in Python 3, some ImportErrors have grown a '_not_found' attribute. > What's the significance of this attribute and where/how is it added? > > The only way I can seem to create this attribute is: > > ex = ImportError > ex._not_found = True It's something importlib does to get fromlists to behave the same way they did in the C implementation. It is set here: http://hg.python.org/cpython/file/3.3/Lib/importlib/_bootstrap.py To make these pass: http://hg.python.org/cpython/file/3.3/Lib/test/test_importlib/import_/test_fromlist.py You'd have to ask Brett which specific test is affected, or else you could comment out the line that sets _not_found and see what breaks (you may need to refreeze _frozen_importlib first). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From chris at simplistix.co.uk Mon Feb 11 11:58:58 2013 From: chris at simplistix.co.uk (Chris Withers) Date: Mon, 11 Feb 2013 10:58:58 +0000 Subject: [Python-Dev] _not_found attribute on ImportError In-Reply-To: References: <5118A6E2.5010105@simplistix.co.uk> Message-ID: <5118CEF2.9070300@simplistix.co.uk> On 11/02/2013 10:54, Nick Coghlan wrote: > On Mon, Feb 11, 2013 at 6:08 PM, Chris Withers wrote: >> Hi All, >> >> I see in Python 3, some ImportErrors have grown a '_not_found' attribute. >> What's the significance of this attribute and where/how is it added? >> >> The only way I can seem to create this attribute is: >> >> ex = ImportError >> ex._not_found = True > > It's something importlib does to get fromlists to behave the same way > they did in the C implementation. > > It is set here: > http://hg.python.org/cpython/file/3.3/Lib/importlib/_bootstrap.py OK. Have any other exceptions grown new attributes in Python 3? cheers, Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk From ncoghlan at gmail.com Mon Feb 11 12:23:00 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 11 Feb 2013 21:23:00 +1000 Subject: [Python-Dev] _not_found attribute on ImportError In-Reply-To: <5118CEF2.9070300@simplistix.co.uk> References: <5118A6E2.5010105@simplistix.co.uk> <5118CEF2.9070300@simplistix.co.uk> Message-ID: On Mon, Feb 11, 2013 at 8:58 PM, Chris Withers wrote: > Have any other exceptions grown new attributes in Python 3? Off the top of my head, ImportError grew "name" and "path" attributes in 3.3, everything grew __cause__, __context__ and __traceback__ attributes in 3.0 and the __suppress_context__ attribute in 3.3. PEP 3151 may have moved a few attributes around in 3.3 as well. If there are any others, you'll need to trawl the What's New documents looking for them. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From just_another_developer at yahoo.de Mon Feb 11 13:16:48 2013 From: just_another_developer at yahoo.de (Developer Developer) Date: Mon, 11 Feb 2013 12:16:48 +0000 (GMT) Subject: [Python-Dev] Question regarding: Lib/_markupbase.py Message-ID: <1360585008.27100.YahooMailNeo@web171404.mail.ir2.yahoo.com> Hello all, I was having a look at the file:?Lib/_markupbase.py (@ 82151), function: "_parse_doctype_element" and have seen something that has caught my attention: if '>' in rawdata[j:]: ? ? return rawdata.find(">", j) + 1 Wouldn't it be better to do the following? pos = rawdata.find(">", j) if pos != -1: ? ? return pos + 1 Otherwise I think we are scanning rawdata[j:] twice. Best regards From techtonik at gmail.com Mon Feb 11 13:49:39 2013 From: techtonik at gmail.com (anatoly techtonik) Date: Mon, 11 Feb 2013 15:49:39 +0300 Subject: [Python-Dev] My CLA Message-ID: Python Contributor Agreement ---------------------------- I allow PSF to release all my code that I submitted to it, under any open source license. -- anatoly t. -------------- next part -------------- An HTML attachment was scrubbed... URL: From phd at phdru.name Mon Feb 11 14:01:51 2013 From: phd at phdru.name (Oleg Broytman) Date: Mon, 11 Feb 2013 17:01:51 +0400 Subject: [Python-Dev] My CLA In-Reply-To: References: Message-ID: <20130211130151.GB28063@iskra.aviel.ru> On Mon, Feb 11, 2013 at 03:49:39PM +0300, anatoly techtonik wrote: > Python Contributor Agreement > ---------------------------- > I allow PSF to release all my code that I submitted to > it, under any open source license. Good intention but wrong way of expressing it. Please do it properly -- via a signed paper. You can send it by snail mail, or you can scan it and send by email. Oleg. -- Oleg Broytman http://phdru.name/ phd at phdru.name Programmers don't die, they just GOSUB without RETURN. From eliben at gmail.com Mon Feb 11 14:29:56 2013 From: eliben at gmail.com (Eli Bendersky) Date: Mon, 11 Feb 2013 05:29:56 -0800 Subject: [Python-Dev] [Python-checkins] Daily reference leaks (b53b029895df): sum=2 In-Reply-To: References: Message-ID: On Sun, Feb 10, 2013 at 9:00 PM, wrote: > results for b53b029895df on branch "default" > -------------------------------------------- > > test_concurrent_futures leaked [2, 1, -1] memory blocks, sum=2 > > When did this start happening? Eli -------------- next part -------------- An HTML attachment was scrubbed... URL: From fred at fdrake.net Mon Feb 11 15:10:53 2013 From: fred at fdrake.net (Fred Drake) Date: Mon, 11 Feb 2013 09:10:53 -0500 Subject: [Python-Dev] Question regarding: Lib/_markupbase.py In-Reply-To: <1360585008.27100.YahooMailNeo@web171404.mail.ir2.yahoo.com> References: <1360585008.27100.YahooMailNeo@web171404.mail.ir2.yahoo.com> Message-ID: On Mon, Feb 11, 2013 at 7:16 AM, Developer Developer wrote: > Wouldn't it be better to do the following? ... > Otherwise I think we are scanning rawdata[j:] twice. Yes, that would be better, and avoids a string object creation as well. -Fred -- Fred L. Drake, Jr. "A storm broke loose in my mind." --Albert Einstein From barry at python.org Mon Feb 11 15:23:34 2013 From: barry at python.org (Barry Warsaw) Date: Mon, 11 Feb 2013 09:23:34 -0500 Subject: [Python-Dev] _not_found attribute on ImportError In-Reply-To: References: <5118A6E2.5010105@simplistix.co.uk> <5118CEF2.9070300@simplistix.co.uk> Message-ID: <20130211092334.7b9049b3@anarchist.wooz.org> On Feb 11, 2013, at 09:23 PM, Nick Coghlan wrote: >On Mon, Feb 11, 2013 at 8:58 PM, Chris Withers wrote: >> Have any other exceptions grown new attributes in Python 3? > >Off the top of my head, ImportError grew "name" and "path" attributes >in 3.3, everything grew __cause__, __context__ and __traceback__ >attributes in 3.0 and the __suppress_context__ attribute in 3.3. > >PEP 3151 may have moved a few attributes around in 3.3 as well. > >If there are any others, you'll need to trawl the What's New documents >looking for them. Those public attributes should be documented. _not_found should not be since it's an implementation detail. Also, Brett has left little TODO easter eggs in the code which clearly indicate changes he plans for 3.4, although I don't know where the plans for ModuleNotFound are documented. In the meantime, I just ran a test against trunk, with the following change and nothing broke afaict. diff -r a79650aacb43 Lib/importlib/_bootstrap.py --- a/Lib/importlib/_bootstrap.py Mon Feb 11 13:33:00 2013 +0000 +++ b/Lib/importlib/_bootstrap.py Mon Feb 11 09:16:51 2013 -0500 @@ -1640,6 +1640,7 @@ # TODO(brett): In Python 3.4, have import raise # ModuleNotFound and catch that. if getattr(exc, '_not_found', False): + del exc._not_found if exc.name == from_name: continue raise I won't commit this, but it really needs another hasattr() check to be completely valid. Cheers, -Barry From brett at python.org Mon Feb 11 15:44:47 2013 From: brett at python.org (Brett Cannon) Date: Mon, 11 Feb 2013 09:44:47 -0500 Subject: [Python-Dev] _not_found attribute on ImportError In-Reply-To: <20130211092334.7b9049b3@anarchist.wooz.org> References: <5118A6E2.5010105@simplistix.co.uk> <5118CEF2.9070300@simplistix.co.uk> <20130211092334.7b9049b3@anarchist.wooz.org> Message-ID: On Mon, Feb 11, 2013 at 9:23 AM, Barry Warsaw wrote: > On Feb 11, 2013, at 09:23 PM, Nick Coghlan wrote: > > >On Mon, Feb 11, 2013 at 8:58 PM, Chris Withers > wrote: > >> Have any other exceptions grown new attributes in Python 3? > > > >Off the top of my head, ImportError grew "name" and "path" attributes > >in 3.3, everything grew __cause__, __context__ and __traceback__ > >attributes in 3.0 and the __suppress_context__ attribute in 3.3. > > > >PEP 3151 may have moved a few attributes around in 3.3 as well. > > > >If there are any others, you'll need to trawl the What's New documents > >looking for them. > > Those public attributes should be documented. They are: http://docs.python.org/3.4/library/exceptions.html?highlight=importerror#ImportError > _not_found should not be since > it's an implementation detail. Yep, hence the leading underscore. > Also, Brett has left little TODO easter eggs > in the code which clearly indicate changes he plans for 3.4, although I > don't > know where the plans for ModuleNotFound are documented. > http://bugs.python.org/issue15767 Basically it's a matter of choosing ModuleNotFound vs. ModuleNotFoundError and then writing the code. -Brett > > In the meantime, I just ran a test against trunk, with the following change > and nothing broke afaict. > > diff -r a79650aacb43 Lib/importlib/_bootstrap.py > --- a/Lib/importlib/_bootstrap.py Mon Feb 11 13:33:00 2013 +0000 > +++ b/Lib/importlib/_bootstrap.py Mon Feb 11 09:16:51 2013 -0500 > @@ -1640,6 +1640,7 @@ > # TODO(brett): In Python 3.4, have import raise > # ModuleNotFound and catch that. > if getattr(exc, '_not_found', False): > + del exc._not_found > if exc.name == from_name: > continue > raise > > I won't commit this, but it really needs another hasattr() check to be > completely valid. > > Cheers, > -Barry > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/brett%40python.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: From just_another_developer at yahoo.de Mon Feb 11 15:47:00 2013 From: just_another_developer at yahoo.de (Developer Developer) Date: Mon, 11 Feb 2013 14:47:00 +0000 (GMT) Subject: [Python-Dev] Question regarding: Lib/_markupbase.py In-Reply-To: References: <1360585008.27100.YahooMailNeo@web171404.mail.ir2.yahoo.com> Message-ID: <1360594020.18756.YahooMailNeo@web171403.mail.ir2.yahoo.com> Same thing in the function: "_parse_doctype_attlist": if ")" in rawdata[j:]: ? ? j = rawdata.find(")", j) + 1 else: ? ? return -1 I would change it to: pos =?rawdata.find(")", j) if pos != -1: ? ? j = pos + 1 else: ? ? return -1 Best regards, Guido ----- Urspr?ngliche Message ----- Von: Fred Drake An: Developer Developer CC: "python-dev at python.org" Gesendet: 15:10 Montag, 11.Februar 2013 Betreff: Re: [Python-Dev] Question regarding: Lib/_markupbase.py On Mon, Feb 11, 2013 at 7:16 AM, Developer Developer wrote: > Wouldn't it be better to do the following? ... > Otherwise I think we are scanning rawdata[j:] twice. Yes, that would be better, and avoids a string object creation as well. ? -Fred -- Fred L. Drake, Jr.? ? "A storm broke loose in my mind."? --Albert Einstein From storchaka at gmail.com Mon Feb 11 16:07:31 2013 From: storchaka at gmail.com (Serhiy Storchaka) Date: Mon, 11 Feb 2013 17:07:31 +0200 Subject: [Python-Dev] [Python-checkins] cpython (3.3): Reject float as uid or gid. In-Reply-To: <511827FB.9040905@trueblade.com> References: <3Z43DT257WzScR@mail.python.org> <511827FB.9040905@trueblade.com> Message-ID: On 11.02.13 01:06, Eric V. Smith wrote: > Instead of special-casing float, isn't using __index__ the preferred way > to do this? Perhaps. I just copied this code from PyArg_ParseTuple*() for 'l' format. From barry at python.org Mon Feb 11 16:13:07 2013 From: barry at python.org (Barry Warsaw) Date: Mon, 11 Feb 2013 10:13:07 -0500 Subject: [Python-Dev] Submitting PEP 422 (Simple class initialization hook) for pronouncement In-Reply-To: <20130210143442.5e3cafcc@pitrou.net> References: <20130210134700.7dd822df@pitrou.net> <20130210143442.5e3cafcc@pitrou.net> Message-ID: <20130211101307.2d36ebcb@anarchist.wooz.org> On Feb 10, 2013, at 02:34 PM, Antoine Pitrou wrote: >zope.interface has been ported to Python 3, so the annoyance can't be >very blocking. The syntax is different, but I actually prefer the Python 3-compatible syntax better. It uses a class decorator instead of a magic class attribute, so it's much clearer IMHO. Works in Python 2, too, fsvo. -Barry From barry at python.org Mon Feb 11 16:15:36 2013 From: barry at python.org (Barry Warsaw) Date: Mon, 11 Feb 2013 10:15:36 -0500 Subject: [Python-Dev] Submitting PEP 422 (Simple class initialization hook) for pronouncement In-Reply-To: <20130210152814.66c26f76@pitrou.net> References: <20130210134700.7dd822df@pitrou.net> <20130210143442.5e3cafcc@pitrou.net> <20130210152814.66c26f76@pitrou.net> Message-ID: <20130211101536.6f769ee3@anarchist.wooz.org> On Feb 10, 2013, at 03:28 PM, Antoine Pitrou wrote: >Sure, every little addition is "trivial". At the end you have a scary >monster made of many little trivial additions along the years, and >everyone has to take care not to break it. Why Antoine, that surely isn't the case with the import system! . -Barry From barry at python.org Mon Feb 11 16:19:53 2013 From: barry at python.org (Barry Warsaw) Date: Mon, 11 Feb 2013 10:19:53 -0500 Subject: [Python-Dev] Submitting PEP 422 (Simple class initialization hook) for pronouncement In-Reply-To: References: <20130210134700.7dd822df@pitrou.net> <20130210143442.5e3cafcc@pitrou.net> <20130210152814.66c26f76@pitrou.net> Message-ID: <20130211101953.6b477bda@anarchist.wooz.org> On Feb 11, 2013, at 08:33 PM, Nick Coghlan wrote: >I like that. Perhaps the PEP should propose some additional guidance >in PEP 8 regarding class based metaprogramming? I wouldn't put it in PEP 8, since it'll glaze the eyes of all but 6 people on the planet. Probably better as a HOWTO in the documentation. Cheers, -Barry From solipsis at pitrou.net Mon Feb 11 16:21:27 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 11 Feb 2013 16:21:27 +0100 Subject: [Python-Dev] Submitting PEP 422 (Simple class initialization hook) for pronouncement References: <20130210134700.7dd822df@pitrou.net> <20130210143442.5e3cafcc@pitrou.net> <20130210152814.66c26f76@pitrou.net> <20130211101536.6f769ee3@anarchist.wooz.org> Message-ID: <20130211162127.6d4705c8@pitrou.net> Le Mon, 11 Feb 2013 10:15:36 -0500, Barry Warsaw a ?crit : > On Feb 10, 2013, at 03:28 PM, Antoine Pitrou wrote: > > >Sure, every little addition is "trivial". At the end you have a scary > >monster made of many little trivial additions along the years, and > >everyone has to take care not to break it. > > Why Antoine, that surely isn't the case with the import system! > . Nor with pythonrun.c. I wouldn't even mention typeobject.c or the build system ;-) Regards Antoine. From barry at python.org Mon Feb 11 16:28:01 2013 From: barry at python.org (Barry Warsaw) Date: Mon, 11 Feb 2013 10:28:01 -0500 Subject: [Python-Dev] checking what atexit handlers are registered in Python 3 In-Reply-To: <5117781F.2080401@simplistix.co.uk> References: <5117781F.2080401@simplistix.co.uk> Message-ID: <20130211102801.5c275c4d@anarchist.wooz.org> On Feb 10, 2013, at 10:36 AM, Chris Withers wrote: >One rough edge I've hit: I see the atexit module has moved to be C-based and, >as far as I can tell, no longer allows you to introspect what atexit >functions have been registered. > >If I'm writing tests for code that registers atexit handlers, how can I check >that they've been correctly registered? Looks like you can't through the atexit module. Even though Python 2's atexit._exithandlers is a private, undocumented attribute, the lack of an introspection API in Python 3's atexit seems like a functional regression. Please file a bug, although it will have to be a new feature for Python 3.4. Cheers, -Barry From rdmurray at bitdance.com Mon Feb 11 16:59:44 2013 From: rdmurray at bitdance.com (R. David Murray) Date: Mon, 11 Feb 2013 10:59:44 -0500 Subject: [Python-Dev] Question regarding: Lib/_markupbase.py In-Reply-To: <1360594020.18756.YahooMailNeo@web171403.mail.ir2.yahoo.com> References: <1360585008.27100.YahooMailNeo@web171404.mail.ir2.yahoo.com> <1360594020.18756.YahooMailNeo@web171403.mail.ir2.yahoo.com> Message-ID: <20130211155945.62F0C250030@webabinitio.net> If these don't get reported as tracker issues they will probably get lost. --David On Mon, 11 Feb 2013 14:47:00 +0000, Developer Developer wrote: > Same thing in the function: "_parse_doctype_attlist": > > if ")" in rawdata[j:]: > ?? ?? j = rawdata.find(")", j) + 1 > else: > ?? ?? return -1 > > I would change it to: > pos =??rawdata.find(")", j) > if pos != -1: > ?? ?? j = pos + 1 > else: > ?? ?? return -1 > > > Best regards, > Guido > > > ----- Urspr??ngliche Message ----- > Von: Fred Drake > An: Developer Developer > CC: "python-dev at python.org" > Gesendet: 15:10 Montag, 11.Februar 2013 > Betreff: Re: [Python-Dev] Question regarding: Lib/_markupbase.py > > On Mon, Feb 11, 2013 at 7:16 AM, Developer Developer > wrote: > > Wouldn't it be better to do the following? > ... > > Otherwise I think we are scanning rawdata[j:] twice. > > Yes, that would be better, and avoids a string object creation as well. > > > ?? -Fred > > -- > Fred L. Drake, Jr.?? ?? > "A storm broke loose in my mind."?? --Albert Einstein > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/rdmurray%40bitdance.com From just_another_developer at yahoo.de Mon Feb 11 17:15:59 2013 From: just_another_developer at yahoo.de (Developer Developer) Date: Mon, 11 Feb 2013 16:15:59 +0000 (GMT) Subject: [Python-Dev] Question regarding: Lib/_markupbase.py In-Reply-To: <20130211155945.62F0C250030@webabinitio.net> References: <1360585008.27100.YahooMailNeo@web171404.mail.ir2.yahoo.com> <1360594020.18756.YahooMailNeo@web171403.mail.ir2.yahoo.com> <20130211155945.62F0C250030@webabinitio.net> Message-ID: <1360599359.83344.YahooMailNeo@web171401.mail.ir2.yahoo.com> Thank you David, I didn't think of the issue tracker. I have just done it. Guido ----- Urspr?ngliche Message ----- Von: R. David Murray An: Developer Developer CC: "python-dev at python.org" Gesendet: 16:59 Montag, 11.Februar 2013 Betreff: Re: [Python-Dev] Question regarding: Lib/_markupbase.py If these don't get reported as tracker issues they will probably get lost. --David On Mon, 11 Feb 2013 14:47:00 +0000, Developer Developer wrote: > Same thing in the function: "_parse_doctype_attlist": > > if ")" in rawdata[j:]: > ? ? j = rawdata.find(")", j) + 1 > else: > ? ? return -1 > > I would change it to: > pos =?rawdata.find(")", j) > if pos != -1: > ? ? j = pos + 1 > else: > ? ? return -1 > > > Best regards, > Guido > > > ----- Urspr?ngliche Message ----- > Von: Fred Drake > An: Developer Developer > CC: "python-dev at python.org" > Gesendet: 15:10 Montag, 11.Februar 2013 > Betreff: Re: [Python-Dev] Question regarding: Lib/_markupbase.py > > On Mon, Feb 11, 2013 at 7:16 AM, Developer Developer > wrote: > > Wouldn't it be better to do the following? > ... > > Otherwise I think we are scanning rawdata[j:] twice. > > Yes, that would be better, and avoids a string object creation as well. > > > ? -Fred > > -- > Fred L. Drake, Jr.? ? > "A storm broke loose in my mind."? --Albert Einstein > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/rdmurray%40bitdance.com From guido at python.org Mon Feb 11 18:44:46 2013 From: guido at python.org (Guido van Rossum) Date: Mon, 11 Feb 2013 09:44:46 -0800 Subject: [Python-Dev] Submitting PEP 422 (Simple class initialization hook) for pronouncement In-Reply-To: <20130211162127.6d4705c8@pitrou.net> References: <20130210134700.7dd822df@pitrou.net> <20130210143442.5e3cafcc@pitrou.net> <20130210152814.66c26f76@pitrou.net> <20130211101536.6f769ee3@anarchist.wooz.org> <20130211162127.6d4705c8@pitrou.net> Message-ID: Hi Nick, I think this will make a fine addition to the language. I agree that it is superior to the alternatives and fulfills a real (if rare) need. I only have a few nits/questions/suggestions. - With PJE, I think __init_class__ should automatically be a class method. The same way that __new__ is automatically a class method. - Would it make any sense to require that __init_class__ *returns* the new class object (to complete the similarity with class decorators)? Frankly, I lost track of things when you were discussing the exact point in time where __class__ is added to the class. (This may place me outside the set of people who fully understand Python's metaclasses. So be it. :-) Perhaps some explanation of the existing mechanism would be in place? It would be nice if you explained reason for why __class__ cannot be set earlier in the game (otherwise the response to this argument would be "so fix it", but somehow I doubt that is possible). I think it may also be unclear at all times whether __class__, when referenced, is meant to refer to the metaclass or to the class under construction. - I wouldn't add the OrderedMeta/OrderedClass to the PEP. Keep it focused on enabling. - Adding more examples would be great. -- --Guido van Rossum (python.org/~guido) From techtonik at gmail.com Mon Feb 11 19:18:58 2013 From: techtonik at gmail.com (anatoly techtonik) Date: Mon, 11 Feb 2013 21:18:58 +0300 Subject: [Python-Dev] My CLA In-Reply-To: <20130211130151.GB28063@iskra.aviel.ru> References: <20130211130151.GB28063@iskra.aviel.ru> Message-ID: On Mon, Feb 11, 2013 at 4:01 PM, Oleg Broytman wrote: > On Mon, Feb 11, 2013 at 03:49:39PM +0300, anatoly techtonik < > techtonik at gmail.com> wrote: > > Python Contributor Agreement > > ---------------------------- > > I allow PSF to release all my code that I submitted to > > it, under any open source license. > > Good intention but wrong way of expressing it. Please do it properly -- > via a signed paper. You can send it by snail mail, or you can scan it > and send by email. What's wrong with it? Is the text not clear? Or there is a problem to threat email as a document? -------------- next part -------------- An HTML attachment was scrubbed... URL: From amauryfa at gmail.com Mon Feb 11 19:27:48 2013 From: amauryfa at gmail.com (Amaury Forgeot d'Arc) Date: Mon, 11 Feb 2013 19:27:48 +0100 Subject: [Python-Dev] My CLA In-Reply-To: References: <20130211130151.GB28063@iskra.aviel.ru> Message-ID: 2013/2/11 anatoly techtonik > On Mon, Feb 11, 2013 at 4:01 PM, Oleg Broytman wrote: > >> On Mon, Feb 11, 2013 at 03:49:39PM +0300, anatoly techtonik < >> techtonik at gmail.com> wrote: >> > Python Contributor Agreement >> > ---------------------------- >> > I allow PSF to release all my code that I submitted to >> > it, under any open source license. >> >> Good intention but wrong way of expressing it. Please do it properly -- >> via a signed paper. You can send it by snail mail, or you can scan it >> and send by email. > > > What's wrong with it? Is the text not clear? Or there is a problem to > threat email as a document? > See the "Submission Instructions" there: http://www.python.org/psf/contrib/ The Contributor Agreement is part of a formal process. It's necessary to follow the rules, even if they were written by a lawyer and we don't understand all the reasons. -- Amaury Forgeot d'Arc -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Mon Feb 11 19:27:51 2013 From: guido at python.org (Guido van Rossum) Date: Mon, 11 Feb 2013 10:27:51 -0800 Subject: [Python-Dev] My CLA In-Reply-To: References: <20130211130151.GB28063@iskra.aviel.ru> Message-ID: Anatoly, stop this discussion *NOW*. It is not appropriate for python-dev and you risk being banned from python-dev if you keep it up. On Mon, Feb 11, 2013 at 10:18 AM, anatoly techtonik wrote: > On Mon, Feb 11, 2013 at 4:01 PM, Oleg Broytman wrote: > >> On Mon, Feb 11, 2013 at 03:49:39PM +0300, anatoly techtonik < >> techtonik at gmail.com> wrote: >> > Python Contributor Agreement >> > ---------------------------- >> > I allow PSF to release all my code that I submitted to >> > it, under any open source license. >> >> Good intention but wrong way of expressing it. Please do it properly -- >> via a signed paper. You can send it by snail mail, or you can scan it >> and send by email. > > > What's wrong with it? Is the text not clear? Or there is a problem to > threat email as a document? > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/guido%40python.org > > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From phd at phdru.name Mon Feb 11 19:30:06 2013 From: phd at phdru.name (Oleg Broytman) Date: Mon, 11 Feb 2013 22:30:06 +0400 Subject: [Python-Dev] My CLA In-Reply-To: References: <20130211130151.GB28063@iskra.aviel.ru> Message-ID: <20130211183006.GA7718@iskra.aviel.ru> On Mon, Feb 11, 2013 at 09:18:58PM +0300, anatoly techtonik wrote: > On Mon, Feb 11, 2013 at 4:01 PM, Oleg Broytman wrote: > > > On Mon, Feb 11, 2013 at 03:49:39PM +0300, anatoly techtonik < > > techtonik at gmail.com> wrote: > > > Python Contributor Agreement > > > ---------------------------- > > > I allow PSF to release all my code that I submitted to > > > it, under any open source license. > > > > Good intention but wrong way of expressing it. Please do it properly -- > > via a signed paper. You can send it by snail mail, or you can scan it > > and send by email. > > What's wrong with it? Is the text not clear? Or there is a problem to > threat email as a document? Yes, email is not a legally recognized document. Electronic signature *could* make it legally recognizable but it very much depends on the organization where you send email to and on the certificate you use to sign mail. Contact PSF for details. I doubt python-dev is a proper list to discuss PSF-related legal issues. Oleg. -- Oleg Broytman http://phdru.name/ phd at phdru.name Programmers don't die, they just GOSUB without RETURN. From phd at phdru.name Mon Feb 11 13:37:35 2013 From: phd at phdru.name (Oleg Broytman) Date: Mon, 11 Feb 2013 16:37:35 +0400 Subject: [Python-Dev] Question regarding: Lib/_markupbase.py In-Reply-To: <1360585008.27100.YahooMailNeo@web171404.mail.ir2.yahoo.com> References: <1360585008.27100.YahooMailNeo@web171404.mail.ir2.yahoo.com> Message-ID: <20130211123735.GA28063@iskra.aviel.ru> On Mon, Feb 11, 2013 at 12:16:48PM +0000, Developer Developer wrote: > I was having a look at the file:?Lib/_markupbase.py (@ 82151), function: "_parse_doctype_element" and have seen something that has caught my attention: > > if '>' in rawdata[j:]: > ? ? return rawdata.find(">", j) + 1 > > > Wouldn't it be better to do the following? > pos = rawdata.find(">", j) > if pos != -1: > ? ? return pos + 1 > > Otherwise I think we are scanning rawdata[j:] twice. Is it really a significant optimization? Can you do an experiment and show figures? Oleg. -- Oleg Broytman http://phdru.name/ phd at phdru.name Programmers don't die, they just GOSUB without RETURN. From brian at python.org Mon Feb 11 19:56:23 2013 From: brian at python.org (Brian Curtin) Date: Mon, 11 Feb 2013 12:56:23 -0600 Subject: [Python-Dev] My CLA In-Reply-To: <20130211183006.GA7718@iskra.aviel.ru> References: <20130211130151.GB28063@iskra.aviel.ru> <20130211183006.GA7718@iskra.aviel.ru> Message-ID: On Mon, Feb 11, 2013 at 12:30 PM, Oleg Broytman wrote: > On Mon, Feb 11, 2013 at 09:18:58PM +0300, anatoly techtonik wrote: >> On Mon, Feb 11, 2013 at 4:01 PM, Oleg Broytman wrote: >> >> > On Mon, Feb 11, 2013 at 03:49:39PM +0300, anatoly techtonik < >> > techtonik at gmail.com> wrote: >> > > Python Contributor Agreement >> > > ---------------------------- >> > > I allow PSF to release all my code that I submitted to >> > > it, under any open source license. >> > >> > Good intention but wrong way of expressing it. Please do it properly -- >> > via a signed paper. You can send it by snail mail, or you can scan it >> > and send by email. >> >> What's wrong with it? Is the text not clear? Or there is a problem to >> threat email as a document? > > Yes, email is not a legally recognized document. Electronic signature > *could* make it legally recognizable but it very much depends on the > organization where you send email to and on the certificate you use to > sign mail. > Contact PSF for details. I doubt python-dev is a proper list to > discuss PSF-related legal issues. There are no further details. Either the proper document is signed or it isn't. Hopefully this is the end of the discussion. Brian Curtin Director Python Software Foundation From guido at python.org Mon Feb 11 20:02:04 2013 From: guido at python.org (Guido van Rossum) Date: Mon, 11 Feb 2013 11:02:04 -0800 Subject: [Python-Dev] Question regarding: Lib/_markupbase.py In-Reply-To: <20130211123735.GA28063@iskra.aviel.ru> References: <1360585008.27100.YahooMailNeo@web171404.mail.ir2.yahoo.com> <20130211123735.GA28063@iskra.aviel.ru> Message-ID: Warning: see http://bugs.python.org/issue17170. Depending on the length of the string being scanned and the probability of finding the specific character, the proposed change could actually be a *pessimization*. OTOH if the character occurs many times, the slice will actually cause O(N**2) behavior. So yes, it depends greatly on the distribution of the input data. On Mon, Feb 11, 2013 at 4:37 AM, Oleg Broytman wrote: > On Mon, Feb 11, 2013 at 12:16:48PM +0000, Developer Developer < > just_another_developer at yahoo.de> wrote: > > I was having a look at the file: Lib/_markupbase.py (@ 82151), function: > "_parse_doctype_element" and have seen something that has caught my > attention: > > > > if '>' in rawdata[j:]: > > return rawdata.find(">", j) + 1 > > > > > > Wouldn't it be better to do the following? > > pos = rawdata.find(">", j) > > if pos != -1: > > return pos + 1 > > > > Otherwise I think we are scanning rawdata[j:] twice. > > Is it really a significant optimization? Can you do an experiment and > show figures? > > Oleg. > -- > Oleg Broytman http://phdru.name/ phd at phdru.name > Programmers don't die, they just GOSUB without RETURN. > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/guido%40python.org > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From techtonik at gmail.com Mon Feb 11 20:07:50 2013 From: techtonik at gmail.com (anatoly techtonik) Date: Mon, 11 Feb 2013 22:07:50 +0300 Subject: [Python-Dev] My CLA In-Reply-To: References: <20130211130151.GB28063@iskra.aviel.ru> Message-ID: On Mon, Feb 11, 2013 at 9:27 PM, Guido van Rossum wrote: > Anatoly, stop this discussion *NOW*. It is not appropriate for python-dev > and you risk being banned from python-dev if you keep it up. > It is not a problem for me to keep silence for another couple of months. But this weekend there will be an open source conference in Belarus [1], and I will need to explain what this specific CLA is about in developer-friendly language translated to Russian. So what would be the best place to discuss the matter in public, so that I can give a link to the discussion afterwards? 1. http://lvee.org/en/main -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Mon Feb 11 20:23:05 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 11 Feb 2013 20:23:05 +0100 Subject: [Python-Dev] My CLA References: <20130211130151.GB28063@iskra.aviel.ru> Message-ID: <20130211202305.66d1d0f6@pitrou.net> On Mon, 11 Feb 2013 22:07:50 +0300 anatoly techtonik wrote: > On Mon, Feb 11, 2013 at 9:27 PM, Guido van Rossum wrote: > > > Anatoly, stop this discussion *NOW*. It is not appropriate for python-dev > > and you risk being banned from python-dev if you keep it up. > > > > It is not a problem for me to keep silence for another couple of months. > But this weekend there will be an open source conference in Belarus [1], > and I will need to explain what this specific CLA is about in > developer-friendly language translated to Russian. The Python contributor agreement allows the PSF to safely redistribute your contributions under its own license, the PSF license. The Python contributor agreement is *not* a copyright assignment: you legally remain the author of the code you contributed (i.e. you can also publish it elsewhere under any license you want). Regards Antoine. From jnoller at gmail.com Mon Feb 11 20:57:31 2013 From: jnoller at gmail.com (Jesse Noller) Date: Mon, 11 Feb 2013 14:57:31 -0500 Subject: [Python-Dev] My CLA In-Reply-To: <20130211202305.66d1d0f6@pitrou.net> References: <20130211130151.GB28063@iskra.aviel.ru> <20130211202305.66d1d0f6@pitrou.net> Message-ID: On Monday, February 11, 2013 at 2:23 PM, Antoine Pitrou wrote: > On Mon, 11 Feb 2013 22:07:50 +0300 > anatoly techtonik wrote: > > On Mon, Feb 11, 2013 at 9:27 PM, Guido van Rossum wrote: > > > > > Anatoly, stop this discussion *NOW*. It is not appropriate for python-dev > > > and you risk being banned from python-dev if you keep it up. > > > > > > > > It is not a problem for me to keep silence for another couple of months. > > But this weekend there will be an open source conference in Belarus [1], > > and I will need to explain what this specific CLA is about in > > developer-friendly language translated to Russian. > > > > The Python contributor agreement allows the PSF to safely redistribute > your contributions under its own license, the PSF license. > > The Python contributor agreement is *not* a copyright assignment: you > legally remain the author of the code you contributed (i.e. you can also > publish it elsewhere under any license you want). > > Regards > > Antoine. > FWIW: Django's FAQ spells out the same reasons we have one for Python: https://www.djangoproject.com/foundation/cla/faq/ just s/Django/Python/ and s/Django Software Foundation/Python Software Foundation/ - it's a good concise FAQ. From pje at telecommunity.com Mon Feb 11 21:57:06 2013 From: pje at telecommunity.com (PJ Eby) Date: Mon, 11 Feb 2013 15:57:06 -0500 Subject: [Python-Dev] Submitting PEP 422 (Simple class initialization hook) for pronouncement In-Reply-To: References: <20130210134700.7dd822df@pitrou.net> <20130210143442.5e3cafcc@pitrou.net> <20130210152814.66c26f76@pitrou.net> <20130211101536.6f769ee3@anarchist.wooz.org> <20130211162127.6d4705c8@pitrou.net> Message-ID: On Mon, Feb 11, 2013 at 12:44 PM, Guido van Rossum wrote: > Hi Nick, > > I think this will make a fine addition to the language. I agree that > it is superior to the alternatives and fulfills a real (if rare) need. > > I only have a few nits/questions/suggestions. > > - With PJE, I think __init_class__ should automatically be a class > method. Actually, I didn't say that as such, because I'm not sure how the heck we'd implement that. ;-) For example, at what point is it converted to a classmethod? Is it going to be a slot method with special C-level handling? Handled by the compiler? What happens if somebody makes it a > The same way that __new__ is automatically a class method. Actually, isn't it automatically a staticmethod? Oh crap. Now that I'm thinking about it, doesn't this *have* to be a static method, explicitly passing in the class? I mean, otherwise, won't calling super().__init_class__() invoke it on the base class, rather than the current class? ISTM that EIBTI argues for the __new__/staticmethod approach, especially if you're returning the class (per below) > - Would it make any sense to require that __init_class__ *returns* the > new class object (to complete the similarity with class decorators)? It would certainly be quite useful to do so, but in that case, perhaps the method should be named __decorate_class__? And in that event the standard usage would look like: def __decorate_class__(cls): cls = super().__decorate_class__(cls) # do stuff return cls On the other hand, one could just drop the super() requirement and make the usage even simpler by having the class machinery walk the MRO and pass each method the result of invoking the previous one. Then the methods are short and sweet, and super() and __class__ don't come into it. (Though I guess the class machinery could keep setting __class__ to whatever the last-returned class was.) In his first draft, Nick implemented inheritable decorators instead, using a __decorators__ attribute in the class body, or something like that. While that approach had an issue or two of its own, it's possible that just going with a single __decorate_class__ method would work out better. From guido at python.org Mon Feb 11 22:44:05 2013 From: guido at python.org (Guido van Rossum) Date: Mon, 11 Feb 2013 13:44:05 -0800 Subject: [Python-Dev] Submitting PEP 422 (Simple class initialization hook) for pronouncement In-Reply-To: References: <20130210134700.7dd822df@pitrou.net> <20130210143442.5e3cafcc@pitrou.net> <20130210152814.66c26f76@pitrou.net> <20130211101536.6f769ee3@anarchist.wooz.org> <20130211162127.6d4705c8@pitrou.net> Message-ID: On Mon, Feb 11, 2013 at 12:57 PM, PJ Eby wrote: > On Mon, Feb 11, 2013 at 12:44 PM, Guido van Rossum > wrote: > > Hi Nick, > > > > I think this will make a fine addition to the language. I agree that > > it is superior to the alternatives and fulfills a real (if rare) need. > > > > I only have a few nits/questions/suggestions. > > > > - With PJE, I think __init_class__ should automatically be a class > > method. > > Actually, I didn't say that as such, because I'm not sure how the heck > we'd implement that. ;-) > > For example, at what point is it converted to a classmethod? Is it > going to be a slot method with special C-level handling? Handled by > the compiler? What happens if somebody makes it a > > > The same way that __new__ is automatically a class method. > > Actually, isn't it automatically a staticmethod? Oh crap. Now that > I'm thinking about it, doesn't this *have* to be a static method, > explicitly passing in the class? I mean, otherwise, won't calling > super().__init_class__() invoke it on the base class, rather than the > current class? > > ISTM that EIBTI argues for the __new__/staticmethod approach, > especially if you're returning the class (per below) > Let's see what Nick and the implementer say. > > - Would it make any sense to require that __init_class__ *returns* the > > new class object (to complete the similarity with class decorators)? > > It would certainly be quite useful to do so, but in that case, perhaps > the method should be named __decorate_class__? And in that event the > standard usage would look like: > > def __decorate_class__(cls): > cls = super().__decorate_class__(cls) > # do stuff > return cls > > On the other hand, one could just drop the super() requirement and > make the usage even simpler by having the class machinery walk the MRO > and pass each method the result of invoking the previous one. Then > the methods are short and sweet, and super() and __class__ don't come > into it. (Though I guess the class machinery could keep setting > __class__ to whatever the last-returned class was.) > > In his first draft, Nick implemented inheritable decorators instead, > using a __decorators__ attribute in the class body, or something like > that. While that approach had an issue or two of its own, it's > possible that just going with a single __decorate_class__ method would > work out better. > Half-baked idea: Maybe the base class __decorate_class__ would call the class decorators? Or does that not make sense? -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Mon Feb 11 23:29:00 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 12 Feb 2013 08:29:00 +1000 Subject: [Python-Dev] Submitting PEP 422 (Simple class initialization hook) for pronouncement In-Reply-To: References: <20130210134700.7dd822df@pitrou.net> <20130210143442.5e3cafcc@pitrou.net> <20130210152814.66c26f76@pitrou.net> <20130211101536.6f769ee3@anarchist.wooz.org> <20130211162127.6d4705c8@pitrou.net> Message-ID: On 12 Feb 2013 07:44, "Guido van Rossum" wrote: > > On Mon, Feb 11, 2013 at 12:57 PM, PJ Eby wrote: >> >> On Mon, Feb 11, 2013 at 12:44 PM, Guido van Rossum wrote: >> > Hi Nick, >> > >> > I think this will make a fine addition to the language. I agree that >> > it is superior to the alternatives and fulfills a real (if rare) need. >> > >> > I only have a few nits/questions/suggestions. >> > >> > - With PJE, I think __init_class__ should automatically be a class >> > method. >> >> Actually, I didn't say that as such, because I'm not sure how the heck >> we'd implement that. ;-) >> >> For example, at what point is it converted to a classmethod? Is it >> going to be a slot method with special C-level handling? Handled by >> the compiler? What happens if somebody makes it a >> >> > The same way that __new__ is automatically a class method. >> >> Actually, isn't it automatically a staticmethod? Oh crap. Now that >> I'm thinking about it, doesn't this *have* to be a static method, >> explicitly passing in the class? I mean, otherwise, won't calling >> super().__init_class__() invoke it on the base class, rather than the >> current class? >> >> ISTM that EIBTI argues for the __new__/staticmethod approach, >> especially if you're returning the class (per below) > > > Let's see what Nick and the implementer say. I think these are some interesting ideas and it's going to take me a while to digest them and update the PEP :) A few random thoughts: 1. I like the idea of a metaprogramming "howto" that provides advice on choosing a suitable level of metaprogramming (with the default choice being "use existing decorators", then escalating through creating custom decorators all the way to creating custom metaclasses). I don't think the PEP needs to be conditional on writing that, but I will at least add PJE's list to the PEP itself. 2. I see the new method as more analogous to__init__ than to__new__, so the __decorate_class__ idea makes me nervous, as it's closer to a __new__ method. Composition gets a *lot* harder when your parent class can switch out the object on you. 3. I'm trying to avoid any custom magic specific to this method, but making it implicitly a static or class method is fairly easy if we so choose - the standard retrieval code during class creation can just bypass the descriptor machinery, and wrap it in staticmethod or classmethod if it isn't already. Given that __new__ is already implicitly static, it may be easier to follow that precedent here rather than trying to explain why an explicit @classmethod is needed in one case but not the other. 4.__class__ is already bound as soon as we have a class object to bind it to, so we can't move it any earlier. However, it's already early enough to let references to it from the new method (including the implied one in zero-arg super) work properly. The black magic that is zero-arg super also deals with PJE's concern about propagating the actual class up the MRO (as it is equivalent to "super(__class__, first_argument)"). 5. Implicitly walking the MRO bothers me, as it becomes a special case for people to learn. We don't do that for __init__ or __new__, so I don't think it makes sense to do it here. We can include a recommended structure in the docs, where the first step is calling the parent through. As PJE suggested, a no-op method on type will make that simple and fairly safe (people using a metaclass hierarchy not anchored on type can figure out their own equivalent) Cheers, Nick. > >> >> > - Would it make any sense to require that __init_class__ *returns* the >> > new class object (to complete the similarity with class decorators)? >> >> It would certainly be quite useful to do so, but in that case, perhaps >> the method should be named __decorate_class__? And in that event the >> standard usage would look like: >> >> def __decorate_class__(cls): >> cls = super().__decorate_class__(cls) >> # do stuff >> return cls >> >> On the other hand, one could just drop the super() requirement and >> make the usage even simpler by having the class machinery walk the MRO >> and pass each method the result of invoking the previous one. Then >> the methods are short and sweet, and super() and __class__ don't come >> into it. (Though I guess the class machinery could keep setting >> __class__ to whatever the last-returned class was.) >> >> In his first draft, Nick implemented inheritable decorators instead, >> using a __decorators__ attribute in the class body, or something like >> that. While that approach had an issue or two of its own, it's >> possible that just going with a single __decorate_class__ method would >> work out better. > > > Half-baked idea: Maybe the base class __decorate_class__ would call the class decorators? Or does that not make sense? > > -- > --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Mon Feb 11 23:35:08 2013 From: guido at python.org (Guido van Rossum) Date: Mon, 11 Feb 2013 14:35:08 -0800 Subject: [Python-Dev] Submitting PEP 422 (Simple class initialization hook) for pronouncement In-Reply-To: References: <20130210134700.7dd822df@pitrou.net> <20130210143442.5e3cafcc@pitrou.net> <20130210152814.66c26f76@pitrou.net> <20130211101536.6f769ee3@anarchist.wooz.org> <20130211162127.6d4705c8@pitrou.net> Message-ID: On Mon, Feb 11, 2013 at 2:29 PM, Nick Coghlan wrote: > > On 12 Feb 2013 07:44, "Guido van Rossum" wrote: > > > > On Mon, Feb 11, 2013 at 12:57 PM, PJ Eby wrote: > >> > >> On Mon, Feb 11, 2013 at 12:44 PM, Guido van Rossum > wrote: > >> > Hi Nick, > >> > > >> > I think this will make a fine addition to the language. I agree that > >> > it is superior to the alternatives and fulfills a real (if rare) need. > >> > > >> > I only have a few nits/questions/suggestions. > >> > > >> > - With PJE, I think __init_class__ should automatically be a class > >> > method. > >> > >> Actually, I didn't say that as such, because I'm not sure how the heck > >> we'd implement that. ;-) > >> > >> For example, at what point is it converted to a classmethod? Is it > >> going to be a slot method with special C-level handling? Handled by > >> the compiler? What happens if somebody makes it a > >> > >> > The same way that __new__ is automatically a class method. > >> > >> Actually, isn't it automatically a staticmethod? Oh crap. Now that > >> I'm thinking about it, doesn't this *have* to be a static method, > >> explicitly passing in the class? I mean, otherwise, won't calling > >> super().__init_class__() invoke it on the base class, rather than the > >> current class? > >> > >> ISTM that EIBTI argues for the __new__/staticmethod approach, > >> especially if you're returning the class (per below) > > > > > > Let's see what Nick and the implementer say. > > I think these are some interesting ideas and it's going to take me a while > to digest them and update the PEP :) > Heh. :-) > A few random thoughts: > > 1. I like the idea of a metaprogramming "howto" that provides advice on > choosing a suitable level of metaprogramming (with the default choice being > "use existing decorators", then escalating through creating custom > decorators all the way to creating custom metaclasses). I don't think the > PEP needs to be conditional on writing that, but I will at least add PJE's > list to the PEP itself. > Sure. > 2. I see the new method as more analogous to__init__ than to__new__, so > the __decorate_class__ idea makes me nervous, as it's closer to a __new__ > method. Composition gets a *lot* harder when your parent class can switch > out the object on you. > Fair enough. > 3. I'm trying to avoid any custom magic specific to this method, but > making it implicitly a static or class method is fairly easy if we so > choose - the standard retrieval code during class creation can just bypass > the descriptor machinery, and wrap it in staticmethod or classmethod if it > isn't already. Given that __new__ is already implicitly static, it may be > easier to follow that precedent here rather than trying to explain why an > explicit @classmethod is needed in one case but not the other. > Also fair enough. > 4.__class__ is already bound as soon as we have a class object to bind it > to, so we can't move it any earlier. However, it's already early enough to > let references to it from the new method (including the implied one in > zero-arg super) work properly. The black magic that is zero-arg super also > deals with PJE's concern about propagating the actual class up the MRO (as > it is equivalent to "super(__class__, first_argument)"). > So where is it not bound when a metaclass runs? I guess in the metaclass's __new__()? Because in the metaclass's __init__() it should exist IIUC. > 5. Implicitly walking the MRO bothers me, as it becomes a special case for > people to learn. We don't do that for __init__ or __new__, so I don't think > it makes sense to do it here. We can include a recommended structure in the > docs, where the first step is calling the parent through. As PJE suggested, > a no-op method on type will make that simple and fairly safe (people using > a metaclass hierarchy not anchored on type can figure out their own > equivalent) > Agreed. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From zuo at chopin.edu.pl Tue Feb 12 00:56:25 2013 From: zuo at chopin.edu.pl (Jan Kaliszewski) Date: Tue, 12 Feb 2013 00:56:25 +0100 Subject: [Python-Dev] Submitting PEP 422 (Simple class initialization hook) for pronouncement In-Reply-To: References: <20130210134700.7dd822df@pitrou.net> <20130210143442.5e3cafcc@pitrou.net> <20130210152814.66c26f76@pitrou.net> <20130211101536.6f769ee3@anarchist.wooz.org> <20130211162127.6d4705c8@pitrou.net> Message-ID: <3230f1789df0f010e01da15dab7e7d22@chopin.edu.pl> 11.02.2013 23:29, Nick Coghlan wrote: > 3. I'm trying to avoid any custom magic specific to this method, but > making it implicitly a static or class method is fairly easy if we so > choose - the standard retrieval code during class creation can just > bypass the descriptor machinery, and wrap it in staticmethod or > classmethod if it isn't already. Given that __new__ is already > implicitly static, it may be easier to follow that precedent here > rather than trying to explain why an explicit @classmethod is needed > in one case but not the other. Though __new__ is implicitly a *static* rather than a *class* method (so we can use it e.g. by calling object.__new__(MyClass), though -- besides -- in Py3k unbound methods have gone so the difference between static and non-static-and-non-class-methods is smaller than in Py2.x), in case of __init_class__ + super() it'd have to be called: super().__init_class__(__class__) ...and it seems to me a bit awkward. And making it implicitly a *class* rather than a *static* method whould make *impossible* to do calls such as: ExplicitAncestor.__init_class__(ExplicitDescendant) ...though I'm not sure we'd ever need such a call. If not -- implicit *class* method may be a good idea, but if we would? *** On the margin: is that middle underscore in '__init_class__' necessary? We had __metaclass__, not __meta_class__... OK, it's one world, but still we also have __getattr__, __getattribute__, __getitem__, __instancecheck__, __qualname__, __truediv__ etc. (not __get_attr__, __instance_check__ etc.). [I remember only one exception: __reduce_ex__, rather rarely used, and easy to defend against weird __reduceex__]. Wouldn't __initclass__ be readable enough? IMHO it could spare users trouble with remembering special case. Cheers. *j From steve at pearwood.info Tue Feb 12 01:09:15 2013 From: steve at pearwood.info (Steven D'Aprano) Date: Tue, 12 Feb 2013 11:09:15 +1100 Subject: [Python-Dev] Submitting PEP 422 (Simple class initialization hook) for pronouncement In-Reply-To: <3230f1789df0f010e01da15dab7e7d22@chopin.edu.pl> References: <20130210134700.7dd822df@pitrou.net> <20130210143442.5e3cafcc@pitrou.net> <20130210152814.66c26f76@pitrou.net> <20130211101536.6f769ee3@anarchist.wooz.org> <20130211162127.6d4705c8@pitrou.net> <3230f1789df0f010e01da15dab7e7d22@chopin.edu.pl> Message-ID: <5119882B.2020306@pearwood.info> On 12/02/13 10:56, Jan Kaliszewski wrote: > Wouldn't __initclass__ be readable enough? IMHO it could spare users > trouble with remembering special case. +1 I approve of the colour of this bikeshed. __init_class__ has too many underscores. -- Steven From ncoghlan at gmail.com Tue Feb 12 08:16:56 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 12 Feb 2013 17:16:56 +1000 Subject: [Python-Dev] Submitting PEP 422 (Simple class initialization hook) for pronouncement In-Reply-To: References: <20130210134700.7dd822df@pitrou.net> <20130210143442.5e3cafcc@pitrou.net> <20130210152814.66c26f76@pitrou.net> <20130211101536.6f769ee3@anarchist.wooz.org> <20130211162127.6d4705c8@pitrou.net> Message-ID: On Tue, Feb 12, 2013 at 8:35 AM, Guido van Rossum wrote: > On Mon, Feb 11, 2013 at 2:29 PM, Nick Coghlan wrote: >> 4.__class__ is already bound as soon as we have a class object to bind it >> to, so we can't move it any earlier. However, it's already early enough to >> let references to it from the new method (including the implied one in >> zero-arg super) work properly. The black magic that is zero-arg super also >> deals with PJE's concern about propagating the actual class up the MRO (as >> it is equivalent to "super(__class__, first_argument)"). > > So where is it not bound when a metaclass runs? I guess in the metaclass's > __new__()? Because in the metaclass's __init__() it should exist IIUC. I stand corrected, we don't *quite* bind it as soon as the class object exists - we bind it as soon as the call to the metaclass returns. So even though the class object exists when __init__ runs, it isn't bound to __class__ yet. Given that the class construction machinery treats the metaclass constructor as an opaque callable, trying to get the cell binding to happen between the call to __new__ and the one to __init__ would be rather challenging (Translation: I'm not sure where you would even start...). Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From chris at simplistix.co.uk Tue Feb 12 08:05:41 2013 From: chris at simplistix.co.uk (Chris Withers) Date: Tue, 12 Feb 2013 07:05:41 +0000 Subject: [Python-Dev] what is a dict_keys and where can I import it from? Message-ID: <5119E9C5.3070400@simplistix.co.uk> Hi all, So, dicts in Python 3 return "something different" from their keys and values methods: >>> dict(x=1, y=2).keys() dict_keys(['y', 'x']) >>> type(dict(x=1, y=2).keys()) I have vague memories of these things being referred to as views or some such? Where can I learn more? More importantly, how can I tell if I have one of them? I guess I can keep a reference to type({}.keys()) somewhere, but that feels a bit yucky. Should these things be in the types module? cheers, Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk From andrew.svetlov at gmail.com Tue Feb 12 11:18:21 2013 From: andrew.svetlov at gmail.com (Andrew Svetlov) Date: Tue, 12 Feb 2013 12:18:21 +0200 Subject: [Python-Dev] what is a dict_keys and where can I import it from? In-Reply-To: <5119E9C5.3070400@simplistix.co.uk> References: <5119E9C5.3070400@simplistix.co.uk> Message-ID: Is collections.KeysView good for you? On Tue, Feb 12, 2013 at 9:05 AM, Chris Withers wrote: > Hi all, > > So, dicts in Python 3 return "something different" from their keys and > values methods: > >>>> dict(x=1, y=2).keys() > dict_keys(['y', 'x']) >>>> type(dict(x=1, y=2).keys()) > > > I have vague memories of these things being referred to as views or some > such? Where can I learn more? > > More importantly, how can I tell if I have one of them? > I guess I can keep a reference to type({}.keys()) somewhere, but that feels > a bit yucky. Should these things be in the types module? > > cheers, > > Chris > > -- > Simplistix - Content Management, Batch Processing & Python Consulting > - http://www.simplistix.co.uk > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/andrew.svetlov%40gmail.com -- Thanks, Andrew Svetlov From solipsis at pitrou.net Tue Feb 12 11:22:28 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 12 Feb 2013 11:22:28 +0100 Subject: [Python-Dev] Question regarding: Lib/_markupbase.py References: <1360585008.27100.YahooMailNeo@web171404.mail.ir2.yahoo.com> <20130211123735.GA28063@iskra.aviel.ru> Message-ID: <20130212112228.6eb5f0eb@pitrou.net> Le Mon, 11 Feb 2013 11:02:04 -0800, Guido van Rossum a ?crit : > Warning: see http://bugs.python.org/issue17170. Depending on the > length of the string being scanned and the probability of finding the > specific character, the proposed change could actually be a > *pessimization*. OTOH if the character occurs many times, the slice > will actually cause O(N**2) behavior. So yes, it depends greatly on > the distribution of the input data. That said, the savings are still puny unless you spend your time calling str.find(). Regards Antoine. From steve at pearwood.info Tue Feb 12 12:27:21 2013 From: steve at pearwood.info (Steven D'Aprano) Date: Tue, 12 Feb 2013 22:27:21 +1100 Subject: [Python-Dev] what is a dict_keys and where can I import it from? In-Reply-To: <5119E9C5.3070400@simplistix.co.uk> References: <5119E9C5.3070400@simplistix.co.uk> Message-ID: <511A2719.6090809@pearwood.info> On 12/02/13 18:05, Chris Withers wrote: > Hi all, > > So, dicts in Python 3 return "something different" from their keys and values methods: > >> >> dict(x=1, y=2).keys() > dict_keys(['y', 'x']) >> >> type(dict(x=1, y=2).keys()) > > > I have vague memories of these things being referred to as views or some such? Where can I learn more? The Fine Manual is usually a good place to refresh your vague memories :-) http://docs.python.org/3/library/stdtypes.html#dict.keys By the way, they're also in Python 2.7, only they're called "viewkeys" instead. > More importantly, how can I tell if I have one of them? Depends why you care. You may not care, but for those times where you do, they are in collections. py> from collections import KeysView py> keys = {}.keys() py> isinstance(keys, KeysView) True An anomaly, which I cannot explain: py> issubclass(type(keys), KeysView) True py> type(keys) is KeysView False py> type(keys).__mro__ (, ) This disturbs my calm, because I expect that if issubclass returns True, the two classes will either be identical, or the second will be in the MRO of the first. What have I missed? > I guess I can keep a reference to type({}.keys()) somewhere, but that feels a bit yucky. I remember Python 1.4 days when the only way to type-test something was: if type(something) is type([]): ... so dynamically grabbing the type from a literal when needed does not seem the least bit yucky to me. -- Steven From amauryfa at gmail.com Tue Feb 12 12:38:48 2013 From: amauryfa at gmail.com (Amaury Forgeot d'Arc) Date: Tue, 12 Feb 2013 12:38:48 +0100 Subject: [Python-Dev] what is a dict_keys and where can I import it from? In-Reply-To: <511A2719.6090809@pearwood.info> References: <5119E9C5.3070400@simplistix.co.uk> <511A2719.6090809@pearwood.info> Message-ID: 2013/2/12 Steven D'Aprano > An anomaly, which I cannot explain: > > py> issubclass(type(keys), KeysView) > True > py> type(keys) is KeysView > False > py> type(keys).__mro__ > (, ) > > > This disturbs my calm, because I expect that if issubclass returns True, > the two classes will either be identical, or the second will be in the MRO > of the first. What have I missed? > Ah, the magic of ABCs... KeysView overrides __instancecheck__, and can pretend to be any other class. This is precisely set in Lib/collections/abc.py: KeysView.register(dict_keys) -- Amaury Forgeot d'Arc -------------- next part -------------- An HTML attachment was scrubbed... URL: From fijall at gmail.com Tue Feb 12 22:03:04 2013 From: fijall at gmail.com (Maciej Fijalkowski) Date: Tue, 12 Feb 2013 23:03:04 +0200 Subject: [Python-Dev] Usage of += on strings in loops in stdlib Message-ID: Hi We recently encountered a performance issue in stdlib for pypy. It turned out that someone commited a performance "fix" that uses += for strings instead of "".join() that was there before. Now this hurts pypy (we can mitigate it to some degree though) and possible Jython and IronPython too. How people feel about generally not having += on long strings in stdlib (since the refcount = 1 thing is a hack)? What about other performance improvements in stdlib that are problematic for pypy or others? Personally I would like cleaner code in stdlib vs speeding up CPython. Typically that also helps pypy so I'm not unbiased. Cheers, fijal From solipsis at pitrou.net Tue Feb 12 22:06:42 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 12 Feb 2013 22:06:42 +0100 Subject: [Python-Dev] Usage of += on strings in loops in stdlib References: Message-ID: <20130212220642.71f03220@pitrou.net> Hi ! On Tue, 12 Feb 2013 23:03:04 +0200 Maciej Fijalkowski wrote: > > We recently encountered a performance issue in stdlib for pypy. It > turned out that someone commited a performance "fix" that uses += for > strings instead of "".join() that was there before. > > Now this hurts pypy (we can mitigate it to some degree though) and > possible Jython and IronPython too. > > How people feel about generally not having += on long strings in > stdlib (since the refcount = 1 thing is a hack)? I agree that += should not be used as an optimization (on strings) in the stdlib code. The optimization is there so that uncareful code does not degenerate, but deliberately relying on it is a bit devilish. (optimisare diabolicum :-)) Regards Antoine. From brett at python.org Tue Feb 12 22:16:55 2013 From: brett at python.org (Brett Cannon) Date: Tue, 12 Feb 2013 16:16:55 -0500 Subject: [Python-Dev] Usage of += on strings in loops in stdlib In-Reply-To: <20130212220642.71f03220@pitrou.net> References: <20130212220642.71f03220@pitrou.net> Message-ID: On Tue, Feb 12, 2013 at 4:06 PM, Antoine Pitrou wrote: > > Hi ! > > On Tue, 12 Feb 2013 23:03:04 +0200 > Maciej Fijalkowski wrote: > > > > We recently encountered a performance issue in stdlib for pypy. It > > turned out that someone commited a performance "fix" that uses += for > > strings instead of "".join() that was there before. > > > > Now this hurts pypy (we can mitigate it to some degree though) and > > possible Jython and IronPython too. > > > > How people feel about generally not having += on long strings in > > stdlib (since the refcount = 1 thing is a hack)? > > I agree that += should not be used as an optimization (on strings) in > the stdlib code. The optimization is there so that uncareful code does > not degenerate, but deliberately relying on it is a bit devilish. > (optimisare diabolicum :-)) > Ditto from me. If you're going so far as to want to optimize Python code then you probably are going to care enough to accelerate it in C, in which case you can leave the Python code idiomatic. -------------- next part -------------- An HTML attachment was scrubbed... URL: From fijall at gmail.com Tue Feb 12 22:25:49 2013 From: fijall at gmail.com (Maciej Fijalkowski) Date: Tue, 12 Feb 2013 23:25:49 +0200 Subject: [Python-Dev] Usage of += on strings in loops in stdlib In-Reply-To: References: <20130212220642.71f03220@pitrou.net> Message-ID: On Tue, Feb 12, 2013 at 11:16 PM, Brett Cannon wrote: > > > > On Tue, Feb 12, 2013 at 4:06 PM, Antoine Pitrou wrote: >> >> >> Hi ! >> >> On Tue, 12 Feb 2013 23:03:04 +0200 >> Maciej Fijalkowski wrote: >> > >> > We recently encountered a performance issue in stdlib for pypy. It >> > turned out that someone commited a performance "fix" that uses += for >> > strings instead of "".join() that was there before. >> > >> > Now this hurts pypy (we can mitigate it to some degree though) and >> > possible Jython and IronPython too. >> > >> > How people feel about generally not having += on long strings in >> > stdlib (since the refcount = 1 thing is a hack)? >> >> I agree that += should not be used as an optimization (on strings) in >> the stdlib code. The optimization is there so that uncareful code does >> not degenerate, but deliberately relying on it is a bit devilish. >> (optimisare diabolicum :-)) > > > Ditto from me. If you're going so far as to want to optimize Python code > then you probably are going to care enough to accelerate it in C, in which > case you can leave the Python code idiomatic. > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/fijall%40gmail.com > I should actually reference the original CPython issue http://bugs.python.org/issue1285086 From fwierzbicki at gmail.com Tue Feb 12 22:32:50 2013 From: fwierzbicki at gmail.com (fwierzbicki at gmail.com) Date: Tue, 12 Feb 2013 13:32:50 -0800 Subject: [Python-Dev] Usage of += on strings in loops in stdlib In-Reply-To: References: Message-ID: On Tue, Feb 12, 2013 at 1:03 PM, Maciej Fijalkowski wrote: > Hi > > We recently encountered a performance issue in stdlib for pypy. It > turned out that someone commited a performance "fix" that uses += for > strings instead of "".join() that was there before. > > Now this hurts pypy (we can mitigate it to some degree though) and > possible Jython and IronPython too. Just to confirm Jython does not have optimizations for += String and will do much better with the idiomatic "".join(). -Frank From solipsis at pitrou.net Tue Feb 12 22:32:49 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 12 Feb 2013 22:32:49 +0100 Subject: [Python-Dev] Usage of += on strings in loops in stdlib References: Message-ID: <20130212223249.1457ff6b@pitrou.net> On Tue, 12 Feb 2013 13:32:50 -0800 "fwierzbicki at gmail.com" wrote: > On Tue, Feb 12, 2013 at 1:03 PM, Maciej Fijalkowski wrote: > > Hi > > > > We recently encountered a performance issue in stdlib for pypy. It > > turned out that someone commited a performance "fix" that uses += for > > strings instead of "".join() that was there before. > > > > Now this hurts pypy (we can mitigate it to some degree though) and > > possible Jython and IronPython too. > Just to confirm Jython does not have optimizations for += String and > will do much better with the idiomatic "".join(). For the record, io.StringIO should be quite fast in 3.3. (except for the method call overhead that Guido is complaining about :-)) Regards Antoine. From ned at nedbatchelder.com Tue Feb 12 22:40:38 2013 From: ned at nedbatchelder.com (Ned Batchelder) Date: Tue, 12 Feb 2013 16:40:38 -0500 Subject: [Python-Dev] Usage of += on strings in loops in stdlib In-Reply-To: References: <20130212220642.71f03220@pitrou.net> Message-ID: <511AB6D6.60805@nedbatchelder.com> On 2/12/2013 4:16 PM, Brett Cannon wrote: > > > > On Tue, Feb 12, 2013 at 4:06 PM, Antoine Pitrou > wrote: > > > Hi ! > > On Tue, 12 Feb 2013 23:03:04 +0200 > Maciej Fijalkowski > wrote: > > > > We recently encountered a performance issue in stdlib for pypy. It > > turned out that someone commited a performance "fix" that uses > += for > > strings instead of "".join() that was there before. > > > > Now this hurts pypy (we can mitigate it to some degree though) and > > possible Jython and IronPython too. > > > > How people feel about generally not having += on long strings in > > stdlib (since the refcount = 1 thing is a hack)? > > I agree that += should not be used as an optimization (on strings) in > the stdlib code. The optimization is there so that uncareful code does > not degenerate, but deliberately relying on it is a bit devilish. > (optimisare diabolicum :-)) > > > Ditto from me. If you're going so far as to want to optimize Python > code then you probably are going to care enough to accelerate it in C, > in which case you can leave the Python code idiomatic. But the only reason "".join() is a Python idiom in the first place is because it was "the fast way" to do what everyone initially coded as "s += ...". Just because we all learned a long time ago that joining was the fast way to build a string doesn't mean that "".join() is the clean idiomatic way to do it. --Ned. > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/ned%40nedbatchelder.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From rdmurray at bitdance.com Tue Feb 12 22:46:34 2013 From: rdmurray at bitdance.com (R. David Murray) Date: Tue, 12 Feb 2013 16:46:34 -0500 Subject: [Python-Dev] Usage of += on strings in loops in stdlib In-Reply-To: <511AB6D6.60805@nedbatchelder.com> References: <20130212220642.71f03220@pitrou.net> <511AB6D6.60805@nedbatchelder.com> Message-ID: <20130212214635.2CAA3250030@webabinitio.net> On Tue, 12 Feb 2013 16:40:38 -0500, Ned Batchelder wrote: > On 2/12/2013 4:16 PM, Brett Cannon wrote: > > On Tue, Feb 12, 2013 at 4:06 PM, Antoine Pitrou > > wrote: > > On Tue, 12 Feb 2013 23:03:04 +0200 > > Maciej Fijalkowski > wrote: > > > > > > We recently encountered a performance issue in stdlib for pypy. It > > > turned out that someone commited a performance "fix" that uses > > += for > > > strings instead of "".join() that was there before. > > > > > > Now this hurts pypy (we can mitigate it to some degree though) and > > > possible Jython and IronPython too. > > > > > > How people feel about generally not having += on long strings in > > > stdlib (since the refcount = 1 thing is a hack)? > > > > I agree that += should not be used as an optimization (on strings) in > > the stdlib code. The optimization is there so that uncareful code does > > not degenerate, but deliberately relying on it is a bit devilish. > > (optimisare diabolicum :-)) > > > > Ditto from me. If you're going so far as to want to optimize Python > > code then you probably are going to care enough to accelerate it in C, > > in which case you can leave the Python code idiomatic. > > But the only reason "".join() is a Python idiom in the first place is > because it was "the fast way" to do what everyone initially coded as "s > += ...". Just because we all learned a long time ago that joining was > the fast way to build a string doesn't mean that "".join() is the clean > idiomatic way to do it. If 'idiomatic' (a terrible term) means "the standard way in this language", which is how it is employed in the programming community, then yes, "".join() is the idiomatic way to write that *in Python*, and thus is cleaner code *in Python*. --David From solipsis at pitrou.net Tue Feb 12 22:44:04 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 12 Feb 2013 22:44:04 +0100 Subject: [Python-Dev] Usage of += on strings in loops in stdlib References: <20130212220642.71f03220@pitrou.net> <511AB6D6.60805@nedbatchelder.com> Message-ID: <20130212224404.4fa839fc@pitrou.net> On Tue, 12 Feb 2013 16:40:38 -0500 Ned Batchelder wrote: > > But the only reason "".join() is a Python idiom in the first place is > because it was "the fast way" to do what everyone initially coded as "s > += ...". Just because we all learned a long time ago that joining was > the fast way to build a string doesn't mean that "".join() is the clean > idiomatic way to do it. It's idiomatic because strings are immutable (by design, not because of an optimization detail) and therefore concatenation *has* to imply building a new string from scratch. Regards Antoine. From python-dev at masklinn.net Tue Feb 12 23:33:56 2013 From: python-dev at masklinn.net (Xavier Morel) Date: Tue, 12 Feb 2013 23:33:56 +0100 Subject: [Python-Dev] Usage of += on strings in loops in stdlib In-Reply-To: <511AB6D6.60805@nedbatchelder.com> References: <20130212220642.71f03220@pitrou.net> <511AB6D6.60805@nedbatchelder.com> Message-ID: <0803CA4F-4ADB-4C59-8A61-8FFD0CAF0CF2@masklinn.net> On 2013-02-12, at 22:40 , Ned Batchelder wrote: > But the only reason "".join() is a Python idiom in the first place is because it was "the fast way" to do what everyone initially coded as "s += ...". Just because we all learned a long time ago that joining was the fast way to build a string doesn't mean that "".join() is the clean idiomatic way to do it. Well no, str.join is the idiomatic way to do it because it is: > idiomatic |?id???matik| > adjective > 1 using, containing, or denoting expressions that are natural to a native speaker or would you argue that the natural way for weathered python developers to concatenate string is to *not* use str.join? Of course usually idioms have original reasons for being, reasons which are sometimes long gone (not unlike religious mandates or prohibitions). For Python, ignoring the refcounting hack (which is not only cpython specific but *current* cpython specific *and* doesn't apply to all cases) that reason still exist: python's strings are formally immutable bytestrings, and repeated concatenation of immutable bytestrings is quadratic. Thus str.join is idiomatic, and although it's possible (if difficult) to change the idiom straight string concatenation would make a terrible new idiom as it will behave either unreliably (current CPython) or simply terribly (every other Python implementation). No? From python at mrabarnett.plus.com Wed Feb 13 00:20:01 2013 From: python at mrabarnett.plus.com (MRAB) Date: Tue, 12 Feb 2013 23:20:01 +0000 Subject: [Python-Dev] Usage of += on strings in loops in stdlib In-Reply-To: <20130212224404.4fa839fc@pitrou.net> References: <20130212220642.71f03220@pitrou.net> <511AB6D6.60805@nedbatchelder.com> <20130212224404.4fa839fc@pitrou.net> Message-ID: <511ACE21.2050900@mrabarnett.plus.com> On 2013-02-12 21:44, Antoine Pitrou wrote: > On Tue, 12 Feb 2013 16:40:38 -0500 > Ned Batchelder wrote: >> >> But the only reason "".join() is a Python idiom in the first place is >> because it was "the fast way" to do what everyone initially coded as "s >> += ...". Just because we all learned a long time ago that joining was >> the fast way to build a string doesn't mean that "".join() is the clean >> idiomatic way to do it. > > It's idiomatic because strings are immutable (by design, not because of > an optimization detail) and therefore concatenation *has* to imply > building a new string from scratch. > Tuples are much like immutable lists; sets were added, and then frozensets; should we be adding mutable strings too (a bit like C#'s StringBuilder)? (Just wondering...) From christian at python.org Wed Feb 13 00:28:15 2013 From: christian at python.org (Christian Heimes) Date: Wed, 13 Feb 2013 00:28:15 +0100 Subject: [Python-Dev] Usage of += on strings in loops in stdlib In-Reply-To: <20130212223249.1457ff6b@pitrou.net> References: <20130212223249.1457ff6b@pitrou.net> Message-ID: <511AD00F.8060103@python.org> Am 12.02.2013 22:32, schrieb Antoine Pitrou: > For the record, io.StringIO should be quite fast in 3.3. > (except for the method call overhead that Guido is complaining > about :-)) AFAIK it's not the actual *call* of the method that is slow, but rather attribute lookup and creation of bound method objects. If speed is of the essence, code can cache the method object locally: strio = io.StringIO() write = strio.write for element in elements: write(element) result = strio.getvalue() Christian From fijall at gmail.com Wed Feb 13 00:30:10 2013 From: fijall at gmail.com (Maciej Fijalkowski) Date: Wed, 13 Feb 2013 01:30:10 +0200 Subject: [Python-Dev] Usage of += on strings in loops in stdlib In-Reply-To: <511AD00F.8060103@python.org> References: <20130212223249.1457ff6b@pitrou.net> <511AD00F.8060103@python.org> Message-ID: On Wed, Feb 13, 2013 at 1:28 AM, Christian Heimes wrote: > Am 12.02.2013 22:32, schrieb Antoine Pitrou: >> For the record, io.StringIO should be quite fast in 3.3. >> (except for the method call overhead that Guido is complaining >> about :-)) > > AFAIK it's not the actual *call* of the method that is slow, but rather > attribute lookup and creation of bound method objects. If speed is of > the essence, code can cache the method object locally: > > strio = io.StringIO() > write = strio.write > for element in elements: > write(element) > result = strio.getvalue() And this is a great example of muddying code in stdlib for the sake of speeding up CPython. Cheers, fijal From fijall at gmail.com Wed Feb 13 00:30:28 2013 From: fijall at gmail.com (Maciej Fijalkowski) Date: Wed, 13 Feb 2013 01:30:28 +0200 Subject: [Python-Dev] Usage of += on strings in loops in stdlib In-Reply-To: <511ACE21.2050900@mrabarnett.plus.com> References: <20130212220642.71f03220@pitrou.net> <511AB6D6.60805@nedbatchelder.com> <20130212224404.4fa839fc@pitrou.net> <511ACE21.2050900@mrabarnett.plus.com> Message-ID: On Wed, Feb 13, 2013 at 1:20 AM, MRAB wrote: > On 2013-02-12 21:44, Antoine Pitrou wrote: >> >> On Tue, 12 Feb 2013 16:40:38 -0500 >> Ned Batchelder wrote: >>> >>> >>> But the only reason "".join() is a Python idiom in the first place is >>> because it was "the fast way" to do what everyone initially coded as "s >>> += ...". Just because we all learned a long time ago that joining was >>> the fast way to build a string doesn't mean that "".join() is the clean >>> idiomatic way to do it. >> >> >> It's idiomatic because strings are immutable (by design, not because of >> an optimization detail) and therefore concatenation *has* to imply >> building a new string from scratch. >> > Tuples are much like immutable lists; sets were added, and then frozensets; > should we be adding mutable strings too (a bit like C#'s StringBuilder)? > (Just wondering...) Isn't bytearray what you're looking for? From ncoghlan at gmail.com Wed Feb 13 00:39:23 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 13 Feb 2013 09:39:23 +1000 Subject: [Python-Dev] Usage of += on strings in loops in stdlib In-Reply-To: References: Message-ID: On 13 Feb 2013 07:08, "Maciej Fijalkowski" wrote: > > Hi > > We recently encountered a performance issue in stdlib for pypy. It > turned out that someone commited a performance "fix" that uses += for > strings instead of "".join() that was there before. > > Now this hurts pypy (we can mitigate it to some degree though) and > possible Jython and IronPython too. > > How people feel about generally not having += on long strings in > stdlib (since the refcount = 1 thing is a hack)? > > What about other performance improvements in stdlib that are > problematic for pypy or others? > > Personally I would like cleaner code in stdlib vs speeding up CPython. For the specific case of "Don't rely on the fragile refcounting hack in CPython's string concatenation" I strongly agree. However, as a general principle, I can't agree until speed.python.org is a going concern and we can get a reasonable overview of any resulting performance implications. Regards, Nick. > Typically that also helps pypy so I'm not unbiased. > > Cheers, > fijal > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From tismer at stackless.com Wed Feb 13 00:53:05 2013 From: tismer at stackless.com (Christian Tismer) Date: Wed, 13 Feb 2013 00:53:05 +0100 Subject: [Python-Dev] efficient string concatenation (yep, from 2004) Message-ID: <511AD5E1.1060209@stackless.com> Hi friends, _efficient string concatenation_ has been a topic in 2004. Armin Rigo proposed a patch with the name of the subject, more precisely: /[Patches] [ python-Patches-980695 ] efficient string concatenation// //on sourceforge.net, on 2004-06-28.// / This patch was finally added to Python 2.4 on 2004-11-30. Some people might remember the larger discussion if such a patch should be accepted at all, because it changes the programming style for many of us from "don't do that, stupid" to "well, you may do it in CPython", which has quite some impact on other implementations (is it fast on Jython, now?). It changed for instance my programming and teaching style a lot, of course! But I think nobody but people heavily involved in PyPy expected this: Now, more than eight years after that patch appeared and made it into 2.4, PyPy (!) still does _not_ have it! Obviously I was mislead by other optimizations, and the fact that this patch was from a/the major author of PyPy who invented the initial patch for CPython. That this would be in PyPy as well sooner or later was without question for me. Wrong... ;-) Yes, I agree that for PyPy it is much harder to implement without the refcounting trick, and probably even more difficult in case of the JIT. But nevertheless, I tried to find any reference to this missing crucial optimization, with no success after an hour (*). And I guess many other people are stepping in the same trap. So I can imagine that PyPy looses some of its speed in many programs, because Armin's great hack did not make it into PyPy, and this is not loudly declared somewhere. I believe the efficiency of string concatenation is something that people assume by default and add it to the vague CPython compatibility claim, if not explicitly told otherwise. ---- Some silly proof, using python 2.7.3 vs PyPy 1.9: > $ cat strconc.py > #!env python > > from timeit import default_timer as timer > > tim = timer() > > s = '' > for i in xrange(100000): > s += 'X' > > tim = timer() - tim > > print 'time for {} concats = {:0.3f}'.format(len(s), tim) > $ python strconc.py > time for 100000 concats = 0.028 > $ pypy strconc.py > time for 100000 concats = 0.804 Something is needed - a patch for PyPy or for the documentation I guess. This is not just some unoptimized function in some module, but it is used all over the place and became a very common pattern since introduced. /How ironic that a foreseen problem occurs _now_, and _there_ :-)// / cheers -- chris (*) http://pypy.readthedocs.org/en/latest/cpython_differences.html http://pypy.org/compat.html http://pypy.org/performance.html -- Christian Tismer :^) Software Consulting : Have a break! Take a ride on Python's Karl-Liebknecht-Str. 121 : *Starship* http://starship.python.net/ 14482 Potsdam : PGP key -> http://pgp.uni-mainz.de phone +49 173 24 18 776 fax +49 (30) 700143-0023 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From alexandre at peadrop.com Wed Feb 13 02:09:01 2013 From: alexandre at peadrop.com (Alexandre Vassalotti) Date: Tue, 12 Feb 2013 17:09:01 -0800 Subject: [Python-Dev] Usage of += on strings in loops in stdlib In-Reply-To: <20130212224404.4fa839fc@pitrou.net> References: <20130212220642.71f03220@pitrou.net> <511AB6D6.60805@nedbatchelder.com> <20130212224404.4fa839fc@pitrou.net> Message-ID: On Tue, Feb 12, 2013 at 1:44 PM, Antoine Pitrou wrote: > It's idiomatic because strings are immutable (by design, not because of > an optimization detail) and therefore concatenation *has* to imply > building a new string from scratch. > Not necessarily. It is totally possible to implement strings such they are immutable and concatenation takes O(1): ropes are the canonical example of this. -------------- next part -------------- An HTML attachment was scrubbed... URL: From tismer at stackless.com Wed Feb 13 02:16:22 2013 From: tismer at stackless.com (Christian Tismer) Date: Wed, 13 Feb 2013 02:16:22 +0100 Subject: [Python-Dev] Usage of += on strings in loops in stdlib In-Reply-To: References: Message-ID: <511AE966.4020101@stackless.com> On 12.02.13 22:03, Maciej Fijalkowski wrote: > Hi > > We recently encountered a performance issue in stdlib for pypy. It > turned out that someone commited a performance "fix" that uses += for > strings instead of "".join() that was there before. > > Now this hurts pypy (we can mitigate it to some degree though) and > possible Jython and IronPython too. > > How people feel about generally not having += on long strings in > stdlib (since the refcount = 1 thing is a hack)? > > What about other performance improvements in stdlib that are > problematic for pypy or others? > > Personally I would like cleaner code in stdlib vs speeding up CPython. > Typically that also helps pypy so I'm not unbiased. > > Cheers, > fijal > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/tismer%40stackless.com Howdy. Funny coincidence that this issue came up an hour after I asked about string_concat optimization absence on the pypy channel. I did not read email while writing the "efficient string concatenation" re-iteration._ _ Maybe we should use the time machine, go backwards and undo the patch, although it still makes a lot of sense and is fastest, opcode-wise, at least on CPython. Which will not matter so much for PyPy of course because _that_ goes away. Alas, the damage to the mindsets already has happened, and the cure will probably be as hard as the eviction of the print statement, after all. But since I'm a complete Python 3.3 convert (with consequent changes to my projects which was not so trivial), I think to also start publicly saying that "s += t" is a pattern that should not be used in the Gigabyte domain, from 2013. Actually a tad, because it contradicted normal programming patterns in an appealing way. Way too sexy... But let's toss it. Keep the past eight years in good memories as an exceptional period of liberal abuse. Maybe we should add it as an addition to the "Zen of Python": There are obviously good things, but "obvious" is the finest liar. -- Christian Tismer :^) Software Consulting : Have a break! Take a ride on Python's Karl-Liebknecht-Str. 121 : *Starship* http://starship.python.net/ 14482 Potsdam : PGP key -> http://pgp.uni-mainz.de phone +49 173 24 18 776 fax +49 (30) 700143-0023 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From tismer at stackless.com Wed Feb 13 02:25:09 2013 From: tismer at stackless.com (Christian Tismer) Date: Wed, 13 Feb 2013 02:25:09 +0100 Subject: [Python-Dev] Usage of += on strings in loops in stdlib In-Reply-To: References: <20130212220642.71f03220@pitrou.net> <511AB6D6.60805@nedbatchelder.com> <20130212224404.4fa839fc@pitrou.net> Message-ID: <511AEB75.6090808@stackless.com> On 13.02.13 02:09, Alexandre Vassalotti wrote: > On Tue, Feb 12, 2013 at 1:44 PM, Antoine Pitrou > wrote: > > It's idiomatic because strings are immutable (by design, not > because of > an optimization detail) and therefore concatenation *has* to imply > building a new string from scratch. > > > Not necessarily. It is totally possible to implement strings such they > are immutable and concatenation takes O(1): ropes are the canonical > example of this. Ropes have been implemented by Carl-Friedrich Bolz in 2007 as I remember. No idea what the impact was, if any at all. Would ropes be an answer (and a simple way to cope with string mutation patterns) as an alternative implementation, and therefore still justify the usage of that pattern? -- Christian Tismer :^) Software Consulting : Have a break! Take a ride on Python's Karl-Liebknecht-Str. 121 : *Starship* http://starship.python.net/ 14482 Potsdam : PGP key -> http://pgp.uni-mainz.de phone +49 173 24 18 776 fax +49 (30) 700143-0023 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From tjreedy at udel.edu Wed Feb 13 02:51:46 2013 From: tjreedy at udel.edu (Terry Reedy) Date: Tue, 12 Feb 2013 20:51:46 -0500 Subject: [Python-Dev] Usage of += on strings in loops in stdlib In-Reply-To: <511ACE21.2050900@mrabarnett.plus.com> References: <20130212220642.71f03220@pitrou.net> <511AB6D6.60805@nedbatchelder.com> <20130212224404.4fa839fc@pitrou.net> <511ACE21.2050900@mrabarnett.plus.com> Message-ID: On 2/12/2013 6:20 PM, MRAB wrote: > Tuples are much like immutable lists; sets were added, and then frozensets; > should we be adding mutable strings too (a bit like C#'s StringBuilder)? > (Just wondering...) StringIO is effectively a mutable string with a file interface. sio.write('abc') is the equivalent of lis.extend(['a', 'b', 'c']). -- Terry Jan Reedy From tjreedy at udel.edu Wed Feb 13 02:58:53 2013 From: tjreedy at udel.edu (Terry Reedy) Date: Tue, 12 Feb 2013 20:58:53 -0500 Subject: [Python-Dev] Usage of += on strings in loops in stdlib In-Reply-To: References: Message-ID: On 2/12/2013 4:03 PM, Maciej Fijalkowski wrote: > Hi > > We recently encountered a performance issue in stdlib for pypy. It > turned out that someone commited a performance "fix" that uses += for > strings instead of "".join() that was there before. > > Now this hurts pypy (we can mitigate it to some degree though) and > possible Jython and IronPython too. > > How people feel about generally not having += on long strings in > stdlib (since the refcount = 1 thing is a hack)? > > What about other performance improvements in stdlib that are > problematic for pypy or others? > > Personally I would like cleaner code in stdlib vs speeding up CPython. > Typically that also helps pypy so I'm not unbiased. I agree. sum() refuses to sum strings specifically to encourage .join(). >>> sum(('x', 'b'), '') Traceback (most recent call last): File "", line 1, in sum(('x', 'b'), '') TypeError: sum() can't sum strings [use ''.join(seq) instead] The doc entry for sum says the same thing. -- Terry Jan Reedy From solipsis at pitrou.net Wed Feb 13 08:14:33 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 13 Feb 2013 08:14:33 +0100 Subject: [Python-Dev] Usage of += on strings in loops in stdlib In-Reply-To: <511AD00F.8060103@python.org> References: <20130212223249.1457ff6b@pitrou.net> <511AD00F.8060103@python.org> Message-ID: <20130213081433.7e8f0c24@pitrou.net> On Wed, 13 Feb 2013 00:28:15 +0100 Christian Heimes wrote: > Am 12.02.2013 22:32, schrieb Antoine Pitrou: > > For the record, io.StringIO should be quite fast in 3.3. > > (except for the method call overhead that Guido is complaining > > about :-)) > > AFAIK it's not the actual *call* of the method that is slow, but rather > attribute lookup and creation of bound method objects. Take a look at http://bugs.python.org/issue17170 From solipsis at pitrou.net Wed Feb 13 08:16:21 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 13 Feb 2013 08:16:21 +0100 Subject: [Python-Dev] Usage of += on strings in loops in stdlib References: Message-ID: <20130213081621.4c4c382e@pitrou.net> On Wed, 13 Feb 2013 09:39:23 +1000 Nick Coghlan wrote: > On 13 Feb 2013 07:08, "Maciej Fijalkowski" wrote: > > > > Hi > > > > We recently encountered a performance issue in stdlib for pypy. It > > turned out that someone commited a performance "fix" that uses += for > > strings instead of "".join() that was there before. > > > > Now this hurts pypy (we can mitigate it to some degree though) and > > possible Jython and IronPython too. > > > > How people feel about generally not having += on long strings in > > stdlib (since the refcount = 1 thing is a hack)? > > > > What about other performance improvements in stdlib that are > > problematic for pypy or others? > > > > Personally I would like cleaner code in stdlib vs speeding up CPython. > > For the specific case of "Don't rely on the fragile refcounting hack in > CPython's string concatenation" I strongly agree. However, as a general > principle, I can't agree until speed.python.org is a going concern and we > can get a reasonable overview of any resulting performance implications. Anybody can run the benchmark suite for himself, speed.p.o is (fortunately) not a roadblock: http://bugs.python.org/issue17170 Regards Antoine. From solipsis at pitrou.net Wed Feb 13 08:24:22 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 13 Feb 2013 08:24:22 +0100 Subject: [Python-Dev] Usage of += on strings in loops in stdlib References: <20130213081621.4c4c382e@pitrou.net> Message-ID: <20130213082422.74673bd1@pitrou.net> On Wed, 13 Feb 2013 08:16:21 +0100 Antoine Pitrou wrote: > On Wed, 13 Feb 2013 09:39:23 +1000 > Nick Coghlan wrote: > > On 13 Feb 2013 07:08, "Maciej Fijalkowski" wrote: > > > > > > Hi > > > > > > We recently encountered a performance issue in stdlib for pypy. It > > > turned out that someone commited a performance "fix" that uses += for > > > strings instead of "".join() that was there before. > > > > > > Now this hurts pypy (we can mitigate it to some degree though) and > > > possible Jython and IronPython too. > > > > > > How people feel about generally not having += on long strings in > > > stdlib (since the refcount = 1 thing is a hack)? > > > > > > What about other performance improvements in stdlib that are > > > problematic for pypy or others? > > > > > > Personally I would like cleaner code in stdlib vs speeding up CPython. > > > > For the specific case of "Don't rely on the fragile refcounting hack in > > CPython's string concatenation" I strongly agree. However, as a general > > principle, I can't agree until speed.python.org is a going concern and we > > can get a reasonable overview of any resulting performance implications. > > Anybody can run the benchmark suite for himself, speed.p.o is > (fortunately) not a roadblock: > http://bugs.python.org/issue17170 And I meant to paste the repo URL actually: http://hg.python.org/benchmarks/ Regards Antoine. From fijall at gmail.com Wed Feb 13 08:35:28 2013 From: fijall at gmail.com (Maciej Fijalkowski) Date: Wed, 13 Feb 2013 09:35:28 +0200 Subject: [Python-Dev] [pypy-dev] efficient string concatenation (yep, from 2004) In-Reply-To: <511AD5E1.1060209@stackless.com> References: <511AD5E1.1060209@stackless.com> Message-ID: Hi Christian. We have it, just not enabled by default. --objspace-with-strbuf I think On Wed, Feb 13, 2013 at 1:53 AM, Christian Tismer wrote: > Hi friends, > > efficient string concatenation has been a topic in 2004. > Armin Rigo proposed a patch with the name of the subject, > more precisely: > > [Patches] [ python-Patches-980695 ] efficient string concatenation > on sourceforge.net, on 2004-06-28. > > This patch was finally added to Python 2.4 on 2004-11-30. > > Some people might remember the larger discussion if such a patch should be > accepted at all, because it changes the programming style for many of us > from "don't do that, stupid" to "well, you may do it in CPython", which has > quite > some impact on other implementations (is it fast on Jython, now?). > > It changed for instance my programming and teaching style a lot, of course! > > But I think nobody but people heavily involved in PyPy expected this: > > Now, more than eight years after that patch appeared and made it into 2.4, > PyPy (!) still does _not_ have it! > > Obviously I was mislead by other optimizations, and the fact that > this patch was from a/the major author of PyPy who invented the initial > patch for CPython. That this would be in PyPy as well sooner or later was > without question for me. Wrong... ;-) > > Yes, I agree that for PyPy it is much harder to implement without the > refcounting trick, and probably even more difficult in case of the JIT. > > But nevertheless, I tried to find any reference to this missing crucial > optimization, > with no success after an hour (*). > > And I guess many other people are stepping in the same trap. > > So I can imagine that PyPy looses some of its speed in many programs, > because > Armin's great hack did not make it into PyPy, and this is not loudly > declared > somewhere. I believe the efficiency of string concatenation is something > that people assume by default and add it to the vague CPython compatibility > claim, if not explicitly told otherwise. > > ---- > > Some silly proof, using python 2.7.3 vs PyPy 1.9: > > $ cat strconc.py > #!env python > > from timeit import default_timer as timer > > tim = timer() > > s = '' > for i in xrange(100000): > s += 'X' > > tim = timer() - tim > > print 'time for {} concats = {:0.3f}'.format(len(s), tim) > > > $ python strconc.py > time for 100000 concats = 0.028 > $ pypy strconc.py > time for 100000 concats = 0.804 > > > Something is needed - a patch for PyPy or for the documentation I guess. > > This is not just some unoptimized function in some module, but it is used > all over the place and became a very common pattern since introduced. > > How ironic that a foreseen problem occurs _now_, and _there_ :-) > > cheers -- chris > > > (*) > http://pypy.readthedocs.org/en/latest/cpython_differences.html > http://pypy.org/compat.html > http://pypy.org/performance.html > > -- > Christian Tismer :^) > Software Consulting : Have a break! Take a ride on Python's > Karl-Liebknecht-Str. 121 : *Starship* http://starship.python.net/ > 14482 Potsdam : PGP key -> http://pgp.uni-mainz.de > phone +49 173 24 18 776 fax +49 (30) 700143-0023 > PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 > whom do you want to sponsor today? http://www.stackless.com/ > > > _______________________________________________ > pypy-dev mailing list > pypy-dev at python.org > http://mail.python.org/mailman/listinfo/pypy-dev > From regebro at gmail.com Wed Feb 13 08:42:17 2013 From: regebro at gmail.com (Lennart Regebro) Date: Wed, 13 Feb 2013 08:42:17 +0100 Subject: [Python-Dev] efficient string concatenation (yep, from 2004) In-Reply-To: <511AD5E1.1060209@stackless.com> References: <511AD5E1.1060209@stackless.com> Message-ID: > Something is needed - a patch for PyPy or for the documentation I guess. Not arguing that it wouldn't be good, but I disagree that it is needed. This is only an issue when you, as in your proof, have a loop that does concatenation. This is usually when looping over a list of strings that should be concatenated together. Doing so in a loop with concatenation may be the natural way for people new to Python, but the "natural" way to do it in Python is with a ''.join() call. This: s = ''.join(('X' for x in xrange(x))) Is more than twice as fast in Python 2.7 than your example. It is in fact also slower in PyPy 1.9 than Python 2.7, but only with a factor of two: Python 2.7: time for 10000000 concats = 0.887 Pypy 1.9: time for 10000000 concats = 1.600 (And of course s = 'X'* x takes only a bout a hundredth of the time, but that's cheating. ;-) //Lennart From alexandre at peadrop.com Wed Feb 13 08:42:59 2013 From: alexandre at peadrop.com (Alexandre Vassalotti) Date: Tue, 12 Feb 2013 23:42:59 -0800 Subject: [Python-Dev] Usage of += on strings in loops in stdlib In-Reply-To: <511AEB75.6090808@stackless.com> References: <20130212220642.71f03220@pitrou.net> <511AB6D6.60805@nedbatchelder.com> <20130212224404.4fa839fc@pitrou.net> <511AEB75.6090808@stackless.com> Message-ID: On Tue, Feb 12, 2013 at 5:25 PM, Christian Tismer wrote: > Would ropes be an answer (and a simple way to cope with string mutation > patterns) as an alternative implementation, and therefore still justify > the usage of that pattern? > I don't think so. Ropes are really useful when you work with gigabytes of data, but unfortunately they don't make good general-purpose strings. Monolithic arrays are much more efficient and simple for the typical use-cases we have in Python. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Wed Feb 13 08:52:56 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 13 Feb 2013 17:52:56 +1000 Subject: [Python-Dev] Usage of += on strings in loops in stdlib In-Reply-To: References: <20130212220642.71f03220@pitrou.net> <511AB6D6.60805@nedbatchelder.com> <20130212224404.4fa839fc@pitrou.net> <511AEB75.6090808@stackless.com> Message-ID: On Wed, Feb 13, 2013 at 5:42 PM, Alexandre Vassalotti wrote: > On Tue, Feb 12, 2013 at 5:25 PM, Christian Tismer > wrote: >> >> Would ropes be an answer (and a simple way to cope with string mutation >> patterns) as an alternative implementation, and therefore still justify >> the usage of that pattern? > > > I don't think so. Ropes are really useful when you work with gigabytes of > data, but unfortunately they don't make good general-purpose strings. > Monolithic arrays are much more efficient and simple for the typical > use-cases we have in Python. If I recall correctly, io.StringIO and io.BytesIO have been updated to use ropes internally in 3.3. Writing to one of those and then calling getvalue() at the end is the main alternative to the list+join trick (when concatenating many small strings, the memory saving relative to a list can be notable since strings have a fairly large per-instance overhead). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From victor.stinner at gmail.com Wed Feb 13 09:02:07 2013 From: victor.stinner at gmail.com (Victor Stinner) Date: Wed, 13 Feb 2013 09:02:07 +0100 Subject: [Python-Dev] Usage of += on strings in loops in stdlib In-Reply-To: References: Message-ID: I added a _PyUnicodeWriter internal API to optimize str%args and str.format(args). It uses a buffer which is overallocated, so it's basically like CPython str += str optimization. I still don't know how efficient it is on Windows, since realloc() is slow on Windows (at least on old Windows versions). We should add an official and public API to concatenate strings. I know that PyPy has already its own API. Example: writer = UnicodeWriter() for item in data: writer += item # i guess that it's faster than writer.append(item) return str(writer) # or writer.getvalue() ? I don't care of the exact implementation of UnicodeWriter, it just have to be as fast or faster than ''.join(data). I don't remember if _PyUnicodeWriter is faster than StringIO or slower. I created an issue for that: http://bugs.python.org/issue15612 Victor 2013/2/12 Maciej Fijalkowski : > Hi > > We recently encountered a performance issue in stdlib for pypy. It > turned out that someone commited a performance "fix" that uses += for > strings instead of "".join() that was there before. > > Now this hurts pypy (we can mitigate it to some degree though) and > possible Jython and IronPython too. > > How people feel about generally not having += on long strings in > stdlib (since the refcount = 1 thing is a hack)? > > What about other performance improvements in stdlib that are > problematic for pypy or others? > > Personally I would like cleaner code in stdlib vs speeding up CPython. > Typically that also helps pypy so I'm not unbiased. > > Cheers, > fijal > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/victor.stinner%40gmail.com From fijall at gmail.com Wed Feb 13 09:12:24 2013 From: fijall at gmail.com (Maciej Fijalkowski) Date: Wed, 13 Feb 2013 10:12:24 +0200 Subject: [Python-Dev] Usage of += on strings in loops in stdlib In-Reply-To: References: Message-ID: On Wed, Feb 13, 2013 at 10:02 AM, Victor Stinner wrote: > I added a _PyUnicodeWriter internal API to optimize str%args and > str.format(args). It uses a buffer which is overallocated, so it's > basically like CPython str += str optimization. I still don't know how > efficient it is on Windows, since realloc() is slow on Windows (at > least on old Windows versions). > > We should add an official and public API to concatenate strings. I > know that PyPy has already its own API. Example: > > writer = UnicodeWriter() > for item in data: > writer += item # i guess that it's faster than writer.append(item) > return str(writer) # or writer.getvalue() ? > > I don't care of the exact implementation of UnicodeWriter, it just > have to be as fast or faster than ''.join(data). > > I don't remember if _PyUnicodeWriter is faster than StringIO or > slower. I created an issue for that: > http://bugs.python.org/issue15612 > > Victor it's in __pypy__.builders (StringBuilder and UnicodeBuilder). The API does not really matter, as long as there is a way to preallocate certain size (which I don't think there is in StringIO for example). bytearray comes close but has a relatively inconvinient API and any pure-python bytearray wrapper will not be fast on CPython. From regebro at gmail.com Wed Feb 13 09:15:35 2013 From: regebro at gmail.com (Lennart Regebro) Date: Wed, 13 Feb 2013 09:15:35 +0100 Subject: [Python-Dev] Usage of += on strings in loops in stdlib In-Reply-To: References: Message-ID: On Tue, Feb 12, 2013 at 10:03 PM, Maciej Fijalkowski wrote: > Hi > > We recently encountered a performance issue in stdlib for pypy. It > turned out that someone commited a performance "fix" that uses += for > strings instead of "".join() that was there before. Can someone show the actual diff? Of this? I'm making a talk about outdated patterns in Python at DjangoCon EU, prompted by this question, and obsessive avoidance of string concatenation. But all the tests I've done show that ''.join() still is faster or as fast, except when you are joining very few strings, like for example two strings, in which case concatenation is faster or as fast. Both under PyPy and CPython. So I'd like to know in which case ''.hoin() is faster on PyPy and += faster on CPython. Code with times x = 100000 s1 = 'X'* x s2 = 'X'* x for i in xrange(500): s1 += s2 Python 3.3: 0.049 seconds PyPy 1.9: 24.217 seconds PyPy indeed is much much slower than CPython here. But let's look at the join case: x = 100000 s1 = 'X'* x s2 = 'X'* x for i in xrange(500): s1 = ''.join((s1, s2)) Python 3.3: 18.969 seconds PyPy 1.9: 62.539 seconds Here PyPy needs twice the time, and CPython needs 387 times as long time. Both are slower. The best case is of course to make a long list of strings and join them: x = 100000 s1 = 'X'* x s2 = 'X'* x l = [s1] for i in xrange(500): l.append(s2) s1 = ''.join(l) Python 3.3: 0.052 seconds PyPy 1.9: 0.117 seconds That's not always feasible though. //Lennart From larry at hastings.org Wed Feb 13 09:52:18 2013 From: larry at hastings.org (Larry Hastings) Date: Wed, 13 Feb 2013 00:52:18 -0800 Subject: [Python-Dev] Usage of += on strings in loops in stdlib In-Reply-To: <511AEB75.6090808@stackless.com> References: <20130212220642.71f03220@pitrou.net> <511AB6D6.60805@nedbatchelder.com> <20130212224404.4fa839fc@pitrou.net> <511AEB75.6090808@stackless.com> Message-ID: <511B5442.4020309@hastings.org> On 02/12/2013 05:25 PM, Christian Tismer wrote: > Ropes have been implemented by Carl-Friedrich Bolz in 2007 as I remember. > No idea what the impact was, if any at all. > Would ropes be an answer (and a simple way to cope with string mutation > patterns) as an alternative implementation, and therefore still justify > the usage of that pattern? I've always hated the "".join(array) idiom for "fast" string concatenation--it's ugly and it flies in the face of TOOWTDI. I think everyone should use "x = a + b + c + d" for string concatenation, and we should just make that fast. In 2006 I proposed "lazy string concatenation", a sort of rope that hid the details inside the string object. If a and b are strings, a+b returned a string object that internally lazily contained references to a and b, and only computed its value if you asked for it. Here's the Unicode version: http://bugs.python.org/issue1629305 Why didn't it get accepted? I lumped in lazy slicing, a bad move as it was more controversial. That and the possibility that macros like PyUnicode_AS_UNICODE could now possibly fail, which would have meant checking 400+ call sites to ensure they handle the possibility of failure. This latter work has already happened with the new efficient Unicode representation patch. I keep thinking it's time to revive the lazy string concatenation patch. //arry/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris at python.org Wed Feb 13 10:09:34 2013 From: chris at python.org (Chris Withers) Date: Wed, 13 Feb 2013 09:09:34 +0000 Subject: [Python-Dev] Usage of += on strings in loops in stdlib In-Reply-To: References: Message-ID: <511B584E.9020206@python.org> On 12/02/2013 21:03, Maciej Fijalkowski wrote: > We recently encountered a performance issue in stdlib for pypy. It > turned out that someone commited a performance "fix" that uses += for > strings instead of "".join() that was there before. That's... interesting. I fixed a performance bug in httplib some years ago by doing the exact opposite; += -> ''.join(). In that case, it changed downloading a file from 20 minutes to 3 seconds. That was likely on Python 2.5. > How people feel about generally not having += on long strings in > stdlib (since the refcount = 1 thing is a hack)? +1 from me. Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk From solipsis at pitrou.net Wed Feb 13 10:31:42 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 13 Feb 2013 10:31:42 +0100 Subject: [Python-Dev] Usage of += on strings in loops in stdlib References: Message-ID: <20130213103142.382aa6cc@pitrou.net> Le Wed, 13 Feb 2013 09:02:07 +0100, Victor Stinner a ?crit : > I added a _PyUnicodeWriter internal API to optimize str%args and > str.format(args). It uses a buffer which is overallocated, so it's > basically like CPython str += str optimization. I still don't know how > efficient it is on Windows, since realloc() is slow on Windows (at > least on old Windows versions). > > We should add an official and public API to concatenate strings. There's io.StringIO already. Regards Antoine. From storchaka at gmail.com Wed Feb 13 12:28:32 2013 From: storchaka at gmail.com (Serhiy Storchaka) Date: Wed, 13 Feb 2013 13:28:32 +0200 Subject: [Python-Dev] Usage of += on strings in loops in stdlib In-Reply-To: References: Message-ID: On 12.02.13 23:03, Maciej Fijalkowski wrote: > How people feel about generally not having += on long strings in > stdlib (since the refcount = 1 thing is a hack)? Sometimes the use of += for strings or bytes is appropriate. For example, I deliberately used += for bytes instead b''.join() (note that there is even no such hack for bytes) in zipfile module where in most cases one of component is empty, and the concatenation of nonempty components only happens once. b''.join() was noticeably slower here. From steve at pearwood.info Wed Feb 13 12:37:13 2013 From: steve at pearwood.info (Steven D'Aprano) Date: Wed, 13 Feb 2013 22:37:13 +1100 Subject: [Python-Dev] Usage of += on strings in loops in stdlib In-Reply-To: <511B5442.4020309@hastings.org> References: <20130212220642.71f03220@pitrou.net> <511AB6D6.60805@nedbatchelder.com> <20130212224404.4fa839fc@pitrou.net> <511AEB75.6090808@stackless.com> <511B5442.4020309@hastings.org> Message-ID: <511B7AE9.2080209@pearwood.info> On 13/02/13 19:52, Larry Hastings wrote: > I've always hated the "".join(array) idiom for "fast" string concatenation >--it's ugly and it flies in the face of TOOWTDI. I think everyone should >use "x = a + b + c + d" for string concatenation, and we should just make > that fast. "".join(array) is much nicer looking than: # ridiculous and impractical for more than a few items array[0] + array[1] + array[2] + ... + array[N] or: # not an expression result = "" for s in array: result += s or even: # currently prohibited, and not obvious sum(array, "") although I will admit to a certain fondness towards # even less obvious than sum map(operator.add, array) and join has been the obvious way to do repeated concatenation of many substrings since at least Python 1.5 when it was spelled "string.join(array [, sep=" "]). -- Steven From storchaka at gmail.com Wed Feb 13 12:37:44 2013 From: storchaka at gmail.com (Serhiy Storchaka) Date: Wed, 13 Feb 2013 13:37:44 +0200 Subject: [Python-Dev] Usage of += on strings in loops in stdlib In-Reply-To: References: <20130212220642.71f03220@pitrou.net> <511AB6D6.60805@nedbatchelder.com> <20130212224404.4fa839fc@pitrou.net> <511AEB75.6090808@stackless.com> Message-ID: On 13.02.13 09:52, Nick Coghlan wrote: > On Wed, Feb 13, 2013 at 5:42 PM, Alexandre Vassalotti > wrote: >> I don't think so. Ropes are really useful when you work with gigabytes of >> data, but unfortunately they don't make good general-purpose strings. >> Monolithic arrays are much more efficient and simple for the typical >> use-cases we have in Python. > > If I recall correctly, io.StringIO and io.BytesIO have been updated to > use ropes internally in 3.3. io.BytesIO has not yet. But it will be in 3.4 (issue #15381). On the other hand, there is a plan for rewriting StringIO to more effective continuous buffer implementation (issue #15612). From python-dev at masklinn.net Wed Feb 13 12:46:33 2013 From: python-dev at masklinn.net (Xavier Morel) Date: Wed, 13 Feb 2013 12:46:33 +0100 Subject: [Python-Dev] Usage of += on strings in loops in stdlib In-Reply-To: <511B7AE9.2080209@pearwood.info> References: <20130212220642.71f03220@pitrou.net> <511AB6D6.60805@nedbatchelder.com> <20130212224404.4fa839fc@pitrou.net> <511AEB75.6090808@stackless.com> <511B5442.4020309@hastings.org> <511B7AE9.2080209@pearwood.info> Message-ID: <193A64B4-EB87-450A-9DF1-E1B7A890648E@masklinn.net> On 2013-02-13, at 12:37 , Steven D'Aprano wrote: > > # even less obvious than sum > map(operator.add, array) That one does not work, it'll try to call the binary `add` with each item of the array when the map iterator is reified, erroring out. functools.reduce(operator.add, array, '') would work though, it's an other way to spell `sum` without the string prohibition. From steve at pearwood.info Wed Feb 13 12:53:25 2013 From: steve at pearwood.info (Steven D'Aprano) Date: Wed, 13 Feb 2013 22:53:25 +1100 Subject: [Python-Dev] Usage of += on strings in loops in stdlib In-Reply-To: <511B584E.9020206@python.org> References: <511B584E.9020206@python.org> Message-ID: <511B7EB5.3040401@pearwood.info> On 13/02/13 20:09, Chris Withers wrote: > On 12/02/2013 21:03, Maciej Fijalkowski wrote: >> We recently encountered a performance issue in stdlib for pypy. It >> turned out that someone commited a performance "fix" that uses += for >> strings instead of "".join() that was there before. > > That's... interesting. > > I fixed a performance bug in httplib some years ago by doing the exact opposite; += -> ''.join(). In that case, it changed downloading a file from 20 minutes to 3 seconds. That was likely on Python 2.5. I remember it well. http://mail.python.org/pipermail/python-dev/2009-August/091125.html I frequently link to this thread as an example of just how bad repeated string concatenation can be, how painful it can be to debug, and how even when the optimization is fast on one system, it may fail and be slow on another system. -- Steven From steve at pearwood.info Wed Feb 13 12:56:39 2013 From: steve at pearwood.info (Steven D'Aprano) Date: Wed, 13 Feb 2013 22:56:39 +1100 Subject: [Python-Dev] Usage of += on strings in loops in stdlib In-Reply-To: <193A64B4-EB87-450A-9DF1-E1B7A890648E@masklinn.net> References: <20130212220642.71f03220@pitrou.net> <511AB6D6.60805@nedbatchelder.com> <20130212224404.4fa839fc@pitrou.net> <511AEB75.6090808@stackless.com> <511B5442.4020309@hastings.org> <511B7AE9.2080209@pearwood.info> <193A64B4-EB87-450A-9DF1-E1B7A890648E@masklinn.net> Message-ID: <511B7F77.9020801@pearwood.info> On 13/02/13 22:46, Xavier Morel wrote: > On 2013-02-13, at 12:37 , Steven D'Aprano wrote: >> >> # even less obvious than sum >> map(operator.add, array) > > That one does not work, it'll try to call the binary `add` with each > item of the array when the map iterator is reified, erroring out. > > functools.reduce(operator.add, array, '') > > would work though, it's an other way to spell `sum` without the > string prohibition. Oops, you are right of course, I was thinking reduce but it came out map. Thanks for the correction. -- Steven From tismer at stackless.com Wed Feb 13 13:06:27 2013 From: tismer at stackless.com (Christian Tismer) Date: Wed, 13 Feb 2013 13:06:27 +0100 Subject: [Python-Dev] efficient string concatenation (yep, from 2004) In-Reply-To: References: <511AD5E1.1060209@stackless.com> Message-ID: <511B81C3.9030905@stackless.com> On 13.02.13 08:42, Lennart Regebro wrote: >> Something is needed - a patch for PyPy or for the documentation I guess. > Not arguing that it wouldn't be good, but I disagree that it is needed. > > This is only an issue when you, as in your proof, have a loop that > does concatenation. This is usually when looping over a list of > strings that should be concatenated together. Doing so in a loop with > concatenation may be the natural way for people new to Python, but the > "natural" way to do it in Python is with a ''.join() call. > > This: > > s = ''.join(('X' for x in xrange(x))) > > Is more than twice as fast in Python 2.7 than your example. It is in > fact also slower in PyPy 1.9 than Python 2.7, but only with a factor > of two: > > Python 2.7: > time for 10000000 concats = 0.887 > Pypy 1.9: > time for 10000000 concats = 1.600 > > (And of course s = 'X'* x takes only a bout a hundredth of the time, > but that's cheating. ;-) > This is not about how to write efficient concatenation and not for me. It is also not about a constant factor, which I don't really care about but in situations where speed matters. This is about a possible algorithmic trap, where code written for CPython may behave well with some roughly O(n) behavior, and by switching to PyPy you get a surprise when the same code now has O(n**2) behavior. Such runtime explosions can damage the trust in PyPy, with code sitting in some module which you even did not write but "pip install"-ed it. So this is important to know, especially for newcomers, and for people who are giving advice to them. For algorithmic compatibility, there should no longer be a feature with this drastic side effect, if that cannot be supported by all other dialects. To avoid such hidden traps in larger code bases, documentation is needed that clearly gives a warning saying "don't do that", like CS students learn for most other languages. cheers - chris -- Christian Tismer :^) Software Consulting : Have a break! Take a ride on Python's Karl-Liebknecht-Str. 121 : *Starship* http://starship.python.net/ 14482 Potsdam : PGP key -> http://pgp.uni-mainz.de phone +49 173 24 18 776 fax +49 (30) 700143-0023 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/ From steve at pearwood.info Wed Feb 13 13:10:26 2013 From: steve at pearwood.info (Steven D'Aprano) Date: Wed, 13 Feb 2013 23:10:26 +1100 Subject: [Python-Dev] efficient string concatenation (yep, from 2004) In-Reply-To: <511AD5E1.1060209@stackless.com> References: <511AD5E1.1060209@stackless.com> Message-ID: <511B82B2.2000505@pearwood.info> On 13/02/13 10:53, Christian Tismer wrote: > Hi friends, > > _efficient string concatenation_ has been a topic in 2004. > Armin Rigo proposed a patch with the name of the subject, > more precisely: > > /[Patches] [ python-Patches-980695 ] efficient string concatenation// > //on sourceforge.net, on 2004-06-28.// > / > This patch was finally added to Python 2.4 on 2004-11-30. > > Some people might remember the larger discussion if such a patch should be > accepted at all, because it changes the programming style for many of us > from "don't do that, stupid" to "well, you may do it in CPython", which has quite > some impact on other implementations (is it fast on Jython, now?). I disagree. If you look at the archives on the python-list@ and tutor at python.org mailing lists, you will see that whenever string concatenation comes up, the common advice given is to use join. The documentation for strings is also clear that you should not rely on this optimization: http://docs.python.org/2/library/stdtypes.html#typesseq And quadratic performance for repeated concatenation is not unique to Python: it applies to pretty much any language with immutable strings, including Java, C++, Lua and Javascript. > It changed for instance my programming and teaching style a lot, of course! Why do you say, "Of course"? It should not have changed anything. Best practice remains the same: - we should still use join for repeated concatenations; - we should still avoid + except for small cases which are not performance critical; - we should still teach beginners to use join; - while this optimization is nice to have, we cannot rely on it being there when it matters. It's not just Jython and IronPython that can't make use of this optimization. It can, and does, fail on CPython as well, as it is sensitive to memory allocation details. See for example: http://utcc.utoronto.ca/~cks/space/blog/python/ExaminingStringConcatOpt and here for a cautionary tale about what can happen when the optimization fails under CPython: http://mail.python.org/pipermail/python-dev/2009-August/091125.html -- Steven From storchaka at gmail.com Wed Feb 13 13:10:56 2013 From: storchaka at gmail.com (Serhiy Storchaka) Date: Wed, 13 Feb 2013 14:10:56 +0200 Subject: [Python-Dev] Usage of += on strings in loops in stdlib In-Reply-To: <511B5442.4020309@hastings.org> References: <20130212220642.71f03220@pitrou.net> <511AB6D6.60805@nedbatchelder.com> <20130212224404.4fa839fc@pitrou.net> <511AEB75.6090808@stackless.com> <511B5442.4020309@hastings.org> Message-ID: On 13.02.13 10:52, Larry Hastings wrote: > I've always hated the "".join(array) idiom for "fast" string > concatenation--it's ugly and it flies in the face of TOOWTDI. I think > everyone should use "x = a + b + c + d" for string concatenation, and we > should just make that fast. I prefer "x = '%s%s%s%s' % (a, b, c, d)" when string's number is more than 3 and some of them are literal strings. From tismer at stackless.com Wed Feb 13 13:39:58 2013 From: tismer at stackless.com (Christian Tismer) Date: Wed, 13 Feb 2013 13:39:58 +0100 Subject: [Python-Dev] efficient string concatenation (yep, from 2004) In-Reply-To: <511B82B2.2000505@pearwood.info> References: <511AD5E1.1060209@stackless.com> <511B82B2.2000505@pearwood.info> Message-ID: <511B899E.1000302@stackless.com> On 13.02.13 13:10, Steven D'Aprano wrote: > On 13/02/13 10:53, Christian Tismer wrote: >> Hi friends, >> >> _efficient string concatenation_ has been a topic in 2004. >> Armin Rigo proposed a patch with the name of the subject, >> more precisely: >> >> /[Patches] [ python-Patches-980695 ] efficient string concatenation// >> //on sourceforge.net, on 2004-06-28.// >> / >> This patch was finally added to Python 2.4 on 2004-11-30. >> >> Some people might remember the larger discussion if such a patch >> should be >> accepted at all, because it changes the programming style for many of us >> from "don't do that, stupid" to "well, you may do it in CPython", >> which has quite >> some impact on other implementations (is it fast on Jython, now?). > > I disagree. If you look at the archives on the python-list@ and > tutor at python.org > mailing lists, you will see that whenever string concatenation comes > up, the common > advice given is to use join. > > The documentation for strings is also clear that you should not rely > on this > optimization: > > http://docs.python.org/2/library/stdtypes.html#typesseq > > And quadratic performance for repeated concatenation is not unique to > Python: > it applies to pretty much any language with immutable strings, > including Java, > C++, Lua and Javascript. > > >> It changed for instance my programming and teaching style a lot, of >> course! > > Why do you say, "Of course"? It should not have changed anything. You are right, I was actually over the top with my rant and never recommend string concatenation when working with real amounts of data. The surprise was just so big. I tend to use whatever fits best for small initialization of some modules, where the fact that concat is cheap lets me stop thinking of big Oh. Although it probably does not matter much, it makes me feel incomfortable to do something with potentially bad asymptotics. > > Best practice remains the same: > > - we should still use join for repeated concatenations; > > - we should still avoid + except for small cases which are not > performance critical; > > - we should still teach beginners to use join; > > - while this optimization is nice to have, we cannot rely on it being > there > when it matters. I agree that CPython does say this clearly. Actually I was complaining about the PyPy documentation which does not mention this, and because PyPy is so very compatible already. 2004 when this stuff came up was the time where PyPy already was quite active, but the Psyco mindset was still around, too. Maybe my slightly shocked reaction originates from there, and my implicit assumption was never corrected ;-) cheers - chris -- Christian Tismer :^) Software Consulting : Have a break! Take a ride on Python's Karl-Liebknecht-Str. 121 : *Starship* http://starship.python.net/ 14482 Potsdam : PGP key -> http://pgp.uni-mainz.de phone +49 173 24 18 776 fax +49 (30) 700143-0023 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/ From dholth at gmail.com Wed Feb 13 14:17:03 2013 From: dholth at gmail.com (Daniel Holth) Date: Wed, 13 Feb 2013 08:17:03 -0500 Subject: [Python-Dev] Usage of += on strings in loops in stdlib In-Reply-To: References: <20130212220642.71f03220@pitrou.net> <511AB6D6.60805@nedbatchelder.com> <20130212224404.4fa839fc@pitrou.net> <511AEB75.6090808@stackless.com> <511B5442.4020309@hastings.org> Message-ID: On Wed, Feb 13, 2013 at 7:10 AM, Serhiy Storchaka wrote: > On 13.02.13 10:52, Larry Hastings wrote: > >> I've always hated the "".join(array) idiom for "fast" string >> concatenation--it's ugly and it flies in the face of TOOWTDI. I think >> everyone should use "x = a + b + c + d" for string concatenation, and we >> should just make that fast. >> > > I prefer "x = '%s%s%s%s' % (a, b, c, d)" when string's number is more than > 3 and some of them are literal strings. > Fixed: x = ('%s' * len(abcd)) % abcd -------------- next part -------------- An HTML attachment was scrubbed... URL: From regebro at gmail.com Wed Feb 13 14:23:17 2013 From: regebro at gmail.com (Lennart Regebro) Date: Wed, 13 Feb 2013 14:23:17 +0100 Subject: [Python-Dev] Usage of += on strings in loops in stdlib In-Reply-To: References: <20130212220642.71f03220@pitrou.net> <511AB6D6.60805@nedbatchelder.com> <20130212224404.4fa839fc@pitrou.net> <511AEB75.6090808@stackless.com> <511B5442.4020309@hastings.org> Message-ID: On Wed, Feb 13, 2013 at 1:10 PM, Serhiy Storchaka wrote: > I prefer "x = '%s%s%s%s' % (a, b, c, d)" when string's number is more than 3 > and some of them are literal strings. This has the benefit of being slow both on CPython and PyPy. Although using .format() is even slower. :-) //Lennart From tismer at stackless.com Wed Feb 13 14:30:21 2013 From: tismer at stackless.com (Christian Tismer) Date: Wed, 13 Feb 2013 14:30:21 +0100 Subject: [Python-Dev] Usage of += on strings in loops in stdlib In-Reply-To: References: <20130212220642.71f03220@pitrou.net> <511AB6D6.60805@nedbatchelder.com> <20130212224404.4fa839fc@pitrou.net> <511AEB75.6090808@stackless.com> <511B5442.4020309@hastings.org> Message-ID: <511B956D.3090207@stackless.com> On 13.02.13 14:17, Daniel Holth wrote: > On Wed, Feb 13, 2013 at 7:10 AM, Serhiy Storchaka > wrote: > > On 13.02.13 10:52, Larry Hastings wrote: > > I've always hated the "".join(array) idiom for "fast" string > concatenation--it's ugly and it flies in the face of TOOWTDI. > I think > everyone should use "x = a + b + c + d" for string > concatenation, and we > should just make that fast. > > > I prefer "x = '%s%s%s%s' % (a, b, c, d)" when string's number is > more than 3 and some of them are literal strings. > > > Fixed: x = ('%s' * len(abcd)) % abcd > Which becomes in the new formatting style x = ('{}' * len(abcd)).format(*abcd) hmm, hmm, not soo nice -- Christian Tismer :^) Software Consulting : Have a break! Take a ride on Python's Karl-Liebknecht-Str. 121 : *Starship* http://starship.python.net/ 14482 Potsdam : PGP key -> http://pgp.uni-mainz.de phone +49 173 24 18 776 fax +49 (30) 700143-0023 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris at python.org Wed Feb 13 15:18:42 2013 From: chris at python.org (Chris Withers) Date: Wed, 13 Feb 2013 14:18:42 +0000 Subject: [Python-Dev] Usage of += on strings in loops in stdlib In-Reply-To: <511B7EB5.3040401@pearwood.info> References: <511B584E.9020206@python.org> <511B7EB5.3040401@pearwood.info> Message-ID: <511BA0C2.6010606@python.org> On 13/02/2013 11:53, Steven D'Aprano wrote: >> I fixed a performance bug in httplib some years ago by doing the exact >> opposite; += -> ''.join(). In that case, it changed downloading a file >> from 20 minutes to 3 seconds. That was likely on Python 2.5. > > > I remember it well. > > http://mail.python.org/pipermail/python-dev/2009-August/091125.html > > I frequently link to this thread as an example of just how bad repeated > string concatenation can be, how painful it can be to debug, and how > even when the optimization is fast on one system, it may fail and be > slow on another system. Amusing is that http://mail.python.org/pipermail/python-dev/2009-August/thread.html#91125 doesn't even list the email where I found the problem... Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk From amauryfa at gmail.com Wed Feb 13 15:27:31 2013 From: amauryfa at gmail.com (Amaury Forgeot d'Arc) Date: Wed, 13 Feb 2013 15:27:31 +0100 Subject: [Python-Dev] Usage of += on strings in loops in stdlib In-Reply-To: References: <20130212220642.71f03220@pitrou.net> <511AB6D6.60805@nedbatchelder.com> <20130212224404.4fa839fc@pitrou.net> <511AEB75.6090808@stackless.com> <511B5442.4020309@hastings.org> Message-ID: 2013/2/13 Lennart Regebro > On Wed, Feb 13, 2013 at 1:10 PM, Serhiy Storchaka > wrote: > > I prefer "x = '%s%s%s%s' % (a, b, c, d)" when string's number is more > than 3 > > and some of them are literal strings. > > This has the benefit of being slow both on CPython and PyPy. Although > using .format() is even slower. :-) Did you really try it? PyPy is really fast with str.__mod__, when the format string is a constant. Yes, it's jitted. -- Amaury Forgeot d'Arc -------------- next part -------------- An HTML attachment was scrubbed... URL: From tismer at stackless.com Wed Feb 13 15:34:06 2013 From: tismer at stackless.com (Christian Tismer) Date: Wed, 13 Feb 2013 15:34:06 +0100 Subject: [Python-Dev] Usage of += on strings in loops in stdlib In-Reply-To: References: <20130212220642.71f03220@pitrou.net> <511AB6D6.60805@nedbatchelder.com> <20130212224404.4fa839fc@pitrou.net> <511AEB75.6090808@stackless.com> <511B5442.4020309@hastings.org> Message-ID: <511BA45E.2080908@stackless.com> On 13.02.13 15:27, Amaury Forgeot d'Arc wrote: > > 2013/2/13 Lennart Regebro > > > On Wed, Feb 13, 2013 at 1:10 PM, Serhiy Storchaka > > wrote: > > I prefer "x = '%s%s%s%s' % (a, b, c, d)" when string's number is > more than 3 > > and some of them are literal strings. > > This has the benefit of being slow both on CPython and PyPy. Although > using .format() is even slower. :-) > > > Did you really try it? > PyPy is really fast with str.__mod__, when the format string is a > constant. > Yes, it's jitted. How about the .format() style: Is that jitted as well? In order to get people to prefer .format over __mod__, it would be nice if PyPy made this actually _faster_ :-) -- Christian Tismer :^) Software Consulting : Have a break! Take a ride on Python's Karl-Liebknecht-Str. 121 : *Starship* http://starship.python.net/ 14482 Potsdam : PGP key -> http://pgp.uni-mainz.de phone +49 173 24 18 776 fax +49 (30) 700143-0023 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From storchaka at gmail.com Wed Feb 13 15:40:47 2013 From: storchaka at gmail.com (Serhiy Storchaka) Date: Wed, 13 Feb 2013 16:40:47 +0200 Subject: [Python-Dev] Usage of += on strings in loops in stdlib In-Reply-To: References: <20130212220642.71f03220@pitrou.net> <511AB6D6.60805@nedbatchelder.com> <20130212224404.4fa839fc@pitrou.net> <511AEB75.6090808@stackless.com> <511B5442.4020309@hastings.org> Message-ID: On 13.02.13 15:23, Lennart Regebro wrote: > On Wed, Feb 13, 2013 at 1:10 PM, Serhiy Storchaka wrote: >> I prefer "x = '%s%s%s%s' % (a, b, c, d)" when string's number is more than 3 >> and some of them are literal strings. > > This has the benefit of being slow both on CPython and PyPy. Although > using .format() is even slower. :-) Only slightly. $ ./python -m timeit -s "spam = 'spam'; ham = 'ham'" "spam + ' = ' + ham + '\n'" 1000000 loops, best of 3: 0.501 usec per loop $ ./python -m timeit -s "spam = 'spam'; ham = 'ham'" "''.join([spam, ' = ', ham, '\n'])" 1000000 loops, best of 3: 0.504 usec per loop $ ./python -m timeit -s "spam = 'spam'; ham = 'ham'" "'%s = %s\n' % (spam, ham)" 1000000 loops, best of 3: 0.524 usec per loop But the last variant looks better for me. From ncoghlan at gmail.com Wed Feb 13 15:44:09 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 14 Feb 2013 00:44:09 +1000 Subject: [Python-Dev] efficient string concatenation (yep, from 2004) In-Reply-To: <511B81C3.9030905@stackless.com> References: <511AD5E1.1060209@stackless.com> <511B81C3.9030905@stackless.com> Message-ID: On Wed, Feb 13, 2013 at 10:06 PM, Christian Tismer wrote: > To avoid such hidden traps in larger code bases, documentation is > needed that clearly gives a warning saying "don't do that", like CS > students learn for most other languages. How much more explicit do you want us to be? """6. CPython implementation detail: If s and t are both strings, some Python implementations such as CPython can usually perform an in-place optimization for assignments of the form s = s + t or s += t. When applicable, this optimization makes quadratic run-time much less likely. This optimization is both version and implementation dependent. For performance sensitive code, it is preferable to use the str.join() method which assures consistent linear concatenation performance across versions and implementations.""" from http://docs.python.org/2/library/stdtypes.html#typesseq So please don't blame us for people not reading a warning that is already there. Since my rewrite of the sequence docs, Python 3 doesn't even acknowledge the hack's existence and is quite explicit about what you need to do to get reliably linear behaviour: """6. Concatenating immutable sequences always results in a new object. This means that building up a sequence by repeated concatenation will have a quadratic runtime cost in the total sequence length. To get a linear runtime cost, you must switch to one of the alternatives below: if concatenating str objects, you can build a list and use str.join() at the end or else write to a io.StringIO instance and retrieve its value when complete if concatenating bytes objects, you can similarly use bytes.join() or io.BytesIO, or you can do in-place concatenation with a bytearray object. bytearray objects are mutable and have an efficient overallocation mechanism if concatenating tuple objects, extend a list instead for other types, investigate the relevant class documentation""" from http://docs.python.org/3/library/stdtypes.html#common-sequence-operations Deliberately *relying* on the += hack to avoid quadratic runtime is just plain wrong, and our documentation already says so. If anyone really thinks it will help, I can add a CPython implementation note back in to the Python 3 docs as well, pointing out that CPython performance measurements may hide broken algorithmic complexity related to string concatenation, but the corresponding note in Python 2 doesn't seem to have done much good :P Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From amauryfa at gmail.com Wed Feb 13 15:47:45 2013 From: amauryfa at gmail.com (Amaury Forgeot d'Arc) Date: Wed, 13 Feb 2013 15:47:45 +0100 Subject: [Python-Dev] Usage of += on strings in loops in stdlib In-Reply-To: <511BA45E.2080908@stackless.com> References: <20130212220642.71f03220@pitrou.net> <511AB6D6.60805@nedbatchelder.com> <20130212224404.4fa839fc@pitrou.net> <511AEB75.6090808@stackless.com> <511B5442.4020309@hastings.org> <511BA45E.2080908@stackless.com> Message-ID: 2013/2/13 Christian Tismer > On 13.02.13 15:27, Amaury Forgeot d'Arc wrote: > > > 2013/2/13 Lennart Regebro > >> On Wed, Feb 13, 2013 at 1:10 PM, Serhiy Storchaka >> wrote: >> > I prefer "x = '%s%s%s%s' % (a, b, c, d)" when string's number is more >> than 3 >> > and some of them are literal strings. >> >> This has the benefit of being slow both on CPython and PyPy. Although >> using .format() is even slower. :-) > > > Did you really try it? > PyPy is really fast with str.__mod__, when the format string is a constant. > Yes, it's jitted. > > > How about the .format() style: Is that jitted as well? > In order to get people to prefer .format over __mod__, > it would be nice if PyPy made this actually _faster_ :-) .format() is jitted as well. But it's still slower than str.__mod__ (about 25%) I suppose it can be further optimized. -- Amaury Forgeot d'Arc -------------- next part -------------- An HTML attachment was scrubbed... URL: From regebro at gmail.com Wed Feb 13 15:53:27 2013 From: regebro at gmail.com (Lennart Regebro) Date: Wed, 13 Feb 2013 15:53:27 +0100 Subject: [Python-Dev] Usage of += on strings in loops in stdlib In-Reply-To: References: <20130212220642.71f03220@pitrou.net> <511AB6D6.60805@nedbatchelder.com> <20130212224404.4fa839fc@pitrou.net> <511AEB75.6090808@stackless.com> <511B5442.4020309@hastings.org> Message-ID: On Wed, Feb 13, 2013 at 3:27 PM, Amaury Forgeot d'Arc wrote: > > 2013/2/13 Lennart Regebro >> >> On Wed, Feb 13, 2013 at 1:10 PM, Serhiy Storchaka >> wrote: >> > I prefer "x = '%s%s%s%s' % (a, b, c, d)" when string's number is more >> > than 3 >> > and some of them are literal strings. >> >> This has the benefit of being slow both on CPython and PyPy. Although >> using .format() is even slower. :-) > > > Did you really try it? Yes. > PyPy is really fast with str.__mod__, when the format string is a constant. > Yes, it's jitted. Simple concatenation: s1 = s1 + s2 PyPy-1.9 time for 100 concats of 10000 length strings = 7.133 CPython time for 100 concats of 10000 length strings = 0.005 Making a list of strings and joining after the loop: s1 = ''.join(l) PyPy-1.9 time for 100 concats of 10000 length strings = 0.005 CPython time for 100 concats of 10000 length strings = 0.003 Old formatting: s1 = '%s%s' % (s1, s2) PyPy-1.9 time for 100 concats of 10000 length strings = 20.924 CPython time for 100 concats of 10000 length strings = 3.787 New formatting: s1 = '{0}{1}'.format(s1, s2) PyPy-1.9 time for 100 concats of 10000 length strings = 13.249 CPython time for 100 concats of 10000 length strings = 3.751 I have, by the way, yet to find a usecase where the fastest method in CPython is not also the fastest in PyPy. //Lennart From regebro at gmail.com Wed Feb 13 15:55:41 2013 From: regebro at gmail.com (Lennart Regebro) Date: Wed, 13 Feb 2013 15:55:41 +0100 Subject: [Python-Dev] Usage of += on strings in loops in stdlib In-Reply-To: References: <20130212220642.71f03220@pitrou.net> <511AB6D6.60805@nedbatchelder.com> <20130212224404.4fa839fc@pitrou.net> <511AEB75.6090808@stackless.com> <511B5442.4020309@hastings.org> Message-ID: On Wed, Feb 13, 2013 at 3:27 PM, Amaury Forgeot d'Arc wrote: > Yes, it's jitted. Admittedly, I have no idea in which cases the JIT kicks in, and what I should do to make that happen to make sure I have the best possible real-life test cases. //Lennart From storchaka at gmail.com Wed Feb 13 15:56:02 2013 From: storchaka at gmail.com (Serhiy Storchaka) Date: Wed, 13 Feb 2013 16:56:02 +0200 Subject: [Python-Dev] Usage of += on strings in loops in stdlib In-Reply-To: References: <20130212220642.71f03220@pitrou.net> <511AB6D6.60805@nedbatchelder.com> <20130212224404.4fa839fc@pitrou.net> <511AEB75.6090808@stackless.com> <511B5442.4020309@hastings.org> Message-ID: On 13.02.13 15:17, Daniel Holth wrote: > On Wed, Feb 13, 2013 at 7:10 AM, Serhiy Storchaka > wrote: > I prefer "x = '%s%s%s%s' % (a, b, c, d)" when string's number is > more than 3 and some of them are literal strings. > > Fixed: x = ('%s' * len(abcd)) % abcd No, you don't need this for the constant number of strings. Because almost certainly some of strings will be literals, you can write this in a more nice way. Compare: 'config[' + key + '] = ' + value + '\n' ''.join(['config[', key, '] = ', value, '\n']) 'config[%s] = %s\n' % (key, value) From amauryfa at gmail.com Wed Feb 13 16:02:57 2013 From: amauryfa at gmail.com (Amaury Forgeot d'Arc) Date: Wed, 13 Feb 2013 16:02:57 +0100 Subject: [Python-Dev] Usage of += on strings in loops in stdlib In-Reply-To: References: <20130212220642.71f03220@pitrou.net> <511AB6D6.60805@nedbatchelder.com> <20130212224404.4fa839fc@pitrou.net> <511AEB75.6090808@stackless.com> <511B5442.4020309@hastings.org> Message-ID: 2013/2/13 Lennart Regebro > On Wed, Feb 13, 2013 at 3:27 PM, Amaury Forgeot d'Arc > wrote: > > Yes, it's jitted. > > Admittedly, I have no idea in which cases the JIT kicks in, and what I > should do to make that happen to make sure I have the best possible > real-life test cases. > PyPy JIT kicks in only after 1000 iterations. I usually use timeit. It's funny to see how the "1000 loops" line is 5 times faster than the "100 loops": $ ./pypy-c -m timeit -v -s "a,b,c,d='1234'" "'{}{}{}{}'.format(a,b,c,d)" 10 loops -> 2.19e-05 secs 100 loops -> 0.000122 secs 1000 loops -> 0.00601 secs 10000 loops -> 0.000363 secs 100000 loops -> 0.00528 secs 1000000 loops -> 0.0533 secs 10000000 loops -> 0.528 secs raw times: 0.521 0.52 0.51 10000000 loops, best of 3: 0.051 usec per loop -- Amaury Forgeot d'Arc -------------- next part -------------- An HTML attachment was scrubbed... URL: From tismer at stackless.com Wed Feb 13 18:07:22 2013 From: tismer at stackless.com (Christian Tismer) Date: Wed, 13 Feb 2013 18:07:22 +0100 Subject: [Python-Dev] efficient string concatenation (yep, from 2004) In-Reply-To: References: <511AD5E1.1060209@stackless.com> <511B81C3.9030905@stackless.com> Message-ID: <511BC84A.80905@stackless.com> Hey Nick, On 13.02.13 15:44, Nick Coghlan wrote: > On Wed, Feb 13, 2013 at 10:06 PM, Christian Tismer wrote: >> To avoid such hidden traps in larger code bases, documentation is >> needed that clearly gives a warning saying "don't do that", like CS >> students learn for most other languages. > How much more explicit do you want us to be? > > """6. CPython implementation detail: If s and t are both strings, some > Python implementations such as CPython can usually perform an in-place > optimization for assignments of the form s = s + t or s += t. When > applicable, this optimization makes quadratic run-time much less > likely. This optimization is both version and implementation > dependent. For performance sensitive code, it is preferable to use the > str.join() method which assures consistent linear concatenation > performance across versions and implementations.""" > > from http://docs.python.org/2/library/stdtypes.html#typesseq > > So please don't blame us for people not reading a warning that is already there. I don't, really not. This was a cross-posting effect. I was using the PyPy documentation, only, and there a lot of things are mentioned, but this behavioral difference was missing. Python-dev was not addressed at all. > ... > Deliberately *relying* on the += hack to avoid quadratic runtime is > just plain wrong, and our documentation already says so. > > If anyone really thinks it will help, I can add a CPython > implementation note back in to the Python 3 docs as well, pointing out > that CPython performance measurements may hide broken algorithmic > complexity related to string concatenation, but the corresponding note > in Python 2 doesn't seem to have done much good :P > Well, while we are at it: Yes, it says so as a note at the end of http://docs.python.org/2/library/stdtypes.html#typesseq I doubt that many people read that far, and they do not search documentation about sequence types when they are adding some strings together. People seem to have a tendency to just try something out instead and see if it works. That even seems to get worse the better and bigger the Python documentation grows. ;-) Maybe it would be a good idea to remove that concat optimization completely? Then people will wake up and read the docs to find out what's wrong ;-) No, does not help, because their test cases will not cover the reality. ----- Thinking a bit more about it. If you think about docs improvement, I don't believe it helps to make the very complete reference documentation even more complete. Completeness is great, don't take me wrong! But what people read is what pops right into their face, and I think that could be added. I think before getting people to work through long and complete documentation, it is probably easier to wake their interest by something like "Hey, are you doing things this way?" And then there is a short, concise list of bad and good things, maybe even dynamic as in WingWare's "Wing Tips" or any better approach. From that easily reachable, only a few pages long tabular collection of short hints and caveats there could be linkage to the existing, real documentation that explains things in more detail. Maybe that could be a way to get people to actually read. Just an idea. cheers - Chris p.s.: Other nice attempts that don't seem to really work: Some hints like http://docs.python.org/2/howto/doanddont.html are not bad, although that is hidden in the HowTO section, does only address a few things, and also the sub-title "in-depth documents on specific topics" is not what they seek in the first place while hacking on some code. Looking things up in a quick ref like http://rgruet.free.fr/PQR27/PQR2.7.html is very concise but does also _not_ mention what to avoid. Others exist, like http://infohost.nmt.edu/tcc/help/pubs/python/web/ By the way, the first thing I find via google is: http://www.python.org/doc/QuickRef.html which is quite funny (v1.3) -- Christian Tismer :^) Software Consulting : Have a break! Take a ride on Python's Karl-Liebknecht-Str. 121 : *Starship* http://starship.python.net/ 14482 Potsdam : PGP key -> http://pgp.uni-mainz.de phone +49 173 24 18 776 fax +49 (30) 700143-0023 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From python at mrabarnett.plus.com Wed Feb 13 18:33:05 2013 From: python at mrabarnett.plus.com (MRAB) Date: Wed, 13 Feb 2013 17:33:05 +0000 Subject: [Python-Dev] Usage of += on strings in loops in stdlib In-Reply-To: References: <20130212220642.71f03220@pitrou.net> <511AB6D6.60805@nedbatchelder.com> <20130212224404.4fa839fc@pitrou.net> <511AEB75.6090808@stackless.com> <511B5442.4020309@hastings.org> Message-ID: <511BCE51.4080805@mrabarnett.plus.com> On 2013-02-13 13:23, Lennart Regebro wrote: > On Wed, Feb 13, 2013 at 1:10 PM, Serhiy Storchaka wrote: >> I prefer "x = '%s%s%s%s' % (a, b, c, d)" when string's number is more than 3 >> and some of them are literal strings. > > This has the benefit of being slow both on CPython and PyPy. Although > using .format() is even slower. :-) > How about adding a class method for catenation: str.cat(a, b, c, d) str.cat([a, b, c, d]) # Equivalent to "".join([a, b, c, d]) Each argument could be a string or a list of strings. From fijall at gmail.com Wed Feb 13 19:06:09 2013 From: fijall at gmail.com (Maciej Fijalkowski) Date: Wed, 13 Feb 2013 20:06:09 +0200 Subject: [Python-Dev] Usage of += on strings in loops in stdlib In-Reply-To: <511BCE51.4080805@mrabarnett.plus.com> References: <20130212220642.71f03220@pitrou.net> <511AB6D6.60805@nedbatchelder.com> <20130212224404.4fa839fc@pitrou.net> <511AEB75.6090808@stackless.com> <511B5442.4020309@hastings.org> <511BCE51.4080805@mrabarnett.plus.com> Message-ID: On Wed, Feb 13, 2013 at 7:33 PM, MRAB wrote: > On 2013-02-13 13:23, Lennart Regebro wrote: >> >> On Wed, Feb 13, 2013 at 1:10 PM, Serhiy Storchaka >> wrote: >>> >>> I prefer "x = '%s%s%s%s' % (a, b, c, d)" when string's number is more >>> than 3 >>> and some of them are literal strings. >> >> >> This has the benefit of being slow both on CPython and PyPy. Although >> using .format() is even slower. :-) >> > How about adding a class method for catenation: > > str.cat(a, b, c, d) > str.cat([a, b, c, d]) # Equivalent to "".join([a, b, c, d]) > > Each argument could be a string or a list of strings. > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/fijall%40gmail.com I actually wonder. There seems to be the consensus to avoid += (to some extent). Can someone commit the change to urrllib then? I'm talking about reverting http://bugs.python.org/issue1285086 specifically From brett at python.org Wed Feb 13 19:24:38 2013 From: brett at python.org (Brett Cannon) Date: Wed, 13 Feb 2013 13:24:38 -0500 Subject: [Python-Dev] Usage of += on strings in loops in stdlib In-Reply-To: References: <20130212220642.71f03220@pitrou.net> <511AB6D6.60805@nedbatchelder.com> <20130212224404.4fa839fc@pitrou.net> <511AEB75.6090808@stackless.com> <511B5442.4020309@hastings.org> <511BCE51.4080805@mrabarnett.plus.com> Message-ID: On Wed, Feb 13, 2013 at 1:06 PM, Maciej Fijalkowski wrote: > On Wed, Feb 13, 2013 at 7:33 PM, MRAB wrote: > > On 2013-02-13 13:23, Lennart Regebro wrote: > >> > >> On Wed, Feb 13, 2013 at 1:10 PM, Serhiy Storchaka > >> wrote: > >>> > >>> I prefer "x = '%s%s%s%s' % (a, b, c, d)" when string's number is more > >>> than 3 > >>> and some of them are literal strings. > >> > >> > >> This has the benefit of being slow both on CPython and PyPy. Although > >> using .format() is even slower. :-) > >> > > How about adding a class method for catenation: > > > > str.cat(a, b, c, d) > > str.cat([a, b, c, d]) # Equivalent to "".join([a, b, c, d]) > > > > Each argument could be a string or a list of strings. > > > > > > _______________________________________________ > > Python-Dev mailing list > > Python-Dev at python.org > > http://mail.python.org/mailman/listinfo/python-dev > > Unsubscribe: > > http://mail.python.org/mailman/options/python-dev/fijall%40gmail.com > > I actually wonder. > > There seems to be the consensus to avoid += (to some extent). Can > someone commit the change to urrllib then? I'm talking about reverting > http://bugs.python.org/issue1285086 specifically Please re-open the bug with a comment as to why and I'm sure someone will get to it. -------------- next part -------------- An HTML attachment was scrubbed... URL: From fijall at gmail.com Wed Feb 13 19:27:38 2013 From: fijall at gmail.com (Maciej Fijalkowski) Date: Wed, 13 Feb 2013 20:27:38 +0200 Subject: [Python-Dev] Usage of += on strings in loops in stdlib In-Reply-To: References: <20130212220642.71f03220@pitrou.net> <511AB6D6.60805@nedbatchelder.com> <20130212224404.4fa839fc@pitrou.net> <511AEB75.6090808@stackless.com> <511B5442.4020309@hastings.org> <511BCE51.4080805@mrabarnett.plus.com> Message-ID: On Wed, Feb 13, 2013 at 8:24 PM, Brett Cannon wrote: > > > > On Wed, Feb 13, 2013 at 1:06 PM, Maciej Fijalkowski > wrote: >> >> On Wed, Feb 13, 2013 at 7:33 PM, MRAB wrote: >> > On 2013-02-13 13:23, Lennart Regebro wrote: >> >> >> >> On Wed, Feb 13, 2013 at 1:10 PM, Serhiy Storchaka >> >> wrote: >> >>> >> >>> I prefer "x = '%s%s%s%s' % (a, b, c, d)" when string's number is more >> >>> than 3 >> >>> and some of them are literal strings. >> >> >> >> >> >> This has the benefit of being slow both on CPython and PyPy. Although >> >> using .format() is even slower. :-) >> >> >> > How about adding a class method for catenation: >> > >> > str.cat(a, b, c, d) >> > str.cat([a, b, c, d]) # Equivalent to "".join([a, b, c, d]) >> > >> > Each argument could be a string or a list of strings. >> > >> > >> > _______________________________________________ >> > Python-Dev mailing list >> > Python-Dev at python.org >> > http://mail.python.org/mailman/listinfo/python-dev >> > Unsubscribe: >> > http://mail.python.org/mailman/options/python-dev/fijall%40gmail.com >> >> I actually wonder. >> >> There seems to be the consensus to avoid += (to some extent). Can >> someone commit the change to urrllib then? I'm talking about reverting >> http://bugs.python.org/issue1285086 specifically > > > Please re-open the bug with a comment as to why and I'm sure someone will > get to it. I can't re-open the bug, my account is kind of lame (and seriously, why do you guys *do* have multiple layers of bug tracker accounts?) Cheers, fijal From brett at python.org Wed Feb 13 19:34:12 2013 From: brett at python.org (Brett Cannon) Date: Wed, 13 Feb 2013 13:34:12 -0500 Subject: [Python-Dev] Usage of += on strings in loops in stdlib In-Reply-To: References: <20130212220642.71f03220@pitrou.net> <511AB6D6.60805@nedbatchelder.com> <20130212224404.4fa839fc@pitrou.net> <511AEB75.6090808@stackless.com> <511B5442.4020309@hastings.org> <511BCE51.4080805@mrabarnett.plus.com> Message-ID: On Wed, Feb 13, 2013 at 1:27 PM, Maciej Fijalkowski wrote: > On Wed, Feb 13, 2013 at 8:24 PM, Brett Cannon wrote: > > > > > > > > On Wed, Feb 13, 2013 at 1:06 PM, Maciej Fijalkowski > > wrote: > >> > >> On Wed, Feb 13, 2013 at 7:33 PM, MRAB > wrote: > >> > On 2013-02-13 13:23, Lennart Regebro wrote: > >> >> > >> >> On Wed, Feb 13, 2013 at 1:10 PM, Serhiy Storchaka < > storchaka at gmail.com> > >> >> wrote: > >> >>> > >> >>> I prefer "x = '%s%s%s%s' % (a, b, c, d)" when string's number is > more > >> >>> than 3 > >> >>> and some of them are literal strings. > >> >> > >> >> > >> >> This has the benefit of being slow both on CPython and PyPy. Although > >> >> using .format() is even slower. :-) > >> >> > >> > How about adding a class method for catenation: > >> > > >> > str.cat(a, b, c, d) > >> > str.cat([a, b, c, d]) # Equivalent to "".join([a, b, c, d]) > >> > > >> > Each argument could be a string or a list of strings. > >> > > >> > > >> > _______________________________________________ > >> > Python-Dev mailing list > >> > Python-Dev at python.org > >> > http://mail.python.org/mailman/listinfo/python-dev > >> > Unsubscribe: > >> > http://mail.python.org/mailman/options/python-dev/fijall%40gmail.com > >> > >> I actually wonder. > >> > >> There seems to be the consensus to avoid += (to some extent). Can > >> someone commit the change to urrllib then? I'm talking about reverting > >> http://bugs.python.org/issue1285086 specifically > > > > > > Please re-open the bug with a comment as to why and I'm sure someone will > > get to it. > > I can't re-open the bug, my account is kind of lame Then leave a comment and I will re-open it. > (and seriously, > why do you guys *do* have multiple layers of bug tracker accounts?) > You obviously have not had users argue with your decision by constantly flipping a bug back open. =) -------------- next part -------------- An HTML attachment was scrubbed... URL: From tismer at stackless.com Wed Feb 13 19:40:38 2013 From: tismer at stackless.com (Christian Tismer) Date: Wed, 13 Feb 2013 19:40:38 +0100 Subject: [Python-Dev] Usage of += on strings in loops in stdlib In-Reply-To: References: <20130212220642.71f03220@pitrou.net> <511AB6D6.60805@nedbatchelder.com> <20130212224404.4fa839fc@pitrou.net> <511AEB75.6090808@stackless.com> <511B5442.4020309@hastings.org> <511BCE51.4080805@mrabarnett.plus.com> Message-ID: <511BDE26.5060808@stackless.com> On 13.02.13 19:06, Maciej Fijalkowski wrote: > On Wed, Feb 13, 2013 at 7:33 PM, MRAB wrote: >> On 2013-02-13 13:23, Lennart Regebro wrote: >>> On Wed, Feb 13, 2013 at 1:10 PM, Serhiy Storchaka >>> wrote: >>>> I prefer "x = '%s%s%s%s' % (a, b, c, d)" when string's number is more >>>> than 3 >>>> and some of them are literal strings. >>> >>> This has the benefit of being slow both on CPython and PyPy. Although >>> using .format() is even slower. :-) >>> >> How about adding a class method for catenation: >> >> str.cat(a, b, c, d) >> str.cat([a, b, c, d]) # Equivalent to "".join([a, b, c, d]) >> >> Each argument could be a string or a list of strings. >> >> >> _______________________________________________ >> Python-Dev mailing list >> Python-Dev at python.org >> http://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe: >> http://mail.python.org/mailman/options/python-dev/fijall%40gmail.com > I actually wonder. > > There seems to be the consensus to avoid += (to some extent). Can > someone commit the change to urrllib then? I'm talking about reverting > http://bugs.python.org/issue1285086 specifically So _is_ += faster in certain library funcs than ''.join() ? If that's the case, the behavior of string concat could be something that might be added to some implementation info, if speed really matters. The library function then could take this info and use the appropriate code path to always be fast, during module initialisation. This is also quite explicit, since it tells the reader not to use in-place add when it is not optimized. If += is anyway a bit slower than other ways, forget it. I would then maybe add a commend somewhere that says "avoiding '+=' because it is not reliable" or something. cheers - chris -- Christian Tismer :^) Software Consulting : Have a break! Take a ride on Python's Karl-Liebknecht-Str. 121 : *Starship* http://starship.python.net/ 14482 Potsdam : PGP key -> http://pgp.uni-mainz.de phone +49 173 24 18 776 fax +49 (30) 700143-0023 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/ From fijall at gmail.com Wed Feb 13 19:48:08 2013 From: fijall at gmail.com (Maciej Fijalkowski) Date: Wed, 13 Feb 2013 20:48:08 +0200 Subject: [Python-Dev] Marking GC details as CPython-only Message-ID: Hi I've tried (and failed) to find what GC details (especially finalizer semantics) are CPython only and which ones are not. The best I could find was the documentation of __del__ here: http://docs.python.org/2/reference/datamodel.html Things were pypy differs: * finalizers in pypy will be called only once, even if the object is resurrected. I'm not sure if this is detail or we're just plain incompatible. * pypy breaks cycles and runs finalizers in random order (but topologically correct), hence gc.garbage is always empty. I *think* this part is really just an implementation detail * we're discussing right now about running multiple finalizers. We want to run them in order, but if there is a link a -> b and a becomes unreachable, we want to reserve the right to call finalizer a then finalizer b, even if a.__del__ resurrects a. What do you think? Overall, the __del__ is baaad. Cheers, fijal From storchaka at gmail.com Wed Feb 13 19:51:48 2013 From: storchaka at gmail.com (Serhiy Storchaka) Date: Wed, 13 Feb 2013 20:51:48 +0200 Subject: [Python-Dev] Usage of += on strings in loops in stdlib In-Reply-To: <511BDE26.5060808@stackless.com> References: <20130212220642.71f03220@pitrou.net> <511AB6D6.60805@nedbatchelder.com> <20130212224404.4fa839fc@pitrou.net> <511AEB75.6090808@stackless.com> <511B5442.4020309@hastings.org> <511BCE51.4080805@mrabarnett.plus.com> <511BDE26.5060808@stackless.com> Message-ID: On 13.02.13 20:40, Christian Tismer wrote: > If += is anyway a bit slower than other ways, forget it. > I would then maybe add a commend somewhere that says > "avoiding '+=' because it is not reliable" or something. += is a fastest way (in any implementation) if you concatenates only two strings. From python-dev at masklinn.net Wed Feb 13 20:09:28 2013 From: python-dev at masklinn.net (Xavier Morel) Date: Wed, 13 Feb 2013 20:09:28 +0100 Subject: [Python-Dev] Marking GC details as CPython-only In-Reply-To: References: Message-ID: <40757358-85D3-4FBF-B865-2FFBC22DC031@masklinn.net> On 2013-02-13, at 19:48 , Maciej Fijalkowski wrote: > Hi > > I've tried (and failed) to find what GC details (especially finalizer > semantics) are CPython only and which ones are not. The best I could > find was the documentation of __del__ here: > http://docs.python.org/2/reference/datamodel.html > > Things were pypy differs: > > * finalizers in pypy will be called only once, even if the object is > resurrected. I'm not sure if this is detail or we're just plain > incompatible. > > * pypy breaks cycles and runs finalizers in random order (but > topologically correct), hence gc.garbage is always empty. I *think* > this part is really just an implementation detail > > * we're discussing right now about running multiple finalizers. We > want to run them in order, but if there is a link a -> b and a becomes > unreachable, we want to reserve the right to call finalizer a then > finalizer b, even if a.__del__ resurrects a. What do you think? > > Overall, the __del__ is baaad. > > Cheers, > fijal There may be one more, although I'm not sure whether it's a GC artifact or something completely unspecified: if a context manager is part of a suspended stack (because it's in a generator) when the program terminates, cpython will run __exit__ but pypy will not -- # -*- encoding: utf-8 -*- class C(object): def __enter__(self): print ("entering") def __exit__(self, *args): print ("exiting") def gen(): with C(): yield r = gen() next(r) -- $ python2 test.py entering exiting $ python3 test.py entering exiting $ pypy test.py entering $ -- From rdmurray at bitdance.com Wed Feb 13 20:10:16 2013 From: rdmurray at bitdance.com (R. David Murray) Date: Wed, 13 Feb 2013 14:10:16 -0500 Subject: [Python-Dev] efficient string concatenation (yep, from 2004) In-Reply-To: <511BC84A.80905@stackless.com> References: <511AD5E1.1060209@stackless.com> <511B81C3.9030905@stackless.com> <511BC84A.80905@stackless.com> Message-ID: <20130213191020.F2F452500B3@webabinitio.net> On Wed, 13 Feb 2013 18:07:22 +0100, Christian Tismer wrote: > I think before getting people to work through long and > complete documentation, it is probably easier to wake their interest > by something like > "Hey, are you doing things this way?" > And then there is a short, concise list of bad and good things, maybe > even dynamic as in WingWare's "Wing Tips" or any better approach. > > From that easily reachable, only a few pages long tabular > collection of short hints and caveats there could be linkage to the > existing, real > documentation that explains things in more detail. > Maybe that could be a way to get people to actually read. There used to be a HOWTO with this goal, but its opinions were considered outdated and/or contentious, and it was deleted: http://docs.python.org/2.6/howto/doanddont.html --David From fijall at gmail.com Wed Feb 13 20:22:14 2013 From: fijall at gmail.com (Maciej Fijalkowski) Date: Wed, 13 Feb 2013 21:22:14 +0200 Subject: [Python-Dev] Marking GC details as CPython-only In-Reply-To: <40757358-85D3-4FBF-B865-2FFBC22DC031@masklinn.net> References: <40757358-85D3-4FBF-B865-2FFBC22DC031@masklinn.net> Message-ID: On Wed, Feb 13, 2013 at 9:09 PM, Xavier Morel wrote: > On 2013-02-13, at 19:48 , Maciej Fijalkowski wrote: > >> Hi >> >> I've tried (and failed) to find what GC details (especially finalizer >> semantics) are CPython only and which ones are not. The best I could >> find was the documentation of __del__ here: >> http://docs.python.org/2/reference/datamodel.html >> >> Things were pypy differs: >> >> * finalizers in pypy will be called only once, even if the object is >> resurrected. I'm not sure if this is detail or we're just plain >> incompatible. >> >> * pypy breaks cycles and runs finalizers in random order (but >> topologically correct), hence gc.garbage is always empty. I *think* >> this part is really just an implementation detail >> >> * we're discussing right now about running multiple finalizers. We >> want to run them in order, but if there is a link a -> b and a becomes >> unreachable, we want to reserve the right to call finalizer a then >> finalizer b, even if a.__del__ resurrects a. What do you think? >> >> Overall, the __del__ is baaad. >> >> Cheers, >> fijal > > There may be one more, although I'm not sure whether it's a GC artifact > or something completely unspecified: if a context manager is part of a > suspended stack (because it's in a generator) when the program > terminates, cpython will run __exit__ but pypy will not > > -- > # -*- encoding: utf-8 -*- > class C(object): > def __enter__(self): > print ("entering") > def __exit__(self, *args): > print ("exiting") > > def gen(): > with C(): > yield > > r = gen() > next(r) > -- > $ python2 test.py > entering > exiting > $ python3 test.py > entering > exiting > $ pypy test.py > entering > $ > -- > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/fijall%40gmail.com I think it's well documented you should not rely on stuff like that being run at the exit of the interpreter. I think we'll try harder to run finalizers at the end of the interpreter (right now we only flush files). File the issue. From solipsis at pitrou.net Wed Feb 13 20:25:40 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 13 Feb 2013 20:25:40 +0100 Subject: [Python-Dev] Marking GC details as CPython-only References: Message-ID: <20130213202540.4e987e17@pitrou.net> On Wed, 13 Feb 2013 20:48:08 +0200 Maciej Fijalkowski wrote: > > Things were pypy differs: > > * finalizers in pypy will be called only once, even if the object is > resurrected. I'm not sure if this is detail or we're just plain > incompatible. I think this should be a detail. > * pypy breaks cycles and runs finalizers in random order (but > topologically correct), hence gc.garbage is always empty. I *think* > this part is really just an implementation detail Agreed. > * we're discussing right now about running multiple finalizers. We > want to run them in order, but if there is a link a -> b and a becomes > unreachable, we want to reserve the right to call finalizer a then > finalizer b, even if a.__del__ resurrects a. What do you think? I think resurrecting objects from __del__ is crazy, so IMO what you suggest is fine. Regards Antoine. From arigo at tunes.org Wed Feb 13 20:30:18 2013 From: arigo at tunes.org (Armin Rigo) Date: Wed, 13 Feb 2013 20:30:18 +0100 Subject: [Python-Dev] Marking GC details as CPython-only In-Reply-To: References: <40757358-85D3-4FBF-B865-2FFBC22DC031@masklinn.net> Message-ID: Hi, On Wed, Feb 13, 2013 at 8:22 PM, Maciej Fijalkowski wrote: > I think it's well documented you should not rely on stuff like that > being run at the exit of the interpreter. Actually right now, at the exit of the interpreter, we just leave the program without caring about running any __del__. This might mean that in a short-running script no __del__ is ever run. I'd add this question to your original list: is it good enough, or should we try harder to run destructors at the exit? A bient?t, Armin. From solipsis at pitrou.net Wed Feb 13 20:40:46 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 13 Feb 2013 20:40:46 +0100 Subject: [Python-Dev] Marking GC details as CPython-only References: <40757358-85D3-4FBF-B865-2FFBC22DC031@masklinn.net> Message-ID: <20130213204046.7ba4239a@pitrou.net> On Wed, 13 Feb 2013 20:30:18 +0100 Armin Rigo wrote: > Hi, > > On Wed, Feb 13, 2013 at 8:22 PM, Maciej Fijalkowski wrote: > > I think it's well documented you should not rely on stuff like that > > being run at the exit of the interpreter. > > Actually right now, at the exit of the interpreter, we just leave the > program without caring about running any __del__. This might mean > that in a short-running script no __del__ is ever run. I'd add this > question to your original list: is it good enough, or should we try > harder to run destructors at the exit? Destructors should be run at exit like they would be in any other finalization situation. Anything else is dangerous, since important resources may not be finalized, committed, or released. (and by destructors I also mean weakref callbacks) Regards Antoine. From regebro at gmail.com Wed Feb 13 21:18:12 2013 From: regebro at gmail.com (Lennart Regebro) Date: Wed, 13 Feb 2013 21:18:12 +0100 Subject: [Python-Dev] Usage of += on strings in loops in stdlib In-Reply-To: References: <20130212220642.71f03220@pitrou.net> <511AB6D6.60805@nedbatchelder.com> <20130212224404.4fa839fc@pitrou.net> <511AEB75.6090808@stackless.com> <511B5442.4020309@hastings.org> <511BCE51.4080805@mrabarnett.plus.com> Message-ID: On Wed, Feb 13, 2013 at 7:06 PM, Maciej Fijalkowski wrote: > I actually wonder. > > There seems to be the consensus to avoid += (to some extent). Can > someone commit the change to urrllib then? I'm talking about reverting > http://bugs.python.org/issue1285086 specifically That's unquoting of URLs, strings that aren't particularly long, normally. And it's not in any tight loops. I'm astonished that any change makes any noticeable speed difference here at all. //Lennart From fijall at gmail.com Wed Feb 13 21:19:22 2013 From: fijall at gmail.com (Maciej Fijalkowski) Date: Wed, 13 Feb 2013 22:19:22 +0200 Subject: [Python-Dev] Marking GC details as CPython-only In-Reply-To: <20130213204046.7ba4239a@pitrou.net> References: <40757358-85D3-4FBF-B865-2FFBC22DC031@masklinn.net> <20130213204046.7ba4239a@pitrou.net> Message-ID: On Wed, Feb 13, 2013 at 9:40 PM, Antoine Pitrou wrote: > On Wed, 13 Feb 2013 20:30:18 +0100 > Armin Rigo wrote: >> Hi, >> >> On Wed, Feb 13, 2013 at 8:22 PM, Maciej Fijalkowski wrote: >> > I think it's well documented you should not rely on stuff like that >> > being run at the exit of the interpreter. >> >> Actually right now, at the exit of the interpreter, we just leave the >> program without caring about running any __del__. This might mean >> that in a short-running script no __del__ is ever run. I'd add this >> question to your original list: is it good enough, or should we try >> harder to run destructors at the exit? > > Destructors should be run at exit like they would be in any other > finalization situation. Anything else is dangerous, since important > resources may not be finalized, committed, or released. > > (and by destructors I also mean weakref callbacks) > > Regards > > Antoine. I think Antoine is right (despite the fact that CPython docs clearly state that __del__s might not be run on the interpreter exit actually) Cheers, fijal From regebro at gmail.com Wed Feb 13 21:40:35 2013 From: regebro at gmail.com (Lennart Regebro) Date: Wed, 13 Feb 2013 21:40:35 +0100 Subject: [Python-Dev] Usage of += on strings in loops in stdlib In-Reply-To: References: <20130212220642.71f03220@pitrou.net> <511AB6D6.60805@nedbatchelder.com> <20130212224404.4fa839fc@pitrou.net> <511AEB75.6090808@stackless.com> <511B5442.4020309@hastings.org> Message-ID: On Wed, Feb 13, 2013 at 4:02 PM, Amaury Forgeot d'Arc wrote: > 2013/2/13 Lennart Regebro >> >> On Wed, Feb 13, 2013 at 3:27 PM, Amaury Forgeot d'Arc >> wrote: >> > Yes, it's jitted. >> >> Admittedly, I have no idea in which cases the JIT kicks in, and what I >> should do to make that happen to make sure I have the best possible >> real-life test cases. > > > PyPy JIT kicks in only after 1000 iterations. Actually, my test code mixed iterations and string length up when printing the results, so the tests I showed was not 100 iterations with 10.000 long string, but 10.000 iterations with 100 long strings. No matter what the iteration/string length is .format() is the slowest or second slowest of all string concatenation methods I've tried and '%s%s' % just marginally faster. This both on PyPy and CPython and irrespective of string length. I'll stick my neck out and say that using formatting for concatenation is probably an anti-pattern. //Lennart From barry at python.org Wed Feb 13 21:43:28 2013 From: barry at python.org (Barry Warsaw) Date: Wed, 13 Feb 2013 15:43:28 -0500 Subject: [Python-Dev] Marking GC details as CPython-only In-Reply-To: References: <40757358-85D3-4FBF-B865-2FFBC22DC031@masklinn.net> Message-ID: <20130213154328.09f3ffb4@anarchist.wooz.org> On Feb 13, 2013, at 08:30 PM, Armin Rigo wrote: >Actually right now, at the exit of the interpreter, we just leave the >program without caring about running any __del__. This might mean >that in a short-running script no __del__ is ever run. I'd add this >question to your original list: is it good enough, or should we try >harder to run destructors at the exit? I've seen *tons* of small Python scripts that don't care about what happens, if anything, at program exit. Some have comments making that quite explicit. Sometimes, they even do so as performance improvements! When you care about start up costs, you often also care about tear down costs. Such scripts just expect that all their resources will get freed when the process exits. Of course, they're not always right (e.g. clean up tmp files), but it's pretty common, and I'm sure not just in Python. OTOH, relying on __del__ to clean up your tmp files seems rather dubious (well, frankly, so does most uses of __del__). Cheers, -Barry From shibturn at gmail.com Wed Feb 13 21:50:51 2013 From: shibturn at gmail.com (Richard Oudkerk) Date: Wed, 13 Feb 2013 20:50:51 +0000 Subject: [Python-Dev] Marking GC details as CPython-only In-Reply-To: <20130213202540.4e987e17@pitrou.net> References: <20130213202540.4e987e17@pitrou.net> Message-ID: On 13/02/2013 7:25pm, Antoine Pitrou wrote: > I think resurrecting objects from __del__ is crazy, so IMO what you > suggest is fine. You mean like subprocess.Popen.__del__? I quite agree. -- Richard From greg.ewing at canterbury.ac.nz Wed Feb 13 22:17:38 2013 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 14 Feb 2013 10:17:38 +1300 Subject: [Python-Dev] efficient string concatenation (yep, from 2004) In-Reply-To: <511B82B2.2000505@pearwood.info> References: <511AD5E1.1060209@stackless.com> <511B82B2.2000505@pearwood.info> Message-ID: <511C02F2.2040709@canterbury.ac.nz> Steven D'Aprano wrote: > The documentation for strings is also clear that you should not rely on > this > optimization: > > ... > > It > can, and does, fail on CPython as well, as it is sensitive to memory > allocation details. If it's that unreliable, why was it ever implemented in the first place? -- Greg From fijall at gmail.com Wed Feb 13 22:52:16 2013 From: fijall at gmail.com (Maciej Fijalkowski) Date: Wed, 13 Feb 2013 23:52:16 +0200 Subject: [Python-Dev] efficient string concatenation (yep, from 2004) In-Reply-To: <511C02F2.2040709@canterbury.ac.nz> References: <511AD5E1.1060209@stackless.com> <511B82B2.2000505@pearwood.info> <511C02F2.2040709@canterbury.ac.nz> Message-ID: On Wed, Feb 13, 2013 at 11:17 PM, Greg Ewing wrote: > Steven D'Aprano wrote: >> >> The documentation for strings is also clear that you should not rely on >> this >> optimization: >> >> ... > >> >> >> It >> can, and does, fail on CPython as well, as it is sensitive to memory >> allocation details. > > > If it's that unreliable, why was it ever implemented > in the first place? > Because someone thought it's a good idea probably and other people asked for a review said +1 :) From tismer at stackless.com Thu Feb 14 00:49:19 2013 From: tismer at stackless.com (Christian Tismer) Date: Thu, 14 Feb 2013 00:49:19 +0100 Subject: [Python-Dev] efficient string concatenation (yep, from 2004) In-Reply-To: References: <511AD5E1.1060209@stackless.com> Message-ID: <0936978B-D261-4C90-ABD5-13F7BD5668DE@stackless.com> Hi Lennart, Sent from my Ei4Steve On Feb 13, 2013, at 8:42, Lennart Regebro wrote: >> Something is needed - a patch for PyPy or for the documentation I guess. > > Not arguing that it wouldn't be good, but I disagree that it is needed. > > This is only an issue when you, as in your proof, have a loop that > does concatenation. This is usually when looping over a list of > strings that should be concatenated together. Doing so in a loop with > concatenation may be the natural way for people new to Python, but the > "natural" way to do it in Python is with a ''.join() call. > > This: > > s = ''.join(('X' for x in xrange(x))) > > Is more than twice as fast in Python 2.7 than your example. It is in > fact also slower in PyPy 1.9 than Python 2.7, but only with a factor > of two: > > Python 2.7: > time for 10000000 concats = 0.887 > Pypy 1.9: > time for 10000000 concats = 1.600 > > (And of course s = 'X'* x takes only a bout a hundredth of the time, > but that's cheating. ;-) > > //Lennart This all does not really concern me, as long as it roughly has the same order of magnitude, or better the same big Oh. I'm not concerned by a constant factor. I'm concerned by a freezing machine that suddenly gets 10000 times slower because the algorithms never explicitly state their algorithmic complexity. ( I think I said this too often, today?) As a side note: Something similar happened to me when somebody used "range" in Python3.3. He ran the same code on Python 2.7. with a crazy effect of having to re-boot: Range() on 2.7 with arguments from some arbitrary input file. A newbie error that was hard to understand, because he was tought thinking 'xrange' when writing 'range'. Hard for me to understand because I am no longer able to make these errors at all, or even expect them. Cheers - Chris From tismer at stackless.com Thu Feb 14 01:13:08 2013 From: tismer at stackless.com (Christian Tismer) Date: Thu, 14 Feb 2013 01:13:08 +0100 Subject: [Python-Dev] efficient string concatenation (yep, from 2004) In-Reply-To: References: <511AD5E1.1060209@stackless.com> <511B82B2.2000505@pearwood.info> <511C02F2.2040709@canterbury.ac.nz> Message-ID: <511C2C14.4050801@stackless.com> On 13.02.13 22:52, Maciej Fijalkowski wrote: > On Wed, Feb 13, 2013 at 11:17 PM, Greg Ewing > wrote: >> Steven D'Aprano wrote: >>> The documentation for strings is also clear that you should not rely on >>> this >>> optimization: >>> >>> ... >>> >>> It >>> can, and does, fail on CPython as well, as it is sensitive to memory >>> allocation details. >> >> If it's that unreliable, why was it ever implemented >> in the first place? >> The _trick_ was very good, the idea was - uhm - arguable. I wished I had objected, but at that time I was only fascinated. -- chris (There are parallels on a larger scale, but I'm shutting up, intentionally.) -- Christian Tismer :^) Software Consulting : Have a break! Take a ride on Python's Karl-Liebknecht-Str. 121 : *Starship* http://starship.python.net/ 14482 Potsdam : PGP key -> http://pgp.uni-mainz.de phone +49 173 24 18 776 fax +49 (30) 700143-0023 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/ From victor.stinner at gmail.com Thu Feb 14 01:21:40 2013 From: victor.stinner at gmail.com (Victor Stinner) Date: Thu, 14 Feb 2013 01:21:40 +0100 Subject: [Python-Dev] Usage of += on strings in loops in stdlib In-Reply-To: <20130213103142.382aa6cc@pitrou.net> References: <20130213103142.382aa6cc@pitrou.net> Message-ID: Hi, I wrote quick hack to expose _PyUnicodeWriter as _string.UnicodeWriter: http://www.haypocalc.com/tmp/string_unicode_writer.patch And I wrote a (micro-)benchmark: http://www.haypocalc.com/tmp/bench_join.py ( The benchmark uses only ASCII string, it would be interesting to test latin1, BMP and non-BMP characters too. ) UnicodeWriter (using the "writer += str" API) is the fastest method in most cases, except for data = ['a'*10**4] * 10**2 (in this case, it's 8x slower!). I guess that the overhead comes for the overallocation which then require to shrink the buffer (shrinking may copy the whole string). The overallocation factor may be adapted depending on the size. If computing the final length is cheap (eg. if it's always the same), it's always faster to use UnicodeWriter with a preallocated buffer. The "UnicodeWriter +=; preallocate" test uses a precomputed length (ok, it's cheating!). I also implemented UnicodeWriter.append method to measure the overhead of a method lookup: it's expensive :-) -- Platform: Linux-3.6.10-2.fc16.x86_64-x86_64-with-fedora-16-Verne Python unicode implementation: PEP 393 Date: 2013-02-14 01:00:06 CFLAGS: -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes SCM: hg revision=659ef9d360ae+ tag=tip branch=default date="2013-02-13 15:25 +0000" CPU model: Intel(R) Core(TM) i7-2600 CPU @ 3.40GHz Python version: 3.4.0a0 (default:659ef9d360ae+, Feb 14 2013, 00:35:19) [GCC 4.6.3 20120306 (Red Hat 4.6.3-2)] Bits: int=32, long=64, long long=64, pointer=64 [ data = ['a'] * 10**2 ] 4.21 us: UnicodeWriter +=; preallocate 4.86 us (+15%): UnicodeWriter append; lookup attr once 4.99 us (+18%): UnicodeWriter += 6.35 us (+51%): str += str 6.45 us (+53%): io.StringIO; lookup attr once 7.02 us (+67%): "".join(list) 7.46 us (+77%): UnicodeWriter append 8.77 us (+108%): io.StringIO [ data = ['abc'] * 10**4 ] 356 us: UnicodeWriter append; lookup attr once 375 us (+5%): UnicodeWriter +=; preallocate 376 us (+6%): UnicodeWriter += 495 us (+39%): io.StringIO; lookup attr once 614 us (+73%): "".join(list) 629 us (+77%): UnicodeWriter append 716 us (+101%): str += str 737 us (+107%): io.StringIO [ data = ['a'*10**4] * 10**1 ] 3.67 us: str += str 3.76 us: UnicodeWriter +=; preallocate 3.95 us (+8%): UnicodeWriter += 4.01 us (+9%): UnicodeWriter append; lookup attr once 4.06 us (+11%): "".join(list) 4.24 us (+15%): UnicodeWriter append 4.59 us (+25%): io.StringIO; lookup attr once 4.77 us (+30%): io.StringIO [ data = ['a'*10**4] * 10**2 ] 41.2 us: UnicodeWriter +=; preallocate 43.8 us (+6%): str += str 45.4 us (+10%): "".join(list) 45.9 us (+11%): io.StringIO; lookup attr once 48.3 us (+17%): io.StringIO 370 us (+797%): UnicodeWriter += 370 us (+798%): UnicodeWriter append; lookup attr once 377 us (+816%): UnicodeWriter append [ data = ['a'*10**4] * 10**4 ] 38.9 ms: UnicodeWriter +=; preallocate 39 ms: "".join(list) 39.1 ms: io.StringIO; lookup attr once 39.4 ms: UnicodeWriter append; lookup attr once 39.5 ms: io.StringIO 39.6 ms: UnicodeWriter += 40.1 ms: str += str 40.1 ms: UnicodeWriter append Victor 2013/2/13 Antoine Pitrou : > Le Wed, 13 Feb 2013 09:02:07 +0100, > Victor Stinner a ?crit : >> I added a _PyUnicodeWriter internal API to optimize str%args and >> str.format(args). It uses a buffer which is overallocated, so it's >> basically like CPython str += str optimization. I still don't know how >> efficient it is on Windows, since realloc() is slow on Windows (at >> least on old Windows versions). >> >> We should add an official and public API to concatenate strings. > > There's io.StringIO already. > > Regards > > Antoine. > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/victor.stinner%40gmail.com From steve at pearwood.info Thu Feb 14 01:25:58 2013 From: steve at pearwood.info (Steven D'Aprano) Date: Thu, 14 Feb 2013 11:25:58 +1100 Subject: [Python-Dev] Usage of += on strings in loops in stdlib In-Reply-To: <511BA0C2.6010606@python.org> References: <511B584E.9020206@python.org> <511B7EB5.3040401@pearwood.info> <511BA0C2.6010606@python.org> Message-ID: <511C2F16.4010901@pearwood.info> On 14/02/13 01:18, Chris Withers wrote: > On 13/02/2013 11:53, Steven D'Aprano wrote: >>> I fixed a performance bug in httplib some years ago by doing the exact >>> opposite; += -> ''.join(). In that case, it changed downloading a file >>> from 20 minutes to 3 seconds. That was likely on Python 2.5. >> >> >> I remember it well. >> >> http://mail.python.org/pipermail/python-dev/2009-August/091125.html >> >> I frequently link to this thread as an example of just how bad repeated >> string concatenation can be, how painful it can be to debug, and how >> even when the optimization is fast on one system, it may fail and be >> slow on another system. > > Amusing is that http://mail.python.org/pipermail/python-dev/2009-August/thread.html#91125 doesn't even list the email where I found the problem... That's because it wasn't solved until the following month. http://mail.python.org/pipermail/python-dev/2009-September/thread.html#91581 -- Steven From steve at pearwood.info Thu Feb 14 01:35:34 2013 From: steve at pearwood.info (Steven D'Aprano) Date: Thu, 14 Feb 2013 11:35:34 +1100 Subject: [Python-Dev] [pypy-dev] efficient string concatenation (yep, from 2004) In-Reply-To: References: <511AD5E1.1060209@stackless.com> <511B81C3.9030905@stackless.com> Message-ID: <511C3156.8060902@pearwood.info> On 14/02/13 01:44, Nick Coghlan wrote: > Deliberately *relying* on the += hack to avoid quadratic runtime is > just plain wrong, and our documentation already says so. +1 I'm not sure that there's any evidence that people in general are *relying* on the += hack. More likely they write the first code they think of, which is +=, and never considered the consequences or test it thoroughly. Even if they test it, they only test it on one version of one implementation on one platform, and likely only with small N. Besides, if you know that N will always be small, then using += is not wrong. I think we have a tendency to sometimes overreact in cases like this. I don't think we need to do any more than we are already doing: the tutor@ and python-list@ mailing lists already try to educate users to use join, the docs recommend to use join, the Internet is filled with code that correctly uses join. What more can we do? I see no evidence that the Python community is awash with coders who write code with atrocious performance characteristics, or at least no more than any other language. -- Steven From solipsis at pitrou.net Thu Feb 14 08:38:58 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 14 Feb 2013 08:38:58 +0100 Subject: [Python-Dev] Usage of += on strings in loops in stdlib In-Reply-To: References: <20130213103142.382aa6cc@pitrou.net> Message-ID: <20130214083858.0380e240@pitrou.net> On Thu, 14 Feb 2013 01:21:40 +0100 Victor Stinner wrote: > > UnicodeWriter (using the "writer += str" API) is the fastest method in > most cases, except for data = ['a'*10**4] * 10**2 (in this case, it's > 8x slower!). I guess that the overhead comes for the overallocation > which then require to shrink the buffer (shrinking may copy the whole > string). The overallocation factor may be adapted depending on the > size. How about testing on Windows? > If computing the final length is cheap (eg. if it's always the same), > it's always faster to use UnicodeWriter with a preallocated buffer. That's not a particularly surprising discovery, is it? ;-) Regards Antoine. From solipsis at pitrou.net Thu Feb 14 10:15:12 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 14 Feb 2013 10:15:12 +0100 Subject: [Python-Dev] Marking GC details as CPython-only References: <20130213202540.4e987e17@pitrou.net> Message-ID: <20130214101512.0e984435@pitrou.net> Le Wed, 13 Feb 2013 20:50:51 +0000, Richard Oudkerk a ?crit : > On 13/02/2013 7:25pm, Antoine Pitrou wrote: > > I think resurrecting objects from __del__ is crazy, so IMO what you > > suggest is fine. > > You mean like subprocess.Popen.__del__? I quite agree. Ouch. I didn't know about that. Regards Antoine. From chris at simplistix.co.uk Thu Feb 14 10:18:46 2013 From: chris at simplistix.co.uk (Chris Withers) Date: Thu, 14 Feb 2013 09:18:46 +0000 Subject: [Python-Dev] build from source on mac uses clang, python.org binary uses gcc Message-ID: <511CABF6.8020808@simplistix.co.uk> Hi All, I've run into "some issues" installing lxml for python 3.3 on my mac: One of the stumbling blocks I've hit is that I built python 3.3 from source (./configure && make && make altinstall), and it used clang: buzzkill:virtualenvs chris$ /src/Python-3.3.0/python.exe Python 3.3.0 (default, Jan 23 2013, 09:56:03) [GCC 4.2.1 Compatible Apple Clang 3.0 (tags/Apple/clang-211.12)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> ...which makes lxml pretty unhappy. However, when I install the binary Python 3.3 from python.org, it's built with GCC: buzzkill:virtualenvs chris$ python3.3 Python 3.3.0 (v3.3.0:bd8afb90ebf2, Sep 29 2012, 01:25:11) [GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> Why the difference? cheers, Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk From chris at python.org Thu Feb 14 10:20:56 2013 From: chris at python.org (Chris Withers) Date: Thu, 14 Feb 2013 09:20:56 +0000 Subject: [Python-Dev] build from source on mac uses clang, python.org binary uses gcc (missing links) In-Reply-To: <511CABF6.8020808@simplistix.co.uk> References: <511CABF6.8020808@simplistix.co.uk> Message-ID: <511CAC78.1040205@python.org> On 14/02/2013 09:18, Chris Withers wrote: > Hi All, > > I've run into "some issues" installing lxml for python 3.3 on my mac: (forgot the links) https://mailman-mail5.webfaction.com/pipermail/lxml/2013-February/006730.html https://mailman-mail5.webfaction.com/pipermail/lxml/2013-February/006731.html cheers, Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk From arigo at tunes.org Thu Feb 14 12:08:34 2013 From: arigo at tunes.org (Armin Rigo) Date: Thu, 14 Feb 2013 12:08:34 +0100 Subject: [Python-Dev] efficient string concatenation (yep, from 2004) In-Reply-To: <511C02F2.2040709@canterbury.ac.nz> References: <511AD5E1.1060209@stackless.com> <511B82B2.2000505@pearwood.info> <511C02F2.2040709@canterbury.ac.nz> Message-ID: Hi Greg, On Wed, Feb 13, 2013 at 10:17 PM, Greg Ewing wrote: > If it's that unreliable, why was it ever implemented > in the first place? I was young and loved hacks and python-dev felt that it was a good idea at the time (http://mail.python.org/pipermail/python-dev/2004-August/046686.html). A bient?t, Armin. From ronaldoussoren at mac.com Thu Feb 14 13:15:55 2013 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Thu, 14 Feb 2013 13:15:55 +0100 Subject: [Python-Dev] build from source on mac uses clang, python.org binary uses gcc In-Reply-To: <511CABF6.8020808@simplistix.co.uk> References: <511CABF6.8020808@simplistix.co.uk> Message-ID: <572C235C-CFA3-47F6-A6F3-0F623E64D389@mac.com> On 14 Feb, 2013, at 10:18, Chris Withers wrote: > Hi All, > > I've run into "some issues" installing lxml for python 3.3 on my mac: > > > One of the stumbling blocks I've hit is that I built python 3.3 from source (./configure && make && make altinstall), and it used clang: > > buzzkill:virtualenvs chris$ /src/Python-3.3.0/python.exe > Python 3.3.0 (default, Jan 23 2013, 09:56:03) > [GCC 4.2.1 Compatible Apple Clang 3.0 (tags/Apple/clang-211.12)] on darwin > Type "help", "copyright", "credits" or "license" for more information. > >>> > > ...which makes lxml pretty unhappy. Why is that? Clang is a working C and C++ compiler. I've noticed in the past that lxml's automatic compilation of libxml2 and libxslt is dodgy on OSX, but haven't had time yet to work on a patch that calculates the right compiler flags instead of hardcoding some guesses in setup.py. Also, libxml2 won't compile on OSX 10.8 at all unless you use the version in the repository (an incompatibility with the pthread headers on 10.8). You get clang instead of GCC when configure detects that clang is the only usable compiler on OSX, where usable means that the compiler is present and is not llvm-gcc. Llvm-gcc contains bugs that miscompile at least parts of the unicode implementation in CPython 3.3 and word on the street is that these bugs won't get fixed unless they are also present in clang. > > However, when I install the binary Python 3.3 from python.org, it's built with GCC: > > buzzkill:virtualenvs chris$ python3.3 > Python 3.3.0 (v3.3.0:bd8afb90ebf2, Sep 29 2012, 01:25:11) > [GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin > Type "help", "copyright", "credits" or "license" for more information. > >>> > > Why the difference? The binary was created on an older release of OSX where gcc is a valid compiler choice. Distutils should still pick clang when it detects that gcc is actually llvm-gcc though, doesn't it do that on your system? BTW. Which OSX release are you using? And which variant of the OSX installer? Ronald P.S. Isn't this a question for python-list? > > cheers, > > Chris > > -- > Simplistix - Content Management, Batch Processing & Python Consulting > - http://www.simplistix.co.uk > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/ronaldoussoren%40mac.com From victor.stinner at gmail.com Thu Feb 14 14:12:39 2013 From: victor.stinner at gmail.com (Victor Stinner) Date: Thu, 14 Feb 2013 14:12:39 +0100 Subject: [Python-Dev] Status of the PEP 433? Message-ID: Hi, I read again all emails related to PEP 433. There are different opposition to this change and I tried to make a summary. My question is: how can I move this PEP forward (accept, defer or reject it)? Can I accept it myself, or do we need a global agreement, or should someone else decide for this PEP (who? Antoine maybe)? -- Most complains concern issues if cloexec is set by default. I don't understand why so many people feel concerned because my PEP proposes *to not change anything to the default behaviour*: cloexec will not be set by default, as it was always the case since Python 1. You have to explicitly use -e command line option, PYTHONCLOEXEC=1 environment variable, or call sys.setdefaultcloexec(True). If you *explicitly* change the default value of the cloexec parameter, you must expect to change the behaviour of your application and maybe discover "bugs" in your application (application incompatible with cloexec set by default). (I don't think that any library would like to play with sys.setdefaultcloexec(), since most developers agree that it might break applications.) -- Ok, let's see all oppositions, presented as a FAQ. (I hope that I didn't misuse email quotes.) Q: the PEP does not solve file descriptor inherance after fork() A: Correct, it only solves issues with exec(), as explained in the PEP. Q: the PEP changes the POSIX behaviour. A: Antoine replied that "Python is not POSIX", Nick wrote "While many of Python's APIs are heavily inspired by POSIX, it still isn't POSIX" (he was talking about integer division). Perl sets cloexec flag by default since its first version, something like 25 years ago, except for explicit POSIX::* calls. Ruby 2 will also set cloexec flag by default. Q: The PEP may break applications. A: Most developers agree that it is (very) unlikely. If file descriptor inherance matters, subprocess must be used (because it rocks!) with pass_fds. If subprocess is used without pass_fds, applications stops working since python 3.2 (since python 3.0 on Windows). pass_fds will clear the close-on-exec flag on the file descriptors with the PEP. If an application breaks because it relies on file descriptor inherance, it's easy to detect it (EBADF error) and simple to fix it (ex: just call sys.setdefaultcloexec(False) at startup, or set explicitly cloexec parameter on the few functions causing trouble). Q: Performance overhead if cloexec is set by default (extra syscalls). A: The PEP now contains a benchmark: the overhead *on a microbenchmark* is between 1 and 3%. I bet that nobody will be able to notice a difference on a real application. Q: The default value of cloexec must not be configurable: "libraries cannot rely on it and have to make file descriptors cloexec on an individual basis, since the default flag can be disabled". A: IMO inherance of file descriptor doesn't matter in most cases, so only a few function must be modified to explicitly set or clear cloexec flag. Said differently: libraries should not care of the default value of the cloexec parameter, which should only be configured by applications (not libraries). Q: The default value of cloexec must not be configurable: "just by looking at this code, you have no way to know whether the file descriptor will be inherited by the child process." A: Guido wrote "Honestly, what happened to the idea that we're all adults here? The likelihood that someone is clueless enough to misuse the flag and yet clueful enough to find out how to change it seems remote -- and they pretty much deserve what they get." -- Another alternative (not listed in the PEP yet) is to leave the posix module (its name depends on the platform: posix, nt or ce) unchanged, and only modify the os module to add cloexec parameter. I implemented this alternative in the "posixmod" branch of the PEP 433 repository. It gives a better traceback on error because the cloexec flag is set in Python, so it's possible to know if the error comes from the creation of the file descriptor or setting the flag in the traceback. (Does it really matter? I never seen an error on setting or clearing the flag.) The problem is that it only concernes posix and os modules: the PEP does also modify socket, io and select modules, which don't have "low" and "high" level modules. So it's possible possible to create a socket using the POSIX behaviour. (Does it matter, it was said that Python is not POSIX.) I'm not convinced that we should do something special for posix/os, I prefer the actual version of the PEP (modify also posix). Victor From just_another_developer at yahoo.de Thu Feb 14 19:25:43 2013 From: just_another_developer at yahoo.de (Developer Developer) Date: Thu, 14 Feb 2013 18:25:43 +0000 (GMT) Subject: [Python-Dev] Question regarding: Lib/_markupbase.py In-Reply-To: <20130212112228.6eb5f0eb@pitrou.net> References: <1360585008.27100.YahooMailNeo@web171404.mail.ir2.yahoo.com> <20130211123735.GA28063@iskra.aviel.ru> <20130212112228.6eb5f0eb@pitrou.net> Message-ID: <1360866343.54571.YahooMailNeo@web171404.mail.ir2.yahoo.com> Sorry, I just thought that: if '>' in rawdata[j:] would do a search, that is, that the implementation of "in" would just reuse/call the implementation of "find" and that the position returned would be used as: -1: not in != -1: in which seemed to me like the easy implementation of "in". That's why I was wondering why to search twice. Now I realize that it doesn't work the way I thought. Thank you for showing me and sorry for the confusion. Best regards, Guido (another Guido ;-) ) ----- Urspr?ngliche Message ----- Von: Antoine Pitrou An: python-dev at python.org CC: Gesendet: 11:22 Dienstag, 12.Februar 2013 Betreff: Re: [Python-Dev] Question regarding: Lib/_markupbase.py Le Mon, 11 Feb 2013 11:02:04 -0800, Guido van Rossum a ?crit : > Warning: see http://bugs.python.org/issue17170. Depending on the > length of the string being scanned and the probability of finding the > specific character, the proposed change could actually be a > *pessimization*. OTOH if the character occurs many times, the slice > will actually cause O(N**2) behavior. So yes, it depends greatly on > the distribution of the input data. That said, the savings are still puny unless you spend your time calling str.find(). Regards Antoine. _______________________________________________ Python-Dev mailing list Python-Dev at python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/just_another_developer%40yahoo.de From ralf at systemexit.de Thu Feb 14 20:05:49 2013 From: ralf at systemexit.de (Ralf Schmitt) Date: Thu, 14 Feb 2013 20:05:49 +0100 Subject: [Python-Dev] Status of the PEP 433? In-Reply-To: (Victor Stinner's message of "Thu, 14 Feb 2013 14:12:39 +0100") References: Message-ID: <87vc9uhg1e.fsf@myhost.lan> Victor Stinner writes: > > Q: The PEP may break applications. > A: Most developers agree that it is (very) unlikely. If file > descriptor inherance matters, subprocess must be used (because it > rocks!) with pass_fds. But that doesn't cover the case for programs that don't fork and really just want to exec another program. when using subprocess, you'll always have two processes running. > If subprocess is used without pass_fds, > applications stops working since python 3.2 (since python 3.0 on > Windows). pass_fds will clear the close-on-exec flag on the file > descriptors with the PEP. If an application breaks because it relies > on file descriptor inherance, it's easy to detect it (EBADF error) and the application may open other files/sockets and the unused file descriptor would be reused. there's no EBADF in that case. From chris at simplistix.co.uk Thu Feb 14 20:03:39 2013 From: chris at simplistix.co.uk (Chris Withers) Date: Thu, 14 Feb 2013 19:03:39 +0000 Subject: [Python-Dev] build from source on mac uses clang, python.org binary uses gcc In-Reply-To: <572C235C-CFA3-47F6-A6F3-0F623E64D389@mac.com> References: <511CABF6.8020808@simplistix.co.uk> <572C235C-CFA3-47F6-A6F3-0F623E64D389@mac.com> Message-ID: <511D350B.1080502@simplistix.co.uk> On 14/02/2013 12:15, Ronald Oussoren wrote: > >> buzzkill:virtualenvs chris$ /src/Python-3.3.0/python.exe >> Python 3.3.0 (default, Jan 23 2013, 09:56:03) >> [GCC 4.2.1 Compatible Apple Clang 3.0 (tags/Apple/clang-211.12)] on darwin >> Type "help", "copyright", "credits" or "license" for more information. >>>>> >> >> ...which makes lxml pretty unhappy. > > Why is that? I'm not lxml's maintainer, I'd suggest asking Stefan... > The binary was created on an older release of OSX where gcc is a valid compiler choice. Distutils > should still pick clang when it detects that gcc is actually llvm-gcc though, doesn't it do that on your system? No. The gcc-based Python 3.3.0 from python.org uses gcc to compile lxml. > BTW. Which OSX release are you using? And which variant of the OSX installer? 10.7.5. If by "OSX Installer" you mean that for Python 3.3.0, then it was the 64-bit one from python.org. > P.S. Isn't this a question for python-list? I think questions about compiling python and how python chooses to compile its extensions, and weird things on particular operating systems related to that, are fair game for python-dev. cheers, Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk From nad at acm.org Thu Feb 14 22:33:35 2013 From: nad at acm.org (Ned Deily) Date: Thu, 14 Feb 2013 13:33:35 -0800 Subject: [Python-Dev] build from source on mac uses clang, python.org binary uses gcc References: <511CABF6.8020808@simplistix.co.uk> <572C235C-CFA3-47F6-A6F3-0F623E64D389@mac.com> <511D350B.1080502@simplistix.co.uk> Message-ID: In article <511D350B.1080502 at simplistix.co.uk>, Chris Withers wrote: > On 14/02/2013 12:15, Ronald Oussoren wrote: > > > >> buzzkill:virtualenvs chris$ /src/Python-3.3.0/python.exe > >> Python 3.3.0 (default, Jan 23 2013, 09:56:03) > >> [GCC 4.2.1 Compatible Apple Clang 3.0 (tags/Apple/clang-211.12)] on darwin > >> Type "help", "copyright", "credits" or "license" for more information. > >>>>> > >> > >> ...which makes lxml pretty unhappy. > > > > Why is that? > > I'm not lxml's maintainer, I'd suggest asking Stefan... > > > The binary was created on an older release of OSX where gcc is a valid > > compiler choice. Distutils > > should still pick clang when it detects that gcc is actually llvm-gcc > > though, doesn't it do that on your system? > > No. The gcc-based Python 3.3.0 from python.org uses gcc to compile lxml. Hmm, that's odd. It should be using clang unless on 10.7 with a current Xcode 4 installed unless something is overriding the compiler selection, e.g. setting CC=clang. (Also, I note you appear to be using an out-of-date clang; Xcode 4.6 and its Command Line Tools are current: $ clang --version Apple LLVM version 4.2 (clang-425.0.24) (based on LLVM 3.2svn Because of the relative newness of clang in the OS X world, it's important to keep up-to-date.) But, installing lxml 3.1.0 (from the tarball you cite) with the Python 3.3.0 64-bit/32-b it install seems to work fine for me. At least, it didn't get any import errors. Looking at your traceback (from https://mailman-mail5.webfaction.com/pipermail/lxml/2013-February/006730. html) appears to show that the lxml is picking up libxml2 and libxslt from /Library/Frameworks/Python.framework/Versions/7.2/lib and or /Library/Frameworks/Python.framework/Versions/7.2/lib. That almost certainly is the root of your problems. I'm not sure what those are but possibly EPD Python distributions. Your 3.3 builds should not be trying to used them to satisfy libraries. Perhaps there are links in /usr/local/*. Removing those links or files in /usr/local should allow you to build lxml with the Apple-supplied libxml2 and libxslt. If you want newer versions or the libs, either build them for 64-/32-bit archs or use a third-party package manager like Homebrew or MacPorts to supply them (and/or supply a Python 3.3). -- Ned Deily, nad at acm.org From ronaldoussoren at mac.com Fri Feb 15 07:45:44 2013 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Fri, 15 Feb 2013 07:45:44 +0100 Subject: [Python-Dev] build from source on mac uses clang, python.org binary uses gcc In-Reply-To: <511D350B.1080502@simplistix.co.uk> References: <511CABF6.8020808@simplistix.co.uk> <572C235C-CFA3-47F6-A6F3-0F623E64D389@mac.com> <511D350B.1080502@simplistix.co.uk> Message-ID: <768265C0-CDFF-4956-8416-76C83DA132C2@mac.com> On 14 Feb, 2013, at 20:03, Chris Withers wrote: > On 14/02/2013 12:15, Ronald Oussoren wrote: >> >>> buzzkill:virtualenvs chris$ /src/Python-3.3.0/python.exe >>> Python 3.3.0 (default, Jan 23 2013, 09:56:03) >>> [GCC 4.2.1 Compatible Apple Clang 3.0 (tags/Apple/clang-211.12)] on darwin >>> Type "help", "copyright", "credits" or "license" for more information. >>>>>> >>> >>> ...which makes lxml pretty unhappy. >> >> Why is that? > > I'm not lxml's maintainer, I'd suggest asking Stefan... I should have been more clear: what error does lxml give when compiling? Ronald From drekin at gmail.com Fri Feb 15 17:41:23 2013 From: drekin at gmail.com (Draic Kin) Date: Fri, 15 Feb 2013 17:41:23 +0100 Subject: [Python-Dev] PYTHONSTARTUP and -i In-Reply-To: References: Message-ID: Hello. Python doc says that the PYTHONSTARTUP file is not read when a script is run with the -i option. Is there a reason for this? Is there a way to get the functionality i.e. to be able to run specified Python file just before the first prompt of interactive interpreter? Use case: I want to run my own repl on Windows, because there's a problem with unicode IO in the console. See http://bugs.python.org/issue1602 . Regards, Drekin -------------- next part -------------- An HTML attachment was scrubbed... URL: From status at bugs.python.org Fri Feb 15 18:07:24 2013 From: status at bugs.python.org (Python tracker) Date: Fri, 15 Feb 2013 18:07:24 +0100 (CET) Subject: [Python-Dev] Summary of Python tracker Issues Message-ID: <20130215170724.1A891568E3@psf.upfronthosting.co.za> ACTIVITY SUMMARY (2013-02-08 - 2013-02-15) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 3845 ( -2) closed 25141 (+53) total 28986 (+51) Open issues with patches: 1664 Issues opened (38) ================== #6083: Reference counting bug in PyArg_ParseTuple and PyArg_ParseTupl http://bugs.python.org/issue6083 reopened by serhiy.storchaka #17162: Py_LIMITED_API needs a PyType_GenericDealloc http://bugs.python.org/issue17162 opened by bfroehle #17163: Fix test discovery for test_file.py http://bugs.python.org/issue17163 opened by zach.ware #17164: MozillaCookieJar does not handle session cookies http://bugs.python.org/issue17164 opened by piotr.dobrogost #17167: python man page contains $Date$ in page footer http://bugs.python.org/issue17167 opened by ned.deily #17170: string method lookup is too slow http://bugs.python.org/issue17170 opened by gvanrossum #17172: Add turtledemo to IDLE menu http://bugs.python.org/issue17172 opened by rhettinger #17174: Posix os.path.join should raise TypeError when passed unusable http://bugs.python.org/issue17174 opened by thomas.scrace #17175: update PEP 430 http://bugs.python.org/issue17175 opened by tshepang #17176: Document imp.NullImporter is NOT used anymore by import http://bugs.python.org/issue17176 opened by brett.cannon #17177: Document/deprecate imp http://bugs.python.org/issue17177 opened by brett.cannon #17178: In all.__doc__ and any.__doc__ need to clarify the behaviour w http://bugs.python.org/issue17178 opened by py.user #17179: Incorrect use of type function in types.new_class http://bugs.python.org/issue17179 opened by cjw296 #17180: shutil copy* unsafe on POSIX - they preserve setuid/setgit bit http://bugs.python.org/issue17180 opened by milko.krachounov #17181: SSLContext.set_servername_callback should be able to set argum http://bugs.python.org/issue17181 opened by grooverdan #17182: signal.default_int_handler should set signal number on the rai http://bugs.python.org/issue17182 opened by pitrou #17183: Small enhancements to Lib/_markupbase.py http://bugs.python.org/issue17183 opened by guido.reina #17184: re.VERBOSE doesn't respect whitespace in '( ?P...)' http://bugs.python.org/issue17184 opened by roysmith #17185: create_autospec http://bugs.python.org/issue17185 opened by cjw296 #17186: no way to introspect registered atexit handlers http://bugs.python.org/issue17186 opened by cjw296 #17187: Python segfaults from improperly formed and called function http://bugs.python.org/issue17187 opened by larry #17188: Document 'from None' in raise statement doc. http://bugs.python.org/issue17188 opened by terry.reedy #17189: Add zip64 support to shutil http://bugs.python.org/issue17189 opened by william.mallard #17191: pdb list shows unexpected code when stack frame includes a try http://bugs.python.org/issue17191 opened by AbramClark #17192: libffi-3.0.12 import http://bugs.python.org/issue17192 opened by doko #17193: Use binary prefixes http://bugs.python.org/issue17193 opened by serhiy.storchaka #17197: c/profile refactoring http://bugs.python.org/issue17197 opened by giampaolo.rodola #17198: dbm.whichdb references unitialized 'ndbm' variable http://bugs.python.org/issue17198 opened by pjenvey #17200: telnetlib.read_until() timeout uses the wrong units http://bugs.python.org/issue17200 opened by Reuben.D'Netto #17201: Use allowZip64=True by default http://bugs.python.org/issue17201 opened by serhiy.storchaka #17202: Add .bat line to .hgeol http://bugs.python.org/issue17202 opened by zach.ware #17203: add long option names to unittest discovery docs http://bugs.python.org/issue17203 opened by chris.jerdonek #17204: argparser's subparsers.add_parser() should accept an ArgumentP http://bugs.python.org/issue17204 opened by chris.jerdonek #17206: Py_XDECREF() expands its argument multiple times http://bugs.python.org/issue17206 opened by dmalcolm #17208: add note/warning about daemon threads http://bugs.python.org/issue17208 opened by pitrou #17209: get_wch() doesn't handle KeyboardInterrupt http://bugs.python.org/issue17209 opened by raymontag #17210: documentation of PyUnicode_Format() states wrong argument type http://bugs.python.org/issue17210 opened by scoder #17211: pkgutil.iter_modules and walk_packages should return a namedtu http://bugs.python.org/issue17211 opened by Ramchandra.Apte Most recent 15 issues with no replies (15) ========================================== #17211: pkgutil.iter_modules and walk_packages should return a namedtu http://bugs.python.org/issue17211 #17210: documentation of PyUnicode_Format() states wrong argument type http://bugs.python.org/issue17210 #17209: get_wch() doesn't handle KeyboardInterrupt http://bugs.python.org/issue17209 #17204: argparser's subparsers.add_parser() should accept an ArgumentP http://bugs.python.org/issue17204 #17203: add long option names to unittest discovery docs http://bugs.python.org/issue17203 #17202: Add .bat line to .hgeol http://bugs.python.org/issue17202 #17191: pdb list shows unexpected code when stack frame includes a try http://bugs.python.org/issue17191 #17188: Document 'from None' in raise statement doc. http://bugs.python.org/issue17188 #17186: no way to introspect registered atexit handlers http://bugs.python.org/issue17186 #17185: create_autospec http://bugs.python.org/issue17185 #17182: signal.default_int_handler should set signal number on the rai http://bugs.python.org/issue17182 #17178: In all.__doc__ and any.__doc__ need to clarify the behaviour w http://bugs.python.org/issue17178 #17177: Document/deprecate imp http://bugs.python.org/issue17177 #17176: Document imp.NullImporter is NOT used anymore by import http://bugs.python.org/issue17176 #17175: update PEP 430 http://bugs.python.org/issue17175 Most recent 15 issues waiting for review (15) ============================================= #17202: Add .bat line to .hgeol http://bugs.python.org/issue17202 #17201: Use allowZip64=True by default http://bugs.python.org/issue17201 #17200: telnetlib.read_until() timeout uses the wrong units http://bugs.python.org/issue17200 #17193: Use binary prefixes http://bugs.python.org/issue17193 #17189: Add zip64 support to shutil http://bugs.python.org/issue17189 #17187: Python segfaults from improperly formed and called function http://bugs.python.org/issue17187 #17180: shutil copy* unsafe on POSIX - they preserve setuid/setgit bit http://bugs.python.org/issue17180 #17175: update PEP 430 http://bugs.python.org/issue17175 #17172: Add turtledemo to IDLE menu http://bugs.python.org/issue17172 #17170: string method lookup is too slow http://bugs.python.org/issue17170 #17163: Fix test discovery for test_file.py http://bugs.python.org/issue17163 #17159: Remove explicit type check from inspect.Signature.from_functio http://bugs.python.org/issue17159 #17148: mingw: nt thread model detection http://bugs.python.org/issue17148 #17145: memoryview(array.array) http://bugs.python.org/issue17145 #17143: trace.py uses _warn without importing it http://bugs.python.org/issue17143 Top 10 most discussed issues (10) ================================= #16997: subtests http://bugs.python.org/issue16997 20 msgs #17170: string method lookup is too slow http://bugs.python.org/issue17170 18 msgs #16278: os.rename documentation slightly inaccurate http://bugs.python.org/issue16278 11 msgs #17130: Add runcall() function to profile.py and cProfile.py http://bugs.python.org/issue17130 9 msgs #17180: shutil copy* unsafe on POSIX - they preserve setuid/setgit bit http://bugs.python.org/issue17180 9 msgs #16612: Integrate "Argument Clinic" specialized preprocessor into CPyt http://bugs.python.org/issue16612 8 msgs #17159: Remove explicit type check from inspect.Signature.from_functio http://bugs.python.org/issue17159 8 msgs #17193: Use binary prefixes http://bugs.python.org/issue17193 8 msgs #17189: Add zip64 support to shutil http://bugs.python.org/issue17189 7 msgs #17200: telnetlib.read_until() timeout uses the wrong units http://bugs.python.org/issue17200 7 msgs Issues closed (52) ================== #4591: 32-bits unsigned user/group identifier http://bugs.python.org/issue4591 closed by serhiy.storchaka #5308: cannot marshal objects with more than 2**31 elements http://bugs.python.org/issue5308 closed by serhiy.storchaka #6426: imaplib.IMAP4 "command illegal in this state" is unhelpful err http://bugs.python.org/issue6426 closed by r.david.murray #6975: symlinks incorrectly resolved on POSIX platforms http://bugs.python.org/issue6975 closed by serhiy.storchaka #7145: UTF-16 BOM is not written by socket.makefile (but is expected http://bugs.python.org/issue7145 closed by serhiy.storchaka #7358: cStringIO not 64-bit safe http://bugs.python.org/issue7358 closed by serhiy.storchaka #7365: grp and pwd should treat uid and gid as unsigned http://bugs.python.org/issue7365 closed by serhiy.storchaka #9067: Use macros from pyctype.h http://bugs.python.org/issue9067 closed by skrah #9874: Message.attach() loses empty attachments http://bugs.python.org/issue9874 closed by r.david.murray #10355: SpooledTemporaryFile's name property is broken http://bugs.python.org/issue10355 closed by serhiy.storchaka #11311: StringIO.readline(0) returns incorrect results http://bugs.python.org/issue11311 closed by serhiy.storchaka #11397: os.path.realpath() may produce incorrect results http://bugs.python.org/issue11397 closed by serhiy.storchaka #11649: startElementNS in xml.sax.saxutils.XMLGenerator ignored encodi http://bugs.python.org/issue11649 closed by serhiy.storchaka #11837: smtplib._quote_periods triggers spurious type error in re.sub http://bugs.python.org/issue11837 closed by serhiy.storchaka #12983: byte string literals with invalid hex escape codes raise Value http://bugs.python.org/issue12983 closed by serhiy.storchaka #13555: cPickle MemoryError when loading large file (while pickle work http://bugs.python.org/issue13555 closed by serhiy.storchaka #13773: Support sqlite3 uri filenames http://bugs.python.org/issue13773 closed by pitrou #15220: Reduce parsing overhead in email.feedparser.BufferedSubFile http://bugs.python.org/issue15220 closed by r.david.murray #16564: email.generator.BytesGenerator fails with bytes payload http://bugs.python.org/issue16564 closed by r.david.murray #16686: audioop overflow issues http://bugs.python.org/issue16686 closed by serhiy.storchaka #16743: mmap on Windows can mishandle files larger than sys.maxsize http://bugs.python.org/issue16743 closed by sbt #16800: tempfile._get_default_tempdir() leaves files behind when HD is http://bugs.python.org/issue16800 closed by serhiy.storchaka #16967: Keyword only argument default values are evaluated before othe http://bugs.python.org/issue16967 closed by python-dev #16996: Reuse shutil.which() in webbrowser module http://bugs.python.org/issue16996 closed by serhiy.storchaka #17043: Invalid read in test_codecs http://bugs.python.org/issue17043 closed by serhiy.storchaka #17052: unittest discovery should use self.testLoader http://bugs.python.org/issue17052 closed by python-dev #17064: Fix sporadic buildbot failures for test_mailbox http://bugs.python.org/issue17064 closed by r.david.murray #17089: Expat parser parses strings only when XML encoding is UTF-8 http://bugs.python.org/issue17089 closed by serhiy.storchaka #17111: test_surrogates of test_fileio fails sometimes on OS X 10.4 http://bugs.python.org/issue17111 closed by ned.deily #17141: random.vonmisesvariate() hangs for large kappa http://bugs.python.org/issue17141 closed by serhiy.storchaka #17147: BytesIO should be mentioned in SpooledTemporaryFile documentat http://bugs.python.org/issue17147 closed by serhiy.storchaka #17149: random.vonmisesvariate() results range is inconsistent for sma http://bugs.python.org/issue17149 closed by mark.dickinson #17152: Array module should support "boolean" natively http://bugs.python.org/issue17152 closed by jcea #17155: logging can raise UnicodeEncodeError http://bugs.python.org/issue17155 closed by vinay.sajip #17156: Tools/i18n/pygettext.py doesn't parse unicode string. http://bugs.python.org/issue17156 closed by serhiy.storchaka #17158: help() module searcher text improvement http://bugs.python.org/issue17158 closed by python-dev #17160: test_urllib2net fails http://bugs.python.org/issue17160 closed by r.david.murray #17161: make install misses the man and the static library http://bugs.python.org/issue17161 closed by ned.deily #17165: Use "except ImportError" instead of bare except in _strptime.p http://bugs.python.org/issue17165 closed by pitrou #17166: Fix the suggested usage of _dummy_thread module http://bugs.python.org/issue17166 closed by r.david.murray #17168: test.support 3.x docs mentions stringio.stringio http://bugs.python.org/issue17168 closed by python-dev #17169: Restore errno in tempfile exceptions http://bugs.python.org/issue17169 closed by serhiy.storchaka #17171: email.encoders.encode7or8bit does not work with binary data http://bugs.python.org/issue17171 closed by r.david.murray #17173: uses of locale-dependent ctype functions http://bugs.python.org/issue17173 closed by pitrou #17190: _FAST opcodes do no range checking http://bugs.python.org/issue17190 closed by rhettinger #17194: operator.attrgetter is slower than a lambda http://bugs.python.org/issue17194 closed by flox #17195: Reading source code from file on exception http://bugs.python.org/issue17195 closed by flox #17196: crash http://bugs.python.org/issue17196 closed by michael.foord #17199: Slightly wrong behavior of logging.StrFormatStyle.usesTime http://bugs.python.org/issue17199 closed by vinay.sajip #17205: In the help() function the order of methods changes http://bugs.python.org/issue17205 closed by r.david.murray #17207: string format precision misbehaving http://bugs.python.org/issue17207 closed by r.david.murray #1470548: Bugfix for #1470540 (XMLGenerator cannot output UTF-16) http://bugs.python.org/issue1470548 closed by serhiy.storchaka From erik.m.bray at gmail.com Fri Feb 15 18:27:41 2013 From: erik.m.bray at gmail.com (Erik Bray) Date: Fri, 15 Feb 2013 12:27:41 -0500 Subject: [Python-Dev] BDFL delegation for PEP 426 (PyPI metadata 1.3) In-Reply-To: References: <20130203122754.7745a1be@pitrou.net> <20130203134853.098aadaa@pitrou.net> <20130203154341.598b063c@pitrou.net> <510EB551.8050705@netwok.org> Message-ID: On Sun, Feb 3, 2013 at 5:24 PM, Vinay Sajip wrote: > ?ric Araujo netwok.org> writes: > >> Looks like we agree that a basic tool able to bootstrap the packaging >> story is needed :) > > Agreed. Just because distutils can't easily/reliably build things that are > better built with SCons/WAF/tup/whatever, doesn't mean that we shouldn't have > the ability to build pure-Python distributions and distributions including C > libs and extensions, with the ability to extend easily by third-party tools. It > just needs to be done in a way which is easy to build on, so the included > battery stays small and simple. Easier said than done, I know :-) > > Regards, > > Vinay Sajip Sorry to revive an old-ish discussion--I'm just catching up on things. But I just wanted to add that distutils is still pretty okay for building reasonably complex projects. Although it does not rise to the level of complexity of Numpy or SciPy, the Astropy project (https://github.com/astropy/astropy) has managed to put together a pretty nice build system on top of mostly-plain distutils (it does use distribute but primarily just for 2to3 support). This has necessitated a number of hacks to overcome shortcomings and bugs in distutils, but most of those shortcomings could probably be fixed in distutils within the framework of a slightly lifted freeze. But in any case I haven't found it worthwhile to switch to something like bento when the batteries included in the stdlib have been mostly Good Enough. Having fewer installation dependencies has also made it significantly easier for non-advanced users to install. Even the distribute requirement doesn't add too much overhead, as most users have it on their systems by default now, and for those who don't distribute_setup.py works okay. TL;DR, strong -1 on the stdlib "getting out of the build business". Also as I think Nick already mentioned one of the wins of Setup-Requires-Dist is to have a standard way to bring in extra build requirements (such as bento) so if we have better support for a feature like that it's not necessary to "bless" any preferred tool. Erik From drekin at gmail.com Fri Feb 15 17:28:40 2013 From: drekin at gmail.com (Draic Kin) Date: Fri, 15 Feb 2013 17:28:40 +0100 Subject: [Python-Dev] PYTHONSTARTUP and -i Message-ID: Hello. Python doc says that the PYTHONSTARTUP file is not read when a script is run with the -i option. Is there a reason for this? Is there a way to get the functionality i.e. to be able to run specified Python file just before the first prompt of interactive interpreter? Use case: I want to run my own repl on Windows, because there's a problem with unicode IO in the console. See http://bugs.python.org/issue1602 . Regards, Drekin -------------- next part -------------- An HTML attachment was scrubbed... URL: From teoliphant at gmail.com Fri Feb 15 08:11:12 2013 From: teoliphant at gmail.com (Travis Oliphant) Date: Fri, 15 Feb 2013 01:11:12 -0600 Subject: [Python-Dev] A new webpage promoting Compiler technology for CPython Message-ID: <3E96A7DD-C8A2-47FB-89C4-D18EB7AEF018@gmail.com> Hey all, With Numba and Blaze we have been doing a lot of work on what essentially is compiler technology and realizing more and more that we are treading on ground that has been plowed before with many other projects. So, we wanted to create a web-site and perhaps even a mailing list or forum where people could coordinate and communicate about compiler projects, compiler tools, and ways to share efforts and ideas. The website is: http://compilers.pydata.org/ This page is specifically for Compiler projects that either integrate with or work directly with the CPython run-time which is why PyPy is not presently listed. The PyPy project is a great project but we just felt that we wanted to explicitly create a collection of links to compilation projects that are accessible from CPython which are likely less well known. But that is just where we started from. The website is intended to be a community website constructed from a github repository. So, we welcome pull requests from anyone who would like to see the website updated to reflect their related project. Jon Riehl (Mython, PyFront, ROFL, and many other interesting projects) and Stephen Diehl (Blaze) and I will be moderating the pull requests to begin with. But, we welcome others with similar interests to participate in that effort of moderation. The github repository is here: https://github.com/pydata/compilers-webpage This is intended to be a community website for information spreading, and so we welcome any and all contributions. Thank you, Travis Oliphant -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefan_ml at behnel.de Fri Feb 15 20:30:22 2013 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 15 Feb 2013 20:30:22 +0100 Subject: [Python-Dev] Proposing "Argument Clinic", a new way of specifying arguments to builtins for CPython In-Reply-To: References: <50BD27CF.1070303@hastings.org> Message-ID: Hi, this reply seems to have drowned, so here it is again. Stefan Behnel, 04.12.2012 16:36: > Why not provide a constructor for signature objects that parses the > signature from a string? For a signature like > > def func(int arg1, float arg2, ExtType arg3, *, > object arg4=None) -> ExtType2: > ... > > you'd just pass in this string: > > (arg1 : int, arg2 : float, arg3 : ExtType, *, arg4=None) -> ExtType2 > > or maybe prefixed by the function name, don't care. Might make it easier to > pass it into the normal parser. > > For more than one alternative input type, use a tuple of types. For builtin > types that are shadowed by C type names, pass "builtins.int" etc. This usage of Py3 annotations for typing isn't currently supported by Cython, but if you'd use the first syntax above, Cython could translate that into a Python function wrapper (almost) straight away. I wonder if that wouldn't be a way to make builtins and stdlib extension modules look and behave more like Python functions, by letting Cython generate their C wrapping code. The non-trivial signatures would also gain some speed when being called, e.g. with keyword arguments. That obviously brings up bootstrapping questions (how to run Cython without builtins?), but they could be worked around by keeping the current code in place until the wrappers are generated, and then replace it by them. Just a thought. Stefan From dholth at gmail.com Fri Feb 15 20:33:02 2013 From: dholth at gmail.com (Daniel Holth) Date: Fri, 15 Feb 2013 14:33:02 -0500 Subject: [Python-Dev] BDFL delegation for PEP 426 (PyPI metadata 1.3) In-Reply-To: References: <20130203122754.7745a1be@pitrou.net> <20130203134853.098aadaa@pitrou.net> <20130203154341.598b063c@pitrou.net> <510EB551.8050705@netwok.org> Message-ID: On Fri, Feb 15, 2013 at 12:27 PM, Erik Bray wrote: > On Sun, Feb 3, 2013 at 5:24 PM, Vinay Sajip > wrote: > > ?ric Araujo netwok.org> writes: > > > >> Looks like we agree that a basic tool able to bootstrap the packaging > >> story is needed :) > > > > Agreed. Just because distutils can't easily/reliably build things that > are > > better built with SCons/WAF/tup/whatever, doesn't mean that we shouldn't > have > > the ability to build pure-Python distributions and distributions > including C > > libs and extensions, with the ability to extend easily by third-party > tools. It > > just needs to be done in a way which is easy to build on, so the included > > battery stays small and simple. Easier said than done, I know :-) > > > > Regards, > > > > Vinay Sajip > > Sorry to revive an old-ish discussion--I'm just catching up on things. > But I just wanted to add that distutils is still pretty okay for > building reasonably complex projects. Although it does not rise to > the level of complexity of Numpy or SciPy, the Astropy project > (https://github.com/astropy/astropy) has managed to put together a > pretty nice build system on top of mostly-plain distutils (it does use > distribute but primarily just for 2to3 support). > > > This has necessitated a number of hacks to overcome shortcomings and > bugs in distutils, but most of those shortcomings could probably be > fixed in distutils within the framework of a slightly lifted freeze. > But in any case I haven't found it worthwhile to switch to something > like bento when the batteries included in the stdlib have been mostly > Good Enough. Having fewer installation dependencies has also made it > significantly easier for non-advanced users to install. Even the > distribute requirement doesn't add too much overhead, as most users > have it on their systems by default now, and for those who don't > distribute_setup.py works okay. > > TL;DR, strong -1 on the stdlib "getting out of the build business". > Also as I think Nick already mentioned one of the wins of > Setup-Requires-Dist is to have a standard way to bring in extra build > requirements (such as bento) so if we have better support for a > feature like that it's not necessary to "bless" any preferred tool. > Distutils is not really going away. We need it to build the existing 28,000 packages. However empirically it seems if you try to write a significant extension to or improvement of distutils then you are likely to get burnt out and switch careers. Instead of literally killing distutils we hope to make it very easy to use other build tools when you need them and not use any build tools at all when you don't. As a thought experiment: what if one of those third party build tools hosted on pypi was distutils itself? What would you need to do to make that happen? The packaging peps PEP-376 and so on are brilliant because they are simple enough to be implemented twice. If we had better ways to separate interface from implementation in Python I'd like to see two of whatever else we come up with for packaging. -------------- next part -------------- An HTML attachment was scrubbed... URL: From paul at boddie.org.uk Fri Feb 15 22:44:14 2013 From: paul at boddie.org.uk (Paul Boddie) Date: Fri, 15 Feb 2013 22:44:14 +0100 Subject: [Python-Dev] A new webpage promoting Compiler technology for CPython Message-ID: <201302152244.14912.paul@boddie.org.uk> Travis Oliphant wrote: > > With Numba and Blaze we have been doing a lot of work on what essentially > is compiler technology and realizing more and more that we are treading on > ground that has been plowed before with many other projects. So, we > wanted to create a web-site and perhaps even a mailing list or forum where > people could coordinate and communicate about compiler projects, compiler > tools, and ways to share efforts and ideas. > > The website is: http://compilers.pydata.org/ This is a rather nice resource. Thank you for letting us know about it! There has been an attempt to record different Python implementations on the Python Wiki, and now that this is available again, I'd like to remind people about it: http://wiki.python.org/moin/PythonImplementations I know that this isn't quite the same thing as a page about compiler technology, but there is a substantial overlap. > This page is specifically for Compiler projects that either integrate with > or work directly with the CPython run-time which is why PyPy is not > presently listed. The PyPy project is a great project but we just felt > that we wanted to explicitly create a collection of links to compilation > projects that are accessible from CPython which are likely less well known. I think that given the scope of the projects listed, it shouldn't preclude PyPy from being listed, really. After all, interoperability with CPython extensions is something of a focus area for PyPy: http://pypy.org/compat.html I don't have an agenda here - I don't use PyPy actively, my only involvement with Shedskin (which is referenced and which can produce CPython extension modules) is in packaging it for Debian, and although I do have a static analysis library I see no pressing need to promote it extensively - but I feel that when it comes to educational resources people should be a bit careful about drawing boundaries that exclude things that would benefit people substantially if they only knew about it. My reason for speaking up about this is that I've had to tell a room full of people who were told to use Cython, NumPy and even plain C to make their Python programs faster that PyPy existed. (Of course, one can justify ignoring the elephant in the room by claiming things like scientific users rely on native libraries or CPython extensions - since "science" is a very broad term, this obviously isn't universally true - but I think people should be entitled to make their own minds up, and I was not completely certain that everyone involved in the case in question was oblivious to PyPy's existence or status.) > But that is just where we started from. The website is intended to be a > community website constructed from a github repository. So, we welcome > pull requests from anyone who would like to see the website updated to > reflect their related project. Jon Riehl (Mython, PyFront, ROFL, and > many other interesting projects) and Stephen Diehl (Blaze) and I will be > moderating the pull requests to begin with. But, we welcome others with > similar interests to participate in that effort of moderation. > > The github repository is here: https://github.com/pydata/compilers-webpage > > This is intended to be a community website for information spreading, and > so we welcome any and all contributions. There is also the python-static-type-checking Google Group: https://groups.google.com/forum/?fromgroups#!forum/python-static-type-checking If no-one beats me to it, I may post details of the site to that group because it may well be of interest to the members. Thanks once again for bringing such information together in one place! Paul From stefan_ml at behnel.de Fri Feb 15 23:36:43 2013 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 15 Feb 2013 23:36:43 +0100 Subject: [Python-Dev] A new webpage promoting Compiler technology for CPython In-Reply-To: <201302152244.14912.paul@boddie.org.uk> References: <201302152244.14912.paul@boddie.org.uk> Message-ID: Paul Boddie, 15.02.2013 22:44: > Travis Oliphant wrote: >> This page is specifically for Compiler projects that either integrate with >> or work directly with the CPython run-time which is why PyPy is not >> presently listed. The PyPy project is a great project but we just felt >> that we wanted to explicitly create a collection of links to compilation >> projects that are accessible from CPython which are likely less well known. > > I think that given the scope of the projects listed, it shouldn't preclude > PyPy from being listed, really. After all, interoperability with CPython > extensions is something of a focus area for PyPy: > > http://pypy.org/compat.html > > I don't have an agenda here - I don't use PyPy actively, my only involvement > with Shedskin (which is referenced and which can produce CPython extension > modules) is in packaging it for Debian, and although I do have a static > analysis library I see no pressing need to promote it extensively - but I > feel that when it comes to educational resources people should be a bit > careful about drawing boundaries that exclude things that would benefit > people substantially if they only knew about it. > > My reason for speaking up about this is that I've had to tell a room full of > people who were told to use Cython, NumPy and even plain C to make their > Python programs faster that PyPy existed. (Of course, one can justify > ignoring the elephant in the room by claiming things like scientific users > rely on native libraries or CPython extensions - since "science" is a very > broad term, this obviously isn't universally true - but I think people should > be entitled to make their own minds up, and I was not completely certain that > everyone involved in the case in question was oblivious to PyPy's existence > or status.) This is off-topic for this list, but the main problem with PyPy is that you'll quickly hit a lot of walls when you try to use it for anything serious in the area. It's true that there is a certain level of interoperability with CPython extensions, but calling it a "focus area" makes it sound bigger than it actually is in my ears. Even trying to get bugs fixed to at least make things work at all often means running against walls on their side. I can tell, trying to port Cython mostly consisted of bugging PyPy developers to fix stuff, which took anything from days to still-not-done, each time. And, by design, PyPy makes it very hard and time consuming to debug it and to try to fix bugs in their code base. So, while I agree that PyPy is worth a try in certain application areas, and can be helpful for some special needs, also in the field of scientific computing, it's lightyears away from a production-ready state in that area. It just doesn't integrate with the huge bulk of software that people use in their daily work. And once you rely on that software, which is hand tuned, well integrated and real-world proven in so many ways, over the time span of way more than a decade, the most visible advantage of PyPy to make Python code run faster becomes almost pointless. In that light, telling people to try PyPy and to drop (most of) their current ecosystem for it doesn't really sound helpful and clearly appears outside of the focus of the web site in question. Stefan From pjenvey at underboss.org Sat Feb 16 01:01:15 2013 From: pjenvey at underboss.org (Philip Jenvey) Date: Fri, 15 Feb 2013 16:01:15 -0800 Subject: [Python-Dev] A new webpage promoting Compiler technology for CPython In-Reply-To: References: <201302152244.14912.paul@boddie.org.uk> Message-ID: On Fri, Feb 15, 2013 at 2:36 PM, Stefan Behnel wrote: > This is off-topic for this list, but the main problem with PyPy is that > you'll quickly hit a lot of walls when you try to use it for anything > serious in the area. It's true that there is a certain level of > interoperability with CPython extensions, but calling it a "focus area" > makes it sound bigger than it actually is in my ears. Even trying to get > bugs fixed to at least make things work at all often means running against > walls on their side. I can tell, trying to port Cython mostly consisted of > bugging PyPy developers to fix stuff, which took anything from days to > still-not-done, each time. And, by design, PyPy makes it very hard and time > consuming to debug it and to try to fix bugs in their code base. > I take issue with how you've described this, Stefan: I recall many on pypy-dev working with you quite a bit on the Cython port. There are some difficult problems involved and the port is not a main focus of the core PyPy team -- there's only so many free cycles. You should ping us (IRC is great) about any outstanding issues. > So, while I agree that PyPy is worth a try in certain application areas, > and can be helpful for some special needs, also in the field of scientific > computing, it's lightyears away from a production-ready state in that area. > It just doesn't integrate with the huge bulk of software that people use in > their daily work. And once you rely on that software, which is hand tuned, > well integrated and real-world proven in so many ways, over the time span > of way more than a decade, the most visible advantage of PyPy to make > Python code run faster becomes almost pointless. In that light, telling > people to try PyPy and to drop (most of) their current ecosystem for it > doesn't really sound helpful and clearly appears outside of the focus of > the web site in question. I disagree that it's pointless. Numba disagrees too: it also attempts to make Python code faster. PyPy is indeed a work in progress in this area, but that doesn't necessarily preclude it from being included. -- Philip Jenvey -------------- next part -------------- An HTML attachment was scrubbed... URL: From paul at boddie.org.uk Sat Feb 16 01:27:00 2013 From: paul at boddie.org.uk (Paul Boddie) Date: Sat, 16 Feb 2013 01:27:00 +0100 Subject: [Python-Dev] A new webpage promoting Compiler technology for CPython Message-ID: <201302160127.01156.paul@boddie.org.uk> Stefan Behnel wrote: > > This is off-topic for this list, but the main problem with PyPy is that > you'll quickly hit a lot of walls when you try to use it for anything > serious in the area. It's true that there is a certain level of > interoperability with CPython extensions, but calling it a "focus area" > makes it sound bigger than it actually is in my ears. It is a focus area. They even have a Web page for it, which I mentioned. > Even trying to get bugs fixed to at least make things work at all often > means running against walls on their side. I can tell, trying to port > Cython mostly consisted of bugging PyPy developers to fix stuff, which took > anything from days to still-not-done, each time. And, by design, PyPy makes > it very hard and time consuming to debug it and to try to fix bugs in their > code base. I'm sure it is tricky to debug PyPy. However, complete compatibility with CPython for Python programs is a goal of the project, and I have no reason to doubt that the project is committed to supporting broader compatibility with CPython. > So, while I agree that PyPy is worth a try in certain application areas, > and can be helpful for some special needs, also in the field of scientific > computing, it's lightyears away from a production-ready state in that area. You are generalising too broadly, which is precisely what I mentioned in my last message. There are also plenty of people doing science who aren't using "numeric" libraries or relying on native code libraries. What I most took exception to was either the lack of awareness of PyPy amongst people giving advice on how to speed up Python programs - not specifically "numeric" programs - to an audience of people who happened to be scientists, or the pretense that the first port of call was to break out Cython or NumPy when the easiest thing for them to do was to try their code on PyPy, provided they could get a package for it. One of my colleagues thought that "you could always rewrite your code in C", which is what he took away from such recommendations, was hilarious advice for speeding up Python programs. You write your Python code in another language! Why not just use C in the first place? Not such a stupid question, really. It's a question that has been hanging around for far too long without a really good answer. > It just doesn't integrate with the huge bulk of software that people use in > their daily work. And once you rely on that software, which is hand tuned, > well integrated and real-world proven in so many ways, over the time span > of way more than a decade, the most visible advantage of PyPy to make > Python code run faster becomes almost pointless. In that light, telling > people to try PyPy and to drop (most of) their current ecosystem for it > doesn't really sound helpful and clearly appears outside of the focus of > the web site in question. For a very long time, and even now to some extent, you could say the same thing about Python 3. Personally, I suspect that some people have such a problem with PyPy that they would rather not mention it or see it mentioned. That's obviously a shame because not only does PyPy offer people an alternative, but it also encourages the development of software in Python, some of which appears on the resource being discussed, that would otherwise be written in C because "that's what we would usually write this kind of software in". And although I don't actively use PyPy, mostly because of what the default Python implementation is on the platforms I use, I would like to see more actual Python code written. But I'm certainly not going to dwell on this matter any more than I've already done. I'm sure people will get something from the referenced resource regardless of whether any particular project is mentioned by it or not. Paul From dholth at gmail.com Sat Feb 16 05:06:23 2013 From: dholth at gmail.com (Daniel Holth) Date: Fri, 15 Feb 2013 23:06:23 -0500 Subject: [Python-Dev] Submitting PEP 425 (compatibility tags) Message-ID: This is the improved compatibility tags PEP 425, specifying how part of the Wheel PEP 427 filenames work. Last time we discussed whether replacing . with _ was ugly but I concluded it was harmless. Submitted for your consideration, PEP: 425 Title: Compatibility Tags for Built Distributions Version: $Revision$ Last-Modified: 07-Aug-2012 Author: Daniel Holth BDFL-Delegate: Nick Coghlan Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 27-Jul-2012 Python-Version: 3.4 Post-History: 8-Aug-2012, 18-Oct-2012, 15-Feb-2013 Abstract ======== This PEP specifies a tagging system to indicate with which versions of Python a built or binary distribution is compatible. A set of three tags indicate which Python implementation and language version, ABI, and platform a built distribution requires. The tags are terse because they will be included in filenames. PEP Editor's Note ================= While the naming scheme described in this PEP will not be supported directly in the standard library until Python 3.4 at the earliest, draft implementations may be made available in third party projects. Rationale ========= Today "python setup.py bdist" generates the same filename on PyPy and CPython, but an incompatible archive, making it inconvenient to share built distributions in the same folder or index. Instead, built distributions should have a file naming convention that includes enough information to decide whether or not a particular archive is compatible with a particular implementation. Previous efforts come from a time where CPython was the only important implementation and the ABI was the same as the Python language release. This specification improves upon the older schemes by including the Python implementation, language version, ABI, and platform as a set of tags. By comparing the tags it supports with the tags listed by the distribution, an installer can make an educated decision about whether to download a particular built distribution without having to read its full metadata. Overview ======== The tag format is {python tag}-{abi tag}-{platform tag} python tag ?py27?, ?cp33? abi tag ?cp32dmu?, ?none? platform tag ?linux_x86_64?, ?any? For example, the tag py27-none-any indicates compatible with Python 2.7 (any Python 2.7 implementation) with no abi requirement, on any platform. Use === The `wheel` built package format includes these tags in its filenames, of the form ``{distribution}-{version}(-{build tag})?-{python tag}-{abi tag}-{platform tag}.whl``. Other package formats may have their own conventions. Details ======= Python Tag ---------- The Python tag indicates the implementation and version required by a distribution. Major implementations have abbreviated codes, initially: * py: Generic Python (does not require implementation-specific features) * cp: CPython * ip: IronPython * pp: PyPy * jy: Jython Other Python implementations should use `sys.implementation.name`. The version is `py_version_nodot`. CPython gets away with no dot, but if one is needed the underscore `_` is used instead. PyPy should probably use its own versions here `pp18`, `pp19`. The version can be just the major version `2` or `3` `py2`, `py3` for many pure-Python distributions. Importantly, major-version-only tags like `py2` and `py3` are not shorthand for `py20` and `py30`. Instead, these tags mean the packager intentionally released a cross-version-compatible distribution. A single-source Python 2/3 compatible distribution can use the compound tag `py2.py3`. See `Compressed Tag Sets`, below. ABI Tag ------- The ABI tag indicates which Python ABI is required by any included extension modules. For implementation-specific ABIs, the implementation is abbreviated in the same way as the Python Tag, e.g. `cp33d` would be the CPython 3.3 ABI with debugging. The CPython stable ABI is `abi3` as in the shared library suffix. Implementations with a very unstable ABI may use the first 6 bytes (as 8 base64-encoded characters) of the SHA-256 hash of ther source code revision and compiler flags, etc, but will probably not have a great need to distribute binary distributions. Each implementation's community may decide how to best use the ABI tag. Platform Tag ------------ The platform tag is simply `distutils.util.get_platform()` with all hyphens `-` and periods `.` replaced with underscore `_`. * win32 * linux_i386 * linux_x86_64 Use === The tags are used by installers to decide which built distribution (if any) to download from a list of potential built distributions. The installer maintains a list of (pyver, abi, arch) tuples that it will support. If the built distribution's tag is `in` the list, then it can be installed. For example, an installer running under CPython 3.3 on a linux_x86_64 system might support:: 1. cp33-cp33m-linux_x86_64 2. cp33-abi3-linux_x86_64 3. cp33-none-linux_x86_64 4. cp33-none-any 5. cp3-none-any 6. cp32-none-any 7. cp31-none-any 8. cp30-none-any 9. py33-none-any 10. py3-none-any 11. py32-none-any 12. py31-none-any 13. py30-none-any The list is in order from most-preferred (a distribution with a compiled extension module, built for the current version of Python) to least-preferred (a pure-Python distribution built with an older version of Python). A user could instruct their installer to fall back to building from an sdist more or less often by configuring this list of tags; for example, a user could include only the `*-none-any` tags to only download built packages that advertise themselves as being pure Python. Rarely there will be more than one supported built distribution for a particular version of a package. For example, a packager could release a package tagged `cp33-abi3-linux_x86_64` that contains an optional C extension and the same distribution tagged `py3-none-any` that does not. The index of the tag in the supported tags list breaks the tie, and the package with the C extension is installed in preference to the package without because that tag appears first in the list. Compressed Tag Sets =================== To allow for compact filenames of bdists that work with more than one compatibility tag triple, each tag in a filename can instead be a '.'-separated, sorted, set of tags. For example, pip, a pure-Python package that is written to run under Python 2 and 3 with the same source code, could distribute a bdist with the tag `py2.py3-none-any`. The full list of simple tags is:: for x in pytag.split('.'): for y in abitag.split('.'): for z in archtag.split('.'): yield '-'.join((x, y, z)) A bdist format that implements this scheme should include the expanded tags in bdist-specific metadata. This compression scheme can generate large numbers of unsupported tags and "impossible" tags that are supported by no Python implementation e.g. "cp33-cp31u-win64", so use it sparingly. FAQ === What tags are used by default? Tools should use the most-preferred architecture dependent tag e.g. `cp33-cp33m-win32` or the most-preferred pure python tag e.g. `py33-none-any` by default. If the packager overrides the default it indicates that they intended to provide cross-Python compatibility. Can I have a tag `py32+` to indicate a minimum Python minor release? No. Inspect the Trove classifiers to determine this level of cross-release compatibility. Similar to the announcements "beaglevote versions 3.2 and above no longer supports Python 1.52", you will have to manually keep track of the maximum (PEP-386) release that still supports your version of Python. Why isn't there a `.` in the Python version number? CPython has lasted 20+ years without a 3-digit major release. This should continue for some time. Other implementations may use _ as a delimeter, since both - and . delimit the surrounding filename. Who will maintain the registry of abbreviated implementations? New two-letter abbreviations can be requested on the python-dev mailing list. As a rule of thumb, abbreviations are reserved for the current 4 most prominent implementations. Does the compatibility tag go into METADATA or PKG-INFO? No. The compatibility tag is part of the built distribution's metadata. METADATA / PKG-INFO should be valid for an entire distribution, not a single build of that distribution. Why didn't you mention my favorite Python implementation? The abbreviated tags facilitate sharing compiled Python code in a public index. Your Python implementation can use this specification too, but with longer tags. Recall that all "pure Python" built distributions just use 'py'. References ========== .. [1] Egg Filename-Embedded Metadata ( http://peak.telecommunity.com/DevCenter/EggFormats#filename-embedded-metadata ) .. [2] Creating Built Distributions (http://docs.python.org/distutils/builtdist.html) .. [3] PEP 3147 -- PYC Repository Directories (http://www.python.org/dev/peps/pep-3147/) Acknowledgements ================ The author thanks Paul Moore, Nick Coghlan, Mark Abramowitz, and Mr. Michele Lacchia for their valuable advice and help with this effort. Copyright ========= This document has been placed in the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End: -------------- next part -------------- An HTML attachment was scrubbed... URL: From dholth at gmail.com Sat Feb 16 05:21:27 2013 From: dholth at gmail.com (Daniel Holth) Date: Fri, 15 Feb 2013 23:21:27 -0500 Subject: [Python-Dev] PEP 427: wheel Message-ID: This is PEP 427: Wheel. A binary package format for Python. Because newegg was taken. Since the last submission, the signature specification has been made clearly optional / informative and does not attempt to specify any specific signing algorithm or how the signatures would fit into a security scheme that would necessarily exist outside of a single archive. JWS signatures are used in their current form by OpenID Connect and Mozilla Personas and are a useful way to implement basically raw public or secret key signatures. The embedded signature scheme in wheel should also not affect the current effort to define end-to-end security for PyPI in any way; it might be a useful complement for packages that are not hosted on PyPI at all. This version also does not depend on the in-process Metadata PEP 426. It has been cleaned up in several places regarding which PEPs it references to describe its contents. The WHEEL metadata file should contain all the information in the wheel filename itself, and the non-alphanumeric/unicode filename escaping rules are made official. Thanks. For your consideration, PEP: 427 Title: The Wheel Binary Package Format 0.1 Version: $Revision$ Last-Modified: $Date$ Author: Daniel Holth BDFL-Delegate: Nick Coghlan Discussions-To: Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 20-Sep-2012 Post-History: 18-Oct-2012, 15-Feb-2013 Abstract ======== This PEP describes a built-package format for Python called "wheel". A wheel is a ZIP-format archive with a specially formatted file name and the ``.whl`` extension. It contains a single distribution nearly as it would be installed according to PEP 376 with a particular installation scheme. Although a specialized installer is recommended, a wheel file may be installed by simply unpacking into site-packages with the standard 'unzip' tool while preserving enough information to spread its contents out onto their final paths at any later time. Note ==== This draft PEP describes version 0.1 of the "wheel" format. When the PEP is accepted, the version will be changed to 1.0. (The major version is used to indicate potentially backwards-incompatible changes to the format.) Rationale ========= Python needs a package format that is easier to install than sdist. Python's sdist packages are defined by and require the distutils and setuptools build systems, running arbitrary code to build-and-install, and re-compile, code just so it can be installed into a new virtualenv. This system of conflating build-install is slow, hard to maintain, and hinders innovation in both build systems and installers. Wheel attempts to remedy these problems by providing a simpler interface between the build system and the installer. The wheel binary package format frees installers from having to know about the build system, saves time by amortizing compile time over many installations, and removes the need to install a build system in the target environment. Details ======= Installing a wheel 'distribution-1.0.py32.none.any.whl' ------------------------------------------------------- Wheel installation notionally consists of two phases: - Unpack. a. Parse ``distribution-1.0.dist-info/WHEEL``. b. Check that installer is compatible with Wheel-Version. Warn if minor version is greater, abort if major version is greater. c. If Root-Is-Purelib == 'true', unpack archive into purelib (site-packages). d. Else unpack archive into platlib (site-packages). - Spread. a. Unpacked archive includes ``distribution-1.0.dist-info/`` and (if there is data) ``distribution-1.0.data/``. b. Move each subtree of ``distribution-1.0.data/`` onto its destination path. Each subdirectory of ``distribution-1.0.data/`` is a key into a dict of destination directories, such as ``distribution-1.0.data/(purelib|platlib|headers|scripts|data)``. The initially supported paths are taken from ``distutils.command.install``. c. If applicable, update scripts starting with ``#!python`` to point to the correct interpreter. d. Update ``distribution-1.0.dist.info/RECORD`` with the installed paths. e. Remove empty ``distribution-1.0.data`` directory. f. Compile any installed .py to .pyc. (Uninstallers should be smart enough to remove .pyc even if it is not mentioned in RECORD.) Recommended installer features '''''''''''''''''''''''''''''' Rewrite ``#!python``. In wheel, scripts are packaged in ``{distribution}-{version}.data/scripts/``. If the first line of a file in ``scripts/`` starts with exactly ``b'#!python'``, rewrite to point to the correct interpreter. Unix installers may need to add the +x bit to these files if the archive was created on Windows. Generate script wrappers. In wheel, scripts packaged on Unix systems will certainly not have accompanying .exe wrappers. Windows installers may want to add them during install. File Format ----------- File name convention '''''''''''''''''''' The wheel filename is ``{distribution}-{version}(-{build tag})?-{python tag}-{abi tag}-{platform tag}.whl``. distribution Distribution name, e.g. 'django', 'pyramid'. version Distribution version, e.g. 1.0. build tag Optional build number. Must start with a digit. A tie breaker if two wheels have the same version. Sort as the empty string if unspecified, else sort the initial digits as a number, and the remainder lexicographically. language implementation and version tag E.g. 'py27', 'py2', 'py3'. abi tag E.g. 'cp33m', 'abi3', 'none'. platform tag E.g. 'linux_x86_64', 'any'. For example, ``distribution-1.0-1-py27-none-any.whl`` is the first build of a package called 'distribution', and is compatible with Python 2.7 (any Python 2.7 implementation), with no ABI (pure Python), on any CPU architecture. The last three components of the filename before the extension are called "compatibility tags." The compatibility tags express the package's basic interpreter requirements and are detailed in PEP 425. Escaping and Unicode '''''''''''''''''''' Each component of the filename is escaped by replacing runs of non-alphanumeric characters with an underscore ``_``:: re.sub("[^\w\d.]+", "_", distribution, re.UNICODE) The filename is Unicode. It will be some time before the tools are updated to support non-ASCII filenames, but they are supported in this specification. File contents ''''''''''''' The contents of a wheel file, where {distribution} is replaced with the name of the package, e.g. ``beaglevote`` and {version} is replaced with its version, e.g. ``1.0.0``, consist of: #. ``/``, the root of the archive, contains all files to be installed in ``purelib`` or ``platlib`` as specified in ``WHEEL``. ``purelib`` and ``platlib`` are usually both ``site-packages``. #. ``{distribution}-{version}.dist-info/`` contains metadata. #. ``{distribution}-{version}.data/`` contains one subdirectory for each non-empty install scheme key not already covered, where the subdirectory name is an index into a dictionary of install paths (e.g. ``data``, ``scripts``, ``include``, ``purelib``, ``platlib``). #. Python scripts must appear in ``scripts`` and begin with exactly ``b'#!python'`` in order to enjoy script wrapper generation and ``#!python`` rewriting at install time. They may have any or no extension. #. ``{distribution}-{version}.dist-info/METADATA`` is Metadata version 1.1 or greater (PEP 314, PEP 345, PEP 426) format metadata. #. ``{distribution}-{version}.dist-info/WHEEL`` is metadata about the archive itself in the same basic key: value format:: Wheel-Version: 0.1 Generator: bdist_wheel 0.7 Root-Is-Purelib: true Tag: py2-none-any Tag: py3-none-any Build: 1 #. ``Wheel-Version`` is the version number of the Wheel specification. ``Generator`` is the name and optionally the version of the software that produced the archive. ``Root-Is-Purelib`` is true if the top level directory of the archive should be installed into purelib; otherwise the root should be installed into platlib. ``Tag`` is the wheel's expanded compatibility tags; in the example the filename would contain ``py2.py3-none-any``. ``Build`` is the build number and is omitted if there is no build number. #. A wheel installer should warn if Wheel-Version is greater than the version it supports, and must fail if Wheel-Version has a greater major version than the version it supports. #. Wheel, being an installation format that is intended to work across multiple versions of Python, does not generally include .pyc files. #. Wheel does not contain setup.py or setup.cfg. This version of the wheel specification is based on the distutils install schemes and does not define how to install files to other locations. The layout offers a superset of the functionality provided by the existing wininst and egg binary formats. The .dist-info directory ^^^^^^^^^^^^^^^^^^^^^^^^ #. Wheel .dist-info directories include at a minimum METADATA, WHEEL, and RECORD. #. METADATA is the package metadata, the same format as PKG-INFO as found at the root of sdists. #. WHEEL is the wheel metadata specific to a build of the package. #. RECORD is a list of (almost) all the files in the wheel and their secure hashes. Unlike PEP 376, every file except RECORD, which cannot contain a hash of itself, must include its hash. The hash algorithm must be sha256 or better; specifically, md5 and sha1 are not permitted, as signed wheel files rely on the strong hashes in RECORD to validate the integrity of the archive. #. PEP 376's INSTALLER and REQUESTED are not included in the archive. #. RECORD.jws is used for digital signatures. It is not mentioned in RECORD. #. RECORD.p7s is allowed as a courtesy to anyone who would prefer to use S/MIME signatures to secure their wheel files. It is not mentioned in RECORD. #. During extraction, wheel installers verify all the hashes in RECORD against the file contents. Apart from RECORD and its signatures, installation will fail if any file in the archive is not both mentioned and correctly hashed in RECORD. The .data directory ^^^^^^^^^^^^^^^^^^^ Any file that is not normally installed inside site-packages goes into the .data directory, named as the .dist-info directory but with the .data/ extension:: distribution-1.0.dist-info/ distribution-1.0.data/ The .data directory contains subdirectories with the scripts, headers, documentation and so forth from the distribution. During installation the contents of these subdirectories are moved onto their destination paths. Signed wheel files ------------------ Wheel files include an extended RECORD that enables digital signatures. PEP 376's RECORD is altered to include a secure hash ``digestname=urlsafe_b64encode_nopad(digest)`` (urlsafe base64 encoding with no trailing = characters) as the second column instead of an md5sum. All possible entries are hashed, including any generated files such as .pyc files, but not RECORD which cannot contain its own hash. For example:: file.py,sha256=AVTFPZpEKzuHr7OvQZmhaU3LvwKz06AJw8mT\_pNh2yI,3144 distribution-1.0.dist-info/RECORD,, The signature file(s) RECORD.jws and RECORD.p7s are not mentioned in RECORD at all since they can only be added after RECORD is generated. Every other file in the archive must have a correct hash in RECORD or the installation will fail. If JSON web signatures are used, one or more JSON Web Signature JSON Serialization (JWS-JS) signatures is stored in a file RECORD.jws adjacent to RECORD. JWS is used to sign RECORD by including the SHA-256 hash of RECORD as the signature's JSON payload:: { "hash": "sha256=ADD-r2urObZHcxBW3Cr-vDCu5RJwT4CaRTHiFmbcIYY" } If RECORD.p7s is used, it must contain a detached S/MIME format signature of RECORD. A wheel installer is not required to understand digital signatures but MUST verify the hashes in RECORD against the extracted file contents. When the installer checks file hashes against RECORD, a separate signature checker only needs to establish that RECORD matches the signature. See - http://self-issued.info/docs/draft-ietf-jose-json-web-signature.html - http://self-issued.info/docs/draft-jones-jose-jws-json-serialization.html - http://self-issued.info/docs/draft-ietf-jose-json-web-key.html - http://self-issued.info/docs/draft-jones-jose-json-private-key.html Comparison to .egg ------------------ #. Wheel is an installation format; egg is importable. Wheel archives do not need to include .pyc and are less tied to a specific Python version or implementation. Wheel can install (pure Python) packages built with previous versions of Python so you don't always have to wait for the packager to catch up. #. Wheel uses .dist-info directories; egg uses .egg-info. Wheel is compatible with the new world of Python packaging and the new concepts it brings. #. Wheel has a richer file naming convention for today's multi-implementation world. A single wheel archive can indicate its compatibility with a number of Python language versions and implementations, ABIs, and system architectures. Historically the ABI has been specific to a CPython release, wheel is ready for the stable ABI. #. Wheel is lossless. The first wheel implementation bdist_wheel always generates egg-info, and then converts it to a .whl. It is also possible to convert existing eggs and bdist_wininst distributions. #. Wheel is versioned. Every wheel file contains the version of the wheel specification and the implementation that packaged it. Hopefully the next migration can simply be to Wheel 2.0. #. Wheel is a reference to the other Python. FAQ === Wheel defines a .data directory. Should I put all my data there? This specification does not have an opinion on how you should organize your code. The .data directory is just a place for any files that are not normally installed inside ``site-packages`` or on the PYTHONPATH. In other words, you may continue to use ``pkgutil.get_data(package, resource)`` even though *those* files will usually not be distributed in *wheel's* ``.data`` directory. Why does wheel include attached signatures? Attached signatures are more convenient than detached signatures because they travel with the archive. Since only the individual files are signed, the archive can be recompressed without invalidating the signature or individual files can be verified without having to download the whole archive. Why does wheel allow JWS signatures? The JOSE specifications of which JWS is a part are designed to be easy to implement, a feature that is also one of wheel's primary design goals. JWS yields a useful, concise pure-Python implementation. Why does wheel also allow S/MIME signatures? S/MIME signatures are allowed for users who need or want to use existing public key infrastructure with wheel. Signed packages are only a basic building block in a secure package update system. Wheel only provides the building block. Appendix ======== Example urlsafe-base64-nopad implementation:: # urlsafe-base64-nopad for Python 3 import base64 def urlsafe_b64encode_nopad(data): return base64.urlsafe_b64encode(data).rstrip(b'=') def urlsafe_b64decode_nopad(data): pad = b'=' * (4 - (len(data) & 3)) return base64.urlsafe_b64decode(data + pad) Copyright ========= This document has been placed into the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End: -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefan_ml at behnel.de Sat Feb 16 06:41:18 2013 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sat, 16 Feb 2013 06:41:18 +0100 Subject: [Python-Dev] A new webpage promoting Compiler technology for CPython In-Reply-To: References: <201302152244.14912.paul@boddie.org.uk> Message-ID: Philip Jenvey, 16.02.2013 01:01: > On Fri, Feb 15, 2013 at 2:36 PM, Stefan Behnel wrote: >> So, while I agree that PyPy is worth a try in certain application areas, >> and can be helpful for some special needs, also in the field of scientific >> computing, it's lightyears away from a production-ready state in that area. >> It just doesn't integrate with the huge bulk of software that people use in >> their daily work. And once you rely on that software, which is hand tuned, >> well integrated and real-world proven in so many ways, over the time span >> of way more than a decade, the most visible advantage of PyPy to make >> Python code run faster becomes almost pointless. In that light, telling >> people to try PyPy and to drop (most of) their current ecosystem for it >> doesn't really sound helpful and clearly appears outside of the focus of >> the web site in question. > > I disagree that it's pointless. Numba disagrees too: it also attempts to > make Python code faster. That's not even the main goal AFAIK, but in any case, it does it from the very inside of the existing ecosystem, like all of the compilers (and related software) that are listed on the website. That's the whole point. > PyPy is indeed a work in progress in this area, but that doesn't > necessarily preclude it from being included. That may be a matter of POV, but as long as PyPy fails to integrate (and you just called that "not a main focus"), I find it hard to defend its relevance. Stefan From steve at pearwood.info Sat Feb 16 07:13:03 2013 From: steve at pearwood.info (Steven D'Aprano) Date: Sat, 16 Feb 2013 17:13:03 +1100 Subject: [Python-Dev] A new webpage promoting Compiler technology for CPython In-Reply-To: References: <201302152244.14912.paul@boddie.org.uk> Message-ID: <511F236F.70800@pearwood.info> On 16/02/13 16:41, Stefan Behnel wrote: >> PyPy is indeed a work in progress in this area, but that doesn't >> necessarily preclude it from being included. > > That may be a matter of POV, but as long as PyPy fails to integrate (and > you just called that "not a main focus"), I find it hard to defend its > relevance. Stefan, you're talking about a website which lists projects such as a Forth- like language in Python, a bytecode transformer, a project "Peregine-Falcon" which appears to be a one-man experimental implementation of Python in C++, ByteRun, a "pure-Python implementation of a Python bytecode execution virtual machine", and SPARK, which has been stuck in version 0.7 pre-alpha for years. I am astonished that you think PyPy is too immature or niche for this list. http://compilers.pydata.org/ -- Steven From stefan_ml at behnel.de Sat Feb 16 08:14:33 2013 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sat, 16 Feb 2013 08:14:33 +0100 Subject: [Python-Dev] A new webpage promoting Compiler technology for CPython In-Reply-To: <511F236F.70800@pearwood.info> References: <201302152244.14912.paul@boddie.org.uk> <511F236F.70800@pearwood.info> Message-ID: Steven D'Aprano, 16.02.2013 07:13: > On 16/02/13 16:41, Stefan Behnel wrote: > >>> PyPy is indeed a work in progress in this area, but that doesn't >>> necessarily preclude it from being included. >> >> That may be a matter of POV, but as long as PyPy fails to integrate (and >> you just called that "not a main focus"), I find it hard to defend its >> relevance. > > Stefan, you're talking about a website which lists projects such as a Forth- > like language in Python, a bytecode transformer, a project "Peregine-Falcon" > which appears to be a one-man experimental implementation of Python in C++, > ByteRun, a "pure-Python implementation of a Python bytecode execution > virtual machine", and SPARK, which has been stuck in version 0.7 pre-alpha > for years. I am astonished that you think PyPy is too immature or niche for > this list. > > http://compilers.pydata.org/ Hmm, I don't have the feeling that this discussion is leading anywhere (especially not on this list). After all, it's quite possible to fire up a PyPy runtime from a CPython runtime and have them talk to each other, letting each one do what it's good at. That doesn't make PyPy a "compiler for CPython", but at least it means that it doesn't fail completely to integrate with the rest of the world. There are arguments for both sides, which means in dubio pro reo, I guess. Stefan From ncoghlan at gmail.com Sat Feb 16 08:28:20 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 16 Feb 2013 17:28:20 +1000 Subject: [Python-Dev] BDFL delegation for PEP 426 (PyPI metadata 1.3) In-Reply-To: References: <20130203122754.7745a1be@pitrou.net> <20130203134853.098aadaa@pitrou.net> <20130203154341.598b063c@pitrou.net> <510EB551.8050705@netwok.org> Message-ID: On Sat, Feb 16, 2013 at 3:27 AM, Erik Bray wrote: > TL;DR, strong -1 on the stdlib "getting out of the build business". > Also as I think Nick already mentioned one of the wins of > Setup-Requires-Dist is to have a standard way to bring in extra build > requirements (such as bento) so if we have better support for a > feature like that it's not necessary to "bless" any preferred tool. I've described below where I personally hope we can get to by the time of Python 3.4. Please, if anyone has any comments on the below, send them to me via private email. I am only publishing this half-baked version to allay any concerns people may have that distutils itself might be going away in the foreseeable future (such concerns are perfectly understandable, given the many harsh things that have been said about distutils and the use of setup.py to drive the build process). (I plan to flesh this out further before the packaging & distribution mini-summit at PyCon US, but it already reflects the general scheme I had in the back of my mind when I decided to step up as BDFL-delegate for Daniel's three PEPs related to the wheel format. There are obviously lots of details to be worked out on distutils-sig and catalog-sig, but the big advantage it has over the approach tried with distutils2 is that individual projects shouldn't have to change much of anything - this scheme is designed to work so long as I can convince at least the setuptools, distribute, distutils2, pip, bento and zc.buildout developers to support it. End users would just need to update to recent versions of their preferred tools or, in the case of current users of plain distutils, upgrade to setuptools/distribute or use the "pip wheel" subcommand to get wheel support in older versions of Python) Components: Archiver: creates sdist source archives for distribution Builder: creates wheel binary archives for installation or distribution Uploader: tool for publishing to PyPI and private index servers Installer: retrieves archives (and their dependencies) and installs them on a target system Proposed flow for source distribution: Development System: -> Source checkout -> Installer -> Setup-Requires-Dist dependencies -> Archiver -> sdist -> Uploader -> PyPI (or private index server) Target System: -> Installer -> sdist + Setup-Requires-Dist dependencies -> Builder -> wheel -> Installer -> installed software + Requires-Dist dependencies Proposed flow for binary distribution: Development System: -> Source checkout -> Installer -> Setup-Requires-Dist dependencies -> Archiver -> sdist -> Builder -> wheel -> Uploader -> PyPI (or private index server) Target System: -> Installer -> installed software + Requires-Dist dependencies Standard library (3.4): distlib: tools to create Archivers, Uploaders, Builders and Installers pydist: Uploader/Installer CLI distutils: Archiver/Builder CLI (via setup.py) (setup.py will also continue to function as a limited Uploader/Installer CLI) Alternate Archiver/Builder tools: setuptools, distribute, distutils2, bento (and various custom distutils derivatives) Alternate Installer tools: pip, easy_install, zc.buildout The pydist CLI would likely be deliberately limited to installation from wheel binary archives. The officially recommended approach to supporting installation from sdist on systems which do not provide pip preinstalled would then be "pydist install pip". Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Sat Feb 16 08:49:04 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 16 Feb 2013 17:49:04 +1000 Subject: [Python-Dev] A new webpage promoting Compiler technology for CPython In-Reply-To: References: <201302152244.14912.paul@boddie.org.uk> <511F236F.70800@pearwood.info> Message-ID: On Sat, Feb 16, 2013 at 5:14 PM, Stefan Behnel wrote: > Hmm, I don't have the feeling that this discussion is leading anywhere > (especially not on this list). After all, it's quite possible to fire up a > PyPy runtime from a CPython runtime and have them talk to each other, > letting each one do what it's good at. That doesn't make PyPy a "compiler > for CPython", but at least it means that it doesn't fail completely to > integrate with the rest of the world. > > There are arguments for both sides, which means in dubio pro reo, I guess. If anyone is interested in fast Python code that integrates cleanly with external C libraries, then the combination of PyPy and cffi (http://cffi.readthedocs.org/en/latest/) should definitely be on their list of candidates to consider. Now, it may be excluded quickly because there's a CPython extension involved in the project that isn't a trivial wrapper around an existing non-Python library (and thus can't be easily replaced with cffi), but that's no excuse for not at least considering the combination in the cases where it makes sense. Yes, the PyPy team and scientific users of Python have a long history of talking past each other (and abusing each other for the mutual lack of understanding). However, that's no excuse for deliberately ignoring the advantages JIT compilation can bring, when cffi has been created specifically to give PyPy a typesafe (and JIT-transparent) way to interface with any library that provides C bindings. Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From stefan_ml at behnel.de Sat Feb 16 09:03:12 2013 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sat, 16 Feb 2013 09:03:12 +0100 Subject: [Python-Dev] A new webpage promoting Compiler technology for CPython In-Reply-To: References: <201302152244.14912.paul@boddie.org.uk> <511F236F.70800@pearwood.info> Message-ID: Nick Coghlan, 16.02.2013 08:49: > Yes, the PyPy team and scientific users of Python have a long history > of talking past each other (and abusing each other for the mutual lack > of understanding). However, that's no excuse for deliberately ignoring > the advantages JIT compilation can bring No-one's ignoring that. Philip already mentioned Numba, for example. The only question is whether this needs to happen outside of the existing ecosystem, or if we can't just embrace the 95-to-5 rule and use the right tools just for the performance critical bits of our code, without sacrificing what we achieved in the last 20 years. Stefan From fijall at gmail.com Sat Feb 16 10:15:06 2013 From: fijall at gmail.com (Maciej Fijalkowski) Date: Sat, 16 Feb 2013 11:15:06 +0200 Subject: [Python-Dev] A new webpage promoting Compiler technology for CPython In-Reply-To: References: <201302152244.14912.paul@boddie.org.uk> <511F236F.70800@pearwood.info> Message-ID: On Sat, Feb 16, 2013 at 10:03 AM, Stefan Behnel wrote: > Nick Coghlan, 16.02.2013 08:49: >> Yes, the PyPy team and scientific users of Python have a long history >> of talking past each other (and abusing each other for the mutual lack >> of understanding). However, that's no excuse for deliberately ignoring >> the advantages JIT compilation can bring > > No-one's ignoring that. Philip already mentioned Numba, for example. The > only question is whether this needs to happen outside of the existing > ecosystem, or if we can't just embrace the 95-to-5 rule and use the right > tools just for the performance critical bits of our code, without > sacrificing what we achieved in the last 20 years. > > Stefan Hi Stefan. I'm sorry PyPy does not cater for your needs, by being not like CPython. This is a conscious decision that Armin made after the psyco experiment and this is the one you disagree with, but it also allows us to do cool stuff, whether you like it or not. I understand it might be frustrating and I understand you guys simply don't like us for that. I also understand that people would love to promote their own work, that's fine too. CPyext is hard not because PyPy is undebuggable, but because the problem to solve (be not like CPython, but look like CPython) is a hard one. Until we come up with an idea how to make it so most CPython C extensions work out of the box and don't need careful refcount fixes, the focus is indeed not that major. However, considering this, the discussion of a website listing random projects from useful to outright obscure does not belong to python-dev, please take it somewhere else. Cheers, fijal From ncoghlan at gmail.com Sat Feb 16 10:18:22 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 16 Feb 2013 19:18:22 +1000 Subject: [Python-Dev] PEP 427: wheel In-Reply-To: References: Message-ID: On Sat, Feb 16, 2013 at 2:21 PM, Daniel Holth wrote: > #. Python scripts must appear in ``scripts`` and begin with exactly > ``b'#!python'`` in order to enjoy script wrapper generation and > ``#!python`` rewriting at install time. They may have any or no > extension. For compatibility with file encoding declarations, I believe this needs to be relaxed to starting with '#!python' in the source file encoding, rather than strictly b'#!python' (which will only be the case for ASCII compatible encodings). My rationale is that installers are going to need to read the source file encoding for the scripts anyway, otherwise they may write the shebang line back out with the wrong encoding, potentially leading to decoding errors when attempting to run the script. > #. ``{distribution}-{version}.dist-info/METADATA`` is Metadata version 1.1 > or greater (PEP 314, PEP 345, PEP 426) format metadata. I suggest removing the PEP references here and simply saying "is Metadata version 1.1 or greater format metadata" > #. ``Wheel-Version`` is the version number of the Wheel specification. > ``Generator`` is the name and optionally the version of the software > that produced the archive. ``Root-Is-Purelib`` is true if the top > level directory of the archive should be installed into purelib; > otherwise the root should be installed into platlib. ``Tag`` is the > wheel's expanded compatibility tags; in the example the filename would > contain ``py2.py3-none-any``. ``Build`` is the build number and is > omitted if there is no build number. I suggest breaking these out into separate bullet points (they're a bit hard to read as they stand) Aside from those minor issues, the current version of the spec looks fine to me - upload those fixes and I will accept it. If we later need to define wheel 1.1 or 2.0 to handle additional situations, well, that's why it's a versioned format :) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From stefan at bytereef.org Sat Feb 16 10:57:14 2013 From: stefan at bytereef.org (Stefan Krah) Date: Sat, 16 Feb 2013 10:57:14 +0100 Subject: [Python-Dev] Point of building without threads? In-Reply-To: <1357637305.2857.54.camel@newpride> References: <20120507214943.579045c2@pitrou.net> <1357637305.2857.54.camel@newpride> Message-ID: <20130216095714.GA8002@sleipnir.bytereef.org> Yury V. Zaytsev wrote: > On Mon, 2012-05-07 at 21:49 +0200, Antoine Pitrou wrote: > > > > I guess a long time ago, threading support in operating systems wasn't > > very widespread, but these days all our supported platforms have it. > > Is it still useful for production purposes to configure > > --without-threads? Do people use this option for something else than > > curiosity of mind? > > I hope that the intent behind asking this question was more of being > curious, rather then considering dropping --without-threads: > unfortunately, multithreading was, still is and probably will remain > troublesome on many supercomputing platforms. > > Often, once a new supercomputer is launched, as a developer you get a > half-baked C/C++ compiler with threading support broken to the point > when it's much easier to not use it altogether [*] rather than trying to > work around the compiler quirks. Out of curiosity: Do these incomplete compilers have any problem with either stdint.h or static inline functions in header files? Stefan Krah From ncoghlan at gmail.com Sat Feb 16 11:06:46 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 16 Feb 2013 20:06:46 +1000 Subject: [Python-Dev] Submitting PEP 425 (compatibility tags) In-Reply-To: References: Message-ID: On Sat, Feb 16, 2013 at 2:06 PM, Daniel Holth wrote: > This is the improved compatibility tags PEP 425, specifying how part of the > Wheel PEP 427 filenames work. Last time we discussed whether replacing . > with _ was ugly but I concluded it was harmless. > > Submitted for your consideration, > > PEP: 425 > Title: Compatibility Tags for Built Distributions > Version: $Revision$ > Last-Modified: 07-Aug-2012 > Author: Daniel Holth > BDFL-Delegate: Nick Coghlan > Status: Draft > Type: Standards Track > Content-Type: text/x-rst > Created: 27-Jul-2012 > Python-Version: 3.4 > Post-History: 8-Aug-2012, 18-Oct-2012, 15-Feb-2013 > > > Abstract > ======== > > This PEP specifies a tagging system to indicate with which versions of > Python a built or binary distribution is compatible. A set of three > tags indicate which Python implementation and language version, ABI, > and platform a built distribution requires. The tags are terse because > they will be included in filenames. > > > PEP Editor's Note > ================= > > While the naming scheme described in this PEP will not be supported directly > in the standard library until Python 3.4 at the earliest, draft > implementations may be made available in third party projects. > > > Rationale > ========= > > Today "python setup.py bdist" generates the same filename on PyPy > and CPython, but an incompatible archive, making it inconvenient to > share built distributions in the same folder or index. Instead, built > distributions should have a file naming convention that includes enough > information to decide whether or not a particular archive is compatible > with a particular implementation. > > Previous efforts come from a time where CPython was the only important > implementation and the ABI was the same as the Python language release. > This specification improves upon the older schemes by including the Python > implementation, language version, ABI, and platform as a set of tags. > > By comparing the tags it supports with the tags listed by the > distribution, an installer can make an educated decision about whether > to download a particular built distribution without having to read its > full metadata. > > Overview > ======== > > The tag format is {python tag}-{abi tag}-{platform tag} > > python tag > ?py27?, ?cp33? > abi tag > ?cp32dmu?, ?none? > platform tag > ?linux_x86_64?, ?any? > > For example, the tag py27-none-any indicates compatible with Python 2.7 > (any Python 2.7 implementation) with no abi requirement, on any platform. > > Use > === > > The `wheel` built package format includes these tags in its filenames, > of the form ``{distribution}-{version}(-{build tag})?-{python tag}-{abi > tag}-{platform tag}.whl``. Other package formats may have their own > conventions. > > Details > ======= > > Python Tag > ---------- > > The Python tag indicates the implementation and version required by > a distribution. Major implementations have abbreviated codes, initially: > > * py: Generic Python (does not require implementation-specific features) > * cp: CPython > * ip: IronPython > * pp: PyPy > * jy: Jython > > Other Python implementations should use `sys.implementation.name`. > > The version is `py_version_nodot`. CPython gets away with no dot, > but if one is needed the underscore `_` is used instead. PyPy should > probably use its own versions here `pp18`, `pp19`. > > The version can be just the major version `2` or `3` `py2`, `py3` for > many pure-Python distributions. > > Importantly, major-version-only tags like `py2` and `py3` are not > shorthand for `py20` and `py30`. Instead, these tags mean the packager > intentionally released a cross-version-compatible distribution. > > A single-source Python 2/3 compatible distribution can use the compound > tag `py2.py3`. See `Compressed Tag Sets`, below. > > ABI Tag > ------- > > The ABI tag indicates which Python ABI is required by any included > extension modules. For implementation-specific ABIs, the implementation > is abbreviated in the same way as the Python Tag, e.g. `cp33d` would be > the CPython 3.3 ABI with debugging. > > The CPython stable ABI is `abi3` as in the shared library suffix. > > Implementations with a very unstable ABI may use the first 6 bytes (as > 8 base64-encoded characters) of the SHA-256 hash of ther source code > revision and compiler flags, etc, but will probably not have a great need > to distribute binary distributions. Each implementation's community may > decide how to best use the ABI tag. > > Platform Tag > ------------ > > The platform tag is simply `distutils.util.get_platform()` with all > hyphens `-` and periods `.` replaced with underscore `_`. > > * win32 > * linux_i386 > * linux_x86_64 > > Use > === > > The tags are used by installers to decide which built distribution > (if any) to download from a list of potential built distributions. > The installer maintains a list of (pyver, abi, arch) tuples that it > will support. If the built distribution's tag is `in` the list, then > it can be installed. I suggest adjusting the specific compatibility tags in the example, and making it clear that is also a *recommended* default order. ================ It is recommended that installers, by default, try to give users access to the most feature complete built distribution available, by favouring those versions more specific to the installation environment, and then those marked as explicitly cross-version compatible, before falling back to pure Python versions published for older Python releases. It is also recommended that installers allow users to configure the list of acceptable compatibility tags, including their relative precedence. For example, an installer running under CPython 3.3 on a linux_x86_64 system might support:: cp33-cp33m-linux_x86_64 cp33-abi3-linux_x86_64 cp3-abi3-linux_x86_64 <--- Added cp33-none-linux_x86_64* cp3-none-linux_x86_64* <--- Added py33-none-linux_x86_64* <--- Added py3-none-linux_x86_64* <--- Added cp33-none-any cp3-none-any py33-none-any py3-none-any py32-none-any py31-none-any py30-none-any *Built distributions may be platform specific for reasons other than C extensions, such as using "sudo" to invoke a subprocess with elevated privileges. ================ > Can I have a tag `py32+` to indicate a minimum Python minor release? > No. Inspect the Trove classifiers to determine this level of > cross-release compatibility. Similar to the announcements "beaglevote > versions 3.2 and above no longer supports Python 1.52", you will > have to manually keep track of the maximum (PEP-386) release that > still supports your version of Python. > The answer to this question needs to be updated given that installers may fall back to trying earlier pure Python (or stable ABI) versions. (That is, installers are free to offer builds like py32-none-any and cp32-abi3-linux_x86_64 as possible solutions for a Python 3.3 install if there's nothing more recent) Additionally, I suggest including FAQ entries covering your answers to a couple of other questions that were raised in the last PEP 425 thread (I went back and looked at that thread to check the major points had been addressed for this version and a couple were missing): ================= Why normalise hyphens and other non-alphanumeric characters to underscores? To avoid conflicting with the "." and "-" characters that separate components of the filename, and for better compatibility with the widest range of filesystem limitations for filenames (including being usable in URL paths without quoting). Why not use special character rather than "." or "-"? Either because that character is inconvenient or potentially confusing in some contexts (for example, "+" must be quoted in URLs, "~" is used to denote the user's home directory in POSIX), or because the advantages weren't sufficiently compelling to justify changing the existing reference implementation for the wheel format defined in PEP 427 (for example, using "," rather than "." to separate components in a compressed tag) ================= The compatibility tag spec itself looks fine to me, so with the above clarifications, I will accept the PEP. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From solipsis at pitrou.net Sat Feb 16 11:12:49 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sat, 16 Feb 2013 11:12:49 +0100 Subject: [Python-Dev] PEP 427: wheel References: Message-ID: <20130216111249.5b608364@pitrou.net> On Sat, 16 Feb 2013 19:18:22 +1000 Nick Coghlan wrote: > On Sat, Feb 16, 2013 at 2:21 PM, Daniel Holth wrote: > > #. Python scripts must appear in ``scripts`` and begin with exactly > > ``b'#!python'`` in order to enjoy script wrapper generation and > > ``#!python`` rewriting at install time. They may have any or no > > extension. > > For compatibility with file encoding declarations, I believe this > needs to be relaxed to starting with '#!python' in the source file > encoding, rather than strictly b'#!python' (which will only be the > case for ASCII compatible encodings). I may be wrong, but I am not aware that Python is able to read encoding declarations in a non-ASCII compatible encoding :-) Also, given the shebang line is for use by the OS, the appropriate encoding should be decided by the OS, not Python conventions. But I would surprised if a shebang-compatible used a non-ASCII encoding by default. Regards Antoine. From solipsis at pitrou.net Sat Feb 16 11:19:19 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sat, 16 Feb 2013 11:19:19 +0100 Subject: [Python-Dev] PEP 427: wheel References: <20130216111249.5b608364@pitrou.net> Message-ID: <20130216111919.77a7633f@pitrou.net> On Sat, 16 Feb 2013 11:12:49 +0100 Antoine Pitrou wrote: > On Sat, 16 Feb 2013 19:18:22 +1000 > Nick Coghlan wrote: > > On Sat, Feb 16, 2013 at 2:21 PM, Daniel Holth wrote: > > > #. Python scripts must appear in ``scripts`` and begin with exactly > > > ``b'#!python'`` in order to enjoy script wrapper generation and > > > ``#!python`` rewriting at install time. They may have any or no > > > extension. > > > > For compatibility with file encoding declarations, I believe this > > needs to be relaxed to starting with '#!python' in the source file > > encoding, rather than strictly b'#!python' (which will only be the > > case for ASCII compatible encodings). > > I may be wrong, but I am not aware that Python is able to read encoding > declarations in a non-ASCII compatible encoding :-) > > Also, given the shebang line is for use by the OS, the appropriate > encoding should be decided by the OS, not Python conventions. But I > would surprised if a shebang-compatible used a non-ASCII encoding by > default. I mean non-ASCII compatible. Regards Antoine. From ncoghlan at gmail.com Sat Feb 16 12:09:06 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 16 Feb 2013 21:09:06 +1000 Subject: [Python-Dev] PEP 427: wheel In-Reply-To: <20130216111249.5b608364@pitrou.net> References: <20130216111249.5b608364@pitrou.net> Message-ID: On Sat, Feb 16, 2013 at 8:12 PM, Antoine Pitrou wrote: > On Sat, 16 Feb 2013 19:18:22 +1000 > Nick Coghlan wrote: > I may be wrong, but I am not aware that Python is able to read encoding > declarations in a non-ASCII compatible encoding :-) > > Also, given the shebang line is for use by the OS, the appropriate > encoding should be decided by the OS, not Python conventions. But I > would surprised if a shebang-compatible used a non-ASCII encoding by > default. Oh, good point - which means the only comments are cosmetic ones, which I can fix while marking it accepted. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From stephen at xemacs.org Sat Feb 16 12:14:39 2013 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Sat, 16 Feb 2013 20:14:39 +0900 Subject: [Python-Dev] PEP 427: wheel In-Reply-To: References: Message-ID: <87sj4wwlwg.fsf@uwakimon.sk.tsukuba.ac.jp> Nick Coghlan writes: > For compatibility with file encoding declarations, I believe this > needs to be relaxed to starting with '#!python' in the source file > encoding, rather than strictly b'#!python' (which will only be the > case for ASCII compatible encodings). In any PEP-263-compatible encoding it will be b'#!python'. Relaxing this is excessive generality for a new feature. I'm not sure what you mean by file encoding declarations if not PEP 263, which requires approximate[1] ASCII compatibility. PEP 3120 simply builds on PEP 263 by making UTF-8, rather than ISO 8859/1, the default encoding. > My rationale is that installers are going to need to read the source > file encoding for the scripts anyway, otherwise they may write the > shebang line back out with the wrong encoding, potentially leading to > decoding errors when attempting to run the script. Too bad if there's no PEP 263 declaration and the file is not in ASCII. I.e., the intersection of Python 2 and Python 3 default encodings. Footnotes: [1] Ie, Shift JIS and Big 5, or any encoding in which a pure ASCII string can be interpreted as a string in that encoding, are OK, but UTF-16 is not. From ncoghlan at gmail.com Sat Feb 16 12:17:30 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 16 Feb 2013 21:17:30 +1000 Subject: [Python-Dev] PEP 427: wheel In-Reply-To: References: Message-ID: Since Antoine and Stephen have pointed out my only non-cosmetic concern was an error on my part, I am accepting the PEP. I'll update the peps repo (including the cosmetic fixes) in a moment. Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Sat Feb 16 12:27:28 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 16 Feb 2013 21:27:28 +1000 Subject: [Python-Dev] PEP 427: wheel In-Reply-To: References: Message-ID: On Sat, Feb 16, 2013 at 9:17 PM, Nick Coghlan wrote: > Since Antoine and Stephen have pointed out my only non-cosmetic > concern was an error on my part, I am accepting the PEP. I'll update > the peps repo (including the cosmetic fixes) in a moment. And done: http://hg.python.org/peps/rev/d272d7a97e0c Thank you to Daniel for his hard work on getting this through to completion. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From stefan at bytereef.org Sat Feb 16 14:12:25 2013 From: stefan at bytereef.org (Stefan Krah) Date: Sat, 16 Feb 2013 14:12:25 +0100 Subject: [Python-Dev] Proposing "Argument Clinic", a new way of specifying arguments to builtins for CPython In-Reply-To: References: <50BD27CF.1070303@hastings.org> Message-ID: <20130216131225.GB9107@sleipnir.bytereef.org> Stefan Behnel wrote: > > you'd just pass in this string: > > > > (arg1 : int, arg2 : float, arg3 : ExtType, *, arg4=None) -> ExtType2 I've mentioned this proposal in http://bugs.python.org/issue16612 , but it wasn't sufficient for the task. Stefan Krah From stefan_ml at behnel.de Sat Feb 16 15:17:25 2013 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sat, 16 Feb 2013 15:17:25 +0100 Subject: [Python-Dev] Point of building without threads? In-Reply-To: <1357637305.2857.54.camel@newpride> References: <20120507214943.579045c2@pitrou.net> <1357637305.2857.54.camel@newpride> Message-ID: Yury V. Zaytsev, 08.01.2013 10:28: > Moreover, these days there is a clear trend towards OpenMP, so it has > become even harder to pressure the manufacturers to fix threads, because > they have 101 argument why you should port your code to OpenMP instead. I can't see OpenMP being an *alternative* to threads. You can happily acquire and release the GIL from OpenMP threads, for example. Stefan From solipsis at pitrou.net Sat Feb 16 16:39:19 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sat, 16 Feb 2013 16:39:19 +0100 Subject: [Python-Dev] PEP 427: wheel References: Message-ID: <20130216163919.63956dd1@pitrou.net> On Sat, 16 Feb 2013 21:27:28 +1000 Nick Coghlan wrote: > On Sat, Feb 16, 2013 at 9:17 PM, Nick Coghlan wrote: > > Since Antoine and Stephen have pointed out my only non-cosmetic > > concern was an error on my part, I am accepting the PEP. I'll update > > the peps repo (including the cosmetic fixes) in a moment. > > And done: http://hg.python.org/peps/rev/d272d7a97e0c > > Thank you to Daniel for his hard work on getting this through to completion. Great! Here's hope for an improved Python 3.4 distutils experience:-) Regards Antoine. From solipsis at pitrou.net Sun Feb 17 00:45:39 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sun, 17 Feb 2013 00:45:39 +0100 Subject: [Python-Dev] cpython: Issue #15022: Add pickle and comparison support to types.SimpleNamespace. References: <3Z7nv62fn4zRYG@mail.python.org> Message-ID: <20130217004539.0e55a3c9@pitrou.net> Hello, On Sun, 17 Feb 2013 00:42:26 +0100 (CET) eric.snow wrote: > > + def test_pickle(self): > + ns = types.SimpleNamespace(breakfast="spam", lunch="spam") > + > + ns_pickled = pickle.dumps(ns) > + ns_roundtrip = pickle.loads(ns_pickled) > + > + self.assertEqual(ns, ns_roundtrip) > + I think you should test with all pickle protocols. Regards Antoine. From ericsnowcurrently at gmail.com Sun Feb 17 01:02:34 2013 From: ericsnowcurrently at gmail.com (Eric Snow) Date: Sat, 16 Feb 2013 17:02:34 -0700 Subject: [Python-Dev] cpython: Issue #15022: Add pickle and comparison support to types.SimpleNamespace. In-Reply-To: <20130217004539.0e55a3c9@pitrou.net> References: <3Z7nv62fn4zRYG@mail.python.org> <20130217004539.0e55a3c9@pitrou.net> Message-ID: On Sat, Feb 16, 2013 at 4:45 PM, Antoine Pitrou wrote: > I think you should test with all pickle protocols. I'll do that. -eric From rovitotv at gmail.com Sun Feb 17 01:45:12 2013 From: rovitotv at gmail.com (Todd Rovito) Date: Sat, 16 Feb 2013 19:45:12 -0500 Subject: [Python-Dev] Cut/Copy/Paste items in IDLE right click context menu In-Reply-To: References: <50948B1F.1030000@udel.edu> Message-ID: On Sat, Nov 3, 2012 at 5:28 AM, Nick Coghlan wrote: > On Sat, Nov 3, 2012 at 1:10 PM, Terry Reedy wrote: >> [snip reasons] >> OK, I am convinced an info PEP would be a good idea. > > Unless anyone objects, I'm happy to be BDFL-delegate for such a PEP. I > mainly want to ensure we clearly fence off "IDLE-the-application", > with (in effect) a 6 month release cycle synchronised across versions, > from the rest of the standard library. I have not seen much progress on this issue so I thought I would attempt to draft a PEP. This is my first PEP so go easy on me. Attached is a PEP titled "IDLE Enhancement Exception for All Branches", I am looking for feedback. Thanks. -------------- next part -------------- PEP: XXX Title: IDLE Enhancement Exception for All Branches Version: Last-Modified: Author: Todd Rovito BDFL-Delegate: Nick Coghlan Status: Draft Type: Informational Content-Type: text/x-rst Created: 16-Feb-2013 Python-Version: 2.7 Post-History: 16-Feb-2013 Abstract ======== Generally only new features are applied to Python 3.4 but this PEP requests an exception for IDLE [1]_. IDLE is part of the standard library and has numerous outstanding issues [2]_. Since IDLE is often the first thing a new Python user sees it desperately needs to be brought up to date with modern GUI standards across the three major platforms Linux, Mac OS X, and Windows. Rationale ========= Python does have many advanced features yet Python is well known for being a easy computer language for beginners [3]_. A major Python philosophy is "batteries included" which is best demonstrated in Python's standard library with many modules that are not typically included with other programming languages [4]_. IDLE is a important "battery" in the Python toolbox because it allows a beginner to get started quickly without downloading and configuring a third party IDE. Additional patches to IDLE cannot break any existing program/library because IDLE is used by humans. Details ======= Python 2.7 does not accept bug fixes, this rule can be ignored for IDLE if the Python development team accepts this PEP [5]_. IDLE issues will be carefully tested on the three major platforms Linux, Mac OS X, and Windows before any commits are made.. References ========== .. [1] IDLE: Right Click Context Menu, Foord, Michael (http://bugs.python.org/issue1207589) .. [2] Meta-issue for "Invent with Python" IDLE feedback (http://bugs.python.org/issue13504) .. [3] Getting Started with Python (http://www.python.org/about/gettingstarted/) .. [4] Batteries Included (http://docs.python.org/2/tutorial/stdlib.html#batteries-included) .. [5] Python 2.7 Release Schedule (http://www.python.org/dev/peps/pep-0373/) Copyright ========= This document has been placed in the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End: From ericsnowcurrently at gmail.com Sun Feb 17 02:20:14 2013 From: ericsnowcurrently at gmail.com (Eric Snow) Date: Sat, 16 Feb 2013 18:20:14 -0700 Subject: [Python-Dev] cpython: Issue #15022: Add pickle and comparison support to types.SimpleNamespace. In-Reply-To: References: <3Z7nv62fn4zRYG@mail.python.org> <20130217004539.0e55a3c9@pitrou.net> Message-ID: On Sat, Feb 16, 2013 at 5:02 PM, Eric Snow wrote: > On Sat, Feb 16, 2013 at 4:45 PM, Antoine Pitrou wrote: >> I think you should test with all pickle protocols. > > I'll do that. Out of curiousity, how critical is it to support protocols 0 and 1? I've made sure SimpleNamespace does, but it made me wonder what the use case is for the older protocols. -eric From ncoghlan at gmail.com Sun Feb 17 03:42:04 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 17 Feb 2013 12:42:04 +1000 Subject: [Python-Dev] Cut/Copy/Paste items in IDLE right click context menu In-Reply-To: References: <50948B1F.1030000@udel.edu> Message-ID: On Sun, Feb 17, 2013 at 10:45 AM, Todd Rovito wrote: > On Sat, Nov 3, 2012 at 5:28 AM, Nick Coghlan wrote: >> On Sat, Nov 3, 2012 at 1:10 PM, Terry Reedy wrote: >>> [snip reasons] >>> OK, I am convinced an info PEP would be a good idea. >> >> Unless anyone objects, I'm happy to be BDFL-delegate for such a PEP. I >> mainly want to ensure we clearly fence off "IDLE-the-application", >> with (in effect) a 6 month release cycle synchronised across versions, >> from the rest of the standard library. > I have not seen much progress on this issue so I thought I would > attempt to draft a PEP. This is my first PEP so go easy on me. > > Attached is a PEP titled "IDLE Enhancement Exception for All > Branches", I am looking for feedback. Thanks. That's a good start - please forward it to the PEP editors list (peps at python.org) to be posted. The rationale needs to be fleshed out a bit more along the lines of "IDLE is primarily used as an application that ships with Python, rather than as a library module used to build Python applications, that's why it is OK for a different standard to apply". Mentioning the point about Linux distros splitting it out into a separate package would also be useful. NY other two major comments: - be specific about which parts of the code base are covered by the exception - no need for extensive cross-OS testing prior to commit, that's a key part of the role of the buildbots Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From nyamatongwe at me.com Sun Feb 17 04:00:24 2013 From: nyamatongwe at me.com (Neil Hodgson) Date: Sun, 17 Feb 2013 14:00:24 +1100 Subject: [Python-Dev] Cut/Copy/Paste items in IDLE right click context menu In-Reply-To: References: <50948B1F.1030000@udel.edu> Message-ID: Nick Coghlan: > - no need for extensive cross-OS testing prior to commit, that's a key > part of the role of the buildbots Are the buildbots able to test UI features like menu selections? Neil From ericsnowcurrently at gmail.com Sun Feb 17 04:36:16 2013 From: ericsnowcurrently at gmail.com (Eric Snow) Date: Sat, 16 Feb 2013 20:36:16 -0700 Subject: [Python-Dev] in a pickle: method lookup irregularities Message-ID: (See http://bugs.python.org/issue16251) The pickle module looks up special methods on instances rather than using the behavior described in the language reference (or even just looking them up on the class). This means that custom __getattribute__() and __getattr__() implementations may get triggered (e.g. if there is no __reduce__() implemented). Bringing pickle in sync with the language reference would not involve a big change, but it would impact anyone that relies on the current behavior. Most notably, proxy types could exist which leverage the current semantics. I look at this as trading one "bug" for another in a . So I want to get some general feedback on the situation and see what others consider to be the trade-offs of fixing or not fixing this bug. I'm particularly interested in existing cases where code uses the current behavior. With that in hand we could assess what alternatives exist in those cases or how we could better document workarounds for the current behavior. -eric From ncoghlan at gmail.com Sun Feb 17 05:42:18 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 17 Feb 2013 14:42:18 +1000 Subject: [Python-Dev] in a pickle: method lookup irregularities In-Reply-To: References: Message-ID: On Sun, Feb 17, 2013 at 1:36 PM, Eric Snow wrote: > (See http://bugs.python.org/issue16251) > > The pickle module looks up special methods on instances rather than > using the behavior described in the language reference (or even just > looking them up on the class). This means that custom > __getattribute__() and __getattr__() implementations may get triggered > (e.g. if there is no __reduce__() implemented). > > Bringing pickle in sync with the language reference would not involve > a big change, but it would impact anyone that relies on the current > behavior. Most notably, proxy types could exist which leverage the > current semantics. > > I look at this as trading one "bug" for another in a . So I want to > get some general feedback on the situation and see what others > consider to be the trade-offs of fixing or not fixing this bug. I'm > particularly interested in existing cases where code uses the current > behavior. With that in hand we could assess what alternatives exist > in those cases or how we could better document workarounds for the > current behavior. I vaguely recall some discussions about this and other similar cases of metaclass confusion (perhaps related to the copy module). One possible approach is to look specifically for cases where type(obj) != obj.__class__ and handle them differently. The general trend has been towards more consistently ignoring instance attributes when looking up special methods, but you're right that this can be a backwards compatibility minefield. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Sun Feb 17 05:52:27 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 17 Feb 2013 14:52:27 +1000 Subject: [Python-Dev] Submitting PEP 425 (compatibility tags) In-Reply-To: References: Message-ID: On Sat, Feb 16, 2013 at 2:06 PM, Daniel Holth wrote: > This is the improved compatibility tags PEP 425, specifying how part of the > Wheel PEP 427 filenames work. Last time we discussed whether replacing . > with _ was ugly but I concluded it was harmless. Daniel has updated the example and FAQ as I requested, so PEP 425 is now accepted. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From rdmurray at bitdance.com Sun Feb 17 07:56:17 2013 From: rdmurray at bitdance.com (R. David Murray) Date: Sun, 17 Feb 2013 01:56:17 -0500 Subject: [Python-Dev] Cut/Copy/Paste items in IDLE right click context menu In-Reply-To: References: <50948B1F.1030000@udel.edu> Message-ID: <20130217065617.D39B6250BC1@webabinitio.net> On Sun, 17 Feb 2013 14:00:24 +1100, Neil Hodgson wrote: > Nick Coghlan: > > > - no need for extensive cross-OS testing prior to commit, that's a key > > part of the role of the buildbots > > Are the buildbots able to test UI features like menu selections? Not to mention that the test suite is a bit sparse for Idle, despite recent efforts to improve this, even for the stuff that can be tested. --David From ncoghlan at gmail.com Sun Feb 17 09:19:17 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 17 Feb 2013 18:19:17 +1000 Subject: [Python-Dev] Cut/Copy/Paste items in IDLE right click context menu In-Reply-To: <20130217065617.D39B6250BC1@webabinitio.net> References: <50948B1F.1030000@udel.edu> <20130217065617.D39B6250BC1@webabinitio.net> Message-ID: On Sun, Feb 17, 2013 at 4:56 PM, R. David Murray wrote: > On Sun, 17 Feb 2013 14:00:24 +1100, Neil Hodgson wrote: >> Nick Coghlan: >> >> > - no need for extensive cross-OS testing prior to commit, that's a key >> > part of the role of the buildbots >> >> Are the buildbots able to test UI features like menu selections? > > Not to mention that the test suite is a bit sparse for Idle, despite > recent efforts to improve this, even for the stuff that can be tested. Perhaps something for the PEP to elaborate on before we declare open season on Idle improvements in bug fix releases. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Sun Feb 17 09:25:07 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 17 Feb 2013 18:25:07 +1000 Subject: [Python-Dev] BDFL delegation for PEP 426 (PyPI metadata 1.3) In-Reply-To: References: Message-ID: On Sun, Feb 3, 2013 at 4:37 PM, Guido van Rossum wrote: > I don't know or care much about PyPI metadata, so do what you feel is > right. If you are uncomfortable being PEP-uncle *and* -author, find > another author or another uncle. But since it doesn't affect the > language or library, it's fine with me if you are both. :-) Based on this, and due to the fact that I think most everyone else involved in reviewing PEP 426 would have already accepted it by now, I've reclaimed the BDFL-Delegate role for PEP 426. I'll be posting the latest draft to distutils-sig for a final sanity check shortly. (Do people see any point in posting it here as well? Everyone that will need to support it near-term is over on distutils-sig, but may not be on python-dev) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From solipsis at pitrou.net Sun Feb 17 12:49:58 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sun, 17 Feb 2013 12:49:58 +0100 Subject: [Python-Dev] cpython: Issue #15022: Add pickle and comparison support to types.SimpleNamespace. In-Reply-To: References: <3Z7nv62fn4zRYG@mail.python.org> <20130217004539.0e55a3c9@pitrou.net> Message-ID: <20130217124958.2d904831@pitrou.net> On Sat, 16 Feb 2013 18:20:14 -0700 Eric Snow wrote: > On Sat, Feb 16, 2013 at 5:02 PM, Eric Snow wrote: > > On Sat, Feb 16, 2013 at 4:45 PM, Antoine Pitrou wrote: > >> I think you should test with all pickle protocols. > > > > I'll do that. > > Out of curiousity, how critical is it to support protocols 0 and 1? > I've made sure SimpleNamespace does, but it made me wonder what the > use case is for the older protocols. Not critical at all. It's just nice to have for consistency, but it shouldn't stop you if it's too difficult to implement. Regards Antoine. From ncoghlan at gmail.com Sun Feb 17 13:00:50 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 17 Feb 2013 22:00:50 +1000 Subject: [Python-Dev] cpython: Issue #15022: Add pickle and comparison support to types.SimpleNamespace. In-Reply-To: <20130217124958.2d904831@pitrou.net> References: <3Z7nv62fn4zRYG@mail.python.org> <20130217004539.0e55a3c9@pitrou.net> <20130217124958.2d904831@pitrou.net> Message-ID: On Sun, Feb 17, 2013 at 9:49 PM, Antoine Pitrou wrote: > On Sat, 16 Feb 2013 18:20:14 -0700 > Eric Snow wrote: >> On Sat, Feb 16, 2013 at 5:02 PM, Eric Snow wrote: >> > On Sat, Feb 16, 2013 at 4:45 PM, Antoine Pitrou wrote: >> >> I think you should test with all pickle protocols. >> > >> > I'll do that. >> >> Out of curiousity, how critical is it to support protocols 0 and 1? >> I've made sure SimpleNamespace does, but it made me wonder what the >> use case is for the older protocols. > > Not critical at all. It's just nice to have for consistency, but it > shouldn't stop you if it's too difficult to implement. Although in such cases, we probably want to test we generate a reasonable error message for the unsupported protocols, rather than silently writing junk to the pickle. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Sun Feb 17 11:11:23 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 17 Feb 2013 20:11:23 +1000 Subject: [Python-Dev] Fwd: PEP 426 is now the draft spec for distribution metadata 2.0 In-Reply-To: References: Message-ID: FYI ---------- Forwarded message ---------- From: Nick Coghlan Date: Sun, Feb 17, 2013 at 8:10 PM Subject: PEP 426 is now the draft spec for distribution metadata 2.0 To: "DistUtils mailing list\"\"" The latest draft of PEP 426 is up at http://www.python.org/dev/peps/pep-0426/ Major changes since the last draft: 1. Metadata-Version is 2.0 rather than 1.3, and the field now has the same major.minor semantics as are defined for wheel versions in PEP 427 (i.e. if a tool sees a major version number it doesn't recognise, it should give up rather than trying to guess what to do with it, while it's OK to process a higher minor version) 2. The "Private-Version" field is added, and several examples are given showing how to use it in conjunction with translated public versions when a project wants to use a version numbering scheme that the standard installation tools won't understand. 3. The environment markers section more clearly covers the need to handle parentheses (this was mentioned in the text, but not the pseudo-grammar), and the fields which support those markers have an explicit cross-reference to that section of the spec. 4. Leading/trailing whitespace and date based versioning are explicitly excluded from valid public versions 5. Version compatibility statistics are provided for this PEP relative to PEP 386 (the analysis script has been added to the PEP repo if anyone wants to check my numbers) 6. I have reclaimed BDFL-Delegate status (with Guido's approval) Since getting wheel support into pip no longer depends on this version of the metadata spec, I plan to leave it open for comment for another week, and then accept it next weekend if I don't hear any significant objections. Regards, Nick. ========================== PEP: 426 Title: Metadata for Python Software Packages 2.0 Version: $Revision$ Last-Modified: $Date$ Author: Daniel Holth , Donald Stufft , Nick Coghlan BDFL-Delegate: Nick Coghlan Discussions-To: Distutils SIG Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 30 Aug 2012 Post-History: 14 Nov 2012, 5 Feb 2013, 7 Feb 2013, 9 Feb 2013 Abstract ======== This PEP describes a mechanism for adding metadata to Python distributions. It includes specifics of the field names, and their semantics and usage. This document specifies version 2.0 of the metadata format. Version 1.0 is specified in PEP 241. Version 1.1 is specified in PEP 314. Version 1.2 is specified in PEP 345. Version 2.0 of the metadata format adds fields designed to make third-party packaging of Python Software easier and defines a formal extension mechanism. It also adds support for optional features of distributions and allows the description to be placed into a payload section. Finally, this version addresses several issues with the previous iteration of the standard version identification scheme. Metadata files ============== The syntax defined in this PEP is for use with Python distribution metadata files. The file format is a simple UTF-8 encoded Key: value format with case-insensitive keys and no maximum line length, optionally followed by a blank line and a payload containing a description of the distribution. This format is parseable by the ``email`` module with an appropriate ``email.policy.Policy()``. When ``metadata`` is a Unicode string, ```email.parser.Parser().parsestr(metadata)`` is a serviceable parser. There are three standard locations for these metadata files: * the ``PKG-INFO`` file included in the base directory of Python source distribution archives (as created by the distutils ``sdist`` command) * the ``{distribution}-{version}.dist-info/METADATA`` file in a ``wheel`` binary distribution archive (as described in PEP 425, or a later version of that specification) * the ``{distribution}-{version}.dist-info/METADATA`` files in a local Python installation database (as described in PEP 376, or a later version of that specification) Other tools involved in Python distribution may also use this format. Encoding ======== Metadata 2.0 files are UTF-8 with the restriction that keys must be ASCII. Parser implementations should be aware that older versions of the Metadata specification do not specify an encoding. Metadata header fields ======================= This section specifies the names and semantics of each of the supported fields in the metadata header. In a single Metadata 2.0 file, fields marked with "(optional)" may occur 0 or 1 times. Fields marked with "(multiple use)" may be specified 0, 1 or more times. Only "Metadata-Version", "Name", "Version", and "Summary" must appear exactly once. The fields may appear in any order within the header section of the file. Metadata-Version ---------------- Version of the file format; "2.0" is the only legal value. Automated tools should warn if ``Metadata-Version`` is greater than the highest version they support, and must fail if ``Metadata-Version`` has a greater major version than the highest version they support. Example:: Metadata-Version: 2.0 Name ---- The name of the distribution. Example:: Name: BeagleVote Version ------- The distribution's public version identifier. Public versions are designed for consumption by automated tools and are strictly ordered according to a defined scheme. See `Version scheme`_ below. Example:: Version: 1.0a2 Summary ------- A one-line summary of what the distribution does. Example:: Summary: A module for collecting votes from beagles. Private-Version (optional) -------------------------- An arbitrary private version label. Private version labels are intended for internal use by a project, and cannot be used in version specifiers. See `Compatibility with other version schemes`_ below. Examples:: Private-Version: 1.0.0-alpha.1 Private-Version: 1.3.7+build.11.e0f985a Private-Version: v1.8.1.301.ga0df26f Private-Version: 2013.02.17.dev123 Description (optional, deprecated) ---------------------------------- Starting with Metadata 2.0, the recommended place for the description is in the payload section of the document, after the last header. The description does not need to be reformatted when it is included in the payload. See `Describing the Distribution`_ for more information on the expected contents of this field. Since a line separator immediately followed by another line separator indicates the end of the headers section, any line separators in a ``Description`` header field must be suffixed by whitespace to indicate continuation. It is an error to provide both a ``Description`` header and a metadata payload. Keywords (optional) ------------------- A list of additional whitespace separated keywords to be used to assist searching for the distribution in a larger catalog. Example:: Keywords: dog puppy voting election Home-page (optional) -------------------- A string containing the URL for the distribution's home page. Example:: Home-page: http://www.example.com/~cschultz/bvote/ Download-URL (optional) ----------------------- A string containing the URL from which this version of the distribution can be downloaded. (This means that the URL can't be something like ".../BeagleVote-latest.tgz", but instead must be ".../BeagleVote-0.45.tgz".) Project-URL (multiple use) -------------------------- A string containing a label and a browsable URL for the project, separated by the last occurrence of comma and space ", ". The label consists of any permitted header text, including commas. Example:: Bug, Issue Tracker, http://bitbucket.org/tarek/distribute/issues/ Author (optional) ----------------- A string containing the author's name at a minimum; additional contact information may be provided. Example:: Author: C. Schultz, Universal Features Syndicate, Los Angeles, CA Author-email (optional) ----------------------- A string containing the author's e-mail address. It contains a name and e-mail address in the RFC 5322 recommended ``Address Specification`` format. Example:: Author-email: "C. Schultz" Maintainer (optional) --------------------- A string containing the maintainer's name at a minimum; additional contact information may be provided. Note that this field is intended for use when a project is being maintained by someone other than the original author: it should be omitted if it is identical to ``Author``. Example:: Maintainer: C. Schultz, Universal Features Syndicate, Los Angeles, CA Maintainer-email (optional) --------------------------- A string containing the maintainer's e-mail address. It has the same format as ``Author-email``. Note that this field is intended for use when a project is being maintained by someone other than the original author: it should be omitted if it is identical to ``Author-email``. Example:: Maintainer-email: "C. Schultz" License (optional) ------------------ Text indicating the license covering the distribution where the license is not a selection from the "License" Trove classifiers. See "Classifier" below. This field may also be used to specify a particular version of a license which is named via the ``Classifier`` field, or to indicate a variation or exception to such a license. Examples:: License: This software may only be obtained by sending the author a postcard, and then the user promises not to redistribute it. License: GPL version 3, excluding DRM provisions The full text of the license would normally be included in a separate file. Classifier (multiple use) ------------------------- Each entry is a string giving a single classification value for the distribution. Classifiers are described in PEP 301 [2]. `Environment markers`_ may be used with this field. Examples:: Classifier: Development Status :: 4 - Beta Classifier: Environment :: Console (Text Based) Provides-Dist (multiple use) ---------------------------- Each entry contains a string naming a requirement that is satisfied by installing this distribution. These strings must be of the form ``Name`` or ``Name (Version)``, following the formats of the corresponding field definitions. A distribution may provide additional names, e.g. to indicate that multiple projects have been merged into and replaced by a single distribution or to indicate that this project is a substitute for another. For instance, distribute (a fork of setuptools) can include a ``Provides-Dist: setuptools`` entry to prevent the conflicting package from being downloaded and installed when distribute is already installed. A distribution that has been merged with another might ``Provides-Dist`` the obsolete name(s) to satisfy any projects that require the obsolete distribution's name. A distribution may also provide a "virtual" project name, which does not correspond to any separately-distributed project: such a name might be used to indicate an abstract capability which could be supplied by one of multiple projects. E.g., multiple projects might supply RDBMS bindings for use by a given ORM: each project might declare that it provides ``ExampleORM-somedb-bindings``, allowing other projects to depend only on having at least one of them installed. A version declaration may be supplied and must follow the rules described in `Version scheme`_. The distribution's version identifier will be implied if none is specified. `Environment markers`_ may be used with this field. Examples:: Provides-Dist: ThisProject Provides-Dist: AnotherProject (3.4) Provides-Dist: virtual_package Provides-Extra (multiple use) ----------------------------- A string containing the name of an optional feature. Must be printable ASCII, not containing whitespace, comma (,), or square brackets []. May be used to make a dependency conditional on whether the optional feature has been requested. See `Optional Features`_ for details on the use of this field. Example:: Name: beaglevote Provides-Extra: pdf Requires-Dist: reportlab; extra == 'pdf' Requires-Dist: nose; extra == 'test' Requires-Dist: sphinx; extra == 'doc' Obsoleted-By (optional) ----------------------- Indicates that this project is no longer being developed. The named project provides a substitute or replacement. A version declaration may be supplied and must follow the rules described in `Version specifiers`_. Possible uses for this field include handling project name changes and project mergers. Examples:: Name: BadName Obsoleted-By: AcceptableName Name: SeparateProject Obsoleted-By: MergedProject (>=4.0.0) Requires-Dist (multiple use) ---------------------------- Each entry contains a string naming some other distutils project required by this distribution. The format of a requirement string is identical to that of a distribution name (e.g., as found in the ``Name:`` field) optionally followed by a version declaration within parentheses. The distribution names should correspond to names as found on the `Python Package Index`_; often the same as, but distinct from, the module names as accessed with ``import x``. `Environment markers`_ may be used with this field. Version declarations must follow the rules described in `Version specifiers`_ Distributions may also depend on optional features of other distributions. See `Optional Features`_ for details. Examples:: Requires-Dist: pkginfo Requires-Dist: PasteDeploy Requires-Dist: zope.interface (>3.5.0) Dependencies mentioned in ``Requires-Dist`` may be installed exclusively at run time and are not guaranteed to be available when creating or installing a package. If a dependency is needed during distribution creation or installation *and* at run time, it should be listed under both ``Requires-Dist`` and ``Setup-Requires-Dist``. Setup-Requires-Dist (multiple use) ---------------------------------- Like ``Requires-Dist``, but names dependencies needed in order to build, package or install the distribution -- in distutils, a dependency imported by ``setup.py`` itself. Commonly used to bring in extra compiler support or a package needed to generate a manifest from version control. Examples:: Setup-Requires-Dist: custom_setup_command Dependencies mentioned in ``Setup-Requires-Dist`` may be installed exclusively for setup and are not guaranteed to be available at run time. If a dependency is needed during distribution creation or installation *and* at run time, it should be listed under both ``Requires-Dist`` and ``Setup-Requires-Dist``. Requires-Python (multiple use) ------------------------------ This field specifies the Python version(s) that the distribution is guaranteed to be compatible with. `Environment markers`_ may be used with this field. Version declarations must be in the format specified in `Version specifiers`_. Examples:: Requires-Python: 3.2 Requires-Python: >3.1 Requires-Python: >=2.3.4 Requires-Python: >=2.5,<2.7 If specified multiple times, the Python version must satisfy all such constraints to be considered compatible. This is most useful in combination with appropriate `Environment markers`_. For example, if a feature was initially introduced to Python as a Unix-specific addition, and then Windows support was added in the subsequent release, this could be indicated with the following pair of entries:: Requires-Python: >= 3.1 Requires-Python: >= 3.2; sys.platform == 'win32' Requires-External (multiple use) -------------------------------- Each entry contains a string describing some dependency in the system that the distribution is to be used. This field is intended to serve as a hint to downstream project maintainers, and has no semantics which are meaningful to the ``distutils`` distribution. The format of a requirement string is a name of an external dependency, optionally followed by a version declaration within parentheses. `Environment markers`_ may be used with this field. Because they refer to non-Python software releases, version identifiers for this field are **not** required to conform to the format described in `Version scheme`_: they should correspond to the version scheme used by the external dependency. Notice that there is no particular rule on the strings to be used. Examples:: Requires-External: C Requires-External: libpng (>=1.5) Platform (multiple use) ----------------------- A Platform specification describing an operating system supported by the distribution which is not listed in the "Operating System" Trove classifiers. See `Classifier`__ above. __ `Classifier (multiple use)`_ Examples:: Platform: ObscureUnix Platform: RareDOS Supported-Platform (multiple use) --------------------------------- Binary distributions containing a metadata file will use the Supported-Platform field in their metadata to specify the OS and CPU for which the binary distribution was compiled. The semantics of the Supported-Platform field are not specified in this PEP. Example:: Supported-Platform: RedHat 7.2 Supported-Platform: i386-win32-2791 Extension (multiple use) ------------------------ An ASCII string, not containing whitespace or the ``/`` character, that indicates the presence of extended metadata. The additional fields defined by the extension are then prefixed with the name of the extension and the ``/`` character. For example:: Extension: Chili Chili/Type: Poblano Chili/Heat: Mild To avoid name conflicts, it is recommended that distribution names be used to identify metadata extensions. This practice will also make it easier to find authoritative documentation for metadata extensions. As the order of the metadata headers is not constrained, the ``Extension: Chili`` field may appear before or after the corresponding extension fields ``Chili/Type:`` etc. Values in extension fields must still respect the general formatting requirements for metadata headers. A bare ``Extension: Name`` entry with no corresponding extension fields is permitted. It may, for example, indicate the expected presence of an additional metadata file rather than the presence of extension fields. An extension field with no corresponding ``Extension: Name`` entry is an error. Describing the distribution =========================== The distribution metadata should include a longer description of the distribution that may run to several paragraphs. Software that deals with metadata should not assume any maximum size for the description. The recommended location for the description is in the metadata payload, separated from the header fields by at least one completely blank line (that is, two successive line separators with no other characters between them, not even whitespace). Alternatively, the description may be provided in the `Description`__ metadata header field. Providing both a ``Description`` field and a payload is an error. __ `Description (optional, deprecated)`_ The distribution description can be written using reStructuredText markup [1]_. For programs that work with the metadata, supporting markup is optional; programs may also display the contents of the field as plain text without any special formatting. This means that authors should be conservative in the markup they use. Version scheme ============== Public version identifiers must comply with the following scheme:: N[.N]+[{a|b|c|rc}N][.postN][.devN] Version identifiers which do not comply with this scheme are an error. Version identifiers must not include leading or trailing whitespace. Any given version will be a "release", "pre-release", "post-release" or "developmental release" as defined in the following sections. .. note:: Some hard to read version identifiers are permitted by this scheme in order to better accommodate the wide range of versioning practices across existing public and private Python projects. Accordingly, some of the versioning practices which are technically permitted by the PEP are strongly discouraged for new projects. Where this is the case, the relevant details are noted in the following sections. Releases -------- A release number is a version identifier that consists solely of one or more non-negative integer values, separated by dots:: N[.N]+ Releases within a project must be numbered in a consistently increasing fashion. Ordering considers the numeric value of each component in turn, with "component does not exist" sorted ahead of all numeric values. Date based release numbers are explicitly excluded from compatibility with this scheme, as they hinder automatic translation to other versioning schemes, as well as preventing the adoption of semantic versioning without changing the name of the project. Accordingly, a leading release component greater than or equal to ``1980`` is an error. While any number of additional components after the first are permitted under this scheme, the most common variants are to use two components ("major.minor") or three components ("major.minor.micro"). For example:: 0.9 0.9.1 0.9.2 ... 0.9.10 0.9.11 1.0 1.0.1 1.1 2.0 2.0.1 A release series is any set of release numbers that start with a common prefix. For example, ``3.3.1``, ``3.3.5`` and ``3.3.9.45`` are all part of the ``3.3`` release series. .. note:: Using both ``X.Y`` and ``X.Y.0`` as distinct release numbers within the scope of a single release series is strongly discouraged, as it makes the version ordering ambiguous for human readers. Automated tools should either treat this case as an error, or else interpret an ``X.Y.0`` release as coming *after* the corresponding ``X.Y`` release. The recommended practice is to always use release numbers of a consistent length (that is, always include the trailing ``.0``). An acceptable alternative is to consistently omit the trailing ``.0``. The example above shows both styles, always including the ``.0`` at the second level and consistently omitting it at the third level. Pre-releases ------------ Some projects use an "alpha, beta, release candidate" pre-release cycle to support testing by their users prior to a full release. If used as part of a project's development cycle, these pre-releases are indicated by a suffix appended directly to the last component of the release number:: X.YaN # Alpha release X.YbN # Beta release X.YcN # Release candidate (alternative notation: X.YrcN) X.Y # Full release The pre-release suffix consists of an alphabetical identifier for the pre-release phase, along with a non-negative integer value. Pre-releases for a given release are ordered first by phase (alpha, beta, release candidate) and then by the numerical component within that phase. .. note:: Using both ``c`` and ``rc`` to identify release candidates within the scope of a single release is strongly discouraged, as it makes the version ordering ambiguous for human readers. Automated tools should either treat this case as an error, or else interpret all ``rc`` versions as coming after all ``c`` versions (that is, ``rc1`` indicates a later version than ``c2``). Post-releases ------------- Some projects use post-releases to address minor errors in a release that do not affect the distributed software (for example, correcting an error in the release notes). If used as part of a project's development cycle, these post-releases are indicated by a suffix appended directly to the last component of the release number:: X.Y.postN # Post-release The post-release suffix consists of the string ``.post``, followed by a non-negative integer value. Post-releases are ordered by their numerical component, immediately following the corresponding release, and ahead of any subsequent release. .. note:: The use of post-releases to publish maintenance releases containing actual bug fixes is strongly discouraged. In general, it is better to use a longer release number and increment the final component for each maintenance release. Post-releases are also permitted for pre-releases:: X.YaN.postM # Post-release of an alpha release X.YbN.postM # Post-release of a beta release X.YcN.postM # Post-release of a release candidate .. note:: Creating post-releases of pre-releases is strongly discouraged, as it makes the version identifier difficult to parse for human readers. In general, it is substantially clearer to simply create a new pre-release by incrementing the numeric component. Developmental releases ---------------------- Some projects make regular developmental releases, and system packagers (especially for Linux distributions) may wish to create early releases which do not conflict with later project releases. If used as part of a project's development cycle, these developmental releases are indicated by a suffix appended directly to the last component of the release number:: X.Y.devN # Developmental release The developmental release suffix consists of the string ``.dev``, followed by a non-negative integer value. Developmental releases are ordered by their numerical component, immediately before the corresponding release (and before any pre-releases), and following any previous release. Developmental releases are also permitted for pre-releases and post-releases:: X.YaN.devM # Developmental release of an alpha release X.YbN.devM # Developmental release of a beta release X.YcN.devM # Developmental release of a release candidate X.Y.postN.devM # Developmental release of a post-release .. note:: Creating developmental releases of pre-releases is strongly discouraged, as it makes the version identifier difficult to parse for human readers. In general, it is substantially clearer to simply create additional pre-releases by incrementing the numeric component. Developmental releases of post-releases are also strongly discouraged, but they may be appropriate for projects which use the post-release notation for full maintenance releases which may include code changes. Examples of compliant version schemes ------------------------------------- The standard version scheme is designed to encompass a wide range of identification practices across public and private Python projects. In practice, a single project attempting to use the full flexibility offered by the scheme would create a situation where human users had difficulty figuring out the relative order of versions, even though the rules above ensure all compliant tools will order them consistently. The following examples illustrate a small selection of the different approaches projects may choose to identify their releases, while still ensuring that the "latest release" and the "latest stable release" can be easily determined, both by human users and automated tools. Simple "major.minor" versioning:: 0.1 0.2 0.3 1.0 1.1 ... Simple "major.minor.micro" versioning:: 1.1.0 1.1.1 1.1.2 1.2.0 ... "major.minor" versioning with alpha, beta and release candidate pre-releases:: 0.9 1.0a1 1.0a2 1.0b1 1.0c1 1.0 1.1a1 ... "major.minor" versioning with developmental releases, release candidates and post-releases for minor corrections:: 0.9 1.0.dev1 1.0.dev2 1.0.dev3 1.0.dev4 1.0rc1 1.0rc2 1.0 1.0.post1 1.1.dev1 ... Summary of permitted suffixes and relative ordering --------------------------------------------------- .. note:: This section is intended primarily for authors of tools that automatically process distribution metadata, rather than authors of Python distributions deciding on a versioning scheme. The numeric release component of version identifiers should be sorted in the same order as Python's tuple sorting when the release number is parsed as follows:: tuple(map(int, release_number.split("."))) Within a numeric release (``1.0``, ``2.7.3``), the following suffixes are permitted and are ordered as shown:: .devN, aN, bN, cN, rcN, , .postN Note that `rc` will always sort after `c` (regardless of the numeric component) although they are semantically equivalent. Tools are free to reject this case as ambiguous and remain in compliance with the PEP. Within an alpha (``1.0a1``), beta (``1.0b1``), or release candidate (``1.0c1``, ``1.0rc1``), the following suffixes are permitted and are ordered as shown:: .devN, , .postN Within a post-release (``1.0.post1``), the following suffixes are permitted and are ordered as shown:: devN, Note that ``devN`` and ``postN`` must always be preceded by a dot, even when used immediately following a numeric version (e.g. ``1.0.dev456``, ``1.0.post1``). Within a given suffix, ordering is by the value of the numeric component. The following example covers many of the possible combinations:: 1.0.dev456 1.0a1 1.0a2.dev456 1.0a12.dev456 1.0a12 1.0b1.dev456 1.0b2 1.0b2.post345.dev456 1.0b2.post345 1.0c1.dev456 1.0c1 1.0 1.0.post456.dev34 1.0.post456 1.1.dev1 Version ordering across different metadata versions --------------------------------------------------- Metadata v1.0 (PEP 241) and metadata v1.1 (PEP 314) do not specify a standard version identification or ordering scheme. This PEP does not mandate any particular approach to handling such versions, but acknowledges that the de facto standard for ordering them is the scheme used by the ``pkg_resources`` component of ``setuptools``. Software that automatically processes distribution metadata may either treat non-compliant version identifiers as an error, or attempt to normalize them to the standard scheme. This means that projects using non-compliant version identifiers may not be handled consistently across different tools, even when correctly publishing the earlier metadata versions. Distribution developers can help ensure consistent automated handling by marking non-compliant versions as "hidden" on the Python Package Index (removing them is generally undesirable, as users may be depending on those specific versions being available). Distribution users may also wish to remove non-compliant versions from any private package indexes they control. For metadata v1.2 (PEP 345), the version ordering described in this PEP should be used in preference to the one defined in PEP 386. Compatibility with other version schemes ---------------------------------------- Some projects may choose to use a version scheme which requires translation in order to comply with the public version scheme defined in this PEP. In such cases, the `Private-Version`__ field can be used to record the project specific version as an arbitrary label, while the translated public version is given in the `Version`_ field. __ `Private-Version (optional)`_ This allows automated distribution tools to provide consistently correct ordering of published releases, while still allowing developers to use the internal versioning scheme they prefer for their projects. Semantic versioning ~~~~~~~~~~~~~~~~~~~ `Semantic versioning`_ is a popular version identification scheme that is more prescriptive than this PEP regarding the significance of different elements of a release number. Even if a project chooses not to abide by the details of semantic versioning, the scheme is worth understanding as it covers many of the issues that can arise when depending on other distributions, and when publishing a distribution that others rely on. The "Major.Minor.Patch" (described in this PEP as "major.minor.micro") aspects of semantic versioning (clauses 1-9 in the 2.0.0-rc-1 specification) are fully compatible with the version scheme defined in this PEP, and abiding by these aspects is encouraged. Semantic versions containing a hyphen (pre-releases - clause 10) or a plus sign (builds - clause 11) are *not* compatible with this PEP and are not permitted in the public `Version`_ field. One possible mechanism to translate such private semantic versions to compatible public versions is to use the ``.devN`` suffix to specify the appropriate version order. .. _Semantic versioning: http://semver.org/ DVCS based version labels ~~~~~~~~~~~~~~~~~~~~~~~~~ Many build tools integrate with distributed version control systems like Git and Mercurial in order to add an identifying hash to the version identifier. As hashes cannot be ordered reliably such versions are not permitted in the public `Version`_ field. As with semantic versioning, the public ``.devN`` suffix may be used to uniquely identify such releases for publication, while the private version field is used to record the original version label. Date based versions ~~~~~~~~~~~~~~~~~~~ As with other incompatible version schemes, date based versions can be stored in the ``Private-Version`` field. Translating them to a compliant version is straightforward: the simplest approach is to subtract the year of the first release from the major component in the release number. Version specifiers ================== A version specifier consists of a series of version clauses, separated by commas. Each version clause consists of an optional comparison operator followed by a version identifier. For example:: 0.9, >= 1.0, != 1.3.4, < 2.0 Each version identifier must be in the standard format described in `Version scheme`_. The comma (",") is equivalent to a logical **and** operator. Comparison operators must be one of ``<``, ``>``, ``<=``, ``>=``, ``==`` or ``!=``. The ``==`` and ``!=`` operators are strict - in order to match, the version supplied must exactly match the specified version, with no additional trailing suffix. However, when no comparison operator is provided along with a version identifier ``V``, it is equivalent to using the following pair of version clauses:: >= V, < V+1 where ``V+1`` is the next version after ``V``, as determined by incrementing the last numeric component in ``V`` (for example, if ``V == 1.0a3``, then ``V+1 == 1.0a4``, while if ``V == 1.0``, then ``V+1 == 1.1``). This approach makes it easy to depend on a particular release series simply by naming it in a version specifier, without requiring any additional annotation. For example, the following pairs of version specifiers are equivalent:: 2 >= 2, < 3 3.3 >= 3.3, < 3.4 Whitespace between a conditional operator and the following version identifier is optional, as is the whitespace around the commas. Pre-releases of any kind, including developmental releases, are implicitly excluded from all version specifiers, *unless* a pre-release or developmental developmental release is explicitly mentioned in one of the clauses. For example, this specifier implicitly excludes all pre-releases and development releases of later versions:: >= 1.0 While these specifiers would include them:: >= 1.0a1 >= 1.0c1 >= 1.0, != 1.0b2 >= 1.0, < 2.0.dev123 Dependency resolution tools should use the above rules by default, but should also allow users to request the following alternative behaviours: * accept already installed pre-releases for all version specifiers * retrieve and install available pre-releases for all version specifiers Dependency resolution tools may also allow the above behaviour to be controlled on a per-distribution basis. Post-releases and purely numeric releases receive no special treatment - they are always included unless explicitly excluded. Given the above rules, projects which include the ``.0`` suffix for the first release in a series, such as ``2.5.0``, can easily refer specifically to that version with the clause ``2.5.0``, while the clause ``2.5`` refers to that entire series. Projects which omit the ".0" suffix for the first release of a series, by using a version string like ``2.5`` rather than ``2.5.0``, will need to use an explicit clause like ``>= 2.5, < 2.5.1`` to refer specifically to that initial release. Some examples: * ``Requires-Dist: zope.interface (3.1)``: any version that starts with 3.1, excluding pre-releases. * ``Requires-Dist: zope.interface (==3.1)``: equivalent to ``Requires-Dist: zope.interface (3.1)``. * ``Requires-Dist: zope.interface (3.1.0)``: any version that starts with 3.1.0, excluding pre-releases. Since that particular project doesn't use more than 3 digits, it also means "only the 3.1.0 release". * ``Requires-Python: 3``: Any Python 3 version, excluding pre-releases. * ``Requires-Python: >=2.6,<3``: Any version of Python 2.6 or 2.7, including post-releases (if they were used for Python). It excludes pre releases of Python 3. * ``Requires-Python: 2.6.2``: Equivalent to ">=2.6.2,<2.6.3". So this includes only Python 2.6.2. Of course, if Python was numbered with 4 digits, it would include all versions of the 2.6.2 series, excluding pre-releases. * ``Requires-Python: 2.5``: Equivalent to ">=2.5,<2.6". * ``Requires-Dist: zope.interface (3.1,!=3.1.3)``: any version that starts with 3.1, excluding pre-releases of 3.1 *and* excluding any version that starts with "3.1.3". For this particular project, this means: "any version of the 3.1 series but not 3.1.3". This is equivalent to: ">=3.1,!=3.1.3,<3.2". * ``Requires-Python: >=3.3a1``: Any version of Python 3.3+, including pre-releases like 3.4a1. Depending on distributions that use non-compliant version schemes ----------------------------------------------------------------- A distribution using this version of the metadata standard may need to depend on another distribution using an earlier version of the metadata standard and a non-compliant versioning scheme. The normal ``Requires-Dist`` and ``Setup-Requires-Dist`` fields can be used for such dependencies, so long as the dependency itself can be expressed using a compliant version specifier. For more exotic dependencies, a metadata extension would be needed in order to express the dependencies accurately while still obeying the restrictions on standard version specifiers. The ``Requires-External`` field may also be used, but would not be as amenable to automatic processing. Environment markers =================== An **environment marker** is a marker that can be added at the end of a field after a semi-colon (";"), to add a condition about the execution environment. Here are some example of fields using such markers:: Requires-Dist: pywin32 (>1.0); sys.platform == 'win32' Requires-Dist: foo (1,!=1.3); platform.machine == 'i386' Requires-Dist: bar; python_version == '2.4' or python_version == '2.5' Requires-External: libxslt; 'linux' in sys.platform The micro-language behind this is a simple subset of Python: it compares only strings, with the ``==`` and ``in`` operators (and their opposites), and with the ability to combine expressions. Parentheses are supported for grouping. The pseudo-grammar is :: MARKER: EXPR [(and|or) EXPR]* EXPR: ("(" MARKER ")") | (SUBEXPR [(in|==|!=|not in) SUBEXPR]) where ``SUBEXPR`` belongs to any of the following (the details after the colon in each entry define the value represented by that subexpression): * ``python_version``: '%s.%s' % (sys.version_info[0], sys.version_info[1]) * ``python_full_version``: sys.version.split()[0] * ``os.name````: os.name * ``sys.platform````: sys.platform * ``platform.version``: platform.version() * ``platform.machine``: platform.machine() * ``platform.python_implementation``: = platform.python_implementation() * ``extra``: (name of requested feature) or None * ``'text'``: a free string, like ``'2.4'``, or ``'win32'`` Notice that ``in`` and ``not in`` are restricted to strings, meaning that it is not possible to use other sequences like tuples or lists on the right side. The fields that benefit from this marker are: * ``Requires-Python`` * ``Requires-External`` * ``Requires-Dist`` * ``Provides-Dist`` * ``Classifier`` Optional features ================= Distributions may use the ``Provides-Extra`` field to declare additional features that they provide. Environment markers may then be used to indicate that particular dependencies are needed only when a particular optional feature has been requested. Other distributions then require an optional feature by placing it inside square brackets after the distribution name when declaring the dependency. Multiple features can be requisted by separating them with a comma within the brackets. The full set of dependency requirements is then the union of the sets created by first evaluating the `Requires-Dist` fields with `extra` set to `None` and then to the name of each requested feature. Example:: Requires-Dist: beaglevote[pdf] -> requires beaglevote, reportlab at run time Setup-Requires-Dist: beaglevote[test, doc] -> requires beaglevote, sphinx, nose at setup time It is legal to specify `Provides-Extra` without referencing it in any `Requires-Dist`. It is an error to request a feature name that has not been declared with `Provides-Extra`. The following feature names are implicitly defined for all distributions: - `test`: dependencies that are needed in order to run automated tests - `doc`: dependencies that are needed in order to generate documentation Listing these implicit features explicitly in a ``Provides-Extra`` field is permitted, but not required. Updating the metadata specification =================================== The metadata specification may be updated with clarifications without requiring a new PEP or a change to the metadata version. Adding new features (other than through the extension mechanism), or changing the meaning of existing fields, requires a new metadata version defined in a new PEP. Summary of differences from \PEP 345 ==================================== * Metadata-Version is now 2.0, with semantics specified for handling version changes * Most fields are now optional * Explicit permission for in-place clarifications without releasing a new version of the specification * General reformatting of the PEP to make it easier to read * Values are now expected to be UTF-8 * Changed the version scheme * added the new ``Private-Version`` field * changed the top level sort position of the ``.devN`` suffix * allowed single value version numbers * explicit exclusion of leading or trailing whitespace * explicit criterion for the exclusion of date based versions * incorporated the version scheme directly into the PEP * Changed interpretation of version specifiers * implicitly exclude pre-releases unless explicitly requested * treat post releases the same way as unqualified releases * Discuss ordering and dependencies across metadata versions * Clarify use of parentheses for grouping in environment marker pseudo-grammar * Support for packaging, build and installation dependencies * the new ``Setup-Requires-Dist`` field * Optional feature mechanism * the new ``Provides-Extra`` field * ``extra`` expression defined for environment markers * optional feature support in ``Requires-Dist`` * Metadata extension mechanism * the new ``Extension`` field and extension specific fields * Updated obsolescence mechanism * the new ``Obsoleted-By`` field * the ``Obsoletes-Dist`` field has been removed * Simpler description format * the ``Description`` field is now deprecated * A payload (containing the description) may appear after the headers. * Other changed fields: - ``Requires-Python`` (explicitly flagged as multiple use) - ``Project-URL`` (commas permitted in labels) * Clarified fields: - ``Provides-Dist`` - ``Keywords`` The rationale for major changes is given in the following sections. Metadata-Version semantics -------------------------- The semantics of major and minor version increments are now specified, and follow the same model as the format version semantics specified for the wheel format in PEP 427: minor version increments must behave reasonably when processed by a tool that only understand earlier metadata versions with the same major version, while major version increments may include changes that are not compatible with existing tools. The major version number of the specification has been incremented accordingly, as interpreting PEP 426 metadata in accordance with earlier metadata specifications is unlikely to give the expected behaviour. Whenever the major version number of the specification is incremented, it is expected that deployment will take some time, as metadata consuming tools much be updated before other tools can safely start producing the new format. Standard encoding and other format clarifications ------------------------------------------------- Several aspects of the file format, including the expected file encoding, were underspecified in previous versions of the metadata standard. To make it easier to develop interoperable tools, these details are now explicitly specified. Changing the version scheme --------------------------- The new ``Private-Version`` field is intended to make it clearer that the constraints on public version identifiers are there primarily to aid in the creation of reliable automated dependency analysis tools. Projects are free to use whatever versioning scheme they like internally, so long as they are able to translate it to something the dependency analysis tools will understand. The key change in the version scheme in this PEP relative to that in PEP 386 is to sort top level developmental releases like ``X.Y.devN`` ahead of alpha releases like ``X.Ya1``. This is a far more logical sort order, as projects already using both development releases and alphas/betas/release candidates do not want their developmental releases sorted in between their release candidates and their full releases. There is no rationale for using ``dev`` releases in that position rather than merely creating additional release candidates. The updated sort order also means the sorting of ``dev`` versions is now consistent between the metadata standard and the pre-existing behaviour of ``pkg_resources`` (and hence the behaviour of current installation tools). Making this change should make it easier for affected existing projects to migrate to the latest version of the metadata standard. Another change to the version scheme is to allow single number versions, similar to those used by non-Python projects like Mozilla Firefox, Google Chrome and the Fedora Linux distribution. This is actually expected to be more useful for version specifiers (allowing things like the simple ``Requires-Python: 3`` rather than the more convoluted ``Requires-Python: >= 3.0, < 4``), but it is easier to allow it for both version specifiers and release numbers, rather than splitting the two definitions. The exclusion of leading and trailing whitespace was made explicit after a couple of projects with version identifiers differing only in a trailing ``\n`` character were found on PyPI. The exclusion of major release numbers that looks like dates was implied by the overall text of PEP 386, but not clear in the definition of the version scheme. This exclusion has been made clear in the definition of the release component. Finally, as the version scheme in use is dependent on the metadata version, it was deemed simpler to merge the scheme definition directly into this PEP rather than continuing to maintain it as a separate PEP. This will also allow all of the distutils-specific elements of PEP 386 to finally be formally rejected. The following statistics provide an analysis of the compatibility of existing projects on PyPI with the specified versioning scheme (as of 16th February, 2013). * Total number of distributions analysed: 28088 * Distributions with no releases: 248 / 28088 (0.88 %) * Fully compatible distributions: 24142 / 28088 (85.95 %) * Compatible distributions after translation: 2830 / 28088 (10.08 %) * Compatible distributions after filtering: 511 / 28088 (1.82 %) * Distributions sorted differently after translation: 38 / 28088 (0.14 %) * Distributions sorted differently without translation: 2 / 28088 (0.01 %) * Distributions with no compatible releases: 317 / 28088 (1.13 %) The two remaining sort order discrepancies picked up by the analysis are due to a pair of projects which have published releases ending with a carriage return, alongside releases with the same version number, only *without* the trailing carriage return. The sorting discrepancies after translation relate mainly to differences in the handling of pre-releases where the standard mechanism is considered to be an improvement. For example, the existing pkg_resources scheme will sort "1.1beta1" *after* "1.1b2", whereas the suggested standard translation for "1.1beta1" is "1.1b1", which sorts *before* "1.1b2". Similarly, the pkg_resources scheme will sort "-dev-N" pre-releases differently from "devN" pre-releases when they occur within the same release, while the standard scheme will normalize both representations to ".devN" and sort them by the numeric component. For comparison, here are the corresponding analysis results for PEP 386: * Total number of distributions analysed: 28088 * Distributions with no releases: 248 / 28088 (0.88 %) * Fully compatible distributions: 23874 / 28088 (85.00 %) * Compatible distributions after translation: 2786 / 28088 (9.92 %) * Compatible distributions after filtering: 527 / 28088 (1.88 %) * Distributions sorted differently after translation: 96 / 28088 (0.34 %) * Distributions sorted differently without translation: 14 / 28088 (0.05 %) * Distributions with no compatible releases: 543 / 28088 (1.93 %) These figures make it clear that only a relatively small number of current projects are affected by these changes. However, some of the affected projects are in widespread use (such as Pinax and selenium). The changes also serve to bring the standard scheme more into line with developer's expectations, which is an important element in encouraging adoption of the new metadata version. The script used for the above analysis is available at [3]_. A more opinionated description of the versioning scheme ------------------------------------------------------- As in PEP 386, the primary focus is on codifying existing practices to make them more amenable to automation, rather than demanding that existing projects make non-trivial changes to their workflow. However, the standard scheme allows significantly more flexibility than is needed for the vast majority of simple Python packages (which often don't even need maintenance releases - many users are happy with needing to upgrade to a new feature release to get bug fixes). For the benefit of novice developers, and for experienced developers wishing to better understand the various use cases, the specification now goes into much greater detail on the components of the defined version scheme, including examples of how each component may be used in practice. The PEP also explicitly guides developers in the direction of semantic versioning (without requiring it), and discourages the use of several aspects of the full versioning scheme that have largely been included in order to cover esoteric corner cases in the practices of existing projects and in repackaging software for Linux distributions. Changing the interpretation of version specifiers ------------------------------------------------- The previous interpretation of version specifiers made it very easy to accidentally download a pre-release version of a dependency. This in turn made it difficult for developers to publish pre-release versions of software to the Python Package Index, as leaving the package set as public would lead to users inadvertently downloading pre-release software, while hiding it would defeat the purpose of publishing it for user testing. The previous interpretation also excluded post-releases from some version specifiers for no adequately justified reason. The updated interpretation is intended to make it difficult to accidentally accept a pre-release version as satisfying a dependency, while allowing pre-release versions to be explicitly requested when needed. Packaging, build and installation dependencies ---------------------------------------------- The new ``Setup-Requires-Dist`` field allows a distribution to indicate when a dependency is needed to package, build or install the distribution, rather than being needed to run the software after installation. This should allow distribution tools to effectively support a wider range of distribution requirements. Support for optional features of distributions ---------------------------------------------- The new ``Provides-Extra`` field allows distributions to declare optional features, and to use environment markers to reduce their dependencies when those features are not requested. Environment markers may also be used to require a later version of Python when particular features are requested. The ``Requires-Dist`` and ``Setup-Requires-Dist`` fields then allow distributions to require optional features of other distributions. The ``test`` and ``doc`` features are implicitly defined for all distributions, as one key motivation for this feature is to encourage distributions to explicitly declare the dependencies needed to run their automatic tests, or build their documentation, without demanding those dependencies be present in order to merely install or use the software. Support for metadata extensions ------------------------------- The new ``Extension`` field effectively allows sections of the metadata namespace to be delegated to other distributions, while preserving a standard overal format metadata format for easy of processing by distribution tools that do not support a particular extension. It also works well in combination with the new ``Setup-Requires-Dist`` field to allow a distribution to depend on tools which *do* know how to handle the chosen extension, and the new optional features mechanism, allowing support for particular extensions to be provided as optional features. Updated obsolescence mechanism ------------------------------ The marker to indicate when a project is obsolete and should be replaced has been moved to the obsolete project (the new ``Obsoleted-By`` field), replacing the previous marker on the replacement project (the removed ``Obsoletes-Dist`` field). This should allow distribution tools to more easily warn users of obsolete projects and their suggested replacements. The ``Obsoletes-Dist`` header is removed rather than deprecated as it is not widely supported, and so removing it does not present any significant barrier to tools and projects adopting the new metadata format. Simpler description format -------------------------- Distribution descriptions are often quite long, sometimes including a short guide to using the module. Moving them into the file payload allows them to be formatted neatly as reStructuredText without needing to carefully avoid the introduction of a blank line that would terminate the header section. The ``Description`` header is deprecated rather than removed to support easier conversion of existing tools and projects to the new metadata format. References ========== This document specifies version 2.0 of the metadata format. Version 1.0 is specified in PEP 241. Version 1.1 is specified in PEP 314. Version 1.2 is specified in PEP 345. The initial attempt at a standardised version scheme, along with the justifications for needing such a standard can be found in PEP 386. .. [1] reStructuredText markup: http://docutils.sourceforge.net/ .. _`Python Package Index`: http://pypi.python.org/pypi/ .. [2] PEP 301: http://www.python.org/dev/peps/pep-0301/ .. [3] Version compatibility analysis script: http://hg.python.org/peps/file/default/pep-0426/pepsort.py Appendix ======== Parsing and generating the Metadata 2.0 serialization format using Python 3.3:: # Metadata 2.0 demo from email.generator import Generator from email import header from email.parser import Parser from email.policy import Compat32 from email.utils import _has_surrogates class MetadataPolicy(Compat32): max_line_length = 0 continuation_whitespace = '\t' def _sanitize_header(self, name, value): if not isinstance(value, str): return value if _has_surrogates(value): raise NotImplementedError() else: return value def _fold(self, name, value, sanitize): body = ((self.linesep+self.continuation_whitespace) .join(value.splitlines())) return ''.join((name, ': ', body, self.linesep)) if __name__ == "__main__": import sys import textwrap pkg_info = """\ Metadata-Version: 2.0 Name: package Version: 0.1.0 Summary: A package. Description: Description =========== A description of the package. """ m = Parser(policy=MetadataPolicy()).parsestr(pkg_info) m['License'] = 'GPL' description = m['Description'] description_lines = description.splitlines() m.set_payload(description_lines[0] + '\n' + textwrap.dedent('\n'.join(description_lines[1:])) + '\n') del m['Description'] # Correct if sys.stdout.encoding == 'UTF-8': Generator(sys.stdout, maxheaderlen=0).flatten(m) Copyright ========= This document has been placed in the public domain. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From solipsis at pitrou.net Sun Feb 17 18:40:03 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sun, 17 Feb 2013 18:40:03 +0100 Subject: [Python-Dev] PEP 426 is now the draft spec for distribution metadata 2.0 References: Message-ID: <20130217184003.0faa841c@pitrou.net> On Sun, 17 Feb 2013 20:11:23 +1000 Nick Coghlan wrote: > > Major changes since the last draft: > > 1. Metadata-Version is 2.0 rather than 1.3, and the field now has the > same major.minor semantics as are defined for wheel versions in PEP > 427 (i.e. if a tool sees a major version number it doesn't recognise, > it should give up rather than trying to guess what to do with it, > while it's OK to process a higher minor version) So is it possible to bundle several versioned metadata files in order to maximize compatibility? Or does a distribution with 2.0 metadata have to abandon all hope of being compatible with all tools? Regards Antoine. From dholth at gmail.com Sun Feb 17 19:45:00 2013 From: dholth at gmail.com (Daniel Holth) Date: Sun, 17 Feb 2013 13:45:00 -0500 Subject: [Python-Dev] PEP 426 is now the draft spec for distribution metadata 2.0 In-Reply-To: <20130217184003.0faa841c@pitrou.net> References: <20130217184003.0faa841c@pitrou.net> Message-ID: On Sun, 17 Feb 2013 20:11:23 +1000 Nick Coghlan wrote: > > Major changes since the last draft: > > 1. Metadata-Version is 2.0 rather than 1.3, and the field now has the > same major.minor semantics as are defined for wheel versions in PEP > 427 (i.e. if a tool sees a major version number it doesn't recognise, > it should give up rather than trying to guess what to do with it, > while it's OK to process a higher minor version) So is it possible to bundle several versioned metadata files in order to maximize compatibility? Or does a distribution with 2.0 metadata have to abandon all hope of being compatible with all tools? Regards Antoine. _______________________________________________ Python-Dev mailing list Python-Dev at python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/dholth%40gmail.com Not likely to matter for a while as the current md v1 tools don't understand this new obsolescence rule :-) -------------- next part -------------- An HTML attachment was scrubbed... URL: From fred at fdrake.net Sun Feb 17 19:49:39 2013 From: fred at fdrake.net (Fred Drake) Date: Sun, 17 Feb 2013 13:49:39 -0500 Subject: [Python-Dev] PEP 426 is now the draft spec for distribution metadata 2.0 In-Reply-To: References: <20130217184003.0faa841c@pitrou.net> Message-ID: On Sun, Feb 17, 2013 at 1:45 PM, Daniel Holth wrote: > Not likely to matter for a while as the current md v1 tools don't understand > this new obsolescence rule :-) Using a separate file for post-obsolescence-rule metadata versions would allow coexistance, which would likely improve adoption. -Fred -- Fred L. Drake, Jr. "A storm broke loose in my mind." --Albert Einstein From ncoghlan at gmail.com Sun Feb 17 22:21:24 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 18 Feb 2013 07:21:24 +1000 Subject: [Python-Dev] PEP 426 is now the draft spec for distribution metadata 2.0 In-Reply-To: References: <20130217184003.0faa841c@pitrou.net> Message-ID: On 18 Feb 2013 04:54, "Fred Drake" wrote: > > On Sun, Feb 17, 2013 at 1:45 PM, Daniel Holth wrote: > > Not likely to matter for a while as the current md v1 tools don't understand > > this new obsolescence rule :-) > > Using a separate file for post-obsolescence-rule metadata versions would > allow coexistance, which would likely improve adoption. Perhaps, but not something we actually need to worry about until we're contemplating metadata 3.0, especially since whether or not it helps will depend on why we're breaking backwards compatibility. At the very least, updated wheel, sdist and installation database specs would be needed to allow multiple versions of the metadata. As Daniel pointed out, easy_install and pip also don't follow this rule yet, so it won't really have any impact if we never get to metadata 3.0. Cheers, Nick. > > > -Fred > > -- > Fred L. Drake, Jr. > "A storm broke loose in my mind." --Albert Einstein > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From fred at fdrake.net Sun Feb 17 23:36:24 2013 From: fred at fdrake.net (Fred Drake) Date: Sun, 17 Feb 2013 17:36:24 -0500 Subject: [Python-Dev] PEP 426 is now the draft spec for distribution metadata 2.0 In-Reply-To: References: <20130217184003.0faa841c@pitrou.net> Message-ID: On Sun, Feb 17, 2013 at 4:21 PM, Nick Coghlan wrote: > As Daniel pointed out, easy_install and pip also don't follow this rule yet, > so it won't really have any impact if we never get to metadata 3.0. Actually, my point was that using a separate filename for version 2.0 would allow provision of both 1.x and 2.0, so version 2.0 metadata can be deployed as tool support becomes available, instead of having to wait until 1.x tools are replaced. Once tools are following the new rule about version compatibility, there's less worry about this (at least on my part). -Fred -- Fred L. Drake, Jr. "A storm broke loose in my mind." --Albert Einstein From ncoghlan at gmail.com Mon Feb 18 00:54:12 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 18 Feb 2013 09:54:12 +1000 Subject: [Python-Dev] PEP 426 is now the draft spec for distribution metadata 2.0 In-Reply-To: References: <20130217184003.0faa841c@pitrou.net> Message-ID: On 18 Feb 2013 08:36, "Fred Drake" wrote: > > On Sun, Feb 17, 2013 at 4:21 PM, Nick Coghlan wrote: > > As Daniel pointed out, easy_install and pip also don't follow this rule yet, > > so it won't really have any impact if we never get to metadata 3.0. > > Actually, my point was that using a separate filename for version 2.0 would > allow provision of both 1.x and 2.0, so version 2.0 metadata can be deployed > as tool support becomes available, instead of having to wait until 1.x tools > are replaced. > > Once tools are following the new rule about version compatibility, there's > less worry about this (at least on my part). None of sdist, wheel or the installation database currently support parallel metadata versions. Interpreting 2.0 metadata as 1.1 also won't be catastrophically wrong in general, especially if the production tools still also ship the legacy dependency and entry points data as separate files. The worst thing that is likely to happen is an old tool may fail to show a 2.0 style description. However, I *will* update the PEP to state explicitly that tools should emit the lowest required metadata version for the fields being published. Cheers, Nick. > > > -Fred > > -- > Fred L. Drake, Jr. > "A storm broke loose in my mind." --Albert Einstein -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Mon Feb 18 13:11:29 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 18 Feb 2013 22:11:29 +1000 Subject: [Python-Dev] PEP 426 is now the draft spec for distribution metadata 2.0 In-Reply-To: References: <20130217184003.0faa841c@pitrou.net> Message-ID: On Mon, Feb 18, 2013 at 9:54 AM, Nick Coghlan wrote: > > On 18 Feb 2013 08:36, "Fred Drake" wrote: >> >> On Sun, Feb 17, 2013 at 4:21 PM, Nick Coghlan wrote: >> > As Daniel pointed out, easy_install and pip also don't follow this rule >> > yet, >> > so it won't really have any impact if we never get to metadata 3.0. >> >> Actually, my point was that using a separate filename for version 2.0 >> would >> allow provision of both 1.x and 2.0, so version 2.0 metadata can be >> deployed >> as tool support becomes available, instead of having to wait until 1.x >> tools >> are replaced. >> >> Once tools are following the new rule about version compatibility, there's >> less worry about this (at least on my part). > > None of sdist, wheel or the installation database currently support parallel > metadata versions. Interpreting 2.0 metadata as 1.1 also won't be > catastrophically wrong in general, especially if the production tools still > also ship the legacy dependency and entry points data as separate files. The > worst thing that is likely to happen is an old tool may fail to show a 2.0 > style description. > > However, I *will* update the PEP to state explicitly that tools should emit > the lowest required metadata version for the fields being published. I clarified the metadata version handling here: http://hg.python.org/peps/rev/630c5dd5a123 Rather than "should", I softened the "use the lowest metadata format" suggestion to a "may". I also pointed out that these new rules will only apply to a future major version transition, and for this one, people can just emit the extra files they do today and even older versions of tools will probably get things fairly right. Cheers, Nick. diff --git a/pep-0426.txt b/pep-0426.txt --- a/pep-0426.txt +++ b/pep-0426.txt @@ -89,9 +89,15 @@ Version of the file format; "2.0" is the only legal value. -Automated tools should warn if ``Metadata-Version`` is greater than the -highest version they support, and must fail if ``Metadata-Version`` has -a greater major version than the highest version they support. +Automated tools consuming metadata should warn if ``Metadata-Version`` is +greater than the highest version they support, and must fail if +``Metadata-Version`` has a greater major version than the highest +version they support. + +For broader compatibility, automated tools may choose to produce +distribution metadata using the lowest metadata version that includes +all of the needed fields. + Example:: @@ -1283,9 +1289,19 @@ metadata specifications is unlikely to give the expected behaviour. Whenever the major version number of the specification is incremented, it -is expected that deployment will take some time, as metadata consuming tools -much be updated before other tools can safely start producing the new -format. +is expected that deployment will take some time, as either metadata +consuming tools must be updated before other tools can safely start +producing the new format, or else the sdist and wheel formats, along with +the installation database definition, will need to be updated to support +provision of multiple versions of the metadata in parallel. + +Existing tools won't abide by this guideline until they're updated to +support the new metadata standard, so the new semantics will first take +effect for a hypothetical 2.x -> 3.0 transition. For the 1.x -> 2.0 +transition, it is recommended that tools continue to produce the +existing supplementary files (such as ``entry_points.txt``) in addition +to any equivalents specified using the new features of the standard +metadata format (including the formal extension mechanism). -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From j.reid at mail.cryst.bbk.ac.uk Mon Feb 18 15:32:26 2013 From: j.reid at mail.cryst.bbk.ac.uk (John Reid) Date: Mon, 18 Feb 2013 14:32:26 +0000 Subject: [Python-Dev] Rationale for different signatures of tuple.__new__ and namedtuple.__new__ Message-ID: Hi, I can do tuple([1,2,3]) but not: from collections import namedtuple namedtuple('B', 'x y z')([1,2,3]) I get a TypeError: __new__() takes exactly 4 arguments (2 given) However I can do: namedtuple('B', 'x y z')._make([1,2,3]) So namedtuple's _make classmethod looks a lot like tuple's __new__(). What's the rationale for this? Wouldn't it be better to share the same signature for __new__? IPython's serialization code depends on a tuple's __new__() having the signature that tuple has and therefore does not work with namedtuples. See discussion here: http://article.gmane.org/gmane.comp.python.general/726849 Thanks, John. From hrvoje.niksic at avl.com Mon Feb 18 16:02:10 2013 From: hrvoje.niksic at avl.com (Hrvoje Niksic) Date: Mon, 18 Feb 2013 16:02:10 +0100 Subject: [Python-Dev] Rationale for different signatures of tuple.__new__ and namedtuple.__new__ In-Reply-To: References: Message-ID: <51224272.10701@avl.com> On 02/18/2013 03:32 PM, John Reid wrote: > I can do > > tuple([1,2,3]) > > but not: > > from collections import namedtuple > namedtuple('B', 'x y z')([1,2,3]) > > I get a TypeError: __new__() takes exactly 4 arguments (2 given) > However I can do: > > namedtuple('B', 'x y z')._make([1,2,3]) > > So namedtuple's _make classmethod looks a lot like tuple's __new__(). > What's the rationale for this? Wouldn't it be better to share the same > signature for __new__? Sharing the constructor signature with tuple would break the common case of: namedtuple('B', 'x y z')(1, 2, 3) From solipsis at pitrou.net Mon Feb 18 18:26:09 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 18 Feb 2013 18:26:09 +0100 Subject: [Python-Dev] cpython (2.7): Fix posixpath.realpath() for multiple pardirs (fixes issue #6975). References: <3Z8h5h0bR3zPl5@mail.python.org> Message-ID: <20130218182609.4eb5ebf6@pitrou.net> On Mon, 18 Feb 2013 11:24:40 +0100 (CET) serhiy.storchaka wrote: > > + def test_realpath_curdir(self): > + self.assertEqual(realpath('.'), os.getcwd()) > + self.assertEqual(realpath('./.'), os.getcwd()) > + self.assertEqual(realpath('/'.join(['.'] * 100)), os.getcwd()) > + > + def test_realpath_pardir(self): > + self.assertEqual(realpath('..'), dirname(os.getcwd())) > + self.assertEqual(realpath('../..'), dirname(dirname(os.getcwd()))) > + self.assertEqual(realpath('/'.join(['..'] * 100)), '/') What if there's a symlink along os.getcwd()? Regards Antoine. From storchaka at gmail.com Mon Feb 18 18:56:07 2013 From: storchaka at gmail.com (Serhiy Storchaka) Date: Mon, 18 Feb 2013 19:56:07 +0200 Subject: [Python-Dev] cpython (2.7): Fix posixpath.realpath() for multiple pardirs (fixes issue #6975). In-Reply-To: <20130218182609.4eb5ebf6@pitrou.net> References: <3Z8h5h0bR3zPl5@mail.python.org> <20130218182609.4eb5ebf6@pitrou.net> Message-ID: On 18.02.13 19:26, Antoine Pitrou wrote: > On Mon, 18 Feb 2013 11:24:40 +0100 (CET) > serhiy.storchaka wrote: >> >> + def test_realpath_curdir(self): >> + self.assertEqual(realpath('.'), os.getcwd()) >> + self.assertEqual(realpath('./.'), os.getcwd()) >> + self.assertEqual(realpath('/'.join(['.'] * 100)), os.getcwd()) >> + >> + def test_realpath_pardir(self): >> + self.assertEqual(realpath('..'), dirname(os.getcwd())) >> + self.assertEqual(realpath('../..'), dirname(dirname(os.getcwd()))) >> + self.assertEqual(realpath('/'.join(['..'] * 100)), '/') > > What if there's a symlink along os.getcwd()? 1. AFAIK, os.getcwd() returns the path with resolved symlinks. 2. realpath() first resolve relative path and then prepends cwd to the result. From solipsis at pitrou.net Mon Feb 18 19:02:04 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 18 Feb 2013 19:02:04 +0100 Subject: [Python-Dev] cpython (2.7): Fix posixpath.realpath() for multiple pardirs (fixes issue #6975). References: <3Z8h5h0bR3zPl5@mail.python.org> <20130218182609.4eb5ebf6@pitrou.net> Message-ID: <20130218190204.43f8ef78@pitrou.net> On Mon, 18 Feb 2013 19:56:07 +0200 Serhiy Storchaka wrote: > On 18.02.13 19:26, Antoine Pitrou wrote: > > On Mon, 18 Feb 2013 11:24:40 +0100 (CET) > > serhiy.storchaka wrote: > >> > >> + def test_realpath_curdir(self): > >> + self.assertEqual(realpath('.'), os.getcwd()) > >> + self.assertEqual(realpath('./.'), os.getcwd()) > >> + self.assertEqual(realpath('/'.join(['.'] * 100)), os.getcwd()) > >> + > >> + def test_realpath_pardir(self): > >> + self.assertEqual(realpath('..'), dirname(os.getcwd())) > >> + self.assertEqual(realpath('../..'), dirname(dirname(os.getcwd()))) > >> + self.assertEqual(realpath('/'.join(['..'] * 100)), '/') > > > > What if there's a symlink along os.getcwd()? > > 1. AFAIK, os.getcwd() returns the path with resolved symlinks. Indeed, it seems you are right (under POSIX at least): ?The getcwd() function shall place an absolute pathname of the current working directory in the array pointed to by buf, and return buf. The pathname shall contain no components that are dot or dot-dot, or are symbolic links.? http://pubs.opengroup.org/onlinepubs/9699919799/functions/getcwd.html Regards Antoine. From tjreedy at udel.edu Mon Feb 18 21:24:06 2013 From: tjreedy at udel.edu (Terry Reedy) Date: Mon, 18 Feb 2013 15:24:06 -0500 Subject: [Python-Dev] Rationale for different signatures of tuple.__new__ and namedtuple.__new__ In-Reply-To: References: Message-ID: Later today you posted "Differences creating tuples and collections.namedtuples" on python-list. http://thread.gmane.org/gmane.comp.python.general/726849 That *is* the proper place to make observations about current Python and ask questions about why it is the way it is and how to work around features that get in your way. So I will answer there, if I think more needs to be said, after reading the multiple current responses. Anyone else should do the same. -- Terry Jan Reedy From mal at egenix.com Tue Feb 19 10:37:49 2013 From: mal at egenix.com (M.-A. Lemburg) Date: Tue, 19 Feb 2013 10:37:49 +0100 Subject: [Python-Dev] Fwd: PEP 426 is now the draft spec for distribution metadata 2.0 In-Reply-To: References: Message-ID: <512347ED.2040905@egenix.com> On 17.02.2013 11:11, Nick Coghlan wrote: > FYI > > > ---------- Forwarded message ---------- > From: Nick Coghlan > Date: Sun, Feb 17, 2013 at 8:10 PM > Subject: PEP 426 is now the draft spec for distribution metadata 2.0 > To: "DistUtils mailing list\"\"" > > > The latest draft of PEP 426 is up at http://www.python.org/dev/peps/pep-0426/ > > Major changes since the last draft: > > 1. Metadata-Version is 2.0 rather than 1.3, and the field now has the > same major.minor semantics as are defined for wheel versions in PEP > 427 (i.e. if a tool sees a major version number it doesn't recognise, > it should give up rather than trying to guess what to do with it, > while it's OK to process a higher minor version) > > 2. The "Private-Version" field is added, and several examples are > given showing how to use it in conjunction with translated public > versions when a project wants to use a version numbering scheme that > the standard installation tools won't understand. > > 3. The environment markers section more clearly covers the need to > handle parentheses (this was mentioned in the text, but not the > pseudo-grammar), and the fields which support those markers have an > explicit cross-reference to that section of the spec. > > 4. Leading/trailing whitespace and date based versioning are > explicitly excluded from valid public versions > > 5. Version compatibility statistics are provided for this PEP relative > to PEP 386 (the analysis script has been added to the PEP repo if > anyone wants to check my numbers) > > 6. I have reclaimed BDFL-Delegate status (with Guido's approval) > > Since getting wheel support into pip no longer depends on this version > of the metadata spec, I plan to leave it open for comment for another > week, and then accept it next weekend if I don't hear any significant > objections. Overall, I think the meta data for Python packages is getting too complicated. Without a support module in the stdlib implementing the required parsing, evaluation and formatting mechanisms needed to analyze and write the format, I'm -1 on adding yet another format version on top of the stack. At the moment, discussing yet another version update is mostly academic, since not even version 1.2 has been picked up by the tools yet: distutils still writes version 1.1 meta data and doesn't even understand 1.2 meta data. The only tool in wide spread use that understands part of the 1.2 data is setuptools/distribute, but it can only understand the Requires-Dist field of that version of the spec (only because the 1.1 Requires field was deprecated) and interprets a Provides-Extra field which isn't even standard. All other 1.2 fields are ignored. setuptools/distribute still writes 1.1 meta-data. I've never seen environment markers being used or supported in the wild. I'm not against modernizing the format, but given that version 1.2 has been out for around 8 years now, without much following, I think we need to make the implementation bit a requirement before accepting the PEP. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Feb 19 2013) >>> Python Projects, Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From p.f.moore at gmail.com Tue Feb 19 11:24:15 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 19 Feb 2013 10:24:15 +0000 Subject: [Python-Dev] [Distutils] PEP 426 is now the draft spec for distribution metadata 2.0 In-Reply-To: <512347ED.2040905@egenix.com> References: <512347ED.2040905@egenix.com> Message-ID: On Tuesday, 19 February 2013, M.-A. Lemburg wrote: > > The only tool in wide spread use that understands part of the 1.2 data > is setuptools/distribute, but it can only understand the Requires-Dist > field of that version of the spec (only because the 1.1 Requires field was > deprecated) and interprets a Provides-Extra field which isn't > even standard. All other 1.2 fields are ignored. > setuptools/distribute still writes 1.1 meta-data. > > I've never seen environment markers being used or supported > in the wild. > > I'm not against modernizing the format, but given that version 1.2 > has been out for around 8 years now, without much following, > I think we need to make the implementation bit a requirement > before accepting the PEP. > The wheel project uses metadata 2.0 and environment markers - indeed, the PEP was written to formalise what wheel was implementing (specifically so that pip was happy to incorporate support). Standard library support was hampered by the difficulty of changing Distutils - an issue which may have gone away now. I agree that standard library support would be good, either via Distutils or by incorporating distlib, but I don't think it is essential for acceptance. Paul -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Tue Feb 19 11:28:25 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 19 Feb 2013 20:28:25 +1000 Subject: [Python-Dev] Fwd: PEP 426 is now the draft spec for distribution metadata 2.0 In-Reply-To: <512347ED.2040905@egenix.com> References: <512347ED.2040905@egenix.com> Message-ID: On Tue, Feb 19, 2013 at 7:37 PM, M.-A. Lemburg wrote: > On 17.02.2013 11:11, Nick Coghlan wrote: > I'm not against modernizing the format, but given that version 1.2 > has been out for around 8 years now, without much following, > I think we need to make the implementation bit a requirement > before accepting the PEP. It is being implemented in distlib, and the (short!) appendix to the PEP itself shows how to read the format using the standard library's email module. v2.0 is designed to fix many of the issues that prevented the adoption of v1.2, including tweaks to the standardised version scheme and the addition of a formal extension mechanism to avoid the ad hoc extensions that occurred with earlier metadata versions. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From solipsis at pitrou.net Tue Feb 19 11:36:09 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 19 Feb 2013 11:36:09 +0100 Subject: [Python-Dev] [Distutils] PEP 426 is now the draft spec for distribution metadata 2.0 References: <512347ED.2040905@egenix.com> Message-ID: <20130219113609.15a421fb@pitrou.net> Le Tue, 19 Feb 2013 10:24:15 +0000, Paul Moore a ?crit : > On Tuesday, 19 February 2013, M.-A. Lemburg wrote: > > > > > The only tool in wide spread use that understands part of the 1.2 > > data is setuptools/distribute, but it can only understand the > > Requires-Dist field of that version of the spec (only because the > > 1.1 Requires field was deprecated) and interprets a Provides-Extra > > field which isn't even standard. All other 1.2 fields are ignored. > > setuptools/distribute still writes 1.1 meta-data. > > > > I've never seen environment markers being used or supported > > in the wild. > > > > I'm not against modernizing the format, but given that version 1.2 > > has been out for around 8 years now, without much following, > > I think we need to make the implementation bit a requirement > > before accepting the PEP. > > > > The wheel project uses metadata 2.0 and environment markers - indeed, > the PEP was written to formalise what wheel was implementing > (specifically so that pip was happy to incorporate support). I'm unsure what this means. Does the "wheel project" (I thought wheel was a format? is it also a project?) reimplement its own wheel of metadata parsing and generating? > I agree that standard library support would be good, either via > Distutils or by incorporating distlib, but I don't think it is > essential for acceptance. I think Marc-Andr? is right that the acceptability of the standard should be judged on the availability of (preferably standard) implementations. If the standard isn't implemented, then perhaps it means it's too ambitious / too complex / victim of the second-system effect. So I kind of agree with him the PEP shouldn't be accepted until there's a decent patch pending for the stdlib. That's not a judgement on the PEP's quality. Just an advice to remain cautious, especially if the previous metadata version already enjoys a very poor adoption rate. By the way, pip may be nice, but it's still a third-party tool, and perhaps not even as widespread as distribute / setuptools. Regards Antoine. From solipsis at pitrou.net Tue Feb 19 11:41:24 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 19 Feb 2013 11:41:24 +0100 Subject: [Python-Dev] [Distutils] PEP 426 is now the draft spec for distribution metadata 2.0 References: <512347ED.2040905@egenix.com> <20130219113609.15a421fb@pitrou.net> Message-ID: <20130219114124.63be1df6@pitrou.net> Le Tue, 19 Feb 2013 11:36:09 +0100, Antoine Pitrou a ?crit : > > I think Marc-Andr? is right that the acceptability of the standard > should be judged on the availability of (preferably standard) > implementations. If the standard isn't implemented, then perhaps it > means it's too ambitious / too complex / victim of the second-system > effect. > > So I kind of agree with him the PEP shouldn't be accepted until > there's a decent patch pending for the stdlib. That's not a judgement > on the PEP's quality. Just an advice to remain cautious, especially > if the previous metadata version already enjoys a very poor adoption > rate. > > By the way, pip may be nice, but it's still a third-party tool, and > perhaps not even as widespread as distribute / setuptools. Well, Nick kind of proved my wrong in his reply to Marc-Andr? :-) Regards Antoine. From ncoghlan at gmail.com Tue Feb 19 12:42:20 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 19 Feb 2013 21:42:20 +1000 Subject: [Python-Dev] [Distutils] PEP 426 is now the draft spec for distribution metadata 2.0 In-Reply-To: <20130219113609.15a421fb@pitrou.net> References: <512347ED.2040905@egenix.com> <20130219113609.15a421fb@pitrou.net> Message-ID: On Tue, Feb 19, 2013 at 8:36 PM, Antoine Pitrou wrote: > Le Tue, 19 Feb 2013 10:24:15 +0000, > Paul Moore a ?crit : >> On Tuesday, 19 February 2013, M.-A. Lemburg wrote: >> >> > >> > The only tool in wide spread use that understands part of the 1.2 >> > data is setuptools/distribute, but it can only understand the >> > Requires-Dist field of that version of the spec (only because the >> > 1.1 Requires field was deprecated) and interprets a Provides-Extra >> > field which isn't even standard. All other 1.2 fields are ignored. >> > setuptools/distribute still writes 1.1 meta-data. >> > >> > I've never seen environment markers being used or supported >> > in the wild. >> > >> > I'm not against modernizing the format, but given that version 1.2 >> > has been out for around 8 years now, without much following, >> > I think we need to make the implementation bit a requirement >> > before accepting the PEP. >> > >> >> The wheel project uses metadata 2.0 and environment markers - indeed, >> the PEP was written to formalise what wheel was implementing >> (specifically so that pip was happy to incorporate support). > > I'm unsure what this means. Does the "wheel project" (I thought wheel > was a format? is it also a project?) reimplement its own wheel of > metadata parsing and generating? > >> I agree that standard library support would be good, either via >> Distutils or by incorporating distlib, but I don't think it is >> essential for acceptance. > > I think Marc-Andr? is right that the acceptability of the standard > should be judged on the availability of (preferably standard) > implementations. If the standard isn't implemented, then perhaps it > means it's too ambitious / too complex / victim of the second-system > effect. The most complex part of the PEP is still the version ordering, and that's implemented in the analysis script (initially by Vinay, with one small tweak by me to fix the regular expression for the new validity rules). Everything else was either already implemented for distutils2/distlib, or is a trivial check (like scanning for "a", "b", "c" or "dev" to identify pre-releases and filter them out if none of those characters are present in the corresponding version specifier). Nothing in the PEP is particularly original - almost all of it is either stolen from other build and packaging systems, or is designed to provide a *discoverable* alternative to existing setuptools/distribute/pip practices (specifically, the new extension mechanism means things like entry points can be defined in the standard metadata files without python-dev needing to get involved). Probably the most novel concepts in the PEP are "Obsoleted-By" (so a project can issue one last release to inform users directly that it is dead, and there is an alternative available) and the "Private-Version" field to handle the case where a projects' preferred version scheme requires translation to match the more restricted format understood by the installation tools. Yes, the spec in aggregate is complex, but it's complex because software distribution and dependency analysis are complex tasks. If people want to simplify it, they can try to rebut the rationale for the individual changes that is included at the end of the PEP, but "it's complex" alone is not an argument. > So I kind of agree with him the PEP shouldn't be accepted until there's > a decent patch pending for the stdlib. That's not a judgement on the > PEP's quality. Just an advice to remain cautious, especially if the > previous metadata version already enjoys a very poor adoption rate. There were at least a couple of very good reasons for not adopting metadata 1.2: - it didn't eliminate the need for ad hoc metadata extension files like entry_points.txt, so it didn't solve one of the problems it set out to solve - it's proposed version ordering scheme was incompatible with the prevailing usage on PyPI in a couple of key ways (hence the systematic statistical comparisons in the new PEP - the numbers that are already there shows that the current version of the standard is highly compatible even with *hidden* revisions included in the statistics. I'm currently redoing the analysis to also show public versions only, which should make the proportions look even better, as many of the remaining glitches only affected old pre-releases). The adoption of metadata 1.2 also got caught up in the "distutils must die" rhetoric that meant the distutils2 project got bogged down in trying to eliminate setup.py completely, when the only thing that *really* needs to die is "./setup.py install" (and even that can be acceptable on a development system, we just want to eliminate the need to ever run it on a deployment target). That's the key change in philosophy between this PEP and the previous one: I'm working backwards from what *installers* need, and creating a specification that allows build tools, including distutils, to give it to them. I *don't really care* how those build tools get the info from their users - setup.py, setup.cfg, bento.info, it doesn't matter to me. > By the way, pip may be nice, but it's still a third-party tool, and > perhaps not even as widespread as distribute / setuptools. The main easy_install feature that pip couldn't previously match was installing from pre-built binaries (since easy_install had eggs), which is why the wheel format is so important. My current plan for Python 3.4 is for pip to be formally endorsed as the recommended tool that supports installing from an sdist, while the stdlib will ship a "pydist" tool that can only install from prebuilt wheel files. This will keep the stdlib tool simple (i.e. no need to try to find installers on arbitrary target systems), while easily allowing users to bootstrap a full fledged installer through "pydist install pip". (This will all play it out in PEPs before 3.4 rolls around - and there will likely be recruiting and documentation efforts well in advance of that) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From a.cavallo at cavallinux.eu Tue Feb 19 12:57:01 2013 From: a.cavallo at cavallinux.eu (a.cavallo at cavallinux.eu) Date: Tue, 19 Feb 2013 12:57:01 +0100 Subject: [Python-Dev] Fwd: PEP 426 is now the draft spec fordistribution metadata 2.0 Message-ID: <5620.1361275021@cavallinux.eu> > I've never seen environment markers being used or supported > in the wild. > > I'm not against modernizing the format, but given that version 1.2 > has been out for around 8 years now, without much following, > I think we need to make the implementation bit a requirement > before accepting the PEP. Elephant in the room? Somethin's been around for so long and hasn't got any sort of mainstream support is *not* a good candidate for a requirement but maybe it's me being naive. From mal at egenix.com Tue Feb 19 14:23:55 2013 From: mal at egenix.com (M.-A. Lemburg) Date: Tue, 19 Feb 2013 14:23:55 +0100 Subject: [Python-Dev] Fwd: PEP 426 is now the draft spec for distribution metadata 2.0 In-Reply-To: References: <512347ED.2040905@egenix.com> Message-ID: <51237CEB.3020701@egenix.com> On 19.02.2013 11:28, Nick Coghlan wrote: > On Tue, Feb 19, 2013 at 7:37 PM, M.-A. Lemburg wrote: >> On 17.02.2013 11:11, Nick Coghlan wrote: >> I'm not against modernizing the format, but given that version 1.2 >> has been out for around 8 years now, without much following, >> I think we need to make the implementation bit a requirement >> before accepting the PEP. > > It is being implemented in distlib, and the (short!) appendix to the > PEP itself shows how to read the format using the standard library's > email module. Hmm, what is distlib and where does it live ? The PEP only shows how to parse the RFC822-style format used by the metadata. That's not what I was referring to. If a tools wants to support metadata 2.0, it has to support all the complicated stuff as well, i.e. handle the requires fields, the environment markers and version comparisons/sorting. > v2.0 is designed to fix many of the issues that prevented the adoption > of v1.2, including tweaks to the standardised version scheme and the > addition of a formal extension mechanism to avoid the ad hoc > extensions that occurred with earlier metadata versions. Some more notes: * I find it confusing that we now have two version schemes, one defined in PEP 426 (hidden in the middle of the document) and one in PEP 386. It would be better to amend or replace PEP 386, since that's where you look for Python version strings. * PEP 426 doesn't include any mention of the egg distribution format, even though it's the most popular distribution format at the moment. It should at least include the location of the metadata file in eggs (EGG-INFO/PKG-INFO) and egg installations (/EGG-INFO/PKG-INFO). Not sure whether related or not, I also think it would be a good idea to make the metadata file available on PyPI for download (could be sent there when registering the package release). The register command only posts the data as 1.0 metadata, but includes fields from metadata 1.1. PyPI itself only displays part of the data. It would be useful to have the metadata readily available for inspection on PyPI without having to download one of the distribution files first. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Feb 19 2013) >>> Python Projects, Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From dholth at gmail.com Tue Feb 19 14:29:54 2013 From: dholth at gmail.com (Daniel Holth) Date: Tue, 19 Feb 2013 08:29:54 -0500 Subject: [Python-Dev] Fwd: PEP 426 is now the draft spec fordistribution metadata 2.0 In-Reply-To: <5620.1361275021@cavallinux.eu> References: <5620.1361275021@cavallinux.eu> Message-ID: On Feb 19, 2013 6:57 AM, wrote: > > I've never seen environment markers being used or supported > > in the wild. > > > > I'm not against modernizing the format, but given that version 1.2 > > has been out for around 8 years now, without much following, > > I think we need to make the implementation bit a requirement > > before accepting the PEP. > > Elephant in the room? Somethin's been around for so long and hasn't got > any sort of > mainstream support is *not* a good candidate for a requirement but maybe > it's me > being naive. > I think 1.2 was started and then amended over a long period of time to become what it is today. I wanted to edit it again just to add Provides-Extra. It turned out to be more practical to make a new PEP. Nick explains that for complicated reasons the implementation of Metadata 1.2 was not successful. For Metadata 2.0 we had a different strategy: put it into distribute first. You will find that this aids adoption. distlib will provide another implementation that is in many ways interchangeable. We also have three implementations of an installer for the related wheel format, three implementations of the wheel builder, and two surprisingly short implementations of environment markers. Yes, the wheel project has the same name as the wheel format. Metadata 2.0's Provides-Extra field is there to represent the setuptools "extras" feature which is necessary for a very large number of distributions on pypi. For a while the most common environment markers will look like just "extra == 'foo'". So it should not be surprising that distribute is the first tool to recognize Provides-Extra. Previously distribute did not use METADATA / PKG-INFO for dependencies, instead reading from requires.txt If you can get around the fact that we've incorporated most of another PEP 386 into v2.0, rewritten and explained everything to death, I would argue that 2.0 is actually simpler than 1.2. Only four headers are actually required in a valid Metadata 2.0 file and a number of the fields are informational and do not have to be understood by the installer. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Tue Feb 19 14:40:28 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 19 Feb 2013 23:40:28 +1000 Subject: [Python-Dev] Fwd: PEP 426 is now the draft spec for distribution metadata 2.0 In-Reply-To: <51237CEB.3020701@egenix.com> References: <512347ED.2040905@egenix.com> <51237CEB.3020701@egenix.com> Message-ID: On Tue, Feb 19, 2013 at 11:23 PM, M.-A. Lemburg wrote: > On 19.02.2013 11:28, Nick Coghlan wrote: >> On Tue, Feb 19, 2013 at 7:37 PM, M.-A. Lemburg wrote: >>> On 17.02.2013 11:11, Nick Coghlan wrote: >>> I'm not against modernizing the format, but given that version 1.2 >>> has been out for around 8 years now, without much following, >>> I think we need to make the implementation bit a requirement >>> before accepting the PEP. >> >> It is being implemented in distlib, and the (short!) appendix to the >> PEP itself shows how to read the format using the standard library's >> email module. > > Hmm, what is distlib and where does it live ? As part of the post-mortem of packaging's removal from Python 3.3, several subcomponents were identified as stable and useful. distlib is those subcomponents extracted into a separate repository by Vinay Sajip. It will be proposed as the standard library infrastructure for building packaging related tools, while distutils will become purely a build system and have nothing to do with installing software directly (except perhaps on developer machines). > The PEP only shows how to parse the RFC822-style format used by the > metadata. That's not what I was referring to. > > If a tools wants to support metadata 2.0, it has to support all > the complicated stuff as well, i.e. handle the requires fields, > the environment markers and version comparisons/sorting. Which is what distutils2 can be used for now, and what distlib will provide without the unwanted build system infrastructure in distutils2. >> v2.0 is designed to fix many of the issues that prevented the adoption >> of v1.2, including tweaks to the standardised version scheme and the >> addition of a formal extension mechanism to avoid the ad hoc >> extensions that occurred with earlier metadata versions. > > Some more notes: > > * I find it confusing that we now have two version schemes, > one defined in PEP 426 (hidden in the middle of the document) > and one in PEP 386. It would be better to amend or replace > PEP 386, since that's where you look for Python version strings. You can't understand version specifiers without understanding the sort order defined for the version scheme, so documenting them separately is just a recipe for confusion. I plan to mark PEP 386 as Withdrawn when I accept this PEP, as the sorting scheme it defines is broken, and the distutils changes proposed in that PEP are never going to happen. > * PEP 426 doesn't include any mention of the egg distribution format, > even though it's the most popular distribution format at the moment. > It should at least include the location of the metadata file > in eggs (EGG-INFO/PKG-INFO) and egg installations > (/EGG-INFO/PKG-INFO). "Other tools involved in Python distribution may also use this format." The egg format has never been, and never will be, officially endorsed by python-dev. The wheel format is the standard format for binary distribution, and PEP 376 defines the standard location for metadata on installed distributions. > > Not sure whether related or not, I also think it would be a good idea > to make the metadata file available on PyPI for download (could be sent > there when registering the package release). The register command > only posts the data as 1.0 metadata, but includes fields from > metadata 1.1. PyPI itself only displays part of the data. It's not related, but I plan to propose the adoption of TUF (with GPG based signatures) for PyPI's end-to-end security solution, and the conversion of the metadata files to JSON for distribution through TUF's metadata support. (Donald Stufft already wrote PEP 426 <-> JSON bidirectional converter as part of an unrelated experiment) > It would be useful to have the metadata readily available for > inspection on PyPI without having to download one of the > distribution files first. Indeed, which is a large part of why TUF (aka The Update Framework: https://www.updateframework.com/) is such an interesting security solution. Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Tue Feb 19 14:59:02 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 19 Feb 2013 23:59:02 +1000 Subject: [Python-Dev] Fwd: PEP 426 is now the draft spec fordistribution metadata 2.0 In-Reply-To: <5620.1361275021@cavallinux.eu> References: <5620.1361275021@cavallinux.eu> Message-ID: On Tue, Feb 19, 2013 at 9:57 PM, wrote: >> I've never seen environment markers being used or supported >> in the wild. >> >> I'm not against modernizing the format, but given that version 1.2 >> has been out for around 8 years now, without much following, >> I think we need to make the implementation bit a requirement >> before accepting the PEP. PEP 345 has NOT been an approved specification for 8 years. It was first drafted in 2005, but not Accepted until 2010: http://hg.python.org/peps/rev/7d78653a0e79 (as Daniel noted, what became PEP 426 actually started as *another* round of edits to PEP 345) PEP 345 was a first attempt at defining a formal standard that adequately described an *existing* implementation defined software distribution ecosystem built around the behaviour of setuptools. It failed, because it did *not* achieve that goal, and because there were associated declarations along the lines of "to migrate to this standard, you must completely change the way you build and distribute your Python code" (by conflating the setup.py -> setup.cfg migration of distutils2 with the use of the new metadata standard). The response was an overwhelming meh, as people willing migrated to distribute and pip (based on the de facto setuptools standards), while the distutils2 project never made it to production readiness because it was trying to solve too many problems at once. By contrast, the PEP 376 metadata format created at the same time has been adopted quite well. PEPs 345 and 386 were also accepted at a time when python-dev had neither interest nor extensive experience in packaging systems (or if anyone did have such experience, they weren't volunteering that information). So what's changed since 2010 that makes me think the time is right for a new metadata version? 1. We've recognised that without setuptools and/or distribute on board to generate it, and pip to consume it, any new metadata version is dead in the water. That groundwork has been done (mostly by Daniel as part of his work on the wheel format) 2. We've systematically gone through and resolved the significant discrepancies between the formal versioning scheme and the implementation defined behaviour of setuptools and distribute. This means that standards compliant versions will sort the same way in those tools, even if they are not updated. 3. We've addressed other objections PJ Eby (the author of setuptools) had to the previous version of the format (notably the handling of project obsolescence). 4. Various other missing features from setuptools are now supported directly (such as extras) 5. The defined interaction of version specifiers and pre- and post- releases was not useful. The new version fixes that to be something developers are more likely to want (i.e. they won't get pre-releases unless they explicitly reference one). 6. We've enhanced the format so it can handle extensions in a systematic fashion with clearly defined lines of authority based on PyPI distribution names, rather than adding arbitrarily named files to a distribution's info directory without any clear indication of where to find that specification for the file's meaning. 7. I've updated the metadata specification itself, including the addition of the Private-Version field, to more clearly explain to developers that the public Version in their metadata is primarily a tool for communicating the relative ordering of versions to installation tools, but they're still free to use their own preferred version labels internally, and certainly are not obliged to actually use the full complexity of the standard versioning scheme that the installers understand. It's OK if people don't want to read the detailed rationale provided for each of the major changes as part of the PEP, or if they want to dispute a particular piece of that rationale. But merely going "it's too complicated!" or "metadata 1.2 failed, so 2.0 will fail as well!" is not a reasonable response. Software distribution is complicated - trying to oversimplify it is one of the reasons setuptools became necessary. Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From christian at python.org Tue Feb 19 16:32:23 2013 From: christian at python.org (Christian Heimes) Date: Tue, 19 Feb 2013 16:32:23 +0100 Subject: [Python-Dev] XML DoS vulnerabilities and exploits in Python Message-ID: <51239B07.8080300@python.org> Hello, in August 2012 I found a DoS vulnerability in expat and XML libraries in Python's standard library. Since then I have found several more issues. I have been working on fixes ever since. The README of https://pypi.python.org/pypi/defusedxml contains detailed explanations of my research and all issues Blog post: http://blog.python.org/2013/02/announcing-defusedxml-fixes-for-xml.html Hotfixes: https://pypi.python.org/pypi/defusedxml https://pypi.python.org/pypi/defusedexpat Repositories: https://bitbucket.org/tiran/defusedxml https://bitbucket.org/tiran/defusedexpat https://bitbucket.org/tiran/expat CVE (work in progress) CVE-2013-1664 Unrestricted entity expansion induces DoS vulnerabilities in Python XML libraries (XML bomb) CVE-2013-1665 External entity expansion in Python XML libraries inflicts potential security flaws and DoS vulnerabilities Regards, Christian Extract from the documentation: Synopsis -------- The results of an attack on a vulnerable XML library can be fairly dramatic. With just a few hundred Bytes of XML data an attacker can occupy several Gigabytes of memory within seconds. An attacker can also keep CPUs busy for a long time with a small to medium size request. Under some circumstances it is even possible to access local files on your server, to circumvent a firewall, or to abuse services to rebound attacks to third parties. The attacks use and abuse less common features of XML and its parsers. The majority of developers are unacquainted with features such as processing instructions and entity expansions that XML inherited from SGML. At best they know about from experience with HTML but they are not aware that a document type definition (DTD) can generate an HTTP request or load a file from the file system. None of the issues is new. They have been known for a long time. Billion laughs was first reported in 2003. Nevertheless some XML libraries and applications are still vulnerable and even heavy users of XML are surprised by these features. It's hard to say whom to blame for the situation. It's too short sighted to shift all blame on XML parsers and XML libraries for using insecure default settings. After all they properly implement XML specifications. Application developers must not rely that a library is always configured for security and potential harmful data by default. Attack vectors ============== billion laughs / exponential entity expansion --------------------------------------------- The Billion Laughs attack -- also known as exponential entity expansion -- uses multiple levels of nested entities. The original example uses 9 levels of 10 expansions in each level to expand the string lol to a string of 3 * 10 9 bytes, hence the name "billion laughs". The resulting string occupies 3 GB (2.79 GiB) of memory; intermediate strings require additional memory. Because most parsers don't cache the intermediate step for every expansion it is repeated over and over again. It increases the CPU load even more. An XML document of just a few hundred bytes can disrupt all services on a machine within seconds. Example XML: ]> &d; quadratic blowup entity expansion --------------------------------- A quadratic blowup attack is similar to a Billion Laughs attack; it abuses entity expansion, too. Instead of nested entities it repeats one large entity with a couple of ten thousand chars over and over again. The attack isn't as efficient as the exponential case but it avoids triggering countermeasures of parsers against heavily nested entities. Some parsers limit the depth and breadth of a single entity but not the total amount of expanded text throughout an entire XML document. A medium-sized XML document with a couple of hundred kilobytes can require a couple of hundred MB to several GB of memory. When the attack is combined with some level of nested expansion an attacker is able to achieve a higher ratio of success. ]> &a;&a;&a;... repeat external entity expansion (remote) ---------------------------------- Entity declarations can contain more than just text for replacement. They can also point to external resources by public identifiers or system identifiers. System identifiers are standard URIs. When the URI is a URL (e.g. a http:// locator) some parsers download the resource from the remote location and embed them into the XML document verbatim. Simple example of a parsed external entity: ]> ⅇ The case of parsed external entities works only for valid XML content. The XML standard also supports unparsed external entities with a NData declaration. External entity expansion opens the door to plenty of exploits. An attacker can abuse a vulnerable XML library and application to rebound and forward network requests with the IP address of the server. It highly depends on the parser and the application what kind of exploit is possible. For example: * An attacker can circumvent firewalls and gain access to restricted resources as all the requests are made from an internal and trustworthy IP address, not from the outside. * An attacker can abuse a service to attack, spy on or DoS your servers but also third party services. The attack is disguised with the IP address of the server and the attacker is able to utilize the high bandwidth of a big machine. * An attacker can exhaust additional resources on the machine, e.g. with requests to a service that doesn't respond or responds with very large files. * An attacker may gain knowledge, when, how often and from which IP address a XML document is accessed. An attacker could send mail from inside your network if the URL handler supports smtp:// URIs. external entity expansion (local file) -------------------------------------- External entities with references to local files are a sub-case of external entity expansion. It's listed as an extra attack because it deserves extra attention. Some XML libraries such as lxml disable network access by default but still allow entity expansion with local file access by default. Local files are either referenced with a file:// URL or by a file path (either relative or absolute). An attacker may be able to access and download all files that can be read by the application process. This may include critical configuration files, too. ]> ⅇ DTD retrieval ------------- This case is similar to external entity expansion, too. Some XML libraries like Python's xml.dom.pulldown retrieve document type definitions from remote or local locations. Several attack scenarios from the external entity case apply to this issue as well. text From p.f.moore at gmail.com Tue Feb 19 17:26:34 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 19 Feb 2013 16:26:34 +0000 Subject: [Python-Dev] Fwd: PEP 426 is now the draft spec fordistribution metadata 2.0 In-Reply-To: References: <5620.1361275021@cavallinux.eu> Message-ID: On 19 February 2013 13:59, Nick Coghlan wrote: > It's OK if people don't want to read the detailed rationale provided > for each of the major changes as part of the PEP, or if they want to > dispute a particular piece of that rationale. But merely going "it's > too complicated!" or "metadata 1.2 failed, so 2.0 will fail as well!" > is not a reasonable response. Software distribution is complicated - > trying to oversimplify it is one of the reasons setuptools became > necessary. Nevertheless, the landscape is confusing. PEPs 241, 314, 345, 426, 390, 376 and 386 are all relevant to one extent or another, and only PEPs 426 (Metadata 2.0) and 390 (Static Metadata for Distutils - setup.cfg) are still in Draft format. The others are all Accepted or Final. And yet PEPs 345, 390 and 386 are unused and likely to remain so. I believe that the only ones that are *actually* of use are 241, 314 and 426 (Metadata 1.0, 1.1 and 2.0, but not 1.2) and 376 (Database of Installed Python Distributions) although 376 still has a number of flaws. I'd suggest that these should be marked as Final, and the others as Rejected, so that we have a clear statement of what is actually supported. Making sure that users have the means to write code that *uses* these standards using functionality available in the stdlib is then the next step as you say. It is critical that this is done, because packaging tools are unique in that the barrier to using external dependencies is particularly high for them - for example, pip's reliance on distribute/setuptools is necessary, but has been problematic at times. Paul PS Apologies for using a load of PEP numbers without explanation. Here's a glossary: Metadata 1.0 - PEP 241 Metadata 1.1 - PEP 314 Metadata 1.2 - PEP 345 Metadata 1.3 - PEP 426 Static Metadata for Distutils - PEP 390 Database of Installed Python Distributions - PEP 376 Changing the version comparison module in Distutils - PEP 386 From demianbrecht at gmail.com Tue Feb 19 18:33:51 2013 From: demianbrecht at gmail.com (Demian Brecht) Date: Tue, 19 Feb 2013 09:33:51 -0800 Subject: [Python-Dev] Python 2.7 buffer and memoryview documentation (issue# 17145) Message-ID: http://bugs.python.org/issue17145 I'm curious as to whether or not anyone has reviewed the documentation update I made here. Context: Having both memory views and buffers in 2.7 (as well as the C-level API for each) is confusing. The initial bug report was to implement consistent behavior for objects supporting the new C-level buffer API or to update the docs. As new features are not being added to 2.7 (as pointed out by Stefan Krah), I made an update to the docs that I think made the disparity between the two (and when one should favour the usage of either) a little clearer. ?But maybe I'm off my rocker ;) Also, are contributor agreements also required for documentation? Thanks, Demian Brecht http://demianbrecht.github.com From rahulgarg at live.ca Tue Feb 19 19:37:15 2013 From: rahulgarg at live.ca (rahul garg) Date: Tue, 19 Feb 2013 13:37:15 -0500 Subject: [Python-Dev] Built with VS2012 Express for desktop Message-ID: Hi. I downloaded Python 3.3 source, opened up the solution in VS2012 Express for Desktop and built the "python" subproject using "Release" and "x64" configurations. I now have a "python.exe" in the PCBuild/amd64 subfolder that appears to be working as far as i can see. I have a few questions: a) Is there a test suite that I can run to verify that the build is working fine? b) I now intend to build some extensions (such as NumPy). Not sure if this is the right list, but would I need to modify something in distutils to get it working with VS2012? thanks, Rahul -------------- next part -------------- An HTML attachment was scrubbed... URL: From brian at python.org Tue Feb 19 19:48:02 2013 From: brian at python.org (Brian Curtin) Date: Tue, 19 Feb 2013 12:48:02 -0600 Subject: [Python-Dev] Built with VS2012 Express for desktop In-Reply-To: References: Message-ID: On Tue, Feb 19, 2013 at 12:37 PM, rahul garg wrote: > Hi. > > I downloaded Python 3.3 source, opened up the solution in VS2012 Express for > Desktop and built the "python" subproject using "Release" and "x64" > configurations. I now have a "python.exe" in the PCBuild/amd64 subfolder > that appears to be working as far as i can see. > > I have a few questions: > a) Is there a test suite that I can run to verify that the build is working > fine? Last I checked there are some issues, but most of the tests should pass. You would run "PCBuild\python.exe -m test" from the top level of your checkout to see this. It's also covered at http://docs.python.org/devguide/ > b) I now intend to build some extensions (such as NumPy). Not sure if this > is the right list, but would I need to modify something in distutils to get > it working with VS2012? Yes. You'll probably need to point distutils to the correct batch file that sets up a VS2012 build environment. From phd at phdru.name Tue Feb 19 19:50:25 2013 From: phd at phdru.name (Oleg Broytman) Date: Tue, 19 Feb 2013 22:50:25 +0400 Subject: [Python-Dev] Built with VS2012 Express for desktop In-Reply-To: References: Message-ID: <20130219185025.GB13258@iskra.aviel.ru> Hello. This mailing list is to work on developing Python (adding new features to Python itself and fixing bugs); if you're having problems learning, understanding or using Python, please find another forum. Probably python-list/comp.lang.python mailing list/news group is the best place; there are Python developers who participate in it; you may get a faster, and probably more complete, answer there. See http://www.python.org/community/ for other lists/news groups/fora. Thank you for understanding. On Tue, Feb 19, 2013 at 01:37:15PM -0500, rahul garg wrote: > a) Is there a test suite that I can run to verify that the build is working fine? http://docs.python.org/3.3/library/test.html Oleg. -- Oleg Broytman http://phdru.name/ phd at phdru.name Programmers don't die, they just GOSUB without RETURN. From dholth at gmail.com Tue Feb 19 20:06:12 2013 From: dholth at gmail.com (Daniel Holth) Date: Tue, 19 Feb 2013 14:06:12 -0500 Subject: [Python-Dev] Fwd: PEP 426 is now the draft spec fordistribution metadata 2.0 In-Reply-To: References: <5620.1361275021@cavallinux.eu> Message-ID: On Tue, Feb 19, 2013 at 11:26 AM, Paul Moore wrote: > On 19 February 2013 13:59, Nick Coghlan wrote: > > It's OK if people don't want to read the detailed rationale provided > > for each of the major changes as part of the PEP, or if they want to > > dispute a particular piece of that rationale. But merely going "it's > > too complicated!" or "metadata 1.2 failed, so 2.0 will fail as well!" > > is not a reasonable response. Software distribution is complicated - > > trying to oversimplify it is one of the reasons setuptools became > > necessary. > > Nevertheless, the landscape is confusing. PEPs 241, 314, 345, 426, > 390, 376 and 386 are all relevant to one extent or another, and only > PEPs 426 (Metadata 2.0) and 390 (Static Metadata for Distutils - > setup.cfg) are still in Draft format. The others are all Accepted or > Final. And yet PEPs 345, 390 and 386 are unused and likely to remain > so. > > I believe that the only ones that are *actually* of use are 241, 314 > and 426 (Metadata 1.0, 1.1 and 2.0, but not 1.2) and 376 (Database of > Installed Python Distributions) although 376 still has a number of > flaws. I'd suggest that these should be marked as Final, and the > others as Rejected, so that we have a clear statement of what is > actually supported. > > Making sure that users have the means to write code that *uses* these > standards using functionality available in the stdlib is then the next > step as you say. It is critical that this is done, because packaging > tools are unique in that the barrier to using external dependencies is > particularly high for them - for example, pip's reliance on > distribute/setuptools is necessary, but has been problematic at times. > > Paul > > PS Apologies for using a load of PEP numbers without explanation. > Here's a glossary: > > Metadata 1.0 - PEP 241 > Metadata 1.1 - PEP 314 > Metadata 1.2 - PEP 345 > Metadata 1.3 - PEP 426 > Static Metadata for Distutils - PEP 390 > Database of Installed Python Distributions - PEP 376 > Changing the version comparison module in Distutils - PEP 386 > The documentation is bad. The standard library will almost certainly grow code that can interpret these PEPs. Packaging tool authors can handle these specs but no sane person reads PEPs to figure out how to submit a simple library to pypi. No end user cares about metadata versions either or even notices that sdists contain PKG-INFO at their root. You get what setup() produces and that's it. Wheel installers can work without being installed into the target environment at all. This is very hard to understand if you have been using the distutils model (the installed package imports the installer and packages are always rebuilt from source) for more than a decade. This feature does something to remedy the setuptools chicken/egg problem. We have eliminated the egg ;-) Once the installer doesn't need to be installed, repeatedly, in every virtualenv, hundreds of times a day, users could just have a single up-to-date standalone copy of pip invoked as "python pip.zip install x". Additional packages needed only for builds (even setuptools) may be installed automatically by a build system (like pip) when building rather than using packages. -------------- next part -------------- An HTML attachment was scrubbed... URL: From v+python at g.nevcal.com Tue Feb 19 20:35:59 2013 From: v+python at g.nevcal.com (Glenn Linderman) Date: Tue, 19 Feb 2013 11:35:59 -0800 Subject: [Python-Dev] Fwd: PEP 426 is now the draft spec fordistribution metadata 2.0 In-Reply-To: References: <5620.1361275021@cavallinux.eu> Message-ID: <5123D41F.4000000@g.nevcal.com> On 2/19/2013 11:06 AM, Daniel Holth wrote: > This feature does something to remedy the setuptools chicken/egg > problem. We have eliminated the egg ;-) This is the most artfully crafted comment I've seen on topic on this list for some months! Thanks! -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Tue Feb 19 21:25:30 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 19 Feb 2013 20:25:30 +0000 Subject: [Python-Dev] Fwd: PEP 426 is now the draft spec for distribution metadata 2.0 In-Reply-To: References: <512347ED.2040905@egenix.com> <51237CEB.3020701@egenix.com> Message-ID: On 19 February 2013 13:40, Nick Coghlan wrote: >> If a tools wants to support metadata 2.0, it has to support all >> the complicated stuff as well, i.e. handle the requires fields, >> the environment markers and version comparisons/sorting. > > Which is what distutils2 can be used for now, and what distlib will > provide without the unwanted build system infrastructure in > distutils2. One particular item in Metadata 2.0 which needs to be addressed is requirements specifiers, as used in (for example) Requires-Dist. These take forms like "DistName (>= ver)". There are a number of potential issues here: 1. This differs from the pkg_resources format (which omits the parentheses). Having 2 subtly different formats is not a good idea in the long term. At the moment, pkg_resources format is used in pip requirements (on the command line and in requirement files) as well as in setuptools/distribute and possibly elsewhere. 2. There is currently no code that I am aware of that implements this spec. I don't believe distlib does (yet - give Vinay 5 minutes and who knows? :-)), pkg_resources as I said implements a different format, and distutils2, apart from being a big dependency to cover just this point, only matches the version (not the project name) and presumably does so using the distutils2 version ordering (which is incompatible with Metadata 2.0). 3. The format is fiddly to parse and process - nobody is likely to implement it for themselves without a library that does it (especially not when using pkg_resources is so easy). The PEP needs a rationale as to why the pkg_resources format wasn't used, and should suggest a migration path for projects which currently use the pkg_resources style (such as pip's requirements files). Paul From donald.stufft at gmail.com Tue Feb 19 21:36:21 2013 From: donald.stufft at gmail.com (Donald Stufft) Date: Tue, 19 Feb 2013 15:36:21 -0500 Subject: [Python-Dev] Fwd: PEP 426 is now the draft spec for distribution metadata 2.0 In-Reply-To: References: <512347ED.2040905@egenix.com> <51237CEB.3020701@egenix.com> Message-ID: On Tuesday, February 19, 2013 at 3:25 PM, Paul Moore wrote: > On 19 February 2013 13:40, Nick Coghlan wrote: > > > If a tools wants to support metadata 2.0, it has to support all > > > the complicated stuff as well, i.e. handle the requires fields, > > > the environment markers and version comparisons/sorting. > > > > > > > > > Which is what distutils2 can be used for now, and what distlib will > > provide without the unwanted build system infrastructure in > > distutils2. > > > > > One particular item in Metadata 2.0 which needs to be addressed is > requirements specifiers, as used in (for example) Requires-Dist. These > take forms like "DistName (>= ver)". There are a number of potential > issues here: > > 1. This differs from the pkg_resources format (which omits the > parentheses). Having 2 subtly different formats is not a good idea in > the long term. At the moment, pkg_resources format is used in pip > requirements (on the command line and in requirement files) as well as > in setuptools/distribute and possibly elsewhere. > 2. There is currently no code that I am aware of that implements this > spec. I don't believe distlib does (yet - give Vinay 5 minutes and who > knows? :-)), pkg_resources as I said implements a different format, > and distutils2, apart from being a big dependency to cover just this > point, only matches the version (not the project name) and presumably > does so using the distutils2 version ordering (which is incompatible > with Metadata 2.0). > > It should be alongside the version stuff, atleast it was in distutils2. > 3. The format is fiddly to parse and process - nobody is likely to > implement it for themselves without a library that does it (especially > not when using pkg_resources is so easy). > > The PEP needs a rationale as to why the pkg_resources format wasn't > used, and should suggest a migration path for projects which currently > use the pkg_resources style (such as pip's requirements files). > > Paul > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org (mailto:Python-Dev at python.org) > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/donald.stufft%40gmail.com > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Tue Feb 19 21:41:47 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 19 Feb 2013 20:41:47 +0000 Subject: [Python-Dev] Fwd: PEP 426 is now the draft spec for distribution metadata 2.0 In-Reply-To: References: <512347ED.2040905@egenix.com> <51237CEB.3020701@egenix.com> Message-ID: On 19 February 2013 20:36, Donald Stufft wrote: > 2. There is currently no code that I am aware of that implements this > spec. I don't believe distlib does (yet - give Vinay 5 minutes and who > knows? :-)), pkg_resources as I said implements a different format, > and distutils2, apart from being a big dependency to cover just this > point, only matches the version (not the project name) and presumably > does so using the distutils2 version ordering (which is incompatible > with Metadata 2.0). > > It should be alongside the version stuff, atleast it was in distutils2. Yes, sorry - distlib does include code for this: https://distlib.readthedocs.org/en/latest/tutorial.html#matching-versions-against-constraints I was looking for the term "requirement" which is not used, and missed it. (Either that or Vinay's been borrowing the time machine keys again!) Paul From dholth at gmail.com Tue Feb 19 21:42:29 2013 From: dholth at gmail.com (Daniel Holth) Date: Tue, 19 Feb 2013 15:42:29 -0500 Subject: [Python-Dev] Fwd: PEP 426 is now the draft spec for distribution metadata 2.0 In-Reply-To: References: <512347ED.2040905@egenix.com> <51237CEB.3020701@egenix.com> Message-ID: On Tue, Feb 19, 2013 at 3:25 PM, Paul Moore wrote: > On 19 February 2013 13:40, Nick Coghlan wrote: > >> If a tools wants to support metadata 2.0, it has to support all > >> the complicated stuff as well, i.e. handle the requires fields, > >> the environment markers and version comparisons/sorting. > > > > Which is what distutils2 can be used for now, and what distlib will > > provide without the unwanted build system infrastructure in > > distutils2. > > One particular item in Metadata 2.0 which needs to be addressed is > requirements specifiers, as used in (for example) Requires-Dist. These > take forms like "DistName (>= ver)". There are a number of potential > issues here: > > 1. This differs from the pkg_resources format (which omits the > parentheses). Having 2 subtly different formats is not a good idea in > the long term. At the moment, pkg_resources format is used in pip > requirements (on the command line and in requirement files) as well as > in setuptools/distribute and possibly elsewhere. > 2. There is currently no code that I am aware of that implements this > spec. I don't believe distlib does (yet - give Vinay 5 minutes and who > knows? :-)), pkg_resources as I said implements a different format, > and distutils2, apart from being a big dependency to cover just this > point, only matches the version (not the project name) and presumably > does so using the distutils2 version ordering (which is incompatible > with Metadata 2.0). > 3. The format is fiddly to parse and process - nobody is likely to > implement it for themselves without a library that does it (especially > not when using pkg_resources is so easy). > > The PEP needs a rationale as to why the pkg_resources format wasn't > used, and should suggest a migration path for projects which currently > use the pkg_resources style (such as pip's requirements files). > > Paul We did add full support for Metadata 2.0 Requires-Dist in pkg_resources. The fiddling happens here: https://bitbucket.org/tarek/distribute/src/tip/pkg_resources.py?at=default#cl-2469 The code removes the parenthesis and adds == when no operator was specified to convert us back to the pkg_resources format. The rest of pkg_resources carries on as before. Wheel installs would not pull in dependencies otherwise. -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.jerdonek at gmail.com Tue Feb 19 22:22:22 2013 From: chris.jerdonek at gmail.com (Chris Jerdonek) Date: Tue, 19 Feb 2013 13:22:22 -0800 Subject: [Python-Dev] Fwd: PEP 426 is now the draft spec for distribution metadata 2.0 In-Reply-To: References: <512347ED.2040905@egenix.com> Message-ID: On Tue, Feb 19, 2013 at 2:28 AM, Nick Coghlan wrote: > On Tue, Feb 19, 2013 at 7:37 PM, M.-A. Lemburg wrote: >> On 17.02.2013 11:11, Nick Coghlan wrote: >> I'm not against modernizing the format, but given that version 1.2 >> has been out for around 8 years now, without much following, >> I think we need to make the implementation bit a requirement >> before accepting the PEP. > > It is being implemented in distlib, and the (short!) appendix to the > PEP itself shows how to read the format using the standard library's > email module. Maybe this is already stated somewhere, but is there a plan for when distlib will be brought into the repository? Is there a reason not to do it now? It seems it would have more visibility that way (e.g. people could see it as part of the development version of the online docs and it would be in check-ins as are PEP edits), and its status relative to Python would be clearer. --Chris > > v2.0 is designed to fix many of the issues that prevented the adoption > of v1.2, including tweaks to the standardised version scheme and the > addition of a formal extension mechanism to avoid the ad hoc > extensions that occurred with earlier metadata versions. > > Cheers, > Nick. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/chris.jerdonek%40gmail.com From mal at egenix.com Tue Feb 19 22:34:34 2013 From: mal at egenix.com (M.-A. Lemburg) Date: Tue, 19 Feb 2013 22:34:34 +0100 Subject: [Python-Dev] Fwd: PEP 426 is now the draft spec for distribution metadata 2.0 In-Reply-To: References: <512347ED.2040905@egenix.com> <51237CEB.3020701@egenix.com> Message-ID: <5123EFEA.70600@egenix.com> On 19.02.2013 14:40, Nick Coghlan wrote: > On Tue, Feb 19, 2013 at 11:23 PM, M.-A. Lemburg wrote: >> * PEP 426 doesn't include any mention of the egg distribution format, >> even though it's the most popular distribution format at the moment. >> It should at least include the location of the metadata file >> in eggs (EGG-INFO/PKG-INFO) and egg installations >> (/EGG-INFO/PKG-INFO). > > "Other tools involved in Python distribution may also use this format." > > The egg format has never been, and never will be, officially endorsed > by python-dev. The wheel format is the standard format for binary > distribution, and PEP 376 defines the standard location for metadata > on installed distributions. Oh, come on, Nick, that's just silly. setuptools was included in stdlib for a short while, so the above is simply wrong. Eggs are the most widely used binary distribution format for Python package on PyPI: # wc *files.csv 25585 25598 1431013 2013-02-19-egg-files.csv 4619 4640 236694 2013-02-19-exe-files.csv 254 255 13402 2013-02-19-msi-files.csv 104691 104853 5251962 2013-02-19-tar-gz-files.csv 24 24 1221 2013-02-19-whl-files.csv 17937 18022 905913 2013-02-19-zip-files.csv 153110 153392 7840205 total (based on todays PyPI stats) It doesn't really help ignoring realities... and I'm saying that as one of the core devs who got setuptools kicked out of the stdlib again. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Feb 19 2013) >>> Python Projects, Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From dholth at gmail.com Tue Feb 19 23:01:46 2013 From: dholth at gmail.com (Daniel Holth) Date: Tue, 19 Feb 2013 17:01:46 -0500 Subject: [Python-Dev] Fwd: PEP 426 is now the draft spec for distribution metadata 2.0 In-Reply-To: <5123EFEA.70600@egenix.com> References: <512347ED.2040905@egenix.com> <51237CEB.3020701@egenix.com> <5123EFEA.70600@egenix.com> Message-ID: On Tue, Feb 19, 2013 at 4:34 PM, M.-A. Lemburg wrote: > On 19.02.2013 14:40, Nick Coghlan wrote: > > On Tue, Feb 19, 2013 at 11:23 PM, M.-A. Lemburg wrote: > >> * PEP 426 doesn't include any mention of the egg distribution format, > >> even though it's the most popular distribution format at the moment. > >> It should at least include the location of the metadata file > >> in eggs (EGG-INFO/PKG-INFO) and egg installations > >> (/EGG-INFO/PKG-INFO). > > > > "Other tools involved in Python distribution may also use this format." > > > > The egg format has never been, and never will be, officially endorsed > > by python-dev. The wheel format is the standard format for binary > > distribution, and PEP 376 defines the standard location for metadata > > on installed distributions. > > Oh, come on, Nick, that's just silly. setuptools was included in stdlib > for a short while, so the above is simply wrong. Eggs are the most > widely used binary distribution format for Python package on PyPI: > > # wc *files.csv > 25585 25598 1431013 2013-02-19-egg-files.csv > 4619 4640 236694 2013-02-19-exe-files.csv > 254 255 13402 2013-02-19-msi-files.csv > 104691 104853 5251962 2013-02-19-tar-gz-files.csv > 24 24 1221 2013-02-19-whl-files.csv > 17937 18022 905913 2013-02-19-zip-files.csv > 153110 153392 7840205 total > > (based on todays PyPI stats) > > It doesn't really help ignoring realities... and I'm saying > that as one of the core devs who got setuptools kicked out of > the stdlib again. > > -- > Marc-Andre Lemburg > eGenix.com > The wheel philosophy is that it should be supported by both python-dev and setuptools and that you should feel happy about using setuptools if you like it whether or not python-dev (currently) endorses that. If you are using setuptools (distribute's pkg_resources) then you can use both at the same time. Distribute, distutils and setuptools' problems have not been well understood which I think is why there has been a need to discredit setuptools by calling it non-standard. It is the defacto standard. If your packages have dependencies there is no other choice. Wheel tries to solve the real problem by allowing you to build a package with setuptools while giving the end-user the choice of installing setuptools or not. Of course eggs are the most popular right now. The wheel format is very egg-like while avoiding some of egg's problems. See the comparison in the PEP or read the story on wheel's rtfd. The wheel project includes tools to losslessly convert eggs or bdist_wininst to wheel. I am confident distlib can thrive outside of the standard library! Why the rush to kill it before its prime? -------------- next part -------------- An HTML attachment was scrubbed... URL: From mal at egenix.com Tue Feb 19 23:07:13 2013 From: mal at egenix.com (M.-A. Lemburg) Date: Tue, 19 Feb 2013 23:07:13 +0100 Subject: [Python-Dev] Fwd: PEP 426 is now the draft spec for distribution metadata 2.0 In-Reply-To: References: <512347ED.2040905@egenix.com> <51237CEB.3020701@egenix.com> Message-ID: <5123F791.1010201@egenix.com> On 19.02.2013 14:40, Nick Coghlan wrote: > On Tue, Feb 19, 2013 at 11:23 PM, M.-A. Lemburg wrote: >> On 19.02.2013 11:28, Nick Coghlan wrote: >>> On Tue, Feb 19, 2013 at 7:37 PM, M.-A. Lemburg wrote: >>>> On 17.02.2013 11:11, Nick Coghlan wrote: >>>> I'm not against modernizing the format, but given that version 1.2 >>>> has been out for around 8 years now, without much following, >>>> I think we need to make the implementation bit a requirement >>>> before accepting the PEP. >>> >>> It is being implemented in distlib, and the (short!) appendix to the >>> PEP itself shows how to read the format using the standard library's >>> email module. >> >> Hmm, what is distlib and where does it live ? > > As part of the post-mortem of packaging's removal from Python 3.3, > several subcomponents were identified as stable and useful. distlib is > those subcomponents extracted into a separate repository by Vinay > Sajip. > > It will be proposed as the standard library infrastructure for > building packaging related tools, while distutils will become purely a > build system and have nothing to do with installing software directly > (except perhaps on developer machines). Shouldn't those details be mentioned in the PEP ? >> The PEP only shows how to parse the RFC822-style format used by the >> metadata. That's not what I was referring to. >> >> If a tools wants to support metadata 2.0, it has to support all >> the complicated stuff as well, i.e. handle the requires fields, >> the environment markers and version comparisons/sorting. > > Which is what distutils2 can be used for now, and what distlib will > provide without the unwanted build system infrastructure in > distutils2. > >>> v2.0 is designed to fix many of the issues that prevented the adoption >>> of v1.2, including tweaks to the standardised version scheme and the >>> addition of a formal extension mechanism to avoid the ad hoc >>> extensions that occurred with earlier metadata versions. >> >> Some more notes: >> >> * I find it confusing that we now have two version schemes, >> one defined in PEP 426 (hidden in the middle of the document) >> and one in PEP 386. It would be better to amend or replace >> PEP 386, since that's where you look for Python version strings. > > You can't understand version specifiers without understanding the sort > order defined for the version scheme, so documenting them separately > is just a recipe for confusion. PEP 386 defines both. The point here is that the version scheme goes far beyond the metadata format and is complicated enough to warrant a separate PEP. > I plan to mark PEP 386 as Withdrawn when I accept this PEP, as the > sorting scheme it defines is broken, and the distutils changes > proposed in that PEP are never going to happen. Hmm, Tarek is the author, so only he can withdraw the PEP, AFAIK. >> * PEP 426 doesn't include any mention of the egg distribution format, >> even though it's the most popular distribution format at the moment. >> It should at least include the location of the metadata file >> in eggs (EGG-INFO/PKG-INFO) and egg installations >> (/EGG-INFO/PKG-INFO). > > "Other tools involved in Python distribution may also use this format." > > The egg format has never been, and never will be, officially endorsed > by python-dev. The wheel format is the standard format for binary > distribution, and PEP 376 defines the standard location for metadata > on installed distributions. See my other reply. >> >> Not sure whether related or not, I also think it would be a good idea >> to make the metadata file available on PyPI for download (could be sent >> there when registering the package release). The register command >> only posts the data as 1.0 metadata, but includes fields from >> metadata 1.1. PyPI itself only displays part of the data. > > It's not related, but I plan to propose the adoption of TUF (with GPG > based signatures) for PyPI's end-to-end security solution, and the > conversion of the metadata files to JSON for distribution through > TUF's metadata support. (Donald Stufft already wrote PEP 426 <-> JSON > bidirectional converter as part of an unrelated experiment) Why convert the metadata format you are defining in PEP 426 to yet another format when it can be uploaded as file straight to PyPI ? TUF doesn't have anything to do with that, agreed ;-) >> It would be useful to have the metadata readily available for >> inspection on PyPI without having to download one of the >> distribution files first. > > Indeed, which is a large part of why TUF (aka The Update Framework: > https://www.updateframework.com/) is such an interesting security > solution. The suggestion to have the metadata available on PyPI doesn't have anything to do with security. It's about being able to determine compatibility and select the right distribution file for download. The metadata also helps in creating dependency graphs, which are useful for a lot of things. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Feb 19 2013) >>> Python Projects, Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From mal at egenix.com Tue Feb 19 23:10:17 2013 From: mal at egenix.com (M.-A. Lemburg) Date: Tue, 19 Feb 2013 23:10:17 +0100 Subject: [Python-Dev] Fwd: PEP 426 is now the draft spec for distribution metadata 2.0 In-Reply-To: References: <512347ED.2040905@egenix.com> <51237CEB.3020701@egenix.com> <5123EFEA.70600@egenix.com> Message-ID: <5123F849.6080703@egenix.com> On 19.02.2013 23:01, Daniel Holth wrote: > On Tue, Feb 19, 2013 at 4:34 PM, M.-A. Lemburg wrote: > >> On 19.02.2013 14:40, Nick Coghlan wrote: >>> On Tue, Feb 19, 2013 at 11:23 PM, M.-A. Lemburg wrote: >>>> * PEP 426 doesn't include any mention of the egg distribution format, >>>> even though it's the most popular distribution format at the moment. >>>> It should at least include the location of the metadata file >>>> in eggs (EGG-INFO/PKG-INFO) and egg installations >>>> (/EGG-INFO/PKG-INFO). >>> >>> "Other tools involved in Python distribution may also use this format." >>> >>> The egg format has never been, and never will be, officially endorsed >>> by python-dev. The wheel format is the standard format for binary >>> distribution, and PEP 376 defines the standard location for metadata >>> on installed distributions. >> >> Oh, come on, Nick, that's just silly. setuptools was included in stdlib >> for a short while, so the above is simply wrong. Eggs are the most >> widely used binary distribution format for Python package on PyPI: >> >> # wc *files.csv >> 25585 25598 1431013 2013-02-19-egg-files.csv >> 4619 4640 236694 2013-02-19-exe-files.csv >> 254 255 13402 2013-02-19-msi-files.csv >> 104691 104853 5251962 2013-02-19-tar-gz-files.csv >> 24 24 1221 2013-02-19-whl-files.csv >> 17937 18022 905913 2013-02-19-zip-files.csv >> 153110 153392 7840205 total >> >> (based on todays PyPI stats) >> >> It doesn't really help ignoring realities... and I'm saying >> that as one of the core devs who got setuptools kicked out of >> the stdlib again. >> >> -- >> Marc-Andre Lemburg >> eGenix.com >> > > The wheel philosophy is that it should be supported by both python-dev and > setuptools and that you should feel happy about using setuptools if you > like it whether or not python-dev (currently) endorses that. If you are > using setuptools (distribute's pkg_resources) then you can use both at the > same time. > > Distribute, distutils and setuptools' problems have not been well > understood which I think is why there has been a need to discredit > setuptools by calling it non-standard. It is the defacto standard. If your > packages have dependencies there is no other choice. Wheel tries to solve > the real problem by allowing you to build a package with setuptools while > giving the end-user the choice of installing setuptools or not. > > Of course eggs are the most popular right now. The wheel format is very > egg-like while avoiding some of egg's problems. See the comparison in the > PEP or read the story on wheel's rtfd. The wheel project includes tools to > losslessly convert eggs or bdist_wininst to wheel. That's all fine, but it doesn't explain the refusal to add the documentation of the location of the PKG-INFO file in eggs ? > I am confident distlib can thrive outside of the standard library! Why the > rush to kill it before its prime? Who's trying to kill distlib ? -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Feb 19 2013) >>> Python Projects, Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From dholth at gmail.com Wed Feb 20 00:16:14 2013 From: dholth at gmail.com (Daniel Holth) Date: Tue, 19 Feb 2013 18:16:14 -0500 Subject: [Python-Dev] Fwd: PEP 426 is now the draft spec for distribution metadata 2.0 In-Reply-To: <5123F849.6080703@egenix.com> References: <512347ED.2040905@egenix.com> <51237CEB.3020701@egenix.com> <5123EFEA.70600@egenix.com> <5123F849.6080703@egenix.com> Message-ID: On Tue, Feb 19, 2013 at 5:10 PM, M.-A. Lemburg wrote: > On 19.02.2013 23:01, Daniel Holth wrote: > > On Tue, Feb 19, 2013 at 4:34 PM, M.-A. Lemburg wrote: > > > >> On 19.02.2013 14:40, Nick Coghlan wrote: > >>> On Tue, Feb 19, 2013 at 11:23 PM, M.-A. Lemburg > wrote: > >>>> * PEP 426 doesn't include any mention of the egg distribution format, > >>>> even though it's the most popular distribution format at the moment. > >>>> It should at least include the location of the metadata file > >>>> in eggs (EGG-INFO/PKG-INFO) and egg installations > >>>> (/EGG-INFO/PKG-INFO). > >>> > >>> "Other tools involved in Python distribution may also use this format." > >>> > >>> The egg format has never been, and never will be, officially endorsed > >>> by python-dev. The wheel format is the standard format for binary > >>> distribution, and PEP 376 defines the standard location for metadata > >>> on installed distributions. > >> > >> Oh, come on, Nick, that's just silly. setuptools was included in stdlib > >> for a short while, so the above is simply wrong. Eggs are the most > >> widely used binary distribution format for Python package on PyPI: > >> > >> # wc *files.csv > >> 25585 25598 1431013 2013-02-19-egg-files.csv > >> 4619 4640 236694 2013-02-19-exe-files.csv > >> 254 255 13402 2013-02-19-msi-files.csv > >> 104691 104853 5251962 2013-02-19-tar-gz-files.csv > >> 24 24 1221 2013-02-19-whl-files.csv > >> 17937 18022 905913 2013-02-19-zip-files.csv > >> 153110 153392 7840205 total > >> > >> (based on todays PyPI stats) > >> > >> It doesn't really help ignoring realities... and I'm saying > >> that as one of the core devs who got setuptools kicked out of > >> the stdlib again. > >> > >> -- > >> Marc-Andre Lemburg > >> eGenix.com > >> > > > > The wheel philosophy is that it should be supported by both python-dev > and > > setuptools and that you should feel happy about using setuptools if you > > like it whether or not python-dev (currently) endorses that. If you are > > using setuptools (distribute's pkg_resources) then you can use both at > the > > same time. > > > > Distribute, distutils and setuptools' problems have not been well > > understood which I think is why there has been a need to discredit > > setuptools by calling it non-standard. It is the defacto standard. If > your > > packages have dependencies there is no other choice. Wheel tries to solve > > the real problem by allowing you to build a package with setuptools while > > giving the end-user the choice of installing setuptools or not. > > > > Of course eggs are the most popular right now. The wheel format is very > > egg-like while avoiding some of egg's problems. See the comparison in the > > PEP or read the story on wheel's rtfd. The wheel project includes tools > to > > losslessly convert eggs or bdist_wininst to wheel. > > That's all fine, but it doesn't explain the refusal to add the > documentation of the location of the PKG-INFO file in eggs ? It would just be a sentence, I wouldn't have a problem with it but I also don't see why it would be necessary. Even setuptools doesn't touch the file usually. Right now distribute's pkg_resources currently only understands Requires-Dist if it is inside a .dist-info directory. Sorry, Chris must have meant http://hg.python.org/distlib/ . I was struggling to imagine a world where that is more visible than something on bitbucket. Half the comments have been about putting something in stdlib right away, something that I do not understand at all as a voracious PyPI or https://crate.io/ user. You should secure both the metadata and the packages but it doesn't have anything to do with the serialization format. Wheel is designed so that the metadata is at the end of the .zip and can be retrieved with an HTTP partial request. The feature may or may not be useful. -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald.stufft at gmail.com Wed Feb 20 00:19:54 2013 From: donald.stufft at gmail.com (Donald Stufft) Date: Tue, 19 Feb 2013 18:19:54 -0500 Subject: [Python-Dev] Fwd: PEP 426 is now the draft spec for distribution metadata 2.0 In-Reply-To: References: <512347ED.2040905@egenix.com> <51237CEB.3020701@egenix.com> <5123EFEA.70600@egenix.com> <5123F849.6080703@egenix.com> Message-ID: <6ECA090456F34E3CB3E0974093CB9E4C@gmail.com> On Tuesday, February 19, 2013 at 6:16 PM, Daniel Holth wrote: > Sorry, Chris must have meant http://hg.python.org/distlib/ . I was struggling to imagine a world where that is more visible than something on bitbucket. Half the comments have been about putting something in stdlib right away, something that I do not understand at all as a voracious PyPI or https://crate.io/ user. Let's not add anything to the stdlib till it has real world usage. Doing otherwise is putting the cart before the horse. -------------- next part -------------- An HTML attachment was scrubbed... URL: From fred at fdrake.net Wed Feb 20 01:54:21 2013 From: fred at fdrake.net (Fred Drake) Date: Tue, 19 Feb 2013 19:54:21 -0500 Subject: [Python-Dev] Fwd: PEP 426 is now the draft spec for distribution metadata 2.0 In-Reply-To: <6ECA090456F34E3CB3E0974093CB9E4C@gmail.com> References: <512347ED.2040905@egenix.com> <51237CEB.3020701@egenix.com> <5123EFEA.70600@egenix.com> <5123F849.6080703@egenix.com> <6ECA090456F34E3CB3E0974093CB9E4C@gmail.com> Message-ID: On Tue, Feb 19, 2013 at 6:19 PM, Donald Stufft wrote: > Let's not add anything to the stdlib till it has real world usage. Doing > otherwise is putting the cart before the horse. I'd posit that anything successful will no longer need to be added to the standard library, to boot. Packaging hasn't done well there. I'd rather see a successful packaging story develop than bundle it into the standard library. The later just isn't that interesting any more. -Fred -- Fred L. Drake, Jr. "A storm broke loose in my mind." --Albert Einstein From p.f.moore at gmail.com Wed Feb 20 03:37:26 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 20 Feb 2013 02:37:26 +0000 Subject: [Python-Dev] Fwd: PEP 426 is now the draft spec for distribution metadata 2.0 In-Reply-To: References: <512347ED.2040905@egenix.com> <51237CEB.3020701@egenix.com> <5123EFEA.70600@egenix.com> <5123F849.6080703@egenix.com> <6ECA090456F34E3CB3E0974093CB9E4C@gmail.com> Message-ID: On 20 February 2013 00:54, Fred Drake wrote: > I'd posit that anything successful will no longer need to be added to > the standard library, to boot. Packaging hasn't done well there. distlib may be the exception, though. Packaging tools are somewhat unique because of the chicken and egg issue involved in having a packaging tool with external dependencies - who installs your dependencies for you? So enabling technology (library code to perform packaging-related tasks, particularly in support of standardised formats) could be better available from the stdlib. > I'd rather see a successful packaging story develop than bundle it into the > standard library. The later just isn't that interesting any more. Bundling too early is a bad idea though. distlib is developing fast and to do so it needs (1) a development cycle independent of python's and (2) compatibility and ease of use with earlier versions of Python (the latter is also critical for adoption in place of custom code in packaging tools). Aiming for an accelerated level of development targeting inclusion in Python 3.4 is plausible, though. MAL pointed out that agreeing standards but not offering tools to support them in the stdlib is risky, as people have no incentive to adopt those standards. We've got 6 months or more until 3.4 feature freeze, let's not make any decision too soon, though. Paul. From steve at pearwood.info Wed Feb 20 03:40:14 2013 From: steve at pearwood.info (Steven D'Aprano) Date: Wed, 20 Feb 2013 13:40:14 +1100 Subject: [Python-Dev] Fwd: PEP 426 is now the draft spec for distribution metadata 2.0 In-Reply-To: References: <512347ED.2040905@egenix.com> <51237CEB.3020701@egenix.com> <5123EFEA.70600@egenix.com> <5123F849.6080703@egenix.com> <6ECA090456F34E3CB3E0974093CB9E4C@gmail.com> Message-ID: <5124378E.6020607@pearwood.info> On 20/02/13 11:54, Fred Drake wrote: > On Tue, Feb 19, 2013 at 6:19 PM, Donald Stufft wrote: >> Let's not add anything to the stdlib till it has real world usage. Doing >> otherwise is putting the cart before the horse. > > I'd posit that anything successful will no longer need to be added to > the standard > library, to boot. Packaging hasn't done well there. > > I'd rather see a successful packaging story develop than bundle it into the > standard library. The later just isn't that interesting any more. I keep hearing people say that the stdlib is not important, but I don't think that is true. There are lots of people who have problems with anything not in the standard library. - Beginners often have difficulty (due to inexperience, lack of confidence or knowledge) in *finding*, let alone installing and using, packages that aren't in the standard library. - To people in the Linux world, adding anything outside of your distro's packaging system is a nuisance. No matter how easy your packaging library makes it, you now have two sorts of packages: first-class packages that your distro will automatically update for you, and second-class ones that aren't. - People working in restrictive corporate systems often have to jump through flaming hoops before installing software. Packages in the stdlib are a no-brainer. Anything outside the stdlib has additional barriers to use, even if installing them is as simple as "some-package-manager install spam.py". For the avoidance of doubt, this is *not* a veiled request for "everything" to be in the stdlib, since that is impractical and stupid, just a reminder that the stdlib is still important and that no matter how easy packaging becomes, it will never be as easy as having something already there. -- Steven From tseaver at palladion.com Wed Feb 20 05:07:39 2013 From: tseaver at palladion.com (Tres Seaver) Date: Tue, 19 Feb 2013 23:07:39 -0500 Subject: [Python-Dev] Fwd: PEP 426 is now the draft spec for distribution metadata 2.0 In-Reply-To: References: <512347ED.2040905@egenix.com> <51237CEB.3020701@egenix.com> <5123EFEA.70600@egenix.com> <5123F849.6080703@egenix.com> <6ECA090456F34E3CB3E0974093CB9E4C@gmail.com> Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 02/19/2013 09:37 PM, Paul Moore wrote: > On 20 February 2013 00:54, Fred Drake wrote: >> I'd posit that anything successful will no longer need to be added >> to the standard library, to boot. Packaging hasn't done well >> there. > > distlib may be the exception, though. Packaging tools are somewhat > unique because of the chicken and egg issue involved in having a > packaging tool with external dependencies - who installs your > dependencies for you? So enabling technology (library code to perform > packaging-related tasks, particularly in support of standardised > formats) could be better available from the stdlib. The big blocker there is that users can't rely on having it in the stdlib until after they drop Python < 3.4 (assuming accelearated absorption) or even 3.5. Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with undefined - http://www.enigmail.net/ iEYEARECAAYFAlEkTAsACgkQ+gerLs4ltQ6VBgCePncI4cX7a8NEN6lj98CVBdAE UTUAnA+6zo8Zjmp6T4n0uL804PnHHvZ8 =OT8w -----END PGP SIGNATURE----- From eliben at gmail.com Wed Feb 20 05:37:36 2013 From: eliben at gmail.com (Eli Bendersky) Date: Tue, 19 Feb 2013 20:37:36 -0800 Subject: [Python-Dev] Built with VS2012 Express for desktop In-Reply-To: <20130219185025.GB13258@iskra.aviel.ru> References: <20130219185025.GB13258@iskra.aviel.ru> Message-ID: On Tue, Feb 19, 2013 at 10:50 AM, Oleg Broytman wrote: > Hello. > > This mailing list is to work on developing Python (adding new > features to Python itself and fixing bugs); if you're having problems > learning, understanding or using Python, please find another forum. > Probably python-list/comp.lang.python mailing list/news group is the > best place; there are Python developers who participate in it; you may > get a faster, and probably more complete, answer there. See > http://www.python.org/community/ for other lists/news groups/fora. Thank > you for understanding. > > Oleg, lately I have the feeling you're getting too automatic with this template response. The OP didn't ask how to reverse a list in Python, he was trying to build Python itself (on Windows, no less), which suggests that he may be trying to work *on* Python. Or maybe he isn't, but let's grant him the benefit of the doubt. If python-dev was overwhelmed with such posts, then perhaps we'd have to think about solving the problem; but there is no problem. Being kind to newbies is an important part of meaningful participation in an open source project. Eli -------------- next part -------------- An HTML attachment was scrubbed... URL: From tjreedy at udel.edu Wed Feb 20 06:48:01 2013 From: tjreedy at udel.edu (Terry Reedy) Date: Wed, 20 Feb 2013 00:48:01 -0500 Subject: [Python-Dev] Python 2.7 buffer and memoryview documentation (issue# 17145) In-Reply-To: References: Message-ID: On 2/19/2013 12:33 PM, Demian Brecht wrote: Comment on patch on issue. > Also, are contributor agreements also required for documentation? I believe so, especially for substantial patches like yours. Just submit one and be done with it. You probably should choose the Academic License. http://www.python.org/psf/contrib/ -- Terry Jan Reedy From solipsis at pitrou.net Wed Feb 20 08:23:19 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 20 Feb 2013 08:23:19 +0100 Subject: [Python-Dev] Built with VS2012 Express for desktop References: <20130219185025.GB13258@iskra.aviel.ru> Message-ID: <20130220082319.02424aeb@pitrou.net> On Tue, 19 Feb 2013 20:37:36 -0800 Eli Bendersky wrote: > On Tue, Feb 19, 2013 at 10:50 AM, Oleg Broytman wrote: > > > Hello. > > > > This mailing list is to work on developing Python (adding new > > features to Python itself and fixing bugs); if you're having problems > > learning, understanding or using Python, please find another forum. > > Probably python-list/comp.lang.python mailing list/news group is the > > best place; there are Python developers who participate in it; you may > > get a faster, and probably more complete, answer there. See > > http://www.python.org/community/ for other lists/news groups/fora. Thank > > you for understanding. > > > > > Oleg, lately I have the feeling you're getting too automatic with this > template response. +1. This isn't the python-robots mailing-list. Regards Antoine. From chris.jerdonek at gmail.com Wed Feb 20 08:48:22 2013 From: chris.jerdonek at gmail.com (Chris Jerdonek) Date: Tue, 19 Feb 2013 23:48:22 -0800 Subject: [Python-Dev] Fwd: PEP 426 is now the draft spec for distribution metadata 2.0 In-Reply-To: <5124378E.6020607@pearwood.info> References: <512347ED.2040905@egenix.com> <51237CEB.3020701@egenix.com> <5123EFEA.70600@egenix.com> <5123F849.6080703@egenix.com> <6ECA090456F34E3CB3E0974093CB9E4C@gmail.com> <5124378E.6020607@pearwood.info> Message-ID: On Tue, Feb 19, 2013 at 3:16 PM, Daniel Holth wrote: > Sorry, Chris must have meant http://hg.python.org/distlib/ . I was > struggling to imagine a world where that is more visible than something on > bitbucket. I meant that bringing distlib into http://hg.python.org/cpython/ would give it more visibility to core devs and others that already keep an eye on python-checkins (the mailing list). And I think seeing the Sphinx-processed docs integrated and cross-referenced with http://docs.python.org/dev/ will help people understand better what has been done and how it fits in with the rest of CPython -- which I think would be useful to the community. It may also encourage involvement (e.g. by being part of the main tracker). In asking about the "plan" for doing this, I was thinking of the following remark by Nick: On Tue, Feb 19, 2013 at 5:40 AM, Nick Coghlan wrote: > On Tue, Feb 19, 2013 at 11:23 PM, M.-A. Lemburg wrote: >> >> Hmm, what is distlib and where does it live ? > > As part of the post-mortem of packaging's removal from Python 3.3, > several subcomponents were identified as stable and useful. distlib is > those subcomponents extracted into a separate repository by Vinay > Sajip. > > It will be proposed as the standard library infrastructure for > building packaging related tools, while distutils will become purely a > build system and have nothing to do with installing software directly > (except perhaps on developer machines). My question was basically whether there was a tentative plan for when it (or completed parts of it) will be proposed (e.g. when a certain amount of functionality is completed, etc). It's better not to do this at the last minute if 3.4 is the plan (as I think was attempted with packaging but for 3.3). On Tue, Feb 19, 2013 at 6:40 PM, Steven D'Aprano wrote: > > I keep hearing people say that the stdlib is not important, but I don't > think > that is true. There are lots of people who have problems with anything not > in > the standard library. > > - Beginners often have difficulty (due to inexperience, lack of confidence > or > knowledge) in *finding*, let alone installing and using, packages that > aren't > in the standard library. > > - To people in the Linux world, adding anything outside of your distro's > packaging system is a nuisance. No matter how easy your packaging library > makes it, you now have two sorts of packages: first-class packages that > your distro will automatically update for you, and second-class ones that > aren't. > > - People working in restrictive corporate systems often have to jump through > flaming hoops before installing software. I would also add that for people new to writing Python modules and that want to distribute them, it's hard to evaluate what they are "supposed" to use (distutils, setuptools, distribute, bento, etc). Just a day or two ago, this exact question was asked on the Distutils mailing list with subject "Confusion of a hobby programmer." Code not being in the standard library creates an extra mental hurdle to overcome. --Chris From donald.stufft at gmail.com Wed Feb 20 08:53:54 2013 From: donald.stufft at gmail.com (Donald Stufft) Date: Wed, 20 Feb 2013 02:53:54 -0500 Subject: [Python-Dev] Fwd: PEP 426 is now the draft spec for distribution metadata 2.0 In-Reply-To: References: <512347ED.2040905@egenix.com> <51237CEB.3020701@egenix.com> <5123EFEA.70600@egenix.com> <5123F849.6080703@egenix.com> <6ECA090456F34E3CB3E0974093CB9E4C@gmail.com> <5124378E.6020607@pearwood.info> Message-ID: On Wednesday, February 20, 2013 at 2:48 AM, Chris Jerdonek wrote: > On Tue, Feb 19, 2013 at 3:16 PM, Daniel Holth wrote: > > Sorry, Chris must have meant http://hg.python.org/distlib/ . I was > > struggling to imagine a world where that is more visible than something on > > bitbucket. > > > > > I meant that bringing distlib into http://hg.python.org/cpython/ would > give it more visibility to core devs and others that already keep an > eye on python-checkins (the mailing list). And I think seeing the > Sphinx-processed docs integrated and cross-referenced with > http://docs.python.org/dev/ will help people understand better what > has been done and how it fits in with the rest of CPython -- which I > think would be useful to the community. It may also encourage > involvement (e.g. by being part of the main tracker). > > On the other hand it makes contributing to it more annoying since it does not have pull requests, unless it was just a mirror. > In asking about the "plan" for doing this, I was thinking of the > following remark by Nick: > > On Tue, Feb 19, 2013 at 5:40 AM, Nick Coghlan wrote: > > On Tue, Feb 19, 2013 at 11:23 PM, M.-A. Lemburg wrote: > > > > > > Hmm, what is distlib and where does it live ? > > > > As part of the post-mortem of packaging's removal from Python 3.3, > > several subcomponents were identified as stable and useful. distlib is > > those subcomponents extracted into a separate repository by Vinay > > Sajip. > > > > It will be proposed as the standard library infrastructure for > > building packaging related tools, while distutils will become purely a > > build system and have nothing to do with installing software directly > > (except perhaps on developer machines). > > > > > My question was basically whether there was a tentative plan for when > it (or completed parts of it) will be proposed (e.g. when a certain > amount of functionality is completed, etc). It's better not to do > this at the last minute if 3.4 is the plan (as I think was attempted > with packaging but for 3.3). > > On Tue, Feb 19, 2013 at 6:40 PM, Steven D'Aprano wrote: > > > > I keep hearing people say that the stdlib is not important, but I don't > > think > > that is true. There are lots of people who have problems with anything not > > in > > the standard library. > > > > - Beginners often have difficulty (due to inexperience, lack of confidence > > or > > knowledge) in *finding*, let alone installing and using, packages that > > aren't > > in the standard library. > > > > - To people in the Linux world, adding anything outside of your distro's > > packaging system is a nuisance. No matter how easy your packaging library > > makes it, you now have two sorts of packages: first-class packages that > > your distro will automatically update for you, and second-class ones that > > aren't. > > > > - People working in restrictive corporate systems often have to jump through > > flaming hoops before installing software. > > > > > I would also add that for people new to writing Python modules and > that want to distribute them, it's hard to evaluate what they are > "supposed" to use (distutils, setuptools, distribute, bento, etc). > Just a day or two ago, this exact question was asked on the Distutils > mailing list with subject "Confusion of a hobby programmer." Code not > being in the standard library creates an extra mental hurdle to > overcome. > > I agree that eventually the stdlib needs standard tooling to work with the future (?) but until that future is in use adding it to the stdlib feels premature to me. -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Wed Feb 20 09:02:39 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 20 Feb 2013 08:02:39 +0000 Subject: [Python-Dev] Fwd: PEP 426 is now the draft spec for distribution metadata 2.0 In-Reply-To: References: <512347ED.2040905@egenix.com> <51237CEB.3020701@egenix.com> <5123EFEA.70600@egenix.com> <5123F849.6080703@egenix.com> <6ECA090456F34E3CB3E0974093CB9E4C@gmail.com> Message-ID: On 20 February 2013 04:07, Tres Seaver wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > On 02/19/2013 09:37 PM, Paul Moore wrote: >> On 20 February 2013 00:54, Fred Drake wrote: >>> I'd posit that anything successful will no longer need to be added >>> to the standard library, to boot. Packaging hasn't done well >>> there. >> >> distlib may be the exception, though. Packaging tools are somewhat >> unique because of the chicken and egg issue involved in having a >> packaging tool with external dependencies - who installs your >> dependencies for you? So enabling technology (library code to perform >> packaging-related tasks, particularly in support of standardised >> formats) could be better available from the stdlib. > > The big blocker there is that users can't rely on having it in the stdlib > until after they drop Python < 3.4 (assuming accelearated absorption) or > even 3.5. Understood - that's why I suggested that distlib reach a point where it's stable as an external package and supported on (some) older versions. I'm hoping for an experience more like unittest2 than packaging/distutils2. But we shouldn't be blocking new stdlib features just because they won't be available in older Python versions... Paul From chris.jerdonek at gmail.com Wed Feb 20 09:03:18 2013 From: chris.jerdonek at gmail.com (Chris Jerdonek) Date: Wed, 20 Feb 2013 00:03:18 -0800 Subject: [Python-Dev] Fwd: PEP 426 is now the draft spec for distribution metadata 2.0 In-Reply-To: References: <512347ED.2040905@egenix.com> <51237CEB.3020701@egenix.com> <5123EFEA.70600@egenix.com> <5123F849.6080703@egenix.com> <6ECA090456F34E3CB3E0974093CB9E4C@gmail.com> <5124378E.6020607@pearwood.info> Message-ID: > On Wednesday, February 20, 2013 at 2:48 AM, Chris Jerdonek wrote: > > I meant that bringing distlib into http://hg.python.org/cpython/ would > give it more visibility to core devs and others that already keep an > eye on python-checkins (the mailing list). And I think seeing the > Sphinx-processed docs integrated and cross-referenced with > http://docs.python.org/dev/ will help people understand better what > has been done and how it fits in with the rest of CPython -- which I > think would be useful to the community. It may also encourage > involvement (e.g. by being part of the main tracker). On Tue, Feb 19, 2013 at 11:53 PM, Donald Stufft wrote: > On the other hand it makes contributing to it more annoying since it > does not have pull requests, unless it was just a mirror. Maybe just the finished/production-ready pieces could be added as they are ready, with the main development happening outside. My understanding of distlib is that it's a collection of independent, bite-sized pieces of functionality, which could lend itself well to such a process. --Chris From rahulgarg at live.ca Wed Feb 20 09:03:14 2013 From: rahulgarg at live.ca (rahul garg) Date: Wed, 20 Feb 2013 03:03:14 -0500 Subject: [Python-Dev] Built with VS2012 Express for desktop In-Reply-To: References: , Message-ID: > Date: Tue, 19 Feb 2013 12:48:02 -0600 > Subject: Re: [Python-Dev] Built with VS2012 Express for desktop > From: brian at python.org > To: rahulgarg at live.ca > CC: python-dev at python.org > > On Tue, Feb 19, 2013 at 12:37 PM, rahul garg wrote: > > Hi. > > > > I downloaded Python 3.3 source, opened up the solution in VS2012 Express for > > Desktop and built the "python" subproject using "Release" and "x64" > > configurations. I now have a "python.exe" in the PCBuild/amd64 subfolder > > that appears to be working as far as i can see. > > > > I have a few questions: > > a) Is there a test suite that I can run to verify that the build is working > > fine? > Last I checked there are some issues, but most of the tests should > pass. You would run "PCBuild\python.exe -m test" from the top level of > your checkout to see this. It's also covered at > http://docs.python.org/devguide/ Thanks! That page is what I should have looked for! > > b) I now intend to build some extensions (such as NumPy). Not sure if this > > is the right list, but would I need to modify something in distutils to get > > it working with VS2012? > > Yes. You'll probably need to point distutils to the correct batch file > that sets up a VS2012 build environment. Thanks again! rahul -------------- next part -------------- An HTML attachment was scrubbed... URL: From phd at phdru.name Wed Feb 20 10:09:13 2013 From: phd at phdru.name (Oleg Broytman) Date: Wed, 20 Feb 2013 13:09:13 +0400 Subject: [Python-Dev] Built with VS2012 Express for desktop In-Reply-To: <20130220082319.02424aeb@pitrou.net> References: <20130219185025.GB13258@iskra.aviel.ru> <20130220082319.02424aeb@pitrou.net> Message-ID: <20130220090913.GA2026@iskra.aviel.ru> On Wed, Feb 20, 2013 at 08:23:19AM +0100, Antoine Pitrou wrote: > On Tue, 19 Feb 2013 20:37:36 -0800 > Eli Bendersky wrote: > > On Tue, Feb 19, 2013 at 10:50 AM, Oleg Broytman wrote: > > Oleg, lately I have the feeling you're getting too automatic with this > > template response. > > +1. This isn't the python-robots mailing-list. Ok, I'll stop. Oleg. -- Oleg Broytman http://phdru.name/ phd at phdru.name Programmers don't die, they just GOSUB without RETURN. From mal at egenix.com Wed Feb 20 10:25:54 2013 From: mal at egenix.com (M.-A. Lemburg) Date: Wed, 20 Feb 2013 10:25:54 +0100 Subject: [Python-Dev] Fwd: PEP 426 is now the draft spec for distribution metadata 2.0 In-Reply-To: References: <512347ED.2040905@egenix.com> <51237CEB.3020701@egenix.com> <5123EFEA.70600@egenix.com> <5123F849.6080703@egenix.com> <6ECA090456F34E3CB3E0974093CB9E4C@gmail.com> Message-ID: <512496A2.2060902@egenix.com> On 20.02.2013 03:37, Paul Moore wrote: > On 20 February 2013 00:54, Fred Drake wrote: >> I'd posit that anything successful will no longer need to be added to >> the standard library, to boot. Packaging hasn't done well there. > > distlib may be the exception, though. Packaging tools are somewhat > unique because of the chicken and egg issue involved in having a > packaging tool with external dependencies - who installs your > dependencies for you? So enabling technology (library code to perform > packaging-related tasks, particularly in support of standardised > formats) could be better available from the stdlib. > >> I'd rather see a successful packaging story develop than bundle it into the >> standard library. The later just isn't that interesting any more. > > Bundling too early is a bad idea though. distlib is developing fast > and to do so it needs (1) a development cycle independent of python's > and (2) compatibility and ease of use with earlier versions of Python > (the latter is also critical for adoption in place of custom code in > packaging tools). > > Aiming for an accelerated level of development targeting inclusion in > Python 3.4 is plausible, though. MAL pointed out that agreeing > standards but not offering tools to support them in the stdlib is > risky, as people have no incentive to adopt those standards. We've got > 6 months or more until 3.4 feature freeze, let's not make any decision > too soon, though. I'm fine with not adding distlib to Python 3.4. The point I wanted to make was that there has to be an reference implementation of the PEP that tool makers can use to avoid reinventing the wheel over and over again (each with its own set of problems). If distlib implements the PEP, then it just needs to be mentioned there as a suitable reference implementation. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Feb 20 2013) >>> Python Projects, Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From solipsis at pitrou.net Wed Feb 20 10:54:06 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 20 Feb 2013 10:54:06 +0100 Subject: [Python-Dev] Built with VS2012 Express for desktop References: <20130219185025.GB13258@iskra.aviel.ru> <20130220082319.02424aeb@pitrou.net> <20130220090913.GA2026@iskra.aviel.ru> Message-ID: <20130220105406.5ce4f5c0@pitrou.net> Le Wed, 20 Feb 2013 13:09:13 +0400, Oleg Broytman a ?crit : > On Wed, Feb 20, 2013 at 08:23:19AM +0100, Antoine Pitrou > wrote: > > On Tue, 19 Feb 2013 20:37:36 -0800 > > Eli Bendersky wrote: > > > On Tue, Feb 19, 2013 at 10:50 AM, Oleg Broytman > > > wrote: Oleg, lately I have the feeling you're getting too > > > automatic with this template response. > > > > +1. This isn't the python-robots mailing-list. > > Ok, I'll stop. Thanks! Perhaps we may have to become more automatic if some day we are flooded with newbie messages on the list, but this isn't the case currently. Moreover, the original message here was a build question that probably wouldn't receive any meaningful answer on python-list, which makes it suitable for python-dev. Regards Antoine. From solipsis at pitrou.net Wed Feb 20 10:56:21 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 20 Feb 2013 10:56:21 +0100 Subject: [Python-Dev] PEP 426 is now the draft spec for distribution metadata 2.0 References: <512347ED.2040905@egenix.com> <51237CEB.3020701@egenix.com> <5123EFEA.70600@egenix.com> <5123F849.6080703@egenix.com> <6ECA090456F34E3CB3E0974093CB9E4C@gmail.com> Message-ID: <20130220105621.1029633d@pitrou.net> Le Tue, 19 Feb 2013 19:54:21 -0500, Fred Drake a ?crit : > On Tue, Feb 19, 2013 at 6:19 PM, Donald Stufft > wrote: > > Let's not add anything to the stdlib till it has real world usage. > > Doing otherwise is putting the cart before the horse. > > I'd posit that anything successful will no longer need to be added to > the standard > library, to boot. Packaging hasn't done well there. If only "successful", agreed. However, packaging is a core part of the Python experience, and as such should be provided in the stdlib. Regards Antoine. From phd at phdru.name Wed Feb 20 11:04:28 2013 From: phd at phdru.name (Oleg Broytman) Date: Wed, 20 Feb 2013 14:04:28 +0400 Subject: [Python-Dev] Built with VS2012 Express for desktop In-Reply-To: <20130220105406.5ce4f5c0@pitrou.net> References: <20130219185025.GB13258@iskra.aviel.ru> <20130220082319.02424aeb@pitrou.net> <20130220090913.GA2026@iskra.aviel.ru> <20130220105406.5ce4f5c0@pitrou.net> Message-ID: <20130220100428.GA3520@iskra.aviel.ru> On Wed, Feb 20, 2013 at 10:54:06AM +0100, Antoine Pitrou wrote: > Le Wed, 20 Feb 2013 13:09:13 +0400, > Oleg Broytman a ??crit : > > On Wed, Feb 20, 2013 at 08:23:19AM +0100, Antoine Pitrou > > wrote: > > > On Tue, 19 Feb 2013 20:37:36 -0800 > > > Eli Bendersky wrote: > > > > On Tue, Feb 19, 2013 at 10:50 AM, Oleg Broytman > > > > wrote: Oleg, lately I have the feeling you're getting too > > > > automatic with this template response. > > > > > > +1. This isn't the python-robots mailing-list. > > > > Ok, I'll stop. > > Thanks! Perhaps we may have to become more automatic if some day we are > flooded with newbie messages on the list, but this isn't the case > currently. That was the case when I started. > Moreover, the original message here was a build question that probably > wouldn't receive any meaningful answer on python-list, which makes it > suitable for python-dev. You are right. I am sorry. Oleg. -- Oleg Broytman http://phdru.name/ phd at phdru.name Programmers don't die, they just GOSUB without RETURN. From mal at egenix.com Wed Feb 20 11:30:28 2013 From: mal at egenix.com (M.-A. Lemburg) Date: Wed, 20 Feb 2013 11:30:28 +0100 Subject: [Python-Dev] Fwd: PEP 426 is now the draft spec for distribution metadata 2.0 In-Reply-To: References: <512347ED.2040905@egenix.com> <51237CEB.3020701@egenix.com> <5123EFEA.70600@egenix.com> <5123F849.6080703@egenix.com> Message-ID: <5124A5C4.4090202@egenix.com> On 20.02.2013 00:16, Daniel Holth wrote: > On Tue, Feb 19, 2013 at 5:10 PM, M.-A. Lemburg wrote: > >> On 19.02.2013 23:01, Daniel Holth wrote: >>> On Tue, Feb 19, 2013 at 4:34 PM, M.-A. Lemburg wrote: >>> >>>> On 19.02.2013 14:40, Nick Coghlan wrote: >>>>> On Tue, Feb 19, 2013 at 11:23 PM, M.-A. Lemburg >> wrote: >>>>>> * PEP 426 doesn't include any mention of the egg distribution format, >>>>>> even though it's the most popular distribution format at the moment. >>>>>> It should at least include the location of the metadata file >>>>>> in eggs (EGG-INFO/PKG-INFO) and egg installations >>>>>> (/EGG-INFO/PKG-INFO). >>>>> >>>>> "Other tools involved in Python distribution may also use this format." >>>>> >>>>> The egg format has never been, and never will be, officially endorsed >>>>> by python-dev. The wheel format is the standard format for binary >>>>> distribution, and PEP 376 defines the standard location for metadata >>>>> on installed distributions. >>>> >>>> Oh, come on, Nick, that's just silly. setuptools was included in stdlib >>>> for a short while, so the above is simply wrong. Eggs are the most >>>> widely used binary distribution format for Python package on PyPI: >>>> >>>> # wc *files.csv >>>> 25585 25598 1431013 2013-02-19-egg-files.csv >>>> 4619 4640 236694 2013-02-19-exe-files.csv >>>> 254 255 13402 2013-02-19-msi-files.csv >>>> 104691 104853 5251962 2013-02-19-tar-gz-files.csv >>>> 24 24 1221 2013-02-19-whl-files.csv >>>> 17937 18022 905913 2013-02-19-zip-files.csv >>>> 153110 153392 7840205 total >>>> >>>> (based on todays PyPI stats) >>>> >>>> It doesn't really help ignoring realities... and I'm saying >>>> that as one of the core devs who got setuptools kicked out of >>>> the stdlib again. >>>> >>>> -- >>>> Marc-Andre Lemburg >>>> eGenix.com >>>> >>> >>> The wheel philosophy is that it should be supported by both python-dev >> and >>> setuptools and that you should feel happy about using setuptools if you >>> like it whether or not python-dev (currently) endorses that. If you are >>> using setuptools (distribute's pkg_resources) then you can use both at >> the >>> same time. >>> >>> Distribute, distutils and setuptools' problems have not been well >>> understood which I think is why there has been a need to discredit >>> setuptools by calling it non-standard. It is the defacto standard. If >> your >>> packages have dependencies there is no other choice. Wheel tries to solve >>> the real problem by allowing you to build a package with setuptools while >>> giving the end-user the choice of installing setuptools or not. >>> >>> Of course eggs are the most popular right now. The wheel format is very >>> egg-like while avoiding some of egg's problems. See the comparison in the >>> PEP or read the story on wheel's rtfd. The wheel project includes tools >> to >>> losslessly convert eggs or bdist_wininst to wheel. >> >> That's all fine, but it doesn't explain the refusal to add the >> documentation of the location of the PKG-INFO file in eggs ? > > > It would just be a sentence, I wouldn't have a problem with it but I also > don't see why it would be necessary. Even setuptools doesn't touch the file > usually. Right now distribute's pkg_resources currently only understands > Requires-Dist if it is inside a .dist-info directory. Perhaps I'm not clear enough. I'll try again :-) The wording in the PEP alienates the egg format by defining an incompatible new standard for the location of the metadata file: """ There are three standard locations for these metadata files: * the PKG-INFO file included in the base directory of Python source distribution archives (as created by the distutils sdist command) * the {distribution}-{version}.dist-info/METADATA file in a wheel binary distribution archive (as described in PEP 425, or a later version of that specification) * the {distribution}-{version}.dist-info/METADATA files in a local Python installation database (as described in PEP 376, or a later version of that specification) """ It's easy to upgrade distribute and distutils to write metadata 1.2 format, simply by changing the version in the PKG-INFO files. These addition are necessary to fix the above and also include the standard location of the metadata for pip and distutils installations: * the EGG-INFO/PKG-INFO file in an egg binary distribution archive (as created by the distribute bdist_egg command) * the {distribution}-{version}.egg/EGG-INFO/PKG-INFO file in an installed egg distribution archive * the {distribution}-{version}.egg-info/PKG-INFO file for packages installed with "pip install" or distribute's "python setup.py install" * the {distribution}-{version}.egg-info file for packages installed with distutils' "python setup.py install" -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Feb 20 2013) >>> Python Projects, Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From barry at python.org Wed Feb 20 17:14:57 2013 From: barry at python.org (Barry Warsaw) Date: Wed, 20 Feb 2013 11:14:57 -0500 Subject: [Python-Dev] The last Python 2.6 security release Message-ID: <20130220111457.07e4ae17@anarchist.wooz.org> I just updated PEP 361 (Python 2.6 release schedule). Python 2.6 is in security maintenance, source only release mode. Official support for Python 2.6 expires on October 1 2013, and I would like to do one last release[1], i.e. 2.6.9 as close to that date as possible. I know of issue 16248, but if there are any other security issues that must be applied to 2.6, please let me know. Cheers, -Barry [1] Guido's time machine works in both directions and prevents a 2.6.10 from ever happening. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From benjamin at python.org Wed Feb 20 17:25:29 2013 From: benjamin at python.org (Benjamin Peterson) Date: Wed, 20 Feb 2013 11:25:29 -0500 Subject: [Python-Dev] XML DoS vulnerabilities and exploits in Python In-Reply-To: <51239B07.8080300@python.org> References: <51239B07.8080300@python.org> Message-ID: 2013/2/19 Christian Heimes : > Hello, > > in August 2012 I found a DoS vulnerability in expat and XML libraries in > Python's standard library. Since then I have found several more issues. > I have been working on fixes ever since. > > The README of https://pypi.python.org/pypi/defusedxml contains detailed > explanations of my research and all issues > > > Blog post: > http://blog.python.org/2013/02/announcing-defusedxml-fixes-for-xml.html > > Hotfixes: > https://pypi.python.org/pypi/defusedxml > https://pypi.python.org/pypi/defusedexpat Are these going to become patches for Python, too? -- Regards, Benjamin From christian at python.org Wed Feb 20 19:24:12 2013 From: christian at python.org (Christian Heimes) Date: Wed, 20 Feb 2013 19:24:12 +0100 Subject: [Python-Dev] XML DoS vulnerabilities and exploits in Python In-Reply-To: References: <51239B07.8080300@python.org> Message-ID: <512514CC.2080508@python.org> Am 20.02.2013 17:25, schrieb Benjamin Peterson: > Are these going to become patches for Python, too? I'm working on it. The patches need to be discussed as they break backward compatibility and AFAIK XML standards, too. From fijall at gmail.com Wed Feb 20 21:17:43 2013 From: fijall at gmail.com (Maciej Fijalkowski) Date: Wed, 20 Feb 2013 22:17:43 +0200 Subject: [Python-Dev] XML DoS vulnerabilities and exploits in Python In-Reply-To: <512514CC.2080508@python.org> References: <51239B07.8080300@python.org> <512514CC.2080508@python.org> Message-ID: On Wed, Feb 20, 2013 at 8:24 PM, Christian Heimes wrote: > Am 20.02.2013 17:25, schrieb Benjamin Peterson: >> Are these going to become patches for Python, too? > > I'm working on it. The patches need to be discussed as they break > backward compatibility and AFAIK XML standards, too. That's not very good. XML parsers are supposed to parse XML according to standards. Is the goal to have them actually do that, or just address DDOS issues? Cheers, fijal From ericsnowcurrently at gmail.com Wed Feb 20 21:50:04 2013 From: ericsnowcurrently at gmail.com (Eric Snow) Date: Wed, 20 Feb 2013 13:50:04 -0700 Subject: [Python-Dev] [Python-checkins] cpython (3.3): Rebuild importlib.h after the changes introduced in 0f65bf6063ca. In-Reply-To: <3ZB9774yS5z7Lld@mail.python.org> References: <3ZB9774yS5z7Lld@mail.python.org> Message-ID: On Wed, Feb 20, 2013 at 1:16 PM, ezio.melotti wrote: > http://hg.python.org/cpython/rev/9d00c79b27e1 > changeset: 82280:9d00c79b27e1 > branch: 3.3 > parent: 82278:96b4acb253f8 > user: Ezio Melotti > date: Wed Feb 20 21:42:46 2013 +0200 > summary: > Rebuild importlib.h after the changes introduced in 0f65bf6063ca. Thanks, Ezio. -eric From skip at pobox.com Wed Feb 20 21:53:55 2013 From: skip at pobox.com (Skip Montanaro) Date: Wed, 20 Feb 2013 14:53:55 -0600 Subject: [Python-Dev] XML DoS vulnerabilities and exploits in Python In-Reply-To: References: <51239B07.8080300@python.org> <512514CC.2080508@python.org> Message-ID: > > I'm working on it. The patches need to be discussed as they break > > backward compatibility and AFAIK XML standards, too. > > That's not very good. XML parsers are supposed to parse XML according > to standards. Is the goal to have them actually do that, or just > address DDOS issues? Having read through Christian's mail and several of his references, it seems to me that addressing the DDoS issues is preferable to blindly following a standard that predates the Morris worm by a couple years. Everyone played nice before that watershed event. Heck, back then you could telnet to gnu at prep.ai.mit.edu without a password! Any incompatibility should have minimal impact. I haven't looked into the defusedxml package to see what limits it introduces to protect against attacks, but it seems that most well-behaved entities will use little, if any, recursion, and result in a size increase of less than a factor of 10 when fully expanded. Skip From carl at oddbird.net Wed Feb 20 22:02:45 2013 From: carl at oddbird.net (Carl Meyer) Date: Wed, 20 Feb 2013 14:02:45 -0700 Subject: [Python-Dev] XML DoS vulnerabilities and exploits in Python In-Reply-To: References: <51239B07.8080300@python.org> <512514CC.2080508@python.org> Message-ID: <512539F5.9020108@oddbird.net> On 02/20/2013 01:53 PM, Skip Montanaro wrote: >> That's not very good. XML parsers are supposed to parse XML according >> to standards. Is the goal to have them actually do that, or just >> address DDOS issues? > > Having read through Christian's mail and several of his references, it > seems to me that addressing the DDoS issues is preferable to blindly > following a standard that predates the Morris worm by a couple years. > Everyone played nice before that watershed event. Heck, back then you > could telnet to gnu at prep.ai.mit.edu without a password! Also, despite the title of this thread, the vulnerabilities include fetching of external DTDs and entities (per standard), which opens up attacks that are worse than just denial-of-service. In our initial Django release advisory we carelessly lumped the potential XML vulnerabilities together under the "DoS" label, and were quickly corrected. An XML parser that follows the XML standard is never safe to expose to untrusted input. This means the choice is just whether the stdlib XML parsers should be safe by default, or follow the standard by default. (Given either choice, the other option can still be made available via flags). Carl From christian at python.org Wed Feb 20 22:55:57 2013 From: christian at python.org (Christian Heimes) Date: Wed, 20 Feb 2013 22:55:57 +0100 Subject: [Python-Dev] XML DoS vulnerabilities and exploits in Python In-Reply-To: References: <51239B07.8080300@python.org> <512514CC.2080508@python.org> Message-ID: <5125466D.5090308@python.org> Am 20.02.2013 21:17, schrieb Maciej Fijalkowski: > On Wed, Feb 20, 2013 at 8:24 PM, Christian Heimes wrote: >> Am 20.02.2013 17:25, schrieb Benjamin Peterson: >>> Are these going to become patches for Python, too? >> >> I'm working on it. The patches need to be discussed as they break >> backward compatibility and AFAIK XML standards, too. > > That's not very good. XML parsers are supposed to parse XML according > to standards. Is the goal to have them actually do that, or just > address DDOS issues? But the standard is flawed. It's not a distributed DoS issue, it's a severe DoS vulnerabilities. A single 1 kB XML document can kill virtually any machine, even servers with more than hundred GB RAM. From christian at python.org Wed Feb 20 23:06:15 2013 From: christian at python.org (Christian Heimes) Date: Wed, 20 Feb 2013 23:06:15 +0100 Subject: [Python-Dev] XML DoS vulnerabilities and exploits in Python In-Reply-To: <512539F5.9020108@oddbird.net> References: <51239B07.8080300@python.org> <512514CC.2080508@python.org> <512539F5.9020108@oddbird.net> Message-ID: <512548D7.8010806@python.org> Am 20.02.2013 22:02, schrieb Carl Meyer: > Also, despite the title of this thread, the vulnerabilities include > fetching of external DTDs and entities (per standard), which opens up > attacks that are worse than just denial-of-service. In our initial > Django release advisory we carelessly lumped the potential XML > vulnerabilities together under the "DoS" label, and were quickly corrected. Right, I tried to address both kinds of issues in the title: XML DoS vulnerabilities and (other XML) exploits Christian From christian at python.org Wed Feb 20 23:06:15 2013 From: christian at python.org (Christian Heimes) Date: Wed, 20 Feb 2013 23:06:15 +0100 Subject: [Python-Dev] XML DoS vulnerabilities and exploits in Python In-Reply-To: <512539F5.9020108@oddbird.net> References: <51239B07.8080300@python.org> <512514CC.2080508@python.org> <512539F5.9020108@oddbird.net> Message-ID: <512548D7.8010806@python.org> Am 20.02.2013 22:02, schrieb Carl Meyer: > Also, despite the title of this thread, the vulnerabilities include > fetching of external DTDs and entities (per standard), which opens up > attacks that are worse than just denial-of-service. In our initial > Django release advisory we carelessly lumped the potential XML > vulnerabilities together under the "DoS" label, and were quickly corrected. Right, I tried to address both kinds of issues in the title: XML DoS vulnerabilities and (other XML) exploits Christian From greg.ewing at canterbury.ac.nz Wed Feb 20 23:35:23 2013 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 21 Feb 2013 11:35:23 +1300 Subject: [Python-Dev] XML DoS vulnerabilities and exploits in Python In-Reply-To: <512539F5.9020108@oddbird.net> References: <51239B07.8080300@python.org> <512514CC.2080508@python.org> <512539F5.9020108@oddbird.net> Message-ID: <51254FAB.4020801@canterbury.ac.nz> Carl Meyer wrote: > An XML parser that follows the XML standard is never safe to expose to > untrusted input. Does the XML standard really mandate that a conforming parser must blindly download any DTD URL given to it from the real live internet? Somehow I doubt that. -- Greg From rdmurray at bitdance.com Wed Feb 20 23:45:58 2013 From: rdmurray at bitdance.com (R. David Murray) Date: Wed, 20 Feb 2013 17:45:58 -0500 Subject: [Python-Dev] XML DoS vulnerabilities and exploits in Python In-Reply-To: <51254FAB.4020801@canterbury.ac.nz> References: <51239B07.8080300@python.org> <512514CC.2080508@python.org> <512539F5.9020108@oddbird.net> <51254FAB.4020801@canterbury.ac.nz> Message-ID: <20130220224558.B3A2A250BC1@webabinitio.net> On Thu, 21 Feb 2013 11:35:23 +1300, Greg Ewing wrote: > Carl Meyer wrote: > > An XML parser that follows the XML standard is never safe to expose to > > untrusted input. > > Does the XML standard really mandate that a conforming parser > must blindly download any DTD URL given to it from the real > live internet? Somehow I doubt that. I don't believe it does. The DTD URL is, if I remember correctly, specified as an identifier. The fact that you can often also download the DTD from the location specified by the identifier is a secondary effect. But, it's been a *long* time since I looked at XML :) (Wikipedia says: "Programs for reading documents may not be required to read the external subset.", which would seem to confirm that.) --David From fred at fdrake.net Wed Feb 20 23:56:15 2013 From: fred at fdrake.net (Fred Drake) Date: Wed, 20 Feb 2013 17:56:15 -0500 Subject: [Python-Dev] XML DoS vulnerabilities and exploits in Python In-Reply-To: <20130220224558.B3A2A250BC1@webabinitio.net> References: <51239B07.8080300@python.org> <512514CC.2080508@python.org> <512539F5.9020108@oddbird.net> <51254FAB.4020801@canterbury.ac.nz> <20130220224558.B3A2A250BC1@webabinitio.net> Message-ID: On Wed, Feb 20, 2013 at 5:45 PM, R. David Murray wrote: > (Wikipedia says: "Programs for reading documents may not be required to > read the external subset.", which would seem to confirm that.) Validating parsers are required to read the external subset; this doesn't apply to the parsers distributed for Python today. Even when loading external resources, I don't think there's anything in the XML specification that says how they have to be loaded, or how to deal with an error when they are (and refusing to load because of resource limits is reasonably just another error with respect to the parser). While I'd hate to make XML processing more painful than it often is, there's no injunction not to be reasonable. Security concerns and resource limits are cross-cutting concerns, so it's not wrong to provide safe defaults. Doing so *will* be backward incompatible, and I'm not sure there's a good way to gauge the extent of the breakage. -Fred -- Fred L. Drake, Jr. "A storm broke loose in my mind." --Albert Einstein From carl at oddbird.net Thu Feb 21 00:03:34 2013 From: carl at oddbird.net (Carl Meyer) Date: Wed, 20 Feb 2013 16:03:34 -0700 Subject: [Python-Dev] XML DoS vulnerabilities and exploits in Python In-Reply-To: <51254FAB.4020801@canterbury.ac.nz> References: <51239B07.8080300@python.org> <512514CC.2080508@python.org> <512539F5.9020108@oddbird.net> <51254FAB.4020801@canterbury.ac.nz> Message-ID: <51255646.8080105@oddbird.net> On 02/20/2013 03:35 PM, Greg Ewing wrote: > Carl Meyer wrote: >> An XML parser that follows the XML standard is never safe to expose to >> untrusted input. > > Does the XML standard really mandate that a conforming parser > must blindly download any DTD URL given to it from the real > live internet? Somehow I doubt that. For a validating parser, the spec does mandate that. It permits non-validating parsers (browsers are the only example given) to simply note the existence of an external entity reference and "retrieve it for display only on demand." [1] But this isn't particularly relevant; the quoted statement is true even if you ignore the external reference issues entirely and consider only entity-expansion DoS. Some level of non-conformance to the spec is necessary to make parsing of untrusted XML safe. Carl [1] http://www.w3.org/TR/xml/#include-if-valid From christian at python.org Thu Feb 21 00:07:33 2013 From: christian at python.org (Christian Heimes) Date: Thu, 21 Feb 2013 00:07:33 +0100 Subject: [Python-Dev] XML DoS vulnerabilities and exploits in Python In-Reply-To: <20130220224558.B3A2A250BC1@webabinitio.net> References: <51239B07.8080300@python.org> <512514CC.2080508@python.org> <512539F5.9020108@oddbird.net> <51254FAB.4020801@canterbury.ac.nz> <20130220224558.B3A2A250BC1@webabinitio.net> Message-ID: <51255735.10908@python.org> Am 20.02.2013 23:45, schrieb R. David Murray: > I don't believe it does. The DTD URL is, if I remember correctly, > specified as an identifier. The fact that you can often also download the > DTD from the location specified by the identifier is a secondary effect. > > But, it's been a *long* time since I looked at XML :) A DTD may have an identifier and a resource locator (local file or URL). It depends which kind of DTD is used (internal, external public or external system), e.g. . For external DTDs a parser may chose to cache a DTD or map DTD identifier to its own set of DTDs. As far as I know a parser doesn't have to download a DTD unless it runs in validation mode. Just xml.sax and xml.dom.pulldom download DTDs, see https://pypi.python.org/pypi/defusedxml#python-xml-libraries DTD retrieval is not as severe as external entity expansion. With external entities like an attacker is actually able to download files and circumvent firewalls if the application returns parts of the XML file back. Most XML parsers expand entities and lots of them even expand external entities. Daniel Veillard (libxml2) has explained that entity expansion is required for XPath() and IIRC for features like XSL, too. Nowadays most XML parsers and libraries have options to disable certain features. Python's standard library doesn't have options for some features or ignores other settings silently. Everything is documented at https://pypi.python.org/pypi/defusedxml, too. Christian From solipsis at pitrou.net Thu Feb 21 00:08:08 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 21 Feb 2013 00:08:08 +0100 Subject: [Python-Dev] XML DoS vulnerabilities and exploits in Python References: <51239B07.8080300@python.org> <512514CC.2080508@python.org> <5125466D.5090308@python.org> Message-ID: <20130221000808.7a8d2989@pitrou.net> On Wed, 20 Feb 2013 22:55:57 +0100 Christian Heimes wrote: > Am 20.02.2013 21:17, schrieb Maciej Fijalkowski: > > On Wed, Feb 20, 2013 at 8:24 PM, Christian Heimes wrote: > >> Am 20.02.2013 17:25, schrieb Benjamin Peterson: > >>> Are these going to become patches for Python, too? > >> > >> I'm working on it. The patches need to be discussed as they break > >> backward compatibility and AFAIK XML standards, too. > > > > That's not very good. XML parsers are supposed to parse XML according > > to standards. Is the goal to have them actually do that, or just > > address DDOS issues? > > But the standard is flawed. It is not flawed as long as you are operating in a sandbox (read: controlled environment). > It's not a distributed DoS issue, it's a severe DoS vulnerabilities. A > single 1 kB XML document can kill virtually any machine, even servers > with more than hundred GB RAM. Assuming an attacker can inject arbitrary XML. Not every XML document is loaded from the Internet. Not everyone is a security nuts. Regards Antoine. From donald.stufft at gmail.com Thu Feb 21 00:21:22 2013 From: donald.stufft at gmail.com (Donald Stufft) Date: Wed, 20 Feb 2013 18:21:22 -0500 Subject: [Python-Dev] XML DoS vulnerabilities and exploits in Python In-Reply-To: <20130221000808.7a8d2989@pitrou.net> References: <51239B07.8080300@python.org> <512514CC.2080508@python.org> <5125466D.5090308@python.org> <20130221000808.7a8d2989@pitrou.net> Message-ID: <9AC9F1A0A9BD422999B3AE43AA5C3BA3@gmail.com> On Wednesday, February 20, 2013 at 6:08 PM, Antoine Pitrou wrote: > > It's not a distributed DoS issue, it's a severe DoS vulnerabilities. A > > single 1 kB XML document can kill virtually any machine, even servers > > with more than hundred GB RAM. > > > > > Assuming an attacker can inject arbitrary XML. Not every XML document > is loaded from the Internet. > > Even documents not loaded from the internet can be at risk. Often times security breaches are the result of a chain of actions. You can say "I'm not loading this XML from the internet, so therefore I am safe" but then you have another flaw (for example) where you unpack a zip file without verifying there are not absolute paths and suddenly your xml file has been replaces with a malicious one. > Not everyone is a security nuts. > > This is precisely why things should be safe by default and allow unsafe actions to be turned on optionally. -------------- next part -------------- An HTML attachment was scrubbed... URL: From christian at python.org Thu Feb 21 00:23:52 2013 From: christian at python.org (Christian Heimes) Date: Thu, 21 Feb 2013 00:23:52 +0100 Subject: [Python-Dev] XML DoS vulnerabilities and exploits in Python In-Reply-To: References: <51239B07.8080300@python.org> <512514CC.2080508@python.org> <512539F5.9020108@oddbird.net> <51254FAB.4020801@canterbury.ac.nz> <20130220224558.B3A2A250BC1@webabinitio.net> Message-ID: <51255B08.7010003@python.org> Am 20.02.2013 23:56, schrieb Fred Drake: > While I'd hate to make XML processing more painful than it often is, there's > no injunction not to be reasonable. Security concerns and resource limits > are cross-cutting concerns, so it's not wrong to provide safe defaults. > > Doing so *will* be backward incompatible, and I'm not sure there's a good > way to gauge the extent of the breakage. We could walk a different path but that would keep Python's XML libraries in an insecure mode by default. My latest patch to expat and pyexpat supports global default values. The global defaults are used when a new parser is created with pyexpat.ParserCreate(). It's also possible to disable the new limitations in expat by default. We can add a function to the XML package tree that enables all restrictions: * limit expansion depths of nested entities * limit total amount of expanded chars * disable external entity expansion * optionally force expat to ignore and reset all DTD information 3rd party users have to disable secure settings explicitly for the current interpreter (although expat limits are process wide and shared across subinterpreters). try: import xml.security except ImportError: # old Python pass else: xml.security.harden_xml_parser() I guess most programs either process untrusted XML input or large XML documents that require expansion and DTD validation. Christian From christian at python.org Thu Feb 21 00:30:56 2013 From: christian at python.org (Christian Heimes) Date: Thu, 21 Feb 2013 00:30:56 +0100 Subject: [Python-Dev] XML DoS vulnerabilities and exploits in Python In-Reply-To: <20130221000808.7a8d2989@pitrou.net> References: <51239B07.8080300@python.org> <512514CC.2080508@python.org> <5125466D.5090308@python.org> <20130221000808.7a8d2989@pitrou.net> Message-ID: <51255CB0.8060706@python.org> Am 21.02.2013 00:08, schrieb Antoine Pitrou: > Not everyone is a security nuts. But, but, but ... it's fun to be paranoid! You get so many new potential enemies. :) Jerry Fletcher From solipsis at pitrou.net Thu Feb 21 00:22:58 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 21 Feb 2013 00:22:58 +0100 Subject: [Python-Dev] XML DoS vulnerabilities and exploits in Python In-Reply-To: <9AC9F1A0A9BD422999B3AE43AA5C3BA3@gmail.com> References: <51239B07.8080300@python.org> <512514CC.2080508@python.org> <5125466D.5090308@python.org> <20130221000808.7a8d2989@pitrou.net> <9AC9F1A0A9BD422999B3AE43AA5C3BA3@gmail.com> Message-ID: <20130221002258.78a805d6@pitrou.net> On Wed, 20 Feb 2013 18:21:22 -0500 Donald Stufft wrote: > On Wednesday, February 20, 2013 at 6:08 PM, Antoine Pitrou wrote: > > > It's not a distributed DoS issue, it's a severe DoS vulnerabilities. A > > > single 1 kB XML document can kill virtually any machine, even servers > > > with more than hundred GB RAM. > > > > > > > Assuming an attacker can inject arbitrary XML. Not every XML document > > is loaded from the Internet. > > Even documents not loaded from the internet can be at risk. Often times > security breaches are the result of a chain of actions. You can say "I'm > not loading this XML from the internet, so therefore I am safe" but then > you have another flaw (for example) where you unpack a zip file > without verifying there are not absolute paths and suddenly your xml file has > been replaces with a malicious one. Assuming your ZIP file is coming from the untrusted Internet, indeed. Again, this is the same assumption that you are grabbing some important data from someone you can't trust. Just because you are living in a Web-centric world doesn't mean everyone does. There are a lot of use cases which are not impacted by your security rules. Bugfix releases shouldn't break those use cases, which means the security features should be mostly opt-in for 2.7 and 3.3. Regards Antoine. From pje at telecommunity.com Thu Feb 21 00:36:21 2013 From: pje at telecommunity.com (PJ Eby) Date: Wed, 20 Feb 2013 18:36:21 -0500 Subject: [Python-Dev] Fwd: PEP 426 is now the draft spec for distribution metadata 2.0 In-Reply-To: <5124A5C4.4090202@egenix.com> References: <512347ED.2040905@egenix.com> <51237CEB.3020701@egenix.com> <5123EFEA.70600@egenix.com> <5123F849.6080703@egenix.com> <5124A5C4.4090202@egenix.com> Message-ID: On Wed, Feb 20, 2013 at 5:30 AM, M.-A. Lemburg wrote: > The wording in the PEP alienates the egg format by defining > an incompatible new standard for the location of the metadata > file: This isn't a problem, because there's not really a use case at the moment for eggs to include a PEP 426-format metadata file, and if they ever do, it ought to be called METADATA, so that pkg_resources will know to read it if there are no egg-format dependencies listed. Setuptools also doesn't care what format PKG-INFO is in, as it only ever reads the "Version:" field, and that only in the case of in-development source packages. > It's easy to upgrade distribute and distutils to write > metadata 1.2 format, simply by changing the version in the > PKG-INFO files. As soon as distutils does it, setuptools will do it, because setuptools delegates metadata writing to distutils. So there's no "alienation" here. What will need to happen at some point is for pkg_resources to implement support for PEP 426-style version requirements, which I haven't tried to fully wrap my head around as yet. I'm hoping that there are simple textual substitutions (e.g. regexes) that can be done to convert them to pkg_resources requirements. If need be, I'll swipe whatever's needed from distlib. ;-) In the meantime, eggs aren't actually going anywhere, since wheels aren't actually trying to replace all of their use cases. And since the new metadata and binary formats don't actually add much new functionality over what eggs already do, eggs wouldn't lose much by not being able to use the same metadata, anyway. ;-) From donald.stufft at gmail.com Thu Feb 21 00:32:51 2013 From: donald.stufft at gmail.com (Donald Stufft) Date: Wed, 20 Feb 2013 18:32:51 -0500 Subject: [Python-Dev] XML DoS vulnerabilities and exploits in Python In-Reply-To: <51255B08.7010003@python.org> References: <51239B07.8080300@python.org> <512514CC.2080508@python.org> <512539F5.9020108@oddbird.net> <51254FAB.4020801@canterbury.ac.nz> <20130220224558.B3A2A250BC1@webabinitio.net> <51255B08.7010003@python.org> Message-ID: On Wednesday, February 20, 2013 at 6:23 PM, Christian Heimes wrote: > We can add a function to the XML package tree that enables all restrictions: > > * limit expansion depths of nested entities > * limit total amount of expanded chars > * disable external entity expansion > * optionally force expat to ignore and reset all DTD information > > 3rd party users have to disable secure settings explicitly for the > current interpreter (although expat limits are process wide and shared > across subinterpreters). > > try: > import xml.security > except ImportError: > # old Python > pass > else: > xml.security.harden_xml_parser() We've learned nothing from Ruby and their YAML problems. Things need to be safe by default and the unsafe things explicitly enabled. Even *smart* developers do things wrong (e.g. YAML.load instead of YAML.safe_load) and protecting developers by default should be the path forward. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jnoller at gmail.com Thu Feb 21 00:47:53 2013 From: jnoller at gmail.com (Jesse Noller) Date: Wed, 20 Feb 2013 18:47:53 -0500 Subject: [Python-Dev] XML DoS vulnerabilities and exploits in Python In-Reply-To: <20130221002258.78a805d6@pitrou.net> References: <51239B07.8080300@python.org> <512514CC.2080508@python.org> <5125466D.5090308@python.org> <20130221000808.7a8d2989@pitrou.net> <9AC9F1A0A9BD422999B3AE43AA5C3BA3@gmail.com> <20130221002258.78a805d6@pitrou.net> Message-ID: On Feb 20, 2013, at 6:22 PM, Antoine Pitrou wrote: > On Wed, 20 Feb 2013 18:21:22 -0500 > Donald Stufft wrote: >> On Wednesday, February 20, 2013 at 6:08 PM, Antoine Pitrou wrote: >>>> It's not a distributed DoS issue, it's a severe DoS vulnerabilities. A >>>> single 1 kB XML document can kill virtually any machine, even servers >>>> with more than hundred GB RAM. >>> >>> Assuming an attacker can inject arbitrary XML. Not every XML document >>> is loaded from the Internet. >> >> Even documents not loaded from the internet can be at risk. Often times >> security breaches are the result of a chain of actions. You can say "I'm >> not loading this XML from the internet, so therefore I am safe" but then >> you have another flaw (for example) where you unpack a zip file >> without verifying there are not absolute paths and suddenly your xml file has >> been replaces with a malicious one. > > Assuming your ZIP file is coming from the untrusted Internet, indeed. > Again, this is the same assumption that you are grabbing some important > data from someone you can't trust. > > Just because you are living in a Web-centric world doesn't mean > everyone does. There are a lot of use cases which are not impacted by > your security rules. Bugfix releases shouldn't break those use cases, > which means the security features should be mostly opt-in for 2.7 and > 3.3. > > Regards > > Antoine. Any type of input is a potential attack vector; this isn't web centric, it's a systemic flaw in the spec that allows any application that's loading XML to be bombed into oblivion. People need to trust that the standard library is reliable and sane-by-default. What we have right now isn't > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/jnoller%40gmail.com From pje at telecommunity.com Thu Feb 21 00:48:06 2013 From: pje at telecommunity.com (PJ Eby) Date: Wed, 20 Feb 2013 18:48:06 -0500 Subject: [Python-Dev] [Distutils] PEP 426 is now the draft spec for distribution metadata 2.0 In-Reply-To: References: <512347ED.2040905@egenix.com> <20130219113609.15a421fb@pitrou.net> Message-ID: On Tue, Feb 19, 2013 at 6:42 AM, Nick Coghlan wrote: > Nothing in the PEP is particularly original - almost all of it is > either stolen from other build and packaging systems, or is designed > to provide a *discoverable* alternative to existing > setuptools/distribute/pip practices (specifically, the new extension > mechanism means things like entry points can be defined in the > standard metadata files without python-dev needing to get involved). FWIW, I actually think this is a step in the wrong direction relative to eggs; the ability to add new metadata files is a useful feature for application frameworks. For example, the EggTranslations project uses egg metadata to implement resource localization plugins. It lets you have an application with plugins that either contain their own translations, contain multiple translations for another plugin, a single language translation for an assortment of plugins, etc. These kinds of runtime-discovery use cases haven't seen much attention in the metadata standard discussion. On one level, that's fine, because it makes sense that distribution-provided metadata should be parseable by all tools, and that at build/download/install time the performance and ease-of-use favor a single file approach. That does not mean, however, that the presence of other files is bad or should be deprecated. IMO, metadata that see significant runtime use independent of the core metadata *should* appear in their own files, even if redundant. From donald.stufft at gmail.com Thu Feb 21 00:45:10 2013 From: donald.stufft at gmail.com (Donald Stufft) Date: Wed, 20 Feb 2013 18:45:10 -0500 Subject: [Python-Dev] XML DoS vulnerabilities and exploits in Python In-Reply-To: <20130221002258.78a805d6@pitrou.net> References: <51239B07.8080300@python.org> <512514CC.2080508@python.org> <5125466D.5090308@python.org> <20130221000808.7a8d2989@pitrou.net> <9AC9F1A0A9BD422999B3AE43AA5C3BA3@gmail.com> <20130221002258.78a805d6@pitrou.net> Message-ID: <61507EB6652B465EB8F1B8D69EEA4002@gmail.com> On Wednesday, February 20, 2013 at 6:22 PM, Antoine Pitrou wrote: > On Wed, 20 Feb 2013 18:21:22 -0500 > Donald Stufft wrote: > > On Wednesday, February 20, 2013 at 6:08 PM, Antoine Pitrou wrote: > > > > It's not a distributed DoS issue, it's a severe DoS vulnerabilities. A > > > > single 1 kB XML document can kill virtually any machine, even servers > > > > with more than hundred GB RAM. > > > > > > > > > > > > > Assuming an attacker can inject arbitrary XML. Not every XML document > > > is loaded from the Internet. > > > > > > > > > Even documents not loaded from the internet can be at risk. Often times > > security breaches are the result of a chain of actions. You can say "I'm > > not loading this XML from the internet, so therefore I am safe" but then > > you have another flaw (for example) where you unpack a zip file > > without verifying there are not absolute paths and suddenly your xml file has > > been replaces with a malicious one. > > > > > Assuming your ZIP file is coming from the untrusted Internet, indeed. > Again, this is the same assumption that you are grabbing some important > data from someone you can't trust. > > No software you run on your computer grabs data from someone you don't trust and it all validates that even though you trust them they haven't been exploited? Like I said these sort of things are often caused by chaining several unrelated things together. > > Just because you are living in a Web-centric world doesn't mean > everyone does. There are a lot of use cases which are not impacted by > your security rules. Bugfix releases shouldn't break those use cases, > which means the security features should be mostly opt-in for 2.7 and > 3.3. > > Regards > > Antoine. > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org (mailto:Python-Dev at python.org) > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/donald.stufft%40gmail.com > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From tseaver at palladion.com Thu Feb 21 00:49:59 2013 From: tseaver at palladion.com (Tres Seaver) Date: Wed, 20 Feb 2013 18:49:59 -0500 Subject: [Python-Dev] XML DoS vulnerabilities and exploits in Python In-Reply-To: <20130221002258.78a805d6@pitrou.net> References: <51239B07.8080300@python.org> <512514CC.2080508@python.org> <5125466D.5090308@python.org> <20130221000808.7a8d2989@pitrou.net> <9AC9F1A0A9BD422999B3AE43AA5C3BA3@gmail.com> <20130221002258.78a805d6@pitrou.net> Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 02/20/2013 06:22 PM, Antoine Pitrou wrote: > On Wed, 20 Feb 2013 18:21:22 -0500 Donald Stufft > wrote: >> On Wednesday, February 20, 2013 at 6:08 PM, Antoine Pitrou wrote: >>>> It's not a distributed DoS issue, it's a severe DoS >>>> vulnerabilities. A single 1 kB XML document can kill virtually >>>> any machine, even servers with more than hundred GB RAM. >>>> >>> >>> Assuming an attacker can inject arbitrary XML. Not every XML >>> document is loaded from the Internet. >> >> Even documents not loaded from the internet can be at risk. Often >> times security breaches are the result of a chain of actions. You >> can say "I'm not loading this XML from the internet, so therefore I >> am safe" but then you have another flaw (for example) where you >> unpack a zip file without verifying there are not absolute paths and >> suddenly your xml file has been replaces with a malicious one. > > Assuming your ZIP file is coming from the untrusted Internet, indeed. > Again, this is the same assumption that you are grabbing some > important data from someone you can't trust. > > Just because you are living in a Web-centric world doesn't mean > everyone does. There are a lot of use cases which are not impacted by > your security rules. Bugfix releases shouldn't break those use cases, > which means the security features should be mostly opt-in for 2.7 and > 3.3. Two words: "hash randomization". If it applies to one, it applies to the other. Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with undefined - http://www.enigmail.net/ iEYEARECAAYFAlElYScACgkQ+gerLs4ltQ4QgwCfctL8/FmnboJWozyPcSE1xbb2 wwIAoNVc2hoQci9G2M6g/keNNsN5RR0O =Q9IX -----END PGP SIGNATURE----- From ncoghlan at gmail.com Thu Feb 21 01:31:19 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 21 Feb 2013 10:31:19 +1000 Subject: [Python-Dev] Postponing acceptance of PEP 426 Message-ID: The feedback I have received (both on-list and in response to some off-list queries to specific people) tells me that PEP 426 isn't quite ready for acceptance yet. Things I'll be working on or facilitating over the next few weeks: - documenting an overall transition plan to put the new metadata format in context - clear acceptance criteria in PEP 426, primarily a list of specific projects and individuals that I have requested endorsement from before I am willing to accept the PEP - adding entry-points to PEP 426 - some more adjustments to PEP 426 for improved pkg_resources compatibility - some adjustments to PEP 426 for easier translation to JSON-compatible structured data - PEP for a standardised approach to metadata caching in installation databases - draft PEP for adding distlib to Python 3.4 - draft PEP for automated conversion of PEP 426 metadata files to JSON-compatible structured data This postpone does mean the initial wheel rollout in various projects will be based on metadata 1.2 + the Provides-Extra field (and associated environment marker change), rather than being cleanly based on metadata 2.0. I can live with that - that part of the PEP is *not* open to change, but there are other legitimate concerns that need to be addressed before I declare the PEP accepted. Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From steve at pearwood.info Thu Feb 21 01:37:47 2013 From: steve at pearwood.info (Steven D'Aprano) Date: Thu, 21 Feb 2013 11:37:47 +1100 Subject: [Python-Dev] XML DoS vulnerabilities and exploits in Python In-Reply-To: <20130221002258.78a805d6@pitrou.net> References: <51239B07.8080300@python.org> <512514CC.2080508@python.org> <5125466D.5090308@python.org> <20130221000808.7a8d2989@pitrou.net> <9AC9F1A0A9BD422999B3AE43AA5C3BA3@gmail.com> <20130221002258.78a805d6@pitrou.net> Message-ID: <51256C5B.70906@pearwood.info> On 21/02/13 10:22, Antoine Pitrou wrote: > On Wed, 20 Feb 2013 18:21:22 -0500 > Donald Stufft wrote: >> On Wednesday, February 20, 2013 at 6:08 PM, Antoine Pitrou wrote: >>>> It's not a distributed DoS issue, it's a severe DoS vulnerabilities. A >>>> single 1 kB XML document can kill virtually any machine, even servers >>>> with more than hundred GB RAM. >>>> >>> >>> Assuming an attacker can inject arbitrary XML. Not every XML document >>> is loaded from the Internet. >> >> Even documents not loaded from the internet can be at risk. Often times >> security breaches are the result of a chain of actions. You can say "I'm >> not loading this XML from the internet, so therefore I am safe" but then >> you have another flaw (for example) where you unpack a zip file >> without verifying there are not absolute paths and suddenly your xml file has >> been replaces with a malicious one. > > Assuming your ZIP file is coming from the untrusted Internet, indeed. > Again, this is the same assumption that you are grabbing some important > data from someone you can't trust. It's easy to forget that malware existed long before the Internet. The internet is just a transmission vector, it is not the source of malicious files. The source of malicious files is *other people*, and unless you never use XML files you didn't generate yourself, you cannot completely trust the source. You might trust your colleagues to not *intentionally* pass you a malicious XML file, but they may still do so accidentally. The risk seems small, these days, but remember that for decades the sole transmission vector for viruses and other malware was *people you trusted* not to deliberately give you a virus. > Just because you are living in a Web-centric world doesn't mean > everyone does. There are a lot of use cases which are not impacted by > your security rules. Bugfix releases shouldn't break those use cases, > which means the security features should be mostly opt-in for 2.7 and > 3.3. I think that is reasonable. Insecure by default or not, code should not suddenly stop working because I've upgraded from 2.7.3 to 2.7.4. -- Steven From ncoghlan at gmail.com Thu Feb 21 01:38:07 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 21 Feb 2013 10:38:07 +1000 Subject: [Python-Dev] XML DoS vulnerabilities and exploits in Python In-Reply-To: References: <51239B07.8080300@python.org> <512514CC.2080508@python.org> <5125466D.5090308@python.org> <20130221000808.7a8d2989@pitrou.net> <9AC9F1A0A9BD422999B3AE43AA5C3BA3@gmail.com> <20130221002258.78a805d6@pitrou.net> Message-ID: On Thu, Feb 21, 2013 at 9:49 AM, Tres Seaver wrote: > Two words: "hash randomization". If it applies to one, it applies to > the other. Agreed. Christian's suggested approach sounds sane to me: - make it possible to enable safer behaviour globally in at least 2.7 and 3.3 (and perhaps in 2.6 and 3.2 security releases as well) - make the safer behaviour the default in 3.4 - make it possible to selectively disable the safeguards in all versions A *possible* alternative in to step 1 is loud warnings in the docs directing people to defusedxml, but I prefer the idea of actually making the safeguards available directly in the standard library. Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From vinay_sajip at yahoo.co.uk Thu Feb 21 01:53:59 2013 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Thu, 21 Feb 2013 00:53:59 +0000 (UTC) Subject: [Python-Dev] Fwd: PEP 426 is now the draft spec for distribution metadata 2.0 References: <512347ED.2040905@egenix.com> <51237CEB.3020701@egenix.com> <5123F791.1010201@egenix.com> Message-ID: M.-A. Lemburg egenix.com> writes: > The suggestion to have the metadata available on PyPI doesn't > have anything to do with security. > > It's about being able to determine compatibility and select the > right distribution file for download. The metadata also helps in > creating dependency graphs, which are useful for a lot of things. FYI, distlib already uses metadata in this way. Note that: 1. Dependency information is locked away in setup.py files, e.g in kwargs to the setup() call. 2. PyPI does not provide access to this metadata. So I have had to write code to extract what information I could out of setup.py setup() and into metadata that can be used. I have, for now, hosted it on my own server, but it can be used for e.g. determining complete dependency graphs without ever actually downloading any distributions. Distlib provides access to it: $ python finddeps.py pyramid 12 dists, elapsed time: 4.35 secs ---------------------------------------- Download order: ---------------------------------------- download setuptools (0.6c11) [for pyramid, zope.interface, zope.deprecation] download PasteDeploy (1.5.0) [for pyramid] download repoze.lru (0.6) [for pyramid] download venusian (1.0a7) [for pyramid] download MarkupSafe (0.15) [for Mako] download translationstring (1.1) [for pyramid] download Chameleon (2.11) [for pyramid] download WebOb (1.2.3) [for pyramid] download zope.interface (4.0.3) [for pyramid] download Mako (0.7.3) [for pyramid] download zope.deprecation (4.0.2) [for pyramid] download pyramid (1.4) In the above, nothing was actually downloaded except the metadata. It's just indicating an intuitive ordering for download based on a topological sort of the dependency graph - for pyramid, in this case. Regards, Vinay Sajip From vinay_sajip at yahoo.co.uk Thu Feb 21 02:04:31 2013 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Thu, 21 Feb 2013 01:04:31 +0000 (UTC) Subject: [Python-Dev] Fwd: PEP 426 is now the draft spec for distribution metadata 2.0 References: <512347ED.2040905@egenix.com> Message-ID: Chris Jerdonek gmail.com> writes: > Maybe this is already stated somewhere, but is there a plan for when > distlib will be brought into the repository? Is there a reason not to > do it now? It seems it would have more visibility that way (e.g. > people could see it as part of the development version of the online > docs and it would be in check-ins as are PEP edits), and its status > relative to Python would be clearer. There's no firm plan other than it being on the 3.4 wishlist, but IMO there's no rush to do it right now. AFAICT interested parties can be in the loop even though the project is actively developed on BitBucket rather than hg.python.org: remember that the current stakeholders in terms of packaging tools are not all in python-dev and may find it easier to contribute on BitBucket. (I often get comments from Daniel Holth in response to distlib commits I push to BitBucket.) I periodically sync the BitBucket repo with hg.python.org/distlib. The docs are also readily accessible on readthedocs.org (the first two Google results for distlib are those docs and the project on BitBucket). While not integrated with docs.python.org, I don't think that the distlib docs suffer inordinately from being on readthedocs.org - for one thing, I've enabled Disqus comments, allowing feedback on the docs - not that I've had any ;-) Closer integration should perhaps happen, but not until the project has matured a little. Regards, Vinay Sajip From vinay_sajip at yahoo.co.uk Thu Feb 21 02:14:21 2013 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Thu, 21 Feb 2013 01:14:21 +0000 (UTC) Subject: [Python-Dev] Fwd: PEP 426 is now the draft spec for distribution metadata 2.0 References: <512347ED.2040905@egenix.com> <51237CEB.3020701@egenix.com> <5123EFEA.70600@egenix.com> <5123F849.6080703@egenix.com> <6ECA090456F34E3CB3E0974093CB9E4C@gmail.com> Message-ID: Paul Moore gmail.com> writes: > Understood - that's why I suggested that distlib reach a point where > it's stable as an external package and supported on (some) older > versions. I'm hoping for an experience more like unittest2 than > packaging/distutils2. Currently, distlib runs on Python 2.6, 2.7 and 3.2+, so there shouldn't be any concerns about support on those older Python versions. Regards, Vinay Sajip From fred at fdrake.net Thu Feb 21 03:07:36 2013 From: fred at fdrake.net (Fred Drake) Date: Wed, 20 Feb 2013 21:07:36 -0500 Subject: [Python-Dev] XML DoS vulnerabilities and exploits in Python In-Reply-To: References: <51239B07.8080300@python.org> <512514CC.2080508@python.org> <5125466D.5090308@python.org> <20130221000808.7a8d2989@pitrou.net> <9AC9F1A0A9BD422999B3AE43AA5C3BA3@gmail.com> <20130221002258.78a805d6@pitrou.net> Message-ID: On Wed, Feb 20, 2013 at 7:38 PM, Nick Coghlan wrote: > Christian's suggested approach sounds sane to me: Definitely. A strong +1 from me, FWIW these days. -Fred -- Fred L. Drake, Jr. "A storm broke loose in my mind." --Albert Einstein From barry at python.org Thu Feb 21 03:08:42 2013 From: barry at python.org (Barry Warsaw) Date: Wed, 20 Feb 2013 21:08:42 -0500 Subject: [Python-Dev] XML DoS vulnerabilities and exploits in Python In-Reply-To: References: <51239B07.8080300@python.org> <512514CC.2080508@python.org> <5125466D.5090308@python.org> <20130221000808.7a8d2989@pitrou.net> <9AC9F1A0A9BD422999B3AE43AA5C3BA3@gmail.com> <20130221002258.78a805d6@pitrou.net> Message-ID: <20130220210842.7dd9d84d@anarchist.wooz.org> On Feb 21, 2013, at 10:38 AM, Nick Coghlan wrote: >- make it possible to enable safer behaviour globally in at least 2.7 >and 3.3 (and perhaps in 2.6 and 3.2 security releases as well) I want to be fairly conservative with 2.6.9. -Barry From tseaver at palladion.com Thu Feb 21 05:35:03 2013 From: tseaver at palladion.com (Tres Seaver) Date: Wed, 20 Feb 2013 23:35:03 -0500 Subject: [Python-Dev] XML DoS vulnerabilities and exploits in Python In-Reply-To: <20130220210842.7dd9d84d@anarchist.wooz.org> References: <51239B07.8080300@python.org> <512514CC.2080508@python.org> <5125466D.5090308@python.org> <20130221000808.7a8d2989@pitrou.net> <9AC9F1A0A9BD422999B3AE43AA5C3BA3@gmail.com> <20130221002258.78a805d6@pitrou.net> <20130220210842.7dd9d84d@anarchist.wooz.org> Message-ID: <5125A3F7.7030308@palladion.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 02/20/2013 09:08 PM, Barry Warsaw wrote: > On Feb 21, 2013, at 10:38 AM, Nick Coghlan wrote: > >> - make it possible to enable safer behaviour globally in at least >> 2.7 and 3.3 (and perhaps in 2.6 and 3.2 security releases as well) > > I want to be fairly conservative with 2.6.9. I believe that the same rationale should apply as that for adding hash randomization in 2.6.8: this is at least as bad a vulnerability, with many more vectors of attack. Tres - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with undefined - http://www.enigmail.net/ iEYEARECAAYFAlElo/cACgkQ+gerLs4ltQ4urQCg2Kyr6CKZPp35fAK1G4OtzYc+ XD8An0fJZw5DHRxg1JPe9AzcLqpvRZc5 =hmpM -----END PGP SIGNATURE----- From barry at python.org Thu Feb 21 05:53:50 2013 From: barry at python.org (Barry Warsaw) Date: Wed, 20 Feb 2013 23:53:50 -0500 Subject: [Python-Dev] XML DoS vulnerabilities and exploits in Python References: <51239B07.8080300@python.org> <512514CC.2080508@python.org> <5125466D.5090308@python.org> <20130221000808.7a8d2989@pitrou.net> <9AC9F1A0A9BD422999B3AE43AA5C3BA3@gmail.com> <20130221002258.78a805d6@pitrou.net> <20130220210842.7dd9d84d@anarchist.wooz.org> <5125A3F7.7030308@palladion.com> Message-ID: <20130220235350.3a5d60e6@anarchist.wooz.org> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 On Feb 20, 2013, at 11:35 PM, Tres Seaver wrote: >I believe that the same rationale should apply as that for adding hash >randomization in 2.6.8: this is at least as bad a vulnerability, with >many more vectors of attack. Except that I really want to EOL 2.6 in October as per schedule, and I really don't want a 2.6.10. So if we get the API changes wrong in 2.6.9 there won't be much of an opportunity to correct it. Also, in 2.6, hash randomization is opt-in so the default didn't change. Cheers, - -Barry -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.12 (GNU/Linux) iQIcBAEBCAAGBQJRJaheAAoJEBJutWOnSwa/A2sQAK0HnPQCG87vEVj7hpfp1h7o 8mVDpf8VyZtfVIhwJLmb106DCe3mWb+UZLf4ks16zSQfy1JVxNl179VESqDgOD30 RTC0/esArDzpVbCOKf0mepyYflQEnrA0FR/URAJVoqjGDlPSPr6vneWX2fOPGwn7 VsgzHzyqHs5k+JLTn9piDKLfW6LXWQOYl+oPF/T60SnYZTs8y6E00n9i2kLPaWy+ UMSnAC3jQMtgfhzPnXgPrlzVh4/ojYYnBVdhZYKGroBx4JGt7UDph1koFck7CRTD yJuVQ4QzsbtEF0/SyJVORWTh2N3jxEA7qWFIDQ8kif+Tqqfz/bujSud8SEK7dLuZ IPinvL2K9lD722RJjhmcMyfsrCJxafF2YbkdWyNr2zMlx6FO/oQWCbDVPBaZlu8+ cZyzF+JJiIM+ljlqL9dy5w156JxfQ5eh5h5ocZ/OiUisiKY93zT9j2hOM82bbo+/ QIWmMf2wPl277awbvw3+9GDi4xdgK0GJB/4BVOYBXy0/b2q2n5oHyHACJCoUtDyv AVCT+Hfw5Nu1ZIm3AU329gFeBPjEvl8+YsbiQf4hqsp40fnY7GzhRFb8HTwCZVmu amOWyeTg9f7dtF/sOwEpc/UGcCjJdBklrbndpNZ0f9gZF+FwncxbJThrTztlxaDY 7eT2EYRJsnLW5XiZTilj =HXz8 -----END PGP SIGNATURE----- From stefan_ml at behnel.de Thu Feb 21 07:37:36 2013 From: stefan_ml at behnel.de (Stefan Behnel) Date: Thu, 21 Feb 2013 07:37:36 +0100 Subject: [Python-Dev] XML DoS vulnerabilities and exploits in Python In-Reply-To: References: <51239B07.8080300@python.org> <512514CC.2080508@python.org> Message-ID: Maciej Fijalkowski, 20.02.2013 21:17: > On Wed, Feb 20, 2013 at 8:24 PM, Christian Heimes wrote: >> Am 20.02.2013 17:25, schrieb Benjamin Peterson: >>> Are these going to become patches for Python, too? >> >> I'm working on it. The patches need to be discussed as they break >> backward compatibility and AFAIK XML standards, too. > > That's not very good. XML parsers are supposed to parse XML according > to standards. I think we can shorten this discussion to "this is a serious problem that needs to be fixed". If that involves taking the freedom that the XML standard leaves about processing DTDs, then I think we shouldn't be throwing any high-level bike shedding at it. Consulting the standard actually helps. Stefan From solipsis at pitrou.net Thu Feb 21 07:42:11 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 21 Feb 2013 07:42:11 +0100 Subject: [Python-Dev] XML DoS vulnerabilities and exploits in Python In-Reply-To: <61507EB6652B465EB8F1B8D69EEA4002@gmail.com> References: <51239B07.8080300@python.org> <512514CC.2080508@python.org> <5125466D.5090308@python.org> <20130221000808.7a8d2989@pitrou.net> <9AC9F1A0A9BD422999B3AE43AA5C3BA3@gmail.com> <20130221002258.78a805d6@pitrou.net> <61507EB6652B465EB8F1B8D69EEA4002@gmail.com> Message-ID: <20130221074211.7cbadcff@pitrou.net> On Wed, 20 Feb 2013 18:45:10 -0500 Donald Stufft wrote: > > No software you run on your computer grabs data from someone you don't trust > and it all validates that even though you trust them they haven't been exploited? What the hell do you mean exactly? There are other reasons to validate data than just "security". > Like I said these sort of things are often caused by chaining several unrelated > things together. At this points this sounds like fear-mongering. Regards Antoine. From solipsis at pitrou.net Thu Feb 21 07:53:12 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 21 Feb 2013 07:53:12 +0100 Subject: [Python-Dev] XML DoS vulnerabilities and exploits in Python References: <51239B07.8080300@python.org> <512514CC.2080508@python.org> <5125466D.5090308@python.org> <20130221000808.7a8d2989@pitrou.net> <9AC9F1A0A9BD422999B3AE43AA5C3BA3@gmail.com> <20130221002258.78a805d6@pitrou.net> <51256C5B.70906@pearwood.info> Message-ID: <20130221075312.4485ad58@pitrou.net> On Thu, 21 Feb 2013 11:37:47 +1100 Steven D'Aprano wrote: > > It's easy to forget that malware existed long before the Internet. The internet is just a transmission vector, it is not the source of malicious files. The source of malicious files is *other people*, and unless you never use XML files you didn't generate yourself, you cannot completely trust the source. You might trust your colleagues to not *intentionally* pass you a malicious XML file, but they may still do so accidentally. That's in theory very nice, but in practice security in everyday computing hasn't really been a concern before the massification of Internet access. (yes, there have been viruses on mainstream platforms such as the Amiga, but it was pretty minor compared to nowadays, and nobody cared about potential DoS attacks for example) So, as for XML files, we are talking about a DoS vulnerability. It will take more than a single file to make a DoS attack really annoying, which means the attacker must pollute the source of those XML files in a systemic way. It's not "a single XML file will smuggle confidential data out of the building". Regards Antoine. From solipsis at pitrou.net Thu Feb 21 07:56:11 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 21 Feb 2013 07:56:11 +0100 Subject: [Python-Dev] XML DoS vulnerabilities and exploits in Python References: <51239B07.8080300@python.org> <512514CC.2080508@python.org> <5125466D.5090308@python.org> <20130221000808.7a8d2989@pitrou.net> <9AC9F1A0A9BD422999B3AE43AA5C3BA3@gmail.com> <20130221002258.78a805d6@pitrou.net> Message-ID: <20130221075611.35bedd2b@pitrou.net> On Thu, 21 Feb 2013 10:38:07 +1000 Nick Coghlan wrote: > On Thu, Feb 21, 2013 at 9:49 AM, Tres Seaver wrote: > > Two words: "hash randomization". If it applies to one, it applies to > > the other. > > Agreed. Christian's suggested approach sounds sane to me: > > - make it possible to enable safer behaviour globally in at least 2.7 > and 3.3 (and perhaps in 2.6 and 3.2 security releases as well) > - make the safer behaviour the default in 3.4 > - make it possible to selectively disable the safeguards in all versions +1 from me. Regards Antoine. From tseaver at palladion.com Thu Feb 21 08:29:08 2013 From: tseaver at palladion.com (Tres Seaver) Date: Thu, 21 Feb 2013 02:29:08 -0500 Subject: [Python-Dev] XML DoS vulnerabilities and exploits in Python In-Reply-To: <20130221075312.4485ad58@pitrou.net> References: <51239B07.8080300@python.org> <512514CC.2080508@python.org> <5125466D.5090308@python.org> <20130221000808.7a8d2989@pitrou.net> <9AC9F1A0A9BD422999B3AE43AA5C3BA3@gmail.com> <20130221002258.78a805d6@pitrou.net> <51256C5B.70906@pearwood.info> <20130221075312.4485ad58@pitrou.net> Message-ID: <5125CCC4.3080501@palladion.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 02/21/2013 01:53 AM, Antoine Pitrou wrote: > On Thu, 21 Feb 2013 11:37:47 +1100 Steven D'Aprano > wrote: >> >> It's easy to forget that malware existed long before the Internet. >> The internet is just a transmission vector, it is not the source of >> malicious files. The source of malicious files is *other people*, >> and unless you never use XML files you didn't generate yourself, you >> cannot completely trust the source. You might trust your colleagues >> to not *intentionally* pass you a malicious XML file, but they may >> still do so accidentally. > > That's in theory very nice, but in practice security in everyday > computing hasn't really been a concern before the massification of > Internet access. > > (yes, there have been viruses on mainstream platforms such as the > Amiga, but it was pretty minor compared to nowadays, and nobody cared > about potential DoS attacks for example) > > So, as for XML files, we are talking about a DoS vulnerability. It > will take more than a single file to make a DoS attack really > annoying, which means the attacker must pollute the source of those > XML files in a systemic way. It's not "a single XML file will smuggle > confidential data out of the building". Antoine, A single, small,, malicious XML file can kill a machine (not just the process parsing it) by sucking all available RAM. We are talking hard lockup, reboot-to-fix-it sorts of DOC here. Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with undefined - http://www.enigmail.net/ iEYEARECAAYFAlElzMQACgkQ+gerLs4ltQ7fDQCgmvvurNi5VtWA+4mqcz4tpUhR rNUAnRtpcKMFCM3z8qRKNfinAE9ly9fX =y+eM -----END PGP SIGNATURE----- From solipsis at pitrou.net Thu Feb 21 08:42:46 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 21 Feb 2013 08:42:46 +0100 Subject: [Python-Dev] XML DoS vulnerabilities and exploits in Python References: <51239B07.8080300@python.org> <512514CC.2080508@python.org> <5125466D.5090308@python.org> <20130221000808.7a8d2989@pitrou.net> <9AC9F1A0A9BD422999B3AE43AA5C3BA3@gmail.com> <20130221002258.78a805d6@pitrou.net> <51256C5B.70906@pearwood.info> <20130221075312.4485ad58@pitrou.net> <5125CCC4.3080501@palladion.com> Message-ID: <20130221084246.579fd260@pitrou.net> On Thu, 21 Feb 2013 02:29:08 -0500 Tres Seaver wrote: > > Antoine, > > A single, small,, malicious XML file can kill a machine (not just the > process parsing it) by sucking all available RAM. We are talking hard > lockup, reboot-to-fix-it sorts of DOC here. Sure, but in many instances, rebooting a machine is not business-threatening. You will have a couple of minutes' downtime and that's all. Which is why the attack must be repeated many times to be a major annoyance. Regards Antoine. From solipsis at pitrou.net Thu Feb 21 10:23:34 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 21 Feb 2013 10:23:34 +0100 Subject: [Python-Dev] XML DoS vulnerabilities and exploits in Python References: <51239B07.8080300@python.org> <512514CC.2080508@python.org> <5125466D.5090308@python.org> <20130221000808.7a8d2989@pitrou.net> <51255CB0.8060706@python.org> Message-ID: <20130221102334.1cc654f3@pitrou.net> Le Thu, 21 Feb 2013 00:30:56 +0100, Christian Heimes a ?crit : > Am 21.02.2013 00:08, schrieb Antoine Pitrou: > > Not everyone is a security nuts. > > But, but, but ... it's fun to be paranoid! You get so many new > potential enemies. :) If you like being paranoid, there are other things than security to be paranoid about: reference cycles, performance on micro-benchmarks, memory consumption of docstrings, etc. :-) Regards Antoine. From christian at python.org Thu Feb 21 10:54:02 2013 From: christian at python.org (Christian Heimes) Date: Thu, 21 Feb 2013 10:54:02 +0100 Subject: [Python-Dev] XML DoS vulnerabilities and exploits in Python In-Reply-To: <20130221102334.1cc654f3@pitrou.net> References: <51239B07.8080300@python.org> <512514CC.2080508@python.org> <5125466D.5090308@python.org> <20130221000808.7a8d2989@pitrou.net> <51255CB0.8060706@python.org> <20130221102334.1cc654f3@pitrou.net> Message-ID: Am 21.02.2013 10:23, schrieb Antoine Pitrou: > If you like being paranoid, there are other things than security to > be paranoid about: reference cycles, performance on micro-benchmarks, > memory consumption of docstrings, etc. :-) snappy(__doc__)? http://code.google.com/p/snappy/ Christian From christian at python.org Thu Feb 21 11:18:35 2013 From: christian at python.org (Christian Heimes) Date: Thu, 21 Feb 2013 11:18:35 +0100 Subject: [Python-Dev] XML DoS vulnerabilities and exploits in Python In-Reply-To: <20130221084246.579fd260@pitrou.net> References: <51239B07.8080300@python.org> <512514CC.2080508@python.org> <5125466D.5090308@python.org> <20130221000808.7a8d2989@pitrou.net> <9AC9F1A0A9BD422999B3AE43AA5C3BA3@gmail.com> <20130221002258.78a805d6@pitrou.net> <51256C5B.70906@pearwood.info> <20130221075312.4485ad58@pitrou.net> <5125CCC4.3080501@palladion.com> <20130221084246.579fd260@pitrou.net> Message-ID: <5125F47B.3010607@python.org> Am 21.02.2013 08:42, schrieb Antoine Pitrou: > Sure, but in many instances, rebooting a machine is not > business-threatening. You will have a couple of minutes' downtime and > that's all. Which is why the attack must be repeated many times to be a > major annoyance. Is this business-threatening enough? https://pypi.python.org/pypi/defusedxml#external-entity-expansion-remote * An attacker can circumvent firewalls and gain access to restricted resources as all the requests are made from an internal and trustworthy IP address, not from the outside. * An attacker can abuse a service to attack, spy on or DoS your servers but also third party services. The attack is disguised with the IP address of the server and the attacker is able to utilize the high bandwidth of a big machine. * An attacker can exhaust additional resources on the machine, e.g. with requests to a service that doesn't respond or responds with very large files. * An attacker may gain knowledge, when, how often and from which IP address a XML document is accessed. * An attacker could send mail from inside your network if the URL handler supports smtp:// URIs. From solipsis at pitrou.net Thu Feb 21 11:32:53 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 21 Feb 2013 11:32:53 +0100 Subject: [Python-Dev] XML DoS vulnerabilities and exploits in Python References: <51239B07.8080300@python.org> <512514CC.2080508@python.org> <5125466D.5090308@python.org> <20130221000808.7a8d2989@pitrou.net> <9AC9F1A0A9BD422999B3AE43AA5C3BA3@gmail.com> <20130221002258.78a805d6@pitrou.net> <51256C5B.70906@pearwood.info> <20130221075312.4485ad58@pitrou.net> <5125CCC4.3080501@palladion.com> <20130221084246.579fd260@pitrou.net> <5125F47B.3010607@python.org> Message-ID: <20130221113253.20e8b28f@pitrou.net> Le Thu, 21 Feb 2013 11:18:35 +0100, Christian Heimes a ?crit : > Am 21.02.2013 08:42, schrieb Antoine Pitrou: > > Sure, but in many instances, rebooting a machine is not > > business-threatening. You will have a couple of minutes' downtime > > and that's all. Which is why the attack must be repeated many times > > to be a major annoyance. > > Is this business-threatening enough? > > https://pypi.python.org/pypi/defusedxml#external-entity-expansion-remote You haven't proved that these were actual threats, nor how they actually worked. I'm gonna remain skeptical if there isn't anything more precise than "It highly depends on the parser and the application what kind of exploit is possible". Regards Antoine. From jnoller at gmail.com Thu Feb 21 12:05:52 2013 From: jnoller at gmail.com (Jesse Noller) Date: Thu, 21 Feb 2013 06:05:52 -0500 Subject: [Python-Dev] XML DoS vulnerabilities and exploits in Python In-Reply-To: <20130221113253.20e8b28f@pitrou.net> References: <51239B07.8080300@python.org> <512514CC.2080508@python.org> <5125466D.5090308@python.org> <20130221000808.7a8d2989@pitrou.net> <9AC9F1A0A9BD422999B3AE43AA5C3BA3@gmail.com> <20130221002258.78a805d6@pitrou.net> <51256C5B.70906@pearwood.info> <20130221075312.4485ad58@pitrou.net> <5125CCC4.3080501@palladion.com> <20130221084246.579fd260@pitrou.net> <5125F47B.3010607@python.org> <20130221113253.20e8b28f@pitrou.net> Message-ID: <74957F62-DF1C-46BC-9705-1494477CD217@gmail.com> On Feb 21, 2013, at 5:32 AM, Antoine Pitrou wrote: > Le Thu, 21 Feb 2013 11:18:35 +0100, > Christian Heimes a ?crit : >> Am 21.02.2013 08:42, schrieb Antoine Pitrou: >>> Sure, but in many instances, rebooting a machine is not >>> business-threatening. You will have a couple of minutes' downtime >>> and that's all. Which is why the attack must be repeated many times >>> to be a major annoyance. >> >> Is this business-threatening enough? >> >> https://pypi.python.org/pypi/defusedxml#external-entity-expansion-remote > > You haven't proved that these were actual threats, nor how they > actually worked. I'm gonna remain skeptical if there isn't anything > more precise than "It highly depends on the parser and the application > what kind of exploit is possible". > > Regards > > Antoine. > I guess someone need to write a proof of concept exploit for you and release it into the wild. Ok > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/jnoller%40gmail.com From solipsis at pitrou.net Thu Feb 21 12:16:41 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 21 Feb 2013 12:16:41 +0100 Subject: [Python-Dev] XML DoS vulnerabilities and exploits in Python References: <51239B07.8080300@python.org> <512514CC.2080508@python.org> <5125466D.5090308@python.org> <20130221000808.7a8d2989@pitrou.net> <9AC9F1A0A9BD422999B3AE43AA5C3BA3@gmail.com> <20130221002258.78a805d6@pitrou.net> <51256C5B.70906@pearwood.info> <20130221075312.4485ad58@pitrou.net> <5125CCC4.3080501@palladion.com> <20130221084246.579fd260@pitrou.net> <5125F47B.3010607@python.org> <20130221113253.20e8b28f@pitrou.net> <74957F62-DF1C-46BC-9705-1494477CD217@gmail.com> Message-ID: <20130221121641.6fc07bf3@pitrou.net> Le Thu, 21 Feb 2013 06:05:52 -0500, Jesse Noller a ?crit : > On Feb 21, 2013, at 5:32 AM, Antoine Pitrou > wrote: > > > Le Thu, 21 Feb 2013 11:18:35 +0100, > > Christian Heimes a ?crit : > >> Am 21.02.2013 08:42, schrieb Antoine Pitrou: > >>> Sure, but in many instances, rebooting a machine is not > >>> business-threatening. You will have a couple of minutes' downtime > >>> and that's all. Which is why the attack must be repeated many > >>> times to be a major annoyance. > >> > >> Is this business-threatening enough? > >> > >> https://pypi.python.org/pypi/defusedxml#external-entity-expansion-remote > > > > You haven't proved that these were actual threats, nor how they > > actually worked. I'm gonna remain skeptical if there isn't anything > > more precise than "It highly depends on the parser and the > > application what kind of exploit is possible". > > > > Regards > > > > Antoine. > > > > I guess someone need to write a proof of concept exploit for you and > release it into the wild. I don't know whether you are trying to be ironic but, for the record, proof of concepts needn't be "released into the wild" as long as they exist. Regards Antoine. From christian at python.org Thu Feb 21 13:04:59 2013 From: christian at python.org (Christian Heimes) Date: Thu, 21 Feb 2013 13:04:59 +0100 Subject: [Python-Dev] XML DoS vulnerabilities and exploits in Python In-Reply-To: <20130221113253.20e8b28f@pitrou.net> References: <51239B07.8080300@python.org> <512514CC.2080508@python.org> <5125466D.5090308@python.org> <20130221000808.7a8d2989@pitrou.net> <9AC9F1A0A9BD422999B3AE43AA5C3BA3@gmail.com> <20130221002258.78a805d6@pitrou.net> <51256C5B.70906@pearwood.info> <20130221075312.4485ad58@pitrou.net> <5125CCC4.3080501@palladion.com> <20130221084246.579fd260@pitrou.net> <5125F47B.3010607@python.org> <20130221113253.20e8b28f@pitrou.net> Message-ID: <51260D6B.1050101@python.org> Am 21.02.2013 11:32, schrieb Antoine Pitrou: > You haven't proved that these were actual threats, nor how they > actually worked. I'm gonna remain skeptical if there isn't anything > more precise than "It highly depends on the parser and the application > what kind of exploit is possible". https://bitbucket.org/tiran/defusedxml/src/82f4037464418bf11ea734969b7ca1c193e6ed91/other/python-external.py?at=default $ ./python-external.py REQUEST: -------- Aachen RESPONSE: --------- The weather in Aachen is terrible. ]> &passwd; RESPONSE: --------- Unknown city root:x:0:0:root:/root:/bin/bash daemon:x:1:1:daemon:/usr/sbin:/bin/sh bin:x:2:2:bin:/bin:/bin/sh sys:x:3:3:sys:/dev:/bin/sh sync:x:4:65534:sync:/bin:/bin/sync games:x:5:60:games:/usr/games:/bin/sh man:x:6:12:man:/var/cache/man:/bin/sh lp:x:7:7:lp:/var/spool/lpd:/bin/sh mail:x:8:8:mail:/var/mail:/bin/sh news:x:9:9:news:/var/spool/news:/bin/sh uucp:x:10:10:uucp:/var/spool/uucp:/bin/sh proxy:x:13:13:proxy:/bin:/bin/sh www-data:x:33:33:www-data:/var/www:/bin/sh backup:x:34:34:backup:/var/backups:/bi REQUEST: -------- ]> &url; RESPONSE: --------- Unknown city -----BEGIN DH PARAMETERS----- MEYCQQD1Kv884bEpQBgRjXyEpwpy1obEAxnIByl6ypUM2Zafq9AKUJsCRtMIPWak XUGfnHy9iUsiGSa6q6Jew1XpKgVfAgEC -----END DH PARAMETERS----- These are the 512 bit DH parameters from "Assigned Number for SKIP Protocols" (http://www.skip-vpn.org/spec/numbers.html). See there for how they were generated. Note that g is not a generator, but this is not a problem since p is a safe prime. Q.E.D. Christian From christian at python.org Thu Feb 21 13:19:54 2013 From: christian at python.org (Christian Heimes) Date: Thu, 21 Feb 2013 13:19:54 +0100 Subject: [Python-Dev] XML DoS vulnerabilities and exploits in Python In-Reply-To: <20130221121641.6fc07bf3@pitrou.net> References: <51239B07.8080300@python.org> <512514CC.2080508@python.org> <5125466D.5090308@python.org> <20130221000808.7a8d2989@pitrou.net> <9AC9F1A0A9BD422999B3AE43AA5C3BA3@gmail.com> <20130221002258.78a805d6@pitrou.net> <51256C5B.70906@pearwood.info> <20130221075312.4485ad58@pitrou.net> <5125CCC4.3080501@palladion.com> <20130221084246.579fd260@pitrou.net> <5125F47B.3010607@python.org> <20130221113253.20e8b28f@pitrou.net> <74957F62-DF1C-46BC-9705-1494477CD217@gmail.com> <20130221121641.6fc07bf3@pitrou.net> Message-ID: <512610EA.3040408@python.org> Am 21.02.2013 12:16, schrieb Antoine Pitrou: > I don't know whether you are trying to be ironic but, for the record, > proof of concepts needn't be "released into the wild" as long as they > exist. Fun fact: In fact the abbreviation 'ap' doesn't stand for 'Antoine Pitrou' but for 'antipole'. I'm a bit paranoid and overcautious. Antoine acts as my antipole and counter balance. Together we make a fairly good team. :) Christian From solipsis at pitrou.net Thu Feb 21 15:03:56 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 21 Feb 2013 15:03:56 +0100 Subject: [Python-Dev] XML DoS vulnerabilities and exploits in Python References: <51239B07.8080300@python.org> <512514CC.2080508@python.org> <5125466D.5090308@python.org> <20130221000808.7a8d2989@pitrou.net> <9AC9F1A0A9BD422999B3AE43AA5C3BA3@gmail.com> <20130221002258.78a805d6@pitrou.net> <51256C5B.70906@pearwood.info> <20130221075312.4485ad58@pitrou.net> <5125CCC4.3080501@palladion.com> <20130221084246.579fd260@pitrou.net> <5125F47B.3010607@python.org> <20130221113253.20e8b28f@pitrou.net> <74957F62-DF1C-46BC-9705-1494477CD217@gmail.com> <20130221121641.6fc07bf3@pitrou.net> <512610EA.3040408@python.org> Message-ID: <20130221150356.26dc30c5@pitrou.net> Le Thu, 21 Feb 2013 13:19:54 +0100, Christian Heimes a ?crit : > Am 21.02.2013 12:16, schrieb Antoine Pitrou: > > I don't know whether you are trying to be ironic but, for the > > record, proof of concepts needn't be "released into the wild" as > > long as they exist. > > Fun fact: > > In fact the abbreviation 'ap' doesn't stand for 'Antoine Pitrou' but > for 'antipole'. I'm a bit paranoid and overcautious. Antoine acts as > my antipole and counter balance. Together we make a fairly good > team. :) Not really, since I'm overcautious towards paranoid people. Regards Antoine. From solipsis at pitrou.net Thu Feb 21 15:03:10 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 21 Feb 2013 15:03:10 +0100 Subject: [Python-Dev] XML DoS vulnerabilities and exploits in Python References: <51239B07.8080300@python.org> <512514CC.2080508@python.org> <5125466D.5090308@python.org> <20130221000808.7a8d2989@pitrou.net> <9AC9F1A0A9BD422999B3AE43AA5C3BA3@gmail.com> <20130221002258.78a805d6@pitrou.net> <51256C5B.70906@pearwood.info> <20130221075312.4485ad58@pitrou.net> <5125CCC4.3080501@palladion.com> <20130221084246.579fd260@pitrou.net> <5125F47B.3010607@python.org> <20130221113253.20e8b28f@pitrou.net> <51260D6B.1050101@python.org> Message-ID: <20130221150310.36e16489@pitrou.net> Le Thu, 21 Feb 2013 13:04:59 +0100, Christian Heimes a ?crit : > Am 21.02.2013 11:32, schrieb Antoine Pitrou: > > You haven't proved that these were actual threats, nor how they > > actually worked. I'm gonna remain skeptical if there isn't anything > > more precise than "It highly depends on the parser and the > > application what kind of exploit is possible". > > https://bitbucket.org/tiran/defusedxml/src/82f4037464418bf11ea734969b7ca1c193e6ed91/other/python-external.py?at=default > > $ ./python-external.py [snip] Again, this requires that your attacker can directly feed XML to the system *and* read the response. Not every computer is a public Internet server. Regards Antoine. From fijall at gmail.com Thu Feb 21 17:00:02 2013 From: fijall at gmail.com (Maciej Fijalkowski) Date: Thu, 21 Feb 2013 18:00:02 +0200 Subject: [Python-Dev] XML DoS vulnerabilities and exploits in Python In-Reply-To: <5125A3F7.7030308@palladion.com> References: <51239B07.8080300@python.org> <512514CC.2080508@python.org> <5125466D.5090308@python.org> <20130221000808.7a8d2989@pitrou.net> <9AC9F1A0A9BD422999B3AE43AA5C3BA3@gmail.com> <20130221002258.78a805d6@pitrou.net> <20130220210842.7dd9d84d@anarchist.wooz.org> <5125A3F7.7030308@palladion.com> Message-ID: On Thu, Feb 21, 2013 at 6:35 AM, Tres Seaver wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > On 02/20/2013 09:08 PM, Barry Warsaw wrote: >> On Feb 21, 2013, at 10:38 AM, Nick Coghlan wrote: >> >>> - make it possible to enable safer behaviour globally in at least >>> 2.7 and 3.3 (and perhaps in 2.6 and 3.2 security releases as well) >> >> I want to be fairly conservative with 2.6.9. > > I believe that the same rationale should apply as that for adding hash > randomization in 2.6.8: this is at least as bad a vulnerability, with > many more vectors of attack. FYI the hash randomization is broken (it only allows 256 really different hashes) From fijall at gmail.com Thu Feb 21 17:02:07 2013 From: fijall at gmail.com (Maciej Fijalkowski) Date: Thu, 21 Feb 2013 18:02:07 +0200 Subject: [Python-Dev] XML DoS vulnerabilities and exploits in Python In-Reply-To: <5125CCC4.3080501@palladion.com> References: <51239B07.8080300@python.org> <512514CC.2080508@python.org> <5125466D.5090308@python.org> <20130221000808.7a8d2989@pitrou.net> <9AC9F1A0A9BD422999B3AE43AA5C3BA3@gmail.com> <20130221002258.78a805d6@pitrou.net> <51256C5B.70906@pearwood.info> <20130221075312.4485ad58@pitrou.net> <5125CCC4.3080501@palladion.com> Message-ID: On Thu, Feb 21, 2013 at 9:29 AM, Tres Seaver wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > On 02/21/2013 01:53 AM, Antoine Pitrou wrote: >> On Thu, 21 Feb 2013 11:37:47 +1100 Steven D'Aprano >> wrote: >>> >>> It's easy to forget that malware existed long before the Internet. >>> The internet is just a transmission vector, it is not the source of >>> malicious files. The source of malicious files is *other people*, >>> and unless you never use XML files you didn't generate yourself, you >>> cannot completely trust the source. You might trust your colleagues >>> to not *intentionally* pass you a malicious XML file, but they may >>> still do so accidentally. >> >> That's in theory very nice, but in practice security in everyday >> computing hasn't really been a concern before the massification of >> Internet access. >> >> (yes, there have been viruses on mainstream platforms such as the >> Amiga, but it was pretty minor compared to nowadays, and nobody cared >> about potential DoS attacks for example) >> >> So, as for XML files, we are talking about a DoS vulnerability. It >> will take more than a single file to make a DoS attack really >> annoying, which means the attacker must pollute the source of those >> XML files in a systemic way. It's not "a single XML file will smuggle >> confidential data out of the building". > > Antoine, > > A single, small,, malicious XML file can kill a machine (not just the > process parsing it) by sucking all available RAM. We are talking hard > lockup, reboot-to-fix-it sorts of DOC here. Er no. We're talking about running out of RAM. Any reasonable person would already have a limit one way or another (rlimits anyone). From stephen at xemacs.org Thu Feb 21 18:23:27 2013 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Fri, 22 Feb 2013 02:23:27 +0900 Subject: [Python-Dev] XML DoS vulnerabilities and exploits in Python In-Reply-To: <74957F62-DF1C-46BC-9705-1494477CD217@gmail.com> References: <51239B07.8080300@python.org> <512514CC.2080508@python.org> <5125466D.5090308@python.org> <20130221000808.7a8d2989@pitrou.net> <9AC9F1A0A9BD422999B3AE43AA5C3BA3@gmail.com> <20130221002258.78a805d6@pitrou.net> <51256C5B.70906@pearwood.info> <20130221075312.4485ad58@pitrou.net> <5125CCC4.3080501@palladion.com> <20130221084246.579fd260@pitrou.net> <5125F47B.3010607@python.org> <20130221113253.20e8b28f@pitrou.net> <74957F62-DF1C-46BC-9705-1494477CD217@gmail.com> Message-ID: <87txp5shrk.fsf@uwakimon.sk.tsukuba.ac.jp> Jesse Noller writes: > I guess someone need to write a proof of concept exploit for you > and release it into the wild. This is a bit ridiculous. This stuff looks easy enough that surely Christian's post informed any malicious body who didn't already know how to do it. If the exploit matters, it's already in the wild. ("Hey, didja know that an XML processor that expands entities does so recursively?" "Uh-oh ....") Yeah, there's a problem here. But ... as far as I can see all the exploits suggested (including those Christian provided in python-external.py) require either blindly processing text from requests received off the Internet as XML, or an attacker capable of doing something equivalent to replacing a Python library. I certainly think defusedxml is a valuable contribution, and not just for security nuts. But to quote from Christian's own README (warning: taken out of context to make *my* point): 7. These are features but they may introduce exploitable holes, see `Other things to consider`_ I'd like to see a little (well, to be honest, a *lot*) more analysis of the kind Fred Drake implicitly suggests: Doing so *will* be backward incompatible, and I'm not sure there's a good way to gauge the extent of the breakage. before making these restrictions the default. Eg, 40 entity indirections in a single expansion (defusedxml's default maximum) may seem like a lot, but I've seen some pretty complex expressions built as entities that recurse three or four levels. Of course, that was a while ago, and today most of the entities would be replaced by actual characters. Nevertheless, I bet those legacy expressions break the 40 indirection limit, or, rather, the limit would break them. From eliben at gmail.com Thu Feb 21 19:39:17 2013 From: eliben at gmail.com (Eli Bendersky) Date: Thu, 21 Feb 2013 10:39:17 -0800 Subject: [Python-Dev] XML DoS vulnerabilities and exploits in Python In-Reply-To: <87txp5shrk.fsf@uwakimon.sk.tsukuba.ac.jp> References: <51239B07.8080300@python.org> <512514CC.2080508@python.org> <5125466D.5090308@python.org> <20130221000808.7a8d2989@pitrou.net> <9AC9F1A0A9BD422999B3AE43AA5C3BA3@gmail.com> <20130221002258.78a805d6@pitrou.net> <51256C5B.70906@pearwood.info> <20130221075312.4485ad58@pitrou.net> <5125CCC4.3080501@palladion.com> <20130221084246.579fd260@pitrou.net> <5125F47B.3010607@python.org> <20130221113253.20e8b28f@pitrou.net> <74957F62-DF1C-46BC-9705-1494477CD217@gmail.com> <87txp5shrk.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: On Thu, Feb 21, 2013 at 9:23 AM, Stephen J. Turnbull wrote: > Jesse Noller writes: > > > I guess someone need to write a proof of concept exploit for you > > and release it into the wild. > > This is a bit ridiculous. This stuff looks easy enough that surely > Christian's post informed any malicious body who didn't already know > how to do it. If the exploit matters, it's already in the wild. > ("Hey, didja know that an XML processor that expands entities does so > recursively?" "Uh-oh ....") > Just to clarify for my own curiosity. These attacks (e.g. http://en.wikipedia.org/wiki/Billion_laughs) have been known and public since 2003? Eli -------------- next part -------------- An HTML attachment was scrubbed... URL: From christian at python.org Thu Feb 21 20:12:22 2013 From: christian at python.org (Christian Heimes) Date: Thu, 21 Feb 2013 20:12:22 +0100 Subject: [Python-Dev] XML DoS vulnerabilities and exploits in Python In-Reply-To: References: <51239B07.8080300@python.org> <512514CC.2080508@python.org> <5125466D.5090308@python.org> <20130221000808.7a8d2989@pitrou.net> <9AC9F1A0A9BD422999B3AE43AA5C3BA3@gmail.com> <20130221002258.78a805d6@pitrou.net> <51256C5B.70906@pearwood.info> <20130221075312.4485ad58@pitrou.net> <5125CCC4.3080501@palladion.com> <20130221084246.579fd260@pitrou.net> <5125F47B.3010607@python.org> <20130221113253.20e8b28f@pitrou.net> <74957F62-DF1C-46BC-9705-1494477CD217@gmail.com> <87txp5shrk.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <51267196.1070001@python.org> Am 21.02.2013 19:39, schrieb Eli Bendersky: > Just to clarify for my own curiosity. These attacks (e.g. > http://en.wikipedia.org/wiki/Billion_laughs) have been known and public > since 2003? Correct, see https://pypi.python.org/pypi/defusedxml#synopsis third paragraph. All XML attacks in my analysis are well known for years, billion laughs for about a decade. As far as I know it's the first time somebody has compiled and published a detailed list of vulnerabilities in Python's XML libraries. However I'm not the only one. OpenStack and Django were contacted by several people in the past few weeks, too. From eliben at gmail.com Thu Feb 21 20:17:57 2013 From: eliben at gmail.com (Eli Bendersky) Date: Thu, 21 Feb 2013 11:17:57 -0800 Subject: [Python-Dev] XML DoS vulnerabilities and exploits in Python In-Reply-To: <51267196.1070001@python.org> References: <51239B07.8080300@python.org> <512514CC.2080508@python.org> <5125466D.5090308@python.org> <20130221000808.7a8d2989@pitrou.net> <9AC9F1A0A9BD422999B3AE43AA5C3BA3@gmail.com> <20130221002258.78a805d6@pitrou.net> <51256C5B.70906@pearwood.info> <20130221075312.4485ad58@pitrou.net> <5125CCC4.3080501@palladion.com> <20130221084246.579fd260@pitrou.net> <5125F47B.3010607@python.org> <20130221113253.20e8b28f@pitrou.net> <74957F62-DF1C-46BC-9705-1494477CD217@gmail.com> <87txp5shrk.fsf@uwakimon.sk.tsukuba.ac.jp> <51267196.1070001@python.org> Message-ID: On Thu, Feb 21, 2013 at 11:12 AM, Christian Heimes wrote: > Am 21.02.2013 19:39, schrieb Eli Bendersky: > > Just to clarify for my own curiosity. These attacks (e.g. > > http://en.wikipedia.org/wiki/Billion_laughs) have been known and public > > since 2003? > > Correct, see https://pypi.python.org/pypi/defusedxml#synopsis third > paragraph. All XML attacks in my analysis are well known for years, > billion laughs for about a decade. > > As far as I know it's the first time somebody has compiled and published > a detailed list of vulnerabilities in Python's XML libraries. However > I'm not the only one. OpenStack and Django were contacted by several > people in the past few weeks, too. > Thanks, Christian. I think this should put the urgency of the fix into context. While I agree that we should work on making future versions resilient by default, I have doubts about the urgency of back-patching existing, in-mainteinance-mode stable versions with something that's not opt-in. Eli -------------- next part -------------- An HTML attachment was scrubbed... URL: From paul at boddie.org.uk Fri Feb 22 00:47:08 2013 From: paul at boddie.org.uk (Paul Boddie) Date: Fri, 22 Feb 2013 00:47:08 +0100 Subject: [Python-Dev] xml.sax and xml.dom fetch DTDs by default (was XML DoS vulnerabilities and exploits in Python) Message-ID: <201302220047.08858.paul@boddie.org.uk> Perhaps related to the discussion of denial-of-service vulnerabilities is the matter of controlling access to remote resources. I suppose that after the following bug was closed, no improvements were made to the standard library: http://bugs.python.org/issue2124 Do Python programs still visit the W3C site millions of times every day to download DTDs that they are not, by default, able to remember from their last visit? Paul From christian at python.org Fri Feb 22 01:07:06 2013 From: christian at python.org (Christian Heimes) Date: Fri, 22 Feb 2013 01:07:06 +0100 Subject: [Python-Dev] xml.sax and xml.dom fetch DTDs by default In-Reply-To: <201302220047.08858.paul@boddie.org.uk> References: <201302220047.08858.paul@boddie.org.uk> Message-ID: <5126B6AA.9070201@python.org> Am 22.02.2013 00:47, schrieb Paul Boddie: > Perhaps related to the discussion of denial-of-service vulnerabilities is the > matter of controlling access to remote resources. I suppose that after the > following bug was closed, no improvements were made to the standard library: > > http://bugs.python.org/issue2124 > > Do Python programs still visit the W3C site millions of times every day to > download DTDs that they are not, by default, able to remember from their last > visit? Affirmative for Python 2.6 to 3.4 dev! It's all in my documentation, too. https://pypi.python.org/pypi/defusedxml#python-xml-libraries Christian From python at mrabarnett.plus.com Fri Feb 22 03:02:20 2013 From: python at mrabarnett.plus.com (MRAB) Date: Fri, 22 Feb 2013 02:02:20 +0000 Subject: [Python-Dev] Can't upload to PyPI Message-ID: <5126D1AC.1000102@mrabarnett.plus.com> Since the PyPI security notice of 2013-02-15 I've been unable to upload to PyPI via "setup.py upload". I changed my password during the grace period, and have reset it, but it's still rejected: Upload failed (401): Incorrect password I can login to PyPI with the password. Can anyone suggest what could be wrong? From graffatcolmingov at gmail.com Fri Feb 22 03:09:29 2013 From: graffatcolmingov at gmail.com (Ian Cordasco) Date: Thu, 21 Feb 2013 21:09:29 -0500 Subject: [Python-Dev] Can't upload to PyPI In-Reply-To: <5126D1AC.1000102@mrabarnett.plus.com> References: <5126D1AC.1000102@mrabarnett.plus.com> Message-ID: This is probably better suited to Catalog-sig but you have to edit your credentials in $HOME/.pypirc On Thu, Feb 21, 2013 at 9:02 PM, MRAB wrote: > Since the PyPI security notice of 2013-02-15 I've been unable to upload > to PyPI via "setup.py upload". > > I changed my password during the grace period, and have reset it, but > it's still rejected: > > Upload failed (401): Incorrect password > > I can login to PyPI with the password. > > Can anyone suggest what could be wrong? > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/graffatcolmingov%40gmail.com From python at mrabarnett.plus.com Fri Feb 22 03:27:21 2013 From: python at mrabarnett.plus.com (MRAB) Date: Fri, 22 Feb 2013 02:27:21 +0000 Subject: [Python-Dev] Can't upload to PyPI In-Reply-To: References: <5126D1AC.1000102@mrabarnett.plus.com> Message-ID: <5126D789.6080604@mrabarnett.plus.com> On 2013-02-22 02:09, Ian Cordasco wrote: > On Thu, Feb 21, 2013 at 9:02 PM, MRAB wrote: >> Since the PyPI security notice of 2013-02-15 I've been unable to upload >> to PyPI via "setup.py upload". >> >> I changed my password during the grace period, and have reset it, but >> it's still rejected: >> >> Upload failed (401): Incorrect password >> >> I can login to PyPI with the password. >> >> Can anyone suggest what could be wrong? >> > This is probably better suited to Catalog-sig but you have to edit > your credentials in $HOME/.pypirc > Are any other changes needed in .pypirc, _apart_ from the password? From graffatcolmingov at gmail.com Fri Feb 22 03:37:34 2013 From: graffatcolmingov at gmail.com (Ian Cordasco) Date: Thu, 21 Feb 2013 21:37:34 -0500 Subject: [Python-Dev] Can't upload to PyPI In-Reply-To: <5126D789.6080604@mrabarnett.plus.com> References: <5126D1AC.1000102@mrabarnett.plus.com> <5126D789.6080604@mrabarnett.plus.com> Message-ID: On Thu, Feb 21, 2013 at 9:27 PM, MRAB wrote: > On 2013-02-22 02:09, Ian Cordasco wrote: >> >> On Thu, Feb 21, 2013 at 9:02 PM, MRAB wrote: >>> >>> Since the PyPI security notice of 2013-02-15 I've been unable to upload >>> to PyPI via "setup.py upload". >>> >>> I changed my password during the grace period, and have reset it, but >>> it's still rejected: >>> >>> Upload failed (401): Incorrect password >>> >>> I can login to PyPI with the password. >>> >>> Can anyone suggest what could be wrong? >>> >> This is probably better suited to Catalog-sig but you have to edit >> your credentials in $HOME/.pypirc >> > Are any other changes needed in .pypirc, _apart_ from the password? > I don't recall needing any other changes. From python at mrabarnett.plus.com Fri Feb 22 03:48:43 2013 From: python at mrabarnett.plus.com (MRAB) Date: Fri, 22 Feb 2013 02:48:43 +0000 Subject: [Python-Dev] Can't upload to PyPI In-Reply-To: References: <5126D1AC.1000102@mrabarnett.plus.com> <5126D789.6080604@mrabarnett.plus.com> Message-ID: <5126DC8B.4060909@mrabarnett.plus.com> On 2013-02-22 02:37, Ian Cordasco wrote: > On Thu, Feb 21, 2013 at 9:27 PM, MRAB wrote: >> On 2013-02-22 02:09, Ian Cordasco wrote: >>> >>> On Thu, Feb 21, 2013 at 9:02 PM, MRAB wrote: >>>> >>>> Since the PyPI security notice of 2013-02-15 I've been unable to upload >>>> to PyPI via "setup.py upload". >>>> >>>> I changed my password during the grace period, and have reset it, but >>>> it's still rejected: >>>> >>>> Upload failed (401): Incorrect password >>>> >>>> I can login to PyPI with the password. >>>> >>>> Can anyone suggest what could be wrong? >>>> >>> This is probably better suited to Catalog-sig but you have to edit >>> your credentials in $HOME/.pypirc >>> >> Are any other changes needed in .pypirc, _apart_ from the password? >> > I don't recall needing any other changes. > OK, I'll take it to Catalog-sig. Thanks anyway. From status at bugs.python.org Fri Feb 22 18:07:22 2013 From: status at bugs.python.org (Python tracker) Date: Fri, 22 Feb 2013 18:07:22 +0100 (CET) Subject: [Python-Dev] Summary of Python tracker Issues Message-ID: <20130222170722.CE28F56A1C@psf.upfronthosting.co.za> ACTIVITY SUMMARY (2013-02-15 - 2013-02-22) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 3866 (+21) closed 25186 (+45) total 29052 (+66) Open issues with patches: 1667 Issues opened (46) ================== #17212: os.path.isfile() in Python 3.3 sometimes fails http://bugs.python.org/issue17212 opened by gpoore #17213: ctypes loads wrong version of C runtime, leading to error mess http://bugs.python.org/issue17213 opened by dancol #17214: http.client.HTTPConnection.putrequest encode error http://bugs.python.org/issue17214 opened by Mi.Zou #17216: sparc linux build fails with "could not import runpy module" http://bugs.python.org/issue17216 opened by uservorname.usernachname #17217: Fix test discovery for test_format.py on Windows http://bugs.python.org/issue17217 opened by zach.ware #17218: support title and description in argparse add_mutually_exclusi http://bugs.python.org/issue17218 opened by chris.jerdonek #17219: cross add Python's library directory when building python stan http://bugs.python.org/issue17219 opened by rpetrov #17220: Little enhancements of _bootstrap.py http://bugs.python.org/issue17220 opened by serhiy.storchaka #17221: Resort Misc/NEWS http://bugs.python.org/issue17221 opened by serhiy.storchaka #17222: py_compile.compile() explicitly sets st_mode for written files http://bugs.python.org/issue17222 opened by Arfrever #17223: Initializing array.array with unicode type code and buffer seg http://bugs.python.org/issue17223 opened by mjacob #17224: can not open idle in python 2.7.3 http://bugs.python.org/issue17224 opened by hayeswu #17226: libintl should also check for libiconv http://bugs.python.org/issue17226 opened by alanh #17227: devguide: buggy heading numbers http://bugs.python.org/issue17227 opened by pitrou #17229: unable to discover preferred HTTPConnection class http://bugs.python.org/issue17229 opened by samwyse #17231: Mark __del__ not being called in cycles as an impl detail http://bugs.python.org/issue17231 opened by fijall #17232: Improve -O docs http://bugs.python.org/issue17232 opened by fijall #17233: http.client header debug output format http://bugs.python.org/issue17233 opened by Kim.Gr??sman #17234: python-2.7.3-r3: crash in visit_decref() http://bugs.python.org/issue17234 opened by mmokrejs #17237: m68k aligns on 16bit boundaries. http://bugs.python.org/issue17237 opened by alanh #17238: Enhance import statement completion http://bugs.python.org/issue17238 opened by Ramchandra Apte #17239: XML vulnerabilities in Python http://bugs.python.org/issue17239 opened by christian.heimes #17240: argparse: subcommand name and arity http://bugs.python.org/issue17240 opened by Thibault.Kruse #17243: The changes made for issue 4074 should be documented http://bugs.python.org/issue17243 opened by r.david.murray #17244: py_compile.compile() fails to raise exceptions when writing of http://bugs.python.org/issue17244 opened by Arfrever #17245: ctypes libffi needs to align the x86 stack to 16 bytes http://bugs.python.org/issue17245 opened by gregory.p.smith #17246: cgitb fails when frame arguments are deleted (due to inspect b http://bugs.python.org/issue17246 opened by Andrew.Lutomirski #17247: int and float should detect inconsistent format strings http://bugs.python.org/issue17247 opened by christian.heimes #17249: reap threads in test_capi http://bugs.python.org/issue17249 opened by ezio.melotti #17250: argparse: Issue 15906 patch; positional with nargs='*' and str http://bugs.python.org/issue17250 opened by paul.j3 #17251: LWPCookieJar load() set domain_specifed wrong http://bugs.python.org/issue17251 opened by B. Kyven #17254: add thai encoding aliases to encodings.aliases http://bugs.python.org/issue17254 opened by fomcl at yahoo.com #17258: multiprocessing.connection challenge implicitly uses MD5 http://bugs.python.org/issue17258 opened by dmalcolm #17259: locale.format() rounding is not reliable for floats http://bugs.python.org/issue17259 opened by Francis.Nimick #17261: multiprocessing.manager BaseManager cannot return proxies from http://bugs.python.org/issue17261 opened by Wilson.Harron #17263: crash when tp_dealloc allows other threads http://bugs.python.org/issue17263 opened by Albert.Zeyer #17264: Update Building C and C++ Extensions with distutils documentat http://bugs.python.org/issue17264 opened by berker.peksag #17267: datetime.time support for '+' and 'now' http://bugs.python.org/issue17267 opened by ronaldoussoren #17268: Context managers written as C types no longer work in Python 2 http://bugs.python.org/issue17268 opened by lemburg #17269: getaddrinfo segfaults on OS X when provided with invalid argum http://bugs.python.org/issue17269 opened by tibbe #17272: request.full_url: unexpected results on assignment http://bugs.python.org/issue17272 opened by dbrecht #17273: multiprocessing.pool.Pool task/worker handlers are not fork sa http://bugs.python.org/issue17273 opened by abn #17274: distutils silently omits relative symlinks http://bugs.python.org/issue17274 opened by fberger #17275: io.BufferedWriter shows wrong type in argument error message http://bugs.python.org/issue17275 opened by mjacob #17276: HMAC: deprecate default hash http://bugs.python.org/issue17276 opened by christian.heimes #17277: incorrect line numbers in backtrace after removing a trace fun http://bugs.python.org/issue17277 opened by xdegaye Most recent 15 issues with no replies (15) ========================================== #17277: incorrect line numbers in backtrace after removing a trace fun http://bugs.python.org/issue17277 #17274: distutils silently omits relative symlinks http://bugs.python.org/issue17274 #17261: multiprocessing.manager BaseManager cannot return proxies from http://bugs.python.org/issue17261 #17251: LWPCookieJar load() set domain_specifed wrong http://bugs.python.org/issue17251 #17250: argparse: Issue 15906 patch; positional with nargs='*' and str http://bugs.python.org/issue17250 #17246: cgitb fails when frame arguments are deleted (due to inspect b http://bugs.python.org/issue17246 #17245: ctypes libffi needs to align the x86 stack to 16 bytes http://bugs.python.org/issue17245 #17243: The changes made for issue 4074 should be documented http://bugs.python.org/issue17243 #17239: XML vulnerabilities in Python http://bugs.python.org/issue17239 #17238: Enhance import statement completion http://bugs.python.org/issue17238 #17233: http.client header debug output format http://bugs.python.org/issue17233 #17229: unable to discover preferred HTTPConnection class http://bugs.python.org/issue17229 #17224: can not open idle in python 2.7.3 http://bugs.python.org/issue17224 #17219: cross add Python's library directory when building python stan http://bugs.python.org/issue17219 #17218: support title and description in argparse add_mutually_exclusi http://bugs.python.org/issue17218 Most recent 15 issues waiting for review (15) ============================================= #17275: io.BufferedWriter shows wrong type in argument error message http://bugs.python.org/issue17275 #17272: request.full_url: unexpected results on assignment http://bugs.python.org/issue17272 #17269: getaddrinfo segfaults on OS X when provided with invalid argum http://bugs.python.org/issue17269 #17264: Update Building C and C++ Extensions with distutils documentat http://bugs.python.org/issue17264 #17258: multiprocessing.connection challenge implicitly uses MD5 http://bugs.python.org/issue17258 #17249: reap threads in test_capi http://bugs.python.org/issue17249 #17245: ctypes libffi needs to align the x86 stack to 16 bytes http://bugs.python.org/issue17245 #17239: XML vulnerabilities in Python http://bugs.python.org/issue17239 #17227: devguide: buggy heading numbers http://bugs.python.org/issue17227 #17223: Initializing array.array with unicode type code and buffer seg http://bugs.python.org/issue17223 #17221: Resort Misc/NEWS http://bugs.python.org/issue17221 #17220: Little enhancements of _bootstrap.py http://bugs.python.org/issue17220 #17219: cross add Python's library directory when building python stan http://bugs.python.org/issue17219 #17217: Fix test discovery for test_format.py on Windows http://bugs.python.org/issue17217 #17211: pkgutil.iter_modules and walk_packages should return a namedtu http://bugs.python.org/issue17211 Top 10 most discussed issues (10) ================================= #17237: m68k aligns on 16bit boundaries. http://bugs.python.org/issue17237 25 msgs #15767: add ImportNotFoundError http://bugs.python.org/issue15767 18 msgs #14468: Update cloning guidelines in devguide http://bugs.python.org/issue14468 11 msgs #17222: py_compile.compile() explicitly sets st_mode for written files http://bugs.python.org/issue17222 11 msgs #17221: Resort Misc/NEWS http://bugs.python.org/issue17221 9 msgs #17227: devguide: buggy heading numbers http://bugs.python.org/issue17227 7 msgs #14905: zipimport.c needs to support namespace packages when no 'direc http://bugs.python.org/issue14905 5 msgs #15004: add weakref support to types.SimpleNamespace http://bugs.python.org/issue15004 5 msgs #16612: Integrate "Argument Clinic" specialized preprocessor into CPyt http://bugs.python.org/issue16612 5 msgs #17177: Document/deprecate imp http://bugs.python.org/issue17177 5 msgs Issues closed (43) ================== #6138: './configure; make install' fails in setup.py step if .pydistu http://bugs.python.org/issue6138 closed by ned.deily #6541: SpooledTemporaryFile breakages http://bugs.python.org/issue6541 closed by r.david.murray #7469: Design and History FAQ entry on Floating Point does not mentio http://bugs.python.org/issue7469 closed by r.david.murray #7842: py_compile.compile SyntaxError output http://bugs.python.org/issue7842 closed by r.david.murray #7963: Misleading error message from object(arg) http://bugs.python.org/issue7963 closed by r.david.murray #7981: False failure with doctest NORMALIZE_WHITESPACE in 3.1.1 http://bugs.python.org/issue7981 closed by r.david.murray #8745: zipimport is a bit slow http://bugs.python.org/issue8745 closed by serhiy.storchaka #8930: messed up formatting after reindenting http://bugs.python.org/issue8930 closed by r.david.murray #9669: regexp: zero-width matches in MIN_UNTIL http://bugs.python.org/issue9669 closed by serhiy.storchaka #13153: IDLE crashes when pasting non-BMP unicode char on Py3 http://bugs.python.org/issue13153 closed by serhiy.storchaka #13169: Regular expressions with 0 to 65536 repetitions raises Overflo http://bugs.python.org/issue13169 closed by serhiy.storchaka #13700: imaplib.IMAP4.authenticate authobject does not work correctly http://bugs.python.org/issue13700 closed by r.david.murray #15003: make PyNamespace_New() public http://bugs.python.org/issue15003 closed by eric.snow #15022: types.SimpleNamespace needs to be picklable http://bugs.python.org/issue15022 closed by eric.snow #17035: Use new style classes in {class, static}method examples http://bugs.python.org/issue17035 closed by ezio.melotti #17143: trace.py uses _warn without importing it http://bugs.python.org/issue17143 closed by ezio.melotti #17163: Fix test discovery for test_file.py http://bugs.python.org/issue17163 closed by ezio.melotti #17175: update PEP 430 http://bugs.python.org/issue17175 closed by ezio.melotti #17178: Clarify empty iterable result in any/all.__doc__, as in manual http://bugs.python.org/issue17178 closed by ezio.melotti #17184: re.VERBOSE doesn't respect whitespace in '( ?P...)' http://bugs.python.org/issue17184 closed by ezio.melotti #17193: Use binary prefixes http://bugs.python.org/issue17193 closed by serhiy.storchaka #17203: add long option names to unittest discovery docs http://bugs.python.org/issue17203 closed by chris.jerdonek #17208: add note/warning about daemon threads http://bugs.python.org/issue17208 closed by pitrou #17215: documentation misprints http://bugs.python.org/issue17215 closed by asvetlov #17225: JSON decoder reports wrong column number on first line http://bugs.python.org/issue17225 closed by serhiy.storchaka #17228: Building without PYMALLOC fails http://bugs.python.org/issue17228 closed by python-dev #17230: when forking, buffered output is not flushed first. http://bugs.python.org/issue17230 closed by gregory.p.smith #17235: "make sharedinstall" ignores ./configure settings http://bugs.python.org/issue17235 closed by ned.deily #17236: format_spec for sequence joining http://bugs.python.org/issue17236 closed by r.david.murray #17241: Python-2.3.5.exe file possibly corrupt http://bugs.python.org/issue17241 closed by skrah #17242: Fix code highlight in devguide/docquality.rst http://bugs.python.org/issue17242 closed by ezio.melotti #17248: test_posix chown -1, 0 tests fail if user has group root http://bugs.python.org/issue17248 closed by serhiy.storchaka #17252: Latin Capital Letter I with Dot Above http://bugs.python.org/issue17252 closed by pitrou #17253: stdin.readline behaviour different between IDLE and the consol http://bugs.python.org/issue17253 closed by serhiy.storchaka #17255: test_any and test_all should validate short-circuiting behavio http://bugs.python.org/issue17255 closed by ezio.melotti #17256: code example in C API docsshould be highlighted http://bugs.python.org/issue17256 closed by ezio.melotti #17257: re module shows unexpected non-greedy behavior when using grou http://bugs.python.org/issue17257 closed by Hendrik.Lemelson #17260: Seg fault when calling unicode() on old style object in virtua http://bugs.python.org/issue17260 closed by ned.deily #17262: OrderedDict not ordering properly when int and float keys are http://bugs.python.org/issue17262 closed by benjamin.peterson #17265: Fix code highlight in the string.Template example http://bugs.python.org/issue17265 closed by ezio.melotti #17266: Idle + tcl 8.6.0 Can't convert '_tkinter.Tcl_Obj' object to st http://bugs.python.org/issue17266 closed by serhiy.storchaka #17270: make the section header doc convention more clearly optional http://bugs.python.org/issue17270 closed by chris.jerdonek #17271: NamedTemporaryFile expects bytes, not string in documentation http://bugs.python.org/issue17271 closed by ezio.melotti From chris at simplistix.co.uk Fri Feb 22 20:29:26 2013 From: chris at simplistix.co.uk (Chris Withers) Date: Fri, 22 Feb 2013 19:29:26 +0000 Subject: [Python-Dev] BDFL delegation for PEP 426 (PyPI metadata 1.3) In-Reply-To: <510E65A9.90600@palladion.com> References: <20130203122754.7745a1be@pitrou.net> <20130203134853.098aadaa@pitrou.net> <20130203140901.678b32b6@pitrou.net> <510E65A9.90600@palladion.com> Message-ID: <5127C716.8030804@simplistix.co.uk> On 03/02/2013 13:27, Tres Seaver wrote: >> As for setuptools (as opposed to distribute), I don't think we should >> care anymore. > > Yes, you need to care. It is *still* true today that distribute and > setuptools remain largely interchangeable, which is the only thing that > makes distribute viable, in the ecosystem sense. + sys.maxint. I was very unpopular with the attitude I took to distribute when it arrived, including it's poor choice of name ;-) Now, I know my tone was bad, but we are where I feared we're be, to paraphrase Ned Batchelder's "two problems" job page: I had a problem with setuptools, so I thought I'd create distribute, now I have two problems... I'm excited by wheel and I still have vague hopes that one day the stdlib might be a collection of packages shipped as a "big distribution" for those who need it and separately upgradeable for the rest of us... ...but let's make sure we keep caring about the tools that people really use, which includes both setuptools and distribute. ...or deliberately break them both, at the same time as we fix the security issues, and provide a workable alternative. People will quickly migrate ;-) cheers, Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk From ezio.melotti at gmail.com Sat Feb 23 05:24:04 2013 From: ezio.melotti at gmail.com (Ezio Melotti) Date: Sat, 23 Feb 2013 06:24:04 +0200 Subject: [Python-Dev] [Python-checkins] peps: PEP 426: replace implied 'version starts with' with new ~= operator Message-ID: Hi, On Sat, Feb 23, 2013 at 5:33 AM, daniel.holth wrote: > http://hg.python.org/peps/rev/de69fe61f300 > changeset: 4764:de69fe61f300 > user: Daniel Holth > date: Fri Feb 22 22:33:09 2013 -0500 > summary: > PEP 426: replace implied 'version starts with' with new ~= operator > I haven't seen any discussion about this, but FWIW CSS [0] and JQuery [1] use ^= for this purpose. ^ also indicates the beginning of the string in regular expressions (this is why ^= was chosen for CSS/JQuery). They also use ~= to indicate "attribute contains word" [0][2]. Perl also has a similar-looking operator [3] (=~) used to test a regex match. Best Regards, Ezio Melotti [0]: http://www.w3.org/TR/selectors/#selectors [1]: http://api.jquery.com/attribute-starts-with-selector/ [2]: http://api.jquery.com/attribute-contains-word-selector/ [3]: http://perldoc.perl.org/perlop.html#Binding-Operators From ncoghlan at gmail.com Sat Feb 23 11:29:48 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 23 Feb 2013 20:29:48 +1000 Subject: [Python-Dev] BDFL delegation for PEP 426 (PyPI metadata 1.3) In-Reply-To: <5127C716.8030804@simplistix.co.uk> References: <20130203122754.7745a1be@pitrou.net> <20130203134853.098aadaa@pitrou.net> <20130203140901.678b32b6@pitrou.net> <510E65A9.90600@palladion.com> <5127C716.8030804@simplistix.co.uk> Message-ID: On Sat, Feb 23, 2013 at 5:29 AM, Chris Withers wrote: > ...but let's make sure we keep caring about the tools that people really > use, which includes both setuptools and distribute. The lack of a meaningful transition plan is where I think we fell down with PEP 345 & 386, and is also the main reason I've postponed accepting PEP 426 (PJE and others pointed out a couple of major gaps that either made it difficult to update pkg_resources to read the new format, or else made it difficult to generate metadata that existing releases of pkg_resources would understand). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Sat Feb 23 12:06:00 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 23 Feb 2013 21:06:00 +1000 Subject: [Python-Dev] [Python-checkins] peps: PEP 426: replace implied 'version starts with' with new ~= operator In-Reply-To: References: Message-ID: On Sat, Feb 23, 2013 at 2:24 PM, Ezio Melotti wrote: > Hi, > > On Sat, Feb 23, 2013 at 5:33 AM, daniel.holth > wrote: >> http://hg.python.org/peps/rev/de69fe61f300 >> changeset: 4764:de69fe61f300 >> user: Daniel Holth >> date: Fri Feb 22 22:33:09 2013 -0500 >> summary: >> PEP 426: replace implied 'version starts with' with new ~= operator >> > > I haven't seen any discussion about this, but FWIW CSS [0] and JQuery > [1] use ^= for this purpose. > ^ also indicates the beginning of the string in regular expressions > (this is why ^= was chosen for CSS/JQuery). > They also use ~= to indicate "attribute contains word" [0][2]. > Perl also has a similar-looking operator [3] (=~) used to test a regex match. Daniel is a fan of this syntax, but I think it is inferior to the implied approach, so don't expect it to survive to any accepted version of the PEP :) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From vinay_sajip at yahoo.co.uk Sat Feb 23 15:57:33 2013 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Sat, 23 Feb 2013 14:57:33 +0000 (UTC) Subject: [Python-Dev] =?utf-8?q?=5BPython-checkins=5D_peps=3A_PEP_426=3A_r?= =?utf-8?q?eplace_implied_=27version_starts_with=27_with_new_=7E=3D?= =?utf-8?q?_operator?= References: Message-ID: Nick Coghlan gmail.com> writes: > Daniel is a fan of this syntax, but I think it is inferior to the > implied approach, so don't expect it to survive to any accepted > version of the PEP :) Another thing against ~= is that it isn't valid Python syntax. It's not a deal- breaker, but it does mean that you can't e.g. use the ast module in the implementation. This might be a factor if the mini-language ever grows (as it recently did, adding parentheses). Regards, Vinay Sajip From stefan at bytereef.org Sat Feb 23 16:02:31 2013 From: stefan at bytereef.org (Stefan Krah) Date: Sat, 23 Feb 2013 16:02:31 +0100 Subject: [Python-Dev] [Python-checkins] peps: Pre-alpha draft for PEP 435 (enum). The name is not important at the moment, as In-Reply-To: <3ZCsSP0rSXzQZS@mail.python.org> References: <3ZCsSP0rSXzQZS@mail.python.org> Message-ID: <20130223150231.GA20921@sleipnir.bytereef.org> eli.bendersky wrote: > +Ordered comparisons between enumeration values are *not* supported. Enums are > +not integers! Hmm. I think this limits interoperation with C libraries and prototyping C code. Actually all I want from a Python enum is to be like a C enum with symbolic representations for the values. Stefan Krah From solipsis at pitrou.net Sat Feb 23 16:06:45 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sat, 23 Feb 2013 16:06:45 +0100 Subject: [Python-Dev] [Python-checkins] peps: Pre-alpha draft for PEP 435 (enum). The name is not important at the moment, as References: <3ZCsSP0rSXzQZS@mail.python.org> <20130223150231.GA20921@sleipnir.bytereef.org> Message-ID: <20130223160645.124f4b77@pitrou.net> On Sat, 23 Feb 2013 16:02:31 +0100 Stefan Krah wrote: > eli.bendersky wrote: > > +Ordered comparisons between enumeration values are *not* supported. Enums are > > +not integers! > > Hmm. I think this limits interoperation with C libraries and prototyping > C code. Agreed, this is a deal-breaker IMO. I'd actually argue that enums are not terribly useful apart from interacting with C constants. Regards Antoine. From ncoghlan at gmail.com Sat Feb 23 16:23:24 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 24 Feb 2013 01:23:24 +1000 Subject: [Python-Dev] [Python-checkins] peps: PEP 426: replace implied 'version starts with' with new ~= operator In-Reply-To: References: Message-ID: On Sun, Feb 24, 2013 at 12:57 AM, Vinay Sajip wrote: > Nick Coghlan gmail.com> writes: > >> Daniel is a fan of this syntax, but I think it is inferior to the >> implied approach, so don't expect it to survive to any accepted >> version of the PEP :) > > Another thing against ~= is that it isn't valid Python syntax. It's not a deal- > breaker, but it does mean that you can't e.g. use the ast module in the > implementation. This might be a factor if the mini-language ever grows (as it > recently did, adding parentheses). Daniel persuaded me that the *semantics* of Ruby's ~> pessimistic version comparison operator are highly desirable. I liked them so much, I'm now proposing them as the default behaviour of version specifiers. Thus, the "~=" operator goes away, and you can use "==" to explicitly request the previously proposed default behaviour, or just append an extra ".0" if you're more pessimistic about a dependency's backwards compatibility policies than the default interpretation. This and other aspects will be brought up on distutils-sig at some point not too far in the future :) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Sat Feb 23 16:31:09 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 24 Feb 2013 01:31:09 +1000 Subject: [Python-Dev] [Python-checkins] peps: Pre-alpha draft for PEP 435 (enum). The name is not important at the moment, as In-Reply-To: <20130223160645.124f4b77@pitrou.net> References: <3ZCsSP0rSXzQZS@mail.python.org> <20130223150231.GA20921@sleipnir.bytereef.org> <20130223160645.124f4b77@pitrou.net> Message-ID: On Sun, Feb 24, 2013 at 1:06 AM, Antoine Pitrou wrote: > On Sat, 23 Feb 2013 16:02:31 +0100 > Stefan Krah wrote: >> eli.bendersky wrote: >> > +Ordered comparisons between enumeration values are *not* supported. Enums are >> > +not integers! >> >> Hmm. I think this limits interoperation with C libraries and prototyping >> C code. > > Agreed, this is a deal-breaker IMO. I'd actually argue that enums are > not terribly useful apart from interacting with C constants. I've argued in the past that we should have a "Named Value" mechanism *before* we have a proposal to group named constants in some fashion (such as sequential enums, or binary flag groups). It never gained much traction, even though it could be more useful in many cases (think things like being able to tag a file descriptor with the repr of the object it belongs to) After seeing this checkin, I also suggested to Eli and Barry that any new PEP on this topic should come with a survey of *existing* pseudo-enums in the standard library, and an explanation of how the proposal can be used to make those either easier to debug when they show up in error messages or otherwise make it easier to detect when you've made a mistake while working with them. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From rdmurray at bitdance.com Sat Feb 23 16:57:56 2013 From: rdmurray at bitdance.com (R. David Murray) Date: Sat, 23 Feb 2013 10:57:56 -0500 Subject: [Python-Dev] [Python-checkins] peps: Pre-alpha draft for PEP 435 (enum). The name is not important at the moment, as In-Reply-To: References: <3ZCsSP0rSXzQZS@mail.python.org> <20130223150231.GA20921@sleipnir.bytereef.org> <20130223160645.124f4b77@pitrou.net> Message-ID: <20130223155802.06C7B250BD6@webabinitio.net> On Sun, 24 Feb 2013 01:31:09 +1000, Nick Coghlan wrote: > On Sun, Feb 24, 2013 at 1:06 AM, Antoine Pitrou wrote: > > On Sat, 23 Feb 2013 16:02:31 +0100 > > Stefan Krah wrote: > >> eli.bendersky wrote: > >> > +Ordered comparisons between enumeration values are *not* supported. Enums are > >> > +not integers! > >> > >> Hmm. I think this limits interoperation with C libraries and prototyping > >> C code. > > > > Agreed, this is a deal-breaker IMO. I'd actually argue that enums are > > not terribly useful apart from interacting with C constants. > > I've argued in the past that we should have a "Named Value" mechanism > *before* we have a proposal to group named constants in some fashion > (such as sequential enums, or binary flag groups). It never gained > much traction, even though it could be more useful in many cases > (think things like being able to tag a file descriptor with the repr > of the object it belongs to) > > After seeing this checkin, I also suggested to Eli and Barry that any > new PEP on this topic should come with a survey of *existing* > pseudo-enums in the standard library, and an explanation of how the > proposal can be used to make those either easier to debug when they > show up in error messages or otherwise make it easier to detect when > you've made a mistake while working with them. See also http://bugs.python.org/issue16801#msg178542 for another use case for named values. I've seen an awful lot of code that uses global variables or class attributes primarily to get name validation on constant values, and I think all of that code would be a prime candidate for using Named Values. Some of them are also enum-like, but many of them are not. So I'm with Nick on this one. --David From eliben at gmail.com Sat Feb 23 17:27:50 2013 From: eliben at gmail.com (Eli Bendersky) Date: Sat, 23 Feb 2013 08:27:50 -0800 Subject: [Python-Dev] [Python-checkins] peps: Pre-alpha draft for PEP 435 (enum). The name is not important at the moment, as In-Reply-To: <20130223155802.06C7B250BD6@webabinitio.net> References: <3ZCsSP0rSXzQZS@mail.python.org> <20130223150231.GA20921@sleipnir.bytereef.org> <20130223160645.124f4b77@pitrou.net> <20130223155802.06C7B250BD6@webabinitio.net> Message-ID: On Sat, Feb 23, 2013 at 7:57 AM, R. David Murray wrote: > On Sun, 24 Feb 2013 01:31:09 +1000, Nick Coghlan > wrote: > > On Sun, Feb 24, 2013 at 1:06 AM, Antoine Pitrou > wrote: > > > On Sat, 23 Feb 2013 16:02:31 +0100 > > > Stefan Krah wrote: > > >> eli.bendersky wrote: > > >> > +Ordered comparisons between enumeration values are *not* > supported. Enums are > > >> > +not integers! > > >> > > >> Hmm. I think this limits interoperation with C libraries and > prototyping > > >> C code. > > > > > > Agreed, this is a deal-breaker IMO. I'd actually argue that enums are > > > not terribly useful apart from interacting with C constants. > > > > I've argued in the past that we should have a "Named Value" mechanism > > *before* we have a proposal to group named constants in some fashion > > (such as sequential enums, or binary flag groups). It never gained > > much traction, even though it could be more useful in many cases > > (think things like being able to tag a file descriptor with the repr > > of the object it belongs to) > > > > After seeing this checkin, I also suggested to Eli and Barry that any > > new PEP on this topic should come with a survey of *existing* > > pseudo-enums in the standard library, and an explanation of how the > > proposal can be used to make those either easier to debug when they > > show up in error messages or otherwise make it easier to detect when > > you've made a mistake while working with them. > > See also http://bugs.python.org/issue16801#msg178542 for another use > case for named values. > > I've seen an awful lot of code that uses global variables or class > attributes primarily to get name validation on constant values, and I > think all of that code would be a prime candidate for using Named Values. > Some of them are also enum-like, but many of them are not. So I'm with > Nick on this one. > Any suggestions for places in the stdlib where enums could come useful will be most welcome Eli -------------- next part -------------- An HTML attachment was scrubbed... URL: From barry at python.org Sat Feb 23 17:34:54 2013 From: barry at python.org (Barry Warsaw) Date: Sat, 23 Feb 2013 11:34:54 -0500 Subject: [Python-Dev] [Python-checkins] peps: Pre-alpha draft for PEP 435 (enum). The name is not important at the moment, as In-Reply-To: <20130223150231.GA20921@sleipnir.bytereef.org> References: <3ZCsSP0rSXzQZS@mail.python.org> <20130223150231.GA20921@sleipnir.bytereef.org> Message-ID: <20130223113454.434c807e@anarchist.wooz.org> On Feb 23, 2013, at 04:02 PM, Stefan Krah wrote: >Hmm. I think this limits interoperation with C libraries and prototyping >C code. As for flufl.enums, it doesn't really, because while items are not ints they are interoperable with ints. >>> from flufl.enum import make >>> Colors = make('Colors', 'red green blue'.split()) >>> int(Colors.green) 2 >>> 7 + int(Colors.green) 9 Cheers, -Barry From ethan at stoneleaf.us Sat Feb 23 17:41:18 2013 From: ethan at stoneleaf.us (Ethan Furman) Date: Sat, 23 Feb 2013 08:41:18 -0800 Subject: [Python-Dev] [Python-checkins] peps: Pre-alpha draft for PEP 435 (enum). The name is not important at the moment, as In-Reply-To: <20130223150231.GA20921@sleipnir.bytereef.org> References: <3ZCsSP0rSXzQZS@mail.python.org> <20130223150231.GA20921@sleipnir.bytereef.org> Message-ID: <5128F12E.6040604@stoneleaf.us> > eli.bendersky wrote: >> +Ordered comparisons between enumeration values are *not* supported. Enums are >> +not integers! class WeekDays(Enum): SUNDAY = enum(doc='last day of the weekend') MONDAY = enum(doc='back to work!') TUESDAY = enum(doc='ho hum day') WEDNESDAY = enum(doc='rounding the bend!') THURSDAY = enum(doc='almost Friday!') FRIDAY = enum(doc='last day!') SATURDAY = enum(doc='sleeping in...') So you're saying that MONDAY should not come before TUESDAY?? -- ~Ethan~ From ncoghlan at gmail.com Sat Feb 23 17:51:58 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 24 Feb 2013 02:51:58 +1000 Subject: [Python-Dev] [Python-checkins] peps: Pre-alpha draft for PEP 435 (enum). The name is not important at the moment, as In-Reply-To: References: <3ZCsSP0rSXzQZS@mail.python.org> <20130223150231.GA20921@sleipnir.bytereef.org> <20130223160645.124f4b77@pitrou.net> <20130223155802.06C7B250BD6@webabinitio.net> Message-ID: On Sun, Feb 24, 2013 at 2:27 AM, Eli Bendersky wrote: > Any suggestions for places in the stdlib where enums could come useful will > be most welcome For named values in general: - 0, 1, 2 as file descriptors (stdin/stdout/stderr) - 0, 1, 2 as relative seek locations (start, current, end, but I forget the exact mapping) - results of inspect.getgeneratorstate() (currently strings) - decimal rounding modes (currently strings) - opcodes - as a building block to simplify other parts of the dis module - HTML error codes I expect there would be many more possibilities if you dove into particular file formats and protocols. The reason I like named values as a starting point is that they can act primarily as the original type, while still being a useful building block for an enum type added later, as well as for any custom enum types created when people don't want quite the same structural behaviour at the higher level. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From rdmurray at bitdance.com Sat Feb 23 17:55:52 2013 From: rdmurray at bitdance.com (R. David Murray) Date: Sat, 23 Feb 2013 11:55:52 -0500 Subject: [Python-Dev] [Python-checkins] peps: Pre-alpha draft for PEP 435 (enum). The name is not important at the moment, as In-Reply-To: References: <3ZCsSP0rSXzQZS@mail.python.org> <20130223150231.GA20921@sleipnir.bytereef.org> <20130223160645.124f4b77@pitrou.net> <20130223155802.06C7B250BD6@webabinitio.net> Message-ID: <20130223165557.956A8250BD5@webabinitio.net> On Sat, 23 Feb 2013 08:27:50 -0800, Eli Bendersky wrote: > On Sat, Feb 23, 2013 at 7:57 AM, R. David Murray wrote: > > > On Sun, 24 Feb 2013 01:31:09 +1000, Nick Coghlan > > wrote: > > > On Sun, Feb 24, 2013 at 1:06 AM, Antoine Pitrou > > wrote: > > > > On Sat, 23 Feb 2013 16:02:31 +0100 > > > > Stefan Krah wrote: > > > >> eli.bendersky wrote: > > > >> > +Ordered comparisons between enumeration values are *not* > > supported. Enums are > > > >> > +not integers! > > > >> > > > >> Hmm. I think this limits interoperation with C libraries and > > prototyping > > > >> C code. > > > > > > > > Agreed, this is a deal-breaker IMO. I'd actually argue that enums are > > > > not terribly useful apart from interacting with C constants. > > > > > > I've argued in the past that we should have a "Named Value" mechanism > > > *before* we have a proposal to group named constants in some fashion > > > (such as sequential enums, or binary flag groups). It never gained > > > much traction, even though it could be more useful in many cases > > > (think things like being able to tag a file descriptor with the repr > > > of the object it belongs to) > > > > > > After seeing this checkin, I also suggested to Eli and Barry that any > > > new PEP on this topic should come with a survey of *existing* > > > pseudo-enums in the standard library, and an explanation of how the > > > proposal can be used to make those either easier to debug when they > > > show up in error messages or otherwise make it easier to detect when > > > you've made a mistake while working with them. > > > > See also http://bugs.python.org/issue16801#msg178542 for another use > > case for named values. > > > > I've seen an awful lot of code that uses global variables or class > > attributes primarily to get name validation on constant values, and I > > think all of that code would be a prime candidate for using Named Values. > > Some of them are also enum-like, but many of them are not. So I'm with > > Nick on this one. > > > > Any suggestions for places in the stdlib where enums could come useful will > be most welcome Heh. I wasn't actually speaking about the stdlib in the second paragraph :) But, if we had a Named Value facility I would use it in the email library for header error sub-codes (something I haven't actually introduced yet). But, speaking of error codes, it would be really nice if the posix error numbers had a repr that included the name. --David From solipsis at pitrou.net Sat Feb 23 18:04:52 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sat, 23 Feb 2013 18:04:52 +0100 Subject: [Python-Dev] [Python-checkins] peps: Pre-alpha draft for PEP 435 (enum). The name is not important at the moment, as References: <3ZCsSP0rSXzQZS@mail.python.org> <20130223150231.GA20921@sleipnir.bytereef.org> <20130223160645.124f4b77@pitrou.net> <20130223155802.06C7B250BD6@webabinitio.net> Message-ID: <20130223180452.4f8cd42a@pitrou.net> On Sat, 23 Feb 2013 08:27:50 -0800 Eli Bendersky wrote: > > See also http://bugs.python.org/issue16801#msg178542 for another use > > case for named values. > > > > I've seen an awful lot of code that uses global variables or class > > attributes primarily to get name validation on constant values, and I > > think all of that code would be a prime candidate for using Named Values. > > Some of them are also enum-like, but many of them are not. So I'm with > > Nick on this one. > > Any suggestions for places in the stdlib where enums could come useful will > be most welcome The constants in the os module (os.SEEK_SET, etc.). The constants in the socket module (socket.AF_INET, etc.). And so on :-) Regards Antoine. From eliben at gmail.com Sat Feb 23 18:15:54 2013 From: eliben at gmail.com (Eli Bendersky) Date: Sat, 23 Feb 2013 09:15:54 -0800 Subject: [Python-Dev] [Python-checkins] peps: Pre-alpha draft for PEP 435 (enum). The name is not important at the moment, as In-Reply-To: <20130223180452.4f8cd42a@pitrou.net> References: <3ZCsSP0rSXzQZS@mail.python.org> <20130223150231.GA20921@sleipnir.bytereef.org> <20130223160645.124f4b77@pitrou.net> <20130223155802.06C7B250BD6@webabinitio.net> <20130223180452.4f8cd42a@pitrou.net> Message-ID: On Sat, Feb 23, 2013 at 9:04 AM, Antoine Pitrou wrote: > On Sat, 23 Feb 2013 08:27:50 -0800 > Eli Bendersky wrote: > > > See also http://bugs.python.org/issue16801#msg178542 for another use > > > case for named values. > > > > > > I've seen an awful lot of code that uses global variables or class > > > attributes primarily to get name validation on constant values, and I > > > think all of that code would be a prime candidate for using Named > Values. > > > Some of them are also enum-like, but many of them are not. So I'm with > > > Nick on this one. > > > > Any suggestions for places in the stdlib where enums could come useful > will > > be most welcome > > The constants in the os module (os.SEEK_SET, etc.). > The constants in the socket module (socket.AF_INET, etc.). > And so on :-) > Hmm, constants such as os.SEEK_* which serve as *inputs* to stdlib rather than outputs can actually be a good candidate for enum without worrying about backwards compatibility. The reason I make the *input* vs. *output* distinction, is that for stdlib code that returns values, the user may rely on their numeric values and if we switch to enum code will break. However, for os.SEEK_* this is not the case. Thee can be aliases for enums: class SeekKind(Enum): SEEK_SET = 0 SEEK_CUR = 1 SEEK_END = 2 SEEK_SET = SeekKind.SEEK_SET ... lseek() can be changed to call 'int' on its input, and now os.SEEK_* gain useful string representations when printed. Thoughts? Eli -------------- next part -------------- An HTML attachment was scrubbed... URL: From ethan at stoneleaf.us Sat Feb 23 18:08:18 2013 From: ethan at stoneleaf.us (Ethan Furman) Date: Sat, 23 Feb 2013 09:08:18 -0800 Subject: [Python-Dev] [Python-checkins] peps: Pre-alpha draft for PEP 435 (enum). The name is not important at the moment, as In-Reply-To: References: <3ZCsSP0rSXzQZS@mail.python.org> <20130223150231.GA20921@sleipnir.bytereef.org> <20130223160645.124f4b77@pitrou.net> <20130223155802.06C7B250BD6@webabinitio.net> Message-ID: <5128F782.50407@stoneleaf.us> On 02/23/2013 08:27 AM, Eli Bendersky wrote: > Any suggestions for places in the stdlib where enums could come useful will be most welcome codecs.EncodedFile: errors = 'strict' | 'ignore' | 'xmlcharrefreplace' | 'replace' socket: AF_INET, AF_UNIX -- socket domains (first argument to socket() call) SOCK_STREAM, SOCK_DGRAM, SOCK_RAW -- socket types (second argument) httplib: client - many cookiejar - many turtle tkinter logging warning From ncoghlan at gmail.com Sat Feb 23 18:46:39 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 24 Feb 2013 03:46:39 +1000 Subject: [Python-Dev] [Python-checkins] peps: Pre-alpha draft for PEP 435 (enum). The name is not important at the moment, as In-Reply-To: References: <3ZCsSP0rSXzQZS@mail.python.org> <20130223150231.GA20921@sleipnir.bytereef.org> <20130223160645.124f4b77@pitrou.net> <20130223155802.06C7B250BD6@webabinitio.net> <20130223180452.4f8cd42a@pitrou.net> Message-ID: On Sun, Feb 24, 2013 at 3:15 AM, Eli Bendersky wrote: > Hmm, constants such as os.SEEK_* which serve as *inputs* to stdlib rather > than outputs can actually be a good candidate for enum without worrying > about backwards compatibility. Not true - users may be taking those values and passing them to third party APIs that expect them to be integers (since they're explicitly documented as such). However, pitching this at the enum level also introduces a mandatory level of structure we may not care about. All of the arguments about enums and what they should and shouldn't allow happen at the higher level, to do with the relations between a *group* of named values, and how the members of the group interact with other data types. For the standard library, we *really don't care* about any of those things, because we're currently using integers and strings for everything, so we can't add structure without risking breaking other people's code. However, just as we can get away with switching from producing and consuming tuples to producing and consuming namedtuples without breaking backwards compatibility, we *could* get away with producing and consuming names values in many cases, so long as those values behaved exactly like they did previously (i.e. as integers - there's little benefit to replacing our string "enums", since they're generally chosen for ease of debugging). We *can't* replace errno values, or the os module seek constants, etc with something that doesn't transparently behave like an integer, because people may currently be using comparisons against their integer counterparts, or passing them to non-stdlib APIs which expect the numeric value. But we could probably get away with changing their __repr__ (as well as switching them to subclasses of int rather than ordinary int objects). Many other existing libraries would be in the same boat - backwards compatibility would be an insurmountable barrier to using enums, but they *could* use named values. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From rdmurray at bitdance.com Sat Feb 23 18:53:19 2013 From: rdmurray at bitdance.com (R. David Murray) Date: Sat, 23 Feb 2013 12:53:19 -0500 Subject: [Python-Dev] [Python-checkins] peps: Pre-alpha draft for PEP 435 (enum). The name is not important at the moment, as In-Reply-To: References: <3ZCsSP0rSXzQZS@mail.python.org> <20130223150231.GA20921@sleipnir.bytereef.org> <20130223160645.124f4b77@pitrou.net> <20130223155802.06C7B250BD6@webabinitio.net> <20130223180452.4f8cd42a@pitrou.net> Message-ID: <20130223175324.67637250BD4@webabinitio.net> On Sat, 23 Feb 2013 09:15:54 -0800, Eli Bendersky wrote: > On Sat, Feb 23, 2013 at 9:04 AM, Antoine Pitrou wrote: > > > On Sat, 23 Feb 2013 08:27:50 -0800 > > Eli Bendersky wrote: > > > > See also http://bugs.python.org/issue16801#msg178542 for another use > > > > case for named values. > > > > > > > > I've seen an awful lot of code that uses global variables or class > > > > attributes primarily to get name validation on constant values, and I > > > > think all of that code would be a prime candidate for using Named > > Values. > > > > Some of them are also enum-like, but many of them are not. So I'm with > > > > Nick on this one. > > > > > > Any suggestions for places in the stdlib where enums could come useful > > will > > > be most welcome > > > > The constants in the os module (os.SEEK_SET, etc.). > > The constants in the socket module (socket.AF_INET, etc.). > > And so on :-) > > > > Hmm, constants such as os.SEEK_* which serve as *inputs* to stdlib rather > than outputs can actually be a good candidate for enum without worrying > about backwards compatibility. > > The reason I make the *input* vs. *output* distinction, is that for stdlib > code that returns values, the user may rely on their numeric values and if > we switch to enum code will break. However, for os.SEEK_* this is not the > case. Thee can be aliases for enums: > > class SeekKind(Enum): > SEEK_SET = 0 > SEEK_CUR = 1 > SEEK_END = 2 > > SEEK_SET = SeekKind.SEEK_SET > ... > > lseek() can be changed to call 'int' on its input, and now os.SEEK_* gain > useful string representations when printed. Eh, what? Why does int have to be called? (Hrm, I guess I'd better read the PEP before making any more comments :) I'm preferring Named Values more and more. And I'm just as interested in the returned values case as I am in the input case (as the errno example indicates). --David From ethan at stoneleaf.us Sat Feb 23 19:07:12 2013 From: ethan at stoneleaf.us (Ethan Furman) Date: Sat, 23 Feb 2013 10:07:12 -0800 Subject: [Python-Dev] [Python-checkins] peps: Pre-alpha draft for PEP 435 (enum). The name is not important at the moment, as In-Reply-To: References: <3ZCsSP0rSXzQZS@mail.python.org> <20130223150231.GA20921@sleipnir.bytereef.org> <20130223160645.124f4b77@pitrou.net> <20130223155802.06C7B250BD6@webabinitio.net> <20130223180452.4f8cd42a@pitrou.net> Message-ID: <51290550.8020602@stoneleaf.us> On 02/23/2013 09:46 AM, Nick Coghlan wrote: > Many other existing libraries would be in the same boat - backwards > compatibility would be an insurmountable barrier to using enums, but > they *could* use named values. I like the idea of named values, but to be clear about enums: if they are int-based (like True and False are) and don't mess with the math operations (like True and False don't) then we could use them with impunity: True + False = 1 os.SEEK_CUR * 3 = 3 (assuming, of course, that os.SEEK_CUR == 1) -- ~Ethan~ P.S. Has the irrevokable decision been made that enums will *not* be int based? If so, I'll stop bringing up this point -- I'm starting to bore myself with it. From ethan at stoneleaf.us Sat Feb 23 18:51:53 2013 From: ethan at stoneleaf.us (Ethan Furman) Date: Sat, 23 Feb 2013 09:51:53 -0800 Subject: [Python-Dev] [Python-checkins] peps: Pre-alpha draft for PEP 435 (enum). The name is not important at the moment, as In-Reply-To: References: <3ZCsSP0rSXzQZS@mail.python.org> <20130223150231.GA20921@sleipnir.bytereef.org> <20130223160645.124f4b77@pitrou.net> <20130223155802.06C7B250BD6@webabinitio.net> <20130223180452.4f8cd42a@pitrou.net> Message-ID: <512901B9.7070802@stoneleaf.us> On 02/23/2013 09:15 AM, Eli Bendersky wrote: > Hmm, constants such as os.SEEK_* which serve as *inputs* to stdlib rather than outputs can actually be a good > candidate for enum without worrying about backwards compatibility. > > The reason I make the *input* vs. *output* distinction, is that for stdlib code that returns values, the user may rely > on their numeric values and if we switch to enum code will break. However, for os.SEEK_* this is not the case. Thee can > be aliases for enums: > > class SeekKind(Enum): > SEEK_SET = 0 > SEEK_CUR = 1 > SEEK_END = 2 > > SEEK_SET = SeekKind.SEEK_SET > ... > > lseek() can be changed to call 'int' on its input, and now os.SEEK_* gain useful string representations when printed. > > Thoughts? If that particular enum is int based, you don't have to a) make the distinction between input/output b) call 'int' on its input and you /still/ get the useful string repr. ;) -- ~Ethan~ From ericsnowcurrently at gmail.com Sat Feb 23 20:35:11 2013 From: ericsnowcurrently at gmail.com (Eric Snow) Date: Sat, 23 Feb 2013 12:35:11 -0700 Subject: [Python-Dev] [Python-checkins] peps: Pre-alpha draft for PEP 435 (enum). The name is not important at the moment, as In-Reply-To: References: <3ZCsSP0rSXzQZS@mail.python.org> <20130223150231.GA20921@sleipnir.bytereef.org> <20130223160645.124f4b77@pitrou.net> <20130223155802.06C7B250BD6@webabinitio.net> <20130223180452.4f8cd42a@pitrou.net> Message-ID: On Sat, Feb 23, 2013 at 10:46 AM, Nick Coghlan wrote: > However, pitching this at the enum level also introduces a mandatory > level of structure we may not care about. All of the arguments about > enums and what they should and shouldn't allow happen at the higher > level, to do with the relations between a *group* of named values, and > how the members of the group interact with other data types. > > For the standard library, we *really don't care* about any of those > things, because we're currently using integers and strings for > everything, so we can't add structure without risking breaking other > people's code. However, just as we can get away with switching from > producing and consuming tuples to producing and consuming namedtuples > without breaking backwards compatibility, we *could* get away with > producing and consuming names values in many cases, so long as those > values behaved exactly like they did previously (i.e. as integers - > there's little benefit to replacing our string "enums", since they're > generally chosen for ease of debugging). +1 FWIW, I'm a fan of starting with a basic NamedValue API that avoids the structure decisions. The grouping question is valuable but a GroupedConstant can build on a NamedValue. -eric From demianbrecht at gmail.com Sat Feb 23 20:36:22 2013 From: demianbrecht at gmail.com (Demian Brecht) Date: Sat, 23 Feb 2013 11:36:22 -0800 Subject: [Python-Dev] request._parse Message-ID: Hope this question belongs here and not in python-ideas, but I'm curious about _parse in the Request object. Specifically, why it was decided to write a custom parse function when the likes or urlparse or urlsplit do essentially the same thing. Doesn't really seem DRY to me.. I was going to change this to use one of the aforementioned functions and submit a patch, but wanted to see if there was any rational behind this that I wasn't seeing. Yes, it's a very minor issue. Call me a stickler for things like this :) -- Demian Brecht http://demianbrecht.github.com From tjreedy at udel.edu Sat Feb 23 22:50:16 2013 From: tjreedy at udel.edu (Terry Reedy) Date: Sat, 23 Feb 2013 16:50:16 -0500 Subject: [Python-Dev] request._parse In-Reply-To: References: Message-ID: On 2/23/2013 2:36 PM, Demian Brecht wrote: > Hope this question belongs here and not in python-ideas, but I'm > curious about _parse in the Request object. Specifically, why it was > decided to write a custom parse function when the likes or urlparse or > urlsplit do essentially the same thing. urllib.parse contains urlparse, urlsplit, splittype, splithost, and other utility functions. urllib.request imports many, including those four. Request._parse sequentially uses splittype and splithost, raising if splittype fails. So it seems to me that it does both less and more that the more general functions. Since urlparse is used four other places in the module and urlsplit once, I would start with the assumption that either would have been here if it were the best and easiest. > Doesn't really seem DRY to me. Nearly all the work in _parse is done by splittype, splithost, and unquote, which *is* DRY as far as this function is concerned. The rest of the code is assignment, two ifs, and a raise. If you want to be helpful, leave _parse along and find a real bug to work on ;-). There are several urllib bug issues. Or check out the code coverage of some test module (see devguide), and see if more tests are needed. -- Terry Jan Reedy From demianbrecht at gmail.com Sat Feb 23 22:58:20 2013 From: demianbrecht at gmail.com (Demian Brecht) Date: Sat, 23 Feb 2013 13:58:20 -0800 Subject: [Python-Dev] Fwd: request._parse In-Reply-To: References: Message-ID: Sounds good to me, thanks for the feedback.. Yes, I guess tackling known issues is a much better use of time than trying to dig my own up ;) > If you want to be helpful, leave _parse along and find a real bug to work on > ;-). There are several urllib bug issues. Or check out the code coverage of > some test module (see devguide), and see if more tests are needed. From tjreedy at udel.edu Sat Feb 23 23:11:30 2013 From: tjreedy at udel.edu (Terry Reedy) Date: Sat, 23 Feb 2013 17:11:30 -0500 Subject: [Python-Dev] [Python-checkins] peps: Pre-alpha draft for PEP 435 (enum). The name is not important at the moment, as In-Reply-To: References: <3ZCsSP0rSXzQZS@mail.python.org> <20130223150231.GA20921@sleipnir.bytereef.org> <20130223160645.124f4b77@pitrou.net> <20130223155802.06C7B250BD6@webabinitio.net> <20130223180452.4f8cd42a@pitrou.net> Message-ID: On 2/23/2013 12:46 PM, Nick Coghlan wrote: > For the standard library, we *really don't care* about any of those > things, because we're currently using integers and strings for > everything, so we can't add structure without risking breaking other > people's code. However, just as we can get away with switching from > producing and consuming tuples to producing and consuming namedtuples > without breaking backwards compatibility, we *could* get away with > producing and consuming names values in many cases, so long as those > values behaved exactly like they did previously (i.e. as integers - > there's little benefit to replacing our string "enums", since they're > generally chosen for ease of debugging). It seems that what would specifically be useful is a namedint class whose instances would be ints with a .__name__ attribute and a custom .__str__ and __repr__. (Note that None, False, and True do not have such because the name knowledge is elsewhere -- in __str__ I would guess.) In all other respects, lets them be ints, just as namedtuples are otherwise tuples. As you say, a namedstring is hardly useful as a string can be considered to be its own name. I personally think we should skip the bikeshedding over how to avoid repeating names to make the bound name match the definition name (as with def, class, and import). Actually, they do not have to match and in cases, one might want a short bound name for retrieval and a longer definition name for display. In any case, compared to the number of functions, classes, and imported modules in the stdlib, there would only be a few namedints, and any typos would be caught very quickly. In io (say) stdin = namedint(1, 'stdin') # or namedint(1, 'standard input (fd 1)') etc plus 3 lines for seek. -- Terry Jan Reedy From dreamingforward at gmail.com Sat Feb 23 23:58:34 2013 From: dreamingforward at gmail.com (Mark Janssen) Date: Sat, 23 Feb 2013 14:58:34 -0800 Subject: [Python-Dev] [Python-checkins] peps: Pre-alpha draft for PEP 435 (enum). The name is not important at the moment, as In-Reply-To: <20130223150231.GA20921@sleipnir.bytereef.org> References: <3ZCsSP0rSXzQZS@mail.python.org> <20130223150231.GA20921@sleipnir.bytereef.org> Message-ID: On Sat, Feb 23, 2013 at 7:02 AM, Stefan Krah wrote: > eli.bendersky wrote: >> +Ordered comparisons between enumeration values are *not* supported. Enums are >> +not integers! I agree with your idea, but note you probably shouldn't call them e-num-erations, then. > Hmm. I think this limits interoperation with C libraries and prototyping > C code. Like I say, despite the lack of alignment with C, that this is the right way to go for a high-level language like Python. But I don't think it should be built-into the language. The bigger question is: what things should be comparable? and this is something that has not been settled. Nor do I believe it can be by the language. This is where I rather liked the old __cmp__ function, not only was it easier to implement, but I could define a comparison and it was clearer that I was doing so (within the class at least). Mark From perica.zivkovic at gmail.com Sun Feb 24 01:17:46 2013 From: perica.zivkovic at gmail.com (Perica Zivkovic) Date: Sat, 23 Feb 2013 18:17:46 -0600 Subject: [Python-Dev] Python 2.7.4 and 3.2.4 Message-ID: Hi there, like posted here: https://groups.google.com/forum/?fromgroups=#!topic/comp.lang.python/xXij443PM6I I'm curious to find out are there any timelines when 2.7.4 and 3.2.4 will be available. I was kinda waiting for it to release new Portable Python so now I'm just curious should I wait bit more and release on this python base or should I go for 2.7.3 and 3.2.3 -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Sun Feb 24 03:19:41 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 24 Feb 2013 12:19:41 +1000 Subject: [Python-Dev] [Python-checkins] peps: Pre-alpha draft for PEP 435 (enum). The name is not important at the moment, as In-Reply-To: References: <3ZCsSP0rSXzQZS@mail.python.org> <20130223150231.GA20921@sleipnir.bytereef.org> <20130223160645.124f4b77@pitrou.net> <20130223155802.06C7B250BD6@webabinitio.net> <20130223180452.4f8cd42a@pitrou.net> Message-ID: On 24 Feb 2013 08:14, "Terry Reedy" wrote: > > On 2/23/2013 12:46 PM, Nick Coghlan wrote: > >> For the standard library, we *really don't care* about any of those >> things, because we're currently using integers and strings for >> everything, so we can't add structure without risking breaking other >> people's code. However, just as we can get away with switching from >> producing and consuming tuples to producing and consuming namedtuples >> without breaking backwards compatibility, we *could* get away with >> producing and consuming names values in many cases, so long as those >> values behaved exactly like they did previously (i.e. as integers - >> there's little benefit to replacing our string "enums", since they're >> generally chosen for ease of debugging). > > > It seems that what would specifically be useful is a namedint class whose instances would be ints with a .__name__ attribute and a custom .__str__ and __repr__. (Note that None, False, and True do not have such because the name knowledge is elsewhere -- in __str__ I would guess.) In all other respects, lets them be ints, just as namedtuples are otherwise tuples. > > As you say, a namedstring is hardly useful as a string can be considered to be its own name. > > I personally think we should skip the bikeshedding over how to avoid repeating names to make the bound name match the definition name (as with def, class, and import). Actually, they do not have to match and in cases, one might want a short bound name for retrieval and a longer definition name for display. I realised last night that "labelled values" might be a better building block than "named values". I'll see if I can find my old python-ideas posts about the concept (although, to be honest, I'm not sure it has ever been elaborated much further than it has in this thread) > In any case, compared to the number of functions, classes, and imported modules in the stdlib, there would only be a few namedints, and any typos would be caught very quickly. > > In io (say) > > stdin = namedint(1, 'stdin') # or namedint(1, 'standard input (fd 1)') > etc > plus 3 lines for seek. > > > -- > Terry Jan Reedy > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Sun Feb 24 03:37:57 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 24 Feb 2013 12:37:57 +1000 Subject: [Python-Dev] [Python-checkins] peps: Pre-alpha draft for PEP 435 (enum). The name is not important at the moment, as In-Reply-To: References: <3ZCsSP0rSXzQZS@mail.python.org> <20130223150231.GA20921@sleipnir.bytereef.org> <20130223160645.124f4b77@pitrou.net> <20130223155802.06C7B250BD6@webabinitio.net> <20130223180452.4f8cd42a@pitrou.net> Message-ID: On Sun, Feb 24, 2013 at 12:19 PM, Nick Coghlan wrote: > On 24 Feb 2013 08:14, "Terry Reedy" wrote: >> I personally think we should skip the bikeshedding over how to avoid >> repeating names to make the bound name match the definition name (as with >> def, class, and import). Actually, they do not have to match and in cases, >> one might want a short bound name for retrieval and a longer definition name >> for display. > > I realised last night that "labelled values" might be a better building > block than "named values". > > I'll see if I can find my old python-ideas posts about the concept > (although, to be honest, I'm not sure it has ever been elaborated much > further than it has in this thread) Ah, there we go - I created a recipe a couple of years ago showing the basic concept: http://code.activestate.com/recipes/577810-named-values/ Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From mal at egenix.com Sun Feb 24 12:56:33 2013 From: mal at egenix.com (M.-A. Lemburg) Date: Sun, 24 Feb 2013 12:56:33 +0100 Subject: [Python-Dev] XML DoS vulnerabilities and exploits in Python In-Reply-To: <51260D6B.1050101@python.org> References: <51239B07.8080300@python.org> <512514CC.2080508@python.org> <5125466D.5090308@python.org> <20130221000808.7a8d2989@pitrou.net> <9AC9F1A0A9BD422999B3AE43AA5C3BA3@gmail.com> <20130221002258.78a805d6@pitrou.net> <51256C5B.70906@pearwood.info> <20130221075312.4485ad58@pitrou.net> <5125CCC4.3080501@palladion.com> <20130221084246.579fd260@pitrou.net> <5125F47B.3010607@python.org> <20130221113253.20e8b28f@pitrou.net> <51260D6B.1050101@python.org> Message-ID: <5129FFF1.5090208@egenix.com> Reminds me of the encoding attacks that were possible in earlier versions of Python... you could have e.g. an email processing script run the Python test suite by simply sending a specially crafted email :-) On 21.02.2013 13:04, Christian Heimes wrote: > Am 21.02.2013 11:32, schrieb Antoine Pitrou: >> You haven't proved that these were actual threats, nor how they >> actually worked. I'm gonna remain skeptical if there isn't anything >> more precise than "It highly depends on the parser and the application >> what kind of exploit is possible". > > https://bitbucket.org/tiran/defusedxml/src/82f4037464418bf11ea734969b7ca1c193e6ed91/other/python-external.py?at=default > > $ ./python-external.py > > REQUEST: > -------- > Aachen > > RESPONSE: > --------- > The weather in Aachen is terrible. > > REQUEST: > -------- > > > ]> > &passwd; > > > RESPONSE: > --------- > Unknown city root:x:0:0:root:/root:/bin/bash > daemon:x:1:1:daemon:/usr/sbin:/bin/sh > bin:x:2:2:bin:/bin:/bin/sh > sys:x:3:3:sys:/dev:/bin/sh > sync:x:4:65534:sync:/bin:/bin/sync > games:x:5:60:games:/usr/games:/bin/sh > man:x:6:12:man:/var/cache/man:/bin/sh > lp:x:7:7:lp:/var/spool/lpd:/bin/sh > mail:x:8:8:mail:/var/mail:/bin/sh > news:x:9:9:news:/var/spool/news:/bin/sh > uucp:x:10:10:uucp:/var/spool/uucp:/bin/sh > proxy:x:13:13:proxy:/bin:/bin/sh > www-data:x:33:33:www-data:/var/www:/bin/sh > backup:x:34:34:backup:/var/backups:/bi > > > REQUEST: > -------- > > "http://hg.python.org/cpython/raw-file/a11ddd687a0b/Lib/test/dh512.pem"> > ]> > &url; > > > RESPONSE: > --------- > Unknown city -----BEGIN DH PARAMETERS----- > MEYCQQD1Kv884bEpQBgRjXyEpwpy1obEAxnIByl6ypUM2Zafq9AKUJsCRtMIPWak > XUGfnHy9iUsiGSa6q6Jew1XpKgVfAgEC > -----END DH PARAMETERS----- > > These are the 512 bit DH parameters from "Assigned Number for SKIP > Protocols" > (http://www.skip-vpn.org/spec/numbers.html). > See there for how they were generated. > Note that g is not a generator, but this is not a problem since p is a > safe prime. > > > > Q.E.D. > Christian > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/mal%40egenix.com > -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Feb 24 2013) >>> Python Projects, Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From barry at python.org Mon Feb 25 02:40:00 2013 From: barry at python.org (Barry Warsaw) Date: Sun, 24 Feb 2013 20:40:00 -0500 Subject: [Python-Dev] [Python-checkins] peps: Pre-alpha draft for PEP 435 (enum). The name is not important at the moment, as References: <3ZCsSP0rSXzQZS@mail.python.org> <20130223150231.GA20921@sleipnir.bytereef.org> Message-ID: <20130224204000.6309bd6c@anarchist.wooz.org> On Feb 23, 2013, at 04:02 PM, Stefan Krah wrote: >> +Ordered comparisons between enumeration values are *not* supported. Enums >> are +not integers! > >Hmm. I think this limits interoperation with C libraries and prototyping >C code. This is mostly a red-herring. flufl.enum values are C-level int compatible without actually *being* ints. E.g. static PyObject * intcompat_printint(PyObject *self, PyObject *args) { int value; if (!PyArg_ParseTuple(args, "i", &value)) return NULL; printf("and the value is: %d\n", value); Py_RETURN_NONE; } >>> from _intcompat import * >>> printint(7) and the value is: 7 >>> from flufl.enum import make >>> Colors = make('Colors', 'red green blue'.split()) >>> printint(Colors.green) and the value is: 2 Full module code is here: http://bazaar.launchpad.net/~barry/+junk/intcompat/files Cheers, -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From ethan at stoneleaf.us Mon Feb 25 04:32:17 2013 From: ethan at stoneleaf.us (Ethan Furman) Date: Sun, 24 Feb 2013 19:32:17 -0800 Subject: [Python-Dev] [Python-checkins] peps: Pre-alpha draft for PEP 435 (enum). The name is not important at the moment, as In-Reply-To: <20130224204000.6309bd6c@anarchist.wooz.org> References: <3ZCsSP0rSXzQZS@mail.python.org> <20130223150231.GA20921@sleipnir.bytereef.org> <20130224204000.6309bd6c@anarchist.wooz.org> Message-ID: <512ADB41.5040109@stoneleaf.us> On 02/24/2013 05:40 PM, Barry Warsaw wrote: > On Feb 23, 2013, at 04:02 PM, Stefan Krah wrote: > >>> +Ordered comparisons between enumeration values are *not* supported. Enums >>> are +not integers! >> >> Hmm. I think this limits interoperation with C libraries and prototyping >> C code. > > This is mostly a red-herring. flufl.enum values are C-level int compatible > without actually *being* ints. > > E.g. > > static PyObject * > intcompat_printint(PyObject *self, PyObject *args) > { > int value; > if (!PyArg_ParseTuple(args, "i", &value)) > return NULL; > > printf("and the value is: %d\n", value); > Py_RETURN_NONE; > } > >>>> from _intcompat import * >>>> printint(7) > and the value is: 7 >>>> from flufl.enum import make >>>> Colors = make('Colors', 'red green blue'.split()) >>>> printint(Colors.green) > and the value is: 2 Okay, that's pretty cool. I would still like the int subclass, though, as it would be an aid to me on the Python side. -- ~Ethan~ From ncoghlan at gmail.com Mon Feb 25 09:16:54 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 25 Feb 2013 18:16:54 +1000 Subject: [Python-Dev] [Python-checkins] peps: PEP-0427: clarify some implementation details. In-Reply-To: <3ZDnVX6xNtz7LmK@mail.python.org> References: <3ZDnVX6xNtz7LmK@mail.python.org> Message-ID: On Mon, Feb 25, 2013 at 12:41 PM, daniel.holth wrote: > http://hg.python.org/peps/rev/7d2494f4cd0a > changeset: 4772:7d2494f4cd0a > user: Daniel Holth > date: Sun Feb 24 21:41:40 2013 -0500 > summary: > PEP-0427: clarify some implementation details. > > Hope it's OK to clarify two details that came up during Vinay's distlib wheel > implementation: zip directory filenames are encoded as utf-8, and it's nicer > to put the .dist-info directory at the end of the archive. It's better to announce the intended update/clarification on python-dev first, but I agree both these adjustments are reasonable (I don't remember if I actually said that in the distutils-sig thread). Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From damani at cse.iitb.ac.in Mon Feb 25 10:49:47 2013 From: damani at cse.iitb.ac.in (=?utf-8?B?T20gRGFtYW5pICjgpJPgpK4g4KSm4KSu4KWN4KSu4KS+4KSj4KWAKQ==?=) Date: Mon, 25 Feb 2013 15:19:47 +0530 Subject: [Python-Dev] Bug scipy v.11 library in python2.7 Message-ID: I wish to report the following bug. Scipy v.11 library in python2.7 gives spearmanrcorrel([1,2,3,4,5],[5,6,7,8,7]) = 0.8207 while scipy v.6 in python2.5 gives spearmanr([1,2,3,4,5],[5,6,7,8,7]) = 0.825(which is correct according to spearman correlation formula). The spearman correlation for [1,2,3,4,5],[5,6,7,8,7] calculated online according to formula available at : https://statistics.laerd.com/calculators/spearmans-rank-order-correlation-calculator-1.php, also gives 0.825. --The definition of spearmanr function in Scipy v.11 is given at : http://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.spearmanr.html#scipy.stats.spearmanr. Thanks, - Om Damani From tjreedy at udel.edu Mon Feb 25 11:57:41 2013 From: tjreedy at udel.edu (Terry Reedy) Date: Mon, 25 Feb 2013 05:57:41 -0500 Subject: [Python-Dev] Bug scipy v.11 library in python2.7 In-Reply-To: References: Message-ID: <512B43A5.20600@udel.edu> On 2/25/2013 4:49 AM, Om Damani (?? ???????) wrote: > I wish to report the following bug. Bugs should be reported on the issue tracker for the particular project. http://www.scipy.org/ has a Report Bugs button at the top. -- Terry Jan Reedy From barry at python.org Mon Feb 25 05:14:10 2013 From: barry at python.org (Barry Warsaw) Date: Sun, 24 Feb 2013 23:14:10 -0500 Subject: [Python-Dev] [Python-checkins] peps: Pre-alpha draft for PEP 435 (enum). The name is not important at the moment, as In-Reply-To: <512ADB41.5040109@stoneleaf.us> References: <3ZCsSP0rSXzQZS@mail.python.org> <20130223150231.GA20921@sleipnir.bytereef.org> <20130224204000.6309bd6c@anarchist.wooz.org> <512ADB41.5040109@stoneleaf.us> Message-ID: <20130224231410.0023038c@anarchist.wooz.org> On Feb 24, 2013, at 07:32 PM, Ethan Furman wrote: >I would still like the int subclass, though, as it would be an aid to me on >the Python side. I think you'll know what I'm going to say. :) 1) Usually, you won't need it (see the responses from people who don't care about the value). 2) When you do, wrapping the item in int() doesn't seem too bad to me. >>> from flufl.enum import make >>> Colors = make('Colors', 'red green blue'.split()) >>> int(Colors.blue) 3 Cheers, -Barry From ethan at stoneleaf.us Mon Feb 25 15:41:06 2013 From: ethan at stoneleaf.us (Ethan Furman) Date: Mon, 25 Feb 2013 06:41:06 -0800 Subject: [Python-Dev] [Python-checkins] peps: Pre-alpha draft for PEP 435 (enum). The name is not important at the moment, as In-Reply-To: <20130224231410.0023038c@anarchist.wooz.org> References: <3ZCsSP0rSXzQZS@mail.python.org> <20130223150231.GA20921@sleipnir.bytereef.org> <20130224204000.6309bd6c@anarchist.wooz.org> <512ADB41.5040109@stoneleaf.us> <20130224231410.0023038c@anarchist.wooz.org> Message-ID: <512B7802.9010306@stoneleaf.us> On 02/24/2013 08:14 PM, Barry Warsaw wrote: > On Feb 24, 2013, at 07:32 PM, Ethan Furman wrote: > >> I would still like the int subclass, though, as it would be an aid to me on >> the Python side. > > I think you'll know what I'm going to say. :) Yup, saw that coming. ;) > 1) Usually, you won't need it (see the responses from people who don't care > about the value). They are not me (or others like me (we do exist! we do! we do!). > 2) When you do, wrapping the item in int() doesn't seem too bad to me. If it was just once or twice, sure, but I use them as names for ints, which means I use them as ints, which means I would have a boat load of int() calls. -- ~Ethan~ From eliben at gmail.com Mon Feb 25 15:45:27 2013 From: eliben at gmail.com (Eli Bendersky) Date: Mon, 25 Feb 2013 06:45:27 -0800 Subject: [Python-Dev] [Python-checkins] peps: Pre-alpha draft for PEP 435 (enum). The name is not important at the moment, as In-Reply-To: <512B7802.9010306@stoneleaf.us> References: <3ZCsSP0rSXzQZS@mail.python.org> <20130223150231.GA20921@sleipnir.bytereef.org> <20130224204000.6309bd6c@anarchist.wooz.org> <512ADB41.5040109@stoneleaf.us> <20130224231410.0023038c@anarchist.wooz.org> <512B7802.9010306@stoneleaf.us> Message-ID: > 2) When you do, wrapping the item in int() doesn't seem too bad to me. >> > > If it was just once or twice, sure, but I use them as names for ints, > which means I use them as ints, which means I would have a boat load of > int() calls. > Personally I don't see "name for ints" as being the main use case for enums. For preparing the draft PEP I ran through some stdlib, Twisted and personal code and there are *tons* of places that just need a simple enum for some sort of "state", meaningful return value, or similar. That's really where you need enums the most, and really where their values don't matter. I prefer to have a good solution to one problem than a poorer solution that tries to cover two unrelated problems. For "names for ints", Nick's named value proposal seems more relevant, but why mix the two together? Eli -------------- next part -------------- An HTML attachment was scrubbed... URL: From skip at pobox.com Mon Feb 25 15:53:10 2013 From: skip at pobox.com (Skip Montanaro) Date: Mon, 25 Feb 2013 08:53:10 -0600 Subject: [Python-Dev] [Python-checkins] peps: Pre-alpha draft for PEP 435 (enum). The name is not important at the moment, as In-Reply-To: References: <3ZCsSP0rSXzQZS@mail.python.org> <20130223150231.GA20921@sleipnir.bytereef.org> <20130224204000.6309bd6c@anarchist.wooz.org> <512ADB41.5040109@stoneleaf.us> <20130224231410.0023038c@anarchist.wooz.org> <512B7802.9010306@stoneleaf.us> Message-ID: >> If it was just once or twice, sure, but I use them as names for ints, >> which means I use them as ints, which means I would have a boat load of >> int() calls. > > > Personally I don't see "name for ints" as being the main use case for enums. Ethan seems to have a use case, even if it is using enums to masquerade as true named constants. Perhaps the correct solution is to consider support for true constants in Python. (I would be surprised if there wasn't already at least one rejected or dormant PEP regarding that topic.) Skip From ethan at stoneleaf.us Mon Feb 25 16:12:03 2013 From: ethan at stoneleaf.us (Ethan Furman) Date: Mon, 25 Feb 2013 07:12:03 -0800 Subject: [Python-Dev] [Python-checkins] peps: Pre-alpha draft for PEP 435 (enum). The name is not important at the moment, as In-Reply-To: References: <3ZCsSP0rSXzQZS@mail.python.org> <20130223150231.GA20921@sleipnir.bytereef.org> <20130224204000.6309bd6c@anarchist.wooz.org> <512ADB41.5040109@stoneleaf.us> <20130224231410.0023038c@anarchist.wooz.org> <512B7802.9010306@stoneleaf.us> Message-ID: <512B7F43.5060406@stoneleaf.us> On 02/25/2013 06:45 AM, Eli Bendersky wrote: > > 2) When you do, wrapping the item in int() doesn't seem too bad to me. > > > If it was just once or twice, sure, but I use them as names for ints, which means I use them as ints, which means I > would have a boat load of int() calls. > > > Personally I don't see "name for ints" as being the main use case for enums. I must admit I find it mildly amusing (but a lot frustrating) that we are talk about /enumerations/ not needing to be based on ints. Checking out Merrian-Webster gives this: Definition of ENUMERATE 1 : to ascertain the number of : count 2 : to specify one after another : list Python even has an enumerate function, and what does it do? Gives us an int to associate with some item in an iterator. Now we're talking about adding an official Enum class, but the int portion is unnecessary and a pain to use? > For preparing the draft PEP I ran through > some stdlib, Twisted and personal code and there are *tons* of places that just need a simple enum for some sort of > "state", meaningful return value, or similar. That's really where you need enums the most, and really where their values > don't matter. And if, in those places, the enums were based on ints (or strings), would it hurt you? Because in the places where I, as well as many others, use enums we *need* the int portion; and having to wrap the enum in an int() call is seven extra keystrokes (minor) and a heap of visual clutter (major), destroying any value the enum was supposed to have. > I prefer to have a good solution to one problem than a poorer solution that tries to cover two unrelated problems. For > "names for ints", Nick's named value proposal seems more relevant, but why mix the two together? Again: Definition of ENUMERATE 1 : to ascertain the number of : count 2 : to specify one after another : list -- ~Ethan~ From ncoghlan at gmail.com Mon Feb 25 16:22:34 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 26 Feb 2013 01:22:34 +1000 Subject: [Python-Dev] [Python-checkins] peps: Pre-alpha draft for PEP 435 (enum). The name is not important at the moment, as In-Reply-To: References: <3ZCsSP0rSXzQZS@mail.python.org> <20130223150231.GA20921@sleipnir.bytereef.org> <20130224204000.6309bd6c@anarchist.wooz.org> <512ADB41.5040109@stoneleaf.us> <20130224231410.0023038c@anarchist.wooz.org> <512B7802.9010306@stoneleaf.us> Message-ID: On Tue, Feb 26, 2013 at 12:53 AM, Skip Montanaro wrote: >>> If it was just once or twice, sure, but I use them as names for ints, >>> which means I use them as ints, which means I would have a boat load of >>> int() calls. >> >> >> Personally I don't see "name for ints" as being the main use case for enums. > > Ethan seems to have a use case, even if it is using enums to > masquerade as true named constants. Perhaps the correct solution is > to consider support for true constants in Python. (I would be > surprised if there wasn't already at least one rejected or dormant PEP > regarding that topic.) I don't think we need true constants - labelled values should suffice for easier to debug magic numbers. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From barry at python.org Mon Feb 25 06:09:29 2013 From: barry at python.org (Barry Warsaw) Date: Mon, 25 Feb 2013 00:09:29 -0500 Subject: [Python-Dev] [Python-checkins] peps: Pre-alpha draft for PEP 435 (enum). The name is not important at the moment, as In-Reply-To: References: <3ZCsSP0rSXzQZS@mail.python.org> <20130223150231.GA20921@sleipnir.bytereef.org> <20130224204000.6309bd6c@anarchist.wooz.org> <512ADB41.5040109@stoneleaf.us> <20130224231410.0023038c@anarchist.wooz.org> <512B7802.9010306@stoneleaf.us> Message-ID: <20130225000929.55529abb@anarchist.wooz.org> On Feb 26, 2013, at 01:22 AM, Nick Coghlan wrote: >I don't think we need true constants - labelled values should suffice >for easier to debug magic numbers. +1 -Barry From solipsis at pitrou.net Mon Feb 25 16:29:53 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 25 Feb 2013 16:29:53 +0100 Subject: [Python-Dev] [Python-checkins] peps: Pre-alpha draft for PEP 435 (enum). The name is not important at the moment, as References: <3ZCsSP0rSXzQZS@mail.python.org> <20130223150231.GA20921@sleipnir.bytereef.org> <20130224204000.6309bd6c@anarchist.wooz.org> <512ADB41.5040109@stoneleaf.us> <20130224231410.0023038c@anarchist.wooz.org> <512B7802.9010306@stoneleaf.us> Message-ID: <20130225162953.16e7fa5e@pitrou.net> Le Mon, 25 Feb 2013 06:45:27 -0800, Eli Bendersky a ?crit : > For preparing the draft PEP I ran through some stdlib, Twisted and > personal code and there are *tons* of places that just need a simple > enum for some sort of "state", meaningful return value, or similar. There are also *tons* of public APIs which are now ints and would benefit from becoming named values, but only if doing so doesn't break compatibility (i.e. if they are still usable everywhere an int can be used). > I prefer to have a good solution to one problem than a poorer > solution that tries to cover two unrelated problems. For "names for > ints", Nick's named value proposal seems more relevant, but why mix > the two together? Because the whole functionality is not important enough to have two slightly different variations on it in the stdlib? And why is it a poorer solution exactly? Really, there can be two kinds of enum values (or named values): - int-alike values (usable everywhere an int is) - str-alike values (usable everywhere a str is) A third kind of value, which is partly int-compatible but not entirely, sounds like a byzantine subtlety to me. Regards Antoine. From rdmurray at bitdance.com Mon Feb 25 16:35:25 2013 From: rdmurray at bitdance.com (R. David Murray) Date: Mon, 25 Feb 2013 10:35:25 -0500 Subject: [Python-Dev] [Python-checkins] peps: Pre-alpha draft for PEP 435 (enum). The name is not important at the moment, as In-Reply-To: <512B7F43.5060406@stoneleaf.us> References: <3ZCsSP0rSXzQZS@mail.python.org> <20130223150231.GA20921@sleipnir.bytereef.org> <20130224204000.6309bd6c@anarchist.wooz.org> <512ADB41.5040109@stoneleaf.us> <20130224231410.0023038c@anarchist.wooz.org> <512B7802.9010306@stoneleaf.us> <512B7F43.5060406@stoneleaf.us> Message-ID: <20130225153526.568E6250BC5@webabinitio.net> On Mon, 25 Feb 2013 07:12:03 -0800, Ethan Furman wrote: > I must admit I find it mildly amusing (but a lot frustrating) that we > are talk about /enumerations/ not needing to be based on ints. > Checking out Merrian-Webster gives this: > > Definition of ENUMERATE > 1 > : to ascertain the number of : count > 2 > : to specify one after another : list I believe they are taking the second definition. Which would mean that, at a minimum, if it is called an Enum the components should be orderable according to the order of definition. Also having an integer value would decrease the surprise factor. Either that, or name them something other than "enum". If they aren't ints and they aren't orderable, it seems to me they are just a set of names. Which if we had labeled values, could be implemented as...a set of names. --David From ethan at stoneleaf.us Mon Feb 25 17:05:56 2013 From: ethan at stoneleaf.us (Ethan Furman) Date: Mon, 25 Feb 2013 08:05:56 -0800 Subject: [Python-Dev] [Python-checkins] peps: Pre-alpha draft for PEP 435 (enum). The name is not important at the moment, as In-Reply-To: References: <3ZCsSP0rSXzQZS@mail.python.org> <20130223150231.GA20921@sleipnir.bytereef.org> <20130224204000.6309bd6c@anarchist.wooz.org> <512ADB41.5040109@stoneleaf.us> <20130224231410.0023038c@anarchist.wooz.org> <512B7802.9010306@stoneleaf.us> Message-ID: <512B8BE4.5040102@stoneleaf.us> I've stated my reasons for why enums should be int based, and some of the problems with having them not be int-based. Besides "we just don't need them int-based in these use-cases" what are the reasons for the strong desire to have them be valueless? -- ~Ethan~ From rahulgarg at live.ca Mon Feb 25 17:35:52 2013 From: rahulgarg at live.ca (rahul garg) Date: Mon, 25 Feb 2013 11:35:52 -0500 Subject: [Python-Dev] Built with VS2012 Express for desktop In-Reply-To: References: , , , Message-ID: Hello again! Apparently the executable I built was broken. I tried with Debug configuration on x64 and got a python_d.exe successfully. However trying to run "python_d.exe -m test" results in an error that cannot import "support" from test. rahul From: rahulgarg at live.ca To: brian at python.org Date: Wed, 20 Feb 2013 03:03:14 -0500 CC: python-dev at python.org Subject: Re: [Python-Dev] Built with VS2012 Express for desktop > Date: Tue, 19 Feb 2013 12:48:02 -0600 > Subject: Re: [Python-Dev] Built with VS2012 Express for desktop > From: brian at python.org > To: rahulgarg at live.ca > CC: python-dev at python.org > > On Tue, Feb 19, 2013 at 12:37 PM, rahul garg wrote: > > Hi. > > > > I downloaded Python 3.3 source, opened up the solution in VS2012 Express for > > Desktop and built the "python" subproject using "Release" and "x64" > > configurations. I now have a "python.exe" in the PCBuild/amd64 subfolder > > that appears to be working as far as i can see. > > > > I have a few questions: > > a) Is there a test suite that I can run to verify that the build is working > > fine? > Last I checked there are some issues, but most of the tests should > pass. You would run "PCBuild\python.exe -m test" from the top level of > your checkout to see this. It's also covered at > http://docs.python.org/devguide/ Thanks! That page is what I should have looked for! > > b) I now intend to build some extensions (such as NumPy). Not sure if this > > is the right list, but would I need to modify something in distutils to get > > it working with VS2012? > > Yes. You'll probably need to point distutils to the correct batch file > that sets up a VS2012 build environment. Thanks again! rahul _______________________________________________ Python-Dev mailing list Python-Dev at python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/rahulgarg%40live.ca -------------- next part -------------- An HTML attachment was scrubbed... URL: From barry at python.org Mon Feb 25 17:37:29 2013 From: barry at python.org (Barry Warsaw) Date: Mon, 25 Feb 2013 11:37:29 -0500 Subject: [Python-Dev] [Python-checkins] peps: Pre-alpha draft for PEP 435 (enum). The name is not important at the moment, as In-Reply-To: <512B7F43.5060406@stoneleaf.us> References: <3ZCsSP0rSXzQZS@mail.python.org> <20130223150231.GA20921@sleipnir.bytereef.org> <20130224204000.6309bd6c@anarchist.wooz.org> <512ADB41.5040109@stoneleaf.us> <20130224231410.0023038c@anarchist.wooz.org> <512B7802.9010306@stoneleaf.us> <512B7F43.5060406@stoneleaf.us> Message-ID: <20130225113729.235b86c6@anarchist.wooz.org> On Feb 25, 2013, at 07:12 AM, Ethan Furman wrote: >And if, in those places, the enums were based on ints (or strings), would it >hurt you? Because in the places where I, as well as many others, use enums >we *need* the int portion; and having to wrap the enum in an int() call is >seven extra keystrokes (minor) and a heap of visual clutter (major), >destroying any value the enum was supposed to have. Yes, I think enum values inheriting from int (or string) would hurt. First, as your question implies, which is it? int or str? Some people want int some want str. It can't be both, and I don't think we need two enum types. It's a deeper question though, because if enum values inherited from int, then people would expect things like ``Colors.red == 1`` to work. Maybe you think that doesn't look so bad, but that also implies: >>> Colors = make('Colors', 'red green blue'.split()) >>> Animals = make('Animals', 'ant bee cat'.split()) >>> Colors.green == Animals.bee and *that* I have a problem with. While I generally recommend that that enum values be comparable by identity, not by equality, lots of people will still use `==` instead of `is`. My preference: def utter(animal): if animal is Animals.cat: print('meow') elif animal is Animals.bee: print('buzz') elif animal is Animals.ant: print('fphfft') but commonly used: def utter(animal): if animal == Animals.cat: print('meow') elif animal == Animals.bee: print('buzz') elif animal == Animals.ant: print('fphfft') In both cases, I want ``utter(Colors.green)`` and ``utter(2)`` to fail. In the latter, if enum values are derived from ints, the second example will succeed. Currently, in both cases ``utter(Animals.cat)`` succeeds. Even worse, if my library defines an Animals enum and your library defines a different Animals enum, I do not want ``utter(YourAnimals.dog)`` to print "meow" :). I get that you think wrapping the value in int() is ugly. I have a compromise that you might find acceptable. EnumValues already have .enum and .name attributes, to give you the value's class and string name respectively. It would be trivial to add a .int attribute to return the value. Thus if you want the int value, it would be easy enough to say ``Colors.green.int`` or ``Animals.bee.int`` https://bugs.launchpad.net/flufl.enum/+bug/1132859 Cheers, -Barry From barry at python.org Mon Feb 25 17:42:14 2013 From: barry at python.org (Barry Warsaw) Date: Mon, 25 Feb 2013 11:42:14 -0500 Subject: [Python-Dev] [Python-checkins] peps: Add PEP 445: The Argument Clinic DSL In-Reply-To: <3ZF85b4VPXzNl2@mail.python.org> References: <3ZF85b4VPXzNl2@mail.python.org> Message-ID: <20130225114214.4aeadb15@anarchist.wooz.org> On Feb 25, 2013, at 05:40 PM, brett.cannon wrote: >http://hg.python.org/peps/rev/7aa92fb33436 >changeset: 4776:7aa92fb33436 >user: Brett Cannon >date: Mon Feb 25 11:39:56 2013 -0500 >summary: > Add PEP 445: The Argument Clinic DSL I beat you with PEP 436. :) -Barry From solipsis at pitrou.net Mon Feb 25 17:43:08 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 25 Feb 2013 17:43:08 +0100 Subject: [Python-Dev] [Python-checkins] peps: Pre-alpha draft for PEP 435 (enum). The name is not important at the moment, as References: <3ZCsSP0rSXzQZS@mail.python.org> <20130223150231.GA20921@sleipnir.bytereef.org> <20130224204000.6309bd6c@anarchist.wooz.org> <512ADB41.5040109@stoneleaf.us> <20130224231410.0023038c@anarchist.wooz.org> <512B7802.9010306@stoneleaf.us> <512B7F43.5060406@stoneleaf.us> <20130225113729.235b86c6@anarchist.wooz.org> Message-ID: <20130225174308.45bb98ad@pitrou.net> Le Mon, 25 Feb 2013 11:37:29 -0500, Barry Warsaw a ?crit : > On Feb 25, 2013, at 07:12 AM, Ethan Furman wrote: > > >And if, in those places, the enums were based on ints (or strings), > >would it hurt you? Because in the places where I, as well as many > >others, use enums we *need* the int portion; and having to wrap the > >enum in an int() call is seven extra keystrokes (minor) and a heap > >of visual clutter (major), destroying any value the enum was > >supposed to have. > > Yes, I think enum values inheriting from int (or string) would hurt. > > First, as your question implies, which is it? int or str? Some > people want int some want str. It can't be both, and I don't think > we need two enum types. I think we do "need" two enum types (as far as some enum mechanism is needed). str is better for most pure-Python uses, but int is necessary for a lot of portability / compatibility requirements. Regards Antoine. From skip at pobox.com Mon Feb 25 17:44:33 2013 From: skip at pobox.com (Skip Montanaro) Date: Mon, 25 Feb 2013 10:44:33 -0600 Subject: [Python-Dev] [Python-checkins] peps: Pre-alpha draft for PEP 435 (enum). The name is not important at the moment, as In-Reply-To: <512B8BE4.5040102@stoneleaf.us> References: <3ZCsSP0rSXzQZS@mail.python.org> <20130223150231.GA20921@sleipnir.bytereef.org> <20130224204000.6309bd6c@anarchist.wooz.org> <512ADB41.5040109@stoneleaf.us> <20130224231410.0023038c@anarchist.wooz.org> <512B7802.9010306@stoneleaf.us> <512B8BE4.5040102@stoneleaf.us> Message-ID: > Besides "we just don't need them int-based in these use-cases" what are the > reasons for the strong desire to have them be valueless? Not sure about other people, but piggybacking off C semantics, while convenient, reflects the limitations of the C implementation more than anything else. An enumeration, while you can count its items, doesn't imply that the elements of the enumeration are naturally ordered. If you force an underlying association with integers, you imply ordering where none naturally exists. Given this: critters = enum(DOG, CAT, RACCOON) what does it mean that DOG < CAT? Python 3 got rid of a lot of nonsense comparisons: ~% python2.7 Python 2.7.3+ (2.7:df57314b93d1, Feb 24 2013, 23:25:05) [GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> 1 < None False >>> ~% python3.3 Python 3.3.0+ (3.3:aae2bb2e3195, Feb 24 2013, 23:15:28) [GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> 1 < None Traceback (most recent call last): File "", line 1, in TypeError: unorderable types: int() < NoneType() >>> It's not clear to me that sneaking artificial ordering in the back door via enumerations is the correct thing to do. Skip From solipsis at pitrou.net Mon Feb 25 17:56:40 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 25 Feb 2013 17:56:40 +0100 Subject: [Python-Dev] [Python-checkins] peps: Pre-alpha draft for PEP 435 (enum). The name is not important at the moment, as References: <3ZCsSP0rSXzQZS@mail.python.org> <20130223150231.GA20921@sleipnir.bytereef.org> <20130224204000.6309bd6c@anarchist.wooz.org> <512ADB41.5040109@stoneleaf.us> <20130224231410.0023038c@anarchist.wooz.org> <512B7802.9010306@stoneleaf.us> <512B8BE4.5040102@stoneleaf.us> Message-ID: <20130225175640.043d4d79@pitrou.net> Le Mon, 25 Feb 2013 10:44:33 -0600, Skip Montanaro a ?crit : > > Besides "we just don't need them int-based in these use-cases" what > > are the reasons for the strong desire to have them be valueless? > > Not sure about other people, but piggybacking off C semantics, while > convenient, reflects the limitations of the C implementation more than > anything else. An enumeration, while you can count its items, doesn't > imply that the elements of the enumeration are naturally ordered. If > you force an underlying association with integers, you imply ordering > where none naturally exists. Given this: > > critters = enum(DOG, CAT, RACCOON) > > what does it mean that DOG < CAT? It doesn't mean anything, but so what? "DOG" > "CAT" doesn't mean much either, and yet it's legal in Python. Regards Antoine. From eliben at gmail.com Mon Feb 25 18:03:06 2013 From: eliben at gmail.com (Eli Bendersky) Date: Mon, 25 Feb 2013 09:03:06 -0800 Subject: [Python-Dev] [Python-checkins] peps: Pre-alpha draft for PEP 435 (enum). The name is not important at the moment, as In-Reply-To: <20130225175640.043d4d79@pitrou.net> References: <3ZCsSP0rSXzQZS@mail.python.org> <20130223150231.GA20921@sleipnir.bytereef.org> <20130224204000.6309bd6c@anarchist.wooz.org> <512ADB41.5040109@stoneleaf.us> <20130224231410.0023038c@anarchist.wooz.org> <512B7802.9010306@stoneleaf.us> <512B8BE4.5040102@stoneleaf.us> <20130225175640.043d4d79@pitrou.net> Message-ID: On Mon, Feb 25, 2013 at 8:56 AM, Antoine Pitrou wrote: > Le Mon, 25 Feb 2013 10:44:33 -0600, > Skip Montanaro a ?crit : > > > Besides "we just don't need them int-based in these use-cases" what > > > are the reasons for the strong desire to have them be valueless? > > > > Not sure about other people, but piggybacking off C semantics, while > > convenient, reflects the limitations of the C implementation more than > > anything else. An enumeration, while you can count its items, doesn't > > imply that the elements of the enumeration are naturally ordered. If > > you force an underlying association with integers, you imply ordering > > where none naturally exists. Given this: > > > > critters = enum(DOG, CAT, RACCOON) > > > > what does it mean that DOG < CAT? > > It doesn't mean anything, but so what? "DOG" > "CAT" doesn't mean much > either, and yet it's legal in Python. > "DOG" > "CAT" invokes lexicographical comparison between two strings, a well-defined and sensical operations. It simply means that in a sorted list of strings, "CAT" will come before "DOG". This is different from an enumeration that attempts to (at least logically) restrict a value to a set of pre-defined entities. That said, I'm slowly starting to realize that enums try to impose a bit of static typing on Python, which may be the cause for so much friction :-/ Eli -------------- next part -------------- An HTML attachment was scrubbed... URL: From rdmurray at bitdance.com Mon Feb 25 18:10:36 2013 From: rdmurray at bitdance.com (R. David Murray) Date: Mon, 25 Feb 2013 12:10:36 -0500 Subject: [Python-Dev] [Python-checkins] peps: Pre-alpha draft for PEP 435 (enum). The name is not important at the moment, as In-Reply-To: References: <3ZCsSP0rSXzQZS@mail.python.org> <20130223150231.GA20921@sleipnir.bytereef.org> <20130224204000.6309bd6c@anarchist.wooz.org> <512ADB41.5040109@stoneleaf.us> <20130224231410.0023038c@anarchist.wooz.org> <512B7802.9010306@stoneleaf.us> <512B8BE4.5040102@stoneleaf.us> <20130225175640.043d4d79@pitrou.net> Message-ID: <20130225171037.1EFC5250BD2@webabinitio.net> On Mon, 25 Feb 2013 09:03:06 -0800, Eli Bendersky wrote: > On Mon, Feb 25, 2013 at 8:56 AM, Antoine Pitrou wrote: > > > Le Mon, 25 Feb 2013 10:44:33 -0600, > > Skip Montanaro a ??crit : > > > > Besides "we just don't need them int-based in these use-cases" what > > > > are the reasons for the strong desire to have them be valueless? > > > > > > Not sure about other people, but piggybacking off C semantics, while > > > convenient, reflects the limitations of the C implementation more than > > > anything else. An enumeration, while you can count its items, doesn't > > > imply that the elements of the enumeration are naturally ordered. If > > > you force an underlying association with integers, you imply ordering > > > where none naturally exists. Given this: > > > > > > critters = enum(DOG, CAT, RACCOON) > > > > > > what does it mean that DOG < CAT? > > > > It doesn't mean anything, but so what? "DOG" > "CAT" doesn't mean much > > either, and yet it's legal in Python. > > > > "DOG" > "CAT" invokes lexicographical comparison between two strings, a > well-defined and sensical operations. It simply means that in a sorted list > of strings, "CAT" will come before "DOG". This is different from an > enumeration that attempts to (at least logically) restrict a value to a set > of pre-defined entities. > > That said, I'm slowly starting to realize that enums try to impose a bit of > static typing on Python, which may be the cause for so much friction :-/ If we had labeled values, none of this would be a problem. If you wanted ints, you could use labeled ints as the values. If you wanted strings, labeled strings. If you wanted incommensurate, unsortable objects, you could use labeled objects as the values. --David From python at mrabarnett.plus.com Mon Feb 25 18:19:55 2013 From: python at mrabarnett.plus.com (MRAB) Date: Mon, 25 Feb 2013 17:19:55 +0000 Subject: [Python-Dev] [Python-checkins] peps: Pre-alpha draft for PEP 435 (enum). The name is not important at the moment, as In-Reply-To: <20130225113729.235b86c6@anarchist.wooz.org> References: <3ZCsSP0rSXzQZS@mail.python.org> <20130223150231.GA20921@sleipnir.bytereef.org> <20130224204000.6309bd6c@anarchist.wooz.org> <512ADB41.5040109@stoneleaf.us> <20130224231410.0023038c@anarchist.wooz.org> <512B7802.9010306@stoneleaf.us> <512B7F43.5060406@stoneleaf.us> <20130225113729.235b86c6@anarchist.wooz.org> Message-ID: <512B9D3B.5020600@mrabarnett.plus.com> On 2013-02-25 16:37, Barry Warsaw wrote: > On Feb 25, 2013, at 07:12 AM, Ethan Furman wrote: > >>And if, in those places, the enums were based on ints (or strings), would it >>hurt you? Because in the places where I, as well as many others, use enums >>we *need* the int portion; and having to wrap the enum in an int() call is >>seven extra keystrokes (minor) and a heap of visual clutter (major), >>destroying any value the enum was supposed to have. > > Yes, I think enum values inheriting from int (or string) would hurt. > > First, as your question implies, which is it? int or str? Some people want > int some want str. It can't be both, and I don't think we need two enum > types. > > It's a deeper question though, because if enum values inherited from int, then > people would expect things like ``Colors.red == 1`` to work. Maybe you think > that doesn't look so bad, but that also implies: > > >>> Colors = make('Colors', 'red green blue'.split()) > >>> Animals = make('Animals', 'ant bee cat'.split()) > >>> Colors.green == Animals.bee > > and *that* I have a problem with. While I generally recommend that that enum > values be comparable by identity, not by equality, lots of people will still > use `==` instead of `is`. My preference: > > def utter(animal): > if animal is Animals.cat: > print('meow') > elif animal is Animals.bee: > print('buzz') > elif animal is Animals.ant: > print('fphfft') > > but commonly used: > > def utter(animal): > if animal == Animals.cat: > print('meow') > elif animal == Animals.bee: > print('buzz') > elif animal == Animals.ant: > print('fphfft') > Would we be doing either of those? Surely we would be using a dict instead, and what does a dict use, identity or equality? > In both cases, I want ``utter(Colors.green)`` and ``utter(2)`` to fail. In > the latter, if enum values are derived from ints, the second example will > succeed. Currently, in both cases ``utter(Animals.cat)`` succeeds. > > Even worse, if my library defines an Animals enum and your library defines a > different Animals enum, I do not want ``utter(YourAnimals.dog)`` to print > "meow" :). > > I get that you think wrapping the value in int() is ugly. I have a compromise > that you might find acceptable. EnumValues already have .enum and .name > attributes, to give you the value's class and string name respectively. It > would be trivial to add a .int attribute to return the value. > > Thus if you want the int value, it would be easy enough to say > ``Colors.green.int`` or ``Animals.bee.int`` > > https://bugs.launchpad.net/flufl.enum/+bug/1132859 > From barry at python.org Mon Feb 25 18:33:42 2013 From: barry at python.org (Barry Warsaw) Date: Mon, 25 Feb 2013 12:33:42 -0500 Subject: [Python-Dev] [Python-checkins] peps: Pre-alpha draft for PEP 435 (enum). The name is not important at the moment, as In-Reply-To: <512B9D3B.5020600@mrabarnett.plus.com> References: <3ZCsSP0rSXzQZS@mail.python.org> <20130223150231.GA20921@sleipnir.bytereef.org> <20130224204000.6309bd6c@anarchist.wooz.org> <512ADB41.5040109@stoneleaf.us> <20130224231410.0023038c@anarchist.wooz.org> <512B7802.9010306@stoneleaf.us> <512B7F43.5060406@stoneleaf.us> <20130225113729.235b86c6@anarchist.wooz.org> <512B9D3B.5020600@mrabarnett.plus.com> Message-ID: <20130225123342.5de928a9@anarchist.wooz.org> On Feb 25, 2013, at 05:19 PM, MRAB wrote: >Would we be doing either of those? Surely we would be using a dict >instead, and what does a dict use, identity or equality? It's just a contrived example for email brevity. In my real world examples, the code inside the conditions can be much more complicated and not conducive to dict usage. -Barry From ethan at stoneleaf.us Mon Feb 25 18:34:54 2013 From: ethan at stoneleaf.us (Ethan Furman) Date: Mon, 25 Feb 2013 09:34:54 -0800 Subject: [Python-Dev] [Python-checkins] peps: Pre-alpha draft for PEP 435 (enum). The name is not important at the moment, as In-Reply-To: References: <3ZCsSP0rSXzQZS@mail.python.org> <20130223150231.GA20921@sleipnir.bytereef.org> <20130224204000.6309bd6c@anarchist.wooz.org> <512ADB41.5040109@stoneleaf.us> <20130224231410.0023038c@anarchist.wooz.org> <512B7802.9010306@stoneleaf.us> <512B8BE4.5040102@stoneleaf.us> Message-ID: <512BA0BE.1010908@stoneleaf.us> On 02/25/2013 08:44 AM, Skip Montanaro wrote: >> Besides "we just don't need them int-based in these use-cases" what are the >> reasons for the strong desire to have them be valueless? > > Not sure about other people, but piggybacking off C semantics, while > convenient, reflects the limitations of the C implementation more than > anything else. An enumeration, while you can count its items, doesn't > imply that the elements of the enumeration are naturally ordered. If > you force an underlying association with integers, you imply ordering > where none naturally exists. Given this: > > critters = enum(DOG, CAT, RACCOON) > > what does it mean that DOG < CAT? Python 3 got rid of a lot of > nonsense comparisons: I certainly agree that there are some cases where an enumeration doesn't have any natural ordering; surely we can also agree that there are some that do? This is why I think our new Enum should have a selectable underlying type: int, bitmask (int is disguise ;), str, float -- whatever the developer needs. If ordering is not important, use str as your backend, or we could even have a truly valueless backend for those occasions. As far as not needing more than one type of enum: enumerations are used for counting and/or listing -- how many ways can we count? int, float, complex, decimal, fraction; how many ways can we list? list, tuple, array. I have no problem with having a valueless enum, or even with having it be the default -- so long as I can choose to use a SequenceEnum (or IntEnum, whatever we call it) then I'll be happy. -- ~Ethan~ From ethan at stoneleaf.us Mon Feb 25 18:35:58 2013 From: ethan at stoneleaf.us (Ethan Furman) Date: Mon, 25 Feb 2013 09:35:58 -0800 Subject: [Python-Dev] [Python-checkins] peps: Pre-alpha draft for PEP 435 (enum). The name is not important at the moment, as In-Reply-To: <20130225113729.235b86c6@anarchist.wooz.org> References: <3ZCsSP0rSXzQZS@mail.python.org> <20130223150231.GA20921@sleipnir.bytereef.org> <20130224204000.6309bd6c@anarchist.wooz.org> <512ADB41.5040109@stoneleaf.us> <20130224231410.0023038c@anarchist.wooz.org> <512B7802.9010306@stoneleaf.us> <512B7F43.5060406@stoneleaf.us> <20130225113729.235b86c6@anarchist.wooz.org> Message-ID: <512BA0FE.40206@stoneleaf.us> On 02/25/2013 08:37 AM, Barry Warsaw wrote: > On Feb 25, 2013, at 07:12 AM, Ethan Furman wrote: > >> And if, in those places, the enums were based on ints (or strings), would it >> hurt you? Because in the places where I, as well as many others, use enums >> we *need* the int portion; and having to wrap the enum in an int() call is >> seven extra keystrokes (minor) and a heap of visual clutter (major), >> destroying any value the enum was supposed to have. > > Yes, I think enum values inheriting from int (or string) would hurt. > > First, as your question implies, which is it? int or str? Some people want > int some want str. It can't be both, and I don't think we need two enum > types. I can see enums of at least three different types being useful: int, str, and bitmask; a valueless enum would be a fourth. > It's a deeper question though, because if enum values inherited from int, then > people would expect things like ``Colors.red == 1`` to work. Maybe you think > that doesn't look so bad, but that also implies: I do expect that to work. > >>> Colors = make('Colors', 'red green blue'.split()) > >>> Animals = make('Animals', 'ant bee cat'.split()) > >>> Colors.green == Animals.bee But this I don't, and in both mine, Ted's, and Alex's versions enums from different groups do not compare equal, regardless of the underlying value. Of course, this does have the potential problem of `green == 1 == bee` but not `green == bee` which would be a problem with set and dicts -- but I'm the only one who has brought up that issue. > I get that you think wrapping the value in int() is ugly. I have a compromise > that you might find acceptable. EnumValues already have .enum and .name > attributes, to give you the value's class and string name respectively. It > would be trivial to add a .int attribute to return the value. > > Thus if you want the int value, it would be easy enough to say > ``Colors.green.int`` or ``Animals.bee.int`` Well, it certainly isn't as ugly, and isn't as hard on the wrists. If your's is the package that makes it in the stdlib I would certainly appreciate the modification. However, as I replied to Skip, I think a stdlib solution should meet many needs, and sometimes (often, for some people ;) those needs will be better served by an int-based or str-based enum. -- ~Ethan~ From barry at python.org Mon Feb 25 19:05:02 2013 From: barry at python.org (Barry Warsaw) Date: Mon, 25 Feb 2013 13:05:02 -0500 Subject: [Python-Dev] [Python-checkins] peps: Pre-alpha draft for PEP 435 (enum). The name is not important at the moment, as In-Reply-To: <512BA0FE.40206@stoneleaf.us> References: <3ZCsSP0rSXzQZS@mail.python.org> <20130223150231.GA20921@sleipnir.bytereef.org> <20130224204000.6309bd6c@anarchist.wooz.org> <512ADB41.5040109@stoneleaf.us> <20130224231410.0023038c@anarchist.wooz.org> <512B7802.9010306@stoneleaf.us> <512B7F43.5060406@stoneleaf.us> <20130225113729.235b86c6@anarchist.wooz.org> <512BA0FE.40206@stoneleaf.us> Message-ID: <20130225130502.3d55e8b6@anarchist.wooz.org> On Feb 25, 2013, at 09:35 AM, Ethan Furman wrote: >> >>> Colors = make('Colors', 'red green blue'.split()) >> >>> Animals = make('Animals', 'ant bee cat'.split()) >> >>> Colors.green == Animals.bee > >But this I don't, and in both mine, Ted's, and Alex's versions enums from >different groups do not compare equal, regardless of the underlying value. >Of course, this does have the potential problem of `green == 1 == bee` but >not `green == bee` which would be a problem with set and dicts -- but I'm the >only one who has brought up that issue. Right, but my point is that if isinstance(Colors.green, int) then a reasonable interpretation is that enum values will compare equal against other ints. It seems weird to me that enum values *are* ints but aren't substitutable with ints (e.g. comparable against other ints including other subclasses of ints). >However, as I replied to Skip, I think a stdlib solution should meet many >needs, and sometimes (often, for some people ;) those needs will be better >served by an int-based or str-based enum. Maybe. I don't think a stdlib solution has to meet *all* needs. I think named values plus int-interoperable enums will solve almost all use cases. -Barry From ethan at stoneleaf.us Mon Feb 25 19:25:00 2013 From: ethan at stoneleaf.us (Ethan Furman) Date: Mon, 25 Feb 2013 10:25:00 -0800 Subject: [Python-Dev] [Python-checkins] peps: Pre-alpha draft for PEP 435 (enum). The name is not important at the moment, as In-Reply-To: <20130225130502.3d55e8b6@anarchist.wooz.org> References: <3ZCsSP0rSXzQZS@mail.python.org> <20130223150231.GA20921@sleipnir.bytereef.org> <20130224204000.6309bd6c@anarchist.wooz.org> <512ADB41.5040109@stoneleaf.us> <20130224231410.0023038c@anarchist.wooz.org> <512B7802.9010306@stoneleaf.us> <512B7F43.5060406@stoneleaf.us> <20130225113729.235b86c6@anarchist.wooz.org> <512BA0FE.40206@stoneleaf.us> <20130225130502.3d55e8b6@anarchist.wooz.org> Message-ID: <512BAC7C.3060003@stoneleaf.us> On 02/25/2013 10:05 AM, Barry Warsaw wrote: > On Feb 25, 2013, at 09:35 AM, Ethan Furman wrote: > >>> >>> Colors = make('Colors', 'red green blue'.split()) >>> >>> Animals = make('Animals', 'ant bee cat'.split()) >>> >>> Colors.green == Animals.bee >> >> But this I don't, and in both mine, Ted's, and Alex's versions enums from >> different groups do not compare equal, regardless of the underlying value. >> Of course, this does have the potential problem of `green == 1 == bee` but >> not `green == bee` which would be a problem with set and dicts -- but I'm the >> only one who has brought up that issue. > > Right, but my point is that if isinstance(Colors.green, int) then a reasonable > interpretation is that enum values will compare equal against other ints. It > seems weird to me that enum values *are* ints but aren't substitutable with > ints (e.g. comparable against other ints including other subclasses of ints). That is certainly a valid point. I will admit to being unhappy with the whole dict/set issue; I'm considering making my enums unhashable -- then if one wants them in a dict, one can take the string or int value and add it that way. >> However, as I replied to Skip, I think a stdlib solution should meet many >> needs, and sometimes (often, for some people ;) those needs will be better >> served by an int-based or str-based enum. > > Maybe. I don't think a stdlib solution has to meet *all* needs. I think > named values plus int-interoperable enums will solve almost all use cases. Hey, I think I just had a light-bulb moment: have the enum implement __index__ -- that will solve my use-case of using them for list indices. Dumb question, but are flufl.enums ordered? That's also an important use case. -- ~Ethan~ From solipsis at pitrou.net Mon Feb 25 19:49:23 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 25 Feb 2013 19:49:23 +0100 Subject: [Python-Dev] [Python-checkins] peps: Pre-alpha draft for PEP 435 (enum). The name is not important at the moment, as In-Reply-To: References: <3ZCsSP0rSXzQZS@mail.python.org> <20130223150231.GA20921@sleipnir.bytereef.org> <20130224204000.6309bd6c@anarchist.wooz.org> <512ADB41.5040109@stoneleaf.us> <20130224231410.0023038c@anarchist.wooz.org> <512B7802.9010306@stoneleaf.us> <512B8BE4.5040102@stoneleaf.us> <20130225175640.043d4d79@pitrou.net> Message-ID: <20130225194923.38359888@pitrou.net> On Mon, 25 Feb 2013 09:03:06 -0800 Eli Bendersky wrote: > > "DOG" > "CAT" invokes lexicographical comparison between two strings, a > well-defined and sensical operations. It simply means that in a sorted list > of strings, "CAT" will come before "DOG". This is different from an > enumeration that attempts to (at least logically) restrict a value to a set > of pre-defined entities. No, it's not different. Like there are use cases for ordered comparisons of strings, there are cases for ordered comparisons of enums. For example, if I have an enum representing SIP or HTTP response codes, it is perfectly reasonable to write: if code < 200: # temporary response ... elif code < 400: # successful final response ... else: # final error response: ... Really, there's no justification for claiming an enum should never compare to anything else. It's entirely application-dependent. Regards Antoine. From solipsis at pitrou.net Mon Feb 25 19:50:56 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 25 Feb 2013 19:50:56 +0100 Subject: [Python-Dev] [Python-checkins] peps: Pre-alpha draft for PEP 435 (enum). The name is not important at the moment, as References: <3ZCsSP0rSXzQZS@mail.python.org> <20130223150231.GA20921@sleipnir.bytereef.org> <20130224204000.6309bd6c@anarchist.wooz.org> <512ADB41.5040109@stoneleaf.us> <20130224231410.0023038c@anarchist.wooz.org> <512B7802.9010306@stoneleaf.us> <512B7F43.5060406@stoneleaf.us> <20130225113729.235b86c6@anarchist.wooz.org> <512BA0FE.40206@stoneleaf.us> <20130225130502.3d55e8b6@anarchist.wooz.org> Message-ID: <20130225195056.3797fc4b@pitrou.net> On Mon, 25 Feb 2013 13:05:02 -0500 Barry Warsaw wrote: > > Maybe. I don't think a stdlib solution has to meet *all* needs. I think > named values plus int-interoperable enums will solve almost all use cases. Being int-interoperable without comparing with ints is completely bonkers. Regards Antoine. From ethan at stoneleaf.us Mon Feb 25 20:07:13 2013 From: ethan at stoneleaf.us (Ethan Furman) Date: Mon, 25 Feb 2013 11:07:13 -0800 Subject: [Python-Dev] [Python-checkins] peps: Pre-alpha draft for PEP 435 (enum). The name is not important at the moment, as In-Reply-To: <20130225194923.38359888@pitrou.net> References: <3ZCsSP0rSXzQZS@mail.python.org> <20130223150231.GA20921@sleipnir.bytereef.org> <20130224204000.6309bd6c@anarchist.wooz.org> <512ADB41.5040109@stoneleaf.us> <20130224231410.0023038c@anarchist.wooz.org> <512B7802.9010306@stoneleaf.us> <512B8BE4.5040102@stoneleaf.us> <20130225175640.043d4d79@pitrou.net> <20130225194923.38359888@pitrou.net> Message-ID: <512BB661.70402@stoneleaf.us> On 02/25/2013 10:49 AM, Antoine Pitrou wrote: > On Mon, 25 Feb 2013 09:03:06 -0800 > Eli Bendersky wrote: >> >> "DOG" > "CAT" invokes lexicographical comparison between two strings, a >> well-defined and sensical operations. It simply means that in a sorted list >> of strings, "CAT" will come before "DOG". This is different from an >> enumeration that attempts to (at least logically) restrict a value to a set >> of pre-defined entities. > > No, it's not different. Like there are use cases for ordered > comparisons of strings, there are cases for ordered comparisons of > enums. > For example, if I have an enum representing SIP or HTTP response codes, > it is perfectly reasonable to write: > > if code < 200: > # temporary response > ... > elif code < 400: > # successful final response > ... > else: > # final error response: > ... > > Really, there's no justification for claiming an enum should never > compare to anything else. It's entirely application-dependent. +1 From ethan at stoneleaf.us Mon Feb 25 20:34:35 2013 From: ethan at stoneleaf.us (Ethan Furman) Date: Mon, 25 Feb 2013 11:34:35 -0800 Subject: [Python-Dev] [Python-checkins] peps: Pre-alpha draft for PEP 435 (enum). The name is not important at the moment, as In-Reply-To: <20130225194923.38359888@pitrou.net> References: <3ZCsSP0rSXzQZS@mail.python.org> <20130223150231.GA20921@sleipnir.bytereef.org> <20130224204000.6309bd6c@anarchist.wooz.org> <512ADB41.5040109@stoneleaf.us> <20130224231410.0023038c@anarchist.wooz.org> <512B7802.9010306@stoneleaf.us> <512B8BE4.5040102@stoneleaf.us> <20130225175640.043d4d79@pitrou.net> <20130225194923.38359888@pitrou.net> Message-ID: <512BBCCB.902@stoneleaf.us> Antoine, question for you: Do you think enums from different groupings should compare equal? If no, how would you propose to handle the following: 8<---------------------------------------------------------------------------- --> import yaenum --> class Color(yaenum.Enum): ... black ... red ... green ... blue ... --> class Literature(yaenum.Enum): ... scifi ... fantasy ... mystery ... pop ... --> Color.black Color('black', value=0) --> Literature.scifi Literature('scifi', value=0) --> black = Color.black --> scifi = Literature.scifi --> black == 0 True --> hash(black) 0 --> scifi == 0 True --> hash(scifi) 0 --> black == scifi False --> hash(0) 0 --> huh = dict() --> huh[black] = 9 --> huh {Color('black', value=0): 9} --> huh[0] 9 --> huh[scifi] Traceback (most recent call last): File "", line 1, in KeyError: Literature('scifi', value=0) --> huh[scifi] = 11 --> huh {Color('black', value=0): 9, Literature('scifi', value=0): 11} --> huh[0] 9 --> del huh[0] --> huh[0] 11 8<---------------------------------------------------------------------------- I do not think enums from different classes should compare equal. I see two ways around the above issue: 1) make enums unhashable, forcing the user to pick a hash method; 2) make the hash based on something else (like the enum's name) in which case the enum would not be /completely/ interoperable, even though it is a subclass of int From solipsis at pitrou.net Mon Feb 25 20:48:17 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 25 Feb 2013 20:48:17 +0100 Subject: [Python-Dev] [Python-checkins] peps: Pre-alpha draft for PEP 435 (enum). The name is not important at the moment, as References: <3ZCsSP0rSXzQZS@mail.python.org> <20130223150231.GA20921@sleipnir.bytereef.org> <20130224204000.6309bd6c@anarchist.wooz.org> <512ADB41.5040109@stoneleaf.us> <20130224231410.0023038c@anarchist.wooz.org> <512B7802.9010306@stoneleaf.us> <512B8BE4.5040102@stoneleaf.us> <20130225175640.043d4d79@pitrou.net> <20130225194923.38359888@pitrou.net> <512BBCCB.902@stoneleaf.us> Message-ID: <20130225204817.38b09100@pitrou.net> On Mon, 25 Feb 2013 11:34:35 -0800 Ethan Furman wrote: > Antoine, question for you: > > Do you think enums from different groupings should compare equal? Equality should be mostly transitive so, yes, I think they should. Regards Antoine. From benjamin at python.org Mon Feb 25 21:05:21 2013 From: benjamin at python.org (Benjamin Peterson) Date: Mon, 25 Feb 2013 15:05:21 -0500 Subject: [Python-Dev] Python 2.7.4 and 3.2.4 In-Reply-To: References: Message-ID: 2013/2/23 Perica Zivkovic : > Hi there, > > like posted here: > https://groups.google.com/forum/?fromgroups=#!topic/comp.lang.python/xXij443PM6I > > I'm curious to find out are there any timelines when 2.7.4 and 3.2.4 will be > available. As soon as there is a releasable 2.7 tree. Unfortunately, the XML security issue doesn't seem to be getting closer to resolution and we have tens of release blockers. > > I was kinda waiting for it to release new Portable Python so now I'm just > curious should I wait bit more and release on this python base or should I > go for 2.7.3 and 3.2.3 -- Regards, Benjamin From barry at python.org Mon Feb 25 21:32:47 2013 From: barry at python.org (Barry Warsaw) Date: Mon, 25 Feb 2013 15:32:47 -0500 Subject: [Python-Dev] [Python-checkins] peps: Pre-alpha draft for PEP 435 (enum). The name is not important at the moment, as In-Reply-To: <512BAC7C.3060003@stoneleaf.us> References: <3ZCsSP0rSXzQZS@mail.python.org> <20130223150231.GA20921@sleipnir.bytereef.org> <20130224204000.6309bd6c@anarchist.wooz.org> <512ADB41.5040109@stoneleaf.us> <20130224231410.0023038c@anarchist.wooz.org> <512B7802.9010306@stoneleaf.us> <512B7F43.5060406@stoneleaf.us> <20130225113729.235b86c6@anarchist.wooz.org> <512BA0FE.40206@stoneleaf.us> <20130225130502.3d55e8b6@anarchist.wooz.org> <512BAC7C.3060003@stoneleaf.us> Message-ID: <20130225153247.005e54a2@anarchist.wooz.org> On Feb 25, 2013, at 10:25 AM, Ethan Furman wrote: >Hey, I think I just had a light-bulb moment: have the enum implement >__index__ -- that will solve my use-case of using them for list indices. They don't currently, but I think it's reasonable, given that they already support __int__. https://bugs.launchpad.net/flufl.enum/+bug/1132972 >Dumb question, but are flufl.enums ordered? That's also an important use >case. Kind of. Ordered comparisons are explicitly not supported, but iteration over the Enum is guaranteed to be returned in int-value order. One thing I've been thinking about is allowing you to override the EnumValue class that the metaclass uses. In that case, if you really wanted ordered comparisons, you could override __lt__() and friends in a custom enum-value class. I haven't quite worked out in my mind how that would look, but I have a bug to track the feature request: https://bugs.launchpad.net/flufl.enum/+bug/1132976 Heck, that might even allow you to implement int-derived enum values if you really wanted them . Cheers, -Barry From timothy.c.delaney at gmail.com Tue Feb 26 00:04:40 2013 From: timothy.c.delaney at gmail.com (Tim Delaney) Date: Tue, 26 Feb 2013 10:04:40 +1100 Subject: [Python-Dev] [Python-checkins] peps: Pre-alpha draft for PEP 435 (enum). The name is not important at the moment, as In-Reply-To: <20130225153247.005e54a2@anarchist.wooz.org> References: <3ZCsSP0rSXzQZS@mail.python.org> <20130223150231.GA20921@sleipnir.bytereef.org> <20130224204000.6309bd6c@anarchist.wooz.org> <512ADB41.5040109@stoneleaf.us> <20130224231410.0023038c@anarchist.wooz.org> <512B7802.9010306@stoneleaf.us> <512B7F43.5060406@stoneleaf.us> <20130225113729.235b86c6@anarchist.wooz.org> <512BA0FE.40206@stoneleaf.us> <20130225130502.3d55e8b6@anarchist.wooz.org> <512BAC7C.3060003@stoneleaf.us> <20130225153247.005e54a2@anarchist.wooz.org> Message-ID: On 26 February 2013 07:32, Barry Warsaw wrote: > One thing I've been thinking about is allowing you to override the > EnumValue > class that the metaclass uses. In that case, if you really wanted ordered > comparisons, you could override __lt__() and friends in a custom enum-value > class. I haven't quite worked out in my mind how that would look, but I > have > a bug to track the feature request: > > https://bugs.launchpad.net/flufl.enum/+bug/1132976 > > Heck, that might even allow you to implement int-derived enum values if you > really wanted them . > You're starting to tread in an area that I investigated, did an implementation of, and then started moving away from due to a different approach (delegating to the methods in the owning Enum class when accessing EnumValue attribtues). I haven't touched my implementation for a couple of weeks now - been busy with other stuff and I got a bit fatigued with the discussion so I decided to wait until things had settled a bit. Hasn't happened yet ... ;) I'm actually in a quandry about what way I want my enums to go. I think each enum should have an ordinal based on the order it is defined, and should be ordered by that ordinal. But (whether or not it inherits from int - I'm ignoring string enums here) should __int__ and __index__ return the ordinal, or the assigned int value (if it has one)? There are arguments both ways. My current implementation doesn't have an ordinal at all (except by accident in the trivial case). That was when I decided to put it aside for a while and see where the discussion went. Tim Delaney -------------- next part -------------- An HTML attachment was scrubbed... URL: From glyph at twistedmatrix.com Tue Feb 26 00:17:06 2013 From: glyph at twistedmatrix.com (Glyph) Date: Mon, 25 Feb 2013 15:17:06 -0800 Subject: [Python-Dev] [Python-checkins] peps: Pre-alpha draft for PEP 435 (enum). The name is not important at the moment, as In-Reply-To: <20130225153247.005e54a2@anarchist.wooz.org> References: <3ZCsSP0rSXzQZS@mail.python.org> <20130223150231.GA20921@sleipnir.bytereef.org> <20130224204000.6309bd6c@anarchist.wooz.org> <512ADB41.5040109@stoneleaf.us> <20130224231410.0023038c@anarchist.wooz.org> <512B7802.9010306@stoneleaf.us> <512B7F43.5060406@stoneleaf.us> <20130225113729.235b86c6@anarchist.wooz.org> <512BA0FE.40206@stoneleaf.us> <20130225130502.3d55e8b6@anarchist.wooz.org> <512BAC7C.3060003@stoneleaf.us> <20130225153247.005e54a2@anarchist.wooz.org> Message-ID: <9A698345-D299-42FC-83B3-DABC5DAA9C25@twistedmatrix.com> On Feb 25, 2013, at 12:32 PM, Barry Warsaw wrote: >> Dumb question, but are flufl.enums ordered? That's also an important use >> case. > > Kind of. Ordered comparisons are explicitly not supported, but iteration over > the Enum is guaranteed to be returned in int-value order. Sorry to jump in to a random leaf of this thread, but there is such a barrage here I cannot find the beginning :). I can see in that Twisted is mentioned; it should probably reference and since we actually implemented a thing as well. (You can order constants by sorting them; off the top of my head, NamedConstant, ValueConstant, and FlagConstant all probably behave differently.) -g -------------- next part -------------- An HTML attachment was scrubbed... URL: From greg.ewing at canterbury.ac.nz Tue Feb 26 00:22:36 2013 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Tue, 26 Feb 2013 12:22:36 +1300 Subject: [Python-Dev] [Python-checkins] peps: Pre-alpha draft for PEP 435 (enum). The name is not important at the moment, as In-Reply-To: <512B7F43.5060406@stoneleaf.us> References: <3ZCsSP0rSXzQZS@mail.python.org> <20130223150231.GA20921@sleipnir.bytereef.org> <20130224204000.6309bd6c@anarchist.wooz.org> <512ADB41.5040109@stoneleaf.us> <20130224231410.0023038c@anarchist.wooz.org> <512B7802.9010306@stoneleaf.us> <512B7F43.5060406@stoneleaf.us> Message-ID: <512BF23C.7060105@canterbury.ac.nz> Ethan Furman wrote: > I must admit I find it mildly amusing (but a lot frustrating) that we > are talk about /enumerations/ not needing to be based on ints. Checking > out Merrian-Webster gives this: > > Definition of ENUMERATE > 1 > : to ascertain the number of : count > 2 > : to specify one after another : list It depends on what you mean by "based on ints". Pascal enums are using the second definition here: the values have ints associated with them, but are not themselves ints. There are really two different use cases. Sometimes you want to have values that are directly substitutable for ints; other times you would rather not, so as to catch mistakes more easily. -- Greg From ethan at stoneleaf.us Tue Feb 26 00:18:55 2013 From: ethan at stoneleaf.us (Ethan Furman) Date: Mon, 25 Feb 2013 15:18:55 -0800 Subject: [Python-Dev] [Python-checkins] peps: Pre-alpha draft for PEP 435 (enum). The name is not important at the moment, as In-Reply-To: <512BA0FE.40206@stoneleaf.us> References: <3ZCsSP0rSXzQZS@mail.python.org> <20130223150231.GA20921@sleipnir.bytereef.org> <20130224204000.6309bd6c@anarchist.wooz.org> <512ADB41.5040109@stoneleaf.us> <20130224231410.0023038c@anarchist.wooz.org> <512B7802.9010306@stoneleaf.us> <512B7F43.5060406@stoneleaf.us> <20130225113729.235b86c6@anarchist.wooz.org> <512BA0FE.40206@stoneleaf.us> Message-ID: <512BF15F.3010308@stoneleaf.us> On 02/25/2013 09:35 AM, Ethan Furman wrote: > But this I don't, and in both mine, Ted's, and Alex's versions [. . .] My sincerest apologies to Tim Delaney (aka 'Ted') for messing up his name. -- ~Ethan~ From ethan at stoneleaf.us Tue Feb 26 00:28:37 2013 From: ethan at stoneleaf.us (Ethan Furman) Date: Mon, 25 Feb 2013 15:28:37 -0800 Subject: [Python-Dev] [Python-checkins] peps: Pre-alpha draft for PEP 435 (enum). The name is not important at the moment, as In-Reply-To: <512BF23C.7060105@canterbury.ac.nz> References: <3ZCsSP0rSXzQZS@mail.python.org> <20130223150231.GA20921@sleipnir.bytereef.org> <20130224204000.6309bd6c@anarchist.wooz.org> <512ADB41.5040109@stoneleaf.us> <20130224231410.0023038c@anarchist.wooz.org> <512B7802.9010306@stoneleaf.us> <512B7F43.5060406@stoneleaf.us> <512BF23C.7060105@canterbury.ac.nz> Message-ID: <512BF3A5.5080000@stoneleaf.us> On 02/25/2013 03:22 PM, Greg Ewing wrote: > Ethan Furman wrote: >> I must admit I find it mildly amusing (but a lot frustrating) that we are talk about /enumerations/ not needing to be >> based on ints. Checking out Merrian-Webster gives this: >> >> Definition of ENUMERATE >> 1 >> : to ascertain the number of : count >> 2 >> : to specify one after another : list > > It depends on what you mean by "based on ints". Pascal > enums are using the second definition here: the values > have ints associated with them, but are not themselves > ints. > > There are really two different use cases. Sometimes you > want to have values that are directly substitutable for ints; > other times you would rather not, so as to catch mistakes > more easily. I agree. I think whichever enum ends up in the stdlib should support both types (and possibly a couple other variations). -- ~Ethan~ From greg.ewing at canterbury.ac.nz Tue Feb 26 00:53:03 2013 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Tue, 26 Feb 2013 12:53:03 +1300 Subject: [Python-Dev] [Python-checkins] peps: Pre-alpha draft for PEP 435 (enum). The name is not important at the moment, as In-Reply-To: <20130225113729.235b86c6@anarchist.wooz.org> References: <3ZCsSP0rSXzQZS@mail.python.org> <20130223150231.GA20921@sleipnir.bytereef.org> <20130224204000.6309bd6c@anarchist.wooz.org> <512ADB41.5040109@stoneleaf.us> <20130224231410.0023038c@anarchist.wooz.org> <512B7802.9010306@stoneleaf.us> <512B7F43.5060406@stoneleaf.us> <20130225113729.235b86c6@anarchist.wooz.org> Message-ID: <512BF95F.1050800@canterbury.ac.nz> Barry Warsaw wrote: > >>> Colors = make('Colors', 'red green blue'.split()) > >>> Animals = make('Animals', 'ant bee cat'.split()) > >>> Colors.green == Animals.bee The currently suggested solution to that seems to be to make comparison non-transitive, so that Colors.green == 1 and Animals.bee == 1 but Colors.green != Animals.bee. And then hope that this does not create a quantum black hole that sucks us all into a logical singularity... -- Greg From larry at hastings.org Tue Feb 26 01:11:23 2013 From: larry at hastings.org (Larry Hastings) Date: Mon, 25 Feb 2013 16:11:23 -0800 Subject: [Python-Dev] Announcing PEP 436: The Argument Clinic DSL Message-ID: <512BFDAB.5050103@hastings.org> Following up on a conversation on python-dev from last December: http://mail.python.org/pipermail/python-dev/2012-December/122920.html I'm pleased to announced PEP 436, proposing Argument Clinic for adoption into the CPython source tree. http://www.python.org/dev/peps/pep-0436/ Argument Clinic itself hasn't changed much in the intervening three months; I did add a prototype extension mechanism for adding user types at runtime, and I allow writing the generated code to a separate file. The PEP is adopted out of the "clinic.txt" included with the prototype, updated and with a new rationale. For what it's worth, the bug tracker issue is here: http://bugs.python.org/issue16612 I'm guessing python-dev is the right place for the ten-thousand-foot view topics: the merits of the specific proposed DSL syntax, the possible runtime extension API, and the approach as a whole. So for now let's just use the bug tracker issue for code reviews and the like. The prototype implementation is checked in here: https://bitbucket.org/larry/python-clinic/ as well as being posted to the above issue on the tracker in patch form. I look forward to your comments, //arry/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From barry at python.org Tue Feb 26 02:42:58 2013 From: barry at python.org (Barry Warsaw) Date: Mon, 25 Feb 2013 20:42:58 -0500 Subject: [Python-Dev] [Python-checkins] peps: Pre-alpha draft for PEP 435 (enum). The name is not important at the moment, as In-Reply-To: References: <3ZCsSP0rSXzQZS@mail.python.org> <20130223150231.GA20921@sleipnir.bytereef.org> <20130224204000.6309bd6c@anarchist.wooz.org> <512ADB41.5040109@stoneleaf.us> <20130224231410.0023038c@anarchist.wooz.org> <512B7802.9010306@stoneleaf.us> <512B7F43.5060406@stoneleaf.us> <20130225113729.235b86c6@anarchist.wooz.org> <512BA0FE.40206@stoneleaf.us> <20130225130502.3d55e8b6@anarchist.wooz.org> <512BAC7C.3060003@stoneleaf.us> <20130225153247.005e54a2@anarchist.wooz.org> Message-ID: <20130225204258.1346f941@anarchist.wooz.org> On Feb 26, 2013, at 10:04 AM, Tim Delaney wrote: >You're starting to tread in an area that I investigated, did an >implementation of, and then started moving away from due to a different >approach (delegating to the methods in the owning Enum class when accessing >EnumValue attribtues). At first glance, this turns out to be pretty easy in flufl.enum. https://code.launchpad.net/~barry/flufl.enum/lp1132976/+merge/150472 I'm not sure it's a good idea or whether it meets a real need, and you could bikeshed on the name of the magic attribute, but I think this shows that it's possible and doesn't complicate the implementation that much. Cheers, -Barry From stephen at xemacs.org Tue Feb 26 04:46:20 2013 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Tue, 26 Feb 2013 12:46:20 +0900 Subject: [Python-Dev] [Python-checkins] peps: Pre-alpha draft for PEP 435 (enum). The name is not important at the moment, as In-Reply-To: <512B7F43.5060406@stoneleaf.us> References: <3ZCsSP0rSXzQZS@mail.python.org> <20130223150231.GA20921@sleipnir.bytereef.org> <20130224204000.6309bd6c@anarchist.wooz.org> <512ADB41.5040109@stoneleaf.us> <20130224231410.0023038c@anarchist.wooz.org> <512B7802.9010306@stoneleaf.us> <512B7F43.5060406@stoneleaf.us> Message-ID: <87txoz3fg3.fsf@uwakimon.sk.tsukuba.ac.jp> Ethan Furman writes: > Again: Repeating yourself doesn't help make the case. It does, however, encourage me to reply. > Definition of ENUMERATE > 1 : to ascertain the number of : count > 2 : to specify one after another : list You say you need the value as an integer; when you evaluate, you are neither counting nor listing. Note that in both counting and listing the object of the operation is not an element. It is a set, and set membership is the most important aspect of the elements for that purpose. The values of the elements (including whether they even have a value) is not relevant to either of those operations. On the other hand, if you *are* going to access (constant!) values by name, I don't see any reason to restrict the domain to integers. Nor do such named constants need to be members of any set in many cases. From ethan at stoneleaf.us Tue Feb 26 07:53:43 2013 From: ethan at stoneleaf.us (Ethan Furman) Date: Mon, 25 Feb 2013 22:53:43 -0800 Subject: [Python-Dev] [Python-checkins] peps: Pre-alpha draft for PEP 435 (enum). The name is not important at the moment, as In-Reply-To: <87txoz3fg3.fsf@uwakimon.sk.tsukuba.ac.jp> References: <3ZCsSP0rSXzQZS@mail.python.org> <20130223150231.GA20921@sleipnir.bytereef.org> <20130224204000.6309bd6c@anarchist.wooz.org> <512ADB41.5040109@stoneleaf.us> <20130224231410.0023038c@anarchist.wooz.org> <512B7802.9010306@stoneleaf.us> <512B7F43.5060406@stoneleaf.us> <87txoz3fg3.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <512C5BF7.9090803@stoneleaf.us> On 02/25/2013 07:46 PM, Stephen J. Turnbull wrote: > Ethan Furman writes: > > > Again: > > Repeating yourself doesn't help make the case. It does, however, > encourage me to reply. Good! For a while I felt like I was talking to myself! ;) > > Definition of ENUMERATE > > 1 : to ascertain the number of : count > > 2 : to specify one after another : list > > You say you need the value as an integer; when you evaluate, you are > neither counting nor listing. That's like saying when I evaluate a dict's key I'm neither assigning nor iterating. True, but a fairly useless factoid. When I /create/ the enumeration (by hand, until a couple days ago), I listed them and counted them all myself. Now I list them and let the computer count them. And both then and now when I use them the computer appropriately substitutes the already counted value for the name I listed. That counted value is every bit as important (at least in my use cases) as the set membership. > Note that in both counting and listing > the object of the operation is not an element. It is a set, and set > membership is the most important aspect of the elements for that > purpose. No, it isn't. It may be in some cases. My enums even respect it to the point of not comparing equal to different enum sets. But, quite frankly, it's the enum's ability to let me use a name to mean a number that is the most important. And the most important thing for the Enum class is to assign those numbers for me as automatically as possible. > On the other hand, if you *are* going to access (constant!) values by > name, I don't see any reason to restrict the domain to integers. I completely agree. I plan on making my enums support a wide range of possibilities. So far it's just int (counting and bitmap) and str. > Nor do such named constants need to be members of any set in many cases. For those cases I plan on using Nick's namedvalues. But mostly I'll be using enumerations. -- ~Ethan~ From stefan at bytereef.org Tue Feb 26 09:30:44 2013 From: stefan at bytereef.org (Stefan Krah) Date: Tue, 26 Feb 2013 09:30:44 +0100 Subject: [Python-Dev] Announcing PEP 436: The Argument Clinic DSL In-Reply-To: <512BFDAB.5050103@hastings.org> References: <512BFDAB.5050103@hastings.org> Message-ID: <20130226083044.GA30336@sleipnir.bytereef.org> Larry Hastings wrote: > http://bugs.python.org/issue16612 > > I'm guessing python-dev is the right place for the ten-thousand-foot view > topics: the merits of the specific proposed DSL syntax, the possible runtime > extension API, and the approach as a whole. So for now let's just use the bug > tracker issue for code reviews and the like. For anyone who isn't following the issue: A PEP proposing a different DSL will be forthcoming either this or next weekend. Stefan Krah From stephen at xemacs.org Tue Feb 26 10:29:00 2013 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Tue, 26 Feb 2013 18:29:00 +0900 Subject: [Python-Dev] [Python-checkins] peps: Pre-alpha draft for PEP 435 (enum). The name is not important at the moment, as In-Reply-To: <512C5BF7.9090803@stoneleaf.us> References: <3ZCsSP0rSXzQZS@mail.python.org> <20130223150231.GA20921@sleipnir.bytereef.org> <20130224204000.6309bd6c@anarchist.wooz.org> <512ADB41.5040109@stoneleaf.us> <20130224231410.0023038c@anarchist.wooz.org> <512B7802.9010306@stoneleaf.us> <512B7F43.5060406@stoneleaf.us> <87txoz3fg3.fsf@uwakimon.sk.tsukuba.ac.jp> <512C5BF7.9090803@stoneleaf.us> Message-ID: <87mwur2zkz.fsf@uwakimon.sk.tsukuba.ac.jp> Ethan Furman writes: >>>>> sjt wrote: > > Note that in both counting and listing the object of the > > operation is not an element. It is a set, and set membership is > > the most important aspect of the elements for that purpose. > > No, it isn't. It may be in some cases. I'm referring only to the dictionary definitions you quote. It really sounds to me like your use case (and even more so, Antoine's) would be perfectly well-served by a named constant class that doesn't know anything about enumerability. > My enums even respect it to the point of not comparing equal to > different enum sets. But, quite frankly, it's the enum's ability > to let me use a name to mean a number that is the most > important. And the most important thing for the Enum class is to > assign those numbers for me as automatically as possible. But "mean a number" and "assigned automatically" are close to mutually exclusive (except that "as possible" gives you a lot of wiggle room). That is, if by "name means number" you imply that the numbers have semantics (as in Antoine's HTTP status code example), you care what the numbers are, and they can't be assigned automatically. OTOH, if the numbers are assigned very automatically (ie, completely implicitly), I have trouble with coming up with a use case where ability to use integer arithmetic (including comparisons other than equality and set membership) on them *by default* is anything but a trap. Eg, with a Likert scale for preference[1], I *do* want to order them, but (at least when I'm teaching statistics) I *really* *don't* want them added and subtracted without thinking about exactly what the operation means. And if you really need arithmetic and automatic assignment is good enough, what's wrong with their ordinary names (ie, 1, 2, 3, ...)? Footnotes: [1] A qualitative expression of strength of feeling ("hate" = 1, "dislike" = 2, "indifferent" = 3, "like" = 4, "love" = 5). From eliben at gmail.com Tue Feb 26 14:25:55 2013 From: eliben at gmail.com (Eli Bendersky) Date: Tue, 26 Feb 2013 05:25:55 -0800 Subject: [Python-Dev] [Python-checkins] peps: Pre-alpha draft for PEP 435 (enum). The name is not important at the moment, as In-Reply-To: <9A698345-D299-42FC-83B3-DABC5DAA9C25@twistedmatrix.com> References: <3ZCsSP0rSXzQZS@mail.python.org> <20130223150231.GA20921@sleipnir.bytereef.org> <20130224204000.6309bd6c@anarchist.wooz.org> <512ADB41.5040109@stoneleaf.us> <20130224231410.0023038c@anarchist.wooz.org> <512B7802.9010306@stoneleaf.us> <512B7F43.5060406@stoneleaf.us> <20130225113729.235b86c6@anarchist.wooz.org> <512BA0FE.40206@stoneleaf.us> <20130225130502.3d55e8b6@anarchist.wooz.org> <512BAC7C.3060003@stoneleaf.us> <20130225153247.005e54a2@anarchist.wooz.org> <9A698345-D299-42FC-83B3-DABC5DAA9C25@twistedmatrix.com> Message-ID: On Mon, Feb 25, 2013 at 3:17 PM, Glyph wrote: > > On Feb 25, 2013, at 12:32 PM, Barry Warsaw wrote: > > Dumb question, but are flufl.enums ordered? That's also an important use > case. > > > Kind of. Ordered comparisons are explicitly not supported, but iteration > over > the Enum is guaranteed to be returned in int-value order. > > > Sorry to jump in to a random leaf of this thread, but there is such a > barrage here I cannot find the beginning :). > > I can see in > that Twisted is mentioned; it should probably reference < > https://twistedmatrix.com/documents/current/api/twisted.python.constants.html> > and > since we actually implemented a thing as well. > > (You can order constants by sorting them; off the top of my head, > NamedConstant, ValueConstant, and FlagConstant all probably behave > differently.) > > Glyph, thanks for the input. I mentioned Twisted because in its code I found a number of places with simple string enumerations used to represent state. I was not aware of twisted.python.constants, but it doesn't appear that this module is used at least in the places I checked. Examples: * twisted/web/_newclient.py * twisted/web/http.py * twisted/protocol/sip.py In general, many protocols have some "state" instance var that's usually just a string, using either predefined constants or direct string literals. Eli -------------- next part -------------- An HTML attachment was scrubbed... URL: From eliben at gmail.com Tue Feb 26 15:12:35 2013 From: eliben at gmail.com (Eli Bendersky) Date: Tue, 26 Feb 2013 06:12:35 -0800 Subject: [Python-Dev] Announcing PEP 436: The Argument Clinic DSL In-Reply-To: <20130226083044.GA30336@sleipnir.bytereef.org> References: <512BFDAB.5050103@hastings.org> <20130226083044.GA30336@sleipnir.bytereef.org> Message-ID: On Tue, Feb 26, 2013 at 12:30 AM, Stefan Krah wrote: > Larry Hastings wrote: > > http://bugs.python.org/issue16612 > > > > I'm guessing python-dev is the right place for the ten-thousand-foot view > > topics: the merits of the specific proposed DSL syntax, the possible > runtime > > extension API, and the approach as a whole. So for now let's just use > the bug > > tracker issue for code reviews and the like. > > For anyone who isn't following the issue: A PEP proposing a different DSL > will be forthcoming either this or next weekend. > If the two proposals share at least the motivation, would it not be more constructive to just have them listed as alternatives in a single PEP? This could serve as a useful single source of information for the new proposed features. Eli -------------- next part -------------- An HTML attachment was scrubbed... URL: From eliben at gmail.com Tue Feb 26 15:27:20 2013 From: eliben at gmail.com (Eli Bendersky) Date: Tue, 26 Feb 2013 06:27:20 -0800 Subject: [Python-Dev] Announcing PEP 436: The Argument Clinic DSL In-Reply-To: <512BFDAB.5050103@hastings.org> References: <512BFDAB.5050103@hastings.org> Message-ID: On Mon, Feb 25, 2013 at 4:11 PM, Larry Hastings wrote: > > Following up on a conversation on python-dev from last December: > > http://mail.python.org/pipermail/python-dev/2012-December/122920.html > > I'm pleased to announced PEP 436, proposing Argument Clinic for adoption > into the CPython source tree. > > http://www.python.org/dev/peps/pep-0436/ > > You really need to provide some realistic usage examples in the PEP. It's hard to get a feel for the DSL just from looking at the spec. Eli -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefan at bytereef.org Tue Feb 26 15:40:19 2013 From: stefan at bytereef.org (Stefan Krah) Date: Tue, 26 Feb 2013 15:40:19 +0100 Subject: [Python-Dev] Announcing PEP 436: The Argument Clinic DSL In-Reply-To: References: <512BFDAB.5050103@hastings.org> <20130226083044.GA30336@sleipnir.bytereef.org> Message-ID: <20130226144019.GA806@sleipnir.bytereef.org> Eli Bendersky wrote: > For anyone who isn't following the issue: A PEP proposing a different DSL > will be forthcoming either this or next weekend. > > > If the two proposals share at least the motivation, would it not be more > constructive to just have them listed as alternatives in a single PEP? This > could serve as a useful single source of information for the new proposed > features. I find PEPs that propose multiple alternatives hard to read. My PEP won't be short: It will contain a token specifications, a yacc grammar and an attempt to specify the semantics of the DSL. Stefan Krah From tjreedy at udel.edu Tue Feb 26 15:50:00 2013 From: tjreedy at udel.edu (Terry Reedy) Date: Tue, 26 Feb 2013 09:50:00 -0500 Subject: [Python-Dev] [Python-checkins] peps: Pre-alpha draft for PEP 435 (enum). The name is not important at the moment, as In-Reply-To: <512BA0FE.40206@stoneleaf.us> References: <3ZCsSP0rSXzQZS@mail.python.org> <20130223150231.GA20921@sleipnir.bytereef.org> <20130224204000.6309bd6c@anarchist.wooz.org> <512ADB41.5040109@stoneleaf.us> <20130224231410.0023038c@anarchist.wooz.org> <512B7802.9010306@stoneleaf.us> <512B7F43.5060406@stoneleaf.us> <20130225113729.235b86c6@anarchist.wooz.org> <512BA0FE.40206@stoneleaf.us> Message-ID: On 2/25/2013 12:35 PM, Ethan Furman wrote: > But this I don't, and in both mine, Ted's, and Alex's versions enums > from different groups do not compare equal, regardless of the underlying > value. Of course, this does have the potential problem of `green == 1 > == bee` but not `green == bee` which would be a problem with set and > dicts -- but I'm the only one who has brought up that issue. I have not been following the discussion in detail so I missed that before. Breaking transitivity of equality a no-no. It is basic to thought and logic. decimal(0) == 0 == 0.0 != decimal(0) was a problem we finally fixed by removing the inequality above. http://bugs.python.org/issue4087 http://bugs.python.org/issue4090 We should NOT knowingly re-introduce the same problem again! If color and animal are isolated from each other, they should each be isolated from everything, including int. -- Terry Jan Reedy From tjreedy at udel.edu Tue Feb 26 16:01:21 2013 From: tjreedy at udel.edu (Terry Reedy) Date: Tue, 26 Feb 2013 10:01:21 -0500 Subject: [Python-Dev] [Python-checkins] peps: Pre-alpha draft for PEP 435 (enum). The name is not important at the moment, as In-Reply-To: <512BF95F.1050800@canterbury.ac.nz> References: <3ZCsSP0rSXzQZS@mail.python.org> <20130223150231.GA20921@sleipnir.bytereef.org> <20130224204000.6309bd6c@anarchist.wooz.org> <512ADB41.5040109@stoneleaf.us> <20130224231410.0023038c@anarchist.wooz.org> <512B7802.9010306@stoneleaf.us> <512B7F43.5060406@stoneleaf.us> <20130225113729.235b86c6@anarchist.wooz.org> <512BF95F.1050800@canterbury.ac.nz> Message-ID: On 2/25/2013 6:53 PM, Greg Ewing wrote: > Barry Warsaw wrote: >> >>> Colors = make('Colors', 'red green blue'.split()) >> >>> Animals = make('Animals', 'ant bee cat'.split()) >> >>> Colors.green == Animals.bee > > The currently suggested solution to that seems to be to > make comparison non-transitive, so that Colors.green == 1 > and Animals.bee == 1 but Colors.green != Animals.bee. > And then hope that this does not create a quantum black > hole that sucks us all into a logical singularity... But it will;-). To repeat myself, transitivity of equality is basic to thought, logic, and sets and we should not deliver Python with it broken. (The non-reflexivity of NAN is a different issue, but NANs are intentionally insane.) Decimal(0) == 0 == 0.0 != Decimal(0) != Fraction(0) == 0 was a problem we finally fixed by making integer-valued decimals compare equal to the same valued floats and fractions. In 3.3: >>> from decimal import Decimal as D >>> from fractions import Fraction as F >>> 0 == 0.0 == D(0) == F(0) True http://bugs.python.org/issue4087 http://bugs.python.org/issue4090 explain the practical problems. We should NOT knowingly go down this road again. If color and animal are isolated from each other, they should each be isolated from everything, including int. -- Terry Jan Reedy From tjreedy at udel.edu Tue Feb 26 16:09:20 2013 From: tjreedy at udel.edu (Terry Reedy) Date: Tue, 26 Feb 2013 10:09:20 -0500 Subject: [Python-Dev] [Python-checkins] peps: Pre-alpha draft for PEP 435 (enum). The name is not important at the moment, as In-Reply-To: <20130225204817.38b09100@pitrou.net> References: <3ZCsSP0rSXzQZS@mail.python.org> <20130223150231.GA20921@sleipnir.bytereef.org> <20130224204000.6309bd6c@anarchist.wooz.org> <512ADB41.5040109@stoneleaf.us> <20130224231410.0023038c@anarchist.wooz.org> <512B7802.9010306@stoneleaf.us> <512B8BE4.5040102@stoneleaf.us> <20130225175640.043d4d79@pitrou.net> <20130225194923.38359888@pitrou.net> <512BBCCB.902@stoneleaf.us> <20130225204817.38b09100@pitrou.net> Message-ID: On 2/25/2013 2:48 PM, Antoine Pitrou wrote: > On Mon, 25 Feb 2013 11:34:35 -0800 > Ethan Furman wrote: >> Antoine, question for you: >> >> Do you think enums from different groupings should compare equal? > > Equality should be mostly transitive so, yes, I think they should. Or if they do not, then they should neither compare equal to ints. One or the other, take your pick, or make both choices available. -- Terry Jan Reedy From fijall at gmail.com Tue Feb 26 16:13:44 2013 From: fijall at gmail.com (Maciej Fijalkowski) Date: Tue, 26 Feb 2013 17:13:44 +0200 Subject: [Python-Dev] cffi in stdlib Message-ID: Hello. I would like to discuss on the language summit a potential inclusion of cffi[1] into stdlib. This is a project Armin Rigo has been working for a while, with some input from other developers. It seems that the main reason why people would prefer ctypes over cffi these days is "because it's included in stdlib", which is not generally the reason I would like to hear. Our calls to not use C extensions and to use an FFI instead has seen very limited success with ctypes and quite a lot more since cffi got released. The API is fairly stable right now with minor changes going in and it'll definitely stablize until Python 3.4 release. Notable projects using it: * pypycore - gevent main loop ported to cffi * pgsql2cffi * sdl-cffi bindings * tls-cffi bindings * lxml-cffi port * pyzmq * cairo-cffi * a bunch of others So relatively a lot given that the project is not even a year old (it got 0.1 release in June). As per documentation, the advantages over ctypes: * The goal is to call C code from Python. You should be able to do so without learning a 3rd language: every alternative requires you to learn their own language (Cython, SWIG) or API (ctypes). So we tried to assume that you know Python and C and minimize the extra bits of API that you need to learn. * Keep all the Python-related logic in Python so that you don?t need to write much C code (unlike CPython native C extensions). * Work either at the level of the ABI (Application Binary Interface) or the API (Application Programming Interface). Usually, C libraries have a specified C API but often not an ABI (e.g. they may document a ?struct? as having at least these fields, but maybe more). (ctypes works at the ABI level, whereas Cython and native C extensions work at the API level.) * We try to be complete. For now some C99 constructs are not supported, but all C89 should be, including macros (and including macro ?abuses?, which you can manually wrap in saner-looking C functions). * We attempt to support both PyPy and CPython, with a reasonable path for other Python implementations like IronPython and Jython. * Note that this project is not about embedding executable C code in Python, unlike Weave. This is about calling existing C libraries from Python. so among other things, making a cffi extension gives you the same level of security as writing C (and unlike ctypes) and brings quite a bit more flexibility (API vs ABI issue) that let's you wrap arbitrary libraries, even those full of macros. Cheers, fijal .. [1]: http://cffi.readthedocs.org/en/release-0.5/ From ncoghlan at gmail.com Tue Feb 26 16:18:55 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 27 Feb 2013 01:18:55 +1000 Subject: [Python-Dev] Announcing PEP 436: The Argument Clinic DSL In-Reply-To: <20130226144019.GA806@sleipnir.bytereef.org> References: <512BFDAB.5050103@hastings.org> <20130226083044.GA30336@sleipnir.bytereef.org> <20130226144019.GA806@sleipnir.bytereef.org> Message-ID: On Wed, Feb 27, 2013 at 12:40 AM, Stefan Krah wrote: > Eli Bendersky wrote: >> For anyone who isn't following the issue: A PEP proposing a different DSL >> will be forthcoming either this or next weekend. >> >> >> If the two proposals share at least the motivation, would it not be more >> constructive to just have them listed as alternatives in a single PEP? This >> could serve as a useful single source of information for the new proposed >> features. > > I find PEPs that propose multiple alternatives hard to read. My PEP won't > be short: It will contain a token specifications, a yacc grammar and > an attempt to specify the semantics of the DSL. More importantly, the competing PEPs have different champions, so you and Larry need to be free to make the best case you can. Cheers, Nick. P.S. I was thinking the last time we had truly competing PEPs was way back when the with statement was designed, but it turns out the competing namespace package designs were also separate PEPs. PEP 315 was more an after the fact here's-a-better-way-to-implement-this redesign of PEP 309, and PEP 3150 vs 403 is just me having an ongoing discussion with myself (and occasionally others) over a feature I doubt would ever make it past Guido even if I do eventually decide to propose one of them. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Tue Feb 26 16:32:48 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 27 Feb 2013 01:32:48 +1000 Subject: [Python-Dev] [Python-checkins] peps: Pre-alpha draft for PEP 435 (enum). The name is not important at the moment, as In-Reply-To: References: <3ZCsSP0rSXzQZS@mail.python.org> <20130223150231.GA20921@sleipnir.bytereef.org> <20130224204000.6309bd6c@anarchist.wooz.org> <512ADB41.5040109@stoneleaf.us> <20130224231410.0023038c@anarchist.wooz.org> <512B7802.9010306@stoneleaf.us> <512B8BE4.5040102@stoneleaf.us> <20130225175640.043d4d79@pitrou.net> <20130225194923.38359888@pitrou.net> <512BBCCB.902@stoneleaf.us> <20130225204817.38b09100@pitrou.net> Message-ID: On Wed, Feb 27, 2013 at 1:09 AM, Terry Reedy wrote: > On 2/25/2013 2:48 PM, Antoine Pitrou wrote: >> >> On Mon, 25 Feb 2013 11:34:35 -0800 >> Ethan Furman wrote: >>> >>> Antoine, question for you: >>> >>> Do you think enums from different groupings should compare equal? >> >> >> Equality should be mostly transitive so, yes, I think they should. > > > Or if they do not, then they should neither compare equal to ints. One or > the other, take your pick, or make both choices available. I'm already working on too many different things to take a serious run at a labelled values proposal, but it is avoiding precisely this kind of problem that makes me favour a permissive solution that *doesn't* define any new behaviour for the instance, aside from changing the result of repr() and the contents of pickles. (You could potentially combine it with a "sensitive values" proposal that ensured repr() and other operations only displayed the label without showing the contents at all - this is also potentially useful for labelling large strings that may otherwise overwhelm debug output). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ethan at stoneleaf.us Tue Feb 26 16:33:14 2013 From: ethan at stoneleaf.us (Ethan Furman) Date: Tue, 26 Feb 2013 07:33:14 -0800 Subject: [Python-Dev] [Python-checkins] peps: Pre-alpha draft for PEP 435 (enum). The name is not important at the moment, as In-Reply-To: References: <3ZCsSP0rSXzQZS@mail.python.org> <20130223150231.GA20921@sleipnir.bytereef.org> <20130224204000.6309bd6c@anarchist.wooz.org> <512ADB41.5040109@stoneleaf.us> <20130224231410.0023038c@anarchist.wooz.org> <512B7802.9010306@stoneleaf.us> <512B7F43.5060406@stoneleaf.us> <20130225113729.235b86c6@anarchist.wooz.org> <512BF95F.1050800@canterbury.ac.nz> Message-ID: <512CD5BA.9060901@stoneleaf.us> On 02/26/2013 07:01 AM, Terry Reedy wrote: > On 2/25/2013 6:53 PM, Greg Ewing wrote: >> The currently suggested solution to that seems to be to >> make comparison non-transitive, so that Colors.green == 1 >> and Animals.bee == 1 but Colors.green != Animals.bee. >> And then hope that this does not create a quantum black >> hole that sucks us all into a logical singularity... > > But it will;-). > To repeat myself, transitivity of equality is basic to thought, logic, and sets and we should not deliver Python with it > broken. (The non-reflexivity of NAN is a different issue, but NANs are intentionally insane.) > > Decimal(0) == 0 == 0.0 != Decimal(0) != Fraction(0) == 0 > was a problem we finally fixed by making integer-valued decimals > compare equal to the same valued floats and fractions. In 3.3: > >>>> from decimal import Decimal as D >>>> from fractions import Fraction as F >>>> 0 == 0.0 == D(0) == F(0) > True > > http://bugs.python.org/issue4087 > http://bugs.python.org/issue4090 > explain the practical problems. We should NOT knowingly go down this road again. If color and animal are isolated from > each other, they should each be isolated from everything, including int. Thank you, Terry. -- ~Ethan~ From ncoghlan at gmail.com Tue Feb 26 16:42:46 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 27 Feb 2013 01:42:46 +1000 Subject: [Python-Dev] cffi in stdlib In-Reply-To: References: Message-ID: On Wed, Feb 27, 2013 at 1:13 AM, Maciej Fijalkowski wrote: > Hello. > > I would like to discuss on the language summit a potential inclusion > of cffi[1] into stdlib. I think cffi is well worth considering as a possible inclusion for Python 3.4. (In particular, I'm a fan of the fact it just uses C syntax to declare what you're trying to talk to) If I'm reading the dependencies correctly, we would also need to bring Eli Bendersky's pycparser into the stdlib, correct? (not an objection, just an observation, although we'd obviously need Eli's explicit endorsement for that). There would also be the matter of converting the cffi test suite from py.test to stdlib unittest. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From tjreedy at udel.edu Tue Feb 26 17:11:12 2013 From: tjreedy at udel.edu (Terry Reedy) Date: Tue, 26 Feb 2013 11:11:12 -0500 Subject: [Python-Dev] Announcing PEP 436: The Argument Clinic DSL In-Reply-To: <512BFDAB.5050103@hastings.org> References: <512BFDAB.5050103@hastings.org> Message-ID: On 2/25/2013 7:11 PM, Larry Hastings wrote: > > Following up on a conversation on python-dev from last December: > > http://mail.python.org/pipermail/python-dev/2012-December/122920.html > > I'm pleased to announced PEP 436, proposing Argument Clinic for adoption > into the CPython source tree. > > http://www.python.org/dev/peps/pep-0436/ The PEP gives an internal, developer-focused rationale. I think there is also an external, user-focused rationale. As much as possible (with obvious caveats about type introspection), I think it should be transparent to users (other than speed) whether a function is implemented in Python or C. People who love keyword calling currently have to learn which module by module and function by function. Tranparency will become even more important as more modules are dual coded, and as users move code between implementations. I presume some of the complexities of PyArg_ParseTupleAndKeywords arise from pursuing this goal. And if I understand correctly, this PEP should reduce some of the complexities. The 'future goal' of C function signature info will further aid transparency. This user rationale suggests that positional-only functions should be discouraged. It also suggests another future goal: a clinic backend that would instead output a skeleton python file with def header lines and docstrings, leaving the bodies to be filled in. One could also wish for the opposite: a python file processor that would convert def header lines and docstrings to a skeleton C file with clinic specifications. > Do we need to support tuple unpacking? Well, Python 3 dropped that for python functions, so it is not needed needed for conversions to or from python 3 files. I suspect the same rationale applies to C as well as Python functions: the unpacking can be done explicitly in the body, as needed. -- Terry Jan Reedy From donald.stufft at gmail.com Tue Feb 26 17:14:43 2013 From: donald.stufft at gmail.com (Donald Stufft) Date: Tue, 26 Feb 2013 11:14:43 -0500 Subject: [Python-Dev] cffi in stdlib In-Reply-To: References: Message-ID: A big +1 from me for cffi in the stdlib it's a great library. I just recently started using it to make bindings to a C library. I looked at the ctypes library, but haven't actually used it, because the docs confused me but with cffi I was able to get somewhere just by a liberal use of copy/paste from the C library's home page. The only C I know I figured out from working with cffi and fwiw even with that minimal amount of gained knowledge I still am confused by the Ctypes docs and where to even really start to try to use it. On Tuesday, February 26, 2013 at 10:13 AM, Maciej Fijalkowski wrote: > Hello. > > I would like to discuss on the language summit a potential inclusion > of cffi[1] into stdlib. This is a project Armin Rigo has been working > for a while, with some input from other developers. It seems that the > main reason why people would prefer ctypes over cffi these days is > "because it's included in stdlib", which is not generally the reason I > would like to hear. Our calls to not use C extensions and to use an > FFI instead has seen very limited success with ctypes and quite a lot > more since cffi got released. The API is fairly stable right now with > minor changes going in and it'll definitely stablize until Python 3.4 > release. Notable projects using it: > > * pypycore - gevent main loop ported to cffi > * pgsql2cffi > * sdl-cffi bindings > * tls-cffi bindings > * lxml-cffi port > * pyzmq > * cairo-cffi > * a bunch of others > > So relatively a lot given that the project is not even a year old (it > got 0.1 release in June). As per documentation, the advantages over > ctypes: > > * The goal is to call C code from Python. You should be able to do so > without learning a 3rd language: every alternative requires you to > learn their own language (Cython, SWIG) or API (ctypes). So we tried > to assume that you know Python and C and minimize the extra bits of > API that you need to learn. > > * Keep all the Python-related logic in Python so that you don?t need > to write much C code (unlike CPython native C extensions). > > * Work either at the level of the ABI (Application Binary Interface) > or the API (Application Programming Interface). Usually, C libraries > have a specified C API but often not an ABI (e.g. they may document a > ?struct? as having at least these fields, but maybe more). (ctypes > works at the ABI level, whereas Cython and native C extensions work at > the API level.) > > * We try to be complete. For now some C99 constructs are not > supported, but all C89 should be, including macros (and including > macro ?abuses?, which you can manually wrap in saner-looking C > functions). > > * We attempt to support both PyPy and CPython, with a reasonable path > for other Python implementations like IronPython and Jython. > > * Note that this project is not about embedding executable C code in > Python, unlike Weave. This is about calling existing C libraries from > Python. > > so among other things, making a cffi extension gives you the same > level of security as writing C (and unlike ctypes) and brings quite a > bit more flexibility (API vs ABI issue) that let's you wrap arbitrary > libraries, even those full of macros. > > Cheers, > fijal > > .. [1]: http://cffi.readthedocs.org/en/release-0.5/ > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org (mailto:Python-Dev at python.org) > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/donald.stufft%40gmail.com > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From fijall at gmail.com Tue Feb 26 17:26:10 2013 From: fijall at gmail.com (Maciej Fijalkowski) Date: Tue, 26 Feb 2013 18:26:10 +0200 Subject: [Python-Dev] cffi in stdlib In-Reply-To: References: Message-ID: On Tue, Feb 26, 2013 at 5:42 PM, Nick Coghlan wrote: > On Wed, Feb 27, 2013 at 1:13 AM, Maciej Fijalkowski wrote: >> Hello. >> >> I would like to discuss on the language summit a potential inclusion >> of cffi[1] into stdlib. > > I think cffi is well worth considering as a possible inclusion for > Python 3.4. (In particular, I'm a fan of the fact it just uses C > syntax to declare what you're trying to talk to) > > If I'm reading the dependencies correctly, we would also need to bring > Eli Bendersky's pycparser into the stdlib, correct? (not an objection, > just an observation, although we'd obviously need Eli's explicit > endorsement for that). Yes, although, depending on the circumstances it's possible to hide it somewhere in cffi and not make it an official stdlib API (then several rules does not apply). It also pulls in some version of ply (or a generated parser). > > There would also be the matter of converting the cffi test suite from > py.test to stdlib unittest. I volunteer to do that, if necessary. > > Cheers, > Nick. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From dirkjan at ochtman.nl Tue Feb 26 17:30:41 2013 From: dirkjan at ochtman.nl (Dirkjan Ochtman) Date: Tue, 26 Feb 2013 17:30:41 +0100 Subject: [Python-Dev] cffi in stdlib In-Reply-To: References: Message-ID: On Tue, Feb 26, 2013 at 4:13 PM, Maciej Fijalkowski wrote: > I would like to discuss on the language summit a potential inclusion > of cffi[1] into stdlib. This is a project Armin Rigo has been working > for a while, with some input from other developers. It seems that the > main reason why people would prefer ctypes over cffi these days is > "because it's included in stdlib", which is not generally the reason I > would like to hear. Our calls to not use C extensions and to use an > FFI instead has seen very limited success with ctypes and quite a lot > more since cffi got released. The API is fairly stable right now with > minor changes going in and it'll definitely stablize until Python 3.4 > release. I think this would be great to have in the stdlib. I think it's useful to note that PyPy is planning to include this in their stdlib for the next release, too, right? Cheers, Dirkjan From solipsis at pitrou.net Tue Feb 26 17:32:56 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 26 Feb 2013 17:32:56 +0100 Subject: [Python-Dev] cffi in stdlib References: Message-ID: <20130226173256.3b4b94d2@pitrou.net> Le Tue, 26 Feb 2013 17:13:44 +0200, Maciej Fijalkowski a ?crit : > Hello. > > I would like to discuss on the language summit a potential inclusion > of cffi[1] into stdlib. This is a project Armin Rigo has been working > for a while, with some input from other developers. It seems that the > main reason why people would prefer ctypes over cffi these days is > "because it's included in stdlib", which is not generally the reason I > would like to hear. Our calls to not use C extensions and to use an > FFI instead has seen very limited success with ctypes and quite a lot > more since cffi got released. The API is fairly stable right now with > minor changes going in and it'll definitely stablize until Python 3.4 > release. Do you intend to actually maintain it inside the CPython repository? Regards Antoine. From eliben at gmail.com Tue Feb 26 17:34:59 2013 From: eliben at gmail.com (Eli Bendersky) Date: Tue, 26 Feb 2013 08:34:59 -0800 Subject: [Python-Dev] cffi in stdlib In-Reply-To: References: Message-ID: On Tue, Feb 26, 2013 at 7:42 AM, Nick Coghlan wrote: > On Wed, Feb 27, 2013 at 1:13 AM, Maciej Fijalkowski > wrote: > > Hello. > > > > I would like to discuss on the language summit a potential inclusion > > of cffi[1] into stdlib. > > I think cffi is well worth considering as a possible inclusion for > Python 3.4. (In particular, I'm a fan of the fact it just uses C > syntax to declare what you're trying to talk to) > I'm cautiously +0.5 because I'd really like to see a strong comparison case being made vs. ctypes. I've used ctypes many times and it was easy and effortless (well, except the segfaults when wrong argument types are declared :-). I'll be really interesting in seeing concrete examples that demonstrate how CFFI is superior. > If I'm reading the dependencies correctly, we would also need to bring > Eli Bendersky's pycparser into the stdlib, correct? (not an objection, > just an observation, although we'd obviously need Eli's explicit > endorsement for that). > Of course I have no problem with this. I obviously signed the contributor agreement by this point ;-) pycparser depends on David Beazley's PLY though, so that will have to be pulled in as well. Eli -------------- next part -------------- An HTML attachment was scrubbed... URL: From ethan at stoneleaf.us Tue Feb 26 17:03:40 2013 From: ethan at stoneleaf.us (Ethan Furman) Date: Tue, 26 Feb 2013 08:03:40 -0800 Subject: [Python-Dev] [Python-checkins] peps: Pre-alpha draft for PEP 435 (enum). The name is not important at the moment, as In-Reply-To: <87mwur2zkz.fsf@uwakimon.sk.tsukuba.ac.jp> References: <3ZCsSP0rSXzQZS@mail.python.org> <20130223150231.GA20921@sleipnir.bytereef.org> <20130224204000.6309bd6c@anarchist.wooz.org> <512ADB41.5040109@stoneleaf.us> <20130224231410.0023038c@anarchist.wooz.org> <512B7802.9010306@stoneleaf.us> <512B7F43.5060406@stoneleaf.us> <87txoz3fg3.fsf@uwakimon.sk.tsukuba.ac.jp> <512C5BF7.9090803@stoneleaf.us> <87mwur2zkz.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <512CDCDC.2090509@stoneleaf.us> On 02/26/2013 01:29 AM, Stephen J. Turnbull wrote: > Ethan Furman writes: >> Stephen J. Turnbull wrote: >>> Note that in both counting and listing the object of the >>> operation is not an element. It is a set, and set membership is >>> the most important aspect of the elements for that purpose. >> >> No, it isn't. It may be in some cases. > > I'm referring only to the dictionary definitions you quote. Ah, okay. Although, going with the first definition -- "ascertain the number of" -- I still maintain that the number is equally important, otherwise it could be a simple set. > It really sounds to me like your use case (and even more so, > Antoine's) would be perfectly well-served by a named constant class > that doesn't know anything about enumerability. My enumeration needs tend to be sequential; hence, well served by a class that easily binds names and sequential natural numbers together. >> My enums even respect it to the point of not comparing equal to >> different enum sets. But, quite frankly, it's the enum's ability >> to let me use a name to mean a number that is the most >> important. And the most important thing for the Enum class is to >> assign those numbers for me as automatically as possible. > > But "mean a number" and "assigned automatically" are close to mutually > exclusive (except that "as possible" gives you a lot of wiggle room). My main "as possible" case is when a number (or short range of numbers) is reserved for future use; being able to easily skip to the next meaningful number is a Good Thing. > That is, if by "name means number" you imply that the numbers have > semantics (as in Antoine's HTTP status code example), you care what > the numbers are, and they can't be assigned automatically. For the status code example I would probably manually assign all the numbers, since there are so few and the gaps between so large. But for my use, which is mainly as list indices, all I have to do is list the names in the correct order -- so I do care what value they have, and they can be assigned automatically. > OTOH, if the numbers are assigned very automatically (ie, completely > implicitly), I have trouble with coming up with a use case where > ability to use integer arithmetic (including comparisons other than > equality and set membership) on them *by default* is anything but a > trap. Eg, with a Likert scale for preference[1], I *do* want to order > them, but (at least when I'm teaching statistics) I *really* *don't* > want them added and subtracted without thinking about exactly what the > operation means. Every time I do arithmetic with True/False I have to think about what's going on; I see no difference between that and doing arithmetic with Color.red. > And if you really need arithmetic and automatic assignment is good > enough, what's wrong with their ordinary names (ie, 1, 2, 3, ...)? > Because in one file CITY is in column 13, in another it's in column 15, in a third it's in column 7. Much better (easier, safer, more readable) to get the assignment done _once_ and then use CITY everywhere else. And an easy to use int-based Enum class makes that easy. (And before you ask, yes, I need all the other columns as well.) > Footnotes: > [1] A qualitative expression of strength of feeling ("hate" = 1, > "dislike" = 2, "indifferent" = 3, "like" = 4, "love" = 5). I'm beginning to see why enums as a class has not yet been added to Python. We don't want to complicate the language with too many choices, yet there is no One Obvious Enum to fit the wide variety of use-cases: - named int enums (http status codes) - named str enums (tkinter options) - named bitmask enums (file-type options) - named valueless enums (any random set of names) - named valueless-yet-orderable enums (any not-so-random set of names ;) -- ~Ethan~ From eliben at gmail.com Tue Feb 26 17:38:17 2013 From: eliben at gmail.com (Eli Bendersky) Date: Tue, 26 Feb 2013 08:38:17 -0800 Subject: [Python-Dev] cffi in stdlib In-Reply-To: References: Message-ID: On Tue, Feb 26, 2013 at 8:26 AM, Maciej Fijalkowski wrote: > On Tue, Feb 26, 2013 at 5:42 PM, Nick Coghlan wrote: > > On Wed, Feb 27, 2013 at 1:13 AM, Maciej Fijalkowski > wrote: > >> Hello. > >> > >> I would like to discuss on the language summit a potential inclusion > >> of cffi[1] into stdlib. > > > > I think cffi is well worth considering as a possible inclusion for > > Python 3.4. (In particular, I'm a fan of the fact it just uses C > > syntax to declare what you're trying to talk to) > > > > If I'm reading the dependencies correctly, we would also need to bring > > Eli Bendersky's pycparser into the stdlib, correct? (not an objection, > > just an observation, although we'd obviously need Eli's explicit > > endorsement for that). > > Yes, although, depending on the circumstances it's possible to hide it > somewhere in cffi and not make it an official stdlib API (then several > rules does not apply). It also pulls in some version of ply (or a > generated parser). > I'll be the first one to admit that pycparser is almost certainly not generally useful enough to be exposed in the stdlib. So just using it as an implementation detail is absolutely fine. PLY is a more interesting question, however, since PLY is somewhat more generally useful. That said, I see all this as implementation details that shouldn't distract us from the main point of whether cffi should be added. Eli -------------- next part -------------- An HTML attachment was scrubbed... URL: From fijall at gmail.com Tue Feb 26 17:38:14 2013 From: fijall at gmail.com (Maciej Fijalkowski) Date: Tue, 26 Feb 2013 18:38:14 +0200 Subject: [Python-Dev] cffi in stdlib In-Reply-To: <20130226173256.3b4b94d2@pitrou.net> References: <20130226173256.3b4b94d2@pitrou.net> Message-ID: On Tue, Feb 26, 2013 at 6:32 PM, Antoine Pitrou wrote: > Le Tue, 26 Feb 2013 17:13:44 +0200, > Maciej Fijalkowski a ?crit : >> Hello. >> >> I would like to discuss on the language summit a potential inclusion >> of cffi[1] into stdlib. This is a project Armin Rigo has been working >> for a while, with some input from other developers. It seems that the >> main reason why people would prefer ctypes over cffi these days is >> "because it's included in stdlib", which is not generally the reason I >> would like to hear. Our calls to not use C extensions and to use an >> FFI instead has seen very limited success with ctypes and quite a lot >> more since cffi got released. The API is fairly stable right now with >> minor changes going in and it'll definitely stablize until Python 3.4 >> release. > > Do you intend to actually maintain it inside the CPython repository? > > Regards > > Antoine. Once we put it in, yes, of course. Me Armin and Alex. From fijall at gmail.com Tue Feb 26 17:39:25 2013 From: fijall at gmail.com (Maciej Fijalkowski) Date: Tue, 26 Feb 2013 18:39:25 +0200 Subject: [Python-Dev] cffi in stdlib In-Reply-To: References: Message-ID: On Tue, Feb 26, 2013 at 6:34 PM, Eli Bendersky wrote: > > On Tue, Feb 26, 2013 at 7:42 AM, Nick Coghlan wrote: >> >> On Wed, Feb 27, 2013 at 1:13 AM, Maciej Fijalkowski >> wrote: >> > Hello. >> > >> > I would like to discuss on the language summit a potential inclusion >> > of cffi[1] into stdlib. >> >> I think cffi is well worth considering as a possible inclusion for >> Python 3.4. (In particular, I'm a fan of the fact it just uses C >> syntax to declare what you're trying to talk to) > > > I'm cautiously +0.5 because I'd really like to see a strong comparison case > being made vs. ctypes. I've used ctypes many times and it was easy and > effortless (well, except the segfaults when wrong argument types are > declared :-). I'll be really interesting in seeing concrete examples that > demonstrate how CFFI is superior. My main issue with ctypes, other than confusing API, which is up to taste, is that you just cannot wrap some libraries, like OpenSSL because of API vs ABI. OpenSSL uses macros extensively. Another point is that even C POSIX stdlib gives you incomplete structs and you have to guess where to put what fields. From arigo at tunes.org Tue Feb 26 17:47:55 2013 From: arigo at tunes.org (Armin Rigo) Date: Tue, 26 Feb 2013 17:47:55 +0100 Subject: [Python-Dev] cffi in stdlib In-Reply-To: References: <20130226173256.3b4b94d2@pitrou.net> Message-ID: Hi, On Tue, Feb 26, 2013 at 5:38 PM, Maciej Fijalkowski wrote: >> Do you intend to actually maintain it inside the CPython repository? > > Once we put it in, yes, of course. Me Armin and Alex. Yes, I confirm. :-) Armin From ncoghlan at gmail.com Tue Feb 26 17:49:13 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 27 Feb 2013 02:49:13 +1000 Subject: [Python-Dev] cffi in stdlib In-Reply-To: References: Message-ID: On Wed, Feb 27, 2013 at 2:39 AM, Maciej Fijalkowski wrote: > On Tue, Feb 26, 2013 at 6:34 PM, Eli Bendersky wrote: >> >> On Tue, Feb 26, 2013 at 7:42 AM, Nick Coghlan wrote: >>> >>> On Wed, Feb 27, 2013 at 1:13 AM, Maciej Fijalkowski >>> wrote: >>> > Hello. >>> > >>> > I would like to discuss on the language summit a potential inclusion >>> > of cffi[1] into stdlib. >>> >>> I think cffi is well worth considering as a possible inclusion for >>> Python 3.4. (In particular, I'm a fan of the fact it just uses C >>> syntax to declare what you're trying to talk to) >> >> >> I'm cautiously +0.5 because I'd really like to see a strong comparison case >> being made vs. ctypes. I've used ctypes many times and it was easy and >> effortless (well, except the segfaults when wrong argument types are >> declared :-). I'll be really interesting in seeing concrete examples that >> demonstrate how CFFI is superior. > > My main issue with ctypes, other than confusing API, which is up to > taste, is that you just cannot wrap some libraries, like OpenSSL > because of API vs ABI. OpenSSL uses macros extensively. Another point > is that even C POSIX stdlib gives you incomplete structs and you have > to guess where to put what fields. For me, it's mostly the fact that ctypes is *less* typesafe than C itself, because you can't easily consume the header files directly, which leads directly to the segfaults caused by wrong declarations. While at the time of inclusion there was a solid "practicality beats purity" case for adding ctypes (and it's taken us quite a long way), being less typesafe than C is still quite an achievement :) As an experienced C programmer, not having to learn a new hybrid language for declarations is also a plus, but the "significantly less likely to segfault" aspect is the big one. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Tue Feb 26 17:52:16 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 27 Feb 2013 02:52:16 +1000 Subject: [Python-Dev] [Python-checkins] peps: Pre-alpha draft for PEP 435 (enum). The name is not important at the moment, as In-Reply-To: <512CDCDC.2090509@stoneleaf.us> References: <3ZCsSP0rSXzQZS@mail.python.org> <20130223150231.GA20921@sleipnir.bytereef.org> <20130224204000.6309bd6c@anarchist.wooz.org> <512ADB41.5040109@stoneleaf.us> <20130224231410.0023038c@anarchist.wooz.org> <512B7802.9010306@stoneleaf.us> <512B7F43.5060406@stoneleaf.us> <87txoz3fg3.fsf@uwakimon.sk.tsukuba.ac.jp> <512C5BF7.9090803@stoneleaf.us> <87mwur2zkz.fsf@uwakimon.sk.tsukuba.ac.jp> <512CDCDC.2090509@stoneleaf.us> Message-ID: On Wed, Feb 27, 2013 at 2:03 AM, Ethan Furman wrote: > I'm beginning to see why enums as a class has not yet been added to Python. > We don't want to complicate the language with too many choices, yet there is > no One Obvious Enum to fit the wide variety of use-cases: > > - named int enums (http status codes) > - named str enums (tkinter options) > - named bitmask enums (file-type options) > - named valueless enums (any random set of names) > - named valueless-yet-orderable enums (any not-so-random set of names ;) That's probably the best succinct description of the core problem I've seen, and I've been following the various enum-related dicussions for years :) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From eliben at gmail.com Tue Feb 26 18:07:03 2013 From: eliben at gmail.com (Eli Bendersky) Date: Tue, 26 Feb 2013 09:07:03 -0800 Subject: [Python-Dev] cffi in stdlib In-Reply-To: References: Message-ID: On Tue, Feb 26, 2013 at 8:39 AM, Maciej Fijalkowski wrote: > On Tue, Feb 26, 2013 at 6:34 PM, Eli Bendersky wrote: > > > > On Tue, Feb 26, 2013 at 7:42 AM, Nick Coghlan > wrote: > >> > >> On Wed, Feb 27, 2013 at 1:13 AM, Maciej Fijalkowski > >> wrote: > >> > Hello. > >> > > >> > I would like to discuss on the language summit a potential inclusion > >> > of cffi[1] into stdlib. > >> > >> I think cffi is well worth considering as a possible inclusion for > >> Python 3.4. (In particular, I'm a fan of the fact it just uses C > >> syntax to declare what you're trying to talk to) > > > > > > I'm cautiously +0.5 because I'd really like to see a strong comparison > case > > being made vs. ctypes. I've used ctypes many times and it was easy and > > effortless (well, except the segfaults when wrong argument types are > > declared :-). I'll be really interesting in seeing concrete examples that > > demonstrate how CFFI is superior. > > My main issue with ctypes, other than confusing API, which is up to > taste, is that you just cannot wrap some libraries, like OpenSSL > because of API vs ABI. OpenSSL uses macros extensively. Another point > is that even C POSIX stdlib gives you incomplete structs and you have > to guess where to put what fields. > Yep, I can think of some reasons too. But just mentioning it so you remember explicitly listing the advantages when it comes to writing a PEP or some other sort of formal proposal. An FWIW, I think there's already enough positive feedback to at least start drafting a PEP. It can always be rejected later ;-) Eli -------------- next part -------------- An HTML attachment was scrubbed... URL: From fijall at gmail.com Tue Feb 26 18:14:34 2013 From: fijall at gmail.com (Maciej Fijalkowski) Date: Tue, 26 Feb 2013 19:14:34 +0200 Subject: [Python-Dev] cffi in stdlib In-Reply-To: References: Message-ID: On Tue, Feb 26, 2013 at 7:07 PM, Eli Bendersky wrote: > > > > On Tue, Feb 26, 2013 at 8:39 AM, Maciej Fijalkowski > wrote: >> >> On Tue, Feb 26, 2013 at 6:34 PM, Eli Bendersky wrote: >> > >> > On Tue, Feb 26, 2013 at 7:42 AM, Nick Coghlan >> > wrote: >> >> >> >> On Wed, Feb 27, 2013 at 1:13 AM, Maciej Fijalkowski >> >> wrote: >> >> > Hello. >> >> > >> >> > I would like to discuss on the language summit a potential inclusion >> >> > of cffi[1] into stdlib. >> >> >> >> I think cffi is well worth considering as a possible inclusion for >> >> Python 3.4. (In particular, I'm a fan of the fact it just uses C >> >> syntax to declare what you're trying to talk to) >> > >> > >> > I'm cautiously +0.5 because I'd really like to see a strong comparison >> > case >> > being made vs. ctypes. I've used ctypes many times and it was easy and >> > effortless (well, except the segfaults when wrong argument types are >> > declared :-). I'll be really interesting in seeing concrete examples >> > that >> > demonstrate how CFFI is superior. >> >> My main issue with ctypes, other than confusing API, which is up to >> taste, is that you just cannot wrap some libraries, like OpenSSL >> because of API vs ABI. OpenSSL uses macros extensively. Another point >> is that even C POSIX stdlib gives you incomplete structs and you have >> to guess where to put what fields. > > > Yep, I can think of some reasons too. But just mentioning it so you remember > explicitly listing the advantages when it comes to writing a PEP or some > other sort of formal proposal. An FWIW, I think there's already enough > positive feedback to at least start drafting a PEP. It can always be > rejected later ;-) > > Eli > It's copy-pasted in the original mail and it's in the docs. Anyway, this is just to familizarize people with the library. We're going to discuss it on the language summit at pycon. From dholth at gmail.com Tue Feb 26 18:21:40 2013 From: dholth at gmail.com (Daniel Holth) Date: Tue, 26 Feb 2013 12:21:40 -0500 Subject: [Python-Dev] cffi in stdlib In-Reply-To: References: Message-ID: +1 On Tue, Feb 26, 2013 at 12:14 PM, Maciej Fijalkowski wrote: > On Tue, Feb 26, 2013 at 7:07 PM, Eli Bendersky wrote: > > > > > > > > On Tue, Feb 26, 2013 at 8:39 AM, Maciej Fijalkowski > > wrote: > >> > >> On Tue, Feb 26, 2013 at 6:34 PM, Eli Bendersky > wrote: > >> > > >> > On Tue, Feb 26, 2013 at 7:42 AM, Nick Coghlan > >> > wrote: > >> >> > >> >> On Wed, Feb 27, 2013 at 1:13 AM, Maciej Fijalkowski < > fijall at gmail.com> > >> >> wrote: > >> >> > Hello. > >> >> > > >> >> > I would like to discuss on the language summit a potential > inclusion > >> >> > of cffi[1] into stdlib. > >> >> > >> >> I think cffi is well worth considering as a possible inclusion for > >> >> Python 3.4. (In particular, I'm a fan of the fact it just uses C > >> >> syntax to declare what you're trying to talk to) > >> > > >> > > >> > I'm cautiously +0.5 because I'd really like to see a strong comparison > >> > case > >> > being made vs. ctypes. I've used ctypes many times and it was easy and > >> > effortless (well, except the segfaults when wrong argument types are > >> > declared :-). I'll be really interesting in seeing concrete examples > >> > that > >> > demonstrate how CFFI is superior. > >> > >> My main issue with ctypes, other than confusing API, which is up to > >> taste, is that you just cannot wrap some libraries, like OpenSSL > >> because of API vs ABI. OpenSSL uses macros extensively. Another point > >> is that even C POSIX stdlib gives you incomplete structs and you have > >> to guess where to put what fields. > > > > > > Yep, I can think of some reasons too. But just mentioning it so you > remember > > explicitly listing the advantages when it comes to writing a PEP or some > > other sort of formal proposal. An FWIW, I think there's already enough > > positive feedback to at least start drafting a PEP. It can always be > > rejected later ;-) > > > > Eli > > > > It's copy-pasted in the original mail and it's in the docs. Anyway, > this is just to familizarize people with the library. We're going to > discuss it on the language summit at pycon. > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/dholth%40gmail.com > -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Tue Feb 26 19:14:26 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 26 Feb 2013 18:14:26 +0000 Subject: [Python-Dev] cffi in stdlib In-Reply-To: References: Message-ID: On 26 February 2013 16:34, Eli Bendersky wrote: > I'm cautiously +0.5 because I'd really like to see a strong comparison case > being made vs. ctypes. I've used ctypes many times and it was easy and > effortless (well, except the segfaults when wrong argument types are > declared :-). I'll be really interesting in seeing concrete examples that > demonstrate how CFFI is superior. I'm probably the same, mainly because I've successfully used ctypes in the past, but I've never used cffi. That's something I need to rectify. One point which I *think* is correct, but which I don't see noted anywhere. Am I right that cffi needs a C compiler involved in the process, at least somewhere? If that's the case, then it is not a suitable option for at least one use case that I have, writing quick hacks involving the Windows API on a machine that doesn't have a C compiler installed. Another possible case would be writing zip-safe code - if cffi involves a compiled C extension, it won't work when loaded from a zipfile. In general, a proper, unbiased comparison between cffi and ctypes would be useful. BTW, I assume that the intention is that both cffi and ctypes remain available indefinitely? Nobody's looking to deprecate ctypes? Paul From p.f.moore at gmail.com Tue Feb 26 19:38:16 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 26 Feb 2013 18:38:16 +0000 Subject: [Python-Dev] cffi in stdlib In-Reply-To: References: Message-ID: On 26 February 2013 18:34, Maciej Fijalkowski wrote: >> One point which I *think* is correct, but which I don't see noted >> anywhere. Am I right that cffi needs a C compiler involved in the >> process, at least somewhere? If that's the case, then it is not a >> suitable option for at least one use case that I have, writing quick >> hacks involving the Windows API on a machine that doesn't have a C >> compiler installed. Another possible case would be writing zip-safe >> code - if cffi involves a compiled C extension, it won't work when >> loaded from a zipfile. > > cffi does require a C compiler during either runtime or installation, > if you use the API. You can still use the ABI level, which is like > ctypes, only if you need the extra stuff, you need a C compiler. Cool, thanks. So if I use it like ctypes, cffi doesn't need a C compiler? That's nice, I never knew that. I definitely need to try it out. (So many projects, so little time... :-)) Paul From fijall at gmail.com Tue Feb 26 19:34:40 2013 From: fijall at gmail.com (Maciej Fijalkowski) Date: Tue, 26 Feb 2013 20:34:40 +0200 Subject: [Python-Dev] cffi in stdlib In-Reply-To: References: Message-ID: On Tue, Feb 26, 2013 at 8:14 PM, Paul Moore wrote: > On 26 February 2013 16:34, Eli Bendersky wrote: >> I'm cautiously +0.5 because I'd really like to see a strong comparison case >> being made vs. ctypes. I've used ctypes many times and it was easy and >> effortless (well, except the segfaults when wrong argument types are >> declared :-). I'll be really interesting in seeing concrete examples that >> demonstrate how CFFI is superior. > > I'm probably the same, mainly because I've successfully used ctypes in > the past, but I've never used cffi. That's something I need to > rectify. > > One point which I *think* is correct, but which I don't see noted > anywhere. Am I right that cffi needs a C compiler involved in the > process, at least somewhere? If that's the case, then it is not a > suitable option for at least one use case that I have, writing quick > hacks involving the Windows API on a machine that doesn't have a C > compiler installed. Another possible case would be writing zip-safe > code - if cffi involves a compiled C extension, it won't work when > loaded from a zipfile. cffi does require a C compiler during either runtime or installation, if you use the API. You can still use the ABI level, which is like ctypes, only if you need the extra stuff, you need a C compiler. > > In general, a proper, unbiased comparison between cffi and ctypes > would be useful. that obviously can't come from me. From larry at hastings.org Tue Feb 26 19:47:30 2013 From: larry at hastings.org (Larry Hastings) Date: Tue, 26 Feb 2013 10:47:30 -0800 Subject: [Python-Dev] Announcing PEP 436: The Argument Clinic DSL In-Reply-To: References: <512BFDAB.5050103@hastings.org> Message-ID: <512D0342.2060603@hastings.org> On 02/26/2013 08:11 AM, Terry Reedy wrote: > The PEP gives an internal, developer-focused rationale. I think there > is also an external, user-focused rationale. As much as possible (with > obvious caveats about type introspection), I think it should be > transparent to users (other than speed) whether a function is > implemented in Python or C. People who love keyword calling currently > have to learn which module by module and function by function. > Tranparency will become even more important as more modules are dual > coded, and as users move code between implementations. I presume some > of the complexities of PyArg_ParseTupleAndKeywords arise from pursuing > this goal. And if I understand correctly, this PEP should reduce some > of the complexities. The 'future goal' of C function signature info > will further aid transparency. > > This user rationale suggests that positional-only functions should be > discouraged. I think positional-only functions should be discouraged, but we don't really need Clinic to do that. I suppose we don't need Clinic to support signatures for built-in functions either, though Clinic would be a much bigger help in that department. So, here's my interpretation of the above suggestion: one user-focused rationale for Clinic is that it'll make it easier for developers to write built-ins that behave identically to functions written in Python, and this benefits users. Do you agree with that? > It also suggests another future goal: a clinic backend that would > instead output a skeleton python file with def header lines and > docstrings, leaving the bodies to be filled in. Well, right now Clinic has no visibility into variables, just functions. So such a skeleton python file would be a bit incomplete. That's mildly interesting to consider, though--telling Clinic about module variables, that is. Another possible use of Clinic is to do conformance testing for alternative implementations, that is, having PyPy (for example) make sure that they got their reimplemented standard library API correct. //arry/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From rdmurray at bitdance.com Tue Feb 26 20:06:19 2013 From: rdmurray at bitdance.com (R. David Murray) Date: Tue, 26 Feb 2013 14:06:19 -0500 Subject: [Python-Dev] cffi in stdlib In-Reply-To: References: Message-ID: <20130226190620.1FF24250BDA@webabinitio.net> On Tue, 26 Feb 2013 18:14:26 +0000, Paul Moore wrote: > BTW, I assume that the intention is that both cffi and ctypes remain > available indefinitely? Nobody's looking to deprecate ctypes? I would expect that ctypes would be deprecated eventually simply because there aren't very many people interested in maintaining it, and probably fewer if cffi is accepted. That said, I would not expect it to leave the stdlib until either the bit rot was so bad it wouldn't be worth shipping it, or (more likely) we reach Python4 and decide at that time that it is time for it to go. Of course, this is just me talking, we only have a *very* vague "sense of the house" for what Python4 means at this point :) --David From guido at python.org Tue Feb 26 20:12:21 2013 From: guido at python.org (Guido van Rossum) Date: Tue, 26 Feb 2013 11:12:21 -0800 Subject: [Python-Dev] cffi in stdlib In-Reply-To: <20130226190620.1FF24250BDA@webabinitio.net> References: <20130226190620.1FF24250BDA@webabinitio.net> Message-ID: Generally speaking, deferring something to Python 4 means "never". On Tue, Feb 26, 2013 at 11:06 AM, R. David Murray wrote: > On Tue, 26 Feb 2013 18:14:26 +0000, Paul Moore wrote: >> BTW, I assume that the intention is that both cffi and ctypes remain >> available indefinitely? Nobody's looking to deprecate ctypes? > > I would expect that ctypes would be deprecated eventually simply because > there aren't very many people interested in maintaining it, and probably > fewer if cffi is accepted. That said, I would not expect it to leave > the stdlib until either the bit rot was so bad it wouldn't be worth > shipping it, or (more likely) we reach Python4 and decide at that time > that it is time for it to go. > > Of course, this is just me talking, we only have a *very* vague "sense of > the house" for what Python4 means at this point :) > > --David > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org -- --Guido van Rossum (python.org/~guido) From brett at python.org Tue Feb 26 20:21:03 2013 From: brett at python.org (Brett Cannon) Date: Tue, 26 Feb 2013 14:21:03 -0500 Subject: [Python-Dev] cffi in stdlib In-Reply-To: References: <20130226190620.1FF24250BDA@webabinitio.net> Message-ID: On Tue, Feb 26, 2013 at 2:12 PM, Guido van Rossum wrote: > Generally speaking, deferring something to Python 4 means "never". > Does that mean your aversion to double digit version numbers (i.e. 3.10) is gone or you expect to freeze Python in carbonite by then? -Brett > > On Tue, Feb 26, 2013 at 11:06 AM, R. David Murray > wrote: > > On Tue, 26 Feb 2013 18:14:26 +0000, Paul Moore > wrote: > >> BTW, I assume that the intention is that both cffi and ctypes remain > >> available indefinitely? Nobody's looking to deprecate ctypes? > > > > I would expect that ctypes would be deprecated eventually simply because > > there aren't very many people interested in maintaining it, and probably > > fewer if cffi is accepted. That said, I would not expect it to leave > > the stdlib until either the bit rot was so bad it wouldn't be worth > > shipping it, or (more likely) we reach Python4 and decide at that time > > that it is time for it to go. > > > > Of course, this is just me talking, we only have a *very* vague "sense of > > the house" for what Python4 means at this point :) > > > > --David > > _______________________________________________ > > Python-Dev mailing list > > Python-Dev at python.org > > http://mail.python.org/mailman/listinfo/python-dev > > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/guido%40python.org > > > > -- > --Guido van Rossum (python.org/~guido) > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/brett%40python.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Tue Feb 26 20:17:57 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 26 Feb 2013 20:17:57 +0100 Subject: [Python-Dev] [Python-checkins] peps: Pre-alpha draft for PEP 435 (enum). The name is not important at the moment, as References: <3ZCsSP0rSXzQZS@mail.python.org> <20130223150231.GA20921@sleipnir.bytereef.org> <20130224204000.6309bd6c@anarchist.wooz.org> <512ADB41.5040109@stoneleaf.us> <20130224231410.0023038c@anarchist.wooz.org> <512B7802.9010306@stoneleaf.us> <512B7F43.5060406@stoneleaf.us> <87txoz3fg3.fsf@uwakimon.sk.tsukuba.ac.jp> <512C5BF7.9090803@stoneleaf.us> <87mwur2zkz.fsf@uwakimon.sk.tsukuba.ac.jp> <512CDCDC.2090509@stoneleaf.us> Message-ID: <20130226201757.535abe3e@pitrou.net> On Tue, 26 Feb 2013 08:03:40 -0800 Ethan Furman wrote: > I'm beginning to see why enums as a class has not yet been added to Python. We don't want to complicate the language > with too many choices, yet there is no One Obvious Enum to fit the wide variety of use-cases: > > - named int enums (http status codes) > - named str enums (tkinter options) > - named bitmask enums (file-type options) > - named valueless enums (any random set of names) > - named valueless-yet-orderable enums (any not-so-random set of names ;) I don't think valueless enums have a point. Just make them str enums. Regards Antoine. From guido at python.org Tue Feb 26 20:24:54 2013 From: guido at python.org (Guido van Rossum) Date: Tue, 26 Feb 2013 11:24:54 -0800 Subject: [Python-Dev] cffi in stdlib In-Reply-To: References: <20130226190620.1FF24250BDA@webabinitio.net> Message-ID: With 1.5 years per release, it'd be 10 years before we'd hit 3.10. >From a software engineering perspective, 10 years is indistinguishable from infinity, so I don't care what happens 10 years from now -- as long as you don't blame me. :-) On Tue, Feb 26, 2013 at 11:21 AM, Brett Cannon wrote: > > > > On Tue, Feb 26, 2013 at 2:12 PM, Guido van Rossum wrote: >> >> Generally speaking, deferring something to Python 4 means "never". > > > Does that mean your aversion to double digit version numbers (i.e. 3.10) is > gone or you expect to freeze Python in carbonite by then? > > -Brett > >> >> >> On Tue, Feb 26, 2013 at 11:06 AM, R. David Murray >> wrote: >> > On Tue, 26 Feb 2013 18:14:26 +0000, Paul Moore >> > wrote: >> >> BTW, I assume that the intention is that both cffi and ctypes remain >> >> available indefinitely? Nobody's looking to deprecate ctypes? >> > >> > I would expect that ctypes would be deprecated eventually simply because >> > there aren't very many people interested in maintaining it, and probably >> > fewer if cffi is accepted. That said, I would not expect it to leave >> > the stdlib until either the bit rot was so bad it wouldn't be worth >> > shipping it, or (more likely) we reach Python4 and decide at that time >> > that it is time for it to go. >> > >> > Of course, this is just me talking, we only have a *very* vague "sense >> > of >> > the house" for what Python4 means at this point :) >> > >> > --David >> > _______________________________________________ >> > Python-Dev mailing list >> > Python-Dev at python.org >> > http://mail.python.org/mailman/listinfo/python-dev >> > Unsubscribe: >> > http://mail.python.org/mailman/options/python-dev/guido%40python.org >> >> >> >> -- >> --Guido van Rossum (python.org/~guido) >> _______________________________________________ >> Python-Dev mailing list >> Python-Dev at python.org >> http://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe: >> http://mail.python.org/mailman/options/python-dev/brett%40python.org > > -- --Guido van Rossum (python.org/~guido) From solipsis at pitrou.net Tue Feb 26 20:21:53 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 26 Feb 2013 20:21:53 +0100 Subject: [Python-Dev] cffi in stdlib References: <20130226190620.1FF24250BDA@webabinitio.net> Message-ID: <20130226202153.2b923404@pitrou.net> On Tue, 26 Feb 2013 14:21:03 -0500 Brett Cannon wrote: > On Tue, Feb 26, 2013 at 2:12 PM, Guido van Rossum wrote: > > > Generally speaking, deferring something to Python 4 means "never". > > > > Does that mean your aversion to double digit version numbers (i.e. 3.10) is > gone or you expect to freeze Python in carbonite by then? Or we'll go straight to 5. (or switch to date-based numbering :-)) Regards Antoine. From ethan at stoneleaf.us Tue Feb 26 21:16:35 2013 From: ethan at stoneleaf.us (Ethan Furman) Date: Tue, 26 Feb 2013 12:16:35 -0800 Subject: [Python-Dev] [Python-checkins] peps: Pre-alpha draft for PEP 435 (enum). The name is not important at the moment, as In-Reply-To: <20130226201757.535abe3e@pitrou.net> References: <3ZCsSP0rSXzQZS@mail.python.org> <20130223150231.GA20921@sleipnir.bytereef.org> <20130224204000.6309bd6c@anarchist.wooz.org> <512ADB41.5040109@stoneleaf.us> <20130224231410.0023038c@anarchist.wooz.org> <512B7802.9010306@stoneleaf.us> <512B7F43.5060406@stoneleaf.us> <87txoz3fg3.fsf@uwakimon.sk.tsukuba.ac.jp> <512C5BF7.9090803@stoneleaf.us> <87mwur2zkz.fsf@uwakimon.sk.tsukuba.ac.jp> <512CDCDC.2090509@stoneleaf.us> <20130226201757.535abe3e@pitrou.net> Message-ID: <512D1823.3060201@stoneleaf.us> On 02/26/2013 11:17 AM, Antoine Pitrou wrote: > On Tue, 26 Feb 2013 08:03:40 -0800 > Ethan Furman wrote: >> I'm beginning to see why enums as a class has not yet been added to Python. We don't want to complicate the language >> with too many choices, yet there is no One Obvious Enum to fit the wide variety of use-cases: >> >> - named int enums (http status codes) >> - named str enums (tkinter options) >> - named bitmask enums (file-type options) >> - named valueless enums (any random set of names) >> - named valueless-yet-orderable enums (any not-so-random set of names ;) > > I don't think valueless enums have a point. Just make them str enums. If they're str enums then they would compare equal with strings and be strings and that may not be desired. -- ~Ethan~ From timothy.c.delaney at gmail.com Tue Feb 26 21:52:54 2013 From: timothy.c.delaney at gmail.com (Tim Delaney) Date: Wed, 27 Feb 2013 07:52:54 +1100 Subject: [Python-Dev] [Python-checkins] peps: Pre-alpha draft for PEP 435 (enum). The name is not important at the moment, as In-Reply-To: References: <3ZCsSP0rSXzQZS@mail.python.org> <20130223150231.GA20921@sleipnir.bytereef.org> <20130224204000.6309bd6c@anarchist.wooz.org> <512ADB41.5040109@stoneleaf.us> <20130224231410.0023038c@anarchist.wooz.org> <512B7802.9010306@stoneleaf.us> <512B7F43.5060406@stoneleaf.us> <20130225113729.235b86c6@anarchist.wooz.org> <512BA0FE.40206@stoneleaf.us> Message-ID: On 27 February 2013 01:50, Terry Reedy wrote: > On 2/25/2013 12:35 PM, Ethan Furman wrote: > > But this I don't, and in both mine, Ted's, and Alex's versions enums >> from different groups do not compare equal, regardless of the underlying >> value. Of course, this does have the potential problem of `green == 1 >> == bee` but not `green == bee` which would be a problem with set and >> dicts -- but I'm the only one who has brought up that issue. >> > > I have not been following the discussion in detail so I missed that > before. Breaking transitivity of equality a no-no. It is basic to thought > and logic. > > decimal(0) == 0 == 0.0 != decimal(0) > was a problem we finally fixed by removing the inequality above. > http://bugs.python.org/**issue4087 > http://bugs.python.org/**issue4090 > > We should NOT knowingly re-introduce the same problem again! If color and > animal are isolated from each other, they should each be isolated from > everything, including int. FWIW the only reason I made my enums int-based (and comparable with ints) was because I read somewhere that Guido had said that any stdlib enum would have to be an int subclass. I have no problems with having int-like enums that: 1. Are not int subclasses; 2. Do not compare equal with ints unless explicitly converted. I do think an int-like enum should implement both __int__ and __index__. Tim Delaney -------------- next part -------------- An HTML attachment was scrubbed... URL: From greg.ewing at canterbury.ac.nz Tue Feb 26 23:20:34 2013 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Wed, 27 Feb 2013 11:20:34 +1300 Subject: [Python-Dev] [Python-checkins] peps: Pre-alpha draft for PEP 435 (enum). The name is not important at the moment, as In-Reply-To: References: <3ZCsSP0rSXzQZS@mail.python.org> <20130223150231.GA20921@sleipnir.bytereef.org> <20130224204000.6309bd6c@anarchist.wooz.org> <512ADB41.5040109@stoneleaf.us> <20130224231410.0023038c@anarchist.wooz.org> <512B7802.9010306@stoneleaf.us> <512B7F43.5060406@stoneleaf.us> <20130225113729.235b86c6@anarchist.wooz.org> <512BF95F.1050800@canterbury.ac.nz> Message-ID: <512D3532.1010402@canterbury.ac.nz> Terry Reedy wrote: > (The > non-reflexivity of NAN is a different issue, but NANs are intentionally > insane.) Yes, the non-transitivity in that case only applies to one very special value. We're talking about making comparison non-transitive for *all* values of the types involved, which is a whole different level of weirdness. -- Greg From ethan at stoneleaf.us Tue Feb 26 22:55:57 2013 From: ethan at stoneleaf.us (Ethan Furman) Date: Tue, 26 Feb 2013 13:55:57 -0800 Subject: [Python-Dev] [Python-checkins] peps: Pre-alpha draft for PEP 435 (enum). The name is not important at the moment, as In-Reply-To: References: <3ZCsSP0rSXzQZS@mail.python.org> <20130223150231.GA20921@sleipnir.bytereef.org> <20130224204000.6309bd6c@anarchist.wooz.org> <512ADB41.5040109@stoneleaf.us> <20130224231410.0023038c@anarchist.wooz.org> <512B7802.9010306@stoneleaf.us> <512B7F43.5060406@stoneleaf.us> <20130225113729.235b86c6@anarchist.wooz.org> <512BA0FE.40206@stoneleaf.us> Message-ID: <512D2F6D.7020402@stoneleaf.us> On 02/26/2013 12:52 PM, Tim Delaney wrote: > On 27 February 2013 01:50, Terry Reedy > wrote: >> We should NOT knowingly re-introduce the same problem again! If color and animal are isolated from each other, they >> should each be isolated from everything, including int. > > FWIW the only reason I made my enums int-based (and comparable with ints) was because I read somewhere that Guido had > said that any stdlib enum would have to be an int subclass. I recall that as well. > I have no problems with having int-like enums that: > > 1. Are not int subclasses; > > 2. Do not compare equal with ints unless explicitly converted. > > I do think an int-like enum should implement both __int__ and __index__. I think it will take more than one type of Enum to handle the myriad needs. 1) Default enumeration: valueless, unordered: class BreakfastMeat(Enum): eggs = 'from a chicken' ham = 'from a pig' spam = 'from a ... um ... ' 2) non-valued, yet ordered: class DaysOfWeek(OrderedEnum): sunday = 'last day of weekend' monday = 'is it the 13th??' tuesday = 'not as bad as monday' wednesday = 'halfway through!' thursday = 'almost there...' friday = 'thank goodness' saturday = 'sleeping in!' 3) bitmask: sub-classes int, but not directly interactable (except for shifts) class DbfFieldOptions(BitMaskEnum): binary = 'no unicode translation' nullable = 'can store None' system = 'internal use only' 4) named string (based on str, basically a named constant) class TkGeometry(StrEnum): top = north = 'top' bottom = south = 'bottom' left = west = 'left' right = east = 'right' 5) named integer (based on int, basically a named constant) class RowLabels(IntEnum): name = 0 address = 1 city = 2 state = 3 zip = 4 In the first three examples the data in quotes is the doc string; in examples 4 and 5 the RHS is the actual value assigned. For that matter, 4 and 5 could be compacted into a NamedValue class, allowing whatever type is needed (since set membership is not restricted anyway): class Constants(NamedValues): top = 'top', 'top, or north, edge' bottom = 'bottom', 'bottom, or south, edge' pi = 3.14159, "circle's circumference divided by it's radius" lucky = 7, 'come on, 7!' unlucky = 2, 'snake eyes!' That would get us to three enum classes and one namedvalue class, and should cover 99% of the use cases out there. Does anyone still reading this thread have a use case not covered by the above? -- ~Ethan~ From greg.ewing at canterbury.ac.nz Wed Feb 27 00:15:05 2013 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Wed, 27 Feb 2013 12:15:05 +1300 Subject: [Python-Dev] cffi in stdlib In-Reply-To: <20130226202153.2b923404@pitrou.net> References: <20130226190620.1FF24250BDA@webabinitio.net> <20130226202153.2b923404@pitrou.net> Message-ID: <512D41F9.3050109@canterbury.ac.nz> Antoine Pitrou wrote: > Or we'll go straight to 5. > (or switch to date-based numbering :-)) We could go the Apple route and start naming them after species of snake. -- Greg From greg.ewing at canterbury.ac.nz Wed Feb 27 00:26:21 2013 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Wed, 27 Feb 2013 12:26:21 +1300 Subject: [Python-Dev] [Python-checkins] peps: Pre-alpha draft for PEP 435 (enum). The name is not important at the moment, as In-Reply-To: <512D2F6D.7020402@stoneleaf.us> References: <3ZCsSP0rSXzQZS@mail.python.org> <20130223150231.GA20921@sleipnir.bytereef.org> <20130224204000.6309bd6c@anarchist.wooz.org> <512ADB41.5040109@stoneleaf.us> <20130224231410.0023038c@anarchist.wooz.org> <512B7802.9010306@stoneleaf.us> <512B7F43.5060406@stoneleaf.us> <20130225113729.235b86c6@anarchist.wooz.org> <512BA0FE.40206@stoneleaf.us> <512D2F6D.7020402@stoneleaf.us> Message-ID: <512D449D.90504@canterbury.ac.nz> Ethan Furman wrote: > In the first three examples the data in quotes is the doc string; in > examples 4 and 5 the RHS is the actual value assigned. What if you want to assign both values and docstrings? -- Greg From ethan at stoneleaf.us Wed Feb 27 00:38:44 2013 From: ethan at stoneleaf.us (Ethan Furman) Date: Tue, 26 Feb 2013 15:38:44 -0800 Subject: [Python-Dev] [Python-checkins] peps: Pre-alpha draft for PEP 435 (enum). The name is not important at the moment, as In-Reply-To: <512D449D.90504@canterbury.ac.nz> References: <3ZCsSP0rSXzQZS@mail.python.org> <20130223150231.GA20921@sleipnir.bytereef.org> <20130224204000.6309bd6c@anarchist.wooz.org> <512ADB41.5040109@stoneleaf.us> <20130224231410.0023038c@anarchist.wooz.org> <512B7802.9010306@stoneleaf.us> <512B7F43.5060406@stoneleaf.us> <20130225113729.235b86c6@anarchist.wooz.org> <512BA0FE.40206@stoneleaf.us> <512D2F6D.7020402@stoneleaf.us> <512D449D.90504@canterbury.ac.nz> Message-ID: <512D4784.2070307@stoneleaf.us> On 02/26/2013 03:26 PM, Greg Ewing wrote: > Ethan Furman wrote: >> In the first three examples the data in quotes is the doc string; in examples >> 4 and 5 the RHS is the actual value assigned. > > What if you want to assign both values and docstrings? Let the bike shedding begin, eh? ;) It could be as simple as name = (0, 'customers full legal name') or as complex as name = enum(value=0, doc="customer's full legal name", xyz='some other attribute') -- ~Ethan~ From glyph at twistedmatrix.com Wed Feb 27 02:10:17 2013 From: glyph at twistedmatrix.com (Glyph) Date: Tue, 26 Feb 2013 17:10:17 -0800 Subject: [Python-Dev] [Python-checkins] peps: Pre-alpha draft for PEP 435 (enum). The name is not important at the moment, as In-Reply-To: References: <3ZCsSP0rSXzQZS@mail.python.org> <20130223150231.GA20921@sleipnir.bytereef.org> <20130224204000.6309bd6c@anarchist.wooz.org> <512ADB41.5040109@stoneleaf.us> <20130224231410.0023038c@anarchist.wooz.org> <512B7802.9010306@stoneleaf.us> <512B7F43.5060406@stoneleaf.us> <20130225113729.235b86c6@anarchist.wooz.org> <512BA0FE.40206@stoneleaf.us> <20130225130502.3d55e8b6@anarchist.wooz.org> <512BAC7C.3060003@stoneleaf.us> <20130225153247.005e54a2@anarchist.wooz.org> <9A698345-D299-42FC-83B3-DABC5DAA9C25@twistedmatrix.com> Message-ID: On Feb 26, 2013, at 5:25 AM, Eli Bendersky wrote: > Glyph, thanks for the input. I mentioned Twisted because in its code I found a number of places with simple string enumerations used to represent state. I was not aware of twisted.python.constants, but it doesn't appear that this module is used at least in the places I checked. Quite so. twisted.python.constants was created because we made the same observation that you did. Hopefully, more of these protocols will be transitioned to make use of twisted.python.constants internally. > In general, many protocols have some "state" instance var that's usually just a string, using either predefined constants or direct string literals. Indeed. This idiom varies considerably. Another thing that Twisted needs is a mechanism for explicitly building state-machine, but that's a discussion for another day. -glyph From tjreedy at udel.edu Wed Feb 27 02:39:08 2013 From: tjreedy at udel.edu (Terry Reedy) Date: Tue, 26 Feb 2013 20:39:08 -0500 Subject: [Python-Dev] cffi in stdlib In-Reply-To: References: Message-ID: On 2/26/2013 10:13 AM, Maciej Fijalkowski wrote: > I would like to discuss on the language summit a potential inclusion > of cffi[1] into stdlib. How does it compare in terms of speed. One reason ctypes has not replaces hand-tuned swig is that it apparently is much slower. I know that someone, for instance, once did a pygame fork using ctypes to wrap SDL, and got it to run, but gave it up for comparative slowness reasons. -- Terry Jan Reedy From tjreedy at udel.edu Wed Feb 27 03:30:36 2013 From: tjreedy at udel.edu (Terry Reedy) Date: Tue, 26 Feb 2013 21:30:36 -0500 Subject: [Python-Dev] Announcing PEP 436: The Argument Clinic DSL In-Reply-To: <512D0342.2060603@hastings.org> References: <512BFDAB.5050103@hastings.org> <512D0342.2060603@hastings.org> Message-ID: On 2/26/2013 1:47 PM, Larry Hastings wrote: > On 02/26/2013 08:11 AM, Terry Reedy wrote: >> The PEP gives an internal, developer-focused rationale. I think there >> is also an external, user-focused rationale. As much as possible (with >> obvious caveats about type introspection), I think it should be >> transparent to users (other than speed) whether a function is >> implemented in Python or C. People who love keyword calling currently >> have to learn which module by module and function by function. >> Tranparency will become even more important as more modules are dual >> coded, and as users move code between implementations. I presume some >> of the complexities of PyArg_ParseTupleAndKeywords arise from pursuing >> this goal. And if I understand correctly, this PEP should reduce some >> of the complexities. The 'future goal' of C function signature info >> will further aid transparency. >> >> This user rationale suggests that positional-only functions should be >> discouraged. > > I think positional-only functions should be discouraged, but we don't If I were writing something like Clinic, I would be tempted to not have that option. But I was actually thinking about something in the positional-only writeup that mentioned the possibility of adding something to the positional-only option. > really need Clinic to do that. I suppose we don't need Clinic to > support signatures for built-in functions either, though Clinic would be > a much bigger help in that department. > > So, here's my interpretation of the above suggestion: one user-focused > rationale for Clinic is that it'll make it easier for developers to > write built-ins that behave identically to functions written in Python, > and this benefits users. Do you agree with that? yes, especially with signatures. > >> It also suggests another future goal: a clinic backend that would >> instead output a skeleton python file with def header lines and >> docstrings, leaving the bodies to be filled in. > > Well, right now Clinic has no visibility into variables, just > functions. So such a skeleton python file would be a bit incomplete. Module variables are option few enough to copy by hand. Methods are more of a problem. As I understand it, C module files are structured something like the following, which is 'unusual' for a python file. def meth1_impl(... def meth2_impl(... class C: meth1 = meth1_impl meth2 = meth2_impl So producing an idiomatic skeleton would require more that an alternate backend. Rather function entries should be stored and not output until the structure (class) definition is encountered. Two passes would be a lot easier than one. > That's mildly interesting to consider, though--telling Clinic about > module variables, that is. Another possible use of Clinic is to do > conformance testing for alternative implementations, that is, having > PyPy (for example) make sure that they got their reimplemented standard > library API correct. > > > //arry/ > > -- Terry Jan Reedy From stephen at xemacs.org Wed Feb 27 04:16:52 2013 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Wed, 27 Feb 2013 12:16:52 +0900 Subject: [Python-Dev] [Python-checkins] peps: Pre-alpha draft for PEP 435 (enum). The name is not important at the moment, as In-Reply-To: <512CDCDC.2090509@stoneleaf.us> References: <3ZCsSP0rSXzQZS@mail.python.org> <20130223150231.GA20921@sleipnir.bytereef.org> <20130224204000.6309bd6c@anarchist.wooz.org> <512ADB41.5040109@stoneleaf.us> <20130224231410.0023038c@anarchist.wooz.org> <512B7802.9010306@stoneleaf.us> <512B7F43.5060406@stoneleaf.us> <87txoz3fg3.fsf@uwakimon.sk.tsukuba.ac.jp> <512C5BF7.9090803@stoneleaf.us> <87mwur2zkz.fsf@uwakimon.sk.tsukuba.ac.jp> <512CDCDC.2090509@stoneleaf.us> Message-ID: <87k3pu30pn.fsf@uwakimon.sk.tsukuba.ac.jp> Ethan Furman writes: > Ah, okay. Although, going with the first definition -- "ascertain > the number of" -- I still maintain that the number is equally > important, otherwise it could be a simple set. Of course "number" is central to counting -- but you can count a set of objects that have no numerical value. You can also count objects with numerical values that cannot be a count for that set: fractions, negative integers, etc. There's no necessary connection between the integers used for counting, and those that are values of the things counted. > But for my use, which is mainly as list indices, all I have to do > is list the names in the correct order -- so I do care what value > they have, and they can be assigned automatically. OK, that sounds like a valid use case to me. I don't mean to deprecate it, I just don't understand it because I'd never do that myself -- in all the cases where I care about the names and there's a variable order, the names are in the data. Eg, a CSV file with a header row. So I read them and need to create an inverse mapping. And in general, I need a string->index mapping for UI. So using an enum so I can hard-code column names doesn't help me much -- I'm doing a lot of string manipulation anyway. Worse for me, most of the applications I have, I'd like the enumerator identifiers to be both string-valued and int-valued: the int used to index into Python sequences, and the string used in formatting SQL. > Every time I do arithmetic with True/False I have to think about > what's going on; I see no difference between that and doing > arithmetic with Color.red. The pedagogical problem is that in a Likert scale "have to think" is a prescriptive requirement that students and even experienced researchers ignore all the time. They easily think of "hate" and "like" as the numbers 1 and 4, and they want to take averages across people. But that doesn't make sense if you examine behavior carefully: even though everybody agrees on order, the size of the steps and origin are different for different people. So I want to make it a logical constraint enforced by Enum, to be relaxed only if the user signs a no-damages release. I don't have an objection to making that easy by providing derived Enum types that have int or string values, or to a subclass of ordered Enums. Just that the base Enum type should be a collection of Python identifiers that you can't do much with except compare for identity, and operations would be added only with some kind of explicit indication like IntEnum, OrderedEnum, etc. From stephen at xemacs.org Wed Feb 27 04:33:35 2013 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Wed, 27 Feb 2013 12:33:35 +0900 Subject: [Python-Dev] [Python-checkins] peps: Pre-alpha draft for PEP 435 (enum). The name is not important at the moment, as In-Reply-To: <20130226201757.535abe3e@pitrou.net> References: <3ZCsSP0rSXzQZS@mail.python.org> <20130223150231.GA20921@sleipnir.bytereef.org> <20130224204000.6309bd6c@anarchist.wooz.org> <512ADB41.5040109@stoneleaf.us> <20130224231410.0023038c@anarchist.wooz.org> <512B7802.9010306@stoneleaf.us> <512B7F43.5060406@stoneleaf.us> <87txoz3fg3.fsf@uwakimon.sk.tsukuba.ac.jp> <512C5BF7.9090803@stoneleaf.us> <87mwur2zkz.fsf@uwakimon.sk.tsukuba.ac.jp> <512CDCDC.2090509@stoneleaf.us> <20130226201757.535abe3e@pitrou.net> Message-ID: <87ip5e2zxs.fsf@uwakimon.sk.tsukuba.ac.jp> Antoine Pitrou writes: > On Tue, 26 Feb 2013 08:03:40 -0800 > Ethan Furman wrote: > > I'm beginning to see why enums as a class has not yet been added > > to Python. We don't want to complicate the language with too > > many choices, yet there is no One Obvious Enum to fit the wide > > variety of use-cases: > > > > - named int enums (http status codes) > > - named str enums (tkinter options) > > - named bitmask enums (file-type options) > > - named valueless enums (any random set of names) > > - named valueless-yet-orderable enums (any not-so-random set of names ;) I don't see what the problem is with having a very restricted base Enum class and providing a number of convenience subclasses. > I don't think valueless enums have a point. Just make them str enums. I suppose Ethan doesn't mean "no value at all", he means "no operations available on the value" (ie, you need to explicitly convert from the enumerator to an int or str to add or compare them). str enums won't satisfy that requirement. They are implicitly orderable and addable. Furthermore, the str order is probably quite wrong. If the enum is orderable, I would expect declaration order. As far as I can see, what you (Antoine) want is an identifier with a constant value, no more and no less. Grouping into an enum is merely a lexical convention, since you are happy to compare enums of different enum classes and enums with ints. I think such a thing is pretty generally desirable, but most of the people who have created Enum classes implement some type-checking that this value came from that Enum. To me that is what differentiates an enumerated constant from a generic named constant. I don't see why you keep deprecating that feature. Do you really think it's useless, or is it just that you wouldn't use it yourself? From eliben at gmail.com Wed Feb 27 05:53:33 2013 From: eliben at gmail.com (Eli Bendersky) Date: Tue, 26 Feb 2013 20:53:33 -0800 Subject: [Python-Dev] cffi in stdlib In-Reply-To: References: Message-ID: * Work either at the level of the ABI (Application Binary Interface) > or the API (Application Programming Interface). Usually, C libraries > have a specified C API but often not an ABI (e.g. they may document a > ?struct? as having at least these fields, but maybe more). (ctypes > works at the ABI level, whereas Cython and native C extensions work at > the API level.) > I read the cffi docs once again and went through some of the examples. I want to divide this to two topics. One is what you call the "ABI" level. IMHO, it's hands down superior to ctypes. Your readdir demo demonstrates this very nicely. I would definitely want to see this in the stdlib as an alternative way to interface to C shared objects & DLLs. Two is what you call the "API" level, which is where my opinion becomes mixed. Some things just don't feel right to me: 1. Tying in a C compiler into the flow of a program. I'm not sure whether we have precedents for it in the stdlib. Does this work on Windows where libraries and DLLs are usually built with MSVC? 2. Using a function called "verify" to create stuff. This may sound like a naming bikeshed, but it's not. It ties in to the question - why is this needed? 3. The partial type specifications in C with ellipsis. What is the point? You have the C declarations somewhere anyhow, so why introduce this? The "ABI level" boasts having just C and Python to write, but those partial ellipsis-ridden declarations are hardly C. Please see this as constructive criticism, and I fully admit that I may not have went deep enough to see the big benefits of the "API" level that would justify all that complexity. Eli -------------- next part -------------- An HTML attachment was scrubbed... URL: From greg.ewing at canterbury.ac.nz Wed Feb 27 05:55:09 2013 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Wed, 27 Feb 2013 17:55:09 +1300 Subject: [Python-Dev] [Python-checkins] peps: Pre-alpha draft for PEP 435 (enum). The name is not important at the moment, as In-Reply-To: <87k3pu30pn.fsf@uwakimon.sk.tsukuba.ac.jp> References: <3ZCsSP0rSXzQZS@mail.python.org> <20130223150231.GA20921@sleipnir.bytereef.org> <20130224204000.6309bd6c@anarchist.wooz.org> <512ADB41.5040109@stoneleaf.us> <20130224231410.0023038c@anarchist.wooz.org> <512B7802.9010306@stoneleaf.us> <512B7F43.5060406@stoneleaf.us> <87txoz3fg3.fsf@uwakimon.sk.tsukuba.ac.jp> <512C5BF7.9090803@stoneleaf.us> <87mwur2zkz.fsf@uwakimon.sk.tsukuba.ac.jp> <512CDCDC.2090509@stoneleaf.us> <87k3pu30pn.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <512D91AD.5090501@canterbury.ac.nz> Stephen J. Turnbull wrote: > Worse for me, most of the applications I have, I'd like the enumerator > identifiers to be both string-valued and int-valued: the int used to > index into Python sequences, and the string used in formatting SQL. Is the string value required the same as the name used in Python code? If so, this requirement would be met by having __index__ return the int value and __str__ return the unqualified name. -- Greg From stephen at xemacs.org Wed Feb 27 06:49:11 2013 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Wed, 27 Feb 2013 14:49:11 +0900 Subject: [Python-Dev] [Python-checkins] peps: Pre-alpha draft for PEP 435 (enum). The name is not important at the moment, as In-Reply-To: <512D91AD.5090501@canterbury.ac.nz> References: <3ZCsSP0rSXzQZS@mail.python.org> <20130223150231.GA20921@sleipnir.bytereef.org> <20130224204000.6309bd6c@anarchist.wooz.org> <512ADB41.5040109@stoneleaf.us> <20130224231410.0023038c@anarchist.wooz.org> <512B7802.9010306@stoneleaf.us> <512B7F43.5060406@stoneleaf.us> <87txoz3fg3.fsf@uwakimon.sk.tsukuba.ac.jp> <512C5BF7.9090803@stoneleaf.us> <87mwur2zkz.fsf@uwakimon.sk.tsukuba.ac.jp> <512CDCDC.2090509@stoneleaf.us> <87k3pu30pn.fsf@uwakimon.sk.tsukuba.ac.jp> <512D91AD.5090501@canterbury.ac.nz> Message-ID: <87d2vm2tns.fsf@uwakimon.sk.tsukuba.ac.jp> Greg Ewing writes: > Stephen J. Turnbull wrote: > > Worse for me, most of the applications I have, I'd like the enumerator > > identifiers to be both string-valued and int-valued: the int used to > > index into Python sequences, and the string used in formatting SQL. > > Is the string value required the same as the name used in > Python code? If so, this requirement would be met by having > __index__ return the int value and __str__ return the > unqualified name. Ah, that's right (and Ethan already mentioned using __index__). Yes, I think that should work for me. Very cool! ... If I had a use for it. But I'm mostly trying to understand Ethan's use case; I can't recall having a case where I would be hard-coding an index by name. In my typical use, I might have several data set with columns CITY, INCOME, NUMBER_OF_CARS, ... and need to compute various correlations based on user-specified models. Then my list of indicies for internal use is just [column_index[name] for name in user_provided_name_list] and the inverse (for formatting output) is [column_name[index] for index in internal_indicies] . The column_index and column_name dicts are computed when reading the data set (typically from column headers in a CSV file). Then I said to myself, "instead of using 'internal_indicies', let me hard-code enum constants for indicies -- what would that look like?" and that's where my requirement came from. If I am guessing Ethan's use case correctly, the computations are very specific to the columns involved (eg, he can't use a generic function like "correlation" for arbitrary pairs of columns). Then he does these idiosyncratic computations for various data sets. Are such applications common? Evidently, they're bread and butter for Ethan, but anybody else? From ronaldoussoren at mac.com Wed Feb 27 08:29:47 2013 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Wed, 27 Feb 2013 08:29:47 +0100 Subject: [Python-Dev] cffi in stdlib In-Reply-To: References: Message-ID: <78C26306-4413-4727-B61F-04B8AEDEECF7@mac.com> On 26 Feb, 2013, at 16:13, Maciej Fijalkowski wrote: > Hello. > > I would like to discuss on the language summit a potential inclusion > of cffi[1] into stdlib. The API in general looks nice, but I do have some concens w.r.t. including cffi in the stdlib. 1. Why is cffi completely separate from ctypes, instead of layered on top of it? That is, add a utility module to ctypes that can parse C declarations and generate the right ctypes definitions. 2. Cffi has a dependencies on pycparser and that module and its dependencies would therefore also be added to the stdlib (even if they'd be hidden in the cffi package) 3. Cffi basicly contains a (limited) C parser, and those are notoriously hard to get exactly right. Luckily cffi only needs to interpret declarations and not the full language, but even so this can be a risk of subtle bugs. 4. And finally a technical concern: how well does cffi work with fat binaries on OSX? In particular, will the distutils support generate cached data for all architectures supported by a fat binary? Also, after playing around with it for 5 minutes I don't quite understand how to use it. Let's say I want to wrap a function "CGPoint CGPointMake(CGFloat x, CGFloat y)". Is is possible to avoid mentioning the exact typedef for CGFloat somewhere? I tried using: ffi.cdef("typedef ... CGFloat; typedef struct { CGFloat x; CGFloat y; } CGPoint; CGPoint CGPointMake(CGFloat x, CGFloat y);") But that results in an error when calling verify: TypeError: field 'struct $CGPoint.x' has ctype 'struct $CGFloat' of unknown size From a first glance this doesn't seem to buy me that much w.r.t. ctypes, I still have to declare the actual type of CGFloat which is documented as "some floating point type". Ronald From fijall at gmail.com Wed Feb 27 09:48:34 2013 From: fijall at gmail.com (Maciej Fijalkowski) Date: Wed, 27 Feb 2013 10:48:34 +0200 Subject: [Python-Dev] cffi in stdlib In-Reply-To: References: Message-ID: On Wed, Feb 27, 2013 at 3:39 AM, Terry Reedy wrote: > On 2/26/2013 10:13 AM, Maciej Fijalkowski wrote: > >> I would like to discuss on the language summit a potential inclusion >> of cffi[1] into stdlib. > > > How does it compare in terms of speed. One reason ctypes has not replaces > hand-tuned swig is that it apparently is much slower. I know that someone, > for instance, once did a pygame fork using ctypes to wrap SDL, and got it to > run, but gave it up for comparative slowness reasons. I never did the comparison with hand-tuned swig. It's faster than ctypes for calls and manipulation, C extension beat it to some degree. It's uber fast on PyPy (a bit over the native call, but we'll get it to the native call level). From fijall at gmail.com Wed Feb 27 10:00:10 2013 From: fijall at gmail.com (Maciej Fijalkowski) Date: Wed, 27 Feb 2013 11:00:10 +0200 Subject: [Python-Dev] cffi in stdlib In-Reply-To: References: Message-ID: On Wed, Feb 27, 2013 at 6:53 AM, Eli Bendersky wrote: > > * Work either at the level of the ABI (Application Binary Interface) >> >> or the API (Application Programming Interface). Usually, C libraries >> have a specified C API but often not an ABI (e.g. they may document a >> ?struct? as having at least these fields, but maybe more). (ctypes >> works at the ABI level, whereas Cython and native C extensions work at >> the API level.) > > > > I read the cffi docs once again and went through some of the examples. I > want to divide this to two topics. > > One is what you call the "ABI" level. IMHO, it's hands down superior to > ctypes. Your readdir demo demonstrates this very nicely. I would definitely > want to see this in the stdlib as an alternative way to interface to C > shared objects & DLLs. > > Two is what you call the "API" level, which is where my opinion becomes > mixed. Some things just don't feel right to me: > > 1. Tying in a C compiler into the flow of a program. I'm not sure whether we > have precedents for it in the stdlib. Does this work on Windows where > libraries and DLLs are usually built with MSVC? > Yes. Precedent in the stdlib is really the C API. All the same rules apply (including build and ship a dll). > 2. Using a function called "verify" to create stuff. This may sound like a > naming bikeshed, but it's not. It ties in to the question - why is this > needed? We welcome a better opinion of name (indeed verify is not that great). This elevates ABI to API so either invokes the C compiler or reads stuff from the cache. > > 3. The partial type specifications in C with ellipsis. What is the point? > You have the C declarations somewhere anyhow, so why introduce this? The > "ABI level" boasts having just C and Python to write, but those partial > ellipsis-ridden declarations are hardly C. No, you don't. Some libraries contain macros for example (like OpenSSL) where you just can't use ABI because it makes no sense. It's less common on windows where binary compatibility is important, however looking on linux, multiple stdlib declaration would use ellipsis in the man page. I can't seem to find one right now, but it's something like: struct X { int public_field; ... } which is impossible to do correctly with ctypes without exposing some sort of platform dependency that might change without warning. Another usages are #define SQLITE_OK ... which you don't know at the time of writing (people assume those won't change and the do change). > > Please see this as constructive criticism, and I fully admit that I may not > have went deep enough to see the big benefits of the "API" level that would > justify all that complexity. Sure. It might sound strange, but on linux (or posix in general) binary compatibility is not a thing, so you need to do stuff to account for that. From fijall at gmail.com Wed Feb 27 10:06:07 2013 From: fijall at gmail.com (Maciej Fijalkowski) Date: Wed, 27 Feb 2013 11:06:07 +0200 Subject: [Python-Dev] cffi in stdlib In-Reply-To: <78C26306-4413-4727-B61F-04B8AEDEECF7@mac.com> References: <78C26306-4413-4727-B61F-04B8AEDEECF7@mac.com> Message-ID: On Wed, Feb 27, 2013 at 9:29 AM, Ronald Oussoren wrote: > > On 26 Feb, 2013, at 16:13, Maciej Fijalkowski wrote: > >> Hello. >> >> I would like to discuss on the language summit a potential inclusion >> of cffi[1] into stdlib. > > The API in general looks nice, but I do have some concens w.r.t. including cffi in the stdlib. > > 1. Why is cffi completely separate from ctypes, instead of layered on top of it? That is, add a utility module to ctypes that can parse C declarations and generate the right ctypes definitions. Because ctypes API is a mess and magic. We needed a cleaner (and much smaller) model. > > 2. Cffi has a dependencies on pycparser and that module and its dependencies would therefore also be added to the stdlib (even if they'd be hidden in the cffi package) Yes. pycparser and ply. > > 3. Cffi basicly contains a (limited) C parser, and those are notoriously hard to get exactly right. Luckily cffi only needs to interpret declarations and not the full language, but even so this can be a risk of subtle bugs. It seems to work. > > 4. And finally a technical concern: how well does cffi work with fat binaries on OSX? In particular, will the distutils support generate cached data for all architectures supported by a fat binary? no idea. > > Also, after playing around with it for 5 minutes I don't quite understand how to use it. Let's say I want to wrap a function "CGPoint CGPointMake(CGFloat x, CGFloat y)". Is is possible to avoid mentioning the exact typedef for CGFloat somewhere? I tried using: > > ffi.cdef("typedef ... CGFloat; typedef struct { CGFloat x; CGFloat y; } CGPoint; CGPoint CGPointMake(CGFloat x, CGFloat y);") > > But that results in an error when calling verify: > > TypeError: field 'struct $CGPoint.x' has ctype 'struct $CGFloat' of unknown size > > From a first glance this doesn't seem to buy me that much w.r.t. ctypes, I still have to declare the actual type of CGFloat which is documented as "some floating point type". > > Ronald typedef ... is assumed to be a struct. Copy pasting from documentation, we might want to have a better support for that, but what you do right now is to do: ffi.cdef("const int mysize;") lib = ffi.verify("const int mysize = sizeof(THE_TYPE);") print lib.mysize which will tell you the size (and then you can pick up which float you want). I agree that some generic support for that would be cool. Cheers, fijal From ronaldoussoren at mac.com Wed Feb 27 10:18:02 2013 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Wed, 27 Feb 2013 10:18:02 +0100 Subject: [Python-Dev] cffi in stdlib In-Reply-To: References: <78C26306-4413-4727-B61F-04B8AEDEECF7@mac.com> Message-ID: <5E7B690F-0A81-4351-A145-8DF0CCFF69AB@mac.com> On 27 Feb, 2013, at 10:06, Maciej Fijalkowski wrote: > On Wed, Feb 27, 2013 at 9:29 AM, Ronald Oussoren wrote: >> >> On 26 Feb, 2013, at 16:13, Maciej Fijalkowski wrote: >> >>> Hello. >>> >>> I would like to discuss on the language summit a potential inclusion >>> of cffi[1] into stdlib. >> >> The API in general looks nice, but I do have some concens w.r.t. including cffi in the stdlib. >> >> 1. Why is cffi completely separate from ctypes, instead of layered on top of it? That is, add a utility module to ctypes that can parse C declarations and generate the right ctypes definitions. > > Because ctypes API is a mess and magic. We needed a cleaner (and much > smaller) model. The major advantages of starting over is probably that you can hide the complexity and that opens opportunities for optimizations. That said, I'm not convinced that ctypes is unnecessarily complex. > >> >> 2. Cffi has a dependencies on pycparser and that module and its dependencies would therefore also be added to the stdlib (even if they'd be hidden in the cffi package) > > Yes. pycparser and ply. Which aren't part of the stdlib right now. > >> >> 3. Cffi basicly contains a (limited) C parser, and those are notoriously hard to get exactly right. Luckily cffi only needs to interpret declarations and not the full language, but even so this can be a risk of subtle bugs. > > It seems to work. That's not a confidency inspiring comment :-). That said, I use a hacked up fork of pycparser to parse Apple's Cocoa headers for PyObjC and it appears to work fine for that. > >> >> 4. And finally a technical concern: how well does cffi work with fat binaries on OSX? In particular, will the distutils support generate cached data for all architectures supported by a fat binary? > > no idea. That's somehting that will have to be resolved before cffi can be included in the stdlib, fat binaries are supported by CPython and are used the binary installers. Ronald From fijall at gmail.com Wed Feb 27 10:21:31 2013 From: fijall at gmail.com (Maciej Fijalkowski) Date: Wed, 27 Feb 2013 11:21:31 +0200 Subject: [Python-Dev] cffi in stdlib In-Reply-To: <5E7B690F-0A81-4351-A145-8DF0CCFF69AB@mac.com> References: <78C26306-4413-4727-B61F-04B8AEDEECF7@mac.com> <5E7B690F-0A81-4351-A145-8DF0CCFF69AB@mac.com> Message-ID: On Wed, Feb 27, 2013 at 11:18 AM, Ronald Oussoren wrote: > > On 27 Feb, 2013, at 10:06, Maciej Fijalkowski wrote: > >> On Wed, Feb 27, 2013 at 9:29 AM, Ronald Oussoren wrote: >>> >>> On 26 Feb, 2013, at 16:13, Maciej Fijalkowski wrote: >>> >>>> Hello. >>>> >>>> I would like to discuss on the language summit a potential inclusion >>>> of cffi[1] into stdlib. >>> >>> The API in general looks nice, but I do have some concens w.r.t. including cffi in the stdlib. >>> >>> 1. Why is cffi completely separate from ctypes, instead of layered on top of it? That is, add a utility module to ctypes that can parse C declarations and generate the right ctypes definitions. >> >> Because ctypes API is a mess and magic. We needed a cleaner (and much >> smaller) model. > > The major advantages of starting over is probably that you can hide the complexity and that opens opportunities for optimizations. That said, I'm not convinced that ctypes is unnecessarily complex. I implemented ctypes. It is. >>> 4. And finally a technical concern: how well does cffi work with fat binaries on OSX? In particular, will the distutils support generate cached data for all architectures supported by a fat binary? >> >> no idea. > > That's somehting that will have to be resolved before cffi can be included in the stdlib, fat binaries are supported by CPython and are used the binary installers. > > Ronald if cpython supports it and you can load it using dlopen, it does work then (it really is just building a C extension on the API level). From dave at dabeaz.com Wed Feb 27 12:00:17 2013 From: dave at dabeaz.com (David Beazley) Date: Wed, 27 Feb 2013 05:00:17 -0600 Subject: [Python-Dev] PLY in stdlib (was cffi in stdlib) In-Reply-To: References: Message-ID: > > From: Eli Bendersky > > I'll be the first one to admit that pycparser is almost certainly not > generally useful enough to be exposed in the stdlib. So just using it as an > implementation detail is absolutely fine. PLY is a more interesting > question, however, since PLY is somewhat more generally useful. That said, > I see all this as implementation details that shouldn't distract us from > the main point of whether cffi should be added. > Regarding the inclusion of PLY or some subcomponent of it in the standard library, it's not an entirely crazy idea in my opinion. LALR(1) parsers have been around for a long time, are generally known to anyone who's used yacc/bison, and would be useful outside the context of cffi or pycparser. PLY has also been around for about 12 years and is what I would call stable. It gets an update about every year or two, but that's about it. PLY is also relatively small--just two files and about 4300 lines of code (much of which could probably be scaled down a bit). The only downside to including PLY might be the fact that there are very few people walking around who've actually had to *implement* an LALR(1) parser generator. Some of the code for that is extremely hairy and mathematical. At this time, I don't think there are any bugs in it, but it's not the sort of thing that one wants to wander into casually. Also, there are some horrible hacks in PLY that I'd really like to get rid of, but am currently stuck with due to backwards compatibility issues. Alex Gaynor has been working on a PLY variant (RPLY) geared at RPython and which has a slightly different programming interface. I'd say if we were to go down this route, he and I should work together to put together some kind of more general "parsing.lalr" package (or similar) that cleans it up and makes it more suitable as a library for building different kinds of parsing tools on top of. Cheers, Dave From fuzzyman at voidspace.org.uk Wed Feb 27 12:16:12 2013 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Wed, 27 Feb 2013 11:16:12 +0000 Subject: [Python-Dev] PLY in stdlib (was cffi in stdlib) In-Reply-To: References: Message-ID: On 27 Feb 2013, at 11:00, David Beazley wrote: >> >> From: Eli Bendersky >> >> I'll be the first one to admit that pycparser is almost certainly not >> generally useful enough to be exposed in the stdlib. So just using it as an >> implementation detail is absolutely fine. PLY is a more interesting >> question, however, since PLY is somewhat more generally useful. That said, >> I see all this as implementation details that shouldn't distract us from >> the main point of whether cffi should be added. >> > > Regarding the inclusion of PLY or some subcomponent of it in the standard library, it's not an entirely crazy idea in my opinion. +1 PLY is capable and well tried-and-tested. We used it in Resolver One to implement a pretty large grammar and it is (in my opinion) best of breed in the Python parser generator world. Being stable and widely used, with an "available maintainer", makes it an ideal candidate for standard library inclusion. Michael > LALR(1) parsers have been around for a long time, are generally known to anyone who's used yacc/bison, and would be useful outside the context of cffi or pycparser. PLY has also been around for about 12 years and is what I would call stable. It gets an update about every year or two, but that's about it. PLY is also relatively small--just two files and about 4300 lines of code (much of which could probably be scaled down a bit). > > The only downside to including PLY might be the fact that there are very few people walking around who've actually had to *implement* an LALR(1) parser generator. Some of the code for that is extremely hairy and mathematical. At this time, I don't think there are any bugs in it, but it's not the sort of thing that one wants to wander into casually. Also, there are some horrible hacks in PLY that I'd really like to get rid of, but am currently stuck with due to backwards compatibility issues. > > Alex Gaynor has been working on a PLY variant (RPLY) geared at RPython and which has a slightly different programming interface. I'd say if we were to go down this route, he and I should work together to put together some kind of more general "parsing.lalr" package (or similar) that cleans it up and makes it more suitable as a library for building different kinds of parsing tools on top of. > > Cheers, > Dave > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk -- http://www.voidspace.org.uk/ May you do good and not evil May you find forgiveness for yourself and forgive others May you share freely, never taking more than you give. -- the sqlite blessing http://www.sqlite.org/different.html From shibturn at gmail.com Wed Feb 27 12:45:14 2013 From: shibturn at gmail.com (Richard Oudkerk) Date: Wed, 27 Feb 2013 11:45:14 +0000 Subject: [Python-Dev] cffi in stdlib In-Reply-To: References: <78C26306-4413-4727-B61F-04B8AEDEECF7@mac.com> <5E7B690F-0A81-4351-A145-8DF0CCFF69AB@mac.com> Message-ID: On 27/02/2013 9:21am, Maciej Fijalkowski wrote: >> >That's somehting that will have to be resolved before cffi can be included in the stdlib, fat binaries are supported by CPython and are used the binary installers. >> > >> >Ronald > if cpython supports it and you can load it using dlopen, it does work > then (it really is just building a C extension on the API level). On Windows ctypes has CDLL for libraries using the cdecl calling convention and WINDLL for libraries with the stdcall calling convention. I can't see any similar distinction in cffi's documentation. Can cffi magically work out which calling convention to use, or are you restricted to the cdecl calling convention? -- Richard From fijall at gmail.com Wed Feb 27 12:53:30 2013 From: fijall at gmail.com (Maciej Fijalkowski) Date: Wed, 27 Feb 2013 13:53:30 +0200 Subject: [Python-Dev] cffi in stdlib In-Reply-To: References: <78C26306-4413-4727-B61F-04B8AEDEECF7@mac.com> <5E7B690F-0A81-4351-A145-8DF0CCFF69AB@mac.com> Message-ID: On Wed, Feb 27, 2013 at 1:45 PM, Richard Oudkerk wrote: > On 27/02/2013 9:21am, Maciej Fijalkowski wrote: >>> >>> >That's somehting that will have to be resolved before cffi can be >>> > included in the stdlib, fat binaries are supported by CPython and are used >>> > the binary installers. >>> > >>> >Ronald >> >> if cpython supports it and you can load it using dlopen, it does work >> then (it really is just building a C extension on the API level). > > > On Windows ctypes has CDLL for libraries using the cdecl calling convention > and WINDLL for libraries with the stdcall calling convention. > > I can't see any similar distinction in cffi's documentation. Can cffi > magically work out which calling convention to use, or are you restricted to > the cdecl calling convention? copy pasting from docs: Windows: you can?t yet specify the calling convention of callbacks. (For regular calls, the correct calling convention should be automatically inferred by the C backend.) Use an indirection, like in the example just above. I think it means you can't use the ABI version and specify the calling convention. It's a reasonable bug report (the calling convention on API version works though) > > -- > Richard > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/fijall%40gmail.com From ncoghlan at gmail.com Wed Feb 27 13:39:30 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 27 Feb 2013 22:39:30 +1000 Subject: [Python-Dev] cffi in stdlib In-Reply-To: References: <78C26306-4413-4727-B61F-04B8AEDEECF7@mac.com> Message-ID: On Wed, Feb 27, 2013 at 7:06 PM, Maciej Fijalkowski wrote: >> 3. Cffi basicly contains a (limited) C parser, and those are notoriously hard to get exactly right. Luckily cffi only needs to interpret declarations and not the full language, but even so this can be a risk of subtle bugs. > > It seems to work. C itself shouldn't be *too* bad - it's trying to parse C++ that is truly insane (which is why Dave Beazley explicitly warned Maciej and Armin away from that based on his experience with SWIG). Looking at pycparsing more closely though, there may be a problem with the C preprocessing step on Windows. While on Linux it uses the system "cpp", for Windows it currently bundles a copy of the LCC preprocessor, which may pose licensing challenges (https://github.com/drh/lcc/blob/master/CPYRIGHT). However, this concern may not be applicable to cffi: given cffi's requirement for a full external C compiler to actually *create* the C extension modules during development, and the ability to retrieve everything necessary directly from the created C extension when no C compiler is available, it seems there would be no reason to ever need the bundled C preprocessor. So we simply don't ship it, and the LCC license becomes irrelevant. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From fijall at gmail.com Wed Feb 27 13:44:12 2013 From: fijall at gmail.com (Maciej Fijalkowski) Date: Wed, 27 Feb 2013 14:44:12 +0200 Subject: [Python-Dev] cffi in stdlib In-Reply-To: References: <78C26306-4413-4727-B61F-04B8AEDEECF7@mac.com> Message-ID: On Wed, Feb 27, 2013 at 2:39 PM, Nick Coghlan wrote: > On Wed, Feb 27, 2013 at 7:06 PM, Maciej Fijalkowski wrote: >>> 3. Cffi basicly contains a (limited) C parser, and those are notoriously hard to get exactly right. Luckily cffi only needs to interpret declarations and not the full language, but even so this can be a risk of subtle bugs. >> >> It seems to work. > > C itself shouldn't be *too* bad - it's trying to parse C++ that is > truly insane (which is why Dave Beazley explicitly warned Maciej and > Armin away from that based on his experience with SWIG). > > Looking at pycparsing more closely though, there may be a problem with > the C preprocessing step on Windows. While on Linux it uses the system > "cpp", for Windows it currently bundles a copy of the LCC > preprocessor, which may pose licensing challenges > (https://github.com/drh/lcc/blob/master/CPYRIGHT). However, this > concern may not be applicable to cffi: given cffi's requirement for a > full external C compiler to actually *create* the C extension modules > during development, and the ability to retrieve everything necessary > directly from the created C extension when no C compiler is available, > it seems there would be no reason to ever need the bundled C > preprocessor. So we simply don't ship it, and the LCC license becomes > irrelevant. > > Cheers, > Nick. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia cffi does not use preprocessor From ncoghlan at gmail.com Wed Feb 27 14:24:09 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 27 Feb 2013 23:24:09 +1000 Subject: [Python-Dev] cffi in stdlib In-Reply-To: References: Message-ID: On Wed, Feb 27, 2013 at 7:00 PM, Maciej Fijalkowski wrote: > On Wed, Feb 27, 2013 at 6:53 AM, Eli Bendersky wrote: >> 2. Using a function called "verify" to create stuff. This may sound like a >> naming bikeshed, but it's not. It ties in to the question - why is this >> needed? > > We welcome a better opinion of name (indeed verify is not that great). > This elevates ABI to API so either invokes the C compiler or reads > stuff from the cache. Have you considered the name "create_api"? After all, you're essentially creating a Python API given a C header declaration and a specified extension module to cache the result. The details on extension module caching, and the fact that it won't even look for a C compiler if the extension module is already available should also be in the main section of the docs on the verification step rather than only being down in the internal documentation for cffi.verifier.Verifier Cheers, Nick. P.S. I created both of those as tracker issues: - https://bitbucket.org/cffi/cffi/issue/59/missing-content-in-cffiverify - https://bitbucket.org/cffi/cffi/issue/60/proposed-rename-for-verify-and-verifier -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From solipsis at pitrou.net Wed Feb 27 14:31:41 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 27 Feb 2013 14:31:41 +0100 Subject: [Python-Dev] cffi in stdlib References: <20130226190620.1FF24250BDA@webabinitio.net> <20130226202153.2b923404@pitrou.net> <512D41F9.3050109@canterbury.ac.nz> Message-ID: <20130227143141.4aef5635@pitrou.net> Le Wed, 27 Feb 2013 12:15:05 +1300, Greg Ewing a ?crit : > Antoine Pitrou wrote: > > Or we'll go straight to 5. > > (or switch to date-based numbering :-)) > > We could go the Apple route and start naming them after > species of snake. We have to find sufficiently silly species of snakes, though. Regards Antoine. From solipsis at pitrou.net Wed Feb 27 14:30:01 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 27 Feb 2013 14:30:01 +0100 Subject: [Python-Dev] [Python-checkins] peps: Pre-alpha draft for PEP 435 (enum). The name is not important at the moment, as References: <3ZCsSP0rSXzQZS@mail.python.org> <20130223150231.GA20921@sleipnir.bytereef.org> <20130224204000.6309bd6c@anarchist.wooz.org> <512ADB41.5040109@stoneleaf.us> <20130224231410.0023038c@anarchist.wooz.org> <512B7802.9010306@stoneleaf.us> <512B7F43.5060406@stoneleaf.us> <87txoz3fg3.fsf@uwakimon.sk.tsukuba.ac.jp> <512C5BF7.9090803@stoneleaf.us> <87mwur2zkz.fsf@uwakimon.sk.tsukuba.ac.jp> <512CDCDC.2090509@stoneleaf.us> <20130226201757.535abe3e@pitrou.net> <87ip5e2zxs.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <20130227143001.3cfce13a@pitrou.net> Le Wed, 27 Feb 2013 12:33:35 +0900, "Stephen J. Turnbull" a ?crit : > > As far as I can see, what you (Antoine) want is an identifier with a > constant value, no more and no less. Grouping into an enum is merely > a lexical convention, since you are happy to compare enums of > different enum classes and enums with ints. > > I think such a thing is pretty generally desirable, but most of the > people who have created Enum classes implement some type-checking that > this value came from that Enum. To me that is what differentiates an > enumerated constant from a generic named constant. I don't see why > you keep deprecating that feature. Do you really think it's useless, > or is it just that you wouldn't use it yourself? I just wouldn't care for it myself. I don't think extra-strong typing of constants is really useful in practice; it smells a bit like private methods to me. Regards Antoine. From python-dev at masklinn.net Wed Feb 27 14:40:44 2013 From: python-dev at masklinn.net (Xavier Morel) Date: Wed, 27 Feb 2013 14:40:44 +0100 Subject: [Python-Dev] cffi in stdlib In-Reply-To: <20130227143141.4aef5635@pitrou.net> References: <20130226190620.1FF24250BDA@webabinitio.net> <20130226202153.2b923404@pitrou.net> <512D41F9.3050109@canterbury.ac.nz> <20130227143141.4aef5635@pitrou.net> Message-ID: <45EACC3C-0AA4-40D2-AA7E-818878800770@masklinn.net> On 2013-02-27, at 14:31 , Antoine Pitrou wrote: > Le Wed, 27 Feb 2013 12:15:05 +1300, > Greg Ewing a ?crit : >> Antoine Pitrou wrote: >>> Or we'll go straight to 5. >>> (or switch to date-based numbering :-)) >> >> We could go the Apple route and start naming them after >> species of snake. > > We have to find sufficiently silly species of snakes, though. With about 3000 extant snake species, that should be manageable. From jsbueno at python.org.br Wed Feb 27 14:41:27 2013 From: jsbueno at python.org.br (Joao S. O. Bueno) Date: Wed, 27 Feb 2013 10:41:27 -0300 Subject: [Python-Dev] cffi in stdlib In-Reply-To: <20130227143141.4aef5635@pitrou.net> References: <20130226190620.1FF24250BDA@webabinitio.net> <20130226202153.2b923404@pitrou.net> <512D41F9.3050109@canterbury.ac.nz> <20130227143141.4aef5635@pitrou.net> Message-ID: On 27 February 2013 10:31, Antoine Pitrou wrote: > Le Wed, 27 Feb 2013 12:15:05 +1300, > Greg Ewing a ?crit : > > Antoine Pitrou wrote: > > > Or we'll go straight to 5. > > > (or switch to date-based numbering :-)) > > > > We could go the Apple route and start naming them after > > species of snake. > > We have to find sufficiently silly species of snakes, though. > Monty Python feature movies. There are less than snake species, but imagine porting Django to be able to run on Life of Brian! > > Regards > > Antoine. > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/jsbueno%40python.org.br > -------------- next part -------------- An HTML attachment was scrubbed... URL: From arigo at tunes.org Wed Feb 27 14:52:34 2013 From: arigo at tunes.org (Armin Rigo) Date: Wed, 27 Feb 2013 14:52:34 +0100 Subject: [Python-Dev] cffi in stdlib In-Reply-To: References: <20130226190620.1FF24250BDA@webabinitio.net> Message-ID: Hi Guido, On Tue, Feb 26, 2013 at 8:24 PM, Guido van Rossum wrote: > From a software engineering perspective, 10 years is indistinguishable > from infinity, so I don't care what happens 10 years from now -- as > long as you don't blame me. :-) I can't resist: around today it is the 10th anniversary of PyPy. :-) A bient?t, Armin. From fred at fdrake.net Wed Feb 27 14:56:35 2013 From: fred at fdrake.net (Fred Drake) Date: Wed, 27 Feb 2013 08:56:35 -0500 Subject: [Python-Dev] [Python-checkins] peps: Pre-alpha draft for PEP 435 (enum). The name is not important at the moment, as In-Reply-To: <20130227143001.3cfce13a@pitrou.net> References: <3ZCsSP0rSXzQZS@mail.python.org> <20130223150231.GA20921@sleipnir.bytereef.org> <20130224204000.6309bd6c@anarchist.wooz.org> <512ADB41.5040109@stoneleaf.us> <20130224231410.0023038c@anarchist.wooz.org> <512B7802.9010306@stoneleaf.us> <512B7F43.5060406@stoneleaf.us> <87txoz3fg3.fsf@uwakimon.sk.tsukuba.ac.jp> <512C5BF7.9090803@stoneleaf.us> <87mwur2zkz.fsf@uwakimon.sk.tsukuba.ac.jp> <512CDCDC.2090509@stoneleaf.us> <20130226201757.535abe3e@pitrou.net> <87ip5e2zxs.fsf@uwakimon.sk.tsukuba.ac.jp> <20130227143001.3cfce13a@pitrou.net> Message-ID: On Wed, Feb 27, 2013 at 8:30 AM, Antoine Pitrou wrote: > I don't think extra-strong typing of constants is really useful in > practice; it smells a bit like private methods to me. I think checking that a value comes from a particular enum *is* a degree of hand-holding. For libraries or frameworks whose users aren't expected to know them exhaustively, making reasonable checks of parameters can substantially reduce the number of ways it can be used incorrectly. Outside performance critical code, this is a win. -Fred -- Fred L. Drake, Jr. "A storm broke loose in my mind." --Albert Einstein From eliben at gmail.com Wed Feb 27 16:50:18 2013 From: eliben at gmail.com (Eli Bendersky) Date: Wed, 27 Feb 2013 07:50:18 -0800 Subject: [Python-Dev] cffi in stdlib In-Reply-To: <78C26306-4413-4727-B61F-04B8AEDEECF7@mac.com> References: <78C26306-4413-4727-B61F-04B8AEDEECF7@mac.com> Message-ID: On Tue, Feb 26, 2013 at 11:29 PM, Ronald Oussoren wrote: > > On 26 Feb, 2013, at 16:13, Maciej Fijalkowski wrote: > > > Hello. > > > > I would like to discuss on the language summit a potential inclusion > > of cffi[1] into stdlib. > > The API in general looks nice, but I do have some concens w.r.t. including > cffi in the stdlib. > > 1. Why is cffi completely separate from ctypes, instead of layered on top > of it? That is, add a utility module to ctypes that can parse C > declarations and generate the right ctypes definitions. > > 2. Cffi has a dependencies on pycparser and that module and its > dependencies would therefore also be added to the stdlib (even if they'd be > hidden in the cffi package) > > 3. Cffi basicly contains a (limited) C parser, and those are notoriously > hard to get exactly right. Luckily cffi only needs to interpret > declarations and not the full language, but even so this can be a risk of > subtle bugs. > pycparser has been around for more than 4 years and is pretty stable. I know that it's been used to parse Windows headers, Linux system headers and other "scary" things for a C parser to handle. It's only known problem has to do with over-creative abuses of C's context sensitivity: http://eli.thegreenplace.net/2011/05/02/the-context-sensitivity-of-c%E2%80%99s-grammar-revisited/ In practice it doesn't come up often, and almost never in declarations, which is the subset cffi uses pycparser for. Eli -------------- next part -------------- An HTML attachment was scrubbed... URL: From eliben at gmail.com Wed Feb 27 16:57:15 2013 From: eliben at gmail.com (Eli Bendersky) Date: Wed, 27 Feb 2013 07:57:15 -0800 Subject: [Python-Dev] cffi in stdlib In-Reply-To: References: Message-ID: > > I read the cffi docs once again and went through some of the examples. I > > want to divide this to two topics. > > > > One is what you call the "ABI" level. IMHO, it's hands down superior to > > ctypes. Your readdir demo demonstrates this very nicely. I would > definitely > > want to see this in the stdlib as an alternative way to interface to C > > shared objects & DLLs. > > > > Two is what you call the "API" level, which is where my opinion becomes > > mixed. Some things just don't feel right to me: > > > > 1. Tying in a C compiler into the flow of a program. I'm not sure > whether we > > have precedents for it in the stdlib. Does this work on Windows where > > libraries and DLLs are usually built with MSVC? > > > > Yes. Precedent in the stdlib is really the C API. All the same rules > apply (including build and ship a dll). > So would you say that the main use of the API level is provide an alternative for writing C API code to interface to C libraries. IOW, it's in competition with Swig? > > > 2. Using a function called "verify" to create stuff. This may sound like > a > > naming bikeshed, but it's not. It ties in to the question - why is this > > needed? > > We welcome a better opinion of name (indeed verify is not that great). > This elevates ABI to API so either invokes the C compiler or reads > stuff from the cache. > Can you elaborate on what "elevates ABI to API" means here? > > 3. The partial type specifications in C with ellipsis. What is the point? > > You have the C declarations somewhere anyhow, so why introduce this? The > > "ABI level" boasts having just C and Python to write, but those partial > > ellipsis-ridden declarations are hardly C. > > No, you don't. Some libraries contain macros for example (like > OpenSSL) where you just can't use ABI because it makes no sense. It's > less common on windows where binary compatibility is important, > however looking on linux, multiple stdlib declaration would use > ellipsis in the man page. It would be useful to find an actual example and discuss it concretely. > I can't seem to find one right now, but it's > something like: > > struct X { > int public_field; > ... > } > > which is impossible to do correctly with ctypes without exposing some > sort of platform dependency that might change without warning. > > Another usages are #define SQLITE_OK ... which you don't know at the > time of writing (people assume those won't change and the do change). > Do you mean that the value of SQLITE_OK changed over time (now it's 0, but used to be different?) If so, then in a realistic use case, how would the API level help solve this? Eli -------------- next part -------------- An HTML attachment was scrubbed... URL: From fijall at gmail.com Wed Feb 27 17:08:26 2013 From: fijall at gmail.com (Maciej Fijalkowski) Date: Wed, 27 Feb 2013 18:08:26 +0200 Subject: [Python-Dev] cffi in stdlib In-Reply-To: References: Message-ID: On Wed, Feb 27, 2013 at 5:57 PM, Eli Bendersky wrote: > >> > I read the cffi docs once again and went through some of the examples. I >> > want to divide this to two topics. >> > >> > One is what you call the "ABI" level. IMHO, it's hands down superior to >> > ctypes. Your readdir demo demonstrates this very nicely. I would >> > definitely >> > want to see this in the stdlib as an alternative way to interface to C >> > shared objects & DLLs. >> > >> > Two is what you call the "API" level, which is where my opinion becomes >> > mixed. Some things just don't feel right to me: >> > >> > 1. Tying in a C compiler into the flow of a program. I'm not sure >> > whether we >> > have precedents for it in the stdlib. Does this work on Windows where >> > libraries and DLLs are usually built with MSVC? >> > >> >> Yes. Precedent in the stdlib is really the C API. All the same rules >> apply (including build and ship a dll). > > > So would you say that the main use of the API level is provide an > alternative for writing C API code to interface to C libraries. IOW, it's in > competition with Swig? the general goal is to provide alternative for writing C API by hand (so SWIG, ctypes, cython, whatever). > >> >> >> > 2. Using a function called "verify" to create stuff. This may sound like >> > a >> > naming bikeshed, but it's not. It ties in to the question - why is this >> > needed? >> >> We welcome a better opinion of name (indeed verify is not that great). >> This elevates ABI to API so either invokes the C compiler or reads >> stuff from the cache. > > > Can you elaborate on what "elevates ABI to API" means here? > >> >> > 3. The partial type specifications in C with ellipsis. What is the >> > point? >> > You have the C declarations somewhere anyhow, so why introduce this? The >> > "ABI level" boasts having just C and Python to write, but those partial >> > ellipsis-ridden declarations are hardly C. >> >> No, you don't. Some libraries contain macros for example (like >> OpenSSL) where you just can't use ABI because it makes no sense. It's >> less common on windows where binary compatibility is important, >> however looking on linux, multiple stdlib declaration would use >> ellipsis in the man page. > > > It would be useful to find an actual example and discuss it concretely. #define Py_INCREF(x) (x -> refcnt++) the API here is Py_INCREF, which is a macro and does not exist in the DLL. copy-pasting the implementation is not using the API, it's hoping that stuff won't change in the future (and say makes it not working when PyPy implements Py_INCREF as a function). Some POSIX things are macros in one version of glibc and functions in other and there is no change in API so your code will silently stop working. > >> >> I can't seem to find one right now, but it's >> something like: >> >> struct X { >> int public_field; >> ... >> } >> >> which is impossible to do correctly with ctypes without exposing some >> sort of platform dependency that might change without warning. >> >> Another usages are #define SQLITE_OK ... which you don't know at the >> time of writing (people assume those won't change and the do change). > > > Do you mean that the value of SQLITE_OK changed over time (now it's 0, but > used to be different?) > > If so, then in a realistic use case, how would the API level help solve > this? > > Eli no the API is SQLITE_OK. The actual numeric value of 0 is *not* part of the API. it might be 0 it might be not, you never know. In cffi you write: #define SQLITE_OK ... and let the compiler figure out. in ctypes you just hardcode 0. I ended up once with a scary mess of ifs because BSD Linux and OS X were declaring the same numeric constants with different values. From ajm at flonidan.dk Wed Feb 27 17:34:35 2013 From: ajm at flonidan.dk (Anders J. Munch) Date: Wed, 27 Feb 2013 17:34:35 +0100 Subject: [Python-Dev] [Python-checkins] peps: Pre-alpha draft for PEP 435 (enum). The name is not important at the moment, as In-Reply-To: References: <3ZCsSP0rSXzQZS@mail.python.org><20130223150231.GA20921@sleipnir.bytereef.org><20130224204000.6309bd6c@anarchist.wooz.org><512ADB41.5040109@stoneleaf.us><20130224231410.0023038c@anarchist.wooz.org><512B7802.9010306@stoneleaf.us><512B7F43.5060406@stoneleaf.us><87txoz3fg3.fsf@uwakimon.sk.tsukuba.ac.jp><512C5BF7.9090803@stoneleaf.us><87mwur2zkz.fsf@uwakimon.sk.tsukuba.ac.jp><512CDCDC.2090509@stoneleaf.us> <20130226201757.535abe3e@pitrou.net><87ip5e2zxs.fsf@uwakimon.sk.tsukuba.ac.jp><20130227143001.3cfce13a@pitrou.net> Message-ID: <9B1795C95533CA46A83BA1EAD4B01030DFE09F@flonidanmail.flonidan.net> Hi, offering my DKK 0.50 on the subject. I've used an in-house enum type for the better part of a decade - put up at http://unicollect.dk/download/oss/dc_enum.zip for inspiration. I'm partial to an int subclass, or at least something very int-like, because wrapping C libraries is such a major use case. Enums should support all the operations that C enum's do: indexing, ordering, bitwise math and unnamed values. There's a whole bunch of these wrapped C enum's in the standard library, e.g. in the os, stat and urllib modules. Wouldn't it be good to see them converted to enumeration values, to make them sort-of self-documenting and sort-of typechecked? - About unnamed values: Suppose for example you gather up some stat constants in an enum like this: class S_I(Enum): XOTH = 1 WOTH = 2 ROTH = 4 RWXO = 7 XGRP = 8 WGRP = 16 RGRP = 32 RWXG = 56 XUSR = 64 EXEC = 64 WRITE = 128 WUSR = 128 RUSR = 256 READ = 256 RWXU = 448 SVTX = 512 SGID = 1024 SUID = 2048 FIFO = 4096 FCHR = 8192 FDIR = 16384 FBLK = 24576 FREG = 32768 FLNK = 40960 FSOCK = 49152 Now what happens if you do >>> S_I(5) ? This should not raise an exception: 5 is the perfectly valid combination of XOTH and ROTH. Supporting unnamed values is also useful when reading integer values from an external source - file, network, serial port - where typically you'd first convert the received value to the enum type before doing any processing. If converting to enum type may raise an exception, that would force you to print raw integer values in diagnostics, where you'd rather have printed the userfriendly symbolic names. - The repr of an enum value should be short and sweet. Because sometimes you'll be looking at long lists of information with 2-3 of these on every line, and then is just going to be annoyingly verbose. This is how my library eventually printed enums: >>> S_I.ROTH S_I.ROTH >>> S_I(4) S_I.ROTH >>> S_I(5) S_I(5) >>> print(S_I(4)) S_I.ROTH(4) >>> print(S_I(5)) S_I.?(5) - A final thing, attempting to compare or convert values from different enum classes should be a TypeError. E.g. in wxPython, it's easy to mistake wx.OK and wx.ID_OK, and a friendly TypeError message is a good way of detecting this early. regards, Anders From michael at voidspace.org.uk Wed Feb 27 17:51:16 2013 From: michael at voidspace.org.uk (Michael Foord) Date: Wed, 27 Feb 2013 16:51:16 +0000 Subject: [Python-Dev] Python Language Summit at PyCon: Agenda Message-ID: <13FB490E-3548-4959-9C2B-2880B8ACA6F5@voidspace.org.uk> Hello all, PyCon, and the Python Language Summit, is nearly upon us. We have a good number of people confirmed to attend. If you are intending to come to the language summit but haven't let me know please do so. The agenda of topics for discussion so far includes the following: * A report on pypy status - Maciej and Armin * Jython and IronPython status reports - Dino / Frank * Packaging (Doug Hellmann and Monty Taylor at least) * Cleaning up interpreter initialisation (both in hopes of finding areas to rationalise and hence speed things up, as well as making things more embedding friendly). Nick Coghlan * Adding new async capabilities to the standard library (Guido) * cffi and the standard library - Maciej * flufl.enum and the standard library - Barry Warsaw * The argument clinic - Larry Hastings If you have other items you'd like to discuss please let me know and I can add them to the agenda. All the best, Michael Foord -- http://www.voidspace.org.uk/ May you do good and not evil May you find forgiveness for yourself and forgive others May you share freely, never taking more than you give. -- the sqlite blessing http://www.sqlite.org/different.html From brian at python.org Wed Feb 27 18:02:56 2013 From: brian at python.org (Brian Curtin) Date: Wed, 27 Feb 2013 11:02:56 -0600 Subject: [Python-Dev] Python Language Summit at PyCon: Agenda In-Reply-To: <13FB490E-3548-4959-9C2B-2880B8ACA6F5@voidspace.org.uk> References: <13FB490E-3548-4959-9C2B-2880B8ACA6F5@voidspace.org.uk> Message-ID: On Wed, Feb 27, 2013 at 10:51 AM, Michael Foord wrote: > Hello all, > > PyCon, and the Python Language Summit, is nearly upon us. We have a good number of people confirmed to attend. If you are intending to come to the language summit but haven't let me know please do so. > > The agenda of topics for discussion so far includes the following: > > * A report on pypy status - Maciej and Armin > * Jython and IronPython status reports - Dino / Frank > * Packaging (Doug Hellmann and Monty Taylor at least) > * Cleaning up interpreter initialisation (both in hopes of finding areas > to rationalise and hence speed things up, as well as making things > more embedding friendly). Nick Coghlan > * Adding new async capabilities to the standard library (Guido) > * cffi and the standard library - Maciej > * flufl.enum and the standard library - Barry Warsaw > * The argument clinic - Larry Hastings > > If you have other items you'd like to discuss please let me know and I can add them to the agenda. I'll take detailed notes again this year. Within a few days of the end of the conference I'll post a write-up to blog.python.org and this list to keep everyone informed. From p.f.moore at gmail.com Wed Feb 27 19:24:59 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 27 Feb 2013 18:24:59 +0000 Subject: [Python-Dev] cffi in stdlib In-Reply-To: References: <78C26306-4413-4727-B61F-04B8AEDEECF7@mac.com> <5E7B690F-0A81-4351-A145-8DF0CCFF69AB@mac.com> Message-ID: On 27 February 2013 11:53, Maciej Fijalkowski wrote: > I think it means you can't use the ABI version and specify the calling > convention. It's a reasonable bug report (the calling convention on > API version works though) That would be a deal-breaker for my use case of quick scripts working with the Windows API on a machine with no C compiler. I'd have to use ctypes in that case until cffi had this feature. Paul. From p.f.moore at gmail.com Wed Feb 27 19:28:40 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 27 Feb 2013 18:28:40 +0000 Subject: [Python-Dev] cffi in stdlib In-Reply-To: References: <78C26306-4413-4727-B61F-04B8AEDEECF7@mac.com> <5E7B690F-0A81-4351-A145-8DF0CCFF69AB@mac.com> Message-ID: 27 February 2013 18:24, Paul Moore wrote: > On 27 February 2013 11:53, Maciej Fijalkowski wrote: >> I think it means you can't use the ABI version and specify the calling >> convention. It's a reasonable bug report (the calling convention on >> API version works though) > > That would be a deal-breaker for my use case of quick scripts working > with the Windows API on a machine with no C compiler. I'd have to use > ctypes in that case until cffi had this feature. One other use case for the ABI level over the API level - the ABI level (no C extension) can be used across Python versions, where the API level needs a separate compiled extension per Python version. This can be a big deal on Windows at least, where users (not developers) with no compilers on their systems are common. I'm not trying to argue against cffi, just pointing out some cases that should be considered. Paul From brett at python.org Wed Feb 27 19:50:54 2013 From: brett at python.org (Brett Cannon) Date: Wed, 27 Feb 2013 13:50:54 -0500 Subject: [Python-Dev] cffi in stdlib In-Reply-To: References: <78C26306-4413-4727-B61F-04B8AEDEECF7@mac.com> <5E7B690F-0A81-4351-A145-8DF0CCFF69AB@mac.com> Message-ID: On Wed, Feb 27, 2013 at 1:28 PM, Paul Moore wrote: > 27 February 2013 18:24, Paul Moore wrote: > > On 27 February 2013 11:53, Maciej Fijalkowski wrote: > >> I think it means you can't use the ABI version and specify the calling > >> convention. It's a reasonable bug report (the calling convention on > >> API version works though) > > > > That would be a deal-breaker for my use case of quick scripts working > > with the Windows API on a machine with no C compiler. I'd have to use > > ctypes in that case until cffi had this feature. > > One other use case for the ABI level over the API level - the ABI > level (no C extension) can be used across Python versions, where the > API level needs a separate compiled extension per Python version. This > can be a big deal on Windows at least, where users (not developers) > with no compilers on their systems are common. > > Is that still true for Windows even with the stable ABI and the dropping of narrow/wide Unicode builds? -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Wed Feb 27 19:50:21 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 27 Feb 2013 19:50:21 +0100 Subject: [Python-Dev] Python Language Summit at PyCon: Agenda References: <13FB490E-3548-4959-9C2B-2880B8ACA6F5@voidspace.org.uk> Message-ID: <20130227195021.0101c919@pitrou.net> On Wed, 27 Feb 2013 16:51:16 +0000 Michael Foord wrote: > Hello all, > > PyCon, and the Python Language Summit, is nearly upon us. We have a good number of people confirmed to attend. If you are intending to come to the language summit but haven't let me know please do so. > > The agenda of topics for discussion so far includes the following: > > * A report on pypy status - Maciej and Armin > * Jython and IronPython status reports - Dino / Frank > * Packaging (Doug Hellmann and Monty Taylor at least) > * Cleaning up interpreter initialisation (both in hopes of finding areas > to rationalise and hence speed things up, as well as making things > more embedding friendly). Nick Coghlan > * Adding new async capabilities to the standard library (Guido) > * cffi and the standard library - Maciej > * flufl.enum and the standard library - Barry Warsaw > * The argument clinic - Larry Hastings > > If you have other items you'd like to discuss please let me know and I can add them to the agenda. Perhaps someone wants to discuss http://www.python.org/dev/peps/pep-0428/, but I won't be there and the PEP isn't terribly up-to-date either :-) Regards Antoine. From p.f.moore at gmail.com Wed Feb 27 19:57:42 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 27 Feb 2013 18:57:42 +0000 Subject: [Python-Dev] cffi in stdlib In-Reply-To: References: <78C26306-4413-4727-B61F-04B8AEDEECF7@mac.com> <5E7B690F-0A81-4351-A145-8DF0CCFF69AB@mac.com> Message-ID: On 27 February 2013 18:50, Brett Cannon wrote: >> One other use case for the ABI level over the API level - the ABI >> level (no C extension) can be used across Python versions, where the >> API level needs a separate compiled extension per Python version. This >> can be a big deal on Windows at least, where users (not developers) >> with no compilers on their systems are common. >> > > Is that still true for Windows even with the stable ABI and the dropping of > narrow/wide Unicode builds? Probably not, but I've never actually seen the stable ABI used in practice (and I don't know if cffi restricts itself to the stable ABI). I'm not sure that any toolchain (such as bdist_wininst or wheel) really supports it (in the sense that they tend to make the assumption that if there is a C extension, the code is version-specific). None of these are insurmountable problems, though. Paul. From fwierzbicki at gmail.com Wed Feb 27 20:01:28 2013 From: fwierzbicki at gmail.com (fwierzbicki at gmail.com) Date: Wed, 27 Feb 2013 11:01:28 -0800 Subject: [Python-Dev] Python Language Summit at PyCon: Agenda In-Reply-To: <13FB490E-3548-4959-9C2B-2880B8ACA6F5@voidspace.org.uk> References: <13FB490E-3548-4959-9C2B-2880B8ACA6F5@voidspace.org.uk> Message-ID: On Wed, Feb 27, 2013 at 8:51 AM, Michael Foord wrote: > If you have other items you'd like to discuss please let me know and I can add them to the agenda. I'd like to discuss merging Jython's standard Lib (the *.py files). We have in the past had agreement that this would be a good idea - I just want to bring it up since I think this is probably the year that I'm actually going to do it. -Frank From arigo at tunes.org Wed Feb 27 20:08:40 2013 From: arigo at tunes.org (Armin Rigo) Date: Wed, 27 Feb 2013 20:08:40 +0100 Subject: [Python-Dev] cffi in stdlib In-Reply-To: References: <78C26306-4413-4727-B61F-04B8AEDEECF7@mac.com> <5E7B690F-0A81-4351-A145-8DF0CCFF69AB@mac.com> Message-ID: Hi Paul, On Wed, Feb 27, 2013 at 7:24 PM, Paul Moore wrote: > On 27 February 2013 11:53, Maciej Fijalkowski wrote: >> I think it means you can't use the ABI version and specify the calling >> convention. It's a reasonable bug report (the calling convention on >> API version works though) > > That would be a deal-breaker for my use case of quick scripts working > with the Windows API on a machine with no C compiler. I'd have to use > ctypes in that case until cffi had this feature. That's not correct: you can't indeed give the calling convention, but it is not needed for the common case. What is not supported is only Python-defined callbacks using the Windows-specific convention --- as documented, there is a workaround for that case. And, in case you wonder, this automatic detection comes from ctypes. I copied the hacked-up version of libffi for Windows from ctypes to cffi, and the logic to detect the actual calling convention and support both is there. The difference is only that in ctypes, after it did the call (always successfully), it checks the convention that was found to be used by the C function, and if it differs from the one specified by the user, then it complains. I basically just removed the "complaining" part. A bient?t, Armin. From brett at python.org Wed Feb 27 20:21:59 2013 From: brett at python.org (Brett Cannon) Date: Wed, 27 Feb 2013 14:21:59 -0500 Subject: [Python-Dev] Python Language Summit at PyCon: Agenda In-Reply-To: References: <13FB490E-3548-4959-9C2B-2880B8ACA6F5@voidspace.org.uk> Message-ID: On Wed, Feb 27, 2013 at 2:01 PM, fwierzbicki at gmail.com < fwierzbicki at gmail.com> wrote: > On Wed, Feb 27, 2013 at 8:51 AM, Michael Foord > wrote: > > If you have other items you'd like to discuss please let me know and I > can add them to the agenda. > > I'd like to discuss merging Jython's standard Lib (the *.py files). We > have in the past had agreement that this would be a good idea - I just > want to bring it up since I think this is probably the year that I'm > actually going to do it. > Do you mean more generally getting more pure Python implementations of modules in the stdlib? If so then as a reference there is http://bugs.python.org/issue16651 which lists the modules in the stdlib w/ only extension module implementations (although operator and random have patches; the latter is waiting for Raymond to approve). And if I am right about what you are suggesting, Frank, this should probably be expanded to a more concerted effort with IronPython and PyPy. IOW it warrants a discussion. =) -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Wed Feb 27 20:26:23 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 27 Feb 2013 19:26:23 +0000 Subject: [Python-Dev] cffi in stdlib In-Reply-To: References: <78C26306-4413-4727-B61F-04B8AEDEECF7@mac.com> <5E7B690F-0A81-4351-A145-8DF0CCFF69AB@mac.com> Message-ID: On 27 February 2013 19:08, Armin Rigo wrote: > That's not correct: you can't indeed give the calling convention, but > it is not needed for the common case. What is not supported is only > Python-defined callbacks using the Windows-specific convention --- as > documented, there is a workaround for that case. OK, that's cool. As I said, I really need to actually experiment with cffi - this thread has certainly made me want to do so. Paul From fwierzbicki at gmail.com Wed Feb 27 20:33:30 2013 From: fwierzbicki at gmail.com (fwierzbicki at gmail.com) Date: Wed, 27 Feb 2013 11:33:30 -0800 Subject: [Python-Dev] Python Language Summit at PyCon: Agenda In-Reply-To: References: <13FB490E-3548-4959-9C2B-2880B8ACA6F5@voidspace.org.uk> Message-ID: On Wed, Feb 27, 2013 at 11:21 AM, Brett Cannon wrote: > Do you mean more generally getting more pure Python implementations of > modules in the stdlib? If so then as a reference there is > http://bugs.python.org/issue16651 which lists the modules in the stdlib w/ > only extension module implementations (although operator and random have > patches; the latter is waiting for Raymond to approve). That is part of it, although my bigger goal is slightly more ambitious. I'm hoping to eventually delete the entire Lib/ directory from Jython and just pull in CPython's. > And if I am right > about what you are suggesting, Frank, this should probably be expanded to a > more concerted effort with IronPython and PyPy. IOW it warrants a > discussion. =) Oh sure sorry - I have some tunnel vision lately :) I am suggesting that we push forward on the "shared library" approach to the files in the Lib/* directory, so that would certainly include IronPython and PyPy as well I hope. The easy part for Jython is pushing some of our "if is_jython:" stuff into the appropriate spots in CPython's Lib/. I'd also like to do something at the top of CPython specific .py files that would fail the import in case it is called from Jython. I suspect that OS packagers would like it if the Lib directory for a particular Python version could be entirely shared between implementations. There are a couple of spots that might be more controversial. For example, Jython has a file Lib/zlib.py that implements zlib in terms of the existing Java support for zlib. I do wonder if such a file is acceptable in CPython's Lib since its 195 lines of code would be entirely skipped by CPython. Anyway I think I might be rambling - it seems like a good thing to discuss :) -Frank From dholth at gmail.com Wed Feb 27 20:55:30 2013 From: dholth at gmail.com (Daniel Holth) Date: Wed, 27 Feb 2013 14:55:30 -0500 Subject: [Python-Dev] cffi in stdlib In-Reply-To: <5E7B690F-0A81-4351-A145-8DF0CCFF69AB@mac.com> References: <78C26306-4413-4727-B61F-04B8AEDEECF7@mac.com> <5E7B690F-0A81-4351-A145-8DF0CCFF69AB@mac.com> Message-ID: On Wed, Feb 27, 2013 at 4:18 AM, Ronald Oussoren wrote: > > On 27 Feb, 2013, at 10:06, Maciej Fijalkowski wrote: > >> On Wed, Feb 27, 2013 at 9:29 AM, Ronald Oussoren wrote: >>> >>> On 26 Feb, 2013, at 16:13, Maciej Fijalkowski wrote: >>> >>>> Hello. >>>> >>>> I would like to discuss on the language summit a potential inclusion >>>> of cffi[1] into stdlib. >>> >>> The API in general looks nice, but I do have some concens w.r.t. including cffi in the stdlib. >>> >>> 1. Why is cffi completely separate from ctypes, instead of layered on top of it? That is, add a utility module to ctypes that can parse C declarations and generate the right ctypes definitions. >> >> Because ctypes API is a mess and magic. We needed a cleaner (and much >> smaller) model. > > The major advantages of starting over is probably that you can hide the complexity and that opens opportunities for optimizations. That said, I'm not convinced that ctypes is unnecessarily complex. cffi actually does have a ctypes backend in addition to the ffi and "compile a CPython extension" backends. But the ctypes backend is guaranteed to be slow and messy because it is ctypes. From techtonik at gmail.com Wed Feb 27 21:20:52 2013 From: techtonik at gmail.com (anatoly techtonik) Date: Wed, 27 Feb 2013 23:20:52 +0300 Subject: [Python-Dev] Python Language Summit at PyCon: Agenda In-Reply-To: <13FB490E-3548-4959-9C2B-2880B8ACA6F5@voidspace.org.uk> References: <13FB490E-3548-4959-9C2B-2880B8ACA6F5@voidspace.org.uk> Message-ID: On Wed, Feb 27, 2013 at 7:51 PM, Michael Foord wrote: > Hello all, > > PyCon, and the Python Language Summit, is nearly upon us. We have a good > number of people confirmed to attend. If you are intending to come to the > language summit but haven't let me know please do so. > > The agenda of topics for discussion so far includes the following: > > * A report on pypy status - Maciej and Armin > * Jython and IronPython status reports - Dino / Frank > * Packaging (Doug Hellmann and Monty Taylor at least) > * Cleaning up interpreter initialisation (both in hopes of finding areas > to rationalise and hence speed things up, as well as making things > more embedding friendly). Nick Coghlan > * Adding new async capabilities to the standard library (Guido) > * cffi and the standard library - Maciej > * flufl.enum and the standard library - Barry Warsaw > * The argument clinic - Larry Hastings > > If you have other items you'd like to discuss please let me know and I can > add them to the agenda. > I won't be able to visit, so this PyCon promises to be all good nice and relaxing for everybody. =) But here is some things that can spice up the meeting in case it becomes boring. * poor introspection capabilities * if you pause the code - what kind of data you'd like to be available? * magical locals() dict that breaks the rules * execution frames that lose information present in original source (such as function vs method and probably others) * as an exercise - try to build a scroller for a running Python script * it is impossible for Python 2 and probably for Python 3 as well * visibility issues with language development * physically split the information flow about work being done on interpreter and stdlib * split the information about stdlib development by modules * describe modules composing in stdlib in formal way https://bitbucket.org/techtonik/python-stdlib * build a roadmap by module (join personal wishlist from involved people) * external people can not join teams dedicated only to several modules people are interested in * IPython and PyPy as a distraction for people who could improve core language and stdlib here * security by obscurity in legal position of PSF towards contributors https://code.google.com/legal/individual-cla-v1.0.html vs http://www.python.org/psf/contrib/contrib-form/ + http://www.samurajdata.se/opensource/mirror/licenses/afl-2.1.php or http://www.python.org/psf/contrib/contrib-form/ + http://opensource.org/licenses/apache2.0.php and why PSF doesn't comply the 4. Redistribution clause from Apache 2.0 license * how to get more pictures and less text for reference, especially for internals * user story approach for writing PEPs Can only wish to have a constructive PyCon with great results. Bye. -- anatoly t. -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Wed Feb 27 21:31:45 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 27 Feb 2013 20:31:45 +0000 Subject: [Python-Dev] cffi in stdlib In-Reply-To: References: <78C26306-4413-4727-B61F-04B8AEDEECF7@mac.com> <5E7B690F-0A81-4351-A145-8DF0CCFF69AB@mac.com> Message-ID: On 27 February 2013 19:26, Paul Moore wrote: > On 27 February 2013 19:08, Armin Rigo wrote: >> That's not correct: you can't indeed give the calling convention, but >> it is not needed for the common case. What is not supported is only >> Python-defined callbacks using the Windows-specific convention --- as >> documented, there is a workaround for that case. > > OK, that's cool. As I said, I really need to actually experiment with > cffi - this thread has certainly made me want to do so. Actually, looking at the cffi documentation, I can't see how I'd translate the following very simple example of something I do quite a lot in ctypes: from ctypes import windll MessageBox = windll.User32.MessageBoxW MessageBox(0, "Hello, world!", "Title", 0) Note - I wrote this from memory, I honestly don't know without looking it up the precise argument types for MessageBoxW - and even if I did, I suspect they would all be macros from windows.h. I don't want to invoke the C compiler by using verify, and I don't see in the docs how I'd get the macro definitions any other way. If anyone could show me a cffi equivalent, I'd be very grateful. Paul From greg.ewing at canterbury.ac.nz Wed Feb 27 22:57:50 2013 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 28 Feb 2013 10:57:50 +1300 Subject: [Python-Dev] cffi in stdlib In-Reply-To: <20130227143141.4aef5635@pitrou.net> References: <20130226190620.1FF24250BDA@webabinitio.net> <20130226202153.2b923404@pitrou.net> <512D41F9.3050109@canterbury.ac.nz> <20130227143141.4aef5635@pitrou.net> Message-ID: <512E815E.703@canterbury.ac.nz> Antoine Pitrou wrote: > We have to find sufficiently silly species of snakes, though. Glancing through http://en.wikipedia.org/wiki/List_of_snakes, we have: Wart snakes Java wart snakes Pipe snakes Stiletto snakes Rubber boas Dog-faced water snakes Shovel-nosed snakes Hook-nosed snakes Leaf-nosed snakes Lyre snakes Cat-eyed snakes Worm snakes North American hog-nosed snakes Moccasins Humpnose vipers Puff adders Carpet vipers False horned vipers Sunbeam snakes And then there's this: http://www.nairaland.com/1009227/new-species-snake-discovered-brazil -- Greg From tjreedy at udel.edu Wed Feb 27 23:58:42 2013 From: tjreedy at udel.edu (Terry Reedy) Date: Wed, 27 Feb 2013 17:58:42 -0500 Subject: [Python-Dev] cffi in stdlib In-Reply-To: References: <78C26306-4413-4727-B61F-04B8AEDEECF7@mac.com> <5E7B690F-0A81-4351-A145-8DF0CCFF69AB@mac.com> Message-ID: On 2/27/2013 3:31 PM, Paul Moore wrote: > from ctypes import windll > MessageBox = windll.User32.MessageBoxW > MessageBox(0, "Hello, world!", "Title", 0) > Note - I wrote this from memory, Gee, that is pretty easy. Worked perfectly -- Terry Jan Reedy From arigo at tunes.org Thu Feb 28 00:18:44 2013 From: arigo at tunes.org (Armin Rigo) Date: Thu, 28 Feb 2013 00:18:44 +0100 Subject: [Python-Dev] cffi in stdlib In-Reply-To: References: <78C26306-4413-4727-B61F-04B8AEDEECF7@mac.com> <5E7B690F-0A81-4351-A145-8DF0CCFF69AB@mac.com> Message-ID: Hi Paul, On Wed, Feb 27, 2013 at 9:31 PM, Paul Moore wrote: > from ctypes import windll > MessageBox = windll.User32.MessageBoxW > MessageBox(0, "Hello, world!", "Title", 0) You are right that it's a bit cumbersome in cffi up to and including 0.5, but in the cffi trunk all standard Windows types are included. So the general answer to your question is: we google MessageBox and copy that line from the microsoft site, and manually remove the unnecessary WINAPI and _In_opt declarations: from cffi import FFI ffi = FFI() ffi.cdef(""" int MessageBox(HWND hWnd, LPCTSTR lpText, LPCTSTR lpCaption, UINT uType); """) lib = ffi.dlopen("USER32.DLL") lib.MessageBox(ffi.NULL, "Hello, world!", "Title", 0) That's a slightly unfair example, because in this case it happens to work with ctypes without specifying the argtypes and the restype. I would argue that this feature of ctypes is not a good thing: it's mostly the same as saying "you only need to declare argtypes and restype if you get nonsense results or segfaults". Note for completeness that the version with verify() simply replaces "lib = ffi.dlopen("USER32.DLL")" with "lib = ffi.verify("")" (no #include necessary here). Then you cannot misdeclare anything without getting clear compile-time errors at the verify(). The compilation occurs only once (it's cached) and by writing two lines in your setup.py you can distribute binary installations, just like you do with hand-written C extension modules; so the extra burden of accessing the API level is in my opinion very small compared to its segfault-avoiding gain. But I know that either level of access can make sense in different situations. Typically, on Windows, the ABI-level works fine; as argued elsewhere, on other platforms and/or with some libraries, the API-level is definitely more suited. A bient?t, Armin. From nyamatongwe at me.com Thu Feb 28 00:34:56 2013 From: nyamatongwe at me.com (Neil Hodgson) Date: Thu, 28 Feb 2013 10:34:56 +1100 Subject: [Python-Dev] cffi in stdlib In-Reply-To: References: <78C26306-4413-4727-B61F-04B8AEDEECF7@mac.com> <5E7B690F-0A81-4351-A145-8DF0CCFF69AB@mac.com> Message-ID: Armin Rigo: > So the general answer to your question is: we google MessageBox and > copy that line from the microsoft site, and manually remove the > unnecessary WINAPI and _In_opt declarations: Wouldn't it be better to understand the SAL annotations like _In_opt so that spurious NULLs (for example) produce a good exception from cffi instead of failing inside the system call? Neil From ericsnowcurrently at gmail.com Thu Feb 28 01:49:23 2013 From: ericsnowcurrently at gmail.com (Eric Snow) Date: Wed, 27 Feb 2013 17:49:23 -0700 Subject: [Python-Dev] Python Language Summit at PyCon: Agenda In-Reply-To: References: <13FB490E-3548-4959-9C2B-2880B8ACA6F5@voidspace.org.uk> Message-ID: On Wed, Feb 27, 2013 at 12:33 PM, fwierzbicki at gmail.com wrote: > There are a couple of spots that might be more controversial. For > example, Jython has a file Lib/zlib.py that implements zlib in terms > of the existing Java support for zlib. I do wonder if such a file is > acceptable in CPython's Lib since its 195 lines of code would be > entirely skipped by CPython. Even if it's pure Python, that sounds like an accelerator module (a la PEP 399) to me. -eric From barry at python.org Thu Feb 28 04:37:49 2013 From: barry at python.org (Barry Warsaw) Date: Wed, 27 Feb 2013 22:37:49 -0500 Subject: [Python-Dev] Python Language Summit at PyCon: Agenda In-Reply-To: References: <13FB490E-3548-4959-9C2B-2880B8ACA6F5@voidspace.org.uk> Message-ID: <20130227223749.2f06a328@anarchist.wooz.org> On Feb 27, 2013, at 11:33 AM, fwierzbicki at gmail.com wrote: >I am suggesting that we push forward on the "shared library" approach to the >files in the Lib/* directory, so that would certainly include IronPython and >PyPy as well I hope. +1 >The easy part for Jython is pushing some of our "if is_jython:" stuff >into the appropriate spots in CPython's Lib/. I wonder if there isn't a better way to do this than sprinkling is_jython, is_pypy, is_ironpython, is_thenextbigthing all over the code base. I have no bright ideas here, but it seems like a feature matrix would be a better way to go than something that assumes a particular Python implementation has a particular feature set (which may change in the future). -Barry From barry at python.org Thu Feb 28 04:42:10 2013 From: barry at python.org (Barry Warsaw) Date: Wed, 27 Feb 2013 22:42:10 -0500 Subject: [Python-Dev] Python Language Summit at PyCon: Agenda In-Reply-To: <13FB490E-3548-4959-9C2B-2880B8ACA6F5@voidspace.org.uk> References: <13FB490E-3548-4959-9C2B-2880B8ACA6F5@voidspace.org.uk> Message-ID: <20130227224210.288a334d@anarchist.wooz.org> On Feb 27, 2013, at 04:51 PM, Michael Foord wrote: >If you have other items you'd like to discuss please let me know and I can >add them to the agenda. I'd like to have some discussions around promotion of Python 3, how we can accelerate its adoption, availability of supporting packages, what critical projects are still missing, etc. -Barry From g.brandl at gmx.net Thu Feb 28 08:36:19 2013 From: g.brandl at gmx.net (Georg Brandl) Date: Thu, 28 Feb 2013 08:36:19 +0100 Subject: [Python-Dev] Python Language Summit at PyCon: Agenda In-Reply-To: <13FB490E-3548-4959-9C2B-2880B8ACA6F5@voidspace.org.uk> References: <13FB490E-3548-4959-9C2B-2880B8ACA6F5@voidspace.org.uk> Message-ID: Am 27.02.2013 17:51, schrieb Michael Foord: > Hello all, > > PyCon, and the Python Language Summit, is nearly upon us. We have a good number of people confirmed to attend. If you are intending to come to the language summit but haven't let me know please do so. > > The agenda of topics for discussion so far includes the following: > > * A report on pypy status - Maciej and Armin > * Jython and IronPython status reports - Dino / Frank > * Packaging (Doug Hellmann and Monty Taylor at least) > * Cleaning up interpreter initialisation (both in hopes of finding areas > to rationalise and hence speed things up, as well as making things > more embedding friendly). Nick Coghlan > * Adding new async capabilities to the standard library (Guido) > * cffi and the standard library - Maciej > * flufl.enum and the standard library - Barry Warsaw > * The argument clinic - Larry Hastings > > If you have other items you'd like to discuss please let me know and I can add them to the agenda. May I in absentia propose at least a short discussion of the XML fixes and accompanying security releases? FWIW, for 3.2 and 3.3 I have no objections to secure-by-default. Georg From arigo at tunes.org Thu Feb 28 09:02:59 2013 From: arigo at tunes.org (Armin Rigo) Date: Thu, 28 Feb 2013 09:02:59 +0100 Subject: [Python-Dev] cffi in stdlib In-Reply-To: References: <78C26306-4413-4727-B61F-04B8AEDEECF7@mac.com> <5E7B690F-0A81-4351-A145-8DF0CCFF69AB@mac.com> Message-ID: Hi Neil, On Thu, Feb 28, 2013 at 12:34 AM, Neil Hodgson wrote: > Wouldn't it be better to understand the SAL annotations like _In_opt so that spurious NULLs (for example) produce a good exception from cffi instead of failing inside the system call? Maybe. Feel like adding an issue to https://bitbucket.org/cffi/cffi/issues, with references? This looks like a Windows-specific extension, which means that I don't automatically know about it. A bient?t, Armin. From p.f.moore at gmail.com Thu Feb 28 09:27:25 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 28 Feb 2013 08:27:25 +0000 Subject: [Python-Dev] cffi in stdlib In-Reply-To: References: <78C26306-4413-4727-B61F-04B8AEDEECF7@mac.com> <5E7B690F-0A81-4351-A145-8DF0CCFF69AB@mac.com> Message-ID: On 27 February 2013 23:18, Armin Rigo wrote: > from cffi import FFI > ffi = FFI() > ffi.cdef(""" > int MessageBox(HWND hWnd, LPCTSTR lpText, LPCTSTR lpCaption, UINT uType); > """) > lib = ffi.dlopen("USER32.DLL") > lib.MessageBox(ffi.NULL, "Hello, world!", "Title", 0) Yeah, that's loads better than 0.5. Presumably ffi.NULL isn't needed and I can use 0? (After all, 0 and NULL are equivalent in C, so that's not a correctness issue). The auto-conversion of strings is a huge improvement. > That's a slightly unfair example, because in this case it happens to > work with ctypes without specifying the argtypes and the restype. I > would argue that this feature of ctypes is not a good thing: it's > mostly the same as saying "you only need to declare argtypes and > restype if you get nonsense results or segfaults". That's a bit unfair. I'd say "you only need to declare argtypes if you're dealing with things more complex than integers, strings and null pointers". Which means you're fine for a huge proportion of the Windows API. On the other hand, going to the API level and being able to use all the Windows constants without having to look them up (their values aren't documented, so googling doesn't help :-() is a huge plus. Paul. From arigo at tunes.org Thu Feb 28 10:06:00 2013 From: arigo at tunes.org (Armin Rigo) Date: Thu, 28 Feb 2013 10:06:00 +0100 Subject: [Python-Dev] cffi in stdlib In-Reply-To: References: <78C26306-4413-4727-B61F-04B8AEDEECF7@mac.com> <5E7B690F-0A81-4351-A145-8DF0CCFF69AB@mac.com> Message-ID: Hi Paul, On Thu, Feb 28, 2013 at 9:27 AM, Paul Moore wrote: > Presumably ffi.NULL isn't needed and I can use 0? (After all, 0 and NULL are equivalent in C, so that's > not a correctness issue). Indeed. I created https://bitbucket.org/cffi/cffi/issue/61/convert-0-to-a-null-pointer. In C, NULL is typically "#define NULL (void *)0", not just 0, which means that there are not 100% equivalent. But it's true that in C you can pass the constant 0 to mean "a NULL pointer argument". The fact that it's not working this way in CFFI is definitely a bug. >> That's a slightly unfair example, because in this case it happens to >> work with ctypes without specifying the argtypes and the restype. I >> would argue that this feature of ctypes is not a good thing: it's >> mostly the same as saying "you only need to declare argtypes and >> restype if you get nonsense results or segfaults". > > That's a bit unfair. I'd say "you only need to declare argtypes if > you're dealing with things more complex than integers, strings and > null pointers". Which means you're fine for a huge proportion of the > Windows API. Yes, you're right, and the 32-bit Windows platform is still important. However, it only works on 32-bit. On typical 64-bit Posix environments, if you don't declare argtypes/restype, you usually end up very quickly with confusion between "int" and "long". And I think that even on 64-bit Windows, passing 0 as a NULL pointer is buggy, because it will pass a 32-bit 0. (It may be that it doesn't actually make a difference and works anyway, but I'm not sure.) Similarly, a function that returns a pointer (e.g. a handle on Windows) will not work without an explicit restype on any 64-bit platform I know of. "Explicit is better than implicit"... A bient?t, Armin. From fijall at gmail.com Thu Feb 28 10:06:20 2013 From: fijall at gmail.com (Maciej Fijalkowski) Date: Thu, 28 Feb 2013 11:06:20 +0200 Subject: [Python-Dev] cffi in stdlib In-Reply-To: References: <78C26306-4413-4727-B61F-04B8AEDEECF7@mac.com> <5E7B690F-0A81-4351-A145-8DF0CCFF69AB@mac.com> Message-ID: On Thu, Feb 28, 2013 at 10:27 AM, Paul Moore wrote: > On 27 February 2013 23:18, Armin Rigo wrote: >> from cffi import FFI >> ffi = FFI() >> ffi.cdef(""" >> int MessageBox(HWND hWnd, LPCTSTR lpText, LPCTSTR lpCaption, UINT uType); >> """) >> lib = ffi.dlopen("USER32.DLL") >> lib.MessageBox(ffi.NULL, "Hello, world!", "Title", 0) > > Yeah, that's loads better than 0.5. Presumably ffi.NULL isn't needed > and I can use 0? (After all, 0 and NULL are equivalent in C, so that's > not a correctness issue). The auto-conversion of strings is a huge > improvement. > >> That's a slightly unfair example, because in this case it happens to >> work with ctypes without specifying the argtypes and the restype. I >> would argue that this feature of ctypes is not a good thing: it's >> mostly the same as saying "you only need to declare argtypes and >> restype if you get nonsense results or segfaults". > > That's a bit unfair. I'd say "you only need to declare argtypes if > you're dealing with things more complex than integers, strings and > null pointers". Which means you're fine for a huge proportion of the > Windows API. No, if the int is of the wrong size or you pass int instead of float (but you meant float) you get nonsense or segfaults. From p.f.moore at gmail.com Thu Feb 28 10:21:25 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 28 Feb 2013 09:21:25 +0000 Subject: [Python-Dev] cffi in stdlib In-Reply-To: References: <78C26306-4413-4727-B61F-04B8AEDEECF7@mac.com> <5E7B690F-0A81-4351-A145-8DF0CCFF69AB@mac.com> Message-ID: On 28 February 2013 09:06, Armin Rigo wrote: > And I think > that even on 64-bit Windows, passing 0 as a NULL pointer is buggy, > because it will pass a 32-bit 0. (It may be that it doesn't actually > make a difference and works anyway, but I'm not sure.) Similarly, a > function that returns a pointer (e.g. a handle on Windows) will not > work without an explicit restype on any 64-bit platform I know of. > "Explicit is better than implicit"... Ha, interesting. My test case was run on 64-bit Windows with 64-bit Python, and worked fine. But you're right, explicit is better, and I may have been luckier than I realised. And anyway my original point about whether cffi was as concise as ctypes for simple ABI-level Windows calls has been answered - yes it is, at least the next version will be. Thanks, Paul. From ncoghlan at gmail.com Thu Feb 28 10:15:07 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 28 Feb 2013 19:15:07 +1000 Subject: [Python-Dev] Python Language Summit at PyCon: Agenda In-Reply-To: <20130227223749.2f06a328@anarchist.wooz.org> References: <13FB490E-3548-4959-9C2B-2880B8ACA6F5@voidspace.org.uk> <20130227223749.2f06a328@anarchist.wooz.org> Message-ID: On Thu, Feb 28, 2013 at 1:37 PM, Barry Warsaw wrote: > On Feb 27, 2013, at 11:33 AM, fwierzbicki at gmail.com wrote: >>The easy part for Jython is pushing some of our "if is_jython:" stuff >>into the appropriate spots in CPython's Lib/. > > I wonder if there isn't a better way to do this than sprinkling is_jython, > is_pypy, is_ironpython, is_thenextbigthing all over the code base. I have no > bright ideas here, but it seems like a feature matrix would be a better way to > go than something that assumes a particular Python implementation has a > particular feature set (which may change in the future). Yes, avoiding that kind of thing is a key motivation for sys.implementation. Any proposal for "is_jython" blocks should instead be reformulated as a proposal for new sys.implementation attributes. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From solipsis at pitrou.net Thu Feb 28 10:22:43 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 28 Feb 2013 10:22:43 +0100 Subject: [Python-Dev] cffi in stdlib References: <20130226190620.1FF24250BDA@webabinitio.net> <20130226202153.2b923404@pitrou.net> <512D41F9.3050109@canterbury.ac.nz> <20130227143141.4aef5635@pitrou.net> <512E815E.703@canterbury.ac.nz> Message-ID: <20130228102243.7ab4014d@pitrou.net> Le Thu, 28 Feb 2013 10:57:50 +1300, Greg Ewing a ?crit : > Antoine Pitrou wrote: > > We have to find sufficiently silly species of snakes, though. > > Glancing through http://en.wikipedia.org/wiki/List_of_snakes, > we have: > > Wart snakes > Java wart snakes > Pipe snakes > Stiletto snakes > Rubber boas > Dog-faced water snakes > Shovel-nosed snakes I'm looking forward to our shovel-nosed release plugging the XML security holes, then. Regards Antoine. From solipsis at pitrou.net Thu Feb 28 10:24:35 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 28 Feb 2013 10:24:35 +0100 Subject: [Python-Dev] cffi in stdlib References: <78C26306-4413-4727-B61F-04B8AEDEECF7@mac.com> <5E7B690F-0A81-4351-A145-8DF0CCFF69AB@mac.com> Message-ID: <20130228102435.32938bda@pitrou.net> Le Thu, 28 Feb 2013 10:06:00 +0100, Armin Rigo a ?crit : > > Yes, you're right, and the 32-bit Windows platform is still important. > However, it only works on 32-bit. On typical 64-bit Posix > environments, if you don't declare argtypes/restype, you usually end > up very quickly with confusion between "int" and "long". And I think > that even on 64-bit Windows, passing 0 as a NULL pointer is buggy, > because it will pass a 32-bit 0. (It may be that it doesn't actually > make a difference and works anyway, but I'm not sure.) Similarly, a > function that returns a pointer (e.g. a handle on Windows) will not > work without an explicit restype on any 64-bit platform I know of. Indeed, argument width is quite important :-) Also, on 64-bit Windows, long is 32-bit while size_t and pointers are 64-bit. Regards Antoine. From solipsis at pitrou.net Thu Feb 28 10:30:37 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 28 Feb 2013 10:30:37 +0100 Subject: [Python-Dev] Python Language Summit at PyCon: Agenda References: <13FB490E-3548-4959-9C2B-2880B8ACA6F5@voidspace.org.uk> Message-ID: <20130228103037.1b60878b@pitrou.net> Le Wed, 27 Feb 2013 11:33:30 -0800, "fwierzbicki at gmail.com" a ?crit : > > There are a couple of spots that might be more controversial. For > example, Jython has a file Lib/zlib.py that implements zlib in terms > of the existing Java support for zlib. I do wonder if such a file is > acceptable in CPython's Lib since its 195 lines of code would be > entirely skipped by CPython. That's a bit annoying. How will we know that the code still works, even though our buildbots don't exercise it? Also, what happens if the code doesn't work anymore? Regards Antoine. From nyamatongwe at me.com Thu Feb 28 09:50:39 2013 From: nyamatongwe at me.com (Neil Hodgson) Date: Thu, 28 Feb 2013 19:50:39 +1100 Subject: [Python-Dev] cffi in stdlib In-Reply-To: References: <78C26306-4413-4727-B61F-04B8AEDEECF7@mac.com> <5E7B690F-0A81-4351-A145-8DF0CCFF69AB@mac.com> Message-ID: <856B71AE-6855-4F4A-A3A9-DFCF1B9A8677@me.com> Armin Rigo: > Maybe. Feel like adding an issue to > https://bitbucket.org/cffi/cffi/issues, with references? OK, issue #62 added. > This looks > like a Windows-specific extension, which means that I don't > automatically know about it. While SAL is Windows-specific, gcc supports some similar attributes including nonnull and sentinel. Neil From ericsnowcurrently at gmail.com Thu Feb 28 10:56:07 2013 From: ericsnowcurrently at gmail.com (Eric Snow) Date: Thu, 28 Feb 2013 02:56:07 -0700 Subject: [Python-Dev] [Python-checkins] peps: Pre-alpha draft for PEP 435 (enum). The name is not important at the moment, as In-Reply-To: <9B1795C95533CA46A83BA1EAD4B01030DFE09F@flonidanmail.flonidan.net> References: <3ZCsSP0rSXzQZS@mail.python.org> <20130223150231.GA20921@sleipnir.bytereef.org> <20130224204000.6309bd6c@anarchist.wooz.org> <512ADB41.5040109@stoneleaf.us> <20130224231410.0023038c@anarchist.wooz.org> <512B7802.9010306@stoneleaf.us> <512B7F43.5060406@stoneleaf.us> <87txoz3fg3.fsf@uwakimon.sk.tsukuba.ac.jp> <512C5BF7.9090803@stoneleaf.us> <87mwur2zkz.fsf@uwakimon.sk.tsukuba.ac.jp> <512CDCDC.2090509@stoneleaf.us> <20130226201757.535abe3e@pitrou.net> <87ip5e2zxs.fsf@uwakimon.sk.tsukuba.ac.jp> <20130227143001.3cfce13a@pitrou.net> <9B1795C95533CA46A83BA1EAD4B01030DFE09F@flonidanmail.flonidan.net> Message-ID: Having had some time to think about this problem space, here's my take on it: =============================== The problem-space can be broken down into four layers: 1. the items 2. interaction between grouped items 3. the grouping itself 4. conversion from a class to a group Here are *potential* characteristics at each layer: items ----- * immutable * unique * proxies or otherwise implicitly exposes underlying value * or only explicitly exposes value * or *is* the value with extra attributes added on * int-like * immutable underlying values * hashable * repr shows group name, item name, and underlying value * str shows just item name * value-less * a class that subclasses the group items and item interaction -------------------------- * equality comparible * richly comparable * comparison by identity only * interaction restricted to within group * flag-like (bitwise operations, etc.) groups ------ * iterable * sized * ordered (sortable) * inheritance to extend the group * a class rather than an instance of an enum type * query: name -> value * immutable * immutable underlying collection of items * allow aliases for actual items * group itself is named * has __qualname__ * repr shows items * no duplicates * a sequence without semantically meaningful values * unserialize (value -> item) using call syntax on group conversion ---------- * inherit a base enum type * use a class decorator * use a factory function, perhaps not even on a class (for more dynamic enum creation) * __getitem__() trick for defining value-less items * auto-numbering * explicit values * implicit values using ... * filter out "private" names * use only upper-case names There is a danger in trying to make an "enum" that is capable of doing everything. People need a simple common ground. When an object is an enum, a developer should be able to know immediately how to interact with it and that interaction should have a small cross-section. =============================== With that in mind and considering a lot of the discussion that has gone on, here's an API that tries to find a middle ground: "items" NamedValue #Nick's recipe NamedItem #an opaque named wrapper around a value group # the "enum" to which the item belongs value # the wrapped value __qualname__ # group qualname + item name __repr__ -> class + qualname + value __str__ -> item name __eq__ -> NotImplemented expose(ns) # export the value out to any namespace OrderedItem(NamedItem) # a totally ordered item __eq__ -> other is self __lt__ -> ask self.group to decide FlagItem(NamedItem) # a bitwise/set operation compatible item __and__ -> ask self.group to do it ... FlagsItem(FlagItem) # the result of FlagItem bitwise operations __repr__ -> shows the the items or'ed together items # the items that were joined together by the operation IntItem(NamedItem) # for those that really need a named item to be a little less opaque __int__ __index__ __bool__ " groups" Group __name__ __qualname__ __items__ # the underlying collection of items (tuple or frozenset) _ns # a GroupValues instance for the group __repr__ -> group name + items __call__(value) -> item # essentially the unserializer for an item _expose(ns) # calls expose() on each item in the group OrderedGroup # corresponds to OrderedItem FlagGroup # corresponds to FlagItem GroupValues # a Mapping proxy around a group's (name -> value) "conversion" named_values_from_class() -> {name: value} # simply extract the values from the class auto_int_from_class() # a classic enum auto_bin_from_class() # similar but steps in powers of 2 as_enum(*, ordered=False, flagged=False, converter=..., strict=True, **kwargs) # the main class decorator factory Examples: @as_enum() class Spam: A = ... B = ... C = ... Spam.A._expose(globals()) @as_enum(converter=auto_bin_from_class, flagged=True) class Ham: A = 1 B = ... C = ... D = 32 Ham.A == 1 # False Ham.B | Ham.C # a FlagsItem containing the two iter(Ham) # TypeError iter(Ham.__items__) # the items iter(Ham._ns) # the names iter(Ham._ns.values()) # the values Key points: * uses a class decorator rather than inheritance to convert a class into an enum * the objects make use of __qualname__ as much as is reasonable * the various Item classes are meant to be compatible (e.g. multiple inheritance) * groups and items are immutable * if order matters, using a class decorator requires that classes use OrderedDict by default for their namespace, this is a feasible change once we have an OrderedDict written in C (issue #16991) * value-less NamedItems are totally valid and the use of ... in converters would support their creation * the Group API is very simple (what you are using over and over) * the definition/conversion API is a little thicker with the basic case simplified * NamedItems are simply opaque wrappers (with the mild exception of IntItem) * Nick's NamedValue is utterly transparent and yet compatible with use in a Group In the interest of having something more concrete, a very raw and still relatively non-functional implementation of the above API can be found at: https://bitbucket.org/ericsnowcurrently/odds_and_ends/src/default/enum.py Finally, like I said before, the smaller the API the better. Obviously what I've got here could be distilled. However, it does capture the way I see the separate layers here. -eric From steve at pearwood.info Thu Feb 28 12:08:55 2013 From: steve at pearwood.info (Steven D'Aprano) Date: Thu, 28 Feb 2013 22:08:55 +1100 Subject: [Python-Dev] Python Language Summit at PyCon: Agenda In-Reply-To: References: <13FB490E-3548-4959-9C2B-2880B8ACA6F5@voidspace.org.uk> Message-ID: <512F3AC7.1070906@pearwood.info> On 28/02/13 07:20, anatoly techtonik wrote: > * as an exercise - try to build a scroller for a running Python script > * it is impossible for Python 2 and probably for Python 3 as well What do you mean by "a scroller"? [...] > and why PSF doesn't comply the 4. Redistribution clause from Apache 2.0 > license If you are correct, that may be a real issue that needs to be resolved. If this issue has been discussed before, can you point me to a previous discussion please? -- Steven From jnoller at gmail.com Thu Feb 28 12:21:59 2013 From: jnoller at gmail.com (Jesse Noller) Date: Thu, 28 Feb 2013 06:21:59 -0500 Subject: [Python-Dev] Python Language Summit at PyCon: Agenda In-Reply-To: References: <13FB490E-3548-4959-9C2B-2880B8ACA6F5@voidspace.org.uk> Message-ID: On Feb 27, 2013, at 3:20 PM, anatoly techtonik wrote: > > * security by obscurity in legal position of PSF towards contributors > https://code.google.com/legal/individual-cla-v1.0.html > vs > http://www.python.org/psf/contrib/contrib-form/ + http://www.samurajdata.se/opensource/mirror/licenses/afl-2.1.php > or > http://www.python.org/psf/contrib/contrib-form/ + http://opensource.org/licenses/apache2.0.php > and why PSF doesn't comply the 4. Redistribution clause from Apache 2.0 license > I'm not even touching your security through obscurity trollbait, but I'd like to know how the PSF / Python core is non compliant with clause 4, and the legal counsel you spoke to to confirm this. Jesse -------------- next part -------------- An HTML attachment was scrubbed... URL: From fuzzyman at voidspace.org.uk Thu Feb 28 12:34:18 2013 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Thu, 28 Feb 2013 11:34:18 +0000 Subject: [Python-Dev] Python Language Summit at PyCon: Agenda In-Reply-To: References: <13FB490E-3548-4959-9C2B-2880B8ACA6F5@voidspace.org.uk> Message-ID: On 28 Feb 2013, at 07:36, Georg Brandl wrote: > Am 27.02.2013 17:51, schrieb Michael Foord: >> Hello all, >> >> PyCon, and the Python Language Summit, is nearly upon us. We have a good number of people confirmed to attend. If you are intending to come to the language summit but haven't let me know please do so. >> >> The agenda of topics for discussion so far includes the following: >> >> * A report on pypy status - Maciej and Armin >> * Jython and IronPython status reports - Dino / Frank >> * Packaging (Doug Hellmann and Monty Taylor at least) >> * Cleaning up interpreter initialisation (both in hopes of finding areas >> to rationalise and hence speed things up, as well as making things >> more embedding friendly). Nick Coghlan >> * Adding new async capabilities to the standard library (Guido) >> * cffi and the standard library - Maciej >> * flufl.enum and the standard library - Barry Warsaw >> * The argument clinic - Larry Hastings >> >> If you have other items you'd like to discuss please let me know and I can add them to the agenda. > > May I in absentia propose at least a short discussion of the XML fixes > and accompanying security releases? FWIW, for 3.2 and 3.3 I have no > objections to secure-by-default. > Sure. It would be good if someone who *will* be there can champion the discussion. Michael > Georg > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk -- http://www.voidspace.org.uk/ May you do good and not evil May you find forgiveness for yourself and forgive others May you share freely, never taking more than you give. -- the sqlite blessing http://www.sqlite.org/different.html From fijall at gmail.com Thu Feb 28 12:36:10 2013 From: fijall at gmail.com (Maciej Fijalkowski) Date: Thu, 28 Feb 2013 13:36:10 +0200 Subject: [Python-Dev] High volumes and off topic discussions Message-ID: Hi I know this is a hard topic, but python-dev is already incredibly high-volume and dragging discussion off-topic is making following important stuff (while ignoring unimportant stuff) very hard. For example in a recent topic "cffi in stdlib" I find a mail that says "we have to find a sufficiently silly species of snake". It's even funny, but it definitely makes it very hard to follow for those of us who don't read python-dev 24/7. Would it be reasonable for python-dev to generally try to stay on topic (for example if the thread is called "silly species of snakes", I absolutely don't mind people posting there whatever they feel like as long as I'm not expected to read every single message). Cheers, fijal From michael at voidspace.org.uk Thu Feb 28 12:39:52 2013 From: michael at voidspace.org.uk (Michael Foord) Date: Thu, 28 Feb 2013 11:39:52 +0000 Subject: [Python-Dev] Python Language Summit at PyCon: Agenda In-Reply-To: <20130227195021.0101c919@pitrou.net> References: <13FB490E-3548-4959-9C2B-2880B8ACA6F5@voidspace.org.uk> <20130227195021.0101c919@pitrou.net> Message-ID: <855C0532-804F-48BB-BE16-2604D2DF0AE4@voidspace.org.uk> On 27 Feb 2013, at 18:50, Antoine Pitrou wrote: > On Wed, 27 Feb 2013 16:51:16 +0000 > Michael Foord wrote: > >> Hello all, >> >> PyCon, and the Python Language Summit, is nearly upon us. We have a good number of people confirmed to attend. If you are intending to come to the language summit but haven't let me know please do so. >> >> The agenda of topics for discussion so far includes the following: >> >> * A report on pypy status - Maciej and Armin >> * Jython and IronPython status reports - Dino / Frank >> * Packaging (Doug Hellmann and Monty Taylor at least) >> * Cleaning up interpreter initialisation (both in hopes of finding areas >> to rationalise and hence speed things up, as well as making things >> more embedding friendly). Nick Coghlan >> * Adding new async capabilities to the standard library (Guido) >> * cffi and the standard library - Maciej >> * flufl.enum and the standard library - Barry Warsaw >> * The argument clinic - Larry Hastings >> >> If you have other items you'd like to discuss please let me know and I can add them to the agenda. > > Perhaps someone wants to discuss > http://www.python.org/dev/peps/pep-0428/, but I won't be there and the > PEP isn't terribly up-to-date either :-) If you can find someone familiar with pathlib to champion the discussion it is more likely to happen and be productive... Getting the PEP up to date before the summit will also help. (I very much like the *idea* of pathlib and the bits I've seen / read through - but I haven't used it in anger yet so I don't feel qualified to champion it myself.) Michael > > Regards > > Antoine. > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk -- http://www.voidspace.org.uk/ May you do good and not evil May you find forgiveness for yourself and forgive others May you share freely, never taking more than you give. -- the sqlite blessing http://www.sqlite.org/different.html From michael at voidspace.org.uk Thu Feb 28 12:41:00 2013 From: michael at voidspace.org.uk (Michael Foord) Date: Thu, 28 Feb 2013 11:41:00 +0000 Subject: [Python-Dev] Python Language Summit at PyCon: Agenda In-Reply-To: References: <13FB490E-3548-4959-9C2B-2880B8ACA6F5@voidspace.org.uk> Message-ID: <2CCC55D8-DF82-4665-BA6C-2888E080B651@voidspace.org.uk> On 27 Feb 2013, at 19:01, fwierzbicki at gmail.com wrote: > On Wed, Feb 27, 2013 at 8:51 AM, Michael Foord wrote: >> If you have other items you'd like to discuss please let me know and I can add them to the agenda. > > I'd like to discuss merging Jython's standard Lib (the *.py files). We > have in the past had agreement that this would be a good idea - I just > want to bring it up since I think this is probably the year that I'm > actually going to do it. Added to the agenda. Michael > > -Frank > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk -- http://www.voidspace.org.uk/ May you do good and not evil May you find forgiveness for yourself and forgive others May you share freely, never taking more than you give. -- the sqlite blessing http://www.sqlite.org/different.html From stefan at bytereef.org Thu Feb 28 12:42:59 2013 From: stefan at bytereef.org (Stefan Krah) Date: Thu, 28 Feb 2013 12:42:59 +0100 Subject: [Python-Dev] Python Language Summit at PyCon: Agenda In-Reply-To: References: <13FB490E-3548-4959-9C2B-2880B8ACA6F5@voidspace.org.uk> Message-ID: <20130228114258.GA2732@sleipnir.bytereef.org> Jesse Noller wrote: > http://www.python.org/psf/contrib/contrib-form/ + http://opensource.org/ > licenses/apache2.0.php > and why PSF doesn't comply the 4. Redistribution clause from Apache 2.0 > license > > > > I'm not even touching your security through obscurity trollbait, but I'd like > to know how the PSF / Python core is non compliant with clause 4, and the legal > counsel you spoke to to confirm this. Perhaps it's an idea to have a python-legal mailing list for these topics? I don't think it's fundamentally wrong to scrutinize licenses, provided that the discussion stays civil and factual. IIRC Debian has such a list because people got annoyed with the traffic on other lists. Stefan Krah From fuzzyman at voidspace.org.uk Thu Feb 28 12:43:08 2013 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Thu, 28 Feb 2013 11:43:08 +0000 Subject: [Python-Dev] Python Language Summit at PyCon: Agenda In-Reply-To: <20130227224210.288a334d@anarchist.wooz.org> References: <13FB490E-3548-4959-9C2B-2880B8ACA6F5@voidspace.org.uk> <20130227224210.288a334d@anarchist.wooz.org> Message-ID: <5A22F57A-5670-4D76-B3A5-26DDE0A163B9@voidspace.org.uk> On 28 Feb 2013, at 03:42, Barry Warsaw wrote: > On Feb 27, 2013, at 04:51 PM, Michael Foord wrote: > >> If you have other items you'd like to discuss please let me know and I can >> add them to the agenda. > > I'd like to have some discussions around promotion of Python 3, how we can > accelerate its adoption, availability of supporting packages, what critical > projects are still missing, etc. > Added to the agenda. Michael > -Barry > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk -- http://www.voidspace.org.uk/ May you do good and not evil May you find forgiveness for yourself and forgive others May you share freely, never taking more than you give. -- the sqlite blessing http://www.sqlite.org/different.html From jnoller at gmail.com Thu Feb 28 12:48:24 2013 From: jnoller at gmail.com (Jesse Noller) Date: Thu, 28 Feb 2013 06:48:24 -0500 Subject: [Python-Dev] Python Language Summit at PyCon: Agenda In-Reply-To: <20130228114258.GA2732@sleipnir.bytereef.org> References: <13FB490E-3548-4959-9C2B-2880B8ACA6F5@voidspace.org.uk> <20130228114258.GA2732@sleipnir.bytereef.org> Message-ID: <08DF327B-EAE1-414D-83B5-B75E6A8AA3A0@gmail.com> On Feb 28, 2013, at 6:42 AM, Stefan Krah wrote: > Jesse Noller wrote: >> http://www.python.org/psf/contrib/contrib-form/ + http://opensource.org/ >> licenses/apache2.0.php >> and why PSF doesn't comply the 4. Redistribution clause from Apache 2.0 >> license >> >> >> >> I'm not even touching your security through obscurity trollbait, but I'd like >> to know how the PSF / Python core is non compliant with clause 4, and the legal >> counsel you spoke to to confirm this. > > Perhaps it's an idea to have a python-legal mailing list for these topics? > > I don't think it's fundamentally wrong to scrutinize licenses, provided > that the discussion stays civil and factual. > > IIRC Debian has such a list because people got annoyed with the traffic > on other lists. > > > Stefan Krah > > We have one: psf at python.org > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/jnoller%40gmail.com From solipsis at pitrou.net Thu Feb 28 12:55:05 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 28 Feb 2013 12:55:05 +0100 Subject: [Python-Dev] Python Language Summit at PyCon: Agenda References: <13FB490E-3548-4959-9C2B-2880B8ACA6F5@voidspace.org.uk> <20130228114258.GA2732@sleipnir.bytereef.org> <08DF327B-EAE1-414D-83B5-B75E6A8AA3A0@gmail.com> Message-ID: <20130228125505.2a380773@pitrou.net> Le Thu, 28 Feb 2013 06:48:24 -0500, Jesse Noller a ?crit : > > > > Perhaps it's an idea to have a python-legal mailing list for these > > topics? > > > > I don't think it's fundamentally wrong to scrutinize licenses, > > provided that the discussion stays civil and factual. > > > > IIRC Debian has such a list because people got annoyed with the > > traffic on other lists. > > > > > > Stefan Krah > > > > > > We have one: psf at python.org That's not exactly a public mailing-list. Regards Antoine. From jnoller at gmail.com Thu Feb 28 12:57:36 2013 From: jnoller at gmail.com (Jesse Noller) Date: Thu, 28 Feb 2013 06:57:36 -0500 Subject: [Python-Dev] Python Language Summit at PyCon: Agenda In-Reply-To: <20130228125505.2a380773@pitrou.net> References: <13FB490E-3548-4959-9C2B-2880B8ACA6F5@voidspace.org.uk> <20130228114258.GA2732@sleipnir.bytereef.org> <08DF327B-EAE1-414D-83B5-B75E6A8AA3A0@gmail.com> <20130228125505.2a380773@pitrou.net> Message-ID: On Feb 28, 2013, at 6:55 AM, Antoine Pitrou wrote: > Le Thu, 28 Feb 2013 06:48:24 -0500, > Jesse Noller a ?crit : >>> >>> Perhaps it's an idea to have a python-legal mailing list for these >>> topics? >>> >>> I don't think it's fundamentally wrong to scrutinize licenses, >>> provided that the discussion stays civil and factual. >>> >>> IIRC Debian has such a list because people got annoyed with the >>> traffic on other lists. >>> >>> >>> Stefan Krah >> >> We have one: psf at python.org > > That's not exactly a public mailing-list. > > Regards > > Antoine. > Nope. But it's also where lawyers flock and these issues can rapidly be resolved. > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/jnoller%40gmail.com From solipsis at pitrou.net Thu Feb 28 13:23:50 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 28 Feb 2013 13:23:50 +0100 Subject: [Python-Dev] Python Language Summit at PyCon: Agenda References: <13FB490E-3548-4959-9C2B-2880B8ACA6F5@voidspace.org.uk> <20130228114258.GA2732@sleipnir.bytereef.org> <08DF327B-EAE1-414D-83B5-B75E6A8AA3A0@gmail.com> <20130228125505.2a380773@pitrou.net> Message-ID: <20130228132350.05218efc@pitrou.net> Le Thu, 28 Feb 2013 06:57:36 -0500, Jesse Noller a ?crit : > > On Feb 28, 2013, at 6:55 AM, Antoine Pitrou > wrote: > > > Le Thu, 28 Feb 2013 06:48:24 -0500, > > Jesse Noller a ?crit : > >>> > >>> Perhaps it's an idea to have a python-legal mailing list for these > >>> topics? > >>> > >>> I don't think it's fundamentally wrong to scrutinize licenses, > >>> provided that the discussion stays civil and factual. > >>> > >>> IIRC Debian has such a list because people got annoyed with the > >>> traffic on other lists. > >>> > >>> > >>> Stefan Krah > >> > >> We have one: psf at python.org > > > > That's not exactly a public mailing-list. > > > > Regards > > > > Antoine. > > > > Nope. But it's also where lawyers flock and these issues can rapidly > be resolved. Really? I didn't know lawyers flocked at the PSF. It also doesn't provide any public feedback for said "resolution", meaning it doesn't help alleviate any future discussions about legal issues. Regards Antoine. From stefan at bytereef.org Thu Feb 28 13:26:37 2013 From: stefan at bytereef.org (Stefan Krah) Date: Thu, 28 Feb 2013 13:26:37 +0100 Subject: [Python-Dev] Python Language Summit at PyCon: Agenda In-Reply-To: References: <13FB490E-3548-4959-9C2B-2880B8ACA6F5@voidspace.org.uk> <20130228114258.GA2732@sleipnir.bytereef.org> <08DF327B-EAE1-414D-83B5-B75E6A8AA3A0@gmail.com> <20130228125505.2a380773@pitrou.net> Message-ID: <20130228122637.GA2960@sleipnir.bytereef.org> Jesse Noller wrote: > >> We have one: psf at python.org > > > > That's not exactly a public mailing-list. > > Nope. But it's also where lawyers flock and these issues can rapidly be resolved. If the list isn't publicly archived, the same questions will arise over and over again (probably on python-dev). Why would it help to resolve such an issue (if it is an issue at all!) for a single person on a private mailing list? Stefan Krah From jnoller at gmail.com Thu Feb 28 13:31:14 2013 From: jnoller at gmail.com (Jesse Noller) Date: Thu, 28 Feb 2013 07:31:14 -0500 Subject: [Python-Dev] Python Language Summit at PyCon: Agenda In-Reply-To: <20130228132350.05218efc@pitrou.net> References: <13FB490E-3548-4959-9C2B-2880B8ACA6F5@voidspace.org.uk> <20130228114258.GA2732@sleipnir.bytereef.org> <08DF327B-EAE1-414D-83B5-B75E6A8AA3A0@gmail.com> <20130228125505.2a380773@pitrou.net> <20130228132350.05218efc@pitrou.net> Message-ID: On Feb 28, 2013, at 7:23 AM, Antoine Pitrou wrote: > Le Thu, 28 Feb 2013 06:57:36 -0500, > Jesse Noller a ?crit : >> >> On Feb 28, 2013, at 6:55 AM, Antoine Pitrou >> wrote: >> >>> Le Thu, 28 Feb 2013 06:48:24 -0500, >>> Jesse Noller a ?crit : >>>>> >>>>> Perhaps it's an idea to have a python-legal mailing list for these >>>>> topics? >>>>> >>>>> I don't think it's fundamentally wrong to scrutinize licenses, >>>>> provided that the discussion stays civil and factual. >>>>> >>>>> IIRC Debian has such a list because people got annoyed with the >>>>> traffic on other lists. >>>>> >>>>> >>>>> Stefan Krah >>>> >>>> We have one: psf at python.org >>> >>> That's not exactly a public mailing-list. >>> >>> Regards >>> >>> Antoine. >> >> Nope. But it's also where lawyers flock and these issues can rapidly >> be resolved. > > Really? I didn't know lawyers flocked at the PSF. It also doesn't > provide any public feedback for said "resolution", meaning it doesn't > help alleviate any future discussions about legal issues. > > Regards > > Antoine. > Yes sir. Ill have a new mailing list made today. I will note on the list page that is is strictly unofficial and all legal / lawyer type things must be sent to psf at python.org for official decisions or legal counsel. > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/jnoller%40gmail.com From jnoller at gmail.com Thu Feb 28 13:51:48 2013 From: jnoller at gmail.com (Jesse Noller) Date: Thu, 28 Feb 2013 07:51:48 -0500 Subject: [Python-Dev] Python Language Summit at PyCon: Agenda In-Reply-To: References: <13FB490E-3548-4959-9C2B-2880B8ACA6F5@voidspace.org.uk> <20130228114258.GA2732@sleipnir.bytereef.org> <08DF327B-EAE1-414D-83B5-B75E6A8AA3A0@gmail.com> <20130228125505.2a380773@pitrou.net> <20130228132350.05218efc@pitrou.net> Message-ID: <5E04F63C-B0CC-4C77-AF99-07BA91D933BB@gmail.com> On Feb 28, 2013, at 7:31 AM, Jesse Noller wrote: > > > On Feb 28, 2013, at 7:23 AM, Antoine Pitrou wrote: > >> Le Thu, 28 Feb 2013 06:57:36 -0500, >> Jesse Noller a ?crit : >>> >>> On Feb 28, 2013, at 6:55 AM, Antoine Pitrou >>> wrote: >>> >>>> Le Thu, 28 Feb 2013 06:48:24 -0500, >>>> Jesse Noller a ?crit : >>>>>> >>>>>> Perhaps it's an idea to have a python-legal mailing list for these >>>>>> topics? >>>>>> >>>>>> I don't think it's fundamentally wrong to scrutinize licenses, >>>>>> provided that the discussion stays civil and factual. >>>>>> >>>>>> IIRC Debian has such a list because people got annoyed with the >>>>>> traffic on other lists. >>>>>> >>>>>> >>>>>> Stefan Krah >>>>> >>>>> We have one: psf at python.org >>>> >>>> That's not exactly a public mailing-list. >>>> >>>> Regards >>>> >>>> Antoine. >>> >>> Nope. But it's also where lawyers flock and these issues can rapidly >>> be resolved. >> >> Really? I didn't know lawyers flocked at the PSF. It also doesn't >> provide any public feedback for said "resolution", meaning it doesn't >> help alleviate any future discussions about legal issues. >> >> Regards >> >> Antoine. > > > Yes sir. Ill have a new mailing list made today. I will note on the list page that is is strictly unofficial and all legal / lawyer type things must be sent to psf at python.org for official decisions or legal counsel. > > Spent a minute driving to request the new list. Will notify when it's up >> >> _______________________________________________ >> Python-Dev mailing list >> Python-Dev at python.org >> http://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe: http://mail.python.org/mailman/options/python-dev/jnoller%40gmail.com From stephen at xemacs.org Thu Feb 28 14:03:34 2013 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Thu, 28 Feb 2013 22:03:34 +0900 Subject: [Python-Dev] Python Language Summit at PyCon: Agenda In-Reply-To: <20130228122637.GA2960@sleipnir.bytereef.org> References: <13FB490E-3548-4959-9C2B-2880B8ACA6F5@voidspace.org.uk> <20130228114258.GA2732@sleipnir.bytereef.org> <08DF327B-EAE1-414D-83B5-B75E6A8AA3A0@gmail.com> <20130228125505.2a380773@pitrou.net> <20130228122637.GA2960@sleipnir.bytereef.org> Message-ID: <87wqtsk2u1.fsf@uwakimon.sk.tsukuba.ac.jp> Stefan Krah writes: > Why would [the PSF list] help to resolve such an issue (if it is an > issue at all!) Precisely. If there *is* a compliance problem, there's nothing to be done before talking to the lawyers. Although license *choice* is primarily a political issue, *compliance* is technical. From jnoller at gmail.com Thu Feb 28 14:08:25 2013 From: jnoller at gmail.com (Jesse Noller) Date: Thu, 28 Feb 2013 08:08:25 -0500 Subject: [Python-Dev] Python Language Summit at PyCon: Agenda In-Reply-To: <87wqtsk2u1.fsf@uwakimon.sk.tsukuba.ac.jp> References: <13FB490E-3548-4959-9C2B-2880B8ACA6F5@voidspace.org.uk> <20130228114258.GA2732@sleipnir.bytereef.org> <08DF327B-EAE1-414D-83B5-B75E6A8AA3A0@gmail.com> <20130228125505.2a380773@pitrou.net> <20130228122637.GA2960@sleipnir.bytereef.org> <87wqtsk2u1.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <6EB35E22-AFA9-42EF-AED6-BE27E5E65416@gmail.com> On Feb 28, 2013, at 8:03 AM, "Stephen J. Turnbull" wrote: > Stefan Krah writes: > >> Why would [the PSF list] help to resolve such an issue (if it is an >> issue at all!) > > Precisely. > > If there *is* a compliance problem, there's nothing to be done before > talking to the lawyers. Although license *choice* is primarily a > political issue, *compliance* is technical. > And the board is legally bound to ensure compliance. I know, I've gotten the email requests clarifying copyright assignments. That being said, I requested a mailing list where this can all be hotly debated and will let everyone know when the playground is open. > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/jnoller%40gmail.com From steve at pearwood.info Thu Feb 28 14:21:48 2013 From: steve at pearwood.info (Steven D'Aprano) Date: Fri, 01 Mar 2013 00:21:48 +1100 Subject: [Python-Dev] Python Language Summit at PyCon: Agenda In-Reply-To: <20130228122637.GA2960@sleipnir.bytereef.org> References: <13FB490E-3548-4959-9C2B-2880B8ACA6F5@voidspace.org.uk> <20130228114258.GA2732@sleipnir.bytereef.org> <08DF327B-EAE1-414D-83B5-B75E6A8AA3A0@gmail.com> <20130228125505.2a380773@pitrou.net> <20130228122637.GA2960@sleipnir.bytereef.org> Message-ID: <512F59EC.3010906@pearwood.info> On 28/02/13 23:26, Stefan Krah wrote: > Jesse Noller wrote: >>>> We have one: psf at python.org >>> >>> That's not exactly a public mailing-list. >> >> Nope. But it's also where lawyers flock and these issues can rapidly be resolved. > > If the list isn't publicly archived, the same questions will arise over > and over again (probably on python-dev). > > Why would it help to resolve such an issue (if it is an issue at all!) > for a single person on a private mailing list? Because we can ask somebody on the PSF to commit to replying here with the resolution: - "No, we have legal advice that we are not violating the Apache license, and here's why..." - "Yes, we are violating the license, and this is what we are going to do to fix it..." The question of whether or not the PSF is violating the Apache license in some way is not one that is helped by having arbitrary people give their uninformed opinions. I sympathize with curious people wanting to see what's going on, but really, this is not a question to be debated and argued by amateurs and non-lawyers. The PSF has a tame lawyer for exactly these sorts of matters, and we should let him do his job. In any case, until Anatoly replies with details, this counts as nothing more than an unsubstantiated, vague accusation. Over to you Anatoly... -- Steven From solipsis at pitrou.net Thu Feb 28 14:37:32 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 28 Feb 2013 14:37:32 +0100 Subject: [Python-Dev] Python Language Summit at PyCon: Agenda References: <13FB490E-3548-4959-9C2B-2880B8ACA6F5@voidspace.org.uk> <20130228114258.GA2732@sleipnir.bytereef.org> <08DF327B-EAE1-414D-83B5-B75E6A8AA3A0@gmail.com> <20130228125505.2a380773@pitrou.net> <20130228122637.GA2960@sleipnir.bytereef.org> <512F59EC.3010906@pearwood.info> Message-ID: <20130228143732.47fa74ae@pitrou.net> Le Fri, 01 Mar 2013 00:21:48 +1100, Steven D'Aprano a ?crit : > > The question of whether or not the PSF is violating the Apache > license in some way is not one that is helped by having arbitrary > people give their uninformed opinions. I sympathize with curious > people wanting to see what's going on, but really, this is not a > question to be debated and argued by amateurs and non-lawyers. Similarly, programming questions are not to be debated by amateurs and non-programmers. We should just close python-dev. Regards Antoine. From brett at python.org Thu Feb 28 14:49:46 2013 From: brett at python.org (Brett Cannon) Date: Thu, 28 Feb 2013 08:49:46 -0500 Subject: [Python-Dev] Python Language Summit at PyCon: Agenda In-Reply-To: References: <13FB490E-3548-4959-9C2B-2880B8ACA6F5@voidspace.org.uk> Message-ID: On Thu, Feb 28, 2013 at 6:34 AM, Michael Foord wrote: > > On 28 Feb 2013, at 07:36, Georg Brandl wrote: > > > Am 27.02.2013 17:51, schrieb Michael Foord: > >> Hello all, > >> > >> PyCon, and the Python Language Summit, is nearly upon us. We have a > good number of people confirmed to attend. If you are intending to come to > the language summit but haven't let me know please do so. > >> > >> The agenda of topics for discussion so far includes the following: > >> > >> * A report on pypy status - Maciej and Armin > >> * Jython and IronPython status reports - Dino / Frank > >> * Packaging (Doug Hellmann and Monty Taylor at least) > >> * Cleaning up interpreter initialisation (both in hopes of finding areas > >> to rationalise and hence speed things up, as well as making things > >> more embedding friendly). Nick Coghlan > >> * Adding new async capabilities to the standard library (Guido) > >> * cffi and the standard library - Maciej > >> * flufl.enum and the standard library - Barry Warsaw > >> * The argument clinic - Larry Hastings > >> > >> If you have other items you'd like to discuss please let me know and I > can add them to the agenda. > > > > May I in absentia propose at least a short discussion of the XML fixes > > and accompanying security releases? FWIW, for 3.2 and 3.3 I have no > > objections to secure-by-default. > > > > Sure. It would be good if someone who *will* be there can champion the > discussion. While Christian is in the best position to discuss this, I did review his various monkeypatch fixes + expat patches so I can attempt to answer any questions people may have. -------------- next part -------------- An HTML attachment was scrubbed... URL: From benjamin at python.org Thu Feb 28 14:53:41 2013 From: benjamin at python.org (Benjamin Peterson) Date: Thu, 28 Feb 2013 08:53:41 -0500 Subject: [Python-Dev] Python Language Summit at PyCon: Agenda In-Reply-To: References: <13FB490E-3548-4959-9C2B-2880B8ACA6F5@voidspace.org.uk> Message-ID: 2013/2/28 Brett Cannon : > > > > On Thu, Feb 28, 2013 at 6:34 AM, Michael Foord > wrote: >> >> >> On 28 Feb 2013, at 07:36, Georg Brandl wrote: >> >> > Am 27.02.2013 17:51, schrieb Michael Foord: >> >> Hello all, >> >> >> >> PyCon, and the Python Language Summit, is nearly upon us. We have a >> >> good number of people confirmed to attend. If you are intending to come to >> >> the language summit but haven't let me know please do so. >> >> >> >> The agenda of topics for discussion so far includes the following: >> >> >> >> * A report on pypy status - Maciej and Armin >> >> * Jython and IronPython status reports - Dino / Frank >> >> * Packaging (Doug Hellmann and Monty Taylor at least) >> >> * Cleaning up interpreter initialisation (both in hopes of finding >> >> areas >> >> to rationalise and hence speed things up, as well as making things >> >> more embedding friendly). Nick Coghlan >> >> * Adding new async capabilities to the standard library (Guido) >> >> * cffi and the standard library - Maciej >> >> * flufl.enum and the standard library - Barry Warsaw >> >> * The argument clinic - Larry Hastings >> >> >> >> If you have other items you'd like to discuss please let me know and I >> >> can add them to the agenda. >> > >> > May I in absentia propose at least a short discussion of the XML fixes >> > and accompanying security releases? FWIW, for 3.2 and 3.3 I have no >> > objections to secure-by-default. >> > >> >> Sure. It would be good if someone who *will* be there can champion the >> discussion. > > > While Christian is in the best position to discuss this, I did review his > various monkeypatch fixes + expat patches so I can attempt to answer any > questions people may have. How close are they to being applied? -- Regards, Benjamin From brett at python.org Thu Feb 28 14:57:14 2013 From: brett at python.org (Brett Cannon) Date: Thu, 28 Feb 2013 08:57:14 -0500 Subject: [Python-Dev] Python Language Summit at PyCon: Agenda In-Reply-To: References: <13FB490E-3548-4959-9C2B-2880B8ACA6F5@voidspace.org.uk> Message-ID: On Thu, Feb 28, 2013 at 8:53 AM, Benjamin Peterson wrote: > 2013/2/28 Brett Cannon : > > > > > > > > On Thu, Feb 28, 2013 at 6:34 AM, Michael Foord < > fuzzyman at voidspace.org.uk> > > wrote: > >> > >> > >> On 28 Feb 2013, at 07:36, Georg Brandl wrote: > >> > >> > Am 27.02.2013 17:51, schrieb Michael Foord: > >> >> Hello all, > >> >> > >> >> PyCon, and the Python Language Summit, is nearly upon us. We have a > >> >> good number of people confirmed to attend. If you are intending to > come to > >> >> the language summit but haven't let me know please do so. > >> >> > >> >> The agenda of topics for discussion so far includes the following: > >> >> > >> >> * A report on pypy status - Maciej and Armin > >> >> * Jython and IronPython status reports - Dino / Frank > >> >> * Packaging (Doug Hellmann and Monty Taylor at least) > >> >> * Cleaning up interpreter initialisation (both in hopes of finding > >> >> areas > >> >> to rationalise and hence speed things up, as well as making things > >> >> more embedding friendly). Nick Coghlan > >> >> * Adding new async capabilities to the standard library (Guido) > >> >> * cffi and the standard library - Maciej > >> >> * flufl.enum and the standard library - Barry Warsaw > >> >> * The argument clinic - Larry Hastings > >> >> > >> >> If you have other items you'd like to discuss please let me know and > I > >> >> can add them to the agenda. > >> > > >> > May I in absentia propose at least a short discussion of the XML fixes > >> > and accompanying security releases? FWIW, for 3.2 and 3.3 I have no > >> > objections to secure-by-default. > >> > > >> > >> Sure. It would be good if someone who *will* be there can champion the > >> discussion. > > > > > > While Christian is in the best position to discuss this, I did review his > > various monkeypatch fixes + expat patches so I can attempt to answer any > > questions people may have. > > How close are they to being applied? > I have no idea. Ask Christian. =) I can just answer what the attacks are and what had to change to protect against them. -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Thu Feb 28 15:03:26 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 28 Feb 2013 15:03:26 +0100 Subject: [Python-Dev] High volumes and off topic discussions References: Message-ID: <20130228150326.2e7b0944@pitrou.net> Le Thu, 28 Feb 2013 13:36:10 +0200, Maciej Fijalkowski a ?crit : > Hi > > I know this is a hard topic, but python-dev is already incredibly > high-volume and dragging discussion off-topic is making following > important stuff (while ignoring unimportant stuff) very hard. > > For example in a recent topic "cffi in stdlib" I find a mail that says > "we have to find a sufficiently silly species of snake". It's even > funny, but it definitely makes it very hard to follow for those of us > who don't read python-dev 24/7. Would it be reasonable for python-dev > to generally try to stay on topic (for example if the thread is called > "silly species of snakes", I absolutely don't mind people posting > there whatever they feel like as long as I'm not expected to read > every single message). I'm afraid you're trying to devise derogatory distinctions regarding drifting discussions. Seriously, yes, I approve of changing the subject line, although I forgot to do it this time. For the record, you can also read the list through a NNTP gateway using Gmane, it can make things easier. Regards Antoine. From jnoller at gmail.com Thu Feb 28 15:14:32 2013 From: jnoller at gmail.com (Jesse Noller) Date: Thu, 28 Feb 2013 09:14:32 -0500 Subject: [Python-Dev] New mailing list: Python-legal-sig Message-ID: <244F6FB17C9D4539955A6995ABCC7E69@gmail.com> See: http://mail.python.org/mailman/listinfo/python-legal-sig Open archives. As the header says this is for the discussion of CLA/other issues. If specific legal questions or alterations to Python/the PSF trademarks, CLA/etc are requested those *must* be sent to psf at python.org for board oversight and approval. This is an open discussion list, and is not official legal stance of the foundation, trademarks, etc. jesse From jnoller at gmail.com Thu Feb 28 15:16:05 2013 From: jnoller at gmail.com (Jesse Noller) Date: Thu, 28 Feb 2013 09:16:05 -0500 Subject: [Python-Dev] Python Language Summit at PyCon: Agenda In-Reply-To: <20130228122637.GA2960@sleipnir.bytereef.org> References: <13FB490E-3548-4959-9C2B-2880B8ACA6F5@voidspace.org.uk> <20130228114258.GA2732@sleipnir.bytereef.org> <08DF327B-EAE1-414D-83B5-B75E6A8AA3A0@gmail.com> <20130228125505.2a380773@pitrou.net> <20130228122637.GA2960@sleipnir.bytereef.org> Message-ID: <6E60016BF5984973A91B755FA21706BE@gmail.com> On Thursday, February 28, 2013 at 7:26 AM, Stefan Krah wrote: > Jesse Noller wrote: > > > > We have one: psf at python.org (mailto:psf at python.org) > > > > > > > > > > > > That's not exactly a public mailing-list. > > > > Nope. But it's also where lawyers flock and these issues can rapidly be resolved. > > If the list isn't publicly archived, the same questions will arise over > and over again (probably on python-dev). > > Why would it help to resolve such an issue (if it is an issue at all!) > for a single person on a private mailing list? See: http://mail.python.org/pipermail/python-dev/2013-February/124463.html From stefan at bytereef.org Thu Feb 28 15:45:20 2013 From: stefan at bytereef.org (Stefan Krah) Date: Thu, 28 Feb 2013 15:45:20 +0100 Subject: [Python-Dev] Python Language Summit at PyCon: Agenda In-Reply-To: <512F59EC.3010906@pearwood.info> References: <13FB490E-3548-4959-9C2B-2880B8ACA6F5@voidspace.org.uk> <20130228114258.GA2732@sleipnir.bytereef.org> <08DF327B-EAE1-414D-83B5-B75E6A8AA3A0@gmail.com> <20130228125505.2a380773@pitrou.net> <20130228122637.GA2960@sleipnir.bytereef.org> <512F59EC.3010906@pearwood.info> Message-ID: <20130228144520.GA3947@sleipnir.bytereef.org> Steven D'Aprano wrote: > The question of whether or not the PSF is violating the Apache license in > some way is not one that is helped by having arbitrary people give their > uninformed opinions. No one will be preventing lawyers from giving their opinions on python-legal. In fact, at least one patent attorney *is* posting on debian-legal. Jesse is right: the list may turn into a playground. In that case, at least the traffic is not on python-dev. Stefan Krah From stefan at bytereef.org Thu Feb 28 15:49:53 2013 From: stefan at bytereef.org (Stefan Krah) Date: Thu, 28 Feb 2013 15:49:53 +0100 Subject: [Python-Dev] Python Language Summit at PyCon: Agenda In-Reply-To: <6E60016BF5984973A91B755FA21706BE@gmail.com> References: <13FB490E-3548-4959-9C2B-2880B8ACA6F5@voidspace.org.uk> <20130228114258.GA2732@sleipnir.bytereef.org> <08DF327B-EAE1-414D-83B5-B75E6A8AA3A0@gmail.com> <20130228125505.2a380773@pitrou.net> <20130228122637.GA2960@sleipnir.bytereef.org> <6E60016BF5984973A91B755FA21706BE@gmail.com> Message-ID: <20130228144953.GB3947@sleipnir.bytereef.org> Jesse Noller wrote: > > Why would it help to resolve such an issue (if it is an issue at all!) > > for a single person on a private mailing list? > > > See: http://mail.python.org/pipermail/python-dev/2013-February/124463.html That was quick. Thanks! Stefan Krah From jnoller at gmail.com Thu Feb 28 15:57:14 2013 From: jnoller at gmail.com (Jesse Noller) Date: Thu, 28 Feb 2013 09:57:14 -0500 Subject: [Python-Dev] Python Language Summit at PyCon: Agenda In-Reply-To: <20130228144953.GB3947@sleipnir.bytereef.org> References: <13FB490E-3548-4959-9C2B-2880B8ACA6F5@voidspace.org.uk> <20130228114258.GA2732@sleipnir.bytereef.org> <08DF327B-EAE1-414D-83B5-B75E6A8AA3A0@gmail.com> <20130228125505.2a380773@pitrou.net> <20130228122637.GA2960@sleipnir.bytereef.org> <6E60016BF5984973A91B755FA21706BE@gmail.com> <20130228144953.GB3947@sleipnir.bytereef.org> Message-ID: <6622E135D99E42C2AAB99B64CFC48C89@gmail.com> On Thursday, February 28, 2013 at 9:49 AM, Stefan Krah wrote: > Jesse Noller wrote: > > > Why would it help to resolve such an issue (if it is an issue at all!) > > > for a single person on a private mailing list? > > > > > > > > > > See: http://mail.python.org/pipermail/python-dev/2013-February/124463.html > > That was quick. Thanks! > > > Stefan Krah It only takes a minute or two to do something good. Now how about dem XML vulnerabilities? ;) From fwierzbicki at gmail.com Thu Feb 28 17:44:34 2013 From: fwierzbicki at gmail.com (fwierzbicki at gmail.com) Date: Thu, 28 Feb 2013 08:44:34 -0800 Subject: [Python-Dev] Python Language Summit at PyCon: Agenda In-Reply-To: <20130227223749.2f06a328@anarchist.wooz.org> References: <13FB490E-3548-4959-9C2B-2880B8ACA6F5@voidspace.org.uk> <20130227223749.2f06a328@anarchist.wooz.org> Message-ID: On Wed, Feb 27, 2013 at 7:37 PM, Barry Warsaw wrote: > On Feb 27, 2013, at 11:33 AM, fwierzbicki at gmail.com wrote: > >>I am suggesting that we push forward on the "shared library" approach to the >>files in the Lib/* directory, so that would certainly include IronPython and >>PyPy as well I hope. > > +1 > >>The easy part for Jython is pushing some of our "if is_jython:" stuff >>into the appropriate spots in CPython's Lib/. > > I wonder if there isn't a better way to do this than sprinkling is_jython, > is_pypy, is_ironpython, is_thenextbigthing all over the code base. I have no > bright ideas here, but it seems like a feature matrix would be a better way to > go than something that assumes a particular Python implementation has a > particular feature set (which may change in the future). Sorry I meant "is_jython" as a sort of shorthand for a case by case check. It would be cool if we had a nice set of checks somewhere like "is_refcounted", etc. Would the sys.implementation area be a good place for such things? On the other hand in some ways Jython is sort of like Python on a weird virtual OS that lets the real OS bleed through some. This may still need to be checked in that way (there's are still checks of right?) -Frank From fwierzbicki at gmail.com Thu Feb 28 17:49:39 2013 From: fwierzbicki at gmail.com (fwierzbicki at gmail.com) Date: Thu, 28 Feb 2013 08:49:39 -0800 Subject: [Python-Dev] Python Language Summit at PyCon: Agenda In-Reply-To: References: <13FB490E-3548-4959-9C2B-2880B8ACA6F5@voidspace.org.uk> <20130227223749.2f06a328@anarchist.wooz.org> Message-ID: On Thu, Feb 28, 2013 at 1:15 AM, Nick Coghlan wrote: > On Thu, Feb 28, 2013 at 1:37 PM, Barry Warsaw wrote: >> On Feb 27, 2013, at 11:33 AM, fwierzbicki at gmail.com wrote: >>>The easy part for Jython is pushing some of our "if is_jython:" stuff >>>into the appropriate spots in CPython's Lib/. >> >> I wonder if there isn't a better way to do this than sprinkling is_jython, >> is_pypy, is_ironpython, is_thenextbigthing all over the code base. I have no >> bright ideas here, but it seems like a feature matrix would be a better way to >> go than something that assumes a particular Python implementation has a >> particular feature set (which may change in the future). > > Yes, avoiding that kind of thing is a key motivation for > sys.implementation. Any proposal for "is_jython" blocks should instead > be reformulated as a proposal for new sys.implementation attributes. Ah nice - the merging effort should definitely cause some careful consideration of these. Maybe I'll start a discussion about a "garbage collection type" for sys.implementation. Some way to ask would catch some of these. -Frank From fwierzbicki at gmail.com Thu Feb 28 17:59:54 2013 From: fwierzbicki at gmail.com (fwierzbicki at gmail.com) Date: Thu, 28 Feb 2013 08:59:54 -0800 Subject: [Python-Dev] Python Language Summit at PyCon: Agenda In-Reply-To: <20130228103037.1b60878b@pitrou.net> References: <13FB490E-3548-4959-9C2B-2880B8ACA6F5@voidspace.org.uk> <20130228103037.1b60878b@pitrou.net> Message-ID: On Thu, Feb 28, 2013 at 1:30 AM, Antoine Pitrou wrote: > Le Wed, 27 Feb 2013 11:33:30 -0800, > "fwierzbicki at gmail.com" a ?crit : >> >> There are a couple of spots that might be more controversial. For >> example, Jython has a file Lib/zlib.py that implements zlib in terms >> of the existing Java support for zlib. I do wonder if such a file is >> acceptable in CPython's Lib since its 195 lines of code would be >> entirely skipped by CPython. > > That's a bit annoying. How will we know that the code still works, even > though our buildbots don't exercise it? > Also, what happens if the code doesn't work anymore? That is definitely something to think about. Maybe when this becomes a serious effort and not just talk I can help get a Jython buildbot going. -Frank From fijall at gmail.com Thu Feb 28 18:17:39 2013 From: fijall at gmail.com (Maciej Fijalkowski) Date: Thu, 28 Feb 2013 19:17:39 +0200 Subject: [Python-Dev] High volumes and off topic discussions In-Reply-To: <20130228150326.2e7b0944@pitrou.net> References: <20130228150326.2e7b0944@pitrou.net> Message-ID: On Thu, Feb 28, 2013 at 4:03 PM, Antoine Pitrou wrote: > Le Thu, 28 Feb 2013 13:36:10 +0200, > Maciej Fijalkowski a ?crit : >> Hi >> >> I know this is a hard topic, but python-dev is already incredibly >> high-volume and dragging discussion off-topic is making following >> important stuff (while ignoring unimportant stuff) very hard. >> >> For example in a recent topic "cffi in stdlib" I find a mail that says >> "we have to find a sufficiently silly species of snake". It's even >> funny, but it definitely makes it very hard to follow for those of us >> who don't read python-dev 24/7. Would it be reasonable for python-dev >> to generally try to stay on topic (for example if the thread is called >> "silly species of snakes", I absolutely don't mind people posting >> there whatever they feel like as long as I'm not expected to read >> every single message). > > I'm afraid you're trying to devise derogatory distinctions regarding > drifting discussions. > > Seriously, yes, I approve of changing the subject line, although I > forgot to do it this time. > For the record, you can also read the list through a NNTP gateway using > Gmane, it can make things easier. How does that help with knowing what mails to read what mails not to read? the agenda thread is a great example - I care about the agenda, because I'm even mentioned there, but I don't care about the legal mailing list that occupies 2/3 of the thread. People who like to care about legal stuff would be baffled that noone told them, because it was buried inside a thread about a conference they might not go to etc. Anyway, I'm not even a regular here so do what works for you. > > Regards > > Antoine. > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/fijall%40gmail.com From chris.jerdonek at gmail.com Thu Feb 28 20:24:21 2013 From: chris.jerdonek at gmail.com (Chris Jerdonek) Date: Thu, 28 Feb 2013 11:24:21 -0800 Subject: [Python-Dev] Merging Jython code into standard Lib [was Re: Python Language Summit at PyCon: Agenda] Message-ID: On Thu, Feb 28, 2013 at 1:30 AM, Antoine Pitrou wrote: > Le Wed, 27 Feb 2013 11:33:30 -0800, > "fwierzbicki at gmail.com" a ?crit : >> >> There are a couple of spots that might be more controversial. For >> example, Jython has a file Lib/zlib.py that implements zlib in terms >> of the existing Java support for zlib. I do wonder if such a file is >> acceptable in CPython's Lib since its 195 lines of code would be >> entirely skipped by CPython. > > That's a bit annoying. How will we know that the code still works, even > though our buildbots don't exercise it? > Also, what happens if the code doesn't work anymore? Agreed on those problems. Would it be possible to use a design pattern in these cases so the Jython-only code wouldn't need to be part of the CPython repo? A naive example would be refactoring zlib to allow subclassing in the way that Jython needs, and then Jython could subclass in its own repo. CPython could have tests to check the subclass "contract" that Jython needs. --Chris From fwierzbicki at gmail.com Thu Feb 28 20:46:04 2013 From: fwierzbicki at gmail.com (fwierzbicki at gmail.com) Date: Thu, 28 Feb 2013 11:46:04 -0800 Subject: [Python-Dev] Merging Jython code into standard Lib [was Re: Python Language Summit at PyCon: Agenda] In-Reply-To: References: Message-ID: On Thu, Feb 28, 2013 at 11:24 AM, Chris Jerdonek wrote: > On Thu, Feb 28, 2013 at 1:30 AM, Antoine Pitrou wrote: >> Le Wed, 27 Feb 2013 11:33:30 -0800, >> "fwierzbicki at gmail.com" a ?crit : >>> >>> There are a couple of spots that might be more controversial. For >>> example, Jython has a file Lib/zlib.py that implements zlib in terms >>> of the existing Java support for zlib. I do wonder if such a file is >>> acceptable in CPython's Lib since its 195 lines of code would be >>> entirely skipped by CPython. >> >> That's a bit annoying. How will we know that the code still works, even >> though our buildbots don't exercise it? >> Also, what happens if the code doesn't work anymore? > > Agreed on those problems. Would it be possible to use a design > pattern in these cases so the Jython-only code wouldn't need to be > part of the CPython repo? A naive example would be refactoring zlib > to allow subclassing in the way that Jython needs, and then Jython > could subclass in its own repo. CPython could have tests to check the > subclass "contract" that Jython needs. What about a plat-java section to parallel plat-aix4, plat-darwin, etc? The analogy being that the Java platform is somewhat analogous to being it's own os? And these areas are not active when on other operating systems... -Frank From fwierzbicki at gmail.com Thu Feb 28 20:47:59 2013 From: fwierzbicki at gmail.com (fwierzbicki at gmail.com) Date: Thu, 28 Feb 2013 11:47:59 -0800 Subject: [Python-Dev] Merging Jython code into standard Lib [was Re: Python Language Summit at PyCon: Agenda] In-Reply-To: References: Message-ID: On Thu, Feb 28, 2013 at 11:46 AM, fwierzbicki at gmail.com wrote: >> Agreed on those problems. Would it be possible to use a design >> pattern in these cases so the Jython-only code wouldn't need to be >> part of the CPython repo? A naive example would be refactoring zlib >> to allow subclassing in the way that Jython needs, and then Jython >> could subclass in its own repo. CPython could have tests to check the >> subclass "contract" that Jython needs. > What about a plat-java section to parallel plat-aix4, plat-darwin, > etc? The analogy being that the Java platform is somewhat analogous to > being it's own os? And these areas are not active when on other > operating systems... Oh yeah and this does not preclude the zlib refactoring, and again for the record once this gets serious I'd want to help in bringing up a Jython buildbot to track breakage. -Frank From solipsis at pitrou.net Thu Feb 28 21:00:23 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 28 Feb 2013 21:00:23 +0100 Subject: [Python-Dev] Merging Jython code into standard Lib [was Re: Python Language Summit at PyCon: Agenda] In-Reply-To: References: Message-ID: <20130228210023.089665ef@pitrou.net> On Thu, 28 Feb 2013 11:46:04 -0800 "fwierzbicki at gmail.com" wrote: > On Thu, Feb 28, 2013 at 11:24 AM, Chris Jerdonek > wrote: > > On Thu, Feb 28, 2013 at 1:30 AM, Antoine Pitrou wrote: > >> Le Wed, 27 Feb 2013 11:33:30 -0800, > >> "fwierzbicki at gmail.com" a ?crit : > >>> > >>> There are a couple of spots that might be more controversial. For > >>> example, Jython has a file Lib/zlib.py that implements zlib in terms > >>> of the existing Java support for zlib. I do wonder if such a file is > >>> acceptable in CPython's Lib since its 195 lines of code would be > >>> entirely skipped by CPython. > >> > >> That's a bit annoying. How will we know that the code still works, even > >> though our buildbots don't exercise it? > >> Also, what happens if the code doesn't work anymore? > > > > Agreed on those problems. Would it be possible to use a design > > pattern in these cases so the Jython-only code wouldn't need to be > > part of the CPython repo? A naive example would be refactoring zlib > > to allow subclassing in the way that Jython needs, and then Jython > > could subclass in its own repo. CPython could have tests to check the > > subclass "contract" that Jython needs. > What about a plat-java section to parallel plat-aix4, plat-darwin, > etc? The analogy being that the Java platform is somewhat analogous to > being it's own os? And these areas are not active when on other > operating systems... IMHO, we should remove the plat-* directories, they are completely unmaintained, undocumented, and serve no useful purpose. Regards Antoine. From fwierzbicki at gmail.com Thu Feb 28 21:17:50 2013 From: fwierzbicki at gmail.com (fwierzbicki at gmail.com) Date: Thu, 28 Feb 2013 12:17:50 -0800 Subject: [Python-Dev] Merging Jython code into standard Lib [was Re: Python Language Summit at PyCon: Agenda] In-Reply-To: <20130228210023.089665ef@pitrou.net> References: <20130228210023.089665ef@pitrou.net> Message-ID: On Thu, Feb 28, 2013 at 12:00 PM, Antoine Pitrou wrote: > IMHO, we should remove the plat-* directories, they are completely > unmaintained, undocumented, and serve no useful purpose. Oh I didn't know that - so definitely adding to that is right out :) Really for cases like Jython's zlib.py (no useful code for CPython) I don't have any trouble keeping them entirely in Jython. It just would have been fun to delete our Lib/ :) It would be nice in this particular case if there was a zlib.py that imported _zlib -- then it would be easy to shim in Jython's version, whether it is written in a .py file or in Java. -Frank From brett at python.org Thu Feb 28 21:35:35 2013 From: brett at python.org (Brett Cannon) Date: Thu, 28 Feb 2013 15:35:35 -0500 Subject: [Python-Dev] Merging Jython code into standard Lib [was Re: Python Language Summit at PyCon: Agenda] In-Reply-To: References: <20130228210023.089665ef@pitrou.net> Message-ID: On Thu, Feb 28, 2013 at 3:17 PM, fwierzbicki at gmail.com < fwierzbicki at gmail.com> wrote: > On Thu, Feb 28, 2013 at 12:00 PM, Antoine Pitrou > wrote: > > IMHO, we should remove the plat-* directories, they are completely > > unmaintained, undocumented, and serve no useful purpose. > Oh I didn't know that - so definitely adding to that is right out :) > > Really for cases like Jython's zlib.py (no useful code for CPython) I > don't have any trouble keeping them entirely in Jython. It just would > have been fun to delete our Lib/ :) > > It would be nice in this particular case if there was a zlib.py that > imported _zlib -- then it would be easy to shim in Jython's version, > whether it is written in a .py file or in Java. > That should be fine as that is what we already do for accelerator modules anyway. If you want to work towards having an equivalent of CPython's Modules/ directory so you can ditch your custom Lib/ modules by treating your specific code as accelerators I think we can move towards that solution. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ericsnowcurrently at gmail.com Thu Feb 28 23:23:03 2013 From: ericsnowcurrently at gmail.com (Eric Snow) Date: Thu, 28 Feb 2013 15:23:03 -0700 Subject: [Python-Dev] Merging Jython code into standard Lib [was Re: Python Language Summit at PyCon: Agenda] In-Reply-To: References: <20130228210023.089665ef@pitrou.net> Message-ID: On Thu, Feb 28, 2013 at 1:35 PM, Brett Cannon wrote: > That should be fine as that is what we already do for accelerator modules > anyway. If you want to work towards having an equivalent of CPython's > Modules/ directory so you can ditch your custom Lib/ modules by treating > your specific code as accelerators I think we can move towards that > solution. +1 -eric