From brett at python.org Sat Jun 1 00:37:13 2013 From: brett at python.org (Brett Cannon) Date: Fri, 31 May 2013 18:37:13 -0400 Subject: [Python-Dev] [Python-checkins] cpython: Add a reset_name argument to importlib.util.module_to_load in order to In-Reply-To: <3bMfy630Xkz7LpQ@mail.python.org> References: <3bMfy630Xkz7LpQ@mail.python.org> Message-ID: I realize this broke the buildbots. Missed part of a diff in the commit. I'm trying to split a massive CL into reasonable commit sizes, so please be patient. On Fri, May 31, 2013 at 6:11 PM, brett.cannon wrote: > http://hg.python.org/cpython/rev/39cc1b04713e > changeset: 83998:39cc1b04713e > user: Brett Cannon > date: Fri May 31 18:11:17 2013 -0400 > summary: > Add a reset_name argument to importlib.util.module_to_load in order to > control whether to reset the module's __name__ attribute in case a > reload is being done. > > files: > Doc/library/importlib.rst | 6 +++++- > Lib/importlib/_bootstrap.py | 14 +++++++++++++- > Lib/test/test_importlib/test_util.py | 12 ++++++++++++ > 3 files changed, 30 insertions(+), 2 deletions(-) > > > diff --git a/Doc/library/importlib.rst b/Doc/library/importlib.rst > --- a/Doc/library/importlib.rst > +++ b/Doc/library/importlib.rst > @@ -788,7 +788,7 @@ > > .. versionadded:: 3.3 > > -.. function:: module_to_load(name) > +.. function:: module_to_load(name, *, reset_name=True) > > Returns a :term:`context manager` which provides the module to load. > The > module will either come from :attr:`sys.modules` in the case of > reloading or > @@ -796,6 +796,10 @@ > :attr:`sys.modules` occurs if the module was new and an exception was > raised. > > + If **reset_name** is true and the module requested is being reloaded > then > + the module's :attr:`__name__` attribute will > + be reset to **name**, else it will be left untouched. > + > .. versionadded:: 3.4 > > .. decorator:: module_for_loader > diff --git a/Lib/importlib/_bootstrap.py b/Lib/importlib/_bootstrap.py > --- a/Lib/importlib/_bootstrap.py > +++ b/Lib/importlib/_bootstrap.py > @@ -493,8 +493,14 @@ > > """ > > - def __init__(self, name): > + def __init__(self, name, *, reset_name=True): > + """Prepare the context manager. > + > + The reset_name argument specifies whether to unconditionally reset > + the __name__ attribute if the module is found to be a reload. > + """ > self._name = name > + self._reset_name = reset_name > > def __enter__(self): > self._module = sys.modules.get(self._name) > @@ -508,6 +514,12 @@ > # (otherwise an optimization shortcut in import.c becomes > wrong) > self._module.__initializing__ = True > sys.modules[self._name] = self._module > + elif self._reset_name: > + try: > + self._module.__name__ = self._name > + except AttributeError: > + pass > + > return self._module > > def __exit__(self, *args): > diff --git a/Lib/test/test_importlib/test_util.py > b/Lib/test/test_importlib/test_util.py > --- a/Lib/test/test_importlib/test_util.py > +++ b/Lib/test/test_importlib/test_util.py > @@ -55,6 +55,18 @@ > else: > self.fail('importlib.util.module_to_load swallowed an > exception') > > + def test_reset_name(self): > + # If reset_name is true then module.__name__ = name, else leave > it be. > + odd_name = 'not your typical name' > + created_module = imp.new_module(self.module_name) > + created_module.__name__ = odd_name > + sys.modules[self.module_name] = created_module > + with util.module_to_load(self.module_name) as module: > + self.assertEqual(module.__name__, self.module_name) > + created_module.__name__ = odd_name > + with util.module_to_load(self.module_name, reset_name=False) as > module: > + self.assertEqual(module.__name__, odd_name) > + > > class ModuleForLoaderTests(unittest.TestCase): > > > -- > Repository URL: http://hg.python.org/cpython > > _______________________________________________ > Python-checkins mailing list > Python-checkins at python.org > http://mail.python.org/mailman/listinfo/python-checkins > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From martin at v.loewis.de Sun Jun 2 12:37:32 2013 From: martin at v.loewis.de (martin at v.loewis.de) Date: Sun, 02 Jun 2013 12:37:32 +0200 Subject: [Python-Dev] Problem building Python 2.7.5 with separate sysroot In-Reply-To: <1370000129.4119.53.camel@homebase> References: <1369986770.4119.43.camel@homebase> <1370000129.4119.53.camel@homebase> Message-ID: <20130602123732.Horde.ShEJBOEliraK4rCETOXT9w9@webmail.df.eu> Quoting Paul Smith : > My case may be unusual but even in a > more formal cross-compilation environment it's not good to > add /usr/include/..., or base such a decision on the behavior of the > _build_ system. The offical procedure to cover "unusual" cases is to edit Modules/Setup. If you are not happy with the way in which modules are build, you can override all flags on a per-module basis there. > It seems to me (keeping with the theme of this mailing list) that the > add_multiarch_paths() function in setup.py is not right. Well, ..., yes. For the last two decades, the build process of Python was *always* wrong, and it always ever will be, in the sense that it doesn't support all cases that people come up with. The only way to deal with this, unfortunately, is to patch the build process for each new use case discovered. As Ned says: if you come up with a patch, don't hesitate to post it to the tracker. Regards, Martin From koobs.freebsd at gmail.com Sun Jun 2 07:12:43 2013 From: koobs.freebsd at gmail.com (Kubilay Kocak) Date: Sun, 02 Jun 2013 15:12:43 +1000 Subject: [Python-Dev] New FreeBSD 10.0-CURRENT buildbot Message-ID: <51AAD44B.4090802@FreeBSD.org> Afternoon (UTC+10), I'd like to request a new user/pass for a FreeBSD 10.0-CURRENT VM guest I've just setup as a dedicated buildbot slave to complement my other koobs-freebsd slaves. Also, with this and future additions to the FreeBSD buildslave set in mind, I think it might also be prudent for a rename to take place: koobs-freebsd9-amd64 koobs-freebsd9-amd64-clang (CC=clang) koobs-freebsd10-amd64 (clang is default here) Convention being: koobs-freebsdX[Y]-arch[-config] (Happy for feedback here) If there are any permutations or additions you'd like to specifically see for FreeBSD to increase coverage just let me know (Hint: I have a PandaBoard arm board here i'm getting up and running) I have ZFS & DTrace to work with among other things, and the long term plan is to have FreeBSD buildbots running the full gamut of versions, from -RELEASE through -STABLE to HEAD branches. I'm on #python-dev IRC if someone needs to discuss. -- Regards, Koobs From carlosnepomuceno at outlook.com Sun Jun 2 21:51:24 2013 From: carlosnepomuceno at outlook.com (Carlos Nepomuceno) Date: Sun, 2 Jun 2013 22:51:24 +0300 Subject: [Python-Dev] New FreeBSD 10.0-CURRENT buildbot In-Reply-To: <51AAD44B.4090802@FreeBSD.org> References: <51AAD44B.4090802@FreeBSD.org> Message-ID: ---------------------------------------- > Date: Sun, 2 Jun 2013 15:12:43 +1000 > From: koobs.freebsd at gmail.com > To: python-dev at python.org > Subject: [Python-Dev] New FreeBSD 10.0-CURRENT buildbot [...] > koobs-freebsd10-amd64 (clang is default here) Does CPython code compiled with clang runs faster than gcc? Why did you chose clang? Any benchmarks? Any benefits? From drsalists at gmail.com Sun Jun 2 22:43:24 2013 From: drsalists at gmail.com (Dan Stromberg) Date: Sun, 2 Jun 2013 13:43:24 -0700 Subject: [Python-Dev] New FreeBSD 10.0-CURRENT buildbot In-Reply-To: References: <51AAD44B.4090802@FreeBSD.org> Message-ID: On Sun, Jun 2, 2013 at 12:51 PM, Carlos Nepomuceno < carlosnepomuceno at outlook.com> wrote: > ---------------------------------------- > > Date: Sun, 2 Jun 2013 15:12:43 +1000 > > From: koobs.freebsd at gmail.com > > To: python-dev at python.org > > Subject: [Python-Dev] New FreeBSD 10.0-CURRENT buildbot > [...] > > koobs-freebsd10-amd64 (clang is default here) > > > Does CPython code compiled with clang runs faster than gcc? > > Why did you chose clang? Any benchmarks? Any benefits? > Clang is sometimes favored over gcc for its non-GPL license; I believe the FreeBSD project sees this as an important issue. In general, the more C compilers a piece of C code compiles on, the fewer bugs are left in it, because different C compilers tend to catch different problems - even with cranked up warnings. -------------- next part -------------- An HTML attachment was scrubbed... URL: From benjamin at python.org Mon Jun 3 04:16:17 2013 From: benjamin at python.org (Benjamin Peterson) Date: Sun, 2 Jun 2013 19:16:17 -0700 Subject: [Python-Dev] PEP 442: Safe object finalization In-Reply-To: <20130518163315.6a21a0cd@fsol> References: <20130518105910.1eecfa5f@fsol> <20130518154552.2a879bf6@fsol> <20130518163315.6a21a0cd@fsol> Message-ID: 2013/5/18 Antoine Pitrou : > Calling finalizers only once is fine with me, but it would be a change > in behaviour; I don't know if it may break existing code. I agree with Armin that this is better behavior. (Mostly significantly consistent with weakrefs.) > > (for example, say someone is using __del__ to manage a freelist) Do you know if it breaks any of the projects you tested it with? -- Regards, Benjamin From benjamin at python.org Mon Jun 3 04:27:49 2013 From: benjamin at python.org (Benjamin Peterson) Date: Sun, 2 Jun 2013 19:27:49 -0700 Subject: [Python-Dev] PEP 442: Safe object finalization In-Reply-To: <20130518105910.1eecfa5f@fsol> References: <20130518105910.1eecfa5f@fsol> Message-ID: 2013/5/18 Antoine Pitrou : > > Hello, > > I would like to submit the following PEP for discussion and evaluation. Will the API of the gc module be at all affected? I assume nothing will just be printed for DEBUG_UNCOLLECTABLE. Maybe there should be a way to discover when a cycle is resurrected? -- Regards, Benjamin From donald at stufft.io Mon Jun 3 07:20:42 2013 From: donald at stufft.io (Donald Stufft) Date: Mon, 3 Jun 2013 01:20:42 -0400 Subject: [Python-Dev] Validating SSL By Default (aka Including a Cert Bundle in CPython) Message-ID: As of right now, as far as I can tell, Python does not validate HTTPS certificates by default. As far as I can tell this is because there is no guaranteed certificates available. So I would like to propose that CPython adopt the Mozilla SSL certificate list and include it in core, and switch over the API's so that they verify HTTPS by default. This is what most people are going to expect when using a https url (Especially after learning that Python 2.x doesn't verify TLS, but Python 3.x "does"). Ideally this would take the shape of attempting to locate the system certificate store if possible, and if that doesn't work falling back to the bundled certificates. That way the various Linux distros can easily have their copies of Python depend soley on their built in certs, but Windows, OSX, Source compiles etc will all still have a fallback value. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 841 bytes Desc: Message signed with OpenPGP using GPGMail URL: From clp2 at rebertia.com Mon Jun 3 07:57:15 2013 From: clp2 at rebertia.com (Chris Rebert) Date: Sun, 2 Jun 2013 22:57:15 -0700 Subject: [Python-Dev] Validating SSL By Default (aka Including a Cert Bundle in CPython) In-Reply-To: References: Message-ID: On Jun 2, 2013 10:22 PM, "Donald Stufft" wrote: > > As of right now, as far as I can tell, Python does not validate HTTPS certificates by default. As far as I can tell this is because there is no guaranteed certificates available. Relevant: http://bugs.python.org/issue13647 > So I would like to propose that CPython adopt the Mozilla SSL certificate list and include it in core, and switch over the API's so that they verify HTTPS by default. This is what most people are going to expect when using a https url (Especially after learning that Python 2.x doesn't verify TLS, but Python 3.x "does"). > > Ideally this would take the shape of attempting to locate the system certificate store if possible, and if that doesn't work falling back to the bundled certificates. That way the various Linux distros can easily have their copies of Python depend solely on their built in certs, but Windows, OSX, Source compiles etc will all still have a fallback value. There's an existing request for this: http://bugs.python.org/issue13655 Cheers, Chris From benjamin at python.org Mon Jun 3 07:58:12 2013 From: benjamin at python.org (Benjamin Peterson) Date: Sun, 2 Jun 2013 22:58:12 -0700 Subject: [Python-Dev] Validating SSL By Default (aka Including a Cert Bundle in CPython) In-Reply-To: References: Message-ID: 2013/6/2 Donald Stufft : > As of right now, as far as I can tell, Python does not validate HTTPS > certificates by default. As far as I can tell this is because there is no > guaranteed certificates available. > > So I would like to propose that CPython adopt the Mozilla SSL certificate > list and include it in core, and switch over the API's so that they verify > HTTPS by default. +1 > > Ideally this would take the shape of attempting to locate the system > certificate store if possible, and if that doesn't work falling back to the > bundled certificates. That way the various Linux distros can easily have > their copies of Python depend soley on their built in certs, but Windows, > OSX, Source compiles etc will all still have a fallback value. My preference would be actually be for the included certificates file to be used by default. This would provide a consistent experience across platforms. We could provide options to look for system cert repositories if desired. -- Regards, Benjamin From donald at stufft.io Mon Jun 3 09:12:05 2013 From: donald at stufft.io (Donald Stufft) Date: Mon, 3 Jun 2013 03:12:05 -0400 Subject: [Python-Dev] Validating SSL By Default (aka Including a Cert Bundle in CPython) In-Reply-To: References: Message-ID: <0583C689-56BE-4F21-9E16-0D06746295C9@stufft.io> On Jun 3, 2013, at 1:58 AM, Benjamin Peterson wrote: > 2013/6/2 Donald Stufft : >> As of right now, as far as I can tell, Python does not validate HTTPS >> certificates by default. As far as I can tell this is because there is no >> guaranteed certificates available. >> >> So I would like to propose that CPython adopt the Mozilla SSL certificate >> list and include it in core, and switch over the API's so that they verify >> HTTPS by default. > > +1 > >> >> Ideally this would take the shape of attempting to locate the system >> certificate store if possible, and if that doesn't work falling back to the >> bundled certificates. That way the various Linux distros can easily have >> their copies of Python depend soley on their built in certs, but Windows, >> OSX, Source compiles etc will all still have a fallback value. > > My preference would be actually be for the included certificates file > to be used by default. This would provide a consistent experience > across platforms. We could provide options to look for system cert > repositories if desired. That's fine with me too. My only reason for wanting to use the system certs first is so if someone has modified their system certs (say to include a corporate cert) that it would ideally take affect for Python as well. But honestly the Linux distros will probably modify things to use system certs anyways and non Linux (esp Windows) probably doesn't have a way to get those system certs into OpenSSL. > > > > -- > Regards, > Benjamin ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 841 bytes Desc: Message signed with OpenPGP using GPGMail URL: From benjamin at python.org Mon Jun 3 09:14:39 2013 From: benjamin at python.org (Benjamin Peterson) Date: Mon, 3 Jun 2013 00:14:39 -0700 Subject: [Python-Dev] Validating SSL By Default (aka Including a Cert Bundle in CPython) In-Reply-To: <0583C689-56BE-4F21-9E16-0D06746295C9@stufft.io> References: <0583C689-56BE-4F21-9E16-0D06746295C9@stufft.io> Message-ID: 2013/6/3 Donald Stufft : > > On Jun 3, 2013, at 1:58 AM, Benjamin Peterson wrote: > > 2013/6/2 Donald Stufft : > > As of right now, as far as I can tell, Python does not validate HTTPS > certificates by default. As far as I can tell this is because there is no > guaranteed certificates available. > > So I would like to propose that CPython adopt the Mozilla SSL certificate > list and include it in core, and switch over the API's so that they verify > HTTPS by default. > > > +1 > > > Ideally this would take the shape of attempting to locate the system > certificate store if possible, and if that doesn't work falling back to the > bundled certificates. That way the various Linux distros can easily have > their copies of Python depend soley on their built in certs, but Windows, > OSX, Source compiles etc will all still have a fallback value. > > > My preference would be actually be for the included certificates file > to be used by default. This would provide a consistent experience > across platforms. We could provide options to look for system cert > repositories if desired. > > > That's fine with me too. My only reason for wanting to use the system certs > first is so > if someone has modified their system certs (say to include a corporate cert) > that it > would ideally take affect for Python as well. I don't think users should be able to modify stdlib behaviors (in this case could be unintentionally) without application consent. > > But honestly the Linux distros will probably modify things to use system > certs anyways > and non Linux (esp Windows) probably doesn't have a way to get those system > certs > into OpenSSL. Yes, I'm happy to let them figure it out. -- Regards, Benjamin From benhoyt at gmail.com Mon Jun 3 09:38:45 2013 From: benhoyt at gmail.com (Ben Hoyt) Date: Mon, 3 Jun 2013 19:38:45 +1200 Subject: [Python-Dev] Validating SSL By Default (aka Including a Cert Bundle in CPython) In-Reply-To: References: Message-ID: Love this idea. Some third-party HTTP libraries turn this on by default in any case (eg: requests, and I think others), so this would mean Python would get their "safe-by-default" behaviour in its stdlib. > > Ideally this would take the shape of attempting to locate the system > > certificate store if possible, and if that doesn't work falling back to the > > bundled certificates. That way the various Linux distros can easily have > > their copies of Python depend soley on their built in certs, but Windows, > > OSX, Source compiles etc will all still have a fallback value. > > My preference would be actually be for the included certificates file > to be used by default. This would provide a consistent experience > across platforms. We could provide options to look for system cert > repositories if desired. Very much agreed. When the Windows version of the mimetypes module tried to use Windows' system mimetype mappings by default, chaos and bugs ensued (for example, http://bugs.python.org/issue15207 and http://bugs.python.org/issue10551). -Ben From solipsis at pitrou.net Mon Jun 3 11:07:43 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 3 Jun 2013 11:07:43 +0200 Subject: [Python-Dev] Validating SSL By Default (aka Including a Cert Bundle in CPython) References: Message-ID: <20130603110743.23886e58@fsol> On Mon, 3 Jun 2013 01:20:42 -0400 Donald Stufft wrote: > As of right now, as far as I can tell, Python does not validate HTTPS > certificates by default. As far as I can tell this is because there is > no guaranteed certificates available. Also for backwards compatibility. Regards Antoine. From solipsis at pitrou.net Mon Jun 3 11:22:05 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 3 Jun 2013 11:22:05 +0200 Subject: [Python-Dev] Validating SSL By Default (aka Including a Cert Bundle in CPython) References: Message-ID: <20130603112205.4f76913e@fsol> On Sun, 2 Jun 2013 22:57:15 -0700 Chris Rebert wrote: > On Jun 2, 2013 10:22 PM, "Donald Stufft" wrote: > > > > As of right now, as far as I can tell, Python does not validate HTTPS certificates by default. As far as I can tell this is because there is no guaranteed certificates available. > > Relevant: http://bugs.python.org/issue13647 > > > So I would like to propose that CPython adopt the Mozilla SSL certificate list and include it in core, and switch over the API's so that they verify HTTPS by default. This is what most people are going to expect when using a https url (Especially after learning that Python 2.x doesn't verify TLS, but Python 3.x "does"). > > > > Ideally this would take the shape of attempting to locate the system certificate store if possible, and if that doesn't work falling back to the bundled certificates. That way the various Linux distros can easily have their copies of Python depend solely on their built in certs, but Windows, OSX, Source compiles etc will all still have a fallback value. > > There's an existing request for this: > http://bugs.python.org/issue13655 See also http://bugs.python.org/issue17134 Regards Antoine. From rdmurray at bitdance.com Mon Jun 3 11:28:51 2013 From: rdmurray at bitdance.com (R. David Murray) Date: Mon, 03 Jun 2013 05:28:51 -0400 Subject: [Python-Dev] Validating SSL By Default (aka Including a Cert Bundle in CPython) In-Reply-To: References: Message-ID: <20130603092852.2A6532500B9@webabinitio.net> On Mon, 03 Jun 2013 19:38:45 +1200, Ben Hoyt wrote: > Love this idea. Some third-party HTTP libraries turn this on by > default in any case (eg: requests, and I think others), so this would > mean Python would get their "safe-by-default" behaviour in its stdlib. > > > > Ideally this would take the shape of attempting to locate the system > > > certificate store if possible, and if that doesn't work falling back to the > > > bundled certificates. That way the various Linux distros can easily have > > > their copies of Python depend soley on their built in certs, but Windows, > > > OSX, Source compiles etc will all still have a fallback value. > > > > My preference would be actually be for the included certificates file > > to be used by default. This would provide a consistent experience > > across platforms. We could provide options to look for system cert > > repositories if desired. > > Very much agreed. When the Windows version of the mimetypes module > tried to use Windows' system mimetype mappings by default, chaos and > bugs ensued (for example, http://bugs.python.org/issue15207 and > http://bugs.python.org/issue10551). On the other hand, the fact that it uses /etc/mimetypes on unix does *not* cause chaos, quite the opposite: it means Python recognizes new mime types provided by the distro automatically, which provides for a more consistent experience for the Python application user. The situation for certs is probably fairly parallel: on unix, it would probably be an advantage as Python would automatically follow distro decisions about cert chains, while on windows trying to use a system cert store would probably be a disaster. An application can choose to explicitly ignore the system mimetypes file, by the way. That said, I don't feel strongly either way about the cert store case. --David From benhoyt at gmail.com Mon Jun 3 11:37:10 2013 From: benhoyt at gmail.com (Ben Hoyt) Date: Mon, 3 Jun 2013 21:37:10 +1200 Subject: [Python-Dev] Validating SSL By Default (aka Including a Cert Bundle in CPython) In-Reply-To: <20130603092852.2A6532500B9@webabinitio.net> References: <20130603092852.2A6532500B9@webabinitio.net> Message-ID: > The situation for certs is probably fairly parallel: on unix, it would > probably be an advantage as Python would automatically follow distro > decisions about cert chains, while on windows trying to use a system > cert store would probably be a disaster. Yeah, fair enough. If it's stable and just works on Linux, this would be an advantage. > An application can choose to explicitly ignore the system mimetypes > file, by the way. My main concern is that it could be broken out of the box on Windows (mimetypes currently is), and callers have to go out of their way to find this workaround. I'm not familiar with Unix/Linux, but on Windows, if it's anything like mimetypes it'll be really hard to get consistent behaviour across different boxes/versions from the registry, or wherever certs might be stored on Windows. I'd much rather have a slightly outdated but consistent experience by default. -Ben From christian at python.org Mon Jun 3 11:45:10 2013 From: christian at python.org (Christian Heimes) Date: Mon, 03 Jun 2013 11:45:10 +0200 Subject: [Python-Dev] Validating SSL By Default (aka Including a Cert Bundle in CPython) In-Reply-To: References: Message-ID: <51AC65A6.80602@python.org> Am 03.06.2013 07:20, schrieb Donald Stufft: > Ideally this would take the shape of attempting to locate the system > certificate store if possible, and if that doesn't work falling back to > the bundled certificates. That way the various Linux distros can easily > have their copies of Python depend soley on their built in certs, but > Windows, OSX, Source compiles etc will all still have a fallback value. On Windows it's rather easy to access the cert storage with a couple of calls to crypt32.dll. I have created a smallish ctypes interface: https://bitbucket.org/tiran/wincertstore/overview . setuptools 0.7 is using it to for its ssl support. I'm going to write a patch for Python 3.4, too. See http://bugs.python.org/issue17134 I welcome a cert bundle in CPython. But how are we suppose to handle updates and invalidation of CAs? A couple of months ago there was an intense discussion about a copy of the Olson time zone database in Python's stdlib. People argued against it as they feared outdated tz information. IMHO outdated CA certs are much more severe than tz ... I'd like to see OCSP support, too. Unfortunately OpenSSL's OCSP API is far from trivial. I had a look at it once. Christian From solipsis at pitrou.net Mon Jun 3 11:51:30 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 3 Jun 2013 11:51:30 +0200 Subject: [Python-Dev] Validating SSL By Default (aka Including a Cert Bundle in CPython) References: <20130603092852.2A6532500B9@webabinitio.net> Message-ID: <20130603115130.299b2504@fsol> On Mon, 3 Jun 2013 21:37:10 +1200 Ben Hoyt wrote: > > I'm not familiar with Unix/Linux, but on Windows, if it's anything > like mimetypes it'll be really hard to get consistent behaviour across > different boxes/versions from the registry, or wherever certs might be > stored on Windows. I'd much rather have a slightly outdated but > consistent experience by default. The problem with a "slightly outdated" CA store is that it can be a security risk. Regards Antoine. From solipsis at pitrou.net Mon Jun 3 11:52:52 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 3 Jun 2013 11:52:52 +0200 Subject: [Python-Dev] PEP 442: Safe object finalization In-Reply-To: References: <20130518105910.1eecfa5f@fsol> Message-ID: <20130603115252.6c07ff38@fsol> On Sun, 2 Jun 2013 19:27:49 -0700 Benjamin Peterson wrote: > 2013/5/18 Antoine Pitrou : > > > > Hello, > > > > I would like to submit the following PEP for discussion and evaluation. > > Will the API of the gc module be at all affected? I assume nothing > will just be printed for DEBUG_UNCOLLECTABLE. Objects with tp_del may still exist (third-party extensions perhaps). > Maybe there should be a > way to discover when a cycle is resurrected? Is it more important than discovering when a non-cycle is resurrected? Regards Antoine. From solipsis at pitrou.net Mon Jun 3 11:53:55 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 3 Jun 2013 11:53:55 +0200 Subject: [Python-Dev] PEP 442: Safe object finalization In-Reply-To: References: <20130518105910.1eecfa5f@fsol> <20130518154552.2a879bf6@fsol> <20130518163315.6a21a0cd@fsol> Message-ID: <20130603115355.356bc4e4@fsol> On Sun, 2 Jun 2013 19:16:17 -0700 Benjamin Peterson wrote: > 2013/5/18 Antoine Pitrou : > > Calling finalizers only once is fine with me, but it would be a change > > in behaviour; I don't know if it may break existing code. > > I agree with Armin that this is better behavior. (Mostly significantly > consistent with weakrefs.) Keep in mind that it is a limitation of weakrefs, not a feature: you can't "unclear" a weakref. > > (for example, say someone is using __del__ to manage a freelist) > > Do you know if it breaks any of the projects you tested it with? I haven't tested it (yet). Regards Antoine. From benhoyt at gmail.com Mon Jun 3 11:57:10 2013 From: benhoyt at gmail.com (Ben Hoyt) Date: Mon, 3 Jun 2013 21:57:10 +1200 Subject: [Python-Dev] Validating SSL By Default (aka Including a Cert Bundle in CPython) In-Reply-To: <20130603115130.299b2504@fsol> References: <20130603092852.2A6532500B9@webabinitio.net> <20130603115130.299b2504@fsol> Message-ID: >> I'm not familiar with Unix/Linux, but on Windows, if it's anything >> like mimetypes it'll be really hard to get consistent behaviour across >> different boxes/versions from the registry, or wherever certs might be >> stored on Windows. I'd much rather have a slightly outdated but >> consistent experience by default. > > The problem with a "slightly outdated" CA store is that it can be a > security risk. True. This is different from mimetypes in that respect. Also, with Christian's post about Windows cert store, it does look like Windows has a stable/documented way to get this from the system. I wonder, are the crypt32.dll Cert* functions what IE uses? -Ben From ben at bendarnell.com Mon Jun 3 15:05:14 2013 From: ben at bendarnell.com (Ben Darnell) Date: Mon, 3 Jun 2013 09:05:14 -0400 Subject: [Python-Dev] Validating SSL By Default (aka Including a Cert Bundle in CPython) In-Reply-To: References: Message-ID: On Mon, Jun 3, 2013 at 1:20 AM, Donald Stufft wrote: > As of right now, as far as I can tell, Python does not validate HTTPS > certificates by default. As far as I can tell this is because there is no > guaranteed certificates available. > > So I would like to propose that CPython adopt the Mozilla SSL certificate > list and include it in core, and switch over the API's so that they verify > HTTPS by default. This is what most people are going to expect when using a > https url (Especially after learning that Python 2.x doesn't verify TLS, > but Python 3.x "does"). > > Ideally this would take the shape of attempting to locate the system > certificate store if possible, and if that doesn't work falling back to the > bundled certificates. That way the various Linux distros can easily have > their copies of Python depend soley on their built in certs, but Windows, > OSX, Source compiles etc will all still have a fallback value. > +1. I bundle a copy of the Mozilla CA list with Tornado, but would love to access the system's CA roots and/or use a Python-provided copy. I'd prefer to use the certificates from the operating system when possible, as that list is most likely to receive timely security updates (or be updated with a local corporate CA, for example). It's better to aim for consistency with the user's browser than consistency of Python applications across different installations. The data is analogous to the time zone database (PEP 431) in that it may need to be updated independently of Python's own release schedule, so we may want to use similar techniques to manage both. Also see certifi ( https://pypi.python.org/pypi/certifi), which is a copy of the Mozilla list in a pip-installable form. -Ben > > ----------------- > Donald Stufft > PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 > DCFA > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/ben%40bendarnell.com > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ethan at stoneleaf.us Mon Jun 3 17:27:24 2013 From: ethan at stoneleaf.us (Ethan Furman) Date: Mon, 03 Jun 2013 08:27:24 -0700 Subject: [Python-Dev] Validating SSL By Default (aka Including a Cert Bundle in CPython) In-Reply-To: References: Message-ID: <51ACB5DC.3040902@stoneleaf.us> On 06/02/2013 10:20 PM, Donald Stufft wrote: > > So I would like to propose that CPython adopt the Mozilla SSL certificate list and include it in core, and switch over > the API's so that they verify HTTPS by default. +1 Also, +1 on using system certs when available. -- ~Ethan~ From donald at stufft.io Mon Jun 3 18:43:32 2013 From: donald at stufft.io (Donald Stufft) Date: Mon, 3 Jun 2013 12:43:32 -0400 Subject: [Python-Dev] Validating SSL By Default (aka Including a Cert Bundle in CPython) In-Reply-To: <20130603115130.299b2504@fsol> References: <20130603092852.2A6532500B9@webabinitio.net> <20130603115130.299b2504@fsol> Message-ID: <95CF5916-B6AD-4DF0-80FF-12D3393FED32@stufft.io> On Jun 3, 2013, at 5:51 AM, Antoine Pitrou wrote: > On Mon, 3 Jun 2013 21:37:10 +1200 > Ben Hoyt wrote: >> >> I'm not familiar with Unix/Linux, but on Windows, if it's anything >> like mimetypes it'll be really hard to get consistent behaviour across >> different boxes/versions from the registry, or wherever certs might be >> stored on Windows. I'd much rather have a slightly outdated but >> consistent experience by default. > > The problem with a "slightly outdated" CA store is that it can be a > security risk. > > Regards > > Antoine. > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/donald%40stufft.io Tracking the Mozilla store isn't difficult. New additions can be ignored for currently released Pythons so we'd just need to watch them for blacklisting certs and roll that into a security update. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 841 bytes Desc: Message signed with OpenPGP using GPGMail URL: From solipsis at pitrou.net Mon Jun 3 18:56:08 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 3 Jun 2013 18:56:08 +0200 Subject: [Python-Dev] Validating SSL By Default (aka Including a Cert Bundle in CPython) In-Reply-To: <95CF5916-B6AD-4DF0-80FF-12D3393FED32@stufft.io> References: <20130603092852.2A6532500B9@webabinitio.net> <20130603115130.299b2504@fsol> <95CF5916-B6AD-4DF0-80FF-12D3393FED32@stufft.io> Message-ID: <20130603185608.4cbfae0f@fsol> On Mon, 3 Jun 2013 12:43:32 -0400 Donald Stufft wrote: > > On Jun 3, 2013, at 5:51 AM, Antoine Pitrou wrote: > > > On Mon, 3 Jun 2013 21:37:10 +1200 > > Ben Hoyt wrote: > >> > >> I'm not familiar with Unix/Linux, but on Windows, if it's anything > >> like mimetypes it'll be really hard to get consistent behaviour across > >> different boxes/versions from the registry, or wherever certs might be > >> stored on Windows. I'd much rather have a slightly outdated but > >> consistent experience by default. > > > > The problem with a "slightly outdated" CA store is that it can be a > > security risk. > > > > Regards > > > > Antoine. > > > > > > _______________________________________________ > > Python-Dev mailing list > > Python-Dev at python.org > > http://mail.python.org/mailman/listinfo/python-dev > > Unsubscribe: http://mail.python.org/mailman/options/python-dev/donald%40stufft.io > > Tracking the Mozilla store isn't difficult. New additions can be ignored for currently released Pythons so we'd just need to watch them for blacklisting certs and roll that into a security update. Let's see if our security release managers want to do that job. Regards Antoine. From ethan at stoneleaf.us Mon Jun 3 18:52:27 2013 From: ethan at stoneleaf.us (Ethan Furman) Date: Mon, 03 Jun 2013 09:52:27 -0700 Subject: [Python-Dev] Validating SSL By Default (aka Including a Cert Bundle in CPython) In-Reply-To: <95CF5916-B6AD-4DF0-80FF-12D3393FED32@stufft.io> References: <20130603092852.2A6532500B9@webabinitio.net> <20130603115130.299b2504@fsol> <95CF5916-B6AD-4DF0-80FF-12D3393FED32@stufft.io> Message-ID: <51ACC9CB.1090407@stoneleaf.us> On 06/03/2013 09:43 AM, Donald Stufft wrote: > On Jun 3, 2013, at 5:51 AM, Antoine Pitrou wrote: >> >> The problem with a "slightly outdated" CA store is that it can be a >> security risk. > > Tracking the Mozilla store isn't difficult. New additions can be ignored for currently released Pythons so we'd just > need to watch them for blacklisting certs and roll that into a security update. Personally, I'm not interested in waiting six months for an update. And why can't I have the new additions? Seems to me a better solution is to have routines that can query and update at will (meaning the app has to call them), as well as having the bundle (black lists as well as new additions) in the regular updates. -- ~Ethan~ From donald at stufft.io Mon Jun 3 19:55:16 2013 From: donald at stufft.io (Donald Stufft) Date: Mon, 3 Jun 2013 13:55:16 -0400 Subject: [Python-Dev] Validating SSL By Default (aka Including a Cert Bundle in CPython) In-Reply-To: <51ACC9CB.1090407@stoneleaf.us> References: <20130603092852.2A6532500B9@webabinitio.net> <20130603115130.299b2504@fsol> <95CF5916-B6AD-4DF0-80FF-12D3393FED32@stufft.io> <51ACC9CB.1090407@stoneleaf.us> Message-ID: <6A511D87-832C-4686-83D0-642C05D8B60B@stufft.io> On Jun 3, 2013, at 12:52 PM, Ethan Furman wrote: > On 06/03/2013 09:43 AM, Donald Stufft wrote: >> On Jun 3, 2013, at 5:51 AM, Antoine Pitrou wrote: >>> >>> The problem with a "slightly outdated" CA store is that it can be a >>> security risk. >> >> Tracking the Mozilla store isn't difficult. New additions can be ignored for currently released Pythons so we'd just >> need to watch them for blacklisting certs and roll that into a security update. > > Personally, I'm not interested in waiting six months for an update. And why can't I have the new additions? > > Seems to me a better solution is to have routines that can query and update at will (meaning the app has to call them), as well as having the bundle (black lists as well as new additions) in the regular updates. > > -- > ~Ethan~ > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/donald%40stufft.io Personally I view that as a nice to have but if an app is willing to call hem they can easily depend on certifi and just replace the default certificates. The goal here is to get secure by default without any work on the part of the developers. If a developer knows to update the certs they can know to use certifi. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 841 bytes Desc: Message signed with OpenPGP using GPGMail URL: From barry at python.org Mon Jun 3 18:36:36 2013 From: barry at python.org (Barry Warsaw) Date: Mon, 3 Jun 2013 12:36:36 -0400 Subject: [Python-Dev] Validating SSL By Default (aka Including a Cert Bundle in CPython) In-Reply-To: References: Message-ID: <20130603123636.7a44dd9c@anarchist> On Jun 03, 2013, at 01:20 AM, Donald Stufft wrote: >So I would like to propose that CPython adopt the Mozilla SSL certificate >list and include it in core, and switch over the API's so that they verify >HTTPS by default. This is what most people are going to expect when using a >https url (Especially after learning that Python 2.x doesn't verify TLS, but >Python 3.x "does"). For the "verify HTTPS by default", do you mean specifically changing the cadefault argument to urllib.request.urlopen() to True? Note that I recently closed a bug related to this: http://bugs.python.org/issue17977 +1 for changing the default to True. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From barry at python.org Mon Jun 3 18:48:36 2013 From: barry at python.org (Barry Warsaw) Date: Mon, 3 Jun 2013 12:48:36 -0400 Subject: [Python-Dev] Validating SSL By Default (aka Including a Cert Bundle in CPython) In-Reply-To: References: Message-ID: <20130603124836.23a83360@anarchist> On Jun 03, 2013, at 09:05 AM, Ben Darnell wrote: >The data is analogous to the time zone database (PEP 431) in that it may >need to be updated independently of Python's own release schedule, so we >may want to use similar techniques to manage both. Also see certifi ( >https://pypi.python.org/pypi/certifi), which is a copy of the Mozilla list >in a pip-installable form. Right, this is very much analogous, except with the additional twist that out-of-date certificates can pose a significant security risk. I'm fairly certain that Debian and Ubuntu would explicitly not use any certificates shipped with Python, for two main reasons: 1) our security teams already manage the certificate store distro-wide and we want to make sure that one update fixes everything; 2) we don't want to duplicate code in multiple packages[1]. So *if* Python decides to do this (and I'm -0, but from a decidedly Linux-distro bias), it must be easily disabled. I generally like the way PEP 431 handles the tzdata, so I think we should do the same here. -Barry [1] This gives us headaches in upstreams like coverage caused by bundling externally available JavaScript libraries, or like urllib3 bundling chardet and urllib3, not to mention their own certificates yet again. :( From donald at stufft.io Mon Jun 3 20:12:34 2013 From: donald at stufft.io (Donald Stufft) Date: Mon, 3 Jun 2013 14:12:34 -0400 Subject: [Python-Dev] Validating SSL By Default (aka Including a Cert Bundle in CPython) In-Reply-To: <20130603123636.7a44dd9c@anarchist> References: <20130603123636.7a44dd9c@anarchist> Message-ID: On Jun 3, 2013, at 12:36 PM, Barry Warsaw wrote: > On Jun 03, 2013, at 01:20 AM, Donald Stufft wrote: > >> So I would like to propose that CPython adopt the Mozilla SSL certificate >> list and include it in core, and switch over the API's so that they verify >> HTTPS by default. This is what most people are going to expect when using a >> https url (Especially after learning that Python 2.x doesn't verify TLS, but >> Python 3.x "does"). > > For the "verify HTTPS by default", do you mean specifically changing the > cadefault argument to urllib.request.urlopen() to True? Note that I recently > closed a bug related to this: > > http://bugs.python.org/issue17977 > > +1 for changing the default to True. > > -Barry > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/donald%40stufft.io If that's all it takes to make sure that people have to opt out of an invalid certificate throwing an error than yes :) My goal here is to make it that when someone accesses a TLS secured network resource (I said HTTP, but if there are other things, xmlrpclib, or what not where people can reasonably expect valid TLS certificates those too) they are protected by a MITM attack by default. I worry with the current situation people will just use TLS connections without realizing it's not being verified and thinking they are "safe". As with most things security the outcome of "failing" to do it right when the default is insecure is well nothing until someone attacks you. So Hopefully we make things secure by default for folks :) ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 841 bytes Desc: Message signed with OpenPGP using GPGMail URL: From barry at python.org Mon Jun 3 18:52:00 2013 From: barry at python.org (Barry Warsaw) Date: Mon, 3 Jun 2013 12:52:00 -0400 Subject: [Python-Dev] Validating SSL By Default (aka Including a Cert Bundle in CPython) In-Reply-To: <0583C689-56BE-4F21-9E16-0D06746295C9@stufft.io> References: <0583C689-56BE-4F21-9E16-0D06746295C9@stufft.io> Message-ID: <20130603125200.69de2e13@anarchist> On Jun 03, 2013, at 03:12 AM, Donald Stufft wrote: >That's fine with me too. My only reason for wanting to use the system certs >first is so if someone has modified their system certs (say to include a >corporate cert) that it would ideally take affect for Python as well. This reminds me of one other thing. We have to make sure that the APIs (e.g urlopen()) continue to allow us to use self-signed certificates, if for no other reason than for testing purposes. OTOH, taking this away would be a backward incompatible change in API so probably wouldn't happen anyway. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From donald at stufft.io Mon Jun 3 20:17:48 2013 From: donald at stufft.io (Donald Stufft) Date: Mon, 3 Jun 2013 14:17:48 -0400 Subject: [Python-Dev] Validating SSL By Default (aka Including a Cert Bundle in CPython) In-Reply-To: <20130603124836.23a83360@anarchist> References: <20130603124836.23a83360@anarchist> Message-ID: <1B90BE69-8DA4-4565-A0C3-E0E576E6881B@stufft.io> On Jun 3, 2013, at 12:48 PM, Barry Warsaw wrote: > On Jun 03, 2013, at 09:05 AM, Ben Darnell wrote: > >> The data is analogous to the time zone database (PEP 431) in that it may >> need to be updated independently of Python's own release schedule, so we >> may want to use similar techniques to manage both. Also see certifi ( >> https://pypi.python.org/pypi/certifi), which is a copy of the Mozilla list >> in a pip-installable form. > > Right, this is very much analogous, except with the additional twist that > out-of-date certificates can pose a significant security risk. > > I'm fairly certain that Debian and Ubuntu would explicitly not use any > certificates shipped with Python, for two main reasons: 1) our security teams > already manage the certificate store distro-wide and we want to make sure that > one update fixes everything; 2) we don't want to duplicate code in multiple > packages[1]. > > So *if* Python decides to do this (and I'm -0, but from a decidedly > Linux-distro bias), it must be easily disabled. I generally like the way PEP > 431 handles the tzdata, so I think we should do the same here. I'd actually prefer for Linux to not use the bundled certs when installed from a package manager because it should use the system certs, but people can't depend on certs being there if they are only there on linux. Adding them into Python means people _can_ depend on them being there, and Windows and other systems without system integrators to modify it to use the system store will still get certs and Ubuntu can make it just work(?). > > -Barry > > [1] This gives us headaches in upstreams like coverage caused by bundling > externally available JavaScript libraries, or like urllib3 bundling chardet > and urllib3, not to mention their own certificates yet again. :( This would probably (eventually) make the bundling of certificates better too. Meaning that once it's been in long enough people are willing to depend on it, they won't need to bundle their own certs and ubuntu/debian can just modify the one location instead of needing to modify it for every package that does it. > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/donald%40stufft.io ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 841 bytes Desc: Message signed with OpenPGP using GPGMail URL: From donald at stufft.io Mon Jun 3 20:21:06 2013 From: donald at stufft.io (Donald Stufft) Date: Mon, 3 Jun 2013 14:21:06 -0400 Subject: [Python-Dev] Validating SSL By Default (aka Including a Cert Bundle in CPython) In-Reply-To: <20130603125200.69de2e13@anarchist> References: <0583C689-56BE-4F21-9E16-0D06746295C9@stufft.io> <20130603125200.69de2e13@anarchist> Message-ID: <706472F8-31AB-4A6D-91EA-EA6E37F0BAC8@stufft.io> On Jun 3, 2013, at 12:52 PM, Barry Warsaw wrote: > On Jun 03, 2013, at 03:12 AM, Donald Stufft wrote: > >> That's fine with me too. My only reason for wanting to use the system certs >> first is so if someone has modified their system certs (say to include a >> corporate cert) that it would ideally take affect for Python as well. > > This reminds me of one other thing. We have to make sure that the APIs > (e.g urlopen()) continue to allow us to use self-signed certificates, if for > no other reason than for testing purposes. OTOH, taking this away would be a > backward incompatible change in API so probably wouldn't happen anyway. > > -Barry > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/donald%40stufft.io The other additional comment I'd like to throw in here is that if we don't bundle SSL certs I think we should still verify by default (which means HTTPS urls will throw an error by default if we can't locate a certificate store) because I think the risk to people unknowingly thinking that their HTTPS urls are protected are significant enough that this "error" shouldn't be silent by default. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 841 bytes Desc: Message signed with OpenPGP using GPGMail URL: From barry at python.org Mon Jun 3 19:04:07 2013 From: barry at python.org (Barry Warsaw) Date: Mon, 3 Jun 2013 13:04:07 -0400 Subject: [Python-Dev] Validating SSL By Default (aka Including a Cert Bundle in CPython) In-Reply-To: <706472F8-31AB-4A6D-91EA-EA6E37F0BAC8@stufft.io> References: <0583C689-56BE-4F21-9E16-0D06746295C9@stufft.io> <20130603125200.69de2e13@anarchist> <706472F8-31AB-4A6D-91EA-EA6E37F0BAC8@stufft.io> Message-ID: <20130603130407.43d1cd0d@anarchist> On Jun 03, 2013, at 02:21 PM, Donald Stufft wrote: >The other additional comment I'd like to throw in here is that if we don't >bundle SSL certs I think we should still verify by default (which means HTTPS >urls will throw an error by default if we can't locate a certificate store) >because I think the risk to people unknowingly thinking that their HTTPS urls >are protected are significant enough that this "error" shouldn't be silent by >default. +1, especially if we ensure that the APIs are available to not verify, as is currently the case with urlopen(). I don't think people will want to do that in production, but it will be useful for testing (e.g. guess how I found issues 17977 :). -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From barry at python.org Mon Jun 3 19:07:00 2013 From: barry at python.org (Barry Warsaw) Date: Mon, 3 Jun 2013 13:07:00 -0400 Subject: [Python-Dev] Validating SSL By Default (aka Including a Cert Bundle in CPython) In-Reply-To: <1B90BE69-8DA4-4565-A0C3-E0E576E6881B@stufft.io> References: <20130603124836.23a83360@anarchist> <1B90BE69-8DA4-4565-A0C3-E0E576E6881B@stufft.io> Message-ID: <20130603130700.4482ae2a@anarchist> On Jun 03, 2013, at 02:17 PM, Donald Stufft wrote: >I'd actually prefer for Linux to not use the bundled certs when installed >from a package manager because it should use the system certs, but people >can't depend on certs being there if they are only there on linux. I think we agree on that. >Adding them into Python means people _can_ depend on them being there, and >Windows and other systems without system integrators to modify it to use the >system store will still get certs and Ubuntu can make it just work(?). Again, I think PEP 431 provides a pretty good model for how this should be done. Maybe it's worth factoring out this specific part of PEP 431 into an informational PEP? >This would probably (eventually) make the bundling of certificates better >too. > >Meaning that once it's been in long enough people are willing to depend on >it, they won't need to bundle their own certs and ubuntu/debian can just >modify the one location instead of needing to modify it for every package >that does it. Can we do the same for the JavaScript libraries? :) -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From solipsis at pitrou.net Mon Jun 3 20:34:08 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 3 Jun 2013 20:34:08 +0200 Subject: [Python-Dev] Validating SSL By Default (aka Including a Cert Bundle in CPython) References: <20130603123636.7a44dd9c@anarchist> Message-ID: <20130603203408.2c800172@fsol> On Mon, 3 Jun 2013 14:12:34 -0400 Donald Stufft wrote: > > I worry with the current situation people will just use TLS connections without realizing it's not being verified and thinking they are "safe". Yet there's quite a visible warning in the docs: http://docs.python.org/dev/library/urllib.request.html Regards Antoine. From ethan at stoneleaf.us Mon Jun 3 20:15:51 2013 From: ethan at stoneleaf.us (Ethan Furman) Date: Mon, 03 Jun 2013 11:15:51 -0700 Subject: [Python-Dev] Validating SSL By Default (aka Including a Cert Bundle in CPython) In-Reply-To: <6A511D87-832C-4686-83D0-642C05D8B60B@stufft.io> References: <20130603092852.2A6532500B9@webabinitio.net> <20130603115130.299b2504@fsol> <95CF5916-B6AD-4DF0-80FF-12D3393FED32@stufft.io> <51ACC9CB.1090407@stoneleaf.us> <6A511D87-832C-4686-83D0-642C05D8B60B@stufft.io> Message-ID: <51ACDD57.10808@stoneleaf.us> On 06/03/2013 10:55 AM, Donald Stufft wrote: > On Jun 3, 2013, at 12:52 PM, Ethan Furman wrote: >> On 06/03/2013 09:43 AM, Donald Stufft wrote: >>> On Jun 3, 2013, at 5:51 AM, Antoine Pitrou wrote: >>>> >>>> The problem with a "slightly outdated" CA store is that it can be a >>>> security risk. >>> >>> Tracking the Mozilla store isn't difficult. New additions can be ignored for currently released Pythons so we'd just >>> need to watch them for blacklisting certs and roll that into a security update. >> >> Personally, I'm not interested in waiting six months for an update. And why can't I have the new additions? >> >> Seems to me a better solution is to have routines that can query and update at will (meaning the app has to call >> them), as well as having the bundle (black lists as well as new additions) in the regular updates. > > Personally I view that as a nice to have but if an app is willing to call hem they can easily depend on certifi and just > replace the default certificates. The goal here is to get secure by default without any work on the part of the > developers. If a developer knows to update the certs they can know to use certifi. Let's pretend I want to do something with https. I'm pretty much a complete novice in that area, but I do know about these certificate things and that I should use them. (Which will probably happen sooner rather than later. ;) From my POV, having the bundled certs in the stdlib is great, but also having something in the stdlib to query for and download updates would also be great. I just looked at certifi and the only thing it has on there is `where` which tells me where the certs are on the system. This on only half the problem. If we're going to solve it, why not solve the whole problem? -- ~Ethan~ From donald at stufft.io Mon Jun 3 21:06:56 2013 From: donald at stufft.io (Donald Stufft) Date: Mon, 3 Jun 2013 15:06:56 -0400 Subject: [Python-Dev] Validating SSL By Default (aka Including a Cert Bundle in CPython) In-Reply-To: <20130603130700.4482ae2a@anarchist> References: <20130603124836.23a83360@anarchist> <1B90BE69-8DA4-4565-A0C3-E0E576E6881B@stufft.io> <20130603130700.4482ae2a@anarchist> Message-ID: <084A63CF-6466-4EBB-8554-3D3DB5C4B17F@stufft.io> On Jun 3, 2013, at 1:07 PM, Barry Warsaw wrote: > On Jun 03, 2013, at 02:17 PM, Donald Stufft wrote: > >> I'd actually prefer for Linux to not use the bundled certs when installed >> from a package manager because it should use the system certs, but people >> can't depend on certs being there if they are only there on linux. > > I think we agree on that. > >> Adding them into Python means people _can_ depend on them being there, and >> Windows and other systems without system integrators to modify it to use the >> system store will still get certs and Ubuntu can make it just work(?). > > Again, I think PEP 431 provides a pretty good model for how this should be > done. Maybe it's worth factoring out this specific part of PEP 431 into an > informational PEP? Looks fine to me minus the not updating in security releases (but that's just a difference in the type of data). > >> This would probably (eventually) make the bundling of certificates better >> too. >> >> Meaning that once it's been in long enough people are willing to depend on >> it, they won't need to bundle their own certs and ubuntu/debian can just >> modify the one location instead of needing to modify it for every package >> that does it. > > Can we do the same for the JavaScript libraries? :) > > -Barry ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 841 bytes Desc: Message signed with OpenPGP using GPGMail URL: From brett at python.org Mon Jun 3 21:20:27 2013 From: brett at python.org (Brett Cannon) Date: Mon, 3 Jun 2013 15:20:27 -0400 Subject: [Python-Dev] Validating SSL By Default (aka Including a Cert Bundle in CPython) In-Reply-To: <20130603130407.43d1cd0d@anarchist> References: <0583C689-56BE-4F21-9E16-0D06746295C9@stufft.io> <20130603125200.69de2e13@anarchist> <706472F8-31AB-4A6D-91EA-EA6E37F0BAC8@stufft.io> <20130603130407.43d1cd0d@anarchist> Message-ID: On Mon, Jun 3, 2013 at 1:04 PM, Barry Warsaw wrote: > On Jun 03, 2013, at 02:21 PM, Donald Stufft wrote: > > >The other additional comment I'd like to throw in here is that if we don't > >bundle SSL certs I think we should still verify by default (which means > HTTPS > >urls will throw an error by default if we can't locate a certificate > store) > >because I think the risk to people unknowingly thinking that their HTTPS > urls > >are protected are significant enough that this "error" shouldn't be > silent by > >default. > > +1, especially if we ensure that the APIs are available to not verify, as > is > currently the case with urlopen(). I don't think people will want to do > that > in production, but it will be useful for testing (e.g. guess how I found > issues 17977 :). > +1 from me as well. Whether we bundle or simply provide a command to download the certs I think making this default is the bare-minimum, especially if setting nothing more than cadefault=True is all that is needed to get this behaviour since that's backwards-compatible to Python 3.3. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ethan at stoneleaf.us Mon Jun 3 20:56:45 2013 From: ethan at stoneleaf.us (Ethan Furman) Date: Mon, 03 Jun 2013 11:56:45 -0700 Subject: [Python-Dev] Validating SSL By Default (aka Including a Cert Bundle in CPython) In-Reply-To: <20130603203408.2c800172@fsol> References: <20130603123636.7a44dd9c@anarchist> <20130603203408.2c800172@fsol> Message-ID: <51ACE6ED.7000200@stoneleaf.us> On 06/03/2013 11:34 AM, Antoine Pitrou wrote: > On Mon, 3 Jun 2013 14:12:34 -0400 > Donald Stufft wrote: >> >> I worry with the current situation people will just use TLS connections without realizing it's not being verified and thinking they are "safe". > > Yet there's quite a visible warning in the docs: > http://docs.python.org/dev/library/urllib.request.html As has been been mentioned elsewhere, at the very least we should change cadefault to True for secure-type connections. The dangerous/unexpected behavior should not be the default. -- ~Ethan~ From solipsis at pitrou.net Mon Jun 3 21:52:44 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 3 Jun 2013 21:52:44 +0200 Subject: [Python-Dev] Validating SSL By Default (aka Including a Cert Bundle in CPython) References: <20130603123636.7a44dd9c@anarchist> <20130603203408.2c800172@fsol> <51ACE6ED.7000200@stoneleaf.us> Message-ID: <20130603215244.576896b2@fsol> On Mon, 03 Jun 2013 11:56:45 -0700 Ethan Furman wrote: > On 06/03/2013 11:34 AM, Antoine Pitrou wrote: > > On Mon, 3 Jun 2013 14:12:34 -0400 > > Donald Stufft wrote: > >> > >> I worry with the current situation people will just use TLS connections without realizing it's not being verified and thinking they are "safe". > > > > Yet there's quite a visible warning in the docs: > > http://docs.python.org/dev/library/urllib.request.html > > As has been been mentioned elsewhere, at the very least we should change cadefault to True for secure-type connections. > The dangerous/unexpected behavior should not be the default. cadefault=True will probably be fail if the system certs are not properly configured in OpenSSL, e.g. under Windows or with a hand-made OpenSSL build. And, because of the way the OpenSSL API works, there's no way of knowing if it is the case or not: http://docs.python.org/3.4/library/ssl.html#ssl.SSLContext.set_default_verify_paths While I'm not saying we shouldn't try to verify certs by default, doing so is a little more delicate than "setting cadefault to True", unfortunately. Bundling our own CA cert store is an option, assuming Donald or someone else wants to take responsibility for it. Regards Antoine. From rosuav at gmail.com Mon Jun 3 22:05:01 2013 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 4 Jun 2013 06:05:01 +1000 Subject: [Python-Dev] Validating SSL By Default (aka Including a Cert Bundle in CPython) In-Reply-To: <51ACE6ED.7000200@stoneleaf.us> References: <20130603123636.7a44dd9c@anarchist> <20130603203408.2c800172@fsol> <51ACE6ED.7000200@stoneleaf.us> Message-ID: On Tue, Jun 4, 2013 at 4:56 AM, Ethan Furman wrote: > On 06/03/2013 11:34 AM, Antoine Pitrou wrote: >> >> On Mon, 3 Jun 2013 14:12:34 -0400 >> Donald Stufft wrote: >>> >>> >>> I worry with the current situation people will just use TLS connections >>> without realizing it's not being verified and thinking they are "safe". >> >> >> Yet there's quite a visible warning in the docs: >> http://docs.python.org/dev/library/urllib.request.html > > > As has been been mentioned elsewhere, at the very least we should change > cadefault to True for secure-type connections. The dangerous/unexpected > behavior should not be the default. +1 for having the default be safe, but this will have to be very loudly announced ("when migrating from 3.3 to 3.4, TLS connections will cease to work if blah blah"). Some legit sites with proper certificates still manage to muck something up administratively (developer.quicksales.com.au has a cert from RapidSSL but doesn't bundle the intermediates, and I've told their devs about it, but all I can do is disable cert checking). This will break code in ways that will surprise people greatly. But I'd still rather the default be True. ChrisA From dmalcolm at redhat.com Mon Jun 3 22:07:05 2013 From: dmalcolm at redhat.com (David Malcolm) Date: Mon, 03 Jun 2013 16:07:05 -0400 Subject: [Python-Dev] Validating SSL By Default (aka Including a Cert Bundle in CPython) In-Reply-To: <20130603124836.23a83360@anarchist> References: <20130603124836.23a83360@anarchist> Message-ID: <1370290025.22768.24.camel@surprise> On Mon, 2013-06-03 at 12:48 -0400, Barry Warsaw wrote: > On Jun 03, 2013, at 09:05 AM, Ben Darnell wrote: > > >The data is analogous to the time zone database (PEP 431) in that it may > >need to be updated independently of Python's own release schedule, so we > >may want to use similar techniques to manage both. Also see certifi ( > >https://pypi.python.org/pypi/certifi), which is a copy of the Mozilla list > >in a pip-installable form. > > Right, this is very much analogous, except with the additional twist that > out-of-date certificates can pose a significant security risk. > > I'm fairly certain that Debian and Ubuntu would explicitly not use any > certificates shipped with Python, for two main reasons: 1) our security teams > already manage the certificate store distro-wide and we want to make sure that > one update fixes everything; 2) we don't want to duplicate code in multiple > packages[1]. Fedora/RHEL are in a similar position; I expect we'd rip out the bundled certs in our builds shortly after unzipping the tarball, and use a system-wide cert store (I "rm -rf" bundled libraries in our builds, to make sure we're not using them). [...snip...] From christian at python.org Mon Jun 3 22:19:26 2013 From: christian at python.org (Christian Heimes) Date: Mon, 03 Jun 2013 22:19:26 +0200 Subject: [Python-Dev] Validating SSL By Default (aka Including a Cert Bundle in CPython) In-Reply-To: <20130603215244.576896b2@fsol> References: <20130603123636.7a44dd9c@anarchist> <20130603203408.2c800172@fsol> <51ACE6ED.7000200@stoneleaf.us> <20130603215244.576896b2@fsol> Message-ID: <51ACFA4E.6050501@python.org> Am 03.06.2013 21:52, schrieb Antoine Pitrou: > cadefault=True will probably be fail if the system certs are not > properly configured in OpenSSL, e.g. under Windows or with a hand-made > OpenSSL build. > And, because of the way the OpenSSL API works, there's no way of > knowing if it is the case or not: > http://docs.python.org/3.4/library/ssl.html#ssl.SSLContext.set_default_verify_paths I only see an issue for uncommon Linux distributions and exotic Unices. For Windows an interface to crypt32 API solves the CA issue as shown in my wincertstore module. It gives the user the same SSL experience as Internet Explorer. Most Linux and BSD-ish operating systems have SSL certs at some standard location. https://bitbucket.org/pypa/setuptools/src/6de3186fdfd9f5b543380e9aca2d48976cfc38cd/setuptools/ssl_support.py?at=default#cl-15 lists a couple of standard locations. Under which conditions do we need to ship a CA cert file? Christian From donald at stufft.io Mon Jun 3 22:22:41 2013 From: donald at stufft.io (Donald Stufft) Date: Mon, 3 Jun 2013 16:22:41 -0400 Subject: [Python-Dev] Validating SSL By Default (aka Including a Cert Bundle in CPython) In-Reply-To: <51ACFA4E.6050501@python.org> References: <20130603123636.7a44dd9c@anarchist> <20130603203408.2c800172@fsol> <51ACE6ED.7000200@stoneleaf.us> <20130603215244.576896b2@fsol> <51ACFA4E.6050501@python.org> Message-ID: <699D9AB5-26E0-4CCE-89C5-8E92C3305BDE@stufft.io> On Jun 3, 2013, at 4:19 PM, Christian Heimes wrote: > Am 03.06.2013 21:52, schrieb Antoine Pitrou: >> cadefault=True will probably be fail if the system certs are not >> properly configured in OpenSSL, e.g. under Windows or with a hand-made >> OpenSSL build. >> And, because of the way the OpenSSL API works, there's no way of >> knowing if it is the case or not: >> http://docs.python.org/3.4/library/ssl.html#ssl.SSLContext.set_default_verify_paths > > I only see an issue for uncommon Linux distributions and exotic Unices. > > For Windows an interface to crypt32 API solves the CA issue as shown in > my wincertstore module. It gives the user the same SSL experience as > Internet Explorer. > > Most Linux and BSD-ish operating systems have SSL certs at some standard > location. > https://bitbucket.org/pypa/setuptools/src/6de3186fdfd9f5b543380e9aca2d48976cfc38cd/setuptools/ssl_support.py?at=default#cl-15 > lists a couple of standard locations. > > Under which conditions do we need to ship a CA cert file? > > Christian > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/donald%40stufft.io What about OSX? ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 841 bytes Desc: Message signed with OpenPGP using GPGMail URL: From nad at acm.org Mon Jun 3 23:24:59 2013 From: nad at acm.org (Ned Deily) Date: Mon, 03 Jun 2013 14:24:59 -0700 Subject: [Python-Dev] Validating SSL By Default (aka Including a Cert Bundle in CPython) References: <20130603123636.7a44dd9c@anarchist> <20130603203408.2c800172@fsol> <51ACE6ED.7000200@stoneleaf.us> <20130603215244.576896b2@fsol> <51ACFA4E.6050501@python.org> <699D9AB5-26E0-4CCE-89C5-8E92C3305BDE@stufft.io> Message-ID: In article <699D9AB5-26E0-4CCE-89C5-8E92C3305BDE at stufft.io>, Donald Stufft wrote: > On Jun 3, 2013, at 4:19 PM, Christian Heimes wrote: > > Most Linux and BSD-ish operating systems have SSL certs at some standard > > location. > > https://bitbucket.org/pypa/setuptools/src/6de3186fdfd9f5b543380e9aca2d48976c > > fc38cd/setuptools/ssl_support.py?at=default#cl-15 > > lists a couple of standard locations. > > Under which conditions do we need to ship a CA cert file? > What about OSX? OS X has an actively managed set of CA certs, by Apple and with provision for users to tailor their security policies (add/modify/delete certs and their policies via either a gui or cli). For python.org OS X installers, we currently depend on the system-provided versions of libssl and libcrypto which use the system/user cert infrastructure. But Apple has deprecated the OpenSSL-compatible APIs and we have an open issue (Issue17128) to move to supplying our own copy of OpenSSL libs. One hangup has been what to do about the certs. One solution would be to use a Python distribution of them as discussed here. Another, probably better for users, solution is to continue to use the existing system/user cert infrastructure; recently, we've seen a new approach to that which we need to look into: http://bugs.python.org/issue17128#msg189244. Third-party distributors of Python on OS X may use other solutions; for example, MacPorts already ships and links with its own version of OpenSSL. -- Ned Deily, nad at acm.org From p.f.moore at gmail.com Mon Jun 3 23:31:40 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Mon, 3 Jun 2013 22:31:40 +0100 Subject: [Python-Dev] Validating SSL By Default (aka Including a Cert Bundle in CPython) In-Reply-To: References: <20130603123636.7a44dd9c@anarchist> <20130603203408.2c800172@fsol> <51ACE6ED.7000200@stoneleaf.us> Message-ID: On 3 June 2013 21:05, Chris Angelico wrote: > +1 for having the default be safe, but this will have to be very > loudly announced ("when migrating from 3.3 to 3.4, TLS connections > will cease to work if blah blah"). > +1 on the default being safe, certainly. But with the proviso that the same code should work in 3.3 and 3.4, with no user impact (other than that the connection is safer, but that's not user-visible unless there's an attack :-)) In other words, that "will cease to work" clause should not exist - but see below... > Some legit sites with proper > certificates still manage to muck something up administratively > (developer.quicksales.com.au has a cert from RapidSSL but doesn't > bundle the intermediates, and I've told their devs about it, but all I > can do is disable cert checking). This will break code in ways that > will surprise people greatly. But I'd still rather the default be > True. > I'm happy if the "will cease to work" clause only says "some sites with broken security configurations may stop working" with a clear explanation that it is *their* fault, not Python's. I'd also expect that the same sites would fail in browsers - if not, we should also be able to make them work (or face cries of "well, Internet Explorer/Firefox doesn't have a problem with my site, why does Python?"). Also, we should consider the issue for application users. Suppose I'm using a Python application that downloads something from the web. I upgrade to 3.4, and the app stops working because of a "will cease to work" case. As an end user, how can I get the app working again? Having to patch the sources isn't an option, and reverting to 3.3 provokes the reaction "Python broke my app". Summary - I'm +1 as long as either the "will cease to work" list is empty, or we have a *very* good story for (legitimate) sites which do cease to work. Paul. -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Mon Jun 3 23:41:19 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 3 Jun 2013 23:41:19 +0200 Subject: [Python-Dev] Validating SSL By Default (aka Including a Cert Bundle in CPython) References: <20130603123636.7a44dd9c@anarchist> <20130603203408.2c800172@fsol> <51ACE6ED.7000200@stoneleaf.us> Message-ID: <20130603234119.7cdeed4b@fsol> On Mon, 3 Jun 2013 22:31:40 +0100 Paul Moore wrote: > > > Some legit sites with proper > > certificates still manage to muck something up administratively > > (developer.quicksales.com.au has a cert from RapidSSL but doesn't > > bundle the intermediates, and I've told their devs about it, but all I > > can do is disable cert checking). This will break code in ways that > > will surprise people greatly. But I'd still rather the default be > > True. > > > > I'm happy if the "will cease to work" clause only says "some sites with > broken security configurations may stop working" with a clear explanation > that it is *their* fault, not Python's. I'd also expect that the same sites > would fail in browsers - if not, we should also be able to make them work > (or face cries of "well, Internet Explorer/Firefox doesn't have a problem > with my site, why does Python?"). Keep in mind that not every HTTPS service is a Web site that is meant to be readable with a browser. Some are Web services, possibly internal, possibly without a domain name (and, therefore, probably a non-matching certificate subject name). Regards Antoine. From rosuav at gmail.com Mon Jun 3 23:46:18 2013 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 4 Jun 2013 07:46:18 +1000 Subject: [Python-Dev] Validating SSL By Default (aka Including a Cert Bundle in CPython) In-Reply-To: References: <20130603123636.7a44dd9c@anarchist> <20130603203408.2c800172@fsol> <51ACE6ED.7000200@stoneleaf.us> Message-ID: On Tue, Jun 4, 2013 at 7:31 AM, Paul Moore wrote: > > On 3 June 2013 21:05, Chris Angelico wrote: >> Some legit sites with proper >> certificates still manage to muck something up administratively >> (developer.quicksales.com.au has a cert from RapidSSL but doesn't >> bundle the intermediates, and I've told their devs about it, but all I >> can do is disable cert checking). This will break code in ways that >> will surprise people greatly. But I'd still rather the default be >> True. > > I'm happy if the "will cease to work" clause only says "some sites with > broken security configurations may stop working" with a clear explanation > that it is *their* fault, not Python's. I'd also expect that the same sites > would fail in browsers - if not, we should also be able to make them work > (or face cries of "well, Internet Explorer/Firefox doesn't have a problem > with my site, why does Python?"). In this specific case, I get no warning from Google Chrome on either Windows or Linux, but I do get a warning from Iceweasel on the same Linux (it's Debian Wheezy, fwiw). So there probably will be cries of "well, Chrome doesn't etc etc". > Also, we should consider the issue for application users. Suppose I'm using > a Python application that downloads something from the web. I upgrade to > 3.4, and the app stops working because of a "will cease to work" case. As an > end user, how can I get the app working again? Having to patch the sources > isn't an option, and reverting to 3.3 provokes the reaction "Python broke my > app". Right. The instance I'm citing comes from what I'm doing at work; I had to disable certificate checking in order to get a non-interactive application to work. If I'd never thought to make sure cert checking was happening, the app could easily have been unexpectedly broken by this change. But you're absolutely right. It's a failing at the remote end, or... On Tue, Jun 4, 2013 at 7:41 AM, Antoine Pitrou wrote: > Keep in mind that not every HTTPS service is a Web site that is meant > to be readable with a browser. Some are Web services, possibly internal, > possibly without a domain name (and, therefore, probably a non-matching > certificate subject name). ... this. This isn't a problem, as checking can easily be disabled, but I'd be inclined to recommend that this one be solved without changing code by explicitly importing that certificate. That would resolve everything, but would require administrative action. ChrisA From donald at stufft.io Mon Jun 3 23:46:52 2013 From: donald at stufft.io (Donald Stufft) Date: Mon, 3 Jun 2013 17:46:52 -0400 Subject: [Python-Dev] Validating SSL By Default (aka Including a Cert Bundle in CPython) In-Reply-To: References: <20130603123636.7a44dd9c@anarchist> <20130603203408.2c800172@fsol> <51ACE6ED.7000200@stoneleaf.us> Message-ID: On Jun 3, 2013, at 5:31 PM, Paul Moore wrote: > > On 3 June 2013 21:05, Chris Angelico wrote: > +1 for having the default be safe, but this will have to be very > loudly announced ("when migrating from 3.3 to 3.4, TLS connections > will cease to work if blah blah"). > > +1 on the default being safe, certainly. But with the proviso that the same code should work in 3.3 and 3.4, with no user impact (other than that the connection is safer, but that's not user-visible unless there's an attack :-)) If we bundle certs that will be the case sans connections where it doesn't validate. > > In other words, that "will cease to work" clause should not exist - but see below... > > Some legit sites with proper > certificates still manage to muck something up administratively > (developer.quicksales.com.au has a cert from RapidSSL but doesn't > bundle the intermediates, and I've told their devs about it, but all I > can do is disable cert checking). This will break code in ways that > will surprise people greatly. But I'd still rather the default be > True. > > I'm happy if the "will cease to work" clause only says "some sites with broken security configurations may stop working" with a clear explanation that it is *their* fault, not Python's. I'd also expect that the same sites would fail in browsers - if not, we should also be able to make them work (or face cries of "well, Internet Explorer/Firefox doesn't have a problem with my site, why does Python?"). Browsers tend to download intermediate certs while I don't think Python does. > > Also, we should consider the issue for application users. Suppose I'm using a Python application that downloads something from the web. I upgrade to 3.4, and the app stops working because of a "will cease to work" case. As an end user, how can I get the app working again? Having to patch the sources isn't an option, and reverting to 3.3 provokes the reaction "Python broke my app". Supply a SSL vert using the environment variable? > > Summary - I'm +1 as long as either the "will cease to work" list is empty, or we have a *very* good story for (legitimate) sites which do cease to work. > > Paul. > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/donald%40stufft.io ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 841 bytes Desc: Message signed with OpenPGP using GPGMail URL: From donald at stufft.io Mon Jun 3 23:47:31 2013 From: donald at stufft.io (Donald Stufft) Date: Mon, 3 Jun 2013 17:47:31 -0400 Subject: [Python-Dev] Validating SSL By Default (aka Including a Cert Bundle in CPython) In-Reply-To: <20130603234119.7cdeed4b@fsol> References: <20130603123636.7a44dd9c@anarchist> <20130603203408.2c800172@fsol> <51ACE6ED.7000200@stoneleaf.us> <20130603234119.7cdeed4b@fsol> Message-ID: On Jun 3, 2013, at 5:41 PM, Antoine Pitrou wrote: > On Mon, 3 Jun 2013 22:31:40 +0100 > Paul Moore wrote: >> >>> Some legit sites with proper >>> certificates still manage to muck something up administratively >>> (developer.quicksales.com.au has a cert from RapidSSL but doesn't >>> bundle the intermediates, and I've told their devs about it, but all I >>> can do is disable cert checking). This will break code in ways that >>> will surprise people greatly. But I'd still rather the default be >>> True. >>> >> >> I'm happy if the "will cease to work" clause only says "some sites with >> broken security configurations may stop working" with a clear explanation >> that it is *their* fault, not Python's. I'd also expect that the same sites >> would fail in browsers - if not, we should also be able to make them work >> (or face cries of "well, Internet Explorer/Firefox doesn't have a problem >> with my site, why does Python?"). > > Keep in mind that not every HTTPS service is a Web site that is meant > to be readable with a browser. Some are Web services, possibly internal, > possibly without a domain name (and, therefore, probably a non-matching > certificate subject name). They should need to explicitly opt in to disabling the checks that allow that to work. > > Regards > > Antoine. > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/donald%40stufft.io ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 841 bytes Desc: Message signed with OpenPGP using GPGMail URL: From solipsis at pitrou.net Mon Jun 3 23:51:30 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 3 Jun 2013 23:51:30 +0200 Subject: [Python-Dev] Validating SSL By Default (aka Including a Cert Bundle in CPython) In-Reply-To: References: <20130603123636.7a44dd9c@anarchist> <20130603203408.2c800172@fsol> <51ACE6ED.7000200@stoneleaf.us> <20130603234119.7cdeed4b@fsol> Message-ID: <20130603235130.3a65fd02@fsol> On Mon, 3 Jun 2013 17:47:31 -0400 Donald Stufft wrote: > > On Jun 3, 2013, at 5:41 PM, Antoine Pitrou wrote: > > > On Mon, 3 Jun 2013 22:31:40 +0100 > > Paul Moore wrote: > >> > >>> Some legit sites with proper > >>> certificates still manage to muck something up administratively > >>> (developer.quicksales.com.au has a cert from RapidSSL but doesn't > >>> bundle the intermediates, and I've told their devs about it, but all I > >>> can do is disable cert checking). This will break code in ways that > >>> will surprise people greatly. But I'd still rather the default be > >>> True. > >>> > >> > >> I'm happy if the "will cease to work" clause only says "some sites with > >> broken security configurations may stop working" with a clear explanation > >> that it is *their* fault, not Python's. I'd also expect that the same sites > >> would fail in browsers - if not, we should also be able to make them work > >> (or face cries of "well, Internet Explorer/Firefox doesn't have a problem > >> with my site, why does Python?"). > > > > Keep in mind that not every HTTPS service is a Web site that is meant > > to be readable with a browser. Some are Web services, possibly internal, > > possibly without a domain name (and, therefore, probably a non-matching > > certificate subject name). > > They should need to explicitly opt in to disabling the checks that allow that to work. Obviously, which means compatibility is broken with existing code. Regards Antoine. From donald at stufft.io Mon Jun 3 23:56:48 2013 From: donald at stufft.io (Donald Stufft) Date: Mon, 3 Jun 2013 17:56:48 -0400 Subject: [Python-Dev] Validating SSL By Default (aka Including a Cert Bundle in CPython) In-Reply-To: <20130603235130.3a65fd02@fsol> References: <20130603123636.7a44dd9c@anarchist> <20130603203408.2c800172@fsol> <51ACE6ED.7000200@stoneleaf.us> <20130603234119.7cdeed4b@fsol> <20130603235130.3a65fd02@fsol> Message-ID: On Jun 3, 2013, at 5:51 PM, Antoine Pitrou wrote: > On Mon, 3 Jun 2013 17:47:31 -0400 > Donald Stufft wrote: >> >> On Jun 3, 2013, at 5:41 PM, Antoine Pitrou wrote: >> >>> On Mon, 3 Jun 2013 22:31:40 +0100 >>> Paul Moore wrote: >>>> >>>>> Some legit sites with proper >>>>> certificates still manage to muck something up administratively >>>>> (developer.quicksales.com.au has a cert from RapidSSL but doesn't >>>>> bundle the intermediates, and I've told their devs about it, but all I >>>>> can do is disable cert checking). This will break code in ways that >>>>> will surprise people greatly. But I'd still rather the default be >>>>> True. >>>>> >>>> >>>> I'm happy if the "will cease to work" clause only says "some sites with >>>> broken security configurations may stop working" with a clear explanation >>>> that it is *their* fault, not Python's. I'd also expect that the same sites >>>> would fail in browsers - if not, we should also be able to make them work >>>> (or face cries of "well, Internet Explorer/Firefox doesn't have a problem >>>> with my site, why does Python?"). >>> >>> Keep in mind that not every HTTPS service is a Web site that is meant >>> to be readable with a browser. Some are Web services, possibly internal, >>> possibly without a domain name (and, therefore, probably a non-matching >>> certificate subject name). >> >> They should need to explicitly opt in to disabling the checks that allow that to work. > > Obviously, which means compatibility is broken with existing code. > > Regards > > Antoine. > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/donald%40stufft.io Yes in that case compat will be broken and they'll need to either specify a cert that can be used to validate the connection or disable the protection. I think it's very surprising for people that they need to *enable* secure mode when most tools have that on by default. It's handing users a security foot gun, and like most things security related "broken" is silent until it's too late. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 841 bytes Desc: Message signed with OpenPGP using GPGMail URL: From p.f.moore at gmail.com Tue Jun 4 00:01:22 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Mon, 3 Jun 2013 23:01:22 +0100 Subject: [Python-Dev] Validating SSL By Default (aka Including a Cert Bundle in CPython) In-Reply-To: References: <20130603123636.7a44dd9c@anarchist> <20130603203408.2c800172@fsol> <51ACE6ED.7000200@stoneleaf.us> Message-ID: On 3 June 2013 22:46, Donald Stufft wrote: > Also, we should consider the issue for application users. Suppose I'm > using a Python application that downloads something from the web. I upgrade > to 3.4, and the app stops working because of a "will cease to work" case. > As an end user, how can I get the app working again? Having to patch the > sources isn't an option, and reverting to 3.3 provokes the reaction "Python > broke my app". > > > Supply a SSL vert using the environment variable? > Hmm, that would be acceptable, I guess, for many users (although Windows users are somewhat more environment-variable-averse than Unix users). But you say that as if it's obvious how to do that (or where to get a cert). It's certainly not obvious to me, and if "it works in Internet Explorer", I'd have no idea where to get a cert from that I could use in an environment variable. Just to repeat - I agree with the principle, but in many environments, users are pretty much clueless about security and actively object to being educated "for their own safety". These users will disable all security quite happily if it stops the internal app failing, and will blame Python for "making things harder" and breaking backward compatibility. On the other hand, I suspect we're talking about an extremely low percentage of cases, so let's not blow the issue out of proportion :-) Paul -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Tue Jun 4 01:15:36 2013 From: donald at stufft.io (Donald Stufft) Date: Mon, 3 Jun 2013 19:15:36 -0400 Subject: [Python-Dev] Validating SSL By Default (aka Including a Cert Bundle in CPython) In-Reply-To: References: <20130603123636.7a44dd9c@anarchist> <20130603203408.2c800172@fsol> <51ACE6ED.7000200@stoneleaf.us> Message-ID: <2593421E-0413-492C-8C28-A3982B7B960E@stufft.io> On Jun 3, 2013, at 6:01 PM, Paul Moore wrote: > > On 3 June 2013 22:46, Donald Stufft wrote: >> Also, we should consider the issue for application users. Suppose I'm using a Python application that downloads something from the web. I upgrade to 3.4, and the app stops working because of a "will cease to work" case. As an end user, how can I get the app working again? Having to patch the sources isn't an option, and reverting to 3.3 provokes the reaction "Python broke my app". > > Supply a SSL vert using the environment variable? > > Hmm, that would be acceptable, I guess, for many users (although Windows users are somewhat more environment-variable-averse than Unix users). But you say that as if it's obvious how to do that (or where to get a cert). It's certainly not obvious to me, and if "it works in Internet Explorer", I'd have no idea where to get a cert from that I could use in an environment variable. > > Just to repeat - I agree with the principle, but in many environments, users are pretty much clueless about security and actively object to being educated "for their own safety". These users will disable all security quite happily if it stops the internal app failing, and will blame Python for "making things harder" and breaking backward compatibility. On the other hand, I suspect we're talking about an extremely low percentage of cases, so let's not blow the issue out of proportion :-) > > Paul Let me make myself a bit more clear because maybe I haven't been. If someone is relying on the insecure by default behavior and Python 3.4 gets this change, their code will break and their options will be: Client side: - Disable Verification, essentially reverting back to the old behavior. - Setup their Python install to be able to validate that certificate - This may be importing it into their OS certificate store - This may be using an env var to specify - This may be passing it directly somewhere Server side: - Make the certificate valid - Purchase a Valid by default certificate - Serve immedaries etc. Generally any of these will be completely valid options, even disabling the checks. The idea behind my proposal is that people generally only use TLS for a reason and that reason is they want to protect against the kinds of attacks that TLS protects against. You really only get the bulk of those protections if you validate the certificates. So for the vast bulk of people validation is the option they want. It also happens that validation on by default is "secure" by default which aims a fairly large foot gun away from peoples feet. To expand on something else, I'm completely willing to do whatever I can to hep make this happen. I don't really know what the process is. If I need to write a PEP I'll do that. If I need to volunteer to manage the certificates, I'll do that. If we need docs or tooling to help the people who this change might break I'll do that too. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 841 bytes Desc: Message signed with OpenPGP using GPGMail URL: From fijall at gmail.com Tue Jun 4 03:56:55 2013 From: fijall at gmail.com (Maciej Fijalkowski) Date: Tue, 4 Jun 2013 09:56:55 +0800 Subject: [Python-Dev] PEP 442: Safe object finalization In-Reply-To: <20130518163315.6a21a0cd@fsol> References: <20130518105910.1eecfa5f@fsol> <20130518154552.2a879bf6@fsol> <20130518163315.6a21a0cd@fsol> Message-ID: On Sat, May 18, 2013 at 10:33 PM, Antoine Pitrou wrote: > On Sat, 18 May 2013 16:22:55 +0200 > Armin Rigo wrote: >> Hi Antoine, >> >> On Sat, May 18, 2013 at 3:45 PM, Antoine Pitrou wrote: >> >> How is this done? I don't see a clear way to determine it by looking >> >> only at the objects in the CI, given that arbitrary modifications of >> >> the object graph may have occurred. >> > >> > The same way a generation is traversed, but restricted to the CI. >> > >> > First the gc_refs field of each CI object is initialized to its >> > ob_refcnt (again). >> > >> > Then, tp_traverse is called on each CI object, and each visited >> > CI object has its gc_refs decremented. This substracts CI-internal >> > references from the gc_refs fields. >> > >> > At the end of the traversal, if all CI objects have their gc_refs equal >> > to 0, then the CI has no external reference to it and can be cleared. >> > If at least one CI object has non-zero gc_refs, the CI cannot be >> > cleared. >> >> Ok, indeed. Then you really should call finalizers only once: in case >> one of the finalizers in a cycle did a trivial change like I >> described, the algorithm above will conservatively assume the cycle >> should be kept alive. At the next GC collection we must not call the >> finalizer again, because it's likely to just do a similar trivial >> change. > > Well, the finalizer will only be called if the resurrected object is > dereferenced again; otherwise the object won't be considered by the GC. > So, this will only happen if someone keeps trying to destroy a > resurrected object. > > Calling finalizers only once is fine with me, but it would be a change > in behaviour; I don't know if it may break existing code. > > (for example, say someone is using __del__ to manage a freelist) > > Regards > > Antoine. PyPy already ever calls finalizers once. If you resurrect an object, it'll be alive, but it's finalizer will not be called again. We discussed a few changes a while ago and we decided (I think even debated on python-dev) than even such behavior is correct: * you have a reference cycle A <-> B, C references A. C references itself. * you collect the stuff. We do topological order, so C finalizer is called first (they're only undefined inside a cycle) * then A and B finalizers are called in undefined order, even if C finalizer would resurrect C. * no more finalizers for those objects are called I'm not sure if it's cool for CPython or not to do such changes Cheers, fijal From tanqing.zjj at taobao.com Tue Jun 4 04:43:59 2013 From: tanqing.zjj at taobao.com (=?gb2312?B?zL3H5w==?=) Date: Tue, 4 Jun 2013 10:43:59 +0800 Subject: [Python-Dev] Let's get rid of unbound methods Message-ID: <000301ce60cd$5d8d5a30$18a80e90$@zjj@taobao.com> -------------- next part -------------- An HTML attachment was scrubbed... URL: From raymond.hettinger at gmail.com Tue Jun 4 06:44:05 2013 From: raymond.hettinger at gmail.com (Raymond Hettinger) Date: Mon, 3 Jun 2013 21:44:05 -0700 Subject: [Python-Dev] Putting the Mac Build in the Apple App Store Message-ID: <770E1B33-4174-4880-9DA1-4E15E1818808@gmail.com> Does anyone know what we would need to do to get Python in the Apple application store as a free App? The default security settings on OS X 10.8 block the installation of the DMG (or any software downloaded outside the app store). A number of my students are having difficulty getting around it will help. If we were in the app store, installation and upgrade would be a piece of cake. Raymond From fijall at gmail.com Tue Jun 4 08:00:45 2013 From: fijall at gmail.com (Maciej Fijalkowski) Date: Tue, 4 Jun 2013 14:00:45 +0800 Subject: [Python-Dev] Putting the Mac Build in the Apple App Store In-Reply-To: <770E1B33-4174-4880-9DA1-4E15E1818808@gmail.com> References: <770E1B33-4174-4880-9DA1-4E15E1818808@gmail.com> Message-ID: On Tue, Jun 4, 2013 at 12:44 PM, Raymond Hettinger wrote: > Does anyone know what we would need to do to get Python in the Apple application store as a free App? > > The default security settings on OS X 10.8 block the installation of the DMG (or any software downloaded outside the app store). A number of my students are having difficulty getting around it will help. > > If we were in the app store, installation and upgrade would be a piece of cake. > > > Raymond > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/fijall%40gmail.com doesn't it break apple's rules? generally you should not be able to download content from the internet and run it (like say javascript in the browser). I suppose python interpreter counts as that but hey, who knows. cheers, fijal From clp2 at rebertia.com Tue Jun 4 08:49:02 2013 From: clp2 at rebertia.com (Chris Rebert) Date: Mon, 3 Jun 2013 23:49:02 -0700 Subject: [Python-Dev] Putting the Mac Build in the Apple App Store In-Reply-To: <770E1B33-4174-4880-9DA1-4E15E1818808@gmail.com> References: <770E1B33-4174-4880-9DA1-4E15E1818808@gmail.com> Message-ID: On Jun 3, 2013 9:45 PM, "Raymond Hettinger" wrote: > > Does anyone know what we would need to do to get Python in the Apple application store as a free App? > > The default security settings on OS X 10.8 block the installation of the DMG (or any software downloaded outside the app store). A number of my students are having difficulty getting around it will help. > > If we were in the app store, installation and upgrade would be a piece of cake. If you need another option, you could try installing Python via Homebrew ( http://brew.sh ). Homebrew itself is also easy to install. (Although admittedly I haven't tried Homebrew's Python, but my experience with their other packages has been good.) Cheers, Chris -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Tue Jun 4 09:55:03 2013 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 4 Jun 2013 17:55:03 +1000 Subject: [Python-Dev] Validating SSL By Default (aka Including a Cert Bundle in CPython) In-Reply-To: <2593421E-0413-492C-8C28-A3982B7B960E@stufft.io> References: <20130603123636.7a44dd9c@anarchist> <20130603203408.2c800172@fsol> <51ACE6ED.7000200@stoneleaf.us> <2593421E-0413-492C-8C28-A3982B7B960E@stufft.io> Message-ID: On Tue, Jun 4, 2013 at 9:15 AM, Donald Stufft wrote: > Generally any of these will be completely valid options, even disabling the > checks. The idea behind my proposal is that people generally only use TLS > for a reason and that reason is they want to protect against the kinds of > attacks that TLS protects against. You really only get the bulk of those > protections if you validate the certificates. So for the vast bulk of people > validation is the option they want. It also happens that validation on by > default is "secure" by default which aims a fairly large foot gun away from > peoples feet. Yep. I absolutely agree, and if Python had been like this ever since 3.0, there would be no issue whatsoever. The only problem is that it's going to trip up people who have otherwise-working code that gets broken by a change of Python version. The problem isn't even in their code, it's in the remote end. I support making the change, but all of the solutions you propose require being aware of the problem. Somehow people will need to be told, preferably before it breaks something, that the default has changed. ChrisA From nad at acm.org Tue Jun 4 10:19:36 2013 From: nad at acm.org (Ned Deily) Date: Tue, 04 Jun 2013 01:19:36 -0700 Subject: [Python-Dev] Putting the Mac Build in the Apple App Store References: <770E1B33-4174-4880-9DA1-4E15E1818808@gmail.com> Message-ID: In article <770E1B33-4174-4880-9DA1-4E15E1818808 at gmail.com>, Raymond Hettinger wrote: > Does anyone know what we would need to do to get Python in the Apple > application store as a free App? > > The default security settings on OS X 10.8 block the installation of the DMG > (or any software downloaded outside the app store). A number of my students > are having difficulty getting around it will help. The ReadMe file included in the current Python OS X installer DMGs explains exactly how to get around the problem: "If you are attempting to install on an OS X 10.8 system, you may see a message that Python can't be installed because it is from an unidentified developer. This is because this Python installer package is not yet compatible with the Gatekeeper security feature introduced in OS X 10.8. To allow Python to be installed, you can override the Gatekeeper policy for this install. In the Finder, instead of double-clicking, control-click or right click the "Python" installer package icon. Then select "Open using ... Installer" from the contextual menu that appears." There are a number of other installers that haven't been updated yet for Gatekeeper, including the currant ActiveState Tcl installers. The same technique works for all of them. This is basic knowledge needed to use OS X 10.8. Even so, the Python installer will get updated. > If we were in the app store, installation and upgrade would be a piece of > cake. Sorry but, under the current Apple app store rules, Python would never be accepted. Among other things, it would need to be "sandboxed" meaning it could only use a very limited set of OS APIs and would have very limited access to the user's file system. Such a Python would not be very usable. -- Ned Deily, nad at acm.org From ronaldoussoren at mac.com Tue Jun 4 12:15:38 2013 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Tue, 4 Jun 2013 12:15:38 +0200 Subject: [Python-Dev] Putting the Mac Build in the Apple App Store In-Reply-To: <770E1B33-4174-4880-9DA1-4E15E1818808@gmail.com> References: <770E1B33-4174-4880-9DA1-4E15E1818808@gmail.com> Message-ID: <50E1F5FD-D245-4998-9B48-3415A19B38FA@mac.com> On 4 Jun, 2013, at 6:44, Raymond Hettinger wrote: > Does anyone know what we would need to do to get Python in the Apple application store as a free App? > > The default security settings on OS X 10.8 block the installation of the DMG (or any software downloaded outside the app store). A number of my students are having difficulty getting around it will help. > > If we were in the app store, installation and upgrade would be a piece of cake. A problem with the app store is that the Python installation should then be an app (for example IDLE.app), and that the application must be sandboxed. The latter is showstopper, as scripts run with the interpreter would be sandboxed as well and hence couldn't access most of the system. A better solution for the problem with OSX 10.8's security settings it sign the installer with a developer ID. It can then be opened by double clicking because the app is provided by an "identified developer". A problem with signing the installer is that this requires changes to the installer, we're currently using an ancient installer format that cannot be signed. That should be changed some time in the future anyway and signing the installer could be a good reason to work on that. BTW. There is a workaround that makes it possible to install without signing the installer: right-click on the installer and select "open" (instead of double clicking the installer). The system will then give a scary warning, but will allow installation anyway. Ronald > > > Raymond > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/ronaldoussoren%40mac.com From ronaldoussoren at mac.com Tue Jun 4 12:20:35 2013 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Tue, 4 Jun 2013 12:20:35 +0200 Subject: [Python-Dev] Validating SSL By Default (aka Including a Cert Bundle in CPython) In-Reply-To: References: Message-ID: <25E65E5D-B8A2-4D7C-B866-AFC4CEF915BD@mac.com> On 3 Jun, 2013, at 7:58, Benjamin Peterson wrote: > 2013/6/2 Donald Stufft : >> As of right now, as far as I can tell, Python does not validate HTTPS >> certificates by default. As far as I can tell this is because there is no >> guaranteed certificates available. >> >> So I would like to propose that CPython adopt the Mozilla SSL certificate >> list and include it in core, and switch over the API's so that they verify >> HTTPS by default. > > +1 > >> >> Ideally this would take the shape of attempting to locate the system >> certificate store if possible, and if that doesn't work falling back to the >> bundled certificates. That way the various Linux distros can easily have >> their copies of Python depend soley on their built in certs, but Windows, >> OSX, Source compiles etc will all still have a fallback value. > > My preference would be actually be for the included certificates file > to be used by default. This would provide a consistent experience > across platforms. We could provide options to look for system cert > repositories if desired. I'd prefer to use the system CA list when that's available. I've had to hunt down the CA list for a number of application when a custom CA for internal use and that's not fun, using the system list is much friendlier to users. Ronald From steve at pearwood.info Tue Jun 4 13:57:44 2013 From: steve at pearwood.info (Steven D'Aprano) Date: Tue, 04 Jun 2013 21:57:44 +1000 Subject: [Python-Dev] Let's get rid of unbound methods In-Reply-To: <000301ce60cd$5d8d5a30$18a80e90$@zjj@taobao.com> References: <000301ce60cd$5d8d5a30$18a80e90$@zjj@taobao.com> Message-ID: <51ADD638.9020001@pearwood.info> On 04/06/13 12:43, ?? wrote: Your email appears to be blank, except for a large chunk of HTML code. Did you have something specific to say other than the subject line? As for unbound methods, Guido's time machine strikes again. They're already gone in Python 3. py> class X: ... def spam(self): ... pass ... py> X.spam py> X().spam > -- Steven From secalert at redhat.com Tue Jun 4 13:44:08 2013 From: secalert at redhat.com (Red Hat Security Response Team) Date: Tue, 4 Jun 2013 07:44:08 -0400 Subject: [Python-Dev] [engineering.redhat.com #216557] [Fwd: Validating SSL By Default (aka Including a Cert Bundle in CPython)] In-Reply-To: <1370289371.22768.17.camel@surprise> References: <1370289371.22768.17.camel@surprise> Message-ID: Hello David, Donald, David, thank you for sharing these intentions with us. On Mon Jun 03 15:56:09 2013, dmalcolm at redhat.com wrote: > > As of right now, as far as I can tell, Python does not validate HTTPS certificates by default. As far as I can tell this > is because there is no guaranteed certificates available. > > So I would like to propose that CPython adopt the Mozilla SSL certificate list and include it in core, and switch over > the API's so that they verify HTTPS by default. Donald, we would only welcome this enhancement / proposal. To mention some examples - urllib2 and httplib modules: http://docs.python.org/2/library/urllib2.html http://docs.python.org/2/library/httplib.html are documented upstream not to perform SSL certificate verification by default (and due this fact there has been couple of CVE identifiers assigned in the past for applications that incorrectly assumed certificates would be validated when using these modules). So any enhancement, which can upstream done in this area, would be only welcome. > This is what most people are going to expect when using a https url > (Especially after learning that Python 2.x doesn't verify TLS, but Python 3.x "does"). > > Ideally this would take the shape of attempting to locate the system certificate store if possible, and if that doesn't > work falling back to the bundled certificates. That way the various Linux distros can easily have their copies of Python > depend soley on their built in certs, but Windows, OSX, Source compiles etc will all still have a fallback value. AFAWCT that proposal looks reasonable. Thank you && Regards, Jan. -- Jan iankko Lieskovsky / Red Hat Security Response Team > > ----------------- > Donald Stufft > PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA > > From carlosnepomuceno at outlook.com Wed Jun 5 01:53:21 2013 From: carlosnepomuceno at outlook.com (Carlos Nepomuceno) Date: Wed, 5 Jun 2013 02:53:21 +0300 Subject: [Python-Dev] Do you consider Python a 4GL? Why (not)? Message-ID: Do you consider Python a 4GL? Why (not)? -------------- next part -------------- An HTML attachment was scrubbed... URL: From drsalists at gmail.com Wed Jun 5 03:17:33 2013 From: drsalists at gmail.com (Dan Stromberg) Date: Tue, 4 Jun 2013 18:17:33 -0700 Subject: [Python-Dev] Do you consider Python a 4GL? Why (not)? In-Reply-To: References: Message-ID: On Tue, Jun 4, 2013 at 4:53 PM, Carlos Nepomuceno < carlosnepomuceno at outlook.com> wrote: > Do you consider Python a 4GL? Why (not)? > By the wikipedia definition of 4GL and 5GL, I'd say Python is neither. And it's not a VHLL either, again according to the wikipedia definition. But IMO it is too high level to be a traditional 3GL too. Perhaps "Scripting language" is the best general category we have that Python fits into. But I hope not. -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Wed Jun 5 03:21:50 2013 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 5 Jun 2013 11:21:50 +1000 Subject: [Python-Dev] Do you consider Python a 4GL? Why (not)? In-Reply-To: References: Message-ID: On Wed, Jun 5, 2013 at 11:17 AM, Dan Stromberg wrote: > > On Tue, Jun 4, 2013 at 4:53 PM, Carlos Nepomuceno > wrote: >> >> Do you consider Python a 4GL? Why (not)? > > > By the wikipedia definition of 4GL and 5GL, I'd say Python is neither. And > it's not a VHLL either, again according to the wikipedia definition. But > IMO it is too high level to be a traditional 3GL too. > > Perhaps "Scripting language" is the best general category we have that > Python fits into. But I hope not. I think this would be a fun discussion for python-list - just what IS Python? Here's some stuff on programming vs scripting: http://www.perl.com/pub/2007/12/06/soto-11.html (including an awesome line about actors and audience) ChrisA From guido at python.org Wed Jun 5 03:32:18 2013 From: guido at python.org (Guido van Rossum) Date: Tue, 4 Jun 2013 18:32:18 -0700 Subject: [Python-Dev] PEP 443 Accepted Message-ID: ?ukasz, Congratulations! I've accepted PEP 443. I've already marked it as Accepted in the repo. I've also applied some very minor edits in order to make the text flow a little better in a few places. I think this is a great PEP -- it's simple, doesn't overreach, and you've managed the bikeshedding admirably. Thank you for your great contribution to Python! -- --Guido van Rossum (python.org/~guido) From python at mrabarnett.plus.com Wed Jun 5 03:47:21 2013 From: python at mrabarnett.plus.com (MRAB) Date: Wed, 05 Jun 2013 02:47:21 +0100 Subject: [Python-Dev] PyPI upload error Message-ID: <51AE98A9.60902@mrabarnett.plus.com> I've just tried to upload to PyPI using setup.py and got this error: Upload failed (503): backend write error Can anyone tell me what it means? From ethan at stoneleaf.us Wed Jun 5 03:54:12 2013 From: ethan at stoneleaf.us (Ethan Furman) Date: Tue, 04 Jun 2013 18:54:12 -0700 Subject: [Python-Dev] PEP 443 Accepted In-Reply-To: References: Message-ID: <51AE9A44.70209@stoneleaf.us> Congratulations, ?ukasz! -- ~Ethan~ From donald at stufft.io Wed Jun 5 04:07:50 2013 From: donald at stufft.io (Donald Stufft) Date: Tue, 4 Jun 2013 22:07:50 -0400 Subject: [Python-Dev] PyPI upload error In-Reply-To: <51AE98A9.60902@mrabarnett.plus.com> References: <51AE98A9.60902@mrabarnett.plus.com> Message-ID: <2C13BE21-1130-4BAA-A2C6-73FC2BEEFBB9@stufft.io> On Jun 4, 2013, at 9:47 PM, MRAB wrote: > I've just tried to upload to PyPI using setup.py and got this error: > > Upload failed (503): backend write error > > Can anyone tell me what it means? > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/donald%40stufft.io This is probably more appropriate for distutils-sig, but does it happen every time? or did it just happen once? ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 841 bytes Desc: Message signed with OpenPGP using GPGMail URL: From python at mrabarnett.plus.com Wed Jun 5 04:25:36 2013 From: python at mrabarnett.plus.com (MRAB) Date: Wed, 05 Jun 2013 03:25:36 +0100 Subject: [Python-Dev] PyPI upload error In-Reply-To: <2C13BE21-1130-4BAA-A2C6-73FC2BEEFBB9@stufft.io> References: <51AE98A9.60902@mrabarnett.plus.com> <2C13BE21-1130-4BAA-A2C6-73FC2BEEFBB9@stufft.io> Message-ID: <51AEA1A0.3020302@mrabarnett.plus.com> On 05/06/2013 03:07, Donald Stufft wrote: > > On Jun 4, 2013, at 9:47 PM, MRAB > wrote: > >> I've just tried to upload to PyPI using setup.py and got this error: >> >> Upload failed (503): backend write error >> >> Can anyone tell me what it means? >> > This is probably more appropriate for distutils-sig, but does it happen > every time? or did it just happen once? > It happened a couple of times, but worked some time later. I'll try to remember to ask distutils-sig next time. From p.f.moore at gmail.com Wed Jun 5 09:07:54 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 5 Jun 2013 08:07:54 +0100 Subject: [Python-Dev] PEP 443 Accepted In-Reply-To: References: Message-ID: On 5 June 2013 02:32, Guido van Rossum wrote: > ?ukasz, > > Congratulations! I've accepted PEP 443. I've already marked it as > Accepted in the repo. I've also applied some very minor edits in order > to make the text flow a little better in a few places. I think this is > a great PEP -- it's simple, doesn't overreach, and you've managed the > bikeshedding admirably. Thank you for your great contribution to > Python! > Excellent news! Congratulations. Paul -------------- next part -------------- An HTML attachment was scrubbed... URL: From markus at unterwaditzer.net Wed Jun 5 09:29:14 2013 From: markus at unterwaditzer.net (Markus Unterwaditzer) Date: Wed, 05 Jun 2013 09:29:14 +0200 Subject: [Python-Dev] PEP 443 Accepted In-Reply-To: References: Message-ID: As somebody who missed the discussion about it and right now took a quick look at the PEP, i ask myself how subclasses are handled, as i don't see anything about it in the PEP, just support for ABCs. E.g if issubclass(Apple, Fruit) And i call a function which has registered an implementation for the Fruits type with an object of type Apple, is this implementation used? I assume so, but as said, i don't see it mentioned anywhere. -- Markus (from phone) Guido van Rossum wrote: >?ukasz, > >Congratulations! I've accepted PEP 443. I've already marked it as >Accepted in the repo. I've also applied some very minor edits in order >to make the text flow a little better in a few places. I think this is >a great PEP -- it's simple, doesn't overreach, and you've managed the >bikeshedding admirably. Thank you for your great contribution to >Python! > >-- >--Guido van Rossum (python.org/~guido) >_______________________________________________ >Python-Dev mailing list >Python-Dev at python.org >http://mail.python.org/mailman/listinfo/python-dev >Unsubscribe: >http://mail.python.org/mailman/options/python-dev/markus%40unterwaditzer.net From ncoghlan at gmail.com Wed Jun 5 09:47:35 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 5 Jun 2013 17:47:35 +1000 Subject: [Python-Dev] PEP 443 Accepted In-Reply-To: References: Message-ID: On Wed, Jun 5, 2013 at 5:29 PM, Markus Unterwaditzer wrote: > As somebody who missed the discussion about it and right now took a quick look at the PEP, i ask myself how subclasses are handled, as i don't see anything about it in the PEP, just support for ABCs. > > E.g if > > issubclass(Apple, Fruit) > > And i call a function which has registered an implementation for the Fruits type with an object of type Apple, is this implementation used? I assume so, but as said, i don't see it mentioned anywhere. That's covered by walking the MRO the same way method dispatch does it, so it didn't really get discussed much (the question never came up). The ABC handling is described explicitly as the generic dispatch implementation that the PEP was based on didn't handle them, and because it required a fair bit more thought than just walking the MRO. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From lukasz at langa.pl Wed Jun 5 09:56:08 2013 From: lukasz at langa.pl (=?utf-8?Q?=C5=81ukasz_Langa?=) Date: Wed, 5 Jun 2013 09:56:08 +0200 Subject: [Python-Dev] PEP 443 Accepted In-Reply-To: References: Message-ID: On 5 cze 2013, at 09:29, Markus Unterwaditzer wrote: > As somebody who missed the discussion about it and right now took a quick look at the PEP, i ask myself how subclasses are handled, as i don't see anything about it in the PEP, just support for ABCs. > > E.g if > > issubclass(Apple, Fruit) > > And i call a function which has registered an implementation for the Fruits type with an object of type Apple, is this implementation used? I assume so, but as said, i don't see it mentioned anywhere. Yes, this is the supported behaviour. The PEP briefly explains that "Where there is no registered implementation for a specific type, its method resolution order is used to find a more generic implementation." As Nick noted, the reason the description is so brief is that this works just like calling methods. -- Best regards, ?ukasz Langa WWW: http://lukasz.langa.pl/ Twitter: @llanga IRC: ambv on #python-dev -------------- next part -------------- An HTML attachment was scrubbed... URL: From koobs.freebsd at gmail.com Wed Jun 5 11:22:44 2013 From: koobs.freebsd at gmail.com (Kubilay Kocak) Date: Wed, 05 Jun 2013 19:22:44 +1000 Subject: [Python-Dev] New FreeBSD 10.0-CURRENT buildbot In-Reply-To: <51AAD44B.4090802@FreeBSD.org> References: <51AAD44B.4090802@FreeBSD.org> Message-ID: <51AF0364.7060504@FreeBSD.org> On 2/06/2013 3:12 PM, Kubilay Kocak wrote: > Afternoon (UTC+10), > > I'd like to request a new user/pass for a FreeBSD 10.0-CURRENT VM guest > I've just setup as a dedicated buildbot slave to complement my other > koobs-freebsd slaves. > > Also, with this and future additions to the FreeBSD buildslave set in > mind, I think it might also be prudent for a rename to take place: > > koobs-freebsd9-amd64 > koobs-freebsd9-amd64-clang (CC=clang) > koobs-freebsd10-amd64 (clang is default here) > > Convention being: koobs-freebsdX[Y]-arch[-config] (Happy for feedback here) > > If there are any permutations or additions you'd like to specifically > see for FreeBSD to increase coverage just let me know (Hint: I have a > PandaBoard arm board here i'm getting up and running) > > I have ZFS & DTrace to work with among other things, and the long term > plan is to have FreeBSD buildbots running the full gamut of versions, > from -RELEASE through -STABLE to HEAD branches. > > I'm on #python-dev IRC if someone needs to discuss. > > -- > Regards, > > Koobs > Ping :) From brett at python.org Wed Jun 5 16:31:04 2013 From: brett at python.org (Brett Cannon) Date: Wed, 5 Jun 2013 10:31:04 -0400 Subject: [Python-Dev] [Python-checkins] cpython: Add reference implementation for PEP 443 In-Reply-To: <3bQQxs6YX7zRbZ@mail.python.org> References: <3bQQxs6YX7zRbZ@mail.python.org> Message-ID: Any chance you could move your definitions for "generic function" and "single dispatch" to the glossary and just link to them here? On Wed, Jun 5, 2013 at 6:20 AM, lukasz.langa wrote: > http://hg.python.org/cpython/rev/dfcb64f51f7b > changeset: 84039:dfcb64f51f7b > user: ?ukasz Langa > date: Wed Jun 05 12:20:24 2013 +0200 > summary: > Add reference implementation for PEP 443 > > PEP accepted: > http://mail.python.org/pipermail/python-dev/2013-June/126734.html > > files: > Doc/library/functools.rst | 110 +++++++ > Lib/functools.py | 128 ++++++++- > Lib/pkgutil.py | 52 +--- > Lib/test/test_functools.py | 374 ++++++++++++++++++++++++- > Misc/NEWS | 3 + > Modules/Setup.dist | 2 +- > 6 files changed, 614 insertions(+), 55 deletions(-) > > > diff --git a/Doc/library/functools.rst b/Doc/library/functools.rst > --- a/Doc/library/functools.rst > +++ b/Doc/library/functools.rst > @@ -6,6 +6,7 @@ > .. moduleauthor:: Peter Harris > .. moduleauthor:: Raymond Hettinger > .. moduleauthor:: Nick Coghlan > +.. moduleauthor:: ?ukasz Langa > .. sectionauthor:: Peter Harris > > **Source code:** :source:`Lib/functools.py` > @@ -186,6 +187,115 @@ > *sequence* contains only one item, the first item is returned. > > > +.. decorator:: singledispatch(default) > + > + Transforms a function into a single-dispatch generic function. A > **generic > + function** is composed of multiple functions implementing the same > operation > + for different types. Which implementation should be used during a call > is > + determined by the dispatch algorithm. When the implementation is > chosen > + based on the type of a single argument, this is known as **single > + dispatch**. > + > + To define a generic function, decorate it with the ``@singledispatch`` > + decorator. Note that the dispatch happens on the type of the first > argument, > + create your function accordingly:: > + > + >>> from functools import singledispatch > + >>> @singledispatch > + ... def fun(arg, verbose=False): > + ... if verbose: > + ... print("Let me just say,", end=" ") > + ... print(arg) > + > + To add overloaded implementations to the function, use the > :func:`register` > + attribute of the generic function. It is a decorator, taking a type > + parameter and decorating a function implementing the operation for that > + type:: > + > + >>> @fun.register(int) > + ... def _(arg, verbose=False): > + ... if verbose: > + ... print("Strength in numbers, eh?", end=" ") > + ... print(arg) > + ... > + >>> @fun.register(list) > + ... def _(arg, verbose=False): > + ... if verbose: > + ... print("Enumerate this:") > + ... for i, elem in enumerate(arg): > + ... print(i, elem) > + > + To enable registering lambdas and pre-existing functions, the > + :func:`register` attribute can be used in a functional form:: > + > + >>> def nothing(arg, verbose=False): > + ... print("Nothing.") > + ... > + >>> fun.register(type(None), nothing) > + > + The :func:`register` attribute returns the undecorated function which > + enables decorator stacking, pickling, as well as creating unit tests > for > + each variant independently:: > + > + >>> @fun.register(float) > + ... @fun.register(Decimal) > + ... def fun_num(arg, verbose=False): > + ... if verbose: > + ... print("Half of your number:", end=" ") > + ... print(arg / 2) > + ... > + >>> fun_num is fun > + False > + > + When called, the generic function dispatches on the type of the first > + argument:: > + > + >>> fun("Hello, world.") > + Hello, world. > + >>> fun("test.", verbose=True) > + Let me just say, test. > + >>> fun(42, verbose=True) > + Strength in numbers, eh? 42 > + >>> fun(['spam', 'spam', 'eggs', 'spam'], verbose=True) > + Enumerate this: > + 0 spam > + 1 spam > + 2 eggs > + 3 spam > + >>> fun(None) > + Nothing. > + >>> fun(1.23) > + 0.615 > + > + Where there is no registered implementation for a specific type, its > + method resolution order is used to find a more generic implementation. > + The original function decorated with ``@singledispatch`` is registered > + for the base ``object`` type, which means it is used if no better > + implementation is found. > + > + To check which implementation will the generic function choose for > + a given type, use the ``dispatch()`` attribute:: > + > + >>> fun.dispatch(float) > + > + >>> fun.dispatch(dict) # note: default implementation > + > + > + To access all registered implementations, use the read-only > ``registry`` > + attribute:: > + > + >>> fun.registry.keys() > + dict_keys([, , , > + , , > + ]) > + >>> fun.registry[float] > + > + >>> fun.registry[object] > + > + > + .. versionadded:: 3.4 > + > + > .. function:: update_wrapper(wrapper, wrapped, > assigned=WRAPPER_ASSIGNMENTS, updated=WRAPPER_UPDATES) > > Update a *wrapper* function to look like the *wrapped* function. The > optional > diff --git a/Lib/functools.py b/Lib/functools.py > --- a/Lib/functools.py > +++ b/Lib/functools.py > @@ -3,19 +3,24 @@ > # Python module wrapper for _functools C module > # to allow utilities written in Python to be added > # to the functools module. > -# Written by Nick Coghlan > -# and Raymond Hettinger > -# Copyright (C) 2006-2010 Python Software Foundation. > +# Written by Nick Coghlan , > +# Raymond Hettinger , > +# and ?ukasz Langa . > +# Copyright (C) 2006-2013 Python Software Foundation. > # See C source code for _functools credits/copyright > > __all__ = ['update_wrapper', 'wraps', 'WRAPPER_ASSIGNMENTS', > 'WRAPPER_UPDATES', > - 'total_ordering', 'cmp_to_key', 'lru_cache', 'reduce', > 'partial'] > + 'total_ordering', 'cmp_to_key', 'lru_cache', 'reduce', > 'partial', > + 'singledispatch'] > > try: > from _functools import reduce > except ImportError: > pass > +from abc import get_cache_token > from collections import namedtuple > +from types import MappingProxyType > +from weakref import WeakKeyDictionary > try: > from _thread import RLock > except: > @@ -354,3 +359,118 @@ > return update_wrapper(wrapper, user_function) > > return decorating_function > + > + > > +################################################################################ > +### singledispatch() - single-dispatch generic function decorator > > +################################################################################ > + > +def _compose_mro(cls, haystack): > + """Calculates the MRO for a given class `cls`, including relevant > abstract > + base classes from `haystack`. > + > + """ > + bases = set(cls.__mro__) > + mro = list(cls.__mro__) > + for needle in haystack: > + if (needle in bases or not hasattr(needle, '__mro__') > + or not issubclass(cls, needle)): > + continue # either present in the __mro__ already or > unrelated > + for index, base in enumerate(mro): > + if not issubclass(base, needle): > + break > + if base in bases and not issubclass(needle, base): > + # Conflict resolution: put classes present in __mro__ and > their > + # subclasses first. See test_mro_conflicts() in > test_functools.py > + # for examples. > + index += 1 > + mro.insert(index, needle) > + return mro > + > +def _find_impl(cls, registry): > + """Returns the best matching implementation for the given class `cls` > in > + `registry`. Where there is no registered implementation for a specific > + type, its method resolution order is used to find a more generic > + implementation. > + > + Note: if `registry` does not contain an implementation for the base > + `object` type, this function may return None. > + > + """ > + mro = _compose_mro(cls, registry.keys()) > + match = None > + for t in mro: > + if match is not None: > + # If `match` is an ABC but there is another unrelated, equally > + # matching ABC. Refuse the temptation to guess. > + if (t in registry and not issubclass(match, t) > + and match not in cls.__mro__): > + raise RuntimeError("Ambiguous dispatch: {} or {}".format( > + match, t)) > + break > + if t in registry: > + match = t > + return registry.get(match) > + > +def singledispatch(func): > + """Single-dispatch generic function decorator. > + > + Transforms a function into a generic function, which can have > different > + behaviours depending upon the type of its first argument. The > decorated > + function acts as the default implementation, and additional > + implementations can be registered using the 'register()' attribute of > + the generic function. > + > + """ > + registry = {} > + dispatch_cache = WeakKeyDictionary() > + cache_token = None > + > + def dispatch(typ): > + """generic_func.dispatch(type) -> > + > + Runs the dispatch algorithm to return the best available > implementation > + for the given `type` registered on `generic_func`. > + > + """ > + nonlocal cache_token > + if cache_token is not None: > + current_token = get_cache_token() > + if cache_token != current_token: > + dispatch_cache.clear() > + cache_token = current_token > + try: > + impl = dispatch_cache[typ] > + except KeyError: > + try: > + impl = registry[typ] > + except KeyError: > + impl = _find_impl(typ, registry) > + dispatch_cache[typ] = impl > + return impl > + > + def register(typ, func=None): > + """generic_func.register(type, func) -> func > + > + Registers a new implementation for the given `type` on a > `generic_func`. > + > + """ > + nonlocal cache_token > + if func is None: > + return lambda f: register(typ, f) > + registry[typ] = func > + if cache_token is None and hasattr(typ, '__abstractmethods__'): > + cache_token = get_cache_token() > + dispatch_cache.clear() > + return func > + > + def wrapper(*args, **kw): > + return dispatch(args[0].__class__)(*args, **kw) > + > + registry[object] = func > + wrapper.register = register > + wrapper.dispatch = dispatch > + wrapper.registry = MappingProxyType(registry) > + wrapper._clear_cache = dispatch_cache.clear > + update_wrapper(wrapper, func) > + return wrapper > diff --git a/Lib/pkgutil.py b/Lib/pkgutil.py > --- a/Lib/pkgutil.py > +++ b/Lib/pkgutil.py > @@ -1,12 +1,13 @@ > """Utilities to support packages.""" > > +from functools import singledispatch as simplegeneric > +import imp > +import importlib > import os > +import os.path > import sys > -import importlib > -import imp > -import os.path > +from types import ModuleType > from warnings import warn > -from types import ModuleType > > __all__ = [ > 'get_importer', 'iter_importers', 'get_loader', 'find_loader', > @@ -27,46 +28,6 @@ > return marshal.load(stream) > > > -def simplegeneric(func): > - """Make a trivial single-dispatch generic function""" > - registry = {} > - def wrapper(*args, **kw): > - ob = args[0] > - try: > - cls = ob.__class__ > - except AttributeError: > - cls = type(ob) > - try: > - mro = cls.__mro__ > - except AttributeError: > - try: > - class cls(cls, object): > - pass > - mro = cls.__mro__[1:] > - except TypeError: > - mro = object, # must be an ExtensionClass or some such > :( > - for t in mro: > - if t in registry: > - return registry[t](*args, **kw) > - else: > - return func(*args, **kw) > - try: > - wrapper.__name__ = func.__name__ > - except (TypeError, AttributeError): > - pass # Python 2.3 doesn't allow functions to be renamed > - > - def register(typ, func=None): > - if func is None: > - return lambda f: register(typ, f) > - registry[typ] = func > - return func > - > - wrapper.__dict__ = func.__dict__ > - wrapper.__doc__ = func.__doc__ > - wrapper.register = register > - return wrapper > - > - > def walk_packages(path=None, prefix='', onerror=None): > """Yields (module_loader, name, ispkg) for all modules recursively > on path, or, if path is None, all accessible modules. > @@ -148,13 +109,12 @@ > yield i, name, ispkg > > > -#@simplegeneric > + at simplegeneric > def iter_importer_modules(importer, prefix=''): > if not hasattr(importer, 'iter_modules'): > return [] > return importer.iter_modules(prefix) > > -iter_importer_modules = simplegeneric(iter_importer_modules) > > # Implement a file walker for the normal importlib path hook > def _iter_file_finder_modules(importer, prefix=''): > diff --git a/Lib/test/test_functools.py b/Lib/test/test_functools.py > --- a/Lib/test/test_functools.py > +++ b/Lib/test/test_functools.py > @@ -1,24 +1,30 @@ > import collections > -import sys > -import unittest > -from test import support > -from weakref import proxy > +from itertools import permutations > import pickle > from random import choice > +import sys > +from test import support > +import unittest > +from weakref import proxy > > import functools > > py_functools = support.import_fresh_module('functools', > blocked=['_functools']) > c_functools = support.import_fresh_module('functools', > fresh=['_functools']) > > +decimal = support.import_fresh_module('decimal', fresh=['_decimal']) > + > + > def capture(*args, **kw): > """capture all positional and keyword arguments""" > return args, kw > > + > def signature(part): > """ return the signature of a partial object """ > return (part.func, part.args, part.keywords, part.__dict__) > > + > class TestPartial: > > def test_basic_examples(self): > @@ -138,6 +144,7 @@ > join = self.partial(''.join) > self.assertEqual(join(data), '0123456789') > > + > @unittest.skipUnless(c_functools, 'requires the C _functools module') > class TestPartialC(TestPartial, unittest.TestCase): > if c_functools: > @@ -194,18 +201,22 @@ > "new style getargs format but argument is not a tuple", > f.__setstate__, BadSequence()) > > + > class TestPartialPy(TestPartial, unittest.TestCase): > partial = staticmethod(py_functools.partial) > > + > if c_functools: > class PartialSubclass(c_functools.partial): > pass > > + > @unittest.skipUnless(c_functools, 'requires the C _functools module') > class TestPartialCSubclass(TestPartialC): > if c_functools: > partial = PartialSubclass > > + > class TestUpdateWrapper(unittest.TestCase): > > def check_wrapper(self, wrapper, wrapped, > @@ -312,6 +323,7 @@ > self.assertTrue(wrapper.__doc__.startswith('max(')) > self.assertEqual(wrapper.__annotations__, {}) > > + > class TestWraps(TestUpdateWrapper): > > def _default_update(self): > @@ -372,6 +384,7 @@ > self.assertEqual(wrapper.attr, 'This is a different test') > self.assertEqual(wrapper.dict_attr, f.dict_attr) > > + > class TestReduce(unittest.TestCase): > func = functools.reduce > > @@ -452,6 +465,7 @@ > d = {"one": 1, "two": 2, "three": 3} > self.assertEqual(self.func(add, d), "".join(d.keys())) > > + > class TestCmpToKey: > > def test_cmp_to_key(self): > @@ -534,14 +548,17 @@ > self.assertRaises(TypeError, hash, k) > self.assertNotIsInstance(k, collections.Hashable) > > + > @unittest.skipUnless(c_functools, 'requires the C _functools module') > class TestCmpToKeyC(TestCmpToKey, unittest.TestCase): > if c_functools: > cmp_to_key = c_functools.cmp_to_key > > + > class TestCmpToKeyPy(TestCmpToKey, unittest.TestCase): > cmp_to_key = staticmethod(py_functools.cmp_to_key) > > + > class TestTotalOrdering(unittest.TestCase): > > def test_total_ordering_lt(self): > @@ -642,6 +659,7 @@ > with self.assertRaises(TypeError): > TestTO(8) <= () > > + > class TestLRU(unittest.TestCase): > > def test_lru(self): > @@ -834,6 +852,353 @@ > DoubleEq(2)) # Verify the correct > return value > > > +class TestSingleDispatch(unittest.TestCase): > + def test_simple_overloads(self): > + @functools.singledispatch > + def g(obj): > + return "base" > + def g_int(i): > + return "integer" > + g.register(int, g_int) > + self.assertEqual(g("str"), "base") > + self.assertEqual(g(1), "integer") > + self.assertEqual(g([1,2,3]), "base") > + > + def test_mro(self): > + @functools.singledispatch > + def g(obj): > + return "base" > + class C: > + pass > + class D(C): > + pass > + def g_C(c): > + return "C" > + g.register(C, g_C) > + self.assertEqual(g(C()), "C") > + self.assertEqual(g(D()), "C") > + > + def test_classic_classes(self): > + @functools.singledispatch > + def g(obj): > + return "base" > + class C: > + pass > + class D(C): > + pass > + def g_C(c): > + return "C" > + g.register(C, g_C) > + self.assertEqual(g(C()), "C") > + self.assertEqual(g(D()), "C") > + > + def test_register_decorator(self): > + @functools.singledispatch > + def g(obj): > + return "base" > + @g.register(int) > + def g_int(i): > + return "int %s" % (i,) > + self.assertEqual(g(""), "base") > + self.assertEqual(g(12), "int 12") > + self.assertIs(g.dispatch(int), g_int) > + self.assertIs(g.dispatch(object), g.dispatch(str)) > + # Note: in the assert above this is not g. > + # @singledispatch returns the wrapper. > + > + def test_wrapping_attributes(self): > + @functools.singledispatch > + def g(obj): > + "Simple test" > + return "Test" > + self.assertEqual(g.__name__, "g") > + self.assertEqual(g.__doc__, "Simple test") > + > + @unittest.skipUnless(decimal, 'requires _decimal') > + @support.cpython_only > + def test_c_classes(self): > + @functools.singledispatch > + def g(obj): > + return "base" > + @g.register(decimal.DecimalException) > + def _(obj): > + return obj.args > + subn = decimal.Subnormal("Exponent < Emin") > + rnd = decimal.Rounded("Number got rounded") > + self.assertEqual(g(subn), ("Exponent < Emin",)) > + self.assertEqual(g(rnd), ("Number got rounded",)) > + @g.register(decimal.Subnormal) > + def _(obj): > + return "Too small to care." > + self.assertEqual(g(subn), "Too small to care.") > + self.assertEqual(g(rnd), ("Number got rounded",)) > + > + def test_compose_mro(self): > + c = collections > + mro = functools._compose_mro > + bases = [c.Sequence, c.MutableMapping, c.Mapping, c.Set] > + for haystack in permutations(bases): > + m = mro(dict, haystack) > + self.assertEqual(m, [dict, c.MutableMapping, c.Mapping, > object]) > + bases = [c.Container, c.Mapping, c.MutableMapping, c.OrderedDict] > + for haystack in permutations(bases): > + m = mro(c.ChainMap, haystack) > + self.assertEqual(m, [c.ChainMap, c.MutableMapping, c.Mapping, > + c.Sized, c.Iterable, c.Container, > object]) > + # Note: The MRO order below depends on haystack ordering. > + m = mro(c.defaultdict, [c.Sized, c.Container, str]) > + self.assertEqual(m, [c.defaultdict, dict, c.Container, c.Sized, > object]) > + m = mro(c.defaultdict, [c.Container, c.Sized, str]) > + self.assertEqual(m, [c.defaultdict, dict, c.Sized, c.Container, > object]) > + > + def test_register_abc(self): > + c = collections > + d = {"a": "b"} > + l = [1, 2, 3] > + s = {object(), None} > + f = frozenset(s) > + t = (1, 2, 3) > + @functools.singledispatch > + def g(obj): > + return "base" > + self.assertEqual(g(d), "base") > + self.assertEqual(g(l), "base") > + self.assertEqual(g(s), "base") > + self.assertEqual(g(f), "base") > + self.assertEqual(g(t), "base") > + g.register(c.Sized, lambda obj: "sized") > + self.assertEqual(g(d), "sized") > + self.assertEqual(g(l), "sized") > + self.assertEqual(g(s), "sized") > + self.assertEqual(g(f), "sized") > + self.assertEqual(g(t), "sized") > + g.register(c.MutableMapping, lambda obj: "mutablemapping") > + self.assertEqual(g(d), "mutablemapping") > + self.assertEqual(g(l), "sized") > + self.assertEqual(g(s), "sized") > + self.assertEqual(g(f), "sized") > + self.assertEqual(g(t), "sized") > + g.register(c.ChainMap, lambda obj: "chainmap") > + self.assertEqual(g(d), "mutablemapping") # irrelevant ABCs > registered > + self.assertEqual(g(l), "sized") > + self.assertEqual(g(s), "sized") > + self.assertEqual(g(f), "sized") > + self.assertEqual(g(t), "sized") > + g.register(c.MutableSequence, lambda obj: "mutablesequence") > + self.assertEqual(g(d), "mutablemapping") > + self.assertEqual(g(l), "mutablesequence") > + self.assertEqual(g(s), "sized") > + self.assertEqual(g(f), "sized") > + self.assertEqual(g(t), "sized") > + g.register(c.MutableSet, lambda obj: "mutableset") > + self.assertEqual(g(d), "mutablemapping") > + self.assertEqual(g(l), "mutablesequence") > + self.assertEqual(g(s), "mutableset") > + self.assertEqual(g(f), "sized") > + self.assertEqual(g(t), "sized") > + g.register(c.Mapping, lambda obj: "mapping") > + self.assertEqual(g(d), "mutablemapping") # not specific enough > + self.assertEqual(g(l), "mutablesequence") > + self.assertEqual(g(s), "mutableset") > + self.assertEqual(g(f), "sized") > + self.assertEqual(g(t), "sized") > + g.register(c.Sequence, lambda obj: "sequence") > + self.assertEqual(g(d), "mutablemapping") > + self.assertEqual(g(l), "mutablesequence") > + self.assertEqual(g(s), "mutableset") > + self.assertEqual(g(f), "sized") > + self.assertEqual(g(t), "sequence") > + g.register(c.Set, lambda obj: "set") > + self.assertEqual(g(d), "mutablemapping") > + self.assertEqual(g(l), "mutablesequence") > + self.assertEqual(g(s), "mutableset") > + self.assertEqual(g(f), "set") > + self.assertEqual(g(t), "sequence") > + g.register(dict, lambda obj: "dict") > + self.assertEqual(g(d), "dict") > + self.assertEqual(g(l), "mutablesequence") > + self.assertEqual(g(s), "mutableset") > + self.assertEqual(g(f), "set") > + self.assertEqual(g(t), "sequence") > + g.register(list, lambda obj: "list") > + self.assertEqual(g(d), "dict") > + self.assertEqual(g(l), "list") > + self.assertEqual(g(s), "mutableset") > + self.assertEqual(g(f), "set") > + self.assertEqual(g(t), "sequence") > + g.register(set, lambda obj: "concrete-set") > + self.assertEqual(g(d), "dict") > + self.assertEqual(g(l), "list") > + self.assertEqual(g(s), "concrete-set") > + self.assertEqual(g(f), "set") > + self.assertEqual(g(t), "sequence") > + g.register(frozenset, lambda obj: "frozen-set") > + self.assertEqual(g(d), "dict") > + self.assertEqual(g(l), "list") > + self.assertEqual(g(s), "concrete-set") > + self.assertEqual(g(f), "frozen-set") > + self.assertEqual(g(t), "sequence") > + g.register(tuple, lambda obj: "tuple") > + self.assertEqual(g(d), "dict") > + self.assertEqual(g(l), "list") > + self.assertEqual(g(s), "concrete-set") > + self.assertEqual(g(f), "frozen-set") > + self.assertEqual(g(t), "tuple") > + > + def test_mro_conflicts(self): > + c = collections > + > + @functools.singledispatch > + def g(arg): > + return "base" > + > + class O(c.Sized): > + def __len__(self): > + return 0 > + > + o = O() > + self.assertEqual(g(o), "base") > + g.register(c.Iterable, lambda arg: "iterable") > + g.register(c.Container, lambda arg: "container") > + g.register(c.Sized, lambda arg: "sized") > + g.register(c.Set, lambda arg: "set") > + self.assertEqual(g(o), "sized") > + c.Iterable.register(O) > + self.assertEqual(g(o), "sized") # because it's explicitly in > __mro__ > + c.Container.register(O) > + self.assertEqual(g(o), "sized") # see above: Sized is in __mro__ > + > + class P: > + pass > + > + p = P() > + self.assertEqual(g(p), "base") > + c.Iterable.register(P) > + self.assertEqual(g(p), "iterable") > + c.Container.register(P) > + with self.assertRaises(RuntimeError) as re: > + g(p) > + self.assertEqual( > + str(re), > + ("Ambiguous dispatch: > " > + "or "), > + ) > + > + class Q(c.Sized): > + def __len__(self): > + return 0 > + > + q = Q() > + self.assertEqual(g(q), "sized") > + c.Iterable.register(Q) > + self.assertEqual(g(q), "sized") # because it's explicitly in > __mro__ > + c.Set.register(Q) > + self.assertEqual(g(q), "set") # because c.Set is a subclass of > + # c.Sized which is explicitly in > + # __mro__ > + > + def test_cache_invalidation(self): > + from collections import UserDict > + class TracingDict(UserDict): > + def __init__(self, *args, **kwargs): > + super(TracingDict, self).__init__(*args, **kwargs) > + self.set_ops = [] > + self.get_ops = [] > + def __getitem__(self, key): > + result = self.data[key] > + self.get_ops.append(key) > + return result > + def __setitem__(self, key, value): > + self.set_ops.append(key) > + self.data[key] = value > + def clear(self): > + self.data.clear() > + _orig_wkd = functools.WeakKeyDictionary > + td = TracingDict() > + functools.WeakKeyDictionary = lambda: td > + c = collections > + @functools.singledispatch > + def g(arg): > + return "base" > + d = {} > + l = [] > + self.assertEqual(len(td), 0) > + self.assertEqual(g(d), "base") > + self.assertEqual(len(td), 1) > + self.assertEqual(td.get_ops, []) > + self.assertEqual(td.set_ops, [dict]) > + self.assertEqual(td.data[dict], g.registry[object]) > + self.assertEqual(g(l), "base") > + self.assertEqual(len(td), 2) > + self.assertEqual(td.get_ops, []) > + self.assertEqual(td.set_ops, [dict, list]) > + self.assertEqual(td.data[dict], g.registry[object]) > + self.assertEqual(td.data[list], g.registry[object]) > + self.assertEqual(td.data[dict], td.data[list]) > + self.assertEqual(g(l), "base") > + self.assertEqual(g(d), "base") > + self.assertEqual(td.get_ops, [list, dict]) > + self.assertEqual(td.set_ops, [dict, list]) > + g.register(list, lambda arg: "list") > + self.assertEqual(td.get_ops, [list, dict]) > + self.assertEqual(len(td), 0) > + self.assertEqual(g(d), "base") > + self.assertEqual(len(td), 1) > + self.assertEqual(td.get_ops, [list, dict]) > + self.assertEqual(td.set_ops, [dict, list, dict]) > + self.assertEqual(td.data[dict], > + functools._find_impl(dict, g.registry)) > + self.assertEqual(g(l), "list") > + self.assertEqual(len(td), 2) > + self.assertEqual(td.get_ops, [list, dict]) > + self.assertEqual(td.set_ops, [dict, list, dict, list]) > + self.assertEqual(td.data[list], > + functools._find_impl(list, g.registry)) > + class X: > + pass > + c.MutableMapping.register(X) # Will not invalidate the cache, > + # not using ABCs yet. > + self.assertEqual(g(d), "base") > + self.assertEqual(g(l), "list") > + self.assertEqual(td.get_ops, [list, dict, dict, list]) > + self.assertEqual(td.set_ops, [dict, list, dict, list]) > + g.register(c.Sized, lambda arg: "sized") > + self.assertEqual(len(td), 0) > + self.assertEqual(g(d), "sized") > + self.assertEqual(len(td), 1) > + self.assertEqual(td.get_ops, [list, dict, dict, list]) > + self.assertEqual(td.set_ops, [dict, list, dict, list, dict]) > + self.assertEqual(g(l), "list") > + self.assertEqual(len(td), 2) > + self.assertEqual(td.get_ops, [list, dict, dict, list]) > + self.assertEqual(td.set_ops, [dict, list, dict, list, dict, list]) > + self.assertEqual(g(l), "list") > + self.assertEqual(g(d), "sized") > + self.assertEqual(td.get_ops, [list, dict, dict, list, list, dict]) > + self.assertEqual(td.set_ops, [dict, list, dict, list, dict, list]) > + g.dispatch(list) > + g.dispatch(dict) > + self.assertEqual(td.get_ops, [list, dict, dict, list, list, dict, > + list, dict]) > + self.assertEqual(td.set_ops, [dict, list, dict, list, dict, list]) > + c.MutableSet.register(X) # Will invalidate the cache. > + self.assertEqual(len(td), 2) # Stale cache. > + self.assertEqual(g(l), "list") > + self.assertEqual(len(td), 1) > + g.register(c.MutableMapping, lambda arg: "mutablemapping") > + self.assertEqual(len(td), 0) > + self.assertEqual(g(d), "mutablemapping") > + self.assertEqual(len(td), 1) > + self.assertEqual(g(l), "list") > + self.assertEqual(len(td), 2) > + g.register(dict, lambda arg: "dict") > + self.assertEqual(g(d), "dict") > + self.assertEqual(g(l), "list") > + g._clear_cache() > + self.assertEqual(len(td), 0) > + functools.WeakKeyDictionary = _orig_wkd > + > + > def test_main(verbose=None): > test_classes = ( > TestPartialC, > @@ -846,6 +1211,7 @@ > TestWraps, > TestReduce, > TestLRU, > + TestSingleDispatch, > ) > support.run_unittest(*test_classes) > > diff --git a/Misc/NEWS b/Misc/NEWS > --- a/Misc/NEWS > +++ b/Misc/NEWS > @@ -344,6 +344,9 @@ > the default for linking if LDSHARED is not also overriden. This > restores > Distutils behavior introduced in 3.2.3 and inadvertently dropped in > 3.3.0. > > +- Implement PEP 443 "Single-dispatch generic functions". > + > + > Tests > ----- > > diff --git a/Modules/Setup.dist b/Modules/Setup.dist > --- a/Modules/Setup.dist > +++ b/Modules/Setup.dist > @@ -116,6 +116,7 @@ > _operator _operator.c # operator.add() and similar goodies > _collections _collectionsmodule.c # Container types > itertools itertoolsmodule.c # Functions creating iterators for > efficient looping > +atexit atexitmodule.c # Register functions to be run at > interpreter-shutdown > > # access to ISO C locale support > _locale _localemodule.c # -lintl > @@ -170,7 +171,6 @@ > #_weakref _weakref.c # basic weak reference support > #_testcapi _testcapimodule.c # Python C API test module > #_random _randommodule.c # Random number generator > -#atexit atexitmodule.c # Register functions to be run at > interpreter-shutdown > #_elementtree -I$(srcdir)/Modules/expat -DHAVE_EXPAT_CONFIG_H > -DUSE_PYEXPAT_CAPI _elementtree.c # elementtree accelerator > #_pickle _pickle.c # pickle accelerator > #_datetime _datetimemodule.c # datetime accelerator > > -- > Repository URL: http://hg.python.org/cpython > > _______________________________________________ > Python-checkins mailing list > Python-checkins at python.org > http://mail.python.org/mailman/listinfo/python-checkins > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From lukasz at langa.pl Wed Jun 5 17:52:56 2013 From: lukasz at langa.pl (=?utf-8?Q?=C5=81ukasz_Langa?=) Date: Wed, 5 Jun 2013 17:52:56 +0200 Subject: [Python-Dev] [Python-checkins] cpython: Add reference implementation for PEP 443 In-Reply-To: References: <3bQQxs6YX7zRbZ@mail.python.org> Message-ID: <0F939F6D-63D1-4D93-9EA7-39C7C09E0F6A@langa.pl> Dnia 5 cze 2013 o godz. 16:31 Brett Cannon napisa?(a): > Any chance you could move your definitions for "generic function" and "single dispatch" to the glossary and just link to them here? Sure thing. -- ? From benjamin at python.org Wed Jun 5 18:10:54 2013 From: benjamin at python.org (Benjamin Peterson) Date: Wed, 5 Jun 2013 09:10:54 -0700 Subject: [Python-Dev] PEP 442 accepted Message-ID: I (and Guido) are accepting PEP 442 (Safe object finalization) on the condition that finalizers are only ever called once globally. Congratulations to Antoine on writing yet another PEP that deeply touches the core language in a way that everyone can agree is an improvement.. I look forward to reviewing the code. -- Regards, Benjamin From solipsis at pitrou.net Wed Jun 5 18:23:07 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 05 Jun 2013 18:23:07 +0200 Subject: [Python-Dev] PEP 442 accepted In-Reply-To: References: Message-ID: <51AF65EB.2070703@pitrou.net> Le 05/06/2013 18:10, Benjamin Peterson a ?crit : > I (and Guido) are accepting PEP 442 (Safe object finalization) on the > condition that finalizers are only ever called once globally. > > Congratulations to Antoine on writing yet another PEP that deeply > touches the core language in a way that everyone can agree is an > improvement.. I look forward to reviewing the code. Thank you! Antoine (on holiday - is anyone I know in Hungary?). > > -- > Regards, > Benjamin > From jpakkane at gmail.com Wed Jun 5 21:21:53 2013 From: jpakkane at gmail.com (Jussi Pakkanen) Date: Wed, 5 Jun 2013 22:21:53 +0300 Subject: [Python-Dev] Compiling Python with Python Message-ID: Hello all I'd like to start this email by saying this is not a proposal to change Python's build system. This is just the results of some experimentation you might be interested it. I have been working on a cross-platform build system called Meson, which is implemented in Python 3. For symmetry I wanted to see if it could be used to build Python itself. After about an evening worth of work, I got the basic C parts (that is, the build targets in the main Makefile such as core Python, pgen etc) built. Main highlights: - pyconfig.h generation in a fully cross-platform way without Autoconf (not tested with Visual Studio but should work as Meson has unit tests for this, tests for functions in header files and some others not done) - builds in a separate build directory, can have arbitrarily many build dirs with different configurations (for gcc/clang/static analysis/debug/release/etc) - configure time 5s, build time 8s on an i5 Macbook running Ubuntu (Autotool-configure takes 37 s but it's not directly comparable because it does a lot more) - the file describing the build is 433 lines, most of which look like this: if cc.has_header('io.h') pyconf.set('HAVE_IO_H', 1) endif - implementation of Meson is 100% Python 3, it does not have a dependency on the shell and in fact already works on Windows If you want to try it yourself, here are the steps (only 64 bit Linux tested thus far): - install python3-ply and Ninja (Ubuntu package ninja-build) - get Meson git head: https://sourceforge.net/p/meson/code/ - get Python3 trunk - download and extract the build files into trunk: https://dl.dropboxusercontent.com/u/37517477/python-meson.tar.gz - cd into Python trunk and do the following commands: mkdir build cd build path/to/meson.py .. ninja And there you have it. You can't do much with it, though (except run pgen to ensure that it actually did something ;) ). If you have any questions that are not directly related to Python, feel free to email me or the Meson mailing list. Enjoy, -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Wed Jun 5 21:37:24 2013 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 6 Jun 2013 05:37:24 +1000 Subject: [Python-Dev] Compiling Python with Python In-Reply-To: References: Message-ID: On Thu, Jun 6, 2013 at 5:21 AM, Jussi Pakkanen wrote: > - implementation of Meson is 100% Python 3, it does not have a dependency on > the shell and in fact already works on Windows Since you're talking about a bootstrap requirement here, the obvious question is: What version of Python 3 does it require? Will it be a lot of hassle to get hold of (say) Python 3.2, only to uninstall it when you have your 3.4 built? ChrisA From jpakkane at gmail.com Wed Jun 5 22:00:47 2013 From: jpakkane at gmail.com (Jussi Pakkanen) Date: Wed, 5 Jun 2013 23:00:47 +0300 Subject: [Python-Dev] Compiling Python with Python In-Reply-To: References: Message-ID: On Wed, Jun 5, 2013 at 10:37 PM, Chris Angelico wrote: > Since you're talking about a bootstrap requirement here, the obvious > question is: What version of Python 3 does it require? Will it be a > lot of hassle to get hold of (say) Python 3.2, only to uninstall it > when you have your 3.4 built? > The implementation does not use deep Python magic such as C extensions or the like so it should work with future releases of Python. The current version requires 3.3 or newer but only because it uses a couple of pythonlib functions that were added at 3.3. Changing it to work with 3.2 or earlier should not be a big task. -------------- next part -------------- An HTML attachment was scrubbed... URL: From cpmicropro at gmail.com Wed Jun 5 22:02:17 2013 From: cpmicropro at gmail.com (Hossein) Date: Thu, 06 Jun 2013 00:32:17 +0430 Subject: [Python-Dev] HAVE_FSTAT? In-Reply-To: <1368966094.28.0.0656360189268.issue12082@psf.upfronthosting.co.za> References: <1368966094.28.0.0656360189268.issue12082@psf.upfronthosting.co.za> Message-ID: <51AF9949.1090204@gmail.com> Hi. My 2 cents about this: (well I'm only a noob) I had this problem; I don't know about other people's environment, but my environment's problem was that it was actually not POSIX-compliant: it didn't have other file functions as well, but anyway the `fstat` error is the FIRST error you get when you compile in such environments, so people as unaware as me think the problem is with fstat only. Anyway I think if you are going to remove anything, you should think in terms of POSIX-compliancy of the target system. Removing HAVE_FSTAT might be fine (as user can easily write his own version of the function and have it included into the python's sources), but if you instead provide the user with the ability to use his custom functions when POSIX one's aren't available, it would help make python compile on even more platforms. Sorry if this last one was off-topic. Best regards. -------------- next part -------------- An HTML attachment was scrubbed... URL: From a.cavallo at cavallinux.eu Wed Jun 5 22:18:21 2013 From: a.cavallo at cavallinux.eu (Antonio Cavallo) Date: Wed, 5 Jun 2013 21:18:21 +0100 Subject: [Python-Dev] Compiling Python with Python In-Reply-To: References: Message-ID: What's the advantage in writing a new build tool? I'm asking this because I'm doing the same using scons: https://bitbucket.org/cavallo71/fatpython At the moment I'm very interested into this problem: the main advantages I see so far are (in scons) are node dependencies and the fact it is plain python syntax. Thanks On 5 Jun 2013, at 20:21, Jussi Pakkanen wrote: > Hello all > > I'd like to start this email by saying this is not a proposal to change Python's build system. This is just the results of some experimentation you might be interested it. > > I have been working on a cross-platform build system called Meson, which is implemented in Python 3. For symmetry I wanted to see if it could be used to build Python itself. After about an evening worth of work, I got the basic C parts (that is, the build targets in the main Makefile such as core Python, pgen etc) built. > > Main highlights: > > - pyconfig.h generation in a fully cross-platform way without Autoconf (not tested with Visual Studio but should work as Meson has unit tests for this, tests for functions in header files and some others not done) > > - builds in a separate build directory, can have arbitrarily many build dirs with different configurations (for gcc/clang/static analysis/debug/release/etc) > > - configure time 5s, build time 8s on an i5 Macbook running Ubuntu (Autotool-configure takes 37 s but it's not directly comparable because it does a lot more) > > - the file describing the build is 433 lines, most of which look like this: > > if cc.has_header('io.h') > pyconf.set('HAVE_IO_H', 1) > endif > > - implementation of Meson is 100% Python 3, it does not have a dependency on the shell and in fact already works on Windows > > If you want to try it yourself, here are the steps (only 64 bit Linux tested thus far): > > - install python3-ply and Ninja (Ubuntu package ninja-build) > - get Meson git head: https://sourceforge.net/p/meson/code/ > - get Python3 trunk > - download and extract the build files into trunk: > https://dl.dropboxusercontent.com/u/37517477/python-meson.tar.gz > - cd into Python trunk and do the following commands: > > mkdir build > cd build > path/to/meson.py .. > ninja > > And there you have it. You can't do much with it, though (except run pgen to ensure that it actually did something ;) ). > > If you have any questions that are not directly related to Python, feel free to email me or the Meson mailing list. > > Enjoy, > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/a.cavallo%40cavallinux.eu From rosuav at gmail.com Wed Jun 5 22:19:55 2013 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 6 Jun 2013 06:19:55 +1000 Subject: [Python-Dev] Compiling Python with Python In-Reply-To: References: Message-ID: On Thu, Jun 6, 2013 at 6:00 AM, Jussi Pakkanen wrote: > > On Wed, Jun 5, 2013 at 10:37 PM, Chris Angelico wrote: > >> >> Since you're talking about a bootstrap requirement here, the obvious >> question is: What version of Python 3 does it require? Will it be a >> lot of hassle to get hold of (say) Python 3.2, only to uninstall it >> when you have your 3.4 built? > > > The implementation does not use deep Python magic such as C extensions or > the like so it should work with future releases of Python. The current > version requires 3.3 or newer but only because it uses a couple of pythonlib > functions that were added at 3.3. Changing it to work with 3.2 or earlier > should not be a big task. Newer versions shouldn't be a problem, older ones will. I'm mainly thinking about systems that can't just casually apt-get a Python 3.3. With Ubuntu, most people probably don't even need to worry about building from source, as there'll be a decently-recent Python in the repo; but what happens when you're on Debian Squeeze and the only Python 3 you can get is 3.1.3? Even Wheezy (the current stable Debian) comes with only 3.2. I do like the symmetry of using Python to build Python. But I also like using Python 3.3+ for everything, and not having to support the older Pythons. Unfortunately those two capacities clash, my lords, they clash! ChrisA From christian at python.org Fri Jun 7 00:37:01 2013 From: christian at python.org (Christian Heimes) Date: Fri, 07 Jun 2013 00:37:01 +0200 Subject: [Python-Dev] ssl improvements and testing question Message-ID: Hi, I'm working on a couple of improvements for the ssl module: http://bugs.python.org/issue17134 http://bugs.python.org/issue18138 http://bugs.python.org/issue18143 http://bugs.python.org/issue18147 #17134 is going to provide a way to use Window's crypt32.dll to load CA certs from Window's CA cert storage. I have a working proof of concept [1] that uses ctypes to interface crypt32.dll. I'll reimplement the code in C. #18138 implements the bits and pieces for #17134 in order to add DER and PEM certs from memory (ASCII unicode or Py_Buffer). Until now the ssl module can only load files from the file system. #18143 and #18147 are diagnostic and debugging helpers that I would like to add. The SSLContext() object is black box. You stuff in some PEM files and don't know which CA certs have been loaded. The enhancements implement a function to retrieve a list of CA certs (same format as getpeercert()) and list of default CA locations for the platform. I'm also thinking about OCSP support and X509v3 extension support for _decode_certificate(). Both are a PITB ... Python has an easier and better documented C API. Question: What's the minimum version of OpenSSL Python 3.4 is going to support? Do we have an easy way to compile and link Python against a custom installation of OpenSSL or do I have to fiddle around with CPPFLAGS and CFLAGS? Christian [1] https://pypi.python.org/pypi/wincertstore From thomas at python.org Fri Jun 7 16:27:06 2013 From: thomas at python.org (Thomas Wouters) Date: Fri, 7 Jun 2013 16:27:06 +0200 Subject: [Python-Dev] [Python-checkins] cpython: Add reference implementation for PEP 443 In-Reply-To: <3bQQxs6YX7zRbZ@mail.python.org> References: <3bQQxs6YX7zRbZ@mail.python.org> Message-ID: On Wed, Jun 5, 2013 at 3:20 AM, lukasz.langa wrote: > +from weakref import WeakKeyDictionary > FYI, this change exposes a bug in the atexit module involving subinterpreters, causing the refleaks reported by Antoine's daily report: interpreter startup now always imports weakref, which imports atexit and registers a classmethod. Unfortunately the atexit module doesn't seem to know subinterpreters from subtitles and so doesn't unregister this classmethod when the subinterpreter is terminated. This isn't a new bug, but it's exposed by always importing weakref and atexit during interpreter startup. I'm wondering if that's really necessary :) -- Thomas Wouters Hi! I'm an email virus! Think twice before sending your email to help me spread! -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Fri Jun 7 17:16:05 2013 From: brett at python.org (Brett Cannon) Date: Fri, 7 Jun 2013 11:16:05 -0400 Subject: [Python-Dev] [Python-checkins] cpython: Issue #17931: Resolve confusion on Windows between pids and process handles. In-Reply-To: <3bQl996bPdz7LkX@mail.python.org> References: <3bQl996bPdz7LkX@mail.python.org> Message-ID: I think this CL introduced a memory leak. The daily leak report went from 0 to not 0 between June 4 and June 5 and this is the only CL that touched C code. On Wed, Jun 5, 2013 at 6:31 PM, richard.oudkerk wrote: > http://hg.python.org/cpython/rev/0410bf251e10 > changeset: 84045:0410bf251e10 > user: Richard Oudkerk > date: Wed Jun 05 23:29:30 2013 +0100 > summary: > Issue #17931: Resolve confusion on Windows between pids and process > handles. > > files: > Include/longobject.h | 13 +++++++++++++ > Misc/NEWS | 5 ++--- > Modules/posixmodule.c | 25 +++++++++---------------- > PC/msvcrtmodule.c | 5 +++-- > PC/pyconfig.h | 4 ++-- > 5 files changed, 29 insertions(+), 23 deletions(-) > > > diff --git a/Include/longobject.h b/Include/longobject.h > --- a/Include/longobject.h > +++ b/Include/longobject.h > @@ -52,6 +52,19 @@ > #error "sizeof(pid_t) is neither sizeof(int), sizeof(long) or sizeof(long > long)" > #endif /* SIZEOF_PID_T */ > > +#if SIZEOF_VOID_P == SIZEOF_INT > +# define _Py_PARSE_INTPTR "i" > +# define _Py_PARSE_UINTPTR "I" > +#elif SIZEOF_VOID_P == SIZEOF_LONG > +# define _Py_PARSE_INTPTR "l" > +# define _Py_PARSE_UINTPTR "k" > +#elif defined(SIZEOF_LONG_LONG) && SIZEOF_VOID_P == SIZEOF_LONG_LONG > +# define _Py_PARSE_INTPTR "L" > +# define _Py_PARSE_UINTPTR "K" > +#else > +# error "void* different in size from int, long and long long" > +#endif /* SIZEOF_VOID_P */ > + > /* Used by Python/mystrtoul.c. */ > #ifndef Py_LIMITED_API > PyAPI_DATA(unsigned char) _PyLong_DigitValue[256]; > diff --git a/Misc/NEWS b/Misc/NEWS > --- a/Misc/NEWS > +++ b/Misc/NEWS > @@ -10,9 +10,8 @@ > Core and Builtins > ----------------- > > -- Issue #17931: Fix PyLong_FromPid() on Windows 64-bit: processes are > - identified by their HANDLE which is a pointer (and not a long, which is > - smaller). > +- Issue #17931: Resolve confusion on Windows between pids and process > + handles. > > - Tweak the exception message when the magic number or size value in a > bytecode > file is truncated. > diff --git a/Modules/posixmodule.c b/Modules/posixmodule.c > --- a/Modules/posixmodule.c > +++ b/Modules/posixmodule.c > @@ -5014,11 +5014,7 @@ > if (spawnval == -1) > return posix_error(); > else > -#if SIZEOF_LONG == SIZEOF_VOID_P > - return Py_BuildValue("l", (long) spawnval); > -#else > - return Py_BuildValue("L", (PY_LONG_LONG) spawnval); > -#endif > + return Py_BuildValue(_Py_PARSE_INTPTR, spawnval); > } > > > @@ -5104,11 +5100,7 @@ > if (spawnval == -1) > (void) posix_error(); > else > -#if SIZEOF_LONG == SIZEOF_VOID_P > - res = Py_BuildValue("l", (long) spawnval); > -#else > - res = Py_BuildValue("L", (PY_LONG_LONG) spawnval); > -#endif > + res = Py_BuildValue(_Py_PARSE_INTPTR, spawnval); > > while (--envc >= 0) > PyMem_DEL(envlist[envc]); > @@ -6178,16 +6170,17 @@ > win32_kill(PyObject *self, PyObject *args) > { > PyObject *result; > - DWORD pid, sig, err; > + pid_t pid; > + DWORD sig, err; > HANDLE handle; > > - if (!PyArg_ParseTuple(args, "kk:kill", &pid, &sig)) > + if (!PyArg_ParseTuple(args, _Py_PARSE_PID "k:kill", &pid, &sig)) > return NULL; > > /* Console processes which share a common console can be sent CTRL+C > or > CTRL+BREAK events, provided they handle said events. */ > if (sig == CTRL_C_EVENT || sig == CTRL_BREAK_EVENT) { > - if (GenerateConsoleCtrlEvent(sig, pid) == 0) { > + if (GenerateConsoleCtrlEvent(sig, (DWORD)pid) == 0) { > err = GetLastError(); > PyErr_SetFromWindowsErr(err); > } > @@ -6197,7 +6190,7 @@ > > /* If the signal is outside of what GenerateConsoleCtrlEvent can use, > attempt to open and terminate the process. */ > - handle = OpenProcess(PROCESS_ALL_ACCESS, FALSE, pid); > + handle = OpenProcess(PROCESS_ALL_ACCESS, FALSE, (DWORD)pid); > if (handle == NULL) { > err = GetLastError(); > return PyErr_SetFromWindowsErr(err); > @@ -6603,7 +6596,7 @@ > Py_intptr_t pid; > int status, options; > > - if (!PyArg_ParseTuple(args, _Py_PARSE_PID "i:waitpid", &pid, > &options)) > + if (!PyArg_ParseTuple(args, _Py_PARSE_INTPTR "i:waitpid", &pid, > &options)) > return NULL; > Py_BEGIN_ALLOW_THREADS > pid = _cwait(&status, pid, options); > @@ -6612,7 +6605,7 @@ > return posix_error(); > > /* shift the status left a byte so this is more like the POSIX > waitpid */ > - return Py_BuildValue("Ni", PyLong_FromPid(pid), status << 8); > + return Py_BuildValue(_Py_PARSE_INTPTR "i", pid, status << 8); > } > #endif /* HAVE_WAITPID || HAVE_CWAIT */ > > diff --git a/PC/msvcrtmodule.c b/PC/msvcrtmodule.c > --- a/PC/msvcrtmodule.c > +++ b/PC/msvcrtmodule.c > @@ -113,11 +113,12 @@ > static PyObject * > msvcrt_open_osfhandle(PyObject *self, PyObject *args) > { > - long handle; > + Py_intptr_t handle; > int flags; > int fd; > > - if (!PyArg_ParseTuple(args, "li:open_osfhandle", &handle, &flags)) > + if (!PyArg_ParseTuple(args, _Py_PARSE_INTPTR "i:open_osfhandle", > + &handle, &flags)) > return NULL; > > fd = _open_osfhandle(handle, flags); > diff --git a/PC/pyconfig.h b/PC/pyconfig.h > --- a/PC/pyconfig.h > +++ b/PC/pyconfig.h > @@ -723,8 +723,8 @@ > /* The size of `wchar_t', as computed by sizeof. */ > #define SIZEOF_WCHAR_T 2 > > -/* The size of `pid_t' (HANDLE). */ > -#define SIZEOF_PID_T SIZEOF_VOID_P > +/* The size of `pid_t', as computed by sizeof. */ > +#define SIZEOF_PID_T SIZEOF_INT > > /* Define if you have the dl library (-ldl). */ > /* #undef HAVE_LIBDL */ > > -- > Repository URL: http://hg.python.org/cpython > > _______________________________________________ > Python-checkins mailing list > Python-checkins at python.org > http://mail.python.org/mailman/listinfo/python-checkins > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From thomas at python.org Fri Jun 7 17:54:48 2013 From: thomas at python.org (Thomas Wouters) Date: Fri, 7 Jun 2013 17:54:48 +0200 Subject: [Python-Dev] [Python-checkins] cpython: Issue #17931: Resolve confusion on Windows between pids and process handles. In-Reply-To: References: <3bQl996bPdz7LkX@mail.python.org> Message-ID: On Fri, Jun 7, 2013 at 5:16 PM, Brett Cannon wrote: > I think this CL introduced a memory leak. The daily leak report went from > 0 to not 0 between June 4 and June 5 and this is the only CL that touched C > code. > It wasn't introduced by C code :) The refleak report is induced by the PEP 443 implementation, see my message to python-dev. -- Thomas Wouters Hi! I'm an email virus! Think twice before sending your email to help me spread! -------------- next part -------------- An HTML attachment was scrubbed... URL: From ethan at stoneleaf.us Fri Jun 7 17:37:08 2013 From: ethan at stoneleaf.us (Ethan Furman) Date: Fri, 07 Jun 2013 08:37:08 -0700 Subject: [Python-Dev] html documentation and table-of-contents Message-ID: <51B1FE24.3060900@stoneleaf.us> I just used the build system on the 3.4.0 docs, and some of the library modules (haven't checked the others) have the TOC showing up at the bottom of the page instead of the top. Am I doing something wrong? -- ~Ethan~ From status at bugs.python.org Fri Jun 7 18:07:36 2013 From: status at bugs.python.org (Python tracker) Date: Fri, 7 Jun 2013 18:07:36 +0200 (CEST) Subject: [Python-Dev] Summary of Python tracker Issues Message-ID: <20130607160736.0C9565692C@psf.upfronthosting.co.za> ACTIVITY SUMMARY (2013-05-31 - 2013-06-07) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 4024 (+27) closed 25905 (+21) total 29929 (+48) Open issues with patches: 1793 Issues opened (41) ================== #8902: add datetime.time.now() for consistency http://bugs.python.org/issue8902 reopened by belopolsky #17206: Py_XDECREF() expands its argument multiple times http://bugs.python.org/issue17206 reopened by haypo #17222: py_compile.compile() replaces target files, breaking special f http://bugs.python.org/issue17222 reopened by brett.cannon #18110: Nested set comprehensions in class scope fail http://bugs.python.org/issue18110 opened by Eric.Wieser #18111: Add a default argument to min & max http://bugs.python.org/issue18111 opened by Julian #18112: PEP 442 implementation http://bugs.python.org/issue18112 opened by pitrou #18113: Memory leak in curses.panel http://bugs.python.org/issue18113 opened by ishimoto #18114: Update PyImport_ImportFrozenModuleObject() to use importlib http://bugs.python.org/issue18114 opened by brett.cannon #18115: Use importlib.util.module_to_load in all loaders in importlib http://bugs.python.org/issue18115 opened by brett.cannon #18116: getpass.unix_getpass() always fallback to sys.stdin http://bugs.python.org/issue18116 opened by Nikratio #18117: Missing symlink:Current after Mac OS X 3.3.2 package installat http://bugs.python.org/issue18117 opened by gavan #18119: urllib.FancyURLopener does not treat URL fragments correctly http://bugs.python.org/issue18119 opened by takahashi.shuhei #18121: antigravity leaks subprocess.Popen object http://bugs.python.org/issue18121 opened by christian.heimes #18122: RuntimeError: not holding the import lock http://bugs.python.org/issue18122 opened by arigo #18123: fnmatchicase for case insensitive file search http://bugs.python.org/issue18123 opened by techtonik #18125: Out-of-tree build cannot regenerate Makefile.pre http://bugs.python.org/issue18125 opened by TBBle #18126: Update links to NumPy resources in documentation http://bugs.python.org/issue18126 opened by zaytsev #18128: pygettext: non-standard timestamp format in POT-Creation-Date http://bugs.python.org/issue18128 opened by jwilk #18129: Fatal Python error: Cannot recover from stack overflow. http://bugs.python.org/issue18129 opened by oscarbenjamin #18131: Tkinter Variables require a proper master http://bugs.python.org/issue18131 opened by terry.reedy #18132: buttons in turtledemo disappear on small screens http://bugs.python.org/issue18132 opened by JanKanis #18135: _ssl module: possible integer overflow for very long strings ( http://bugs.python.org/issue18135 opened by haypo #18136: Put local build paths before system build paths in configure.a http://bugs.python.org/issue18136 opened by brett.cannon #18137: format(float, str): integer overflow for huge precision http://bugs.python.org/issue18137 opened by haypo #18138: ssl.SSLContext.add_cert() http://bugs.python.org/issue18138 opened by christian.heimes #18139: email module should have a way to prepend and insert headers http://bugs.python.org/issue18139 opened by kbandla #18140: urlparse.urlsplit confused to fragment when password include # http://bugs.python.org/issue18140 opened by anh.le #18141: tkinter.Image.__del__ can throw an exception if module globals http://bugs.python.org/issue18141 opened by JanKanis #18142: Tests fail on Mageia Linux Cauldron x86-64 with some configure http://bugs.python.org/issue18142 opened by shlomif #18143: ssl.get_default_verify_paths() http://bugs.python.org/issue18143 opened by christian.heimes #18144: FD leak in urllib2 http://bugs.python.org/issue18144 opened by Claudio.Freire #18147: SSL: diagnostic functions to list loaded CA certs http://bugs.python.org/issue18147 opened by christian.heimes #18148: Make of Python 3.2.2 fails on Solaris SPARC http://bugs.python.org/issue18148 opened by eeiddne #18149: filecmp.cmp() - cache invalidation fails when file modificatio http://bugs.python.org/issue18149 opened by fbm #18150: Duplicate test inside TestSingleDispatch http://bugs.python.org/issue18150 opened by vajrasky #18151: Idlelib: update to "with open ... except OSError" (in 2.7, le http://bugs.python.org/issue18151 opened by terry.reedy #18152: Idle: add 2.7 backport script http://bugs.python.org/issue18152 opened by terry.reedy #18153: python imaplib - error 'unexpected repsonse' http://bugs.python.org/issue18153 opened by tahnoon #18154: make install failed on solaris http://bugs.python.org/issue18154 opened by palm.kevin #18155: csv.Sniffer.has_header doesn't escape characters used in regex http://bugs.python.org/issue18155 opened by davechallis #18156: Add an 'attr' attribute to AttributeError http://bugs.python.org/issue18156 opened by brett.cannon Most recent 15 issues with no replies (15) ========================================== #18156: Add an 'attr' attribute to AttributeError http://bugs.python.org/issue18156 #18149: filecmp.cmp() - cache invalidation fails when file modificatio http://bugs.python.org/issue18149 #18148: Make of Python 3.2.2 fails on Solaris SPARC http://bugs.python.org/issue18148 #18147: SSL: diagnostic functions to list loaded CA certs http://bugs.python.org/issue18147 #18144: FD leak in urllib2 http://bugs.python.org/issue18144 #18138: ssl.SSLContext.add_cert() http://bugs.python.org/issue18138 #18137: format(float, str): integer overflow for huge precision http://bugs.python.org/issue18137 #18131: Tkinter Variables require a proper master http://bugs.python.org/issue18131 #18129: Fatal Python error: Cannot recover from stack overflow. http://bugs.python.org/issue18129 #18126: Update links to NumPy resources in documentation http://bugs.python.org/issue18126 #18119: urllib.FancyURLopener does not treat URL fragments correctly http://bugs.python.org/issue18119 #18114: Update PyImport_ImportFrozenModuleObject() to use importlib http://bugs.python.org/issue18114 #18113: Memory leak in curses.panel http://bugs.python.org/issue18113 #18112: PEP 442 implementation http://bugs.python.org/issue18112 #18108: shutil.chown should support dir_fd and follow_symlinks keyword http://bugs.python.org/issue18108 Most recent 15 issues waiting for review (15) ============================================= #18155: csv.Sniffer.has_header doesn't escape characters used in regex http://bugs.python.org/issue18155 #18154: make install failed on solaris http://bugs.python.org/issue18154 #18151: Idlelib: update to "with open ... except OSError" (in 2.7, le http://bugs.python.org/issue18151 #18150: Duplicate test inside TestSingleDispatch http://bugs.python.org/issue18150 #18147: SSL: diagnostic functions to list loaded CA certs http://bugs.python.org/issue18147 #18143: ssl.get_default_verify_paths() http://bugs.python.org/issue18143 #18141: tkinter.Image.__del__ can throw an exception if module globals http://bugs.python.org/issue18141 #18138: ssl.SSLContext.add_cert() http://bugs.python.org/issue18138 #18137: format(float, str): integer overflow for huge precision http://bugs.python.org/issue18137 #18135: _ssl module: possible integer overflow for very long strings ( http://bugs.python.org/issue18135 #18132: buttons in turtledemo disappear on small screens http://bugs.python.org/issue18132 #18126: Update links to NumPy resources in documentation http://bugs.python.org/issue18126 #18116: getpass.unix_getpass() always fallback to sys.stdin http://bugs.python.org/issue18116 #18112: PEP 442 implementation http://bugs.python.org/issue18112 #18111: Add a default argument to min & max http://bugs.python.org/issue18111 Top 10 most discussed issues (10) ================================= #18111: Add a default argument to min & max http://bugs.python.org/issue18111 28 msgs #18116: getpass.unix_getpass() always fallback to sys.stdin http://bugs.python.org/issue18116 17 msgs #18021: Update broken link to Apple Publication Style Guide http://bugs.python.org/issue18021 9 msgs #16427: Faster hash implementation http://bugs.python.org/issue16427 8 msgs #17222: py_compile.compile() replaces target files, breaking special f http://bugs.python.org/issue17222 7 msgs #18078: threading.Condition to allow notify on a specific waiter http://bugs.python.org/issue18078 7 msgs #3329: API for setting the memory allocator used by Python http://bugs.python.org/issue3329 6 msgs #18054: Add more exception related assertions to unittest http://bugs.python.org/issue18054 6 msgs #18122: RuntimeError: not holding the import lock http://bugs.python.org/issue18122 6 msgs #18143: ssl.get_default_verify_paths() http://bugs.python.org/issue18143 6 msgs Issues closed (24) ================== #11959: smtpd cannot be used without affecting global state http://bugs.python.org/issue11959 closed by python-dev #16450: test_missing_localfile masks problems in urlopen http://bugs.python.org/issue16450 closed by orsenthil #17314: Stop using imp.find_module() in multiprocessing http://bugs.python.org/issue17314 closed by brett.cannon #17342: datetime.strptime does not implement %z http://bugs.python.org/issue17342 closed by belopolsky #17486: datetime.timezone returns the wrong tzname() http://bugs.python.org/issue17486 closed by belopolsky #17931: PyLong_FromPid() is not correctly defined on Windows 64-bit http://bugs.python.org/issue17931 closed by python-dev #17932: Win64: possible integer overflow in iterobject.c http://bugs.python.org/issue17932 closed by python-dev #17967: urllib2.open failed to access a url when a perent directory of http://bugs.python.org/issue17967 closed by orsenthil #18065: set __path__ = [] for frozen packages http://bugs.python.org/issue18065 closed by brett.cannon #18066: Remove SGI-specific code from pty.py http://bugs.python.org/issue18066 closed by akuchling #18088: Create importlib.abc.Loader.init_module_attrs() http://bugs.python.org/issue18088 closed by brett.cannon #18089: Create importlib.abc.InspectLoader.load_module() http://bugs.python.org/issue18089 closed by brett.cannon #18094: Skip tests in test_uuid not silently http://bugs.python.org/issue18094 closed by serhiy.storchaka #18109: os.uname() crashes if hostname contains non-ascii characters http://bugs.python.org/issue18109 closed by python-dev #18118: curses utf8 output broken in Python2 http://bugs.python.org/issue18118 closed by haypo #18120: multiprocessing: garbage collector fails to GC Pipe() end when http://bugs.python.org/issue18120 closed by sbt #18124: Broken build on target machine with incorrect hostname (non-as http://bugs.python.org/issue18124 closed by dmi.baranov #18127: Strange behaviour with default list argument http://bugs.python.org/issue18127 closed by eric.smith #18130: idlelib.configSectionNameDialog: fix and add tests and mocks http://bugs.python.org/issue18130 closed by terry.reedy #18133: Modulus not calculating properly? http://bugs.python.org/issue18133 closed by pccreator25 #18134: zipfile extractall accepts wrong password http://bugs.python.org/issue18134 closed by r.david.murray #18145: Strange behavior when importing internal modules in the __init http://bugs.python.org/issue18145 closed by brett.cannon #18146: Document miss, Stats objects has no method called "print_resul http://bugs.python.org/issue18146 closed by hanks #1512124: OSX: debugger hangs IDLE http://bugs.python.org/issue1512124 closed by ned.deily From ethan at stoneleaf.us Fri Jun 7 18:06:15 2013 From: ethan at stoneleaf.us (Ethan Furman) Date: Fri, 07 Jun 2013 09:06:15 -0700 Subject: [Python-Dev] doctest and pickle Message-ID: <51B204F7.4030107@stoneleaf.us> Is there a doctest mailing list? I couldn't find it. I'm try to use doctest to verify my docs (imagine that!) but I'm having trouble with the one that uses pickle (imagine that!). Any advice on how to make it work? Here's the excerpt: =============================================================================== Pickling -------- Enumerations can be pickled and unpickled:: >>> from enum import Enum >>> class Fruit(Enum): ... tomato = 1 ... banana = 2 ... cherry = 3 ... >>> from pickle import dumps, loads >>> Fruit.tomato is loads(dumps(Fruit.tomato)) True The usual restrictions for pickling apply: picklable enums must be defined in the top level of a module, since unpickling requires them to be importable from that module. =============================================================================== Oh, and I just realized this is probably why the flufl.enum docs import from a preexisting module instead of creating a new class on the spot. Still, it would be nice if this could work. Any ideas? -- ~Ethan~ From olemis at gmail.com Fri Jun 7 18:54:11 2013 From: olemis at gmail.com (Olemis Lang) Date: Fri, 7 Jun 2013 11:54:11 -0500 Subject: [Python-Dev] doctest and pickle In-Reply-To: <51B204F7.4030107@stoneleaf.us> References: <51B204F7.4030107@stoneleaf.us> Message-ID: On 6/7/13, Ethan Furman wrote: > Is there a doctest mailing list? I couldn't find it. > JFTR, Testing-in-Python (TiP) ML should be the right target for general purpose questions about testing, considering docs even for unittest and doctest http://lists.idyll.org/listinfo/testing-in-python [...] > Any advice on how to make it work? > > Here's the excerpt: > > =============================================================================== > Pickling > -------- > > Enumerations can be pickled and unpickled:: > > >>> from enum import Enum > >>> class Fruit(Enum): > ... tomato = 1 > ... banana = 2 > ... cherry = 3 > ... > >>> from pickle import dumps, loads > >>> Fruit.tomato is loads(dumps(Fruit.tomato)) > True > ... but it seems I'm still getting in tests an instance of Fruit after invoking loads (do you ?) [...] -- Regards, Olemis. Apache? Bloodhound contributor http://issues.apache.org/bloodhound Blog ES: http://simelo-es.blogspot.com/ Blog EN: http://simelo-en.blogspot.com/ Featured article: From dreamingforward at gmail.com Fri Jun 7 19:50:50 2013 From: dreamingforward at gmail.com (Mark Janssen) Date: Fri, 7 Jun 2013 10:50:50 -0700 Subject: [Python-Dev] doctest and pickle In-Reply-To: <51B204F7.4030107@stoneleaf.us> References: <51B204F7.4030107@stoneleaf.us> Message-ID: > >>> from pickle import dumps, loads > >>> Fruit.tomato is loads(dumps(Fruit.tomato)) > True Why are you using is here instead of ==? You're making a circular loop using "is" -- MarkJ Tacoma, Washington From dreamingforward at gmail.com Fri Jun 7 19:54:57 2013 From: dreamingforward at gmail.com (Mark Janssen) Date: Fri, 7 Jun 2013 10:54:57 -0700 Subject: [Python-Dev] doctest and pickle In-Reply-To: References: <51B204F7.4030107@stoneleaf.us> Message-ID: On Fri, Jun 7, 2013 at 10:50 AM, Mark Janssen wrote: >> >>> from pickle import dumps, loads >> >>> Fruit.tomato is loads(dumps(Fruit.tomato)) >> True > > Why are you using is here instead of ==? You're making a circular > loop using "is" I should add that when you're serializing with pickle and then reloading, the objects should be seen as "essentially equivalent". This means that they are either byte-by-byte equivalent (not sure actually if Python actually guarantees this), or every element would still compare equal and that is what matters. -- MarkJ Tacoma, Washington From rdmurray at bitdance.com Fri Jun 7 20:11:59 2013 From: rdmurray at bitdance.com (R. David Murray) Date: Fri, 07 Jun 2013 14:11:59 -0400 Subject: [Python-Dev] doctest and pickle In-Reply-To: References: <51B204F7.4030107@stoneleaf.us> Message-ID: <20130607181205.3B93F250BEB@webabinitio.net> On Fri, 07 Jun 2013 10:54:57 -0700, Mark Janssen wrote: > On Fri, Jun 7, 2013 at 10:50 AM, Mark Janssen wrote: > >> >>> from pickle import dumps, loads > >> >>> Fruit.tomato is loads(dumps(Fruit.tomato)) > >> True > > > > Why are you using is here instead of ==? You're making a circular > > loop using "is" > > I should add that when you're serializing with pickle and then > reloading, the objects should be seen as "essentially equivalent". > This means that they are either byte-by-byte equivalent (not sure > actually if Python actually guarantees this), or every element would > still compare equal and that is what matters. Enums are supposed to be singletons, though, so the 'is' test is exactly the point of this test. --David From ethan at stoneleaf.us Fri Jun 7 19:53:14 2013 From: ethan at stoneleaf.us (Ethan Furman) Date: Fri, 07 Jun 2013 10:53:14 -0700 Subject: [Python-Dev] doctest and pickle In-Reply-To: References: <51B204F7.4030107@stoneleaf.us> Message-ID: <51B21E0A.3090409@stoneleaf.us> On 06/07/2013 09:54 AM, Olemis Lang wrote: > On 6/7/13, Ethan Furman wrote: >> Is there a doctest mailing list? I couldn't find it. >> > > JFTR, Testing-in-Python (TiP) ML should be the right target for > general purpose questions about testing, considering docs even for > unittest and doctest > http://lists.idyll.org/listinfo/testing-in-python Cool, thanks. > [...] >> Any advice on how to make it work? >> >> Here's the excerpt: >> >> =============================================================================== >> Pickling >> -------- >> >> Enumerations can be pickled and unpickled:: >> >> >>> from enum import Enum >> >>> class Fruit(Enum): >> ... tomato = 1 >> ... banana = 2 >> ... cherry = 3 >> ... >> >>> from pickle import dumps, loads >> >>> Fruit.tomato is loads(dumps(Fruit.tomato)) >> True >> > > ... but it seems I'm still getting in tests an instance of Fruit after > invoking loads (do you ?) I'm not sure what you mean by "an instance of Fruit" -- you should be getting an instance of Fruit; specifically the tomato instance. -- ~Ethan~ From brett at python.org Fri Jun 7 20:16:44 2013 From: brett at python.org (Brett Cannon) Date: Fri, 7 Jun 2013 14:16:44 -0400 Subject: [Python-Dev] [Python-checkins] cpython: Add reference implementation for PEP 443 In-Reply-To: References: <3bQQxs6YX7zRbZ@mail.python.org> Message-ID: On Fri, Jun 7, 2013 at 10:27 AM, Thomas Wouters wrote: > > On Wed, Jun 5, 2013 at 3:20 AM, lukasz.langa wrote: > >> +from weakref import WeakKeyDictionary >> > > FYI, this change exposes a bug in the atexit module involving > subinterpreters, causing the refleaks reported by Antoine's daily report: > interpreter startup now always imports weakref, which imports atexit and > registers a classmethod. Unfortunately the atexit module doesn't seem to > know subinterpreters from subtitles and so doesn't unregister this > classmethod when the subinterpreter is terminated. > > This isn't a new bug, but it's exposed by always importing weakref and > atexit during interpreter startup. I'm wondering if that's really necessary > :) > Is there an issue tracking this? -Brett > > > -- > Thomas Wouters > > Hi! I'm an email virus! Think twice before sending your email to help me > spread! > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/brett%40python.org > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ethan at stoneleaf.us Fri Jun 7 19:55:59 2013 From: ethan at stoneleaf.us (Ethan Furman) Date: Fri, 07 Jun 2013 10:55:59 -0700 Subject: [Python-Dev] doctest and pickle In-Reply-To: References: <51B204F7.4030107@stoneleaf.us> Message-ID: <51B21EAF.2000809@stoneleaf.us> On 06/07/2013 10:50 AM, Mark Janssen wrote: >> >>> from pickle import dumps, loads >> >>> Fruit.tomato is loads(dumps(Fruit.tomato)) >> True > > Why are you using is here instead of ==? I'm using `is` because I'm verifying that the instance returned by `pickle.loads` is the exact same object as the instance fed into `pickle.dumps`. Enum members should be singletons. > You're making a circular loop using "is" Huh? -- ~Ethan~ From ethan at stoneleaf.us Fri Jun 7 20:02:54 2013 From: ethan at stoneleaf.us (Ethan Furman) Date: Fri, 07 Jun 2013 11:02:54 -0700 Subject: [Python-Dev] doctest and pickle In-Reply-To: References: <51B204F7.4030107@stoneleaf.us> Message-ID: <51B2204E.2060007@stoneleaf.us> On 06/07/2013 10:54 AM, Mark Janssen wrote: > On Fri, Jun 7, 2013 at 10:50 AM, Mark Janssen wrote: >>> >>> from pickle import dumps, loads >>> >>> Fruit.tomato is loads(dumps(Fruit.tomato)) >>> True >> >> Why are you using is here instead of ==? You're making a circular >> loop using "is" > > I should add that when you're serializing with pickle and then > reloading, the objects should be seen as "essentially equivalent". > This means that they are either byte-by-byte equivalent (not sure > actually if Python actually guarantees this), or every element would > still compare equal and that is what matters. In most cases, sure. But one of the invariants of Enums is that there is only ever one of each member specifically so that `is` tests work. -- ~Ethan~ From pje at telecommunity.com Fri Jun 7 20:30:39 2013 From: pje at telecommunity.com (PJ Eby) Date: Fri, 7 Jun 2013 14:30:39 -0400 Subject: [Python-Dev] doctest and pickle In-Reply-To: References: <51B204F7.4030107@stoneleaf.us> Message-ID: On Fri, Jun 7, 2013 at 1:54 PM, Mark Janssen wrote: > On Fri, Jun 7, 2013 at 10:50 AM, Mark Janssen wrote: >>> >>> from pickle import dumps, loads >>> >>> Fruit.tomato is loads(dumps(Fruit.tomato)) >>> True >> >> Why are you using is here instead of ==? You're making a circular >> loop using "is" > > I should add that when you're serializing with pickle and then > reloading, the objects should be seen as "essentially equivalent". > This means that they are either byte-by-byte equivalent (not sure > actually if Python actually guarantees this), or every element would > still compare equal and that is what matters. For global objects such as functions and classes -- and singletons such as None, Ellipsis, True, and False -- pickling and unpickling is actually supposed to retain the "is" relationship as well. I don't know if enums *actually* preserve this invariant, but my default expectation of the One Obvious Way would be that enums, being uniquely-named objects that know their name and container, should be considered global objects in the same fashion as classes and functions, *and* that as singletons, they'd also be treated in the same way as None, Ellipsis, etc. That is, there are two independent precedents for objects like that preserving "is" upon pickling and unpickling. (As another precedent, my own SymbolType library (available on PyPI) preserves the "is"-ness of its named symbol objects upon pickling and unpickling.) From dreamingforward at gmail.com Fri Jun 7 21:25:44 2013 From: dreamingforward at gmail.com (Mark Janssen) Date: Fri, 7 Jun 2013 12:25:44 -0700 Subject: [Python-Dev] doctest and pickle In-Reply-To: <51B21EAF.2000809@stoneleaf.us> References: <51B204F7.4030107@stoneleaf.us> <51B21EAF.2000809@stoneleaf.us> Message-ID: >> Why are you using is here instead of ==? > > > I'm using `is` because I'm verifying that the instance returned by > `pickle.loads` is the exact same object as the instance fed into > `pickle.dumps`. Enum members should be singletons. I see now. That makes sense, but I don't think you'll be able to do that. "Supposed to be singletons" while at the same time you are holding two instances on the interpreter line. How are you going to manage that? -- MarkJ Tacoma, Washington From barry at python.org Fri Jun 7 21:31:21 2013 From: barry at python.org (Barry Warsaw) Date: Fri, 7 Jun 2013 15:31:21 -0400 Subject: [Python-Dev] doctest and pickle In-Reply-To: <51B204F7.4030107@stoneleaf.us> References: <51B204F7.4030107@stoneleaf.us> Message-ID: <20130607153121.1a1418e0@anarchist> On Jun 07, 2013, at 09:06 AM, Ethan Furman wrote: >Oh, and I just realized this is probably why the flufl.enum docs import from >a preexisting module instead of creating a new class on the spot. Exactly. ;) -Barry From barry at python.org Fri Jun 7 21:33:26 2013 From: barry at python.org (Barry Warsaw) Date: Fri, 7 Jun 2013 15:33:26 -0400 Subject: [Python-Dev] doctest and pickle In-Reply-To: References: <51B204F7.4030107@stoneleaf.us> Message-ID: <20130607153326.56d6424b@anarchist> On Jun 07, 2013, at 02:30 PM, PJ Eby wrote: >I don't know if enums *actually* preserve this invariant, but my >default expectation of the One Obvious Way would be that enums, being >uniquely-named objects that know their name and container, should be >considered global objects in the same fashion as classes and >functions, *and* that as singletons, they'd also be treated in the >same way as None, Ellipsis, etc. That is, there are two independent >precedents for objects like that preserving "is" upon pickling and >unpickling. This is certainly how I thought of them in flufl.enum, and indeed the `is` test in my own test suite proves that it works. -Barry From lukasz at langa.pl Fri Jun 7 21:42:42 2013 From: lukasz at langa.pl (=?utf-8?Q?=C5=81ukasz_Langa?=) Date: Fri, 7 Jun 2013 21:42:42 +0200 Subject: [Python-Dev] [Python-checkins] cpython: Add reference implementation for PEP 443 In-Reply-To: References: <3bQQxs6YX7zRbZ@mail.python.org> Message-ID: On 7 cze 2013, at 16:27, Thomas Wouters wrote: > > On Wed, Jun 5, 2013 at 3:20 AM, lukasz.langa wrote: > +from weakref import WeakKeyDictionary > > This isn't a new bug, but it's exposed by always importing weakref and atexit during interpreter startup. I'm wondering if that's really necessary :) We can easily move the import inside singledispatch but I would much rather try fixing the actual bug. What do you think? -- Best regards, ?ukasz Langa WWW: http://lukasz.langa.pl/ Twitter: @llanga IRC: ambv on #python-dev -------------- next part -------------- An HTML attachment was scrubbed... URL: From pje at telecommunity.com Fri Jun 7 22:50:53 2013 From: pje at telecommunity.com (PJ Eby) Date: Fri, 7 Jun 2013 16:50:53 -0400 Subject: [Python-Dev] [Python-checkins] cpython: Add reference implementation for PEP 443 In-Reply-To: References: <3bQQxs6YX7zRbZ@mail.python.org> Message-ID: On Fri, Jun 7, 2013 at 10:27 AM, Thomas Wouters wrote: > This isn't a new bug, but it's exposed by always importing weakref and > atexit during interpreter startup. I'm wondering if that's really necessary > :) Importing it during startup isn't necessary per se; imports needed only by the generic function implementation can and should be imported late, rather than at the time functools is imported. However, if pkgutil was/is migrated to using this implementation of generics, then it's likely going to end up imported during startup anyway, because at least the -m startup path involves pkgutil. In short, the overall answer right now is, "maybe", and the answer later is "rather likely". ;-) From lukasz at langa.pl Fri Jun 7 23:16:35 2013 From: lukasz at langa.pl (=?utf-8?Q?=C5=81ukasz_Langa?=) Date: Fri, 7 Jun 2013 23:16:35 +0200 Subject: [Python-Dev] [Python-checkins] cpython: Add reference implementation for PEP 443 In-Reply-To: References: <3bQQxs6YX7zRbZ@mail.python.org> Message-ID: On 7 cze 2013, at 22:50, PJ Eby wrote: > On Fri, Jun 7, 2013 at 10:27 AM, Thomas Wouters wrote: >> This isn't a new bug, but it's exposed by always importing weakref and >> atexit during interpreter startup. I'm wondering if that's really necessary >> :) > > In short, the overall answer right now is, "maybe", and the answer > later is "rather likely". ;-) I would rather say that it's "rather certain". functools is necessary for setup.py to work while bootstrapping, whereas pkgutil is used in runpy.py which is always imported in Modules/main.c. So we're left with having to fix atexit to support subinterpreters. I wonder how difficult that will be. -- Best regards, ?ukasz Langa WWW: http://lukasz.langa.pl/ Twitter: @llanga IRC: ambv on #python-dev From pje at telecommunity.com Sat Jun 8 06:38:03 2013 From: pje at telecommunity.com (PJ Eby) Date: Sat, 8 Jun 2013 00:38:03 -0400 Subject: [Python-Dev] [Python-checkins] cpython: Add reference implementation for PEP 443 In-Reply-To: References: <3bQQxs6YX7zRbZ@mail.python.org> Message-ID: On Fri, Jun 7, 2013 at 5:16 PM, ?ukasz Langa wrote: > On 7 cze 2013, at 22:50, PJ Eby wrote: > >> On Fri, Jun 7, 2013 at 10:27 AM, Thomas Wouters wrote: >>> This isn't a new bug, but it's exposed by always importing weakref and >>> atexit during interpreter startup. I'm wondering if that's really necessary >>> :) >> >> In short, the overall answer right now is, "maybe", and the answer >> later is "rather likely". ;-) > > I would rather say that it's "rather certain". > > functools is necessary for setup.py to work while bootstrapping, whereas > pkgutil is used in runpy.py which is always imported in Modules/main.c. > > So we're left with having to fix atexit to support subinterpreters. I wonder > how difficult that will be. If the problem really has to do with interpreter startup, then there actually is a workaround possible, at the cost of slightly hairier code. If dispatch() looked in the registry *first* and avoided the cache in that case, and lazily created the cache (including the weakref import), then interpreter startup would not trigger an import of weakref in the default case. (Of course, depending on whether site/sitecustomize results in the use of importer subclasses and such, this might not help. It might be necessary to take the even more complex tack of avoiding the use of a cache entirely until an ABC is registered, and walking mro's.) Anyway, it remains to be seen whether these workarounds are easier or more difficult than fixing the atexit problem. ;-) Hmm... actually, there are a couple other ways around this. singledispatch doesn't use finalize(), so it doesn't really need atexit. It doesn't even do much with WeakKeyDictionary, so it could actually just "from _weakref import ref", and inline the relevant operations. Or, WeakKeyDictionary could be pulled out into a separate module, where singledispatch could pull it from without importing finalize. Or, weakref.finalize could be fixed so that the atexit import and register() are deferred until actual use. (Of all of these, that last one actually sounds like the least invasive workaround, with fewest lines of code likely to be changed.) From stephen at xemacs.org Sat Jun 8 07:18:15 2013 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Sat, 08 Jun 2013 14:18:15 +0900 Subject: [Python-Dev] doctest and pickle In-Reply-To: <51B204F7.4030107@stoneleaf.us> References: <51B204F7.4030107@stoneleaf.us> Message-ID: <87a9n19o4o.fsf@uwakimon.sk.tsukuba.ac.jp> Ethan Furman writes: > Enumerations can be pickled and unpickled:: > > >>> from enum import Enum > >>> class Fruit(Enum): > ... tomato = 1 > ... banana = 2 > ... cherry = 3 > ... > >>> from pickle import dumps, loads > >>> Fruit.tomato is loads(dumps(Fruit.tomato)) > True > [...] > Still, it would be nice if this could work. Well, you could cheat and reverse the test. ;-) I assume the problem is that loads proceeds to recreate the Fruit enum, rather than checking if there already is one? Maybe the metaclass can check somehow? At the very least, if this can't work in this implementation, unpickling should be an error. From steve at pearwood.info Sat Jun 8 08:45:49 2013 From: steve at pearwood.info (Steven D'Aprano) Date: Sat, 08 Jun 2013 16:45:49 +1000 Subject: [Python-Dev] doctest and pickle In-Reply-To: <87a9n19o4o.fsf@uwakimon.sk.tsukuba.ac.jp> References: <51B204F7.4030107@stoneleaf.us> <87a9n19o4o.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <51B2D31D.9080106@pearwood.info> On 08/06/13 15:18, Stephen J. Turnbull wrote: > Ethan Furman writes: > > > Enumerations can be pickled and unpickled:: > > > > >>> from enum import Enum > > >>> class Fruit(Enum): > > ... tomato = 1 > > ... banana = 2 > > ... cherry = 3 > > ... > > >>> from pickle import dumps, loads > > >>> Fruit.tomato is loads(dumps(Fruit.tomato)) > > True > > [...] > > Still, it would be nice if this could work. > > Well, you could cheat and reverse the test. ;-) > > I assume the problem is that loads proceeds to recreate the Fruit > enum, rather than checking if there already is one? I don't believe so. I understand that the problem is that pickle cannot find the Fruit enum in the __main__ module. Untested, but adding this before the call to dumps might work: import __main__ __main__.Fruit = Fruit although that's the sort of thing that makes me think it's time to turn this into a unittest rather than a doctest. -- Steven From ethan at stoneleaf.us Sat Jun 8 09:03:14 2013 From: ethan at stoneleaf.us (Ethan Furman) Date: Sat, 08 Jun 2013 00:03:14 -0700 Subject: [Python-Dev] doctest and pickle In-Reply-To: <51B2D31D.9080106@pearwood.info> References: <51B204F7.4030107@stoneleaf.us> <87a9n19o4o.fsf@uwakimon.sk.tsukuba.ac.jp> <51B2D31D.9080106@pearwood.info> Message-ID: <51B2D732.40405@stoneleaf.us> On 06/07/2013 11:45 PM, Steven D'Aprano wrote: > On 08/06/13 15:18, Stephen J. Turnbull wrote: >> Ethan Furman writes: >> >> > Enumerations can be pickled and unpickled:: >> > >> > >>> from enum import Enum >> > >>> class Fruit(Enum): >> > ... tomato = 1 >> > ... banana = 2 >> > ... cherry = 3 >> > ... >> > >>> from pickle import dumps, loads >> > >>> Fruit.tomato is loads(dumps(Fruit.tomato)) >> > True >> > [...] >> > Still, it would be nice if this could work. >> >> Well, you could cheat and reverse the test. ;-) >> >> I assume the problem is that loads proceeds to recreate the Fruit >> enum, rather than checking if there already is one? > > > I don't believe so. I understand that the problem is that pickle cannot find the Fruit enum in the __main__ module. > > Untested, but adding this before the call to dumps might work: > > import __main__ > __main__.Fruit = Fruit > > > although that's the sort of thing that makes me think it's time to turn this into a unittest rather than a doctest. Indeed, and it is already in several different ways. But it would be nice to have a pickle example in the docs that worked with doctest. I ended up doing what Barry did: >>> from test.test_enum import Fruit >>> from pickle import dumps, loads >>> Fruit.tomato is loads(dumps(Fruit.tomato)) True -- ~Ethan~ From storchaka at gmail.com Sat Jun 8 09:45:45 2013 From: storchaka at gmail.com (Serhiy Storchaka) Date: Sat, 08 Jun 2013 10:45:45 +0300 Subject: [Python-Dev] Obsoleted RFCs Message-ID: Here is attached a list of obsoleted RFCs referred in the *.rst, *.txt, *.py, *.c and *.h files. I think it would be worthwhile to update the source code and documentation for more modern RFCs. For example for updating RFC3548 to RFC4648 there is an issue #16995. -------------- next part -------------- 821: Simple Mail Transfer Protocol. (Obsoleted by RFC2821) Lib/smtpd.py Lib/smtplib.py 822: STANDARD FOR THE FORMAT OF ARPA INTERNET TEXT MESSAGES. (Obsoleted by RFC2822) Doc/library/email-examples.rst Doc/library/email.rst Doc/library/imaplib.rst Lib/configparser.py Lib/email/_header_value_parser.py Lib/email/_parseaddr.py Lib/email/header.py Lib/imaplib.py Lib/test/test_email/data/msg_16.txt Lib/test/test_email/test_email.py Lib/test/test_http_cookiejar.py 850: Standard for interchange of USENET messages. (Obsoleted by RFC1036) Lib/email/_parseaddr.py 977: Network News Transfer Protocol. (Obsoleted by RFC3977) Lib/nntplib.py Lib/test/test_nntplib.py 1036: Standard for interchange of USENET messages. (Obsoleted by RFC5536, RFC5537) rfcuse.txt 1521: MIME (Multipurpose Internet Mail Extensions) Part One: Mechanisms for Specifying and Describing the Format of Internet Message Bodies. (Obsoleted by RFC2045, RFC2046, RFC2047, RFC2048, RFC2049) Lib/base64.py Lib/quopri.py 1522: MIME (Multipurpose Internet Mail Extensions) Part Two: Message Header Extensions for Non-ASCII Text. (Obsoleted by RFC2045, RFC2046, RFC2047, RFC2048, RFC2049) Doc/library/binascii.rst Lib/quopri.py 1738: Uniform Resource Locators (URL). (Obsoleted by RFC4248, RFC4266) Lib/urllib/parse.py 1750: Randomness Recommendations for Security. (Obsoleted by RFC4086) Doc/library/ssl.rst Modules/_ssl.c 1766: Tags for the Identification of Languages. (Obsoleted by RFC3066, RFC3282) Lib/locale.py 1808: Relative Uniform Resource Locators. (Obsoleted by RFC3986) Lib/test/test_urlparse.py Lib/urllib/parse.py 1869: SMTP Service Extensions. (Obsoleted by RFC2821) Lib/smtpd.py Lib/smtplib.py 1894: An Extensible Message Format for Delivery Status Notifications. (Obsoleted by RFC3464) Lib/test/test_email/test_email.py 2048: Multipurpose Internet Mail Extensions (MIME) Part Four: Registration Procedures. (Obsoleted by RFC4288, RFC4289) rfcuse.txt 2060: Internet Message Access Protocol - Version 4rev1. (Obsoleted by RFC3501) Lib/imaplib.py 2068: Hypertext Transfer Protocol -- HTTP/1.1. (Obsoleted by RFC2616) Lib/http/cookies.py Lib/urllib/request.py 2069: An Extension to HTTP : Digest Access Authentication. (Obsoleted by RFC2617) Lib/urllib/request.py 2070: Internationalization of the Hypertext Markup Language. (Obsoleted by RFC2854) Lib/html/entities.py 2109: HTTP State Management Mechanism. (Obsoleted by RFC2965) Doc/library/http.cookiejar.rst Lib/http/cookiejar.py Lib/http/cookies.py Lib/test/test_http_cookiejar.py 2133: Basic Socket Interface Extensions for IPv6. (Obsoleted by RFC2553) Modules/getaddrinfo.c Modules/getnameinfo.c 2292: Advanced Sockets API for IPv6. (Obsoleted by RFC3542) Modules/socketmodule.c 2373: IP Version 6 Addressing Architecture. (Obsoleted by RFC3513) Lib/ipaddress.py 2396: Uniform Resource Identifiers (URI): Generic Syntax. (Obsoleted by RFC3986) Lib/http/cookiejar.py Lib/test/test_urllib.py Lib/test/test_urlparse.py Lib/urllib/parse.py 2434: Guidelines for Writing an IANA Considerations Section in RFCs. (Obsoleted by RFC5226) rfc3454.txt 2440: OpenPGP Message Format. (Obsoleted by RFC4880) Lib/test/test_email/data/msg_45.txt 2487: SMTP Service Extension for Secure SMTP over TLS. (Obsoleted by RFC3207) Lib/smtplib.py 2518: HTTP Extensions for Distributed Authoring -- WEBDAV. (Obsoleted by RFC4918) Doc/library/http.client.rst 2553: Basic Socket Interface Extensions for IPv6. (Obsoleted by RFC3493) Modules/addrinfo.h Modules/socketmodule.c rfcuse.txt 2554: SMTP Service Extension for Authentication. (Obsoleted by RFC4954) Lib/smtplib.py 2718: Guidelines for new URL Schemes. (Obsoleted by RFC4395) Lib/http/cookiejar.py 2732: Format for Literal IPv6 Addresses in URL's. (Obsoleted by RFC3986) Lib/test/test_urlparse.py Lib/urllib/parse.py 2821: Simple Mail Transfer Protocol. (Obsoleted by RFC5321) Lib/smtplib.py rfcuse.txt 2822: Internet Message Format. (Obsoleted by RFC5322) Doc/tutorial/stdlib.rst Lib/email/_header_value_parser.py Lib/email/_parseaddr.py Lib/email/feedparser.py Lib/email/generator.py Lib/email/header.py Lib/email/message.py Lib/email/mime/message.py Lib/email/parser.py Lib/email/utils.py Lib/http/client.py Lib/smtplib.py Lib/test/test_email/data/msg_35.txt Lib/test/test_email/test_email.py rfcuse.txt 3066: Tags for the Identification of Languages. (Obsoleted by RFC4646, RFC4647) rfcuse.txt 3171: IANA Guidelines for IPv4 Multicast Address Assignments. (Obsoleted by RFC5771) Lib/ipaddress.py 3280: Internet X.509 Public Key Infrastructure Certificate and Certificate Revocation List (CRL) Profile. (Obsoleted by RFC5280) Doc/library/ssl.rst 3330: Special-Use IPv4 Addresses. (Obsoleted by RFC5735) Lib/ipaddress.py 3513: Internet Protocol Version 6 (IPv6) Addressing Architecture. (Obsoleted by RFC4291) Lib/ipaddress.py rfcuse.txt 3548: The Base16, Base32, and Base64 Data Encodings. (Obsoleted by RFC4648) Doc/library/base64.rst Doc/whatsnew/2.4.rst Lib/base64.py 4366: Transport Layer Security (TLS) Extensions. (Obsoleted by RFC5246) Doc/library/ssl.rst From storchaka at gmail.com Sat Jun 8 10:07:11 2013 From: storchaka at gmail.com (Serhiy Storchaka) Date: Sat, 08 Jun 2013 11:07:11 +0300 Subject: [Python-Dev] doctest and pickle In-Reply-To: <51B2D732.40405@stoneleaf.us> References: <51B204F7.4030107@stoneleaf.us> <87a9n19o4o.fsf@uwakimon.sk.tsukuba.ac.jp> <51B2D31D.9080106@pearwood.info> <51B2D732.40405@stoneleaf.us> Message-ID: 08.06.13 10:03, Ethan Furman ???????(??): > Indeed, and it is already in several different ways. But it would be > nice to have a pickle example in the docs that worked with doctest. > > I ended up doing what Barry did: > > >>> from test.test_enum import Fruit > >>> from pickle import dumps, loads > >>> Fruit.tomato is loads(dumps(Fruit.tomato)) > True I think that the documentation is there for people. If you need tests, add them separately, but the documentation should be clear and understandable. In this case it is better to exclude a code example from doctests or add auxiliary code (i.e. as Steven suggested) to pass the doctest. From benjamin at python.org Sat Jun 8 10:23:25 2013 From: benjamin at python.org (Benjamin Peterson) Date: Sat, 8 Jun 2013 01:23:25 -0700 Subject: [Python-Dev] Obsoleted RFCs In-Reply-To: References: Message-ID: 2013/6/8 Serhiy Storchaka : > Here is attached a list of obsoleted RFCs referred in the *.rst, *.txt, > *.py, *.c and *.h files. I think it would be worthwhile to update the source > code and documentation for more modern RFCs. Just because you change the reference, doesn't mean the code is automatically compliant with the updated RFC. :) > > For example for updating RFC3548 to RFC4648 there is an issue #16995. -- Regards, Benjamin From mal at egenix.com Sat Jun 8 10:42:21 2013 From: mal at egenix.com (M.-A. Lemburg) Date: Sat, 08 Jun 2013 10:42:21 +0200 Subject: [Python-Dev] Obsoleted RFCs In-Reply-To: References: Message-ID: <51B2EE6D.2020101@egenix.com> On 08.06.2013 09:45, Serhiy Storchaka wrote: > Here is attached a list of obsoleted RFCs referred in the *.rst, *.txt, *.py, *.c and *.h files. I > think it would be worthwhile to update the source code and documentation for more modern RFCs. Thanks for creating such a list. BTW: What is rfcuse.txt that's mentioned several times in the list ? > For example for updating RFC3548 to RFC4648 there is an issue #16995. Given that more recent RFCs tend to introduce new functionality and sometimes backwards incompatible changes, I think each RFC update would need to be handled in a separate ticket. Some updates could probably be done in one go, e.g. RFC 821 -> 2821 -> 5321 -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jun 08 2013) >>> Python Projects, Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ 2013-07-01: EuroPython 2013, Florence, Italy ... 23 days to go 2013-07-16: Python Meeting Duesseldorf ... 38 days to go ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From storchaka at gmail.com Sat Jun 8 10:59:03 2013 From: storchaka at gmail.com (Serhiy Storchaka) Date: Sat, 08 Jun 2013 11:59:03 +0300 Subject: [Python-Dev] Obsoleted RFCs In-Reply-To: References: Message-ID: 08.06.13 11:23, Benjamin Peterson ???????(??): > 2013/6/8 Serhiy Storchaka : >> Here is attached a list of obsoleted RFCs referred in the *.rst, *.txt, >> *.py, *.c and *.h files. I think it would be worthwhile to update the source >> code and documentation for more modern RFCs. > > Just because you change the reference, doesn't mean the code is > automatically compliant with the updated RFC. :) Of course. Maintainers should review his modules and conclude what should be made for supporting more modern RFCs. I'm surprised that even new ipaddress module uses obsoleted RFC. From ethan at stoneleaf.us Sat Jun 8 10:47:35 2013 From: ethan at stoneleaf.us (Ethan Furman) Date: Sat, 08 Jun 2013 01:47:35 -0700 Subject: [Python-Dev] doctest and pickle In-Reply-To: References: <51B204F7.4030107@stoneleaf.us> <87a9n19o4o.fsf@uwakimon.sk.tsukuba.ac.jp> <51B2D31D.9080106@pearwood.info> <51B2D732.40405@stoneleaf.us> Message-ID: <51B2EFA7.6060703@stoneleaf.us> On 06/08/2013 01:07 AM, Serhiy Storchaka wrote: > 08.06.13 10:03, Ethan Furman ???????(??): >> Indeed, and it is already in several different ways. But it would be >> nice to have a pickle example in the docs that worked with doctest. >> >> I ended up doing what Barry did: >> >> >>> from test.test_enum import Fruit >> >>> from pickle import dumps, loads >> >>> Fruit.tomato is loads(dumps(Fruit.tomato)) >> True > > I think that the documentation is there for people. I agree. >If you need tests, add them separately, I have. > but the documentation should be clear and understandable. And the example code should be testable. > In this case it is better to exclude a code example from doctests or > add auxiliary code (i.e. as Steven suggested) to pass the doctest. Are you saying there is something wrong about what I have in place now? I would think that one line showing something you might actually do (importing an Enum from another module) is better than two lines showing esoteric workarounds (importing __main__ and setting an attribute on it). Apologies if I sound gruff -- it's way past my bedtime. In fact, I'll think I'll go sleep now. :) -- ~Ethan~ From storchaka at gmail.com Sat Jun 8 11:51:23 2013 From: storchaka at gmail.com (Serhiy Storchaka) Date: Sat, 08 Jun 2013 12:51:23 +0300 Subject: [Python-Dev] Obsoleted RFCs In-Reply-To: <51B2EE6D.2020101@egenix.com> References: <51B2EE6D.2020101@egenix.com> Message-ID: 08.06.13 11:42, M.-A. Lemburg ???????(??): > On 08.06.2013 09:45, Serhiy Storchaka wrote: >> Here is attached a list of obsoleted RFCs referred in the *.rst, *.txt, *.py, *.c and *.h files. I >> think it would be worthwhile to update the source code and documentation for more modern RFCs. > > Thanks for creating such a list. > > BTW: What is rfcuse.txt that's mentioned several times in the list ? Oh, sorry. It is here by mistake. Just ignore it. >> For example for updating RFC3548 to RFC4648 there is an issue #16995. > > Given that more recent RFCs tend to introduce new functionality and > sometimes backwards incompatible changes, I think each RFC update would > need to be handled in a separate ticket. > > Some updates could probably be done in one go, e.g. RFC 821 -> 2821 -> > 5321 Of course. This list is only a start point. From storchaka at gmail.com Sat Jun 8 12:00:27 2013 From: storchaka at gmail.com (Serhiy Storchaka) Date: Sat, 08 Jun 2013 13:00:27 +0300 Subject: [Python-Dev] Obsoleted RFCs In-Reply-To: References: Message-ID: By mistake some local files were added to the list. Here's an updated list. It now also contains low-case references. Attached also a script used to generate this list. -------------- next part -------------- 3: Documentation conventions. (Obsoleted by RFC0010) Lib/test/math_testcases.txt 10: Documentation conventions. (Obsoleted by RFC0016) Lib/test/math_testcases.txt 11: Implementation of the Host - Host Software Procedures in GORDO. (Obsoleted by RFC0033) Lib/test/math_testcases.txt 821: Simple Mail Transfer Protocol. (Obsoleted by RFC2821) Lib/smtpd.py Lib/smtplib.py 822: STANDARD FOR THE FORMAT OF ARPA INTERNET TEXT MESSAGES. (Obsoleted by RFC2822) Doc/distutils/apiref.rst Doc/library/email-examples.rst Doc/library/email.iterators.rst Doc/library/email.message.rst Doc/library/email.mime.rst Doc/library/email.parser.rst Doc/library/email.rst Doc/library/imaplib.rst Doc/whatsnew/2.2.rst Doc/whatsnew/2.4.rst Lib/configparser.py Lib/distutils/dist.py Lib/distutils/tests/test_util.py Lib/distutils/util.py Lib/email/_header_value_parser.py Lib/email/_parseaddr.py Lib/email/feedparser.py Lib/email/generator.py Lib/email/header.py Lib/email/message.py Lib/email/mime/message.py Lib/email/utils.py Lib/imaplib.py Lib/mimetypes.py Lib/test/test_email/data/msg_05.txt Lib/test/test_email/data/msg_06.txt Lib/test/test_email/data/msg_11.txt Lib/test/test_email/data/msg_16.txt Lib/test/test_email/data/msg_25.txt Lib/test/test_email/data/msg_28.txt Lib/test/test_email/data/msg_42.txt Lib/test/test_email/data/msg_43.txt Lib/test/test_email/data/msg_46.txt Lib/test/test_email/test_defect_handling.py Lib/test/test_email/test_email.py Lib/test/test_email/torture_test.py Lib/test/test_http_cookiejar.py Tools/scripts/mailerdaemon.py 850: Standard for interchange of USENET messages. (Obsoleted by RFC1036) Lib/email/_parseaddr.py Lib/http/cookiejar.py Lib/test/test_http_cookiejar.py 931: Authentication server. (Obsoleted by RFC1413) Lib/http/server.py 977: Network News Transfer Protocol. (Obsoleted by RFC3977) Lib/nntplib.py Lib/test/test_nntplib.py 1521: MIME (Multipurpose Internet Mail Extensions) Part One: Mechanisms for Specifying and Describing the Format of Internet Message Bodies. (Obsoleted by RFC2045, RFC2046, RFC2047, RFC2048, RFC2049) Lib/base64.py Lib/quopri.py Modules/binascii.c 1522: MIME (Multipurpose Internet Mail Extensions) Part Two: Message Header Extensions for Non-ASCII Text. (Obsoleted by RFC2045, RFC2046, RFC2047, RFC2048, RFC2049) Doc/library/binascii.rst Lib/quopri.py 1738: Uniform Resource Locators (URL). (Obsoleted by RFC4248, RFC4266) Lib/urllib/parse.py 1750: Randomness Recommendations for Security. (Obsoleted by RFC4086) Doc/library/ssl.rst Modules/_ssl.c 1766: Tags for the Identification of Languages. (Obsoleted by RFC3066, RFC3282) Lib/locale.py 1808: Relative Uniform Resource Locators. (Obsoleted by RFC3986) Lib/test/test_urlparse.py Lib/urllib/parse.py 1869: SMTP Service Extensions. (Obsoleted by RFC2821) Lib/smtpd.py Lib/smtplib.py 1894: An Extensible Message Format for Delivery Status Notifications. (Obsoleted by RFC3464) Lib/test/test_email/test_email.py 2060: Internet Message Access Protocol - Version 4rev1. (Obsoleted by RFC3501) Lib/imaplib.py 2068: Hypertext Transfer Protocol -- HTTP/1.1. (Obsoleted by RFC2616) Lib/http/cookies.py Lib/urllib/request.py 2069: An Extension to HTTP : Digest Access Authentication. (Obsoleted by RFC2617) Lib/urllib/request.py 2070: Internationalization of the Hypertext Markup Language. (Obsoleted by RFC2854) Lib/html/entities.py 2109: HTTP State Management Mechanism. (Obsoleted by RFC2965) Doc/library/http.cookiejar.rst Lib/http/cookiejar.py Lib/http/cookies.py Lib/test/test_http_cookiejar.py 2133: Basic Socket Interface Extensions for IPv6. (Obsoleted by RFC2553) Modules/getaddrinfo.c Modules/getnameinfo.c 2279: UTF-8, a transformation format of ISO 10646. (Obsoleted by RFC3629) Lib/test/test_unicode.py 2292: Advanced Sockets API for IPv6. (Obsoleted by RFC3542) Modules/socketmodule.c 2368: The mailto URL scheme. (Obsoleted by RFC6068) Lib/test/test_urlparse.py Lib/urllib/parse.py 2373: IP Version 6 Addressing Architecture. (Obsoleted by RFC3513) Lib/ipaddress.py 2396: Uniform Resource Identifiers (URI): Generic Syntax. (Obsoleted by RFC3986) Lib/http/cookiejar.py Lib/test/test_urllib.py Lib/test/test_urlparse.py Lib/urllib/parse.py 2440: OpenPGP Message Format. (Obsoleted by RFC4880) Lib/test/test_email/data/msg_45.txt 2487: SMTP Service Extension for Secure SMTP over TLS. (Obsoleted by RFC3207) Lib/smtplib.py 2518: HTTP Extensions for Distributed Authoring -- WEBDAV. (Obsoleted by RFC4918) Doc/library/http.client.rst 2553: Basic Socket Interface Extensions for IPv6. (Obsoleted by RFC3493) Modules/addrinfo.h Modules/socketmodule.c 2554: SMTP Service Extension for Authentication. (Obsoleted by RFC4954) Lib/smtplib.py 2718: Guidelines for new URL Schemes. (Obsoleted by RFC4395) Lib/http/cookiejar.py 2732: Format for Literal IPv6 Addresses in URL's. (Obsoleted by RFC3986) Lib/test/test_urlparse.py Lib/urllib/parse.py 2821: Simple Mail Transfer Protocol. (Obsoleted by RFC5321) Lib/smtplib.py 2822: Internet Message Format. (Obsoleted by RFC5322) Doc/tutorial/stdlib.rst Doc/whatsnew/2.2.rst Lib/email/_header_value_parser.py Lib/email/_parseaddr.py Lib/email/feedparser.py Lib/email/generator.py Lib/email/header.py Lib/email/message.py Lib/email/mime/message.py Lib/email/parser.py Lib/email/utils.py Lib/http/client.py Lib/smtplib.py Lib/test/test_email/data/msg_35.txt Lib/test/test_email/test_email.py 2965: HTTP State Management Mechanism. (Obsoleted by RFC6265) Doc/library/http.cookiejar.rst Doc/library/urllib.request.rst Lib/http/cookiejar.py Lib/test/test_http_cookiejar.py Lib/urllib/request.py 3171: IANA Guidelines for IPv4 Multicast Address Assignments. (Obsoleted by RFC5771) Lib/ipaddress.py 3280: Internet X.509 Public Key Infrastructure Certificate and Certificate Revocation List (CRL) Profile. (Obsoleted by RFC5280) Doc/library/ssl.rst 3330: Special-Use IPv4 Addresses. (Obsoleted by RFC5735) Lib/ipaddress.py 3490: Internationalizing Domain Names in Applications (IDNA). (Obsoleted by RFC5890, RFC5891) Doc/library/codecs.rst 3513: Internet Protocol Version 6 (IPv6) Addressing Architecture. (Obsoleted by RFC4291) Lib/ipaddress.py 3548: The Base16, Base32, and Base64 Data Encodings. (Obsoleted by RFC4648) Doc/library/base64.rst Doc/whatsnew/2.4.rst Lib/base64.py 4366: Transport Layer Security (TLS) Extensions. (Obsoleted by RFC5246, RFC6066) Doc/library/ssl.rst 5735: Special Use IPv4 Addresses. (Obsoleted by RFC6890) Lib/ipaddress.py -------------- next part -------------- A non-text attachment was scrubbed... Name: rfc_obsoleted.py Type: text/x-python Size: 985 bytes Desc: not available URL: From storchaka at gmail.com Sat Jun 8 12:09:21 2013 From: storchaka at gmail.com (Serhiy Storchaka) Date: Sat, 08 Jun 2013 13:09:21 +0300 Subject: [Python-Dev] doctest and pickle In-Reply-To: <51B2EFA7.6060703@stoneleaf.us> References: <51B204F7.4030107@stoneleaf.us> <87a9n19o4o.fsf@uwakimon.sk.tsukuba.ac.jp> <51B2D31D.9080106@pearwood.info> <51B2D732.40405@stoneleaf.us> <51B2EFA7.6060703@stoneleaf.us> Message-ID: 08.06.13 11:47, Ethan Furman ???????(??): >> In this case it is better to exclude a code example from doctests or >> add auxiliary code (i.e. as Steven suggested) to pass the doctest. > > Are you saying there is something wrong about what I have in place now? > I would think that one line showing something you might actually do > (importing an Enum from another module) is better than two lines showing > esoteric workarounds (importing __main__ and setting an attribute on it). test.test_enum is not here. The reader should look into the external test module (which may not be supplied along with the module and documentation) to understand the example. Or rely on assumptions. Is it possible to add "invisible" code which doesn't displayed in the resulting documentation, but taken into account by doctest? From stefan at drees.name Sat Jun 8 12:36:45 2013 From: stefan at drees.name (Stefan Drees) Date: Sat, 08 Jun 2013 12:36:45 +0200 Subject: [Python-Dev] Obsoleted RFCs In-Reply-To: References: Message-ID: <51B3093D.6060001@drees.name> On 2013-06-08 10:59, Serhiy Storchaka wrote: > 08.06.13 11:23, Benjamin Peterson ... : >> 2013/6/8 Serhiy Storchaka ... : >>> Here is attached a list of obsoleted RFCs referred in the *.rst, *.txt, >>> *.py, *.c and *.h files. I think it would be worthwhile to update the >>> source >>> code and documentation for more modern RFCs. >> >> Just because you change the reference, doesn't mean the code is >> automatically compliant with the updated RFC. :) > > Of course. Maintainers should review his modules and conclude what > should be made for supporting more modern RFCs. > > I'm surprised that even new ipaddress module uses obsoleted RFC. I for one am not :-) Let's pick that one ... as it seems to depend on the perspective chosen (web docs vs. python code comments) (cf. Details) Summary: Besides the magic, that lies in the process that transforms the documentation inside the python source into the web presented form, it should be easy and semantically correct, to simply replace the two occurences of "RFC 3513 2.5.6" with "RFC 4291 2.5.7". Details: The current web doc (3.4.0a0) at http://docs.python.org/dev/library/ipaddress#ipaddress.IPv6Address only shows this (no RFC3513 in sight): """ The following constitutes a valid IPv6 address: 1. A string consisting of ... See RFC 4291 for details. ... """ the current source at Lib/ipaddress.py you checked has: # --- 8< ----- ------ ------- ------ ---- ------- ----- ------ ---- class IPv6Network(_BaseV6, _BaseNetwork): # ... snipped methods, reformatted, shortened ... @property def is_site_local(self): """Test if the address is reserved for site-local. Note that the site-local address space has been deprecated by RFC 3879. Use is_private to test if this address is in the space of unique local addresses as defined by RFC 4193. Returns: A boolean, True if the ad... is res... per RFC 3513 2.5.6. """ sitelocal_network = IPv6Network('fec0::/10') return self in sitelocal_network # --- 8< ----- ------ ------- ------ ---- ------- ----- ------ ---- and later: # --- 8< ----- ------ ------- ------ ---- ------- ----- ------ ---- class IPv6Network(_BaseV6, _BaseNetwork): # ... again snipped methods, reformatted, shortened ... @property def is_site_local(self): """Test if the address is reserved for site-local. Note that the site-local address space has been deprecated by RFC 3879. Use is_private to test if this address is in the space of unique local addresses as defined by RFC 4193. Returns: A boolean, True if the ad... is res... per RFC 3513 2.5.6. """ return (self.network_address.is_site_local and self.broadcast_address.is_site_local) # --- 8< ----- ------ ------- ------ ---- ------- ----- ------ ---- The RFC 3513 (April 2003) has been obsoleted by RFC 4291 (February 2006) and the latter has been updated by RFC 5952 (August 2010) and RFC 6052 (October 2010). The given motivation for referencing inside the python source file is in both cases the detailed specification of what a reserved address actaully is. Looking at RFC 3513 has at section 2.5.6 (the referenced one): # --- 8< ----- ------ ------- ------ ---- ------- ----- ------ ---- 2.5.6 Local-Use IPv6 Unicast Addresses There are two types of local-use unicast addresses defined. These are Link-Local and Site-Local. The Link-Local is for use on a single link and the Site-Local is for use in a single site. Link-Local addresses have the following format: | 10 | | bits | 54 bits | 64 bits | +----------+-------------------------+----------------------------+ |1111111010| 0 | interface ID | +----------+-------------------------+----------------------------+ Link-Local addresses are designed to be used for addressing on a single link for purposes such as automatic address configuration, neighbor discovery, or when no routers are present. Routers must not forward any packets with link-local source or destination addresses to other links. Site-Local addresses have the following format: | 10 | | bits | 54 bits | 64 bits | +----------+-------------------------+----------------------------+ |1111111011| subnet ID | interface ID | +----------+-------------------------+----------------------------+ Site-local addresses are designed to be used for addressing inside of a site without the need for a global prefix. Although a subnet ID may be up to 54-bits long, it is expected that globally-connected sites will use the same subnet IDs for site-local and global prefixes. Routers must not forward any packets with site-local source or destination addresses outside of the site. # --- 8< ----- ------ ------- ------ ---- ------- ----- ------ ---- The obsoleting RFC 4291 has separated the description of Link-Local and Site-Local IPv6 Unicast adresses, thus if the section 2.5.6 reference for the is_site_local method was correct from the start - which seems to be a quite reasonable assumption - then the new section level reference should be "RFC 4291 2.5.7": # --- 8< ----- ------ ------- ------ ---- ------- ----- ------ ---- 2.5.7. Site-Local IPv6 Unicast Addresses Site-Local addresses were originally designed to be used for addressing inside of a site without the need for a global prefix. Site-local addresses are now deprecated as defined in [SLDEP]. Site-Local addresses have the following format: | 10 | | bits | 54 bits | 64 bits | +----------+-------------------------+----------------------------+ |1111111011| subnet ID | interface ID | +----------+-------------------------+----------------------------+ The special behavior of this prefix defined in [RFC3513] must no longer be supported in new implementations (i.e., new implementations must treat this prefix as Global Unicast). Existing implementations and deployments may continue to use this prefix. # --- 8< ----- ------ ------- ------ ---- ------- ----- ------ ---- But there seems to be a problem when actually using Site-Local addresses, as the new RFC (deviating from the obsoleted one) states (c.f. above): """ Site-local addresses are now deprecated as defined in [SLDEP]. """ Otherwise as a format reference all is well. The updateing RFCs seem to not interfere with the detail pointed to (by trusting the relation between title and content :-): [RFC5952] "A Recommendation for IPv6 Address Text Representation" [RFC6052] "IPv6 Addressing of IPv4/IPv6 Translators" All the best, Stefan. From rdmurray at bitdance.com Sat Jun 8 16:47:18 2013 From: rdmurray at bitdance.com (R. David Murray) Date: Sat, 08 Jun 2013 10:47:18 -0400 Subject: [Python-Dev] Obsoleted RFCs In-Reply-To: References: <51B2EE6D.2020101@egenix.com> Message-ID: <20130608144719.1B092250BF2@webabinitio.net> On Sat, 08 Jun 2013 12:51:23 +0300, Serhiy Storchaka wrote: > 08.06.13 11:42, M.-A. Lemburg ??????????????(????): > > On 08.06.2013 09:45, Serhiy Storchaka wrote: > >> Here is attached a list of obsoleted RFCs referred in the *.rst, *.txt, *.py, *.c and *.h files. I > >> think it would be worthwhile to update the source code and documentation for more modern RFCs. > > > > Given that more recent RFCs tend to introduce new functionality and > > sometimes backwards incompatible changes, I think each RFC update would > > need to be handled in a separate ticket. > > > > Some updates could probably be done in one go, e.g. RFC 821 -> 2821 -> > > 5321 > > Of course. This list is only a start point. We've already done some (most?) of the work for the smtplib update. A thorough review is needed to make sure there aren't any leftovers and to update any comments and docs that haven't been fixed yet. For the email package, the support for RFC 5322 is mostly done, in the new policies. The legacy policies will continue to only support the older RFCs formally, though in many cases the code is also conformant with the newer RFCs because the relevant details have not changed. --David From ethan at stoneleaf.us Sat Jun 8 17:41:00 2013 From: ethan at stoneleaf.us (Ethan Furman) Date: Sat, 08 Jun 2013 08:41:00 -0700 Subject: [Python-Dev] doctest and pickle In-Reply-To: References: <51B204F7.4030107@stoneleaf.us> <87a9n19o4o.fsf@uwakimon.sk.tsukuba.ac.jp> <51B2D31D.9080106@pearwood.info> <51B2D732.40405@stoneleaf.us> <51B2EFA7.6060703@stoneleaf.us> Message-ID: <51B3508C.5000905@stoneleaf.us> On 06/08/2013 03:09 AM, Serhiy Storchaka wrote: > 08.06.13 11:47, Ethan Furman ???????(??): >>> In this case it is better to exclude a code example from doctests or >>> add auxiliary code (i.e. as Steven suggested) to pass the doctest. >> >> Are you saying there is something wrong about what I have in place now? >> I would think that one line showing something you might actually do >> (importing an Enum from another module) is better than two lines showing >> esoteric workarounds (importing __main__ and setting an attribute on it). > > test.test_enum is not here. The reader should look into the external test module (which may not be supplied along with > the module and documentation) to understand the example. Or rely on assumptions. Fair point. But I suppose that if the end-user is running a doc test, it is not too much to require that the other tests be installed as well. Plus, we definitely have the other tests. :) > Is it possible to add "invisible" code which doesn't displayed in the resulting documentation, but taken into account by > doctest? I have no idea. This is my first time using doctest. -- ~Ethan~ From stephen at xemacs.org Sat Jun 8 18:18:47 2013 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Sun, 09 Jun 2013 01:18:47 +0900 Subject: [Python-Dev] Obsoleted RFCs In-Reply-To: References: Message-ID: <87wqq48tjs.fsf@uwakimon.sk.tsukuba.ac.jp> Serhiy Storchaka writes: > 08.06.13 11:23, Benjamin Peterson ???????(??): > > 2013/6/8 Serhiy Storchaka : > >> Here is attached a list of obsoleted RFCs referred in the *.rst, > >> *.txt, *.py, *.c and *.h files. I think it would be worthwhile > >> to update the source code and documentation for more modern > >> RFCs. > > > > Just because you change the reference, doesn't mean the code is > > automatically compliant with the updated RFC. :) > > Of course. Maintainers should review his modules and conclude what > should be made for supporting more modern RFCs. I suspect in many cases the answer is going to be "nothing". Grepping out the references and checking for obsoleted RFCs is useful information, of course. Good GSoC fodder, for one thing. But I'd be cautious about even creating an issue without consideration of each case individually. This can be a *lot* of work, for very little gain. In the case of mail, consider that STD 11 still points to RFC 822![1] Also, even the most modern RFC 5322 REQUIREs parsers to accept the "obsolete" syntax of section 4, which I believe is that of RFC 822. In any case, it's pretty close. So you wouldn't want to change the parser anyway. Whether it would be worth auditing the generative functions for 5322 conformance, and creating tests, is a more difficult question, but it still sounds like much work for little gain. The analysis is surely different for other RFCs, but for this particular series I see little harm in letting each component of the email module continue to explicitly target whichever of the RFCs happened to be current when its author started coding. Footnotes: [1] http://tools.ietf.org/html/std11 From olemis at gmail.com Sat Jun 8 19:17:41 2013 From: olemis at gmail.com (Olemis Lang) Date: Sat, 8 Jun 2013 12:17:41 -0500 Subject: [Python-Dev] doctest and pickle In-Reply-To: <51B3508C.5000905@stoneleaf.us> References: <51B204F7.4030107@stoneleaf.us> <87a9n19o4o.fsf@uwakimon.sk.tsukuba.ac.jp> <51B2D31D.9080106@pearwood.info> <51B2D732.40405@stoneleaf.us> <51B2EFA7.6060703@stoneleaf.us> <51B3508C.5000905@stoneleaf.us> Message-ID: On 6/8/13, Ethan Furman wrote: > On 06/08/2013 03:09 AM, Serhiy Storchaka wrote: >> 08.06.13 11:47, Ethan Furman ???????(??): [...] > > Fair point. But I suppose that if the end-user is running a doc test, it is > not too much to require that the other > tests be installed as well. Plus, we definitely have the other tests. :) > > >> Is it possible to add "invisible" code which doesn't displayed in the >> resulting documentation, but taken into account by >> doctest? > > I have no idea. This is my first time using doctest. > No ... if using doctest . To get such things done you'll need something like dutest [1]_ [2]_ , but that's not an option for testing docs in stdlib . .. [1] dutest @ PyPI (https://pypi.python.org/pypi/dutest?) .. [2] Purpose of Doctests [Was: Best practices for Enum] (http://goo.gl/F7Afy) -- Regards, Olemis. Apache? Bloodhound contributor http://issues.apache.org/bloodhound Blog ES: http://simelo-es.blogspot.com/ Blog EN: http://simelo-en.blogspot.com/ Featured article: From lrekucki at gmail.com Sat Jun 8 19:54:18 2013 From: lrekucki at gmail.com (=?UTF-8?Q?=C5=81ukasz_Rekucki?=) Date: Sat, 8 Jun 2013 19:54:18 +0200 Subject: [Python-Dev] doctest and pickle In-Reply-To: <51B3508C.5000905@stoneleaf.us> References: <51B204F7.4030107@stoneleaf.us> <87a9n19o4o.fsf@uwakimon.sk.tsukuba.ac.jp> <51B2D31D.9080106@pearwood.info> <51B2D732.40405@stoneleaf.us> <51B2EFA7.6060703@stoneleaf.us> <51B3508C.5000905@stoneleaf.us> Message-ID: On 8 June 2013 17:41, Ethan Furman wrote: > On 06/08/2013 03:09 AM, Serhiy Storchaka wrote: >> >> Is it possible to add "invisible" code which doesn't displayed in the >> resulting documentation, but taken into account by >> doctest? > > > I have no idea. This is my first time using doctest. > AFAIK, stdlib uses Sphinx, so you can provide a testsetup block[1]. At least if you're running them via Sphinx. [1]: http://sphinx-doc.org/ext/doctest.html -- ?ukasz Rekucki From rdmurray at bitdance.com Sat Jun 8 20:17:14 2013 From: rdmurray at bitdance.com (R. David Murray) Date: Sat, 08 Jun 2013 14:17:14 -0400 Subject: [Python-Dev] doctest and pickle In-Reply-To: References: <51B204F7.4030107@stoneleaf.us> <87a9n19o4o.fsf@uwakimon.sk.tsukuba.ac.jp> <51B2D31D.9080106@pearwood.info> <51B2D732.40405@stoneleaf.us> <51B2EFA7.6060703@stoneleaf.us> <51B3508C.5000905@stoneleaf.us> Message-ID: <20130608181714.B3112250BF3@webabinitio.net> On Sat, 08 Jun 2013 19:54:18 +0200, =?UTF-8?Q?=C5=81ukasz_Rekucki?= wrote: > On 8 June 2013 17:41, Ethan Furman wrote: > > On 06/08/2013 03:09 AM, Serhiy Storchaka wrote: > >> > >> Is it possible to add "invisible" code which doesn't displayed in the > >> resulting documentation, but taken into account by > >> doctest? > > > > > > I have no idea. This is my first time using doctest. > > > > AFAIK, stdlib uses Sphinx, so you can provide a testsetup block[1]. At > least if you're running them via Sphinx. Running the doctests via Sphinx is still, I believe a Future Objective :). I think we are getting close to being able to do that, though. (Well, you already can for Python2; but not, I think, for Python3). If the test module from which you are importing is short, you could use the Sphinx literalinclude directive to include the contents of the file in the docs just before you show the 'is' example. You could also explain (briefly) *why* it is necessary to use an imported class for this example, which IMO is useful information. --David From lukasz at langa.pl Sat Jun 8 23:49:22 2013 From: lukasz at langa.pl (=?utf-8?Q?=C5=81ukasz_Langa?=) Date: Sat, 8 Jun 2013 23:49:22 +0200 Subject: [Python-Dev] Unicode minus sign in numeric conversions Message-ID: Expected behaviour: >>> float('\N{MINUS SIGN}12.34') -12.34 Current behaviour: Traceback (most recent call last): ... ValueError: could not convert string to float: '?12.34' Please note: '\N{MINUS SIGN}' == '\u2212' -- Best regards, ?ukasz Langa WWW: http://lukasz.langa.pl/ Twitter: @llanga IRC: ambv on #python-dev From guido at python.org Sun Jun 9 00:30:52 2013 From: guido at python.org (Guido van Rossum) Date: Sat, 8 Jun 2013 15:30:52 -0700 Subject: [Python-Dev] Unicode minus sign in numeric conversions In-Reply-To: References: Message-ID: [Diverting to python-ideas, since this isn't as clear-cut as you think.] Why exactly is that expected behavior? What's the use case? (Surely you don't have a keyboard that generates \u2212 when you hit the minus key? :-) Is there a Unicode standard for parsing numbers? IIRC there are a variety of other things marked as "digits" in the Unicode standard -- do we do anything with those? If we do anything we should be consistent. For now, I think we *are* consistent -- we only support the ASCII representation of numbers. (And that's the only representation we generate as output as well -- think about symmetry too.) This page scares me: http://en.wikipedia.org/wiki/Numerals_in_Unicode --Guido On Sat, Jun 8, 2013 at 2:49 PM, ?ukasz Langa wrote: > Expected behaviour: >>>> float('\N{MINUS SIGN}12.34') > -12.34 > > > Current behaviour: > Traceback (most recent call last): > ... > ValueError: could not convert string to float: '?12.34' > > > Please note: '\N{MINUS SIGN}' == '\u2212' > > -- > Best regards, > ?ukasz Langa > > WWW: http://lukasz.langa.pl/ > Twitter: @llanga > IRC: ambv on #python-dev > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org -- --Guido van Rossum (python.org/~guido) From python at mrabarnett.plus.com Sun Jun 9 02:39:59 2013 From: python at mrabarnett.plus.com (MRAB) Date: Sun, 09 Jun 2013 01:39:59 +0100 Subject: [Python-Dev] Unicode minus sign in numeric conversions In-Reply-To: References: Message-ID: <51B3CEDF.4050106@mrabarnett.plus.com> On 08/06/2013 23:30, Guido van Rossum wrote: > [Diverting to python-ideas, since this isn't as clear-cut as you think.] > > Why exactly is that expected behavior? What's the use case? (Surely > you don't have a keyboard that generates \u2212 when you hit the minus > key? :-) > > Is there a Unicode standard for parsing numbers? IIRC there are a > variety of other things marked as "digits" in the Unicode standard -- > do we do anything with those? If we do anything we should be > consistent. For now, I think we *are* consistent -- we only support > the ASCII representation of numbers. (And that's the only > representation we generate as output as well -- think about symmetry > too.) > We already recognise at least some of the digits: >>> float("\N{ARABIC-INDIC DIGIT ONE}") 1.0 (I haven't check all of them!) > This page scares me: http://en.wikipedia.org/wiki/Numerals_in_Unicode > > --Guido > > On Sat, Jun 8, 2013 at 2:49 PM, ?ukasz Langa wrote: >> Expected behaviour: >>>>> float('\N{MINUS SIGN}12.34') >> -12.34 >> >> >> Current behaviour: >> Traceback (most recent call last): >> ... >> ValueError: could not convert string to float: '?12.34' >> >> >> Please note: '\N{MINUS SIGN}' == '\u2212' >> From rdmurray at bitdance.com Sun Jun 9 03:46:38 2013 From: rdmurray at bitdance.com (R. David Murray) Date: Sat, 08 Jun 2013 21:46:38 -0400 Subject: [Python-Dev] Unicode minus sign in numeric conversions In-Reply-To: <51B3CEDF.4050106@mrabarnett.plus.com> References: <51B3CEDF.4050106@mrabarnett.plus.com> Message-ID: <20130609014639.46E14250BF3@webabinitio.net> On Sun, 09 Jun 2013 01:39:59 +0100, MRAB wrote: > On 08/06/2013 23:30, Guido van Rossum wrote: > > [Diverting to python-ideas, since this isn't as clear-cut as you think.] > > > > Why exactly is that expected behavior? What's the use case? (Surely > > you don't have a keyboard that generates \u2212 when you hit the minus > > key? :-) > > > > Is there a Unicode standard for parsing numbers? IIRC there are a > > variety of other things marked as "digits" in the Unicode standard -- > > do we do anything with those? If we do anything we should be > > consistent. For now, I think we *are* consistent -- we only support > > the ASCII representation of numbers. (And that's the only > > representation we generate as output as well -- think about symmetry > > too.) > > > We already recognise at least some of the digits: > > >>> float("\N{ARABIC-INDIC DIGIT ONE}") > 1.0 > > (I haven't check all of them!) > > > This page scares me: http://en.wikipedia.org/wiki/Numerals_in_Unicode http://bugs.python.org/issue6632 contains a bunch of good information relevant to this discussion. It looks like the argument there was that there is no standard for the signs, therefore we should not support them. As Guido said, the issue is non-trivial. --David From ncoghlan at gmail.com Sun Jun 9 06:21:02 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 9 Jun 2013 14:21:02 +1000 Subject: [Python-Dev] doctest and pickle In-Reply-To: <20130608181714.B3112250BF3@webabinitio.net> References: <51B204F7.4030107@stoneleaf.us> <87a9n19o4o.fsf@uwakimon.sk.tsukuba.ac.jp> <51B2D31D.9080106@pearwood.info> <51B2D732.40405@stoneleaf.us> <51B2EFA7.6060703@stoneleaf.us> <51B3508C.5000905@stoneleaf.us> <20130608181714.B3112250BF3@webabinitio.net> Message-ID: On 9 June 2013 04:17, R. David Murray wrote: > On Sat, 08 Jun 2013 19:54:18 +0200, =?UTF-8?Q?=C5=81ukasz_Rekucki?= wrote: >> On 8 June 2013 17:41, Ethan Furman wrote: >> > On 06/08/2013 03:09 AM, Serhiy Storchaka wrote: >> >> >> >> Is it possible to add "invisible" code which doesn't displayed in the >> >> resulting documentation, but taken into account by >> >> doctest? >> > >> > >> > I have no idea. This is my first time using doctest. >> > >> >> AFAIK, stdlib uses Sphinx, so you can provide a testsetup block[1]. At >> least if you're running them via Sphinx. > > Running the doctests via Sphinx is still, I believe a Future Objective :). > I think we are getting close to being able to do that, though. (Well, > you already can for Python2; but not, I think, for Python3). > > If the test module from which you are importing is short, you could use > the Sphinx literalinclude directive to include the contents of the file > in the docs just before you show the 'is' example. You could also explain > (briefly) *why* it is necessary to use an imported class for this example, > which IMO is useful information. Right - for the singleton behaviour of enums to be honoured by pickle, the enum definition *must* be reachable through the import system based on its __name__ and __module__ attributes (PEP 3154 will eventually change that limitation to __qualname__ and __module__). The temporary namespaces created to run doctests generally aren't reachable that way, so the unpickling either fails or creates a new instance (depending on the details of the situation). Using the test suite in the enum docstrings initially is fine. In the future, once we migrate a module like socket to using enum.IntEnum instead of bare integers, it would be appropriate to change the enum docs to reference that rather than the test suite. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ethan at stoneleaf.us Sun Jun 9 06:47:29 2013 From: ethan at stoneleaf.us (Ethan Furman) Date: Sat, 08 Jun 2013 21:47:29 -0700 Subject: [Python-Dev] doctest and pickle In-Reply-To: References: <51B204F7.4030107@stoneleaf.us> <87a9n19o4o.fsf@uwakimon.sk.tsukuba.ac.jp> <51B2D31D.9080106@pearwood.info> <51B2D732.40405@stoneleaf.us> <51B2EFA7.6060703@stoneleaf.us> <51B3508C.5000905@stoneleaf.us> <20130608181714.B3112250BF3@webabinitio.net> Message-ID: <51B408E1.4090508@stoneleaf.us> On 06/08/2013 09:21 PM, Nick Coghlan wrote: > > Using the test suite in the enum docstrings initially is fine. In the > future, once we migrate a module like socket to using enum.IntEnum > instead of bare integers, it would be appropriate to change the enum > docs to reference that rather than the test suite. Good point. I'll keep that in mind. -- ~Ethan~ From alexander.belopolsky at gmail.com Mon Jun 10 05:13:43 2013 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Sun, 9 Jun 2013 23:13:43 -0400 Subject: [Python-Dev] Ordering keyword dicts In-Reply-To: References: Message-ID: On Sun, May 19, 2013 at 1:47 AM, Guido van Rossum wrote: > I'm slow at warming up to the idea. My main concern is speed -- since > most code doesn't need it and function calls are already slow (and > obviously very common :-) it would be a shame if this slowed down > function calls that don't need it noticeably. > Here is an idea that will not affect functions that don't need to know the order of keywords: a special __kworder__ local variable. The use of this variable inside the function will signal compiler to generate additional bytecode to copy keyword names from the stack to a tuple and save it in __kworder__. With that feature, an OrderedDict constructor, for example can be written as def odict(**kwargs): return OrderedDict([(key, kwargs[key]) for key in __kworder__]) -------------- next part -------------- An HTML attachment was scrubbed... URL: From ron3200 at gmail.com Mon Jun 10 17:08:46 2013 From: ron3200 at gmail.com (Ron Adam) Date: Mon, 10 Jun 2013 10:08:46 -0500 Subject: [Python-Dev] Ordering keyword dicts In-Reply-To: References: Message-ID: <51B5EBFE.7080605@gmail.com> On 06/09/2013 10:13 PM, Alexander Belopolsky wrote: > > On Sun, May 19, 2013 at 1:47 AM, Guido van Rossum > wrote: > > I'm slow at warming up to the idea. My main concern is speed -- since > most code doesn't need it and function calls are already slow (and > obviously very common :-) it would be a shame if this slowed down > function calls that don't need it noticeably. And we could remove this bit from the docs... """ The OrderedDict constructor and update() method both accept keyword arguments, but their order is lost because Python?s function call semantics pass-in keyword arguments using a regular unordered dictionary. """ > Here is an idea that will not affect functions that don't need to know the > order of keywords: a special __kworder__ local variable. The use of this > variable inside the function will signal compiler to generate additional > bytecode to copy keyword names from the stack to a tuple and save it in > __kworder__. I like the idea of an ordered dict acting the same as a regular dict with an optional way to access order. It also will catch uses of unordered dicts in situations where an ordered dict is expected by having the ordered key words accessed in a way that isn't available in unordered dicts. I don't care for a magic local variable inside of a function. > With that feature, an OrderedDict constructor, for example > can be written as > > def odict(**kwargs): > return OrderedDict([(key, kwargs[key]) for key in __kworder__]) There is two situations where this would matter... The packing of arguments when **kwargs is used in the function definition, and the unpacking of arguments when **kwargs is used in the function call. By having the unpackng side also work with ordered dicts, maybe we could then replace the many *very common* occurrences of (*args, **kwargs) with just (**all_args) or (***all_args). :) It's the unordered nature of dictionaries that requires *args to be used along with **kwargs when forwarding function arguments to other functions. Another variation of this is to have a way to forward a functions "signature name space" to another function directly and bypass the signature parsing those cases. Cheers, Ron From ron3200 at gmail.com Mon Jun 10 17:08:46 2013 From: ron3200 at gmail.com (Ron Adam) Date: Mon, 10 Jun 2013 10:08:46 -0500 Subject: [Python-Dev] Ordering keyword dicts In-Reply-To: References: Message-ID: <51B5EBFE.7080605@gmail.com> On 06/09/2013 10:13 PM, Alexander Belopolsky wrote: > > On Sun, May 19, 2013 at 1:47 AM, Guido van Rossum > wrote: > > I'm slow at warming up to the idea. My main concern is speed -- since > most code doesn't need it and function calls are already slow (and > obviously very common :-) it would be a shame if this slowed down > function calls that don't need it noticeably. And we could remove this bit from the docs... """ The OrderedDict constructor and update() method both accept keyword arguments, but their order is lost because Python?s function call semantics pass-in keyword arguments using a regular unordered dictionary. """ > Here is an idea that will not affect functions that don't need to know the > order of keywords: a special __kworder__ local variable. The use of this > variable inside the function will signal compiler to generate additional > bytecode to copy keyword names from the stack to a tuple and save it in > __kworder__. I like the idea of an ordered dict acting the same as a regular dict with an optional way to access order. It also will catch uses of unordered dicts in situations where an ordered dict is expected by having the ordered key words accessed in a way that isn't available in unordered dicts. I don't care for a magic local variable inside of a function. > With that feature, an OrderedDict constructor, for example > can be written as > > def odict(**kwargs): > return OrderedDict([(key, kwargs[key]) for key in __kworder__]) There is two situations where this would matter... The packing of arguments when **kwargs is used in the function definition, and the unpacking of arguments when **kwargs is used in the function call. By having the unpackng side also work with ordered dicts, maybe we could then replace the many *very common* occurrences of (*args, **kwargs) with just (**all_args) or (***all_args). :) It's the unordered nature of dictionaries that requires *args to be used along with **kwargs when forwarding function arguments to other functions. Another variation of this is to have a way to forward a functions "signature name space" to another function directly and bypass the signature parsing those cases. Cheers, Ron From ethan at stoneleaf.us Mon Jun 10 21:02:30 2013 From: ethan at stoneleaf.us (Ethan Furman) Date: Mon, 10 Jun 2013 12:02:30 -0700 Subject: [Python-Dev] Python 3 metaclasses, doctest, and unittest Message-ID: <51B622C6.7060209@stoneleaf.us> I just wanted to take a minute and say THANK YOU to everyone involved in getting Python 3 to where it is today. It is so much easier to use, especially metaclasses (nearly pulled my hair out trying to get Enum working on Py2!). Also a big thank you to the doctest folks as it was invaluable in getting my Py2 examples correct, and, of course, the unittest folks as I would have been lost without those. Thank you thank you thank you. -- ~Ethan~ P.S. I'll get a backported package on PyPI after the code commit for 3.4. From jas at corpus-callosum.com Tue Jun 11 07:57:57 2013 From: jas at corpus-callosum.com (Jeff Sickel) Date: Tue, 11 Jun 2013 00:57:57 -0500 Subject: [Python-Dev] new Plan 9 (and family) port of v2.7.5 Message-ID: I have a set of patches I've developed based on the 2.7 branch that provides a working port of Python 2.7.5 to Plan 9 386 and amd64 builds (an arm version is mostly working, but will need a few more updates before being hauled off on the wagon). Most of the changes, including mkfiles, are local to a Plan9 subdirectory. There are a few changes to Lib/*.py files specifically site.py and sysconfig.py to make living in a semi-POSIX world tolerable with many current Python modules. Because Plan 9 does not have shared libraries, we must link in all the C modules we want at build time. This means that I've included specific targets to pull in Mercurial sources during the compilation and linking phases. That brings us up to a current version of Mercurial 2.6.2. There is no ssh support for cloning, but we do have a viable version of https: using TLS/SSLv3 that has worked well with bitbucket. That said, I can prepare a patch from my mq work area and submit it upstream in one of various ways for review. Is there a preferred method that does not require an initial `hg clone ssh://hg at hg.python.org/cpython`? acme# pwd /usr/jas/src/cmd/cpython acme# hg sum parent: 83814:75a6bbf41cc8 plan9 qbase qtip update setup base install locations branch: 2.7 commit: 8 unknown (clean) update: 37 new changesets, 2 branch heads (merge) mq: 1 applied ac -jas From brett at python.org Tue Jun 11 19:25:09 2013 From: brett at python.org (Brett Cannon) Date: Tue, 11 Jun 2013 13:25:09 -0400 Subject: [Python-Dev] new Plan 9 (and family) port of v2.7.5 In-Reply-To: References: Message-ID: On Tue, Jun 11, 2013 at 1:57 AM, Jeff Sickel wrote: > I have a set of patches I've developed based on the 2.7 branch that > provides a working port of Python 2.7.5 to Plan 9 386 and amd64 builds (an > arm version is mostly working, but will need a few more updates before > being hauled off on the wagon). Most of the changes, including mkfiles, > are local to a Plan9 subdirectory. There are a few changes to Lib/*.py > files specifically site.py and sysconfig.py to make living in a semi-POSIX > world tolerable with many current Python modules. > > Because Plan 9 does not have shared libraries, we must link in all the C > modules we want at build time. This means that I've included specific > targets to pull in Mercurial sources during the compilation and linking > phases. That brings us up to a current version of Mercurial 2.6.2. There > is no ssh support for cloning, but we do have a viable version of https: > using TLS/SSLv3 that has worked well with bitbucket. That said, I can > prepare a patch from my mq work area and submit it upstream in one of > various ways for review. > We have moved away from trying to support non-mainstream OSs directly in the code base and prefer people to maintain a patched branch elsewhere. That being said, if any of the changes are reasonable in general feel free to submit patches at bugs.python.org. > > Is there a preferred method that does not require an initial `hg clone > ssh://hg at hg.python.org/cpython`? > > Not quite sure what you mean, but I assume you are asking about submitting patches, in which case you can use an HTTPS clone. SSH access is only for core developers to push changes to the main repo. -Brett > acme# pwd > /usr/jas/src/cmd/cpython > acme# hg sum > parent: 83814:75a6bbf41cc8 plan9 qbase qtip > update setup base install locations > branch: 2.7 > commit: 8 unknown (clean) > update: 37 new changesets, 2 branch heads (merge) > mq: 1 applied > ac > > > -jas > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/brett%40python.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: From skip at pobox.com Tue Jun 11 21:10:41 2013 From: skip at pobox.com (Skip Montanaro) Date: Tue, 11 Jun 2013 14:10:41 -0500 Subject: [Python-Dev] Doubly linked lists in Python core? Message-ID: I encountered this disconcerting message yesterday on a Linux system running Python 2.7.2: *** glibc detected *** /opt/local/bin/python: corrupted double-linked list: 0x0000000003b01c90 *** Of course, no core file or other information about where the problem occurred was left behind, just the raw pointer value. Running under valgrind didn't report anything obvious, at least nothing different than a run with slightly different command line parameters which didn't elicit the message. I suspect the problem actually lies in one of our C++ libraries here at work, but before I attempt any debugging gymnastics (ow! my back hurts) I thought I would check to see if Python makes much use of double linked lists in a non-debug build. I don't recall a lot from days of yore, perhaps some in the dict implementation? Thanks, Skip From benjamin at python.org Tue Jun 11 21:14:55 2013 From: benjamin at python.org (Benjamin Peterson) Date: Tue, 11 Jun 2013 12:14:55 -0700 Subject: [Python-Dev] Doubly linked lists in Python core? In-Reply-To: References: Message-ID: 2013/6/11 Skip Montanaro : > I encountered this disconcerting message yesterday on a Linux system > running Python 2.7.2: > > *** glibc detected *** /opt/local/bin/python: corrupted double-linked > list: 0x0000000003b01c90 *** I suspect that's a corrupt linked list interal to glibc. -- Regards, Benjamin From dmalcolm at redhat.com Tue Jun 11 21:38:47 2013 From: dmalcolm at redhat.com (David Malcolm) Date: Tue, 11 Jun 2013 15:38:47 -0400 Subject: [Python-Dev] Doubly linked lists in Python core? In-Reply-To: References: Message-ID: <1370979527.28429.39.camel@surprise> On Tue, 2013-06-11 at 12:14 -0700, Benjamin Peterson wrote: > 2013/6/11 Skip Montanaro : > > I encountered this disconcerting message yesterday on a Linux system > > running Python 2.7.2: > > > > *** glibc detected *** /opt/local/bin/python: corrupted double-linked > > list: 0x0000000003b01c90 *** > > I suspect that's a corrupt linked list interal to glibc. Yes: almost certainly the one inside glibc's implementation of malloc. Somewhere in the process you have a double-free, or a buffer overrun that's splatting the links that live in memory between the allocated bufffers. You may want to try running the process under valgrind. Hope this is helpful Dave From skip at pobox.com Tue Jun 11 21:49:06 2013 From: skip at pobox.com (Skip Montanaro) Date: Tue, 11 Jun 2013 14:49:06 -0500 Subject: [Python-Dev] Doubly linked lists in Python core? In-Reply-To: <1370979527.28429.39.camel@surprise> References: <1370979527.28429.39.camel@surprise> Message-ID: > You may want to try running the process under valgrind. Thanks. I'm trying that but have been so far unable to get valgrind to report any problems within Python or our libraries before that message, just a couple things at startup which seem to occur before Python gets going: ==21374== Invalid read of size 1 ==21374== at 0x4C2DEA9: putenv (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so) ==21374== by 0x400E71: append_env_vars (binary-wrapper-main.c:175) ==21374== by 0x4008CA: set_binary_wrapper_env (c-python-wrapper.c:58) ==21374== by 0x40168D: main (binary-wrapper-main.c:382) ==21374== Address 0x51d6a84 is 0 bytes after a block of size 52 alloc'd ==21374== at 0x4C2ACCE: realloc (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so) ==21374== by 0x400E61: append_env_vars (binary-wrapper-main.c:173) ==21374== by 0x4008CA: set_binary_wrapper_env (c-python-wrapper.c:58) ==21374== by 0x40168D: main (binary-wrapper-main.c:382) ==21374== ==21374== Syscall param execve(envp[i]) points to unaddressable byte(s) ==21374== at 0x4EE7067: execve (in /lib64/libc-2.15.so) ==21374== by 0x4016AA: main (binary-wrapper-main.c:385) ==21374== Address 0x51d6a84 is 0 bytes after a block of size 52 alloc'd ==21374== at 0x4C2ACCE: realloc (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so) ==21374== by 0x400E61: append_env_vars (binary-wrapper-main.c:173) ==21374== by 0x4008CA: set_binary_wrapper_env (c-python-wrapper.c:58) ==21374== by 0x40168D: main (binary-wrapper-main.c:382) Skip From victor.stinner at gmail.com Thu Jun 13 01:06:30 2013 From: victor.stinner at gmail.com (Victor Stinner) Date: Thu, 13 Jun 2013 01:06:30 +0200 Subject: [Python-Dev] Enhancement of Python memory allocators Message-ID: Hi, I would like to improve memory allocators of Python. My two use cases are replacing memory allocators with custom allocators in embedded system and hooking allocators to track usage of memory. I wrote a patch for this, I'm going to commit it if nobody complains: http://bugs.python.org/issue3329 Using this patch, detecting memory corruptions (buffer underflow and overflow) can be done without recompilation. We may add an environment variable to enable Python debug functions at runtime, example: PYDEBUGMALLOC=1. There is just a restriction: the environment variable would not be ignored with -E command line option, because command line options are parsed after the first memory allocation. What do you think? ***** The patch adds the following functions: void PyMem_GetAllocators( void **ctx_p, void* (**malloc_p) (void *ctx, size_t size), void* (**realloc_p) (void *ctx, void *ptr, size_t size), void (**free_p) (void *ctx, void *ptr)); void PyMem_SetAllocators( void *ctx, void* (*malloc) (void *ctx, size_t size), void* (*realloc) (void *ctx, void *ptr, size_t size), void (*free) (void *ctx, void *ptr)); It adds 4 similar functions (get/set) for PyObject_Malloc() and allocators of pymalloc arenas. ***** For the "track usage of memory" use case, see the following project which hooks memory allocators using PyMem_SetAllocators() and PyObject_SetAllocators() to get allocated bytes per filename and line number. https://pypi.python.org/pypi/pytracemalloc ***** Another issue proposes to use VirtualAlloc() and VirtualFree() for pymalloc arenas, see: http://bugs.python.org/issue13483 I don't know if it would be interesting, but it would now possible to choose the memory allocator (malloc, mmap, HeapAlloc, VirtualAlloc, ...) at runtime, with an environment variable for example. Victor From ncoghlan at gmail.com Thu Jun 13 01:21:42 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 13 Jun 2013 09:21:42 +1000 Subject: [Python-Dev] Enhancement of Python memory allocators In-Reply-To: References: Message-ID: On 13 Jun 2013 09:09, "Victor Stinner" wrote: > > Hi, > > I would like to improve memory allocators of Python. My two use cases > are replacing memory allocators with custom allocators in embedded > system and hooking allocators to track usage of memory. > > I wrote a patch for this, I'm going to commit it if nobody complains: > http://bugs.python.org/issue3329 > > Using this patch, detecting memory corruptions (buffer underflow and > overflow) can be done without recompilation. We may add an environment > variable to enable Python debug functions at runtime, example: > PYDEBUGMALLOC=1. There is just a restriction: the environment variable > would not be ignored with -E command line option, because command line > options are parsed after the first memory allocation. What do you > think? The rest of it sounds fine, but please don't add the runtime switching support to our existing main function. Interpreter startup is a mess already. If you were interested in helping directly with PEP 432, though, that would be good - I haven't been able to spend much time on it lately. Cheers, Nick. > > ***** > > The patch adds the following functions: > > void PyMem_GetAllocators( > void **ctx_p, > void* (**malloc_p) (void *ctx, size_t size), > void* (**realloc_p) (void *ctx, void *ptr, size_t size), > void (**free_p) (void *ctx, void *ptr)); > > void PyMem_SetAllocators( > void *ctx, > void* (*malloc) (void *ctx, size_t size), > void* (*realloc) (void *ctx, void *ptr, size_t size), > void (*free) (void *ctx, void *ptr)); > > It adds 4 similar functions (get/set) for PyObject_Malloc() and > allocators of pymalloc arenas. > > ***** > > For the "track usage of memory" use case, see the following project > which hooks memory allocators using PyMem_SetAllocators() and > PyObject_SetAllocators() to get allocated bytes per filename and line > number. > https://pypi.python.org/pypi/pytracemalloc > > ***** > > Another issue proposes to use VirtualAlloc() and VirtualFree() for > pymalloc arenas, see: > http://bugs.python.org/issue13483 > > I don't know if it would be interesting, but it would now possible to > choose the memory allocator (malloc, mmap, HeapAlloc, VirtualAlloc, > ...) at runtime, with an environment variable for example. > > Victor > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From victor.stinner at gmail.com Thu Jun 13 02:09:18 2013 From: victor.stinner at gmail.com (Victor Stinner) Date: Thu, 13 Jun 2013 02:09:18 +0200 Subject: [Python-Dev] Enhancement of Python memory allocators In-Reply-To: References: Message-ID: 2013/6/13 Nick Coghlan : > On 13 Jun 2013 09:09, "Victor Stinner" wrote: >> Using this patch, detecting memory corruptions (buffer underflow and >> overflow) can be done without recompilation. We may add an environment >> variable to enable Python debug functions at runtime, example: >> PYDEBUGMALLOC=1. There is just a restriction: the environment variable >> would not be ignored with -E command line option, because command line >> options are parsed after the first memory allocation. What do you >> think? > > The rest of it sounds fine, but please don't add the runtime switching > support to our existing main function. Interpreter startup is a mess > already. If you were interested in helping directly with PEP 432, though, > that would be good - I haven't been able to spend much time on it lately. I proposed an environment variable to solve the following issue: when memory allocators are replaced with custom allocators, debug hooks cannot be used. Debug hooks must be set before the first memory allocation. Another option is to add a new function (ex: PyMem_SetDebugHook()) to install explicitly debug hooks, so it can be called after PyMem_SetAllocators() and before the first memory allocation. Victor From brett at python.org Thu Jun 13 02:13:49 2013 From: brett at python.org (Brett Cannon) Date: Wed, 12 Jun 2013 20:13:49 -0400 Subject: [Python-Dev] [Python-checkins] cpython: Move test___all__ over to unittest.main() and use ModuleNotFoundError In-Reply-To: <3bW54Q4xGQzRrw@mail.python.org> References: <3bW54Q4xGQzRrw@mail.python.org> Message-ID: Obviously the commit message is a little misleading since changes I was about to stage accidentally went in on this change. Sorry about that. Same basic concept of the changes, just to more modules. On Wed, Jun 12, 2013 at 8:12 PM, brett.cannon wrote: > http://hg.python.org/cpython/rev/c4d7228421df > changeset: 84106:c4d7228421df > user: Brett Cannon > date: Wed Jun 12 20:12:30 2013 -0400 > summary: > Move test___all__ over to unittest.main() and use ModuleNotFoundError > > files: > Lib/test/regrtest.py | 16 ++++++++-------- > Lib/test/support.py | 14 +++++++------- > Lib/test/test___all__.py | 7 ++----- > Lib/xmlrpc/server.py | 2 +- > Lib/zipfile.py | 6 +++--- > 5 files changed, 21 insertions(+), 24 deletions(-) > > > diff --git a/Lib/test/regrtest.py b/Lib/test/regrtest.py > --- a/Lib/test/regrtest.py > +++ b/Lib/test/regrtest.py > @@ -146,11 +146,11 @@ > > try: > import threading > -except ImportError: > +except ModuleNotFoundError: > threading = None > try: > import multiprocessing.process > -except ImportError: > +except ModuleNotFoundError: > multiprocessing = None > > > @@ -180,7 +180,7 @@ > if sys.platform == 'darwin': > try: > import resource > - except ImportError: > + except ModuleNotFoundError: > pass > else: > soft, hard = resource.getrlimit(resource.RLIMIT_STACK) > @@ -571,7 +571,7 @@ > if findleaks: > try: > import gc > - except ImportError: > + except ModuleNotFoundError: > print('No GC available, disabling findleaks.') > findleaks = False > else: > @@ -692,7 +692,7 @@ > if use_mp: > try: > from threading import Thread > - except ImportError: > + except ModuleNotFoundError: > print("Multiprocess option requires thread support") > sys.exit(2) > from queue import Queue > @@ -1396,7 +1396,7 @@ > pic = sys.path_importer_cache.copy() > try: > import zipimport > - except ImportError: > + except ModuleNotFoundError: > zdc = None # Run unmodified on platforms without zipimport support > else: > zdc = zipimport._zip_directory_cache.copy() > @@ -1473,7 +1473,7 @@ > sys.path_importer_cache.update(pic) > try: > import zipimport > - except ImportError: > + except ModuleNotFoundError: > pass # Run unmodified on platforms without zipimport support > else: > zipimport._zip_directory_cache.clear() > @@ -1510,7 +1510,7 @@ > doctest.master = None > try: > import ctypes > - except ImportError: > + except ModuleNotFoundError: > # Don't worry about resetting the cache if ctypes is not supported > pass > else: > diff --git a/Lib/test/support.py b/Lib/test/support.py > --- a/Lib/test/support.py > +++ b/Lib/test/support.py > @@ -29,27 +29,27 @@ > > try: > import _thread, threading > -except ImportError: > +except ModuleNotFoundError: > _thread = None > threading = None > try: > import multiprocessing.process > -except ImportError: > +except ModuleNotFoundError: > multiprocessing = None > > try: > import zlib > -except ImportError: > +except ModuleNotFoundError: > zlib = None > > try: > import bz2 > -except ImportError: > +except ModuleNotFoundError: > bz2 = None > > try: > import lzma > -except ImportError: > +except ModuleNotFoundError: > lzma = None > > __all__ = [ > @@ -116,7 +116,7 @@ > with _ignore_deprecated_imports(deprecated): > try: > return importlib.import_module(name) > - except ImportError as msg: > + except ModuleNotFoundError as msg: > if sys.platform.startswith(tuple(required_on)): > raise > raise unittest.SkipTest(str(msg)) > @@ -188,7 +188,7 @@ > if not _save_and_block_module(blocked_name, orig_modules): > names_to_remove.append(blocked_name) > fresh_module = importlib.import_module(name) > - except ImportError: > + except ModuleNotFoundError: > fresh_module = None > finally: > for orig_name, module in orig_modules.items(): > diff --git a/Lib/test/test___all__.py b/Lib/test/test___all__.py > --- a/Lib/test/test___all__.py > +++ b/Lib/test/test___all__.py > @@ -75,7 +75,7 @@ > try: > import rlcompleter > import locale > - except ImportError: > + except ModuleNotFoundError: > pass > else: > locale.setlocale(locale.LC_CTYPE, 'C') > @@ -113,8 +113,5 @@ > print('Following modules failed to be imported:', > failed_imports) > > > -def test_main(): > - support.run_unittest(AllTest) > - > if __name__ == "__main__": > - test_main() > + unittest.main() > diff --git a/Lib/xmlrpc/server.py b/Lib/xmlrpc/server.py > --- a/Lib/xmlrpc/server.py > +++ b/Lib/xmlrpc/server.py > @@ -116,7 +116,7 @@ > import traceback > try: > import fcntl > -except ImportError: > +except ModuleNotFoundError: > fcntl = None > > def resolve_dotted_attribute(obj, attr, allow_dotted_names=True): > diff --git a/Lib/zipfile.py b/Lib/zipfile.py > --- a/Lib/zipfile.py > +++ b/Lib/zipfile.py > @@ -18,18 +18,18 @@ > try: > import zlib # We may need its compression method > crc32 = zlib.crc32 > -except ImportError: > +except ModuleNotFoundError: > zlib = None > crc32 = binascii.crc32 > > try: > import bz2 # We may need its compression method > -except ImportError: > +except ModuleNotFoundError: > bz2 = None > > try: > import lzma # We may need its compression method > -except ImportError: > +except ModuleNotFoundError: > lzma = None > > __all__ = ["BadZipFile", "BadZipfile", "error", > > -- > Repository URL: http://hg.python.org/cpython > > _______________________________________________ > Python-checkins mailing list > Python-checkins at python.org > http://mail.python.org/mailman/listinfo/python-checkins > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Thu Jun 13 04:40:35 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 13 Jun 2013 12:40:35 +1000 Subject: [Python-Dev] Enhancement of Python memory allocators In-Reply-To: References: Message-ID: On 13 Jun 2013 10:09, "Victor Stinner" wrote: > > 2013/6/13 Nick Coghlan : > > On 13 Jun 2013 09:09, "Victor Stinner" wrote: > >> Using this patch, detecting memory corruptions (buffer underflow and > >> overflow) can be done without recompilation. We may add an environment > >> variable to enable Python debug functions at runtime, example: > >> PYDEBUGMALLOC=1. There is just a restriction: the environment variable > >> would not be ignored with -E command line option, because command line > >> options are parsed after the first memory allocation. What do you > >> think? > > > > The rest of it sounds fine, but please don't add the runtime switching > > support to our existing main function. Interpreter startup is a mess > > already. If you were interested in helping directly with PEP 432, though, > > that would be good - I haven't been able to spend much time on it lately. > > I proposed an environment variable to solve the following issue: when > memory allocators are replaced with custom allocators, debug hooks > cannot be used. Debug hooks must be set before the first memory > allocation. > > Another option is to add a new function (ex: PyMem_SetDebugHook()) to > install explicitly debug hooks, so it can be called after > PyMem_SetAllocators() and before the first memory allocation. Yes, that sounds better. One of the biggest problems with the current startup sequence is the way it relies on environment variables for configuration, which makes life hard for other applications that want to embed the CPython runtime. Cheers, Nick. > > Victor -------------- next part -------------- An HTML attachment was scrubbed... URL: From victor.stinner at gmail.com Thu Jun 13 13:03:46 2013 From: victor.stinner at gmail.com (Victor Stinner) Date: Thu, 13 Jun 2013 13:03:46 +0200 Subject: [Python-Dev] Enhancement of Python memory allocators In-Reply-To: References: Message-ID: 2013/6/13 Nick Coghlan : > Yes, that sounds better. One of the biggest problems with the current > startup sequence is the way it relies on environment variables for > configuration, which makes life hard for other applications that want to > embed the CPython runtime. I wrote a new patch (attached to issue #3329) adding a new PyMem_SetupDebugHooks() function. So if an application replaces Python memory allocator functions, it can still can PyMem_SetupDebugHooks() to benefit of the Python debug hooks detecting memory bugs. The function does nothing if hooks are already installed or if Python is not compiled in debug mode. With this function, the API is now complete for all use cases. The PEP 432 helps to configure embedded Python, but the new "Set" functions (ex: PyMem_SetAllocators) are still needed for my pytracemalloc module which installs hooks at runtime, when Python is fully initialized (the hooks can be installed anytime). pytracemalloc is just an example, you may use PyMem_SetAllocators for other debug or performance purpose. With my patch, allocator functions like PyMem_Malloc() are no more macro, and are always the same function. This helps the stable ABI: C extension modules do not need to be recompiled to benefit of the debug hooks ;-) Victor From solipsis at pitrou.net Thu Jun 13 20:56:26 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 13 Jun 2013 20:56:26 +0200 Subject: [Python-Dev] ssl improvements and testing question References: Message-ID: <20130613205626.5f379704@fsol> On Fri, 07 Jun 2013 00:37:01 +0200 Christian Heimes wrote: > > I'm also thinking about OCSP support and X509v3 extension support for > _decode_certificate(). Both are a PITB ... Python has an easier and > better documented C API. Yes, OpenSSL's documentation is often a joke, unfortunately. > Question: > What's the minimum version of OpenSSL Python 3.4 is going to support? Judging by the kind of machines we run on, I would say 0.9.7something. Basically I don't think we should remove any existing #ifdef for 3.4. > Do > we have an easy way to compile and link Python against a custom > installation of OpenSSL or do I have to fiddle around with CPPFLAGS and > CFLAGS? You have to fiddle around, sadly. (and you will also have to fiddle around with LD_LIBRARY_PATH) If you find a way to improve that, your contribution is much welcome :-) Regards Antoine. From solipsis at pitrou.net Thu Jun 13 20:59:13 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 13 Jun 2013 20:59:13 +0200 Subject: [Python-Dev] cpython: fixd refleak References: <3bTSdr5GRqzRNm@mail.python.org> Message-ID: <20130613205913.39f389b1@fsol> On Mon, 10 Jun 2013 10:47:28 +0200 (CEST) christian.heimes wrote: > http://hg.python.org/cpython/rev/6860263c05b3 > changeset: 84077:6860263c05b3 > user: Christian Heimes > date: Mon Jun 10 10:47:22 2013 +0200 > summary: > fixd refleak > [...] > - return Py_BuildValue("(OOOO)", ofile_env, ofile, odir_env, odir); > + if ((tup = PyTuple_New(4)) == NULL) { > + goto error; > + } > + PyTuple_SET_ITEM(tup, 0, ofile_env); > + PyTuple_SET_ITEM(tup, 1, ofile); > + PyTuple_SET_ITEM(tup, 2, odir_env); > + PyTuple_SET_ITEM(tup, 3, odir); > + return tup; How about return Py_BuildValue("NNNN", ofile_env, ofile, odir_env, odir); ? From christian at python.org Thu Jun 13 23:39:49 2013 From: christian at python.org (Christian Heimes) Date: Thu, 13 Jun 2013 23:39:49 +0200 Subject: [Python-Dev] cpython: fixd refleak In-Reply-To: <20130613205913.39f389b1@fsol> References: <3bTSdr5GRqzRNm@mail.python.org> <20130613205913.39f389b1@fsol> Message-ID: <51BA3C25.50506@python.org> Am 13.06.2013 20:59, schrieb Antoine Pitrou: > How about > > return Py_BuildValue("NNNN", ofile_env, ofile, odir_env, odir); > > ? Oh right, I forgot about 'N'. The PyArg_Parse*() methods don't have it. Do you want me to modify the code to use Py_BuildValue()? Christian From solipsis at pitrou.net Thu Jun 13 23:42:21 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 13 Jun 2013 23:42:21 +0200 Subject: [Python-Dev] cpython: fixd refleak In-Reply-To: <51BA3C25.50506@python.org> References: <3bTSdr5GRqzRNm@mail.python.org> <20130613205913.39f389b1@fsol> <51BA3C25.50506@python.org> Message-ID: <20130613234221.171de591@fsol> On Thu, 13 Jun 2013 23:39:49 +0200 Christian Heimes wrote: > Am 13.06.2013 20:59, schrieb Antoine Pitrou: > > How about > > > > return Py_BuildValue("NNNN", ofile_env, ofile, odir_env, odir); > > > > ? > > Oh right, I forgot about 'N'. The PyArg_Parse*() methods don't have it. > > Do you want me to modify the code to use Py_BuildValue()? Always better to use the simple form, IMO. Go for it! Regards Antoine. From victor.stinner at gmail.com Fri Jun 14 00:04:19 2013 From: victor.stinner at gmail.com (Victor Stinner) Date: Fri, 14 Jun 2013 00:04:19 +0200 Subject: [Python-Dev] Allow calling PyMem_Malloc() without the GIL held in Python 3.4 Message-ID: Hi, I would like to remove the "GIL must be held" restriction from PyMem_Malloc(). In my opinion, the restriction was motived by a bug in Python, bug fixed by the issue #3329. Let me explain why. The PyMem_Malloc() function is a thin wrapper to malloc(). It returns NULL if the size is lager than PY_SSIZE_T_MAX and have a well defined behaviour for PyMem_Malloc(0) (don't return NULL). So it is surprising to read in Include/pymem.h: "The GIL must be held when using these APIs." The reason is more surprising: in debug mode, PyMem_Malloc() is no more a thin wrapper to malloc(), but it calls internally PyObject_Malloc(), the "Python allocator" (called pymalloc). (Many other checks are done in debug mode, but it's unrelated to my point.) The problem is that PyObject_Malloc() is not thread-safe, the GIL must be held. Short history: fb45791150d1 (Mar 23 2002) "gives Python a debug-mode pymalloc" f294fdd18b5b (Mar 28 2002) removes the "check API family" e16dbf875303 (Apr 22 2002) redirects indirectly PyMem_Malloc() to PyObject_Malloc() in debug mode b6aff7a59803 (Sep 28 2009) reintroduces API checks So the GIL issue is almost as old as the debug mode for Python memory allocators. My patch attached to http://bugs.python.org/issue3329 changes the design of the debug memory allocators: they are now wrapper (hooks) on the underlying memory allocator (PyMem: malloc, PyObject: pymalloc), instead of always redirecting to pymalloc (ex: PyObject_Malloc). Using my patch, PyMem_Malloc() now always calls malloc(), even in debug mode. Removing the "GIL must be held" restriction is now safe. Do you agree? May it cause backward compatibility issue? PyMem_Malloc() and PyMem_MALLOC() call malloc(), except if the Python source code was manually modified. Does this use case concern many developers? Removing the GIL restriction would help to replace direct calls to malloc() with PyMem_Malloc(). Using PyMem_SetAllocators(), an application would be able to replace memory allocators, and these allocators would be used "everywhere". => see http://bugs.python.org/issue18203 Victor From solipsis at pitrou.net Fri Jun 14 10:46:03 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 14 Jun 2013 10:46:03 +0200 Subject: [Python-Dev] cpython (2.7): Fix comment blocks. Adjust blocksize to a power-of-two for better divmod References: <3bWqYP3sqgzQdN@mail.python.org> Message-ID: <20130614104603.35f83a9c@fsol> On Fri, 14 Jun 2013 07:06:49 +0200 (CEST) raymond.hettinger wrote: > http://hg.python.org/cpython/rev/5accb0ac8bfb > changeset: 84116:5accb0ac8bfb > branch: 2.7 > parent: 84095:ca8e86711403 > user: Raymond Hettinger > date: Fri Jun 14 01:06:33 2013 -0400 > summary: > Fix comment blocks. Adjust blocksize to a power-of-two for better divmod computations. Is there any rationale for changing the heuristic from "fits in a whole number of cachelines" to "allows fast division by the blocksize"? I personally would prefer if such changes were justified a bit more than by a one-liner changeset message without any corresponding open issue. Regards Antoine. From kristjan at ccpgames.com Fri Jun 14 11:29:17 2013 From: kristjan at ccpgames.com (=?iso-8859-1?Q?Kristj=E1n_Valur_J=F3nsson?=) Date: Fri, 14 Jun 2013 09:29:17 +0000 Subject: [Python-Dev] Allow calling PyMem_Malloc() without the GIL held in Python 3.4 In-Reply-To: References: Message-ID: > -----Original Message----- > I would like to remove the "GIL must be held" restriction from > PyMem_Malloc(). In my opinion, the restriction was motived by a bug in > Python, bug fixed by the issue #3329. Let me explain why. > ... > > Removing the GIL restriction would help to replace direct calls to > malloc() with PyMem_Malloc(). Using PyMem_SetAllocators(), an application > would be able to replace memory allocators, and these allocators would be > used "everywhere". > => see http://bugs.python.org/issue18203 > To keep this interesting, I have a somewhat different opinion to Victor :) have put comments in the original defect, but would like to repeat them here. IMHO, keeping the GIL restriction on PyMem_MALLOC is useful. 1) It allows it to be replaced with PyObject_MALLOC(). Or anything else. In particular, an implementer is free to add memory profiling support and other things without worrying about implementation details. Requiring it to be GIL free severely limits what it can do. For example, it would be forbidden to delegate to PyObject_MALLOC when debugging. The approach CCP has taken (we have replaced all raw malloc calls with api calls) is this: a) Add a "raw" api, PyMem_MALLOC_RAW. This is guaranteed to be thread safe and call directly to the external memory api of python, as set by Py_SetAllocator() b) Replace calls to malloc() in the source code with PyMem_MALLOC/PyMem_MALLOC_RAW as appropriate (in our case, using an include file with #defines to mimimize changes) There are only two or three places in the source code that require non-GIL protected malloc. IMHO, requiring PyMem_MALLOC to be threadsafe just to cater to those three places is an overkill, and does more harm than good by limiting our options. Cheers! Kristj?n From storchaka at gmail.com Fri Jun 14 15:47:50 2013 From: storchaka at gmail.com (Serhiy Storchaka) Date: Fri, 14 Jun 2013 16:47:50 +0300 Subject: [Python-Dev] cpython (2.7): Fix comment blocks. Adjust blocksize to a power-of-two for better divmod In-Reply-To: <20130614104603.35f83a9c@fsol> References: <3bWqYP3sqgzQdN@mail.python.org> <20130614104603.35f83a9c@fsol> Message-ID: 14.06.13 11:46, Antoine Pitrou ???????(??): > On Fri, 14 Jun 2013 07:06:49 +0200 (CEST) > raymond.hettinger wrote: >> http://hg.python.org/cpython/rev/5accb0ac8bfb >> changeset: 84116:5accb0ac8bfb >> Fix comment blocks. Adjust blocksize to a power-of-two for better divmod computations. > > Is there any rationale for changing the heuristic from "fits in a whole > number of cachelines" to "allows fast division by the blocksize"? > > I personally would prefer if such changes were justified a bit more > than by a one-liner changeset message without any corresponding open > issue. I share the doubts of Antoine and I was going to write the same comment. I thought there were good reasons for previous code. What has changed? From storchaka at gmail.com Fri Jun 14 15:59:52 2013 From: storchaka at gmail.com (Serhiy Storchaka) Date: Fri, 14 Jun 2013 16:59:52 +0300 Subject: [Python-Dev] cpython: Move test_pep352 over to unittest.main() In-Reply-To: <3bWkVM0wM9zP5m@mail.python.org> References: <3bWkVM0wM9zP5m@mail.python.org> Message-ID: 14.06.13 04:18, brett.cannon ???????(??): > http://hg.python.org/cpython/rev/af27c661d4fb > changeset: 84115:af27c661d4fb > user: Brett Cannon > date: Thu Jun 13 21:18:43 2013 -0400 > summary: > Move test_pep352 over to unittest.main() You forgot about: from test.support import run_unittest From pynix.wang at gmail.com Fri Jun 14 15:58:59 2013 From: pynix.wang at gmail.com (Pynix Wang) Date: Fri, 14 Jun 2013 21:58:59 +0800 Subject: [Python-Dev] python symbolizition Message-ID: 1.lambda expression c# a.(x, y) => x == y b.() => SomeMethod() ruby: -> {|msg| puts msg} python can use c# like and remove "lambda" keyword. 2.global variable ruby $glo_var python can use $ or @ or another and remove "global". -------------- next part -------------- An HTML attachment was scrubbed... URL: From markus at unterwaditzer.net Fri Jun 14 16:50:53 2013 From: markus at unterwaditzer.net (Markus Unterwaditzer) Date: Fri, 14 Jun 2013 16:50:53 +0200 Subject: [Python-Dev] python symbolizition In-Reply-To: References: Message-ID: But why? -- Markus (from phone) Pynix Wang wrote: >1.lambda expression > >c# >a.(x, y) => x == y >b.() => SomeMethod() > >ruby: > -> {|msg| puts msg} > >python can use c# like and remove "lambda" keyword. > >2.global variable > >ruby >$glo_var > >python can use $ or @ or another and remove "global". > > >------------------------------------------------------------------------ > >_______________________________________________ >Python-Dev mailing list >Python-Dev at python.org >http://mail.python.org/mailman/listinfo/python-dev >Unsubscribe: >http://mail.python.org/mailman/options/python-dev/markus%40unterwaditzer.net From rdmurray at bitdance.com Fri Jun 14 17:03:44 2013 From: rdmurray at bitdance.com (R. David Murray) Date: Fri, 14 Jun 2013 11:03:44 -0400 Subject: [Python-Dev] python symbolizition In-Reply-To: References: Message-ID: <20130614150345.51EE2250BD7@webabinitio.net> This discussion is better suited for python-ideas than it is for python-dev. On Fri, 14 Jun 2013 16:50:53 +0200, Markus Unterwaditzer wrote: > But why? > > -- Markus (from phone) > > Pynix Wang wrote: > >1.lambda expression > > > >c# > >a.(x, y) => x == y > >b.() => SomeMethod() > > > >ruby: > > -> {|msg| puts msg} > > > >python can use c# like and remove "lambda" keyword. > > > >2.global variable > > > >ruby > >$glo_var > > > >python can use $ or @ or another and remove "global". > > > > > >------------------------------------------------------------------------ > > > >_______________________________________________ > >Python-Dev mailing list > >Python-Dev at python.org > >http://mail.python.org/mailman/listinfo/python-dev > >Unsubscribe: > >http://mail.python.org/mailman/options/python-dev/markus%40unterwaditzer.net > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/rdmurray%40bitdance.com From walter at livinglogic.de Fri Jun 14 16:55:24 2013 From: walter at livinglogic.de (=?UTF-8?B?V2FsdGVyIETDtnJ3YWxk?=) Date: Fri, 14 Jun 2013 16:55:24 +0200 Subject: [Python-Dev] eval and triple quoted strings Message-ID: <51BB2EDC.80604@livinglogic.de> Hello all! This surprised me: >>> eval("'''\r\n'''") '\n' Where did the \r go? ast.literal_eval() has the same problem: >>> ast.literal_eval("'''\r\n'''") '\n' Is this a bug/worth fixing? Servus, Walter From guido at python.org Fri Jun 14 17:36:48 2013 From: guido at python.org (Guido van Rossum) Date: Fri, 14 Jun 2013 08:36:48 -0700 Subject: [Python-Dev] eval and triple quoted strings In-Reply-To: <51BB2EDC.80604@livinglogic.de> References: <51BB2EDC.80604@livinglogic.de> Message-ID: Not a bug. The same is done for file input -- CRLF is changed to LF before tokenizing. On Jun 14, 2013 8:27 AM, "Walter D?rwald" wrote: > Hello all! > > This surprised me: > > >>> eval("'''\r\n'''") > '\n' > > Where did the \r go? ast.literal_eval() has the same problem: > > >>> ast.literal_eval("'''\r\n'''") > '\n' > > Is this a bug/worth fixing? > > Servus, > Walter > ______________________________**_________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/**mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/**mailman/options/python-dev/** > guido%40python.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: From status at bugs.python.org Fri Jun 14 18:07:28 2013 From: status at bugs.python.org (Python tracker) Date: Fri, 14 Jun 2013 18:07:28 +0200 (CEST) Subject: [Python-Dev] Summary of Python tracker Issues Message-ID: <20130614160728.2A40956A39@psf.upfronthosting.co.za> ACTIVITY SUMMARY (2013-06-07 - 2013-06-14) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 4042 (+18) closed 25946 (+41) total 29988 (+59) Open issues with patches: 1795 Issues opened (41) ================== #17947: Code, test, and doc review for PEP-0435 Enum http://bugs.python.org/issue17947 reopened by ncoghlan #18159: ConfigParser getters not available on SectionProxy http://bugs.python.org/issue18159 opened by JBernardo #18161: call fchdir if subprocess.Popen(cwd=integer|fileobject) http://bugs.python.org/issue18161 opened by mmarkk #18162: Add index attribute to IndexError http://bugs.python.org/issue18162 opened by brett.cannon #18163: Add a 'key' attribute to KeyError http://bugs.python.org/issue18163 opened by brett.cannon #18164: Embedding Python doc incorrectly refers to LINKFORSHARED http://bugs.python.org/issue18164 opened by ned.deily #18165: Add 'unexpected_type' to TypeError http://bugs.python.org/issue18165 opened by brett.cannon #18166: 'value' attribute for ValueError http://bugs.python.org/issue18166 opened by brett.cannon #18167: cgi.FieldStorage fails to handle multipart/form-data when \r\n http://bugs.python.org/issue18167 opened by catlee #18168: plistlib output self-sorted dictionary http://bugs.python.org/issue18168 opened by halfjuice #18170: define built-in exceptions in Python code http://bugs.python.org/issue18170 opened by brett.cannon #18173: Add MixedTypeKey to reprlib http://bugs.python.org/issue18173 opened by ncoghlan #18174: Make regrtest with --huntrleaks check for fd leaks http://bugs.python.org/issue18174 opened by sbt #18176: Builtins documentation refers to old version of UCD. http://bugs.python.org/issue18176 opened by belopolsky #18177: Incorect quote marks in code section-headers in PDF version of http://bugs.python.org/issue18177 opened by icedream91 #18178: Redefinition of malloc(3) family of functions at build time http://bugs.python.org/issue18178 opened by cvs-src #18179: SMTP.local_hostname is undocumented http://bugs.python.org/issue18179 opened by jonash #18181: super vs. someclass.__getattribute__ http://bugs.python.org/issue18181 opened by ronaldoussoren #18182: xml.dom.createElement() does not take implicit namespaces into http://bugs.python.org/issue18182 opened by Alexander.Tobias.Heinrich #18184: Add range check for %c in PyUnicode_FromFormat http://bugs.python.org/issue18184 opened by serhiy.storchaka #18185: Error in test_set.TestVariousIteratorArgs.test_inline_methods http://bugs.python.org/issue18185 opened by syeberman #18188: ERROR: test_no_optimize_flag on Mageia Linux Cauldron x86-64 w http://bugs.python.org/issue18188 opened by shlomif #18189: IDLE Improvements: Unit test for Delegator.py http://bugs.python.org/issue18189 opened by JayKrish #18191: urllib2/urllib.parse.splitport does not handle IPv6 correctly http://bugs.python.org/issue18191 opened by jd #18192: Move imp.get_magic() to importlib http://bugs.python.org/issue18192 opened by brett.cannon #18193: Move imp.reload() to importlib http://bugs.python.org/issue18193 opened by brett.cannon #18194: Move imp.source_from_cache/cache_from_source to importlib http://bugs.python.org/issue18194 opened by brett.cannon #18195: error when deep copying module is confusing http://bugs.python.org/issue18195 opened by mrjbq7 #18197: insufficient error checking causes crash on windows http://bugs.python.org/issue18197 opened by maxdeliso #18199: Windows: support path longer than 260 bytes using "\\?\" prefi http://bugs.python.org/issue18199 opened by Voo #18201: distutils write into symlinks instead of replacing them http://bugs.python.org/issue18201 opened by mgorny #18202: Minor fixes for test_coding http://bugs.python.org/issue18202 opened by serhiy.storchaka #18203: Replace calls to malloc() with PyMem_Malloc() http://bugs.python.org/issue18203 opened by haypo #18206: license url in site.py should always use X.Y.Z form of version http://bugs.python.org/issue18206 opened by py.user #18207: OpenSSL may ignore seconds in notAfter http://bugs.python.org/issue18207 opened by christian.heimes #18209: Bytearray type not supported as a mutable object in the fcntl. http://bugs.python.org/issue18209 opened by vxgmichel #18211: -Werror=statement-after-declaration problem http://bugs.python.org/issue18211 opened by ronaldoussoren #18212: No way to check whether Future is finished? http://bugs.python.org/issue18212 opened by cool-RR #18213: py-bt errors on backtrace with PyRun_SimpleString and friends http://bugs.python.org/issue18213 opened by pmuldoon #18214: Stop purging modules which are garbage collected before shutdo http://bugs.python.org/issue18214 opened by sbt #18215: Script to test multiple versions of OpenSSL http://bugs.python.org/issue18215 opened by christian.heimes Most recent 15 issues with no replies (15) ========================================== #18215: Script to test multiple versions of OpenSSL http://bugs.python.org/issue18215 #18214: Stop purging modules which are garbage collected before shutdo http://bugs.python.org/issue18214 #18212: No way to check whether Future is finished? http://bugs.python.org/issue18212 #18211: -Werror=statement-after-declaration problem http://bugs.python.org/issue18211 #18209: Bytearray type not supported as a mutable object in the fcntl. http://bugs.python.org/issue18209 #18207: OpenSSL may ignore seconds in notAfter http://bugs.python.org/issue18207 #18202: Minor fixes for test_coding http://bugs.python.org/issue18202 #18189: IDLE Improvements: Unit test for Delegator.py http://bugs.python.org/issue18189 #18184: Add range check for %c in PyUnicode_FromFormat http://bugs.python.org/issue18184 #18173: Add MixedTypeKey to reprlib http://bugs.python.org/issue18173 #18166: 'value' attribute for ValueError http://bugs.python.org/issue18166 #18165: Add 'unexpected_type' to TypeError http://bugs.python.org/issue18165 #18161: call fchdir if subprocess.Popen(cwd=integer|fileobject) http://bugs.python.org/issue18161 #18144: FD leak in urllib2 http://bugs.python.org/issue18144 #18137: format(float, str): integer overflow for huge precision http://bugs.python.org/issue18137 Most recent 15 issues waiting for review (15) ============================================= #18214: Stop purging modules which are garbage collected before shutdo http://bugs.python.org/issue18214 #18202: Minor fixes for test_coding http://bugs.python.org/issue18202 #18197: insufficient error checking causes crash on windows http://bugs.python.org/issue18197 #18193: Move imp.reload() to importlib http://bugs.python.org/issue18193 #18191: urllib2/urllib.parse.splitport does not handle IPv6 correctly http://bugs.python.org/issue18191 #18189: IDLE Improvements: Unit test for Delegator.py http://bugs.python.org/issue18189 #18185: Error in test_set.TestVariousIteratorArgs.test_inline_methods http://bugs.python.org/issue18185 #18184: Add range check for %c in PyUnicode_FromFormat http://bugs.python.org/issue18184 #18181: super vs. someclass.__getattribute__ http://bugs.python.org/issue18181 #18179: SMTP.local_hostname is undocumented http://bugs.python.org/issue18179 #18176: Builtins documentation refers to old version of UCD. http://bugs.python.org/issue18176 #18174: Make regrtest with --huntrleaks check for fd leaks http://bugs.python.org/issue18174 #18167: cgi.FieldStorage fails to handle multipart/form-data when \r\n http://bugs.python.org/issue18167 #18164: Embedding Python doc incorrectly refers to LINKFORSHARED http://bugs.python.org/issue18164 #18155: csv.Sniffer.has_header doesn't escape characters used in regex http://bugs.python.org/issue18155 Top 10 most discussed issues (10) ================================= #17947: Code, test, and doc review for PEP-0435 Enum http://bugs.python.org/issue17947 16 msgs #18181: super vs. someclass.__getattribute__ http://bugs.python.org/issue18181 11 msgs #3329: API for setting the memory allocator used by Python http://bugs.python.org/issue3329 10 msgs #17860: subprocess docs lack info how to use output result http://bugs.python.org/issue17860 8 msgs #18149: filecmp.cmp() incorrect results when previously compared file http://bugs.python.org/issue18149 7 msgs #18163: Add a 'key' attribute to KeyError http://bugs.python.org/issue18163 7 msgs #10581: Review and document string format accepted in numeric data typ http://bugs.python.org/issue10581 6 msgs #14455: plistlib unable to read json and binary plist files http://bugs.python.org/issue14455 6 msgs #18188: ERROR: test_no_optimize_flag on Mageia Linux Cauldron x86-64 w http://bugs.python.org/issue18188 6 msgs #18203: Replace calls to malloc() with PyMem_Malloc() http://bugs.python.org/issue18203 6 msgs Issues closed (38) ================== #5492: Error on leaving IDLE with quit() or exit() under Linux http://bugs.python.org/issue5492 closed by roger.serwy #6632: Include more chars in the decimal codec http://bugs.python.org/issue6632 closed by belopolsky #10382: Command line error marker misplaced on unicode entry http://bugs.python.org/issue10382 closed by belopolsky #14797: Deprecate imp.find_module()/load_module() http://bugs.python.org/issue14797 closed by brett.cannon #14813: Can't build under VS2008 anymore http://bugs.python.org/issue14813 closed by christian.heimes #15239: Abandoned Tools/unicode/mkstringprep.py http://bugs.python.org/issue15239 closed by serhiy.storchaka #15767: add ModuleNotFoundError http://bugs.python.org/issue15767 closed by brett.cannon #16102: uuid._netbios_getnode() is outdated http://bugs.python.org/issue16102 closed by serhiy.storchaka #17010: Windows launcher ignores active virtual environment http://bugs.python.org/issue17010 closed by vinay.sajip #17354: TypeError when running setup.py upload --show-response http://bugs.python.org/issue17354 closed by berker.peksag #17511: Idle find function closes after each find operation http://bugs.python.org/issue17511 closed by roger.serwy #17691: Fix test discovery for test_univnewlines.py http://bugs.python.org/issue17691 closed by ezio.melotti #17838: Can't assign a different value for sys.stdin in IDLE http://bugs.python.org/issue17838 closed by roger.serwy #17903: Python launcher for windows should search path for #!/usr/bin/ http://bugs.python.org/issue17903 closed by vinay.sajip #18038: Unhelpful error message on invalid encoding specification http://bugs.python.org/issue18038 closed by serhiy.storchaka #18048: Merging test_pep263.py and test_coding.py http://bugs.python.org/issue18048 closed by serhiy.storchaka #18055: Stop using imp in IDLE http://bugs.python.org/issue18055 closed by brett.cannon #18126: Update links to NumPy resources in documentation http://bugs.python.org/issue18126 closed by ezio.melotti #18150: Duplicate test inside TestSingleDispatch http://bugs.python.org/issue18150 closed by lukasz.langa #18157: remove usage of imp.load_module() from pydoc http://bugs.python.org/issue18157 closed by brett.cannon #18160: Packaging more coherent Python http://bugs.python.org/issue18160 closed by ezio.melotti #18169: struct.pack() behaves strangely for 'L' on 64bit Linux http://bugs.python.org/issue18169 closed by skrah #18171: os.path.expanduser does not use the system encoding http://bugs.python.org/issue18171 closed by r.david.murray #18172: New easter egg: insecure string pickle http://bugs.python.org/issue18172 closed by brian.curtin #18175: os.listdir(fd) leaks fd on error http://bugs.python.org/issue18175 closed by larry #18180: Refleak in test_imp on Windows http://bugs.python.org/issue18180 closed by sbt #18183: Calling .lower() on certain unicode string raises SystemError http://bugs.python.org/issue18183 closed by benjamin.peterson #18186: 2.x subprocess contains set notation http://bugs.python.org/issue18186 closed by berker.peksag #18187: Fix broken link in Doc/library/venv.rst http://bugs.python.org/issue18187 closed by ned.deily #18190: RuntimeError raised with re.search + re.DOTALL on empty string http://bugs.python.org/issue18190 closed by serhiy.storchaka #18196: IDLE: forward apply patch for handling SystemExit http://bugs.python.org/issue18196 closed by roger.serwy #18198: unittest discover should provide a way to define discovery at http://bugs.python.org/issue18198 closed by michael.foord #18200: Update stdlib to use ModuleNotFoundError http://bugs.python.org/issue18200 closed by brett.cannon #18204: distutils error showing upload error message http://bugs.python.org/issue18204 closed by berker.peksag #18205: PyOS_ReadlineFunctionPointer violates PyMem_Malloc() API: the http://bugs.python.org/issue18205 closed by haypo #18208: Wrong bytecode generated for 'in' operation http://bugs.python.org/issue18208 closed by amaury.forgeotdarc #18210: Problem creating extension in python3.3 with mingw32 http://bugs.python.org/issue18210 closed by amaury.forgeotdarc #18158: Delete test_importhooks http://bugs.python.org/issue18158 closed by brett.cannon From ijmorlan at uwaterloo.ca Fri Jun 14 17:43:49 2013 From: ijmorlan at uwaterloo.ca (Isaac Morland) Date: Fri, 14 Jun 2013 11:43:49 -0400 (EDT) Subject: [Python-Dev] python symbolizition In-Reply-To: <20130614150345.51EE2250BD7@webabinitio.net> References: <20130614150345.51EE2250BD7@webabinitio.net> Message-ID: On Fri, 14 Jun 2013, R. David Murray wrote: > This discussion is better suited for python-ideas than it > is for python-dev. Better yet, perl-ideas. > On Fri, 14 Jun 2013 16:50:53 +0200, Markus Unterwaditzer wrote: >> But why? >> >> -- Markus (from phone) >> >> Pynix Wang wrote: >>> 1.lambda expression >>> >>> c# >>> a.(x, y) => x == y >>> b.() => SomeMethod() >>> >>> ruby: >>> -> {|msg| puts msg} >>> >>> python can use c# like and remove "lambda" keyword. >>> >>> 2.global variable >>> >>> ruby >>> $glo_var >>> >>> python can use $ or @ or another and remove "global". Isaac Morland CSCF Web Guru DC 2554C, x36650 WWW Software Specialist From solipsis at pitrou.net Fri Jun 14 19:34:41 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 14 Jun 2013 19:34:41 +0200 Subject: [Python-Dev] PEP 442 accepted In-Reply-To: References: Message-ID: <20130614193441.04b219c8@fsol> On Wed, 5 Jun 2013 09:10:54 -0700 Benjamin Peterson wrote: > I (and Guido) are accepting PEP 442 (Safe object finalization) on the > condition that finalizers are only ever called once globally. Ok, so there's an issue with that condition: it can't be upholded on non-GC objects. Creating a non-GC object is quite obscure and rare, though, since it requires basically a class with no __dict__ and an empty __slots__: class C: __slots__ = () survivors = [] def __del__(self): self.survivors.append(self) In this case, a C instance's __del__ will be called every time destruction is attempted, not only once. Is that a realistic problem? Regards Antoine. From solipsis at pitrou.net Fri Jun 14 20:09:08 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 14 Jun 2013 20:09:08 +0200 Subject: [Python-Dev] PEP 442 accepted References: <20130614193441.04b219c8@fsol> Message-ID: <20130614200908.6f7fce29@fsol> On Fri, 14 Jun 2013 19:34:41 +0200 Antoine Pitrou wrote: > On Wed, 5 Jun 2013 09:10:54 -0700 > Benjamin Peterson wrote: > > I (and Guido) are accepting PEP 442 (Safe object finalization) on the > > condition that finalizers are only ever called once globally. > > Ok, so there's an issue with that condition: it can't be upholded on > non-GC objects. Creating a non-GC object is quite obscure and rare, > though, since it requires basically a class with no __dict__ and an > empty __slots__: [...] One more problem, albeit minor: there's a test_descr failure in some part marked as "CPython implementation detail": ====================================================================== FAIL: test_subtype_resurrection (test.test_descr.ClassPropertiesAndMethods) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/antoine/cpython/finalize/Lib/test/test_descr.py", line 3749, in test_subtype_resurrection self.assertEqual(len(C.container), 1) AssertionError: 0 != 1 However, the third-party test suites (Tornado, Tulip, SQLAlchemy, Django, zope.interface) show no regressions. Regards Antoine. From ron3200 at gmail.com Fri Jun 14 20:11:11 2013 From: ron3200 at gmail.com (Ron Adam) Date: Fri, 14 Jun 2013 13:11:11 -0500 Subject: [Python-Dev] eval and triple quoted strings In-Reply-To: References: <51BB2EDC.80604@livinglogic.de> Message-ID: <51BB5CBF.5070308@gmail.com> On 06/14/2013 10:36 AM, Guido van Rossum wrote: > Not a bug. The same is done for file input -- CRLF is changed to LF before > tokenizing. Should this be the same? python3 -c 'print(bytes("""\r\n""", "utf8"))' b'\r\n' >>> eval('print(bytes("""\r\n""", "utf8"))') b'\n' Ron > On Jun 14, 2013 8:27 AM, "Walter D?rwald" > wrote: > > Hello all! > > This surprised me: > > >>> eval("'''\r\n'''") > '\n' > > Where did the \r go? ast.literal_eval() has the same problem: > > >>> ast.literal_eval("'''\r\n'''") > '\n' > > Is this a bug/worth fixing? > > Servus, > Walter > _________________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/__mailman/listinfo/python-dev > > Unsubscribe: > http://mail.python.org/__mailman/options/python-dev/__guido%40python.org > From ron3200 at gmail.com Fri Jun 14 20:11:11 2013 From: ron3200 at gmail.com (Ron Adam) Date: Fri, 14 Jun 2013 13:11:11 -0500 Subject: [Python-Dev] eval and triple quoted strings In-Reply-To: References: <51BB2EDC.80604@livinglogic.de> Message-ID: <51BB5CBF.5070308@gmail.com> On 06/14/2013 10:36 AM, Guido van Rossum wrote: > Not a bug. The same is done for file input -- CRLF is changed to LF before > tokenizing. Should this be the same? python3 -c 'print(bytes("""\r\n""", "utf8"))' b'\r\n' >>> eval('print(bytes("""\r\n""", "utf8"))') b'\n' Ron > On Jun 14, 2013 8:27 AM, "Walter D?rwald" > wrote: > > Hello all! > > This surprised me: > > >>> eval("'''\r\n'''") > '\n' > > Where did the \r go? ast.literal_eval() has the same problem: > > >>> ast.literal_eval("'''\r\n'''") > '\n' > > Is this a bug/worth fixing? > > Servus, > Walter > _________________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/__mailman/listinfo/python-dev > > Unsubscribe: > http://mail.python.org/__mailman/options/python-dev/__guido%40python.org > From maschu09 at gmail.com Fri Jun 14 21:12:00 2013 From: maschu09 at gmail.com (Martin Schultz) Date: Fri, 14 Jun 2013 21:12:00 +0200 Subject: [Python-Dev] Clean way in python to test for None, empty, scalar, and list/ndarray? A prayer to the gods of Python Message-ID: As much as I love python, the following drives me crazy, and I would wish that some future version would come up with a more consistent approach for this. And please don't reply with "Too bad if you don't know what type your data are..." - if I want to implement some generic functionality, I want to avoid constrains on the user where not absolutely necessary, and I believe this approach is truely pythonic. OK - here comes the problem set. These are in fact several related issues. Clearly, a solution exists for each of them, but you will have to admit that they are all very different in style and therefore unnecessarily complicate matters. If you want to write code which works under many circumstances, it will become quite convoluted. 1. Testing for None: From what I read elsewhere, the preferred solution is `if x is None` (or `if not x is None` in the opposite case). This is fine and it works for scalars, lists, sets, numpy ndarrays,... 2. Testing for empty lists or empty ndarrays: In principle, `len(x) == 0` will do the trick. **BUT** there are several caveats here: - `len(scalar)` raises a TypeError, so you will have to use try and except or find some other way of testing for a scalar value - `len(numpy.array(0))` (i.e. a scalar coded as numpy array) also raises a TypeError ("unsized object") - `len([[]])` returns a length of 1, which is somehow understandable, but - I would argue - perhaps not what one might expect initially Alternatively, numpy arrays have a size attribute, and `numpy.array([]).size`, `numpy.array(8.).size`, and `numpy.array([8.]).size` all return what you would expect. And even `numpy.array([[]]).size` gives you 0. Now, if I could convert everything to a numpy array, this might work. But have you ever tried to assign a list of mixed data types to a numpy array? `numpy.array(["a",1,[2,3],(888,9)])` will fail, even though the list inside is perfectly fine as a list. 3. Testing for scalar: Let's suppose we knew the number of non-empty elements, and this is 1. Then there are occasions when you want to know if you have in fact `6` or `[6]` as an answer (or maybe even `[[6]]`). Obviously, this question is also relevant for numpy arrays. For the latter, a combination of size and ndim can help. For other objects, I would be tempted to use something like `isiterable()`, however, this function doesn't exist, and there are numerous discussions how one could or should find out if an object is iterable - none of them truly intuitive. (and is it true that **every** iterable object is a descendant of collections.Iterable?) 4. Finding the number of elements in an object: From the discussion above, it is already clear that `len(x)` is not very robust for doing this. Just to mention another complication: `len("abcd")` returns 4, even though this is only one string. Of course this is correct, but it's a nuisance if you need to to find the number of elements of a list of strings and if it can happen that you have a scalar string instead of a 1-element list. And, believe me, such situations do occur! 5. Forcing a scalar to become a 1-element list: Unfortunately, `list(77)` throws an error, because 77 is not iterable. `numpy.array(77)` works, but - as we saw above - there will be no len defined for it. Simply writing `[x]` is dangerous, because if x is a list already, it will create `[[77]]`, which you generally don't want. Also, `numpy.array([x])` would create a 2D array if x is already a 1D array or a list. Often, it would be quite useful to know for sure that a function result is provided as a list, regardless of how many elements it contains (because then you can write `res[0]` without risking the danger to throw an exception). Does anyone have a good suggestion for this one? 6. Detecting None values in a list: This is just for completeness. I have seen solutions using `all` which solve this problem (see [question #1270920][1]). I haven't digged into them extensively, but I fear that these will also suffer from the above-mentioned issues if you don't know for sure if you are starting from a list, a numpy array, or a scalar. Enough complaining. Here comes my prayer to the python gods: **Please** - add a good `isiterable` function - add a `size` attribute to all objects (I wouldn't mind if this is None in case you don't really know how to define the size of something, but it would be good to have it, so that `anything.size` would never throw an error - add an `isscalar` function which would at least try to test if something is a scalar (meaning a single entity). Note that this might give different results compared to `isiterable`, because one would consider a scalar string as a scalar even though it is iterable. And if `isscalar` would throw exceptions in cases where it doesn't know what to do: fine - this can be easily captured. - enable the `len()` function for scalar variables such as integers or floats. I would tend to think that 1 is a natural answer to what the length of a number is. [1]: http://stackoverflow.com/questions/1270920/most-concise-way-to-check-whether-a-list-is-empty-or-contains-only-none -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Fri Jun 14 22:03:41 2013 From: brett at python.org (Brett Cannon) Date: Fri, 14 Jun 2013 16:03:41 -0400 Subject: [Python-Dev] Clean way in python to test for None, empty, scalar, and list/ndarray? A prayer to the gods of Python In-Reply-To: References: Message-ID: On Fri, Jun 14, 2013 at 3:12 PM, Martin Schultz wrote: > As much as I love python, the following drives me crazy, and I would wish > that some future version would come up with a more consistent approach for > this. And please don't reply with "Too bad if you don't know what type your > data are..." - if I want to implement some generic functionality, I want to > avoid constrains on the user where not absolutely necessary, and I believe > this approach is truely pythonic. > > OK - here comes the problem set. These are in fact several related issues. > Clearly, a solution exists for each of them, but you will have to admit > that they are all very different in style and therefore unnecessarily > complicate matters. If you want to write code which works under many > circumstances, it will become quite convoluted. > > 1. Testing for None: > > From what I read elsewhere, the preferred solution is `if x is None` (or > `if not x is None` in the opposite case). This is fine and it works for > scalars, lists, sets, numpy ndarrays,... > Should actually be ``if x is not None``. [SNIP] > > - add a good `isiterable` function > Done: isinstance(x, collections.abc.Iterable) > - add a `size` attribute to all objects (I wouldn't mind if this is None > in case you don't really know how to define the size of something, but it > would be good to have it, so that `anything.size` would never throw an error > This is what len() is for. I don't know why numpy doesn't define the __len__ method on their array types for that. We can't force every object to have it as it doesn't make sense in all cases. > - add an `isscalar` function which would at least try to test if > something is a scalar (meaning a single entity). Note that this might give > different results compared to `isiterable`, because one would consider a > scalar string as a scalar even though it is iterable. And if `isscalar` > would throw exceptions in cases where it doesn't know what to do: fine - > this can be easily captured. > The numbers module has a bunch of ABCs you can use to test for integrals, etc. just as I suggested for iterables. > - enable the `len()` function for scalar variables such as integers or > floats. I would tend to think that 1 is a natural answer to what the length > of a number is. > The len() function is specifically for containers so it would not make sense to ask for the length of something you can't put something into or iterate over. This is actually a perfect case of using the new single-dispatch generic function work that has landed in Python 3.4 ( http://docs.python.org/3.4/library/functools.html#functools.singledispatch). With that you could write your own custom_len() function that dispatches on the type to return exactly what you are after. E.g.:: @functools.singledispatch def custom_length(ob): return len(ob) @custom_length.register(int) _(ob): return 1 And on you go for all the types you want to special-case. -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Fri Jun 14 22:20:50 2013 From: robert.kern at gmail.com (Robert Kern) Date: Fri, 14 Jun 2013 21:20:50 +0100 Subject: [Python-Dev] Clean way in python to test for None, empty, scalar, and list/ndarray? A prayer to the gods of Python In-Reply-To: References: Message-ID: On 2013-06-14 21:03, Brett Cannon wrote: > > On Fri, Jun 14, 2013 at 3:12 PM, Martin Schultz > wrote: > - add a `size` attribute to all objects (I wouldn't mind if this is None > in case you don't really know how to define the size of something, but it > would be good to have it, so that `anything.size` would never throw an error > > This is what len() is for. I don't know why numpy doesn't define the __len__ > method on their array types for that. It does. It gives the size of the first axis, i.e. the one accessed by simple indexing with an integer: some_array[i]. The `size` attribute givens the total number of items in the possibly-multidimensional array. However, one of the other axes can be 0-length, so the array will have no elements but the length will be nonzero. [~] |4> np.empty([3,4,0]) array([], shape=(3, 4, 0), dtype=float64) [~] |5> np.empty([3,4,0])[1] array([], shape=(4, 0), dtype=float64) [~] |6> len(np.empty([3,4,0])) 3 -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From rdmurray at bitdance.com Fri Jun 14 22:55:33 2013 From: rdmurray at bitdance.com (R. David Murray) Date: Fri, 14 Jun 2013 16:55:33 -0400 Subject: [Python-Dev] Clean way in python to test for None, empty, scalar, and list/ndarray? A prayer to the gods of Python In-Reply-To: References: Message-ID: <20130614205534.A595A250BE9@webabinitio.net> On Fri, 14 Jun 2013 21:12:00 +0200, Martin Schultz wrote: > 2. Testing for empty lists or empty ndarrays: > > In principle, `len(x) == 0` will do the trick. **BUT** there are several > caveats here: > - `len(scalar)` raises a TypeError, so you will have to use try and > except or find some other way of testing for a scalar value > - `len(numpy.array(0))` (i.e. a scalar coded as numpy array) also raises > a TypeError ("unsized object") > - `len([[]])` returns a length of 1, which is somehow understandable, > but - I would argue - perhaps not what one might expect initially > > Alternatively, numpy arrays have a size attribute, and > `numpy.array([]).size`, `numpy.array(8.).size`, and > `numpy.array([8.]).size` all return what you would expect. And even > `numpy.array([[]]).size` gives you 0. Now, if I could convert everything to > a numpy array, this might work. But have you ever tried to assign a list of > mixed data types to a numpy array? `numpy.array(["a",1,[2,3],(888,9)])` > will fail, even though the list inside is perfectly fine as a list. In general you test whether nor not something is empty in Python by testing its truth value. Empty things are False. Numpy seems to follow this using size, from the limited examples you have given >>> bool(numpy.array([[]]) False >>> bool(numpy.array([[1]]) True I have no idea what the definition of numpy.array.size is that it would return 0 for [[]], so its return value obviously defies my intuition as much as len([[]]) seems to have initially defied yours :) > 3. Testing for scalar: > > Let's suppose we knew the number of non-empty elements, and this is 1. > Then there are occasions when you want to know if you have in fact `6` or > `[6]` as an answer (or maybe even `[[6]]`). Obviously, this question is > also relevant for numpy arrays. For the latter, a combination of size and > ndim can help. For other objects, I would be tempted to use something like > `isiterable()`, however, this function doesn't exist, and there are > numerous discussions how one could or should find out if an object is > iterable - none of them truly intuitive. (and is it true that **every** > iterable object is a descendant of collections.Iterable?) No, but...I'm not 100% sure about this as I tend to stay away from ABCs myself, but my understanding is that collections.Iterable checks if an object is iterable when you use it in an isinstance check. There are probably ways to fool it, but I think you could argue that any such data types are broken. > 4. Finding the number of elements in an object: > > From the discussion above, it is already clear that `len(x)` is not very > robust for doing this. Just to mention another complication: `len("abcd")` > returns 4, even though this is only one string. Of course this is correct, > but it's a nuisance if you need to to find the number of elements of a list > of strings and if it can happen that you have a scalar string instead of a > 1-element list. And, believe me, such situations do occur! len is robust when you consider that it only applies to sequences (see below). (I don't know what it means to "code a scaler as a numpy array", but if it is still a scaler, it makes sense that it raises a TypeError on len...it should.) > 5. Forcing a scalar to become a 1-element list: > > Unfortunately, `list(77)` throws an error, because 77 is not iterable. > `numpy.array(77)` works, but - as we saw above - there will be no len > defined for it. Simply writing `[x]` is dangerous, because if x is a list > already, it will create `[[77]]`, which you generally don't want. Also, > `numpy.array([x])` would create a 2D array if x is already a 1D array or a > list. Often, it would be quite useful to know for sure that a function > result is provided as a list, regardless of how many elements it contains > (because then you can write `res[0]` without risking the danger to throw an > exception). Does anyone have a good suggestion for this one? Well, no. If the list is empty res[0] will throw an error. You need to know both that it is indexable (note: not iterable...an object can be iterable without being indexable) and that it is non-empty. Well behaved objects should I think pass an isinstance check against collections.Sequence. (I can't think of a good way to check for indexability without the abc.) > Enough complaining. Here comes my prayer to the python gods: **Please** > > - add a good `isiterable` function That would be spelled isinstance(x, collections.Iterable), it seems. > - add a `size` attribute to all objects (I wouldn't mind if this is None > in case you don't really know how to define the size of something, but it > would be good to have it, so that `anything.size` would never throw an error Why? What is the definition of 'size' that makes it useful outside of numpy? > - add an `isscalar` function which would at least try to test if something > is a scalar (meaning a single entity). Note that this might give different > results compared to `isiterable`, because one would consider a scalar > string as a scalar even though it is iterable. And if `isscalar` would > throw exceptions in cases where it doesn't know what to do: fine - this can > be easily captured. This I sort of agree with. I've often enough wanted to know if something is a non-string iterable. But you'd have to decide if bytes/bytearray is a sequence of integers or a scaler... > - enable the `len()` function for scalar variables such as integers or > floats. I would tend to think that 1 is a natural answer to what the length > of a number is. That would screw up the ABC type hierarchy...the existence of len indicates that an iterable is indexable. --David From pje at telecommunity.com Fri Jun 14 23:03:14 2013 From: pje at telecommunity.com (PJ Eby) Date: Fri, 14 Jun 2013 17:03:14 -0400 Subject: [Python-Dev] eval and triple quoted strings In-Reply-To: <51BB5CBF.5070308@gmail.com> References: <51BB2EDC.80604@livinglogic.de> <51BB5CBF.5070308@gmail.com> Message-ID: On Fri, Jun 14, 2013 at 2:11 PM, Ron Adam wrote: > > > On 06/14/2013 10:36 AM, Guido van Rossum wrote: >> >> Not a bug. The same is done for file input -- CRLF is changed to LF before >> tokenizing. > > > > Should this be the same? > > > python3 -c 'print(bytes("""\r\n""", "utf8"))' > b'\r\n' > > >>>> eval('print(bytes("""\r\n""", "utf8"))') > b'\n' No, but: eval(r'print(bytes("""\r\n""", "utf8"))') should be. (And is.) What I believe you and Walter are missing is that the \r\n in the eval strings are converted early if you don't make the enclosing string raw. So what you're eval-ing is not what you think you are eval-ing, hence the confusion. From robert.kern at gmail.com Sat Jun 15 00:31:47 2013 From: robert.kern at gmail.com (Robert Kern) Date: Fri, 14 Jun 2013 23:31:47 +0100 Subject: [Python-Dev] Clean way in python to test for None, empty, scalar, and list/ndarray? A prayer to the gods of Python In-Reply-To: <20130614205534.A595A250BE9@webabinitio.net> References: <20130614205534.A595A250BE9@webabinitio.net> Message-ID: On 2013-06-14 21:55, R. David Murray wrote: > On Fri, 14 Jun 2013 21:12:00 +0200, Martin Schultz wrote: >> 2. Testing for empty lists or empty ndarrays: >> >> In principle, `len(x) == 0` will do the trick. **BUT** there are several >> caveats here: >> - `len(scalar)` raises a TypeError, so you will have to use try and >> except or find some other way of testing for a scalar value >> - `len(numpy.array(0))` (i.e. a scalar coded as numpy array) also raises >> a TypeError ("unsized object") >> - `len([[]])` returns a length of 1, which is somehow understandable, >> but - I would argue - perhaps not what one might expect initially >> >> Alternatively, numpy arrays have a size attribute, and >> `numpy.array([]).size`, `numpy.array(8.).size`, and >> `numpy.array([8.]).size` all return what you would expect. And even >> `numpy.array([[]]).size` gives you 0. Now, if I could convert everything to >> a numpy array, this might work. But have you ever tried to assign a list of >> mixed data types to a numpy array? `numpy.array(["a",1,[2,3],(888,9)])` >> will fail, even though the list inside is perfectly fine as a list. > > In general you test whether nor not something is empty in Python by > testing its truth value. Empty things are False. Numpy seems to > follow this using size, from the limited examples you have given > > >>> bool(numpy.array([[]]) > False > >>> bool(numpy.array([[1]]) > True numpy does not do so. Empty arrays are extremely rare and testing for them rarer (rarer still is testing for emptiness not knowing if it is an array or some other sequence). What people usually want from bool(some_array) is either some_array.all() or some_array.any(). In the face of this ambiguity, numpy refuses the temptation to guess and raises an exception explaining matters. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From robert.kern at gmail.com Sat Jun 15 00:53:53 2013 From: robert.kern at gmail.com (Robert Kern) Date: Fri, 14 Jun 2013 23:53:53 +0100 Subject: [Python-Dev] Clean way in python to test for None, empty, scalar, and list/ndarray? A prayer to the gods of Python In-Reply-To: References: <20130614205534.A595A250BE9@webabinitio.net> Message-ID: On 2013-06-14 23:31, Robert Kern wrote: > On 2013-06-14 21:55, R. David Murray wrote: >> On Fri, 14 Jun 2013 21:12:00 +0200, Martin Schultz wrote: >>> 2. Testing for empty lists or empty ndarrays: >>> >>> In principle, `len(x) == 0` will do the trick. **BUT** there are several >>> caveats here: >>> - `len(scalar)` raises a TypeError, so you will have to use try and >>> except or find some other way of testing for a scalar value >>> - `len(numpy.array(0))` (i.e. a scalar coded as numpy array) also raises >>> a TypeError ("unsized object") >>> - `len([[]])` returns a length of 1, which is somehow understandable, >>> but - I would argue - perhaps not what one might expect initially >>> >>> Alternatively, numpy arrays have a size attribute, and >>> `numpy.array([]).size`, `numpy.array(8.).size`, and >>> `numpy.array([8.]).size` all return what you would expect. And even >>> `numpy.array([[]]).size` gives you 0. Now, if I could convert everything to >>> a numpy array, this might work. But have you ever tried to assign a list of >>> mixed data types to a numpy array? `numpy.array(["a",1,[2,3],(888,9)])` >>> will fail, even though the list inside is perfectly fine as a list. >> >> In general you test whether nor not something is empty in Python by >> testing its truth value. Empty things are False. Numpy seems to >> follow this using size, from the limited examples you have given >> >> >>> bool(numpy.array([[]]) >> False >> >>> bool(numpy.array([[1]]) >> True > > numpy does not do so. Empty arrays are extremely rare and testing for them rarer > (rarer still is testing for emptiness not knowing if it is an array or some > other sequence). What people usually want from bool(some_array) is either > some_array.all() or some_array.any(). In the face of this ambiguity, numpy > refuses the temptation to guess and raises an exception explaining matters. Actually, that's a bit of a lie. In the empty case and the one-element case, we do return a bool, False for empty and bool(element) for whatever that one element is. Anything else raises the exception since we don't know whether it is all() or any() that was desired. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From brett at python.org Sat Jun 15 00:57:01 2013 From: brett at python.org (Brett Cannon) Date: Fri, 14 Jun 2013 18:57:01 -0400 Subject: [Python-Dev] [Python-checkins] cpython: Closes issue 17947. Adds PEP-0435 (Enum, IntEnum) to the stdlib. In-Reply-To: <3bWtm22z9dz7Ljj@mail.python.org> References: <3bWtm22z9dz7Ljj@mail.python.org> Message-ID: Ethan, did you forget to run ``hg add`` before committing? If not then why the heck did we argue over enums for so long if this was all it took to make everyone happy? =) On Fri, Jun 14, 2013 at 3:31 AM, ethan.furman wrote: > http://hg.python.org/cpython/rev/fae92309c3be > changeset: 84117:fae92309c3be > parent: 84115:af27c661d4fb > user: Ethan Furman > date: Fri Jun 14 00:30:27 2013 -0700 > summary: > Closes issue 17947. Adds PEP-0435 (Enum, IntEnum) to the stdlib. > > files: > Doc/library/datatypes.rst | 1 + > 1 files changed, 1 insertions(+), 0 deletions(-) > > > diff --git a/Doc/library/datatypes.rst b/Doc/library/datatypes.rst > --- a/Doc/library/datatypes.rst > +++ b/Doc/library/datatypes.rst > @@ -30,3 +30,4 @@ > copy.rst > pprint.rst > reprlib.rst > + enum.rst > > -- > Repository URL: http://hg.python.org/cpython > > _______________________________________________ > Python-checkins mailing list > Python-checkins at python.org > http://mail.python.org/mailman/listinfo/python-checkins > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From victor.stinner at gmail.com Sat Jun 15 01:03:42 2013 From: victor.stinner at gmail.com (Victor Stinner) Date: Sat, 15 Jun 2013 01:03:42 +0200 Subject: [Python-Dev] Allow calling PyMem_Malloc() without the GIL held in Python 3.4 In-Reply-To: References: Message-ID: I commited the new API (little bit different than my last patch on issue #3329): http://hg.python.org/cpython/rev/6661a8154eb3 The documentation will be available in a few minutes at: http://docs.python.org/3/c-api/memory.html 2013/6/14 Kristj?n Valur J?nsson : >> Removing the GIL restriction would help to replace direct calls to >> malloc() with PyMem_Malloc(). Using PyMem_SetAllocators(), an application >> would be able to replace memory allocators, and these allocators would be >> used "everywhere". >> => see http://bugs.python.org/issue18203 > > To keep this interesting, I have a somewhat different opinion to Victor :) > have put comments in the original defect, but would like to repeat them here. > IMHO, keeping the GIL restriction on PyMem_MALLOC is useful. > 1) It allows it to be replaced with PyObject_MALLOC(). Or anything else. In particular, an implementer is free to add memory profiling support and other things without worrying about implementation details. Requiring it to be GIL free severely limits what it can do. For example, it would be forbidden to delegate to PyObject_MALLOC when debugging. For my own pytracemalloc tool, holding the GIL while calling PyMem_Malloc() is required to be able to retrieve the Python filename and line number of the caller. So you convinced me :-) I am also worried by the backward compatibility, even if I expect that only a very few developers replaced Python memory allocators. A custom memory allocator may not be thread-safe, so the GIL can also be convinient. I added new functions in the "final" API: PyMem_RawMalloc(), PyMem_RawRealloc(), PyMem_RawFree(). These functions are just wrapper for malloc(), realloc() and free(). The GIL does not need to be hold. No process is done before/after at all. Behaviour of PyMem_RawMalloc(0) is platform depend for example. "size > PY_SSIZE_T_MAX" check is not done neither, but it may be interesting to add this check for security reasons (it is already in place for PyMem_Malloc and PyObject_Malloc). Using these new functions instead of malloc/realloc/free is interesting because the internal functions can be replaced with PyMem_SetRawAllocators() and many checks are added in debug mode (ex: check for buffer under- and overflow). PyObject_Malloc() was not documented, so I did not document PyObject_SetAllocators(). In the final API, I added a new PyMemAllocators structure to simplify the API. I also made _PyObject_SetArenaAllocators() private because I don't like its API (it is not homogenous with PyMem_SetAllocators) and it is concerned by less use cases. I prefer to wait a little before making this API public. I didn't use "#ifndef Py_LIMITED_API", so all new functions are part of the stable API. Is it correct? Victor From solipsis at pitrou.net Sat Jun 15 03:04:31 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sat, 15 Jun 2013 03:04:31 +0200 Subject: [Python-Dev] cpython: Issue #3329: Add new APIs to customize memory allocators References: <3bXH1R0dqGz7LkT@mail.python.org> Message-ID: <20130615030431.679bfc1b@fsol> On Sat, 15 Jun 2013 00:44:11 +0200 (CEST) victor.stinner wrote: > http://hg.python.org/cpython/rev/6661a8154eb3 > changeset: 84127:6661a8154eb3 > user: Victor Stinner > date: Sat Jun 15 00:37:46 2013 +0200 > summary: > Issue #3329: Add new APIs to customize memory allocators > > * Add a new PyMemAllocators structure > * New functions: > > - PyMem_RawMalloc(), PyMem_RawRealloc(), PyMem_RawFree(): GIL-free memory > allocator functions > - PyMem_GetRawAllocators(), PyMem_SetRawAllocators() > - PyMem_GetAllocators(), PyMem_SetAllocators() > - PyMem_SetupDebugHooks() > - _PyObject_GetArenaAllocators(), _PyObject_SetArenaAllocators() My two cents, but I would prefer if this whole changeset was reverted. I think it adds too much complexity in the memory allocation APIs, for a pretty specialized benefit. IMHO, we should be able to get by with less allocation APIs (why the new _Raw APIs) and less hook-setting functions. Regards Antoine. From victor.stinner at gmail.com Sat Jun 15 03:54:50 2013 From: victor.stinner at gmail.com (Victor Stinner) Date: Sat, 15 Jun 2013 03:54:50 +0200 Subject: [Python-Dev] cpython: Issue #3329: Add new APIs to customize memory allocators In-Reply-To: <20130615030431.679bfc1b@fsol> References: <3bXH1R0dqGz7LkT@mail.python.org> <20130615030431.679bfc1b@fsol> Message-ID: 2013/6/15 Antoine Pitrou : >> http://hg.python.org/cpython/rev/6661a8154eb3 >> ... >> Issue #3329: Add new APIs to customize memory allocators >> >> * Add a new PyMemAllocators structure >> * New functions: >> >> - PyMem_RawMalloc(), PyMem_RawRealloc(), PyMem_RawFree(): GIL-free memory >> allocator functions >> - PyMem_GetRawAllocators(), PyMem_SetRawAllocators() >> - PyMem_GetAllocators(), PyMem_SetAllocators() >> - PyMem_SetupDebugHooks() >> - _PyObject_GetArenaAllocators(), _PyObject_SetArenaAllocators() > > My two cents, but I would prefer if this whole changeset was reverted. > I think it adds too much complexity in the memory allocation APIs, > for a pretty specialized benefit. IMHO, we should be able to get by with > less allocation APIs (why the new _Raw APIs) and less hook-setting > functions. Ok, I reverted my commit. I posted my initial patch 3 months ago on the bug tracker. I got some reviews and discussed with Kristj?n Valur J?nsson who heavily modified Python for his game at CCP. I started two threads on python-dev this week (ok, only two days ago). I thaugh that the last known issues were fixed with the addition of PyMem_SetupDebugHooks() (to avoid an environment variable, as asked by Nick) and PyMem_RawMalloc() (have a GIL-free allocator). I will work on a PEP to explain all these new functions and their use cases. ** The addition of PyMem_RawMalloc() is motivated by the issue #18203 (Replace calls to malloc() with PyMem_Malloc()). The goal is to be able to setup a custom allocator for *all* allocation made by Python, so malloc() should not be called directly. PyMem_RawMalloc() is required in places where the GIL is not held (ex: in os.getcwd() on Windows). PyMem_Malloc() is misused (called without the GIL held) in different places. Examples: the readline modules and functions called at Python startup, including main(). Replacing PyMem_Malloc() with malloc() would not allow to use the custom allocator everywhere, so PyMem_RawMalloc() is also required here. The last point is an extension of the issue #18203: some external libraries like zlib or OpenSSL are also calling malloc() directly. But Python can configure these libraries to use a custom memory allocator. I plan to configure external libraries to use PyMem_GetRawAllocators() if PyMem_SetRawAllocators() was called (if PyMem_RawMalloc is not simply malloc) and if setting a custom allocator only affect a function and not the whole library. Victor From greg.ewing at canterbury.ac.nz Sat Jun 15 06:08:21 2013 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sat, 15 Jun 2013 16:08:21 +1200 Subject: [Python-Dev] eval and triple quoted strings In-Reply-To: References: <51BB2EDC.80604@livinglogic.de> Message-ID: <51BBE8B5.1080706@canterbury.ac.nz> Guido van Rossum wrote: > Not a bug. The same is done for file input -- CRLF is changed to LF > before tokenizing. I'm not convinced it's reasonable behaviour to re-scan the string as though it's being read from a file. It's a Python string, so it's already been through whatever line-ending transformation is appropriate to get it into memory. -- Greg From ncoghlan at gmail.com Sat Jun 15 06:57:49 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 15 Jun 2013 14:57:49 +1000 Subject: [Python-Dev] PEP 442 accepted In-Reply-To: <20130614193441.04b219c8@fsol> References: <20130614193441.04b219c8@fsol> Message-ID: On 15 June 2013 03:34, Antoine Pitrou wrote: > On Wed, 5 Jun 2013 09:10:54 -0700 > Benjamin Peterson wrote: >> I (and Guido) are accepting PEP 442 (Safe object finalization) on the >> condition that finalizers are only ever called once globally. > > Ok, so there's an issue with that condition: it can't be upholded on > non-GC objects. Creating a non-GC object is quite obscure and rare, > though, since it requires basically a class with no __dict__ and an > empty __slots__: > > class C: > __slots__ = () > survivors = [] > > def __del__(self): > self.survivors.append(self) > > > In this case, a C instance's __del__ will be called every time > destruction is attempted, not only once. Is that a realistic problem? So, to trigger that __del__() method a second time, such an object would have to be: 1. Defined in the first place (the use cases for stateless objects with destructors seem rare...) 2. Hanging off a reference cycle 3. Which then gets resurrected If it isn't easy to fix by changing the way such classes are constructed by typeobject.c (i.e. force them to be GC-aware when they define a __del__ method, since they may still be hanging off the edge of a reference cycle, even if they can't form one themselves), I'd say chalk it up as a CPython wart and not worry about it for now. As far as the test_subtype_resurrection failure goes, I think the associated comment says it all: "(On non-CPython implementations, however, __del__ is typically not called again.)" The test should just be rewritten to check that __del__ is only called once in the face of resurrection. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Sat Jun 15 07:18:01 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 15 Jun 2013 15:18:01 +1000 Subject: [Python-Dev] eval and triple quoted strings In-Reply-To: <51BBE8B5.1080706@canterbury.ac.nz> References: <51BB2EDC.80604@livinglogic.de> <51BBE8B5.1080706@canterbury.ac.nz> Message-ID: On 15 June 2013 14:08, Greg Ewing wrote: > Guido van Rossum wrote: >> >> Not a bug. The same is done for file input -- CRLF is changed to LF before >> tokenizing. > > > I'm not convinced it's reasonable behaviour to re-scan the > string as though it's being read from a file. It's a Python > string, so it's already been through whatever line-ending > transformation is appropriate to get it into memory. No, that's not the way the Python compiler works. The transformation Guido is talking about is the way the tokenizer identifiers "NEWLINE" tokens: >>> list(tokenize.tokenize((l for l in (b"""'\r\n'""", b"")).__next__))[2] TokenInfo(type=4 (NEWLINE), string='\r\n', start=(1, 1), end=(1, 3), line="'\r\n'") This long predates universal newlines mode - it's part of the compilation process, not part of the IO system. The compiler then sees the NEWLINE token in the tokenizer output, and inserts a "\n" into the triple-quoted string. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Sat Jun 15 07:50:58 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 15 Jun 2013 15:50:58 +1000 Subject: [Python-Dev] cpython: Issue #3329: Add new APIs to customize memory allocators In-Reply-To: References: <3bXH1R0dqGz7LkT@mail.python.org> <20130615030431.679bfc1b@fsol> Message-ID: On 15 June 2013 11:54, Victor Stinner wrote: > 2013/6/15 Antoine Pitrou : >>> http://hg.python.org/cpython/rev/6661a8154eb3 >>> ... >>> Issue #3329: Add new APIs to customize memory allocators >>> >>> * Add a new PyMemAllocators structure >>> * New functions: >>> >>> - PyMem_RawMalloc(), PyMem_RawRealloc(), PyMem_RawFree(): GIL-free memory >>> allocator functions >>> - PyMem_GetRawAllocators(), PyMem_SetRawAllocators() >>> - PyMem_GetAllocators(), PyMem_SetAllocators() >>> - PyMem_SetupDebugHooks() >>> - _PyObject_GetArenaAllocators(), _PyObject_SetArenaAllocators() >> >> My two cents, but I would prefer if this whole changeset was reverted. >> I think it adds too much complexity in the memory allocation APIs, >> for a pretty specialized benefit. IMHO, we should be able to get by with >> less allocation APIs (why the new _Raw APIs) and less hook-setting >> functions. > > Ok, I reverted my commit. > > I posted my initial patch 3 months ago on the bug tracker. I got some > reviews and discussed with Kristj?n Valur J?nsson who heavily modified > Python for his game at CCP. I started two threads on python-dev this > week (ok, only two days ago). I thaugh that the last known issues were > fixed with the addition of PyMem_SetupDebugHooks() (to avoid an > environment variable, as asked by Nick) and PyMem_RawMalloc() (have a > GIL-free allocator). > > I will work on a PEP to explain all these new functions and their use cases. I think the new APIs are mostly valid and well-justified, but agree a PEP is a good idea. Yes, it's a complex solution, but it's solving a complex problem that arises when embedding CPython inside executables that need to run on non-traditional platforms where the simple C defined malloc/realloc/free trio is inadequate. This is a complementary effort to PEP 432 - that aims to simplify embedding CPython in general, while Victor's efforts here specifically focus on situations where it is necessary to better map CPython to an underlying memory model that differs from the traditional C one. While the "single heap" model of memory enshrined in the C standard is certainly the most common model, it's far from being the only one, and these days CPython also gets used in those other environments. About the only simplification I can see is that PyMem_RawMalloc(), PyMem_RawRealloc(), PyMem_RawFree() could perhaps be handled with preprocessor macros instead of permitting runtime reconfiguration. Allowing the memory allocations for the CPython runtime to be handled separately from those for arbitrary C libraries loaded into the process seems reasonable, though. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From greg at krypto.org Sat Jun 15 09:54:10 2013 From: greg at krypto.org (Gregory P. Smith) Date: Sat, 15 Jun 2013 00:54:10 -0700 Subject: [Python-Dev] cpython: Issue #3329: Add new APIs to customize memory allocators In-Reply-To: References: <3bXH1R0dqGz7LkT@mail.python.org> <20130615030431.679bfc1b@fsol> Message-ID: fwiw i'm also supportive of adding these apis. Lets PEP away to iron out any details or document disagreements but overall I'd also like to see something a lot like these go in. -gps On Fri, Jun 14, 2013 at 10:50 PM, Nick Coghlan wrote: > On 15 June 2013 11:54, Victor Stinner wrote: > > 2013/6/15 Antoine Pitrou : > >>> http://hg.python.org/cpython/rev/6661a8154eb3 > >>> ... > >>> Issue #3329: Add new APIs to customize memory allocators > >>> > >>> * Add a new PyMemAllocators structure > >>> * New functions: > >>> > >>> - PyMem_RawMalloc(), PyMem_RawRealloc(), PyMem_RawFree(): GIL-free > memory > >>> allocator functions > >>> - PyMem_GetRawAllocators(), PyMem_SetRawAllocators() > >>> - PyMem_GetAllocators(), PyMem_SetAllocators() > >>> - PyMem_SetupDebugHooks() > >>> - _PyObject_GetArenaAllocators(), _PyObject_SetArenaAllocators() > >> > >> My two cents, but I would prefer if this whole changeset was reverted. > >> I think it adds too much complexity in the memory allocation APIs, > >> for a pretty specialized benefit. IMHO, we should be able to get by with > >> less allocation APIs (why the new _Raw APIs) and less hook-setting > >> functions. > > > > Ok, I reverted my commit. > > > > I posted my initial patch 3 months ago on the bug tracker. I got some > > reviews and discussed with Kristj?n Valur J?nsson who heavily modified > > Python for his game at CCP. I started two threads on python-dev this > > week (ok, only two days ago). I thaugh that the last known issues were > > fixed with the addition of PyMem_SetupDebugHooks() (to avoid an > > environment variable, as asked by Nick) and PyMem_RawMalloc() (have a > > GIL-free allocator). > > > > I will work on a PEP to explain all these new functions and their use > cases. > > I think the new APIs are mostly valid and well-justified, but agree a > PEP is a good idea. > > Yes, it's a complex solution, but it's solving a complex problem that > arises when embedding CPython inside executables that need to run on > non-traditional platforms where the simple C defined > malloc/realloc/free trio is inadequate. > > This is a complementary effort to PEP 432 - that aims to simplify > embedding CPython in general, while Victor's efforts here specifically > focus on situations where it is necessary to better map CPython to an > underlying memory model that differs from the traditional C one. While > the "single heap" model of memory enshrined in the C standard is > certainly the most common model, it's far from being the only one, and > these days CPython also gets used in those other environments. > > About the only simplification I can see is that PyMem_RawMalloc(), > PyMem_RawRealloc(), PyMem_RawFree() could perhaps be handled with > preprocessor macros instead of permitting runtime reconfiguration. > Allowing the memory allocations for the CPython runtime to be handled > separately from those for arbitrary C libraries loaded into the > process seems reasonable, though. > > Cheers, > Nick. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/greg%40krypto.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Sat Jun 15 12:55:57 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sat, 15 Jun 2013 12:55:57 +0200 Subject: [Python-Dev] PEP 442 accepted In-Reply-To: References: <20130614193441.04b219c8@fsol> Message-ID: <20130615125557.6455f6a5@fsol> On Sat, 15 Jun 2013 14:57:49 +1000 Nick Coghlan wrote: > On 15 June 2013 03:34, Antoine Pitrou wrote: > > On Wed, 5 Jun 2013 09:10:54 -0700 > > Benjamin Peterson wrote: > >> I (and Guido) are accepting PEP 442 (Safe object finalization) on the > >> condition that finalizers are only ever called once globally. > > > > Ok, so there's an issue with that condition: it can't be upholded on > > non-GC objects. Creating a non-GC object is quite obscure and rare, > > though, since it requires basically a class with no __dict__ and an > > empty __slots__: > > > > class C: > > __slots__ = () > > survivors = [] > > > > def __del__(self): > > self.survivors.append(self) > > > > > > In this case, a C instance's __del__ will be called every time > > destruction is attempted, not only once. Is that a realistic problem? > > So, to trigger that __del__() method a second time, such an object > would have to be: > > 1. Defined in the first place (the use cases for stateless objects > with destructors seem rare...) > 2. Hanging off a reference cycle > 3. Which then gets resurrected They don't need to hang off a reference cycle. You can resurrect a non-cyclic object from __del__ too. But, yeah, stateless objects must be pretty rare, since by definition they cannot represent anything (except perhaps "nothing"). > i.e. force them to be GC-aware when they > define a __del__ method, since they may still be hanging off the edge > of a reference cycle, even if they can't form one themselves Objects which hang off the edge of a reference cycle don't need to be GC-aware, since traversing them isn't needed to be break the cycle. I think "chalking it up as a CPython wart" is a reasonable solution, I just wanted to ask. Regards Antoine. From solipsis at pitrou.net Sat Jun 15 13:01:25 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sat, 15 Jun 2013 13:01:25 +0200 Subject: [Python-Dev] cpython: Issue #3329: Add new APIs to customize memory allocators In-Reply-To: References: <3bXH1R0dqGz7LkT@mail.python.org> <20130615030431.679bfc1b@fsol> Message-ID: <20130615130125.1d4ef1f7@fsol> On Sat, 15 Jun 2013 03:54:50 +0200 Victor Stinner wrote: > The addition of PyMem_RawMalloc() is motivated by the issue #18203 > (Replace calls to malloc() with PyMem_Malloc()). The goal is to be > able to setup a custom allocator for *all* allocation made by Python, > so malloc() should not be called directly. PyMem_RawMalloc() is > required in places where the GIL is not held (ex: in os.getcwd() on > Windows). We already had this discussion on IRC and this argument isn't very convincing to me. If os.getcwd() doesn't hold the GIL while allocating memory, then you should fix it to hold the GIL while allocating memory. I don't like the idea of adding of third layer of allocation APIs. The dichotomy between PyObject_Malloc and PyMem_Malloc is already a bit gratuitous (i.e. not motivated by any actual real-world concern, as far as I can tell). As for the debug functions you added: PyMem_GetRawAllocators(), PyMem_SetRawAllocators(), PyMem_GetAllocators(), PyMem_SetAllocators(), PyMem_SetupDebugHooks(), _PyObject_GetArenaAllocators(), _PyObject_SetArenaAllocators(). Well, do we need all *7* of them? Can't you try to make that 2 or 3? Regards Antoine. From ncoghlan at gmail.com Sat Jun 15 14:22:33 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 15 Jun 2013 22:22:33 +1000 Subject: [Python-Dev] cpython: Issue #3329: Add new APIs to customize memory allocators In-Reply-To: <20130615130125.1d4ef1f7@fsol> References: <3bXH1R0dqGz7LkT@mail.python.org> <20130615030431.679bfc1b@fsol> <20130615130125.1d4ef1f7@fsol> Message-ID: On 15 June 2013 21:01, Antoine Pitrou wrote: > On Sat, 15 Jun 2013 03:54:50 +0200 > Victor Stinner wrote: >> The addition of PyMem_RawMalloc() is motivated by the issue #18203 >> (Replace calls to malloc() with PyMem_Malloc()). The goal is to be >> able to setup a custom allocator for *all* allocation made by Python, >> so malloc() should not be called directly. PyMem_RawMalloc() is >> required in places where the GIL is not held (ex: in os.getcwd() on >> Windows). > > We already had this discussion on IRC and this argument isn't very > convincing to me. If os.getcwd() doesn't hold the GIL while allocating > memory, then you should fix it to hold the GIL while allocating memory. > > I don't like the idea of adding of third layer of allocation APIs. The > dichotomy between PyObject_Malloc and PyMem_Malloc is already a bit > gratuitous (i.e. not motivated by any actual real-world concern, as > far as I can tell). The only reason for the small object allocator to exist is because operating system allocators generally aren't optimised for frequent allocation and deallocation of small objects. You can gain a *lot* of speed from handling those inside the application. As the allocations grow in size, though, the application level allocator just becomes useless overhead, so it's better to delegate those operations directly to the OS. However, it's still desirable to be able to monitor those direct allocations in debug mode, thus it makes sense to have a GIL protected direct allocation API as well. You could try to hide the existence of the latter behaviour and treat it as a private API, but why? For custom allocators, it's useful to be able to *ensure* you can bypass CPython's small object allocator, rather than having to rely on it being bypassed for allocations above a certain size. > As for the debug functions you added: PyMem_GetRawAllocators(), > PyMem_SetRawAllocators(), PyMem_GetAllocators(), PyMem_SetAllocators(), > PyMem_SetupDebugHooks(), _PyObject_GetArenaAllocators(), > _PyObject_SetArenaAllocators(). Well, do we need all *7* of them? Can't > you try to make that 2 or 3? Faux simplicity that is achieved only by failing to model a complex problem domain correctly is a bad idea (if we were satisfied with that, we could stick with the status quo). The only question mark in my mind is over the GIL-free raw allocation APIs. I think it makes sense to at least conditionally define those as macros so an embedding application can redirect *just* the allocations made by the CPython runtime (rather than having to redefine malloc/realloc/free when building Python), but I don't believe the case has been adequately made for making the raw APIs configurable at runtime. Dropping that aspect would at least eliminate the PyMem_(Get|Set)RawAllocators() APIs. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From solipsis at pitrou.net Sat Jun 15 14:41:15 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sat, 15 Jun 2013 14:41:15 +0200 Subject: [Python-Dev] cpython: Issue #3329: Add new APIs to customize memory allocators In-Reply-To: References: <3bXH1R0dqGz7LkT@mail.python.org> <20130615030431.679bfc1b@fsol> <20130615130125.1d4ef1f7@fsol> Message-ID: <20130615144115.0a7e3176@fsol> On Sat, 15 Jun 2013 22:22:33 +1000 Nick Coghlan wrote: > On 15 June 2013 21:01, Antoine Pitrou wrote: > > On Sat, 15 Jun 2013 03:54:50 +0200 > > Victor Stinner wrote: > >> The addition of PyMem_RawMalloc() is motivated by the issue #18203 > >> (Replace calls to malloc() with PyMem_Malloc()). The goal is to be > >> able to setup a custom allocator for *all* allocation made by Python, > >> so malloc() should not be called directly. PyMem_RawMalloc() is > >> required in places where the GIL is not held (ex: in os.getcwd() on > >> Windows). > > > > We already had this discussion on IRC and this argument isn't very > > convincing to me. If os.getcwd() doesn't hold the GIL while allocating > > memory, then you should fix it to hold the GIL while allocating memory. > > > > I don't like the idea of adding of third layer of allocation APIs. The > > dichotomy between PyObject_Malloc and PyMem_Malloc is already a bit > > gratuitous (i.e. not motivated by any actual real-world concern, as > > far as I can tell). > > The only reason for the small object allocator to exist is because > operating system allocators generally aren't optimised for frequent > allocation and deallocation of small objects. You can gain a *lot* of > speed from handling those inside the application. As the allocations > grow in size, though, the application level allocator just becomes > useless overhead, so it's better to delegate those operations directly > to the OS. The small object allocator *already* delegates those operations directly to the OS. You don't need a separate API to do it by hand. > For > custom allocators, it's useful to be able to *ensure* you can bypass > CPython's small object allocator, rather than having to rely on it > being bypassed for allocations above a certain size. Which custom allocators? > > As for the debug functions you added: PyMem_GetRawAllocators(), > > PyMem_SetRawAllocators(), PyMem_GetAllocators(), PyMem_SetAllocators(), > > PyMem_SetupDebugHooks(), _PyObject_GetArenaAllocators(), > > _PyObject_SetArenaAllocators(). Well, do we need all *7* of them? Can't > > you try to make that 2 or 3? > > Faux simplicity that is achieved only by failing to model a complex > problem domain correctly is a bad idea (if we were satisfied with > that, we could stick with the status quo). Actually, I'm sure almost everyone *is* satisfied with the status quo here (witness the total absence of bug reports on the matter). Victor's patch addresses a rare concern compared to the common use cases of CPython. And I'm not even sure what "faux simplicity" you are talking about. What is the supposed complexity that this API is supposed to address? Why do we need two different pairs of hook-setting functions, rather than letting each function set / get all hooks at once? And why the private API functions for setting arena allocators? Memory allocation APIs are a fundamental part of the C API that many extension writers have to understand and deal with. I'm opposed to gratuitous complication when the use cases are not compelling. Regards Antoine. From tseaver at palladion.com Sat Jun 15 14:53:07 2013 From: tseaver at palladion.com (Tres Seaver) Date: Sat, 15 Jun 2013 08:53:07 -0400 Subject: [Python-Dev] Clean way in python to test for None, empty, scalar, and list/ndarray? A prayer to the gods of Python In-Reply-To: <20130614205534.A595A250BE9@webabinitio.net> References: <20130614205534.A595A250BE9@webabinitio.net> Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 06/14/2013 04:55 PM, R. David Murray wrote: > This I sort of agree with. I've often enough wanted to know if > something is a non-string iterable. But you'd have to decide if > bytes/bytearray is a sequence of integers or a scaler... In fifteen years of Python programming, I have literally *never* wanted to iterate over 'str' (or now 'bytes'). I've always considered the fact that Python made them iterable by default (rather than e.g. defining a method / property to get to an iterable "view" on the underlying string) a wart and a serious bug magnet. Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with undefined - http://www.enigmail.net/ iEYEARECAAYFAlG8Y7MACgkQ+gerLs4ltQ6EbwCfYlC3JKL22HK7WgxJLAh9Gk2H R4cAn2+ijAkebHuF92txeddBq99L/zqn =bLkO -----END PGP SIGNATURE----- From victor.stinner at gmail.com Sat Jun 15 14:57:12 2013 From: victor.stinner at gmail.com (Victor Stinner) Date: Sat, 15 Jun 2013 14:57:12 +0200 Subject: [Python-Dev] cpython: Issue #3329: Add new APIs to customize memory allocators In-Reply-To: References: <3bXH1R0dqGz7LkT@mail.python.org> <20130615030431.679bfc1b@fsol> Message-ID: Le 15 juin 2013 03:54, "Victor Stinner" > a ?crit : > Ok, I reverted my commit. > > I will work on a PEP to explain all these new functions and their use cases. I created the PEP 445 to reserve the number. It is ready for a review but already contains some explanation of the new API. http://www.python.org/dev/peps/pep-0445/ Victor -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Sat Jun 15 15:15:12 2013 From: donald at stufft.io (Donald Stufft) Date: Sat, 15 Jun 2013 09:15:12 -0400 Subject: [Python-Dev] Clean way in python to test for None, empty, scalar, and list/ndarray? A prayer to the gods of Python In-Reply-To: References: <20130614205534.A595A250BE9@webabinitio.net> Message-ID: <9FDBF287-7E0E-4519-9834-E3AF1CC6F3CA@stufft.io> I never want to iterate, but I love slice syntax and indexing. Don't think you can have that w/o being able to loop over it can you? Maybe I'm just thinking slow since I just woke up. On Jun 15, 2013, at 8:53 AM, Tres Seaver wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > On 06/14/2013 04:55 PM, R. David Murray wrote: >> This I sort of agree with. I've often enough wanted to know if >> something is a non-string iterable. But you'd have to decide if >> bytes/bytearray is a sequence of integers or a scaler... > > In fifteen years of Python programming, I have literally *never* wanted > to iterate over 'str' (or now 'bytes'). I've always considered the fact > that Python made them iterable by default (rather than e.g. defining a > method / property to get to an iterable "view" on the underlying string) > a wart and a serious bug magnet. > > > Tres. > - -- > =================================================================== > Tres Seaver +1 540-429-0999 tseaver at palladion.com > Palladion Software "Excellence by Design" http://palladion.com > -----BEGIN PGP SIGNATURE----- > Version: GnuPG v1.4.11 (GNU/Linux) > Comment: Using GnuPG with undefined - http://www.enigmail.net/ > > iEYEARECAAYFAlG8Y7MACgkQ+gerLs4ltQ6EbwCfYlC3JKL22HK7WgxJLAh9Gk2H > R4cAn2+ijAkebHuF92txeddBq99L/zqn > =bLkO > -----END PGP SIGNATURE----- > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/donald%40stufft.io From tseaver at palladion.com Sat Jun 15 15:47:06 2013 From: tseaver at palladion.com (Tres Seaver) Date: Sat, 15 Jun 2013 09:47:06 -0400 Subject: [Python-Dev] Clean way in python to test for None, empty, scalar, and list/ndarray? A prayer to the gods of Python In-Reply-To: <9FDBF287-7E0E-4519-9834-E3AF1CC6F3CA@stufft.io> References: <20130614205534.A595A250BE9@webabinitio.net> <9FDBF287-7E0E-4519-9834-E3AF1CC6F3CA@stufft.io> Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 06/15/2013 09:15 AM, Donald Stufft wrote: > I never want to iterate, but I love slice syntax and indexing. Don't > think you can have that w/o being able to loop over it can you? Maybe > I'm just thinking slow since I just woke up. You could if '__iter__' raised an error ('NotIterable', maybe) which defeated the '__getitem__'/'__len__'-based fallback. Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with undefined - http://www.enigmail.net/ iEYEARECAAYFAlG8cFoACgkQ+gerLs4ltQ7KcgCfdFCEkp2gBeuUgn/KooY7F5HX Jm8An2bwq8QoplwJ1MqIbS76m+xdl/Mk =6hA9 -----END PGP SIGNATURE----- From ncoghlan at gmail.com Sat Jun 15 16:12:02 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 16 Jun 2013 00:12:02 +1000 Subject: [Python-Dev] cpython: Issue #3329: Add new APIs to customize memory allocators In-Reply-To: <20130615144115.0a7e3176@fsol> References: <3bXH1R0dqGz7LkT@mail.python.org> <20130615030431.679bfc1b@fsol> <20130615130125.1d4ef1f7@fsol> <20130615144115.0a7e3176@fsol> Message-ID: On 15 June 2013 22:41, Antoine Pitrou wrote: > On Sat, 15 Jun 2013 22:22:33 +1000 > Nick Coghlan wrote: >> For >> custom allocators, it's useful to be able to *ensure* you can bypass >> CPython's small object allocator, rather than having to rely on it >> being bypassed for allocations above a certain size. > > Which custom allocators? Those used by companies like Dropbox to speed up frequent allocations (look up their PyCon 2011 keynote). If we don't provide suitable APIs that we can still hook in debug mode, they'll bypass our infrastructure completely and we'll miss significant memory accesses. >> > As for the debug functions you added: PyMem_GetRawAllocators(), >> > PyMem_SetRawAllocators(), PyMem_GetAllocators(), PyMem_SetAllocators(), >> > PyMem_SetupDebugHooks(), _PyObject_GetArenaAllocators(), >> > _PyObject_SetArenaAllocators(). Well, do we need all *7* of them? Can't >> > you try to make that 2 or 3? >> >> Faux simplicity that is achieved only by failing to model a complex >> problem domain correctly is a bad idea (if we were satisfied with >> that, we could stick with the status quo). > > Actually, I'm sure almost everyone *is* satisfied with the status quo > here (witness the total absence of bug reports on the matter). Victor's > patch addresses a rare concern compared to the common use cases of > CPython. Indeed, but they're use cases I care about, Victor cares about, Kristjan cares about, Greg cares about. It's OK that you don't care about them, just as 99% of the Python programmers on the planet won't care about PEP 432 or the various arcane metaclass changes we've made over the years. issue 3329 (the one where Victor implemented this) was actually filed by the folks working on the Symbian port. The last comment on that issue before Victor restarted was from you, in reply to someone asking if we had implemented it yet. > And I'm not even sure what "faux simplicity" you are talking about. > What is the supposed complexity that this API is supposed to address? The fact that there is more to the world than x86/x86_64 and the very simplistic C memory model. Python is growing more popular in non-traditional execution environments, and we finally have someone (Victor) interested in doing the work to support them properly. That should be celebrated, not blocked because it isn't meaningful for the more common systems where the C memory model is fine. > Why do we need two different pairs of hook-setting functions, rather > than letting each function set / get all hooks at once? I've already said I don't think the raw allocators should be configurable at runtime. The other is because it's likely people will only want to replace the lower level allocators and leave the small object allocator alone. However, they should be able to completely replace the small object allocator if they want. Making the more common case more complicated to avoid adding a more appropriate two level API is the kind of thing I mean by "faux simplicity" - it's almost certainly going to be harder to use in practice, so trading multiple functions for fewer functions each taking more parameters isn't actually a win. > And why the > private API functions for setting arena allocators? Because they're in a different compilation unit... > Memory allocation APIs are a fundamental part of the C API that many > extension writers have to understand and deal with. I'm opposed to > gratuitous complication when the use cases are not compelling. That's a documentation problem. C extension authors shouldn't be touching these, and most people embedding CPython shouldn't be touching them either. They're the C level equivalent of metaclasses: if you're not sure you need them, you don't need them. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From solipsis at pitrou.net Sat Jun 15 16:38:39 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sat, 15 Jun 2013 16:38:39 +0200 Subject: [Python-Dev] cpython: Issue #3329: Add new APIs to customize memory allocators In-Reply-To: References: <3bXH1R0dqGz7LkT@mail.python.org> <20130615030431.679bfc1b@fsol> <20130615130125.1d4ef1f7@fsol> <20130615144115.0a7e3176@fsol> Message-ID: <20130615163839.61a2d669@fsol> On Sun, 16 Jun 2013 00:12:02 +1000 Nick Coghlan wrote: > On 15 June 2013 22:41, Antoine Pitrou wrote: > > On Sat, 15 Jun 2013 22:22:33 +1000 > > Nick Coghlan wrote: > >> For > >> custom allocators, it's useful to be able to *ensure* you can bypass > >> CPython's small object allocator, rather than having to rely on it > >> being bypassed for allocations above a certain size. > > > > Which custom allocators? > > Those used by companies like Dropbox to speed up frequent allocations > (look up their PyCon 2011 keynote). If we don't provide suitable APIs > that we can still hook in debug mode, they'll bypass our > infrastructure completely and we'll miss significant memory accesses. I don't understand the concern. People can ignore the Python allocators, and then use their own debugging infrastructure. This is what happens everytime you want to use your own infrastructure instead of a 3rd party-provided one. > > And I'm not even sure what "faux simplicity" you are talking about. > > What is the supposed complexity that this API is supposed to address? > > The fact that there is more to the world than x86/x86_64 and the very > simplistic C memory model. Then I think it needs a PEP to properly address it and explain it to everyone. Moreover, I think you are conflating two issues: the ability to add memory allocation hooks (for tracing/debugging purposes), and the adaptation to "non-traditional" memory models (whatever that means). Those concerns don't necessarily come together. Regards Antoine. From oscar.j.benjamin at gmail.com Sat Jun 15 17:12:38 2013 From: oscar.j.benjamin at gmail.com (Oscar Benjamin) Date: Sat, 15 Jun 2013 16:12:38 +0100 Subject: [Python-Dev] Clean way in python to test for None, empty, scalar, and list/ndarray? A prayer to the gods of Python In-Reply-To: References: Message-ID: Your questions/suggestions are off-topic for this list. This belongs on python-ideas. On 14 June 2013 20:12, Martin Schultz wrote: > > 2. Testing for empty lists or empty ndarrays: > 4. Finding the number of elements in an object: > 6. Detecting None values in a list: Each of the problems above relates to wanting to use generic Python containers interchangeably with numpy arrays. The solution is simply not to do this; there are too many semantic inconsistencies for this interchangeability to be safe. Here are just a few of them: >>> import numpy as np >>> [1, 2, 3] + [4, 5, 6] [1, 2, 3, 4, 5, 6] >>> np.array([1, 2, 3]) + np.array([4, 5, 6]) array([5, 7, 9]) >>> [1, 2, 3] * 3 [1, 2, 3, 1, 2, 3, 1, 2, 3] >>> np.array([1, 2, 3]) * 3 array([3, 6, 9]) >>> [[1, 2, 3], [4, 5, 6]] + [7, 8, 9] [[1, 2, 3], [4, 5, 6], 7, 8, 9] >>> np.array([[1, 2, 3], [4, 5, 6]]) + np.array([7, 8, 9]) array([[ 8, 10, 12], [11, 13, 15]]) >>> bool([1, 2, 3]) True >>> bool(np.array([1, 2, 3])) Traceback (most recent call last): File "", line 1, in ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all() Numpy provides a very convenient function that will convert objects of all the mentioned types to ndarray objects: the numpy.array function. If you have a function and want to accept lists (of lists ...) and ndarrays interchangeably, simply convert your input with numpy.array at the start of the function i.e.: def square(numbers): numbers = numpy.array(numbers) print(numbers.size, numbers.shape) # etc. return numbers ** 2 # Would be an error for a list of lists > 3. Testing for scalar: > 5. Forcing a scalar to become a 1-element list: numbers = np.array(possibly_scalar_input) if not numbers.shape: print('Numbers was scalar: coercing...') numbers = np.array([numbers]) Usually it is bad design to have an API that interchangeably accepts scalars and sequences. There are some situations where it does make sense such as numpy's ufuncs. These are scalar mathematical functions that can operate distributively on lists of lists, ndarrays etc. and numpy provides a convenient way for you to define these yourself: >>> def square(x): ... return x ** 2 ... >>> square = np.frompyfunc(square, 1, 1) >>> square(2) 4 >>> square([2]) array([4], dtype=object) >>> square([2, 3, 4]) array([4, 9, 16], dtype=object) >>> square([[2, 3, 4]]) array([[4, 9, 16]], dtype=object) >>> square([[]]) array([], shape=(1, 0), dtype=object) The frompyfunc function unfortunately doesn't quite work like a decorator but you can easily wrap it do so: def ufunc(nin, nout): def wrapper_factory(func): return np.frompyfunc(func, nin, nout) return wrapper_factory @ufunc(1, 1) def square(x): return x ** 2 Oscar From christian at python.org Sat Jun 15 17:28:25 2013 From: christian at python.org (Christian Heimes) Date: Sat, 15 Jun 2013 17:28:25 +0200 Subject: [Python-Dev] cpython: Issue #3329: Add new APIs to customize memory allocators In-Reply-To: References: <3bXH1R0dqGz7LkT@mail.python.org> <20130615030431.679bfc1b@fsol> <20130615130125.1d4ef1f7@fsol> Message-ID: <51BC8819.5060106@python.org> Am 15.06.2013 14:22, schrieb Nick Coghlan: > However, it's still desirable to be able to monitor those direct > allocations in debug mode, thus it makes sense to have a GIL protected > direct allocation API as well. You could try to hide the existence of > the latter behaviour and treat it as a private API, but why? For > custom allocators, it's useful to be able to *ensure* you can bypass > CPython's small object allocator, rather than having to rely on it > being bypassed for allocations above a certain size. There is even more to it. We like to keep track of memory allocations in libraries that are wrapped by Python's extension modules, e.g. expat, openssl etc. Almost every library has a hook to set a custom memory allocator, either globally (CRYPTO_set_mem_functions) or for each object (XML_ParserCreate_MM's XML_Memory_Handling_Suite). Python releases the GIL around IO or CPU critical sections of the library. But these sections may call the memory management functions. If these memory functions use the GIL, then some speed ups of releasing the GIL in the first place are lost. It might even be possible to walk into dead lock situations. For that reason it makes sense to have a set of low level memory management functions, that don't rely on the GIL for locking. These functions can still impose their own locking if they need to modify a global state (e.g. allocation statistics). In normal release mode and on most platforms the raw memory allocators should be thin wrappers around malloc(), realloc() and free() -- or perhaps just macros. Eventually I would like to ban direct usage of malloc() from Python's core and patch all memory management through our API. Christian From christian at python.org Sat Jun 15 17:49:41 2013 From: christian at python.org (Christian Heimes) Date: Sat, 15 Jun 2013 17:49:41 +0200 Subject: [Python-Dev] cpython: Issue #3329: Add new APIs to customize memory allocators In-Reply-To: References: <3bXH1R0dqGz7LkT@mail.python.org> <20130615030431.679bfc1b@fsol> Message-ID: <51BC8D15.1050707@python.org> Am 15.06.2013 14:57, schrieb Victor Stinner: > Le 15 juin 2013 03:54, "Victor Stinner" > a ?crit : > >> Ok, I reverted my commit. >> >> I will work on a PEP to explain all these new functions and their use > cases. > > I created the PEP 445 to reserve the number. It is ready for a review > but already containssome explanation of the new API. +1 How about you compare your approach with some other libraries, too? Other projects have dealt with similar issues in the past, too. libxml2 has an extensive API for memory management and debugging. I have assembled a list of APIs for you: OpenSSL has CRYPTO_set_mem_functions() to set memory management functions globally http://git.openssl.org/gitweb/?p=openssl.git;a=blob;f=crypto/mem.c;h=f7984fa958eb1edd6c61f6667f3f2b29753be662;hb=HEAD#l124 expat has a per-instance memory handler: http://hg.python.org/cpython/file/cc27d50bd91a/Modules/expat/xmlparse.c#l717 libtiff has three global hooks _TIFFmalloc(), _TIFFrealloc() and _TIFFfree() that are used instead of malloc() in its core. http://trac.imagemagick.org/browser/tiff/trunk/libtiff/tif_unix.c#L258 libxml2 has http://xmlsoft.org/html/libxml-xmlmemory.html Christian From ron3200 at gmail.com Sat Jun 15 21:58:33 2013 From: ron3200 at gmail.com (Ron Adam) Date: Sat, 15 Jun 2013 14:58:33 -0500 Subject: [Python-Dev] eval and triple quoted strings In-Reply-To: References: <51BB2EDC.80604@livinglogic.de> <51BB5CBF.5070308@gmail.com> Message-ID: <51BCC769.8070904@gmail.com> On 06/14/2013 04:03 PM, PJ Eby wrote: >> >Should this be the same? >> > >> > >> >python3 -c 'print(bytes("""\r\n""", "utf8"))' >> >b'\r\n' >> > >> > >>>>> >>>>eval('print(bytes("""\r\n""", "utf8"))') >> >b'\n' > No, but: > > eval(r'print(bytes("""\r\n""", "utf8"))') > > should be. (And is.) > > What I believe you and Walter are missing is that the \r\n in the eval > strings are converted early if you don't make the enclosing string > raw. So what you're eval-ing is not what you think you are eval-ing, > hence the confusion. Yes thanks, seems like an easy mistake to make. To be clear... The string to eval is parsed when the eval line is tokenized in the scope containing the eval() function. The eval function then parses the resulting string object it receives as it's input. There is no mention of using raw strings in the docs on evel and exec. I think there should be, because the intention (in most cases) is for eval to parse the string, and not for it to be parsed or changed before it's evaluated by eval or exec. An example using a string with escape characters might make it clearer. Cheers, Ron From ron3200 at gmail.com Sat Jun 15 21:58:33 2013 From: ron3200 at gmail.com (Ron Adam) Date: Sat, 15 Jun 2013 14:58:33 -0500 Subject: [Python-Dev] eval and triple quoted strings In-Reply-To: References: <51BB2EDC.80604@livinglogic.de> <51BB5CBF.5070308@gmail.com> Message-ID: <51BCC769.8070904@gmail.com> On 06/14/2013 04:03 PM, PJ Eby wrote: >> >Should this be the same? >> > >> > >> >python3 -c 'print(bytes("""\r\n""", "utf8"))' >> >b'\r\n' >> > >> > >>>>> >>>>eval('print(bytes("""\r\n""", "utf8"))') >> >b'\n' > No, but: > > eval(r'print(bytes("""\r\n""", "utf8"))') > > should be. (And is.) > > What I believe you and Walter are missing is that the \r\n in the eval > strings are converted early if you don't make the enclosing string > raw. So what you're eval-ing is not what you think you are eval-ing, > hence the confusion. Yes thanks, seems like an easy mistake to make. To be clear... The string to eval is parsed when the eval line is tokenized in the scope containing the eval() function. The eval function then parses the resulting string object it receives as it's input. There is no mention of using raw strings in the docs on evel and exec. I think there should be, because the intention (in most cases) is for eval to parse the string, and not for it to be parsed or changed before it's evaluated by eval or exec. An example using a string with escape characters might make it clearer. Cheers, Ron From tjreedy at udel.edu Sat Jun 15 22:11:59 2013 From: tjreedy at udel.edu (Terry Reedy) Date: Sat, 15 Jun 2013 16:11:59 -0400 Subject: [Python-Dev] Clean way in python to test for None, empty, scalar, and list/ndarray? A prayer to the gods of Python In-Reply-To: References: <20130614205534.A595A250BE9@webabinitio.net> Message-ID: On 6/15/2013 8:53 AM, Tres Seaver wrote: > In fifteen years of Python programming, I have literally *never* wanted > to iterate over 'str' (or now 'bytes'). If so, it is because you have always been able to use pre-written methods and functions that internally do the iteration for you. > I've always considered the fact > that Python made them iterable by default (rather than e.g. defining a > method / property to get to an iterable "view" on the underlying string) .__iter__ is that method. But this is off-topic for pydev. -- Terry Jan Reedy From guido at python.org Sat Jun 15 22:23:46 2013 From: guido at python.org (Guido van Rossum) Date: Sat, 15 Jun 2013 13:23:46 -0700 Subject: [Python-Dev] eval and triple quoted strings In-Reply-To: <51BCC769.8070904@gmail.com> References: <51BB2EDC.80604@livinglogic.de> <51BB5CBF.5070308@gmail.com> <51BCC769.8070904@gmail.com> Message-ID: The semantics of raw strings are clear. I don't see that they should be called out especially in any context. (Except for regexps.) Usually exec() is not used with a literal anyway (what would be the point). --Guido van Rossum (sent from Android phone) On Jun 15, 2013 1:03 PM, "Ron Adam" wrote: > > > On 06/14/2013 04:03 PM, PJ Eby wrote: > >> >Should this be the same? >>> > >>> > >>> >python3 -c 'print(bytes("""\r\n""", "utf8"))' >>> >b'\r\n' >>> > >>> > >>> >>>> >>>>eval('print(bytes("""\r\n"**"", "utf8"))') >>>>>> >>>>> >b'\n' >>> >> No, but: >> >> eval(r'print(bytes("""\r\n""", "utf8"))') >> >> should be. (And is.) >> >> What I believe you and Walter are missing is that the \r\n in the eval >> strings are converted early if you don't make the enclosing string >> raw. So what you're eval-ing is not what you think you are eval-ing, >> hence the confusion. >> > > Yes thanks, seems like an easy mistake to make. > > To be clear... > > The string to eval is parsed when the eval line is tokenized in the scope > containing the eval() function. The eval function then parses the > resulting string object it receives as it's input. > > There is no mention of using raw strings in the docs on evel and exec. I > think there should be, because the intention (in most cases) is for eval to > parse the string, and not for it to be parsed or changed before it's > evaluated by eval or exec. > > An example using a string with escape characters might make it clearer. > > Cheers, > Ron > ______________________________**_________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/**mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/**mailman/options/python-dev/** > guido%40python.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ethan at stoneleaf.us Sat Jun 15 21:46:32 2013 From: ethan at stoneleaf.us (Ethan Furman) Date: Sat, 15 Jun 2013 12:46:32 -0700 Subject: [Python-Dev] backported Enum Message-ID: <51BCC498.1030100@stoneleaf.us> So I have the stdlb 3.4 Enum backported for both earlier 3.x and back to 2.4 in the 2.x series. I would like to put this on PyPI, but the `enum` name is already taken. Would it be inappropriate to call it `stdlib.enum`? -- ~Ethan~ From solipsis at pitrou.net Sat Jun 15 22:53:54 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sat, 15 Jun 2013 22:53:54 +0200 Subject: [Python-Dev] backported Enum References: <51BCC498.1030100@stoneleaf.us> Message-ID: <20130615225354.33ab8e09@fsol> On Sat, 15 Jun 2013 12:46:32 -0700 Ethan Furman wrote: > So I have the stdlb 3.4 Enum backported for both earlier 3.x and back to 2.4 in the 2.x series. > > I would like to put this on PyPI, but the `enum` name is already taken. > > Would it be inappropriate to call it `stdlib.enum`? `backport.enum`? From barry at python.org Sat Jun 15 23:14:07 2013 From: barry at python.org (Barry Warsaw) Date: Sat, 15 Jun 2013 17:14:07 -0400 Subject: [Python-Dev] backported Enum In-Reply-To: <51BCC498.1030100@stoneleaf.us> References: <51BCC498.1030100@stoneleaf.us> Message-ID: <20130615171407.0e2c827d@anarchist> On Jun 15, 2013, at 12:46 PM, Ethan Furman wrote: >So I have the stdlb 3.4 Enum backported for both earlier 3.x and back to 2.4 >in the 2.x series. > >I would like to put this on PyPI, but the `enum` name is already taken. > >Would it be inappropriate to call it `stdlib.enum`? The last upload was on 2009-08-25. Maybe Ben Finney's abandoned it and wouldn't mind giving up the enum PyPI name? -Barry From tseaver at palladion.com Sat Jun 15 23:45:33 2013 From: tseaver at palladion.com (Tres Seaver) Date: Sat, 15 Jun 2013 17:45:33 -0400 Subject: [Python-Dev] Clean way in python to test for None, empty, scalar, and list/ndarray? A prayer to the gods of Python In-Reply-To: References: <20130614205534.A595A250BE9@webabinitio.net> Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 06/15/2013 04:11 PM, Terry Reedy wrote: > On 6/15/2013 8:53 AM, Tres Seaver wrote: > >> In fifteen years of Python programming, I have literally *never* >> wanted to iterate over 'str' (or now 'bytes'). > > If so, it is because you have always been able to use pre-written > methods and functions that internally do the iteration for you. Given that strings are implemented in C, there is no real "iteration" happing (in the Python sense) in their methods. What stdlib code can you point to that does iterate over them in Python? I.e.: for c in stringval: ... Even if there were, aren't you proving my point? The fact that strings are implicitly iterable injects all kinds of fur into methods which take either a single value or an iterable as an argument, e.g.: def api_function(value_or_values): if isinstance(value_or_values, STRING_TYPES): value_or_values = [value_or_values] elif isinstance(value_or_values, collections.abc.Iterable): value_or_values = list(value_or_values) else: value_or_values = [value_or_values] The bugs that leaving the first test out yields pop up all over the place. >> I've always considered the fact >>> that Python made them iterable by default (rather than e.g. >>> defining a method / property to get to an iterable "view" on the >>> underlying string) > > .__iter__ is that method. But it is *on be default*: I was describing something which has to be requested in ways that don't get triggered by syntax, and doesn't make strings look "non-scalar" for APIs like the one above. Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with undefined - http://www.enigmail.net/ iEYEARECAAYFAlG84H0ACgkQ+gerLs4ltQ56RQCgks8R52f41RwJ+v9oteOBC3qY kIIAoIHmg+zcmJpV3v/gAhkKJfbNKufj =ZeRB -----END PGP SIGNATURE----- From tseaver at palladion.com Sat Jun 15 23:47:25 2013 From: tseaver at palladion.com (Tres Seaver) Date: Sat, 15 Jun 2013 17:47:25 -0400 Subject: [Python-Dev] backported Enum In-Reply-To: <20130615171407.0e2c827d@anarchist> References: <51BCC498.1030100@stoneleaf.us> <20130615171407.0e2c827d@anarchist> Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 06/15/2013 05:14 PM, Barry Warsaw wrote: > On Jun 15, 2013, at 12:46 PM, Ethan Furman wrote: > >> So I have the stdlb 3.4 Enum backported for both earlier 3.x and >> back to 2.4 in the 2.x series. >> >> I would like to put this on PyPI, but the `enum` name is already >> taken. >> >> Would it be inappropriate to call it `stdlib.enum`? > > The last upload was on 2009-08-25. Maybe Ben Finney's abandoned it > and wouldn't mind giving up the enum PyPI name? That would screw anybody already using the existing distributions pretty badly. Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with undefined - http://www.enigmail.net/ iEYEARECAAYFAlG84O0ACgkQ+gerLs4ltQ71qwCgo4uubYXVw/qvARESfzPLzFYZ Fb8AnjB5ZcwupMoQ6SP6r7Xl26Hg3wpQ =u3L7 -----END PGP SIGNATURE----- From matthew.lefavor at nasa.gov Sat Jun 15 23:58:24 2013 From: matthew.lefavor at nasa.gov (Lefavor, Matthew (GSFC-582.0)[MICROTEL LLC]) Date: Sat, 15 Jun 2013 16:58:24 -0500 Subject: [Python-Dev] backported Enum In-Reply-To: <20130615171407.0e2c827d@anarchist> References: <51BCC498.1030100@stoneleaf.us> <20130615171407.0e2c827d@anarchist> Message-ID: <8897D17F-8C15-49D8-A477-E4C1693891B9@nasa.gov> What if there's some obscure PyPi module that requires that `enum` package, or some other project (not hosted on PyPI) that requires Ben Finney's original `enum` package? Is there anyway to get usage data to make sure nobody's been using it recently? ML On Jun 15, 2013, at 5:14 PM, Barry Warsaw wrote: On Jun 15, 2013, at 12:46 PM, Ethan Furman wrote: So I have the stdlb 3.4 Enum backported for both earlier 3.x and back to 2.4 in the 2.x series. I would like to put this on PyPI, but the `enum` name is already taken. Would it be inappropriate to call it `stdlib.enum`? The last upload was on 2009-08-25. Maybe Ben Finney's abandoned it and wouldn't mind giving up the enum PyPI name? -Barry _______________________________________________ Python-Dev mailing list Python-Dev at python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/matthew.lefavor%40nasa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From ethan at stoneleaf.us Sat Jun 15 23:43:36 2013 From: ethan at stoneleaf.us (Ethan Furman) Date: Sat, 15 Jun 2013 14:43:36 -0700 Subject: [Python-Dev] backported Enum In-Reply-To: <20130615225354.33ab8e09@fsol> References: <51BCC498.1030100@stoneleaf.us> <20130615225354.33ab8e09@fsol> Message-ID: <51BCE008.5070308@stoneleaf.us> On 06/15/2013 01:53 PM, Antoine Pitrou wrote: > On Sat, 15 Jun 2013 12:46:32 -0700 > Ethan Furman wrote: >> So I have the stdlb 3.4 Enum backported for both earlier 3.x and back to 2.4 in the 2.x series. >> >> I would like to put this on PyPI, but the `enum` name is already taken. >> >> Would it be inappropriate to call it `stdlib.enum`? > > `backport.enum`? Some other well-meaning soul has already claimed that name, but that one only covers 2.6 and 2.7. -- ~Ethan~ From greg at krypto.org Sun Jun 16 00:43:10 2013 From: greg at krypto.org (Gregory P. Smith) Date: Sat, 15 Jun 2013 15:43:10 -0700 Subject: [Python-Dev] backported Enum In-Reply-To: <51BCC498.1030100@stoneleaf.us> References: <51BCC498.1030100@stoneleaf.us> Message-ID: I tend to just pick a name and stick with it. subprocess32 is subprocess backported from 3.2 (with the 3.3 timeout feature also in it). unittest2 is unittest from 2.7. It tends to work. and it also emphasizes that i'm unlikely to backport future non-bugfix updates beyond the release mentioned and merely focus on keeping it stable and available for use on older pythons. A "backport" namespace is a neat idea but unless someone's going to create a system for backports to register which versions they are backports from and automagically have sub-module imports from backport pick the backported code or the standard library version when the Python VM is recent enough to not need a backport I wouldn't bother claiming that name. (and even that feels like overengineering) -gps On Sat, Jun 15, 2013 at 12:46 PM, Ethan Furman wrote: > So I have the stdlb 3.4 Enum backported for both earlier 3.x and back to > 2.4 in the 2.x series. > > I would like to put this on PyPI, but the `enum` name is already taken. > > Would it be inappropriate to call it `stdlib.enum`? > > -- > ~Ethan~ > ______________________________**_________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/**mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/**mailman/options/python-dev/** > greg%40krypto.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: From victor.stinner at gmail.com Sun Jun 16 01:48:06 2013 From: victor.stinner at gmail.com (Victor Stinner) Date: Sun, 16 Jun 2013 01:48:06 +0200 Subject: [Python-Dev] cpython: Issue #3329: Add new APIs to customize memory allocators In-Reply-To: <51BC8819.5060106@python.org> References: <3bXH1R0dqGz7LkT@mail.python.org> <20130615030431.679bfc1b@fsol> <20130615130125.1d4ef1f7@fsol> <51BC8819.5060106@python.org> Message-ID: 2013/6/15 Christian Heimes : > Am 15.06.2013 14:22, schrieb Nick Coghlan: >> However, it's still desirable to be able to monitor those direct >> allocations in debug mode, thus it makes sense to have a GIL protected >> direct allocation API as well. You could try to hide the existence of >> the latter behaviour and treat it as a private API, but why? For >> custom allocators, it's useful to be able to *ensure* you can bypass >> CPython's small object allocator, rather than having to rely on it >> being bypassed for allocations above a certain size. > > There is even more to it. We like to keep track of memory allocations in > libraries that are wrapped by Python's extension modules, e.g. expat, > openssl etc. Almost every library has a hook to set a custom memory > allocator, either globally (CRYPTO_set_mem_functions) or for each object > (XML_ParserCreate_MM's XML_Memory_Handling_Suite). I just create the issue http://bugs.python.org/issue18227: "Use Python memory allocators in external libraries like zlib or OpenSSL". Is it possible to detect if Python is used as a standalone application (the classic "python" program) or if Python is embedded? If it is possible, we can modify the "global" memory allocators of a library. Otherwise, it is more tricky. Should Python sets its "own" memory allocators? Maybe only if PyMem_SetRawAllocators() was called? > Eventually I would like to ban direct usage of malloc() from Python's > core and patch all memory management through our API. I already create issue http://bugs.python.org/issue18203 for this part. Victor From victor.stinner at gmail.com Sun Jun 16 02:18:32 2013 From: victor.stinner at gmail.com (Victor Stinner) Date: Sun, 16 Jun 2013 02:18:32 +0200 Subject: [Python-Dev] cpython: Issue #3329: Add new APIs to customize memory allocators In-Reply-To: <20130615130125.1d4ef1f7@fsol> References: <3bXH1R0dqGz7LkT@mail.python.org> <20130615030431.679bfc1b@fsol> <20130615130125.1d4ef1f7@fsol> Message-ID: 2013/6/15 Antoine Pitrou : > On Sat, 15 Jun 2013 03:54:50 +0200 > Victor Stinner wrote: >> The addition of PyMem_RawMalloc() is motivated by the issue #18203 >> (Replace calls to malloc() with PyMem_Malloc()). The goal is to be >> able to setup a custom allocator for *all* allocation made by Python, >> so malloc() should not be called directly. PyMem_RawMalloc() is >> required in places where the GIL is not held (ex: in os.getcwd() on >> Windows). > > We already had this discussion on IRC and this argument isn't very > convincing to me. If os.getcwd() doesn't hold the GIL while allocating > memory, then you should fix it to hold the GIL while allocating memory. The GIL is released for best performances, holding the GIL would have an impact on performances. PyMem_RawMalloc() is needed when PyMem_Malloc() cannot be used because the GIL was released. For example, for the issue #18227 (reuse the custom allocator in external libraries), PyMem_Malloc() is usually not appropriate. PyMem_RawMalloc() should also be used instead of PyMem_Malloc() in the Python startup sequence, because PyMem_Malloc() requires the GIL whereas the GIL does not exist yet. PyMem_RawMalloc() also provides more accurate memory usage if it can be replaced or hooked (with PyMem_SetRawAllocators). The issue #18203 explains why I would like to replace direct call to malloc() with PyMem_Malloc() or PyMem_RawMalloc(). > I don't like the idea of adding of third layer of allocation APIs. The > dichotomy between PyObject_Malloc and PyMem_Malloc is already a bit > gratuitous (i.e. not motivated by any actual real-world concern, as > far as I can tell). In Python 3.3, PyMem_Malloc() cannot be used instead of malloc() where the GIL is not held. Instead of adding PyMem_RawMalloc(), an alternative is to remove the "the GIL must be held" restriction from PyMem_Malloc() by changing PyMem_Malloc() to make it always call malloc() (instead of PyObject_Malloc() in debug mode). With such change, a debug hook cannot rely on the GIL anymore: it cannot inspect Python objects, get a frame or traceback, etc. To still get accurate debug report, PyMem_Malloc() should be replaced with PyObject_Malloc(). I don't understand yet the effect of such change on backport compatibility. May it break applications? > As for the debug functions you added: PyMem_GetRawAllocators(), > PyMem_SetRawAllocators(), PyMem_GetAllocators(), PyMem_SetAllocators(), > PyMem_SetupDebugHooks(), _PyObject_GetArenaAllocators(), > _PyObject_SetArenaAllocators(). Well, do we need all *7* of them? Can't > you try to make that 2 or 3? Get/SetAllocators of PyMem, PyMem_Raw and PyObject can be grouped into 2 functions (get and set) with an argument to select the API. It is what I proposed initially. I changed this when I had to choose a name for the name of the argument ("api", "domain", something else?) because there were only two choices. With 3 family of functions (PyMem, PyMem_Raw and PyObject), it becomes again interesting to have generic functions. The arena case is different: pymalloc only uses two functions to allocate areneas: void* alloc(size_t) and void release(void*, size_t). The release function has a size argument, which is unusual, but require to implement it using munmap(). VirtualFree() on Windows requires also the size. An application can choose to replace PyObject_Malloc() with its own allocator, but in my experience, it has an important impact on performance (Python is slower). To benefit of pymalloc with a custom memory allocator, _PyObject_SetArenaAllocators() can be used. I kept _PyObject_SetArenaAllocators() private because I don't like its API, it is not homogenous with the other SetAllocators functions. I'm not sure that it would be used, so I prefer to keep it private until it is tested by some projects. "Private" functions can be used by applications, it's just that Python doesn't give any backward compatibility warranty. Am I right? Victor From victor.stinner at gmail.com Sun Jun 16 02:25:16 2013 From: victor.stinner at gmail.com (Victor Stinner) Date: Sun, 16 Jun 2013 02:25:16 +0200 Subject: [Python-Dev] cpython: Issue #3329: Add new APIs to customize memory allocators In-Reply-To: References: <3bXH1R0dqGz7LkT@mail.python.org> <20130615030431.679bfc1b@fsol> <20130615130125.1d4ef1f7@fsol> Message-ID: 2013/6/15 Nick Coghlan : > The only reason for the small object allocator to exist is because > operating system allocators generally aren't optimised for frequent > allocation and deallocation of small objects. You can gain a *lot* of > speed from handling those inside the application. As the allocations > grow in size, though, the application level allocator just becomes > useless overhead, so it's better to delegate those operations directly > to the OS. Why not using PyObject_Malloc() for all allocations? PyObject_Malloc() fallbacks to malloc() if the size is larger than a threshold (512 bytes in Python 3.4). Are PyObject_Realloc() and PyObject_Free() more expensive than realloc() and free() (when the memory was allocated by malloc)? > The only question mark in my mind is over the GIL-free raw allocation > APIs. I think it makes sense to at least conditionally define those as > macros so an embedding application can redirect *just* the allocations > made by the CPython runtime (rather than having to redefine > malloc/realloc/free when building Python), but I don't believe the > case has been adequately made for making the raw APIs configurable at > runtime. Dropping that aspect would at least eliminate the > PyMem_(Get|Set)RawAllocators() APIs. PyMem_SetRawAllocators() is required for the two use cases: use a custom memory allocator (embedded device and Python embedded in an application) and setup an hook for debug purpose. Without PyMem_SetRawAllocators(), allocations made by PyMem_RawMalloc() would go to the same place than the rest of the "Python memory", nor seen by debug tools. It becomes worse with large allocations kept for a long time. Victor From victor.stinner at gmail.com Sun Jun 16 02:51:29 2013 From: victor.stinner at gmail.com (Victor Stinner) Date: Sun, 16 Jun 2013 02:51:29 +0200 Subject: [Python-Dev] cpython: Issue #3329: Add new APIs to customize memory allocators In-Reply-To: <20130615163839.61a2d669@fsol> References: <3bXH1R0dqGz7LkT@mail.python.org> <20130615030431.679bfc1b@fsol> <20130615130125.1d4ef1f7@fsol> <20130615144115.0a7e3176@fsol> <20130615163839.61a2d669@fsol> Message-ID: 2013/6/15 Antoine Pitrou : > Moreover, I think you are conflating two issues: the ability to add > memory allocation hooks (for tracing/debugging purposes), and the > adaptation to "non-traditional" memory models (whatever that means). > Those concerns don't necessarily come together. In my implementation, both uses case use the same API: PyMem_SetAllocators(), except that hooks need also PyMem_GetAllocators(). Victor From donald at stufft.io Sun Jun 16 03:50:37 2013 From: donald at stufft.io (Donald Stufft) Date: Sat, 15 Jun 2013 21:50:37 -0400 Subject: [Python-Dev] backported Enum In-Reply-To: <51BCE008.5070308@stoneleaf.us> References: <51BCC498.1030100@stoneleaf.us> <20130615225354.33ab8e09@fsol> <51BCE008.5070308@stoneleaf.us> Message-ID: <644E4682-4898-49A8-B97D-9C9C06EFEBFA@stufft.io> On Jun 15, 2013, at 5:43 PM, Ethan Furman wrote: > On 06/15/2013 01:53 PM, Antoine Pitrou wrote: >> On Sat, 15 Jun 2013 12:46:32 -0700 >> Ethan Furman wrote: >>> So I have the stdlb 3.4 Enum backported for both earlier 3.x and back to 2.4 in the 2.x series. >>> >>> I would like to put this on PyPI, but the `enum` name is already taken. >>> >>> Would it be inappropriate to call it `stdlib.enum`? >> >> `backport.enum`? > > Some other well-meaning soul has already claimed that name, but that one only covers 2.6 and 2.7. > > -- > ~Ethan~ > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/donald%40stufft.io I claimed backport.enum, but you're welcome to the name. I was going to try and backport this PEP under that name anyways. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 841 bytes Desc: Message signed with OpenPGP using GPGMail URL: From ron3200 at gmail.com Sun Jun 16 04:00:13 2013 From: ron3200 at gmail.com (Ron Adam) Date: Sat, 15 Jun 2013 21:00:13 -0500 Subject: [Python-Dev] eval and triple quoted strings In-Reply-To: References: <51BB2EDC.80604@livinglogic.de> <51BB5CBF.5070308@gmail.com> <51BCC769.8070904@gmail.com> Message-ID: <51BD1C2D.2050101@gmail.com> On 06/15/2013 03:23 PM, Guido van Rossum wrote: > The semantics of raw strings are clear. I don't see that they should be > called out especially in any context. (Except for regexps.) Usually exec() > is not used with a literal anyway (what would be the point). There are about a hundred instances of eval/exec(some_string_literal) in pythons library. Most of them in the tests, and maybe about half of those testing the compiler, eval, and exec. egrep -owr --include="*.py" "(eval|exec)\(('.*'|\".*\")\)" * | wc -l 114 I have no idea in how many places a string literal is assigned to a name first and then used later in eval or exec. It's harder to grep for but would be less than... egrep -owr --include="*.py" "(eval|exec)\(.*\)" * | wc -l 438 That's overstated because some of those are comments, and some may be functions with the name ending with eval or exec. I do think that eval and exec is a similar case to regexps. And possibly often enough, the string may contain a raw string, regular expression, or a file/path name. Only a short note needed in the docs for eval, nothing more. And not even that if no thinks it's an issue. cheers, Ron From ron3200 at gmail.com Sun Jun 16 04:00:13 2013 From: ron3200 at gmail.com (Ron Adam) Date: Sat, 15 Jun 2013 21:00:13 -0500 Subject: [Python-Dev] eval and triple quoted strings In-Reply-To: References: <51BB2EDC.80604@livinglogic.de> <51BB5CBF.5070308@gmail.com> <51BCC769.8070904@gmail.com> Message-ID: <51BD1C2D.2050101@gmail.com> On 06/15/2013 03:23 PM, Guido van Rossum wrote: > The semantics of raw strings are clear. I don't see that they should be > called out especially in any context. (Except for regexps.) Usually exec() > is not used with a literal anyway (what would be the point). There are about a hundred instances of eval/exec(some_string_literal) in pythons library. Most of them in the tests, and maybe about half of those testing the compiler, eval, and exec. egrep -owr --include="*.py" "(eval|exec)\(('.*'|\".*\")\)" * | wc -l 114 I have no idea in how many places a string literal is assigned to a name first and then used later in eval or exec. It's harder to grep for but would be less than... egrep -owr --include="*.py" "(eval|exec)\(.*\)" * | wc -l 438 That's overstated because some of those are comments, and some may be functions with the name ending with eval or exec. I do think that eval and exec is a similar case to regexps. And possibly often enough, the string may contain a raw string, regular expression, or a file/path name. Only a short note needed in the docs for eval, nothing more. And not even that if no thinks it's an issue. cheers, Ron From ncoghlan at gmail.com Sun Jun 16 04:23:47 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 16 Jun 2013 12:23:47 +1000 Subject: [Python-Dev] cpython: Issue #3329: Add new APIs to customize memory allocators In-Reply-To: References: <3bXH1R0dqGz7LkT@mail.python.org> <20130615030431.679bfc1b@fsol> <20130615130125.1d4ef1f7@fsol> <20130615144115.0a7e3176@fsol> <20130615163839.61a2d669@fsol> Message-ID: On 16 Jun 2013 10:54, "Victor Stinner" wrote: > > 2013/6/15 Antoine Pitrou : > > Moreover, I think you are conflating two issues: the ability to add > > memory allocation hooks (for tracing/debugging purposes), and the > > adaptation to "non-traditional" memory models (whatever that means). > > Those concerns don't necessarily come together. > > In my implementation, both uses case use the same API: > PyMem_SetAllocators(), except that hooks need also > PyMem_GetAllocators(). Right - they're different use cases that share a technical solution, so it makes sense to consider them together. Cheers, Nick. > > Victor > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From ben+python at benfinney.id.au Sun Jun 16 04:45:58 2013 From: ben+python at benfinney.id.au (Ben Finney) Date: Sun, 16 Jun 2013 12:45:58 +1000 Subject: [Python-Dev] backported Enum References: <51BCC498.1030100@stoneleaf.us> Message-ID: <7w8v2aepsp.fsf@benfinney.id.au> Ethan Furman writes: > So I have the stdlb 3.4 Enum backported for both earlier 3.x and back > to 2.4 in the 2.x series. > > I would like to put this on PyPI, but the `enum` name is already > taken. I have for a long time approved of ?flufl.enum? becoming the One Obvious Way to do enumerations, and am glad to see PEP 435 bring it into the standard library. There are some people still contacting me about their use of the ?enum? package on PyPI, so I think that to just replace it with a different code base under the same name would not be helpful. > Would it be inappropriate to call it `stdlib.enum`? That sounds good to me. Is there anything I can do to keep the ?enum? package online for continuity but make it clear, to automated tools, that this is end-of-life and obsoleted by another package? -- \ ?Free thought is a necessary, but not a sufficient, condition | `\ for democracy.? ?Carl Sagan | _o__) | Ben Finney From donald at stufft.io Sun Jun 16 05:53:35 2013 From: donald at stufft.io (Donald Stufft) Date: Sat, 15 Jun 2013 23:53:35 -0400 Subject: [Python-Dev] backported Enum In-Reply-To: <7w8v2aepsp.fsf@benfinney.id.au> References: <51BCC498.1030100@stoneleaf.us> <7w8v2aepsp.fsf@benfinney.id.au> Message-ID: On Jun 15, 2013, at 10:45 PM, Ben Finney wrote: > Ethan Furman writes: > >> So I have the stdlb 3.4 Enum backported for both earlier 3.x and back >> to 2.4 in the 2.x series. >> >> I would like to put this on PyPI, but the `enum` name is already >> taken. > > I have for a long time approved of ?flufl.enum? becoming the One Obvious > Way to do enumerations, and am glad to see PEP 435 bring it into the > standard library. > > > > There are some people still contacting me about their use of the ?enum? > package on PyPI, so I think that to just replace it with a different > code base under the same name would not be helpful. > >> Would it be inappropriate to call it `stdlib.enum`? > > That sounds good to me. > > Is there anything I can do to keep the ?enum? package online for > continuity but make it clear, to automated tools, that this is > end-of-life and obsoleted by another package? Right now the best you can do is make a note of it. Pep 426 let's you do what you want. > > -- > \ ?Free thought is a necessary, but not a sufficient, condition | > `\ for democracy.? ?Carl Sagan | > _o__) | > Ben Finney > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/donald%40stufft.io From tjreedy at udel.edu Sun Jun 16 05:55:43 2013 From: tjreedy at udel.edu (Terry Reedy) Date: Sat, 15 Jun 2013 23:55:43 -0400 Subject: [Python-Dev] Clean way in python to test for None, empty, scalar, and list/ndarray? A prayer to the gods of Python In-Reply-To: References: <20130614205534.A595A250BE9@webabinitio.net> Message-ID: On 6/15/2013 5:45 PM, Tres Seaver wrote: > Given that strings are implemented in C, That is a current implementation detail. String functions were originally written in python in string.py. Some used 'for c in s:'. The functions only because methods after 2.2. I presume Pypy starts from Python code again. > there is no real "iteration" > happing (in the Python sense) in their methods. I disagree, but not relevant. > What stdlib code can you > point to that does iterate over them in Python? I.e.: > > for c in stringval: Using Idle's grep (Find in files) for ...Python33/Lib/*.py: 'for c in' 193 hits 'for ch in' 30 hits 'for chr in' 0 hits 'for char in' 14 hits. Some, especially in the 193, are false positives, but most are not. There are at least a few other 'for name in string' uses: for instance, 'for a in _hexdig for b in _hexdig' where _hexdig is '0123456789ABCDEFabcdef'. -- Terry Jan Reedy From ben+python at benfinney.id.au Sun Jun 16 07:52:33 2013 From: ben+python at benfinney.id.au (Ben Finney) Date: Sun, 16 Jun 2013 15:52:33 +1000 Subject: [Python-Dev] backported Enum References: <51BCC498.1030100@stoneleaf.us> <7w8v2aepsp.fsf@benfinney.id.au> Message-ID: <7wd2rmeh5q.fsf@benfinney.id.au> Donald Stufft writes: > On Jun 15, 2013, at 10:45 PM, Ben Finney wrote: > > Is there anything I can do to keep the ?enum? package online for > > continuity but make it clear, to automated tools, that this is > > end-of-life and obsoleted by another package? > > Right now the best you can do is make a note of it. Pep 426 let's you > do what you want. Thanks. What specifically in PEP 426 do you mean; how would I make a note of ?this package is end-of-life as is, please migrate to ?flufl.enum? instead? using PEP 426 metadata? -- \ ?There are no significant bugs in our released software that | `\ any significant number of users want fixed.? ?Bill Gates, | _o__) 1995-10-23 | Ben Finney From donald at stufft.io Sun Jun 16 07:58:55 2013 From: donald at stufft.io (Donald Stufft) Date: Sun, 16 Jun 2013 01:58:55 -0400 Subject: [Python-Dev] backported Enum In-Reply-To: <7wd2rmeh5q.fsf@benfinney.id.au> References: <51BCC498.1030100@stoneleaf.us> <7w8v2aepsp.fsf@benfinney.id.au> <7wd2rmeh5q.fsf@benfinney.id.au> Message-ID: On Jun 16, 2013, at 1:52 AM, Ben Finney wrote: > Donald Stufft writes: > >> On Jun 15, 2013, at 10:45 PM, Ben Finney wrote: >>> Is there anything I can do to keep the ?enum? package online for >>> continuity but make it clear, to automated tools, that this is >>> end-of-life and obsoleted by another package? >> >> Right now the best you can do is make a note of it. Pep 426 let's you >> do what you want. > > Thanks. What specifically in PEP 426 do you mean; how would I make a > note of ?this package is end-of-life as is, please migrate to > ?flufl.enum? instead? using PEP 426 metadata? http://www.python.org/dev/peps/pep-0426/#obsoleted-by Note PEP426 isn't completed and isn't implemented :) > > -- > \ ?There are no significant bugs in our released software that | > `\ any significant number of users want fixed.? ?Bill Gates, | > _o__) 1995-10-23 | > Ben Finney > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/donald%40stufft.io ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 841 bytes Desc: Message signed with OpenPGP using GPGMail URL: From ncoghlan at gmail.com Sun Jun 16 09:31:29 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 16 Jun 2013 17:31:29 +1000 Subject: [Python-Dev] backported Enum In-Reply-To: References: <51BCC498.1030100@stoneleaf.us> <7w8v2aepsp.fsf@benfinney.id.au> <7wd2rmeh5q.fsf@benfinney.id.au> Message-ID: On 16 June 2013 15:58, Donald Stufft wrote: > > On Jun 16, 2013, at 1:52 AM, Ben Finney wrote: > > Donald Stufft writes: > > On Jun 15, 2013, at 10:45 PM, Ben Finney wrote: > > Is there anything I can do to keep the ?enum? package online for > continuity but make it clear, to automated tools, that this is > end-of-life and obsoleted by another package? > > > Right now the best you can do is make a note of it. Pep 426 let's you > do what you want. > > > Thanks. What specifically in PEP 426 do you mean; how would I make a > note of ?this package is end-of-life as is, please migrate to > ?flufl.enum? instead? using PEP 426 metadata? > > > http://www.python.org/dev/peps/pep-0426/#obsoleted-by > > Note PEP426 isn't completed and isn't implemented :) That specific bit is stable, though :) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ethan at stoneleaf.us Sun Jun 16 08:48:47 2013 From: ethan at stoneleaf.us (Ethan Furman) Date: Sat, 15 Jun 2013 23:48:47 -0700 Subject: [Python-Dev] backported Enum In-Reply-To: <644E4682-4898-49A8-B97D-9C9C06EFEBFA@stufft.io> References: <51BCC498.1030100@stoneleaf.us> <20130615225354.33ab8e09@fsol> <51BCE008.5070308@stoneleaf.us> <644E4682-4898-49A8-B97D-9C9C06EFEBFA@stufft.io> Message-ID: <51BD5FCF.1040203@stoneleaf.us> On 06/15/2013 06:50 PM, Donald Stufft wrote: > > I claimed backport.enum, but you're welcome to the name. I was going to try and backport this PEP under that name anyways. Thank you for the offer, but I think I'll go with GPS's idea of calling it enum34. -- ~Ethan~ From ethan at stoneleaf.us Sun Jun 16 08:53:16 2013 From: ethan at stoneleaf.us (Ethan Furman) Date: Sat, 15 Jun 2013 23:53:16 -0700 Subject: [Python-Dev] backported Enum In-Reply-To: <7wd2rmeh5q.fsf@benfinney.id.au> References: <51BCC498.1030100@stoneleaf.us> <7w8v2aepsp.fsf@benfinney.id.au> <7wd2rmeh5q.fsf@benfinney.id.au> Message-ID: <51BD60DC.40701@stoneleaf.us> On 06/15/2013 10:52 PM, Ben Finney wrote: > Donald Stufft writes: >> On Jun 15, 2013, at 10:45 PM, Ben Finney wrote: >>> >>> Is there anything I can do to keep the ?enum? package online for >>> continuity but make it clear, to automated tools, that this is >>> end-of-life and obsoleted by another package? >> >> Right now the best you can do is make a note of it. Pep 426 let's you >> do what you want. > > Thanks. What specifically in PEP 426 do you mean; how would I make a > note of ?this package is end-of-life as is, please migrate to > ?flufl.enum? instead? using PEP 426 metadata? Note that while flufl.enum was the inspiration, it is not the version that went into 3.4. Differences include: Enum does not allow subclassing when members have been defined int(Enum.member) is an error Similarities include: different Enum enumerations do not compare to each other members in an Enum enumeration are not ordered There may be (and probably are) other differences from flufl.enum, but it's late and I'm not remembering them. So to cut a long story short, your note should say: "migrate to enum34 instead." Oh, and I'm still learning the intricacies of distutils -- you should be able to ignore any errors if you try to install using `easy_install enum34`; I'll hopefully get those kinks worked out in the next few days. -- ~Ethan~ From lukasz at langa.pl Sun Jun 16 11:30:41 2013 From: lukasz at langa.pl (=?utf-8?Q?=C5=81ukasz_Langa?=) Date: Sun, 16 Jun 2013 11:30:41 +0200 Subject: [Python-Dev] Backports galore Message-ID: <7D0CAFF5-8F53-4732-AEB9-4B667AF4667F@langa.pl> Now we have (at least) the following libraries backported from 3.2+ to older versions of Python by members of the core team: - configparser - enum34 - singledispatch - subprocess32 - unittest2 There are also unofficial backports like billiard (multiprocessing). I would be happy if all those were more discoverable by the community at large. Having a single namespace for backports would be great but my spidey sense forebodes large flocks of bike sheds flying that way. Can we put links to those backports in the docs of older versions of Python? Most users would be better off using the updated packages while still deploying on an older release of Python. -- Best regards, ?ukasz Langa WWW: http://lukasz.langa.pl/ Twitter: @llanga IRC: ambv on #python-dev From solipsis at pitrou.net Sun Jun 16 11:50:31 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sun, 16 Jun 2013 11:50:31 +0200 Subject: [Python-Dev] cpython: Issue #3329: Add new APIs to customize memory allocators In-Reply-To: References: <3bXH1R0dqGz7LkT@mail.python.org> <20130615030431.679bfc1b@fsol> <20130615130125.1d4ef1f7@fsol> <51BC8819.5060106@python.org> Message-ID: <20130616115031.6924f071@fsol> On Sun, 16 Jun 2013 01:48:06 +0200 Victor Stinner wrote: > 2013/6/15 Christian Heimes : > > Am 15.06.2013 14:22, schrieb Nick Coghlan: > >> However, it's still desirable to be able to monitor those direct > >> allocations in debug mode, thus it makes sense to have a GIL protected > >> direct allocation API as well. You could try to hide the existence of > >> the latter behaviour and treat it as a private API, but why? For > >> custom allocators, it's useful to be able to *ensure* you can bypass > >> CPython's small object allocator, rather than having to rely on it > >> being bypassed for allocations above a certain size. > > > > There is even more to it. We like to keep track of memory allocations in > > libraries that are wrapped by Python's extension modules, e.g. expat, > > openssl etc. Almost every library has a hook to set a custom memory > > allocator, either globally (CRYPTO_set_mem_functions) or for each object > > (XML_ParserCreate_MM's XML_Memory_Handling_Suite). > > I just create the issue http://bugs.python.org/issue18227: "Use Python > memory allocators in external libraries like zlib or OpenSSL". > > Is it possible to detect if Python is used as a standalone application > (the classic "python" program) or if Python is embedded? If it is > possible, we can modify the "global" memory allocators of a library. The question is why you want to do so, not how/whether to do it. Regards Antoine. From solipsis at pitrou.net Sun Jun 16 11:56:24 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sun, 16 Jun 2013 11:56:24 +0200 Subject: [Python-Dev] cpython: Issue #3329: Add new APIs to customize memory allocators In-Reply-To: References: <3bXH1R0dqGz7LkT@mail.python.org> <20130615030431.679bfc1b@fsol> <20130615130125.1d4ef1f7@fsol> Message-ID: <20130616115624.027a8945@fsol> On Sun, 16 Jun 2013 02:18:32 +0200 Victor Stinner wrote: > 2013/6/15 Antoine Pitrou : > > On Sat, 15 Jun 2013 03:54:50 +0200 > > Victor Stinner wrote: > >> The addition of PyMem_RawMalloc() is motivated by the issue #18203 > >> (Replace calls to malloc() with PyMem_Malloc()). The goal is to be > >> able to setup a custom allocator for *all* allocation made by Python, > >> so malloc() should not be called directly. PyMem_RawMalloc() is > >> required in places where the GIL is not held (ex: in os.getcwd() on > >> Windows). > > > > We already had this discussion on IRC and this argument isn't very > > convincing to me. If os.getcwd() doesn't hold the GIL while allocating > > memory, then you should fix it to hold the GIL while allocating memory. > > The GIL is released for best performances, holding the GIL would have > an impact on performances. Well, do you have benchmark numbers? Do you have a workload where getcwd() is performance-critical to the point that a single GIL-protected allocation may slow down your program? > "Private" functions can be used by applications, it's just that Python > doesn't give any backward compatibility warranty. Am I right? Anyone "can" use anything obviously, but when it's private, it can be changed or removed in any release. If the only goal for these functions is to be used by applications, though, it's quite a bad idea to make them private. Regards Antoine. From ncoghlan at gmail.com Sun Jun 16 12:06:03 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 16 Jun 2013 20:06:03 +1000 Subject: [Python-Dev] cpython: Issue #3329: Add new APIs to customize memory allocators In-Reply-To: <20130616115031.6924f071@fsol> References: <3bXH1R0dqGz7LkT@mail.python.org> <20130615030431.679bfc1b@fsol> <20130615130125.1d4ef1f7@fsol> <51BC8819.5060106@python.org> <20130616115031.6924f071@fsol> Message-ID: On 16 June 2013 19:50, Antoine Pitrou wrote: > On Sun, 16 Jun 2013 01:48:06 +0200 > Victor Stinner wrote: > >> 2013/6/15 Christian Heimes : >> > Am 15.06.2013 14:22, schrieb Nick Coghlan: >> >> However, it's still desirable to be able to monitor those direct >> >> allocations in debug mode, thus it makes sense to have a GIL protected >> >> direct allocation API as well. You could try to hide the existence of >> >> the latter behaviour and treat it as a private API, but why? For >> >> custom allocators, it's useful to be able to *ensure* you can bypass >> >> CPython's small object allocator, rather than having to rely on it >> >> being bypassed for allocations above a certain size. >> > >> > There is even more to it. We like to keep track of memory allocations in >> > libraries that are wrapped by Python's extension modules, e.g. expat, >> > openssl etc. Almost every library has a hook to set a custom memory >> > allocator, either globally (CRYPTO_set_mem_functions) or for each object >> > (XML_ParserCreate_MM's XML_Memory_Handling_Suite). >> >> I just create the issue http://bugs.python.org/issue18227: "Use Python >> memory allocators in external libraries like zlib or OpenSSL". >> >> Is it possible to detect if Python is used as a standalone application >> (the classic "python" program) or if Python is embedded? If it is >> possible, we can modify the "global" memory allocators of a library. > > The question is why you want to do so, not how/whether to do it. I don't think we should be doing that ourselves - it's up to system integrators/embedders to configure those libraries if they want to, we shouldn't be doing it on their behalf. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Sun Jun 16 12:32:02 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 16 Jun 2013 20:32:02 +1000 Subject: [Python-Dev] Backports galore In-Reply-To: <7D0CAFF5-8F53-4732-AEB9-4B667AF4667F@langa.pl> References: <7D0CAFF5-8F53-4732-AEB9-4B667AF4667F@langa.pl> Message-ID: On 16 June 2013 19:30, ?ukasz Langa wrote: > Now we have (at least) the following libraries backported from 3.2+ to > older versions of Python by members of the core team: > > - configparser > - enum34 > - singledispatch > - subprocess32 > - unittest2 > > There are also unofficial backports like billiard (multiprocessing). Step one is compiling a list. I started a page for that here: http://wiki.python.org/moin/StandardLibraryBackports (It's reachable from the front page of the wiki via the Useful Modules page) > I would be happy if all those were more discoverable by the community > at large. Having a single namespace for backports would be great but > my spidey sense forebodes large flocks of bike sheds flying that way. > > Can we put links to those backports in the docs of older versions of > Python? Most users would be better off using the updated packages > while still deploying on an older release of Python. It's probably better to start with a list anyone can edit, rather than links that can only be updated by core developers. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ben+python at benfinney.id.au Sun Jun 16 15:17:31 2013 From: ben+python at benfinney.id.au (Ben Finney) Date: Sun, 16 Jun 2013 23:17:31 +1000 Subject: [Python-Dev] backported Enum References: <51BCC498.1030100@stoneleaf.us> <20130615225354.33ab8e09@fsol> <51BCE008.5070308@stoneleaf.us> <644E4682-4898-49A8-B97D-9C9C06EFEBFA@stufft.io> <51BD5FCF.1040203@stoneleaf.us> Message-ID: <7wehc2w5xw.fsf@benfinney.id.au> Ethan Furman writes: > Thank you for the offer, but I think I'll go with GPS's idea of > calling it enum34. That name will be confusing once Python 3.5 exists (even as a development version). I'd argue that it is confusing even now, since this is a backport for Python version *earlier* than Python 3.4. The name ?stdlib.enum? avoids these problems, do you think that is suitable? -- \ ?If trees could scream, would we be so cavalier about cutting | `\ them down? We might, if they screamed all the time, for no good | _o__) reason.? ?Jack Handey | Ben Finney From barry at python.org Sun Jun 16 17:19:12 2013 From: barry at python.org (Barry Warsaw) Date: Sun, 16 Jun 2013 11:19:12 -0400 Subject: [Python-Dev] backported Enum In-Reply-To: <51BD60DC.40701@stoneleaf.us> References: <51BCC498.1030100@stoneleaf.us> <7w8v2aepsp.fsf@benfinney.id.au> <7wd2rmeh5q.fsf@benfinney.id.au> <51BD60DC.40701@stoneleaf.us> Message-ID: <20130616111912.1a59b0fe@anarchist> On Jun 15, 2013, at 11:53 PM, Ethan Furman wrote: >Note that while flufl.enum was the inspiration, it is not the version that >went into 3.4. I haven't yet decided whether to continue with development of flufl.enum or not. I think there's a lot of value in promoting a standard enum library and having PEP 435 in both the 3.4 stdlib and in a good backport may outweigh the differences. I mostly agree with everything in PEP 435 . One big test will be porting Mailman 3 to enum34. I expect that to mostly Just Work and I don't think the differences will affect it (I've already adjusted for the lack of inheritance - mostly I have to see what effect it will have on the database layer). OTOH, if folks really depend on the differences in flufl.enum, I will probably at least continue to maintain it, but not add any new features. -Barry From victor.stinner at gmail.com Sun Jun 16 20:59:26 2013 From: victor.stinner at gmail.com (Victor Stinner) Date: Sun, 16 Jun 2013 20:59:26 +0200 Subject: [Python-Dev] Backports galore In-Reply-To: <7D0CAFF5-8F53-4732-AEB9-4B667AF4667F@langa.pl> References: <7D0CAFF5-8F53-4732-AEB9-4B667AF4667F@langa.pl> Message-ID: There is also faulthandler on PyPI. It is not really a backport since the project developement started on PyPI. Victor Le dimanche 16 juin 2013, ?ukasz Langa a ?crit : > Now we have (at least) the following libraries backported from 3.2+ to > older versions of Python by members of the core team: > > - configparser > - enum34 > - singledispatch > - subprocess32 > - unittest2 > > There are also unofficial backports like billiard (multiprocessing). > > I would be happy if all those were more discoverable by the community > at large. Having a single namespace for backports would be great but > my spidey sense forebodes large flocks of bike sheds flying that way. > > Can we put links to those backports in the docs of older versions of > Python? Most users would be better off using the updated packages > while still deploying on an older release of Python. > > -- > Best regards, > ?ukasz Langa > > WWW: http://lukasz.langa.pl/ > Twitter: @llanga > IRC: ambv on #python-dev > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/victor.stinner%40gmail.com > -------------- next part -------------- An HTML attachment was scrubbed... URL: From eliben at gmail.com Sun Jun 16 22:13:00 2013 From: eliben at gmail.com (Eli Bendersky) Date: Sun, 16 Jun 2013 13:13:00 -0700 Subject: [Python-Dev] backported Enum In-Reply-To: <20130616111912.1a59b0fe@anarchist> References: <51BCC498.1030100@stoneleaf.us> <7w8v2aepsp.fsf@benfinney.id.au> <7wd2rmeh5q.fsf@benfinney.id.au> <51BD60DC.40701@stoneleaf.us> <20130616111912.1a59b0fe@anarchist> Message-ID: On Sun, Jun 16, 2013 at 8:19 AM, Barry Warsaw wrote: > On Jun 15, 2013, at 11:53 PM, Ethan Furman wrote: > > >Note that while flufl.enum was the inspiration, it is not the version that > >went into 3.4. > > I haven't yet decided whether to continue with development of flufl.enum or > not. I think there's a lot of value in promoting a standard enum library > and > having PEP 435 in both the 3.4 stdlib and in a good backport may outweigh > the > differences. I mostly agree with everything in PEP 435 . > > One big test will be porting Mailman 3 to enum34. I expect that to mostly > Just Work and I don't think the differences will affect it (I've already > adjusted for the lack of inheritance - mostly I have to see what effect it > will have on the database layer). > > OTOH, if folks really depend on the differences in flufl.enum, I will > probably > at least continue to maintain it, but not add any new features. > If you write down the process of porting mailmain from flufl to stdlib.enum in some place, it can make the process much easier for others, and even encourage them to follow the same path. Eli -------------- next part -------------- An HTML attachment was scrubbed... URL: From vinay_sajip at yahoo.co.uk Mon Jun 17 18:27:45 2013 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Mon, 17 Jun 2013 16:27:45 +0000 (UTC) Subject: [Python-Dev] Problem with pydoc script in PEP 405 venvs Message-ID: Issue #18224 (http://bugs.python.org/issue18224) highlights a problem on Windows with the pydoc script provided with venvs created by pyvenv. On POSIX, the script is named pydoc and causes no problems: on Windows, it is called pydoc.py and this causes problems because it shadows the stdlib pydoc module. Possible solutions: 1. Remove the pydoc script altogether from created venvs, on Windows but also on POSIX (for consistency). 2. Rename the pydoc script on both Windows and POSIX (e.g. to pydocs.py and pydocs respectively). 3. Rename the pydoc.py script to pydoc-script.py and introduce a simple .exe launcher pydoc.exe adjacent to it (which is how setuptools and distlib handle installed scripts). The first two approaches are backwards-incompatible, while the third is less likely to lead to breakage, but involves adding a Windows script launcher to Python. While this is a bigger change, I think any built-in Python installer functionality should include such a launcher (as setuptools and distlib do). Still, that's probably a discussion for another day. Does anyone have any comments? Approach #2 seems the most appropriate. I assume it would be reasonable to implement this in both 3.3 and 3.4, as it's not a change in core Python APIs. In the absence of adverse feedback here, I propose to implement approach #2 on both 3.3 and 3.4. Regards, Vinay Sajip From adityaavinash143 at gmail.com Mon Jun 17 18:42:12 2013 From: adityaavinash143 at gmail.com (Aditya Avinash) Date: Mon, 17 Jun 2013 22:12:12 +0530 Subject: [Python-Dev] Writing Extensions for Python 3 in Windows Message-ID: Hi. This is the last place where I want to ask a question. I have searched for lots of tutorials and documentation on the web but, didn't find a decent one to develop extensions for Python 3 using a custom compiler (mingw32, nvcc). Please help me. PS: Don't point me to Python Documentation. It is not good for beginners. It doesn't elaborate about calls and implementation. -- Aditya Avinash Atluri -------------- next part -------------- An HTML attachment was scrubbed... URL: From walter at livinglogic.de Mon Jun 17 19:04:41 2013 From: walter at livinglogic.de (=?UTF-8?B?V2FsdGVyIETDtnJ3YWxk?=) Date: Mon, 17 Jun 2013 19:04:41 +0200 Subject: [Python-Dev] eval and triple quoted strings In-Reply-To: References: <51BB2EDC.80604@livinglogic.de> <51BB5CBF.5070308@gmail.com> Message-ID: <51BF41A9.4040400@livinglogic.de> On 14.06.13 23:03, PJ Eby wrote: > On Fri, Jun 14, 2013 at 2:11 PM, Ron Adam wrote: >> >> >> On 06/14/2013 10:36 AM, Guido van Rossum wrote: >>> >>> Not a bug. The same is done for file input -- CRLF is changed to LF before >>> tokenizing. >> >> >> >> Should this be the same? >> >> >> python3 -c 'print(bytes("""\r\n""", "utf8"))' >> b'\r\n' >> >> >>>>> eval('print(bytes("""\r\n""", "utf8"))') >> b'\n' > > No, but: > > eval(r'print(bytes("""\r\n""", "utf8"))') > > should be. (And is.) > > What I believe you and Walter are missing is that the \r\n in the eval > strings are converted early if you don't make the enclosing string > raw. So what you're eval-ing is not what you think you are eval-ing, > hence the confusion. I expected that eval()ing a string that contains the characters U+0027: APOSTROPHE U+0027: APOSTROPHE U+0027: APOSTROPHE U+000D: CR U+000A: LR U+0027: APOSTROPHE U+0027: APOSTROPHE U+0027: APOSTROPHE to return a string containing the characters: U+000D: CR U+000A: LR Making the string raw, of course turns it into: U+0027: APOSTROPHE U+0027: APOSTROPHE U+0027: APOSTROPHE U+005C: REVERSE SOLIDUS U+0072: LATIN SMALL LETTER R U+005C: REVERSE SOLIDUS U+006E: LATIN SMALL LETTER N U+0027: APOSTROPHE U+0027: APOSTROPHE U+0027: APOSTROPHE and eval()ing that does indeed give "\r\n" as expected. Hmm, it seems that codecs.unicode_escape_decode() does what I want: >>> codecs.unicode_escape_decode("\r\n\\r\\n\\x0d\\x0a\\u000d\\u000a") ('\r\n\r\n\r\n\r\n', 26) Servus, Walter From brett at python.org Mon Jun 17 19:05:35 2013 From: brett at python.org (Brett Cannon) Date: Mon, 17 Jun 2013 13:05:35 -0400 Subject: [Python-Dev] Writing Extensions for Python 3 in Windows In-Reply-To: References: Message-ID: So the best place to ask this question is python-list and not python-dev as this is about developing *for* Python and not *with* Python. On Mon, Jun 17, 2013 at 12:42 PM, Aditya Avinash wrote: > Hi. This is the last place where I want to ask a question. I have searched > for lots of tutorials and documentation on the web but, didn't find a > decent one to develop extensions for Python 3 using a custom compiler > (mingw32, nvcc). Please help me. > PS: Don't point me to Python Documentation. It is not good for beginners. > It doesn't elaborate about calls and implementation. > > -- > Aditya Avinash Atluri > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/brett%40python.org > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From shibturn at gmail.com Mon Jun 17 19:12:49 2013 From: shibturn at gmail.com (Richard Oudkerk) Date: Mon, 17 Jun 2013 18:12:49 +0100 Subject: [Python-Dev] Problem with pydoc script in PEP 405 venvs In-Reply-To: References: Message-ID: On 17/06/2013 5:27pm, Vinay Sajip wrote: > 3. Rename the pydoc.py script to pydoc-script.py and introduce a simple .exe > launcher pydoc.exe adjacent to it (which is how setuptools and distlib > handle installed scripts). Can't a batch file pydoc.bat be used? -- Richard From walter at livinglogic.de Mon Jun 17 19:14:52 2013 From: walter at livinglogic.de (=?UTF-8?B?V2FsdGVyIETDtnJ3YWxk?=) Date: Mon, 17 Jun 2013 19:14:52 +0200 Subject: [Python-Dev] eval and triple quoted strings In-Reply-To: <51BF41A9.4040400@livinglogic.de> References: <51BB2EDC.80604@livinglogic.de> <51BB5CBF.5070308@gmail.com> <51BF41A9.4040400@livinglogic.de> Message-ID: <51BF440C.1090900@livinglogic.de> On 17.06.13 19:04, Walter D?rwald wrote: > Hmm, it seems that codecs.unicode_escape_decode() does what I want: > > >>> codecs.unicode_escape_decode("\r\n\\r\\n\\x0d\\x0a\\u000d\\u000a") > ('\r\n\r\n\r\n\r\n', 26) Hmm, no it doesn't: >>> codecs.unicode_escape_decode("\u1234") ('?\x88?', 3) Servus, Walter From guido at python.org Mon Jun 17 19:22:01 2013 From: guido at python.org (Guido van Rossum) Date: Mon, 17 Jun 2013 10:22:01 -0700 Subject: [Python-Dev] eval and triple quoted strings In-Reply-To: <51BF41A9.4040400@livinglogic.de> References: <51BB2EDC.80604@livinglogic.de> <51BB5CBF.5070308@gmail.com> <51BF41A9.4040400@livinglogic.de> Message-ID: On Mon, Jun 17, 2013 at 10:04 AM, Walter D?rwald wrote: > I expected that eval()ing a string that contains the characters > > U+0027: APOSTROPHE > U+0027: APOSTROPHE > U+0027: APOSTROPHE > U+000D: CR > U+000A: LR > U+0027: APOSTROPHE > U+0027: APOSTROPHE > U+0027: APOSTROPHE > > to return a string containing the characters: > > U+000D: CR > U+000A: LR No. Executing a file containing those exact characters produces a string containing only '\n' and exec/eval is meant to behave the same way. The string may not have originated from a file, so the universal newlines behavior of the io module is irrelevant here -- the parser must implement its own equivalent processing, and it does. -- --Guido van Rossum (python.org/~guido) From vinay_sajip at yahoo.co.uk Mon Jun 17 20:37:49 2013 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Mon, 17 Jun 2013 18:37:49 +0000 (UTC) Subject: [Python-Dev] Problem with pydoc script in PEP 405 venvs References: Message-ID: Richard Oudkerk gmail.com> writes: > > Can't a batch file pydoc.bat be used? > I generally find .bat files so limiting that I never thought of that. In this case, it makes perfect sense to use one. Thanks! @echo off __VENV_PYTHON__ -c "import sys, pydoc; sys.exit(pydoc.cli())" should do it. Regards, Vinay Sajip From rdmurray at bitdance.com Mon Jun 17 20:45:05 2013 From: rdmurray at bitdance.com (R. David Murray) Date: Mon, 17 Jun 2013 14:45:05 -0400 Subject: [Python-Dev] Problem with pydoc script in PEP 405 venvs In-Reply-To: References: Message-ID: <20130617184506.27624250169@webabinitio.net> On Mon, 17 Jun 2013 16:27:45 -0000, Vinay Sajip wrote: > Issue #18224 (http://bugs.python.org/issue18224) highlights a problem on > Windows with the pydoc script provided with venvs created by pyvenv. On > POSIX, the script is named pydoc and causes no problems: on Windows, it is > called pydoc.py and this causes problems because it shadows the stdlib pydoc > module. > > Possible solutions: > > 1. Remove the pydoc script altogether from created venvs, on Windows but > also on POSIX (for consistency). > 2. Rename the pydoc script on both Windows and POSIX (e.g. to pydocs.py and > pydocs respectively). > 3. Rename the pydoc.py script to pydoc-script.py and introduce a simple .exe > launcher pydoc.exe adjacent to it (which is how setuptools and distlib > handle installed scripts). > > The first two approaches are backwards-incompatible, while the third is less > likely to lead to breakage, but involves adding a Windows script launcher to > Python. While this is a bigger change, I think any built-in Python installer > functionality should include such a launcher (as setuptools and distlib do). > Still, that's probably a discussion for another day. > > Does anyone have any comments? Approach #2 seems the most appropriate. I > assume it would be reasonable to implement this in both 3.3 and 3.4, as it's > not a change in core Python APIs. > > In the absence of adverse feedback here, I propose to implement approach #2 > on both 3.3 and 3.4. I don't think renaming pydoc to pydocs is appropriate on posix. Too many people likely have 'pydoc' in their fingers and brains as the command name. The .bat file suggestion seems better, IMO. --David From phd at phdru.name Mon Jun 17 21:15:12 2013 From: phd at phdru.name (Oleg Broytman) Date: Mon, 17 Jun 2013 23:15:12 +0400 Subject: [Python-Dev] Problem with pydoc script in PEP 405 venvs In-Reply-To: References: Message-ID: <20130617191512.GA12889@iskra.aviel.ru> On Mon, Jun 17, 2013 at 06:37:49PM +0000, Vinay Sajip wrote: > @echo off > __VENV_PYTHON__ -c "import sys, pydoc; sys.exit(pydoc.cli())" I think you want to pass command line arguments: @echo off __VENV_PYTHON__ -c "import sys, pydoc; sys.exit(pydoc.cli())" %1 %2 %3 %4 %5 %6 %7 %8 %9 Oleg. -- Oleg Broytman http://phdru.name/ phd at phdru.name Programmers don't die, they just GOSUB without RETURN. From vinay_sajip at yahoo.co.uk Mon Jun 17 22:30:54 2013 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Mon, 17 Jun 2013 20:30:54 +0000 (UTC) Subject: [Python-Dev] Problem with pydoc script in PEP 405 venvs References: <20130617191512.GA12889@iskra.aviel.ru> Message-ID: Oleg Broytman phdru.name> writes: > > On Mon, Jun 17, 2013 at 06:37:49PM +0000, Vinay Sajip yahoo.co.uk> wrote: > > echo off > > __VENV_PYTHON__ -c "import sys, pydoc; sys.exit(pydoc.cli())" > > I think you want to pass command line arguments: > > echo off > __VENV_PYTHON__ -c "import sys, pydoc; sys.exit(pydoc.cli())" %1 %2 %3 %4 %5 %6 %7 %8 %9 > That was off-the-cuff, what I actually implemented (in my sandbox repo) was @echo off __VENV_PYTHON__ -c "import sys, pydoc; sys.argv[0] = 'pydoc'; sys.exit(pydoc.cli())" %* Regards, Vinay Sajip From ron3200 at gmail.com Mon Jun 17 23:52:48 2013 From: ron3200 at gmail.com (Ron Adam) Date: Mon, 17 Jun 2013 16:52:48 -0500 Subject: [Python-Dev] eval and triple quoted strings In-Reply-To: <51BF41A9.4040400@livinglogic.de> References: <51BB2EDC.80604@livinglogic.de> <51BB5CBF.5070308@gmail.com> <51BF41A9.4040400@livinglogic.de> Message-ID: On 06/17/2013 12:04 PM, Walter D?rwald wrote: > Making the string raw, of course turns it into: > > U+0027: APOSTROPHE > U+0027: APOSTROPHE > U+0027: APOSTROPHE > U+005C: REVERSE SOLIDUS > U+0072: LATIN SMALL LETTER R > U+005C: REVERSE SOLIDUS > U+006E: LATIN SMALL LETTER N > U+0027: APOSTROPHE > U+0027: APOSTROPHE > U+0027: APOSTROPHE > > and eval()ing that does indeed give "\r\n" as expected. You can also escape the reverse slashes in a regular string to get the same result. >>> s1 = "'''\\r\\n'''" >>> list(s1) ["'", "'", "'", '\\', 'r', '\\', 'n', "'", "'", "'"] >>> s2 = eval(s1) >>> list(s2) ['\r', '\n'] >>> s3 = "'''%s'''" % s2 >>> list(s3) ["'", "'", "'", '\r', '\n', "'", "'", "'"] >>> s4 = eval(s3) >>> list(s4) ['\n'] When a standard string literal used with eval, it's evaluated first to a string object in the same scope as the eval function is called from, then the eval function is called with that string object and it's evaluated again. So it's really being parsed twice. (That's the part that got me.) The transformation between s1 and s2 is what Phillip is referring to, and Guido is referring to the transformation from s2 to s4. (s3 is needed to avoid the end of line error of evaluating a single quoted string with \n in it.) When a sting literal is used directly with eval, it looks like it is evaluated from s1 to s4 in one step, but that isn't what is happening. Cheers, Ron (ps: Was informed my posts where showing up twice.. hopefully I got that fixed now.) From greg.ewing at canterbury.ac.nz Tue Jun 18 00:18:24 2013 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Tue, 18 Jun 2013 10:18:24 +1200 Subject: [Python-Dev] eval and triple quoted strings In-Reply-To: References: <51BB2EDC.80604@livinglogic.de> <51BB5CBF.5070308@gmail.com> <51BF41A9.4040400@livinglogic.de> Message-ID: <51BF8B30.70707@canterbury.ac.nz> Guido van Rossum wrote: > No. Executing a file containing those exact characters produces a > string containing only '\n' and exec/eval is meant to behave the same > way. The string may not have originated from a file, so the universal > newlines behavior of the io module is irrelevant here -- the parser > must implement its own equivalent processing, and it does. I'm still not convinced that this is necessary or desirable behaviour. I can understand the parser doing this as a workaround before we had universal newlines, but now that we do, I'd expect any Python string to already have newlines converted to their canonical representation, and that any CRs it contains are meant to be there. The parser shouldn't need to do newline translation a second time. -- Greg From guido at python.org Tue Jun 18 00:35:12 2013 From: guido at python.org (Guido van Rossum) Date: Mon, 17 Jun 2013 15:35:12 -0700 Subject: [Python-Dev] eval and triple quoted strings In-Reply-To: <51BF8B30.70707@canterbury.ac.nz> References: <51BB2EDC.80604@livinglogic.de> <51BB5CBF.5070308@gmail.com> <51BF41A9.4040400@livinglogic.de> <51BF8B30.70707@canterbury.ac.nz> Message-ID: On Mon, Jun 17, 2013 at 3:18 PM, Greg Ewing wrote: > Guido van Rossum wrote: >> >> No. Executing a file containing those exact characters produces a >> string containing only '\n' and exec/eval is meant to behave the same >> way. The string may not have originated from a file, so the universal >> newlines behavior of the io module is irrelevant here -- the parser >> must implement its own equivalent processing, and it does. > I'm still not convinced that this is necessary or desirable > behaviour. I can understand the parser doing this as a > workaround before we had universal newlines, but now that > we do, I'd expect any Python string to already have newlines > converted to their canonical representation, and that any CRs > it contains are meant to be there. The parser shouldn't need > to do newline translation a second time. There are other ways to get a string besides reading it from a file. Anyway, I think that if you want a string literal that contains \r\n as its line endings, you should use a syntactic solution, and the syntax ought to be the same regardless of whether you are reading it from a file or from a string literal. That syntactic solution is very clear: """line one\r line two\r line three\r """ This works everywhere. -- --Guido van Rossum (python.org/~guido) From benjamin at python.org Tue Jun 18 01:40:01 2013 From: benjamin at python.org (Benjamin Peterson) Date: Mon, 17 Jun 2013 16:40:01 -0700 Subject: [Python-Dev] eval and triple quoted strings In-Reply-To: <51BF8B30.70707@canterbury.ac.nz> References: <51BB2EDC.80604@livinglogic.de> <51BB5CBF.5070308@gmail.com> <51BF41A9.4040400@livinglogic.de> <51BF8B30.70707@canterbury.ac.nz> Message-ID: 2013/6/17 Greg Ewing : > Guido van Rossum wrote: >> >> No. Executing a file containing those exact characters produces a >> string containing only '\n' and exec/eval is meant to behave the same >> way. The string may not have originated from a file, so the universal >> newlines behavior of the io module is irrelevant here -- the parser >> must implement its own equivalent processing, and it does. > > > I'm still not convinced that this is necessary or desirable > behaviour. I can understand the parser doing this as a > workaround before we had universal newlines, but now that > we do, I'd expect any Python string to already have newlines > converted to their canonical representation, and that any CRs > it contains are meant to be there. The parser shouldn't need > to do newline translation a second time. It used to be that way until 2.7. People like to do things like with open("myfile.py", "rb") as fp: exec fp.read() in ns which used to fail with CRLF newlines because binary mode doesn't have them. I think this is actually the correct way to execute Python sources because the parser then handles the somewhat complicated process of decoding Python source for you. -- Regards, Benjamin From guido at python.org Tue Jun 18 01:52:21 2013 From: guido at python.org (Guido van Rossum) Date: Mon, 17 Jun 2013 16:52:21 -0700 Subject: [Python-Dev] eval and triple quoted strings In-Reply-To: References: <51BB2EDC.80604@livinglogic.de> <51BB5CBF.5070308@gmail.com> <51BF41A9.4040400@livinglogic.de> <51BF8B30.70707@canterbury.ac.nz> Message-ID: On Mon, Jun 17, 2013 at 4:40 PM, Benjamin Peterson wrote: > 2013/6/17 Greg Ewing : >> Guido van Rossum wrote: >>> >>> No. Executing a file containing those exact characters produces a >>> string containing only '\n' and exec/eval is meant to behave the same >>> way. The string may not have originated from a file, so the universal >>> newlines behavior of the io module is irrelevant here -- the parser >>> must implement its own equivalent processing, and it does. >> >> >> I'm still not convinced that this is necessary or desirable >> behaviour. I can understand the parser doing this as a >> workaround before we had universal newlines, but now that >> we do, I'd expect any Python string to already have newlines >> converted to their canonical representation, and that any CRs >> it contains are meant to be there. The parser shouldn't need >> to do newline translation a second time. > > It used to be that way until 2.7. People like to do things like > > with open("myfile.py", "rb") as fp: > exec fp.read() in ns > > which used to fail with CRLF newlines because binary mode doesn't have > them. I think this is actually the correct way to execute Python > sources because the parser then handles the somewhat complicated > process of decoding Python source for you. What exactly does the parser handles better than the io module? Is it just the coding cookies? I suppose that works as long as the file is encoded using as ASCII superset like the Latin-N variants or UTF-8. It would fail pretty badly if it was UTF-16 (and yes, that's an abominable encoding for other reasons :-). -- --Guido van Rossum (python.org/~guido) From benjamin at python.org Tue Jun 18 02:02:26 2013 From: benjamin at python.org (Benjamin Peterson) Date: Mon, 17 Jun 2013 17:02:26 -0700 Subject: [Python-Dev] eval and triple quoted strings In-Reply-To: References: <51BB2EDC.80604@livinglogic.de> <51BB5CBF.5070308@gmail.com> <51BF41A9.4040400@livinglogic.de> <51BF8B30.70707@canterbury.ac.nz> Message-ID: 2013/6/17 Guido van Rossum : > On Mon, Jun 17, 2013 at 4:40 PM, Benjamin Peterson wrote: >> 2013/6/17 Greg Ewing : >>> Guido van Rossum wrote: >>>> >>>> No. Executing a file containing those exact characters produces a >>>> string containing only '\n' and exec/eval is meant to behave the same >>>> way. The string may not have originated from a file, so the universal >>>> newlines behavior of the io module is irrelevant here -- the parser >>>> must implement its own equivalent processing, and it does. >>> >>> >>> I'm still not convinced that this is necessary or desirable >>> behaviour. I can understand the parser doing this as a >>> workaround before we had universal newlines, but now that >>> we do, I'd expect any Python string to already have newlines >>> converted to their canonical representation, and that any CRs >>> it contains are meant to be there. The parser shouldn't need >>> to do newline translation a second time. >> >> It used to be that way until 2.7. People like to do things like >> >> with open("myfile.py", "rb") as fp: >> exec fp.read() in ns >> >> which used to fail with CRLF newlines because binary mode doesn't have >> them. I think this is actually the correct way to execute Python >> sources because the parser then handles the somewhat complicated >> process of decoding Python source for you. > > What exactly does the parser handles better than the io module? Is it > just the coding cookies? I suppose that works as long as the file is > encoded using as ASCII superset like the Latin-N variants or UTF-8. It > would fail pretty badly if it was UTF-16 (and yes, that's an > abominable encoding for other reasons :-). The coding cookie is the main one. In fact, if you can't parse that, you don't really know what encoding to open the file with at all. There's also small things like BOM handling (you have to use the utf-16-sig encoding with TextIO to get it removed) and defaulting to UTF-8 (which the io module doesn't do) which is better left to the parser. -- Regards, Benjamin From guido at python.org Tue Jun 18 02:06:03 2013 From: guido at python.org (Guido van Rossum) Date: Mon, 17 Jun 2013 17:06:03 -0700 Subject: [Python-Dev] eval and triple quoted strings In-Reply-To: References: <51BB2EDC.80604@livinglogic.de> <51BB5CBF.5070308@gmail.com> <51BF41A9.4040400@livinglogic.de> <51BF8B30.70707@canterbury.ac.nz> Message-ID: On Mon, Jun 17, 2013 at 5:02 PM, Benjamin Peterson wrote: > 2013/6/17 Guido van Rossum : >> On Mon, Jun 17, 2013 at 4:40 PM, Benjamin Peterson wrote: >>> 2013/6/17 Greg Ewing : >>>> Guido van Rossum wrote: >>>>> >>>>> No. Executing a file containing those exact characters produces a >>>>> string containing only '\n' and exec/eval is meant to behave the same >>>>> way. The string may not have originated from a file, so the universal >>>>> newlines behavior of the io module is irrelevant here -- the parser >>>>> must implement its own equivalent processing, and it does. >>>> >>>> >>>> I'm still not convinced that this is necessary or desirable >>>> behaviour. I can understand the parser doing this as a >>>> workaround before we had universal newlines, but now that >>>> we do, I'd expect any Python string to already have newlines >>>> converted to their canonical representation, and that any CRs >>>> it contains are meant to be there. The parser shouldn't need >>>> to do newline translation a second time. >>> >>> It used to be that way until 2.7. People like to do things like >>> >>> with open("myfile.py", "rb") as fp: >>> exec fp.read() in ns >>> >>> which used to fail with CRLF newlines because binary mode doesn't have >>> them. I think this is actually the correct way to execute Python >>> sources because the parser then handles the somewhat complicated >>> process of decoding Python source for you. >> >> What exactly does the parser handles better than the io module? Is it >> just the coding cookies? I suppose that works as long as the file is >> encoded using as ASCII superset like the Latin-N variants or UTF-8. It >> would fail pretty badly if it was UTF-16 (and yes, that's an >> abominable encoding for other reasons :-). > > The coding cookie is the main one. In fact, if you can't parse that, > you don't really know what encoding to open the file with at all. > There's also small things like BOM handling (you have to use the > utf-16-sig encoding with TextIO to get it removed) and defaulting to > UTF-8 (which the io module doesn't do) which is better left to the > parser. Maybe there are some lessons here that the TextIO module could learn? -- --Guido van Rossum (python.org/~guido) From victor.stinner at gmail.com Tue Jun 18 02:22:18 2013 From: victor.stinner at gmail.com (Victor Stinner) Date: Tue, 18 Jun 2013 02:22:18 +0200 Subject: [Python-Dev] eval and triple quoted strings In-Reply-To: References: <51BB2EDC.80604@livinglogic.de> <51BB5CBF.5070308@gmail.com> <51BF41A9.4040400@livinglogic.de> <51BF8B30.70707@canterbury.ac.nz> Message-ID: It may be possible to implement parsing the codec cookie as a Python codec :-) Victor 2013/6/18 Guido van Rossum : > On Mon, Jun 17, 2013 at 5:02 PM, Benjamin Peterson wrote: >> 2013/6/17 Guido van Rossum : >>> On Mon, Jun 17, 2013 at 4:40 PM, Benjamin Peterson wrote: >>>> 2013/6/17 Greg Ewing : >>>>> Guido van Rossum wrote: >>>>>> >>>>>> No. Executing a file containing those exact characters produces a >>>>>> string containing only '\n' and exec/eval is meant to behave the same >>>>>> way. The string may not have originated from a file, so the universal >>>>>> newlines behavior of the io module is irrelevant here -- the parser >>>>>> must implement its own equivalent processing, and it does. >>>>> >>>>> >>>>> I'm still not convinced that this is necessary or desirable >>>>> behaviour. I can understand the parser doing this as a >>>>> workaround before we had universal newlines, but now that >>>>> we do, I'd expect any Python string to already have newlines >>>>> converted to their canonical representation, and that any CRs >>>>> it contains are meant to be there. The parser shouldn't need >>>>> to do newline translation a second time. >>>> >>>> It used to be that way until 2.7. People like to do things like >>>> >>>> with open("myfile.py", "rb") as fp: >>>> exec fp.read() in ns >>>> >>>> which used to fail with CRLF newlines because binary mode doesn't have >>>> them. I think this is actually the correct way to execute Python >>>> sources because the parser then handles the somewhat complicated >>>> process of decoding Python source for you. >>> >>> What exactly does the parser handles better than the io module? Is it >>> just the coding cookies? I suppose that works as long as the file is >>> encoded using as ASCII superset like the Latin-N variants or UTF-8. It >>> would fail pretty badly if it was UTF-16 (and yes, that's an >>> abominable encoding for other reasons :-). >> >> The coding cookie is the main one. In fact, if you can't parse that, >> you don't really know what encoding to open the file with at all. >> There's also small things like BOM handling (you have to use the >> utf-16-sig encoding with TextIO to get it removed) and defaulting to >> UTF-8 (which the io module doesn't do) which is better left to the >> parser. > > Maybe there are some lessons here that the TextIO module could learn? > > -- > --Guido van Rossum (python.org/~guido) > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/victor.stinner%40gmail.com From ron3200 at gmail.com Tue Jun 18 02:44:21 2013 From: ron3200 at gmail.com (Ron Adam) Date: Mon, 17 Jun 2013 19:44:21 -0500 Subject: [Python-Dev] eval and triple quoted strings In-Reply-To: <51BF8B30.70707@canterbury.ac.nz> References: <51BB2EDC.80604@livinglogic.de> <51BB5CBF.5070308@gmail.com> <51BF41A9.4040400@livinglogic.de> <51BF8B30.70707@canterbury.ac.nz> Message-ID: On 06/17/2013 05:18 PM, Greg Ewing wrote: > I'm still not convinced that this is necessary or desirable > behaviour. I can understand the parser doing this as a > workaround before we had universal newlines, but now that > we do, I'd expect any Python string to already have newlines > converted to their canonical representation, and that any CRs > it contains are meant to be there. The parser shouldn't need > to do newline translation a second time. It's the other way around. Eval and exec should generate the same results as pythons compiler with the same input, including errors and exceptions. The only way we can have that is if eval and exec parses everything the same way. It's the first parsing that needs to be avoided or compensated for in these cases. Raw strings (my preference) works for string literals, or you can escape the escape codes so they are still individual characters after the first translation. Or read the code directly from a file rather than importing it. For example, if you wrote your own python console program, you would want all the errors and exceptions to come from eval, including those for bad strings. You would still need to feed the bad strings to eval. If you don't then you won't get the same output from eval as the compiler does. Cheers, Ron From benjamin at python.org Tue Jun 18 02:49:10 2013 From: benjamin at python.org (Benjamin Peterson) Date: Mon, 17 Jun 2013 17:49:10 -0700 Subject: [Python-Dev] eval and triple quoted strings In-Reply-To: References: <51BB2EDC.80604@livinglogic.de> <51BB5CBF.5070308@gmail.com> <51BF41A9.4040400@livinglogic.de> <51BF8B30.70707@canterbury.ac.nz> Message-ID: 2013/6/17 Guido van Rossum : > On Mon, Jun 17, 2013 at 5:02 PM, Benjamin Peterson wrote: >> 2013/6/17 Guido van Rossum : >>> On Mon, Jun 17, 2013 at 4:40 PM, Benjamin Peterson wrote: >>> What exactly does the parser handles better than the io module? Is it >>> just the coding cookies? I suppose that works as long as the file is >>> encoded using as ASCII superset like the Latin-N variants or UTF-8. It >>> would fail pretty badly if it was UTF-16 (and yes, that's an >>> abominable encoding for other reasons :-). >> >> The coding cookie is the main one. In fact, if you can't parse that, >> you don't really know what encoding to open the file with at all. >> There's also small things like BOM handling (you have to use the >> utf-16-sig encoding with TextIO to get it removed) and defaulting to >> UTF-8 (which the io module doesn't do) which is better left to the >> parser. > > Maybe there are some lessons here that the TextIO module could learn? UTF-8 by default would be great, but that ship has sailed. Reading Python coding cookies is outside the purview of TextIOWrapper. However, it would be good to have a function in the stdlib to read a python source file to Unicode; I've definitely implemented that several times. -- Regards, Benjamin From victor.stinner at gmail.com Tue Jun 18 03:03:07 2013 From: victor.stinner at gmail.com (Victor Stinner) Date: Tue, 18 Jun 2013 03:03:07 +0200 Subject: [Python-Dev] cpython: Issue #3329: Add new APIs to customize memory allocators In-Reply-To: <20130616115031.6924f071@fsol> References: <3bXH1R0dqGz7LkT@mail.python.org> <20130615030431.679bfc1b@fsol> <20130615130125.1d4ef1f7@fsol> <51BC8819.5060106@python.org> <20130616115031.6924f071@fsol> Message-ID: 2013/6/16 Antoine Pitrou : > On Sun, 16 Jun 2013 01:48:06 +0200 > Victor Stinner wrote: >> I just create the issue http://bugs.python.org/issue18227: "Use Python >> memory allocators in external libraries like zlib or OpenSSL". >> >> Is it possible to detect if Python is used as a standalone application >> (the classic "python" program) or if Python is embedded? If it is >> possible, we can modify the "global" memory allocators of a library. > > The question is why you want to do so, not how/whether to do it. I want to be able to track the memory usage of all Python memory, even in external libraries, and/or use a custom memory allocator, even in external libraries. Victor From merwok at netwok.org Tue Jun 18 03:07:42 2013 From: merwok at netwok.org (=?UTF-8?B?w4lyaWMgQXJhdWpv?=) Date: Mon, 17 Jun 2013 21:07:42 -0400 Subject: [Python-Dev] eval and triple quoted strings In-Reply-To: References: <51BB2EDC.80604@livinglogic.de> <51BB5CBF.5070308@gmail.com> <51BF41A9.4040400@livinglogic.de> <51BF8B30.70707@canterbury.ac.nz> Message-ID: <51BFB2DE.2050403@netwok.org> Le 17/06/2013 20:49, Benjamin Peterson a ?crit : > Reading Python coding cookies is outside the purview of TextIOWrapper. > However, it would be good to have a function in the stdlib to read a > python source file to Unicode; I've definitely implemented that > several times. IIUC you want http://docs.python.org/3/library/tokenize#tokenize.open (3.2+). Regards From benjamin at python.org Tue Jun 18 03:28:59 2013 From: benjamin at python.org (Benjamin Peterson) Date: Mon, 17 Jun 2013 18:28:59 -0700 Subject: [Python-Dev] eval and triple quoted strings In-Reply-To: <51BFB2DE.2050403@netwok.org> References: <51BB2EDC.80604@livinglogic.de> <51BB5CBF.5070308@gmail.com> <51BF41A9.4040400@livinglogic.de> <51BF8B30.70707@canterbury.ac.nz> <51BFB2DE.2050403@netwok.org> Message-ID: 2013/6/17 ?ric Araujo : > Le 17/06/2013 20:49, Benjamin Peterson a ?crit : >> Reading Python coding cookies is outside the purview of TextIOWrapper. >> However, it would be good to have a function in the stdlib to read a >> python source file to Unicode; I've definitely implemented that >> several times. > > IIUC you want http://docs.python.org/3/library/tokenize#tokenize.open > (3.2+). Yep. :) -- Regards, Benjamin From eliben at gmail.com Tue Jun 18 06:05:53 2013 From: eliben at gmail.com (Eli Bendersky) Date: Mon, 17 Jun 2013 21:05:53 -0700 Subject: [Python-Dev] cpython (2.7): Fix comment blocks. Adjust blocksize to a power-of-two for better divmod In-Reply-To: References: <3bWqYP3sqgzQdN@mail.python.org> <20130614104603.35f83a9c@fsol> Message-ID: On Fri, Jun 14, 2013 at 6:47 AM, Serhiy Storchaka wrote: > 14.06.13 11:46, Antoine Pitrou ???????(??): > >> On Fri, 14 Jun 2013 07:06:49 +0200 (CEST) >> raymond.hettinger wrote: >> >>> http://hg.python.org/cpython/**rev/5accb0ac8bfb >>> changeset: 84116:5accb0ac8bfb >>> Fix comment blocks. Adjust blocksize to a power-of-two for better >>> divmod computations. >>> >> >> Is there any rationale for changing the heuristic from "fits in a whole >> number of cachelines" to "allows fast division by the blocksize"? >> >> I personally would prefer if such changes were justified a bit more >> than by a one-liner changeset message without any corresponding open >> issue. >> > > I share the doubts of Antoine and I was going to write the same comment. I > thought there were good reasons for previous code. What has changed? > I agree it would be interesting to hear about the reasons for the change. Raymond? Eli -------------- next part -------------- An HTML attachment was scrubbed... URL: From ethan at stoneleaf.us Tue Jun 18 08:08:22 2013 From: ethan at stoneleaf.us (Ethan Furman) Date: Mon, 17 Jun 2013 23:08:22 -0700 Subject: [Python-Dev] cpython (2.7): Fix comment blocks. Adjust blocksize to a power-of-two for better divmod In-Reply-To: References: <3bWqYP3sqgzQdN@mail.python.org> <20130614104603.35f83a9c@fsol> Message-ID: <51BFF956.7040403@stoneleaf.us> On 06/17/2013 09:05 PM, Eli Bendersky wrote: > > > > On Fri, Jun 14, 2013 at 6:47 AM, Serhiy Storchaka > wrote: > > 14.06.13 11:46, Antoine Pitrou ???????(??): > > On Fri, 14 Jun 2013 07:06:49 +0200 (CEST) > raymond.hettinger > wrote: > > http://hg.python.org/cpython/__rev/5accb0ac8bfb > changeset: 84116:5accb0ac8bfb > Fix comment blocks. Adjust blocksize to a power-of-two for better divmod computations. > > > Is there any rationale for changing the heuristic from "fits in a whole > number of cachelines" to "allows fast division by the blocksize"? > > I personally would prefer if such changes were justified a bit more > than by a one-liner changeset message without any corresponding open > issue. > > > I share the doubts of Antoine and I was going to write the same comment. I thought there were good reasons for > previous code. What has changed? > > > I agree it would be interesting to hear about the reasons for the change. Raymond? Asking as a learner: are such non-bugfix changes appropriate for the 2.7 line? -- ~Ethan~ From greg at krypto.org Tue Jun 18 09:00:53 2013 From: greg at krypto.org (Gregory P. Smith) Date: Tue, 18 Jun 2013 00:00:53 -0700 Subject: [Python-Dev] cpython (2.7): Fix comment blocks. Adjust blocksize to a power-of-two for better divmod In-Reply-To: <51BFF956.7040403@stoneleaf.us> References: <3bWqYP3sqgzQdN@mail.python.org> <20130614104603.35f83a9c@fsol> <51BFF956.7040403@stoneleaf.us> Message-ID: Raymond - Why did you do this in the 2.7 branch? It hasn't been done in 3.3 or default and it isn't not the sort of change we make in a stable release branch without justification. What issue was this for? What problem were you solving? -gps On Mon, Jun 17, 2013 at 11:08 PM, Ethan Furman wrote: > On 06/17/2013 09:05 PM, Eli Bendersky wrote: > >> >> >> >> On Fri, Jun 14, 2013 at 6:47 AM, Serhiy Storchaka > storchaka at gmail.com>> wrote: >> >> 14.06.13 11:46, Antoine Pitrou ???????(??): >> >> On Fri, 14 Jun 2013 07:06:49 +0200 (CEST) >> raymond.hettinger > python-checkins@**python.org >> wrote: >> >> http://hg.python.org/cpython/_**_rev/5accb0ac8bfb< >> http://hg.python.org/cpython/**rev/5accb0ac8bfb >> > >> >> changeset: 84116:5accb0ac8bfb >> Fix comment blocks. Adjust blocksize to a power-of-two >> for better divmod computations. >> >> >> Is there any rationale for changing the heuristic from "fits in a >> whole >> number of cachelines" to "allows fast division by the blocksize"? >> >> I personally would prefer if such changes were justified a bit >> more >> than by a one-liner changeset message without any corresponding >> open >> issue. >> >> >> I share the doubts of Antoine and I was going to write the same >> comment. I thought there were good reasons for >> previous code. What has changed? >> >> >> I agree it would be interesting to hear about the reasons for the change. >> Raymond? >> > > Asking as a learner: are such non-bugfix changes appropriate for the 2.7 > line? > > -- > ~Ethan~ > > ______________________________**_________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/**mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/**mailman/options/python-dev/** > greg%40krypto.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jeremy.kloth at gmail.com Tue Jun 18 12:56:37 2013 From: jeremy.kloth at gmail.com (Jeremy Kloth) Date: Tue, 18 Jun 2013 04:56:37 -0600 Subject: [Python-Dev] [Python-checkins] cpython (3.3): ctypes: AIX needs an explicit #include to get alloca() In-Reply-To: <3bZ3HS2pvhzQxV@mail.python.org> References: <3bZ3HS2pvhzQxV@mail.python.org> Message-ID: On Mon, Jun 17, 2013 at 2:02 PM, victor.stinner wrote: > diff --git a/Modules/_ctypes/callproc.c b/Modules/_ctypes/callproc.c > --- a/Modules/_ctypes/callproc.c > +++ b/Modules/_ctypes/callproc.c > @@ -70,6 +70,7 @@ > > #include > #include "ctypes.h" > +#include This header is not present on Windows, thus breaking all the Windows buildbots. Perhaps it should be wrapped in an AIX-specific #ifdef? -- Jeremy Kloth From christian at python.org Tue Jun 18 13:28:13 2013 From: christian at python.org (Christian Heimes) Date: Tue, 18 Jun 2013 13:28:13 +0200 Subject: [Python-Dev] cpython (3.3): ctypes: AIX needs an explicit #include to get alloca() In-Reply-To: References: <3bZ3HS2pvhzQxV@mail.python.org> Message-ID: <51C0444D.8030402@python.org> Am 18.06.2013 12:56, schrieb Jeremy Kloth: > On Mon, Jun 17, 2013 at 2:02 PM, victor.stinner > wrote: >> diff --git a/Modules/_ctypes/callproc.c b/Modules/_ctypes/callproc.c >> --- a/Modules/_ctypes/callproc.c >> +++ b/Modules/_ctypes/callproc.c >> @@ -70,6 +70,7 @@ >> >> #include >> #include "ctypes.h" >> +#include > > This header is not present on Windows, thus breaking all the Windows > buildbots. Perhaps it should be wrapped in an AIX-specific #ifdef? I have added HAVE_ALLOCA_H to configure: http://hg.python.org/cpython/rev/7b6ae19dd116 Christian From victor.stinner at gmail.com Tue Jun 18 13:32:14 2013 From: victor.stinner at gmail.com (Victor Stinner) Date: Tue, 18 Jun 2013 13:32:14 +0200 Subject: [Python-Dev] cpython (3.3): ctypes: AIX needs an explicit #include to get alloca() In-Reply-To: <51C0444D.8030402@python.org> References: <3bZ3HS2pvhzQxV@mail.python.org> <51C0444D.8030402@python.org> Message-ID: 2013/6/18 Christian Heimes : > Am 18.06.2013 12:56, schrieb Jeremy Kloth: >> On Mon, Jun 17, 2013 at 2:02 PM, victor.stinner >>> +#include >> >> This header is not present on Windows, thus breaking all the Windows >> buildbots. Perhaps it should be wrapped in an AIX-specific #ifdef? Oh really? Portability is complex :-) I only tested on Linux, but I expected this header to be part of the C standard... > I have added HAVE_ALLOCA_H to configure: > > http://hg.python.org/cpython/rev/7b6ae19dd116 Cool, thanks. Victor From christian at python.org Tue Jun 18 13:45:05 2013 From: christian at python.org (Christian Heimes) Date: Tue, 18 Jun 2013 13:45:05 +0200 Subject: [Python-Dev] cpython (3.3): ctypes: AIX needs an explicit #include to get alloca() In-Reply-To: References: <3bZ3HS2pvhzQxV@mail.python.org> <51C0444D.8030402@python.org> Message-ID: <51C04841.8080302@python.org> Am 18.06.2013 13:32, schrieb Victor Stinner: > 2013/6/18 Christian Heimes : >> Am 18.06.2013 12:56, schrieb Jeremy Kloth: >>> On Mon, Jun 17, 2013 at 2:02 PM, victor.stinner >>>> +#include >>> >>> This header is not present on Windows, thus breaking all the Windows >>> buildbots. Perhaps it should be wrapped in an AIX-specific #ifdef? > > Oh really? Portability is complex :-) I only tested on Linux, but I > expected this header to be part of the C standard... It's neither C99 nor POSIX.1-2001. I guess it's just too fragile and not portable enough. http://c-faq.com/malloc/alloca.html > >> I have added HAVE_ALLOCA_H to configure: >> >> http://hg.python.org/cpython/rev/7b6ae19dd116 > > Cool, thanks. You are welcome. alloca() is documented to require . It merely works with GCC because the compiler translates the function call to inline code. Christian From victor.stinner at gmail.com Tue Jun 18 22:40:49 2013 From: victor.stinner at gmail.com (Victor Stinner) Date: Tue, 18 Jun 2013 22:40:49 +0200 Subject: [Python-Dev] RFC: PEP 445: Add new APIs to customize Python memory allocators Message-ID: If you prefer the HTML version: http://www.python.org/dev/peps/pep-0445/ PEP: 445 Title: Add new APIs to customize Python memory allocators Version: $Revision$ Last-Modified: $Date$ Author: Victor Stinner Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 15-june-2013 Python-Version: 3.4 Abstract ======== Add new APIs to customize Python memory allocators. Rationale ========= Use cases: * Application embedding Python may want to isolate Python memory from the memory of the application, or may want to use a different memory allocator optimized for its Python usage * Python running on embedded devices with low memory and slow CPU. A custom memory allocator may be required to use efficiently the memory and/or to be able to use all the memory of the device. * Debug tool to: - track the memory usage (memory leaks) - get the Python filename and line number where an object was allocated - detect buffer underflow, buffer overflow and detect misuse of Python allocator APIs (builtin Python debug hooks) - force allocation to fail to test handling of ``MemoryError`` exception Proposal ======== API changes ----------- * Add new GIL-free (no need to hold the GIL) memory allocator functions: - ``void* PyMem_RawMalloc(size_t size)`` - ``void* PyMem_RawRealloc(void *ptr, size_t new_size)`` - ``void PyMem_RawFree(void *ptr)`` - the behaviour of requesting zero bytes is not defined: return *NULL* or a distinct non-*NULL* pointer depending on the platform. * Add a new ``PyMemBlockAllocator`` structure:: typedef struct { /* user context passed as the first argument to the 3 functions */ void *ctx; /* allocate a memory block */ void* (*malloc) (void *ctx, size_t size); /* allocate or resize a memory block */ void* (*realloc) (void *ctx, void *ptr, size_t new_size); /* release a memory block */ void (*free) (void *ctx, void *ptr); } PyMemBlockAllocator; * Add new functions to get and set internal functions of ``PyMem_RawMalloc()``, ``PyMem_RawRealloc()`` and ``PyMem_RawFree()``: - ``void PyMem_GetRawAllocator(PyMemBlockAllocator *allocator)`` - ``void PyMem_SetRawAllocator(PyMemBlockAllocator *allocator)`` - default allocator: ``malloc()``, ``realloc()``, ``free()`` * Add new functions to get and set internal functions of ``PyMem_Malloc()``, ``PyMem_Realloc()`` and ``PyMem_Free()``: - ``void PyMem_GetAllocator(PyMemBlockAllocator *allocator)`` - ``void PyMem_SetAllocator(PyMemBlockAllocator *allocator)`` - ``malloc(ctx, 0)`` and ``realloc(ctx, ptr, 0)`` must not return *NULL*: it would be treated as an error. - default allocator: ``malloc()``, ``realloc()``, ``free()``; ``PyMem_Malloc(0)`` calls ``malloc(1)`` and ``PyMem_Realloc(NULL, 0)`` calls ``realloc(NULL, 1)`` * Add new functions to get and set internal functions of ``PyObject_Malloc()``, ``PyObject_Realloc()`` and ``PyObject_Free()``: - ``void PyObject_GetAllocator(PyMemBlockAllocator *allocator)`` - ``void PyObject_SetAllocator(PyMemBlockAllocator *allocator)`` - ``malloc(ctx, 0)`` and ``realloc(ctx, ptr, 0)`` must not return *NULL*: it would be treated as an error. - default allocator: the *pymalloc* allocator * Add a new ``PyMemMappingAllocator`` structure:: typedef struct { /* user context passed as the first argument to the 2 functions */ void *ctx; /* allocate a memory mapping */ void* (*alloc) (void *ctx, size_t size); /* release a memory mapping */ void (*free) (void *ctx, void *ptr, size_t size); } PyMemMappingAllocator; * Add a new function to get and set the memory mapping allocator: - ``void PyMem_GetMappingAllocator(PyMemMappingAllocator *allocator)`` - ``void PyMem_SetMappingAllocator(PyMemMappingAllocator *allocator)`` - Currently, this allocator is only used internally by *pymalloc* to allocate arenas. * Add a new function to setup the builtin Python debug hooks when memory allocators are replaced: - ``void PyMem_SetupDebugHooks(void)`` - the function does nothing is Python is compiled not compiled in debug mode * The following memory allocators always returns *NULL* if size is greater than ``PY_SSIZE_T_MAX`` (check before calling the internal function): ``PyMem_RawMalloc()``, ``PyMem_RawRealloc()``, ``PyMem_Malloc()``, ``PyMem_Realloc()``, ``PyObject_Malloc()``, ``PyObject_Realloc()``. The builtin Python debug hooks were introduced in Python 2.3 and implement the following checks: * Newly allocated memory is filled with the byte ``0xCB``, freed memory is filled with the byte ``0xDB``. * Detect API violations, ex: ``PyObject_Free()`` called on a memory block allocated by ``PyMem_Malloc()`` * Detect write before the start of the buffer (buffer underflow) * Detect write after the end of the buffer (buffer overflow) Other changes ------------- * ``PyMem_Malloc()`` and ``PyMem_Realloc()`` always call ``malloc()`` and ``realloc()``, instead of calling ``PyObject_Malloc()`` and ``PyObject_Realloc()`` in debug mode * ``PyObject_Malloc()`` falls back on ``PyMem_Malloc()`` instead of ``malloc()`` if size is greater or equal than ``SMALL_REQUEST_THRESHOLD`` (512 bytes), and ``PyObject_Realloc()`` falls back on ``PyMem_Realloc()`` instead of ``realloc()`` * Replace direct calls to ``malloc()`` with ``PyMem_Malloc()``, or ``PyMem_RawMalloc()`` if the GIL is not held * Configure external libraries like zlib or OpenSSL to allocate memory using ``PyMem_RawMalloc()`` Examples ======== Use case 1: Replace Memory Allocator, keep pymalloc ---------------------------------------------------- Dummy example wasting 2 bytes per memory block, and 10 bytes per memory mapping:: #include int block_padding = 2; int mapping_padding = 10; void* my_malloc(void *ctx, size_t size) { int padding = *(int *)ctx; return malloc(size + padding); } void* my_realloc(void *ctx, void *ptr, size_t new_size) { int padding = *(int *)ctx; return realloc(ptr, new_size + padding); } void my_free(void *ctx, void *ptr) { free(ptr); } void* my_alloc_mapping(void *ctx, size_t size) { int padding = *(int *)ctx; return malloc(size + padding); } void my_free_mapping(void *ctx, void *ptr, size_t size) { free(ptr); } void setup_custom_allocator(void) { PyMemBlockAllocator block; PyMemMappingAllocator mapping; block.ctx = &block_padding; block.malloc = my_malloc; block.realloc = my_realloc; block.free = my_free; PyMem_SetRawAllocator(&block); PyMem_SetAllocator(&block); mapping.ctx = &mapping_padding; mapping.alloc = my_alloc_mapping; mapping.free = my_free_mapping; PyMem_SetMappingAllocator(mapping); PyMem_SetupDebugHooks(); } .. warning:: Remove the call ``PyMem_SetRawAllocator(&alloc)`` if the new allocator are not thread-safe. Use case 2: Replace Memory Allocator, override pymalloc -------------------------------------------------------- If your allocator is optimized for allocation of small objects (less than 512 bytes) with a short lifetime, pymalloc can be overriden (replace ``PyObject_Malloc()``). Dummy example wasting 2 bytes per memory block:: #include int padding = 2; void* my_malloc(void *ctx, size_t size) { int padding = *(int *)ctx; return malloc(size + padding); } void* my_realloc(void *ctx, void *ptr, size_t new_size) { int padding = *(int *)ctx; return realloc(ptr, new_size + padding); } void my_free(void *ctx, void *ptr) { free(ptr); } void setup_custom_allocator(void) { PyMemBlockAllocator alloc; alloc.ctx = &padding; alloc.malloc = my_malloc; alloc.realloc = my_realloc; alloc.free = my_free; PyMem_SetRawAllocator(&alloc); PyMem_SetAllocator(&alloc); PyObject_SetAllocator(&alloc); PyMem_SetupDebugHooks(); } .. warning:: Remove the call ``PyMem_SetRawAllocator(&alloc)`` if the new allocator are not thread-safe. Use case 3: Setup Allocator Hooks --------------------------------- Example to setup hooks on all memory allocators:: struct { PyMemBlockAllocator raw; PyMemBlockAllocator mem; PyMemBlockAllocator obj; /* ... */ } hook; static void* hook_malloc(void *ctx, size_t size) { PyMemBlockAllocator *alloc = (PyMemBlockAllocator *)ctx; /* ... */ ptr = alloc->malloc(alloc->ctx, size); /* ... */ return ptr; } static void* hook_realloc(void *ctx, void *ptr, size_t new_size) { PyMemBlockAllocator *alloc = (PyMemBlockAllocator *)ctx; void *ptr2; /* ... */ ptr2 = alloc->realloc(alloc->ctx, ptr, new_size); /* ... */ return ptr2; } static void hook_free(void *ctx, void *ptr) { PyMemBlockAllocator *alloc = (PyMemBlockAllocator *)ctx; /* ... */ alloc->free(alloc->ctx, ptr); /* ... */ } void setup_hooks(void) { PyMemBlockAllocator alloc; static int installed = 0; if (installed) return; installed = 1; alloc.malloc = hook_malloc; alloc.realloc = hook_realloc; alloc.free = hook_free; PyMem_GetRawAllocator(&hook.raw); alloc.ctx = &hook.raw; PyMem_SetRawAllocator(&alloc); PyMem_GetAllocator(&hook.mem); alloc.ctx = &hook.mem; PyMem_SetAllocator(&alloc); PyObject_GetAllocator(&hook.obj); alloc.ctx = &hook.obj; PyObject_SetAllocator(&alloc); } .. warning:: Remove the call ``PyMem_SetRawAllocator(&alloc)`` if hooks are not thread-safe. .. note:: ``PyMem_SetupDebugHooks()`` does not need to be called: Python debug hooks are installed automatically at startup. Performances ============ Results of the `Python benchmarks suite `_ (-b 2n3): some tests are 1.04x faster, some tests are 1.04 slower, significant is between 115 and -191. Results of pybench benchmark: "+0.1%" slower globally (diff between -4.9% and +5.6%). The full reports are attached to the issue #3329. Alternatives ============ Only one get/set function for block allocators ---------------------------------------------- Replace the 6 functions: * ``void PyMem_GetRawAllocator(PyMemBlockAllocator *allocator)`` * ``void PyMem_GetAllocator(PyMemBlockAllocator *allocator)`` * ``void PyObject_GetAllocator(PyMemBlockAllocator *allocator)`` * ``void PyMem_SetRawAllocator(PyMemBlockAllocator *allocator)`` * ``void PyMem_SetAllocator(PyMemBlockAllocator *allocator)`` * ``void PyObject_SetAllocator(PyMemBlockAllocator *allocator)`` with 2 functions with an additional *domain* argument: * ``int PyMem_GetBlockAllocator(int domain, PyMemBlockAllocator *allocator)`` * ``int PyMem_SetBlockAllocator(int domain, PyMemBlockAllocator *allocator)`` These functions return 0 on success, or -1 if the domain is unknown. where domain is one of these values: * ``PYALLOC_PYMEM`` * ``PYALLOC_PYMEM_RAW`` * ``PYALLOC_PYOBJECT`` Drawback: the caller has to check if the result is 0, or handle the error. Make PyMem_Malloc() reuse PyMem_RawMalloc() by default ------------------------------------------------------ ``PyMem_Malloc()`` should call ``PyMem_RawMalloc()`` by default. So calling ``PyMem_SetRawAllocator()`` would also also patch ``PyMem_Malloc()`` indirectly. .. note:: In the implementation of this PEP (issue #3329), ``PyMem_RawMalloc(0)`` calls ``malloc(0)``, whereas ``PyMem_Malloc(0)`` calls ``malloc(1)``. Add a new PYDEBUGMALLOC environment variable -------------------------------------------- To be able to use the Python builtin debug hooks even when a custom memory allocator replaces the default Python allocator, an environment variable ``PYDEBUGMALLOC`` can be added to setup these debug function hooks, instead of adding the new function ``PyMem_SetupDebugHooks()``. If the environment variable is present, ``PyMem_SetRawAllocator()``, ``PyMem_SetAllocator()`` and ``PyObject_SetAllocator()`` will reinstall automatically the hook on top of the new allocator. An new environment variable would make the Python initialization even more complex. The `PEP 432 `_ tries to simply the CPython startup sequence. Use macros to get customizable allocators ----------------------------------------- To have no overhead in the default configuration, customizable allocators would be an optional feature enabled by a configuration option or by macros. Not having to recompile Python makes debug hooks easier to use in practice. Extensions modules don't have to be recompiled with macros. Pass the C filename and line number ----------------------------------- Define allocator functions as macros using ``__FILE__`` and ``__LINE__`` to get the C filename and line number of a memory allocation. Example of ``PyMem_Malloc`` macro with the modified ``PyMemBlockAllocator`` structure:: typedef struct { /* user context passed as the first argument to the 3 functions */ void *ctx; /* allocate a memory block */ void* (*malloc) (void *ctx, const char *filename, int lineno, size_t size); /* allocate or resize a memory block */ void* (*realloc) (void *ctx, const char *filename, int lineno, void *ptr, size_t new_size); /* release a memory block */ void (*free) (void *ctx, const char *filename, int lineno, void *ptr); } PyMemBlockAllocator; void* _PyMem_MallocTrace(const char *filename, int lineno, size_t size); /* need also a function for the Python stable ABI */ void* PyMem_Malloc(size_t size); #define PyMem_Malloc(size) \ _PyMem_MallocTrace(__FILE__, __LINE__, size) Passing a filename and a line number to each allocator makes the API more complex: pass 3 new arguments, instead of just a context argument, to each allocator function. The GC allocator functions should also be patched. For example, ``_PyObject_GC_Malloc()`` is used in many C functions and so objects of differenet types would have the same allocation location. Such changes add too much complexity for a little gain. GIL-free PyMem_Malloc() ----------------------- When Python is compiled in debug mode, ``PyMem_Malloc()`` calls indirectly ``PyObject_Malloc()`` which requires the GIL to be held. That's why ``PyMem_Malloc()`` must be called with the GIL held. This PEP proposes to "fix" ``PyMem_Malloc()`` to make it always call ``malloc()``. So the "GIL must be held" restriction may be removed from ``PyMem_Malloc()``. Allowing to call ``PyMem_Malloc()`` without holding the GIL might break applications which setup their own allocators or allocator hooks. Holding the GIL is convinient to develop a custom allocator: no need to care of other threads. It is also convinient for a debug allocator hook: Python internal objects can be safetly inspected. Calling ``PyGILState_Ensure()`` in a memory allocator may have unexpected behaviour, especially at Python startup and at creation of a new Python thread state. Don't add PyMem_RawMalloc() --------------------------- Replace ``malloc()`` with ``PyMem_Malloc()``, but only if the GIL is held. Otherwise, keep ``malloc()`` unchanged. The ``PyMem_Malloc()`` is used without the GIL held in some Python functions. For example, the ``main()`` and ``Py_Main()`` functions of Python call ``PyMem_Malloc()`` whereas the GIL do not exist yet. In this case, ``PyMem_Malloc()`` should be replaced with ``malloc()`` (or ``PyMem_RawMalloc()``). If an hook is used to the track memory usage, the ``malloc()`` memory will not be seen. Remaining ``malloc()`` may allocate a lot of memory and so would be missed in reports. Use existing debug tools to analyze the memory ---------------------------------------------- There are many existing debug tools to analyze the memory. Some examples: `Valgrind `_, `Purify `_, `Clang AddressSanitizer `_, `failmalloc `_, etc. The problem is to retrieve the Python object related to a memory pointer to read its type and/or content. Another issue is to retrieve the location of the memory allocation: the C backtrace is usually useless (same reasoning than macros using ``__FILE__`` and ``__LINE__``), the Python filename and line number (or even the Python traceback) is more useful. Classic tools are unable to introspect Python internals to collect such information. Being able to setup a hook on allocators called with the GIL held allow to collect a lot of useful data from Python internals. Add msize() ----------- Add another field to ``PyMemBlockAllocator`` and ``PyMemMappingAllocator``:: size_t msize(void *ptr); This function returns the size of a memory block or a memory mapping. Return (size_t)-1 if the function is not implemented or if the pointer is unknown (ex: NULL pointer). On Windows, this function can be implemented using ``_msize()`` and ``VirtualQuery()``. No context argument ------------------- Simplify the signature of allocator functions, remove the context argument: * ``void* malloc(size_t size)`` * ``void* realloc(void *ptr, size_t new_size)`` * ``void free(void *ptr)`` It is likely for an allocator hook to be reused for ``PyMem_SetAllocator()`` and ``PyObject_SetAllocator()``, or even ``PyMem_SetRawAllocator()``, but the hook must call a different function depending on the allocator. The context is a convenient way to reuse the same custom allocator or hook for different Python allocators. External libraries ================== Python should try to reuse the same prototypes for allocator functions than other libraries. Libraries used by Python: * OpenSSL: `CRYPTO_set_mem_functions() `_ to set memory management functions globally * expat: `parserCreate() `_ has a per-instance memory handler Other libraries: * glib: `g_mem_set_vtable() `_ * libxml2: `xmlGcMemSetup() `_, global See also the `GNU libc: Memory Allocation Hooks `_. Memory allocators ================= The C standard library provides the well known ``malloc()`` function. Its implementation depends on the platform and of the C library. The GNU C library uses a modified ptmalloc2, based on "Doug Lea's Malloc" (dlmalloc). FreeBSD uses `jemalloc `_. Google provides tcmalloc which is part of `gperftools `_. ``malloc()`` uses two kinds of memory: heap and memory mappings. Memory mappings are usually used for large allocations (ex: larger than 256 KB), whereas the heap is used for small allocations. On UNIX, the heap is handled by ``brk()`` and ``sbrk()`` system calls on Linux, and it is contiguous. On Windows, the heap is handled by ``HeapAlloc()`` and may be discontiguous. Memory mappings are handled by ``mmap()`` on UNIX and ``VirtualAlloc()`` on Windows, they may be discontiguous. Releasing a memory mapping gives back immediatly the memory to the system. On UNIX, heap memory is only given back to the system if it is at the end of the heap. Otherwise, the memory will only be given back to the system when all the memory located after the released memory are also released. To allocate memory in the heap, the allocator tries to reuse free space. If there is no contiguous space big enough, the heap must be increased, even if we have more free space than required size. This issue is called the "memory fragmentation": the memory usage seen by the system may be much higher than real usage. On Windows, ``HeapAlloc()`` creates a new memory mapping with ``VirtualAlloc()`` if there is not enough free contiguous memory. CPython has a *pymalloc* allocator for allocations smaller than 512 bytes. This allocator is optimized for small objects with a short lifetime. It uses memory mappings called "arenas" with a fixed size of 256 KB. Other allocators: * Windows provides a `Low-fragmentation Heap `_. * The Linux kernel uses `slab allocation `_. * The glib library has a `Memory Slice API `_: efficient way to allocate groups of equal-sized chunks of memory Links ===== CPython issues related to memory allocation: * `Issue #3329: Add new APIs to customize memory allocators `_ * `Issue #13483: Use VirtualAlloc to allocate memory arenas `_ * `Issue #16742: PyOS_Readline drops GIL and calls PyOS_StdioReadline, which isn't thread safe `_ * `Issue #18203: Replace calls to malloc() with PyMem_Malloc() or PyMem_RawMalloc() `_ * `Issue #18227: Use Python memory allocators in external libraries like zlib or OpenSSL `_ Projects analyzing the memory usage of Python applications: * `pytracemalloc `_ * `Meliae: Python Memory Usage Analyzer `_ * `Guppy-PE: umbrella package combining Heapy and GSL `_ * `PySizer (developed for Python 2.4) `_ From victor.stinner at gmail.com Tue Jun 18 23:20:04 2013 From: victor.stinner at gmail.com (Victor Stinner) Date: Tue, 18 Jun 2013 23:20:04 +0200 Subject: [Python-Dev] RFC: PEP 445: Add new APIs to customize Python memory allocators In-Reply-To: References: Message-ID: typedef struct { /* user context passed as the first argument to the 2 functions */ void *ctx; /* allocate a memory mapping */ void* (*alloc) (void *ctx, size_t size); /* release a memory mapping */ void (*free) (void *ctx, void *ptr, size_t size); } PyMemMappingAllocator; The PyMemMappingAllocator structure is very specific to the pymalloc allocator. There is no "resize", "lock" nor "protect" method. There is no way to configure protection or flags of the mapping. The PyMem_SetMappingAllocator() function was initially called _PyObject_SetArenaAllocator(). I'm not sure that the structure and the 2 related functions should be public. Can an extension module call private (_Py*) functions or use a private structure? Or the structure might be renamed to indicate that it is specific to arenas? What do you think? Victor From scott+python-dev at scottdial.com Wed Jun 19 01:23:55 2013 From: scott+python-dev at scottdial.com (Scott Dial) Date: Tue, 18 Jun 2013 19:23:55 -0400 Subject: [Python-Dev] RFC: PEP 445: Add new APIs to customize Python memory allocators In-Reply-To: References: Message-ID: <51C0EC0B.5040704@scottdial.com> On 6/18/2013 4:40 PM, Victor Stinner wrote: > No context argument > ------------------- > > Simplify the signature of allocator functions, remove the context > argument: > > * ``void* malloc(size_t size)`` > * ``void* realloc(void *ptr, size_t new_size)`` > * ``void free(void *ptr)`` > > It is likely for an allocator hook to be reused for > ``PyMem_SetAllocator()`` and ``PyObject_SetAllocator()``, or even > ``PyMem_SetRawAllocator()``, but the hook must call a different function > depending on the allocator. The context is a convenient way to reuse the > same custom allocator or hook for different Python allocators. I think there is a lack of justification for the extra argument, and the extra argument is not free. The typical use-case for doing this continuation-passing style is when the set of contexts is either unknown, arbitrarily large, or infinite. In other words, when it would be either impossible or impractical to enumerate all of the contexts. However, in this case, we have only 3. Your proposal already puts forward having 3 pairs of Get/Set functions, so there is no distinct advantage in having a single typedef instance that you pass in to all 3 of them. And, having all 3 pairs use the same typedef is a bit of an attractive nuisance, in that one could pass the wrong allocators to the wrong setter. With that, I could argue that there should be 3 typedefs to prevent coding errors. Nevertheless, the ctx argument buys the implementer nothing if they have to begin their alloc function with "if(ctx == X)". In other words, there is nothing simpler about: """ void *_alloc(void *ctx, size_t size) { if(ctx == PYALLOC_PYMEM) return _alloc_pymem(size); else if(ctx == PYALLOC_PYMEM_RAW) return _alloc_pymem_raw(size); else if(ctx == PYALLOC_PYOBJECT) return _alloc_pyobject(size); else abort(); } PyMemBlockAllocator pymem_allocator = {.ctx=PYALLOC_PYMEM, .alloc=&_alloc, .free=&_free}; PyMemBlockAllocator pymem_raw_allocator = {.ctx=PYALLOC_PYMEM_RAW, .alloc=&_alloc, .free=&_free}; PyMemBlockAllocator pyobject_allocator = {.ctx=PYALLOC_PYOBJECT, .alloc=&_alloc, .free=&_free}; """ In comparison to: """ PyMemBlockAllocator pymem_allocator = {.alloc=&_alloc_pymem, .free=&_free_pymem}; PyMemBlockAllocator pymem_raw_allocator = {.alloc=&_alloc_pymem_raw, .free=&_free_pymem}; PyMemBlockAllocator pyobject_allocator = {.alloc=&_alloc_pyobject, .free=&_free_pyobject}; """ And in the latter case, there is no extra indirect branching in the hot-path of the allocators. Also, none of the external libraries cited introduce this CPS/ctx stuff. -- Scott Dial scott at scottdial.com From barry at python.org Wed Jun 19 05:04:03 2013 From: barry at python.org (Barry Warsaw) Date: Tue, 18 Jun 2013 23:04:03 -0400 Subject: [Python-Dev] backported Enum In-Reply-To: References: <51BCC498.1030100@stoneleaf.us> <7w8v2aepsp.fsf@benfinney.id.au> <7wd2rmeh5q.fsf@benfinney.id.au> <51BD60DC.40701@stoneleaf.us> <20130616111912.1a59b0fe@anarchist> Message-ID: <20130618230403.4cf9e657@anarchist> On Jun 16, 2013, at 01:13 PM, Eli Bendersky wrote: >If you write down the process of porting mailmain from flufl to stdlib.enum >in some place, it can make the process much easier for others, and even >encourage them to follow the same path. Let's write it down here! It was mostly mechanical, and some probably would have been done to fix deprecations in flufl.enum. Here's the list. Switching from call syntax to getitem syntax for looking up an enum member by name, e.g. - delivery_mode = DeliveryMode(data['delivery_mode']) + delivery_mode = DeliveryMode[data['delivery_mode']] Switching from getitem syntax to call syntax for looking up an enum member by value, e.g. - return self._enum[value] + return self._enum(value) Interesting that these two were exactly opposite from flufl.enum. Switching from int() to .value to get the integer value of an enum member, e.g. - return (member.list_id, member.address.email, int(member.role)) + return (member.list_id, member.address.email, member.role.value) Changing the imports (obviously), e.g. -from flufl.enum import Enum +from enum import Enum Adapting to the different repr as seen in a doctest, e.g. - + Fixing some type tests. I have one case that was testing for an enum member to do value type conversion. Since enum members are now instances of their enum class, and because members no longer have a .enum attribute (because you can just ask for its __class__), the type test actually became simpler: - elif hasattr(obj, 'enum') and issubclass(obj.enum, Enum): + elif isinstance(obj, Enum): An unexpected difference is that failing name lookups raise a KeyError instead of a ValueError. There are many cases where I catch failing lookups in my REST API. Fixing this was mostly easy... - except ValueError: + except KeyError: ...except in one case where I was previously allowing the ValueError to percolate up to signal to a higher level that some exception handling had to be done. Here, I had to convert the exception: - # This will raise a ValueError if the enum value is unknown. Let - # that percolate up. - return self._enum_class[enum_value] + # This will raise a KeyError if the enum value is unknown. The + # Validator API requires turning this into a ValueError. + try: + return self._enum_class[enum_value] + except KeyError as exception: + # Retain the error message. + raise ValueError(exception.message) Interestingly, I found an existing hidden bug in an upgrade script where I was not converting a value from an integer to an enum. Nice to catch that one. For now, I'll hold out on merging this branch to trunk until I can get enum34 packaged up for Debian. Other than that, it looks pretty good! Merge proposal: http://tinyurl.com/l2fq38l Branch: https://code.launchpad.net/~barry/mailman/pep435 -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From ncoghlan at gmail.com Wed Jun 19 05:32:01 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 19 Jun 2013 13:32:01 +1000 Subject: [Python-Dev] RFC: PEP 445: Add new APIs to customize Python memory allocators In-Reply-To: <51C0EC0B.5040704@scottdial.com> References: <51C0EC0B.5040704@scottdial.com> Message-ID: On 19 June 2013 09:23, Scott Dial wrote: > On 6/18/2013 4:40 PM, Victor Stinner wrote: >> No context argument >> ------------------- >> >> Simplify the signature of allocator functions, remove the context >> argument: >> >> * ``void* malloc(size_t size)`` >> * ``void* realloc(void *ptr, size_t new_size)`` >> * ``void free(void *ptr)`` >> >> It is likely for an allocator hook to be reused for >> ``PyMem_SetAllocator()`` and ``PyObject_SetAllocator()``, or even >> ``PyMem_SetRawAllocator()``, but the hook must call a different function >> depending on the allocator. The context is a convenient way to reuse the >> same custom allocator or hook for different Python allocators. > > I think there is a lack of justification for the extra argument, and the > extra argument is not free. The typical use-case for doing this > continuation-passing style is when the set of contexts is either > unknown, arbitrarily large, or infinite. In other words, when it would > be either impossible or impractical to enumerate all of the contexts. > However, in this case, we have only 3. Note that the context is part of the BlockAllocator structure, NOT predefined by Python. > Your proposal already puts forward having 3 pairs of Get/Set functions, > so there is no distinct advantage in having a single typedef instance > that you pass in to all 3 of them. And, having all 3 pairs use the same > typedef is a bit of an attractive nuisance, in that one could pass the > wrong allocators to the wrong setter. With that, I could argue that > there should be 3 typedefs to prevent coding errors. I'm not sure we *should* be restricting this to the CPython internal domains indefinitely. If we use a domain based model from the start, then that will allow us in the future to let extension modules declare additional domains rather than having to employ library specific logic in either the CPython core or in embedding applications. > Nevertheless, the ctx argument buys the implementer nothing if they have > to begin their alloc function with "if(ctx == X)". In other words, there > is nothing simpler about: > > """ > void *_alloc(void *ctx, size_t size) { > if(ctx == PYALLOC_PYMEM) > return _alloc_pymem(size); > else if(ctx == PYALLOC_PYMEM_RAW) > return _alloc_pymem_raw(size); > else if(ctx == PYALLOC_PYOBJECT) > return _alloc_pyobject(size); > else > abort(); > } > > PyMemBlockAllocator pymem_allocator = > {.ctx=PYALLOC_PYMEM, .alloc=&_alloc, .free=&_free}; > PyMemBlockAllocator pymem_raw_allocator = > {.ctx=PYALLOC_PYMEM_RAW, .alloc=&_alloc, .free=&_free}; > PyMemBlockAllocator pyobject_allocator = > {.ctx=PYALLOC_PYOBJECT, .alloc=&_alloc, .free=&_free}; > """ Why would anyone do that? The context is so embedding applications can distinguish the CPython runtime from their *other* domains that use the same allocator functions. If you wanted to use completely different allocators for each domain, you would just do that and ignore the context argument entirely. Agreed more of that rationale needs to be moved from the issue tracker into the PEP, though. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Wed Jun 19 05:37:27 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 19 Jun 2013 13:37:27 +1000 Subject: [Python-Dev] backported Enum In-Reply-To: <51BCC498.1030100@stoneleaf.us> References: <51BCC498.1030100@stoneleaf.us> Message-ID: On 16 June 2013 05:46, Ethan Furman wrote: > So I have the stdlb 3.4 Enum backported for both earlier 3.x and back to 2.4 > in the 2.x series. We should resolve http://bugs.python.org/issue17961 (switching the functional creation API to use strings instead of 1-based integers) before we get too gung ho about encouraging people to use the backport. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From scott+python-dev at scottdial.com Wed Jun 19 06:33:54 2013 From: scott+python-dev at scottdial.com (Scott Dial) Date: Wed, 19 Jun 2013 00:33:54 -0400 Subject: [Python-Dev] RFC: PEP 445: Add new APIs to customize Python memory allocators In-Reply-To: References: <51C0EC0B.5040704@scottdial.com> Message-ID: <51C134B2.8040408@scottdial.com> On 6/18/2013 11:32 PM, Nick Coghlan wrote: > Agreed more of that rationale needs to be moved from the issue tracker > into the PEP, though. Thanks for the clarification. I hadn't read the issue tracker at all. On it's face value, I didn't see what purpose it served, but having read Kristj?n's comments on the issue tracker, he would like to store state for the allocators in that ctx pointer.[1] Having read that (previously, I thought the only utility was distinguishing which domain it was -- a small, glorified enumeration), but his use-case makes sense and definitely is informative to have in the PEP, because the utility of that wasn't obvious to me. Thanks, -Scott [1] http://bugs.python.org/issue3329#msg190529 """ One particular trick we have been using, which might be of interest, is to be able to tag each allocation with a "context" id. This is then set according to a global sys.memcontext variable, which the program will modify according to what it is doing. This can then be used to track memory usage by different parts of the program. """ -- Scott Dial scott at scottdial.com From victor.stinner at gmail.com Wed Jun 19 07:42:08 2013 From: victor.stinner at gmail.com (Victor Stinner) Date: Wed, 19 Jun 2013 07:42:08 +0200 Subject: [Python-Dev] RFC: PEP 445: Add new APIs to customize Python memory allocators In-Reply-To: <51C0EC0B.5040704@scottdial.com> References: <51C0EC0B.5040704@scottdial.com> Message-ID: Le mercredi 19 juin 2013, Scott Dial a ?crit : > On 6/18/2013 4:40 PM, Victor Stinner wrote: > > No context argument > > I think there is a lack of justification for the extra argument, and the > extra argument is not free. The typical use-case for doing this > continuation-passing style is when the set of contexts is either > unknown, arbitrarily large, or infinite. In other words, when it would > be either impossible or impractical to enumerate all of the contexts. > However, in this case, we have only 3. See the use case 3 in examples. Without the context argument, you have to copy/paste 3 times each functions: 3 functions -> 9 functions. I don't like having to copy/paste code, it sounds like a bad design. > And in the latter case, there is no extra indirect branching in the > hot-path of the allocators. Are you concerned by performances? Did you see the Performances section, there is no overhead according to the benchmark suite. > > Also, none of the external libraries cited introduce this CPS/ctx stuff. Oops, the list is incomplete. Copy/paste from the issue: Some customizable memory allocators I know have an extra parameter "void *opaque" that is passed to all functions: - in zlib: zalloc and zfree: http://www.zlib.net/manual.html#Usage - same thing for bz2. - lzma's ISzAlloc: http://www.asawicki.info/news_1368_lzma_sdk_-_how_to_use.html - Oracle's OCI: http://docs.oracle.com/cd/B10501_01/appdev.920/a96584/oci15re4.htm OTOH, expat, libxml, libmpdec don't have this extra parameter. Victor -------------- next part -------------- An HTML attachment was scrubbed... URL: From trent at snakebite.org Wed Jun 19 15:10:02 2013 From: trent at snakebite.org (Trent Nelson) Date: Wed, 19 Jun 2013 21:10:02 +0800 Subject: [Python-Dev] pyparallel and new memory API discussions... Message-ID: <20130619131001.GB10039@snakebite.org> The new memory API discussions (and PEP) warrant a quick pyparallel update: a couple of weeks after PyCon, I came up with a solution for the biggest show-stopper that has been plaguing pyparallel since its inception: being able to detect the modification of "main thread" Python objects from within a parallel context. For example, `data.append(4)` in the example below will generate an AssignmentError exception, because data is a main thread object, and `data.append(4)` gets executed from within a parallel context:: data = [ 1, 2, 3 ] def work(): data.append(4) async.submit_work(work) The solution turned out to be deceptively simple: 1. Prior to running parallel threads, lock all "main thread" memory pages as read-only (via VirtualProtect on Windows, mprotect on POSIX). 2. Detect attempts to write to main thread pages during parallel thread execution (via SEH on Windows or a SIGSEGV trap on POSIX), and raise an exception instead (detection is done in the ceval frame exec loop). 3. Prior to returning control back to the main thread (which will be paused whilst all the parallel threads are running), unlock all the "main thread" pages. 4. Pause all parallel threads while the main thread runs. 5. Go back to 1. I got a proof-of-concept working on Windows a while back (and also played around with large page support in the same commit). The main changes were to obmalloc.c: https://bitbucket.org/tpn/pyparallel/commits/0e70a0caa1c07dc0c14bb5c99cbe808c1c11779f#chg-Objects/obmalloc.c The key was the introduction of two new API calls, intended to be called by the pyparallel.c infrastructure: _PyMem_LockMainThreadPages() _PyMem_UnlockMainThreadPages() The implementation is pretty simple: +int +_PyMem_LockMainThreadPages(void) +{ + DWORD old = 0; + + if (!VirtualProtect(base_addr, nbytes_committed, PAGE_READONLY, &old)) { + PyErr_SetFromWindowsErr(0); + return -1; + } Note the `base_addr` and `nbytes_committed` argument. Basically, I re-organized obmalloc.c a little bit such that we never actually call malloc() directly. Instead, we exploit the ability to reserve huge virtual address ranges without actually committing the memory, giving us a fixed `base_addr` void pointer that we can pass to calls like VirtualProtect or mprotect. We then incrementally commit more pages as demand increases, and simply adjust our `nbytes_committed` counter as we go along. The net effect is that we can call VirtualProtect/mprotect once, with a single base void pointer and size_t range, and immediately affect the protection of all memory pages that fall within that range. As an added bonus, we also get a very cheap and elegant way to test if a pointer (or any arbitrary memory address, actually) belongs to the main thread's memory range (at least in comparison to the existing _PyMem_InRange black magic). (This is very useful for my pyparallel infrastructure, which makes extensive use of conditional logic based on address tests.) (Side-bar: a side-effect of the approach I've used in the proof- of-concept (by only having a single base addr pointer) is that we effectively limit the maximum memory we could eventually commit. I actually quite like this -- in fact, I'd like to tweak it such that we can actually expose min/max memory values to the Python interpreter at startup (analogous to the JVM). Having known upper bounds on maximum memory usage will vastly simplify some other areas of my pyparallel work (like the async socket stuff). For example, consider network programs these days that take a "max clients" configuration parameter. That seems a bit backwards to me. It would be better if we simply said, "here, Python, you have 1GB to work with". That allows us to calculate how many clients we could simultaneously serve based on socket memory requirements, which allows for much more graceful behavior under load than leaving it open-ended. Maximum memory constraints would also be useful for the parallel.map(callable, iterable) stuff I've got in the works, as it'll allow us to optimally chunk work and assign to threads based on available memory.) So, Victor, I'm interested to hear how the new API you're proposing will affect this solution I've come up with for pyparallel; I'm going to be absolutely dependent upon the ability to lock main thread pages as read-only in one fell-swoop -- am I still going to be able to do that with your new API in place? Regards, Trent. From ncoghlan at gmail.com Wed Jun 19 15:33:02 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 19 Jun 2013 23:33:02 +1000 Subject: [Python-Dev] pyparallel and new memory API discussions... In-Reply-To: <20130619131001.GB10039@snakebite.org> References: <20130619131001.GB10039@snakebite.org> Message-ID: On 19 June 2013 23:10, Trent Nelson wrote: > So, Victor, I'm interested to hear how the new API you're proposing > will affect this solution I've come up with for pyparallel; I'm > going to be absolutely dependent upon the ability to lock main > thread pages as read-only in one fell-swoop -- am I still going to > be able to do that with your new API in place? By default, nothing will change for the ordinary CPython runtime. It's only if an embedding application starts messing with the allocators that things might change, but at that point, pyparallel would break anyway. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From cf.natali at gmail.com Wed Jun 19 16:01:49 2013 From: cf.natali at gmail.com (=?ISO-8859-1?Q?Charles=2DFran=E7ois_Natali?=) Date: Wed, 19 Jun 2013 16:01:49 +0200 Subject: [Python-Dev] pyparallel and new memory API discussions... In-Reply-To: <20130619131001.GB10039@snakebite.org> References: <20130619131001.GB10039@snakebite.org> Message-ID: 2013/6/19 Trent Nelson : > > The new memory API discussions (and PEP) warrant a quick pyparallel > update: a couple of weeks after PyCon, I came up with a solution for > the biggest show-stopper that has been plaguing pyparallel since its > inception: being able to detect the modification of "main thread" > Python objects from within a parallel context. > > For example, `data.append(4)` in the example below will generate an > AssignmentError exception, because data is a main thread object, and > `data.append(4)` gets executed from within a parallel context:: > > data = [ 1, 2, 3 ] > > def work(): > data.append(4) > > async.submit_work(work) > > The solution turned out to be deceptively simple: > > 1. Prior to running parallel threads, lock all "main thread" > memory pages as read-only (via VirtualProtect on Windows, > mprotect on POSIX). > > 2. Detect attempts to write to main thread pages during parallel > thread execution (via SEH on Windows or a SIGSEGV trap on POSIX), > and raise an exception instead (detection is done in the ceval > frame exec loop). Quick stupid question: because of refcounts, the pages will be written to even in case of read-only access. How do you deal with this? cf From solipsis at pitrou.net Wed Jun 19 16:15:58 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 19 Jun 2013 16:15:58 +0200 Subject: [Python-Dev] RFC: PEP 445: Add new APIs to customize Python memory allocators References: Message-ID: <20130619161558.1f042553@pitrou.net> Le Tue, 18 Jun 2013 22:40:49 +0200, Victor Stinner a ?crit : > > Other changes > ------------- > [...] > > * Configure external libraries like zlib or OpenSSL to allocate memory > using ``PyMem_RawMalloc()`` Why so, and is it done by default? > Only one get/set function for block allocators > ---------------------------------------------- > > Replace the 6 functions: > > * ``void PyMem_GetRawAllocator(PyMemBlockAllocator *allocator)`` > * ``void PyMem_GetAllocator(PyMemBlockAllocator *allocator)`` > * ``void PyObject_GetAllocator(PyMemBlockAllocator *allocator)`` > * ``void PyMem_SetRawAllocator(PyMemBlockAllocator *allocator)`` > * ``void PyMem_SetAllocator(PyMemBlockAllocator *allocator)`` > * ``void PyObject_SetAllocator(PyMemBlockAllocator *allocator)`` > > with 2 functions with an additional *domain* argument: > > * ``int PyMem_GetBlockAllocator(int domain, PyMemBlockAllocator > *allocator)`` > * ``int PyMem_SetBlockAllocator(int domain, PyMemBlockAllocator > *allocator)`` I would much prefer this solution. > Drawback: the caller has to check if the result is 0, or handle the > error. Or you can just call Py_FatalError() if the domain is invalid. > If an hook is used to the track memory usage, the ``malloc()`` memory > will not be seen. Remaining ``malloc()`` may allocate a lot of memory > and so would be missed in reports. A lot of memory? In main()? Regards Antoine. From trent at snakebite.org Wed Jun 19 16:33:10 2013 From: trent at snakebite.org (Trent Nelson) Date: Wed, 19 Jun 2013 22:33:10 +0800 Subject: [Python-Dev] pyparallel and new memory API discussions... In-Reply-To: References: <20130619131001.GB10039@snakebite.org> Message-ID: <20130619143309.GA10788@snakebite.org> Hi Charles-Fran?ois! Good to hear from you again. It was actually your e-mail a few months ago that acted as the initial catalyst for this memory protection idea, so, thanks for that :-) Answer below. On Wed, Jun 19, 2013 at 07:01:49AM -0700, Charles-Fran?ois Natali wrote: > 2013/6/19 Trent Nelson : > > > > The new memory API discussions (and PEP) warrant a quick pyparallel > > update: a couple of weeks after PyCon, I came up with a solution for > > the biggest show-stopper that has been plaguing pyparallel since its > > inception: being able to detect the modification of "main thread" > > Python objects from within a parallel context. > > > > For example, `data.append(4)` in the example below will generate an > > AssignmentError exception, because data is a main thread object, and > > `data.append(4)` gets executed from within a parallel context:: > > > > data = [ 1, 2, 3 ] > > > > def work(): > > data.append(4) > > > > async.submit_work(work) > > > > The solution turned out to be deceptively simple: > > > > 1. Prior to running parallel threads, lock all "main thread" > > memory pages as read-only (via VirtualProtect on Windows, > > mprotect on POSIX). > > > > 2. Detect attempts to write to main thread pages during parallel > > thread execution (via SEH on Windows or a SIGSEGV trap on POSIX), > > and raise an exception instead (detection is done in the ceval > > frame exec loop). > > Quick stupid question: because of refcounts, the pages will be written > to even in case of read-only access. How do you deal with this? Easy: I don't refcount in parallel contexts :-) There's no need, for two reasons: 1. All memory allocated in a parallel context is localized to a private heap. When the parallel context is finished, the entire heap can be blown away in one fell-swoop. There's no need for reference counting or GC because none of the objects will exist after the parallel context completes. 2. The main thread won't be running when parallel threads/contexts are executing, which means main thread objects being accessed in parallel contexts (read-only access is fine) won't be suddenly free()'d or GC-collected or whatever. You get credit for that second point; you asked a similar question a few months ago that made me realize I absolutely couldn't have the main thread running at the same time the parallel threads were running. Once I accepted that as a design constraint, everything else came together nicely... "Hmmm, if the main thread isn't running, it won't need write-access to any of its pages! If we mark them read-only, we could catch the traps/SEHs from parallel threads, then raise an exception, ahh, simple!". I'm both chuffed at how simple it is (considering it was *the* major show-stopper), and miffed at how it managed to elude me for so long ;-) Regards, Trent. From victor.stinner at gmail.com Wed Jun 19 17:24:21 2013 From: victor.stinner at gmail.com (Victor Stinner) Date: Wed, 19 Jun 2013 17:24:21 +0200 Subject: [Python-Dev] RFC: PEP 445: Add new APIs to customize Python memory allocators In-Reply-To: <20130619161558.1f042553@pitrou.net> References: <20130619161558.1f042553@pitrou.net> Message-ID: 2013/6/19 Antoine Pitrou : > Le Tue, 18 Jun 2013 22:40:49 +0200, > Victor Stinner a ?crit : >> >> Other changes >> ------------- >> > [...] >> >> * Configure external libraries like zlib or OpenSSL to allocate memory >> using ``PyMem_RawMalloc()`` > > Why so, and is it done by default? (Oh, I realized that PyMem_Malloc() may be used instead of PyMem_RawMalloc() if we are sure that the library will only be used when the GIL is held.) "is it done by default?" First, it would be safer to only reuse PyMem_RawMalloc() allocator if PyMem_SetRawMalloc() was called. Just to avoid regressions in Python 3.4. Then, it depends on the library: if the allocator can be replaced for one library object (ex: expat supports this), it can always be replaced. Otherwise, we should only replace the library allocator if Python is a standalone program (don't replace the library allocator if Python is embedded). That's why I asked if it is possible to check if Python is embedded or not. "Why so," For the "track memory usage" use case, it is important to track memory allocated in external libraries to have accurate reports, because these allocations may be huge. >> Only one get/set function for block allocators >> ---------------------------------------------- >> >> Replace the 6 functions: >> >> * ``void PyMem_GetRawAllocator(PyMemBlockAllocator *allocator)`` >> * ``void PyMem_GetAllocator(PyMemBlockAllocator *allocator)`` >> * ``void PyObject_GetAllocator(PyMemBlockAllocator *allocator)`` >> * ``void PyMem_SetRawAllocator(PyMemBlockAllocator *allocator)`` >> * ``void PyMem_SetAllocator(PyMemBlockAllocator *allocator)`` >> * ``void PyObject_SetAllocator(PyMemBlockAllocator *allocator)`` >> >> with 2 functions with an additional *domain* argument: >> >> * ``int PyMem_GetBlockAllocator(int domain, PyMemBlockAllocator >> *allocator)`` >> * ``int PyMem_SetBlockAllocator(int domain, PyMemBlockAllocator >> *allocator)`` > > I would much prefer this solution. I don't have a strong preference between these two choices. Oh, one argument in favor of one generic function is that code using these functions would be simpler. Extract of the unit test of the implementation (_testcapi.c): + if (api == 'o') + PyObject_SetAllocator(&hook.alloc); + else if (api == 'r') + PyMem_SetRawAllocator(&hook.alloc); + else + PyMem_SetAllocator(&hook.alloc); With a generic function, this block can be replace with one unique function call. >> Drawback: the caller has to check if the result is 0, or handle the >> error. > > Or you can just call Py_FatalError() if the domain is invalid. I don't like Py_FatalError(), especially when Python is embedded. It's safer to return -1 and expect the caller to check for the error case. >> If an hook is used to the track memory usage, the ``malloc()`` memory >> will not be seen. Remaining ``malloc()`` may allocate a lot of memory >> and so would be missed in reports. > > A lot of memory? In main()? Not in main(). The Python expat and zlib modules call directly malloc() and may allocate large blocks. External libraries like OpenSSL or bz2 may also allocate large blocks. See issues #18203 and #18227. Victor From kristjan at ccpgames.com Wed Jun 19 17:31:35 2013 From: kristjan at ccpgames.com (=?iso-8859-1?Q?Kristj=E1n_Valur_J=F3nsson?=) Date: Wed, 19 Jun 2013 15:31:35 +0000 Subject: [Python-Dev] RFC: PEP 445: Add new APIs to customize Python memory allocators In-Reply-To: <51C134B2.8040408@scottdial.com> References: <51C0EC0B.5040704@scottdial.com> <51C134B2.8040408@scottdial.com> Message-ID: Right, think of the "ctxt" as a "this" pointer from c++. If you have an allocator object, that you got from some c++ api, and want to ask Python to use that, you need to be able to thunk the "this" pointer to get at the particular allocator instance. It used to be a common mistake when writing C callback apis to forget to add an opaque "context" pointer along with the callback function. This omission makes it difficult (but not impossible) to attach c++ methods to such callbacks. K > -----Original Message----- > From: Python-Dev [mailto:python-dev- > bounces+kristjan=ccpgames.com at python.org] On Behalf Of Scott Dial > Sent: 19. j?n? 2013 04:34 > To: ncoghlan at gmail.com > Cc: Python-Dev at python.org > Subject: Re: [Python-Dev] RFC: PEP 445: Add new APIs to customize Python > memory allocators > > On 6/18/2013 11:32 PM, Nick Coghlan wrote: > > Agreed more of that rationale needs to be moved from the issue tracker > > into the PEP, though. > > Thanks for the clarification. I hadn't read the issue tracker at all. On it's face > value, I didn't see what purpose it served, but having read Kristj?n's > comments on the issue tracker, he would like to store state for the allocators > in that ctx pointer.[1] Having read that (previously, I thought the only utility > was distinguishing which domain it was -- a small, glorified enumeration), but > his use-case makes sense and definitely is informative to have in the PEP, > because the utility of that wasn't obvious to me. > > Thanks, > -Scott > > [1] http://bugs.python.org/issue3329#msg190529 > """ > One particular trick we have been using, which might be of interest, is to be > able to tag each allocation with a "context" id. This is then set according to a > global sys.memcontext variable, which the program will modify according to > what it is doing. This can then be used to track memory usage by different > parts of the program. > """ > > -- > Scott Dial > scott at scottdial.com > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python- > dev/kristjan%40ccpgames.com From solipsis at pitrou.net Wed Jun 19 17:43:54 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 19 Jun 2013 17:43:54 +0200 Subject: [Python-Dev] RFC: PEP 445: Add new APIs to customize Python memory allocators In-Reply-To: References: <20130619161558.1f042553@pitrou.net> Message-ID: <20130619174354.34a7d8f0@fsol> On Wed, 19 Jun 2013 17:24:21 +0200 Victor Stinner wrote: > > For the "track memory usage" use case, it is important to track memory > allocated in external libraries to have accurate reports, because > these allocations may be huge. [...] > Not in main(). The Python expat and zlib modules call directly > malloc() and may allocate large blocks. External libraries like > OpenSSL or bz2 may also allocate large blocks. Fair enough. > >> Drawback: the caller has to check if the result is 0, or handle the > >> error. > > > > Or you can just call Py_FatalError() if the domain is invalid. > > I don't like Py_FatalError(), especially when Python is embedded. It's > safer to return -1 and expect the caller to check for the error case. I don't think you need to check for errors. The domain is always one of the existing constants, i.e. it should be hard-coded in the source, not computed. Regards Antoine. From victor.stinner at gmail.com Wed Jun 19 17:45:55 2013 From: victor.stinner at gmail.com (Victor Stinner) Date: Wed, 19 Jun 2013 17:45:55 +0200 Subject: [Python-Dev] pyparallel and new memory API discussions... In-Reply-To: <20130619143309.GA10788@snakebite.org> References: <20130619131001.GB10039@snakebite.org> <20130619143309.GA10788@snakebite.org> Message-ID: > 1. All memory allocated in a parallel context is localized to a > private heap. How do you allocate memory in this "private" heap? Did you add new functions to allocate memory? Victor From victor.stinner at gmail.com Wed Jun 19 17:49:02 2013 From: victor.stinner at gmail.com (Victor Stinner) Date: Wed, 19 Jun 2013 17:49:02 +0200 Subject: [Python-Dev] RFC: PEP 445: Add new APIs to customize Python memory allocators In-Reply-To: <20130619174354.34a7d8f0@fsol> References: <20130619161558.1f042553@pitrou.net> <20130619174354.34a7d8f0@fsol> Message-ID: 2013/6/19 Antoine Pitrou : > On Wed, 19 Jun 2013 17:24:21 +0200 >> >> Drawback: the caller has to check if the result is 0, or handle the >> >> error. >> > >> > Or you can just call Py_FatalError() if the domain is invalid. >> >> I don't like Py_FatalError(), especially when Python is embedded. It's >> safer to return -1 and expect the caller to check for the error case. > > I don't think you need to check for errors. The domain is always one of > the existing constants, i.e. it should be hard-coded in the source, not > computed. Imagine that PyMem_GetBlockAllocator() is part of the stable ABI and that a new domain is added to Python 3.5. An application is written for Python 3.5 and is run with Python 3.4: how would the application notice that PyMem_GetBlockAllocator() does not know the new domain? "I don't think you need to check for errors." Do you mean that an unknown domain should be simply ignored? Victor From kristjan at ccpgames.com Wed Jun 19 17:28:22 2013 From: kristjan at ccpgames.com (=?iso-8859-1?Q?Kristj=E1n_Valur_J=F3nsson?=) Date: Wed, 19 Jun 2013 15:28:22 +0000 Subject: [Python-Dev] RFC: PEP 445: Add new APIs to customize Python memory allocators In-Reply-To: References: Message-ID: Oh, it should be public, in my opinion. We do exactly that when we embed python into UnrealEngine. We keep pythons internal PyObject_Mem allocator, but have it ask UnrealEngine for its arenas. That way, we can still keep track of python's memory usage from with the larger application, even if the granularity of memory is now on an "arena" level, rather than individual allocs. K > -----Original Message----- > From: Python-Dev [mailto:python-dev- > bounces+kristjan=ccpgames.com at python.org] On Behalf Of Victor Stinner > Sent: 18. j?n? 2013 21:20 > To: Python Dev > Subject: Re: [Python-Dev] RFC: PEP 445: Add new APIs to customize Python > memory allocators > > typedef struct { > /* user context passed as the first argument > to the 2 functions */ > void *ctx; > > /* allocate a memory mapping */ > void* (*alloc) (void *ctx, size_t size); > > /* release a memory mapping */ > void (*free) (void *ctx, void *ptr, size_t size); > } PyMemMappingAllocator; > > The PyMemMappingAllocator structure is very specific to the pymalloc > allocator. There is no "resize", "lock" nor "protect" method. There is no way > to configure protection or flags of the mapping. The > PyMem_SetMappingAllocator() function was initially called > _PyObject_SetArenaAllocator(). I'm not sure that the structure and the > 2 related functions should be public. Can an extension module call private > (_Py*) functions or use a private structure? > > Or the structure might be renamed to indicate that it is specific to arenas? > > What do you think? > > Victor > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python- > dev/kristjan%40ccpgames.com From victor.stinner at gmail.com Wed Jun 19 17:58:36 2013 From: victor.stinner at gmail.com (Victor Stinner) Date: Wed, 19 Jun 2013 17:58:36 +0200 Subject: [Python-Dev] RFC: PEP 445: Add new APIs to customize Python memory allocators In-Reply-To: References: Message-ID: 2013/6/19 Kristj?n Valur J?nsson : > Oh, it should be public, in my opinion. Ok. And do you think that the PyMemMappingAllocator structure is complete, or that we should add something to be future-proof? At least, PyMemMappingAllocator is enough for pymalloc usage :-) Is PyMemMappingAllocator complete enough for your usage at CCP Games? > We do exactly that when we embed python into UnrealEngine. We keep pythons internal PyObject_Mem allocator, but have it ask UnrealEngine for its arenas. That way, we can still keep track of python's memory usage from with the larger application, even if the granularity of memory is now on an "arena" level, rather than individual allocs. I hope that the PEP 445 is flexible enough to allow you to decide which functions are hooked and replaced, and which functions will be leaved unchanged. That's why I'm not in favor of the "Make PyMem_Malloc() reuse PyMem_RawMalloc() by default" alternative. Victor From trent at snakebite.org Wed Jun 19 18:00:01 2013 From: trent at snakebite.org (Trent Nelson) Date: Thu, 20 Jun 2013 00:00:01 +0800 Subject: [Python-Dev] pyparallel and new memory API discussions... In-Reply-To: References: <20130619131001.GB10039@snakebite.org> <20130619143309.GA10788@snakebite.org> Message-ID: <20130619160000.GA11044@snakebite.org> On Wed, Jun 19, 2013 at 08:45:55AM -0700, Victor Stinner wrote: > > 1. All memory allocated in a parallel context is localized to a > > private heap. > > How do you allocate memory in this "private" heap? Did you add new > functions to allocate memory? Yup: _PyHeap_Malloc(): http://hg.python.org/sandbox/trent/file/0e70a0caa1c0/Python/pyparallel.c#l2365. All memory operations (PyObject_New/Malloc etc) get intercepted during parallel thread execution and redirected to _PyHeap_Malloc(), which is a very simple slab allocator. (No need for convoluted buckets because we never free individual objects during parallel execution; instead, we just blow everything away at the end.) Trent. From tjreedy at udel.edu Wed Jun 19 18:13:06 2013 From: tjreedy at udel.edu (Terry Reedy) Date: Wed, 19 Jun 2013 12:13:06 -0400 Subject: [Python-Dev] RFC: PEP 445: Add new APIs to customize Python memory allocators In-Reply-To: References: <20130619161558.1f042553@pitrou.net> Message-ID: On 6/19/2013 11:24 AM, Victor Stinner wrote: > 2013/6/19 Antoine Pitrou : >> Le Tue, 18 Jun 2013 22:40:49 +0200, >> Victor Stinner a ?crit : >>> Only one get/set function for block allocators >>> ---------------------------------------------- >>> >>> Replace the 6 functions: >>> >>> * ``void PyMem_GetRawAllocator(PyMemBlockAllocator *allocator)`` >>> * ``void PyMem_GetAllocator(PyMemBlockAllocator *allocator)`` >>> * ``void PyObject_GetAllocator(PyMemBlockAllocator *allocator)`` >>> * ``void PyMem_SetRawAllocator(PyMemBlockAllocator *allocator)`` >>> * ``void PyMem_SetAllocator(PyMemBlockAllocator *allocator)`` >>> * ``void PyObject_SetAllocator(PyMemBlockAllocator *allocator)`` >>> >>> with 2 functions with an additional *domain* argument: >>> >>> * ``int PyMem_GetBlockAllocator(int domain, PyMemBlockAllocator >>> *allocator)`` >>> * ``int PyMem_SetBlockAllocator(int domain, PyMemBlockAllocator >>> *allocator)`` >> >> I would much prefer this solution. I do to. The two names can be remembered as one pair with only get/set difference. -- Terry Jan Reedy From victor.stinner at gmail.com Wed Jun 19 18:20:15 2013 From: victor.stinner at gmail.com (Victor Stinner) Date: Wed, 19 Jun 2013 18:20:15 +0200 Subject: [Python-Dev] pyparallel and new memory API discussions... In-Reply-To: <20130619160000.GA11044@snakebite.org> References: <20130619131001.GB10039@snakebite.org> <20130619143309.GA10788@snakebite.org> <20130619160000.GA11044@snakebite.org> Message-ID: """ So, Victor, I'm interested to hear how the new API you're proposing will affect this solution I've come up with for pyparallel; I'm going to be absolutely dependent upon the ability to lock main thread pages as read-only in one fell-swoop -- am I still going to be able to do that with your new API in place? """ 2013/6/19 Trent Nelson : > On Wed, Jun 19, 2013 at 08:45:55AM -0700, Victor Stinner wrote: >> > 1. All memory allocated in a parallel context is localized to a >> > private heap. >> >> How do you allocate memory in this "private" heap? Did you add new >> functions to allocate memory? > > Yup: > _PyHeap_Malloc(): http://hg.python.org/sandbox/trent/file/0e70a0caa1c0/Python/pyparallel.c#l2365. > > All memory operations (PyObject_New/Malloc etc) get intercepted > during parallel thread execution and redirected to _PyHeap_Malloc(), > which is a very simple slab allocator. (No need for convoluted > buckets because we never free individual objects during parallel > execution; instead, we just blow everything away at the end.) Ok, so I don't think that the PEP 445 would change anything for you. The following change might have an impact: If _PyHeap_Malloc is not thread safe, replacing PyMem_Malloc() with PyMem_RawMalloc() when the GIL is not held would avoid bugs in your code. If you want to choose dynamically the allocator at runtime, you can replace PyObject_Malloc allocator using: -------------------------- 8< ----------------- static void * _PxMem_AllocMalloc(void *ctx, size_t size) { PyMemBlockAllocator *ctx; if (Py_PXCTX) return _PxMem_Malloc(size)) else return alloc->malloc(alloc->ctx, size); } ... PyMemBlockAllocator pyparallel_pyobject; static void * setup_pyparallel_allocator(void) { PyMemBlockAllocator alloc; PyObject_GetAllocator(&pyparallel_pyobject); alloc.ctx = &pyparallel_pyobject; alloc.malloc = _PxMem_AllocMalloc; ... PyObject_SetAllocator(&alloc); } -------------------------- 8< ----------------- But I don't know if you want pyparallel to be an "optional" feature chosen at runtime... Victor From solipsis at pitrou.net Wed Jun 19 18:22:29 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 19 Jun 2013 18:22:29 +0200 Subject: [Python-Dev] RFC: PEP 445: Add new APIs to customize Python memory allocators In-Reply-To: References: <20130619161558.1f042553@pitrou.net> <20130619174354.34a7d8f0@fsol> Message-ID: <20130619182229.2fa7cd92@fsol> On Wed, 19 Jun 2013 17:49:02 +0200 Victor Stinner wrote: > 2013/6/19 Antoine Pitrou : > > On Wed, 19 Jun 2013 17:24:21 +0200 > >> >> Drawback: the caller has to check if the result is 0, or handle the > >> >> error. > >> > > >> > Or you can just call Py_FatalError() if the domain is invalid. > >> > >> I don't like Py_FatalError(), especially when Python is embedded. It's > >> safer to return -1 and expect the caller to check for the error case. > > > > I don't think you need to check for errors. The domain is always one of > > the existing constants, i.e. it should be hard-coded in the source, not > > computed. > > Imagine that PyMem_GetBlockAllocator() is part of the stable ABI and > that a new domain is added to Python 3.5. An application is written > for Python 3.5 and is run with Python 3.4: how would the application > notice that PyMem_GetBlockAllocator() does not know the new domain? That's a good question. I don't know why guidelines Martin used when designing the stable ABI, but I would expect important high-level functions to end there, not memory allocation debugging. Regards Antoine. From trent at snakebite.org Wed Jun 19 18:42:06 2013 From: trent at snakebite.org (Trent Nelson) Date: Thu, 20 Jun 2013 00:42:06 +0800 Subject: [Python-Dev] pyparallel and new memory API discussions... In-Reply-To: References: <20130619131001.GB10039@snakebite.org> <20130619143309.GA10788@snakebite.org> <20130619160000.GA11044@snakebite.org> Message-ID: <20130619164205.GA11338@snakebite.org> On Wed, Jun 19, 2013 at 09:20:15AM -0700, Victor Stinner wrote: > """ > So, Victor, I'm interested to hear how the new API you're proposing > will affect this solution I've come up with for pyparallel; I'm > going to be absolutely dependent upon the ability to lock main > thread pages as read-only in one fell-swoop -- am I still going to > be able to do that with your new API in place? > """ > > 2013/6/19 Trent Nelson : > > On Wed, Jun 19, 2013 at 08:45:55AM -0700, Victor Stinner wrote: > >> > 1. All memory allocated in a parallel context is localized to a > >> > private heap. > >> > >> How do you allocate memory in this "private" heap? Did you add new > >> functions to allocate memory? > > > > Yup: > > _PyHeap_Malloc(): http://hg.python.org/sandbox/trent/file/0e70a0caa1c0/Python/pyparallel.c#l2365. > > > > All memory operations (PyObject_New/Malloc etc) get intercepted > > during parallel thread execution and redirected to _PyHeap_Malloc(), > > which is a very simple slab allocator. (No need for convoluted > > buckets because we never free individual objects during parallel > > execution; instead, we just blow everything away at the end.) > > Ok, so I don't think that the PEP 445 would change anything for you. > > The following change might have an impact: If _PyHeap_Malloc is not > thread safe, replacing PyMem_Malloc() with PyMem_RawMalloc() when the > GIL is not held would avoid bugs in your code. Hmmm, well, _PyHeap_Malloc is sort of implicitly thread-safe, by design, but I'm not sure if we're referring to the same sort of thread-safe problem here. For one, _PyHeap_Malloc won't ever run if the GIL isn't being held. (Parallel threads are only allowed to run when the main thread has the GIL held and has relinquished control to parallel threads.) Also, I interpret PyMem_RawMalloc() as a direct shortcut to malloc() (or something else that returns void *s that are then free()'d down the track). Is that right? I don't think that would impact pyparallel. > If you want to choose dynamically the allocator at runtime, you can > replace PyObject_Malloc allocator using: > -------------------------- 8< ----------------- > static void * > _PxMem_AllocMalloc(void *ctx, size_t size) > { > PyMemBlockAllocator *ctx; > if (Py_PXCTX) > return _PxMem_Malloc(size)) > else > return alloc->malloc(alloc->ctx, size); > } > > ... > > PyMemBlockAllocator pyparallel_pyobject; > > static void * > setup_pyparallel_allocator(void) > { > PyMemBlockAllocator alloc; > PyObject_GetAllocator(&pyparallel_pyobject); > alloc.ctx = &pyparallel_pyobject; > alloc.malloc = _PxMem_AllocMalloc; > ... > PyObject_SetAllocator(&alloc); > } > -------------------------- 8< ----------------- > > But I don't know if you want pyparallel to be an "optional" feature > chosen at runtime... Hmmm, those code snippets are interesting. Time for some more homework. Trent. From victor.stinner at gmail.com Thu Jun 20 00:21:13 2013 From: victor.stinner at gmail.com (Victor Stinner) Date: Thu, 20 Jun 2013 00:21:13 +0200 Subject: [Python-Dev] RFC: PEP 445: Add new APIs to customize Python memory allocators In-Reply-To: <20130619182229.2fa7cd92@fsol> References: <20130619161558.1f042553@pitrou.net> <20130619174354.34a7d8f0@fsol> <20130619182229.2fa7cd92@fsol> Message-ID: PyMem_RawAlloc()/Realloc/Free should be part of the stable ABI. I agree that all other new fumctions ans structures should not. Victor -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Thu Jun 20 01:01:49 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 20 Jun 2013 09:01:49 +1000 Subject: [Python-Dev] RFC: PEP 445: Add new APIs to customize Python memory allocators In-Reply-To: References: Message-ID: On 20 Jun 2013 02:03, "Victor Stinner" wrote: > > 2013/6/19 Kristj?n Valur J?nsson : > > Oh, it should be public, in my opinion. > > Ok. And do you think that the PyMemMappingAllocator structure is > complete, or that we should add something to be future-proof? At > least, PyMemMappingAllocator is enough for pymalloc usage :-) > > Is PyMemMappingAllocator complete enough for your usage at CCP Games? Can we go back to calling this the "Arena" allocator? Or at least "Mapped"? When I see "Mapping" in the context of Python I think of the container API, not a memory allocation API. > > > We do exactly that when we embed python into UnrealEngine. We keep pythons internal PyObject_Mem allocator, but have it ask UnrealEngine for its arenas. That way, we can still keep track of python's memory usage from with the larger application, even if the granularity of memory is now on an "arena" level, rather than individual allocs. > > I hope that the PEP 445 is flexible enough to allow you to decide > which functions are hooked and replaced, and which functions will be > leaved unchanged. That's why I'm not in favor of the "Make > PyMem_Malloc() reuse PyMem_RawMalloc() by default" alternative. It's also why I'm in favour of the "domain" API rather than separate functions. 1. In the initial iteration, just have the three basic domains (raw, interpreter, objects). Replacing allocators for third party libraries is the responsibility of embedding applications. 2. In a later iteration, add "PyMem_AddDomain" and "PyMem_GetDomains" APIs so that extension modules can register new domains for wrapped libraries. Replacing allocators is still the responsibility of embedding applications, but there's a consistent API to do it. (Alternatively, we could do both now) And agreed PyMem_Raw* are the only new APIs that should be added to the stable ABI. Cheers, Nick. > > Victor > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From victor.stinner at gmail.com Thu Jun 20 07:37:42 2013 From: victor.stinner at gmail.com (Victor Stinner) Date: Thu, 20 Jun 2013 07:37:42 +0200 Subject: [Python-Dev] RFC: PEP 445: Add new APIs to customize Python memory allocators In-Reply-To: References: Message-ID: Le jeudi 20 juin 2013, Nick Coghlan a ?crit : > > > Is PyMemMappingAllocator complete enough for your usage at CCP Games? > > Can we go back to calling this the "Arena" allocator? Or at least > "Mapped"? When I see "Mapping" in the context of Python I think of the > container API, not a memory allocation API. > This function is written to be able to use mmap() and VirtualAlloc(). There is no Python function to use directly this allocator yet, but I chose "memory mapping" name because it is very different than the heap and it may be useful for other functions than pymalloc. If I change the name, it would be called PyObject_SetArenaAllocator() with a PyObjectArenaAllocator structure. I'm not sure that PyMemMappingAllocator API is future-proof, so I'm fine to call it "arena" again. > > I hope that the PEP 445 is flexible enough to allow you to decide > > which functions are hooked and replaced, and which functions will be > > leaved unchanged. That's why I'm not in favor of the "Make > > PyMem_Malloc() reuse PyMem_RawMalloc() by default" alternative. > > It's also why I'm in favour of the "domain" API rather than separate > functions. > > 1. In the initial iteration, just have the three basic domains (raw, > interpreter, objects). Replacing allocators for third party libraries is > the responsibility of embedding applications. > > 2. In a later iteration, add "PyMem_AddDomain" and "PyMem_GetDomains" APIs > so that extension modules can register new domains for wrapped libraries. > Replacing allocators is still the responsibility of embedding applications, > but there's a consistent API to do it. > > (Alternatively, we could do both now) > How would you use an allocator of a new domain? PyMemBlockAllocator structure is not convinient, and if Py_GetAllocator() only once, you may loose a hook installed later. Victor -------------- next part -------------- An HTML attachment was scrubbed... URL: From victor.stinner at gmail.com Thu Jun 20 12:16:56 2013 From: victor.stinner at gmail.com (Victor Stinner) Date: Thu, 20 Jun 2013 12:16:56 +0200 Subject: [Python-Dev] RFC: PEP 445: Add new APIs to customize Python memory allocators In-Reply-To: References: Message-ID: > * Add new GIL-free (no need to hold the GIL) memory allocator functions: > > - ``void* PyMem_RawMalloc(size_t size)`` > - ``void* PyMem_RawRealloc(void *ptr, size_t new_size)`` > - ``void PyMem_RawFree(void *ptr)`` > - the behaviour of requesting zero bytes is not defined: return *NULL* > or a distinct non-*NULL* pointer depending on the platform. > (...) > * Add new functions to get and set internal functions of > ``PyMem_Malloc()``, ``PyMem_Realloc()`` and ``PyMem_Free()``: > > - ``void PyMem_GetAllocator(PyMemBlockAllocator *allocator)`` > - ``void PyMem_SetAllocator(PyMemBlockAllocator *allocator)`` > - ``malloc(ctx, 0)`` and ``realloc(ctx, ptr, 0)`` must not return > *NULL*: it would be treated as an error. > - default allocator: ``malloc()``, ``realloc()``, ``free()``; > ``PyMem_Malloc(0)`` calls ``malloc(1)`` > and ``PyMem_Realloc(NULL, 0)`` calls ``realloc(NULL, 1)`` Oh, one more question: PyMem_RawMalloc(0) has an undefined behaviour, whereas PyMem_Malloc(0) has a well defined behaviour (don't return NULL). Adding "if (size == 1) size = 0;" in the default implementation of PyMem_RawMalloc(0) should not have a visible overhead, but it gives the same guarantee than PyMem_Malloc(0) (don't return NULL). Do you agree to add the test? I chose to implement "if (size > (size_t)PY_SSIZE_T_MAX) return NULL;" in Py*_Malloc(), whereas "if (size == 0) size =1;" is implemented in the inner function (_PyMem_Malloc). An application may use an allocator which has already a well defined behaviour (a "malloc(0)" that don't return NULL) and I expect malloc(1) to allocate "more" memory than malloc(0) (malloc(0) may create a singleton) :-) Victor From kristjan at ccpgames.com Thu Jun 20 12:44:08 2013 From: kristjan at ccpgames.com (=?utf-8?B?S3Jpc3Rqw6FuIFZhbHVyIErDs25zc29u?=) Date: Thu, 20 Jun 2013 10:44:08 +0000 Subject: [Python-Dev] RFC: PEP 445: Add new APIs to customize Python memory allocators In-Reply-To: References: Message-ID: Oh, please don't misunderstand. I'm not making any demands or requirements, what I'm trying to do is to make recommendations based on experience that I have had with embedding. This sound altogether too much like I'm trying to push things one way or the other :) The api as laid out certainly seems to work, and be adequate for the purpose. I can add here as a point of information that since we work on windows, there was no need to pass in the "size" argument to the "munmap" callback. VirtualFree(address, NULL) will release the entire chunk of memory that was initially allocated at that place. Therefor in our implementation we can reuse the same allocator structo for those arenas. But I understand that munmap doesn't have this feature, so passing in the size is prudent. K > -----Original Message----- > From: Victor Stinner [mailto:victor.stinner at gmail.com] > Sent: 19. j?n? 2013 15:59 > To: Kristj?n Valur J?nsson > Cc: Python Dev > Subject: Re: [Python-Dev] RFC: PEP 445: Add new APIs to customize Python > memory allocators > > Is PyMemMappingAllocator complete enough for your usage at CCP Games? From victor.stinner at gmail.com Thu Jun 20 13:26:52 2013 From: victor.stinner at gmail.com (Victor Stinner) Date: Thu, 20 Jun 2013 13:26:52 +0200 Subject: [Python-Dev] PEP 445: Add new APIs to customize Python memory allocators (second round) Message-ID: Hi, I changed the PEP 445 according to the discussing on python-dev. Read it online: http://www.python.org/dev/peps/pep-0445/ Changes: * add PyMemAllocatorDomain enum: PYALLOC_PYMEM_RAW, PYALLOC_PYMEM or PYALLOC_PYOBJECT * rename: - PyMemBlockAllocator structure => PyMemAllocator - PyMemMappingAllocator structure => PyObjectArenaAllocator - PyMem_GetMappingAllocator() => PyObject_GetArenaAllocator() - PyMem_SetMappingAllocator() => PyObject_SetArenaAllocator() * group get/set functions to only keep 2 functions: PyMem_GetAllocator() and PyMem_SetAllocator() * PyMem_RawMalloc(0) now calls malloc(1) to have a well defined behaviour * PYALLOC_PYMEM_RAW and PYALLOC_PYMEM are now using exactly the same allocator * Add more references for external libraries As expected, most changes occurred in the Proposal section. Full diff: http://hg.python.org/peps/rev/a17ebebe52ca I also updated the implementation attached to: http://bugs.python.org/issue3329 Who is going give the final decision on this PEP? Guido? Another candidate? Victor From victor.stinner at gmail.com Thu Jun 20 13:28:23 2013 From: victor.stinner at gmail.com (Victor Stinner) Date: Thu, 20 Jun 2013 13:28:23 +0200 Subject: [Python-Dev] PEP 445: Add new APIs to customize Python memory allocators (second round) In-Reply-To: References: Message-ID: > I also updated the implementation attached to: > http://bugs.python.org/issue3329 I excluded new enums, structures and functions from the stable ABI, except PyMem_Raw*() functions. Victor From solipsis at pitrou.net Thu Jun 20 14:16:48 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 20 Jun 2013 14:16:48 +0200 Subject: [Python-Dev] PEP 445: Add new APIs to customize Python memory allocators (second round) References: Message-ID: <20130620141648.0dbedfc1@pitrou.net> Le Thu, 20 Jun 2013 13:26:52 +0200, Victor Stinner a ?crit : > Hi, > > I changed the PEP 445 according to the discussing on python-dev. > > Read it online: > http://www.python.org/dev/peps/pep-0445/ > > Changes: > > * add PyMemAllocatorDomain enum: PYALLOC_PYMEM_RAW, PYALLOC_PYMEM or > PYALLOC_PYOBJECT PYMEM_DOMAIN_RAW? > * rename: > > - PyMemBlockAllocator structure => PyMemAllocator > - PyMemMappingAllocator structure => PyObjectArenaAllocator > - PyMem_GetMappingAllocator() => PyObject_GetArenaAllocator() > - PyMem_SetMappingAllocator() => PyObject_SetArenaAllocator() > > * group get/set functions to only keep 2 functions: > PyMem_GetAllocator() and PyMem_SetAllocator() > * PyMem_RawMalloc(0) now calls malloc(1) to have a well defined > behaviour > * PYALLOC_PYMEM_RAW and PYALLOC_PYMEM are now using exactly the same > allocator > * Add more references for external libraries > > As expected, most changes occurred in the Proposal section. > > Full diff: > http://hg.python.org/peps/rev/a17ebebe52ca > > I also updated the implementation attached to: > http://bugs.python.org/issue3329 > > Who is going give the final decision on this PEP? Guido? Another > candidate? > > Victor From ncoghlan at gmail.com Thu Jun 20 14:19:58 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 20 Jun 2013 22:19:58 +1000 Subject: [Python-Dev] RFC: PEP 445: Add new APIs to customize Python memory allocators In-Reply-To: References: Message-ID: On 20 June 2013 15:37, Victor Stinner wrote: > Le jeudi 20 juin 2013, Nick Coghlan a ?crit : >> >> > Is PyMemMappingAllocator complete enough for your usage at CCP Games? >> >> Can we go back to calling this the "Arena" allocator? Or at least >> "Mapped"? When I see "Mapping" in the context of Python I think of the >> container API, not a memory allocation API. > > This function is written to be able to use mmap() and VirtualAlloc(). There > is no Python function to use directly this allocator yet, but I chose > "memory mapping" name because it is very different than the heap and it may > be useful for other functions than pymalloc. > > If I change the name, it would be called PyObject_SetArenaAllocator() with a > PyObjectArenaAllocator structure. I'm not sure that PyMemMappingAllocator > API is future-proof, so I'm fine to call it "arena" again. Yeah, I think making that API specifically about pymalloc is a good idea. It also makes it clearer that if you're bypassing pymalloc entirely (by replacing the object allocators), then you shouldn't need to worry about those. >> > I hope that the PEP 445 is flexible enough to allow you to decide >> > which functions are hooked and replaced, and which functions will be >> > leaved unchanged. That's why I'm not in favor of the "Make >> > PyMem_Malloc() reuse PyMem_RawMalloc() by default" alternative. >> >> It's also why I'm in favour of the "domain" API rather than separate >> functions. >> >> 1. In the initial iteration, just have the three basic domains (raw, >> interpreter, objects). Replacing allocators for third party libraries is the >> responsibility of embedding applications. >> >> 2. In a later iteration, add "PyMem_AddDomain" and "PyMem_GetDomains" APIs >> so that extension modules can register new domains for wrapped libraries. >> Replacing allocators is still the responsibility of embedding applications, >> but there's a consistent API to do it. >> >> (Alternatively, we could do both now) > > How would you use an allocator of a new domain? PyMemBlockAllocator > structure is not convinient, and if Py_GetAllocator() only once, you may > loose a hook installed later. Say that, for the current PEP, we assume we configure standard library extension modules to use the PyMem or PyMem_Raw APIs (depending on GIL usage), thus allowing those to be redirected automatically to an externally configured allocator when the new PEP 445 APIs are used. The notion I had an mind as a possible future change is that extension modules could register a "set_allocator" callback with the interpreter so they will be automatically notified if the allocator they're interested in changes. However, I also realised that this would actually be independent of the APIs in the current PEP. You could do something like: typedef void (*PyMem_AllocatorSetter)(PyMemAllocator *allocator); void PyMem_AddExternalAllocator( PyMemAllocatorDomain domain, PyMemAllocatorSetter set_allocator ); Then, whenever the allocator for the specified domain was changed, the "set_allocator" callback would be invoked to set the allocator in the extension module as well. The setter would also be called immediately on registration, using the currently defined allocator. We don't have to do this right away (and we should give the basic API a chance to establish itself first). I just like it as something that the "memory domain" model may allow us to pursue in the future. (That said, we may end up wanting something like this internally anyway, even just for the standard library extension modules that do memory allocations) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From christian at python.org Thu Jun 20 15:05:34 2013 From: christian at python.org (Christian Heimes) Date: Thu, 20 Jun 2013 15:05:34 +0200 Subject: [Python-Dev] stat module in C -- what to do with stat.py? Message-ID: <51C2FE1E.8020801@python.org> Hello, I have re-implemented the entire stat module in C. [1] It's a necessary step to fix portability issues. For most constants POSIX standards dictate only the name of the constant and its meaning but not its value. Only the values of permission bits (0644 == rw-r--r--) have fixed values. For example S_IFDIR is usually 0o40000 but it might be 0o20000 on some platforms. Common file types seem to have the same value on all important platforms. It's the only reason we were able to get away with stat.py. But uncommon file types like door, event port and whiteout don't have sensible default values. The C implementation is able to pick up the platform's values easily. What shall I do about stat.py? statmodule.c implements all features of stat.py so there is no point in using it in CPython. It's one Python file less to load on every startup. However it might still be useful to alternative implementations of Python such as Jython or PyPy. 1) keep the file stat.py but let it be shadowed by the builtin stat module. Antoine loathes my hack... 2) rename stat.py to _stat.py 3) remove stat.py Opinions? Christian [1] http://bugs.python.org/issue11016 From florent.xicluna at gmail.com Thu Jun 20 15:21:13 2013 From: florent.xicluna at gmail.com (Florent) Date: Thu, 20 Jun 2013 15:21:13 +0200 Subject: [Python-Dev] stat module in C -- what to do with stat.py? In-Reply-To: <51C2FE1E.8020801@python.org> References: <51C2FE1E.8020801@python.org> Message-ID: we already have "_pyio.py", we could have "_pystat.py". my 2c -- Florent Xicluna 2013/6/20 Christian Heimes : > Hello, > > I have re-implemented the entire stat module in C. [1] It's a necessary > step to fix portability issues. For most constants POSIX standards > dictate only the name of the constant and its meaning but not its value. > Only the values of permission bits (0644 == rw-r--r--) have fixed values. > > For example S_IFDIR is usually 0o40000 but it might be 0o20000 on some > platforms. Common file types seem to have the same value on all > important platforms. It's the only reason we were able to get away with > stat.py. But uncommon file types like door, event port and whiteout > don't have sensible default values. The C implementation is able to pick > up the platform's values easily. > > What shall I do about stat.py? statmodule.c implements all features of > stat.py so there is no point in using it in CPython. It's one Python > file less to load on every startup. However it might still be useful to > alternative implementations of Python such as Jython or PyPy. > > 1) keep the file stat.py but let it be shadowed by the builtin stat > module. Antoine loathes my hack... > 2) rename stat.py to _stat.py > 3) remove stat.py > > Opinions? > > Christian > > [1] http://bugs.python.org/issue11016 > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/florent.xicluna%40gmail.com From christian at python.org Thu Jun 20 15:29:42 2013 From: christian at python.org (Christian Heimes) Date: Thu, 20 Jun 2013 15:29:42 +0200 Subject: [Python-Dev] stat module in C -- what to do with stat.py? In-Reply-To: References: <51C2FE1E.8020801@python.org> Message-ID: <51C303C6.8050609@python.org> Am 20.06.2013 15:21, schrieb Florent: > we already have "_pyio.py", we could have "_pystat.py". Works for me. From brett at python.org Thu Jun 20 15:36:07 2013 From: brett at python.org (Brett Cannon) Date: Thu, 20 Jun 2013 09:36:07 -0400 Subject: [Python-Dev] [Python-checkins] cpython (3.3): Add -b and -X options to python man page. In-Reply-To: <3bbSw10tm2z7Ljf@mail.python.org> References: <3bbSw10tm2z7Ljf@mail.python.org> Message-ID: On Wed, Jun 19, 2013 at 11:20 PM, senthil.kumaran < python-checkins at python.org> wrote: > http://hg.python.org/cpython/rev/dfead0696a71 > changeset: 84224:dfead0696a71 > branch: 3.3 > parent: 84221:0113247f894b > user: Senthil Kumaran > date: Wed Jun 19 22:19:46 2013 -0500 > summary: > Add -b and -X options to python man page. > Patch contributed by Corey Brune. > > files: > Misc/python.man | 22 ++++++++++++++++++---- > 1 files changed, 18 insertions(+), 4 deletions(-) > > > diff --git a/Misc/python.man b/Misc/python.man > --- a/Misc/python.man > +++ b/Misc/python.man > @@ -11,6 +11,9 @@ > .B \-B > ] > [ > +.B \-b > +] > +[ > .B \-d > ] > [ > @@ -23,14 +26,14 @@ > .B \-i > ] > [ > -.B \-m > +.B \-m > .I module-name > ] > -[ > -.B \-q > -] > .br > [ > +.B \-q > +] > +[ > .B \-O > ] > [ > @@ -60,6 +63,10 @@ > .B \-x > ] > [ > +[ > +.B \-X > +.I option > +] > .B \-? > ] > .br > @@ -105,6 +112,10 @@ > .I .py[co] > files on import. See also PYTHONDONTWRITEBYTECODE. > .TP > +.B \-b > +Issue warnings about str(bytes_instance), str(bytearray_instance) > +and comparing bytes/bytearray with str. (-bb: issue errors) > +.TP > .BI "\-c " command > Specify the command to execute (see next section). > This terminates the option list (following options are passed as > @@ -243,6 +254,9 @@ > field matches the line number, where zero matches all line numbers and > is thus equivalent to an omitted line number. > .TP > +.BI "\-X " option > +Set implementation specific option. > Should probably be "Set the implementation-specific option." > +.TP > .B \-x > Skip the first line of the source. This is intended for a DOS > specific hack only. Warning: the line numbers in error messages will > > -- > Repository URL: http://hg.python.org/cpython > > _______________________________________________ > Python-checkins mailing list > Python-checkins at python.org > http://mail.python.org/mailman/listinfo/python-checkins > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Thu Jun 20 15:34:44 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 20 Jun 2013 23:34:44 +1000 Subject: [Python-Dev] PEP 445: Add new APIs to customize Python memory allocators (second round) In-Reply-To: <20130620141648.0dbedfc1@pitrou.net> References: <20130620141648.0dbedfc1@pitrou.net> Message-ID: On 20 June 2013 22:16, Antoine Pitrou wrote: > Le Thu, 20 Jun 2013 13:26:52 +0200, > Victor Stinner a ?crit : > >> Hi, >> >> I changed the PEP 445 according to the discussing on python-dev. >> >> Read it online: >> http://www.python.org/dev/peps/pep-0445/ >> >> Changes: >> >> * add PyMemAllocatorDomain enum: PYALLOC_PYMEM_RAW, PYALLOC_PYMEM or >> PYALLOC_PYOBJECT > > PYMEM_DOMAIN_RAW? The challenge there is coming up with a good name for PYALLOC_PYMEM. I think Victor's names for the domains work well: - "PYALLOC_" prefix to say "this is an allocation domain" - "PYMEM_RAW" suffix for PyMem_RawMalloc/Realloc/Free - "PYMEM" suffix for PyMem_Malloc/Realloc/Free - "PYOBJECT" suffix for PyObject_Malloc/Realloc/Free Although, there's a copy-and-paste error in the domain definitions (2 *Reallocs in each entry and no *Frees). If Guido doesn't want to do it, I'm happy to. Antoine would be a good choice too, since he's the one who suggested taking the time to thrash out the API details in a PEP (which I agree has substantially improved the end result). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Thu Jun 20 15:40:13 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 20 Jun 2013 23:40:13 +1000 Subject: [Python-Dev] stat module in C -- what to do with stat.py? In-Reply-To: <51C303C6.8050609@python.org> References: <51C2FE1E.8020801@python.org> <51C303C6.8050609@python.org> Message-ID: On 20 June 2013 23:29, Christian Heimes wrote: > Am 20.06.2013 15:21, schrieb Florent: >> we already have "_pyio.py", we could have "_pystat.py". > > Works for me. I suggest following the guidelines in PEP 399 for cross implementation compatibility of the standard library: http://www.python.org/dev/peps/pep-0399/#details 1. Keep stat.py 2. Make the C version "_stat" 3. Add the following to the end of stat.py: try: from _stat import * except ImportError: pass Run the tests with and without the C module in the test suite (again, as per PEP 399). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From eric at trueblade.com Thu Jun 20 16:14:25 2013 From: eric at trueblade.com (Eric V. Smith) Date: Thu, 20 Jun 2013 10:14:25 -0400 Subject: [Python-Dev] stat module in C -- what to do with stat.py? In-Reply-To: References: <51C2FE1E.8020801@python.org> <51C303C6.8050609@python.org> Message-ID: <51C30E41.9090301@trueblade.com> On 6/20/2013 9:40 AM, Nick Coghlan wrote: > On 20 June 2013 23:29, Christian Heimes wrote: >> Am 20.06.2013 15:21, schrieb Florent: >>> we already have "_pyio.py", we could have "_pystat.py". >> >> Works for me. > > I suggest following the guidelines in PEP 399 for cross implementation > compatibility of the standard library: > http://www.python.org/dev/peps/pep-0399/#details > > 1. Keep stat.py > 2. Make the C version "_stat" > 3. Add the following to the end of stat.py: > > try: > from _stat import * > except ImportError: > pass > > Run the tests with and without the C module in the test suite (again, > as per PEP 399). Agreed with the above. But isn't the real problem with this module in Python the fact that the constants might be wrong? I'm not sure what, if anything, we can do about that. -- Eric. From brett at python.org Thu Jun 20 17:01:32 2013 From: brett at python.org (Brett Cannon) Date: Thu, 20 Jun 2013 11:01:32 -0400 Subject: [Python-Dev] stat module in C -- what to do with stat.py? In-Reply-To: <51C30E41.9090301@trueblade.com> References: <51C2FE1E.8020801@python.org> <51C303C6.8050609@python.org> <51C30E41.9090301@trueblade.com> Message-ID: On Thu, Jun 20, 2013 at 10:14 AM, Eric V. Smith wrote: > On 6/20/2013 9:40 AM, Nick Coghlan wrote: > > On 20 June 2013 23:29, Christian Heimes wrote: > >> Am 20.06.2013 15:21, schrieb Florent: > >>> we already have "_pyio.py", we could have "_pystat.py". > >> > >> Works for me. > > > > I suggest following the guidelines in PEP 399 for cross implementation > > compatibility of the standard library: > > http://www.python.org/dev/peps/pep-0399/#details > > > > 1. Keep stat.py > > 2. Make the C version "_stat" > > 3. Add the following to the end of stat.py: > > > > try: > > from _stat import * > > except ImportError: > > pass > > > > Run the tests with and without the C module in the test suite (again, > > as per PEP 399). > > Agreed with the above. > > But isn't the real problem with this module in Python the fact that the > constants might be wrong? I'm not sure what, if anything, we can do > about that. > There isn't anything we can do beyond at least trying to provide reasonable defaults when something better isn't available (which is what the stats module has done all this time). It might make testing difficult for the Python code when the C code has the right values, but I don't think it's necessarily worth tossing out the Python code as backup. -------------- next part -------------- An HTML attachment was scrubbed... URL: From thomas at python.org Thu Jun 20 17:04:56 2013 From: thomas at python.org (Thomas Wouters) Date: Thu, 20 Jun 2013 08:04:56 -0700 Subject: [Python-Dev] stat module in C -- what to do with stat.py? In-Reply-To: References: <51C2FE1E.8020801@python.org> <51C303C6.8050609@python.org> <51C30E41.9090301@trueblade.com> Message-ID: On Thu, Jun 20, 2013 at 8:01 AM, Brett Cannon wrote: > > > > On Thu, Jun 20, 2013 at 10:14 AM, Eric V. Smith wrote: > >> On 6/20/2013 9:40 AM, Nick Coghlan wrote: >> > On 20 June 2013 23:29, Christian Heimes wrote: >> >> Am 20.06.2013 15:21, schrieb Florent: >> >>> we already have "_pyio.py", we could have "_pystat.py". >> >> >> >> Works for me. >> > >> > I suggest following the guidelines in PEP 399 for cross implementation >> > compatibility of the standard library: >> > http://www.python.org/dev/peps/pep-0399/#details >> > >> > 1. Keep stat.py >> > 2. Make the C version "_stat" >> > 3. Add the following to the end of stat.py: >> > >> > try: >> > from _stat import * >> > except ImportError: >> > pass >> > >> > Run the tests with and without the C module in the test suite (again, >> > as per PEP 399). >> >> Agreed with the above. >> >> But isn't the real problem with this module in Python the fact that the >> constants might be wrong? I'm not sure what, if anything, we can do >> about that. >> > > There isn't anything we can do beyond at least trying to provide > reasonable defaults when something better isn't available (which is what > the stats module has done all this time). It might make testing difficult > for the Python code when the C code has the right values, but I don't think > it's necessarily worth tossing out the Python code as backup. > If the .py file is going to be wrong or incomplete, why would we want to keep it -- or use it as fallback -- at all? If we're dead set on having a .py file instead of requiring it to be part of the interpreter (whichever that is, however it was built), it should be generated as part of the build process. Personally, I don't see the value in it; other implementations will need to do *something* special to use it anyway. -- Thomas Wouters Hi! I'm an email virus! Think twice before sending your email to help me spread! -------------- next part -------------- An HTML attachment was scrubbed... URL: From dholth at gmail.com Thu Jun 20 17:09:03 2013 From: dholth at gmail.com (Daniel Holth) Date: Thu, 20 Jun 2013 11:09:03 -0400 Subject: [Python-Dev] stat module in C -- what to do with stat.py? In-Reply-To: References: <51C2FE1E.8020801@python.org> <51C303C6.8050609@python.org> <51C30E41.9090301@trueblade.com> Message-ID: cffi makes this kind of constant-grabbing very easy. However this is a tiny module so no problem with having a C version. On Thu, Jun 20, 2013 at 11:04 AM, Thomas Wouters wrote: > > > > On Thu, Jun 20, 2013 at 8:01 AM, Brett Cannon wrote: >> >> >> >> >> On Thu, Jun 20, 2013 at 10:14 AM, Eric V. Smith >> wrote: >>> >>> On 6/20/2013 9:40 AM, Nick Coghlan wrote: >>> > On 20 June 2013 23:29, Christian Heimes wrote: >>> >> Am 20.06.2013 15:21, schrieb Florent: >>> >>> we already have "_pyio.py", we could have "_pystat.py". >>> >> >>> >> Works for me. >>> > >>> > I suggest following the guidelines in PEP 399 for cross implementation >>> > compatibility of the standard library: >>> > http://www.python.org/dev/peps/pep-0399/#details >>> > >>> > 1. Keep stat.py >>> > 2. Make the C version "_stat" >>> > 3. Add the following to the end of stat.py: >>> > >>> > try: >>> > from _stat import * >>> > except ImportError: >>> > pass >>> > >>> > Run the tests with and without the C module in the test suite (again, >>> > as per PEP 399). >>> >>> Agreed with the above. >>> >>> But isn't the real problem with this module in Python the fact that the >>> constants might be wrong? I'm not sure what, if anything, we can do >>> about that. >> >> >> There isn't anything we can do beyond at least trying to provide >> reasonable defaults when something better isn't available (which is what the >> stats module has done all this time). It might make testing difficult for >> the Python code when the C code has the right values, but I don't think it's >> necessarily worth tossing out the Python code as backup. > > > If the .py file is going to be wrong or incomplete, why would we want to > keep it -- or use it as fallback -- at all? If we're dead set on having a > .py file instead of requiring it to be part of the interpreter (whichever > that is, however it was built), it should be generated as part of the build > process. Personally, I don't see the value in it; other implementations will > need to do *something* special to use it anyway. > > -- > Thomas Wouters > > Hi! I'm an email virus! Think twice before sending your email to help me > spread! > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/dholth%40gmail.com > From cf.natali at gmail.com Thu Jun 20 17:11:05 2013 From: cf.natali at gmail.com (=?ISO-8859-1?Q?Charles=2DFran=E7ois_Natali?=) Date: Thu, 20 Jun 2013 17:11:05 +0200 Subject: [Python-Dev] stat module in C -- what to do with stat.py? In-Reply-To: References: <51C2FE1E.8020801@python.org> <51C303C6.8050609@python.org> <51C30E41.9090301@trueblade.com> Message-ID: 2013/6/20 Thomas Wouters : > If the .py file is going to be wrong or incomplete, why would we want to > keep it -- or use it as fallback -- at all? If we're dead set on having a > .py file instead of requiring it to be part of the interpreter (whichever > that is, however it was built), it should be generated as part of the build > process. Personally, I don't see the value in it; other implementations will > need to do *something* special to use it anyway. That's exactly my rationale for pushing for removal. cf From benjamin at python.org Thu Jun 20 17:35:22 2013 From: benjamin at python.org (Benjamin Peterson) Date: Thu, 20 Jun 2013 08:35:22 -0700 Subject: [Python-Dev] stat module in C -- what to do with stat.py? In-Reply-To: References: <51C2FE1E.8020801@python.org> <51C303C6.8050609@python.org> <51C30E41.9090301@trueblade.com> Message-ID: 2013/6/20 Charles-Fran?ois Natali : > 2013/6/20 Thomas Wouters : >> If the .py file is going to be wrong or incomplete, why would we want to >> keep it -- or use it as fallback -- at all? If we're dead set on having a >> .py file instead of requiring it to be part of the interpreter (whichever >> that is, however it was built), it should be generated as part of the build >> process. Personally, I don't see the value in it; other implementations will >> need to do *something* special to use it anyway. > > That's exactly my rationale for pushing for removal. +1 to nixing it. -- Regards, Benjamin From christian at python.org Thu Jun 20 20:16:23 2013 From: christian at python.org (Christian Heimes) Date: Thu, 20 Jun 2013 20:16:23 +0200 Subject: [Python-Dev] stat module in C -- what to do with stat.py? In-Reply-To: References: <51C2FE1E.8020801@python.org> <51C303C6.8050609@python.org> <51C30E41.9090301@trueblade.com> Message-ID: <51C346F7.3080302@python.org> Am 20.06.2013 17:35, schrieb Benjamin Peterson: > 2013/6/20 Charles-Fran?ois Natali : >> 2013/6/20 Thomas Wouters : >>> If the .py file is going to be wrong or incomplete, why would we want to >>> keep it -- or use it as fallback -- at all? If we're dead set on having a >>> .py file instead of requiring it to be part of the interpreter (whichever >>> that is, however it was built), it should be generated as part of the build >>> process. Personally, I don't see the value in it; other implementations will >>> need to do *something* special to use it anyway. >> >> That's exactly my rationale for pushing for removal. > > +1 to nixing it. I'm +0 for removal. The stat module contains merely constants, wrappers for macros and one simple functions. Somebody just say the word. Christian From storchaka at gmail.com Thu Jun 20 21:16:08 2013 From: storchaka at gmail.com (Serhiy Storchaka) Date: Thu, 20 Jun 2013 22:16:08 +0300 Subject: [Python-Dev] stat module in C -- what to do with stat.py? In-Reply-To: <51C2FE1E.8020801@python.org> References: <51C2FE1E.8020801@python.org> Message-ID: 20.06.13 16:05, Christian Heimes ???????(??): > I have re-implemented the entire stat module in C. [1] It's a necessary > step to fix portability issues. For most constants POSIX standards > dictate only the name of the constant and its meaning but not its value. > Only the values of permission bits (0644 == rw-r--r--) have fixed values. > > For example S_IFDIR is usually 0o40000 but it might be 0o20000 on some > platforms. Common file types seem to have the same value on all > important platforms. It's the only reason we were able to get away with > stat.py. But uncommon file types like door, event port and whiteout > don't have sensible default values. The C implementation is able to pick > up the platform's values easily. > > What shall I do about stat.py? statmodule.c implements all features of > stat.py so there is no point in using it in CPython. It's one Python > file less to load on every startup. However it might still be useful to > alternative implementations of Python such as Jython or PyPy. > > 1) keep the file stat.py but let it be shadowed by the builtin stat > module. Antoine loathes my hack... > 2) rename stat.py to _stat.py > 3) remove stat.py > > Opinions? Now with enumerations in the stdlib the stat module constants are candidates for flag enumerations. How easy will be implement it on C? From lukasz at langa.pl Thu Jun 20 22:05:19 2013 From: lukasz at langa.pl (=?utf-8?Q?=C5=81ukasz_Langa?=) Date: Thu, 20 Jun 2013 22:05:19 +0200 Subject: [Python-Dev] How about we extend C3 to support ABCs? Message-ID: <54088A4F-99E9-4B95-B3D3-A3A9D079B413@langa.pl> Review and comments on the ticket needed, please. http://bugs.python.org/issue18244#msg191535 -- Best regards, ?ukasz Langa WWW: http://lukasz.langa.pl/ Twitter: @llanga IRC: ambv on #python-dev -------------- next part -------------- An HTML attachment was scrubbed... URL: From amauryfa at gmail.com Thu Jun 20 22:18:54 2013 From: amauryfa at gmail.com (Amaury Forgeot d'Arc) Date: Thu, 20 Jun 2013 22:18:54 +0200 Subject: [Python-Dev] stat module in C -- what to do with stat.py? In-Reply-To: References: <51C2FE1E.8020801@python.org> Message-ID: 2013/6/20 Serhiy Storchaka > Now with enumerations in the stdlib the stat module constants are > candidates for flag enumerations. How easy will be implement it on C? Aha. Should an internal C module fetch the value of the constants, and a public stat.py nicely wrap them in enums? -- Amaury Forgeot d'Arc -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Thu Jun 20 22:27:16 2013 From: guido at python.org (Guido van Rossum) Date: Thu, 20 Jun 2013 13:27:16 -0700 Subject: [Python-Dev] stat module in C -- what to do with stat.py? In-Reply-To: References: <51C2FE1E.8020801@python.org> Message-ID: But aren't most of these masks? enum,IntEnum doesn't handle those very gracefully (the | operator returns a plain int). On Thu, Jun 20, 2013 at 1:18 PM, Amaury Forgeot d'Arc wrote: > 2013/6/20 Serhiy Storchaka >> >> Now with enumerations in the stdlib the stat module constants are >> candidates for flag enumerations. How easy will be implement it on C? > > > Aha. Should an internal C module fetch the value of the constants, and a > public stat.py nicely wrap them in enums? > > > -- > Amaury Forgeot d'Arc > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/guido%40python.org > -- --Guido van Rossum (python.org/~guido) From ethan at stoneleaf.us Thu Jun 20 22:24:48 2013 From: ethan at stoneleaf.us (Ethan Furman) Date: Thu, 20 Jun 2013 13:24:48 -0700 Subject: [Python-Dev] stat module in C -- what to do with stat.py? In-Reply-To: References: <51C2FE1E.8020801@python.org> Message-ID: <51C36510.5060305@stoneleaf.us> On 06/20/2013 01:18 PM, Amaury Forgeot d'Arc wrote: > 2013/6/20 Serhiy Storchaka wrote: >> >> Now with enumerations in the stdlib the stat module constants are candidates for flag enumerations. How easy will be >> implement it on C? > > Aha. Should an internal C module fetch the value of the constants, and a public stat.py nicely wrap them in enums? That's pretty much the approach I'm thinking about for sockets. -- ~Ethan~ From victor.stinner at gmail.com Thu Jun 20 22:52:22 2013 From: victor.stinner at gmail.com (Victor Stinner) Date: Thu, 20 Jun 2013 22:52:22 +0200 Subject: [Python-Dev] PEP 445: Add new APIs to customize Python memory allocators (second round) In-Reply-To: References: <20130620141648.0dbedfc1@pitrou.net> Message-ID: 2013/6/20 Antoine Pitrou >: > Victor Stinner > a ?crit : >> Changes: >> >> * add PyMemAllocatorDomain enum: PYALLOC_PYMEM_RAW, PYALLOC_PYMEM or >> PYALLOC_PYOBJECT > > PYMEM_DOMAIN_RAW? I prefer your suggestion because it shares the PYMEM/PyMem prefix with PyMem_SetAllocator(). So the whole list would be: - PYMEM_DOMAIN_RAW - PYMEM_DOMAIN_MEM - PYMEM_DOMAIN_OBJ Victor -------------- next part -------------- An HTML attachment was scrubbed... URL: From victor.stinner at gmail.com Thu Jun 20 22:53:04 2013 From: victor.stinner at gmail.com (Victor Stinner) Date: Thu, 20 Jun 2013 22:53:04 +0200 Subject: [Python-Dev] stat module in C -- what to do with stat.py? In-Reply-To: <51C30E41.9090301@trueblade.com> References: <51C2FE1E.8020801@python.org> <51C303C6.8050609@python.org> <51C30E41.9090301@trueblade.com> Message-ID: 2013/6/20 Eric V. Smith >: > But isn't the real problem with this module in Python the fact that the > constants might be wrong? I'm not sure what, if anything, we can do > about that. Python is providing a stat module implemented in Python since 10 years, or maybe 20 years, and I don't remember that someone complained that constants are wrong. At the same time, Python distributes IN.py which contains things like that: LONG_MAX = 9223372036854775807 LONG_MAX = 2147483647 LONG_MIN = (-LONG_MAX - 1) In my opinion, the situation of plat-*/*.py modules is worse than stat.py :-) Victor -------------- next part -------------- An HTML attachment was scrubbed... URL: From eric at trueblade.com Thu Jun 20 23:55:22 2013 From: eric at trueblade.com (Eric V. Smith) Date: Thu, 20 Jun 2013 17:55:22 -0400 Subject: [Python-Dev] stat module in C -- what to do with stat.py? In-Reply-To: References: <51C2FE1E.8020801@python.org> <51C303C6.8050609@python.org> <51C30E41.9090301@trueblade.com> Message-ID: <51C37A4A.3050102@trueblade.com> On 6/20/2013 4:53 PM, Victor Stinner wrote: > 2013/6/20 Eric V. Smith >: >> But isn't the real problem with this module in Python the fact that the >> constants might be wrong? I'm not sure what, if anything, we can do >> about that. > > Python is providing a stat module implemented in Python since 10 years, > or maybe 20 years, and I don't remember that someone complained that > constants are wrong. This is serious, not argumentative: If there's really no concern that the values be correct, then why implement it in C? Let's just leave the python version with the hard-coded, usually-correct values. My personal opinion is that having the correct values in C is the way to go, and delete stat.py. Eric. From victor.stinner at gmail.com Fri Jun 21 00:18:16 2013 From: victor.stinner at gmail.com (Victor Stinner) Date: Fri, 21 Jun 2013 00:18:16 +0200 Subject: [Python-Dev] stat module in C -- what to do with stat.py? In-Reply-To: <51C37A4A.3050102@trueblade.com> References: <51C2FE1E.8020801@python.org> <51C303C6.8050609@python.org> <51C30E41.9090301@trueblade.com> <51C37A4A.3050102@trueblade.com> Message-ID: 2013/6/20 Eric V. Smith : > This is serious, not argumentative: If there's really no concern that > the values be correct, then why implement it in C? I read again the issue. The problem is to add new flags. Current flags (type: block device/symlink/..., file mode) are well defined and portable, whereas new flags are usually specific to an OS and no standardize yet. Examples: - Solaris: "door" - Solaris: "event port" - OSX/Darwin/FreeBSD: "whiteout" The other problem is that values of the new flags are not portable. Martin v. L?wis wrote: "Looking in more detail: for the S_IFMT flags, OSX/Darwin/FreeBSD defines 0xe000 as S_IFWHT (whiteout), but Solaris defines it as S_IFPORT (event port)." A C module reusing existing S_ISxxx() macros is the obvious way to solve this issue. Victor From ethan at stoneleaf.us Fri Jun 21 00:19:55 2013 From: ethan at stoneleaf.us (Ethan Furman) Date: Thu, 20 Jun 2013 15:19:55 -0700 Subject: [Python-Dev] stat module in C -- what to do with stat.py? In-Reply-To: References: <51C2FE1E.8020801@python.org> Message-ID: <51C3800B.3060404@stoneleaf.us> On 06/20/2013 01:27 PM, Guido van Rossum wrote: > On Thu, Jun 20, 2013 at 1:18 PM, Amaury Forgeot d'Arc wrote: >> 2013/6/20 Serhiy Storchaka wrote: >>> >>> Now with enumerations in the stdlib the stat module constants are >>> candidates for flag enumerations. How easy will be implement it on C? >> >> Aha. Should an internal C module fetch the value of the constants, and a >> public stat.py nicely wrap them in enums? > > But aren't most of these masks? enum.IntEnum doesn't handle those very > gracefully (the | operator returns a plain int). As is, IntEnum would not be a great choice. We could either add a another Enum (BitMaskEnum?) that was also int-based, or flesh out IntEnum with enough smarts to handle |, &, and ^ and return an IntEnum when possible and fall back to a plain int when not possible. -- ~Ethan~ From victor.stinner at gmail.com Fri Jun 21 00:58:44 2013 From: victor.stinner at gmail.com (Victor Stinner) Date: Fri, 21 Jun 2013 00:58:44 +0200 Subject: [Python-Dev] stat module in C -- what to do with stat.py? In-Reply-To: References: <51C2FE1E.8020801@python.org> Message-ID: 2013/6/20 Serhiy Storchaka : > Now with enumerations in the stdlib the stat module constants are candidates > for flag enumerations. How easy will be implement it on C? Numerical values are less important than S_ISxxx() macros. Example: #define S_ISDOOR(mode) (((mode)&0xF000) == 0xd000) 0xd000 is (stat.S_IFSOCK + stat.S_IFIFO). And how do you represent the file mode with enums? I don't think that enum should be used in the stat module. I would prefer a stat object with methods than having to calls low-level functions. Something like: os.stat("document.txt").st_mode.is_reg() versus stat.S_ISREG(os.stat("document.txt").st_mode) The idea was discussed in http://bugs.python.org/issue11406 to solve a real API design issue. How should os.scandir() return the "is a directory" information with a portable API. I'm not saying that stat.S_ISREG should go away. The two approaches are complementary. Victor From ncoghlan at gmail.com Fri Jun 21 03:06:24 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 21 Jun 2013 11:06:24 +1000 Subject: [Python-Dev] stat module in C -- what to do with stat.py? In-Reply-To: References: <51C2FE1E.8020801@python.org> <51C303C6.8050609@python.org> <51C30E41.9090301@trueblade.com> Message-ID: On 21 June 2013 01:04, Thomas Wouters wrote: > If the .py file is going to be wrong or incomplete, why would we want to > keep it -- or use it as fallback -- at all? If we're dead set on having a > .py file instead of requiring it to be part of the interpreter (whichever > that is, however it was built), it should be generated as part of the build > process. Personally, I don't see the value in it; other implementations will > need to do *something* special to use it anyway. Because practicality beats purity. This "wrong" Python code has been good enough for all Python version up until 3.4, it makes sense to keep it as a fallback instead of throwing it away. As Daniel notes, it also means PyPy can just have a small cffi module that adds (or overwrites) the platform specific constants, instead of having to reimplement the whole module. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From rosuav at gmail.com Fri Jun 21 09:16:01 2013 From: rosuav at gmail.com (Chris Angelico) Date: Fri, 21 Jun 2013 17:16:01 +1000 Subject: [Python-Dev] stat module in C -- what to do with stat.py? In-Reply-To: References: <51C2FE1E.8020801@python.org> <51C303C6.8050609@python.org> <51C30E41.9090301@trueblade.com> Message-ID: On Fri, Jun 21, 2013 at 11:06 AM, Nick Coghlan wrote: > On 21 June 2013 01:04, Thomas Wouters wrote: >> If the .py file is going to be wrong or incomplete, why would we want to >> keep it -- or use it as fallback -- at all? If we're dead set on having a >> .py file instead of requiring it to be part of the interpreter (whichever >> that is, however it was built), it should be generated as part of the build >> process. Personally, I don't see the value in it; other implementations will >> need to do *something* special to use it anyway. > > Because practicality beats purity. This "wrong" Python code has been > good enough for all Python version up until 3.4, it makes sense to > keep it as a fallback instead of throwing it away. How would you know if the Python you're running on has an incorrect bitflag? If the "wrong" code is simply incomplete (it has the standard flags but none of the contended ones), that would at least be safe - you'll never get a false flag, but you might be unable to recognize the platform-specific ones. And then the platform-specific modules would always be adding, never overwriting, bitflags. ChrisA From victor.stinner at gmail.com Fri Jun 21 09:25:35 2013 From: victor.stinner at gmail.com (Victor Stinner) Date: Fri, 21 Jun 2013 09:25:35 +0200 Subject: [Python-Dev] stat module in C -- what to do with stat.py? In-Reply-To: References: <51C2FE1E.8020801@python.org> <51C303C6.8050609@python.org> <51C30E41.9090301@trueblade.com> Message-ID: 2013/6/21 Nick Coghlan : > Because practicality beats purity. This "wrong" Python code has been > good enough for all Python version up until 3.4, it makes sense to > keep it as a fallback instead of throwing it away. How do you plan to handle the following case in Python? "Looking in more detail: for the S_IFMT flags, OSX/Darwin/FreeBSD defines 0xe000 as S_IFWHT (whiteout), but Solaris defines it as S_IFPORT (event port)." We may keep the Python module if it is kept unchanged, but the Python and C modules should have the same public API (the C module should not have more features). Or should play with "if sys.platform == ..."? Victor From hrvoje.niksic at avl.com Fri Jun 21 10:42:47 2013 From: hrvoje.niksic at avl.com (Hrvoje Niksic) Date: Fri, 21 Jun 2013 08:42:47 -0000 Subject: [Python-Dev] PEP 409 and the stdlib In-Reply-To: References: <519A2149.3040903@stoneleaf.us> <519A2F37.6020406@stoneleaf.us> <20130520145004.1B1FA250BCF@webabinitio.net> <519A3DFC.5090705@stoneleaf.us> <519B1F89.4030003@avl.com> Message-ID: <519B3DE3.8020508@avl.com> On 05/21/2013 10:36 AM, Serhiy Storchaka wrote: > 21.05.13 10:17, Hrvoje Niksic ???????(??): >> On 05/20/2013 05:15 PM, Ethan Furman wrote: >>> 1) Do nothing and be happy I use 'raise ... from None' in my own >>> libraries >>> >>> 2) Change the wording of 'During handling of the above exception, >>> another exception occurred' (no ideas as to what at >>> the moment) >> >> The word "occurred" misleads one to think that, during handling of the >> real exception, an unrelated and unintended exception occurred. This is >> not the case when the "raise" keyword is used. In that case, the >> exception was intentionally *converted* from one type to another. For >> the "raise" case a wording like the following might work better: >> >> The above exception was converted to the following exception: >> ... >> >> That makes it clear that the conversion was explicit and (hopefully) >> intentional, and that the latter exception supersedes the former. > > How do you distinguish intentional and unintentional exceptions? By the use of the "raise" keyword. Given the code: try: x = bla['foo'] except KeyError: raise BusinessError(...) ...explicit raise is a giveaway that the exception replacement was quite intentional. Hrvoje From ncoghlan at gmail.com Fri Jun 21 13:39:10 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 21 Jun 2013 21:39:10 +1000 Subject: [Python-Dev] stat module in C -- what to do with stat.py? In-Reply-To: References: <51C2FE1E.8020801@python.org> <51C303C6.8050609@python.org> <51C30E41.9090301@trueblade.com> Message-ID: On 21 June 2013 17:25, Victor Stinner wrote: > 2013/6/21 Nick Coghlan : >> Because practicality beats purity. This "wrong" Python code has been >> good enough for all Python version up until 3.4, it makes sense to >> keep it as a fallback instead of throwing it away. > > How do you plan to handle the following case in Python? > > "Looking in more detail: for the S_IFMT flags, OSX/Darwin/FreeBSD defines > 0xe000 as S_IFWHT (whiteout), but Solaris defines it as > S_IFPORT (event port)." > > We may keep the Python module if it is kept unchanged, but the Python > and C modules should have the same public API (the C module should not > have more features). I think it's OK to expose additional platform specific features in the C version, and have them fail cleanly with the pure Python version (rather than silently giving the wrong answer). What's not OK is for the standard library to regress for other implementations just because we added a C implementation for the benefit of CPython. That's exactly the kind of thing PEP 399 says we *won't* do. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From solipsis at pitrou.net Fri Jun 21 13:45:50 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 21 Jun 2013 13:45:50 +0200 Subject: [Python-Dev] stat module in C -- what to do with stat.py? References: <51C2FE1E.8020801@python.org> <51C303C6.8050609@python.org> <51C30E41.9090301@trueblade.com> Message-ID: <20130621134550.1021345e@pitrou.net> Le Fri, 21 Jun 2013 21:39:10 +1000, Nick Coghlan a ?crit : > On 21 June 2013 17:25, Victor Stinner > wrote: > > 2013/6/21 Nick Coghlan : > >> Because practicality beats purity. This "wrong" Python code has > >> been good enough for all Python version up until 3.4, it makes > >> sense to keep it as a fallback instead of throwing it away. > > > > How do you plan to handle the following case in Python? > > > > "Looking in more detail: for the S_IFMT flags, OSX/Darwin/FreeBSD > > defines 0xe000 as S_IFWHT (whiteout), but Solaris defines it as > > S_IFPORT (event port)." > > > > We may keep the Python module if it is kept unchanged, but the > > Python and C modules should have the same public API (the C module > > should not have more features). > > I think it's OK to expose additional platform specific features in the > C version, and have them fail cleanly with the pure Python version > (rather than silently giving the wrong answer). PEP 399 says we don't do it: "Acting as a drop-in replacement also dictates that no public API be provided in accelerated code that does not exist in the pure Python code. Without this requirement people could accidentally come to rely on a detail in the accelerated code which is not made available to other VMs that use the pure Python implementation." > What's not OK is for > the standard library to regress for other implementations just because > we added a C implementation for the benefit of CPython. That's exactly > the kind of thing PEP 399 says we *won't* do. For me, PEP 399 should not be considered a requirement but a guideline. stat.py is algorithmically trivial and I don't think it saves much work for authors of third-party Python implementations. Regards Antoine. From christian at python.org Fri Jun 21 14:21:13 2013 From: christian at python.org (Christian Heimes) Date: Fri, 21 Jun 2013 14:21:13 +0200 Subject: [Python-Dev] stat module in C -- what to do with stat.py? In-Reply-To: <20130621134550.1021345e@pitrou.net> References: <51C2FE1E.8020801@python.org> <51C303C6.8050609@python.org> <51C30E41.9090301@trueblade.com> <20130621134550.1021345e@pitrou.net> Message-ID: <51C44539.70508@python.org> Am 21.06.2013 13:45, schrieb Antoine Pitrou: > For me, PEP 399 should not be considered a requirement but a guideline. > stat.py is algorithmically trivial and I don't think it saves much work > for authors of third-party Python implementations. The module's content is rather boring. It's just a bunch of constants, some wrapper functions for macros and one trivial function that turns st_mode into "-rwxr-xr-x" string. In my opinion stat module falls into the same line as the errno module. The latter is also implemented in C in order to pull a bunch of constants from header files. I have yet another argument in favor of a C implementation. POSIX standards won't gain any new S_IF* numeric constants for new files types. Some file types such as mq, semaphore, shmem and typed memory from POSIX real time extensions aren't testable with S_IF*, too. (But I don't know any platform that implements them as files, though). http://pubs.opengroup.org/onlinepubs/9699919799/basedefs/sys_stat.h.html#tag_13_61_06 Christian From eric at trueblade.com Fri Jun 21 14:33:43 2013 From: eric at trueblade.com (Eric V. Smith) Date: Fri, 21 Jun 2013 08:33:43 -0400 Subject: [Python-Dev] stat module in C -- what to do with stat.py? In-Reply-To: References: <51C2FE1E.8020801@python.org> <51C303C6.8050609@python.org> <51C30E41.9090301@trueblade.com> Message-ID: <51C44827.7030908@trueblade.com> On 6/21/2013 7:39 AM, Nick Coghlan wrote: > On 21 June 2013 17:25, Victor Stinner wrote: >> 2013/6/21 Nick Coghlan : >>> Because practicality beats purity. This "wrong" Python code has been >>> good enough for all Python version up until 3.4, it makes sense to >>> keep it as a fallback instead of throwing it away. >> >> How do you plan to handle the following case in Python? >> >> "Looking in more detail: for the S_IFMT flags, OSX/Darwin/FreeBSD defines >> 0xe000 as S_IFWHT (whiteout), but Solaris defines it as >> S_IFPORT (event port)." >> >> We may keep the Python module if it is kept unchanged, but the Python >> and C modules should have the same public API (the C module should not >> have more features). > > I think it's OK to expose additional platform specific features in the > C version, and have them fail cleanly with the pure Python version > (rather than silently giving the wrong answer). What's not OK is for > the standard library to regress for other implementations just because > we added a C implementation for the benefit of CPython. That's exactly > the kind of thing PEP 399 says we *won't* do. I was just writing up something similar. But as always, Nick said it better than me. -- Eric. From ani at aristanetworks.com Fri Jun 21 07:18:27 2013 From: ani at aristanetworks.com (Ani Sinha) Date: Thu, 20 Jun 2013 22:18:27 -0700 Subject: [Python-Dev] [python 2.7] a question about exporting a new method to socket object Message-ID: Hi python devs : As a part of exploring an idea, I am trying to add a new method to the python socket object. Here's what I have been trying to do (the patch may be whitespace damaged so please bear with me) : Index: Python-2.7/Modules/socketmodule.c =================================================================== --- Python-2.7.orig/Modules/socketmodule.c +++ Python-2.7/Modules/socketmodule.c @@ -126,6 +126,7 @@ makefile([mode, [bufsize]]) -- return a recv(buflen[, flags]) -- receive data\n\ recv_into(buffer[, nbytes[, flags]]) -- receive data (into a buffer)\n\ recvfrom(buflen[, flags]) -- receive data and sender\'s address\n\ +arecvfrom(buflen[, flags]) -- same as recvfrom \n\ recvfrom_into(buffer[, nbytes, [, flags])\n\ -- receive data and sender\'s address (into a buffer)\n\ sendall(data[, flags]) -- send all data\n\ @@ -468,6 +469,13 @@ static PyTypeObject sock_type; #define IS_SELECTABLE(s) ((s)->sock_fd < FD_SETSIZE || s->sock_timeout <= 0.0) #endif @@ -2620,6 +2802,58 @@ PyDoc_STRVAR(recvfrom_doc, \n\ Like recv(buffersize, flags) but also return the sender's address info."); +static PyObject * +sock_arecvfrom(PySocketSockObject *s, PyObject *args) +{ + return sock_recvfrom(s,args); +} + +PyDoc_STRVAR(arecvfrom_doc, +"arecvfrom(buffersize[, flags]) -> (data, address info)\n\ +\n\ + experimental stuff"); /* s.recvfrom_into(buffer[, nbytes [,flags]]) method */ @@ -2963,6 +3197,8 @@ static PyMethodDef sock_methods[] = { recv_into_doc}, {"recvfrom", (PyCFunction)sock_recvfrom, METH_VARARGS, recvfrom_doc}, + {"arecvfrom", (PyCFunction)sock_arecvfrom, METH_VARARGS, + arecvfrom_doc}, {"recvfrom_into", (PyCFunction)sock_recvfrom_into, METH_VARARGS | METH_KEYWORDS, recvfrom_into_doc}, {"send", (PyCFunction)sock_send, METH_VARARGS, When I compile this and run a simple script like the following that uses arecvfrom() : sock = socket.socket( socket.PF_PACKET, socket.SOCK_RAW ) sock.bind( ( intf, ETH_P_ALL ) ) (pkt,dontcare) = dst.arecvfrom( 500, socket.MSG_DONTWAIT ) I get this exception : AttributeError: '_socketobject' object has no attribute 'arecvfrom' So what am I doing wrong? How do I export this new socket method? any help/pointer will be greatly appreciated. cheers, ani -- Ani -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Fri Jun 21 17:07:04 2013 From: guido at python.org (Guido van Rossum) Date: Fri, 21 Jun 2013 08:07:04 -0700 Subject: [Python-Dev] [python 2.7] a question about exporting a new method to socket object In-Reply-To: References: Message-ID: Technically this list isn't the right place, but you probably are bitten by the duplication of functionality in Lib/socket.py. There's a list of methods at the top of that file, _socketmethods. But you really shouldn't be asking here -- this list is for changes to Python, not using or hacking it. (Try StackOverflow for that.) --Guido On Thu, Jun 20, 2013 at 10:18 PM, Ani Sinha wrote: > Hi python devs : > > As a part of exploring an idea, I am trying to add a new method to the > python socket object. Here's what I have been trying to do (the patch may be > whitespace damaged so please bear with me) : > > > Index: Python-2.7/Modules/socketmodule.c > =================================================================== > --- Python-2.7.orig/Modules/socketmodule.c > +++ Python-2.7/Modules/socketmodule.c > @@ -126,6 +126,7 @@ makefile([mode, [bufsize]]) -- return a > recv(buflen[, flags]) -- receive data\n\ > recv_into(buffer[, nbytes[, flags]]) -- receive data (into a buffer)\n\ > recvfrom(buflen[, flags]) -- receive data and sender\'s address\n\ > +arecvfrom(buflen[, flags]) -- same as recvfrom \n\ > recvfrom_into(buffer[, nbytes, [, flags])\n\ > -- receive data and sender\'s address (into a buffer)\n\ > sendall(data[, flags]) -- send all data\n\ > @@ -468,6 +469,13 @@ static PyTypeObject sock_type; > #define IS_SELECTABLE(s) ((s)->sock_fd < FD_SETSIZE || s->sock_timeout <= > 0.0) > #endif > > @@ -2620,6 +2802,58 @@ PyDoc_STRVAR(recvfrom_doc, > \n\ > Like recv(buffersize, flags) but also return the sender's address info."); > > +static PyObject * > +sock_arecvfrom(PySocketSockObject *s, PyObject *args) > +{ > + return sock_recvfrom(s,args); > +} > + > +PyDoc_STRVAR(arecvfrom_doc, > +"arecvfrom(buffersize[, flags]) -> (data, address info)\n\ > +\n\ > + experimental stuff"); > > /* s.recvfrom_into(buffer[, nbytes [,flags]]) method */ > > @@ -2963,6 +3197,8 @@ static PyMethodDef sock_methods[] = { > recv_into_doc}, > {"recvfrom", (PyCFunction)sock_recvfrom, METH_VARARGS, > recvfrom_doc}, > + {"arecvfrom", (PyCFunction)sock_arecvfrom, METH_VARARGS, > + arecvfrom_doc}, > {"recvfrom_into", (PyCFunction)sock_recvfrom_into, METH_VARARGS | > METH_KEYWORDS, > recvfrom_into_doc}, > {"send", (PyCFunction)sock_send, METH_VARARGS, > > > When I compile this and run a simple script like the following that uses > arecvfrom() : > > sock = socket.socket( socket.PF_PACKET, socket.SOCK_RAW ) > sock.bind( ( intf, ETH_P_ALL ) ) > > (pkt,dontcare) = dst.arecvfrom( 500, socket.MSG_DONTWAIT ) > > I get this exception : > > AttributeError: '_socketobject' object has no attribute 'arecvfrom' > > So what am I doing wrong? How do I export this new socket method? > > any help/pointer will be greatly appreciated. > > cheers, > ani > > -- > Ani > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/guido%40python.org > -- --Guido van Rossum (python.org/~guido) From python.00 at klix.ch Fri Jun 21 17:06:25 2013 From: python.00 at klix.ch (Gerald Klix) Date: Fri, 21 Jun 2013 17:06:25 +0200 Subject: [Python-Dev] [python 2.7] a question about exporting a new method to socket object In-Reply-To: References: Message-ID: <174A8F55-3226-49C7-A104-F2D62ABEB4B1@klix.ch> Hi Ani, just a simple question: Are you sure, that you imported the modified socket module and not the original module from the distribution? HTH, Gerald Am 21.06.2013 um 07:18 schrieb Ani Sinha : > Hi python devs : > > As a part of exploring an idea, I am trying to add a new method to the > python socket object. Here's what I have been trying to do (the patch may > be whitespace damaged so please bear with me) : > > > Index: Python-2.7/Modules/socketmodule.c > =================================================================== > --- Python-2.7.orig/Modules/socketmodule.c > +++ Python-2.7/Modules/socketmodule.c > @@ -126,6 +126,7 @@ makefile([mode, [bufsize]]) -- return a > recv(buflen[, flags]) -- receive data\n\ > recv_into(buffer[, nbytes[, flags]]) -- receive data (into a buffer)\n\ > recvfrom(buflen[, flags]) -- receive data and sender\'s address\n\ > +arecvfrom(buflen[, flags]) -- same as recvfrom \n\ > recvfrom_into(buffer[, nbytes, [, flags])\n\ > -- receive data and sender\'s address (into a buffer)\n\ > sendall(data[, flags]) -- send all data\n\ > @@ -468,6 +469,13 @@ static PyTypeObject sock_type; > #define IS_SELECTABLE(s) ((s)->sock_fd < FD_SETSIZE || s->sock_timeout <= > 0.0) > #endif > > @@ -2620,6 +2802,58 @@ PyDoc_STRVAR(recvfrom_doc, > \n\ > Like recv(buffersize, flags) but also return the sender's address info."); > > +static PyObject * > +sock_arecvfrom(PySocketSockObject *s, PyObject *args) > +{ > + return sock_recvfrom(s,args); > +} > + > +PyDoc_STRVAR(arecvfrom_doc, > +"arecvfrom(buffersize[, flags]) -> (data, address info)\n\ > +\n\ > + experimental stuff"); > > /* s.recvfrom_into(buffer[, nbytes [,flags]]) method */ > > @@ -2963,6 +3197,8 @@ static PyMethodDef sock_methods[] = { > recv_into_doc}, > {"recvfrom", (PyCFunction)sock_recvfrom, METH_VARARGS, > recvfrom_doc}, > + {"arecvfrom", (PyCFunction)sock_arecvfrom, METH_VARARGS, > + arecvfrom_doc}, > {"recvfrom_into", (PyCFunction)sock_recvfrom_into, METH_VARARGS | > METH_KEYWORDS, > recvfrom_into_doc}, > {"send", (PyCFunction)sock_send, METH_VARARGS, > > > When I compile this and run a simple script like the following that uses > arecvfrom() : > > sock = socket.socket( socket.PF_PACKET, socket.SOCK_RAW ) > sock.bind( ( intf, ETH_P_ALL ) ) > > (pkt,dontcare) = dst.arecvfrom( 500, socket.MSG_DONTWAIT ) > > I get this exception : > > AttributeError: '_socketobject' object has no attribute 'arecvfrom' > > So what am I doing wrong? How do I export this new socket method? > > any help/pointer will be greatly appreciated. > > cheers, > ani > > -- > Ani > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/python.00%40klix.ch From ncoghlan at gmail.com Fri Jun 21 17:29:40 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 22 Jun 2013 01:29:40 +1000 Subject: [Python-Dev] stat module in C -- what to do with stat.py? In-Reply-To: <20130621134550.1021345e@pitrou.net> References: <51C2FE1E.8020801@python.org> <51C303C6.8050609@python.org> <51C30E41.9090301@trueblade.com> <20130621134550.1021345e@pitrou.net> Message-ID: On 21 June 2013 21:45, Antoine Pitrou wrote: > Le Fri, 21 Jun 2013 21:39:10 +1000, > Nick Coghlan a ?crit : >> What's not OK is for >> the standard library to regress for other implementations just because >> we added a C implementation for the benefit of CPython. That's exactly >> the kind of thing PEP 399 says we *won't* do. > > For me, PEP 399 should not be considered a requirement but a guideline. > stat.py is algorithmically trivial and I don't think it saves much work > for authors of third-party Python implementations. So why not just replace the *broken* parts of stat.py and keep the rest of it? Why make pointless work for the other implementations? Basically, I want to hear from the Jython, PyPy and IronPython devs that they're OK with us deleting Lib/stat.py. Hearing other CPython devs say they're fine with it doesn't mean anything, since we're not the ones that will have to do additional work as a result. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From solipsis at pitrou.net Fri Jun 21 17:38:26 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 21 Jun 2013 17:38:26 +0200 Subject: [Python-Dev] stat module in C -- what to do with stat.py? References: <51C2FE1E.8020801@python.org> <51C303C6.8050609@python.org> <51C30E41.9090301@trueblade.com> <20130621134550.1021345e@pitrou.net> Message-ID: <20130621173826.3610dd5b@pitrou.net> Le Sat, 22 Jun 2013 01:29:40 +1000, Nick Coghlan a ?crit : > On 21 June 2013 21:45, Antoine Pitrou wrote: > > Le Fri, 21 Jun 2013 21:39:10 +1000, > > Nick Coghlan a ?crit : > >> What's not OK is for > >> the standard library to regress for other implementations just > >> because we added a C implementation for the benefit of CPython. > >> That's exactly the kind of thing PEP 399 says we *won't* do. > > > > For me, PEP 399 should not be considered a requirement but a > > guideline. stat.py is algorithmically trivial and I don't think it > > saves much work for authors of third-party Python implementations. > > So why not just replace the *broken* parts of stat.py and keep the > rest of it? Why make pointless work for the other implementations? I guess the answer is: because it's more work for us :-) > Basically, I want to hear from the Jython, PyPy and IronPython devs > that they're OK with us deleting Lib/stat.py. Hearing other CPython > devs say they're fine with it doesn't mean anything, since we're not > the ones that will have to do additional work as a result. Yes, I agree with that. Regards Antoine. From benjamin at python.org Fri Jun 21 17:47:17 2013 From: benjamin at python.org (Benjamin Peterson) Date: Fri, 21 Jun 2013 08:47:17 -0700 Subject: [Python-Dev] stat module in C -- what to do with stat.py? In-Reply-To: References: <51C2FE1E.8020801@python.org> <51C303C6.8050609@python.org> <51C30E41.9090301@trueblade.com> <20130621134550.1021345e@pitrou.net> Message-ID: 2013/6/21 Nick Coghlan : > Basically, I want to hear from the Jython, PyPy and IronPython devs > that they're OK with us deleting Lib/stat.py. Hearing other CPython > devs say they're fine with it doesn't mean anything, since we're not > the ones that will have to do additional work as a result. Kill it. I would argue having incorrect constants makes the implementation incompatible with CPython anyway. This not that much work (as long as there are tests that the constants exist at least) to emulate. -- Regards, Benjamin From status at bugs.python.org Fri Jun 21 18:07:31 2013 From: status at bugs.python.org (Python tracker) Date: Fri, 21 Jun 2013 18:07:31 +0200 (CEST) Subject: [Python-Dev] Summary of Python tracker Issues Message-ID: <20130621160731.5407A56A50@psf.upfronthosting.co.za> ACTIVITY SUMMARY (2013-06-14 - 2013-06-21) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 4057 (+15) closed 25993 (+47) total 30050 (+62) Open issues with patches: 1804 Issues opened (42) ================== #18113: Memory leak in curses.panel http://bugs.python.org/issue18113 reopened by serhiy.storchaka #18216: gettext doesn't check MO versions http://bugs.python.org/issue18216 opened by jwilk #18217: Deprecate and remove gettext.install http://bugs.python.org/issue18217 opened by alex #18219: csv.DictWriter is slow when writing files with large number of http://bugs.python.org/issue18219 opened by mtraskin #18220: In itertools.islice() make prototype like in help() http://bugs.python.org/issue18220 opened by py.user #18221: abspath strips trailing spaces on win32 http://bugs.python.org/issue18221 opened by Jeremy.Gray #18222: os.path.abspath should accept multiple path parts and join the http://bugs.python.org/issue18222 opened by ??ukasz.Balcerzak #18224: pyvenv pydoc.py script causing AttributeErrors on Windows http://bugs.python.org/issue18224 opened by peter at psantoro.net #18226: IDLE Unit test for FormatParagrah.py http://bugs.python.org/issue18226 opened by Todd.Rovito #18227: Use Python memory allocators in external libraries like zlib o http://bugs.python.org/issue18227 opened by haypo #18228: AIX locale parsing failure http://bugs.python.org/issue18228 opened by David.Edelsohn #18229: attribute headers of http.server.BaseHTTPRequestHandler someti http://bugs.python.org/issue18229 opened by joru #18230: test_builtin fails/hangs when run after test_getopt http://bugs.python.org/issue18230 opened by brett.cannon #18231: What's new in Python should explain what's new in UCD http://bugs.python.org/issue18231 opened by belopolsky #18232: running a suite with no tests is not an error http://bugs.python.org/issue18232 opened by rbcollins #18233: SSLSocket.getpeercertchain() http://bugs.python.org/issue18233 opened by christian.heimes #18234: Unicodedata module should provide access to codepoint aliases http://bugs.python.org/issue18234 opened by belopolsky #18235: _sysconfigdata.py wrong on AIX installations http://bugs.python.org/issue18235 opened by David.Edelsohn #18236: int() and float() do not accept strings with trailing separato http://bugs.python.org/issue18236 opened by belopolsky #18237: unittest.assertRaisesRegex(p) example is wrong in docs http://bugs.python.org/issue18237 opened by jtratner #18240: hmac unnecessarily restricts input to "bytes" http://bugs.python.org/issue18240 opened by jborgstrom #18241: Add unique option to heapq functions http://bugs.python.org/issue18241 opened by pitrou #18242: IDLE should not be replacing warnings.formatwarning http://bugs.python.org/issue18242 opened by brett.cannon #18243: mktime_tz documentation out-of-date http://bugs.python.org/issue18243 opened by jwilk #18244: singledispatch: When virtual-inheriting ABCs at distinct point http://bugs.python.org/issue18244 opened by ecatmur #18246: tkinter.Text() add a newline to the content - bug? http://bugs.python.org/issue18246 opened by Friedrich.Spee.von.Langenfeld #18249: Incorrect and incomplete help docs for close() method http://bugs.python.org/issue18249 opened by DaveA #18254: Accessing attr dict at definition time of a class from a metac http://bugs.python.org/issue18254 opened by archardlias #18255: CPython setup.py problems http://bugs.python.org/issue18255 opened by ronaldoussoren #18257: Two copies of python-config http://bugs.python.org/issue18257 opened by ronaldoussoren #18258: Fix test discovery for test_codecmaps*.py http://bugs.python.org/issue18258 opened by zach.ware #18260: configparser: TypeError occurs when handling errors in files w http://bugs.python.org/issue18260 opened by Arfrever #18262: ZipInfo.external_attr are not documented http://bugs.python.org/issue18262 opened by techtonik #18264: enum.IntEnum is not compatible with JSON serialisation http://bugs.python.org/issue18264 opened by ncoghlan #18266: Fix test discovery for test_largefile.py http://bugs.python.org/issue18266 opened by zach.ware #18268: ElementTree.fromstring non-deterministically gives unicode tex http://bugs.python.org/issue18268 opened by Brendan.OConnor #18269: Add new parameter format for converter function w/ position nu http://bugs.python.org/issue18269 opened by techtonik #18270: IDLE on OS X fails with Attribute Error if no initial shell an http://bugs.python.org/issue18270 opened by dbackhaus #18273: Simplify calling and discovery of json test package http://bugs.python.org/issue18273 opened by zach.ware #18275: Make isinstance() work with super type instances http://bugs.python.org/issue18275 opened by brett.cannon #18276: posixpath.c:_fd_converter() should use PyObject_AsFileDescript http://bugs.python.org/issue18276 opened by christian.heimes #18277: Queue is empty right after put from the same process/thread http://bugs.python.org/issue18277 opened by shwouchk Most recent 15 issues with no replies (15) ========================================== #18277: Queue is empty right after put from the same process/thread http://bugs.python.org/issue18277 #18273: Simplify calling and discovery of json test package http://bugs.python.org/issue18273 #18266: Fix test discovery for test_largefile.py http://bugs.python.org/issue18266 #18260: configparser: TypeError occurs when handling errors in files w http://bugs.python.org/issue18260 #18258: Fix test discovery for test_codecmaps*.py http://bugs.python.org/issue18258 #18254: Accessing attr dict at definition time of a class from a metac http://bugs.python.org/issue18254 #18246: tkinter.Text() add a newline to the content - bug? http://bugs.python.org/issue18246 #18243: mktime_tz documentation out-of-date http://bugs.python.org/issue18243 #18242: IDLE should not be replacing warnings.formatwarning http://bugs.python.org/issue18242 #18237: unittest.assertRaisesRegex(p) example is wrong in docs http://bugs.python.org/issue18237 #18236: int() and float() do not accept strings with trailing separato http://bugs.python.org/issue18236 #18232: running a suite with no tests is not an error http://bugs.python.org/issue18232 #18230: test_builtin fails/hangs when run after test_getopt http://bugs.python.org/issue18230 #18229: attribute headers of http.server.BaseHTTPRequestHandler someti http://bugs.python.org/issue18229 #18226: IDLE Unit test for FormatParagrah.py http://bugs.python.org/issue18226 Most recent 15 issues waiting for review (15) ============================================= #18273: Simplify calling and discovery of json test package http://bugs.python.org/issue18273 #18266: Fix test discovery for test_largefile.py http://bugs.python.org/issue18266 #18258: Fix test discovery for test_codecmaps*.py http://bugs.python.org/issue18258 #18255: CPython setup.py problems http://bugs.python.org/issue18255 #18244: singledispatch: When virtual-inheriting ABCs at distinct point http://bugs.python.org/issue18244 #18240: hmac unnecessarily restricts input to "bytes" http://bugs.python.org/issue18240 #18237: unittest.assertRaisesRegex(p) example is wrong in docs http://bugs.python.org/issue18237 #18233: SSLSocket.getpeercertchain() http://bugs.python.org/issue18233 #18222: os.path.abspath should accept multiple path parts and join the http://bugs.python.org/issue18222 #18220: In itertools.islice() make prototype like in help() http://bugs.python.org/issue18220 #18219: csv.DictWriter is slow when writing files with large number of http://bugs.python.org/issue18219 #18214: Stop purging modules which are garbage collected before shutdo http://bugs.python.org/issue18214 #18206: license url in site.py should always use X.Y.Z form of version http://bugs.python.org/issue18206 #18203: Replace direct calls to malloc() with PyMem_Malloc() or PyMem_ http://bugs.python.org/issue18203 #18191: urllib2/urllib.parse.splitport does not handle IPv6 correctly http://bugs.python.org/issue18191 Top 10 most discussed issues (10) ================================= #18264: enum.IntEnum is not compatible with JSON serialisation http://bugs.python.org/issue18264 16 msgs #18081: test_logging failure in WarningsTest on buildbots http://bugs.python.org/issue18081 15 msgs #18203: Replace direct calls to malloc() with PyMem_Malloc() or PyMem_ http://bugs.python.org/issue18203 9 msgs #18103: Create a GUI test framework for Idle http://bugs.python.org/issue18103 8 msgs #16499: CLI option for isolated mode http://bugs.python.org/issue16499 7 msgs #18113: Memory leak in curses.panel http://bugs.python.org/issue18113 7 msgs #18153: python imaplib - error 'unexpected repsonse' http://bugs.python.org/issue18153 7 msgs #18162: Add index attribute to IndexError http://bugs.python.org/issue18162 7 msgs #18269: Add new parameter format for converter function w/ position nu http://bugs.python.org/issue18269 7 msgs #13483: Use VirtualAlloc to allocate memory arenas http://bugs.python.org/issue13483 6 msgs Issues closed (43) ================== #14015: surrogateescape largely missing from documentation http://bugs.python.org/issue14015 closed by python-dev #15198: multiprocessing Pipe send of non-picklable objects doesn't rai http://bugs.python.org/issue15198 closed by sbt #16507: Patch selectmodule.c to support WSAPoll on Windows http://bugs.python.org/issue16507 closed by sbt #17018: Inconsistent behaviour of methods waiting for child process http://bugs.python.org/issue17018 closed by sbt #17177: Deprecate imp http://bugs.python.org/issue17177 closed by brett.cannon #17689: Fix test discovery for test_tarfile.py http://bugs.python.org/issue17689 closed by zach.ware #17907: Deprecate imp.new_module() in favour of types.ModuleType http://bugs.python.org/issue17907 closed by brett.cannon #17961: Use enum names as values in enum.Enum() functional API http://bugs.python.org/issue17961 closed by ncoghlan #18014: Problem compiling on Windows, VC++Express 2010 http://bugs.python.org/issue18014 closed by terry.reedy #18057: Register NamespaceLoader with importlib.abc.Loader http://bugs.python.org/issue18057 closed by brett.cannon #18058: Define is_package for NamespaceLoader http://bugs.python.org/issue18058 closed by brett.cannon #18076: Implement importlib.util.decode_source() http://bugs.python.org/issue18076 closed by brett.cannon #18115: Use importlib.util.module_to_load in all loaders in importlib http://bugs.python.org/issue18115 closed by brett.cannon #18147: SSL: diagnostic functions to list loaded CA certs http://bugs.python.org/issue18147 closed by christian.heimes #18149: filecmp.cmp() incorrect results when previously compared file http://bugs.python.org/issue18149 closed by ned.deily #18167: cgi.FieldStorage fails to handle multipart/form-data when \r\n http://bugs.python.org/issue18167 closed by serhiy.storchaka #18192: Move imp.get_magic() to importlib http://bugs.python.org/issue18192 closed by brett.cannon #18193: Move imp.reload() to importlib http://bugs.python.org/issue18193 closed by brett.cannon #18194: Move imp.source_from_cache/cache_from_source to importlib http://bugs.python.org/issue18194 closed by brett.cannon #18197: insufficient error checking causes crash on windows http://bugs.python.org/issue18197 closed by ncoghlan #18202: Minor fixes for test_coding http://bugs.python.org/issue18202 closed by serhiy.storchaka #18207: OpenSSL may ignore seconds in notAfter http://bugs.python.org/issue18207 closed by christian.heimes #18218: In itertools.count() clarify the starting point http://bugs.python.org/issue18218 closed by python-dev #18223: Refactor test_tarfile http://bugs.python.org/issue18223 closed by serhiy.storchaka #18225: ctypes Structure data size is incorrect http://bugs.python.org/issue18225 closed by alex #18238: test_signal.py wait_helper hangs on AIX http://bugs.python.org/issue18238 closed by haypo #18239: In itertools docstring update arguments in count() example http://bugs.python.org/issue18239 closed by python-dev #18245: In itertools.groupby() make data plural http://bugs.python.org/issue18245 closed by r.david.murray #18247: Add Lib/test/data/ to .gitignore http://bugs.python.org/issue18247 closed by python-dev #18248: fficonfig.py.in wrong for AIX http://bugs.python.org/issue18248 closed by python-dev #18250: In itertools.repeat() object shadows object() http://bugs.python.org/issue18250 closed by akuchling #18251: test_subprocess_jy fails when Argument has embedded quote http://bugs.python.org/issue18251 closed by r.david.murray #18252: timeit makes code run faster? http://bugs.python.org/issue18252 closed by r.david.murray #18253: make standard library PEP8 compliant http://bugs.python.org/issue18253 closed by r.david.murray #18256: thread_pthread.h fixes for AIX http://bugs.python.org/issue18256 closed by pitrou #18259: Declare sethostname in socketmodule.c for AIX http://bugs.python.org/issue18259 closed by christian.heimes #18261: Confusing description in Minimal DOM implementation http://bugs.python.org/issue18261 closed by ned.deily #18263: python.man: no documentation for -b, -X http://bugs.python.org/issue18263 closed by orsenthil #18265: typedef lookupfunc defined by not used http://bugs.python.org/issue18265 closed by David.Edelsohn #18267: xmlrpc.client documentation multicall example missleading for http://bugs.python.org/issue18267 closed by python-dev #18271: get_payload method returns bytes which cannot be decoded using http://bugs.python.org/issue18271 closed by r.david.murray #18272: In itertools recipes there is a typo in __builtins__ http://bugs.python.org/issue18272 closed by python-dev #18274: python: not found http://bugs.python.org/issue18274 closed by brett.cannon From tjreedy at udel.edu Fri Jun 21 18:28:57 2013 From: tjreedy at udel.edu (Terry Reedy) Date: Fri, 21 Jun 2013 12:28:57 -0400 Subject: [Python-Dev] stat module in C -- what to do with stat.py? In-Reply-To: <20130621134550.1021345e@pitrou.net> References: <51C2FE1E.8020801@python.org> <51C303C6.8050609@python.org> <51C30E41.9090301@trueblade.com> <20130621134550.1021345e@pitrou.net> Message-ID: On 6/21/2013 7:45 AM, Antoine Pitrou wrote: > Le Fri, 21 Jun 2013 21:39:10 +1000, > Nick Coghlan a ?crit : >> I think it's OK to expose additional platform specific features in the >> C version, and have them fail cleanly with the pure Python version >> (rather than silently giving the wrong answer). > > PEP 399 says we don't do it: > > "Acting as a drop-in replacement also dictates that no public API be > provided in accelerated code that does not exist in the pure Python > code. Without this requirement people could accidentally come to rely > on a detail in the accelerated code which is not made available to > other VMs that use the pure Python implementation." Any C accelerator extensions should by documented as CPython extensions not necessarily available elsewhere. Or the doc should have, in this case, a separate _stat that says "In addition to the above, CPython's _stat, imported by stat, also provides ... -- Terry Jan Reedy From christian at python.org Fri Jun 21 18:29:27 2013 From: christian at python.org (Christian Heimes) Date: Fri, 21 Jun 2013 18:29:27 +0200 Subject: [Python-Dev] stat module in C -- what to do with stat.py? In-Reply-To: References: <51C2FE1E.8020801@python.org> <51C303C6.8050609@python.org> <51C30E41.9090301@trueblade.com> <20130621134550.1021345e@pitrou.net> Message-ID: <51C47F67.2090808@python.org> Am 21.06.2013 17:47, schrieb Benjamin Peterson: > > Kill it. I would argue having incorrect constants makes the > implementation incompatible with CPython anyway. This not that much > work (as long as there are tests that the constants exist at least) to > emulate. > My patch adds extensive tests for all features of the stat module. The test check the existence of attributes and the return value of all functions. A missing or ill-defined attribute is going to raise an error. But see for yourself: http://hg.python.org/cpython/rev/f8ff61f44aca Christian From solipsis at pitrou.net Fri Jun 21 19:28:02 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 21 Jun 2013 19:28:02 +0200 Subject: [Python-Dev] PEP 445 delegate Message-ID: <20130621192802.1f960f40@fsol> Hello, I've been appointed PEP 445 delegate by Nick and Guido. I would like to know if there are still pending changes to the PEP. If not, I expect to give it a review in the coming days or weeks, and then make a final pronouncement (which will probably be positive anyway). Regards Antoine. From ani at aristanetworks.com Fri Jun 21 19:28:20 2013 From: ani at aristanetworks.com (Ani Sinha) Date: Fri, 21 Jun 2013 10:28:20 -0700 Subject: [Python-Dev] [python 2.7] a question about exporting a new method to socket object In-Reply-To: References: Message-ID: Hi Guido : Thanks a lot. That helped. I will try StackOverflow next time. cheers, ani On Fri, Jun 21, 2013 at 8:07 AM, Guido van Rossum wrote: > Technically this list isn't the right place, but you probably are > bitten by the duplication of functionality in Lib/socket.py. There's a > list of methods at the top of that file, _socketmethods. > > But you really shouldn't be asking here -- this list is for changes to > Python, not using or hacking it. (Try StackOverflow for that.) > > --Guido > > On Thu, Jun 20, 2013 at 10:18 PM, Ani Sinha > wrote: > > Hi python devs : > > > > As a part of exploring an idea, I am trying to add a new method to the > > python socket object. Here's what I have been trying to do (the patch > may be > > whitespace damaged so please bear with me) : > > > > > > Index: Python-2.7/Modules/socketmodule.c > > =================================================================== > > --- Python-2.7.orig/Modules/socketmodule.c > > +++ Python-2.7/Modules/socketmodule.c > > @@ -126,6 +126,7 @@ makefile([mode, [bufsize]]) -- return a > > recv(buflen[, flags]) -- receive data\n\ > > recv_into(buffer[, nbytes[, flags]]) -- receive data (into a buffer)\n\ > > recvfrom(buflen[, flags]) -- receive data and sender\'s address\n\ > > +arecvfrom(buflen[, flags]) -- same as recvfrom \n\ > > recvfrom_into(buffer[, nbytes, [, flags])\n\ > > -- receive data and sender\'s address (into a buffer)\n\ > > sendall(data[, flags]) -- send all data\n\ > > @@ -468,6 +469,13 @@ static PyTypeObject sock_type; > > #define IS_SELECTABLE(s) ((s)->sock_fd < FD_SETSIZE || s->sock_timeout > <= > > 0.0) > > #endif > > > > @@ -2620,6 +2802,58 @@ PyDoc_STRVAR(recvfrom_doc, > > \n\ > > Like recv(buffersize, flags) but also return the sender's address > info."); > > > > +static PyObject * > > +sock_arecvfrom(PySocketSockObject *s, PyObject *args) > > +{ > > + return sock_recvfrom(s,args); > > +} > > + > > +PyDoc_STRVAR(arecvfrom_doc, > > +"arecvfrom(buffersize[, flags]) -> (data, address info)\n\ > > +\n\ > > + experimental stuff"); > > > > /* s.recvfrom_into(buffer[, nbytes [,flags]]) method */ > > > > @@ -2963,6 +3197,8 @@ static PyMethodDef sock_methods[] = { > > recv_into_doc}, > > {"recvfrom", (PyCFunction)sock_recvfrom, METH_VARARGS, > > recvfrom_doc}, > > + {"arecvfrom", (PyCFunction)sock_arecvfrom, METH_VARARGS, > > + arecvfrom_doc}, > > {"recvfrom_into", (PyCFunction)sock_recvfrom_into, METH_VARARGS | > > METH_KEYWORDS, > > recvfrom_into_doc}, > > {"send", (PyCFunction)sock_send, METH_VARARGS, > > > > > > When I compile this and run a simple script like the following that uses > > arecvfrom() : > > > > sock = socket.socket( socket.PF_PACKET, socket.SOCK_RAW ) > > sock.bind( ( intf, ETH_P_ALL ) ) > > > > (pkt,dontcare) = dst.arecvfrom( 500, socket.MSG_DONTWAIT ) > > > > I get this exception : > > > > AttributeError: '_socketobject' object has no attribute 'arecvfrom' > > > > So what am I doing wrong? How do I export this new socket method? > > > > any help/pointer will be greatly appreciated. > > > > cheers, > > ani > > > > -- > > Ani > > > > _______________________________________________ > > Python-Dev mailing list > > Python-Dev at python.org > > http://mail.python.org/mailman/listinfo/python-dev > > Unsubscribe: > > http://mail.python.org/mailman/options/python-dev/guido%40python.org > > > > > > -- > --Guido van Rossum (python.org/~guido) > -- Ani -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Fri Jun 21 20:33:45 2013 From: steve at pearwood.info (Steven D'Aprano) Date: Sat, 22 Jun 2013 04:33:45 +1000 Subject: [Python-Dev] stat module in C -- what to do with stat.py? In-Reply-To: References: <51C2FE1E.8020801@python.org> <51C303C6.8050609@python.org> <51C30E41.9090301@trueblade.com> <20130621134550.1021345e@pitrou.net> Message-ID: <51C49C89.2060900@pearwood.info> On 22/06/13 01:29, Nick Coghlan wrote: > Basically, I want to hear from the Jython, PyPy and IronPython devs > that they're OK with us deleting Lib/stat.py. Hearing other CPython > devs say they're fine with it doesn't mean anything, since we're not > the ones that will have to do additional work as a result. It's not just established Python implementations. A pure-python standard library makes a good foundation for any future implementations. -- Steven From steve at pearwood.info Fri Jun 21 21:20:24 2013 From: steve at pearwood.info (Steven D'Aprano) Date: Sat, 22 Jun 2013 05:20:24 +1000 Subject: [Python-Dev] stat module in C -- what to do with stat.py? In-Reply-To: References: <51C2FE1E.8020801@python.org> <51C303C6.8050609@python.org> <51C30E41.9090301@trueblade.com> Message-ID: <51C4A778.9070803@pearwood.info> On 21/06/13 01:35, Benjamin Peterson wrote: > 2013/6/20 Charles-Fran?ois Natali : >> 2013/6/20 Thomas Wouters : >>> If the .py file is going to be wrong or incomplete, why would we want to >>> keep it -- or use it as fallback -- at all? If we're dead set on having a >>> .py file instead of requiring it to be part of the interpreter (whichever >>> that is, however it was built), it should be generated as part of the build >>> process. Personally, I don't see the value in it; other implementations will >>> need to do *something* special to use it anyway. That's not correct. Other implementations can do exactly what CPython 3.3 does, namely just use stat.py as given. Not all implementations necessarily care about multiple platforms where stat constants are likely to change. >> That's exactly my rationale for pushing for removal. > > +1 to nixing it. -1 Reading the Python source code is a very good way for beginner programmers to learn about things like this. Being able to read stat.py in Python makes a good, complementary source of information for those new to stat. Getting rid of stat.py means there's no source code to read at all, unless the user has built Python from source and kept the C source code. And even if they have, it's easy to forget how intimidating C can be to neophytes. I'm with Nick on this one: PEP 399 already answers the question of what to do with stat.py. We keep it as a fallback, which guarantees that any Python implementation that uses the standard library is no worse off than what we have now. When available, we override the default constants with platform specific ones generated from whatever language is used by the implementation. The accuracy of those constants then becomes a matter of quality of implementation. A low quality implementation may take the lazy way out and just use the default constants, while a high quality implementation will not. At the moment, the stat.py module contains ten small functions. Getting rid of the stat.py means that they have to be re-implemented in C/Java/RPython/etc. for every implementation. Even if the re-implementations are individually trivial, it's still a cost for no real gain. Keeping the pure Python implementation also lowers the bar for adding new functions in the future, and for documentation changes. -- Steven From ericsnowcurrently at gmail.com Fri Jun 21 22:04:20 2013 From: ericsnowcurrently at gmail.com (Eric Snow) Date: Fri, 21 Jun 2013 14:04:20 -0600 Subject: [Python-Dev] stat module in C -- what to do with stat.py? In-Reply-To: References: <51C2FE1E.8020801@python.org> <51C303C6.8050609@python.org> <51C30E41.9090301@trueblade.com> <20130621134550.1021345e@pitrou.net> Message-ID: On Fri, Jun 21, 2013 at 9:29 AM, Nick Coghlan wrote: > So why not just replace the *broken* parts of stat.py and keep the > rest of it? In some ways this sounds like yet another use case for what amounts to properties on modules. The sketchy contants that are causing concern could raise NotImplementedError in stat.py, but the C version would implement them. Then PEP 399 remains satisfied and appropriately so. -eric From victor.stinner at gmail.com Fri Jun 21 22:43:58 2013 From: victor.stinner at gmail.com (Victor Stinner) Date: Fri, 21 Jun 2013 22:43:58 +0200 Subject: [Python-Dev] PEP 445 delegate In-Reply-To: <20130621192802.1f960f40@fsol> References: <20130621192802.1f960f40@fsol> Message-ID: Hi, 2013/6/21 Antoine Pitrou : > I've been appointed PEP 445 delegate by Nick and Guido. I would like to > know if there are still pending changes to the PEP. Cool. Hum, there is maybe something. In my pytracemalloc project, I added another API to track usage of free lists: PyAPI_FUNC(int) _PyFreeList_SetAllocators( void (*alloc) (PyObject *), void (*free) (PyObject *) ); PyAPI_FUNC(void) _PyFreeList_Alloc(void *); PyAPI_FUNC(void) _PyFreeList_Free(void *); Then I'm using Py_TYPE(op)->tp_basicsize to get the size of the object. I didn't add this API to the PEP because I'm not really convinced that is useful. I wrote it becase Python 2 has unlimited free lists, especially for the int type. In Python 3, the situation is different: free lists have a limited size, and the "int" type of Python 3 is the old "long" type, and int in Python 3 does not use a free list. (By the way, it may provide better performances if we had a free list for "small" ints.) I expect better results with the new PyMem_RawMalloc() function and the "Don't call malloc() directly anymore" section of the PEP than with tracking free lists. I'm testing Python 3.4 with all patches related to the PEP 445 (#3329, #16742, #18203) and it works fine. You can also try my pytracemalloc project with its "userdata" branch. I just found a bug in pytracemalloc for allocation of zero bytes :-) Victor From gjcarneiro at gmail.com Fri Jun 21 23:02:44 2013 From: gjcarneiro at gmail.com (Gustavo Carneiro) Date: Fri, 21 Jun 2013 22:02:44 +0100 Subject: [Python-Dev] stat module in C -- what to do with stat.py? In-Reply-To: <51C4A778.9070803@pearwood.info> References: <51C2FE1E.8020801@python.org> <51C303C6.8050609@python.org> <51C30E41.9090301@trueblade.com> <51C4A778.9070803@pearwood.info> Message-ID: On Fri, Jun 21, 2013 at 8:20 PM, Steven D'Aprano wrote: > On 21/06/13 01:35, Benjamin Peterson wrote: > >> 2013/6/20 Charles-Fran?ois Natali : >> >>> 2013/6/20 Thomas Wouters : >>> >>>> If the .py file is going to be wrong or incomplete, why would we want to >>>> keep it -- or use it as fallback -- at all? If we're dead set on having >>>> a >>>> .py file instead of requiring it to be part of the interpreter >>>> (whichever >>>> that is, however it was built), it should be generated as part of the >>>> build >>>> process. Personally, I don't see the value in it; other implementations >>>> will >>>> need to do *something* special to use it anyway. >>>> >>> > That's not correct. Other implementations can do exactly what CPython 3.3 > does, namely just use stat.py as given. Not all implementations necessarily > care about multiple platforms where stat constants are likely to change. > > > > That's exactly my rationale for pushing for removal. >>> >> >> +1 to nixing it. >> > > -1 > > Reading the Python source code is a very good way for beginner programmers > to learn about things like this. On the other hand, it is counter-productive to learn about code that is conceptually _wrong_. -------------- next part -------------- An HTML attachment was scrubbed... URL: From cs at zip.com.au Sat Jun 22 01:58:59 2013 From: cs at zip.com.au (Cameron Simpson) Date: Sat, 22 Jun 2013 09:58:59 +1000 Subject: [Python-Dev] stat module in C -- what to do with stat.py? In-Reply-To: References: Message-ID: <20130621235859.GA89388@cskk.homeip.net> On 21Jun2013 21:39, Nick Coghlan wrote: | On 21 June 2013 17:25, Victor Stinner wrote: | > How do you plan to handle the following case in Python? | > | > "Looking in more detail: for the S_IFMT flags, OSX/Darwin/FreeBSD defines | > 0xe000 as S_IFWHT (whiteout), but Solaris defines it as | > S_IFPORT (event port)." | > | > We may keep the Python module if it is kept unchanged, but the Python | > and C modules should have the same public API (the C module should not | > have more features). | | I think it's OK to expose additional platform specific features in the | C version, and have them fail cleanly with the pure Python version | (rather than silently giving the wrong answer). What's not OK is for | the standard library to regress for other implementations just because | we added a C implementation for the benefit of CPython. That's exactly | the kind of thing PEP 399 says we *won't* do. +1 I'm all for the C module exposing any and all of the S_* macros for the local platform, and good with the python module (if used because the C module isn't present, or conceivably is compiled out because it is known broken on this platform) exposing only the portable stuff. At least you can detect "I don't know what to do" rather than ploughing on mistakenly. -- Cameron Simpson Ignorance is preferable to error; and he is less remote from the truth who believes nothing, than he who believes what is wrong. - Thomas Jefferson From stephen at xemacs.org Sat Jun 22 02:01:53 2013 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Sat, 22 Jun 2013 09:01:53 +0900 Subject: [Python-Dev] stat module in C -- what to do with stat.py? In-Reply-To: References: <51C2FE1E.8020801@python.org> <51C303C6.8050609@python.org> <51C30E41.9090301@trueblade.com> <51C4A778.9070803@pearwood.info> Message-ID: <87wqpn109a.fsf@uwakimon.sk.tsukuba.ac.jp> Gustavo Carneiro writes: > On Fri, Jun 21, 2013 at 8:20 PM, Steven D'Aprano wrote: >> -1 Reading the Python source code is a very good way for beginner >> programmers to learn about things like this. > On the other hand, it is counter-productive to learn about code > that is conceptually _wrong_. It's hardly _conceptually_ wrong when it's simply a pure-Python version that reflects *exactly* the implementation used by many OSes written in C. It's purely a quality of implementation issue in that it fails to observe the DRY principle and is non-portable. But if you look at on Mac OS X or Linux, you'll see that the visible definitions are protected by a thicket of #ifdefs, and the actual definitions may be drawn from other files #include'd there, and not using the definitions visible in at all. Ie, on those systems the OS implementation possesses exactly the same structure that CPython would have with a stat.py + _stat module implementation. If the presence of the _stat module is documented and the rationale briefly explained in stat.py, I suppose you have the best of both worlds for the novice programmer (at the expense of a test for the presence of _stat at import time). From cs at zip.com.au Sat Jun 22 03:42:02 2013 From: cs at zip.com.au (Cameron Simpson) Date: Sat, 22 Jun 2013 11:42:02 +1000 Subject: [Python-Dev] stat module in C -- what to do with stat.py? In-Reply-To: References: Message-ID: <20130622014202.GA43943@cskk.homeip.net> On 20Jun2013 08:35, Benjamin Peterson wrote: | 2013/6/20 Charles-Fran?ois Natali : | > 2013/6/20 Thomas Wouters : | >> If the .py file is going to be wrong or incomplete, why would we want to | >> keep it -- or use it as fallback -- at all? If we're dead set on having a | >> .py file instead of requiring it to be part of the interpreter (whichever | >> that is, however it was built), it should be generated as part of the build | >> process. Personally, I don't see the value in it; other implementations will | >> need to do *something* special to use it anyway. | > | > That's exactly my rationale for pushing for removal. | | +1 to nixing it. -1 to nixing it. I think there should be a pure python reference implementation. If it exposes only the portable constants/macros or, better, has a machine generated _section_ for the local platform macros, all to the good; it would not lie. - A huge amount of code only needs to care about the portable stuff (is this a dir, is this a regular file, is it neither). Missing local constants won't break such code. - A reference implementation exposes algorithms and purpose in a common language (Python, of course; everyone caring about such an impl can already read Python:-). - A reference implementation provides a base for other implementations to use outright, or to build on. - A reference implementation provides something to test against for comparison of the common stuff. - The implementation cost is low; the ref implementation already exists! Getting rid of it seems mad. Pruning/fixing/adapting the nonportable bits might be good. Christian Heimes' test patch would make that much easier to identify. So, -1 on removal of stat.py. Cheers, -- Cameron Simpson I will not do it as a hack I will not do it for my friends I will not do it on a Mac I will not write for Uncle Sam I will not do it on weekends I won't do ADA, Sam-I-Am - Gregory Bond From arigo at tunes.org Sat Jun 22 10:55:03 2013 From: arigo at tunes.org (Armin Rigo) Date: Sat, 22 Jun 2013 10:55:03 +0200 Subject: [Python-Dev] stat module in C -- what to do with stat.py? In-Reply-To: <51C4A778.9070803@pearwood.info> References: <51C2FE1E.8020801@python.org> <51C303C6.8050609@python.org> <51C30E41.9090301@trueblade.com> <51C4A778.9070803@pearwood.info> Message-ID: Hi, On Fri, Jun 21, 2013 at 9:20 PM, Steven D'Aprano wrote: >>>> process. Personally, I don't see the value in it; other implementations >>>> will need to do *something* special to use it anyway. > > That's not correct. Other implementations can do exactly what CPython 3.3 > does, namely just use stat.py as given. Not all implementations necessarily > care about multiple platforms where stat constants are likely to change. Thanks Steven. That's PyPy's position: for now we really care only about where stat.py has been obviously sufficient, as shown by CPython's long experience with stat.py. A bient?t, Armin. From benjamin at python.org Sat Jun 22 20:17:12 2013 From: benjamin at python.org (Benjamin Peterson) Date: Sat, 22 Jun 2013 11:17:12 -0700 Subject: [Python-Dev] cpython (2.7): Fix comment blocks. Adjust blocksize to a power-of-two for better divmod In-Reply-To: References: <3bWqYP3sqgzQdN@mail.python.org> <20130614104603.35f83a9c@fsol> <51BFF956.7040403@stoneleaf.us> Message-ID: Many people have raised concerns about this change, so I've now backed it out. 2013/6/18 Gregory P. Smith : > Raymond - > > Why did you do this in the 2.7 branch? > > It hasn't been done in 3.3 or default and it isn't not the sort of change we > make in a stable release branch without justification. What issue was this > for? What problem were you solving? > > -gps > > > > On Mon, Jun 17, 2013 at 11:08 PM, Ethan Furman wrote: >> >> On 06/17/2013 09:05 PM, Eli Bendersky wrote: >>> >>> >>> >>> >>> On Fri, Jun 14, 2013 at 6:47 AM, Serhiy Storchaka >> > wrote: >>> >>> 14.06.13 11:46, Antoine Pitrou ???????(??): >>> >>> On Fri, 14 Jun 2013 07:06:49 +0200 (CEST) >>> raymond.hettinger >> > wrote: >>> >>> http://hg.python.org/cpython/__rev/5accb0ac8bfb >>> >>> >>> changeset: 84116:5accb0ac8bfb >>> Fix comment blocks. Adjust blocksize to a power-of-two >>> for better divmod computations. >>> >>> >>> Is there any rationale for changing the heuristic from "fits in a >>> whole >>> number of cachelines" to "allows fast division by the blocksize"? >>> >>> I personally would prefer if such changes were justified a bit >>> more >>> than by a one-liner changeset message without any corresponding >>> open >>> issue. >>> >>> >>> I share the doubts of Antoine and I was going to write the same >>> comment. I thought there were good reasons for >>> previous code. What has changed? >>> >>> >>> I agree it would be interesting to hear about the reasons for the change. >>> Raymond? >> >> >> Asking as a learner: are such non-bugfix changes appropriate for the 2.7 >> line? >> >> -- >> ~Ethan~ >> >> _______________________________________________ >> Python-Dev mailing list >> Python-Dev at python.org >> http://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe: >> http://mail.python.org/mailman/options/python-dev/greg%40krypto.org > > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/benjamin%40python.org > -- Regards, Benjamin From scott+python-dev at scottdial.com Sat Jun 22 22:43:44 2013 From: scott+python-dev at scottdial.com (Scott Dial) Date: Sat, 22 Jun 2013 16:43:44 -0400 Subject: [Python-Dev] cpython (2.7): Fix comment blocks. Adjust blocksize to a power-of-two for better divmod In-Reply-To: References: <3bWqYP3sqgzQdN@mail.python.org> <20130614104603.35f83a9c@fsol> <51BFF956.7040403@stoneleaf.us> Message-ID: <51C60C80.1010207@scottdial.com> On 6/22/2013 2:17 PM, Benjamin Peterson wrote: > Many people have raised concerns about this change, so I've now backed it out. I think that change also goes with this change: http://hg.python.org/cpython/rev/f1dc30a1be72 changeset 84248:f1dc30a1be72 2.7 Arrange structure to match the common access patterns. 1.1 --- a/Modules/_collectionsmodule.c 1.2 +++ b/Modules/_collectionsmodule.c 1.3 @@ -48,8 +48,8 @@ 1.4 1.5 typedef struct BLOCK { 1.6 struct BLOCK *leftlink; 1.7 + PyObject *data[BLOCKLEN]; 1.8 struct BLOCK *rightlink; 1.9 - PyObject *data[BLOCKLEN]; 1.10 } block; 1.11 1.12 #define MAXFREEBLOCKS 10 , which seems like a strange micro-optimization, at best. Based on that structure, it would seem that neither BLOCKLEN being 62 (previous value) nor 64 (the new value) make much sense. It would seem best that sizeof(block) == 64, so BLOCKLEN should be (64 - 2*sizeof(PyObject *)). Nevertheless, I am skeptical that any tuning of this structure provides any meaningful performance improvement. -- Scott Dial scott at scottdial.com From guido at python.org Sat Jun 22 23:55:09 2013 From: guido at python.org (Guido van Rossum) Date: Sat, 22 Jun 2013 14:55:09 -0700 Subject: [Python-Dev] cpython (2.7): Fix comment blocks. Adjust blocksize to a power-of-two for better divmod In-Reply-To: <51C60C80.1010207@scottdial.com> References: <3bWqYP3sqgzQdN@mail.python.org> <20130614104603.35f83a9c@fsol> <51BFF956.7040403@stoneleaf.us> <51C60C80.1010207@scottdial.com> Message-ID: On Sat, Jun 22, 2013 at 1:43 PM, Scott Dial wrote: > On 6/22/2013 2:17 PM, Benjamin Peterson wrote: >> Many people have raised concerns about this change, so I've now backed it out. > > I think that change also goes with this change: > > http://hg.python.org/cpython/rev/f1dc30a1be72 > > changeset 84248:f1dc30a1be72 2.7 > Arrange structure to match the common access patterns. > > 1.1 --- a/Modules/_collectionsmodule.c > 1.2 +++ b/Modules/_collectionsmodule.c > 1.3 @@ -48,8 +48,8 @@ > 1.4 > 1.5 typedef struct BLOCK { > 1.6 struct BLOCK *leftlink; > 1.7 + PyObject *data[BLOCKLEN]; > 1.8 struct BLOCK *rightlink; > 1.9 - PyObject *data[BLOCKLEN]; > 1.10 } block; > 1.11 > 1.12 #define MAXFREEBLOCKS 10 > > , which seems like a strange micro-optimization, at best. > > Based on that structure, it would seem that neither BLOCKLEN being 62 > (previous value) nor 64 (the new value) make much sense. It would seem > best that sizeof(block) == 64, so BLOCKLEN should be (64 - > 2*sizeof(PyObject *)). Nevertheless, I am skeptical that any tuning of > this structure provides any meaningful performance improvement. Actually the data buffer is an array of pointers too, so with the original BLOCKLEN value of 62, sizeof(block) would be 64 times sizeof(PyObject *). In the Py3 version of the source there's even a comment explaining this right before the #define BLOCKLEN. Raymond, can you explain? -- --Guido van Rossum (python.org/~guido) From victor.stinner at gmail.com Sun Jun 23 14:40:30 2013 From: victor.stinner at gmail.com (Victor Stinner) Date: Sun, 23 Jun 2013 14:40:30 +0200 Subject: [Python-Dev] PEP 445 delegate In-Reply-To: <20130621192802.1f960f40@fsol> References: <20130621192802.1f960f40@fsol> Message-ID: Hi, 2013/6/21 Antoine Pitrou : > I've been appointed PEP 445 delegate by Nick and Guido. I would like to > know if there are still pending changes to the PEP. If not, I expect to > give it a review in the coming days or weeks, and then make a final > pronouncement (which will probably be positive anyway). As you asked on IRC, I added a section to the PEP 445 explaining in detail the redesign of debug checks on memory allocators: http://www.python.org/dev/peps/pep-0445/#redesign-debug-checks-on-memory-allocators-as-hooks The section show also how "hooks" work in the simple "call traces" example. Hooks are just inserted between the public function (ex: PyMem_Malloc) and the inner allocator function (ex: _PyMem_RawMalloc()). Minor nit: in practice, there is only one "debug hook" (reused on 3 allocators), so we may drop the trailing "s" from the name "PyMem_SetupDebugHooks". Diff on pep-0445.txt: http://hg.python.org/peps/rev/c6199202522a Victor From fijall at gmail.com Sun Jun 23 17:40:13 2013 From: fijall at gmail.com (Maciej Fijalkowski) Date: Sun, 23 Jun 2013 17:40:13 +0200 Subject: [Python-Dev] [Python-checkins] cpython (3.3): Add -b and -X options to python man page. In-Reply-To: References: <3bbSw10tm2z7Ljf@mail.python.org> Message-ID: On Thu, Jun 20, 2013 at 3:36 PM, Brett Cannon wrote: > > > > On Wed, Jun 19, 2013 at 11:20 PM, senthil.kumaran > wrote: >> >> http://hg.python.org/cpython/rev/dfead0696a71 >> changeset: 84224:dfead0696a71 >> branch: 3.3 >> parent: 84221:0113247f894b >> user: Senthil Kumaran >> date: Wed Jun 19 22:19:46 2013 -0500 >> summary: >> Add -b and -X options to python man page. >> Patch contributed by Corey Brune. >> >> files: >> Misc/python.man | 22 ++++++++++++++++++---- >> 1 files changed, 18 insertions(+), 4 deletions(-) >> >> >> diff --git a/Misc/python.man b/Misc/python.man >> --- a/Misc/python.man >> +++ b/Misc/python.man >> @@ -11,6 +11,9 @@ >> .B \-B >> ] >> [ >> +.B \-b >> +] >> +[ >> .B \-d >> ] >> [ >> @@ -23,14 +26,14 @@ >> .B \-i >> ] >> [ >> -.B \-m >> +.B \-m >> .I module-name >> ] >> -[ >> -.B \-q >> -] >> .br >> [ >> +.B \-q >> +] >> +[ >> .B \-O >> ] >> [ >> @@ -60,6 +63,10 @@ >> .B \-x >> ] >> [ >> +[ >> +.B \-X >> +.I option >> +] >> .B \-? >> ] >> .br >> @@ -105,6 +112,10 @@ >> .I .py[co] >> files on import. See also PYTHONDONTWRITEBYTECODE. >> .TP >> +.B \-b >> +Issue warnings about str(bytes_instance), str(bytearray_instance) >> +and comparing bytes/bytearray with str. (-bb: issue errors) >> +.TP >> .BI "\-c " command >> Specify the command to execute (see next section). >> This terminates the option list (following options are passed as >> @@ -243,6 +254,9 @@ >> field matches the line number, where zero matches all line numbers and >> is thus equivalent to an omitted line number. >> .TP >> +.BI "\-X " option >> +Set implementation specific option. > > > Should probably be "Set the implementation-specific option." Is there anyone respecting this notation? (I know pypy does not, it uses --jit and stuff) > >> >> +.TP >> .B \-x >> Skip the first line of the source. This is intended for a DOS >> specific hack only. Warning: the line numbers in error messages will >> >> -- >> Repository URL: http://hg.python.org/cpython >> >> _______________________________________________ >> Python-checkins mailing list >> Python-checkins at python.org >> http://mail.python.org/mailman/listinfo/python-checkins >> > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/fijall%40gmail.com > From rdmurray at bitdance.com Sun Jun 23 18:14:38 2013 From: rdmurray at bitdance.com (R. David Murray) Date: Sun, 23 Jun 2013 12:14:38 -0400 Subject: [Python-Dev] [Python-checkins] cpython (3.3): Add -b and -X options to python man page. In-Reply-To: References: <3bbSw10tm2z7Ljf@mail.python.org> Message-ID: <20130623161440.2587A250338@webabinitio.net> On Sun, 23 Jun 2013 17:40:13 +0200, Maciej Fijalkowski wrote: > On Thu, Jun 20, 2013 at 3:36 PM, Brett Cannon wrote: > > On Wed, Jun 19, 2013 at 11:20 PM, senthil.kumaran > > wrote: > >> .TP > >> +.BI "\-X " option > >> +Set implementation specific option. > > > > > > Should probably be "Set the implementation-specific option." > > Is there anyone respecting this notation? (I know pypy does not, it > uses --jit and stuff) CPython does. We introduced it for ourselves, it is up to other implementations whether or not to use it, or use something else. --David From benjamin at python.org Sun Jun 23 20:38:26 2013 From: benjamin at python.org (Benjamin Peterson) Date: Sun, 23 Jun 2013 11:38:26 -0700 Subject: [Python-Dev] cpython (2.7): Fix comment blocks. Adjust blocksize to a power-of-two for better divmod In-Reply-To: <51C60C80.1010207@scottdial.com> References: <3bWqYP3sqgzQdN@mail.python.org> <20130614104603.35f83a9c@fsol> <51BFF956.7040403@stoneleaf.us> <51C60C80.1010207@scottdial.com> Message-ID: I've backed this one out, too. 2013/6/22 Scott Dial : > On 6/22/2013 2:17 PM, Benjamin Peterson wrote: >> Many people have raised concerns about this change, so I've now backed it out. > > I think that change also goes with this change: > > http://hg.python.org/cpython/rev/f1dc30a1be72 > > changeset 84248:f1dc30a1be72 2.7 > Arrange structure to match the common access patterns. > > 1.1 --- a/Modules/_collectionsmodule.c > 1.2 +++ b/Modules/_collectionsmodule.c > 1.3 @@ -48,8 +48,8 @@ > 1.4 > 1.5 typedef struct BLOCK { > 1.6 struct BLOCK *leftlink; > 1.7 + PyObject *data[BLOCKLEN]; > 1.8 struct BLOCK *rightlink; > 1.9 - PyObject *data[BLOCKLEN]; > 1.10 } block; > 1.11 > 1.12 #define MAXFREEBLOCKS 10 > > , which seems like a strange micro-optimization, at best. > > Based on that structure, it would seem that neither BLOCKLEN being 62 > (previous value) nor 64 (the new value) make much sense. It would seem > best that sizeof(block) == 64, so BLOCKLEN should be (64 - > 2*sizeof(PyObject *)). Nevertheless, I am skeptical that any tuning of > this structure provides any meaningful performance improvement. > > -- > Scott Dial > scott at scottdial.com -- Regards, Benjamin From storchaka at gmail.com Sun Jun 23 20:59:37 2013 From: storchaka at gmail.com (Serhiy Storchaka) Date: Sun, 23 Jun 2013 21:59:37 +0300 Subject: [Python-Dev] A type of out-of-range exception for %c Message-ID: Currently %c formatting raises OverflowError if an argument is out of range. >>> '%c' % 1114112 Traceback (most recent call last): File "", line 1, in OverflowError: %c arg not in range(0x110000) >>> '{:c}'.format(1114112) Traceback (most recent call last): File "", line 1, in OverflowError: %c arg not in range(0x110000) The same for PyUnicode_Format(), PyUnicode_FromFromat() and PyUnicode_FromFromatV() in C API. But chr() raises ValueError. >>> chr(1114112) Traceback (most recent call last): File "", line 1, in ValueError: chr() arg not in range(0x110000) And ValueError looks more suitable, because it is not really an overflow (limitation of the C language, or a C type). Can we change an exception type for %c formatting from OverflowError to ValueError? From solipsis at pitrou.net Sun Jun 23 21:07:03 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sun, 23 Jun 2013 21:07:03 +0200 Subject: [Python-Dev] A type of out-of-range exception for %c References: Message-ID: <20130623210703.4e25fa32@fsol> On Sun, 23 Jun 2013 21:59:37 +0300 Serhiy Storchaka wrote: > Currently %c formatting raises OverflowError if an argument is out of range. > > >>> '%c' % 1114112 > Traceback (most recent call last): > File "", line 1, in > OverflowError: %c arg not in range(0x110000) > >>> '{:c}'.format(1114112) > Traceback (most recent call last): > File "", line 1, in > OverflowError: %c arg not in range(0x110000) > > The same for PyUnicode_Format(), PyUnicode_FromFromat() and > PyUnicode_FromFromatV() in C API. > > But chr() raises ValueError. > > >>> chr(1114112) > Traceback (most recent call last): > File "", line 1, in > ValueError: chr() arg not in range(0x110000) > > And ValueError looks more suitable, because it is not really an overflow > (limitation of the C language, or a C type). Can we change an exception > type for %c formatting from OverflowError to ValueError? Well, it overflows the limited range of the "unicode character" type (which is virtual in both Python and C, but still conceptually exists). The most annoying thing here is that OverflowError doesn't subclass ValueError. Regards Antoine. From lukasz at langa.pl Sun Jun 23 23:27:27 2013 From: lukasz at langa.pl (=?utf-8?Q?=C5=81ukasz_Langa?=) Date: Sun, 23 Jun 2013 23:27:27 +0200 Subject: [Python-Dev] A type of out-of-range exception for %c In-Reply-To: <20130623210703.4e25fa32@fsol> References: <20130623210703.4e25fa32@fsol> Message-ID: On 23 cze 2013, at 21:07, Antoine Pitrou wrote: > The most annoying thing here is that OverflowError doesn't subclass > ValueError. My intuition would rather make OverflowError subclass RuntimeError. Am I wrong? That means I support Serhiy's point that %c would be less surprising raising a ValueError. Backwards compatibility is an unfortunate blocker here, I suppose. -- Best regards, ?ukasz Langa WWW: http://lukasz.langa.pl/ Twitter: @llanga IRC: ambv on #python-dev -------------- next part -------------- An HTML attachment was scrubbed... URL: From raymond.hettinger at gmail.com Mon Jun 24 02:16:32 2013 From: raymond.hettinger at gmail.com (Raymond Hettinger) Date: Sun, 23 Jun 2013 17:16:32 -0700 Subject: [Python-Dev] cpython (2.7): Fix comment blocks. Adjust blocksize to a power-of-two for better divmod In-Reply-To: <51C60C80.1010207@scottdial.com> References: <3bWqYP3sqgzQdN@mail.python.org> <20130614104603.35f83a9c@fsol> <51BFF956.7040403@stoneleaf.us> <51C60C80.1010207@scottdial.com> Message-ID: <2278DBFF-B33B-4DEC-BADA-B7208E0D61EA@rcn.com> On Jun 18, 2013, at 12:00 AM, Gregory P. Smith wrote: > Why did you do this in the 2.7 branch? > > It hasn't been done in 3.3 or default I worked on the 2.7 branch first because that was the one I had loaded and the one where I did timings and code disassembly. I intended to post it to 3.3 and 3.4 as well over the weekend. Ideally, it makes maintenance simpler for me if I keep the branches the same as possible. I viewed the one-line struct transposition, comment correction, and one-line blocklen change as being somewhat innocuous non-algorithmic changes. The struct change fixed an unintended cache miss for left links and the blocksize change causes the deque_index code to compile more efficiently (using a right-shift and bitwise-and and rather than a measurably more expensive division and modulo calculation). I truly wasn't expecting the Spanish Inquisition :-) On Jun 22, 2013, at 1:43 PM, Scott Dial wrote: > changeset 84248:f1dc30a1be72 2.7 > Arrange structure to match the common access patterns. > > 1.1 --- a/Modules/_collectionsmodule.c > 1.2 +++ b/Modules/_collectionsmodule.c > 1.3 @@ -48,8 +48,8 @@ > 1.4 > 1.5 typedef struct BLOCK { > 1.6 struct BLOCK *leftlink; > 1.7 + PyObject *data[BLOCKLEN]; > 1.8 struct BLOCK *rightlink; > 1.9 - PyObject *data[BLOCKLEN]; > 1.10 } block; > 1.11 > 1.12 #define MAXFREEBLOCKS 10 > > , which seems like a strange micro-optimization, at best. Yes, this is a micro-optimization. In working on implementing deque slicing for 3.4, I restudied the block access patterns. On an appendleft(), popleft() or extendleft() operation, the left link is accessed immediately before or after the leftmost entry in the data block. The old struct arrangement can cause an unnecessary cache miss when jumping leftward. This was something I overlooked when I originally wrote the code almost a decade ago. On Jun 23, 2013, at 11:38 AM, Benjamin Peterson wrote: > I've backed this one out, too. Really, you reverted my one-line change within 24 hours of it being posted? I can't be on-line every day and sometimes it takes a little while to catch up with python email. On Jun 22, 2013, at 2:55 PM, Guido van Rossum wrote: > Actually the data buffer is an array of pointers too, so with the > original BLOCKLEN value of 62, sizeof(block) would be 64 times > sizeof(PyObject *). In the Py3 version of the source there's even a > comment explaining this right before the #define BLOCKLEN. Raymond, > can you explain? I also tried to fix that comment so it would stop emphasizing the blocklen being a multiple of the cache line. Also long as there is a reasonably long data block, it matters less whether the data block size is an exact multiple of the cache line length (62 or 64 words of data versus a typical 8 byte cache line). The data block size does matter elsewhere though. The benefit of the having the data block being a power-of-two is that the deque_index computation can use bits shifts rather division and modulo calculations. The benefit of switching data block size from 62 to 64 was measurable (a few percent) and observable in the disassembly of the code. I experimented with one other ordering as well (putting the data block first and the links afterwards). That saved the scaled-index byte in the generated code but produced no measureable speed-up. In short, I believe the following should go in: * The comment fix. (Incorrectly suggesting that a 64-bit Py_ssize_t would overflow). The revised comment is simpler, more general, and correct. * Putting the leftlink before the data block in the structure. The follow is up for debate: Changing the BLOCKLEN from 62 to 64 is debatable. It measureably improved deque_index without an observable negative effect on the other operations. Lastly, there was a change I just put in to Py 3.4 replacing the memcpy() with a simple loop and replacing the "deque->" references with local variables. Besides giving a small speed-up, it made the code more clear and less at the mercy of various implementations of memcpy(). Ideally, I would like 2.7 and 3.3 to replace their use of memcpy() as well, but the flavor of this thread suggests that is right out. Raymond -------------- next part -------------- An HTML attachment was scrubbed... URL: From scott+python-dev at scottdial.com Mon Jun 24 03:52:52 2013 From: scott+python-dev at scottdial.com (Scott Dial) Date: Sun, 23 Jun 2013 21:52:52 -0400 Subject: [Python-Dev] cpython (2.7): Fix comment blocks. Adjust blocksize to a power-of-two for better divmod In-Reply-To: <2278DBFF-B33B-4DEC-BADA-B7208E0D61EA@rcn.com> References: <3bWqYP3sqgzQdN@mail.python.org> <20130614104603.35f83a9c@fsol> <51BFF956.7040403@stoneleaf.us> <51C60C80.1010207@scottdial.com> <2278DBFF-B33B-4DEC-BADA-B7208E0D61EA@rcn.com> Message-ID: <51C7A674.4010706@scottdial.com> On 6/23/2013 8:16 PM, Raymond Hettinger wrote: > Yes, this is a micro-optimization. In working on implementing > deque slicing for 3.4, I restudied the block access patterns. > On an appendleft(), popleft() or extendleft() operation, the left link is > accessed immediately before or after the leftmost entry in the data block. > The old struct arrangement can cause an unnecessary cache miss > when jumping leftward. This was something I overlooked when I > originally wrote the code almost a decade ago. A decade ago, cache lines were 64 bytes, pointers were 4 bytes, and allocations were 16 byte aligned, so there could never be a cache miss. Nowadays, cache lines are still 64 bytes but pointers are 8 bytes, and we still allocating on 16 byte alignment, so you have a 25% chance of a cache miss now. -- Scott Dial scott at scottdial.com From raymond.hettinger at gmail.com Mon Jun 24 05:37:37 2013 From: raymond.hettinger at gmail.com (Raymond Hettinger) Date: Sun, 23 Jun 2013 20:37:37 -0700 Subject: [Python-Dev] cpython (2.7): Fix comment blocks. Adjust blocksize to a power-of-two for better divmod In-Reply-To: <51C7A674.4010706@scottdial.com> References: <3bWqYP3sqgzQdN@mail.python.org> <20130614104603.35f83a9c@fsol> <51BFF956.7040403@stoneleaf.us> <51C60C80.1010207@scottdial.com> <2278DBFF-B33B-4DEC-BADA-B7208E0D61EA@rcn.com> <51C7A674.4010706@scottdial.com> Message-ID: <85B0D1EF-A3E5-486A-8C7B-DCC297F181DD@gmail.com> On Jun 23, 2013, at 6:52 PM, Scott Dial wrote: > Nowadays, cache lines are still 64 bytes but pointers are 8 bytes, and > we still allocating on 16 byte alignment, so you have a 25% chance of a > cache miss now. Honestly, I'm not sure what you're arguing for or against. The struct should to be reordered so that the leftmost data element and left link are positioned side-by-side, and the same goes for the rightmost element and right link. Whether the blocksize should change is debatable. It causes an unfortunate odd-size at the end (not good for the final cache line), but it does improve the speed of the deque_index() code. The former seems to make no difference in timings while the latter gives a measureable speed-up. Unfortunately, I'm losing interest in this part of the deque work. I've already invested substantial time reading papers (such as http://www.akkadia.org/drepper/cpumemory.pdf and http://www.intel.com/content/www/us/en/architecture-and-technology/64-ia-32-architectures-optimization-manual.html), analyzing the code, reading disassembly, and making timings. But it isn't worth all the second guessing (and what feels like sniping). I've worked on this code for almost a decade. As far as I can tell, none of the participants in this thread has ever previously shown any interest in the deque object. It is discouraging to have a simple parameter change and struct reordering reverted. This thread has offered zero support or encouragement for my work. The python-dev social environment appears to be degrading over time. I only have a few hours of development time each week and now I'm wasting that time responding to these emails (it may go with the territory, but it is a waste none-the-less). If it is what you guys what, then leave the code as is (with an incorrect comment, a blocklen that is unfavorable to indexing, and a struct order that doesn't exploit cache locality by following the natural access patterns in the code). I understand that the code for Py2.7 is sensitive and understand if you all prefer to leave it untouched. Raymond P.S. This has all arisen in context of my working on patch for implementing slicing for deques. In that context, the code for deque_item() and deque_rotate() will become more important than they were before. Along the way, I am re-examining all my existing code. -------------- next part -------------- An HTML attachment was scrubbed... URL: From alexander.belopolsky at gmail.com Mon Jun 24 06:06:56 2013 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Mon, 24 Jun 2013 00:06:56 -0400 Subject: [Python-Dev] cpython (2.7): Fix comment blocks. Adjust blocksize to a power-of-two for better divmod In-Reply-To: <85B0D1EF-A3E5-486A-8C7B-DCC297F181DD@gmail.com> References: <3bWqYP3sqgzQdN@mail.python.org> <20130614104603.35f83a9c@fsol> <51BFF956.7040403@stoneleaf.us> <51C60C80.1010207@scottdial.com> <2278DBFF-B33B-4DEC-BADA-B7208E0D61EA@rcn.com> <51C7A674.4010706@scottdial.com> <85B0D1EF-A3E5-486A-8C7B-DCC297F181DD@gmail.com> Message-ID: On Sun, Jun 23, 2013 at 11:37 PM, Raymond Hettinger < raymond.hettinger at gmail.com> wrote: > As far as I can tell, none > of the participants in this thread has ever previously shown any interest > in the deque object. It is discouraging to have a simple parameter > change and struct reordering reverted. This thread has offered zero > support or encouragement for my work. The python-dev > social environment appears to be degrading over time. > FWIW, I was surprised to see Raymond's patch reverted in what looked like a knee-jerk reaction. In my view Raymond through his contributions has earned the presumption of validity for his commits. A risk of introducing regressions may ultimately outweigh the benefit of optimization and keeping 2.x and 3.x code in sync, but there is no need in rushing the decision. This patch could be reverted at any time before the 2.7.6 release which as far as I know has not even been scheduled. -------------- next part -------------- An HTML attachment was scrubbed... URL: From greg at krypto.org Mon Jun 24 06:11:58 2013 From: greg at krypto.org (Gregory P. Smith) Date: Sun, 23 Jun 2013 21:11:58 -0700 Subject: [Python-Dev] cpython (2.7): Fix comment blocks. Adjust blocksize to a power-of-two for better divmod In-Reply-To: <85B0D1EF-A3E5-486A-8C7B-DCC297F181DD@gmail.com> References: <3bWqYP3sqgzQdN@mail.python.org> <20130614104603.35f83a9c@fsol> <51BFF956.7040403@stoneleaf.us> <51C60C80.1010207@scottdial.com> <2278DBFF-B33B-4DEC-BADA-B7208E0D61EA@rcn.com> <51C7A674.4010706@scottdial.com> <85B0D1EF-A3E5-486A-8C7B-DCC297F181DD@gmail.com> Message-ID: Your work is great! I even agree with the changes on a coding best practices level. It's just that it belongs on the default (3.4) branch as it is an enhancement, not a bug fix. We don't do new things on release branches. I agree that can be extremely frustrating at times, knowing that code won't see the light of day for most users for years, but it is what has always kept the Python project stable. -gps On Jun 23, 2013 8:38 PM, "Raymond Hettinger" wrote: > > On Jun 23, 2013, at 6:52 PM, Scott Dial > wrote: > > Nowadays, cache lines are still 64 bytes but pointers are 8 bytes, and > we still allocating on 16 byte alignment, so you have a 25% chance of a > cache miss now. > > > Honestly, I'm not sure what you're arguing for or against. > > The struct should to be reordered so that the leftmost data element > and left link are positioned side-by-side, and the same goes for the > rightmost element and right link. > > Whether the blocksize should change is debatable. > It causes an unfortunate odd-size at the end (not good for > the final cache line), but it does improve the speed of the > deque_index() code. The former seems to make no difference > in timings while the latter gives a measureable speed-up. > > Unfortunately, I'm losing interest in this part of the deque work. > I've already invested substantial time reading papers > (such as http://www.akkadia.org/drepper/cpumemory.pdf > and > http://www.intel.com/content/www/us/en/architecture-and-technology/64-ia-32-architectures-optimization-manual.html > ), > analyzing the code, reading disassembly, and making timings. > But it isn't worth all the second guessing (and what feels like sniping). > I've worked on this code for almost a decade. As far as I can tell, none > of the participants in this thread has ever previously shown any interest > in the deque object. It is discouraging to have a simple parameter > change and struct reordering reverted. This thread has offered zero > support or encouragement for my work. The python-dev > social environment appears to be degrading over time. > I only have a few hours of development time each week > and now I'm wasting that time responding to these emails > (it may go with the territory, but it is a waste none-the-less). > > If it is what you guys what, then leave the code as is > (with an incorrect comment, a blocklen that is unfavorable > to indexing, and a struct order that doesn't exploit > cache locality by following the natural access patterns > in the code). > > I understand that the code for Py2.7 is sensitive and understand > if you all prefer to leave it untouched. > > > > Raymond > > > P.S. This has all arisen in context of my working on patch > for implementing slicing for deques. In that context, the > code for deque_item() and deque_rotate() will become > more important than they were before. Along the way, > I am re-examining all my existing code. > > > > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/greg%40krypto.org > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From benjamin at python.org Mon Jun 24 08:03:14 2013 From: benjamin at python.org (Benjamin Peterson) Date: Sun, 23 Jun 2013 23:03:14 -0700 Subject: [Python-Dev] cpython (2.7): Fix comment blocks. Adjust blocksize to a power-of-two for better divmod In-Reply-To: References: <3bWqYP3sqgzQdN@mail.python.org> <20130614104603.35f83a9c@fsol> <51BFF956.7040403@stoneleaf.us> <51C60C80.1010207@scottdial.com> <2278DBFF-B33B-4DEC-BADA-B7208E0D61EA@rcn.com> <51C7A674.4010706@scottdial.com> <85B0D1EF-A3E5-486A-8C7B-DCC297F181DD@gmail.com> Message-ID: 2013/6/23 Alexander Belopolsky : > > On Sun, Jun 23, 2013 at 11:37 PM, Raymond Hettinger > wrote: >> >> As far as I can tell, none >> of the participants in this thread has ever previously shown any interest >> in the deque object. It is discouraging to have a simple parameter >> change and struct reordering reverted. This thread has offered zero >> support or encouragement for my work. The python-dev >> social environment appears to be degrading over time. > > > FWIW, I was surprised to see Raymond's patch reverted in what looked like a > knee-jerk reaction. In my view Raymond through his contributions has earned > the presumption of validity for his commits. A risk of introducing > regressions may ultimately outweigh the benefit of optimization and keeping > 2.x and 3.x code in sync, but there is no need in rushing the decision. > This patch could be reverted at any time before the 2.7.6 release which as > far as I know has not even been scheduled. The first concern about 5accb0ac8bfb was raised more than a week ago and more piled up over the course of the week. I knew Raymond wasn't offline, since he continued to commit. Since we don't have mandatory reviews before push (which more and more I think we should), post-facto reviews on python-checkins need to be acknowledged. A backout is not the end of the world; it takes seconds to reland once a conclusion has been reached. This incident should not be construed to diminish Raymond's long history of contribution or his technical ability. My second reversion (86d512e0ec66) was knee-jerk and shouldn't have happened. -- Regards, Benjamin From ncoghlan at gmail.com Mon Jun 24 08:23:18 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 24 Jun 2013 16:23:18 +1000 Subject: [Python-Dev] cpython (2.7): Fix comment blocks. Adjust blocksize to a power-of-two for better divmod In-Reply-To: <85B0D1EF-A3E5-486A-8C7B-DCC297F181DD@gmail.com> References: <3bWqYP3sqgzQdN@mail.python.org> <20130614104603.35f83a9c@fsol> <51BFF956.7040403@stoneleaf.us> <51C60C80.1010207@scottdial.com> <2278DBFF-B33B-4DEC-BADA-B7208E0D61EA@rcn.com> <51C7A674.4010706@scottdial.com> <85B0D1EF-A3E5-486A-8C7B-DCC297F181DD@gmail.com> Message-ID: On 24 June 2013 13:37, Raymond Hettinger wrote: > But it isn't worth all the second guessing (and what feels like sniping). > I've worked on this code for almost a decade. As far as I can tell, none > of the participants in this thread has ever previously shown any interest > in the deque object. It is discouraging to have a simple parameter > change and struct reordering reverted. This thread has offered zero > support or encouragement for my work. The python-dev > social environment appears to be degrading over time. > I only have a few hours of development time each week > and now I'm wasting that time responding to these emails > (it may go with the territory, but it is a waste none-the-less). The problem wasn't the change in and of itself, it was: - the fact it happened in a stable branch - the fact there was no explanation, even a week after one was requested (not even a quick "I do have the data to back up this assertion, please leave the commit alone until I have time to provide it"). The mailing lists, commit history, source code comments and files like dict_notes.txt are an important part of understanding *why* various parts of CPython are the way they are. So, no, "responding to emails" is *not* a waste of anybody's time. It helps to ensure important knowledge is transferred to more than the person that was responsible for the commit. This is *especially* important for areas where one person has substantially more knowledge and experience than others (alleviating that kind of problem is one of the reasons Brett, myself and others have been embarked on a multi-year campaign to make the import system less arcane and mysterious, and why I want to make the startup code more comprehensible). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From storchaka at gmail.com Mon Jun 24 10:44:53 2013 From: storchaka at gmail.com (Serhiy Storchaka) Date: Mon, 24 Jun 2013 11:44:53 +0300 Subject: [Python-Dev] cpython (2.7): Fix comment blocks. Adjust blocksize to a power-of-two for better divmod In-Reply-To: <2278DBFF-B33B-4DEC-BADA-B7208E0D61EA@rcn.com> References: <3bWqYP3sqgzQdN@mail.python.org> <20130614104603.35f83a9c@fsol> <51BFF956.7040403@stoneleaf.us> <51C60C80.1010207@scottdial.com> <2278DBFF-B33B-4DEC-BADA-B7208E0D61EA@rcn.com> Message-ID: 24.06.13 03:16, Raymond Hettinger ???????(??): > I truly wasn't expecting the Spanish Inquisition :-) I only asked about the reasons. Previous reasons seemed reasonable to me and I wanted to know why a new code is better than the old. It will help me to use the best style in other places. Thank you for your clarification. From victor.stinner at gmail.com Mon Jun 24 12:39:31 2013 From: victor.stinner at gmail.com (Victor Stinner) Date: Mon, 24 Jun 2013 12:39:31 +0200 Subject: [Python-Dev] cpython (2.7): Fix comment blocks. Adjust blocksize to a power-of-two for better divmod In-Reply-To: <2278DBFF-B33B-4DEC-BADA-B7208E0D61EA@rcn.com> References: <3bWqYP3sqgzQdN@mail.python.org> <20130614104603.35f83a9c@fsol> <51BFF956.7040403@stoneleaf.us> <51C60C80.1010207@scottdial.com> <2278DBFF-B33B-4DEC-BADA-B7208E0D61EA@rcn.com> Message-ID: Hi Raymond, Thank you for your long explanation, it is exactly what Antoine asked for :-) I like micro-optimization even if I know that some other developers only see such work as noise, not providing an real speed up :-) So it was interesting to read your email! I'm sorry that you was injured by the revert of your commits (even I didn't do the revert myself, I agreed with Benjamin, and he's the maintainer of Python 2.7). I hope that you understood why they were rejected after reading this mail thread: the last Python 2.7 release (which one exactly?) contains regressions, and changes on the 2.7 stable branch are sensitive. By experience, there is always someone in the world relying to very specific implementation details. Cython, PyPy CPyExt, ctypes, etc. are sensitive to such change. People should not rely on such implementation details, but they do :-) For this specific case, I "hope" that nobody relies on the exact BLOCK structure (since it is a private structure). But it is just safer to not change it that in a minor Python version. If your compare the risk of introducing a regression versus the speedup, I guess that the risk is higher. As I wrote, I'm interested in micro-optimization, so please continue your work! It is just safer to only modify the active development branch (Python 3.4). Victor 2013/6/24 Raymond Hettinger : > > On Jun 18, 2013, at 12:00 AM, Gregory P. Smith wrote: > > Why did you do this in the 2.7 branch? > > It hasn't been done in 3.3 or default > > > I worked on the 2.7 branch first because that was the one I had loaded > and the one where I did timings and code disassembly. I intended to > post it to 3.3 and 3.4 as well over the weekend. > > Ideally, it makes maintenance simpler for me if I keep the branches > the same as possible. I viewed the one-line struct transposition, > comment correction, and one-line blocklen change as being somewhat > innocuous non-algorithmic changes. The struct change fixed an unintended > cache miss for left links and the blocksize change causes the > deque_index code to compile more efficiently (using a right-shift > and bitwise-and and rather than a measurably more expensive > division and modulo calculation). > > I truly wasn't expecting the Spanish Inquisition :-) > > > On Jun 22, 2013, at 1:43 PM, Scott Dial > wrote: > > changeset 84248:f1dc30a1be72 2.7 > Arrange structure to match the common access patterns. > > 1.1 --- a/Modules/_collectionsmodule.c > 1.2 +++ b/Modules/_collectionsmodule.c > 1.3 @@ -48,8 +48,8 @@ > 1.4 > 1.5 typedef struct BLOCK { > 1.6 struct BLOCK *leftlink; > 1.7 + PyObject *data[BLOCKLEN]; > 1.8 struct BLOCK *rightlink; > 1.9 - PyObject *data[BLOCKLEN]; > 1.10 } block; > 1.11 > 1.12 #define MAXFREEBLOCKS 10 > > , which seems like a strange micro-optimization, at best. > > > Yes, this is a micro-optimization. In working on implementing > deque slicing for 3.4, I restudied the block access patterns. > On an appendleft(), popleft() or extendleft() operation, the left link is > accessed immediately before or after the leftmost entry in the data block. > The old struct arrangement can cause an unnecessary cache miss > when jumping leftward. This was something I overlooked when I > originally wrote the code almost a decade ago. > > On Jun 23, 2013, at 11:38 AM, Benjamin Peterson wrote: > > I've backed this one out, too. > > > Really, you reverted my one-line change within 24 hours of it being posted? > I can't be on-line every day and sometimes it takes a little while to catch > up > with python email. > > On Jun 22, 2013, at 2:55 PM, Guido van Rossum wrote: > > Actually the data buffer is an array of pointers too, so with the > original BLOCKLEN value of 62, sizeof(block) would be 64 times > sizeof(PyObject *). In the Py3 version of the source there's even a > comment explaining this right before the #define BLOCKLEN. Raymond, > can you explain? > > > I also tried to fix that comment so it would stop emphasizing the blocklen > being a multiple of the cache line. Also long as there is a reasonably > long > data block, it matters less whether the data block size is an exact multiple > of the cache line length (62 or 64 words of data versus a typical 8 byte > cache line). > > The data block size does matter elsewhere though. > The benefit of the having the data block being a power-of-two > is that the deque_index computation can use bits shifts > rather division and modulo calculations. The benefit > of switching data block size from 62 to 64 was measurable > (a few percent) and observable in the disassembly of the code. > > I experimented with one other ordering as well > (putting the data block first and the links afterwards). > That saved the scaled-index byte in the generated code > but produced no measureable speed-up. > > In short, I believe the following should go in: > > * The comment fix. (Incorrectly suggesting that a 64-bit > Py_ssize_t would overflow). The revised comment > is simpler, more general, and correct. > > * Putting the leftlink before the data block in the structure. > > The follow is up for debate: > > Changing the BLOCKLEN from 62 to 64 is debatable. > It measureably improved deque_index without an > observable negative effect on the other operations. > > Lastly, there was a change I just put in to Py 3.4 replacing > the memcpy() with a simple loop and replacing the > "deque->" references with local variables. Besides > giving a small speed-up, it made the code more clear > and less at the mercy of various implementations > of memcpy(). > > Ideally, I would like 2.7 and 3.3 to replace their use of > memcpy() as well, but the flavor of this thread suggests > that is right out. > > > Raymond > > > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/victor.stinner%40gmail.com > From victor.stinner at gmail.com Mon Jun 24 12:50:01 2013 From: victor.stinner at gmail.com (Victor Stinner) Date: Mon, 24 Jun 2013 12:50:01 +0200 Subject: [Python-Dev] cpython (2.7): Fix comment blocks. Adjust blocksize to a power-of-two for better divmod In-Reply-To: <2278DBFF-B33B-4DEC-BADA-B7208E0D61EA@rcn.com> References: <3bWqYP3sqgzQdN@mail.python.org> <20130614104603.35f83a9c@fsol> <51BFF956.7040403@stoneleaf.us> <51C60C80.1010207@scottdial.com> <2278DBFF-B33B-4DEC-BADA-B7208E0D61EA@rcn.com> Message-ID: 2013/6/24 Raymond Hettinger : > Lastly, there was a change I just put in to Py 3.4 replacing > the memcpy() with a simple loop and replacing the > "deque->" references with local variables. Besides > giving a small speed-up, it made the code more clear > and less at the mercy of various implementations > of memcpy(). > > Ideally, I would like 2.7 and 3.3 to replace their use of > memcpy() as well, but the flavor of this thread suggests > that is right out. The specific memcpy() function is usually highly optimized with assembler code for each architecture. The GNU libc now does better: it can choose the fastest version depending on the CPU version (MMX, SSE, etc.) at runtime. If I understood correctly, the glibc contains different version of memcpy, and the dynamic linker (ld.so) chooses the version depending on the CPU. GCC is also able to inline memcpy() when the size is known at compile time. I also saw two code paths when the size is only known at runtime: inline version for small size, and function call for larger copy. Python has a Py_MEMCPY which implements exactly that, but only for Visual Studio. I suppose that Visual Studio does not implement this optimization. By the way, Py_MEMCPY() is only used in few places. So it's surprising to read that a dummy loop is faster than memcpy()... even if I already see this in my own micro-benchmarks :-) Do you have an idea on how we can decide between the dummy loop and memcpy()? Using a benchmark? Or can it be decided just by reading the C code? What is the policy for using Py_MEMCPY() vs memcpy()? Victor From victor.stinner at gmail.com Mon Jun 24 13:07:14 2013 From: victor.stinner at gmail.com (Victor Stinner) Date: Mon, 24 Jun 2013 13:07:14 +0200 Subject: [Python-Dev] cpython (2.7): Fix comment blocks. Adjust blocksize to a power-of-two for better divmod In-Reply-To: <2278DBFF-B33B-4DEC-BADA-B7208E0D61EA@rcn.com> References: <3bWqYP3sqgzQdN@mail.python.org> <20130614104603.35f83a9c@fsol> <51BFF956.7040403@stoneleaf.us> <51C60C80.1010207@scottdial.com> <2278DBFF-B33B-4DEC-BADA-B7208E0D61EA@rcn.com> Message-ID: 2013/6/24 Raymond Hettinger : > Changing the BLOCKLEN from 62 to 64 is debatable. > It measureably improved deque_index without an > observable negative effect on the other operations. Out of curiosity, do you know (remember) how was the number 62 chosen? Is it a compromise between memory usage and performances? 62 is surprising because it is not a power of two :-) Is it to just have 64 (2+62) pointers in the structure? (64 is a power of 2) Does it help the memory allocator (reduce memory fragmentation) to have a structure of 256 bytes (32-bit pointer) or 512 bytes (64-bit pointer), which are power of 2, instead of 264 or 528 bytes? It would be interesting to test pymalloc memory allocator on deque: I plan to test Python globally with PyMem_Malloc using pymalloc. pymalloc has a threshold of 512 bytes (and 528 bytes is greater than 512!). I suppose that the memory allocator is (much) more expensive than integer divisions. Would it be possible to change dynamically BLOCKLEN depending on the total size of the deque? Python dict does something like that, but for other reasons (reduce the number of hash collisions). According to your comment, "Larger numbers reduce the number of calls to the memory allocator, give faster indexing and rotation, and reduce the link::data overhead ratio." With dynamic BLOCKLEN, it would also be possible to reduce the memory usage for small deque (less than 62 items). Victor From amk at amk.ca Mon Jun 24 20:30:28 2013 From: amk at amk.ca (A.M. Kuchling) Date: Mon, 24 Jun 2013 14:30:28 -0400 Subject: [Python-Dev] [Python-checkins] cpython (2.7): Issue #18277: Document quirks of multiprocessing queue. In-Reply-To: <3bfBmR47hXz7Lls@mail.python.org> References: <3bfBmR47hXz7Lls@mail.python.org> Message-ID: <20130624183028.GA12278@datl9makuchling.home> On Mon, Jun 24, 2013 at 03:53:27PM +0200, richard.oudkerk wrote: > http://hg.python.org/cpython/rev/8dcc4e017d42 > + but should not cause any pratical difficulties -- you can always ^^ practical > + infinitessimal delay before the queue's :meth:`~Queue.empty` ^^ infinitesimal (or just "short") --amk From solipsis at pitrou.net Mon Jun 24 21:15:01 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 24 Jun 2013 21:15:01 +0200 Subject: [Python-Dev] cpython (2.7): Fix comment blocks. Adjust blocksize to a power-of-two for better divmod References: <3bWqYP3sqgzQdN@mail.python.org> <20130614104603.35f83a9c@fsol> <51BFF956.7040403@stoneleaf.us> <51C60C80.1010207@scottdial.com> <2278DBFF-B33B-4DEC-BADA-B7208E0D61EA@rcn.com> <51C7A674.4010706@scottdial.com> <85B0D1EF-A3E5-486A-8C7B-DCC297F181DD@gmail.com> Message-ID: <20130624211501.373a88b2@fsol> On Sun, 23 Jun 2013 20:37:37 -0700 Raymond Hettinger wrote: > But it isn't worth all the second guessing (and what feels like sniping). > I've worked on this code for almost a decade. As far as I can tell, none > of the participants in this thread has ever previously shown any interest > in the deque object. How do you measure "showing any interest"? I certainly use deques regularly (thank you for implementing them) and I'm sure others do too. I'm also interested in how the type is implemented and what the tradeoffs are there. Otherwise I wouldn't have noticed your commit. (if you remember, I was also interested enough to propose grafting deque-like functionality on OrderedDict: http://bugs.python.org/issue17100) Regards Antoine. From shibturn at gmail.com Mon Jun 24 21:35:23 2013 From: shibturn at gmail.com (Richard Oudkerk) Date: Mon, 24 Jun 2013 20:35:23 +0100 Subject: [Python-Dev] [Python-checkins] cpython (2.7): Issue #18277: Document quirks of multiprocessing queue. In-Reply-To: <20130624183028.GA12278@datl9makuchling.home> References: <3bfBmR47hXz7Lls@mail.python.org> <20130624183028.GA12278@datl9makuchling.home> Message-ID: On 24/06/2013 7:30pm, A.M. Kuchling wrote: > On Mon, Jun 24, 2013 at 03:53:27PM +0200, richard.oudkerk wrote: >> http://hg.python.org/cpython/rev/8dcc4e017d42 >> + but should not cause any pratical difficulties -- you can always > ^^ practical > >> + infinitessimal delay before the queue's :meth:`~Queue.empty` > ^^ infinitesimal (or just "short") > > --amk > Thanks. The first has already been pointed out. -- Richard From fijall at gmail.com Mon Jun 24 22:14:46 2013 From: fijall at gmail.com (Maciej Fijalkowski) Date: Mon, 24 Jun 2013 22:14:46 +0200 Subject: [Python-Dev] [Python-checkins] cpython (3.3): Add -b and -X options to python man page. In-Reply-To: <20130623161440.2587A250338@webabinitio.net> References: <3bbSw10tm2z7Ljf@mail.python.org> <20130623161440.2587A250338@webabinitio.net> Message-ID: On Sun, Jun 23, 2013 at 6:14 PM, R. David Murray wrote: > On Sun, 23 Jun 2013 17:40:13 +0200, Maciej Fijalkowski wrote: >> On Thu, Jun 20, 2013 at 3:36 PM, Brett Cannon wrote: >> > On Wed, Jun 19, 2013 at 11:20 PM, senthil.kumaran >> > wrote: >> >> .TP >> >> +.BI "\-X " option >> >> +Set implementation specific option. >> > >> > >> > Should probably be "Set the implementation-specific option." >> >> Is there anyone respecting this notation? (I know pypy does not, it >> uses --jit and stuff) > > CPython does. We introduced it for ourselves, it is up to other > implementations whether or not to use it, or use something else. > > --David you mean "CPython does not have any implementation-specific options"? I would claim -O behavior should be implementation-specific since it's nonsense in the optimizations sense, but other than that, it does not seem that there is any -X options? From alex.gaynor at gmail.com Mon Jun 24 22:17:30 2013 From: alex.gaynor at gmail.com (Alex Gaynor) Date: Mon, 24 Jun 2013 13:17:30 -0700 Subject: [Python-Dev] [Python-checkins] cpython (3.3): Add -b and -X options to python man page. In-Reply-To: References: <3bbSw10tm2z7Ljf@mail.python.org> <20130623161440.2587A250338@webabinitio.net> Message-ID: 3.3 adds some -X options around faulthandler if I recall correctly. Alex On Mon, Jun 24, 2013 at 1:14 PM, Maciej Fijalkowski wrote: > On Sun, Jun 23, 2013 at 6:14 PM, R. David Murray > wrote: > > On Sun, 23 Jun 2013 17:40:13 +0200, Maciej Fijalkowski > wrote: > >> On Thu, Jun 20, 2013 at 3:36 PM, Brett Cannon wrote: > >> > On Wed, Jun 19, 2013 at 11:20 PM, senthil.kumaran > >> > wrote: > >> >> .TP > >> >> +.BI "\-X " option > >> >> +Set implementation specific option. > >> > > >> > > >> > Should probably be "Set the implementation-specific option." > >> > >> Is there anyone respecting this notation? (I know pypy does not, it > >> uses --jit and stuff) > > > > CPython does. We introduced it for ourselves, it is up to other > > implementations whether or not to use it, or use something else. > > > > --David > > you mean "CPython does not have any implementation-specific options"? > I would claim -O behavior should be implementation-specific since it's > nonsense in the optimizations sense, but other than that, it does not > seem that there is any -X options? > _______________________________________________ > Python-checkins mailing list > Python-checkins at python.org > http://mail.python.org/mailman/listinfo/python-checkins > -- "I disapprove of what you say, but I will defend to the death your right to say it." -- Evelyn Beatrice Hall (summarizing Voltaire) "The people's good is the highest law." -- Cicero GPG Key fingerprint: 125F 5C67 DFE9 4084 -------------- next part -------------- An HTML attachment was scrubbed... URL: From rdmurray at bitdance.com Mon Jun 24 22:43:15 2013 From: rdmurray at bitdance.com (R. David Murray) Date: Mon, 24 Jun 2013 16:43:15 -0400 Subject: [Python-Dev] [Python-checkins] cpython (3.3): Add -b and -X options to python man page. In-Reply-To: References: <3bbSw10tm2z7Ljf@mail.python.org> <20130623161440.2587A250338@webabinitio.net> Message-ID: <20130624204316.028CB2500E3@webabinitio.net> On Mon, 24 Jun 2013 22:14:46 +0200, Maciej Fijalkowski wrote: > On Sun, Jun 23, 2013 at 6:14 PM, R. David Murray wrote: > > On Sun, 23 Jun 2013 17:40:13 +0200, Maciej Fijalkowski wrote: > >> On Thu, Jun 20, 2013 at 3:36 PM, Brett Cannon wrote: > >> > On Wed, Jun 19, 2013 at 11:20 PM, senthil.kumaran > >> > wrote: > >> >> .TP > >> >> +.BI "\-X " option > >> >> +Set implementation specific option. > >> > > >> > > >> > Should probably be "Set the implementation-specific option." > >> > >> Is there anyone respecting this notation? (I know pypy does not, it > >> uses --jit and stuff) > > > > CPython does. We introduced it for ourselves, it is up to other > > implementations whether or not to use it, or use something else. > > > > --David > > you mean "CPython does not have any implementation-specific options"? > I would claim -O behavior should be implementation-specific since it's > nonsense in the optimizations sense, but other than that, it does not > seem that there is any -X options? There is one. -X faulthandler. I'm sure others would agree about -O, but that long predates -X. So, the idea is that -X *can* be used by other implementations for their own purposes, but there is certainly no requirement that they do so. Our promise is that anything CPython uses it for is something we don't expect other implementations to support. --David From benjamin at python.org Mon Jun 24 22:43:45 2013 From: benjamin at python.org (Benjamin Peterson) Date: Mon, 24 Jun 2013 13:43:45 -0700 Subject: [Python-Dev] [Python-checkins] cpython (3.3): Add -b and -X options to python man page. In-Reply-To: References: <3bbSw10tm2z7Ljf@mail.python.org> <20130623161440.2587A250338@webabinitio.net> Message-ID: 2013/6/24 Maciej Fijalkowski : > On Sun, Jun 23, 2013 at 6:14 PM, R. David Murray wrote: >> On Sun, 23 Jun 2013 17:40:13 +0200, Maciej Fijalkowski wrote: >>> On Thu, Jun 20, 2013 at 3:36 PM, Brett Cannon wrote: >>> > On Wed, Jun 19, 2013 at 11:20 PM, senthil.kumaran >>> > wrote: >>> >> .TP >>> >> +.BI "\-X " option >>> >> +Set implementation specific option. >>> > >>> > >>> > Should probably be "Set the implementation-specific option." >>> >>> Is there anyone respecting this notation? (I know pypy does not, it >>> uses --jit and stuff) >> >> CPython does. We introduced it for ourselves, it is up to other >> implementations whether or not to use it, or use something else. >> >> --David > > you mean "CPython does not have any implementation-specific options"? > I would claim -O behavior should be implementation-specific since it's > nonsense in the optimizations sense, but other than that, it does not > seem that there is any -X options? I wouldn't object to making that -Xno-docstrings or such, but the ship sailed long ago on -O. -- Regards, Benjamin From brett at python.org Mon Jun 24 23:00:26 2013 From: brett at python.org (Brett Cannon) Date: Mon, 24 Jun 2013 17:00:26 -0400 Subject: [Python-Dev] [Python-checkins] cpython (3.3): Add -b and -X options to python man page. In-Reply-To: References: <3bbSw10tm2z7Ljf@mail.python.org> <20130623161440.2587A250338@webabinitio.net> Message-ID: On Mon, Jun 24, 2013 at 4:43 PM, Benjamin Peterson wrote: > 2013/6/24 Maciej Fijalkowski : > > On Sun, Jun 23, 2013 at 6:14 PM, R. David Murray > wrote: > >> On Sun, 23 Jun 2013 17:40:13 +0200, Maciej Fijalkowski < > fijall at gmail.com> wrote: > >>> On Thu, Jun 20, 2013 at 3:36 PM, Brett Cannon > wrote: > >>> > On Wed, Jun 19, 2013 at 11:20 PM, senthil.kumaran > >>> > wrote: > >>> >> .TP > >>> >> +.BI "\-X " option > >>> >> +Set implementation specific option. > >>> > > >>> > > >>> > Should probably be "Set the implementation-specific option." > >>> > >>> Is there anyone respecting this notation? (I know pypy does not, it > >>> uses --jit and stuff) > >> > >> CPython does. We introduced it for ourselves, it is up to other > >> implementations whether or not to use it, or use something else. > >> > >> --David > > > > you mean "CPython does not have any implementation-specific options"? > > I would claim -O behavior should be implementation-specific since it's > > nonsense in the optimizations sense, but other than that, it does not > > seem that there is any -X options? > > I wouldn't object to making that -Xno-docstrings or such, but the ship > sailed long ago on -O. > Python 4 change! =) -------------- next part -------------- An HTML attachment was scrubbed... URL: From jeremy.kloth at gmail.com Mon Jun 24 23:48:24 2013 From: jeremy.kloth at gmail.com (Jeremy Kloth) Date: Mon, 24 Jun 2013 15:48:24 -0600 Subject: [Python-Dev] [Python-checkins] cpython: Issue #9566: Fix a compiler warning in tupleiter_setstate() on Windows x64 In-Reply-To: <3bfNzS6mryz7Lpx@mail.python.org> References: <3bfNzS6mryz7Lpx@mail.python.org> Message-ID: On Mon, Jun 24, 2013 at 3:33 PM, victor.stinner wrote: > http://hg.python.org/cpython/rev/6b4d279508a3 > changeset: 84325:6b4d279508a3 > user: Victor Stinner > date: Mon Jun 24 23:31:48 2013 +0200 > summary: > Issue #9566: Fix a compiler warning in tupleiter_setstate() on Windows x64 > > files: > Objects/tupleobject.c | 2 +- > 1 files changed, 1 insertions(+), 1 deletions(-) > > > diff --git a/Objects/tupleobject.c b/Objects/tupleobject.c > --- a/Objects/tupleobject.c > +++ b/Objects/tupleobject.c > @@ -997,7 +997,7 @@ > static PyObject * > tupleiter_setstate(tupleiterobject *it, PyObject *state) > { > - long index = PyLong_AsLong(state); > + Py_ssize_t index = PyLong_AsLong(state); Actually, this will still lose data when state > MAX_INT (on Windows). It should be changed to PyLong_AsSssize_t(state) to ensure consistent behavior on all platforms. -- Jeremy Kloth From victor.stinner at gmail.com Tue Jun 25 00:00:41 2013 From: victor.stinner at gmail.com (Victor Stinner) Date: Tue, 25 Jun 2013 00:00:41 +0200 Subject: [Python-Dev] [Python-checkins] cpython: Issue #9566: Fix a compiler warning in tupleiter_setstate() on Windows x64 In-Reply-To: References: <3bfNzS6mryz7Lpx@mail.python.org> Message-ID: Ah yes correct, it should be better with the following commit: http://hg.python.org/cpython/rev/3a393fc86b29 Victor 2013/6/24 Jeremy Kloth : > On Mon, Jun 24, 2013 at 3:33 PM, victor.stinner > wrote: >> http://hg.python.org/cpython/rev/6b4d279508a3 >> changeset: 84325:6b4d279508a3 >> user: Victor Stinner >> date: Mon Jun 24 23:31:48 2013 +0200 >> summary: >> Issue #9566: Fix a compiler warning in tupleiter_setstate() on Windows x64 >> ... >> static PyObject * >> tupleiter_setstate(tupleiterobject *it, PyObject *state) >> { >> - long index = PyLong_AsLong(state); >> + Py_ssize_t index = PyLong_AsLong(state); > > Actually, this will still lose data when state > MAX_INT (on Windows). > It should be changed to PyLong_AsSssize_t(state) to ensure consistent > behavior on all platforms. From jeremy.kloth at gmail.com Tue Jun 25 00:08:01 2013 From: jeremy.kloth at gmail.com (Jeremy Kloth) Date: Mon, 24 Jun 2013 16:08:01 -0600 Subject: [Python-Dev] [Python-checkins] cpython: Issue #9566: recv(), recvfrom(), send(), sendall() and sendto() methods In-Reply-To: <3bfPJ14MyCz7LqX@mail.python.org> References: <3bfPJ14MyCz7LqX@mail.python.org> Message-ID: On Mon, Jun 24, 2013 at 3:48 PM, victor.stinner wrote: > +#if defined(MS_WIN64) || defined(MS_WINDOWS) This test could be reduced to simply `#ifdef MS_WINDOWS`. See PC\pyconfig.h From victor.stinner at gmail.com Tue Jun 25 00:28:09 2013 From: victor.stinner at gmail.com (Victor Stinner) Date: Tue, 25 Jun 2013 00:28:09 +0200 Subject: [Python-Dev] [Python-checkins] cpython (3.3): Add -b and -X options to python man page. In-Reply-To: <20130624204316.028CB2500E3@webabinitio.net> References: <3bbSw10tm2z7Ljf@mail.python.org> <20130623161440.2587A250338@webabinitio.net> <20130624204316.028CB2500E3@webabinitio.net> Message-ID: 2013/6/24 R. David Murray : > There is one. -X faulthandler. I'm sure others would agree about > -O, but that long predates -X. FYI I didn't chose "-X" because it is specific to CPython, but just because it becomes really hard to choose a random letter to add a new option... I prefer long options, but Python only supports the long option --help. IMO "python -X faulthandler" is more explicit than "python -@" (or "python -f", or any other random letter). If you didn't know, you can also write "python -X faulthandler=1", it does also enable faulthandler... As "python -X faulthandler=0" and "python -X faulthandler=False" :-) (PHP style!) > So, the idea is that -X *can* be used by other implementations for their > own purposes, but there is certainly no requirement that they do so. > Our promise is that anything CPython uses it for is something we don't > expect other implementations to support. It would be nice to have faulthandler or something similar (dump traces on a crash) in other Python implementations :-) faulthandler implementation is very specific to CPython. It uses internal structures and low-level OS features like signal handler and a specific stack for its signal handler. It is better to have it integrated directly into CPython. (But it is also available as a third party module for older versions.) Victor From victor.stinner at gmail.com Tue Jun 25 00:40:14 2013 From: victor.stinner at gmail.com (Victor Stinner) Date: Tue, 25 Jun 2013 00:40:14 +0200 Subject: [Python-Dev] [Python-checkins] cpython: Issue #9566: recv(), recvfrom(), send(), sendall() and sendto() methods In-Reply-To: References: <3bfPJ14MyCz7LqX@mail.python.org> Message-ID: Oh, I didn't know that."if defined(MS_WIN64) || defined(MS_WINDOWS)" is a common pattern in the Python source code. I simplified the #ifdef in many places: http://hg.python.org/cpython/rev/dfc020b4b123 I also read that MS_WIN32 is always defined on Windows. "#ifdef MS_WIN32" (used in many files, especially in ctypes) can probably be replaced with "#ifdef MS_WINDOWS" for consistency. MS_WIN32 should only be used to check if the API is Win64: /* MSVC defines _WINxx to differentiate the windows platform types Note that for compatibility reasons _WIN32 is defined on Win32 *and* on Win64. For the same reasons, in Python, MS_WIN32 is defined on Win32 *and* Win64. Win32 only code must therefore be guarded as follows: #if defined(MS_WIN32) && !defined(MS_WIN64) Some modules are disabled on Itanium processors, therefore we have MS_WINI64 set for those targets, otherwise MS_WINX64 */ Victor 2013/6/25 Jeremy Kloth : > On Mon, Jun 24, 2013 at 3:48 PM, victor.stinner > wrote: >> +#if defined(MS_WIN64) || defined(MS_WINDOWS) > > This test could be reduced to simply `#ifdef MS_WINDOWS`. See PC\pyconfig.h From steve at pearwood.info Tue Jun 25 01:01:15 2013 From: steve at pearwood.info (Steven D'Aprano) Date: Tue, 25 Jun 2013 09:01:15 +1000 Subject: [Python-Dev] [Python-checkins] cpython (3.3): Add -b and -X options to python man page. In-Reply-To: References: <3bbSw10tm2z7Ljf@mail.python.org> <20130623161440.2587A250338@webabinitio.net> <20130624204316.028CB2500E3@webabinitio.net> Message-ID: <20130624230115.GB10635@ando> On Tue, Jun 25, 2013 at 12:28:09AM +0200, Victor Stinner wrote: > 2013/6/24 R. David Murray : > > There is one. -X faulthandler. I'm sure others would agree about > > -O, but that long predates -X. > > FYI I didn't chose "-X" because it is specific to CPython, but just > because it becomes really hard to choose a random letter to add a new > option... I prefer long options, but Python only supports the long > option --help. Is that a policy though? Couldn't Python support long options, or is it prohibited? -- Steven From ncoghlan at gmail.com Tue Jun 25 01:43:40 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 25 Jun 2013 09:43:40 +1000 Subject: [Python-Dev] [Python-checkins] cpython (3.3): Add -b and -X options to python man page. In-Reply-To: <20130624230115.GB10635@ando> References: <3bbSw10tm2z7Ljf@mail.python.org> <20130623161440.2587A250338@webabinitio.net> <20130624204316.028CB2500E3@webabinitio.net> <20130624230115.GB10635@ando> Message-ID: On 25 Jun 2013 09:03, "Steven D'Aprano" wrote: > > On Tue, Jun 25, 2013 at 12:28:09AM +0200, Victor Stinner wrote: > > 2013/6/24 R. David Murray : > > > There is one. -X faulthandler. I'm sure others would agree about > > > -O, but that long predates -X. > > > > FYI I didn't chose "-X" because it is specific to CPython, but just > > because it becomes really hard to choose a random letter to add a new > > option... I prefer long options, but Python only supports the long > > option --help. > > Is that a policy though? Couldn't Python support long options, or is it > prohibited? Just impractical. It will likely be more feasible to support them in a maintainable way once PEP 432 is done and we can lean on the C API more heavily while processing command line arguments. Cheers, Nick. > > > -- > Steven > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From raymond.hettinger at gmail.com Tue Jun 25 04:47:51 2013 From: raymond.hettinger at gmail.com (Raymond Hettinger) Date: Mon, 24 Jun 2013 19:47:51 -0700 Subject: [Python-Dev] cpython (2.7): Fix comment blocks. Adjust blocksize to a power-of-two for better divmod In-Reply-To: References: <3bWqYP3sqgzQdN@mail.python.org> <20130614104603.35f83a9c@fsol> <51BFF956.7040403@stoneleaf.us> <51C60C80.1010207@scottdial.com> <2278DBFF-B33B-4DEC-BADA-B7208E0D61EA@rcn.com> Message-ID: On Jun 24, 2013, at 4:07 AM, Victor Stinner wrote: > Out of curiosity, do you know (remember) how was the number 62 chosen? > Is it a compromise between memory usage and performances? 62 is > surprising because it is not a power of two :-) > > Is it to just have 64 (2+62) pointers in the structure? Yes, the goal was to have the struct size be an exact multiple of the cache line length (always a power-of-two, typically 64 bytes). What was different then is that deques weren't indexable. When indexing was added, the size of 62 became an unfavorable choice because it made the division and modulo calculation in deque_index() slower than for a power of two. > For this specific case, I "hope" that nobody relies on the exact BLOCK > structure (since it is a private structure). I don't know what you're talking about. This structure isn't externally visible. Raymond -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Tue Jun 25 06:10:18 2013 From: guido at python.org (Guido van Rossum) Date: Mon, 24 Jun 2013 21:10:18 -0700 Subject: [Python-Dev] cpython (2.7): Fix comment blocks. Adjust blocksize to a power-of-two for better divmod In-Reply-To: References: <3bWqYP3sqgzQdN@mail.python.org> <20130614104603.35f83a9c@fsol> <51BFF956.7040403@stoneleaf.us> <51C60C80.1010207@scottdial.com> <2278DBFF-B33B-4DEC-BADA-B7208E0D61EA@rcn.com> Message-ID: On Mon, Jun 24, 2013 at 7:47 PM, Raymond Hettinger < raymond.hettinger at gmail.com> wrote: > > > On Jun 24, 2013, at 4:07 AM, Victor Stinner > wrote: > > Out of curiosity, do you know (remember) how was the number 62 chosen? > Is it a compromise between memory usage and performances? 62 is > surprising because it is not a power of two :-) > > Is it to just have 64 (2+62) pointers in the structure? > > > Yes, the goal was to have the struct size be an exact multiple > of the cache line length (always a power-of-two, typically 64 bytes). > What was different then is that deques weren't indexable. > When indexing was added, the size of 62 became an > unfavorable choice because it made the division and modulo > calculation in deque_index() slower than for a power of two. > A-ha! Finally an explanation of the change. It makes intuitive sense now. I think the general feeling is that folks overreacted (perhaps confused by your silence) and that the reversal will be rolled back. Benjamin? -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From greg at krypto.org Tue Jun 25 07:07:30 2013 From: greg at krypto.org (Gregory P. Smith) Date: Mon, 24 Jun 2013 22:07:30 -0700 Subject: [Python-Dev] cpython (2.7): Fix comment blocks. Adjust blocksize to a power-of-two for better divmod In-Reply-To: References: <3bWqYP3sqgzQdN@mail.python.org> <20130614104603.35f83a9c@fsol> <51BFF956.7040403@stoneleaf.us> <51C60C80.1010207@scottdial.com> <2278DBFF-B33B-4DEC-BADA-B7208E0D61EA@rcn.com> Message-ID: On Jun 24, 2013 9:11 PM, "Guido van Rossum" wrote: > > On Mon, Jun 24, 2013 at 7:47 PM, Raymond Hettinger < raymond.hettinger at gmail.com> wrote: >> >> >> >> On Jun 24, 2013, at 4:07 AM, Victor Stinner wrote: >> >>> Out of curiosity, do you know (remember) how was the number 62 chosen? >>> Is it a compromise between memory usage and performances? 62 is >>> surprising because it is not a power of two :-) >>> >>> Is it to just have 64 (2+62) pointers in the structure? >> >> >> Yes, the goal was to have the struct size be an exact multiple >> of the cache line length (always a power-of-two, typically 64 bytes). >> What was different then is that deques weren't indexable. >> When indexing was added, the size of 62 became an >> unfavorable choice because it made the division and modulo >> calculation in deque_index() slower than for a power of two. > > > A-ha! Finally an explanation of the change. It makes intuitive sense now. I think the general feeling is that folks overreacted (perhaps confused by your silence) and that the reversal will be rolled back. Benjamin? Seems likely to me. > > -- > --Guido van Rossum (python.org/~guido) > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/greg%40krypto.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: From benjamin at python.org Tue Jun 25 07:12:43 2013 From: benjamin at python.org (Benjamin Peterson) Date: Mon, 24 Jun 2013 22:12:43 -0700 Subject: [Python-Dev] cpython (2.7): Fix comment blocks. Adjust blocksize to a power-of-two for better divmod In-Reply-To: References: <3bWqYP3sqgzQdN@mail.python.org> <20130614104603.35f83a9c@fsol> <51BFF956.7040403@stoneleaf.us> <51C60C80.1010207@scottdial.com> <2278DBFF-B33B-4DEC-BADA-B7208E0D61EA@rcn.com> Message-ID: 2013/6/24 Guido van Rossum : > On Mon, Jun 24, 2013 at 7:47 PM, Raymond Hettinger > wrote: >> >> >> >> On Jun 24, 2013, at 4:07 AM, Victor Stinner >> wrote: >> >> Out of curiosity, do you know (remember) how was the number 62 chosen? >> Is it a compromise between memory usage and performances? 62 is >> surprising because it is not a power of two :-) >> >> Is it to just have 64 (2+62) pointers in the structure? >> >> >> Yes, the goal was to have the struct size be an exact multiple >> of the cache line length (always a power-of-two, typically 64 bytes). >> What was different then is that deques weren't indexable. >> When indexing was added, the size of 62 became an >> unfavorable choice because it made the division and modulo >> calculation in deque_index() slower than for a power of two. > > > A-ha! Finally an explanation of the change. It makes intuitive sense now. I > think the general feeling is that folks overreacted (perhaps confused by > your silence) and that the reversal will be rolled back. Benjamin? Raymond, go ahead and reapply your change. -- Regards, Benjamin From fijall at gmail.com Tue Jun 25 11:12:23 2013 From: fijall at gmail.com (Maciej Fijalkowski) Date: Tue, 25 Jun 2013 11:12:23 +0200 Subject: [Python-Dev] [Python-checkins] cpython (3.3): Add -b and -X options to python man page. In-Reply-To: <20130624225629.72d9cf05@fsol> References: <3bbSw10tm2z7Ljf@mail.python.org> <20130623161440.2587A250338@webabinitio.net> <20130624204316.028CB2500E3@webabinitio.net> <20130624225629.72d9cf05@fsol> Message-ID: On Mon, Jun 24, 2013 at 10:56 PM, Antoine Pitrou wrote: > On Mon, 24 Jun 2013 16:43:15 -0400 > "R. David Murray" wrote: >> On Mon, 24 Jun 2013 22:14:46 +0200, Maciej Fijalkowski wrote: >> > On Sun, Jun 23, 2013 at 6:14 PM, R. David Murray wrote: >> > > On Sun, 23 Jun 2013 17:40:13 +0200, Maciej Fijalkowski wrote: >> > >> On Thu, Jun 20, 2013 at 3:36 PM, Brett Cannon wrote: >> > >> > On Wed, Jun 19, 2013 at 11:20 PM, senthil.kumaran >> > >> > wrote: >> > >> >> .TP >> > >> >> +.BI "\-X " option >> > >> >> +Set implementation specific option. >> > >> > >> > >> > >> > >> > Should probably be "Set the implementation-specific option." >> > >> >> > >> Is there anyone respecting this notation? (I know pypy does not, it >> > >> uses --jit and stuff) >> > > >> > > CPython does. We introduced it for ourselves, it is up to other >> > > implementations whether or not to use it, or use something else. >> > > >> > > --David >> > >> > you mean "CPython does not have any implementation-specific options"? >> > I would claim -O behavior should be implementation-specific since it's >> > nonsense in the optimizations sense, but other than that, it does not >> > seem that there is any -X options? >> >> There is one. -X faulthandler. I'm sure others would agree about >> -O, but that long predates -X. >> >> So, the idea is that -X *can* be used by other implementations for their >> own purposes, but there is certainly no requirement that they do so. >> Our promise is that anything CPython uses it for is something we don't >> expect other implementations to support. > > Yes, basically -X is a private namespace for every implementation to > use as it sees fit without fearing of conflicting with a future > cross-implementation option. > > Regards > > Antoine. I'm for one looking forward to the day where cpython conflicts on --jit ;-) From brett at yvrsfo.ca Tue Jun 25 17:45:00 2013 From: brett at yvrsfo.ca (Brett Cannon) Date: Tue, 25 Jun 2013 11:45:00 -0400 Subject: [Python-Dev] [Python-checkins] cpython: Issue #18081: Workaround "./python -m test_idle test_logging" failure In-Reply-To: <3bfPyS4crdz7Lll@mail.python.org> References: <3bfPyS4crdz7Lll@mail.python.org> Message-ID: This fix is still not correct as warnings.formatwarning should not be overridden, only showwarning can be. On Jun 24, 2013 6:18 PM, "victor.stinner" wrote: > http://hg.python.org/cpython/rev/2a9e1eb3719c > changeset: 84330:2a9e1eb3719c > user: Victor Stinner > date: Tue Jun 25 00:17:37 2013 +0200 > summary: > Issue #18081: Workaround "./python -m test_idle test_logging" failure > > "import idlelib" should not install hooks on the warning modules, hooks > should > only be installed when IDLE is started. > > files: > Lib/idlelib/PyShell.py | 5 +++-- > Lib/idlelib/run.py | 4 +++- > 2 files changed, 6 insertions(+), 3 deletions(-) > > > diff --git a/Lib/idlelib/PyShell.py b/Lib/idlelib/PyShell.py > --- a/Lib/idlelib/PyShell.py > +++ b/Lib/idlelib/PyShell.py > @@ -61,7 +61,6 @@ > lineno, line=line)) > except OSError: > pass ## file (probably __stderr__) is invalid, warning > dropped. > - warnings.showwarning = idle_showwarning > def idle_formatwarning(message, category, filename, lineno, > line=None): > """Format warnings the IDLE way""" > s = "\nWarning (from warnings module):\n" > @@ -73,7 +72,6 @@ > s += " %s\n" % line > s += "%s: %s\n>>> " % (category.__name__, message) > return s > - warnings.formatwarning = idle_formatwarning > > def extended_linecache_checkcache(filename=None, > orig_checkcache=linecache.checkcache): > @@ -1425,6 +1423,9 @@ > def main(): > global flist, root, use_subprocess > > + warnings.showwarning = idle_showwarning > + warnings.formatwarning = idle_formatwarning > + > use_subprocess = True > enable_shell = False > enable_edit = False > diff --git a/Lib/idlelib/run.py b/Lib/idlelib/run.py > --- a/Lib/idlelib/run.py > +++ b/Lib/idlelib/run.py > @@ -40,7 +40,6 @@ > s += " %s\n" % line > s += "%s: %s\n" % (category.__name__, message) > return s > - warnings.formatwarning = idle_formatwarning_subproc > > > tcl = tkinter.Tcl() > @@ -82,6 +81,9 @@ > global exit_now > global quitting > global no_exitfunc > + > + warnings.formatwarning = idle_formatwarning_subproc > + > no_exitfunc = del_exitfunc > #time.sleep(15) # test subprocess not responding > try: > > -- > Repository URL: http://hg.python.org/cpython > > _______________________________________________ > Python-checkins mailing list > Python-checkins at python.org > http://mail.python.org/mailman/listinfo/python-checkins > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From raymond.hettinger at gmail.com Tue Jun 25 19:59:33 2013 From: raymond.hettinger at gmail.com (Raymond Hettinger) Date: Tue, 25 Jun 2013 10:59:33 -0700 Subject: [Python-Dev] cpython (2.7): Fix comment blocks. Adjust blocksize to a power-of-two for better divmod In-Reply-To: References: <3bWqYP3sqgzQdN@mail.python.org> <20130614104603.35f83a9c@fsol> <51BFF956.7040403@stoneleaf.us> <51C60C80.1010207@scottdial.com> <2278DBFF-B33B-4DEC-BADA-B7208E0D61EA@rcn.com> Message-ID: <70F355A6-F3E8-4E1C-B5E7-42187FDEF0F9@gmail.com> On Jun 24, 2013, at 10:12 PM, Benjamin Peterson wrote: > Raymond, go ahead and reapply your change. If you don't mind, I think you should be the one to undo your own reversions. Thank you, Raymond -------------- next part -------------- An HTML attachment was scrubbed... URL: From jdhardy at gmail.com Tue Jun 25 20:05:15 2013 From: jdhardy at gmail.com (Jeff Hardy) Date: Tue, 25 Jun 2013 11:05:15 -0700 Subject: [Python-Dev] [Python-checkins] cpython (3.3): Add -b and -X options to python man page. In-Reply-To: References: <3bbSw10tm2z7Ljf@mail.python.org> Message-ID: On Sun, Jun 23, 2013 at 8:40 AM, Maciej Fijalkowski wrote: > >> +.BI "\-X " option > >> +Set implementation specific option. > > > > > > Should probably be "Set the implementation-specific option." > > Is there anyone respecting this notation? (I know pypy does not, it > uses --jit and stuff) > > IronPython does, for -X:Frames and a few others. - Jeff -------------- next part -------------- An HTML attachment was scrubbed... URL: From benjamin at python.org Tue Jun 25 20:32:44 2013 From: benjamin at python.org (Benjamin Peterson) Date: Tue, 25 Jun 2013 11:32:44 -0700 Subject: [Python-Dev] cpython (2.7): Fix comment blocks. Adjust blocksize to a power-of-two for better divmod In-Reply-To: <70F355A6-F3E8-4E1C-B5E7-42187FDEF0F9@gmail.com> References: <3bWqYP3sqgzQdN@mail.python.org> <20130614104603.35f83a9c@fsol> <51BFF956.7040403@stoneleaf.us> <51C60C80.1010207@scottdial.com> <2278DBFF-B33B-4DEC-BADA-B7208E0D61EA@rcn.com> <70F355A6-F3E8-4E1C-B5E7-42187FDEF0F9@gmail.com> Message-ID: 2013/6/25 Raymond Hettinger : > > On Jun 24, 2013, at 10:12 PM, Benjamin Peterson wrote: > > Raymond, go ahead and reapply your change. > > > If you don't mind, I think you should be the one to undo your own > reversions. Done. -- Regards, Benjamin From vitaly.murashev at gmail.com Tue Jun 25 22:11:04 2013 From: vitaly.murashev at gmail.com (Vitaly Murashev) Date: Wed, 26 Jun 2013 00:11:04 +0400 Subject: [Python-Dev] Relative path in co_filename for zipped modules Message-ID: Dear Python developers community, Recently I found out that it not possible to debug python code if it is a part of zip-module. Python version being used is 3.3.0 Well known GUI debuggers like Eclipse+PyDev or PyCharm are unable to start debugging and give the following warning: --- pydev debugger: CRITICAL WARNING: This version of python seems to be incorrectly compiled (internal generated filenames are not absolute) pydev debugger: The debugger may still function, but it will work slower and may miss breakpoints. --- So I started my own investigation of this issue and results are the following. At first I took traditional python debugger 'pdb' to analyze how it behaves during debugging of zipped module. 'pdb' showed me some backtaces and filename part for stack entries looks malformed. I expected something like 'full-path-to-zip-dir/my_zipped_module.zip/subdir/test_module.py' but realy it looks like 'full-path-to-current-dir/subdir/test_module.py' Source code in pdb.py and bdb.py (which one are a part of python stdlib) gave me the answer why it happens. The root cause are inside Bdb.format_stack_entry() + Bdb.canonic() Please take a look at the following line inside 'format_stack_entry' method: filename = self.canonic(frame.f_code.co_filename) For zipped module variable 'frame.f_code.co_filename' holds _relative_ file path started from the root of zip archive like 'subdir/test_module.py' And as result Bdb.canonic() method gives what we have - 'full-path-to-current-dir/subdir/test_module.py' --- So my final question. Could anyone confirm that it is a bug in python core subsystem which one is responsible for loading zipped modules, or something is wrong with my zipped module ? If it is a bug could you please submit it into official python bugtracker ? I never did it before and afraid to do something wrong. Thanks, Vitaly Murashev From rdmurray at bitdance.com Tue Jun 25 22:19:57 2013 From: rdmurray at bitdance.com (R. David Murray) Date: Tue, 25 Jun 2013 16:19:57 -0400 Subject: [Python-Dev] Relative path in co_filename for zipped modules In-Reply-To: References: Message-ID: <20130625201958.29E8B250B3F@webabinitio.net> Please file this as a bug report on bugs.python.org so that it doesn't get lost. See also http://bugs.python.org/issue13328 (which looks like a different bug but could also be causing you problems). --David On Wed, 26 Jun 2013 00:11:04 +0400, Vitaly Murashev wrote: > Dear Python developers community, > > Recently I found out that it not possible to debug python code if it > is a part of zip-module. > Python version being used is 3.3.0 > > Well known GUI debuggers like Eclipse+PyDev or PyCharm are unable to > start debugging and give the following warning: > --- > pydev debugger: CRITICAL WARNING: This version of python seems to be > incorrectly compiled (internal generated filenames are not absolute) > pydev debugger: The debugger may still function, but it will work > slower and may miss breakpoints. > --- > So I started my own investigation of this issue and results are the following. > > At first I took traditional python debugger 'pdb' to analyze how it > behaves during debugging of zipped module. > 'pdb' showed me some backtaces and filename part for stack entries > looks malformed. > I expected something like > 'full-path-to-zip-dir/my_zipped_module.zip/subdir/test_module.py' > but realy it looks like 'full-path-to-current-dir/subdir/test_module.py' > > Source code in pdb.py and bdb.py (which one are a part of python > stdlib) gave me the answer why it happens. > > The root cause are inside Bdb.format_stack_entry() + Bdb.canonic() > > Please take a look at the following line inside 'format_stack_entry' method: > > filename = self.canonic(frame.f_code.co_filename) > > For zipped module variable 'frame.f_code.co_filename' holds _relative_ > file path started from the root of zip archive like > 'subdir/test_module.py' > And as result Bdb.canonic() method gives what we have - > 'full-path-to-current-dir/subdir/test_module.py' > --- > So my final question. > Could anyone confirm that it is a bug in python core subsystem which > one is responsible for loading zipped modules, > or something is wrong with my zipped module ? > > If it is a bug could you please submit it into official python bugtracker ? > I never did it before and afraid to do something wrong. > > Thanks, > Vitaly Murashev > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/rdmurray%40bitdance.com From victor.stinner at gmail.com Wed Jun 26 00:44:07 2013 From: victor.stinner at gmail.com (Victor Stinner) Date: Wed, 26 Jun 2013 00:44:07 +0200 Subject: [Python-Dev] End of the mystery "@README.txt Mercurial bug" Message-ID: Hi, One month ago, unit tests were added to IDLE (cool!) with a file called @README.txt. The @ was used to see the name on top in a listing of the directory. Some developers began to get strange Mercurial errors like: "abort: data/Lib/idlelib/idle_test/@README.txt.i at 7573717b9e6f: no match" " 83941: empty or missing Lib/idlelib/idle_test/@README.txt " etc. Senthil reported the issue on python-committers mailing list: http://mail.python.org/pipermail/python-committers/2013-May/002565.html The @ character was discussed. Replacing it with "_" was proposed. Guido asked to simply rename the name "README.txt", he wrote: "I think we have a zen rule about this: Special cases aren't special enough to break the rules." Senthil was asked to upgrade its Mercurial version. Someone supposed that it is a disk issue. Anyway, the issue disappeared with a fresh clone. I also had the issue on 3 different computers, and so I reported the issue upstream: http://bz.selenic.com/show_bug.cgi?id=3954 I discussed with a Mercurial developer, Matt, on IRC. He asked how the server is managed, if we are using only one physical server, if repositories are copied with rsync, etc. I was unable to answer, I don't have access to hg.python.org server. The issue was closed, but 20 days later (today) I reproduced the issue again. I cloned the repository at a specific revision, tried to update to another specific revision: no bug. I also played with with hg bisect, because I suspected a bug in bisect: no bug. I tried to update at each revision between 83900 and 84348 to check if @README.txt disappears from .hg/store: still no bug. I also ran fsck: no error (but the disk is mounted, so I don't know if the report is reliable). And then I ran "make distclean"... Victor From benjamin at python.org Wed Jun 26 00:58:30 2013 From: benjamin at python.org (Benjamin Peterson) Date: Tue, 25 Jun 2013 15:58:30 -0700 Subject: [Python-Dev] End of the mystery "@README.txt Mercurial bug" In-Reply-To: References: Message-ID: 2013/6/25 Victor Stinner : > And then I ran "make distclean"... You've left us hanging... -- Regards, Benjamin From senthil at uthcode.com Wed Jun 26 03:33:05 2013 From: senthil at uthcode.com (Senthil Kumaran) Date: Tue, 25 Jun 2013 20:33:05 -0500 Subject: [Python-Dev] End of the mystery "@README.txt Mercurial bug" In-Reply-To: References: Message-ID: On Tue, Jun 25, 2013 at 5:58 PM, Benjamin Peterson wrote: > 2013/6/25 Victor Stinner : > > And then I ran "make distclean"... > > You've left us hanging... > > Yeah, the final part is here: http://bz.selenic.com/show_bug.cgi?id=3954#c4 But still I have question as why hg complained about @README in the first place. Also, I hope make distclean is not working "inside" .hg folder. -- Senthil -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Wed Jun 26 03:34:11 2013 From: steve at pearwood.info (Steven D'Aprano) Date: Wed, 26 Jun 2013 11:34:11 +1000 Subject: [Python-Dev] End of the mystery "@README.txt Mercurial bug" In-Reply-To: References: Message-ID: <51CA4513.5030001@pearwood.info> On 26/06/13 08:44, Victor Stinner wrote: > And then I ran "make distclean"... Victor, you're a cruel, cruel man. You've told us everything except the solution to the mystery. -- Steven From dreamingforward at gmail.com Wed Jun 26 03:44:25 2013 From: dreamingforward at gmail.com (Mark Janssen) Date: Tue, 25 Jun 2013 18:44:25 -0700 Subject: [Python-Dev] End of the mystery "@README.txt Mercurial bug" In-Reply-To: References: Message-ID: > It's like this. Whenever you use special characters in a file name, > you're asking for trouble. The shell and the OS have negotiate how to > interpret it. It bigger than git, and not a bug. Sorry, I meant mercurial, not git. -- MarkJ Tacoma, Washington From dreamingforward at gmail.com Wed Jun 26 03:43:27 2013 From: dreamingforward at gmail.com (Mark Janssen) Date: Tue, 25 Jun 2013 18:43:27 -0700 Subject: [Python-Dev] End of the mystery "@README.txt Mercurial bug" In-Reply-To: References: Message-ID: > One month ago, unit tests were added to IDLE (cool!) with a file > called @README.txt. The @ was used to see the name on top in a listing > of the directory. It's like this. Whenever you use special characters in a file name, you're asking for trouble. The shell and the OS have negotiate how to interpret it. It bigger than git, and not a bug. The issue is between the file system, the kernel, and the shell. > I also had the issue on 3 different computers, and so I reported the > issue upstream: > http://bz.selenic.com/show_bug.cgi?id=3954 Try it on different OS's on different machines (MacOS, Linux, Windows) . If my theory is right, it should be inconsistent across machines, but consistent within the same machine. -- MarkJ Tacoma, Washington From eric at trueblade.com Wed Jun 26 04:11:04 2013 From: eric at trueblade.com (Eric V. Smith) Date: Tue, 25 Jun 2013 22:11:04 -0400 Subject: [Python-Dev] End of the mystery "@README.txt Mercurial bug" In-Reply-To: References: Message-ID: <51CA4DB8.8060802@trueblade.com> On 6/25/2013 9:33 PM, Senthil Kumaran wrote: > > > > On Tue, Jun 25, 2013 at 5:58 PM, Benjamin Peterson > wrote: > > 2013/6/25 Victor Stinner >: > > And then I ran "make distclean"... > > You've left us hanging... > > > Yeah, the final part is here: http://bz.selenic.com/show_bug.cgi?id=3954#c4 > But still I have question as why hg complained about @README in the > first place. > Also, I hope make distclean is not working "inside" .hg folder. I think that's exactly what's happening. >From the bug report: find $(srcdir) '(' -name '*.fdc' -o -name '*~' \ -o -name '[@,#]*' -o -name '*.old' \ -o -name '*.orig' -o -name '*.rej' \ -o -name '*.bak' ')' \ -exec rm -f {} ';' Will find files beginning with '@' inside subdirectories of $(srcdir)/.hg. Just this week I saw someone use the logical equivalent of: find $(srcdir)/* ... to avoid this problem. It won't expand the .hg top-level directory. -- Eric. From phd at phdru.name Wed Jun 26 12:24:01 2013 From: phd at phdru.name (Oleg Broytman) Date: Wed, 26 Jun 2013 14:24:01 +0400 Subject: [Python-Dev] End of the mystery "@README.txt Mercurial bug" In-Reply-To: <51CA4DB8.8060802@trueblade.com> References: <51CA4DB8.8060802@trueblade.com> Message-ID: <20130626102401.GA11231@iskra.aviel.ru> On Tue, Jun 25, 2013 at 10:11:04PM -0400, "Eric V. Smith" wrote: > On 6/25/2013 9:33 PM, Senthil Kumaran wrote: > > > > > > > > On Tue, Jun 25, 2013 at 5:58 PM, Benjamin Peterson > > wrote: > > > > 2013/6/25 Victor Stinner > >: > > > And then I ran "make distclean"... > > > > You've left us hanging... > > > > > > Yeah, the final part is here: http://bz.selenic.com/show_bug.cgi?id=3954#c4 > > But still I have question as why hg complained about @README in the > > first place. > > Also, I hope make distclean is not working "inside" .hg folder. > > I think that's exactly what's happening. > > >From the bug report: > > find $(srcdir) '(' -name '*.fdc' -o -name '*~' \ > -o -name '[@,#]*' -o -name '*.old' \ > -o -name '*.orig' -o -name '*.rej' \ > -o -name '*.bak' ')' \ > -exec rm -f {} ';' > > Will find files beginning with '@' inside subdirectories of $(srcdir)/.hg. > > Just this week I saw someone use the logical equivalent of: > > find $(srcdir)/* ... > > to avoid this problem. It won't expand the .hg top-level directory. Or find \( -type d -name .hg -prune \) -o ... Oleg. -- Oleg Broytman http://phdru.name/ phd at phdru.name Programmers don't die, they just GOSUB without RETURN. From a.cavallo at cavallinux.eu Wed Jun 26 12:43:41 2013 From: a.cavallo at cavallinux.eu (a.cavallo at cavallinux.eu) Date: Wed, 26 Jun 2013 12:43:41 +0200 Subject: [Python-Dev] End of the mystery "@README.txt Mercurial bug" In-Reply-To: <20130626102401.GA11231@iskra.aviel.ru> References: <51CA4DB8.8060802@trueblade.com> <20130626102401.GA11231@iskra.aviel.ru> Message-ID: <3a58f5264f76fd48032c90ed9f3b1ff2@cavallinux.eu> .. or having hg "purging" unwanted build artifact (probably cleaning up the .hgignore file first) >> find $(srcdir)/* ... >> >> to avoid this problem. It won't expand the .hg top-level directory. > > Or find \( -type d -name .hg -prune \) -o ... > > Oleg. From eric at trueblade.com Wed Jun 26 14:18:27 2013 From: eric at trueblade.com (Eric V. Smith) Date: Wed, 26 Jun 2013 08:18:27 -0400 Subject: [Python-Dev] End of the mystery "@README.txt Mercurial bug" In-Reply-To: <3a58f5264f76fd48032c90ed9f3b1ff2@cavallinux.eu> References: <51CA4DB8.8060802@trueblade.com> <20130626102401.GA11231@iskra.aviel.ru> <3a58f5264f76fd48032c90ed9f3b1ff2@cavallinux.eu> Message-ID: <51CADC13.5060400@trueblade.com> On 6/26/2013 6:43 AM, a.cavallo at cavallinux.eu wrote: > .. or having hg "purging" unwanted build artifact (probably cleaning up > the .hgignore file first) How would that work? How could hg purge the .bak, .orig, .rej, .old, etc. files? >>> find $(srcdir)/* ... >>> >>> to avoid this problem. It won't expand the .hg top-level directory. >> >> Or find \( -type d -name .hg -prune \) -o ... I'm torn. Yours is more obvious, but we'd likely need to add .svn, .git, etc. Maybe find $(srcdir)/[a-zA-Z]* ... would be good enough to ignore all dot directories/files? -- Eric. From eric at trueblade.com Wed Jun 26 14:21:19 2013 From: eric at trueblade.com (Eric V. Smith) Date: Wed, 26 Jun 2013 08:21:19 -0400 Subject: [Python-Dev] End of the mystery "@README.txt Mercurial bug" In-Reply-To: <51CADC13.5060400@trueblade.com> References: <51CA4DB8.8060802@trueblade.com> <20130626102401.GA11231@iskra.aviel.ru> <3a58f5264f76fd48032c90ed9f3b1ff2@cavallinux.eu> <51CADC13.5060400@trueblade.com> Message-ID: <51CADCBF.1030503@trueblade.com> On 6/26/2013 8:18 AM, Eric V. Smith wrote: > On 6/26/2013 6:43 AM, a.cavallo at cavallinux.eu wrote: >> .. or having hg "purging" unwanted build artifact (probably cleaning up >> the .hgignore file first) > > How would that work? How could hg purge the .bak, .orig, .rej, .old, > etc. files? > >>>> find $(srcdir)/* ... >>>> >>>> to avoid this problem. It won't expand the .hg top-level directory. >>> >>> Or find \( -type d -name .hg -prune \) -o ... > > I'm torn. Yours is more obvious, but we'd likely need to add .svn, .git, > etc. Maybe find $(srcdir)/[a-zA-Z]* ... would be good enough to ignore > all dot directories/files? > Sorry for the mis-attribution. "Yours" refers to Oleg. -- Eric. From phd at phdru.name Wed Jun 26 14:44:05 2013 From: phd at phdru.name (Oleg Broytman) Date: Wed, 26 Jun 2013 16:44:05 +0400 Subject: [Python-Dev] End of the mystery "@README.txt Mercurial bug" In-Reply-To: <51CADC13.5060400@trueblade.com> References: <51CA4DB8.8060802@trueblade.com> <20130626102401.GA11231@iskra.aviel.ru> <3a58f5264f76fd48032c90ed9f3b1ff2@cavallinux.eu> <51CADC13.5060400@trueblade.com> Message-ID: <20130626124405.GA14843@iskra.aviel.ru> On Wed, Jun 26, 2013 at 08:18:27AM -0400, "Eric V. Smith" wrote: > >>> find $(srcdir)/* ... > >>> > >>> to avoid this problem. It won't expand the .hg top-level directory. > >> > >> Or find \( -type d -name .hg -prune \) -o ... > > I'm torn. Yours is more obvious, but we'd likely need to add .svn, .git, How many of those dot-files/directories are there beside .*ignore? > etc. Maybe find $(srcdir)/[a-zA-Z]* ... would be good enough to ignore > all dot directories/files? On the other hand yes, I think it'd be enough. Oleg. -- Oleg Broytman http://phdru.name/ phd at phdru.name Programmers don't die, they just GOSUB without RETURN. From ronaldoussoren at mac.com Wed Jun 26 14:57:28 2013 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Wed, 26 Jun 2013 14:57:28 +0200 Subject: [Python-Dev] End of the mystery "@README.txt Mercurial bug" In-Reply-To: <51CADC13.5060400@trueblade.com> References: <51CA4DB8.8060802@trueblade.com> <20130626102401.GA11231@iskra.aviel.ru> <3a58f5264f76fd48032c90ed9f3b1ff2@cavallinux.eu> <51CADC13.5060400@trueblade.com> Message-ID: On 26 Jun, 2013, at 14:18, Eric V. Smith wrote: > On 6/26/2013 6:43 AM, a.cavallo at cavallinux.eu wrote: >> .. or having hg "purging" unwanted build artifact (probably cleaning up >> the .hgignore file first) > > How would that work? How could hg purge the .bak, .orig, .rej, .old, > etc. files? > >>>> find $(srcdir)/* ... >>>> >>>> to avoid this problem. It won't expand the .hg top-level directory. >>> >>> Or find \( -type d -name .hg -prune \) -o ... > > I'm torn. Yours is more obvious, but we'd likely need to add .svn, .git, > etc. Maybe find $(srcdir)/[a-zA-Z]* ... would be good enough to ignore > all dot directories/files? Is the find command in the distclean target still needed? The comment for the distclean target says it is used to clean up the tree for distribution, but that's easier to accomplish by using a clean checkout. The target is still useful to get a clean tree when you're building with srcdir == builddir, but you don't need the find command for that. Ronald From eric at trueblade.com Wed Jun 26 15:02:46 2013 From: eric at trueblade.com (Eric V. Smith) Date: Wed, 26 Jun 2013 09:02:46 -0400 Subject: [Python-Dev] End of the mystery "@README.txt Mercurial bug" In-Reply-To: <20130626124405.GA14843@iskra.aviel.ru> References: <51CA4DB8.8060802@trueblade.com> <20130626102401.GA11231@iskra.aviel.ru> <3a58f5264f76fd48032c90ed9f3b1ff2@cavallinux.eu> <51CADC13.5060400@trueblade.com> <20130626124405.GA14843@iskra.aviel.ru> Message-ID: <51CAE676.1080501@trueblade.com> On 6/26/2013 8:44 AM, Oleg Broytman wrote: > On Wed, Jun 26, 2013 at 08:18:27AM -0400, "Eric V. Smith" wrote: >>>>> find $(srcdir)/* ... >>>>> >>>>> to avoid this problem. It won't expand the .hg top-level directory. >>>> >>>> Or find \( -type d -name .hg -prune \) -o ... >> >> I'm torn. Yours is more obvious, but we'd likely need to add .svn, .git, > > How many of those dot-files/directories are there beside .*ignore? That's all I can think of. And the files don't matter in this case, anyway. So probably just .svn, .hg, .git is all we'd need. >> etc. Maybe find $(srcdir)/[a-zA-Z]* ... would be good enough to ignore >> all dot directories/files? > > On the other hand yes, I think it'd be enough. I can go either way. -- Eric. From eric at trueblade.com Wed Jun 26 15:04:49 2013 From: eric at trueblade.com (Eric V. Smith) Date: Wed, 26 Jun 2013 09:04:49 -0400 Subject: [Python-Dev] End of the mystery "@README.txt Mercurial bug" In-Reply-To: References: <51CA4DB8.8060802@trueblade.com> <20130626102401.GA11231@iskra.aviel.ru> <3a58f5264f76fd48032c90ed9f3b1ff2@cavallinux.eu> <51CADC13.5060400@trueblade.com> Message-ID: <51CAE6F1.1040707@trueblade.com> On 6/26/2013 8:57 AM, Ronald Oussoren wrote: > > On 26 Jun, 2013, at 14:18, Eric V. Smith wrote: > >> On 6/26/2013 6:43 AM, a.cavallo at cavallinux.eu wrote: >>> .. or having hg "purging" unwanted build artifact (probably cleaning up >>> the .hgignore file first) >> >> How would that work? How could hg purge the .bak, .orig, .rej, .old, >> etc. files? >> >>>>> find $(srcdir)/* ... >>>>> >>>>> to avoid this problem. It won't expand the .hg top-level directory. >>>> >>>> Or find \( -type d -name .hg -prune \) -o ... >> >> I'm torn. Yours is more obvious, but we'd likely need to add .svn, .git, >> etc. Maybe find $(srcdir)/[a-zA-Z]* ... would be good enough to ignore >> all dot directories/files? > > Is the find command in the distclean target still needed? The comment for the distclean target says it is used to clean up the tree for distribution, but that's easier to accomplish by using a clean checkout. > > The target is still useful to get a clean tree when you're building with srcdir == builddir, but you don't need the find command for that. I run 'make distclean' fairly often, but maybe it's just out of habit. If I'm adding/deleting modules, I want to make sure there are no build artifacts. And since I have modified files, a clean checkout won't help (easily, at least). But me running distclean is not the same as answering your question about the find command being needed, I realize. -- Eric. From a.cavallo at cavallinux.eu Wed Jun 26 15:12:45 2013 From: a.cavallo at cavallinux.eu (a.cavallo at cavallinux.eu) Date: Wed, 26 Jun 2013 15:12:45 +0200 Subject: [Python-Dev] End of the mystery "@README.txt Mercurial bug" In-Reply-To: <51CADC13.5060400@trueblade.com> References: <51CA4DB8.8060802@trueblade.com> <20130626102401.GA11231@iskra.aviel.ru> <3a58f5264f76fd48032c90ed9f3b1ff2@cavallinux.eu> <51CADC13.5060400@trueblade.com> Message-ID: <5a0c3c0845fef23576cccd5e35287efb@cavallinux.eu> > How would that work? How could hg purge the .bak, .orig, .rej, .old, > etc. files? hg purge (it's an extension) removes anything that isn't tracked (and not ignored in the .hgignore): kind of distclean. I hope this helps From solipsis at pitrou.net Wed Jun 26 15:28:17 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 26 Jun 2013 15:28:17 +0200 Subject: [Python-Dev] End of the mystery "@README.txt Mercurial bug" References: <51CA4DB8.8060802@trueblade.com> <20130626102401.GA11231@iskra.aviel.ru> <3a58f5264f76fd48032c90ed9f3b1ff2@cavallinux.eu> <51CADC13.5060400@trueblade.com> <5a0c3c0845fef23576cccd5e35287efb@cavallinux.eu> Message-ID: <20130626152817.4a19541f@pitrou.net> Le Wed, 26 Jun 2013 15:12:45 +0200, a.cavallo at cavallinux.eu a ?crit : > > > How would that work? How could hg purge the .bak, .orig, .rej, .old, > > etc. files? > > hg purge (it's an extension) removes anything that isn't tracked (and > not ignored in the .hgignore): kind of distclean. distclean removes only what we want to remove, while hg purge will remove any untracked file, including any files that you may have wanted to keep (notes, work-in-progress patches, personal data files, etc.). Regards Antoine. From zachary.ware+pydev at gmail.com Wed Jun 26 15:28:51 2013 From: zachary.ware+pydev at gmail.com (Zachary Ware) Date: Wed, 26 Jun 2013 08:28:51 -0500 Subject: [Python-Dev] End of the mystery "@README.txt Mercurial bug" In-Reply-To: <5a0c3c0845fef23576cccd5e35287efb@cavallinux.eu> References: <51CA4DB8.8060802@trueblade.com> <20130626102401.GA11231@iskra.aviel.ru> <3a58f5264f76fd48032c90ed9f3b1ff2@cavallinux.eu> <51CADC13.5060400@trueblade.com> <5a0c3c0845fef23576cccd5e35287efb@cavallinux.eu> Message-ID: On Wed, Jun 26, 2013 at 8:12 AM, wrote: >Eric V. Smith wrote: >> How would that work? How could hg purge the .bak, .orig, .rej, .old, >> etc. files? > > > hg purge (it's an extension) removes anything that isn't tracked (and not > ignored in the .hgignore): kind of distclean. > > I hope this helps I've recently discovered purge and have started using it on Windows since there is no `make distclean`, and it is very effective. `hg purge -p` shows what will be removed (which should match anything with a ? in `hg status`), `hg purge` removes it, and `hg purge --all` clears out everything that's not tracked (including things listed in .hgignore) giving a fresh checkout without having to re-download. Very convenient, especially since it's a built-in extension. From barry at python.org Wed Jun 26 15:39:54 2013 From: barry at python.org (Barry Warsaw) Date: Wed, 26 Jun 2013 09:39:54 -0400 Subject: [Python-Dev] End of the mystery "@README.txt Mercurial bug" In-Reply-To: <51CAE6F1.1040707@trueblade.com> References: <51CA4DB8.8060802@trueblade.com> <20130626102401.GA11231@iskra.aviel.ru> <3a58f5264f76fd48032c90ed9f3b1ff2@cavallinux.eu> <51CADC13.5060400@trueblade.com> <51CAE6F1.1040707@trueblade.com> Message-ID: <20130626093954.51fc6741@anarchist> On Jun 26, 2013, at 09:04 AM, Eric V. Smith wrote: >I run 'make distclean' fairly often, but maybe it's just out of habit. >If I'm adding/deleting modules, I want to make sure there are no build >artifacts. And since I have modified files, a clean checkout won't help >(easily, at least). As do I. I think it still makes sense for us to include a working distclean, especially since it's a very common target for make-based builds. -Barry From ronaldoussoren at mac.com Wed Jun 26 16:05:53 2013 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Wed, 26 Jun 2013 16:05:53 +0200 Subject: [Python-Dev] End of the mystery "@README.txt Mercurial bug" In-Reply-To: <20130626093954.51fc6741@anarchist> References: <51CA4DB8.8060802@trueblade.com> <20130626102401.GA11231@iskra.aviel.ru> <3a58f5264f76fd48032c90ed9f3b1ff2@cavallinux.eu> <51CADC13.5060400@trueblade.com> <51CAE6F1.1040707@trueblade.com> <20130626093954.51fc6741@anarchist> Message-ID: <78896606-9093-4A3A-BE5C-F1AD77528C70@mac.com> On 26 Jun, 2013, at 15:39, Barry Warsaw wrote: > On Jun 26, 2013, at 09:04 AM, Eric V. Smith wrote: > >> I run 'make distclean' fairly often, but maybe it's just out of habit. >> If I'm adding/deleting modules, I want to make sure there are no build >> artifacts. And since I have modified files, a clean checkout won't help >> (easily, at least). > > As do I. I think it still makes sense for us to include a working distclean, > especially since it's a very common target for make-based builds. Sure, but is it necessary to run the find command for removing backup files in make distclean? When the find command is removed you'd still end up with a tree that's clean enough to perform a build from scratch, although the tree won't be perfectly clean. BTW. I usually build in a separate directory, that makes cleaning up even easier :-) Ronald > > -Barry > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/ronaldoussoren%40mac.com From rdmurray at bitdance.com Wed Jun 26 16:18:49 2013 From: rdmurray at bitdance.com (R. David Murray) Date: Wed, 26 Jun 2013 10:18:49 -0400 Subject: [Python-Dev] End of the mystery "@README.txt Mercurial bug" In-Reply-To: <20130626093954.51fc6741@anarchist> References: <51CA4DB8.8060802@trueblade.com> <20130626102401.GA11231@iskra.aviel.ru> <3a58f5264f76fd48032c90ed9f3b1ff2@cavallinux.eu> <51CADC13.5060400@trueblade.com> <51CAE6F1.1040707@trueblade.com> <20130626093954.51fc6741@anarchist> Message-ID: <20130626141849.987C4250B43@webabinitio.net> On Wed, 26 Jun 2013 09:39:54 -0400, Barry Warsaw wrote: > On Jun 26, 2013, at 09:04 AM, Eric V. Smith wrote: > > >I run 'make distclean' fairly often, but maybe it's just out of habit. > >If I'm adding/deleting modules, I want to make sure there are no build > >artifacts. And since I have modified files, a clean checkout won't help > >(easily, at least). > > As do I. I think it still makes sense for us to include a working distclean, > especially since it's a very common target for make-based builds. We also sometimes ask someone reporting an issue to do a make distclean and recompile, and many of these reporters will be working from a tarball rather than a checkout. Sure, they could re-unpack the tarball (if they haven't deleted it already), but make distclean is easier. --David From victor.stinner at gmail.com Wed Jun 26 16:24:03 2013 From: victor.stinner at gmail.com (Victor Stinner) Date: Wed, 26 Jun 2013 16:24:03 +0200 Subject: [Python-Dev] End of the mystery "@README.txt Mercurial bug" In-Reply-To: <51CA4DB8.8060802@trueblade.com> References: <51CA4DB8.8060802@trueblade.com> Message-ID: 2013/6/26 Eric V. Smith : > I think that's exactly what's happening. > > From the bug report: > > find $(srcdir) '(' -name '*.fdc' -o -name '*~' \ > -o -name '[@,#]*' -o -name '*.old' \ > -o -name '*.orig' -o -name '*.rej' \ > -o -name '*.bak' ')' \ > -exec rm -f {} ';' > > Will find files beginning with '@' inside subdirectories of $(srcdir)/.hg. In my opinion, make distclean should only remove files generated by configure and a build. It should not remove random files. *~, .orig, .rej, .back should be kept. They are not generated by configure nor make. What are these "@*", ",*" and "#*" files? Why does "make distclean" remove them? "make distclean" removes also the "tags" file which is generated by the ctags program, useful tool to browse the C source code (ex: in vim). Why does "make distclean" remove it? In short, the whole "find ... -exec rm -f {} ';'" command should be removed from "make distclean". (They are other commands to remove Makefile, "*.o" files, etc.) If someone really need such cleanup, another Makefile rule should be added. Victor From a.cavallo at cavallinux.eu Wed Jun 26 16:33:50 2013 From: a.cavallo at cavallinux.eu (a.cavallo at cavallinux.eu) Date: Wed, 26 Jun 2013 16:33:50 +0200 Subject: [Python-Dev] End of the mystery "@README.txt Mercurial bug" In-Reply-To: References: <51CA4DB8.8060802@trueblade.com> Message-ID: > *~, .orig, .rej, .back should be kept. They are not generated by > configure nor make. Ideally they should be left untracked not ignored. While devs can certainly add them to the .hgignore list to make life easier, a repository should be clean of extra files (or shown as untracked). I'd add that generated files (like the one generated from pgen) shouldn't be part of defaul make target but keept as fully tracked files (and regenerated on demand through a special make target). I hope this helps. From vitaly.murashev at gmail.com Wed Jun 26 16:31:27 2013 From: vitaly.murashev at gmail.com (Vitaly Murashev) Date: Wed, 26 Jun 2013 18:31:27 +0400 Subject: [Python-Dev] Relative path in co_filename for zipped modules In-Reply-To: <20130625201958.29E8B250B3F@webabinitio.net> References: <20130625201958.29E8B250B3F@webabinitio.net> Message-ID: Reported as http://bugs.python.org/issue18307 On Wed, Jun 26, 2013 at 12:19 AM, R. David Murray wrote: > Please file this as a bug report on bugs.python.org so that it doesn't > get lost. > > See also http://bugs.python.org/issue13328 (which looks like a different bug > but could also be causing you problems). > > --David > From rdmurray at bitdance.com Wed Jun 26 18:18:08 2013 From: rdmurray at bitdance.com (R. David Murray) Date: Wed, 26 Jun 2013 12:18:08 -0400 Subject: [Python-Dev] =?utf-8?q?=5BPython-checkins=5D_peps=3A_Reject_PEP_3?= =?utf-8?b?MTUu?= In-Reply-To: <3bgT1R1H5tz7LxL@mail.python.org> References: <3bgT1R1H5tz7LxL@mail.python.org> Message-ID: <20130626161809.788D7250B43@webabinitio.net> On Wed, 26 Jun 2013 17:39:07 +0200, lukasz.langa wrote: > - Deferred; see > + Rejected; see > + http://mail.python.org/pipermail/python-ideas/2013-June/021610.html > + > + This PEP has been deferred since 2006; see > http://mail.python.org/pipermail/python-dev/2006-February/060718.html > > Subsequent efforts to revive the PEP in April 2009 did not > meet with success because no syntax emerged that could > - compete with a while-True and an inner if-break. > + compete with the following form: > > - A syntax was found for a basic do-while loop but it found > - had little support because the condition was at the top: > + while True: > + > + if not : > + break > + > > - do ... while : > - > + Users of the language are advised to use that form when a do-while > + loop would have been appropriate. Why delete the information about what was found wanting? --David From lukasz at langa.pl Wed Jun 26 19:10:03 2013 From: lukasz at langa.pl (=?utf-8?Q?=C5=81ukasz_Langa?=) Date: Wed, 26 Jun 2013 19:10:03 +0200 Subject: [Python-Dev] [Python-checkins] peps: Reject PEP 315. In-Reply-To: <20130626161809.788D7250B43@webabinitio.net> References: <3bgT1R1H5tz7LxL@mail.python.org> <20130626161809.788D7250B43@webabinitio.net> Message-ID: On 26 cze 2013, at 18:18, R. David Murray wrote: > On Wed, 26 Jun 2013 17:39:07 +0200, lukasz.langa wrote: >> - Deferred; see >> + Rejected; see >> + http://mail.python.org/pipermail/python-ideas/2013-June/021610.html >> + >> + This PEP has been deferred since 2006; see >> http://mail.python.org/pipermail/python-dev/2006-February/060718.html >> >> Subsequent efforts to revive the PEP in April 2009 did not >> meet with success because no syntax emerged that could >> - compete with a while-True and an inner if-break. >> + compete with the following form: >> >> - A syntax was found for a basic do-while loop but it found >> - had little support because the condition was at the top: >> + while True: >> + >> + if not : >> + break >> + >> >> - do ... while : >> - >> + Users of the language are advised to use that form when a do-while >> + loop would have been appropriate. > > Why delete the information about what was found wanting? This was a mention of an alternative possible syntax. I don't feel it was worth keeping this particular one while not discussing any other alternatives at all. If you'd rather keep this information, it would be better off to have it to an "Alternative approaches" section. -- Best regards, ?ukasz Langa WWW: http://lukasz.langa.pl/ Twitter: @llanga IRC: ambv on #python-dev -------------- next part -------------- An HTML attachment was scrubbed... URL: From turnbull at sk.tsukuba.ac.jp Wed Jun 26 19:41:22 2013 From: turnbull at sk.tsukuba.ac.jp (Stephen J. Turnbull) Date: Thu, 27 Jun 2013 02:41:22 +0900 Subject: [Python-Dev] End of the mystery "@README.txt Mercurial bug" In-Reply-To: References: <51CA4DB8.8060802@trueblade.com> Message-ID: <87a9mczs59.fsf@uwakimon.sk.tsukuba.ac.jp> Victor Stinner writes: > In my opinion, make distclean should only remove files generated by > configure and a build. It should not remove random files. FWIW, the GNU standard for these targets is something like: ## make clean or make mostlyclean ## Delete all files from the current directory that are normally ## created by building the program. Don't delete the files that ## record the configuration. Also preserve files that could be ## made by building, but normally aren't because the distribution ## comes with them. ## Delete `.dvi' files here if they are not part of the ## distribution. ## make distclean ## Delete all files from the current directory that are created by ## configuring or building the program. If you have unpacked the ## source and built the program without creating any other files, ## `make distclean' should leave only the files that were in the ## distribution. ## make realclean ## Delete everything from the current directory that can be ## reconstructed with this Makefile. This typically includes ## everything deleted by distclean, plus more: C source files ## produced by Bison, tags tables, info files, and so on. ## make extraclean ## Still more severe - delete backup and autosave files, too. This is from the XEmacs Makefile.in.in, so it's not authoritative. Still, it seems pretty intuitive and presumably is in wide use, not to forget matching Victor's preferred usage for 'distclean'. From victor.stinner at gmail.com Thu Jun 27 00:07:11 2013 From: victor.stinner at gmail.com (Victor Stinner) Date: Thu, 27 Jun 2013 00:07:11 +0200 Subject: [Python-Dev] PyArg_ParseTupe(): parse unsigned integer and check for overflow Message-ID: Hi, I don't understand why the B, H, I, k formats of PyArg_ParseTupe() do not check for overflow, whereas formats for signed numbers do check for overflow. What is the useful of ignoring overflow? http://docs.python.org/3/c-api/arg.html I would to parse an integer in [0; UINT_MAX] to fix the zlib module on 64-bit system: http://bugs.python.org/issue18294 How should I implement that? Use "O" format and then use PyLong_Check(), PyLong_AsLong(), and check value <= UINT_MAX? Victor From guido at python.org Thu Jun 27 00:26:05 2013 From: guido at python.org (Guido van Rossum) Date: Wed, 26 Jun 2013 15:26:05 -0700 Subject: [Python-Dev] PyArg_ParseTupe(): parse unsigned integer and check for overflow In-Reply-To: References: Message-ID: On Wed, Jun 26, 2013 at 3:07 PM, Victor Stinner wrote: > I don't understand why the B, H, I, k formats of PyArg_ParseTupe() do > not check for overflow, whereas formats for signed numbers do check > for overflow. What is the useful of ignoring overflow? > http://docs.python.org/3/c-api/arg.html I think this is really old DNA. It comes from times when 0xffffffff and -1 were equivalent -- we borrowed this cavalier attitude from old C code. I worry that "fixing" this would break some old libraries relying on the feature, so I hope we can avoid changes in this area (the older the DNA, the more hidden dependencies it has ;-). > I would to parse an integer in [0; UINT_MAX] to fix the zlib module on > 64-bit system: > http://bugs.python.org/issue18294 > > How should I implement that? Use "O" format and then use > PyLong_Check(), PyLong_AsLong(), and check value <= UINT_MAX? Why can't you use the K format? It won't reject out-of-range values, but it will convert them to in-range so there aren't any attacks possible based on bypassing the range check. I'm probably misunderstanding something -- I don't completely understand that bug report. :-( -- --Guido van Rossum (python.org/~guido) From ncoghlan at gmail.com Thu Jun 27 00:51:53 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 27 Jun 2013 08:51:53 +1000 Subject: [Python-Dev] [Python-checkins] peps: Reject PEP 315. In-Reply-To: References: <3bgT1R1H5tz7LxL@mail.python.org> <20130626161809.788D7250B43@webabinitio.net> Message-ID: On 27 Jun 2013 03:12, "?ukasz Langa" wrote: > > On 26 cze 2013, at 18:18, R. David Murray wrote: > >> On Wed, 26 Jun 2013 17:39:07 +0200, lukasz.langa < python-checkins at python.org> wrote: >>> >>> - Deferred; see >>> + Rejected; see >>> + http://mail.python.org/pipermail/python-ideas/2013-June/021610.html >>> + >>> + This PEP has been deferred since 2006; see >>> http://mail.python.org/pipermail/python-dev/2006-February/060718.html >>> >>> Subsequent efforts to revive the PEP in April 2009 did not >>> meet with success because no syntax emerged that could >>> - compete with a while-True and an inner if-break. >>> + compete with the following form: >>> >>> - A syntax was found for a basic do-while loop but it found >>> - had little support because the condition was at the top: >>> + while True: >>> + >>> + if not : >>> + break >>> + >>> >>> - do ... while : >>> - >>> + Users of the language are advised to use that form when a do-while >>> + loop would have been appropriate. >> >> >> Why delete the information about what was found wanting? > > > This was a mention of an alternative possible syntax. I don't feel it was worth keeping this particular one while not discussing any other alternatives at all. If you'd rather keep this information, it would be better off to have it to an "Alternative approaches" section. It was relevant because it was the most acceptable alternative found, and we *still* didn't think it was an improvement over the status quo. We don't generally significantly edit PEPs when accepting or rejecting them. Cheers, Nick. > > -- > Best regards, > ?ukasz Langa > > WWW: http://lukasz.langa.pl/ > Twitter: @llanga > IRC: ambv on #python-dev > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com > -------------- next part -------------- An HTML attachment was scrubbed... URL: From eric at trueblade.com Thu Jun 27 03:27:55 2013 From: eric at trueblade.com (Eric V. Smith) Date: Wed, 26 Jun 2013 21:27:55 -0400 Subject: [Python-Dev] End of the mystery "@README.txt Mercurial bug" In-Reply-To: <51CAE676.1080501@trueblade.com> References: <51CA4DB8.8060802@trueblade.com> <20130626102401.GA11231@iskra.aviel.ru> <3a58f5264f76fd48032c90ed9f3b1ff2@cavallinux.eu> <51CADC13.5060400@trueblade.com> <20130626124405.GA14843@iskra.aviel.ru> <51CAE676.1080501@trueblade.com> Message-ID: <51CB951B.1000501@trueblade.com> On 6/26/2013 9:02 AM, Eric V. Smith wrote: > On 6/26/2013 8:44 AM, Oleg Broytman wrote: >> On Wed, Jun 26, 2013 at 08:18:27AM -0400, "Eric V. Smith" wrote: >>>>>> find $(srcdir)/* ... >>>>>> >>>>>> to avoid this problem. It won't expand the .hg top-level directory. >>>>> >>>>> Or find \( -type d -name .hg -prune \) -o ... >>> >>> I'm torn. Yours is more obvious, but we'd likely need to add .svn, .git, >> >> How many of those dot-files/directories are there beside .*ignore? > > That's all I can think of. And the files don't matter in this case, > anyway. So probably just .svn, .hg, .git is all we'd need. > >>> etc. Maybe find $(srcdir)/[a-zA-Z]* ... would be good enough to ignore >>> all dot directories/files? >> >> On the other hand yes, I think it'd be enough. > > I can go either way. > I created http://bugs.python.org/issue18312 to track this. -- Eric. From guido at python.org Thu Jun 27 03:56:10 2013 From: guido at python.org (Guido van Rossum) Date: Wed, 26 Jun 2013 18:56:10 -0700 Subject: [Python-Dev] Reminder: an oft-forgotten rule about docstring formatting (PEP 257) Message-ID: PEP 257 says this on the formatting of multi-line docstrings: """ Multi-line docstrings consist of a summary line just like a one-line docstring, followed by a blank line, followed by a more elaborate description. The summary line may be used by automatic indexing tools; it is important that it fits on one line and is separated from the rest of the docstring by a blank line. [...] """ I still like this rule, but it is violated frequently, in the stdlib and elsewhere. I'd like to urge stdlib contributors and core devs to heed it -- or explain why you can't. -- --Guido van Rossum (python.org/~guido) From skip at pobox.com Thu Jun 27 04:08:27 2013 From: skip at pobox.com (Skip Montanaro) Date: Wed, 26 Jun 2013 21:08:27 -0500 Subject: [Python-Dev] Reminder: an oft-forgotten rule about docstring formatting (PEP 257) In-Reply-To: References: Message-ID: > I'd like to urge stdlib contributors and core devs to > heed it -- or explain why you can't. Where I can, I do, however I often find it difficult to come up with a one-liner, especially one that mentions the parameters to functions. If the one-line rule is going to be violated, I go whole hog and don't bother with the blank line either. (If I'm going to punch a hole in a rule, I might as well create one big enough to comfortably walk through.) Skip From ethan at stoneleaf.us Thu Jun 27 04:14:01 2013 From: ethan at stoneleaf.us (Ethan Furman) Date: Wed, 26 Jun 2013 19:14:01 -0700 Subject: [Python-Dev] Reminder: an oft-forgotten rule about docstring formatting (PEP 257) In-Reply-To: References: Message-ID: <51CB9FE9.5050300@stoneleaf.us> On 06/26/2013 07:08 PM, Skip Montanaro wrote: >> I'd like to urge stdlib contributors and core devs to >> heed it -- or explain why you can't. > > Where I can, I do, however I often find it difficult to come up with a > one-liner, especially one that mentions the parameters to functions. > If the one-line rule is going to be violated, I go whole hog and don't > bother with the blank line either. (If I'm going to punch a hole in a > rule, I might as well create one big enough to comfortably walk > through.) Why do the parameters have to be in the summary? You could do them better justice in the following paragraphs. -- ~Ethan~ From ben+python at benfinney.id.au Thu Jun 27 07:29:00 2013 From: ben+python at benfinney.id.au (Ben Finney) Date: Thu, 27 Jun 2013 15:29:00 +1000 Subject: [Python-Dev] Reminder: an oft-forgotten rule about docstring formatting (PEP 257) References: Message-ID: <7w1u7oktpf.fsf@benfinney.id.au> Skip Montanaro writes: > Where I can [conform to PEP 257], I do, however I often find it > difficult to come up with a one-liner, especially one that mentions > the parameters to functions. I think it's far better to make the synopsis a summary of the purpose of the function; no need to force the parameters in there if they don't fit naturally in the sentence fragment. A list of the parameters and return value can go in the detailed description. -- \ ?Theology is the effort to explain the unknowable in terms of | `\ the not worth knowing.? ?Henry L. Mencken | _o__) | Ben Finney From ericsnowcurrently at gmail.com Thu Jun 27 08:03:05 2013 From: ericsnowcurrently at gmail.com (Eric Snow) Date: Thu, 27 Jun 2013 00:03:05 -0600 Subject: [Python-Dev] Classes with ordered namespaces Message-ID: There are two relevant class namespaces in this proposal: definition namespace and __dict__. Currently both are dicts. For class definition namespaces, I'd like to make the default an OrderedDict. With the implementation in issue16991, the change is trivial (basically 1-liners in 2 spots). This change would make it unnecessary to write a custom metaclass just for a __prepare__(). PEP 422 alleviates that problem somewhat. However, I expect OrderedDict is by far the most common type returned by __prepare__(), so having it be the default would proportionately reduce the need for people to write metaclasses or learn about the PEP 422 API. You may ask what is the point if they aren't using a metaclass. That leads to the other half of the proposal. Once I have a class, I'd like to know the definition order without needing a metaclass. Unfortunately it's not as simple as using the C OrderedDict (issue16991 again) for tp_dict, etc. That change is actually pretty trivial. However, it causes problems because the concrete C API (PyDict_*) doesn't play nice with subclasses and the concrete API is used on class __dict__ all over the place. The alternative I propose is to add __definition_order__ or similar to classes. It would be a list of the names from the definition namespace, in definition order (making use of the new default there). Having a class's __dict__ be ordered isn't important if the definition order is available somewhere. Again, using the C OrderedDict, the change to add __definition_order__ is pretty trivial. FWIW, Guido already expressed some approval[1]. -eric [1] http://mail.python.org/pipermail/python-ideas/2013-February/019704.html From solipsis at pitrou.net Thu Jun 27 10:21:43 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 27 Jun 2013 10:21:43 +0200 Subject: [Python-Dev] Reminder: an oft-forgotten rule about docstring formatting (PEP 257) References: Message-ID: <20130627102143.65125bc3@pitrou.net> Le Wed, 26 Jun 2013 18:56:10 -0700, Guido van Rossum a ?crit : > PEP 257 says this on the formatting of multi-line docstrings: > > """ > Multi-line docstrings consist of a summary line just like a one-line > docstring, followed by a blank line, followed by a more elaborate > description. The summary line may be used by automatic indexing tools; > it is important that it fits on one line and is separated from the > rest of the docstring by a blank line. [...] > """ > > I still like this rule, but it is violated frequently, in the stdlib > and elsewhere. I'd like to urge stdlib contributors and core devs to > heed it -- or explain why you can't. I don't always find it easy to summarize a function in one line. Regards Antoine. From ncoghlan at gmail.com Thu Jun 27 10:48:47 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 27 Jun 2013 18:48:47 +1000 Subject: [Python-Dev] Classes with ordered namespaces In-Reply-To: References: Message-ID: On 27 June 2013 16:03, Eric Snow wrote: > There are two relevant class namespaces in this proposal: definition > namespace and __dict__. Currently both are dicts. > > For class definition namespaces, I'd like to make the default an > OrderedDict. With the implementation in issue16991, the change is > trivial (basically 1-liners in 2 spots). This change would make it > unnecessary to write a custom metaclass just for a __prepare__(). PEP > 422 alleviates that problem somewhat. However, I expect OrderedDict > is by far the most common type returned by __prepare__(), so having it > be the default would proportionately reduce the need for people to > write metaclasses or learn about the PEP 422 API. You may ask what is > the point if they aren't using a metaclass. I'd be tempted to kill PEP 422 as not worth the hassle if we did this. Yes, I count that as a point in favour of the idea :) > That leads to the other > half of the proposal. > > Once I have a class, I'd like to know the definition order without > needing a metaclass. Unfortunately it's not as simple as using the C > OrderedDict (issue16991 again) for tp_dict, etc. That change is > actually pretty trivial. However, it causes problems because the > concrete C API (PyDict_*) doesn't play nice with subclasses and the > concrete API is used on class __dict__ all over the place. The > alternative I propose is to add __definition_order__ or similar to > classes. It would be a list of the names from the definition > namespace, in definition order (making use of the new default there). > Having a class's __dict__ be ordered isn't important if the definition > order is available somewhere. Again, using the C OrderedDict, the > change to add __definition_order__ is pretty trivial. I think the main concern I would have is whether other implementations are happy they can provide a suitable ordered dictionary for class namespace execution. It's also worth considering what would have to happen for dynamically created types where the namespace passed in *wasn't* ordered. It's a plausible way forward, though. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From lukasz at langa.pl Thu Jun 27 10:49:41 2013 From: lukasz at langa.pl (=?utf-8?Q?=C5=81ukasz_Langa?=) Date: Thu, 27 Jun 2013 10:49:41 +0200 Subject: [Python-Dev] [Python-checkins] peps: Reject PEP 315. In-Reply-To: References: <3bgT1R1H5tz7LxL@mail.python.org> <20130626161809.788D7250B43@webabinitio.net> Message-ID: <929D0643-3516-4133-AE53-C73E2C4E5C29@langa.pl> On 27 cze 2013, at 00:51, Nick Coghlan wrote: > On 27 Jun 2013 03:12, "?ukasz Langa" wrote: > > This was a mention of an alternative possible syntax. I don't feel it was worth keeping this particular one while not discussing any other alternatives at all. If you'd rather keep this information, it would be better off to have it to an "Alternative approaches" section. > > It was relevant because it was the most acceptable alternative found, and we *still* didn't think it was an improvement over the status quo. > > We don't generally significantly edit PEPs when accepting or rejecting them. > Comment restored, point taken. Thanks for clearing this up. -- Best regards, ?ukasz Langa WWW: http://lukasz.langa.pl/ Twitter: @llanga IRC: ambv on #python-dev -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Thu Jun 27 16:20:55 2013 From: guido at python.org (Guido van Rossum) Date: Thu, 27 Jun 2013 07:20:55 -0700 Subject: [Python-Dev] Reminder: an oft-forgotten rule about docstring formatting (PEP 257) In-Reply-To: <20130627102143.65125bc3@pitrou.net> References: <20130627102143.65125bc3@pitrou.net> Message-ID: On Thu, Jun 27, 2013 at 1:21 AM, Antoine Pitrou wrote: > I don't always find it easy to summarize a function in one line. Neither do I. But I always manage to do it anyway. My reasoning is, you can always leave out more detail and add it to the subsequent paragraph. -- --Guido van Rossum (python.org/~guido) From larry at hastings.org Thu Jun 27 17:09:13 2013 From: larry at hastings.org (Larry Hastings) Date: Thu, 27 Jun 2013 10:09:13 -0500 Subject: [Python-Dev] Reminder: an oft-forgotten rule about docstring formatting (PEP 257) In-Reply-To: References: Message-ID: <51CC5599.9010305@hastings.org> On 06/26/2013 08:56 PM, Guido van Rossum wrote: > PEP 257 says this on the formatting of multi-line docstrings: > > """ > Multi-line docstrings consist of a summary line just like a one-line > docstring, followed by a blank line, followed by a more elaborate > description. The summary line may be used by automatic indexing tools; > it is important that it fits on one line and is separated from the > rest of the docstring by a blank line. [...] > """ > > I still like this rule, but it is violated frequently, in the stdlib > and elsewhere. I'd like to urge stdlib contributors and core devs to > heed it -- or explain why you can't. Argument Clinic could conceivably enforce this. It could mandate that the first paragraph of the function docstring contain exactly one sentence (must end in a period, all embedded periods cannot be followed by whitespace). This would make some things nicer; I could automatically insert the per-parameter docstrings in after the summary. Should it? //arry/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From ericsnowcurrently at gmail.com Thu Jun 27 17:18:30 2013 From: ericsnowcurrently at gmail.com (Eric Snow) Date: Thu, 27 Jun 2013 09:18:30 -0600 Subject: [Python-Dev] Classes with ordered namespaces In-Reply-To: References: Message-ID: On Thu, Jun 27, 2013 at 2:48 AM, Nick Coghlan wrote: > I think the main concern I would have is whether other implementations > are happy they can provide a suitable ordered dictionary for class > namespace execution. > > It's also worth considering what would have to happen for dynamically > created types where the namespace passed in *wasn't* ordered. Good points. In either case there is no definition order available. I'm okay with that. Would it be better to represent that as None (and the attribute is always defined) or by having the attribute undefined? I'd rather always have the attribute, but I expect the significantly simpler solution is to leave the attribute undefined when definition order is not available. So I'd go with the latter. -eric From guido at python.org Thu Jun 27 17:57:32 2013 From: guido at python.org (Guido van Rossum) Date: Thu, 27 Jun 2013 08:57:32 -0700 Subject: [Python-Dev] Reminder: an oft-forgotten rule about docstring formatting (PEP 257) In-Reply-To: <51CC5599.9010305@hastings.org> References: <51CC5599.9010305@hastings.org> Message-ID: Yes on one line, capitalized, period. No on single sentence. --Guido van Rossum (sent from Android phone) On Jun 27, 2013 8:17 AM, "Larry Hastings" wrote: > On 06/26/2013 08:56 PM, Guido van Rossum wrote: > > PEP 257 says this on the formatting of multi-line docstrings: > > """ > Multi-line docstrings consist of a summary line just like a one-line > docstring, followed by a blank line, followed by a more elaborate > description. The summary line may be used by automatic indexing tools; > it is important that it fits on one line and is separated from the > rest of the docstring by a blank line. [...] > """ > > I still like this rule, but it is violated frequently, in the stdlib > and elsewhere. I'd like to urge stdlib contributors and core devs to > heed it -- or explain why you can't. > > > Argument Clinic could conceivably enforce this. It could mandate that the > first paragraph of the function docstring contain exactly one sentence > (must end in a period, all embedded periods cannot be followed by > whitespace). This would make some things nicer; I could automatically > insert the per-parameter docstrings in after the summary. > > Should it? > > > */arry* > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/guido%40python.org > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ethan at stoneleaf.us Thu Jun 27 17:35:38 2013 From: ethan at stoneleaf.us (Ethan Furman) Date: Thu, 27 Jun 2013 08:35:38 -0700 Subject: [Python-Dev] Classes with ordered namespaces In-Reply-To: References: Message-ID: <51CC5BCA.7050804@stoneleaf.us> On 06/27/2013 08:18 AM, Eric Snow wrote: > On Thu, Jun 27, 2013 at 2:48 AM, Nick Coghlan wrote: >> I think the main concern I would have is whether other implementations >> are happy they can provide a suitable ordered dictionary for class >> namespace execution. >> >> It's also worth considering what would have to happen for dynamically >> created types where the namespace passed in *wasn't* ordered. > > Good points. In either case there is no definition order available. > I'm okay with that. Would it be better to represent that as None (and > the attribute is always defined) or by having the attribute undefined? > I'd rather always have the attribute, but I expect the significantly > simpler solution is to leave the attribute undefined when definition > order is not available. So I'd go with the latter. So in Python space the options are either: if some_class.__definition_order__ is not None: or if getattr(some_class, '__definition_order__', None) is not None: ? Seems like always defined would be easier. -- ~Ethan~ From tjreedy at udel.edu Thu Jun 27 18:40:14 2013 From: tjreedy at udel.edu (Terry Reedy) Date: Thu, 27 Jun 2013 12:40:14 -0400 Subject: [Python-Dev] Reminder: an oft-forgotten rule about docstring formatting (PEP 257) In-Reply-To: References: <51CC5599.9010305@hastings.org> Message-ID: On 6/27/2013 11:57 AM, Guido van Rossum wrote: > Yes on one line, capitalized, period. No on single sentence. Complete and correct docstrings are somewhat rare in idlelib. About half are missing. Single lines typically omit the period. Multiple lines often omit the blank line after the first. I will take your reminder as permission to make changes as I review the code. I wish there were more good summary lines already. My understanding is that SubclassTests and test_methods should not have docstrings. -- Terry Jan Reedy From guido at python.org Thu Jun 27 18:57:30 2013 From: guido at python.org (Guido van Rossum) Date: Thu, 27 Jun 2013 09:57:30 -0700 Subject: [Python-Dev] Reminder: an oft-forgotten rule about docstring formatting (PEP 257) In-Reply-To: References: <51CC5599.9010305@hastings.org> Message-ID: It was never my intention to enforce that everything has a docstring. Just that if it does, it looks good. On Thu, Jun 27, 2013 at 9:40 AM, Terry Reedy wrote: > On 6/27/2013 11:57 AM, Guido van Rossum wrote: >> >> Yes on one line, capitalized, period. No on single sentence. > > > Complete and correct docstrings are somewhat rare in idlelib. > About half are missing. > Single lines typically omit the period. > Multiple lines often omit the blank line after the first. > > I will take your reminder as permission to make changes as I review the > code. I wish there were more good summary lines already. > > My understanding is that SubclassTests and test_methods should not have > docstrings. > > -- > Terry Jan Reedy > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/guido%40python.org -- --Guido van Rossum (python.org/~guido) From tjreedy at udel.edu Thu Jun 27 20:25:21 2013 From: tjreedy at udel.edu (Terry Reedy) Date: Thu, 27 Jun 2013 14:25:21 -0400 Subject: [Python-Dev] Reminder: an oft-forgotten rule about docstring formatting (PEP 257) In-Reply-To: References: <51CC5599.9010305@hastings.org> Message-ID: On 6/27/2013 12:57 PM, Guido van Rossum wrote: > It was never my intention to enforce that everything has a docstring. > Just that if it does, it looks good. Ok, I won't add them when a function's name actually makes what it does obvious. But when I have to spend at least a few minutes reading the body to make sense of it, I will try to add the summary line that I wish had been there already. -- Terry Jan Reedy From pje at telecommunity.com Thu Jun 27 23:45:36 2013 From: pje at telecommunity.com (PJ Eby) Date: Thu, 27 Jun 2013 17:45:36 -0400 Subject: [Python-Dev] Classes with ordered namespaces In-Reply-To: References: Message-ID: On Thu, Jun 27, 2013 at 4:48 AM, Nick Coghlan wrote: > I'd be tempted to kill PEP 422 as not worth the hassle if we did this. I assume you mean the "namespace" keyword part of PEP 422, since PEP 422's class initialization feature is entirely orthogonal to definition order or namespace customization. (Indeed, I cannot recall a single instance of class initialization in my code that actually *cared* about definition order. Certainly I haven't had any situations where a pre-existing definition order would've eliminated the need for a class-level initialization.) From ericsnowcurrently at gmail.com Thu Jun 27 23:48:25 2013 From: ericsnowcurrently at gmail.com (Eric Snow) Date: Thu, 27 Jun 2013 15:48:25 -0600 Subject: [Python-Dev] Classes with ordered namespaces In-Reply-To: <51CC5BCA.7050804@stoneleaf.us> References: <51CC5BCA.7050804@stoneleaf.us> Message-ID: On Thu, Jun 27, 2013 at 9:35 AM, Ethan Furman wrote: > On 06/27/2013 08:18 AM, Eric Snow wrote: >> Good points. In either case there is no definition order available. >> I'm okay with that. Would it be better to represent that as None (and >> the attribute is always defined) or by having the attribute undefined? >> I'd rather always have the attribute, but I expect the significantly >> simpler solution is to leave the attribute undefined when definition >> order is not available. So I'd go with the latter. > > > So in Python space the options are either: > > if some_class.__definition_order__ is not None: > > or > > if getattr(some_class, '__definition_order__', None) is not None: > > ? > > Seems like always defined would be easier. Certainly it's easier to use if you need the definition order. However, I expect it wouldn't be used often enough that that would make a big difference. Furthermore, if it's always there then it will always show up when you dir(SomeClass) or look at SomeClass.__dict__. People would see that on every class, whether it actually provided definition order or not. I'm not sure that is as helpful as only having the attribute around when order is defined. This got me thinking. The attribute should be a class-only attribute (i.e. a property on type) like __name__ is. That would keep it from showing up on instance lookups and on every vars(obj). The big thing for me, though, is that optionally having the attribute accommodates Python implementations that can't (or don't) preserve definition order. They don't have to worry about adding a dummy attribute to every class just for the sake of complying with the spec. It also means the implementation for dynamic types like Nick mentioned don't necesarily need to be touched. The more I think about it the more the optionally set attribute approach is the way to go. -eric From alexander.belopolsky at gmail.com Thu Jun 27 23:51:44 2013 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Thu, 27 Jun 2013 17:51:44 -0400 Subject: [Python-Dev] Reminder: an oft-forgotten rule about docstring formatting (PEP 257) In-Reply-To: References: <20130627102143.65125bc3@pitrou.net> Message-ID: On Thu, Jun 27, 2013 at 10:20 AM, Guido van Rossum wrote: > On Thu, Jun 27, 2013 at 1:21 AM, Antoine Pitrou > wrote: > > I don't always find it easy to summarize a function in one line. > > Neither do I. But I always manage to do it anyway. +1 If you cannot summarize what your function does in one line, chances are it is time to split your function, not the summary line. Ideally, the name of the function should already give a good idea of what it does. A summary line can just rephrase the same in a complete sentence. I took a quick look at the stdlib and in many cases a summary line is already there. It is just the issue of formatting. 250 class _ErrorHolder(object): 251 """ 252 Placeholder for a TestCase inside a result. As far as a TestResult 253 is concerned, this looks exactly like a unit test. Used to insert 254 arbitrary errors into a test suite run. 255 """ *http://hg.python.org/cpython/file/44f455e6163d/Lib/unittest/suite.py#l250* 81 class TestProgram(object): 82 """A command-line program that runs a set of tests; this is primarily 83 for making test modules conveniently executable. 84 """ *http://hg.python.org/cpython/file/44f455e6163d/Lib/unittest/main.py#l81* 109 @failfast 110 def addError(self, test, err): 111 """Called when an error has occurred. 'err' is a tuple of values as 112 returned by sys.exc_info(). 113 """ 114 self.errors.append((test, self._exc_info_to_string(err, test))) 115 self._mirrorOutput = True 116 117 @failfast 118 def addFailure(self, test, err): 119 """Called when an error has occurred. 'err' is a tuple of values as 120 returned by sys.exc_info().""" 121 self.failures.append((test, self._exc_info_to_string(err, test))) 122 self._mirrorOutput = True 123 124 @failfast 125 def addSubTest(self, test, subtest, err): 126 """Called at the end of a subtest. 127 'err' is None if the subtest ended successfully, otherwise it's a 128 tuple of values as returned by sys.exc_info(). 129 """ *http://hg.python.org/cpython/file/44f455e6163d/Lib/unittest/result.py#l109* * * 589 def do_break(self, arg, temporary = 0): 590 """b(reak) [ ([filename:]lineno | function) [, condition] ] 591 Without argument, list all breaks. *http://hg.python.org/cpython/file/44f455e6163d/Lib/pdb.py#l589* -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Fri Jun 28 00:02:59 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 28 Jun 2013 08:02:59 +1000 Subject: [Python-Dev] Classes with ordered namespaces In-Reply-To: References: Message-ID: On 28 Jun 2013 07:46, "PJ Eby" wrote: > > On Thu, Jun 27, 2013 at 4:48 AM, Nick Coghlan wrote: > > I'd be tempted to kill PEP 422 as not worth the hassle if we did this. > > I assume you mean the "namespace" keyword part of PEP 422, since PEP > 422's class initialization feature is entirely orthogonal to > definition order or namespace customization. (Indeed, I cannot recall > a single instance of class initialization in my code that actually > *cared* about definition order. Certainly I haven't had any > situations where a pre-existing definition order would've eliminated > the need for a class-level initialization.) That and the second argument to the new initialisation hook. So you're right, the consequences for PEP 422 would be "greatly simplify" rather than "kill". Cheers, Nick. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Fri Jun 28 00:19:59 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 28 Jun 2013 08:19:59 +1000 Subject: [Python-Dev] Classes with ordered namespaces In-Reply-To: References: <51CC5BCA.7050804@stoneleaf.us> Message-ID: On 28 Jun 2013 07:51, "Eric Snow" wrote: > > On Thu, Jun 27, 2013 at 9:35 AM, Ethan Furman wrote: > > On 06/27/2013 08:18 AM, Eric Snow wrote: > >> Good points. In either case there is no definition order available. > >> I'm okay with that. Would it be better to represent that as None (and > >> the attribute is always defined) or by having the attribute undefined? > >> I'd rather always have the attribute, but I expect the significantly > >> simpler solution is to leave the attribute undefined when definition > >> order is not available. So I'd go with the latter. > > > > > > So in Python space the options are either: > > > > if some_class.__definition_order__ is not None: > > > > or > > > > if getattr(some_class, '__definition_order__', None) is not None: > > > > ? > > > > Seems like always defined would be easier. > > Certainly it's easier to use if you need the definition order. > However, I expect it wouldn't be used often enough that that would > make a big difference. Furthermore, if it's always there then it will > always show up when you dir(SomeClass) or look at SomeClass.__dict__. > People would see that on every class, whether it actually provided > definition order or not. I'm not sure that is as helpful as only > having the attribute around when order is defined. > > This got me thinking. The attribute should be a class-only attribute > (i.e. a property on type) like __name__ is. That would keep it from > showing up on instance lookups and on every vars(obj). > > The big thing for me, though, is that optionally having the attribute > accommodates Python implementations that can't (or don't) preserve > definition order. They don't have to worry about adding a dummy > attribute to every class just for the sake of complying with the spec. > It also means the implementation for dynamic types like Nick > mentioned don't necesarily need to be touched. The more I think about > it the more the optionally set attribute approach is the way to go. My experience with maybe set, maybe not attributes for modules is that they're *always* a PITA to deal with and just not worth the hassle. Better to just expose it as a read-only attribute on class objects (that isn't accessible through instances) and set it to None on construction if order info is not available. This will also block accidental inheritance of the attribute from an ordered base class. A new keyword-only argument for the dynamic type construction APIs in the "types" module would also be appropriate, and would allow the creation of ordered classes even if the implementation used unordered namespaces for class syntax. Cheers, Nick. > > -eric > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From ben+python at benfinney.id.au Fri Jun 28 03:25:00 2013 From: ben+python at benfinney.id.au (Ben Finney) Date: Fri, 28 Jun 2013 11:25:00 +1000 Subject: [Python-Dev] Reminder: an oft-forgotten rule about docstring formatting (PEP 257) References: <20130627102143.65125bc3@pitrou.net> Message-ID: <7wppv7kowj.fsf@benfinney.id.au> Antoine Pitrou writes: > I don't always find it easy to summarize a function in one line. Difficulty in coming up with an accurate one-line summary of the intent of a function is a code smell: the function probably needs to be simplified so it is easier to describe accurately. This is, IMO, one of the primary benefits of having a convention of expecting a one-line summary for every function (and module and class). -- \ ?Courage is not the absence of fear, but the decision that | `\ something else is more important than fear.? ?Ambrose Redmoon | _o__) | Ben Finney From stephen at xemacs.org Fri Jun 28 11:14:41 2013 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Fri, 28 Jun 2013 18:14:41 +0900 Subject: [Python-Dev] Reminder: an oft-forgotten rule about docstring formatting (PEP 257) In-Reply-To: References: <51CC5599.9010305@hastings.org> Message-ID: <87li5uy4u6.fsf@uwakimon.sk.tsukuba.ac.jp> Terry Reedy writes: > Ok, I won't add them when a function's name actually makes what it does > obvious. But when I have to spend at least a few minutes reading the > body to make sense of it, I will try to add the summary line that I wish > had been there already. +1 That's a *great* rule-of-thumb! From pynix.wang at gmail.com Fri Jun 28 16:31:29 2013 From: pynix.wang at gmail.com (Pynix Wang) Date: Fri, 28 Jun 2013 22:31:29 +0800 Subject: [Python-Dev] add new lambda syntax Message-ID: I want use coffeescript function syntax to write python lambda expression so I modified the Grammar file. ``` atom: ('(' [yield_expr|testlist_comp|vararglist] ')' | '[' [testlist_comp] ']' | '{' [dictorsetmaker] '}' | NAME | NUMBER | STRING+ | '...' | 'None' | 'True' | 'False') trailer: '(' [arglist] ')' | '[' subscriptlist ']' | '.' NAME | '->' text ``` but when I write ``` (x,y=1)->x+y ``` the parser doesn't go into vararglist. any help? -------------- next part -------------- An HTML attachment was scrubbed... URL: From status at bugs.python.org Fri Jun 28 18:07:36 2013 From: status at bugs.python.org (Python tracker) Date: Fri, 28 Jun 2013 18:07:36 +0200 (CEST) Subject: [Python-Dev] Summary of Python tracker Issues Message-ID: <20130628160736.1193F56A4C@psf.upfronthosting.co.za> ACTIVITY SUMMARY (2013-06-21 - 2013-06-28) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 4069 (+12) closed 26027 (+34) total 30096 (+46) Open issues with patches: 1804 Issues opened (40) ================== #7732: imp.find_module crashes Python if there exists a directory nam http://bugs.python.org/issue7732 reopened by haypo #18135: _ssl module: possible integer overflow for very long strings ( http://bugs.python.org/issue18135 reopened by pitrou #18267: xmlrpc.client documentation multicall example missleading for http://bugs.python.org/issue18267 reopened by ber #18279: IDLE Unit test for RstripExtension.py http://bugs.python.org/issue18279 opened by philwebster #18280: Documentation is too personalized http://bugs.python.org/issue18280 opened by serhiy.storchaka #18281: tarfile defines stat constants http://bugs.python.org/issue18281 opened by christian.heimes #18283: shutil.which() should support bytes http://bugs.python.org/issue18283 opened by Arfrever #18286: Python 3.3 - Slowing down computer http://bugs.python.org/issue18286 opened by jon_irenicus #18287: PyType_Ready() should sanity-check the tp_name field http://bugs.python.org/issue18287 opened by nkoep #18288: Idle 2.7: Run Module does not set __file__ http://bugs.python.org/issue18288 opened by terry.reedy #18289: python.org Interactive interpreter linked with libedit can seg http://bugs.python.org/issue18289 opened by Max.Kaye #18290: json encoder does not support JSONP/JavaScript safe escaping http://bugs.python.org/issue18290 opened by Ztane #18291: codecs.open interprets space as line ends http://bugs.python.org/issue18291 opened by wpk #18292: IDLE Improvements: Unit test for AutoExpand.py http://bugs.python.org/issue18292 opened by JayKrish #18293: ssl.wrap_socket (cert_reqs=...), getpeercert, and unvalidated http://bugs.python.org/issue18293 opened by mpb #18294: zlib module is not completly 64-bit safe http://bugs.python.org/issue18294 opened by haypo #18295: Possible integer overflow in PyCode_New() http://bugs.python.org/issue18295 opened by haypo #18296: test_os.test_trailers() is failing on AMD64 FreeBSD 9.0 dtrace http://bugs.python.org/issue18296 opened by haypo #18299: Change script_helper to use universal_newlines=True in _assert http://bugs.python.org/issue18299 opened by r.david.murray #18300: script_helper._asert_python should set TERM='' by default. http://bugs.python.org/issue18300 opened by r.david.murray #18301: In itertools.chain.from_iterable() there is no cls argument http://bugs.python.org/issue18301 opened by py.user #18302: test_multiprocessing: test.support.import_module() does not ig http://bugs.python.org/issue18302 opened by haypo #18304: ElementTree gets awkward to use if there is an xmlns http://bugs.python.org/issue18304 opened by brycenesbitt #18305: [patch] Fast sum() for non-numbers http://bugs.python.org/issue18305 opened by Sergey #18306: os.stat gives exception for Windows junctions in v3.3 http://bugs.python.org/issue18306 opened by John.Jefferies #18307: Relative path in co_filename for zipped modules http://bugs.python.org/issue18307 opened by vmurashev #18308: checkRecvmsgAddress wrong in test_socket.py (AIX failures) http://bugs.python.org/issue18308 opened by David.Edelsohn #18309: Make python slightly more relocatable http://bugs.python.org/issue18309 opened by mathias #18312: "make distclean" deletes files under .hg directory http://bugs.python.org/issue18312 opened by eric.smith #18313: In itertools recipes repeatfunc() defines a non-keyword argume http://bugs.python.org/issue18313 opened by py.user #18314: Have os.unlink remove junction points http://bugs.python.org/issue18314 opened by Kim.Gr??sman #18315: bufsize parameter not documented in 2.7.5 http://bugs.python.org/issue18315 opened by Terrel.Shumway #18316: Idle 2.7: update to simplify cross-version patches http://bugs.python.org/issue18316 opened by terry.reedy #18317: gettext: DoS via crafted Plural-Forms http://bugs.python.org/issue18317 opened by jwilk #18318: Idle: stop depending on console output http://bugs.python.org/issue18318 opened by terry.reedy #18319: gettext() cannot find translations with plural forms http://bugs.python.org/issue18319 opened by jwilk #18320: idle, pydoc search python on wrong path at make altinstall wit http://bugs.python.org/issue18320 opened by cosmicduck #18321: Multivolume support in tarfile module http://bugs.python.org/issue18321 opened by edulix #18322: test_stat nits http://bugs.python.org/issue18322 opened by pitrou #18323: 'GzipFile' object has no attribute 'extrastart' http://bugs.python.org/issue18323 opened by Laurent.Gautier Most recent 15 issues with no replies (15) ========================================== #18320: idle, pydoc search python on wrong path at make altinstall wit http://bugs.python.org/issue18320 #18319: gettext() cannot find translations with plural forms http://bugs.python.org/issue18319 #18318: Idle: stop depending on console output http://bugs.python.org/issue18318 #18307: Relative path in co_filename for zipped modules http://bugs.python.org/issue18307 #18305: [patch] Fast sum() for non-numbers http://bugs.python.org/issue18305 #18304: ElementTree gets awkward to use if there is an xmlns http://bugs.python.org/issue18304 #18302: test_multiprocessing: test.support.import_module() does not ig http://bugs.python.org/issue18302 #18300: script_helper._asert_python should set TERM='' by default. http://bugs.python.org/issue18300 #18299: Change script_helper to use universal_newlines=True in _assert http://bugs.python.org/issue18299 #18292: IDLE Improvements: Unit test for AutoExpand.py http://bugs.python.org/issue18292 #18273: Simplify calling and discovery of json test package http://bugs.python.org/issue18273 #18266: Fix test discovery for test_largefile.py http://bugs.python.org/issue18266 #18258: Fix test discovery for test_codecmaps*.py http://bugs.python.org/issue18258 #18254: Accessing attr dict at definition time of a class from a metac http://bugs.python.org/issue18254 #18243: mktime_tz documentation out-of-date http://bugs.python.org/issue18243 Most recent 15 issues waiting for review (15) ============================================= #18322: test_stat nits http://bugs.python.org/issue18322 #18321: Multivolume support in tarfile module http://bugs.python.org/issue18321 #18317: gettext: DoS via crafted Plural-Forms http://bugs.python.org/issue18317 #18315: bufsize parameter not documented in 2.7.5 http://bugs.python.org/issue18315 #18312: "make distclean" deletes files under .hg directory http://bugs.python.org/issue18312 #18309: Make python slightly more relocatable http://bugs.python.org/issue18309 #18305: [patch] Fast sum() for non-numbers http://bugs.python.org/issue18305 #18301: In itertools.chain.from_iterable() there is no cls argument http://bugs.python.org/issue18301 #18294: zlib module is not completly 64-bit safe http://bugs.python.org/issue18294 #18287: PyType_Ready() should sanity-check the tp_name field http://bugs.python.org/issue18287 #18283: shutil.which() should support bytes http://bugs.python.org/issue18283 #18279: IDLE Unit test for RstripExtension.py http://bugs.python.org/issue18279 #18273: Simplify calling and discovery of json test package http://bugs.python.org/issue18273 #18266: Fix test discovery for test_largefile.py http://bugs.python.org/issue18266 #18258: Fix test discovery for test_codecmaps*.py http://bugs.python.org/issue18258 Top 10 most discussed issues (10) ================================= #16487: Allow ssl certificates to be specified from memory rather than http://bugs.python.org/issue16487 17 msgs #17206: Py_XDECREF() expands its argument multiple times http://bugs.python.org/issue17206 15 msgs #18234: Unicodedata module should provide access to codepoint aliases http://bugs.python.org/issue18234 15 msgs #11016: Re-implementation of the stat module in C http://bugs.python.org/issue11016 14 msgs #18236: str.isspace should use the Unicode White_Space property http://bugs.python.org/issue18236 11 msgs #18264: enum.IntEnum is not compatible with JSON serialisation http://bugs.python.org/issue18264 10 msgs #17621: Create a lazy import loader mixin http://bugs.python.org/issue17621 9 msgs #9566: Compilation warnings under x64 Windows http://bugs.python.org/issue9566 8 msgs #18135: _ssl module: possible integer overflow for very long strings ( http://bugs.python.org/issue18135 8 msgs #18081: test_logging failure in WarningsTest on buildbots http://bugs.python.org/issue18081 7 msgs Issues closed (34) ================== #1542: Ship 32 and 64bit libs with MSI installer http://bugs.python.org/issue1542 closed by christian.heimes #1953: Compact int and float freelists http://bugs.python.org/issue1953 closed by christian.heimes #4610: Unicode case mappings are incorrect http://bugs.python.org/issue4610 closed by belopolsky #9369: const char* for PyObject_CallMethod and PyObject_CallFunction http://bugs.python.org/issue9369 closed by serhiy.storchaka #11390: doctest: add cmdline parameters http://bugs.python.org/issue11390 closed by r.david.murray #11454: email.message import time http://bugs.python.org/issue11454 closed by r.david.murray #13483: Use VirtualAlloc to allocate memory arenas http://bugs.python.org/issue13483 closed by loewis #14264: Comparison bug in distutils2.version http://bugs.python.org/issue14264 closed by eric.araujo #14360: email.encoders.encode_quopri doesn't work with python 3.2 http://bugs.python.org/issue14360 closed by r.david.murray #15223: datetime instances lack __module__ attribute http://bugs.python.org/issue15223 closed by belopolsky #15818: multiprocessing documentation of Process.exitcode http://bugs.python.org/issue15818 closed by sbt #16114: incorrect path in subprocess.Popen() FileNotFoundError message http://bugs.python.org/issue16114 closed by gregory.p.smith #17037: Use a test.support helper to wrap the PEP 399 boilerplate http://bugs.python.org/issue17037 closed by eric.snow #17985: multiprocessing Queue.qsize() and Queue.empty() with different http://bugs.python.org/issue17985 closed by sbt #18137: format(float, str): integer overflow for huge precision http://bugs.python.org/issue18137 closed by haypo #18143: ssl.get_default_verify_paths() http://bugs.python.org/issue18143 closed by christian.heimes #18164: Embedding Python doc incorrectly refers to LINKFORSHARED http://bugs.python.org/issue18164 closed by ned.deily #18179: SMTP.local_hostname is undocumented http://bugs.python.org/issue18179 closed by r.david.murray #18184: Add range check for %c in PyUnicode_FromFormat http://bugs.python.org/issue18184 closed by serhiy.storchaka #18220: Expand itertools.islice docstring signature to 2 lines http://bugs.python.org/issue18220 closed by python-dev #18222: os.path.abspath should accept multiple path parts and join the http://bugs.python.org/issue18222 closed by terry.reedy #18246: tkinter.Text() add a newline to the content - bug? http://bugs.python.org/issue18246 closed by terry.reedy #18260: configparser: TypeError occurs when handling errors in files w http://bugs.python.org/issue18260 closed by lukasz.langa #18277: Queue is empty right after put from the same process/thread http://bugs.python.org/issue18277 closed by sbt #18278: Clarify the loader requirements for importlib.machinery.FileFi http://bugs.python.org/issue18278 closed by brett.cannon #18282: Ugly behavior of binary and unicode handling on reading unknow http://bugs.python.org/issue18282 closed by r.david.murray #18284: Doc make html: KeyError: 'howto/advocacy' http://bugs.python.org/issue18284 closed by r.david.murray #18285: In itertools.product() add argument repeat to the docstring http://bugs.python.org/issue18285 closed by python-dev #18297: In random.sample() correct the ValueError message for negative http://bugs.python.org/issue18297 closed by mark.dickinson #18298: pythonw.exe fails with redirected stderr http://bugs.python.org/issue18298 closed by amaury.forgeotdarc #18303: json.dumps() claims numpy.ndarray and numpy.bool_ are not seri http://bugs.python.org/issue18303 closed by ethan.furman #18310: itertools.tee() can't accept keyword arguments http://bugs.python.org/issue18310 closed by rhettinger #18311: Typo in SSL documentation http://bugs.python.org/issue18311 closed by r.david.murray #1040439: Missing documentation on how to link with libpython http://bugs.python.org/issue1040439 closed by ned.deily From amauryfa at gmail.com Fri Jun 28 19:01:47 2013 From: amauryfa at gmail.com (Amaury Forgeot d'Arc) Date: Fri, 28 Jun 2013 19:01:47 +0200 Subject: [Python-Dev] add new lambda syntax In-Reply-To: References: Message-ID: 2013/6/28 Pynix Wang > I want use coffeescript function syntax to write python lambda expression > so I modified the Grammar file. > > ``` > atom: ('(' [yield_expr|testlist_comp|vararglist] ')' | > '[' [testlist_comp] ']' | > '{' [dictorsetmaker] '}' | > NAME | NUMBER | STRING+ | '...' | 'None' | 'True' | 'False') > trailer: '(' [arglist] ')' | '[' subscriptlist ']' | '.' NAME | '->' text > ``` > > but when I write > ``` > (x,y=1)->x+y > ``` > the parser doesn't go into vararglist. > This grammar is not LL(1) anymore (it's probably LALR now) when seeing "x", it has the choice between testlist_comp and vararglist, and the first one is picked. Python's parser generator only supports LL(1) grammars. -- Amaury Forgeot d'Arc -------------- next part -------------- An HTML attachment was scrubbed... URL: From thomas at python.org Fri Jun 28 19:39:02 2013 From: thomas at python.org (Thomas Wouters) Date: Fri, 28 Jun 2013 19:39:02 +0200 Subject: [Python-Dev] add new lambda syntax In-Reply-To: References: Message-ID: On Fri, Jun 28, 2013 at 7:01 PM, Amaury Forgeot d'Arc wrote: > 2013/6/28 Pynix Wang > >> I want use coffeescript function syntax to write python lambda expression >> so I modified the Grammar file. >> >> ``` >> atom: ('(' [yield_expr|testlist_comp|vararglist] ')' | >> '[' [testlist_comp] ']' | >> '{' [dictorsetmaker] '}' | >> NAME | NUMBER | STRING+ | '...' | 'None' | 'True' | 'False') >> trailer: '(' [arglist] ')' | '[' subscriptlist ']' | '.' NAME | '->' text >> ``` >> >> but when I write >> ``` >> (x,y=1)->x+y >> ``` >> the parser doesn't go into vararglist. >> > > This grammar is not LL(1) anymore (it's probably LALR now) > when seeing "x", it has the choice between testlist_comp and vararglist, > and the first one is picked. > Python's parser generator only supports LL(1) grammars. > Indeed. You may be able to make this work, but you'd have to fold the bits of vararglist you need into testlist_comp, then reject invalid syntax that matches the grammar (like "(x=expr for ...)" or "((x+1)=expr)" or "(*foo=1)") in the compiler. Something like (untested): testlist_comp: (test|star_expr) ['=' test]( comp_for | (',' (test|star_expr) ['=' test])* [','] ) -- Thomas Wouters Hi! I'm an email virus! Think twice before sending your email to help me spread! -------------- next part -------------- An HTML attachment was scrubbed... URL: From jimjjewett at gmail.com Fri Jun 28 22:07:50 2013 From: jimjjewett at gmail.com (Jim J. Jewett) Date: Fri, 28 Jun 2013 13:07:50 -0700 (PDT) Subject: [Python-Dev] backported Enum In-Reply-To: <20130618230403.4cf9e657@anarchist> Message-ID: <51cded16.4267320a.7d7d.0586@mx.google.com> (On June 19, 2013) Barry Warsaw wrote about porting mailman from flufl.enum to the stdlib.enum: > Switching from call syntax to getitem syntax for looking up an > enum member by name, e.g. > - delivery_mode = DeliveryMode(data['delivery_mode']) > + delivery_mode = DeliveryMode[data['delivery_mode']] > Switching from getitem syntax to call syntax for looking up an > enum member by value, e.g. > - return self._enum[value] > + return self._enum(value) > Interesting that these two were exactly opposite from flufl.enum. Is there a reason why these were reversed? I can sort of convince myself that it makes sense because dicts work better with strings than with ints, but ... it seems like such a minor win that I'm not sure it is worth backwards incompatibility. (Of course, I also don't know how much use stdlib.enum has already gotten with the current syntax.) > Switching from int() to .value to get the integer value of an > enum member, e.g. > - return (member.list_id, member.address.email, int(member.role)) > + return (member.list_id, member.address.email, member.role.value) Is just this a style preference? Using a .value attribute certainly makes sense, but I don't see it mentioned in the PEP as even optional, let alone recommended. If you care that the value be specifically an int (as opposed to any object), then a int constructor may be better. > [Some additional changes that mean there will be *some* changes, > which does reduce the pressure for backwards compatibility.] ... > An unexpected difference is that failing name lookups raise a > KeyError instead of a ValueError. I could understand either, as well as AttributeError, since the instance that would represent that value isn't a class attribute. Looking at Enum creation, I think ValueError would be better than TypeError for complaints about duplicate names. Was TypeError chosen because it should only happen during setup? I would also not be shocked if some people expect failed value lookups to raise an IndexError, though I expect they would adapt if they get something else that makes sense. Would it be wrong to create an EnumError that subclasses (ValueError, KeyError, AttributeError) and to raise that subclass from everything but _StealthProperty and _get_mixins? -jJ -- If there are still threading problems with my replies, please email me with details, so that I can try to resolve them. -jJ From barry at python.org Fri Jun 28 22:20:27 2013 From: barry at python.org (Barry Warsaw) Date: Fri, 28 Jun 2013 16:20:27 -0400 Subject: [Python-Dev] backported Enum In-Reply-To: <51cded16.4267320a.7d7d.0586@mx.google.com> References: <20130618230403.4cf9e657@anarchist> <51cded16.4267320a.7d7d.0586@mx.google.com> Message-ID: <20130628162027.44a0607b@anarchist> On Jun 28, 2013, at 01:07 PM, Jim J. Jewett wrote: >> Switching from getitem syntax to call syntax for looking up an >> enum member by value, e.g. > >> - return self._enum[value] >> + return self._enum(value) > >> Interesting that these two were exactly opposite from flufl.enum. > >Is there a reason why these were reversed? The individual decisions made (begrudging) sense at the time, although I don't think it was noticed that the combined effect was to switch the meanings. Call syntax makes sense for converting a value to an enum because that mirrors calling built-in types, e.g. int('7'). Getitem syntax makes sense for lookup-by-name. >> Switching from int() to .value to get the integer value of an >> enum member, e.g. > >> - return (member.list_id, member.address.email, int(member.role)) >> + return (member.list_id, member.address.email, member.role.value) > >Is just this a style preference? No. I'm not using IntEnums and regular enums don't have an __int__, which makes sense because their values can be anything. .value is the way to get at the value. (Note that even though I don't use IntEnums, my values are all integers. It's just not worth it to change all my imports.) -Barry From ethan at stoneleaf.us Fri Jun 28 22:30:14 2013 From: ethan at stoneleaf.us (Ethan Furman) Date: Fri, 28 Jun 2013 13:30:14 -0700 Subject: [Python-Dev] backported Enum In-Reply-To: <51cded16.4267320a.7d7d.0586@mx.google.com> References: <51cded16.4267320a.7d7d.0586@mx.google.com> Message-ID: <51CDF256.30104@stoneleaf.us> On 06/28/2013 01:07 PM, Jim J. Jewett wrote: > (On June 19, 2013) Barry Warsaw wrote about porting mailman from > flufl.enum to the stdlib.enum: >> >> Switching from call syntax to getitem syntax for looking up an >> enum member by name, e.g. > >> - delivery_mode = DeliveryMode(data['delivery_mode']) >> + delivery_mode = DeliveryMode[data['delivery_mode']] >> >> Switching from getitem syntax to call syntax for looking up an >> enum member by value, e.g. >> >> - return self._enum[value] >> + return self._enum(value) >> >> Interesting that these two were exactly opposite from flufl.enum. > > Is there a reason why these were reversed? Originally the call syntax was used for both value and name lookup. Various folks were unhappy with that arrangement, and since the use-case that I was thinking of at the time was getting enum members back from databases, etc., I went with EnumClass(value); I still wanted to be able to use name lookups, so lobbied for getitem syntax for that. Nobody noticed this was the opposite of flufl.enum. Oh, I say "I", and it is certainly my reasons, but I no longer remember if was me that initially proposed those specific ideas, and there were certainly many others that agreed. >> Switching from int() to .value to get the integer value of an >> enum member, e.g. >> >> - return (member.list_id, member.address.email, int(member.role)) >> + return (member.list_id, member.address.email, member.role.value) > > Is just this a style preference? Nope. If you want the exact value, accessing `.value` is the way to get it. > Using a .value attribute certainly makes sense, but I don't see it > mentioned in the PEP as even optional, let alone recommended. I'll look again at the PEP and the docs and make sure that point is clear. > If you care that the value be specifically an int (as opposed to any > object), then a int constructor may be better. Not entirely sure what you mean by this? Had it been me, I would have subclassed Enum (as, say, FluflEnum) and added `__int__` to it so that those lines would have remained the same. >> [Some additional changes that mean there will be *some* changes, >> which does reduce the pressure for backwards compatibility.] ... >> >> >> An unexpected difference is that failing name lookups raise a >> KeyError instead of a ValueError. > > I could understand either, as well as AttributeError, since the > instance that would represent that value isn't a class attribute. > > Looking at Enum creation, I think ValueError would be better than > TypeError for complaints about duplicate names. Was TypeError > chosen because it should only happen during setup? Yes. That particular error can only happen during setup. > I would also not be shocked if some people expect failed value > lookups to raise an IndexError, though I expect they would > adapt if they get something else that makes sense. > > Would it be wrong to create an EnumError that subclasses > (ValueError, KeyError, AttributeError) and to raise that > subclass from everything but _StealthProperty and _get_mixins? Wouldn't bother me; I'm not sure what the bar is for adding new exceptions, though. -- ~Ethan~ From guido at python.org Fri Jun 28 23:11:59 2013 From: guido at python.org (Guido van Rossum) Date: Fri, 28 Jun 2013 14:11:59 -0700 Subject: [Python-Dev] backported Enum In-Reply-To: <51CDF256.30104@stoneleaf.us> References: <51cded16.4267320a.7d7d.0586@mx.google.com> <51CDF256.30104@stoneleaf.us> Message-ID: On Fri, Jun 28, 2013 at 1:30 PM, Ethan Furman wrote: > On 06/28/2013 01:07 PM, Jim J. Jewett wrote: >> (On June 19, 2013) Barry Warsaw wrote about porting mailman from >>> Switching from getitem syntax to call syntax for looking up an >>> enum member by value, e.g. >>> >>> - return self._enum[value] >>> + return self._enum(value) >>> >>> Interesting that these two were exactly opposite from flufl.enum. >> Is there a reason why these were reversed? > Originally the call syntax was used for both value and name lookup. Various > folks were unhappy with that arrangement, and since the use-case that I was > thinking of at the time was getting enum members back from databases, etc., > I went with EnumClass(value); I still wanted to be able to use name lookups, > so lobbied for getitem syntax for that. Nobody noticed this was the > opposite of flufl.enum. > > Oh, I say "I", and it is certainly my reasons, but I no longer remember if > was me that initially proposed those specific ideas, and there were > certainly many others that agreed. I also have a strong preference for the new way of doing this. In Python, type(arg) is usually a constructor or conversion/cast, and that maps reasonably well on what we're doing here with enums -- taking a value and turning it into the corresponding enum (which still shares some properties with that value, and more if it's an IntEnum). Also, this is a no-op (technically, the identity function) if the arg is already a member of that enum. On the other hand, Color['red'] feels more like a lookup -- this is not so much rhyming with the behavior of other classes (classes are rarely mappings) but still the mnemonic tip "foo[bar] is a lookup" should help in remembering the meaning. I have no idea why it was the other way around in flufl.enum, but admittedly neither of these rules are absolute, and it's possible that flufl.enum just evolved that way without conscious decisions, or that it came from taking a different perspective. -- --Guido van Rossum (python.org/~guido) From barry at python.org Fri Jun 28 23:18:30 2013 From: barry at python.org (Barry Warsaw) Date: Fri, 28 Jun 2013 17:18:30 -0400 Subject: [Python-Dev] backported Enum In-Reply-To: References: <51cded16.4267320a.7d7d.0586@mx.google.com> <51CDF256.30104@stoneleaf.us> Message-ID: <20130628171830.4640d897@anarchist> On Jun 28, 2013, at 02:11 PM, Guido van Rossum wrote: >I have no idea why it was the other way around in flufl.enum, but >admittedly neither of these rules are absolute, and it's possible that >flufl.enum just evolved that way without conscious decisions, or that >it came from taking a different perspective. The latter, but I think PEP 435 made the right choices, at least for the stdlib version. It doesn't bother me that I had to switch them when porting. -Barry From tjreedy at udel.edu Sat Jun 29 00:45:06 2013 From: tjreedy at udel.edu (Terry Reedy) Date: Fri, 28 Jun 2013 18:45:06 -0400 Subject: [Python-Dev] Reminder: an oft-forgotten rule about docstring formatting (PEP 257) In-Reply-To: References: Message-ID: On 6/26/2013 9:56 PM, Guido van Rossum wrote: > PEP 257 says this on the formatting of multi-line docstrings: > > """ > Multi-line docstrings consist of a summary line just like a one-line > docstring, followed by a blank line, followed by a more elaborate > description. fileinput has docstrings like """ Return the file number of the current file. When no file is currently opened, returns -1. """ and, perhaps neater, """ Return the name of the file currently being read. Before the first line has been read, returns None. """ From the above, I presume these should become """ Return the file number of the current file. When no file is currently opened, returns -1. """ and """ Return the name of the file currently being read. Before the first line has been read, returns None. """ -- Terry Jan Reedy From ncoghlan at gmail.com Sat Jun 29 12:21:36 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 29 Jun 2013 20:21:36 +1000 Subject: [Python-Dev] backported Enum In-Reply-To: <51CDF256.30104@stoneleaf.us> References: <51cded16.4267320a.7d7d.0586@mx.google.com> <51CDF256.30104@stoneleaf.us> Message-ID: On 29 June 2013 06:30, Ethan Furman wrote: >> I would also not be shocked if some people expect failed value >> lookups to raise an IndexError, though I expect they would >> adapt if they get something else that makes sense. >> >> Would it be wrong to create an EnumError that subclasses >> (ValueError, KeyError, AttributeError) and to raise that >> subclass from everything but _StealthProperty and _get_mixins? > > > Wouldn't bother me; I'm not sure what the bar is for adding new exceptions, > though. I'd actually do something a bit more complex, but also cleaner from a type system perspective: class EnumError(Exception): pass class EnumValueError(EnumError, ValueError): pass class EnumAttributeError(EnumError, AttributeError): pass class EnumKeyError(EnumError, KeyError): pass However, it's probably not necessary. The value lookup API should just document clearly which exception it throws. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From pynix.wang at gmail.com Sat Jun 29 09:05:28 2013 From: pynix.wang at gmail.com (Pynix Wang) Date: Sat, 29 Jun 2013 15:05:28 +0800 Subject: [Python-Dev] add new lambda syntax In-Reply-To: References: Message-ID: http://aruld.info/lambda-expressions-in-java-8-adopts-c-style-syntax/ Lambda expressions in Java 8 adopts C# style syntax. On Sat, Jun 29, 2013 at 1:39 AM, Thomas Wouters wrote: > > > > On Fri, Jun 28, 2013 at 7:01 PM, Amaury Forgeot d'Arc wrote: > >> 2013/6/28 Pynix Wang >> >>> I want use coffeescript function syntax to write python lambda >>> expression so I modified the Grammar file. >>> >>> ``` >>> atom: ('(' [yield_expr|testlist_comp|vararglist] ')' | >>> '[' [testlist_comp] ']' | >>> '{' [dictorsetmaker] '}' | >>> NAME | NUMBER | STRING+ | '...' | 'None' | 'True' | 'False') >>> trailer: '(' [arglist] ')' | '[' subscriptlist ']' | '.' NAME | '->' text >>> ``` >>> >>> but when I write >>> ``` >>> (x,y=1)->x+y >>> ``` >>> the parser doesn't go into vararglist. >>> >> >> This grammar is not LL(1) anymore (it's probably LALR now) >> when seeing "x", it has the choice between testlist_comp and vararglist, >> and the first one is picked. >> Python's parser generator only supports LL(1) grammars. >> > > Indeed. You may be able to make this work, but you'd have to fold the bits > of vararglist you need into testlist_comp, then reject invalid syntax that > matches the grammar (like "(x=expr for ...)" or "((x+1)=expr)" or > "(*foo=1)") in the compiler. Something like (untested): > > testlist_comp: (test|star_expr) ['=' test]( comp_for | (',' > (test|star_expr) ['=' test])* [','] ) > > -- > Thomas Wouters > > Hi! I'm an email virus! Think twice before sending your email to help me > spread! > -------------- next part -------------- An HTML attachment was scrubbed... URL: From nadeem.vawda at gmail.com Sat Jun 29 17:16:46 2013 From: nadeem.vawda at gmail.com (Nadeem Vawda) Date: Sat, 29 Jun 2013 17:16:46 +0200 Subject: [Python-Dev] PyArg_ParseTupe(): parse unsigned integer and check for overflow In-Reply-To: References: Message-ID: On Thu, Jun 27, 2013 at 12:07 AM, Victor Stinner wrote: > I would to parse an integer in [0; UINT_MAX] to fix the zlib module on > 64-bit system: > http://bugs.python.org/issue18294 > > How should I implement that? Use "O" format and then use > PyLong_Check(), PyLong_AsLong(), and check value <= UINT_MAX? > I ran into the same problem in the _lzma module. My solution was to define a custom converter that does an explicit check before returning the value (see http://hg.python.org/cpython/file/default/Modules/_lzmamodule.c#l134). On Thu, Jun 27, 2013 at 12:26 AM, Guido van Rossum wrote: > > I would to parse an integer in [0; UINT_MAX] to fix the zlib module on > > 64-bit system: > > http://bugs.python.org/issue18294 > > > > How should I implement that? Use "O" format and then use > > PyLong_Check(), PyLong_AsLong(), and check value <= UINT_MAX? > > Why can't you use the K format? It won't reject out-of-range values, > but it will convert them to in-range so there aren't any attacks > possible based on bypassing the range check. I'm probably > misunderstanding something -- I don't completely understand that bug > report. :-( The point is not to protect against deliberate attacks, but rather to fail loudly (instead of silently) when the caller provides an input that the underlying C library cannot handle. - Nadeem -------------- next part -------------- An HTML attachment was scrubbed... URL: From storchaka at gmail.com Sat Jun 29 18:24:11 2013 From: storchaka at gmail.com (Serhiy Storchaka) Date: Sat, 29 Jun 2013 19:24:11 +0300 Subject: [Python-Dev] PyArg_ParseTupe(): parse unsigned integer and check for overflow In-Reply-To: References: Message-ID: 29.06.13 18:16, Nadeem Vawda ???????(??): > I ran into the same problem in the _lzma module. My solution was to > define a custom converter that does an explicit check before returning > the value (see > http://hg.python.org/cpython/file/default/Modules/_lzmamodule.c#l134). Definitely Argument Clinic should use converters for unsigned integers without wraparound by default. From tamir.fri at gmail.com Sat Jun 29 17:59:13 2013 From: tamir.fri at gmail.com (Tamir Friedman) Date: Sat, 29 Jun 2013 18:59:13 +0300 Subject: [Python-Dev] winsound.c fix to support python3 Message-ID: Hello, My name is Tamir Friedman, and I suggest to fix a bug in PlaySound in winsound library. It's doesn't support the SND_MEMORY feature because its accepts only "str" and rejects "bytes" type. therefore i include the fixed source file: OLD: ---------------------------------------------------------------------------- static PyObject * sound_playsound(PyObject *s, PyObject *args) { wchar_t *wsound; int flags; int ok; if (PyArg_ParseTuple(args, "Zi:PlaySound", &wsound, &flags)) { if (flags & SND_ASYNC && flags & SND_MEMORY) { /* Sidestep reference counting headache; unfortunately this also prevent SND_LOOP from memory. */ PyErr_SetString(PyExc_RuntimeError, "Cannot play asynchronously from memory"); return NULL; } Py_BEGIN_ALLOW_THREADS ok = PlaySoundW(wsound, NULL, flags); Py_END_ALLOW_THREADS if (!ok) { PyErr_SetString(PyExc_RuntimeError, "Failed to play sound"); return NULL; } Py_INCREF(Py_None); return Py_None; } return NULL; } ---------------------------------------------------------------------------- NEW: ---------------------------------------------------------------------------- static PyObject * sound_playsound(PyObject *s, PyObject *args) { wchar_t *wsound; int flags; int ok; if (PyArg_ParseTuple(args, "z*i:PlaySound", &wsound, &flags)) { if (flags & SND_ASYNC && flags & SND_MEMORY) { /* Sidestep reference counting headache; unfortunately this also prevent SND_LOOP from memory. */ PyErr_SetString(PyExc_RuntimeError, "Cannot play asynchronously from memory"); return NULL; } Py_BEGIN_ALLOW_THREADS ok = PlaySoundW(wsound, NULL, flags); Py_END_ALLOW_THREADS if (!ok) { PyErr_SetString(PyExc_RuntimeError, "Failed to play sound"); return NULL; } Py_INCREF(Py_None); return Py_None; } return NULL; } ---------------------------------------------------------------------------- -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: OLD_winsound_OLD.c Type: text/x-csrc Size: 5581 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: winsound.c Type: text/x-csrc Size: 5637 bytes Desc: not available URL: From rdmurray at bitdance.com Sat Jun 29 21:29:09 2013 From: rdmurray at bitdance.com (R. David Murray) Date: Sat, 29 Jun 2013 15:29:09 -0400 Subject: [Python-Dev] =?utf-8?q?=5BPython-checkins=5D_cpython=3A_Fix_memor?= =?utf-8?q?y_leak_in_pyexpat_PyUnknownEncodingHandler?= In-Reply-To: <3bjNyf4jvKz7Llt@mail.python.org> References: <3bjNyf4jvKz7Llt@mail.python.org> Message-ID: <20130629192910.333AD2500C5@webabinitio.net> I don't know if this is the commit at fault or not, but we are seeing segfaults in test_xml_etree on the buildbots now. --David On Sat, 29 Jun 2013 20:43:22 +0200, christian.heimes wrote: > http://hg.python.org/cpython/rev/bd0834b59828 > changeset: 84375:bd0834b59828 > user: Christian Heimes > date: Sat Jun 29 20:43:13 2013 +0200 > summary: > Fix memory leak in pyexpat PyUnknownEncodingHandler > CID 1040367 (#1 of 1): Resource leak (RESOURCE_LEAK) > leaked_storage: Variable u going out of scope leaks the storage it points to. > > files: > Modules/pyexpat.c | 4 +++- > 1 files changed, 3 insertions(+), 1 deletions(-) > > > diff --git a/Modules/pyexpat.c b/Modules/pyexpat.c > --- a/Modules/pyexpat.c > +++ b/Modules/pyexpat.c > @@ -1128,8 +1128,10 @@ > } > > u = PyUnicode_Decode((char*) template_buffer, 256, name, "replace"); > - if (u == NULL || PyUnicode_READY(u)) > + if (u == NULL || PyUnicode_READY(u)) { > + Py_DECREF(u); > return XML_STATUS_ERROR; > + } > > if (PyUnicode_GET_LENGTH(u) != 256) { > Py_DECREF(u); > > -- > Repository URL: http://hg.python.org/cpython > _______________________________________________ > Python-checkins mailing list > Python-checkins at python.org > http://mail.python.org/mailman/listinfo/python-checkins From christian at python.org Sat Jun 29 21:51:19 2013 From: christian at python.org (Christian Heimes) Date: Sat, 29 Jun 2013 21:51:19 +0200 Subject: [Python-Dev] cpython: Fix memory leak in pyexpat PyUnknownEncodingHandler In-Reply-To: <20130629192910.333AD2500C5@webabinitio.net> References: <3bjNyf4jvKz7Llt@mail.python.org> <20130629192910.333AD2500C5@webabinitio.net> Message-ID: <51CF3AB7.8020308@python.org> Am 29.06.2013 21:29, schrieb R. David Murray: > I don't know if this is the commit at fault or not, but we are seeing > segfaults in test_xml_etree on the buildbots now. Yeah, it's my fault. Thanks! >> u = PyUnicode_Decode((char*) template_buffer, 256, name, "replace"); >> - if (u == NULL || PyUnicode_READY(u))wr >> + Py_DECREF(u); >> return XML_STATUS_ERROR; >> + } Py_DECREF() is wrong as u can be NULL. I have changed the code to use Py_XDECREF(). From tjreedy at udel.edu Sat Jun 29 22:20:59 2013 From: tjreedy at udel.edu (Terry Reedy) Date: Sat, 29 Jun 2013 16:20:59 -0400 Subject: [Python-Dev] winsound.c fix to support python3 In-Reply-To: References: Message-ID: On 6/29/2013 11:59 AM, Tamir Friedman wrote: > Hello, > My name is Tamir Friedman, and I suggest to fix a bug in PlaySound in > winsound library. It's doesn't support the SND_MEMORY feature because > its accepts only "str" and rejects "bytes" type. > therefore i include the fixed source file: Thank you for tracking down the source of a problem. Please make an account at bugs.python.org and open a tracker issue. A post here will get lost. On the issue, either upload a patch file or describe the change. As near as I can tell, the description for your change might be: "winsound.c, line NN in sound_playsound function, is currently if (PyArg_ParseTuple(args, "Zi:PlaySound", &wsound, &flags)) { I think 'Zi' should be changed for 'z*i' because ..." -- Terry Jan Reedy From amk at amk.ca Sun Jun 30 00:00:23 2013 From: amk at amk.ca (A.M. Kuchling) Date: Sat, 29 Jun 2013 18:00:23 -0400 Subject: [Python-Dev] Issue 4199: combining assignment with global & nonlocal Message-ID: <20130629220023.GA66048@datlandrewk.home> Issue 4199 begins with a self-explanatory comment: PEP 3104 says that the nonlocal and global statements should allow a shorthand. ("global x; x = 3" == "global x = 3") This patch implements that. Benjamin posted his patch on 2008-10-24. It got postponed to 3.2 because it was too late for 3.1, and then to 3.3 because 3.2 still fell under the language moratorium, and then was forgotten for 3.3. Georg Brandl and Jeremy Hylton raised some issues with the patch and with the PEP's specification: http://bugs.python.org/issue4199#msg100015 We've gone five years in 3.x without supporting this statement form. Is it still worth implementing it for 3.4? Or should the issue simply be closed (and maybe a note added to the PEP that "we didn't do this after all")? --amk From larry at hastings.org Sun Jun 30 00:03:25 2013 From: larry at hastings.org (Larry Hastings) Date: Sun, 30 Jun 2013 00:03:25 +0200 Subject: [Python-Dev] PyArg_ParseTupe(): parse unsigned integer and check for overflow In-Reply-To: References: Message-ID: <51CF59AD.2070301@hastings.org> On 06/29/2013 06:24 PM, Serhiy Storchaka wrote: > Definitely Argument Clinic should use converters for unsigned integers > without wraparound by default. Argument Clinic 1.0 is a thin layer over PyArg_ParseTuple. But it will support these sorts of converters, and if people have bright ideas about converters to include in-the-box that's totally fine. but-nothing-has-been-formally-accepted-yet-ly yours, //arry/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Sun Jun 30 02:15:05 2013 From: guido at python.org (Guido van Rossum) Date: Sat, 29 Jun 2013 17:15:05 -0700 Subject: [Python-Dev] Issue 4199: combining assignment with global & nonlocal In-Reply-To: <20130629220023.GA66048@datlandrewk.home> References: <20130629220023.GA66048@datlandrewk.home> Message-ID: I agree that there are problems with the idea of combining assignment and global statements, because the ambiguity of "global x, y = p, q" is just too much to handle. The only case would be if there was only a single variable, but that fails the test of easy generalization. So let's close it. --Guido On Sat, Jun 29, 2013 at 3:00 PM, A.M. Kuchling wrote: > Issue 4199 begins with a self-explanatory comment: > > PEP 3104 says that the nonlocal and global statements should > allow a shorthand. ("global x; x = 3" == "global x = 3") This > patch implements that. > > Benjamin posted his patch on 2008-10-24. It got postponed to 3.2 > because it was too late for 3.1, and then to 3.3 because 3.2 still > fell under the language moratorium, and then was forgotten for 3.3. > > Georg Brandl and Jeremy Hylton raised some issues with the patch and > with the PEP's specification: > http://bugs.python.org/issue4199#msg100015 > > We've gone five years in 3.x without supporting this statement form. > Is it still worth implementing it for 3.4? Or should the issue simply > be closed (and maybe a note added to the PEP that "we didn't do this > after all")? > > --amk > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org -- --Guido van Rossum (python.org/~guido) From benjamin at python.org Sun Jun 30 02:22:44 2013 From: benjamin at python.org (Benjamin Peterson) Date: Sat, 29 Jun 2013 17:22:44 -0700 Subject: [Python-Dev] Issue 4199: combining assignment with global & nonlocal In-Reply-To: References: <20130629220023.GA66048@datlandrewk.home> Message-ID: Agreed. The currently supported syntax (a global/nonlocal line and an assignment) is not too painful to figure out the ambiguity of the proposed syntax. 2013/6/29 Guido van Rossum : > I agree that there are problems with the idea of combining assignment > and global statements, because the ambiguity of "global x, y = p, q" > is just too much to handle. The only case would be if there was only a > single variable, but that fails the test of easy generalization. So > let's close it. > > --Guido > > On Sat, Jun 29, 2013 at 3:00 PM, A.M. Kuchling wrote: >> Issue 4199 begins with a self-explanatory comment: >> >> PEP 3104 says that the nonlocal and global statements should >> allow a shorthand. ("global x; x = 3" == "global x = 3") This >> patch implements that. >> >> Benjamin posted his patch on 2008-10-24. It got postponed to 3.2 >> because it was too late for 3.1, and then to 3.3 because 3.2 still >> fell under the language moratorium, and then was forgotten for 3.3. >> >> Georg Brandl and Jeremy Hylton raised some issues with the patch and >> with the PEP's specification: >> http://bugs.python.org/issue4199#msg100015 >> >> We've gone five years in 3.x without supporting this statement form. >> Is it still worth implementing it for 3.4? Or should the issue simply >> be closed (and maybe a note added to the PEP that "we didn't do this >> after all")? >> >> --amk >> _______________________________________________ >> Python-Dev mailing list >> Python-Dev at python.org >> http://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org > > > > -- > --Guido van Rossum (python.org/~guido) > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/benjamin%40python.org -- Regards, Benjamin